2024-03-23T23:59:17.415 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-23T23:59:17.416 DEBUG:teuthology.run:Teuthology command: teuthology --owner mchangir --verbose --name mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi --archive /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616430 --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} -- /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616430/orig.config.yaml 2024-03-23T23:59:17.470 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-23T23:59:17.556 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616430 branch: wip-mchangir-testing-main-20240318.032620 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} email: mchangir@redhat.com first_in_suite: false job_id: '7616430' kernel: client: sha1: distro flavor: default kdb: true sha1: 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-mchangir-testing-main-20240318.032620 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 subvols: create: 2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - wsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-mchangir-testing-main-20240318.032620 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 owner: mchangir priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 3550 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 sleep_before_teardown: 0 subset: 613/1024 suite: fs suite_branch: wip-mchangir-testing-main-20240318.032620 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi078.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNlgIrdQHTUm2YYUyP8OOqEpcJBHk0sJPKor9xkGjd7G77koK5yCB7Xl8n2XEaD96yV9TYEuyXcol4jclHKgm1w= smithi106.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOU+yW0i5FrJ74ve1qM70CraxjpC6KFblwTr27caooJCq5unNp8xntIF0qN5C6lhlcJ38CkQZEV1HdL7ie/qIUk= smithi107.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEhgHjr9J4ZU0C3A4afjwXidRGS3mBNzk3UgRevilq1xOPxqzr44vijyI6iJRJBsQr+a5Ie4pRxQLYFT3wjMfNY= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-22_09:49:57 tube: smithi user: mchangir verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-23T23:59:17.556 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa; will attempt to use it 2024-03-23T23:59:17.557 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks 2024-03-23T23:59:17.557 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-23T23:59:17.559 INFO:teuthology.task.internal:Checking packages... 2024-03-23T23:59:17.587 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3be9a5c9bc793e11e2800a8c0c696e8b46742033' 2024-03-23T23:59:17.587 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-23T23:59:17.587 INFO:teuthology.packaging:ref: None 2024-03-23T23:59:17.587 INFO:teuthology.packaging:tag: None 2024-03-23T23:59:17.587 INFO:teuthology.packaging:branch: wip-mchangir-testing-main-20240318.032620 2024-03-23T23:59:17.588 INFO:teuthology.packaging:sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-23T23:59:17.588 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-mchangir-testing-main-20240318.032620 2024-03-23T23:59:17.808 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2145.g3be9a5c9 2024-03-23T23:59:17.810 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-23T23:59:17.815 INFO:teuthology.task.internal:no buildpackages task found 2024-03-23T23:59:17.816 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-23T23:59:17.826 INFO:teuthology.task.internal:Saving configuration 2024-03-23T23:59:17.845 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-23T23:59:17.851 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-23T23:59:17.876 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616430', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 23:54:08.246705', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNlgIrdQHTUm2YYUyP8OOqEpcJBHk0sJPKor9xkGjd7G77koK5yCB7Xl8n2XEaD96yV9TYEuyXcol4jclHKgm1w='} 2024-03-23T23:59:17.895 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi106.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616430', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 23:54:08.250442', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOU+yW0i5FrJ74ve1qM70CraxjpC6KFblwTr27caooJCq5unNp8xntIF0qN5C6lhlcJ38CkQZEV1HdL7ie/qIUk='} 2024-03-23T23:59:17.915 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616430', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 23:54:08.248785', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEhgHjr9J4ZU0C3A4afjwXidRGS3mBNzk3UgRevilq1xOPxqzr44vijyI6iJRJBsQr+a5Ie4pRxQLYFT3wjMfNY='} 2024-03-23T23:59:17.915 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-23T23:59:17.922 INFO:teuthology.task.internal:roles: ubuntu@smithi078.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-23T23:59:17.922 INFO:teuthology.task.internal:roles: ubuntu@smithi106.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-23T23:59:17.922 INFO:teuthology.task.internal:roles: ubuntu@smithi107.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-23T23:59:17.922 INFO:teuthology.run_tasks:Running task console_log... 2024-03-23T23:59:18.022 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0a908930d0>, signals=[15]) 2024-03-23T23:59:18.022 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-23T23:59:18.028 INFO:teuthology.task.internal:Opening connections... 2024-03-23T23:59:18.029 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi078.front.sepia.ceph.com 2024-03-23T23:59:18.030 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T23:59:18.124 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi106.front.sepia.ceph.com 2024-03-23T23:59:18.125 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T23:59:18.210 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi107.front.sepia.ceph.com 2024-03-23T23:59:18.211 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T23:59:18.297 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-23T23:59:18.304 DEBUG:teuthology.orchestra.run.smithi078:> uname -m 2024-03-23T23:59:18.321 INFO:teuthology.orchestra.run.smithi078.stdout:x86_64 2024-03-23T23:59:18.322 DEBUG:teuthology.orchestra.run.smithi078:> cat /etc/os-release 2024-03-23T23:59:18.378 INFO:teuthology.orchestra.run.smithi078.stdout:NAME="CentOS Stream" 2024-03-23T23:59:18.378 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION="9" 2024-03-23T23:59:18.379 INFO:teuthology.orchestra.run.smithi078.stdout:ID="centos" 2024-03-23T23:59:18.379 INFO:teuthology.orchestra.run.smithi078.stdout:ID_LIKE="rhel fedora" 2024-03-23T23:59:18.379 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_ID="9" 2024-03-23T23:59:18.379 INFO:teuthology.orchestra.run.smithi078.stdout:PLATFORM_ID="platform:el9" 2024-03-23T23:59:18.379 INFO:teuthology.orchestra.run.smithi078.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-23T23:59:18.379 INFO:teuthology.orchestra.run.smithi078.stdout:ANSI_COLOR="0;31" 2024-03-23T23:59:18.379 INFO:teuthology.orchestra.run.smithi078.stdout:LOGO="fedora-logo-icon" 2024-03-23T23:59:18.379 INFO:teuthology.orchestra.run.smithi078.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-23T23:59:18.379 INFO:teuthology.orchestra.run.smithi078.stdout:HOME_URL="https://centos.org/" 2024-03-23T23:59:18.379 INFO:teuthology.orchestra.run.smithi078.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-23T23:59:18.379 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-23T23:59:18.379 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-23T23:59:18.380 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2024-03-23T23:59:18.400 DEBUG:teuthology.orchestra.run.smithi106:> uname -m 2024-03-23T23:59:18.418 INFO:teuthology.orchestra.run.smithi106.stdout:x86_64 2024-03-23T23:59:18.418 DEBUG:teuthology.orchestra.run.smithi106:> cat /etc/os-release 2024-03-23T23:59:18.474 INFO:teuthology.orchestra.run.smithi106.stdout:NAME="CentOS Stream" 2024-03-23T23:59:18.474 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION="9" 2024-03-23T23:59:18.474 INFO:teuthology.orchestra.run.smithi106.stdout:ID="centos" 2024-03-23T23:59:18.474 INFO:teuthology.orchestra.run.smithi106.stdout:ID_LIKE="rhel fedora" 2024-03-23T23:59:18.474 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION_ID="9" 2024-03-23T23:59:18.474 INFO:teuthology.orchestra.run.smithi106.stdout:PLATFORM_ID="platform:el9" 2024-03-23T23:59:18.474 INFO:teuthology.orchestra.run.smithi106.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-23T23:59:18.474 INFO:teuthology.orchestra.run.smithi106.stdout:ANSI_COLOR="0;31" 2024-03-23T23:59:18.474 INFO:teuthology.orchestra.run.smithi106.stdout:LOGO="fedora-logo-icon" 2024-03-23T23:59:18.474 INFO:teuthology.orchestra.run.smithi106.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-23T23:59:18.474 INFO:teuthology.orchestra.run.smithi106.stdout:HOME_URL="https://centos.org/" 2024-03-23T23:59:18.475 INFO:teuthology.orchestra.run.smithi106.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-23T23:59:18.475 INFO:teuthology.orchestra.run.smithi106.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-23T23:59:18.475 INFO:teuthology.orchestra.run.smithi106.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-23T23:59:18.475 INFO:teuthology.lock.ops:Updating smithi106.front.sepia.ceph.com on lock server 2024-03-23T23:59:18.500 DEBUG:teuthology.orchestra.run.smithi107:> uname -m 2024-03-23T23:59:18.517 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64 2024-03-23T23:59:18.518 DEBUG:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2024-03-23T23:59:18.574 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="CentOS Stream" 2024-03-23T23:59:18.574 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="9" 2024-03-23T23:59:18.574 INFO:teuthology.orchestra.run.smithi107.stdout:ID="centos" 2024-03-23T23:59:18.574 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE="rhel fedora" 2024-03-23T23:59:18.575 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="9" 2024-03-23T23:59:18.575 INFO:teuthology.orchestra.run.smithi107.stdout:PLATFORM_ID="platform:el9" 2024-03-23T23:59:18.575 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-23T23:59:18.575 INFO:teuthology.orchestra.run.smithi107.stdout:ANSI_COLOR="0;31" 2024-03-23T23:59:18.575 INFO:teuthology.orchestra.run.smithi107.stdout:LOGO="fedora-logo-icon" 2024-03-23T23:59:18.575 INFO:teuthology.orchestra.run.smithi107.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-23T23:59:18.575 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://centos.org/" 2024-03-23T23:59:18.575 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-23T23:59:18.575 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-23T23:59:18.575 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-23T23:59:18.575 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2024-03-23T23:59:18.600 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-23T23:59:18.610 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-23T23:59:18.615 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-23T23:59:18.616 DEBUG:teuthology.orchestra.run.smithi078:> test '!' -e /home/ubuntu/cephtest 2024-03-23T23:59:18.618 DEBUG:teuthology.orchestra.run.smithi106:> test '!' -e /home/ubuntu/cephtest 2024-03-23T23:59:18.620 DEBUG:teuthology.orchestra.run.smithi107:> test '!' -e /home/ubuntu/cephtest 2024-03-23T23:59:18.636 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-23T23:59:18.642 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-23T23:59:18.643 DEBUG:teuthology.orchestra.run.smithi078:> test -z $(ls -A /var/lib/ceph) 2024-03-23T23:59:18.675 DEBUG:teuthology.orchestra.run.smithi106:> test -z $(ls -A /var/lib/ceph) 2024-03-23T23:59:18.677 DEBUG:teuthology.orchestra.run.smithi107:> test -z $(ls -A /var/lib/ceph) 2024-03-23T23:59:18.917 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-23T23:59:18.992 INFO:teuthology.run_tasks:Running task kernel... 2024-03-23T23:59:19.006 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'} 2024-03-23T23:59:19.006 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-03-23T23:59:19.006 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-03-23T23:59:19.007 DEBUG:teuthology.orchestra.run.smithi078:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-23T23:59:19.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T23:59:19.023 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2024-03-23T23:59:19.079 INFO:teuthology.orchestra.run.smithi078.stdout:5.14.0-361.el9.x86_64 2024-03-23T23:59:19.080 INFO:teuthology.task.kernel:Running kernel on smithi078: 5.14.0-361.el9.x86_64 2024-03-23T23:59:19.080 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum install -y kernel 2024-03-23T23:59:21.270 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-23T23:59:23.013 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS-9 - BaseOS 3.2 MB/s | 2.0 MB 00:00 2024-03-23T23:59:24.105 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-23T23:59:25.979 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-23T23:59:37.221 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - CRB 803 kB/s | 6.1 MB 00:07 2024-03-23T23:59:38.838 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 16 kB 00:00 2024-03-23T23:59:51.155 INFO:teuthology.orchestra.run.smithi078.stdout:Extra Packages for Enterprise Linux 1.7 MB/s | 21 MB 00:12 2024-03-23T23:59:55.939 INFO:teuthology.orchestra.run.smithi078.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-03-23T23:59:58.029 INFO:teuthology.orchestra.run.smithi078.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-23T23:59:58.029 INFO:teuthology.orchestra.run.smithi078.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-23T23:59:58.119 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-23T23:59:58.134 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-23T23:59:58.135 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2024-03-23T23:59:58.135 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-23T23:59:58.135 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-03-23T23:59:58.135 INFO:teuthology.orchestra.run.smithi078.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-23T23:59:58.135 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-03-23T23:59:58.135 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-23T23:59:58.135 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-23T23:59:58.135 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-23T23:59:58.135 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-23T23:59:58.135 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-03-23T23:59:58.135 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-23T23:59:58.135 INFO:teuthology.orchestra.run.smithi078.stdout:Install 4 Packages 2024-03-23T23:59:58.135 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-23T23:59:58.136 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 76 M 2024-03-23T23:59:58.136 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 123 M 2024-03-23T23:59:58.136 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-03-23T23:59:58.373 INFO:teuthology.orchestra.run.smithi078.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 359 kB/s | 44 kB 00:00 2024-03-24T00:00:01.657 INFO:teuthology.orchestra.run.smithi078.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 4.4 MB/s | 15 MB 00:03 2024-03-24T00:00:04.032 INFO:teuthology.orchestra.run.smithi078.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 5.8 MB/s | 33 MB 00:05 2024-03-24T00:00:04.349 INFO:teuthology.orchestra.run.smithi078.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 4.6 MB/s | 27 MB 00:05 2024-03-24T00:00:04.349 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-03-24T00:00:04.349 INFO:teuthology.orchestra.run.smithi078.stdout:Total 12 MB/s | 76 MB 00:06 2024-03-24T00:00:05.012 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-03-24T00:00:05.383 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-03-24T00:00:05.383 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-03-24T00:00:06.367 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-03-24T00:00:06.368 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-03-24T00:00:08.828 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-03-24T00:00:09.589 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T00:00:09.676 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T00:00:12.399 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T00:00:12.531 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T00:00:17.256 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T00:00:17.298 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T00:00:22.067 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T00:00:51.170 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T00:00:51.170 INFO:teuthology.orchestra.run.smithi078.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T00:00:51.170 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:00:51.213 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T00:00:51.563 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T00:00:51.564 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T00:00:51.564 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T00:00:51.564 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T00:00:51.982 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T00:00:51.982 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:00:51.982 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-03-24T00:00:51.983 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T00:00:51.983 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T00:00:51.983 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T00:00:51.983 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T00:00:51.983 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:00:51.983 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-24T00:00:52.304 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T00:00:52.402 INFO:teuthology.orchestra.run.smithi078.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T00:00:52.403 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T00:00:52.403 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T00:00:52.403 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T00:00:52.403 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T00:00:52.403 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-03-24T00:00:52.403 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi078.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T00:00:52.404 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T00:00:52.482 INFO:teuthology.orchestra.run.smithi078.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T00:00:52.483 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T00:00:52.483 DEBUG:teuthology.orchestra.run.smithi078:> sudo rpm -qi grub2-tools 2024-03-24T00:00:52.523 INFO:teuthology.orchestra.run.smithi078.stdout:Name : grub2-tools 2024-03-24T00:00:52.523 INFO:teuthology.orchestra.run.smithi078.stdout:Epoch : 1 2024-03-24T00:00:52.524 INFO:teuthology.orchestra.run.smithi078.stdout:Version : 2.06 2024-03-24T00:00:52.524 INFO:teuthology.orchestra.run.smithi078.stdout:Release : 68.el9 2024-03-24T00:00:52.524 INFO:teuthology.orchestra.run.smithi078.stdout:Architecture: x86_64 2024-03-24T00:00:52.524 INFO:teuthology.orchestra.run.smithi078.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T00:00:52.524 INFO:teuthology.orchestra.run.smithi078.stdout:Group : Unspecified 2024-03-24T00:00:52.524 INFO:teuthology.orchestra.run.smithi078.stdout:Size : 8264254 2024-03-24T00:00:52.524 INFO:teuthology.orchestra.run.smithi078.stdout:License : GPLv3+ 2024-03-24T00:00:52.524 INFO:teuthology.orchestra.run.smithi078.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T00:00:52.524 INFO:teuthology.orchestra.run.smithi078.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T00:00:52.524 INFO:teuthology.orchestra.run.smithi078.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T00:00:52.524 INFO:teuthology.orchestra.run.smithi078.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T00:00:52.524 INFO:teuthology.orchestra.run.smithi078.stdout:Packager : builder@centos.org 2024-03-24T00:00:52.524 INFO:teuthology.orchestra.run.smithi078.stdout:Vendor : CentOS 2024-03-24T00:00:52.525 INFO:teuthology.orchestra.run.smithi078.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T00:00:52.525 INFO:teuthology.orchestra.run.smithi078.stdout:Summary : Support tools for GRUB. 2024-03-24T00:00:52.525 INFO:teuthology.orchestra.run.smithi078.stdout:Description : 2024-03-24T00:00:52.525 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:00:52.525 INFO:teuthology.orchestra.run.smithi078.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T00:00:52.525 INFO:teuthology.orchestra.run.smithi078.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T00:00:52.525 INFO:teuthology.orchestra.run.smithi078.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T00:00:52.525 INFO:teuthology.orchestra.run.smithi078.stdout:hardware devices. 2024-03-24T00:00:52.525 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:00:52.525 INFO:teuthology.orchestra.run.smithi078.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T00:00:52.526 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T00:00:52.526 INFO:teuthology.task.kernel:Updating grub on smithi078 to boot 5.14.0-430.el9.x86_64 2024-03-24T00:00:52.526 DEBUG:teuthology.orchestra.run.smithi078:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T00:00:53.150 INFO:teuthology.orchestra.run.smithi078.stderr:Generating grub configuration file ... 2024-03-24T00:00:54.526 INFO:teuthology.orchestra.run.smithi078.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T00:00:54.563 INFO:teuthology.orchestra.run.smithi078.stderr:done 2024-03-24T00:00:54.565 DEBUG:teuthology.orchestra.run.smithi078:> mktemp 2024-03-24T00:00:54.580 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/tmp.EhFAXjKJpk 2024-03-24T00:00:54.580 DEBUG:teuthology.orchestra.run.smithi078:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.EhFAXjKJpk 2024-03-24T00:00:54.649 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0666 /tmp/tmp.EhFAXjKJpk 2024-03-24T00:00:54.784 DEBUG:teuthology.orchestra.remote:smithi078:/tmp/tmp.EhFAXjKJpk is 6KB 2024-03-24T00:00:54.832 DEBUG:teuthology.orchestra.run.smithi078:> rm -fr /tmp/tmp.EhFAXjKJpk 2024-03-24T00:00:54.846 DEBUG:teuthology.orchestra.run.smithi078:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T00:00:54.910 INFO:teuthology.orchestra.run.smithi078.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T00:00:54.911 INFO:teuthology.orchestra.run.smithi078.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T00:00:54.911 INFO:teuthology.orchestra.run.smithi078.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T00:00:54.911 INFO:teuthology.orchestra.run.smithi078.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T00:00:54.912 DEBUG:teuthology.orchestra.run.smithi078:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T00:00:55.073 DEBUG:teuthology.orchestra.run.smithi078:> sudo shutdown -r now 2024-03-24T00:01:25.075 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-24T00:01:25.076 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-03-24T00:01:25.077 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:01:43.454 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.78 2024-03-24T00:01:52.456 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-03-24T00:01:52.457 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:01:55.522 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.78 2024-03-24T00:02:07.523 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-03-24T00:02:07.524 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:02:39.454 DEBUG:teuthology.orchestra.run.smithi078:> true 2024-03-24T00:02:39.884 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-03-24T00:02:39.884 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-430.el9.x86_64"... 2024-03-24T00:02:39.884 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2024-03-24T00:02:39.897 INFO:teuthology.orchestra.run.smithi078.stdout:5.14.0-430.el9.x86_64 2024-03-24T00:02:39.897 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T00:02:39.898 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T00:02:39.898 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T00:02:40.899 DEBUG:teuthology.parallel:result is None 2024-03-24T00:02:40.899 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-24T00:02:40.911 INFO:teuthology.task.internal:Creating test directory... 2024-03-24T00:02:40.911 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T00:02:40.916 DEBUG:teuthology.orchestra.run.smithi106:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T00:02:40.918 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T00:02:40.937 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-24T00:02:40.948 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-24T00:02:40.963 INFO:teuthology.task.internal:Creating archive directory... 2024-03-24T00:02:40.963 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T00:02:40.975 DEBUG:teuthology.orchestra.run.smithi106:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T00:02:40.978 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T00:02:41.150 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-24T00:02:41.185 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-24T00:02:41.185 DEBUG:teuthology.orchestra.run.smithi106:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T00:02:41.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:02:41.202 DEBUG:teuthology.orchestra.run.smithi107:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T00:02:41.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:02:41.219 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T00:02:41.221 DEBUG:teuthology.orchestra.run.smithi106:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T00:02:41.244 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T00:02:41.282 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T00:02:41.295 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T00:02:41.301 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T00:02:41.307 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T00:02:41.314 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T00:02:41.357 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T00:02:41.359 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-24T00:02:41.375 INFO:teuthology.task.internal:Configuring sudo... 2024-03-24T00:02:41.375 DEBUG:teuthology.orchestra.run.smithi078:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T00:02:41.377 DEBUG:teuthology.orchestra.run.smithi106:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T00:02:41.380 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T00:02:41.426 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-24T00:02:41.435 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-24T00:02:41.435 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T00:02:41.437 DEBUG:teuthology.orchestra.run.smithi106:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T00:02:41.450 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T00:02:41.466 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T00:02:41.546 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T00:02:41.620 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T00:02:41.668 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T00:02:41.730 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-24T00:02:41.730 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T00:02:41.774 DEBUG:teuthology.orchestra.run.smithi106:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T00:02:41.818 DEBUG:teuthology.orchestra.run.smithi106:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T00:02:41.867 DEBUG:teuthology.orchestra.run.smithi106:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T00:02:41.950 DEBUG:teuthology.orchestra.run.smithi106:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T00:02:41.994 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-24T00:02:41.994 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T00:02:42.062 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T00:02:42.104 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T00:02:42.165 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T00:02:42.249 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T00:02:42.293 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-24T00:02:42.293 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T00:02:42.364 DEBUG:teuthology.orchestra.run.smithi078:> sudo service rsyslog restart 2024-03-24T00:02:42.366 DEBUG:teuthology.orchestra.run.smithi106:> sudo service rsyslog restart 2024-03-24T00:02:42.368 DEBUG:teuthology.orchestra.run.smithi107:> sudo service rsyslog restart 2024-03-24T00:02:42.419 INFO:teuthology.orchestra.run.smithi106.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T00:02:42.437 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T00:02:42.493 INFO:teuthology.orchestra.run.smithi078.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T00:02:42.887 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-24T00:02:42.895 INFO:teuthology.task.internal:Starting timer... 2024-03-24T00:02:42.895 INFO:teuthology.run_tasks:Running task pcp... 2024-03-24T00:02:42.909 INFO:teuthology.run_tasks:Running task selinux... 2024-03-24T00:02:42.918 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-24T00:02:42.919 DEBUG:teuthology.orchestra.run.smithi078:> sudo service auditd rotate 2024-03-24T00:02:42.988 INFO:teuthology.orchestra.run.smithi078.stdout:Rotating logs: 2024-03-24T00:02:42.989 DEBUG:teuthology.orchestra.run.smithi106:> sudo service auditd rotate 2024-03-24T00:02:43.049 INFO:teuthology.orchestra.run.smithi106.stdout:Rotating logs: 2024-03-24T00:02:43.051 DEBUG:teuthology.orchestra.run.smithi107:> sudo service auditd rotate 2024-03-24T00:02:43.111 INFO:teuthology.orchestra.run.smithi107.stdout:Rotating logs: 2024-03-24T00:02:43.113 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-24T00:02:43.113 DEBUG:teuthology.orchestra.run.smithi078:> /usr/sbin/getenforce 2024-03-24T00:02:43.139 INFO:teuthology.orchestra.run.smithi078.stdout:Permissive 2024-03-24T00:02:43.139 DEBUG:teuthology.orchestra.run.smithi106:> /usr/sbin/getenforce 2024-03-24T00:02:43.164 INFO:teuthology.orchestra.run.smithi106.stdout:Permissive 2024-03-24T00:02:43.165 DEBUG:teuthology.orchestra.run.smithi107:> /usr/sbin/getenforce 2024-03-24T00:02:43.196 INFO:teuthology.orchestra.run.smithi107.stdout:Permissive 2024-03-24T00:02:43.196 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi078.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi106.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi107.front.sepia.ceph.com': 'permissive'} 2024-03-24T00:02:43.197 DEBUG:teuthology.orchestra.run.smithi078:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T00:02:43.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:02:43.224 DEBUG:teuthology.orchestra.run.smithi106:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T00:02:43.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:02:43.255 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T00:02:43.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:02:43.283 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-24T00:02:43.283 DEBUG:teuthology.orchestra.run.smithi078:> sudo /usr/sbin/setenforce permissive 2024-03-24T00:02:43.316 DEBUG:teuthology.orchestra.run.smithi106:> sudo /usr/sbin/setenforce permissive 2024-03-24T00:02:43.350 DEBUG:teuthology.orchestra.run.smithi107:> sudo /usr/sbin/setenforce permissive 2024-03-24T00:02:43.381 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-24T00:02:43.397 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-24T00:02:43.415 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-24T00:02:43.416 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi078.front.sepia.ceph.com,smithi106.front.sepia.ceph.com,smithi107.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-24T00:07:13.177 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi078.front.sepia.ceph.com'), Remote(name='ubuntu@smithi106.front.sepia.ceph.com'), Remote(name='ubuntu@smithi107.front.sepia.ceph.com')] 2024-03-24T00:07:13.178 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-03-24T00:07:13.180 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:07:13.274 DEBUG:teuthology.orchestra.run.smithi078:> true 2024-03-24T00:07:13.358 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-03-24T00:07:13.359 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-03-24T00:07:13.359 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:07:13.448 DEBUG:teuthology.orchestra.run.smithi106:> true 2024-03-24T00:07:13.517 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-03-24T00:07:13.518 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-03-24T00:07:13.518 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:07:13.605 DEBUG:teuthology.orchestra.run.smithi107:> true 2024-03-24T00:07:13.673 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-03-24T00:07:13.674 INFO:teuthology.run_tasks:Running task clock... 2024-03-24T00:07:13.686 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-24T00:07:13.686 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T00:07:13.686 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T00:07:13.688 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T00:07:13.689 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T00:07:13.691 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T00:07:13.691 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T00:07:13.717 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T00:07:13.721 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T00:07:13.733 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T00:07:13.738 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T00:07:13.745 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T00:07:13.761 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T00:07:13.762 INFO:teuthology.orchestra.run.smithi078.stderr:sudo: ntpd: command not found 2024-03-24T00:07:13.769 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: ntpd: command not found 2024-03-24T00:07:13.776 INFO:teuthology.orchestra.run.smithi078.stdout:506 Cannot talk to daemon 2024-03-24T00:07:13.783 INFO:teuthology.orchestra.run.smithi106.stdout:506 Cannot talk to daemon 2024-03-24T00:07:13.791 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: ntpd: command not found 2024-03-24T00:07:13.792 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T00:07:13.799 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T00:07:13.804 INFO:teuthology.orchestra.run.smithi107.stdout:506 Cannot talk to daemon 2024-03-24T00:07:13.808 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T00:07:13.816 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T00:07:13.820 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T00:07:13.837 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T00:07:13.860 INFO:teuthology.orchestra.run.smithi106.stderr:bash: line 1: ntpq: command not found 2024-03-24T00:07:13.861 INFO:teuthology.orchestra.run.smithi078.stderr:bash: line 1: ntpq: command not found 2024-03-24T00:07:13.863 INFO:teuthology.orchestra.run.smithi106.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T00:07:13.863 INFO:teuthology.orchestra.run.smithi106.stdout:=============================================================================== 2024-03-24T00:07:13.863 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:07:13.863 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:07:13.863 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:07:13.864 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:07:13.865 INFO:teuthology.orchestra.run.smithi078.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T00:07:13.865 INFO:teuthology.orchestra.run.smithi078.stdout:=============================================================================== 2024-03-24T00:07:13.865 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:07:13.865 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:07:13.865 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:07:13.865 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:07:13.880 INFO:teuthology.orchestra.run.smithi107.stderr:bash: line 1: ntpq: command not found 2024-03-24T00:07:13.883 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T00:07:13.884 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2024-03-24T00:07:13.884 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:07:13.884 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:07:13.884 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:07:13.884 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:07:13.885 INFO:teuthology.run_tasks:Running task install... 2024-03-24T00:07:13.909 DEBUG:teuthology.task.install:project ceph 2024-03-24T00:07:13.909 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}} 2024-03-24T00:07:13.910 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'} 2024-03-24T00:07:13.910 INFO:teuthology.task.install:Using flavor: default 2024-03-24T00:07:13.918 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-24T00:07:13.918 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-24T00:07:13.919 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-24T00:07:13.919 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:07:13.922 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-24T00:07:13.922 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:07:13.923 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-24T00:07:13.923 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:07:14.076 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-24T00:07:14.076 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-24T00:07:14.139 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-24T00:07:14.139 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-24T00:07:14.157 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-24T00:07:14.157 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-24T00:07:14.191 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T00:07:14.191 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-24T00:07:14.191 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T00:07:14.219 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-24T00:07:14.219 DEBUG:teuthology.orchestra.run.smithi107:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T00:07:14.249 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T00:07:14.249 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-24T00:07:14.249 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T00:07:14.265 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T00:07:14.265 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-24T00:07:14.266 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T00:07:14.278 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-24T00:07:14.278 DEBUG:teuthology.orchestra.run.smithi078:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T00:07:14.287 DEBUG:teuthology.orchestra.run.smithi107:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T00:07:14.293 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-24T00:07:14.293 DEBUG:teuthology.orchestra.run.smithi106:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T00:07:14.346 DEBUG:teuthology.orchestra.run.smithi078:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T00:07:14.360 DEBUG:teuthology.orchestra.run.smithi106:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T00:07:14.366 DEBUG:teuthology.orchestra.run.smithi107:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T00:07:14.426 DEBUG:teuthology.orchestra.run.smithi078:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T00:07:14.432 INFO:teuthology.orchestra.run.smithi107.stdout:check_obsoletes = 1 2024-03-24T00:07:14.434 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum clean all 2024-03-24T00:07:14.438 DEBUG:teuthology.orchestra.run.smithi106:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T00:07:14.492 INFO:teuthology.orchestra.run.smithi078.stdout:check_obsoletes = 1 2024-03-24T00:07:14.495 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum clean all 2024-03-24T00:07:14.504 INFO:teuthology.orchestra.run.smithi106.stdout:check_obsoletes = 1 2024-03-24T00:07:14.506 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum clean all 2024-03-24T00:07:14.665 INFO:teuthology.orchestra.run.smithi107.stdout:74 files removed 2024-03-24T00:07:14.692 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-24T00:07:14.737 INFO:teuthology.orchestra.run.smithi078.stdout:74 files removed 2024-03-24T00:07:14.739 INFO:teuthology.orchestra.run.smithi106.stdout:74 files removed 2024-03-24T00:07:14.764 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-24T00:07:14.765 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-24T00:07:15.322 INFO:teuthology.orchestra.run.smithi107.stdout:ceph packages for x86_64 244 kB/s | 83 kB 00:00 2024-03-24T00:07:15.380 INFO:teuthology.orchestra.run.smithi078.stdout:ceph packages for x86_64 261 kB/s | 83 kB 00:00 2024-03-24T00:07:15.428 INFO:teuthology.orchestra.run.smithi106.stdout:ceph packages for x86_64 223 kB/s | 83 kB 00:00 2024-03-24T00:07:15.752 INFO:teuthology.orchestra.run.smithi107.stdout:ceph noarch packages 26 kB/s | 10 kB 00:00 2024-03-24T00:07:15.832 INFO:teuthology.orchestra.run.smithi078.stdout:ceph noarch packages 25 kB/s | 10 kB 00:00 2024-03-24T00:07:15.863 INFO:teuthology.orchestra.run.smithi106.stdout:ceph noarch packages 26 kB/s | 10 kB 00:00 2024-03-24T00:07:16.377 INFO:teuthology.orchestra.run.smithi107.stdout:ceph source packages 3.2 kB/s | 1.9 kB 00:00 2024-03-24T00:07:16.588 INFO:teuthology.orchestra.run.smithi106.stdout:ceph source packages 2.7 kB/s | 1.9 kB 00:00 2024-03-24T00:07:16.590 INFO:teuthology.orchestra.run.smithi078.stdout:ceph source packages 2.6 kB/s | 1.9 kB 00:00 2024-03-24T00:07:16.771 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-24T00:07:17.034 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-24T00:07:17.040 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-24T00:07:18.171 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-9 - BaseOS 6.4 MB/s | 2.0 MB 00:00 2024-03-24T00:07:18.409 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2024-03-24T00:07:18.416 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2024-03-24T00:07:19.987 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - BaseOS 5.4 MB/s | 8.0 MB 00:01 2024-03-24T00:07:20.133 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - BaseOS 5.7 MB/s | 8.0 MB 00:01 2024-03-24T00:07:21.957 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-24T00:07:22.823 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - BaseOS 2.0 MB/s | 8.0 MB 00:04 2024-03-24T00:07:23.876 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - AppStream 7.0 MB/s | 19 MB 00:02 2024-03-24T00:07:26.021 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - CRB 9.7 MB/s | 6.1 MB 00:00 2024-03-24T00:07:27.505 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 16 kB 00:00 2024-03-24T00:07:28.304 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - CRB 6.2 MB/s | 6.1 MB 00:00 2024-03-24T00:07:28.332 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 29 MB/s | 21 MB 00:00 2024-03-24T00:07:30.978 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-03-24T00:07:32.465 INFO:teuthology.orchestra.run.smithi078.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-24T00:07:33.181 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-03-24T00:07:35.486 INFO:teuthology.orchestra.run.smithi107.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T00:07:35.487 INFO:teuthology.orchestra.run.smithi107.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T00:07:35.501 INFO:teuthology.orchestra.run.smithi107.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-24T00:07:35.502 INFO:teuthology.orchestra.run.smithi107.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-24T00:07:35.503 INFO:teuthology.orchestra.run.smithi107.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-24T00:07:35.504 INFO:teuthology.orchestra.run.smithi107.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-24T00:07:35.506 INFO:teuthology.orchestra.run.smithi107.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-24T00:07:35.509 INFO:teuthology.orchestra.run.smithi107.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-24T00:07:35.510 INFO:teuthology.orchestra.run.smithi107.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-24T00:07:35.592 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-24T00:07:35.604 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================================ 2024-03-24T00:07:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-03-24T00:07:35.604 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================================ 2024-03-24T00:07:35.604 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-24T00:07:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-24T00:07:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-24T00:07:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-24T00:07:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-24T00:07:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-24T00:07:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-24T00:07:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-24T00:07:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-24T00:07:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-24T00:07:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-24T00:07:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-24T00:07:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-24T00:07:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-24T00:07:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-24T00:07:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-24T00:07:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-24T00:07:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-24T00:07:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-24T00:07:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-24T00:07:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-24T00:07:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-24T00:07:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-24T00:07:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-24T00:07:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-24T00:07:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-24T00:07:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-24T00:07:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-24T00:07:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-24T00:07:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-24T00:07:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-24T00:07:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-24T00:07:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-24T00:07:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-24T00:07:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-24T00:07:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-24T00:07:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-24T00:07:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-24T00:07:35.607 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrading: 2024-03-24T00:07:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-24T00:07:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-24T00:07:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-24T00:07:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-24T00:07:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-24T00:07:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-24T00:07:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-24T00:07:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-24T00:07:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-24T00:07:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-24T00:07:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-24T00:07:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-24T00:07:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-24T00:07:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-24T00:07:35.609 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-24T00:07:35.609 INFO:teuthology.orchestra.run.smithi107.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-24T00:07:35.609 INFO:teuthology.orchestra.run.smithi107.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-24T00:07:35.609 INFO:teuthology.orchestra.run.smithi107.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-24T00:07:35.609 INFO:teuthology.orchestra.run.smithi107.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-24T00:07:35.609 INFO:teuthology.orchestra.run.smithi107.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-24T00:07:35.609 INFO:teuthology.orchestra.run.smithi107.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-24T00:07:35.609 INFO:teuthology.orchestra.run.smithi107.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-24T00:07:35.609 INFO:teuthology.orchestra.run.smithi107.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-24T00:07:35.609 INFO:teuthology.orchestra.run.smithi107.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-24T00:07:35.609 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-03-24T00:07:35.609 INFO:teuthology.orchestra.run.smithi107.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T00:07:35.609 INFO:teuthology.orchestra.run.smithi107.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-24T00:07:35.610 INFO:teuthology.orchestra.run.smithi107.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T00:07:35.610 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-24T00:07:35.610 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-24T00:07:35.610 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-24T00:07:35.610 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-24T00:07:35.610 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-24T00:07:35.610 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-24T00:07:35.610 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-24T00:07:35.610 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-24T00:07:35.610 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T00:07:35.610 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T00:07:35.610 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T00:07:35.610 INFO:teuthology.orchestra.run.smithi107.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T00:07:35.611 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T00:07:35.611 INFO:teuthology.orchestra.run.smithi107.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T00:07:35.611 INFO:teuthology.orchestra.run.smithi107.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T00:07:35.611 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T00:07:35.611 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T00:07:35.611 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-24T00:07:35.611 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T00:07:35.611 INFO:teuthology.orchestra.run.smithi107.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T00:07:35.611 INFO:teuthology.orchestra.run.smithi107.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T00:07:35.611 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T00:07:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T00:07:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-24T00:07:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T00:07:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-24T00:07:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T00:07:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-24T00:07:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T00:07:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T00:07:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T00:07:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T00:07:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T00:07:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T00:07:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T00:07:35.613 INFO:teuthology.orchestra.run.smithi107.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T00:07:35.613 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-24T00:07:35.613 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T00:07:35.613 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T00:07:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T00:07:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-24T00:07:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T00:07:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T00:07:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T00:07:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-24T00:07:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-24T00:07:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T00:07:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T00:07:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T00:07:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T00:07:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-24T00:07:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-24T00:07:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T00:07:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T00:07:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T00:07:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T00:07:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T00:07:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T00:07:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T00:07:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T00:07:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T00:07:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T00:07:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T00:07:35.616 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T00:07:35.616 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T00:07:35.616 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T00:07:35.616 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T00:07:35.616 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T00:07:35.616 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T00:07:35.616 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T00:07:35.616 INFO:teuthology.orchestra.run.smithi107.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T00:07:35.616 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T00:07:35.616 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T00:07:35.616 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T00:07:35.616 INFO:teuthology.orchestra.run.smithi107.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T00:07:35.616 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T00:07:35.616 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T00:07:35.617 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T00:07:35.617 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T00:07:35.617 INFO:teuthology.orchestra.run.smithi107.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T00:07:35.617 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T00:07:35.617 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T00:07:35.617 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T00:07:35.617 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T00:07:35.617 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-24T00:07:35.617 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T00:07:35.617 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T00:07:35.617 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T00:07:35.617 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T00:07:35.617 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T00:07:35.618 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T00:07:35.618 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T00:07:35.618 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T00:07:35.618 INFO:teuthology.orchestra.run.smithi107.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T00:07:35.618 INFO:teuthology.orchestra.run.smithi107.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T00:07:35.618 INFO:teuthology.orchestra.run.smithi107.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T00:07:35.618 INFO:teuthology.orchestra.run.smithi107.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-24T00:07:35.618 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T00:07:35.618 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T00:07:35.618 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T00:07:35.618 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T00:07:35.618 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T00:07:35.618 INFO:teuthology.orchestra.run.smithi107.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T00:07:35.619 INFO:teuthology.orchestra.run.smithi107.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T00:07:35.619 INFO:teuthology.orchestra.run.smithi107.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T00:07:35.619 INFO:teuthology.orchestra.run.smithi107.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T00:07:35.619 INFO:teuthology.orchestra.run.smithi107.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-24T00:07:35.619 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T00:07:35.619 INFO:teuthology.orchestra.run.smithi107.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-24T00:07:35.619 INFO:teuthology.orchestra.run.smithi107.stdout:Installing weak dependencies: 2024-03-24T00:07:35.619 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T00:07:35.619 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-24T00:07:35.619 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-24T00:07:35.619 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================================ 2024-03-24T00:07:35.619 INFO:teuthology.orchestra.run.smithi107.stdout:Install 150 Packages 2024-03-24T00:07:35.619 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrade 24 Packages 2024-03-24T00:07:35.620 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-24T00:07:35.620 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 224 M 2024-03-24T00:07:35.620 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-24T00:07:36.524 INFO:teuthology.orchestra.run.smithi107.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 39 kB/s | 6.6 kB 00:00 2024-03-24T00:07:36.716 INFO:teuthology.orchestra.run.smithi107.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 4.5 MB/s | 890 kB 00:00 2024-03-24T00:07:36.791 INFO:teuthology.orchestra.run.smithi107.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 1.9 MB/s | 145 kB 00:00 2024-03-24T00:07:36.883 INFO:teuthology.orchestra.run.smithi107.stdout:(4/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 10 MB/s | 5.4 MB 00:00 2024-03-24T00:07:37.025 INFO:teuthology.orchestra.run.smithi107.stdout:(5/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 9.9 MB/s | 2.3 MB 00:00 2024-03-24T00:07:37.067 INFO:teuthology.orchestra.run.smithi107.stdout:(6/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 8.1 MB/s | 1.5 MB 00:00 2024-03-24T00:07:37.379 INFO:teuthology.orchestra.run.smithi078.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-03-24T00:07:37.467 INFO:teuthology.orchestra.run.smithi107.stdout:(7/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 11 MB/s | 4.7 MB 00:00 2024-03-24T00:07:38.351 INFO:teuthology.orchestra.run.smithi107.stdout:(8/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x86 13 MB/s | 17 MB 00:01 2024-03-24T00:07:38.492 INFO:teuthology.orchestra.run.smithi107.stdout:(9/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el9 9.6 MB/s | 9.9 MB 00:01 2024-03-24T00:07:38.526 INFO:teuthology.orchestra.run.smithi107.stdout:(10/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el 144 kB/s | 25 kB 00:00 2024-03-24T00:07:38.584 INFO:teuthology.orchestra.run.smithi107.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 3.7 MB/s | 220 kB 00:00 2024-03-24T00:07:38.651 INFO:teuthology.orchestra.run.smithi107.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 406 kB/s | 27 kB 00:00 2024-03-24T00:07:38.718 INFO:teuthology.orchestra.run.smithi107.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 469 kB/s | 31 kB 00:00 2024-03-24T00:07:38.819 INFO:teuthology.orchestra.run.smithi107.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 7.3 MB/s | 746 kB 00:00 2024-03-24T00:07:38.894 INFO:teuthology.orchestra.run.smithi107.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 2.1 MB/s | 163 kB 00:00 2024-03-24T00:07:38.969 INFO:teuthology.orchestra.run.smithi107.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.7 MB/s | 127 kB 00:00 2024-03-24T00:07:39.061 INFO:teuthology.orchestra.run.smithi107.stdout:(17/174): libradosstriper1-19.0.0-2145.g3be9a5c 5.3 MB/s | 501 kB 00:00 2024-03-24T00:07:39.611 INFO:teuthology.orchestra.run.smithi107.stdout:(18/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 9.4 MB/s | 5.1 MB 00:00 2024-03-24T00:07:39.718 INFO:teuthology.orchestra.run.smithi078.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T00:07:39.718 INFO:teuthology.orchestra.run.smithi078.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T00:07:39.733 INFO:teuthology.orchestra.run.smithi078.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-24T00:07:39.734 INFO:teuthology.orchestra.run.smithi078.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-24T00:07:39.735 INFO:teuthology.orchestra.run.smithi078.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-24T00:07:39.736 INFO:teuthology.orchestra.run.smithi078.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-24T00:07:39.738 INFO:teuthology.orchestra.run.smithi078.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-24T00:07:39.741 INFO:teuthology.orchestra.run.smithi078.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-24T00:07:39.742 INFO:teuthology.orchestra.run.smithi078.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-24T00:07:39.795 INFO:teuthology.orchestra.run.smithi107.stdout:(19/174): ceph-common-19.0.0-2145.g3be9a5c9.el9 6.0 MB/s | 20 MB 00:03 2024-03-24T00:07:39.812 INFO:teuthology.orchestra.run.smithi107.stdout:(20/174): python3-ceph-argparse-19.0.0-2145.g3b 225 kB/s | 45 kB 00:00 2024-03-24T00:07:39.822 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-24T00:07:39.844 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================================ 2024-03-24T00:07:39.844 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-03-24T00:07:39.845 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================================ 2024-03-24T00:07:39.845 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-03-24T00:07:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-24T00:07:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-24T00:07:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-24T00:07:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-24T00:07:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-24T00:07:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-24T00:07:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-24T00:07:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-24T00:07:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-24T00:07:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-24T00:07:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-24T00:07:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-24T00:07:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-24T00:07:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-24T00:07:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-24T00:07:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-24T00:07:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-24T00:07:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-24T00:07:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-24T00:07:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-24T00:07:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-24T00:07:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-24T00:07:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-24T00:07:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-24T00:07:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-24T00:07:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-24T00:07:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-24T00:07:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-24T00:07:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-24T00:07:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-24T00:07:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-24T00:07:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-24T00:07:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-24T00:07:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-24T00:07:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-24T00:07:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-24T00:07:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-24T00:07:39.848 INFO:teuthology.orchestra.run.smithi078.stdout:Upgrading: 2024-03-24T00:07:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-24T00:07:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-24T00:07:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-24T00:07:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-24T00:07:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-24T00:07:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-24T00:07:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-24T00:07:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-24T00:07:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-24T00:07:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-24T00:07:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-24T00:07:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-24T00:07:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-24T00:07:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-24T00:07:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-24T00:07:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-24T00:07:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-24T00:07:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-24T00:07:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-24T00:07:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-24T00:07:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-24T00:07:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-24T00:07:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-24T00:07:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-24T00:07:39.850 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-03-24T00:07:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T00:07:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-24T00:07:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T00:07:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-24T00:07:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-24T00:07:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-24T00:07:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-24T00:07:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-24T00:07:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-24T00:07:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-24T00:07:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-24T00:07:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T00:07:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T00:07:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T00:07:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T00:07:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T00:07:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T00:07:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T00:07:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T00:07:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T00:07:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-24T00:07:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T00:07:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T00:07:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T00:07:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T00:07:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T00:07:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-24T00:07:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T00:07:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-24T00:07:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T00:07:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-24T00:07:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T00:07:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T00:07:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T00:07:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T00:07:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T00:07:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T00:07:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T00:07:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T00:07:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-24T00:07:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T00:07:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T00:07:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T00:07:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-24T00:07:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T00:07:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T00:07:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T00:07:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-24T00:07:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-24T00:07:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T00:07:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T00:07:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T00:07:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T00:07:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-24T00:07:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-24T00:07:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T00:07:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T00:07:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T00:07:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T00:07:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T00:07:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T00:07:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T00:07:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T00:07:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T00:07:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T00:07:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T00:07:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T00:07:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T00:07:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T00:07:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T00:07:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T00:07:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T00:07:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T00:07:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T00:07:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T00:07:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T00:07:39.856 INFO:teuthology.orchestra.run.smithi078.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T00:07:39.856 INFO:teuthology.orchestra.run.smithi078.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T00:07:39.856 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T00:07:39.856 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T00:07:39.856 INFO:teuthology.orchestra.run.smithi078.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T00:07:39.856 INFO:teuthology.orchestra.run.smithi078.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T00:07:39.856 INFO:teuthology.orchestra.run.smithi078.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T00:07:39.856 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T00:07:39.856 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T00:07:39.856 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T00:07:39.856 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T00:07:39.856 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-24T00:07:39.856 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T00:07:39.856 INFO:teuthology.orchestra.run.smithi078.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T00:07:39.857 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T00:07:39.857 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T00:07:39.857 INFO:teuthology.orchestra.run.smithi078.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T00:07:39.857 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T00:07:39.857 INFO:teuthology.orchestra.run.smithi078.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T00:07:39.857 INFO:teuthology.orchestra.run.smithi078.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T00:07:39.857 INFO:teuthology.orchestra.run.smithi078.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T00:07:39.857 INFO:teuthology.orchestra.run.smithi078.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T00:07:39.857 INFO:teuthology.orchestra.run.smithi078.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T00:07:39.857 INFO:teuthology.orchestra.run.smithi078.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-24T00:07:39.857 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T00:07:39.857 INFO:teuthology.orchestra.run.smithi078.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T00:07:39.857 INFO:teuthology.orchestra.run.smithi078.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T00:07:39.858 INFO:teuthology.orchestra.run.smithi078.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T00:07:39.858 INFO:teuthology.orchestra.run.smithi078.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T00:07:39.858 INFO:teuthology.orchestra.run.smithi078.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T00:07:39.858 INFO:teuthology.orchestra.run.smithi078.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T00:07:39.858 INFO:teuthology.orchestra.run.smithi078.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T00:07:39.858 INFO:teuthology.orchestra.run.smithi078.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T00:07:39.858 INFO:teuthology.orchestra.run.smithi078.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-24T00:07:39.858 INFO:teuthology.orchestra.run.smithi078.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T00:07:39.858 INFO:teuthology.orchestra.run.smithi078.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-24T00:07:39.858 INFO:teuthology.orchestra.run.smithi078.stdout:Installing weak dependencies: 2024-03-24T00:07:39.858 INFO:teuthology.orchestra.run.smithi078.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T00:07:39.858 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:07:39.858 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-03-24T00:07:39.859 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================================ 2024-03-24T00:07:39.859 INFO:teuthology.orchestra.run.smithi078.stdout:Install 150 Packages 2024-03-24T00:07:39.859 INFO:teuthology.orchestra.run.smithi078.stdout:Upgrade 24 Packages 2024-03-24T00:07:39.859 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:07:39.859 INFO:teuthology.orchestra.run.smithi107.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 3.8 MB/s | 162 kB 00:00 2024-03-24T00:07:39.859 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 224 M 2024-03-24T00:07:39.860 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-03-24T00:07:39.879 INFO:teuthology.orchestra.run.smithi107.stdout:(22/174): python3-ceph-common-19.0.0-2145.g3be9 1.5 MB/s | 131 kB 00:00 2024-03-24T00:07:39.921 INFO:teuthology.orchestra.run.smithi107.stdout:(23/174): python3-rados-19.0.0-2145.g3be9a5c9.e 4.7 MB/s | 321 kB 00:00 2024-03-24T00:07:39.971 INFO:teuthology.orchestra.run.smithi107.stdout:(24/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 3.2 MB/s | 298 kB 00:00 2024-03-24T00:07:39.996 INFO:teuthology.orchestra.run.smithi107.stdout:(25/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 1.3 MB/s | 99 kB 00:00 2024-03-24T00:07:40.047 INFO:teuthology.orchestra.run.smithi107.stdout:(26/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.1 MB/s | 85 kB 00:00 2024-03-24T00:07:40.138 INFO:teuthology.orchestra.run.smithi107.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 1.8 MB/s | 170 kB 00:00 2024-03-24T00:07:40.214 INFO:teuthology.orchestra.run.smithi107.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 453 kB/s | 34 kB 00:00 2024-03-24T00:07:40.272 INFO:teuthology.orchestra.run.smithi107.stdout:(29/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 11 MB/s | 3.1 MB 00:00 2024-03-24T00:07:40.298 INFO:teuthology.orchestra.run.smithi107.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 1.6 MB/s | 139 kB 00:00 2024-03-24T00:07:40.464 INFO:teuthology.orchestra.run.smithi107.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 9.4 MB/s | 1.8 MB 00:00 2024-03-24T00:07:40.548 INFO:teuthology.orchestra.run.smithi107.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 2.9 MB/s | 248 kB 00:00 2024-03-24T00:07:40.615 INFO:teuthology.orchestra.run.smithi107.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 740 kB/s | 49 kB 00:00 2024-03-24T00:07:40.674 INFO:teuthology.orchestra.run.smithi107.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2145.g3 283 kB/s | 16 kB 00:00 2024-03-24T00:07:40.757 INFO:teuthology.orchestra.run.smithi107.stdout:(35/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 3.1 MB/s | 264 kB 00:00 2024-03-24T00:07:40.858 INFO:teuthology.orchestra.run.smithi107.stdout:(36/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 7.3 MB/s | 748 kB 00:00 2024-03-24T00:07:40.925 INFO:teuthology.orchestra.run.smithi107.stdout:(37/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 275 kB/s | 18 kB 00:00 2024-03-24T00:07:41.067 INFO:teuthology.orchestra.run.smithi107.stdout:(38/174): ceph-mgr-diskprediction-local-19.0.0- 9.6 MB/s | 7.4 MB 00:00 2024-03-24T00:07:41.105 INFO:teuthology.orchestra.run.smithi078.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 35 kB/s | 6.6 kB 00:00 2024-03-24T00:07:41.150 INFO:teuthology.orchestra.run.smithi107.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 2.9 MB/s | 662 kB 00:00 2024-03-24T00:07:41.184 INFO:teuthology.orchestra.run.smithi107.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 7.7 MB/s | 920 kB 00:00 2024-03-24T00:07:41.209 INFO:teuthology.orchestra.run.smithi107.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 106 kB 00:00 2024-03-24T00:07:41.234 INFO:teuthology.orchestra.run.smithi107.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-24T00:07:41.259 INFO:teuthology.orchestra.run.smithi107.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 638 kB/s | 32 kB 00:00 2024-03-24T00:07:41.327 INFO:teuthology.orchestra.run.smithi107.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 33 MB/s | 3.0 MB 00:00 2024-03-24T00:07:41.352 INFO:teuthology.orchestra.run.smithi107.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 192 kB/s | 17 kB 00:00 2024-03-24T00:07:41.377 INFO:teuthology.orchestra.run.smithi107.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 202 kB/s | 10 kB 00:00 2024-03-24T00:07:41.402 INFO:teuthology.orchestra.run.smithi107.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-24T00:07:41.427 INFO:teuthology.orchestra.run.smithi107.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 896 kB/s | 44 kB 00:00 2024-03-24T00:07:41.460 INFO:teuthology.orchestra.run.smithi107.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-24T00:07:41.486 INFO:teuthology.orchestra.run.smithi107.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.8 MB/s | 578 kB 00:00 2024-03-24T00:07:41.511 INFO:teuthology.orchestra.run.smithi107.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 932 kB/s | 46 kB 00:00 2024-03-24T00:07:41.537 INFO:teuthology.orchestra.run.smithi107.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-24T00:07:41.553 INFO:teuthology.orchestra.run.smithi107.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2024-03-24T00:07:41.579 INFO:teuthology.orchestra.run.smithi107.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2024-03-24T00:07:41.596 INFO:teuthology.orchestra.run.smithi107.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 823 kB/s | 34 kB 00:00 2024-03-24T00:07:41.630 INFO:teuthology.orchestra.run.smithi078.stdout:(2/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 7.5 MB/s | 5.4 MB 00:00 2024-03-24T00:07:41.664 INFO:teuthology.orchestra.run.smithi078.stdout:(3/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.6 MB/s | 890 kB 00:00 2024-03-24T00:07:41.689 INFO:teuthology.orchestra.run.smithi078.stdout:(4/174): ceph-immutable-object-cache-19.0.0-214 2.4 MB/s | 145 kB 00:00 2024-03-24T00:07:41.696 INFO:teuthology.orchestra.run.smithi107.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 41 MB/s | 4.7 MB 00:00 2024-03-24T00:07:41.748 INFO:teuthology.orchestra.run.smithi107.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 10 MB/s | 1.5 MB 00:00 2024-03-24T00:07:41.780 INFO:teuthology.orchestra.run.smithi107.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 9.8 MB/s | 811 kB 00:00 2024-03-24T00:07:41.805 INFO:teuthology.orchestra.run.smithi107.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.3 MB/s | 134 kB 00:00 2024-03-24T00:07:41.873 INFO:teuthology.orchestra.run.smithi078.stdout:(5/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 8.2 MB/s | 1.5 MB 00:00 2024-03-24T00:07:41.922 INFO:teuthology.orchestra.run.smithi107.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 40 MB/s | 5.7 MB 00:00 2024-03-24T00:07:41.956 INFO:teuthology.orchestra.run.smithi107.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 6.8 MB/s | 1.0 MB 00:00 2024-03-24T00:07:41.981 INFO:teuthology.orchestra.run.smithi107.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-24T00:07:42.090 INFO:teuthology.orchestra.run.smithi107.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 43 MB/s | 5.8 MB 00:00 2024-03-24T00:07:42.106 INFO:teuthology.orchestra.run.smithi107.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 199 kB/s | 24 kB 00:00 2024-03-24T00:07:42.123 INFO:teuthology.orchestra.run.smithi107.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 542 kB/s | 18 kB 00:00 2024-03-24T00:07:42.148 INFO:teuthology.orchestra.run.smithi107.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-24T00:07:42.166 INFO:teuthology.orchestra.run.smithi107.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 755 kB/s | 32 kB 00:00 2024-03-24T00:07:42.233 INFO:teuthology.orchestra.run.smithi107.stdout:(68/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.2 MB/s | 217 kB 00:00 2024-03-24T00:07:42.299 INFO:teuthology.orchestra.run.smithi107.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-03-24T00:07:42.325 INFO:teuthology.orchestra.run.smithi107.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.9 MB/s | 174 kB 00:00 2024-03-24T00:07:42.350 INFO:teuthology.orchestra.run.smithi107.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-24T00:07:42.357 INFO:teuthology.orchestra.run.smithi078.stdout:(6/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 9.7 MB/s | 4.7 MB 00:00 2024-03-24T00:07:42.367 INFO:teuthology.orchestra.run.smithi107.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 984 kB/s | 41 kB 00:00 2024-03-24T00:07:42.392 INFO:teuthology.orchestra.run.smithi107.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 5.6 MB/s | 238 kB 00:00 2024-03-24T00:07:42.418 INFO:teuthology.orchestra.run.smithi107.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-24T00:07:42.443 INFO:teuthology.orchestra.run.smithi107.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-24T00:07:42.468 INFO:teuthology.orchestra.run.smithi107.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 945 kB/s | 47 kB 00:00 2024-03-24T00:07:42.493 INFO:teuthology.orchestra.run.smithi107.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 850 kB/s | 43 kB 00:00 2024-03-24T00:07:42.540 INFO:teuthology.orchestra.run.smithi078.stdout:(7/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 2.6 MB/s | 2.3 MB 00:00 2024-03-24T00:07:42.636 INFO:teuthology.orchestra.run.smithi107.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 277 kB/s | 39 kB 00:00 2024-03-24T00:07:42.852 INFO:teuthology.orchestra.run.smithi107.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-03-24T00:07:42.877 INFO:teuthology.orchestra.run.smithi107.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 171 kB/s | 41 kB 00:00 2024-03-24T00:07:42.903 INFO:teuthology.orchestra.run.smithi107.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-24T00:07:42.928 INFO:teuthology.orchestra.run.smithi107.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-24T00:07:42.953 INFO:teuthology.orchestra.run.smithi107.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-24T00:07:42.979 INFO:teuthology.orchestra.run.smithi107.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 878 kB/s | 44 kB 00:00 2024-03-24T00:07:43.004 INFO:teuthology.orchestra.run.smithi107.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-24T00:07:43.029 INFO:teuthology.orchestra.run.smithi107.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-24T00:07:43.063 INFO:teuthology.orchestra.run.smithi107.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-03-24T00:07:43.088 INFO:teuthology.orchestra.run.smithi107.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-24T00:07:43.113 INFO:teuthology.orchestra.run.smithi107.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-24T00:07:43.139 INFO:teuthology.orchestra.run.smithi107.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-03-24T00:07:43.565 INFO:teuthology.orchestra.run.smithi078.stdout:(8/174): ceph-common-19.0.0-2145.g3be9a5c9.el9. 7.7 MB/s | 20 MB 00:02 2024-03-24T00:07:43.572 INFO:teuthology.orchestra.run.smithi107.stdout:(91/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 9.4 MB/s | 48 MB 00:05 2024-03-24T00:07:43.607 INFO:teuthology.orchestra.run.smithi078.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 608 kB/s | 25 kB 00:00 2024-03-24T00:07:43.766 INFO:teuthology.orchestra.run.smithi078.stdout:(10/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x8 12 MB/s | 17 MB 00:01 2024-03-24T00:07:43.832 INFO:teuthology.orchestra.run.smithi078.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 3.2 MB/s | 220 kB 00:00 2024-03-24T00:07:43.883 INFO:teuthology.orchestra.run.smithi078.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 541 kB/s | 27 kB 00:00 2024-03-24T00:07:43.924 INFO:teuthology.orchestra.run.smithi078.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 753 kB/s | 31 kB 00:00 2024-03-24T00:07:44.008 INFO:teuthology.orchestra.run.smithi078.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 8.7 MB/s | 746 kB 00:00 2024-03-24T00:07:44.075 INFO:teuthology.orchestra.run.smithi078.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 2.4 MB/s | 163 kB 00:00 2024-03-24T00:07:44.142 INFO:teuthology.orchestra.run.smithi078.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.9 MB/s | 127 kB 00:00 2024-03-24T00:07:44.226 INFO:teuthology.orchestra.run.smithi078.stdout:(17/174): libradosstriper1-19.0.0-2145.g3be9a5c 5.9 MB/s | 501 kB 00:00 2024-03-24T00:07:44.263 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - AppStream 946 kB/s | 19 MB 00:20 2024-03-24T00:07:44.339 INFO:teuthology.orchestra.run.smithi107.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 255 kB/s | 195 kB 00:00 2024-03-24T00:07:44.684 INFO:teuthology.orchestra.run.smithi078.stdout:(18/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 11 MB/s | 5.1 MB 00:00 2024-03-24T00:07:44.734 INFO:teuthology.orchestra.run.smithi078.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 908 kB/s | 45 kB 00:00 2024-03-24T00:07:44.793 INFO:teuthology.orchestra.run.smithi078.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 2.2 MB/s | 131 kB 00:00 2024-03-24T00:07:44.860 INFO:teuthology.orchestra.run.smithi078.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 2.4 MB/s | 162 kB 00:00 2024-03-24T00:07:44.935 INFO:teuthology.orchestra.run.smithi078.stdout:(22/174): python3-rados-19.0.0-2145.g3be9a5c9.e 4.2 MB/s | 321 kB 00:00 2024-03-24T00:07:45.011 INFO:teuthology.orchestra.run.smithi078.stdout:(23/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 3.9 MB/s | 298 kB 00:00 2024-03-24T00:07:45.047 INFO:teuthology.orchestra.run.smithi107.stdout:(93/174): jq-1.6-16.el9.x86_64.rpm 97 kB/s | 187 kB 00:01 2024-03-24T00:07:45.136 INFO:teuthology.orchestra.run.smithi078.stdout:(24/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el 3.8 MB/s | 9.9 MB 00:02 2024-03-24T00:07:45.161 INFO:teuthology.orchestra.run.smithi078.stdout:(25/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 661 kB/s | 99 kB 00:00 2024-03-24T00:07:45.195 INFO:teuthology.orchestra.run.smithi078.stdout:(26/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.4 MB/s | 85 kB 00:00 2024-03-24T00:07:45.278 INFO:teuthology.orchestra.run.smithi078.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 2.0 MB/s | 170 kB 00:00 2024-03-24T00:07:45.298 INFO:teuthology.orchestra.run.smithi107.stdout:(94/174): elfutils-libelf-devel-0.190-2.el9.x86 86 kB/s | 21 kB 00:00 2024-03-24T00:07:45.323 INFO:teuthology.orchestra.run.smithi107.stdout:(95/174): python3-requests-2.25.1-8.el9.noarch. 127 kB/s | 125 kB 00:00 2024-03-24T00:07:45.362 INFO:teuthology.orchestra.run.smithi078.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 407 kB/s | 34 kB 00:00 2024-03-24T00:07:45.429 INFO:teuthology.orchestra.run.smithi078.stdout:(29/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 12 MB/s | 3.1 MB 00:00 2024-03-24T00:07:45.454 INFO:teuthology.orchestra.run.smithi078.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 1.5 MB/s | 139 kB 00:00 2024-03-24T00:07:45.543 INFO:teuthology.orchestra.run.smithi107.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 104 kB/s | 25 kB 00:00 2024-03-24T00:07:45.604 INFO:teuthology.orchestra.run.smithi078.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 10 MB/s | 1.8 MB 00:00 2024-03-24T00:07:45.671 INFO:teuthology.orchestra.run.smithi078.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 3.6 MB/s | 248 kB 00:00 2024-03-24T00:07:45.730 INFO:teuthology.orchestra.run.smithi078.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 847 kB/s | 49 kB 00:00 2024-03-24T00:07:45.797 INFO:teuthology.orchestra.run.smithi078.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2145.g3 247 kB/s | 16 kB 00:00 2024-03-24T00:07:45.872 INFO:teuthology.orchestra.run.smithi078.stdout:(35/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 3.4 MB/s | 264 kB 00:00 2024-03-24T00:07:45.882 INFO:teuthology.orchestra.run.smithi107.stdout:(97/174): libgfortran-11.4.1-3.el9.x86_64.rpm 293 kB/s | 803 kB 00:02 2024-03-24T00:07:45.964 INFO:teuthology.orchestra.run.smithi078.stdout:(36/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 8.0 MB/s | 748 kB 00:00 2024-03-24T00:07:46.040 INFO:teuthology.orchestra.run.smithi078.stdout:(37/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 243 kB/s | 18 kB 00:00 2024-03-24T00:07:46.182 INFO:teuthology.orchestra.run.smithi078.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 4.6 MB/s | 662 kB 00:00 2024-03-24T00:07:46.232 INFO:teuthology.orchestra.run.smithi078.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 18 MB/s | 920 kB 00:00 2024-03-24T00:07:46.265 INFO:teuthology.orchestra.run.smithi078.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-24T00:07:46.307 INFO:teuthology.orchestra.run.smithi078.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-24T00:07:46.341 INFO:teuthology.orchestra.run.smithi078.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 950 kB/s | 32 kB 00:00 2024-03-24T00:07:46.425 INFO:teuthology.orchestra.run.smithi078.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 36 MB/s | 3.0 MB 00:00 2024-03-24T00:07:46.458 INFO:teuthology.orchestra.run.smithi078.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 517 kB/s | 17 kB 00:00 2024-03-24T00:07:46.492 INFO:teuthology.orchestra.run.smithi078.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 302 kB/s | 10 kB 00:00 2024-03-24T00:07:46.525 INFO:teuthology.orchestra.run.smithi078.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.7 MB/s | 159 kB 00:00 2024-03-24T00:07:46.631 INFO:teuthology.orchestra.run.smithi107.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 226 kB/s | 245 kB 00:01 2024-03-24T00:07:46.951 INFO:teuthology.orchestra.run.smithi078.stdout:(47/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 14 MB/s | 48 MB 00:03 2024-03-24T00:07:46.967 INFO:teuthology.orchestra.run.smithi078.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 100 kB/s | 44 kB 00:00 2024-03-24T00:07:46.993 INFO:teuthology.orchestra.run.smithi078.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 16 MB/s | 661 kB 00:00 2024-03-24T00:07:47.227 INFO:teuthology.orchestra.run.smithi078.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.8 MB/s | 578 kB 00:00 2024-03-24T00:07:47.227 INFO:teuthology.orchestra.run.smithi078.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 941 kB/s | 46 kB 00:00 2024-03-24T00:07:47.227 INFO:teuthology.orchestra.run.smithi078.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-24T00:07:47.227 INFO:teuthology.orchestra.run.smithi078.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 2.7 MB/s | 187 kB 00:00 2024-03-24T00:07:47.227 INFO:teuthology.orchestra.run.smithi078.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2024-03-24T00:07:47.227 INFO:teuthology.orchestra.run.smithi078.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 822 kB/s | 34 kB 00:00 2024-03-24T00:07:47.253 INFO:teuthology.orchestra.run.smithi078.stdout:(56/174): ceph-mgr-diskprediction-local-19.0.0- 4.1 MB/s | 7.4 MB 00:01 2024-03-24T00:07:47.329 INFO:teuthology.orchestra.run.smithi078.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 25 MB/s | 4.7 MB 00:00 2024-03-24T00:07:47.388 INFO:teuthology.orchestra.run.smithi078.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 6.5 MB/s | 1.5 MB 00:00 2024-03-24T00:07:47.412 INFO:teuthology.orchestra.run.smithi078.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 5.1 MB/s | 811 kB 00:00 2024-03-24T00:07:47.431 INFO:teuthology.orchestra.run.smithi107.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 160 kB/s | 248 kB 00:01 2024-03-24T00:07:47.438 INFO:teuthology.orchestra.run.smithi078.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.2 MB/s | 134 kB 00:00 2024-03-24T00:07:47.479 INFO:teuthology.orchestra.run.smithi078.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-03-24T00:07:47.580 INFO:teuthology.orchestra.run.smithi078.stdout:(62/174): postgresql-server-13.11-1.el9.x86_64. 34 MB/s | 5.7 MB 00:00 2024-03-24T00:07:47.605 INFO:teuthology.orchestra.run.smithi078.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 464 kB/s | 78 kB 00:00 2024-03-24T00:07:47.697 INFO:teuthology.orchestra.run.smithi078.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 27 MB/s | 5.8 MB 00:00 2024-03-24T00:07:47.714 INFO:teuthology.orchestra.run.smithi078.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 186 kB/s | 24 kB 00:00 2024-03-24T00:07:47.730 INFO:teuthology.orchestra.run.smithi078.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 147 kB/s | 18 kB 00:00 2024-03-24T00:07:47.748 INFO:teuthology.orchestra.run.smithi107.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 194 kB/s | 216 kB 00:01 2024-03-24T00:07:47.756 INFO:teuthology.orchestra.run.smithi078.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-24T00:07:47.781 INFO:teuthology.orchestra.run.smithi078.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 627 kB/s | 32 kB 00:00 2024-03-24T00:07:47.815 INFO:teuthology.orchestra.run.smithi107.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 91 kB/s | 35 kB 00:00 2024-03-24T00:07:47.857 INFO:teuthology.orchestra.run.smithi078.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 4.8 MB 00:00 2024-03-24T00:07:47.882 INFO:teuthology.orchestra.run.smithi078.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.7 MB/s | 217 kB 00:00 2024-03-24T00:07:47.907 INFO:teuthology.orchestra.run.smithi078.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.4 MB/s | 174 kB 00:00 2024-03-24T00:07:47.915 INFO:teuthology.orchestra.run.smithi107.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 267 kB/s | 44 kB 00:00 2024-03-24T00:07:47.932 INFO:teuthology.orchestra.run.smithi078.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 919 kB/s | 69 kB 00:00 2024-03-24T00:07:47.957 INFO:teuthology.orchestra.run.smithi078.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 822 kB/s | 41 kB 00:00 2024-03-24T00:07:47.983 INFO:teuthology.orchestra.run.smithi078.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-24T00:07:48.008 INFO:teuthology.orchestra.run.smithi078.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-24T00:07:48.033 INFO:teuthology.orchestra.run.smithi078.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-24T00:07:48.058 INFO:teuthology.orchestra.run.smithi078.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 630 kB/s | 47 kB 00:00 2024-03-24T00:07:48.074 INFO:teuthology.orchestra.run.smithi107.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 141 kB/s | 22 kB 00:00 2024-03-24T00:07:48.083 INFO:teuthology.orchestra.run.smithi078.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 569 kB/s | 43 kB 00:00 2024-03-24T00:07:48.109 INFO:teuthology.orchestra.run.smithi078.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 775 kB/s | 39 kB 00:00 2024-03-24T00:07:48.149 INFO:teuthology.orchestra.run.smithi107.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 281 kB/s | 94 kB 00:00 2024-03-24T00:07:48.206 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-24T00:07:48.226 INFO:teuthology.orchestra.run.smithi078.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 289 kB/s | 41 kB 00:00 2024-03-24T00:07:48.251 INFO:teuthology.orchestra.run.smithi078.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 2.1 MB/s | 300 kB 00:00 2024-03-24T00:07:48.416 INFO:teuthology.orchestra.run.smithi107.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.5 MB/s | 863 kB 00:00 2024-03-24T00:07:48.418 INFO:teuthology.orchestra.run.smithi078.stdout:(82/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2024-03-24T00:07:48.443 INFO:teuthology.orchestra.run.smithi078.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 262 kB/s | 56 kB 00:00 2024-03-24T00:07:48.468 INFO:teuthology.orchestra.run.smithi078.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 291 kB/s | 63 kB 00:00 2024-03-24T00:07:48.494 INFO:teuthology.orchestra.run.smithi078.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 884 kB/s | 44 kB 00:00 2024-03-24T00:07:48.510 INFO:teuthology.orchestra.run.smithi078.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.2 MB/s | 79 kB 00:00 2024-03-24T00:07:48.536 INFO:teuthology.orchestra.run.smithi078.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-24T00:07:48.561 INFO:teuthology.orchestra.run.smithi078.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-24T00:07:48.603 INFO:teuthology.orchestra.run.smithi078.stdout:(89/174): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.1 MB 00:00 2024-03-24T00:07:48.628 INFO:teuthology.orchestra.run.smithi078.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 124 kB 00:00 2024-03-24T00:07:48.662 INFO:teuthology.orchestra.run.smithi078.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 1.9 MB/s | 187 kB 00:00 2024-03-24T00:07:48.683 INFO:teuthology.orchestra.run.smithi107.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.0 MB/s | 548 kB 00:00 2024-03-24T00:07:48.775 INFO:teuthology.orchestra.run.smithi107.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 212 kB/s | 19 kB 00:00 2024-03-24T00:07:48.829 INFO:teuthology.orchestra.run.smithi078.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 937 kB/s | 187 kB 00:00 2024-03-24T00:07:48.854 INFO:teuthology.orchestra.run.smithi078.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-03-24T00:07:48.875 INFO:teuthology.orchestra.run.smithi107.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 673 kB/s | 309 kB 00:00 2024-03-24T00:07:48.887 INFO:teuthology.orchestra.run.smithi078.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-03-24T00:07:48.904 INFO:teuthology.orchestra.run.smithi078.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 428 kB/s | 21 kB 00:00 2024-03-24T00:07:48.938 INFO:teuthology.orchestra.run.smithi078.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 508 kB/s | 25 kB 00:00 2024-03-24T00:07:48.975 INFO:teuthology.orchestra.run.smithi107.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-03-24T00:07:48.980 INFO:teuthology.orchestra.run.smithi078.stdout:(97/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.2 MB/s | 803 kB 00:00 2024-03-24T00:07:49.030 INFO:teuthology.orchestra.run.smithi078.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.6 MB/s | 245 kB 00:00 2024-03-24T00:07:49.075 INFO:teuthology.orchestra.run.smithi107.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 487 kB/s | 49 kB 00:00 2024-03-24T00:07:49.088 INFO:teuthology.orchestra.run.smithi078.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.2 MB/s | 248 kB 00:00 2024-03-24T00:07:49.134 INFO:teuthology.orchestra.run.smithi107.stdout:(111/174): libarrow-9.0.0-10.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-03-24T00:07:49.147 INFO:teuthology.orchestra.run.smithi078.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 1.8 MB/s | 216 kB 00:00 2024-03-24T00:07:49.164 INFO:teuthology.orchestra.run.smithi078.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 464 kB/s | 35 kB 00:00 2024-03-24T00:07:49.167 INFO:teuthology.orchestra.run.smithi107.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 733 kB/s | 67 kB 00:00 2024-03-24T00:07:49.206 INFO:teuthology.orchestra.run.smithi078.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 758 kB/s | 44 kB 00:00 2024-03-24T00:07:49.232 INFO:teuthology.orchestra.run.smithi078.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.4 MB/s | 94 kB 00:00 2024-03-24T00:07:49.234 INFO:teuthology.orchestra.run.smithi107.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-03-24T00:07:49.343 INFO:teuthology.orchestra.run.smithi107.stdout:(114/174): python3-asyncssh-2.13.2-5.el9.noarch 4.9 MB/s | 548 kB 00:00 2024-03-24T00:07:49.385 INFO:teuthology.orchestra.run.smithi107.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.8 MB/s | 838 kB 00:00 2024-03-24T00:07:49.435 INFO:teuthology.orchestra.run.smithi107.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 472 kB/s | 43 kB 00:00 2024-03-24T00:07:49.478 INFO:teuthology.orchestra.run.smithi107.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 350 kB/s | 32 kB 00:00 2024-03-24T00:07:49.507 INFO:teuthology.orchestra.run.smithi078.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 74 kB/s | 22 kB 00:00 2024-03-24T00:07:49.528 INFO:teuthology.orchestra.run.smithi107.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 163 kB/s | 15 kB 00:00 2024-03-24T00:07:49.578 INFO:teuthology.orchestra.run.smithi107.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-03-24T00:07:49.615 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 16 kB 00:00 2024-03-24T00:07:49.628 INFO:teuthology.orchestra.run.smithi107.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-03-24T00:07:49.679 INFO:teuthology.orchestra.run.smithi107.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.9 MB/s | 303 kB 00:00 2024-03-24T00:07:49.720 INFO:teuthology.orchestra.run.smithi107.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 384 kB/s | 35 kB 00:00 2024-03-24T00:07:49.779 INFO:teuthology.orchestra.run.smithi107.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 2.0 MB/s | 203 kB 00:00 2024-03-24T00:07:49.832 INFO:teuthology.orchestra.run.smithi078.stdout:(105/174): openssl-devel-3.0.7-27.el9.x86_64.rp 4.5 MB/s | 4.1 MB 00:00 2024-03-24T00:07:49.879 INFO:teuthology.orchestra.run.smithi107.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-03-24T00:07:49.949 INFO:teuthology.orchestra.run.smithi078.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.2 MB/s | 863 kB 00:00 2024-03-24T00:07:49.971 INFO:teuthology.orchestra.run.smithi107.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 116 kB/s | 11 kB 00:00 2024-03-24T00:07:50.072 INFO:teuthology.orchestra.run.smithi107.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 177 kB/s | 18 kB 00:00 2024-03-24T00:07:50.163 INFO:teuthology.orchestra.run.smithi107.stdout:(127/174): python3-jaraco-collections-3.0.0-8.e 253 kB/s | 23 kB 00:00 2024-03-24T00:07:50.255 INFO:teuthology.orchestra.run.smithi107.stdout:(128/174): python3-jaraco-functools-3.5.0-2.el9 212 kB/s | 19 kB 00:00 2024-03-24T00:07:50.266 INFO:teuthology.orchestra.run.smithi078.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 722 kB/s | 548 kB 00:00 2024-03-24T00:07:50.349 INFO:teuthology.orchestra.run.smithi078.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 52 kB/s | 19 kB 00:00 2024-03-24T00:07:50.356 INFO:teuthology.orchestra.run.smithi107.stdout:(129/174): python3-jaraco-text-3.2.0-6.el9.noar 197 kB/s | 20 kB 00:00 2024-03-24T00:07:50.472 INFO:teuthology.orchestra.run.smithi107.stdout:(130/174): python3-kubernetes-26.1.0-2.el9.noar 8.8 MB/s | 1.0 MB 00:00 2024-03-24T00:07:50.564 INFO:teuthology.orchestra.run.smithi107.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 505 kB/s | 46 kB 00:00 2024-03-24T00:07:50.583 INFO:teuthology.orchestra.run.smithi078.stdout:(109/174): gperftools-libs-2.9.1-2.el9.x86_64.r 411 kB/s | 309 kB 00:00 2024-03-24T00:07:50.600 INFO:teuthology.orchestra.run.smithi078.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 104 kB/s | 26 kB 00:00 2024-03-24T00:07:50.665 INFO:teuthology.orchestra.run.smithi107.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 791 kB/s | 79 kB 00:00 2024-03-24T00:07:50.725 INFO:teuthology.orchestra.run.smithi078.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 343 kB/s | 49 kB 00:00 2024-03-24T00:07:50.756 INFO:teuthology.orchestra.run.smithi107.stdout:(133/174): python3-natsort-7.1.1-5.el9.noarch.r 629 kB/s | 58 kB 00:00 2024-03-24T00:07:50.792 INFO:teuthology.orchestra.run.smithi078.stdout:(112/174): libarrow-9.0.0-10.el9.x86_64.rpm 8.4 MB/s | 4.4 MB 00:00 2024-03-24T00:07:50.826 INFO:teuthology.orchestra.run.smithi078.stdout:(113/174): libunwind-1.6.2-1.el9.x86_64.rpm 299 kB/s | 67 kB 00:00 2024-03-24T00:07:50.848 INFO:teuthology.orchestra.run.smithi107.stdout:(134/174): python3-grpcio-1.46.7-10.el9.x86_64. 1.8 MB/s | 2.0 MB 00:01 2024-03-24T00:07:50.860 INFO:teuthology.orchestra.run.smithi078.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 1.1 MB/s | 151 kB 00:00 2024-03-24T00:07:50.873 INFO:teuthology.orchestra.run.smithi107.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-03-24T00:07:50.941 INFO:teuthology.orchestra.run.smithi107.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 179 kB/s | 16 kB 00:00 2024-03-24T00:07:50.960 INFO:teuthology.orchestra.run.smithi078.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.9 MB/s | 838 kB 00:00 2024-03-24T00:07:50.974 INFO:teuthology.orchestra.run.smithi107.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 898 kB/s | 90 kB 00:00 2024-03-24T00:07:50.985 INFO:teuthology.orchestra.run.smithi078.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 347 kB/s | 43 kB 00:00 2024-03-24T00:07:51.033 INFO:teuthology.orchestra.run.smithi107.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 307 kB/s | 28 kB 00:00 2024-03-24T00:07:51.067 INFO:teuthology.orchestra.run.smithi107.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 334 kB/s | 31 kB 00:00 2024-03-24T00:07:51.068 INFO:teuthology.orchestra.run.smithi078.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 297 kB/s | 32 kB 00:00 2024-03-24T00:07:51.094 INFO:teuthology.orchestra.run.smithi078.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 138 kB/s | 15 kB 00:00 2024-03-24T00:07:51.133 INFO:teuthology.orchestra.run.smithi107.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-03-24T00:07:51.159 INFO:teuthology.orchestra.run.smithi107.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 643 kB/s | 59 kB 00:00 2024-03-24T00:07:51.177 INFO:teuthology.orchestra.run.smithi078.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-03-24T00:07:51.227 INFO:teuthology.orchestra.run.smithi078.stdout:(120/174): python3-asyncssh-2.13.2-5.el9.noarch 1.3 MB/s | 548 kB 00:00 2024-03-24T00:07:51.234 INFO:teuthology.orchestra.run.smithi107.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 357 kB/s | 36 kB 00:00 2024-03-24T00:07:51.259 INFO:teuthology.orchestra.run.smithi107.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 509 kB/s | 51 kB 00:00 2024-03-24T00:07:51.278 INFO:teuthology.orchestra.run.smithi078.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 1.9 MB/s | 358 kB 00:00 2024-03-24T00:07:51.311 INFO:teuthology.orchestra.run.smithi078.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.2 MB/s | 303 kB 00:00 2024-03-24T00:07:51.326 INFO:teuthology.orchestra.run.smithi107.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-03-24T00:07:51.328 INFO:teuthology.orchestra.run.smithi078.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 351 kB/s | 35 kB 00:00 2024-03-24T00:07:51.360 INFO:teuthology.orchestra.run.smithi107.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 894 kB/s | 90 kB 00:00 2024-03-24T00:07:51.420 INFO:teuthology.orchestra.run.smithi078.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 1.4 MB/s | 203 kB 00:00 2024-03-24T00:07:51.427 INFO:teuthology.orchestra.run.smithi107.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 4.2 MB/s | 427 kB 00:00 2024-03-24T00:07:51.460 INFO:teuthology.orchestra.run.smithi107.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 199 kB/s | 20 kB 00:00 2024-03-24T00:07:51.487 INFO:teuthology.orchestra.run.smithi078.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 12 MB/s | 2.0 MB 00:00 2024-03-24T00:07:51.512 INFO:teuthology.orchestra.run.smithi078.stdout:(126/174): python3-grpcio-tools-1.46.7-10.el9.x 786 kB/s | 144 kB 00:00 2024-03-24T00:07:51.527 INFO:teuthology.orchestra.run.smithi107.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-24T00:07:51.529 INFO:teuthology.orchestra.run.smithi078.stdout:(127/174): python3-jaraco-8.2.1-3.el9.noarch.rp 98 kB/s | 11 kB 00:00 2024-03-24T00:07:51.588 INFO:teuthology.orchestra.run.smithi078.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 176 kB/s | 18 kB 00:00 2024-03-24T00:07:51.594 INFO:teuthology.orchestra.run.smithi107.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-03-24T00:07:51.613 INFO:teuthology.orchestra.run.smithi078.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 230 kB/s | 23 kB 00:00 2024-03-24T00:07:51.655 INFO:teuthology.orchestra.run.smithi078.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 155 kB/s | 19 kB 00:00 2024-03-24T00:07:51.689 INFO:teuthology.orchestra.run.smithi078.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 196 kB/s | 20 kB 00:00 2024-03-24T00:07:51.719 INFO:teuthology.orchestra.run.smithi106.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:01 2024-03-24T00:07:51.756 INFO:teuthology.orchestra.run.smithi078.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 463 kB/s | 46 kB 00:00 2024-03-24T00:07:51.798 INFO:teuthology.orchestra.run.smithi078.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 726 kB/s | 79 kB 00:00 2024-03-24T00:07:51.856 INFO:teuthology.orchestra.run.smithi078.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 575 kB/s | 58 kB 00:00 2024-03-24T00:07:51.886 INFO:teuthology.orchestra.run.smithi107.stdout:(150/174): librados2-19.0.0-2145.g3be9a5c9.el9. 9.5 MB/s | 3.4 MB 00:00 2024-03-24T00:07:51.890 INFO:teuthology.orchestra.run.smithi078.stdout:(135/174): python3-kubernetes-26.1.0-2.el9.noar 3.7 MB/s | 1.0 MB 00:00 2024-03-24T00:07:51.915 INFO:teuthology.orchestra.run.smithi078.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-03-24T00:07:51.936 INFO:teuthology.orchestra.run.smithi107.stdout:(151/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 9.2 MB/s | 3.1 MB 00:00 2024-03-24T00:07:51.957 INFO:teuthology.orchestra.run.smithi078.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 163 kB/s | 16 kB 00:00 2024-03-24T00:07:51.962 INFO:teuthology.orchestra.run.smithi107.stdout:(152/174): xfsdump-3.1.12-4.el9.x86_64.rpm 4.2 MB/s | 321 kB 00:00 2024-03-24T00:07:51.999 INFO:teuthology.orchestra.run.smithi078.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 827 kB/s | 90 kB 00:00 2024-03-24T00:07:52.016 INFO:teuthology.orchestra.run.smithi078.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 281 kB/s | 28 kB 00:00 2024-03-24T00:07:52.058 INFO:teuthology.orchestra.run.smithi078.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 306 kB/s | 31 kB 00:00 2024-03-24T00:07:52.117 INFO:teuthology.orchestra.run.smithi078.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-03-24T00:07:52.133 INFO:teuthology.orchestra.run.smithi078.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 504 kB/s | 59 kB 00:00 2024-03-24T00:07:52.159 INFO:teuthology.orchestra.run.smithi078.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 356 kB/s | 36 kB 00:00 2024-03-24T00:07:52.217 INFO:teuthology.orchestra.run.smithi078.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 508 kB/s | 51 kB 00:00 2024-03-24T00:07:52.243 INFO:teuthology.orchestra.run.smithi078.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-03-24T00:07:52.276 INFO:teuthology.orchestra.run.smithi078.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 766 kB/s | 90 kB 00:00 2024-03-24T00:07:52.343 INFO:teuthology.orchestra.run.smithi078.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 199 kB/s | 20 kB 00:00 2024-03-24T00:07:52.368 INFO:teuthology.orchestra.run.smithi078.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 2.8 MB/s | 427 kB 00:00 2024-03-24T00:07:52.402 INFO:teuthology.orchestra.run.smithi078.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2024-03-24T00:07:52.527 INFO:teuthology.orchestra.run.smithi078.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 8.6 MB/s | 1.6 MB 00:00 2024-03-24T00:07:52.569 INFO:teuthology.orchestra.run.smithi078.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-24T00:07:52.728 INFO:teuthology.orchestra.run.smithi078.stdout:(152/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 9.7 MB/s | 3.1 MB 00:00 2024-03-24T00:07:52.936 INFO:teuthology.orchestra.run.smithi078.stdout:(153/174): librados2-19.0.0-2145.g3be9a5c9.el9. 6.0 MB/s | 3.4 MB 00:00 2024-03-24T00:07:52.995 INFO:teuthology.orchestra.run.smithi078.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 723 kB/s | 42 kB 00:00 2024-03-24T00:07:53.078 INFO:teuthology.orchestra.run.smithi078.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 2.9 MB/s | 248 kB 00:00 2024-03-24T00:07:53.145 INFO:teuthology.orchestra.run.smithi078.stdout:(156/174): NetworkManager-libnm-1.47.2-1.el9.x8 4.4 MB/s | 1.8 MB 00:00 2024-03-24T00:07:53.170 INFO:teuthology.orchestra.run.smithi078.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 777 kB/s | 71 kB 00:00 2024-03-24T00:07:53.196 INFO:teuthology.orchestra.run.smithi078.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 726 kB/s | 36 kB 00:00 2024-03-24T00:07:53.271 INFO:teuthology.orchestra.run.smithi078.stdout:(159/174): NetworkManager-1.47.2-1.el9.x86_64.r 3.3 MB/s | 2.3 MB 00:00 2024-03-24T00:07:53.296 INFO:teuthology.orchestra.run.smithi078.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.5 MB/s | 192 kB 00:00 2024-03-24T00:07:53.321 INFO:teuthology.orchestra.run.smithi078.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.0 MB/s | 256 kB 00:00 2024-03-24T00:07:53.347 INFO:teuthology.orchestra.run.smithi078.stdout:(162/174): libacl-2.3.1-4.el9.x86_64.rpm 306 kB/s | 23 kB 00:00 2024-03-24T00:07:53.372 INFO:teuthology.orchestra.run.smithi078.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 108 kB 00:00 2024-03-24T00:07:53.397 INFO:teuthology.orchestra.run.smithi078.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.0 MB/s | 155 kB 00:00 2024-03-24T00:07:53.423 INFO:teuthology.orchestra.run.smithi078.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 136 kB 00:00 2024-03-24T00:07:53.456 INFO:teuthology.orchestra.run.smithi078.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 761 kB/s | 64 kB 00:00 2024-03-24T00:07:53.473 INFO:teuthology.orchestra.run.smithi078.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 379 kB/s | 29 kB 00:00 2024-03-24T00:07:53.724 INFO:teuthology.orchestra.run.smithi078.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 4.0 MB/s | 1.2 MB 00:00 2024-03-24T00:07:53.799 INFO:teuthology.orchestra.run.smithi078.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 6.3 MB/s | 2.2 MB 00:00 2024-03-24T00:07:53.849 INFO:teuthology.orchestra.run.smithi078.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 624 kB/s | 31 kB 00:00 2024-03-24T00:07:53.883 INFO:teuthology.orchestra.run.smithi078.stdout:(171/174): util-linux-core-2.37.4-18.el9.x86_64 2.9 MB/s | 465 kB 00:00 2024-03-24T00:07:53.941 INFO:teuthology.orchestra.run.smithi078.stdout:(172/174): util-linux-2.37.4-18.el9.x86_64.rpm 4.9 MB/s | 2.3 MB 00:00 2024-03-24T00:07:53.966 INFO:teuthology.orchestra.run.smithi078.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 890 kB/s | 74 kB 00:00 2024-03-24T00:07:54.033 INFO:teuthology.orchestra.run.smithi078.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 5.8 MB/s | 1.1 MB 00:00 2024-03-24T00:07:54.040 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-03-24T00:07:54.041 INFO:teuthology.orchestra.run.smithi078.stdout:Total 16 MB/s | 224 MB 00:14 2024-03-24T00:07:54.524 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-03-24T00:07:54.797 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-03-24T00:07:54.797 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-03-24T00:07:56.675 INFO:teuthology.orchestra.run.smithi106.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-24T00:07:56.783 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-03-24T00:07:56.784 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-03-24T00:07:58.997 INFO:teuthology.orchestra.run.smithi106.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T00:07:58.998 INFO:teuthology.orchestra.run.smithi106.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T00:07:59.011 INFO:teuthology.orchestra.run.smithi106.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-24T00:07:59.012 INFO:teuthology.orchestra.run.smithi106.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-24T00:07:59.013 INFO:teuthology.orchestra.run.smithi106.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-24T00:07:59.014 INFO:teuthology.orchestra.run.smithi106.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-24T00:07:59.016 INFO:teuthology.orchestra.run.smithi106.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-24T00:07:59.019 INFO:teuthology.orchestra.run.smithi106.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-24T00:07:59.020 INFO:teuthology.orchestra.run.smithi106.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-24T00:07:59.098 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-03-24T00:07:59.115 INFO:teuthology.orchestra.run.smithi106.stdout:============================================================================================ 2024-03-24T00:07:59.116 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repository Size 2024-03-24T00:07:59.116 INFO:teuthology.orchestra.run.smithi106.stdout:============================================================================================ 2024-03-24T00:07:59.116 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-03-24T00:07:59.116 INFO:teuthology.orchestra.run.smithi106.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-24T00:07:59.116 INFO:teuthology.orchestra.run.smithi106.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-24T00:07:59.116 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-24T00:07:59.117 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-24T00:07:59.117 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-24T00:07:59.117 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-24T00:07:59.117 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-24T00:07:59.117 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-24T00:07:59.118 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-24T00:07:59.118 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-24T00:07:59.118 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-24T00:07:59.118 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-24T00:07:59.118 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-24T00:07:59.118 INFO:teuthology.orchestra.run.smithi106.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-24T00:07:59.118 INFO:teuthology.orchestra.run.smithi106.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-24T00:07:59.118 INFO:teuthology.orchestra.run.smithi106.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-24T00:07:59.119 INFO:teuthology.orchestra.run.smithi106.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-24T00:07:59.119 INFO:teuthology.orchestra.run.smithi106.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-24T00:07:59.119 INFO:teuthology.orchestra.run.smithi106.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-24T00:07:59.119 INFO:teuthology.orchestra.run.smithi106.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-24T00:07:59.119 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-24T00:07:59.119 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-24T00:07:59.119 INFO:teuthology.orchestra.run.smithi106.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-24T00:07:59.119 INFO:teuthology.orchestra.run.smithi106.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-24T00:07:59.119 INFO:teuthology.orchestra.run.smithi106.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-24T00:07:59.120 INFO:teuthology.orchestra.run.smithi106.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-24T00:07:59.120 INFO:teuthology.orchestra.run.smithi106.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-24T00:07:59.120 INFO:teuthology.orchestra.run.smithi106.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-24T00:07:59.120 INFO:teuthology.orchestra.run.smithi106.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-24T00:07:59.120 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-24T00:07:59.120 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-24T00:07:59.120 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-24T00:07:59.120 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-24T00:07:59.121 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-24T00:07:59.121 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-24T00:07:59.121 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-24T00:07:59.121 INFO:teuthology.orchestra.run.smithi106.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-24T00:07:59.121 INFO:teuthology.orchestra.run.smithi106.stdout:Upgrading: 2024-03-24T00:07:59.121 INFO:teuthology.orchestra.run.smithi106.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-24T00:07:59.121 INFO:teuthology.orchestra.run.smithi106.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-24T00:07:59.121 INFO:teuthology.orchestra.run.smithi106.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-24T00:07:59.122 INFO:teuthology.orchestra.run.smithi106.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-24T00:07:59.122 INFO:teuthology.orchestra.run.smithi106.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-24T00:07:59.122 INFO:teuthology.orchestra.run.smithi106.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-24T00:07:59.122 INFO:teuthology.orchestra.run.smithi106.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-24T00:07:59.122 INFO:teuthology.orchestra.run.smithi106.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-24T00:07:59.122 INFO:teuthology.orchestra.run.smithi106.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-24T00:07:59.122 INFO:teuthology.orchestra.run.smithi106.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-24T00:07:59.122 INFO:teuthology.orchestra.run.smithi106.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-24T00:07:59.122 INFO:teuthology.orchestra.run.smithi106.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-24T00:07:59.123 INFO:teuthology.orchestra.run.smithi106.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-24T00:07:59.123 INFO:teuthology.orchestra.run.smithi106.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-24T00:07:59.123 INFO:teuthology.orchestra.run.smithi106.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-24T00:07:59.123 INFO:teuthology.orchestra.run.smithi106.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-24T00:07:59.123 INFO:teuthology.orchestra.run.smithi106.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-24T00:07:59.123 INFO:teuthology.orchestra.run.smithi106.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-24T00:07:59.123 INFO:teuthology.orchestra.run.smithi106.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-24T00:07:59.123 INFO:teuthology.orchestra.run.smithi106.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-24T00:07:59.123 INFO:teuthology.orchestra.run.smithi106.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-24T00:07:59.124 INFO:teuthology.orchestra.run.smithi106.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-24T00:07:59.124 INFO:teuthology.orchestra.run.smithi106.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-24T00:07:59.124 INFO:teuthology.orchestra.run.smithi106.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-24T00:07:59.124 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-03-24T00:07:59.124 INFO:teuthology.orchestra.run.smithi106.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T00:07:59.124 INFO:teuthology.orchestra.run.smithi106.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-24T00:07:59.124 INFO:teuthology.orchestra.run.smithi106.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T00:07:59.124 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-24T00:07:59.124 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-24T00:07:59.124 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-24T00:07:59.125 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-24T00:07:59.125 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-24T00:07:59.125 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-24T00:07:59.125 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-24T00:07:59.125 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-24T00:07:59.125 INFO:teuthology.orchestra.run.smithi106.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T00:07:59.125 INFO:teuthology.orchestra.run.smithi106.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T00:07:59.125 INFO:teuthology.orchestra.run.smithi106.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T00:07:59.126 INFO:teuthology.orchestra.run.smithi106.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T00:07:59.126 INFO:teuthology.orchestra.run.smithi106.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T00:07:59.126 INFO:teuthology.orchestra.run.smithi106.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T00:07:59.126 INFO:teuthology.orchestra.run.smithi106.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T00:07:59.126 INFO:teuthology.orchestra.run.smithi106.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T00:07:59.126 INFO:teuthology.orchestra.run.smithi106.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T00:07:59.126 INFO:teuthology.orchestra.run.smithi106.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-24T00:07:59.126 INFO:teuthology.orchestra.run.smithi106.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T00:07:59.126 INFO:teuthology.orchestra.run.smithi106.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T00:07:59.126 INFO:teuthology.orchestra.run.smithi106.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T00:07:59.127 INFO:teuthology.orchestra.run.smithi106.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T00:07:59.127 INFO:teuthology.orchestra.run.smithi106.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T00:07:59.127 INFO:teuthology.orchestra.run.smithi106.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-24T00:07:59.127 INFO:teuthology.orchestra.run.smithi106.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T00:07:59.127 INFO:teuthology.orchestra.run.smithi106.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-24T00:07:59.127 INFO:teuthology.orchestra.run.smithi106.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T00:07:59.127 INFO:teuthology.orchestra.run.smithi106.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-24T00:07:59.127 INFO:teuthology.orchestra.run.smithi106.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T00:07:59.128 INFO:teuthology.orchestra.run.smithi106.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T00:07:59.128 INFO:teuthology.orchestra.run.smithi106.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T00:07:59.128 INFO:teuthology.orchestra.run.smithi106.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T00:07:59.128 INFO:teuthology.orchestra.run.smithi106.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T00:07:59.128 INFO:teuthology.orchestra.run.smithi106.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T00:07:59.128 INFO:teuthology.orchestra.run.smithi106.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T00:07:59.128 INFO:teuthology.orchestra.run.smithi106.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T00:07:59.128 INFO:teuthology.orchestra.run.smithi106.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-24T00:07:59.128 INFO:teuthology.orchestra.run.smithi106.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T00:07:59.129 INFO:teuthology.orchestra.run.smithi106.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T00:07:59.129 INFO:teuthology.orchestra.run.smithi106.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T00:07:59.129 INFO:teuthology.orchestra.run.smithi106.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-24T00:07:59.129 INFO:teuthology.orchestra.run.smithi106.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T00:07:59.129 INFO:teuthology.orchestra.run.smithi106.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T00:07:59.129 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T00:07:59.129 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-24T00:07:59.129 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-24T00:07:59.129 INFO:teuthology.orchestra.run.smithi106.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T00:07:59.129 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T00:07:59.129 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T00:07:59.129 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T00:07:59.130 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-24T00:07:59.130 INFO:teuthology.orchestra.run.smithi106.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-24T00:07:59.130 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T00:07:59.130 INFO:teuthology.orchestra.run.smithi106.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T00:07:59.130 INFO:teuthology.orchestra.run.smithi106.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T00:07:59.130 INFO:teuthology.orchestra.run.smithi106.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T00:07:59.130 INFO:teuthology.orchestra.run.smithi106.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T00:07:59.130 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T00:07:59.130 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T00:07:59.130 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T00:07:59.130 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T00:07:59.131 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T00:07:59.131 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T00:07:59.131 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T00:07:59.131 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T00:07:59.131 INFO:teuthology.orchestra.run.smithi106.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T00:07:59.131 INFO:teuthology.orchestra.run.smithi106.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T00:07:59.131 INFO:teuthology.orchestra.run.smithi106.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T00:07:59.131 INFO:teuthology.orchestra.run.smithi106.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T00:07:59.131 INFO:teuthology.orchestra.run.smithi106.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T00:07:59.131 INFO:teuthology.orchestra.run.smithi106.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T00:07:59.131 INFO:teuthology.orchestra.run.smithi106.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T00:07:59.131 INFO:teuthology.orchestra.run.smithi106.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T00:07:59.132 INFO:teuthology.orchestra.run.smithi106.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T00:07:59.132 INFO:teuthology.orchestra.run.smithi106.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T00:07:59.132 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T00:07:59.132 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T00:07:59.132 INFO:teuthology.orchestra.run.smithi106.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T00:07:59.132 INFO:teuthology.orchestra.run.smithi106.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T00:07:59.132 INFO:teuthology.orchestra.run.smithi106.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T00:07:59.132 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T00:07:59.132 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T00:07:59.132 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T00:07:59.133 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T00:07:59.133 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-24T00:07:59.133 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T00:07:59.133 INFO:teuthology.orchestra.run.smithi106.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T00:07:59.133 INFO:teuthology.orchestra.run.smithi106.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T00:07:59.134 INFO:teuthology.orchestra.run.smithi106.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T00:07:59.134 INFO:teuthology.orchestra.run.smithi106.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T00:07:59.134 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T00:07:59.134 INFO:teuthology.orchestra.run.smithi106.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T00:07:59.134 INFO:teuthology.orchestra.run.smithi106.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T00:07:59.134 INFO:teuthology.orchestra.run.smithi106.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T00:07:59.134 INFO:teuthology.orchestra.run.smithi106.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T00:07:59.134 INFO:teuthology.orchestra.run.smithi106.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T00:07:59.134 INFO:teuthology.orchestra.run.smithi106.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-24T00:07:59.134 INFO:teuthology.orchestra.run.smithi106.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T00:07:59.134 INFO:teuthology.orchestra.run.smithi106.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T00:07:59.134 INFO:teuthology.orchestra.run.smithi106.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T00:07:59.135 INFO:teuthology.orchestra.run.smithi106.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T00:07:59.135 INFO:teuthology.orchestra.run.smithi106.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T00:07:59.135 INFO:teuthology.orchestra.run.smithi106.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T00:07:59.135 INFO:teuthology.orchestra.run.smithi106.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T00:07:59.135 INFO:teuthology.orchestra.run.smithi106.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T00:07:59.135 INFO:teuthology.orchestra.run.smithi106.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T00:07:59.135 INFO:teuthology.orchestra.run.smithi106.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-24T00:07:59.135 INFO:teuthology.orchestra.run.smithi106.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T00:07:59.135 INFO:teuthology.orchestra.run.smithi106.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-24T00:07:59.135 INFO:teuthology.orchestra.run.smithi106.stdout:Installing weak dependencies: 2024-03-24T00:07:59.135 INFO:teuthology.orchestra.run.smithi106.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T00:07:59.135 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-24T00:07:59.136 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-03-24T00:07:59.136 INFO:teuthology.orchestra.run.smithi106.stdout:============================================================================================ 2024-03-24T00:07:59.136 INFO:teuthology.orchestra.run.smithi106.stdout:Install 150 Packages 2024-03-24T00:07:59.136 INFO:teuthology.orchestra.run.smithi106.stdout:Upgrade 24 Packages 2024-03-24T00:07:59.136 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-24T00:07:59.137 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 224 M 2024-03-24T00:07:59.137 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-03-24T00:07:59.894 INFO:teuthology.orchestra.run.smithi107.stdout:(153/174): NetworkManager-libnm-1.47.2-1.el9.x8 237 kB/s | 1.8 MB 00:07 2024-03-24T00:07:59.960 INFO:teuthology.orchestra.run.smithi106.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 40 kB/s | 6.6 kB 00:00 2024-03-24T00:08:00.128 INFO:teuthology.orchestra.run.smithi107.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 181 kB/s | 42 kB 00:00 2024-03-24T00:08:00.301 INFO:teuthology.orchestra.run.smithi106.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 2.5 MB/s | 890 kB 00:00 2024-03-24T00:08:00.378 INFO:teuthology.orchestra.run.smithi107.stdout:(155/174): NetworkManager-1.47.2-1.el9.x86_64.r 280 kB/s | 2.3 MB 00:08 2024-03-24T00:08:00.385 INFO:teuthology.orchestra.run.smithi106.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 1.7 MB/s | 145 kB 00:00 2024-03-24T00:08:00.461 INFO:teuthology.orchestra.run.smithi106.stdout:(4/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 8.0 MB/s | 5.4 MB 00:00 2024-03-24T00:08:00.619 INFO:teuthology.orchestra.run.smithi106.stdout:(5/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 9.4 MB/s | 1.5 MB 00:00 2024-03-24T00:08:00.820 INFO:teuthology.orchestra.run.smithi107.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 162 kB/s | 71 kB 00:00 2024-03-24T00:08:01.012 INFO:teuthology.orchestra.run.smithi107.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 190 kB/s | 36 kB 00:00 2024-03-24T00:08:01.053 INFO:teuthology.orchestra.run.smithi106.stdout:(6/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 11 MB/s | 4.7 MB 00:00 2024-03-24T00:08:01.103 INFO:teuthology.orchestra.run.smithi106.stdout:(7/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 3.2 MB/s | 2.3 MB 00:00 2024-03-24T00:08:01.303 INFO:teuthology.orchestra.run.smithi107.stdout:(158/174): NetworkManager-tui-1.47.2-1.el9.x86_ 211 kB/s | 248 kB 00:01 2024-03-24T00:08:01.662 INFO:teuthology.orchestra.run.smithi107.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 296 kB/s | 192 kB 00:00 2024-03-24T00:08:01.720 INFO:teuthology.orchestra.run.smithi106.stdout:(8/174): ceph-common-19.0.0-2145.g3be9a5c9.el9. 11 MB/s | 20 MB 00:01 2024-03-24T00:08:01.778 INFO:teuthology.orchestra.run.smithi106.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 433 kB/s | 25 kB 00:00 2024-03-24T00:08:01.820 INFO:teuthology.orchestra.run.smithi107.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 146 kB/s | 23 kB 00:00 2024-03-24T00:08:02.162 INFO:teuthology.orchestra.run.smithi107.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 317 kB/s | 108 kB 00:00 2024-03-24T00:08:02.187 INFO:teuthology.orchestra.run.smithi107.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 289 kB/s | 256 kB 00:00 2024-03-24T00:08:02.338 INFO:teuthology.orchestra.run.smithi107.stdout:(163/174): openssl-devel-3.0.7-27.el9.x86_64.rp 249 kB/s | 4.1 MB 00:17 2024-03-24T00:08:02.672 INFO:teuthology.orchestra.run.smithi107.stdout:(164/174): libsmartcols-2.37.4-18.el9.x86_64.rp 192 kB/s | 64 kB 00:00 2024-03-24T00:08:02.704 INFO:teuthology.orchestra.run.smithi106.stdout:(10/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x8 10 MB/s | 17 MB 00:01 2024-03-24T00:08:02.771 INFO:teuthology.orchestra.run.smithi106.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 3.2 MB/s | 220 kB 00:00 2024-03-24T00:08:02.805 INFO:teuthology.orchestra.run.smithi107.stdout:(165/174): libuuid-2.37.4-18.el9.x86_64.rpm 215 kB/s | 29 kB 00:00 2024-03-24T00:08:02.831 INFO:teuthology.orchestra.run.smithi107.stdout:(166/174): libmount-2.37.4-18.el9.x86_64.rpm 211 kB/s | 136 kB 00:00 2024-03-24T00:08:02.838 INFO:teuthology.orchestra.run.smithi106.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 407 kB/s | 27 kB 00:00 2024-03-24T00:08:02.856 INFO:teuthology.orchestra.run.smithi107.stdout:(167/174): libfdisk-2.37.4-18.el9.x86_64.rpm 223 kB/s | 155 kB 00:00 2024-03-24T00:08:02.904 INFO:teuthology.orchestra.run.smithi106.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 469 kB/s | 31 kB 00:00 2024-03-24T00:08:03.038 INFO:teuthology.orchestra.run.smithi106.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 5.5 MB/s | 746 kB 00:00 2024-03-24T00:08:03.113 INFO:teuthology.orchestra.run.smithi106.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 2.1 MB/s | 163 kB 00:00 2024-03-24T00:08:03.188 INFO:teuthology.orchestra.run.smithi106.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.7 MB/s | 127 kB 00:00 2024-03-24T00:08:03.280 INFO:teuthology.orchestra.run.smithi106.stdout:(17/174): libradosstriper1-19.0.0-2145.g3be9a5c 5.4 MB/s | 501 kB 00:00 2024-03-24T00:08:03.422 INFO:teuthology.orchestra.run.smithi106.stdout:(18/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el 4.3 MB/s | 9.9 MB 00:02 2024-03-24T00:08:03.506 INFO:teuthology.orchestra.run.smithi106.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 542 kB/s | 45 kB 00:00 2024-03-24T00:08:03.589 INFO:teuthology.orchestra.run.smithi106.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 1.5 MB/s | 131 kB 00:00 2024-03-24T00:08:03.674 INFO:teuthology.orchestra.run.smithi106.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 1.9 MB/s | 162 kB 00:00 2024-03-24T00:08:03.741 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-03-24T00:08:03.790 INFO:teuthology.orchestra.run.smithi106.stdout:(22/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 10 MB/s | 5.1 MB 00:00 2024-03-24T00:08:03.824 INFO:teuthology.orchestra.run.smithi106.stdout:(23/174): python3-rados-19.0.0-2145.g3be9a5c9.e 2.1 MB/s | 321 kB 00:00 2024-03-24T00:08:03.857 INFO:teuthology.orchestra.run.smithi106.stdout:(24/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 4.4 MB/s | 298 kB 00:00 2024-03-24T00:08:03.878 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-24T00:08:03.890 INFO:teuthology.orchestra.run.smithi106.stdout:(25/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 1.5 MB/s | 99 kB 00:00 2024-03-24T00:08:03.916 INFO:teuthology.orchestra.run.smithi106.stdout:(26/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.4 MB/s | 85 kB 00:00 2024-03-24T00:08:03.987 INFO:teuthology.orchestra.run.smithi106.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 2.3 MB/s | 170 kB 00:00 2024-03-24T00:08:04.037 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-24T00:08:04.046 INFO:teuthology.orchestra.run.smithi106.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 583 kB/s | 34 kB 00:00 2024-03-24T00:08:04.113 INFO:teuthology.orchestra.run.smithi106.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 2.0 MB/s | 139 kB 00:00 2024-03-24T00:08:04.169 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T00:08:04.182 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T00:08:04.229 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-24T00:08:04.286 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-24T00:08:04.323 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-24T00:08:04.339 INFO:teuthology.orchestra.run.smithi106.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 8.0 MB/s | 1.8 MB 00:00 2024-03-24T00:08:04.356 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-24T00:08:04.407 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-24T00:08:04.581 INFO:teuthology.orchestra.run.smithi106.stdout:(31/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 4.5 MB/s | 3.1 MB 00:00 2024-03-24T00:08:04.735 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-24T00:08:04.790 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T00:08:04.910 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T00:08:04.942 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T00:08:05.017 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T00:08:05.061 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-24T00:08:05.064 INFO:teuthology.orchestra.run.smithi106.stdout:(32/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 14 MB/s | 48 MB 00:03 2024-03-24T00:08:05.090 INFO:teuthology.orchestra.run.smithi106.stdout:(33/174): ceph-mgr-modules-core-19.0.0-2145.g3b 489 kB/s | 248 kB 00:00 2024-03-24T00:08:05.103 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-24T00:08:05.115 INFO:teuthology.orchestra.run.smithi106.stdout:(34/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 988 kB/s | 49 kB 00:00 2024-03-24T00:08:05.140 INFO:teuthology.orchestra.run.smithi106.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2145.g3 328 kB/s | 16 kB 00:00 2024-03-24T00:08:05.191 INFO:teuthology.orchestra.run.smithi106.stdout:(36/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 3.4 MB/s | 264 kB 00:00 2024-03-24T00:08:05.272 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-24T00:08:05.308 INFO:teuthology.orchestra.run.smithi106.stdout:(37/174): ceph-mgr-diskprediction-local-19.0.0- 7.6 MB/s | 7.4 MB 00:00 2024-03-24T00:08:05.318 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-24T00:08:05.333 INFO:teuthology.orchestra.run.smithi106.stdout:(38/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 129 kB/s | 18 kB 00:00 2024-03-24T00:08:05.338 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T00:08:05.366 INFO:teuthology.orchestra.run.smithi106.stdout:(39/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 3.2 MB/s | 748 kB 00:00 2024-03-24T00:08:05.425 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T00:08:05.447 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-24T00:08:05.504 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-24T00:08:05.533 INFO:teuthology.orchestra.run.smithi106.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 908 kB/s | 106 kB 00:00 2024-03-24T00:08:05.543 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-24T00:08:05.566 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T00:08:05.575 INFO:teuthology.orchestra.run.smithi106.stdout:(41/174): automake-1.16.2-8.el9.noarch.rpm 2.4 MB/s | 662 kB 00:00 2024-03-24T00:08:05.581 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T00:08:05.617 INFO:teuthology.orchestra.run.smithi106.stdout:(42/174): bison-3.7.4-5.el9.x86_64.rpm 3.2 MB/s | 920 kB 00:00 2024-03-24T00:08:05.643 INFO:teuthology.orchestra.run.smithi106.stdout:(43/174): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 307 kB 00:00 2024-03-24T00:08:05.668 INFO:teuthology.orchestra.run.smithi106.stdout:(44/174): flexiblas-3.0.4-8.el9.x86_64.rpm 347 kB/s | 32 kB 00:00 2024-03-24T00:08:05.718 INFO:teuthology.orchestra.run.smithi106.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 229 kB/s | 17 kB 00:00 2024-03-24T00:08:05.768 INFO:teuthology.orchestra.run.smithi106.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 100 kB/s | 10 kB 00:00 2024-03-24T00:08:05.783 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T00:08:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T00:08:05.818 INFO:teuthology.orchestra.run.smithi106.stdout:(47/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 17 MB/s | 3.0 MB 00:00 2024-03-24T00:08:05.844 INFO:teuthology.orchestra.run.smithi106.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 159 kB 00:00 2024-03-24T00:08:05.853 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-24T00:08:05.860 INFO:teuthology.orchestra.run.smithi106.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 483 kB/s | 44 kB 00:00 2024-03-24T00:08:05.894 INFO:teuthology.orchestra.run.smithi106.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-24T00:08:05.919 INFO:teuthology.orchestra.run.smithi106.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 7.5 MB/s | 578 kB 00:00 2024-03-24T00:08:05.936 INFO:teuthology.orchestra.run.smithi106.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 621 kB/s | 46 kB 00:00 2024-03-24T00:08:05.962 INFO:teuthology.orchestra.run.smithi106.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2024-03-24T00:08:05.987 INFO:teuthology.orchestra.run.smithi106.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-24T00:08:06.012 INFO:teuthology.orchestra.run.smithi106.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-03-24T00:08:06.029 INFO:teuthology.orchestra.run.smithi106.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 513 kB/s | 34 kB 00:00 2024-03-24T00:08:06.035 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-24T00:08:06.113 INFO:teuthology.orchestra.run.smithi106.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 15 MB/s | 1.5 MB 00:00 2024-03-24T00:08:06.163 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T00:08:06.171 INFO:teuthology.orchestra.run.smithi106.stdout:(58/174): openblas-openmp-0.3.21-2.el9.x86_64.r 30 MB/s | 4.7 MB 00:00 2024-03-24T00:08:06.196 INFO:teuthology.orchestra.run.smithi106.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 4.8 MB/s | 811 kB 00:00 2024-03-24T00:08:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T00:08:06.222 INFO:teuthology.orchestra.run.smithi106.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.2 MB/s | 134 kB 00:00 2024-03-24T00:08:06.308 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T00:08:06.355 INFO:teuthology.orchestra.run.smithi106.stdout:(61/174): postgresql-server-13.11-1.el9.x86_64. 31 MB/s | 5.7 MB 00:00 2024-03-24T00:08:06.366 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T00:08:06.381 INFO:teuthology.orchestra.run.smithi106.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 497 kB/s | 78 kB 00:00 2024-03-24T00:08:06.423 INFO:teuthology.orchestra.run.smithi106.stdout:(63/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.1 MB/s | 1.0 MB 00:00 2024-03-24T00:08:06.548 INFO:teuthology.orchestra.run.smithi106.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 30 MB/s | 5.8 MB 00:00 2024-03-24T00:08:06.573 INFO:teuthology.orchestra.run.smithi106.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 129 kB/s | 24 kB 00:00 2024-03-24T00:08:06.593 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T00:08:06.598 INFO:teuthology.orchestra.run.smithi106.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 105 kB/s | 18 kB 00:00 2024-03-24T00:08:06.623 INFO:teuthology.orchestra.run.smithi106.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-24T00:08:06.649 INFO:teuthology.orchestra.run.smithi106.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 630 kB/s | 32 kB 00:00 2024-03-24T00:08:06.695 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-24T00:08:06.749 INFO:teuthology.orchestra.run.smithi106.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-03-24T00:08:06.774 INFO:teuthology.orchestra.run.smithi106.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 217 kB 00:00 2024-03-24T00:08:06.781 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-24T00:08:06.800 INFO:teuthology.orchestra.run.smithi106.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 174 kB 00:00 2024-03-24T00:08:06.813 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-24T00:08:06.824 INFO:teuthology.orchestra.run.smithi106.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 920 kB/s | 69 kB 00:00 2024-03-24T00:08:06.840 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-24T00:08:06.850 INFO:teuthology.orchestra.run.smithi106.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 827 kB/s | 41 kB 00:00 2024-03-24T00:08:06.875 INFO:teuthology.orchestra.run.smithi106.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-24T00:08:06.883 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-24T00:08:06.901 INFO:teuthology.orchestra.run.smithi106.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-24T00:08:06.926 INFO:teuthology.orchestra.run.smithi106.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 950 kB/s | 47 kB 00:00 2024-03-24T00:08:06.951 INFO:teuthology.orchestra.run.smithi106.stdout:(77/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 211 kB 00:00 2024-03-24T00:08:06.957 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-24T00:08:06.976 INFO:teuthology.orchestra.run.smithi106.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 572 kB/s | 43 kB 00:00 2024-03-24T00:08:07.010 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-24T00:08:07.049 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-24T00:08:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-24T00:08:07.150 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-24T00:08:07.184 INFO:teuthology.orchestra.run.smithi106.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 188 kB/s | 39 kB 00:00 2024-03-24T00:08:07.191 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-24T00:08:07.210 INFO:teuthology.orchestra.run.smithi106.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 178 kB/s | 41 kB 00:00 2024-03-24T00:08:07.224 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-24T00:08:07.278 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-24T00:08:07.346 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-24T00:08:07.369 INFO:teuthology.orchestra.run.smithi106.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-03-24T00:08:07.382 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-24T00:08:07.393 INFO:teuthology.orchestra.run.smithi106.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 300 kB 00:00 2024-03-24T00:08:07.419 INFO:teuthology.orchestra.run.smithi106.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 273 kB/s | 56 kB 00:00 2024-03-24T00:08:07.435 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-24T00:08:07.444 INFO:teuthology.orchestra.run.smithi106.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 836 kB/s | 63 kB 00:00 2024-03-24T00:08:07.446 INFO:teuthology.orchestra.run.smithi107.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 267 kB/s | 1.2 MB 00:04 2024-03-24T00:08:07.462 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-24T00:08:07.479 INFO:teuthology.orchestra.run.smithi106.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 741 kB/s | 44 kB 00:00 2024-03-24T00:08:07.504 INFO:teuthology.orchestra.run.smithi106.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 929 kB/s | 79 kB 00:00 2024-03-24T00:08:07.529 INFO:teuthology.orchestra.run.smithi106.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.8 MB/s | 241 kB 00:00 2024-03-24T00:08:07.530 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-24T00:08:07.546 INFO:teuthology.orchestra.run.smithi106.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-24T00:08:07.579 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-24T00:08:07.580 INFO:teuthology.orchestra.run.smithi106.stdout:(89/174): python3-cryptography-36.0.1-4.el9.x86 15 MB/s | 1.1 MB 00:00 2024-03-24T00:08:07.605 INFO:teuthology.orchestra.run.smithi106.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.6 MB/s | 124 kB 00:00 2024-03-24T00:08:07.630 INFO:teuthology.orchestra.run.smithi106.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-03-24T00:08:07.652 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-24T00:08:07.786 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-24T00:08:07.840 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-24T00:08:08.359 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-24T00:08:08.443 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-24T00:08:08.525 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-24T00:08:08.530 INFO:teuthology.orchestra.run.smithi106.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 217 kB/s | 195 kB 00:00 2024-03-24T00:08:08.556 INFO:teuthology.orchestra.run.smithi106.stdout:(93/174): jq-1.6-16.el9.x86_64.rpm 192 kB/s | 187 kB 00:00 2024-03-24T00:08:08.563 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T00:08:08.572 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T00:08:08.604 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-24T00:08:08.653 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-24T00:08:08.690 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-24T00:08:08.720 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-24T00:08:08.974 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-24T00:08:08.998 INFO:teuthology.orchestra.run.smithi106.stdout:(94/174): elfutils-libelf-devel-0.190-2.el9.x86 49 kB/s | 21 kB 00:00 2024-03-24T00:08:09.154 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-24T00:08:09.222 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-24T00:08:09.266 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-24T00:08:09.275 INFO:teuthology.orchestra.run.smithi106.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 488 kB/s | 803 kB 00:01 2024-03-24T00:08:09.300 INFO:teuthology.orchestra.run.smithi106.stdout:(96/174): python3-requests-2.25.1-8.el9.noarch. 162 kB/s | 125 kB 00:00 2024-03-24T00:08:09.413 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-24T00:08:09.417 INFO:teuthology.orchestra.run.smithi106.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 61 kB/s | 25 kB 00:00 2024-03-24T00:08:09.452 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-24T00:08:09.507 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-24T00:08:09.543 INFO:teuthology.orchestra.run.smithi106.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2024-03-24T00:08:09.596 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-24T00:08:09.746 INFO:teuthology.orchestra.run.smithi107.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 202 kB/s | 465 kB 00:02 2024-03-24T00:08:09.830 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-24T00:08:09.859 INFO:teuthology.orchestra.run.smithi106.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 684 kB/s | 216 kB 00:00 2024-03-24T00:08:09.865 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-24T00:08:09.871 INFO:teuthology.orchestra.run.smithi107.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 249 kB/s | 31 kB 00:00 2024-03-24T00:08:09.914 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-24T00:08:09.926 INFO:teuthology.orchestra.run.smithi106.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 523 kB/s | 35 kB 00:00 2024-03-24T00:08:09.948 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-24T00:08:09.992 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-24T00:08:10.034 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-24T00:08:10.170 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-24T00:08:10.221 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-24T00:08:10.226 INFO:teuthology.orchestra.run.smithi106.stdout:(101/174): python3-jinja2-2.11.3-5.el9.noarch.r 306 kB/s | 248 kB 00:00 2024-03-24T00:08:10.251 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-24T00:08:10.252 INFO:teuthology.orchestra.run.smithi106.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 137 kB/s | 44 kB 00:00 2024-03-24T00:08:10.292 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-24T00:08:10.344 INFO:teuthology.orchestra.run.smithi106.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 243 kB/s | 22 kB 00:00 2024-03-24T00:08:10.388 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-24T00:08:10.426 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-24T00:08:10.467 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-24T00:08:10.492 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-24T00:08:10.516 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-24T00:08:10.594 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-24T00:08:10.610 INFO:teuthology.orchestra.run.smithi106.stdout:(104/174): protobuf-compiler-3.14.0-13.el9.x86_ 3.2 MB/s | 863 kB 00:00 2024-03-24T00:08:10.661 INFO:teuthology.orchestra.run.smithi106.stdout:(105/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 216 kB/s | 94 kB 00:00 2024-03-24T00:08:10.735 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-24T00:08:10.850 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-24T00:08:10.900 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-24T00:08:10.957 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-24T00:08:11.012 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-24T00:08:11.194 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-24T00:08:11.203 INFO:teuthology.orchestra.run.smithi106.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 926 kB/s | 548 kB 00:00 2024-03-24T00:08:11.228 INFO:teuthology.orchestra.run.smithi106.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 545 kB/s | 309 kB 00:00 2024-03-24T00:08:11.245 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-24T00:08:11.263 INFO:teuthology.orchestra.run.smithi107.stdout:(171/174): util-linux-2.37.4-18.el9.x86_64.rpm 278 kB/s | 2.3 MB 00:08 2024-03-24T00:08:11.295 INFO:teuthology.orchestra.run.smithi106.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 211 kB/s | 19 kB 00:00 2024-03-24T00:08:11.373 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-24T00:08:11.404 INFO:teuthology.orchestra.run.smithi106.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2024-03-24T00:08:11.410 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-24T00:08:11.442 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-24T00:08:11.473 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-24T00:08:11.509 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-24T00:08:11.512 INFO:teuthology.orchestra.run.smithi106.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 448 kB/s | 49 kB 00:00 2024-03-24T00:08:11.535 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-24T00:08:11.575 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-24T00:08:11.606 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-24T00:08:11.636 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-24T00:08:11.642 INFO:teuthology.orchestra.run.smithi106.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 518 kB/s | 67 kB 00:00 2024-03-24T00:08:11.663 INFO:teuthology.orchestra.run.smithi107.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 186 kB/s | 74 kB 00:00 2024-03-24T00:08:11.712 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-24T00:08:11.721 INFO:teuthology.orchestra.run.smithi106.stdout:(112/174): libarrow-9.0.0-10.el9.x86_64.rpm 9.0 MB/s | 4.4 MB 00:00 2024-03-24T00:08:11.747 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-24T00:08:11.754 INFO:teuthology.orchestra.run.smithi106.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2024-03-24T00:08:11.780 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-24T00:08:11.821 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-24T00:08:11.838 INFO:teuthology.orchestra.run.smithi106.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 838 kB 00:00 2024-03-24T00:08:11.856 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-24T00:08:11.922 INFO:teuthology.orchestra.run.smithi106.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 3.2 MB/s | 548 kB 00:00 2024-03-24T00:08:11.922 INFO:teuthology.orchestra.run.smithi107.stdout:(173/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 243 kB/s | 2.2 MB 00:09 2024-03-24T00:08:11.939 INFO:teuthology.orchestra.run.smithi106.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 433 kB/s | 43 kB 00:00 2024-03-24T00:08:11.944 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-24T00:08:12.023 INFO:teuthology.orchestra.run.smithi106.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 321 kB/s | 32 kB 00:00 2024-03-24T00:08:12.026 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-24T00:08:12.039 INFO:teuthology.orchestra.run.smithi106.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 149 kB/s | 15 kB 00:00 2024-03-24T00:08:12.070 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-24T00:08:12.102 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-24T00:08:12.131 INFO:teuthology.orchestra.run.smithi106.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-03-24T00:08:12.148 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-24T00:08:12.156 INFO:teuthology.orchestra.run.smithi106.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-03-24T00:08:12.246 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-24T00:08:12.256 INFO:teuthology.orchestra.run.smithi106.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.4 MB/s | 303 kB 00:00 2024-03-24T00:08:12.273 INFO:teuthology.orchestra.run.smithi106.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 302 kB/s | 35 kB 00:00 2024-03-24T00:08:12.294 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-24T00:08:12.374 INFO:teuthology.orchestra.run.smithi106.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 1.7 MB/s | 203 kB 00:00 2024-03-24T00:08:12.424 INFO:teuthology.orchestra.run.smithi106.stdout:(124/174): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-03-24T00:08:12.491 INFO:teuthology.orchestra.run.smithi106.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.2 MB/s | 144 kB 00:00 2024-03-24T00:08:12.508 INFO:teuthology.orchestra.run.smithi106.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 127 kB/s | 11 kB 00:00 2024-03-24T00:08:12.592 INFO:teuthology.orchestra.run.smithi106.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 176 kB/s | 18 kB 00:00 2024-03-24T00:08:12.609 INFO:teuthology.orchestra.run.smithi106.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 231 kB/s | 23 kB 00:00 2024-03-24T00:08:12.701 INFO:teuthology.orchestra.run.smithi106.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 179 kB/s | 19 kB 00:00 2024-03-24T00:08:12.734 INFO:teuthology.orchestra.run.smithi106.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 157 kB/s | 20 kB 00:00 2024-03-24T00:08:12.826 INFO:teuthology.orchestra.run.smithi106.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 505 kB/s | 46 kB 00:00 2024-03-24T00:08:12.893 INFO:teuthology.orchestra.run.smithi106.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 5.3 MB/s | 1.0 MB 00:00 2024-03-24T00:08:12.918 INFO:teuthology.orchestra.run.smithi106.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 858 kB/s | 79 kB 00:00 2024-03-24T00:08:12.994 INFO:teuthology.orchestra.run.smithi106.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 575 kB/s | 58 kB 00:00 2024-03-24T00:08:13.023 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-24T00:08:13.027 INFO:teuthology.orchestra.run.smithi106.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-03-24T00:08:13.067 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-24T00:08:13.078 INFO:teuthology.orchestra.run.smithi106.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 197 kB/s | 16 kB 00:00 2024-03-24T00:08:13.109 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-24T00:08:13.128 INFO:teuthology.orchestra.run.smithi106.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 898 kB/s | 90 kB 00:00 2024-03-24T00:08:13.158 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-24T00:08:13.170 INFO:teuthology.orchestra.run.smithi106.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 307 kB/s | 28 kB 00:00 2024-03-24T00:08:13.217 INFO:teuthology.orchestra.run.smithi106.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 368 kB/s | 31 kB 00:00 2024-03-24T00:08:13.245 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-24T00:08:13.284 INFO:teuthology.orchestra.run.smithi106.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-03-24T00:08:13.286 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-24T00:08:13.313 INFO:teuthology.orchestra.run.smithi106.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 590 kB/s | 59 kB 00:00 2024-03-24T00:08:13.371 INFO:teuthology.orchestra.run.smithi106.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 391 kB/s | 36 kB 00:00 2024-03-24T00:08:13.413 INFO:teuthology.orchestra.run.smithi106.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 509 kB/s | 51 kB 00:00 2024-03-24T00:08:13.462 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-24T00:08:13.480 INFO:teuthology.orchestra.run.smithi106.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-03-24T00:08:13.499 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-24T00:08:13.513 INFO:teuthology.orchestra.run.smithi106.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 895 kB/s | 90 kB 00:00 2024-03-24T00:08:13.578 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-24T00:08:13.597 INFO:teuthology.orchestra.run.smithi106.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-03-24T00:08:13.623 INFO:teuthology.orchestra.run.smithi106.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 184 kB/s | 20 kB 00:00 2024-03-24T00:08:13.698 INFO:teuthology.orchestra.run.smithi106.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-24T00:08:13.756 INFO:teuthology.orchestra.run.smithi106.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-03-24T00:08:14.057 INFO:teuthology.orchestra.run.smithi106.stdout:(150/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 10 MB/s | 3.1 MB 00:00 2024-03-24T00:08:14.115 INFO:teuthology.orchestra.run.smithi106.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 5.4 MB/s | 321 kB 00:00 2024-03-24T00:08:14.211 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-24T00:08:14.241 INFO:teuthology.orchestra.run.smithi106.stdout:(152/174): librados2-19.0.0-2145.g3be9a5c9.el9. 6.3 MB/s | 3.4 MB 00:00 2024-03-24T00:08:14.321 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-24T00:08:14.456 INFO:teuthology.orchestra.run.smithi107.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 240 kB/s | 1.1 MB 00:04 2024-03-24T00:08:14.459 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-24T00:08:14.459 INFO:teuthology.orchestra.run.smithi107.stdout:Total 5.8 MB/s | 224 MB 00:38 2024-03-24T00:08:14.914 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-24T00:08:15.180 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-24T00:08:15.181 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-24T00:08:15.537 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-24T00:08:15.625 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-24T00:08:15.656 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-24T00:08:15.687 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-24T00:08:15.714 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-24T00:08:16.026 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-24T00:08:16.053 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-24T00:08:16.138 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-24T00:08:16.166 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-24T00:08:16.202 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-24T00:08:16.654 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-24T00:08:16.680 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-24T00:08:16.735 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-24T00:08:16.757 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-24T00:08:16.889 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-24T00:08:16.889 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-24T00:08:17.449 INFO:teuthology.orchestra.run.smithi106.stdout:(153/174): NetworkManager-libnm-1.47.2-1.el9.x8 585 kB/s | 1.8 MB 00:03 2024-03-24T00:08:17.833 INFO:teuthology.orchestra.run.smithi106.stdout:(154/174): NetworkManager-1.47.2-1.el9.x86_64.r 635 kB/s | 2.3 MB 00:03 2024-03-24T00:08:18.083 INFO:teuthology.orchestra.run.smithi106.stdout:(155/174): NetworkManager-team-1.47.2-1.el9.x86 66 kB/s | 42 kB 00:00 2024-03-24T00:08:18.800 INFO:teuthology.orchestra.run.smithi106.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 103 kB/s | 71 kB 00:00 2024-03-24T00:08:18.833 INFO:teuthology.orchestra.run.smithi106.stdout:(157/174): NetworkManager-tui-1.47.2-1.el9.x86_ 248 kB/s | 248 kB 00:01 2024-03-24T00:08:19.048 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T00:08:19.080 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T00:08:19.434 INFO:teuthology.orchestra.run.smithi106.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 57 kB/s | 36 kB 00:00 2024-03-24T00:08:19.621 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T00:08:19.659 INFO:teuthology.orchestra.run.smithi106.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 233 kB/s | 192 kB 00:00 2024-03-24T00:08:19.667 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-24T00:08:19.751 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-24T00:08:19.751 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T00:08:19.751 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T00:08:19.752 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:08:19.785 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-24T00:08:20.268 INFO:teuthology.orchestra.run.smithi106.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 40 kB/s | 23 kB 00:00 2024-03-24T00:08:20.410 INFO:teuthology.orchestra.run.smithi106.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 262 kB/s | 256 kB 00:00 2024-03-24T00:08:21.010 INFO:teuthology.orchestra.run.smithi106.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 146 kB/s | 108 kB 00:00 2024-03-24T00:08:21.219 INFO:teuthology.orchestra.run.smithi106.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 191 kB/s | 155 kB 00:00 2024-03-24T00:08:21.769 INFO:teuthology.orchestra.run.smithi106.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 179 kB/s | 136 kB 00:00 2024-03-24T00:08:21.936 INFO:teuthology.orchestra.run.smithi106.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 89 kB/s | 64 kB 00:00 2024-03-24T00:08:22.378 INFO:teuthology.orchestra.run.smithi106.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 47 kB/s | 29 kB 00:00 2024-03-24T00:08:23.818 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-24T00:08:23.942 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-24T00:08:23.975 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-24T00:08:24.006 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T00:08:24.015 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T00:08:24.056 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-24T00:08:24.109 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-24T00:08:24.144 INFO:teuthology.orchestra.run.smithi106.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 561 kB/s | 1.2 MB 00:02 2024-03-24T00:08:24.151 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-24T00:08:24.177 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-24T00:08:24.219 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-24T00:08:24.228 INFO:teuthology.orchestra.run.smithi106.stdout:(168/174): openssl-devel-3.0.7-27.el9.x86_64.rp 283 kB/s | 4.1 MB 00:14 2024-03-24T00:08:24.569 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-24T00:08:24.611 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T00:08:24.732 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T00:08:24.762 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T00:08:24.836 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T00:08:24.866 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-24T00:08:24.909 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-24T00:08:25.090 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-24T00:08:25.139 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-24T00:08:25.167 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T00:08:25.269 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T00:08:25.293 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-24T00:08:25.357 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-24T00:08:25.405 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-24T00:08:25.428 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T00:08:25.443 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T00:08:25.627 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T00:08:25.656 INFO:teuthology.orchestra.run.smithi106.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 326 kB/s | 465 kB 00:01 2024-03-24T00:08:25.658 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T00:08:25.682 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-24T00:08:25.787 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-24T00:08:25.892 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T00:08:25.903 INFO:teuthology.orchestra.run.smithi106.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 626 kB/s | 2.2 MB 00:03 2024-03-24T00:08:25.952 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T00:08:26.026 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T00:08:26.079 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T00:08:26.279 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T00:08:26.303 INFO:teuthology.orchestra.run.smithi106.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 48 kB/s | 31 kB 00:00 2024-03-24T00:08:26.384 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-24T00:08:26.477 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-24T00:08:26.516 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-24T00:08:26.543 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-24T00:08:26.578 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-24T00:08:26.652 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-24T00:08:26.704 INFO:teuthology.orchestra.run.smithi106.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 186 kB/s | 74 kB 00:00 2024-03-24T00:08:26.707 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-24T00:08:26.745 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-24T00:08:26.809 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-24T00:08:26.844 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-24T00:08:26.887 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-24T00:08:26.919 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-24T00:08:26.948 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-24T00:08:27.017 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-24T00:08:27.072 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-24T00:08:27.113 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-24T00:08:27.142 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-24T00:08:27.212 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-24T00:08:27.256 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-24T00:08:27.329 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-24T00:08:27.474 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-24T00:08:27.524 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-24T00:08:27.720 INFO:teuthology.orchestra.run.smithi106.stdout:(173/174): util-linux-2.37.4-18.el9.x86_64.rpm 654 kB/s | 2.3 MB 00:03 2024-03-24T00:08:27.912 INFO:teuthology.orchestra.run.smithi106.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 554 kB/s | 1.1 MB 00:01 2024-03-24T00:08:27.918 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-03-24T00:08:27.919 INFO:teuthology.orchestra.run.smithi106.stdout:Total 7.8 MB/s | 224 MB 00:28 2024-03-24T00:08:28.043 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-24T00:08:28.120 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-24T00:08:28.202 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-24T00:08:28.250 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T00:08:28.258 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T00:08:28.290 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-24T00:08:28.340 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-24T00:08:28.373 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-03-24T00:08:28.377 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-24T00:08:28.407 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-24T00:08:28.644 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-03-24T00:08:28.644 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-03-24T00:08:28.647 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-24T00:08:28.847 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-24T00:08:28.927 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-24T00:08:28.971 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-24T00:08:29.113 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-24T00:08:29.163 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-24T00:08:29.218 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-24T00:08:29.291 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-24T00:08:29.325 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-24T00:08:29.367 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-24T00:08:29.416 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-24T00:08:29.450 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-24T00:08:29.493 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-24T00:08:29.535 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-24T00:08:29.668 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-24T00:08:29.724 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-24T00:08:29.754 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-24T00:08:29.796 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-24T00:08:29.889 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-24T00:08:29.924 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-24T00:08:30.069 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-24T00:08:30.270 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-24T00:08:30.353 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-03-24T00:08:30.353 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-03-24T00:08:30.370 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-24T00:08:30.474 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-24T00:08:30.603 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-24T00:08:30.702 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-24T00:08:30.753 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-24T00:08:30.802 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-24T00:08:30.882 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-24T00:08:31.064 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-24T00:08:31.107 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-24T00:08:31.235 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-24T00:08:31.281 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-24T00:08:31.313 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-24T00:08:31.344 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-24T00:08:31.372 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-24T00:08:31.414 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-24T00:08:31.454 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-24T00:08:31.486 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-24T00:08:31.515 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-24T00:08:31.592 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-24T00:08:31.626 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-24T00:08:31.662 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-24T00:08:31.700 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-24T00:08:31.735 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-24T00:08:31.824 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-24T00:08:31.830 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-24T00:08:31.830 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /sys 2024-03-24T00:08:31.830 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /proc 2024-03-24T00:08:31.830 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /mnt 2024-03-24T00:08:31.830 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /var/tmp 2024-03-24T00:08:31.830 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /home 2024-03-24T00:08:31.830 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /root 2024-03-24T00:08:31.830 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /tmp 2024-03-24T00:08:31.830 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:08:31.913 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-24T00:08:31.958 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-24T00:08:31.988 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-24T00:08:31.990 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-24T00:08:32.004 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-24T00:08:32.004 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:08:32.004 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T00:08:32.004 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:08:32.037 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-24T00:08:32.143 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-24T00:08:32.191 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-24T00:08:32.912 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-24T00:08:32.963 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-24T00:08:32.997 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-24T00:08:33.046 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-24T00:08:33.142 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-24T00:08:33.173 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-24T00:08:33.351 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-24T00:08:33.380 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-24T00:08:33.457 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-24T00:08:33.517 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-24T00:08:33.542 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-24T00:08:33.543 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:08:33.543 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T00:08:33.543 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T00:08:33.543 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T00:08:33.543 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:08:33.767 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-24T00:08:33.790 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-24T00:08:33.791 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:08:33.791 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T00:08:33.791 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T00:08:33.791 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T00:08:33.791 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:08:34.090 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-24T00:08:34.211 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-24T00:08:34.226 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-24T00:08:34.249 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-24T00:08:34.249 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:08:34.249 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T00:08:34.249 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T00:08:34.249 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T00:08:34.249 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:08:34.361 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-24T00:08:34.778 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-24T00:08:34.809 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-24T00:08:34.850 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-24T00:08:34.909 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-24T00:08:34.934 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-24T00:08:35.166 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-24T00:08:35.200 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-24T00:08:35.416 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-24T00:08:35.490 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-24T00:08:35.520 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-24T00:08:35.576 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-24T00:08:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-24T00:08:35.925 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-24T00:08:35.951 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-24T00:08:36.034 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-24T00:08:36.064 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-24T00:08:36.100 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-24T00:08:36.159 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-24T00:08:36.201 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-24T00:08:36.302 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-24T00:08:36.463 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-24T00:08:36.494 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-24T00:08:36.517 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-24T00:08:36.518 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:08:36.518 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T00:08:36.518 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T00:08:36.518 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T00:08:36.518 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:08:36.561 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-24T00:08:36.561 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-24T00:08:36.573 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-24T00:08:36.588 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-24T00:08:36.624 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-24T00:08:36.658 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-24T00:08:36.681 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-24T00:08:37.487 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-24T00:08:37.496 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-03-24T00:08:37.530 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-24T00:08:37.553 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-24T00:08:37.553 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:08:37.553 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T00:08:37.553 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T00:08:37.553 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T00:08:37.553 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:08:37.594 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-24T00:08:37.597 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-24T00:08:37.617 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-24T00:08:37.620 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-24T00:08:37.620 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:08:37.620 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T00:08:37.620 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:08:37.656 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T00:08:37.664 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T00:08:37.673 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T00:08:37.696 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T00:08:37.696 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:08:37.696 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-24T00:08:37.696 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T00:08:37.696 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T00:08:37.696 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:08:37.705 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-24T00:08:37.758 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-24T00:08:37.792 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-24T00:08:37.818 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-24T00:08:37.859 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-24T00:08:38.002 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-24T00:08:38.022 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-24T00:08:38.022 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:08:38.022 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T00:08:38.022 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T00:08:38.022 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T00:08:38.022 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:08:38.179 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-24T00:08:38.263 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T00:08:38.382 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T00:08:38.412 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T00:08:38.486 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T00:08:38.515 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-24T00:08:38.553 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-24T00:08:38.729 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-24T00:08:38.780 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-24T00:08:38.808 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T00:08:38.911 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T00:08:38.912 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T00:08:38.934 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-24T00:08:38.953 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T00:08:38.998 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-24T00:08:39.046 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-24T00:08:39.069 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T00:08:39.084 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T00:08:39.261 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T00:08:39.291 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T00:08:39.331 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-24T00:08:39.432 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-24T00:08:39.487 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T00:08:39.523 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-24T00:08:39.533 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T00:08:39.594 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T00:08:39.604 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-24T00:08:39.604 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T00:08:39.604 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T00:08:39.604 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-24T00:08:39.632 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-24T00:08:39.669 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T00:08:39.728 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T00:08:39.925 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T00:08:40.031 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-24T00:08:40.118 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-24T00:08:40.149 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-24T00:08:40.176 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-24T00:08:40.211 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-24T00:08:40.285 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-24T00:08:40.339 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-24T00:08:40.377 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-24T00:08:40.441 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-24T00:08:40.477 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-24T00:08:40.520 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-24T00:08:40.553 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-24T00:08:40.581 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-24T00:08:40.649 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-24T00:08:40.685 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-24T00:08:40.728 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-24T00:08:40.757 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-24T00:08:40.818 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-24T00:08:40.866 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-24T00:08:40.936 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-24T00:08:41.069 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-24T00:08:41.123 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-24T00:08:41.636 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-24T00:08:41.711 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-24T00:08:41.796 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-24T00:08:41.839 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T00:08:41.847 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T00:08:41.879 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-24T00:08:41.928 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-24T00:08:41.966 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-24T00:08:41.996 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-24T00:08:42.232 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-24T00:08:42.392 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-24T00:08:42.465 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-24T00:08:42.478 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-24T00:08:42.508 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-24T00:08:42.520 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-24T00:08:42.550 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-24T00:08:42.577 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-24T00:08:42.639 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-24T00:08:42.702 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-24T00:08:42.757 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-24T00:08:42.775 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-24T00:08:42.820 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-24T00:08:42.853 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-24T00:08:42.884 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-24T00:08:42.889 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-24T00:08:42.925 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-24T00:08:42.935 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-24T00:08:42.971 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-24T00:08:42.977 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-24T00:08:43.014 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-24T00:08:43.033 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-24T00:08:43.057 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-24T00:08:43.068 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-24T00:08:43.110 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-24T00:08:43.142 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-24T00:08:43.171 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-24T00:08:43.189 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-24T00:08:43.226 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-24T00:08:43.237 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-24T00:08:43.268 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-24T00:08:43.278 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-24T00:08:43.309 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-24T00:08:43.311 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-24T00:08:43.402 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-24T00:08:43.442 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-24T00:08:43.482 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-24T00:08:43.508 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-24T00:08:43.532 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-24T00:08:43.603 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-24T00:08:43.733 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-24T00:08:43.848 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-24T00:08:43.864 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T00:08:43.890 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-24T00:08:43.901 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T00:08:43.922 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T00:08:43.938 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-24T00:08:43.993 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-24T00:08:44.175 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-24T00:08:44.227 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-24T00:08:44.354 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-24T00:08:44.401 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-24T00:08:44.441 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-24T00:08:44.472 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-24T00:08:44.500 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-24T00:08:44.534 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-24T00:08:44.573 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-24T00:08:44.605 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-24T00:08:44.635 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-24T00:08:44.710 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-24T00:08:44.745 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-24T00:08:44.752 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-24T00:08:44.778 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-24T00:08:44.819 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-24T00:08:44.837 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-24T00:08:44.854 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-24T00:08:44.878 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-24T00:08:44.942 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-24T00:08:44.967 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-24T00:08:45.007 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-24T00:08:45.023 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-24T00:08:45.033 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-24T00:08:45.058 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-24T00:08:45.076 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-24T00:08:45.085 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-24T00:08:45.085 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T00:08:45.108 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-24T00:08:45.154 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-24T00:08:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T00:08:45.203 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-24T00:08:45.203 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T00:08:45.251 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-24T00:08:45.278 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T00:08:45.300 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-24T00:08:45.310 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-24T00:08:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-24T00:08:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-24T00:08:45.388 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-24T00:08:45.423 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-24T00:08:45.490 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-24T00:08:45.528 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-24T00:08:45.553 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-24T00:08:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-24T00:08:45.604 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-24T00:08:45.629 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-24T00:08:45.636 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T00:08:45.636 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T00:08:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T00:08:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T00:08:45.898 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T00:08:45.923 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-24T00:08:45.954 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-24T00:08:45.974 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-24T00:08:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-24T00:08:46.021 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-24T00:08:46.065 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-24T00:08:46.073 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T00:08:46.098 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-24T00:08:46.154 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-24T00:08:46.233 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-24T00:08:46.274 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-24T00:08:46.451 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-24T00:08:46.472 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-24T00:08:46.549 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-24T00:08:47.178 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-24T00:08:47.301 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-24T00:08:48.493 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-24T00:08:48.571 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-24T00:08:48.602 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-24T00:08:48.634 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-24T00:08:48.661 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-24T00:08:48.973 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-24T00:08:49.008 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-24T00:08:49.098 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-24T00:08:49.129 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-24T00:08:49.165 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-24T00:08:49.625 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-24T00:08:50.031 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-24T00:08:50.031 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-24T00:08:50.031 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-24T00:08:51.192 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T00:08:51.192 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-24T00:08:51.192 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-24T00:08:51.192 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-24T00:08:51.192 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-24T00:08:51.192 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-24T00:08:51.192 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-24T00:08:51.192 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-24T00:08:51.192 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-24T00:08:51.192 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-24T00:08:51.193 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T00:08:51.193 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T00:08:51.193 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-24T00:08:51.193 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-24T00:08:51.193 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-24T00:08:51.193 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-24T00:08:51.193 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T00:08:51.193 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-24T00:08:51.193 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-24T00:08:51.193 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-24T00:08:51.193 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-24T00:08:51.193 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-24T00:08:51.193 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-24T00:08:51.193 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-24T00:08:51.193 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-24T00:08:51.194 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-24T00:08:51.194 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-24T00:08:51.194 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-24T00:08:51.194 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-24T00:08:51.194 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-24T00:08:51.194 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-24T00:08:51.194 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-24T00:08:51.194 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-24T00:08:51.194 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-24T00:08:51.194 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-24T00:08:51.194 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-24T00:08:51.194 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-24T00:08:51.195 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-24T00:08:51.195 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-24T00:08:51.195 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-24T00:08:51.195 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-24T00:08:51.195 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-24T00:08:51.195 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-24T00:08:51.195 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-24T00:08:51.195 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-24T00:08:51.195 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-24T00:08:51.195 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-24T00:08:51.195 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-24T00:08:51.195 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-24T00:08:51.195 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-24T00:08:51.195 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-24T00:08:51.196 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-24T00:08:51.196 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-24T00:08:51.196 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-24T00:08:51.196 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-24T00:08:51.196 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-24T00:08:51.196 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-24T00:08:51.196 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-24T00:08:51.196 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-24T00:08:51.196 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-24T00:08:51.196 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-24T00:08:51.196 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-24T00:08:51.196 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-24T00:08:51.196 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-24T00:08:51.196 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-24T00:08:51.197 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-24T00:08:51.197 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-24T00:08:51.197 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-24T00:08:51.197 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-24T00:08:51.197 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-24T00:08:51.197 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-24T00:08:51.197 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-24T00:08:51.197 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-24T00:08:51.197 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-24T00:08:51.197 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-24T00:08:51.197 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-24T00:08:51.197 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-24T00:08:51.197 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-24T00:08:51.197 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-24T00:08:51.198 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-24T00:08:51.198 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-24T00:08:51.198 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-24T00:08:51.198 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-24T00:08:51.198 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-24T00:08:51.198 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-24T00:08:51.198 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-24T00:08:51.198 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-24T00:08:51.198 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-24T00:08:51.198 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-24T00:08:51.198 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-24T00:08:51.198 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-24T00:08:51.198 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-24T00:08:51.198 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-24T00:08:51.199 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-24T00:08:51.199 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-24T00:08:51.199 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-24T00:08:51.199 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-24T00:08:51.199 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-24T00:08:51.199 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-24T00:08:51.199 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-24T00:08:51.199 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-24T00:08:51.199 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-24T00:08:51.199 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-24T00:08:51.199 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-24T00:08:51.199 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-24T00:08:51.199 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-24T00:08:51.199 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-24T00:08:51.200 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-24T00:08:51.200 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-24T00:08:51.200 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-24T00:08:51.200 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-24T00:08:51.200 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-24T00:08:51.200 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-24T00:08:51.200 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-24T00:08:51.200 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-24T00:08:51.200 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-24T00:08:51.200 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-24T00:08:51.200 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-24T00:08:51.200 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-24T00:08:51.200 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-24T00:08:51.200 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-24T00:08:51.201 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-24T00:08:51.201 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-24T00:08:51.201 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-24T00:08:51.201 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-24T00:08:51.201 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-24T00:08:51.201 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-24T00:08:51.201 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-24T00:08:51.201 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-24T00:08:51.201 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-24T00:08:51.201 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-24T00:08:51.201 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-24T00:08:51.201 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-24T00:08:51.201 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-24T00:08:51.202 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-24T00:08:51.202 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-24T00:08:51.202 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-24T00:08:51.202 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-24T00:08:51.202 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-24T00:08:51.202 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-24T00:08:51.202 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-24T00:08:51.202 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-24T00:08:51.202 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-24T00:08:51.202 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-24T00:08:51.202 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-24T00:08:51.202 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-24T00:08:51.202 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-24T00:08:51.202 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-24T00:08:51.203 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-24T00:08:51.203 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-24T00:08:51.203 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-24T00:08:51.203 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T00:08:51.203 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-24T00:08:51.203 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-24T00:08:51.203 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-24T00:08:51.203 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-24T00:08:51.203 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-24T00:08:51.203 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-24T00:08:51.203 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-24T00:08:51.203 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-24T00:08:51.203 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-24T00:08:51.203 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-24T00:08:51.204 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-24T00:08:51.204 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-24T00:08:51.204 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-24T00:08:51.204 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-24T00:08:51.204 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-24T00:08:51.204 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-24T00:08:51.204 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-24T00:08:51.204 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-24T00:08:51.204 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-24T00:08:51.204 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-24T00:08:51.204 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-24T00:08:51.204 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-24T00:08:51.204 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-24T00:08:51.204 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-24T00:08:51.205 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-24T00:08:51.205 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-24T00:08:51.205 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-24T00:08:51.205 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-24T00:08:51.205 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-24T00:08:51.205 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-24T00:08:51.205 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-24T00:08:51.205 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-24T00:08:51.205 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-24T00:08:51.205 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-24T00:08:51.205 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-24T00:08:51.205 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-24T00:08:51.205 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-24T00:08:51.205 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-24T00:08:51.206 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-24T00:08:51.206 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-24T00:08:51.206 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-24T00:08:51.206 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-24T00:08:51.206 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-24T00:08:51.206 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-24T00:08:51.206 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-24T00:08:51.206 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-24T00:08:51.534 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-24T00:08:51.534 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /sys 2024-03-24T00:08:51.534 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /proc 2024-03-24T00:08:51.534 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /mnt 2024-03-24T00:08:51.534 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /var/tmp 2024-03-24T00:08:51.534 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /home 2024-03-24T00:08:51.534 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /root 2024-03-24T00:08:51.534 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /tmp 2024-03-24T00:08:51.535 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-24T00:08:51.697 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-24T00:08:51.712 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-24T00:08:51.712 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:08:51.712 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T00:08:51.713 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-24T00:08:51.954 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T00:08:51.994 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T00:08:52.530 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T00:08:52.572 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-24T00:08:52.635 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-24T00:08:52.636 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T00:08:52.636 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T00:08:52.636 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-24T00:08:52.664 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-24T00:08:53.177 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-24T00:08:53.202 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-24T00:08:53.202 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:08:53.202 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T00:08:53.202 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T00:08:53.202 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T00:08:53.203 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-24T00:08:53.419 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-24T00:08:53.441 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-24T00:08:53.441 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:08:53.441 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T00:08:53.441 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T00:08:53.442 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T00:08:53.442 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-24T00:08:53.878 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-24T00:08:53.900 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-24T00:08:53.900 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:08:53.901 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T00:08:53.901 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T00:08:53.901 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T00:08:53.901 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-24T00:08:53.996 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-24T00:08:54.209 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-24T00:08:54.209 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:08:54.209 INFO:teuthology.orchestra.run.smithi078.stdout:Upgraded: 2024-03-24T00:08:54.209 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-24T00:08:54.209 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-24T00:08:54.209 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-24T00:08:54.209 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-24T00:08:54.209 INFO:teuthology.orchestra.run.smithi078.stdout: acl-2.3.1-4.el9.x86_64 2024-03-24T00:08:54.209 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-24T00:08:54.209 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-24T00:08:54.210 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-24T00:08:54.210 INFO:teuthology.orchestra.run.smithi078.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-24T00:08:54.210 INFO:teuthology.orchestra.run.smithi078.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-24T00:08:54.210 INFO:teuthology.orchestra.run.smithi078.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-24T00:08:54.210 INFO:teuthology.orchestra.run.smithi078.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-24T00:08:54.210 INFO:teuthology.orchestra.run.smithi078.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-24T00:08:54.210 INFO:teuthology.orchestra.run.smithi078.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:08:54.210 INFO:teuthology.orchestra.run.smithi078.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:08:54.210 INFO:teuthology.orchestra.run.smithi078.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-24T00:08:54.210 INFO:teuthology.orchestra.run.smithi078.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-24T00:08:54.210 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-24T00:08:54.211 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-24T00:08:54.211 INFO:teuthology.orchestra.run.smithi078.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-24T00:08:54.211 INFO:teuthology.orchestra.run.smithi078.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-24T00:08:54.211 INFO:teuthology.orchestra.run.smithi078.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-24T00:08:54.211 INFO:teuthology.orchestra.run.smithi078.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-24T00:08:54.211 INFO:teuthology.orchestra.run.smithi078.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-24T00:08:54.211 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-03-24T00:08:54.211 INFO:teuthology.orchestra.run.smithi078.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T00:08:54.211 INFO:teuthology.orchestra.run.smithi078.stdout: automake-1.16.2-8.el9.noarch 2024-03-24T00:08:54.211 INFO:teuthology.orchestra.run.smithi078.stdout: bison-3.7.4-5.el9.x86_64 2024-03-24T00:08:54.211 INFO:teuthology.orchestra.run.smithi078.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T00:08:54.212 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:08:54.212 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:08:54.212 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:08:54.212 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:08:54.212 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:08:54.212 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:08:54.212 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:08:54.212 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:08:54.212 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:08:54.213 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:08:54.213 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:08:54.213 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:08:54.213 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:08:54.213 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:08:54.213 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:08:54.213 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:08:54.214 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:08:54.214 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:08:54.214 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:08:54.214 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:08:54.214 INFO:teuthology.orchestra.run.smithi078.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:08:54.214 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:08:54.214 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:08:54.215 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:08:54.215 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-24T00:08:54.215 INFO:teuthology.orchestra.run.smithi078.stdout: flex-2.6.4-9.el9.x86_64 2024-03-24T00:08:54.215 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T00:08:54.215 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T00:08:54.215 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T00:08:54.215 INFO:teuthology.orchestra.run.smithi078.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-24T00:08:54.215 INFO:teuthology.orchestra.run.smithi078.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T00:08:54.216 INFO:teuthology.orchestra.run.smithi078.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T00:08:54.216 INFO:teuthology.orchestra.run.smithi078.stdout: jq-1.6-16.el9.x86_64 2024-03-24T00:08:54.216 INFO:teuthology.orchestra.run.smithi078.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-24T00:08:54.216 INFO:teuthology.orchestra.run.smithi078.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T00:08:54.216 INFO:teuthology.orchestra.run.smithi078.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T00:08:54.216 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:08:54.216 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:08:54.216 INFO:teuthology.orchestra.run.smithi078.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:08:54.217 INFO:teuthology.orchestra.run.smithi078.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T00:08:54.217 INFO:teuthology.orchestra.run.smithi078.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T00:08:54.217 INFO:teuthology.orchestra.run.smithi078.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T00:08:54.217 INFO:teuthology.orchestra.run.smithi078.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T00:08:54.217 INFO:teuthology.orchestra.run.smithi078.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T00:08:54.217 INFO:teuthology.orchestra.run.smithi078.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:08:54.217 INFO:teuthology.orchestra.run.smithi078.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:08:54.217 INFO:teuthology.orchestra.run.smithi078.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T00:08:54.218 INFO:teuthology.orchestra.run.smithi078.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:08:54.218 INFO:teuthology.orchestra.run.smithi078.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-24T00:08:54.218 INFO:teuthology.orchestra.run.smithi078.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T00:08:54.218 INFO:teuthology.orchestra.run.smithi078.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-24T00:08:54.218 INFO:teuthology.orchestra.run.smithi078.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-24T00:08:54.218 INFO:teuthology.orchestra.run.smithi078.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T00:08:54.218 INFO:teuthology.orchestra.run.smithi078.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T00:08:54.219 INFO:teuthology.orchestra.run.smithi078.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T00:08:54.219 INFO:teuthology.orchestra.run.smithi078.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T00:08:54.219 INFO:teuthology.orchestra.run.smithi078.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T00:08:54.219 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T00:08:54.219 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T00:08:54.219 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-24T00:08:54.219 INFO:teuthology.orchestra.run.smithi078.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T00:08:54.219 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-13.11-1.el9.x86_64 2024-03-24T00:08:54.220 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-24T00:08:54.220 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-24T00:08:54.220 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-24T00:08:54.220 INFO:teuthology.orchestra.run.smithi078.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T00:08:54.220 INFO:teuthology.orchestra.run.smithi078.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T00:08:54.220 INFO:teuthology.orchestra.run.smithi078.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T00:08:54.220 INFO:teuthology.orchestra.run.smithi078.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-24T00:08:54.220 INFO:teuthology.orchestra.run.smithi078.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T00:08:54.221 INFO:teuthology.orchestra.run.smithi078.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T00:08:54.221 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T00:08:54.221 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:08:54.221 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:08:54.221 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:08:54.221 INFO:teuthology.orchestra.run.smithi078.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T00:08:54.221 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T00:08:54.221 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T00:08:54.222 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T00:08:54.222 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-24T00:08:54.222 INFO:teuthology.orchestra.run.smithi078.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-24T00:08:54.222 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T00:08:54.222 INFO:teuthology.orchestra.run.smithi078.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T00:08:54.222 INFO:teuthology.orchestra.run.smithi078.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T00:08:54.222 INFO:teuthology.orchestra.run.smithi078.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T00:08:54.222 INFO:teuthology.orchestra.run.smithi078.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T00:08:54.223 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T00:08:54.223 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T00:08:54.223 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T00:08:54.223 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T00:08:54.223 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T00:08:54.223 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T00:08:54.223 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T00:08:54.223 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T00:08:54.224 INFO:teuthology.orchestra.run.smithi078.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T00:08:54.224 INFO:teuthology.orchestra.run.smithi078.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T00:08:54.224 INFO:teuthology.orchestra.run.smithi078.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T00:08:54.224 INFO:teuthology.orchestra.run.smithi078.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T00:08:54.224 INFO:teuthology.orchestra.run.smithi078.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T00:08:54.224 INFO:teuthology.orchestra.run.smithi078.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T00:08:54.224 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T00:08:54.225 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T00:08:54.225 INFO:teuthology.orchestra.run.smithi078.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T00:08:54.225 INFO:teuthology.orchestra.run.smithi078.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T00:08:54.225 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T00:08:54.225 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T00:08:54.225 INFO:teuthology.orchestra.run.smithi078.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T00:08:54.225 INFO:teuthology.orchestra.run.smithi078.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T00:08:54.225 INFO:teuthology.orchestra.run.smithi078.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T00:08:54.225 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T00:08:54.226 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T00:08:54.226 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T00:08:54.226 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T00:08:54.226 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-24T00:08:54.226 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T00:08:54.226 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:08:54.226 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:08:54.227 INFO:teuthology.orchestra.run.smithi078.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T00:08:54.227 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T00:08:54.227 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T00:08:54.227 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:08:54.227 INFO:teuthology.orchestra.run.smithi078.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T00:08:54.227 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T00:08:54.227 INFO:teuthology.orchestra.run.smithi078.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T00:08:54.227 INFO:teuthology.orchestra.run.smithi078.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T00:08:54.228 INFO:teuthology.orchestra.run.smithi078.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T00:08:54.228 INFO:teuthology.orchestra.run.smithi078.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T00:08:54.228 INFO:teuthology.orchestra.run.smithi078.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T00:08:54.228 INFO:teuthology.orchestra.run.smithi078.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-24T00:08:54.228 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T00:08:54.228 INFO:teuthology.orchestra.run.smithi078.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T00:08:54.228 INFO:teuthology.orchestra.run.smithi078.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T00:08:54.228 INFO:teuthology.orchestra.run.smithi078.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T00:08:54.229 INFO:teuthology.orchestra.run.smithi078.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T00:08:54.229 INFO:teuthology.orchestra.run.smithi078.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T00:08:54.229 INFO:teuthology.orchestra.run.smithi078.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T00:08:54.229 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:08:54.229 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:08:54.229 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:08:54.229 INFO:teuthology.orchestra.run.smithi078.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T00:08:54.229 INFO:teuthology.orchestra.run.smithi078.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T00:08:54.229 INFO:teuthology.orchestra.run.smithi078.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T00:08:54.229 INFO:teuthology.orchestra.run.smithi078.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-24T00:08:54.230 INFO:teuthology.orchestra.run.smithi078.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-24T00:08:54.230 INFO:teuthology.orchestra.run.smithi078.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T00:08:54.230 INFO:teuthology.orchestra.run.smithi078.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-24T00:08:54.230 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:08:54.230 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-24T00:08:54.415 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-24T00:08:54.445 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-24T00:08:54.477 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-24T00:08:54.537 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-24T00:08:54.560 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-24T00:08:54.721 DEBUG:teuthology.parallel:result is None 2024-03-24T00:08:54.792 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-24T00:08:54.827 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-24T00:08:55.784 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-24T00:08:55.828 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-24T00:08:55.928 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-24T00:08:56.090 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-24T00:08:56.122 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-24T00:08:56.145 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-24T00:08:56.145 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:08:56.145 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T00:08:56.146 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T00:08:56.146 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T00:08:56.146 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-24T00:08:56.189 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-24T00:08:56.199 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-24T00:08:56.245 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-24T00:08:57.118 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-24T00:08:57.150 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-24T00:08:57.172 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-24T00:08:57.172 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:08:57.172 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T00:08:57.172 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T00:08:57.172 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T00:08:57.172 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-24T00:08:57.217 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-24T00:08:57.239 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-24T00:08:57.239 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:08:57.239 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T00:08:57.239 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-24T00:08:57.285 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T00:08:57.306 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T00:08:57.307 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:08:57.307 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-24T00:08:57.307 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T00:08:57.307 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T00:08:57.307 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-24T00:08:57.618 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-24T00:08:57.640 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-24T00:08:57.640 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:08:57.640 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T00:08:57.640 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T00:08:57.641 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T00:08:57.641 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-24T00:09:02.064 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-24T00:09:02.105 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-24T00:09:02.136 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-24T00:09:02.163 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-24T00:09:02.354 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-24T00:09:02.471 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-24T00:09:02.511 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-24T00:09:02.551 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-24T00:09:02.603 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-24T00:09:02.639 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-24T00:09:02.671 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-24T00:09:02.703 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-24T00:09:02.733 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-24T00:09:02.788 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-24T00:09:02.848 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-24T00:09:02.890 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-24T00:09:03.469 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T00:09:03.508 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T00:09:03.527 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T00:09:04.333 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-24T00:09:04.412 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-24T00:09:04.452 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-24T00:09:04.531 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-24T00:09:04.565 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-24T00:09:04.577 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-24T00:09:04.577 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /sys 2024-03-24T00:09:04.577 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /proc 2024-03-24T00:09:04.578 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /mnt 2024-03-24T00:09:04.578 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /var/tmp 2024-03-24T00:09:04.578 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /home 2024-03-24T00:09:04.578 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /root 2024-03-24T00:09:04.578 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /tmp 2024-03-24T00:09:04.578 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-24T00:09:04.590 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-24T00:09:04.616 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-24T00:09:04.642 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-24T00:09:04.642 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T00:09:04.730 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T00:09:04.742 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-24T00:09:04.758 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-24T00:09:04.758 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:09:04.758 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T00:09:04.758 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-24T00:09:04.760 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-24T00:09:04.760 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T00:09:04.845 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T00:09:04.887 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-24T00:09:04.921 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-24T00:09:04.946 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-24T00:09:04.971 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-24T00:09:05.004 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-24T00:09:05.044 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-24T00:09:05.076 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-24T00:09:05.102 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-24T00:09:05.127 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-24T00:09:05.153 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-24T00:09:05.178 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-24T00:09:05.184 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T00:09:05.184 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T00:09:05.364 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T00:09:05.364 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T00:09:05.447 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T00:09:05.472 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-24T00:09:05.503 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-24T00:09:05.531 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-24T00:09:05.556 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-24T00:09:05.620 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T00:09:06.230 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-24T00:09:06.255 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-24T00:09:06.255 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:09:06.256 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T00:09:06.256 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T00:09:06.256 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T00:09:06.256 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-24T00:09:06.473 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-24T00:09:06.495 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-24T00:09:06.495 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:09:06.495 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T00:09:06.495 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T00:09:06.495 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T00:09:06.495 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-24T00:09:06.940 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-24T00:09:06.962 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-24T00:09:06.962 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:09:06.962 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T00:09:06.963 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T00:09:06.963 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T00:09:06.963 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-24T00:09:07.059 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-24T00:09:07.459 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-24T00:09:07.490 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-24T00:09:07.522 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-24T00:09:07.582 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-24T00:09:07.605 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-24T00:09:07.839 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-24T00:09:07.872 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-24T00:09:08.833 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-24T00:09:08.873 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-24T00:09:08.972 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-24T00:09:09.144 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-24T00:09:09.175 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-24T00:09:09.197 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-24T00:09:09.197 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:09:09.198 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T00:09:09.198 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T00:09:09.198 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T00:09:09.198 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-24T00:09:09.242 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-24T00:09:09.253 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-24T00:09:09.298 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-24T00:09:10.170 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-24T00:09:10.203 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-24T00:09:10.225 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-24T00:09:10.225 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:09:10.225 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T00:09:10.225 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T00:09:10.225 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T00:09:10.226 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-24T00:09:10.270 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-24T00:09:10.292 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-24T00:09:10.292 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:09:10.292 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T00:09:10.292 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-24T00:09:10.337 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T00:09:10.359 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T00:09:10.359 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:09:10.359 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-24T00:09:10.359 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T00:09:10.360 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T00:09:10.360 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-24T00:09:10.548 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T00:09:10.548 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-24T00:09:10.548 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-24T00:09:10.549 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-24T00:09:10.549 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-24T00:09:10.549 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-24T00:09:10.549 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-24T00:09:10.549 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-24T00:09:10.549 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-24T00:09:10.549 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-24T00:09:10.549 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T00:09:10.549 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T00:09:10.549 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-24T00:09:10.549 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-24T00:09:10.549 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-24T00:09:10.549 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-24T00:09:10.549 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T00:09:10.550 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-24T00:09:10.550 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-24T00:09:10.550 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-24T00:09:10.550 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-24T00:09:10.550 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-24T00:09:10.550 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-24T00:09:10.550 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-24T00:09:10.550 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-24T00:09:10.550 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-24T00:09:10.550 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-24T00:09:10.550 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-24T00:09:10.550 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-24T00:09:10.550 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-24T00:09:10.551 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-24T00:09:10.551 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-24T00:09:10.551 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-24T00:09:10.551 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-24T00:09:10.551 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-24T00:09:10.551 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-24T00:09:10.551 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-24T00:09:10.551 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-24T00:09:10.551 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-24T00:09:10.551 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-24T00:09:10.551 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-24T00:09:10.551 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-24T00:09:10.551 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-24T00:09:10.551 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-24T00:09:10.552 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-24T00:09:10.552 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-24T00:09:10.552 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-24T00:09:10.552 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-24T00:09:10.552 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-24T00:09:10.552 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-24T00:09:10.552 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-24T00:09:10.552 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-24T00:09:10.552 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-24T00:09:10.552 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-24T00:09:10.552 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-24T00:09:10.552 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-24T00:09:10.552 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-24T00:09:10.552 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-24T00:09:10.552 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-24T00:09:10.553 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-24T00:09:10.553 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-24T00:09:10.553 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-24T00:09:10.553 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-24T00:09:10.553 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-24T00:09:10.553 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-24T00:09:10.553 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-24T00:09:10.553 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-24T00:09:10.553 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-24T00:09:10.553 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-24T00:09:10.553 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-24T00:09:10.553 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-24T00:09:10.553 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-24T00:09:10.554 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-24T00:09:10.554 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-24T00:09:10.554 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-24T00:09:10.554 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-24T00:09:10.554 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-24T00:09:10.554 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-24T00:09:10.554 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-24T00:09:10.554 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-24T00:09:10.554 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-24T00:09:10.554 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-24T00:09:10.554 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-24T00:09:10.554 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-24T00:09:10.554 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-24T00:09:10.554 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-24T00:09:10.555 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-24T00:09:10.555 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-24T00:09:10.555 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-24T00:09:10.555 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-24T00:09:10.555 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-24T00:09:10.555 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-24T00:09:10.555 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-24T00:09:10.555 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-24T00:09:10.555 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-24T00:09:10.555 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-24T00:09:10.555 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-24T00:09:10.555 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-24T00:09:10.555 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-24T00:09:10.555 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-24T00:09:10.556 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-24T00:09:10.556 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-24T00:09:10.556 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-24T00:09:10.556 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-24T00:09:10.556 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-24T00:09:10.556 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-24T00:09:10.556 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-24T00:09:10.556 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-24T00:09:10.556 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-24T00:09:10.556 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-24T00:09:10.556 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-24T00:09:10.556 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-24T00:09:10.556 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-24T00:09:10.557 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-24T00:09:10.557 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-24T00:09:10.557 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-24T00:09:10.557 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-24T00:09:10.557 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-24T00:09:10.557 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-24T00:09:10.557 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-24T00:09:10.557 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-24T00:09:10.557 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-24T00:09:10.557 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-24T00:09:10.557 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-24T00:09:10.557 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-24T00:09:10.557 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-24T00:09:10.558 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-24T00:09:10.558 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-24T00:09:10.558 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-24T00:09:10.558 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-24T00:09:10.558 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-24T00:09:10.558 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-24T00:09:10.558 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-24T00:09:10.558 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-24T00:09:10.558 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-24T00:09:10.558 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-24T00:09:10.558 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-24T00:09:10.558 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-24T00:09:10.558 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-24T00:09:10.559 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-24T00:09:10.559 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-24T00:09:10.559 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-24T00:09:10.559 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-24T00:09:10.559 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-24T00:09:10.559 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-24T00:09:10.559 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-24T00:09:10.559 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-24T00:09:10.559 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-24T00:09:10.559 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-24T00:09:10.559 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-24T00:09:10.559 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-24T00:09:10.559 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T00:09:10.560 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-24T00:09:10.560 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-24T00:09:10.560 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-24T00:09:10.560 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-24T00:09:10.560 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-24T00:09:10.560 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-24T00:09:10.560 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-24T00:09:10.560 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-24T00:09:10.560 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-24T00:09:10.560 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-24T00:09:10.560 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-24T00:09:10.560 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-24T00:09:10.560 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-24T00:09:10.560 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-24T00:09:10.561 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-24T00:09:10.561 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-24T00:09:10.561 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-24T00:09:10.561 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-24T00:09:10.561 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-24T00:09:10.561 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-24T00:09:10.561 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-24T00:09:10.561 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-24T00:09:10.561 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-24T00:09:10.561 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-24T00:09:10.561 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-24T00:09:10.561 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-24T00:09:10.561 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-24T00:09:10.561 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-24T00:09:10.562 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-24T00:09:10.562 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-24T00:09:10.562 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-24T00:09:10.562 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-24T00:09:10.562 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-24T00:09:10.562 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-24T00:09:10.562 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-24T00:09:10.562 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-24T00:09:10.562 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-24T00:09:10.562 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-24T00:09:10.562 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-24T00:09:10.562 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-24T00:09:10.562 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-24T00:09:10.563 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-24T00:09:10.563 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-24T00:09:10.563 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-24T00:09:10.563 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-24T00:09:10.563 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-24T00:09:10.672 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-24T00:09:10.694 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-24T00:09:10.694 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:09:10.694 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T00:09:10.694 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T00:09:10.694 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T00:09:10.695 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-24T00:09:13.703 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-24T00:09:13.703 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-24T00:09:13.704 INFO:teuthology.orchestra.run.smithi107.stdout:Upgraded: 2024-03-24T00:09:13.704 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-24T00:09:13.704 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-24T00:09:13.704 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-24T00:09:13.704 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-24T00:09:13.704 INFO:teuthology.orchestra.run.smithi107.stdout: acl-2.3.1-4.el9.x86_64 2024-03-24T00:09:13.705 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-24T00:09:13.705 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-24T00:09:13.705 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-24T00:09:13.705 INFO:teuthology.orchestra.run.smithi107.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-24T00:09:13.705 INFO:teuthology.orchestra.run.smithi107.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-24T00:09:13.705 INFO:teuthology.orchestra.run.smithi107.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-24T00:09:13.705 INFO:teuthology.orchestra.run.smithi107.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-24T00:09:13.705 INFO:teuthology.orchestra.run.smithi107.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-24T00:09:13.706 INFO:teuthology.orchestra.run.smithi107.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:13.706 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:13.706 INFO:teuthology.orchestra.run.smithi107.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-24T00:09:13.706 INFO:teuthology.orchestra.run.smithi107.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-24T00:09:13.706 INFO:teuthology.orchestra.run.smithi107.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-24T00:09:13.706 INFO:teuthology.orchestra.run.smithi107.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-24T00:09:13.706 INFO:teuthology.orchestra.run.smithi107.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-24T00:09:13.706 INFO:teuthology.orchestra.run.smithi107.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-24T00:09:13.706 INFO:teuthology.orchestra.run.smithi107.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-24T00:09:13.706 INFO:teuthology.orchestra.run.smithi107.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-24T00:09:13.707 INFO:teuthology.orchestra.run.smithi107.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-24T00:09:13.707 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-24T00:09:13.707 INFO:teuthology.orchestra.run.smithi107.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T00:09:13.707 INFO:teuthology.orchestra.run.smithi107.stdout: automake-1.16.2-8.el9.noarch 2024-03-24T00:09:13.707 INFO:teuthology.orchestra.run.smithi107.stdout: bison-3.7.4-5.el9.x86_64 2024-03-24T00:09:13.707 INFO:teuthology.orchestra.run.smithi107.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T00:09:13.707 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:13.707 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:13.707 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:13.708 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:13.708 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:09:13.708 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:13.708 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:13.708 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:13.708 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:09:13.708 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:09:13.708 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:09:13.708 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:09:13.708 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:09:13.709 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:13.709 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:13.709 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:09:13.709 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:13.709 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:13.709 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:13.709 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:09:13.709 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:09:13.709 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:13.710 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:13.710 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:09:13.710 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-24T00:09:13.710 INFO:teuthology.orchestra.run.smithi107.stdout: flex-2.6.4-9.el9.x86_64 2024-03-24T00:09:13.710 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T00:09:13.710 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T00:09:13.710 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T00:09:13.710 INFO:teuthology.orchestra.run.smithi107.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-24T00:09:13.710 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T00:09:13.710 INFO:teuthology.orchestra.run.smithi107.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T00:09:13.710 INFO:teuthology.orchestra.run.smithi107.stdout: jq-1.6-16.el9.x86_64 2024-03-24T00:09:13.711 INFO:teuthology.orchestra.run.smithi107.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-24T00:09:13.711 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T00:09:13.711 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T00:09:13.711 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:13.711 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:13.711 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:13.711 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T00:09:13.711 INFO:teuthology.orchestra.run.smithi107.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T00:09:13.711 INFO:teuthology.orchestra.run.smithi107.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T00:09:13.711 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T00:09:13.711 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T00:09:13.711 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:13.711 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:13.711 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T00:09:13.712 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:13.712 INFO:teuthology.orchestra.run.smithi107.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-24T00:09:13.712 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T00:09:13.712 INFO:teuthology.orchestra.run.smithi107.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-24T00:09:13.712 INFO:teuthology.orchestra.run.smithi107.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-24T00:09:13.712 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T00:09:13.712 INFO:teuthology.orchestra.run.smithi107.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T00:09:13.712 INFO:teuthology.orchestra.run.smithi107.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T00:09:13.712 INFO:teuthology.orchestra.run.smithi107.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T00:09:13.712 INFO:teuthology.orchestra.run.smithi107.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T00:09:13.712 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T00:09:13.712 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T00:09:13.712 INFO:teuthology.orchestra.run.smithi107.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-24T00:09:13.712 INFO:teuthology.orchestra.run.smithi107.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T00:09:13.713 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-13.11-1.el9.x86_64 2024-03-24T00:09:13.713 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-24T00:09:13.713 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-24T00:09:13.713 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-24T00:09:13.713 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T00:09:13.713 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T00:09:13.713 INFO:teuthology.orchestra.run.smithi107.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T00:09:13.713 INFO:teuthology.orchestra.run.smithi107.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-24T00:09:13.713 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T00:09:13.713 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T00:09:13.713 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T00:09:13.713 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:13.713 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:13.713 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:13.714 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T00:09:13.714 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T00:09:13.714 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T00:09:13.714 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T00:09:13.714 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-24T00:09:13.714 INFO:teuthology.orchestra.run.smithi107.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-24T00:09:13.714 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T00:09:13.714 INFO:teuthology.orchestra.run.smithi107.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T00:09:13.714 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T00:09:13.714 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T00:09:13.714 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T00:09:13.714 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T00:09:13.714 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T00:09:13.715 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T00:09:13.715 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T00:09:13.715 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T00:09:13.715 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T00:09:13.715 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T00:09:13.715 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T00:09:13.715 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T00:09:13.715 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T00:09:13.715 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T00:09:13.715 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T00:09:13.715 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T00:09:13.715 INFO:teuthology.orchestra.run.smithi107.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T00:09:13.716 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T00:09:13.716 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T00:09:13.716 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T00:09:13.716 INFO:teuthology.orchestra.run.smithi107.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T00:09:13.716 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T00:09:13.716 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T00:09:13.716 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T00:09:13.716 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T00:09:13.716 INFO:teuthology.orchestra.run.smithi107.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T00:09:13.716 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T00:09:13.716 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T00:09:13.716 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T00:09:13.716 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T00:09:13.716 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-24T00:09:13.717 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T00:09:13.717 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:13.717 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:13.717 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T00:09:13.717 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T00:09:13.717 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T00:09:13.717 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:13.717 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T00:09:13.717 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T00:09:13.717 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T00:09:13.717 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T00:09:13.717 INFO:teuthology.orchestra.run.smithi107.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T00:09:13.717 INFO:teuthology.orchestra.run.smithi107.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T00:09:13.718 INFO:teuthology.orchestra.run.smithi107.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T00:09:13.718 INFO:teuthology.orchestra.run.smithi107.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-24T00:09:13.718 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T00:09:13.718 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T00:09:13.718 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T00:09:13.718 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T00:09:13.718 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T00:09:13.718 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T00:09:13.718 INFO:teuthology.orchestra.run.smithi107.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T00:09:13.718 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:13.718 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:13.718 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:13.718 INFO:teuthology.orchestra.run.smithi107.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T00:09:13.718 INFO:teuthology.orchestra.run.smithi107.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T00:09:13.719 INFO:teuthology.orchestra.run.smithi107.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T00:09:13.719 INFO:teuthology.orchestra.run.smithi107.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-24T00:09:13.719 INFO:teuthology.orchestra.run.smithi107.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-24T00:09:13.719 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T00:09:13.719 INFO:teuthology.orchestra.run.smithi107.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-24T00:09:13.719 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-24T00:09:13.719 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-24T00:09:14.164 DEBUG:teuthology.parallel:result is None 2024-03-24T00:09:15.101 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-24T00:09:15.142 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-24T00:09:15.172 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-24T00:09:15.199 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-24T00:09:15.378 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-24T00:09:15.506 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-24T00:09:15.547 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-24T00:09:15.586 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-24T00:09:15.661 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-24T00:09:15.707 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-24T00:09:15.748 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-24T00:09:15.780 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-24T00:09:15.810 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-24T00:09:15.856 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-24T00:09:15.907 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-24T00:09:15.941 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-24T00:09:16.478 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T00:09:16.514 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T00:09:16.534 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T00:09:17.348 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-24T00:09:17.434 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-24T00:09:17.475 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-24T00:09:17.555 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-24T00:09:17.596 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-24T00:09:17.621 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-24T00:09:17.646 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-24T00:09:17.673 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-24T00:09:17.673 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T00:09:17.760 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T00:09:17.790 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-24T00:09:17.790 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T00:09:17.875 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T00:09:17.901 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-24T00:09:17.934 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-24T00:09:17.960 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-24T00:09:17.984 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-24T00:09:18.017 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-24T00:09:18.057 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-24T00:09:18.098 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-24T00:09:18.123 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-24T00:09:18.148 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-24T00:09:18.173 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-24T00:09:18.199 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-24T00:09:18.205 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T00:09:18.205 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T00:09:18.385 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T00:09:18.385 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T00:09:18.459 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T00:09:18.484 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-24T00:09:18.516 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-24T00:09:18.544 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-24T00:09:18.569 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-24T00:09:18.632 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T00:09:23.473 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T00:09:23.473 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-24T00:09:23.473 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-24T00:09:23.473 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-24T00:09:23.473 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-24T00:09:23.473 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-24T00:09:23.473 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-24T00:09:23.474 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-24T00:09:23.474 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-24T00:09:23.474 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-24T00:09:23.474 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T00:09:23.474 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T00:09:23.474 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-24T00:09:23.474 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-24T00:09:23.474 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-24T00:09:23.474 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-24T00:09:23.474 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T00:09:23.474 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-24T00:09:23.474 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-24T00:09:23.475 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-24T00:09:23.475 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-24T00:09:23.475 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-24T00:09:23.475 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-24T00:09:23.475 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-24T00:09:23.475 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-24T00:09:23.475 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-24T00:09:23.475 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-24T00:09:23.475 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-24T00:09:23.475 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-24T00:09:23.475 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-24T00:09:23.475 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-24T00:09:23.475 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-24T00:09:23.476 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-24T00:09:23.476 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-24T00:09:23.476 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-24T00:09:23.476 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-24T00:09:23.476 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-24T00:09:23.476 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-24T00:09:23.476 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-24T00:09:23.476 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-24T00:09:23.476 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-24T00:09:23.476 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-24T00:09:23.476 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-24T00:09:23.476 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-24T00:09:23.476 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-24T00:09:23.477 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-24T00:09:23.477 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-24T00:09:23.477 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-24T00:09:23.477 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-24T00:09:23.477 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-24T00:09:23.477 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-24T00:09:23.477 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-24T00:09:23.477 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-24T00:09:23.477 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-24T00:09:23.477 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-24T00:09:23.477 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-24T00:09:23.477 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-24T00:09:23.477 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-24T00:09:23.478 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-24T00:09:23.478 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-24T00:09:23.478 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-24T00:09:23.478 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-24T00:09:23.478 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-24T00:09:23.478 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-24T00:09:23.478 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-24T00:09:23.478 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-24T00:09:23.478 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-24T00:09:23.478 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-24T00:09:23.478 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-24T00:09:23.478 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-24T00:09:23.478 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-24T00:09:23.479 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-24T00:09:23.479 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-24T00:09:23.479 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-24T00:09:23.479 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-24T00:09:23.479 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-24T00:09:23.479 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-24T00:09:23.479 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-24T00:09:23.479 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-24T00:09:23.479 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-24T00:09:23.479 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-24T00:09:23.479 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-24T00:09:23.479 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-24T00:09:23.479 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-24T00:09:23.480 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-24T00:09:23.480 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-24T00:09:23.480 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-24T00:09:23.480 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-24T00:09:23.480 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-24T00:09:23.480 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-24T00:09:23.480 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-24T00:09:23.480 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-24T00:09:23.480 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-24T00:09:23.480 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-24T00:09:23.480 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-24T00:09:23.480 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-24T00:09:23.480 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-24T00:09:23.480 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-24T00:09:23.481 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-24T00:09:23.481 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-24T00:09:23.481 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-24T00:09:23.481 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-24T00:09:23.481 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-24T00:09:23.481 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-24T00:09:23.481 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-24T00:09:23.481 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-24T00:09:23.481 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-24T00:09:23.481 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-24T00:09:23.481 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-24T00:09:23.481 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-24T00:09:23.481 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-24T00:09:23.482 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-24T00:09:23.482 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-24T00:09:23.482 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-24T00:09:23.482 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-24T00:09:23.482 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-24T00:09:23.482 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-24T00:09:23.482 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-24T00:09:23.482 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-24T00:09:23.482 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-24T00:09:23.482 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-24T00:09:23.482 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-24T00:09:23.482 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-24T00:09:23.482 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-24T00:09:23.482 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-24T00:09:23.483 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-24T00:09:23.483 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-24T00:09:23.483 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-24T00:09:23.483 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-24T00:09:23.483 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-24T00:09:23.483 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-24T00:09:23.483 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-24T00:09:23.483 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-24T00:09:23.483 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-24T00:09:23.483 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-24T00:09:23.483 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-24T00:09:23.483 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-24T00:09:23.483 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-24T00:09:23.484 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-24T00:09:23.484 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-24T00:09:23.484 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-24T00:09:23.484 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-24T00:09:23.484 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-24T00:09:23.484 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-24T00:09:23.484 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-24T00:09:23.484 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-24T00:09:23.484 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-24T00:09:23.484 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-24T00:09:23.484 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-24T00:09:23.484 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-24T00:09:23.484 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-24T00:09:23.484 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T00:09:23.485 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-24T00:09:23.485 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-24T00:09:23.485 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-24T00:09:23.485 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-24T00:09:23.485 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-24T00:09:23.485 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-24T00:09:23.485 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-24T00:09:23.485 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-24T00:09:23.485 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-24T00:09:23.485 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-24T00:09:23.485 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-24T00:09:23.485 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-24T00:09:23.485 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-24T00:09:23.486 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-24T00:09:23.486 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-24T00:09:23.486 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-24T00:09:23.486 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-24T00:09:23.486 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-24T00:09:23.486 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-24T00:09:23.486 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-24T00:09:23.486 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-24T00:09:23.486 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-24T00:09:23.486 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-24T00:09:23.486 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-24T00:09:23.486 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-24T00:09:23.486 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-24T00:09:23.487 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-24T00:09:23.487 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-24T00:09:23.487 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-24T00:09:23.487 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-24T00:09:23.487 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-24T00:09:23.487 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-24T00:09:23.487 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-24T00:09:23.487 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-24T00:09:23.487 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-24T00:09:23.488 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-24T00:09:23.488 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-24T00:09:23.488 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-24T00:09:23.488 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-24T00:09:23.488 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-24T00:09:23.488 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-24T00:09:23.488 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-24T00:09:23.488 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-24T00:09:23.488 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-24T00:09:23.488 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-24T00:09:23.488 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-24T00:09:26.316 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-24T00:09:26.316 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-24T00:09:26.316 INFO:teuthology.orchestra.run.smithi106.stdout:Upgraded: 2024-03-24T00:09:26.316 INFO:teuthology.orchestra.run.smithi106.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-24T00:09:26.317 INFO:teuthology.orchestra.run.smithi106.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-24T00:09:26.317 INFO:teuthology.orchestra.run.smithi106.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-24T00:09:26.317 INFO:teuthology.orchestra.run.smithi106.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-24T00:09:26.317 INFO:teuthology.orchestra.run.smithi106.stdout: acl-2.3.1-4.el9.x86_64 2024-03-24T00:09:26.317 INFO:teuthology.orchestra.run.smithi106.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-24T00:09:26.317 INFO:teuthology.orchestra.run.smithi106.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-24T00:09:26.317 INFO:teuthology.orchestra.run.smithi106.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-24T00:09:26.317 INFO:teuthology.orchestra.run.smithi106.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-24T00:09:26.317 INFO:teuthology.orchestra.run.smithi106.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-24T00:09:26.317 INFO:teuthology.orchestra.run.smithi106.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-24T00:09:26.317 INFO:teuthology.orchestra.run.smithi106.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-24T00:09:26.317 INFO:teuthology.orchestra.run.smithi106.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-24T00:09:26.317 INFO:teuthology.orchestra.run.smithi106.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:26.318 INFO:teuthology.orchestra.run.smithi106.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:26.318 INFO:teuthology.orchestra.run.smithi106.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-24T00:09:26.318 INFO:teuthology.orchestra.run.smithi106.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-24T00:09:26.318 INFO:teuthology.orchestra.run.smithi106.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-24T00:09:26.318 INFO:teuthology.orchestra.run.smithi106.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-24T00:09:26.318 INFO:teuthology.orchestra.run.smithi106.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-24T00:09:26.318 INFO:teuthology.orchestra.run.smithi106.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-24T00:09:26.318 INFO:teuthology.orchestra.run.smithi106.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-24T00:09:26.318 INFO:teuthology.orchestra.run.smithi106.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-24T00:09:26.318 INFO:teuthology.orchestra.run.smithi106.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-24T00:09:26.318 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-03-24T00:09:26.318 INFO:teuthology.orchestra.run.smithi106.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T00:09:26.318 INFO:teuthology.orchestra.run.smithi106.stdout: automake-1.16.2-8.el9.noarch 2024-03-24T00:09:26.318 INFO:teuthology.orchestra.run.smithi106.stdout: bison-3.7.4-5.el9.x86_64 2024-03-24T00:09:26.319 INFO:teuthology.orchestra.run.smithi106.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T00:09:26.319 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:26.319 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:26.319 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:26.319 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:26.319 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:09:26.319 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:26.319 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:26.319 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:26.319 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:09:26.319 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:09:26.319 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:09:26.319 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:09:26.320 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:09:26.320 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:26.320 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:26.320 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:09:26.320 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:26.320 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:26.320 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:26.320 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:09:26.320 INFO:teuthology.orchestra.run.smithi106.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:09:26.320 INFO:teuthology.orchestra.run.smithi106.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:26.320 INFO:teuthology.orchestra.run.smithi106.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:26.320 INFO:teuthology.orchestra.run.smithi106.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:09:26.320 INFO:teuthology.orchestra.run.smithi106.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-24T00:09:26.320 INFO:teuthology.orchestra.run.smithi106.stdout: flex-2.6.4-9.el9.x86_64 2024-03-24T00:09:26.321 INFO:teuthology.orchestra.run.smithi106.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T00:09:26.321 INFO:teuthology.orchestra.run.smithi106.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T00:09:26.321 INFO:teuthology.orchestra.run.smithi106.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T00:09:26.321 INFO:teuthology.orchestra.run.smithi106.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-24T00:09:26.321 INFO:teuthology.orchestra.run.smithi106.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T00:09:26.321 INFO:teuthology.orchestra.run.smithi106.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T00:09:26.321 INFO:teuthology.orchestra.run.smithi106.stdout: jq-1.6-16.el9.x86_64 2024-03-24T00:09:26.321 INFO:teuthology.orchestra.run.smithi106.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-24T00:09:26.321 INFO:teuthology.orchestra.run.smithi106.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T00:09:26.321 INFO:teuthology.orchestra.run.smithi106.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T00:09:26.321 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:26.321 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:26.321 INFO:teuthology.orchestra.run.smithi106.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:26.321 INFO:teuthology.orchestra.run.smithi106.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T00:09:26.322 INFO:teuthology.orchestra.run.smithi106.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T00:09:26.322 INFO:teuthology.orchestra.run.smithi106.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T00:09:26.322 INFO:teuthology.orchestra.run.smithi106.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T00:09:26.322 INFO:teuthology.orchestra.run.smithi106.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T00:09:26.322 INFO:teuthology.orchestra.run.smithi106.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:26.322 INFO:teuthology.orchestra.run.smithi106.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:26.322 INFO:teuthology.orchestra.run.smithi106.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T00:09:26.322 INFO:teuthology.orchestra.run.smithi106.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:26.322 INFO:teuthology.orchestra.run.smithi106.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-24T00:09:26.322 INFO:teuthology.orchestra.run.smithi106.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T00:09:26.322 INFO:teuthology.orchestra.run.smithi106.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-24T00:09:26.322 INFO:teuthology.orchestra.run.smithi106.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-24T00:09:26.322 INFO:teuthology.orchestra.run.smithi106.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T00:09:26.322 INFO:teuthology.orchestra.run.smithi106.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T00:09:26.323 INFO:teuthology.orchestra.run.smithi106.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T00:09:26.323 INFO:teuthology.orchestra.run.smithi106.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T00:09:26.323 INFO:teuthology.orchestra.run.smithi106.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T00:09:26.323 INFO:teuthology.orchestra.run.smithi106.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T00:09:26.323 INFO:teuthology.orchestra.run.smithi106.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T00:09:26.323 INFO:teuthology.orchestra.run.smithi106.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-24T00:09:26.323 INFO:teuthology.orchestra.run.smithi106.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T00:09:26.323 INFO:teuthology.orchestra.run.smithi106.stdout: postgresql-13.11-1.el9.x86_64 2024-03-24T00:09:26.323 INFO:teuthology.orchestra.run.smithi106.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-24T00:09:26.323 INFO:teuthology.orchestra.run.smithi106.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-24T00:09:26.323 INFO:teuthology.orchestra.run.smithi106.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-24T00:09:26.323 INFO:teuthology.orchestra.run.smithi106.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T00:09:26.323 INFO:teuthology.orchestra.run.smithi106.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T00:09:26.323 INFO:teuthology.orchestra.run.smithi106.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T00:09:26.323 INFO:teuthology.orchestra.run.smithi106.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-24T00:09:26.324 INFO:teuthology.orchestra.run.smithi106.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T00:09:26.324 INFO:teuthology.orchestra.run.smithi106.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T00:09:26.324 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T00:09:26.324 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:26.324 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:26.324 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:26.324 INFO:teuthology.orchestra.run.smithi106.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T00:09:26.324 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T00:09:26.324 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T00:09:26.324 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T00:09:26.324 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-24T00:09:26.324 INFO:teuthology.orchestra.run.smithi106.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-24T00:09:26.324 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T00:09:26.324 INFO:teuthology.orchestra.run.smithi106.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T00:09:26.325 INFO:teuthology.orchestra.run.smithi106.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T00:09:26.325 INFO:teuthology.orchestra.run.smithi106.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T00:09:26.325 INFO:teuthology.orchestra.run.smithi106.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T00:09:26.325 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T00:09:26.325 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T00:09:26.325 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T00:09:26.325 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T00:09:26.325 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T00:09:26.325 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T00:09:26.325 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T00:09:26.326 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T00:09:26.326 INFO:teuthology.orchestra.run.smithi106.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T00:09:26.326 INFO:teuthology.orchestra.run.smithi106.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T00:09:26.326 INFO:teuthology.orchestra.run.smithi106.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T00:09:26.326 INFO:teuthology.orchestra.run.smithi106.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T00:09:26.326 INFO:teuthology.orchestra.run.smithi106.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T00:09:26.326 INFO:teuthology.orchestra.run.smithi106.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T00:09:26.326 INFO:teuthology.orchestra.run.smithi106.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T00:09:26.326 INFO:teuthology.orchestra.run.smithi106.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T00:09:26.326 INFO:teuthology.orchestra.run.smithi106.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T00:09:26.326 INFO:teuthology.orchestra.run.smithi106.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T00:09:26.326 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T00:09:26.326 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T00:09:26.327 INFO:teuthology.orchestra.run.smithi106.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T00:09:26.327 INFO:teuthology.orchestra.run.smithi106.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T00:09:26.327 INFO:teuthology.orchestra.run.smithi106.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T00:09:26.327 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T00:09:26.327 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T00:09:26.327 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T00:09:26.327 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T00:09:26.327 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-24T00:09:26.327 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T00:09:26.327 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:26.327 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:26.327 INFO:teuthology.orchestra.run.smithi106.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T00:09:26.327 INFO:teuthology.orchestra.run.smithi106.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T00:09:26.327 INFO:teuthology.orchestra.run.smithi106.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T00:09:26.327 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:26.328 INFO:teuthology.orchestra.run.smithi106.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T00:09:26.328 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T00:09:26.328 INFO:teuthology.orchestra.run.smithi106.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T00:09:26.328 INFO:teuthology.orchestra.run.smithi106.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T00:09:26.328 INFO:teuthology.orchestra.run.smithi106.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T00:09:26.328 INFO:teuthology.orchestra.run.smithi106.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T00:09:26.328 INFO:teuthology.orchestra.run.smithi106.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T00:09:26.328 INFO:teuthology.orchestra.run.smithi106.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-24T00:09:26.328 INFO:teuthology.orchestra.run.smithi106.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T00:09:26.328 INFO:teuthology.orchestra.run.smithi106.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T00:09:26.328 INFO:teuthology.orchestra.run.smithi106.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T00:09:26.328 INFO:teuthology.orchestra.run.smithi106.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T00:09:26.328 INFO:teuthology.orchestra.run.smithi106.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T00:09:26.328 INFO:teuthology.orchestra.run.smithi106.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T00:09:26.329 INFO:teuthology.orchestra.run.smithi106.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T00:09:26.329 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:26.329 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:26.329 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:09:26.329 INFO:teuthology.orchestra.run.smithi106.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T00:09:26.329 INFO:teuthology.orchestra.run.smithi106.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T00:09:26.329 INFO:teuthology.orchestra.run.smithi106.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T00:09:26.329 INFO:teuthology.orchestra.run.smithi106.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-24T00:09:26.329 INFO:teuthology.orchestra.run.smithi106.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-24T00:09:26.329 INFO:teuthology.orchestra.run.smithi106.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T00:09:26.330 INFO:teuthology.orchestra.run.smithi106.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-24T00:09:26.330 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-24T00:09:26.330 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-03-24T00:09:26.842 DEBUG:teuthology.parallel:result is None 2024-03-24T00:09:26.843 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:09:26.984 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T00:09:27.010 INFO:teuthology.orchestra.run.smithi078.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-24T00:09:27.010 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-24T00:09:27.011 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-24T00:09:27.012 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:09:27.153 DEBUG:teuthology.orchestra.run.smithi106:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T00:09:27.174 INFO:teuthology.orchestra.run.smithi106.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-24T00:09:27.174 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-24T00:09:27.174 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-24T00:09:27.176 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:09:27.311 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T00:09:27.336 INFO:teuthology.orchestra.run.smithi107.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-24T00:09:27.337 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-24T00:09:27.337 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-24T00:09:27.339 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-24T00:09:27.339 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-24T00:09:27.340 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T00:09:27.374 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-24T00:09:27.374 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T00:09:27.403 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-24T00:09:27.403 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T00:09:27.436 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-24T00:09:27.437 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-24T00:09:27.437 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T00:09:27.468 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T00:09:27.538 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-24T00:09:27.538 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T00:09:27.564 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T00:09:27.631 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-24T00:09:27.631 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T00:09:27.661 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T00:09:27.731 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-24T00:09:27.731 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-24T00:09:27.731 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T00:09:27.758 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T00:09:27.825 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-24T00:09:27.825 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T00:09:27.852 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T00:09:27.919 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-24T00:09:27.919 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T00:09:27.946 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T00:09:28.014 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-24T00:09:28.014 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-24T00:09:28.014 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T00:09:28.041 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T00:09:28.111 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-24T00:09:28.111 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T00:09:28.138 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T00:09:28.204 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-24T00:09:28.204 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T00:09:28.231 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T00:09:28.298 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-24T00:09:28.411 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN'], 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'subvols': {'create': 2}} 2024-03-24T00:09:28.411 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:09:28.411 INFO:tasks.cephadm:Cluster fsid is c7e2806a-e972-11ee-95cd-87774f69a715 2024-03-24T00:09:28.411 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-24T00:09:28.412 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.78', 'mon.b': '172.21.15.106', 'mon.c': '172.21.15.107'} 2024-03-24T00:09:28.412 INFO:tasks.cephadm:First mon is mon.a on smithi078 2024-03-24T00:09:28.412 INFO:tasks.cephadm:First mgr is x 2024-03-24T00:09:28.412 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-24T00:09:28.412 DEBUG:teuthology.orchestra.run.smithi078:> sudo hostname $(hostname -s) 2024-03-24T00:09:28.439 DEBUG:teuthology.orchestra.run.smithi106:> sudo hostname $(hostname -s) 2024-03-24T00:09:28.467 DEBUG:teuthology.orchestra.run.smithi107:> sudo hostname $(hostname -s) 2024-03-24T00:09:28.494 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-24T00:09:28.494 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:09:28.711 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77877/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2145-g3be9a5c9', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2145.g3be9a5c9'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-22 05:51:42.267384', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-mchangir-testing-main-20240318.032620', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-24T00:09:28.844 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-mchangir-testing-main-20240318.032620, sha1 3be9a5c9bc793e11e2800a8c0c696e8b46742033 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:09:28.845 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm 2024-03-24T00:09:28.845 DEBUG:teuthology.orchestra.run.smithi078:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T00:09:29.274 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 24 00:09 /home/ubuntu/cephtest/cephadm 2024-03-24T00:09:29.274 DEBUG:teuthology.orchestra.run.smithi106:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T00:09:29.852 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 24 00:09 /home/ubuntu/cephtest/cephadm 2024-03-24T00:09:29.852 DEBUG:teuthology.orchestra.run.smithi107:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T00:09:30.446 INFO:teuthology.orchestra.run.smithi107.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 24 00:09 /home/ubuntu/cephtest/cephadm 2024-03-24T00:09:30.446 DEBUG:teuthology.orchestra.run.smithi078:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T00:09:30.464 DEBUG:teuthology.orchestra.run.smithi106:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T00:09:30.483 DEBUG:teuthology.orchestra.run.smithi107:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T00:09:30.566 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 on all hosts... 2024-03-24T00:09:30.566 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-24T00:09:30.569 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-24T00:09:30.571 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-24T00:09:31.340 INFO:teuthology.orchestra.run.smithi107.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-24T00:09:31.438 INFO:teuthology.orchestra.run.smithi078.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-24T00:09:31.464 INFO:teuthology.orchestra.run.smithi106.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-24T00:10:23.598 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-03-24T00:10:23.599 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T00:10:23.599 INFO:teuthology.orchestra.run.smithi106.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-24T00:10:23.599 INFO:teuthology.orchestra.run.smithi106.stdout: "repo_digests": [ 2024-03-24T00:10:23.599 INFO:teuthology.orchestra.run.smithi106.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-24T00:10:23.599 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-03-24T00:10:23.599 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-03-24T00:10:35.237 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-24T00:10:35.237 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T00:10:35.237 INFO:teuthology.orchestra.run.smithi078.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-24T00:10:35.237 INFO:teuthology.orchestra.run.smithi078.stdout: "repo_digests": [ 2024-03-24T00:10:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-24T00:10:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-24T00:10:35.238 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-24T00:10:35.400 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-03-24T00:10:35.400 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T00:10:35.400 INFO:teuthology.orchestra.run.smithi107.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-24T00:10:35.400 INFO:teuthology.orchestra.run.smithi107.stdout: "repo_digests": [ 2024-03-24T00:10:35.400 INFO:teuthology.orchestra.run.smithi107.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-24T00:10:35.400 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-03-24T00:10:35.400 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-03-24T00:10:35.424 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /etc/ceph 2024-03-24T00:10:35.454 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /etc/ceph 2024-03-24T00:10:35.489 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /etc/ceph 2024-03-24T00:10:35.519 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 777 /etc/ceph 2024-03-24T00:10:35.546 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 777 /etc/ceph 2024-03-24T00:10:35.575 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 777 /etc/ceph 2024-03-24T00:10:35.602 INFO:tasks.cephadm:Writing seed config... 2024-03-24T00:10:35.604 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-24T00:10:35.604 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-24T00:10:35.604 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-24T00:10:35.604 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-24T00:10:35.604 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-24T00:10:35.604 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-24T00:10:35.604 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-24T00:10:35.604 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-24T00:10:35.604 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-24T00:10:35.604 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-24T00:10:35.604 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-24T00:10:35.604 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-24T00:10:35.605 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-24T00:10:35.605 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-24T00:10:35.605 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-24T00:10:35.605 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-24T00:10:35.605 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-24T00:10:35.605 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-24T00:10:35.605 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-24T00:10:35.605 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-24T00:10:35.605 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-24T00:10:35.605 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-03-24T00:10:35.605 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-24T00:10:35.605 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-24T00:10:35.605 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-24T00:10:35.605 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-24T00:10:35.605 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-24T00:10:35.606 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-24T00:10:35.606 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-24T00:10:35.606 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-24T00:10:35.606 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-24T00:10:35.606 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-24T00:10:35.606 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-24T00:10:35.606 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-24T00:10:35.606 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-24T00:10:35.606 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-24T00:10:35.606 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-24T00:10:35.606 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-24T00:10:35.606 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-24T00:10:35.606 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-24T00:10:35.606 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-24T00:10:35.607 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-24T00:10:35.607 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-03-24T00:10:35.607 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-24T00:10:35.607 DEBUG:teuthology.orchestra.run.smithi078:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-24T00:10:35.622 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = c7e2806a-e972-11ee-95cd-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-24T00:10:35.622 DEBUG:teuthology.orchestra.run.smithi078:mon.a> sudo journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mon.a.service 2024-03-24T00:10:35.665 DEBUG:teuthology.orchestra.run.smithi078:mgr.x> sudo journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mgr.x.service 2024-03-24T00:10:35.707 INFO:tasks.cephadm:Bootstrapping... 2024-03-24T00:10:35.707 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 -v bootstrap --fsid c7e2806a-e972-11ee-95cd-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.78 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-24T00:10:35.898 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-03-24T00:10:35.899 INFO:teuthology.orchestra.run.smithi078.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033', '-v', 'bootstrap', '--fsid', 'c7e2806a-e972-11ee-95cd-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.78', '--skip-admin-label'] 2024-03-24T00:10:35.923 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T00:10:35.924 INFO:teuthology.orchestra.run.smithi078.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-24T00:10:35.924 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying podman|docker is present... 2024-03-24T00:10:35.947 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T00:10:35.947 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying lvm2 is present... 2024-03-24T00:10:35.947 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying time synchronization is in place... 2024-03-24T00:10:35.955 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-24T00:10:35.955 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T00:10:35.961 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-24T00:10:35.961 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2024-03-24T00:10:35.967 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout enabled 2024-03-24T00:10:35.973 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout active 2024-03-24T00:10:35.973 INFO:teuthology.orchestra.run.smithi078.stdout:Unit chronyd.service is enabled and running 2024-03-24T00:10:35.973 INFO:teuthology.orchestra.run.smithi078.stdout:Repeating the final host check... 2024-03-24T00:10:35.997 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T00:10:35.997 INFO:teuthology.orchestra.run.smithi078.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-24T00:10:35.997 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl is present 2024-03-24T00:10:35.997 INFO:teuthology.orchestra.run.smithi078.stdout:lvcreate is present 2024-03-24T00:10:36.003 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-24T00:10:36.003 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T00:10:36.009 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-24T00:10:36.009 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2024-03-24T00:10:36.015 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout enabled 2024-03-24T00:10:36.021 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout active 2024-03-24T00:10:36.021 INFO:teuthology.orchestra.run.smithi078.stdout:Unit chronyd.service is enabled and running 2024-03-24T00:10:36.021 INFO:teuthology.orchestra.run.smithi078.stdout:Host looks OK 2024-03-24T00:10:36.021 INFO:teuthology.orchestra.run.smithi078.stdout:Cluster fsid: c7e2806a-e972-11ee-95cd-87774f69a715 2024-03-24T00:10:36.021 INFO:teuthology.orchestra.run.smithi078.stdout:Acquiring lock 140524682190272 on /run/cephadm/c7e2806a-e972-11ee-95cd-87774f69a715.lock 2024-03-24T00:10:36.021 INFO:teuthology.orchestra.run.smithi078.stdout:Lock 140524682190272 acquired on /run/cephadm/c7e2806a-e972-11ee-95cd-87774f69a715.lock 2024-03-24T00:10:36.022 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying IP 172.21.15.78 port 3300 ... 2024-03-24T00:10:36.022 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying IP 172.21.15.78 port 6789 ... 2024-03-24T00:10:36.022 INFO:teuthology.orchestra.run.smithi078.stdout:Base mon IP(s) is [172.21.15.78:3300, 172.21.15.78:6789], mon addrv is [v2:172.21.15.78:3300,v1:172.21.15.78:6789] 2024-03-24T00:10:36.026 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.78 metric 100 2024-03-24T00:10:36.026 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.78 metric 100 2024-03-24T00:10:36.028 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-24T00:10:36.028 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-24T00:10:36.028 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1769sec hoplimit 64 pref medium 2024-03-24T00:10:36.031 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-24T00:10:36.031 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-24T00:10:36.031 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-24T00:10:36.031 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-24T00:10:36.031 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d4dd/64 scope link 2024-03-24T00:10:36.031 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-24T00:10:36.032 INFO:teuthology.orchestra.run.smithi078.stdout:Mon IP `172.21.15.78` is in CIDR network `172.21.0.0/20` 2024-03-24T00:10:36.032 INFO:teuthology.orchestra.run.smithi078.stdout:Mon IP `172.21.15.78` is in CIDR network `172.21.0.0/20` 2024-03-24T00:10:36.032 INFO:teuthology.orchestra.run.smithi078.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-24T00:10:36.032 INFO:teuthology.orchestra.run.smithi078.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-24T00:10:36.033 INFO:teuthology.orchestra.run.smithi078.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-24T00:10:36.449 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stdout c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 2024-03-24T00:10:36.449 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-24T00:10:36.449 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Getting image source signatures 2024-03-24T00:10:36.449 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Copying blob sha256:4dd83c6ee8921fe17396d77c31e205985ae94c87d3f752465b5f04aaf9ae340a 2024-03-24T00:10:36.449 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-24T00:10:36.449 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Copying config sha256:c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 2024-03-24T00:10:36.449 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-24T00:10:37.302 INFO:teuthology.orchestra.run.smithi078.stdout:ceph: stdout ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:10:37.302 INFO:teuthology.orchestra.run.smithi078.stdout:Ceph version: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:10:37.302 INFO:teuthology.orchestra.run.smithi078.stdout:Extracting ceph user uid/gid from container image... 2024-03-24T00:10:38.200 INFO:teuthology.orchestra.run.smithi078.stdout:stat: stdout 167 167 2024-03-24T00:10:38.200 INFO:teuthology.orchestra.run.smithi078.stdout:Creating initial keys... 2024-03-24T00:10:39.088 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph-authtool: stdout AQB+b/9lqddKHxAAxXZmeGliSoHoA6q6qGoBNg== 2024-03-24T00:10:39.951 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph-authtool: stdout AQB/b/9l5V7jFRAADpX5hJ0rgfVVrkMtW5OhUQ== 2024-03-24T00:10:40.790 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph-authtool: stdout AQCAb/9lgq+CEBAAWZcbSCvRLNxcL210VjkIyg== 2024-03-24T00:10:40.790 INFO:teuthology.orchestra.run.smithi078.stdout:Creating initial monmap... 2024-03-24T00:10:41.763 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T00:10:41.763 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-24T00:10:41.763 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to c7e2806a-e972-11ee-95cd-87774f69a715 2024-03-24T00:10:41.763 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T00:10:41.763 INFO:teuthology.orchestra.run.smithi078.stdout:monmaptool for a [v2:172.21.15.78:3300,v1:172.21.15.78:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T00:10:41.763 INFO:teuthology.orchestra.run.smithi078.stdout:setting min_mon_release = quincy 2024-03-24T00:10:41.763 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: set fsid to c7e2806a-e972-11ee-95cd-87774f69a715 2024-03-24T00:10:41.763 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T00:10:41.764 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:10:41.764 INFO:teuthology.orchestra.run.smithi078.stdout:Creating mon... 2024-03-24T00:10:42.693 INFO:teuthology.orchestra.run.smithi078.stdout:create mon.a on 2024-03-24T00:10:42.896 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-24T00:10:43.040 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-24T00:10:43.235 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-c7e2806a-e972-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-c7e2806a-e972-11ee-95cd-87774f69a715.target. 2024-03-24T00:10:43.235 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-c7e2806a-e972-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-c7e2806a-e972-11ee-95cd-87774f69a715.target. 2024-03-24T00:10:43.477 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mon.a 2024-03-24T00:10:43.477 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mon.a.service: Unit ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mon.a.service not loaded. 2024-03-24T00:10:43.640 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c7e2806a-e972-11ee-95cd-87774f69a715.target.wants/ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mon.a.service → /etc/systemd/system/ceph-c7e2806a-e972-11ee-95cd-87774f69a715@.service. 2024-03-24T00:10:44.126 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T00:10:44.126 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout disabled 2024-03-24T00:10:44.134 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T00:10:44.134 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2024-03-24T00:10:44.134 INFO:teuthology.orchestra.run.smithi078.stdout:firewalld.service is not enabled 2024-03-24T00:10:44.134 INFO:teuthology.orchestra.run.smithi078.stdout:Not possible to enable service . firewalld.service is not available 2024-03-24T00:10:44.134 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mon to start... 2024-03-24T00:10:44.134 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mon... 2024-03-24T00:10:45.226 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:45 smithi078 ceph-mon[32049]: from='client.? 172.21.15.78:0/3851866844' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-24T00:10:45.228 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout cluster: 2024-03-24T00:10:45.235 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout id: c7e2806a-e972-11ee-95cd-87774f69a715 2024-03-24T00:10:45.235 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-24T00:10:45.235 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-24T00:10:45.235 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout services: 2024-03-24T00:10:45.235 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.522328s) 2024-03-24T00:10:45.235 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-24T00:10:45.235 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-24T00:10:45.235 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-24T00:10:45.235 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout data: 2024-03-24T00:10:45.236 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-24T00:10:45.236 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-24T00:10:45.236 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-24T00:10:45.236 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout pgs: 2024-03-24T00:10:45.236 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-24T00:10:45.236 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.578+0000 7f403bd9c700 1 Processor -- start 2024-03-24T00:10:45.236 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.578+0000 7f403bd9c700 1 -- start start 2024-03-24T00:10:45.236 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.578+0000 7f403bd9c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4034107a00 0x7f4034107e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:10:45.236 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.578+0000 7f403bd9c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40341083f0 con 0x7f4034107a00 2024-03-24T00:10:45.236 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.578+0000 7f4039b38700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4034107a00 0x7f4034107e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:10:45.236 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.578+0000 7f4039b38700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4034107a00 0x7f4034107e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53506/0 (socket says 172.21.15.78:53506) 2024-03-24T00:10:45.236 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.578+0000 7f4039b38700 1 -- 172.21.15.78:0/2558875584 learned_addr learned my addr 172.21.15.78:0/2558875584 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:10:45.236 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.579+0000 7f4039b38700 1 -- 172.21.15.78:0/2558875584 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4034108c00 con 0x7f4034107a00 2024-03-24T00:10:45.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.579+0000 7f4039b38700 1 --2- 172.21.15.78:0/2558875584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4034107a00 0x7f4034107e20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f40300098a0 tx=0x7f4030009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8a577d76c2aa4f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:10:45.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.579+0000 7f4038b36700 1 -- 172.21.15.78:0/2558875584 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4030004160 con 0x7f4034107a00 2024-03-24T00:10:45.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.579+0000 7f4038b36700 1 -- 172.21.15.78:0/2558875584 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4030004300 con 0x7f4034107a00 2024-03-24T00:10:45.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.579+0000 7f4038b36700 1 -- 172.21.15.78:0/2558875584 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f40300036a0 con 0x7f4034107a00 2024-03-24T00:10:45.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.579+0000 7f403bd9c700 1 -- 172.21.15.78:0/2558875584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4034107a00 msgr2=0x7f4034107e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:10:45.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.579+0000 7f403bd9c700 1 --2- 172.21.15.78:0/2558875584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4034107a00 0x7f4034107e20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f40300098a0 tx=0x7f4030009bb0 comp rx=0 tx=0).stop 2024-03-24T00:10:45.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.579+0000 7f403bd9c700 1 -- 172.21.15.78:0/2558875584 shutdown_connections 2024-03-24T00:10:45.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.579+0000 7f403bd9c700 1 --2- 172.21.15.78:0/2558875584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4034107a00 0x7f4034107e20 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:10:45.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.579+0000 7f403bd9c700 1 -- 172.21.15.78:0/2558875584 >> 172.21.15.78:0/2558875584 conn(0x7f4034103040 msgr2=0x7f4034105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:10:45.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.580+0000 7f403bd9c700 1 -- 172.21.15.78:0/2558875584 shutdown_connections 2024-03-24T00:10:45.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.580+0000 7f403bd9c700 1 -- 172.21.15.78:0/2558875584 wait complete. 2024-03-24T00:10:45.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.580+0000 7f403bd9c700 1 Processor -- start 2024-03-24T00:10:45.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.580+0000 7f403bd9c700 1 -- start start 2024-03-24T00:10:45.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.580+0000 7f403bd9c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4034107a00 0x7f403419b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:10:45.238 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.580+0000 7f403bd9c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4034109130 con 0x7f4034107a00 2024-03-24T00:10:45.238 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.580+0000 7f4039b38700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4034107a00 0x7f403419b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:10:45.238 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.580+0000 7f4039b38700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4034107a00 0x7f403419b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53518/0 (socket says 172.21.15.78:53518) 2024-03-24T00:10:45.238 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.580+0000 7f4039b38700 1 -- 172.21.15.78:0/3851866844 learned_addr learned my addr 172.21.15.78:0/3851866844 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:10:45.238 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.581+0000 7f4039b38700 1 -- 172.21.15.78:0/3851866844 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f403419b640 con 0x7f4034107a00 2024-03-24T00:10:45.238 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.581+0000 7f4039b38700 1 --2- 172.21.15.78:0/3851866844 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4034107a00 0x7f403419b100 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4030000c00 tx=0x7f4030004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:10:45.238 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.581+0000 7f402affd700 1 -- 172.21.15.78:0/3851866844 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4030003db0 con 0x7f4034107a00 2024-03-24T00:10:45.238 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.581+0000 7f402affd700 1 -- 172.21.15.78:0/3851866844 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4030011630 con 0x7f4034107a00 2024-03-24T00:10:45.238 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.581+0000 7f403bd9c700 1 -- 172.21.15.78:0/3851866844 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f403419b8d0 con 0x7f4034107a00 2024-03-24T00:10:45.238 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.581+0000 7f402affd700 1 -- 172.21.15.78:0/3851866844 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4030011950 con 0x7f4034107a00 2024-03-24T00:10:45.238 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.581+0000 7f403bd9c700 1 -- 172.21.15.78:0/3851866844 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f403419e5c0 con 0x7f4034107a00 2024-03-24T00:10:45.238 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.581+0000 7f402affd700 1 -- 172.21.15.78:0/3851866844 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f4030011af0 con 0x7f4034107a00 2024-03-24T00:10:45.239 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.581+0000 7f402affd700 1 -- 172.21.15.78:0/3851866844 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f403001c450 con 0x7f4034107a00 2024-03-24T00:10:45.239 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.582+0000 7f403bd9c700 1 -- 172.21.15.78:0/3851866844 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4034069220 con 0x7f4034107a00 2024-03-24T00:10:45.239 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.584+0000 7f402affd700 1 -- 172.21.15.78:0/3851866844 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f4030032020 con 0x7f4034107a00 2024-03-24T00:10:45.239 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.653+0000 7f403bd9c700 1 -- 172.21.15.78:0/3851866844 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f403402dbb0 con 0x7f4034107a00 2024-03-24T00:10:45.239 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.654+0000 7f402affd700 1 -- 172.21.15.78:0/3851866844 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f4030004260 con 0x7f4034107a00 2024-03-24T00:10:45.239 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.655+0000 7f403bd9c700 1 -- 172.21.15.78:0/3851866844 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4034107a00 msgr2=0x7f403419b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:10:45.239 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.655+0000 7f403bd9c700 1 --2- 172.21.15.78:0/3851866844 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4034107a00 0x7f403419b100 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4030000c00 tx=0x7f4030004750 comp rx=0 tx=0).stop 2024-03-24T00:10:45.239 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.655+0000 7f403bd9c700 1 -- 172.21.15.78:0/3851866844 shutdown_connections 2024-03-24T00:10:45.239 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.655+0000 7f403bd9c700 1 --2- 172.21.15.78:0/3851866844 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4034107a00 0x7f403419b100 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:10:45.239 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.655+0000 7f403bd9c700 1 -- 172.21.15.78:0/3851866844 >> 172.21.15.78:0/3851866844 conn(0x7f4034103040 msgr2=0x7f4034191e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:10:45.239 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.655+0000 7f403bd9c700 1 -- 172.21.15.78:0/3851866844 shutdown_connections 2024-03-24T00:10:45.239 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:44.655+0000 7f403bd9c700 1 -- 172.21.15.78:0/3851866844 wait complete. 2024-03-24T00:10:45.239 INFO:teuthology.orchestra.run.smithi078.stdout:mon is available 2024-03-24T00:10:45.240 INFO:teuthology.orchestra.run.smithi078.stdout:Assimilating anything we can from ceph.conf... 2024-03-24T00:10:46.224 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-24T00:10:46.224 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [global] 2024-03-24T00:10:46.224 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout fsid = c7e2806a-e972-11ee-95cd-87774f69a715 2024-03-24T00:10:46.225 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.78:3300,v1:172.21.15.78:6789] 2024-03-24T00:10:46.225 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-24T00:10:46.225 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-24T00:10:46.225 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-24T00:10:46.225 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-24T00:10:46.225 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-24T00:10:46.225 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-24T00:10:46.225 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-24T00:10:46.225 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-24T00:10:46.225 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [osd] 2024-03-24T00:10:46.225 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-24T00:10:46.225 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-24T00:10:46.225 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-24T00:10:46.226 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-24T00:10:46.226 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-24T00:10:46.226 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.625+0000 7f0bc4190700 1 Processor -- start 2024-03-24T00:10:46.226 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.625+0000 7f0bc4190700 1 -- start start 2024-03-24T00:10:46.226 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.625+0000 7f0bc4190700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc1084a0 0x7f0bbc10a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:10:46.226 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.625+0000 7f0bc4190700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc07ae90 con 0x7f0bbc1084a0 2024-03-24T00:10:46.226 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.625+0000 7f0bc1f2c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc1084a0 0x7f0bbc10a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:10:46.226 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.625+0000 7f0bc1f2c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc1084a0 0x7f0bbc10a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53524/0 (socket says 172.21.15.78:53524) 2024-03-24T00:10:46.226 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.625+0000 7f0bc1f2c700 1 -- 172.21.15.78:0/1552062178 learned_addr learned my addr 172.21.15.78:0/1552062178 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:10:46.226 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.625+0000 7f0bc1f2c700 1 -- 172.21.15.78:0/1552062178 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bbc10add0 con 0x7f0bbc1084a0 2024-03-24T00:10:46.226 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.626+0000 7f0bc1f2c700 1 --2- 172.21.15.78:0/1552062178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc1084a0 0x7f0bbc10a890 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f0bac0098a0 tx=0x7f0bac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=618f3cf47f9a687e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:10:46.226 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.626+0000 7f0bc0f2a700 1 -- 172.21.15.78:0/1552062178 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0bac004160 con 0x7f0bbc1084a0 2024-03-24T00:10:46.226 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.626+0000 7f0bc0f2a700 1 -- 172.21.15.78:0/1552062178 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bac004300 con 0x7f0bbc1084a0 2024-03-24T00:10:46.227 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.626+0000 7f0bc0f2a700 1 -- 172.21.15.78:0/1552062178 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0bac0036a0 con 0x7f0bbc1084a0 2024-03-24T00:10:46.227 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.626+0000 7f0bc4190700 1 -- 172.21.15.78:0/1552062178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc1084a0 msgr2=0x7f0bbc10a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:10:46.227 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.626+0000 7f0bc4190700 1 --2- 172.21.15.78:0/1552062178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc1084a0 0x7f0bbc10a890 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f0bac0098a0 tx=0x7f0bac009bb0 comp rx=0 tx=0).stop 2024-03-24T00:10:46.227 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.626+0000 7f0bc4190700 1 -- 172.21.15.78:0/1552062178 shutdown_connections 2024-03-24T00:10:46.227 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.626+0000 7f0bc4190700 1 --2- 172.21.15.78:0/1552062178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc1084a0 0x7f0bbc10a890 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:10:46.227 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.626+0000 7f0bc4190700 1 -- 172.21.15.78:0/1552062178 >> 172.21.15.78:0/1552062178 conn(0x7f0bbc101fb0 msgr2=0x7f0bbc1043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:10:46.227 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.626+0000 7f0bc4190700 1 -- 172.21.15.78:0/1552062178 shutdown_connections 2024-03-24T00:10:46.227 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.626+0000 7f0bc4190700 1 -- 172.21.15.78:0/1552062178 wait complete. 2024-03-24T00:10:46.227 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.627+0000 7f0bc4190700 1 Processor -- start 2024-03-24T00:10:46.227 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.627+0000 7f0bc4190700 1 -- start start 2024-03-24T00:10:46.227 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.627+0000 7f0bc4190700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc1084a0 0x7f0bbc19b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:10:46.227 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.627+0000 7f0bc4190700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc10b240 con 0x7f0bbc1084a0 2024-03-24T00:10:46.227 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.627+0000 7f0bc1f2c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc1084a0 0x7f0bbc19b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:10:46.228 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.627+0000 7f0bc1f2c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc1084a0 0x7f0bbc19b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53530/0 (socket says 172.21.15.78:53530) 2024-03-24T00:10:46.228 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.627+0000 7f0bc1f2c700 1 -- 172.21.15.78:0/3308806467 learned_addr learned my addr 172.21.15.78:0/3308806467 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:10:46.228 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.627+0000 7f0bc1f2c700 1 -- 172.21.15.78:0/3308806467 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bbc19b5e0 con 0x7f0bbc1084a0 2024-03-24T00:10:46.228 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.628+0000 7f0bc1f2c700 1 --2- 172.21.15.78:0/3308806467 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc1084a0 0x7f0bbc19b0a0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f0bac000c00 tx=0x7f0bac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:10:46.228 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.628+0000 7f0bb2ffd700 1 -- 172.21.15.78:0/3308806467 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0bac003db0 con 0x7f0bbc1084a0 2024-03-24T00:10:46.228 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.628+0000 7f0bb2ffd700 1 -- 172.21.15.78:0/3308806467 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bac011630 con 0x7f0bbc1084a0 2024-03-24T00:10:46.228 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.628+0000 7f0bc4190700 1 -- 172.21.15.78:0/3308806467 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bbc19b870 con 0x7f0bbc1084a0 2024-03-24T00:10:46.228 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.628+0000 7f0bb2ffd700 1 -- 172.21.15.78:0/3308806467 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0bac011950 con 0x7f0bbc1084a0 2024-03-24T00:10:46.228 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.628+0000 7f0bc4190700 1 -- 172.21.15.78:0/3308806467 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bbc19e560 con 0x7f0bbc1084a0 2024-03-24T00:10:46.228 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.628+0000 7f0bb2ffd700 1 -- 172.21.15.78:0/3308806467 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f0bac011af0 con 0x7f0bbc1084a0 2024-03-24T00:10:46.228 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.628+0000 7f0bb2ffd700 1 -- 172.21.15.78:0/3308806467 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0bac01c450 con 0x7f0bbc1084a0 2024-03-24T00:10:46.228 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.629+0000 7f0bc4190700 1 -- 172.21.15.78:0/3308806467 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bbc069220 con 0x7f0bbc1084a0 2024-03-24T00:10:46.228 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.631+0000 7f0bb2ffd700 1 -- 172.21.15.78:0/3308806467 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f0bac032020 con 0x7f0bbc1084a0 2024-03-24T00:10:46.229 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.699+0000 7f0bc4190700 1 -- 172.21.15.78:0/3308806467 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f0bbc02dbb0 con 0x7f0bbc1084a0 2024-03-24T00:10:46.229 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.702+0000 7f0bb2ffd700 1 -- 172.21.15.78:0/3308806467 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f0bac004260 con 0x7f0bbc1084a0 2024-03-24T00:10:46.229 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.703+0000 7f0bb2ffd700 1 -- 172.21.15.78:0/3308806467 <== mon.0 v2:172.21.15.78:3300/0 8 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0bac01b450 con 0x7f0bbc1084a0 2024-03-24T00:10:46.229 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.704+0000 7f0bc4190700 1 -- 172.21.15.78:0/3308806467 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc1084a0 msgr2=0x7f0bbc19b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:10:46.229 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.704+0000 7f0bc4190700 1 --2- 172.21.15.78:0/3308806467 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc1084a0 0x7f0bbc19b0a0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f0bac000c00 tx=0x7f0bac004750 comp rx=0 tx=0).stop 2024-03-24T00:10:46.229 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.704+0000 7f0bc4190700 1 -- 172.21.15.78:0/3308806467 shutdown_connections 2024-03-24T00:10:46.229 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.704+0000 7f0bc4190700 1 --2- 172.21.15.78:0/3308806467 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc1084a0 0x7f0bbc19b0a0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:10:46.229 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.704+0000 7f0bc4190700 1 -- 172.21.15.78:0/3308806467 >> 172.21.15.78:0/3308806467 conn(0x7f0bbc101fb0 msgr2=0x7f0bbc1042c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:10:46.229 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.704+0000 7f0bc4190700 1 -- 172.21.15.78:0/3308806467 shutdown_connections 2024-03-24T00:10:46.229 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:45.704+0000 7f0bc4190700 1 -- 172.21.15.78:0/3308806467 wait complete. 2024-03-24T00:10:46.229 INFO:teuthology.orchestra.run.smithi078.stdout:Generating new minimal ceph.conf... 2024-03-24T00:10:46.324 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:46 smithi078 ceph-mon[32049]: from='client.? 172.21.15.78:0/3308806467' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T00:10:46.325 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:46 smithi078 ceph-mon[32049]: from='client.? 172.21.15.78:0/3308806467' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-24T00:10:47.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:47 smithi078 ceph-mon[32049]: from='client.? 172.21.15.78:0/714311037' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:10:47.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.624+0000 7f7daf769700 1 Processor -- start 2024-03-24T00:10:47.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.624+0000 7f7daf769700 1 -- start start 2024-03-24T00:10:47.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.624+0000 7f7daf769700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da8107ac0 0x7f7da8107ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:10:47.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.624+0000 7f7daf769700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da81084b0 con 0x7f7da8107ac0 2024-03-24T00:10:47.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.624+0000 7f7dad505700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da8107ac0 0x7f7da8107ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:10:47.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.624+0000 7f7dad505700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da8107ac0 0x7f7da8107ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53532/0 (socket says 172.21.15.78:53532) 2024-03-24T00:10:47.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.624+0000 7f7dad505700 1 -- 172.21.15.78:0/1195111607 learned_addr learned my addr 172.21.15.78:0/1195111607 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:10:47.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.624+0000 7f7dad505700 1 -- 172.21.15.78:0/1195111607 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7da8108c70 con 0x7f7da8107ac0 2024-03-24T00:10:47.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.625+0000 7f7dad505700 1 --2- 172.21.15.78:0/1195111607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da8107ac0 0x7f7da8107ee0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7da40098a0 tx=0x7f7da4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b72b29060884f8c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:10:47.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.625+0000 7f7d9ffff700 1 -- 172.21.15.78:0/1195111607 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7da4004160 con 0x7f7da8107ac0 2024-03-24T00:10:47.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.625+0000 7f7d9ffff700 1 -- 172.21.15.78:0/1195111607 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7da4004300 con 0x7f7da8107ac0 2024-03-24T00:10:47.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.625+0000 7f7d9ffff700 1 -- 172.21.15.78:0/1195111607 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7da4011b50 con 0x7f7da8107ac0 2024-03-24T00:10:47.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.625+0000 7f7daf769700 1 -- 172.21.15.78:0/1195111607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da8107ac0 msgr2=0x7f7da8107ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:10:47.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.625+0000 7f7daf769700 1 --2- 172.21.15.78:0/1195111607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da8107ac0 0x7f7da8107ee0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7da40098a0 tx=0x7f7da4009bb0 comp rx=0 tx=0).stop 2024-03-24T00:10:47.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.625+0000 7f7daf769700 1 -- 172.21.15.78:0/1195111607 shutdown_connections 2024-03-24T00:10:47.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.625+0000 7f7daf769700 1 --2- 172.21.15.78:0/1195111607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da8107ac0 0x7f7da8107ee0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:10:47.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.625+0000 7f7daf769700 1 -- 172.21.15.78:0/1195111607 >> 172.21.15.78:0/1195111607 conn(0x7f7da81030c0 msgr2=0x7f7da81054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:10:47.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.626+0000 7f7daf769700 1 -- 172.21.15.78:0/1195111607 shutdown_connections 2024-03-24T00:10:47.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.626+0000 7f7daf769700 1 -- 172.21.15.78:0/1195111607 wait complete. 2024-03-24T00:10:47.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.626+0000 7f7daf769700 1 Processor -- start 2024-03-24T00:10:47.294 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.626+0000 7f7daf769700 1 -- start start 2024-03-24T00:10:47.294 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.626+0000 7f7daf769700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da8107ac0 0x7f7da819dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:10:47.294 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.626+0000 7f7daf769700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da810c020 con 0x7f7da8107ac0 2024-03-24T00:10:47.294 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.626+0000 7f7dad505700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da8107ac0 0x7f7da819dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:10:47.294 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.626+0000 7f7dad505700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da8107ac0 0x7f7da819dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53542/0 (socket says 172.21.15.78:53542) 2024-03-24T00:10:47.294 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.626+0000 7f7dad505700 1 -- 172.21.15.78:0/714311037 learned_addr learned my addr 172.21.15.78:0/714311037 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:10:47.294 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.627+0000 7f7dad505700 1 -- 172.21.15.78:0/714311037 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7da819e190 con 0x7f7da8107ac0 2024-03-24T00:10:47.294 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.627+0000 7f7dad505700 1 --2- 172.21.15.78:0/714311037 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da8107ac0 0x7f7da819dc50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7da4009fb0 tx=0x7f7da4011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:10:47.294 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.627+0000 7f7d9e7fc700 1 -- 172.21.15.78:0/714311037 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7da40038b0 con 0x7f7da8107ac0 2024-03-24T00:10:47.294 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.627+0000 7f7d9e7fc700 1 -- 172.21.15.78:0/714311037 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7da4011e50 con 0x7f7da8107ac0 2024-03-24T00:10:47.294 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.627+0000 7f7daf769700 1 -- 172.21.15.78:0/714311037 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7da819e420 con 0x7f7da8107ac0 2024-03-24T00:10:47.294 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.627+0000 7f7d9e7fc700 1 -- 172.21.15.78:0/714311037 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7da4019550 con 0x7f7da8107ac0 2024-03-24T00:10:47.294 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.627+0000 7f7daf769700 1 -- 172.21.15.78:0/714311037 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7da81a1110 con 0x7f7da8107ac0 2024-03-24T00:10:47.295 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.627+0000 7f7d9e7fc700 1 -- 172.21.15.78:0/714311037 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f7da4019be0 con 0x7f7da8107ac0 2024-03-24T00:10:47.295 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.627+0000 7f7d9e7fc700 1 -- 172.21.15.78:0/714311037 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7da4024730 con 0x7f7da8107ac0 2024-03-24T00:10:47.295 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.628+0000 7f7daf769700 1 -- 172.21.15.78:0/714311037 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7da8069220 con 0x7f7da8107ac0 2024-03-24T00:10:47.295 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.630+0000 7f7d9e7fc700 1 -- 172.21.15.78:0/714311037 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f7da4043dd0 con 0x7f7da8107ac0 2024-03-24T00:10:47.295 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.697+0000 7f7daf769700 1 -- 172.21.15.78:0/714311037 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f7da81a12e0 con 0x7f7da8107ac0 2024-03-24T00:10:47.295 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.697+0000 7f7d9e7fc700 1 -- 172.21.15.78:0/714311037 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f7da4032030 con 0x7f7da8107ac0 2024-03-24T00:10:47.295 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.698+0000 7f7daf769700 1 -- 172.21.15.78:0/714311037 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da8107ac0 msgr2=0x7f7da819dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:10:47.295 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.698+0000 7f7daf769700 1 --2- 172.21.15.78:0/714311037 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da8107ac0 0x7f7da819dc50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7da4009fb0 tx=0x7f7da4011bf0 comp rx=0 tx=0).stop 2024-03-24T00:10:47.295 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.698+0000 7f7daf769700 1 -- 172.21.15.78:0/714311037 shutdown_connections 2024-03-24T00:10:47.295 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.698+0000 7f7daf769700 1 --2- 172.21.15.78:0/714311037 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da8107ac0 0x7f7da819dc50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:10:47.295 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.698+0000 7f7daf769700 1 -- 172.21.15.78:0/714311037 >> 172.21.15.78:0/714311037 conn(0x7f7da81030c0 msgr2=0x7f7da8103ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:10:47.295 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.698+0000 7f7daf769700 1 -- 172.21.15.78:0/714311037 shutdown_connections 2024-03-24T00:10:47.295 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:46.698+0000 7f7daf769700 1 -- 172.21.15.78:0/714311037 wait complete. 2024-03-24T00:10:47.295 INFO:teuthology.orchestra.run.smithi078.stdout:Restarting the monitor... 2024-03-24T00:10:47.510 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:47 smithi078 systemd[1]: Stopping Ceph mon.a for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:10:47.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:47 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mon-a[32045]: 2024-03-24T00:10:47.510+0000 7fa2f7bf3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T00:10:47.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:47 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mon-a[32045]: 2024-03-24T00:10:47.510+0000 7fa2f7bf3700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-24T00:10:47.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:47 smithi078 podman[32297]: 2024-03-24 00:10:47.694531162 +0000 UTC m=+0.276816422 container died 6265ef6a43a77c98690b793fa4a39d5bacb8f426dde754c23153141fd7815825 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mon-a, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-24T00:10:48.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:47 smithi078 podman[32297]: 2024-03-24 00:10:47.976816288 +0000 UTC m=+0.559101623 container cleanup 6265ef6a43a77c98690b793fa4a39d5bacb8f426dde754c23153141fd7815825 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mon-a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-24T00:10:48.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:47 smithi078 bash[32297]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mon-a 2024-03-24T00:10:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 podman[32311]: 2024-03-24 00:10:48.210838552 +0000 UTC m=+0.520190375 container remove 6265ef6a43a77c98690b793fa4a39d5bacb8f426dde754c23153141fd7815825 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-24T00:10:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-24T00:10:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 systemd[1]: Stopped Ceph mon.a for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:10:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 systemd[1]: Starting Ceph mon.a for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:10:48.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 podman[32410]: 2024-03-24 00:10:48.536551746 +0000 UTC m=+0.096656061 container create c766f205ef0efd9d287f7e718d1e87cb3ca7b62fe6851449dbfa54159e05dc51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:10:48.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 podman[32410]: 2024-03-24 00:10:48.45761606 +0000 UTC m=+0.017720389 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:10:48.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 podman[32410]: 2024-03-24 00:10:48.7221853 +0000 UTC m=+0.282289636 container init c766f205ef0efd9d287f7e718d1e87cb3ca7b62fe6851449dbfa54159e05dc51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mon-a, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-24T00:10:48.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 podman[32410]: 2024-03-24 00:10:48.724914002 +0000 UTC m=+0.285018321 container start c766f205ef0efd9d287f7e718d1e87cb3ca7b62fe6851449dbfa54159e05dc51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mon-a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:10:48.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T00:10:48.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev), process ceph-mon, pid 2 2024-03-24T00:10:48.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: pidfile_write: ignore empty --pid-file 2024-03-24T00:10:48.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: load: jerasure load: lrc 2024-03-24T00:10:48.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: RocksDB version: 7.9.2 2024-03-24T00:10:48.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Git sha 0 2024-03-24T00:10:48.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Compile date 2024-03-22 04:58:55 2024-03-24T00:10:48.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: DB SUMMARY 2024-03-24T00:10:48.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: DB Session ID: 3V110Q3RVX3C98YBBBSA 2024-03-24T00:10:48.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: CURRENT file: CURRENT 2024-03-24T00:10:48.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T00:10:48.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-24T00:10:48.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-24T00:10:48.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 99504 ; 2024-03-24T00:10:48.807 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.error_if_exists: 0 2024-03-24T00:10:48.807 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.create_if_missing: 0 2024-03-24T00:10:48.807 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.paranoid_checks: 1 2024-03-24T00:10:48.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T00:10:48.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T00:10:48.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T00:10:48.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.env: 0x55ea6824db20 2024-03-24T00:10:48.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T00:10:48.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.info_log: 0x55ea69a19920 2024-03-24T00:10:48.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T00:10:48.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.statistics: (nil) 2024-03-24T00:10:48.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.use_fsync: 0 2024-03-24T00:10:48.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.max_log_file_size: 0 2024-03-24T00:10:48.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T00:10:48.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T00:10:48.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T00:10:48.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T00:10:48.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.allow_fallocate: 1 2024-03-24T00:10:48.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T00:10:48.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T00:10:48.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.use_direct_reads: 0 2024-03-24T00:10:48.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T00:10:48.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T00:10:48.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.db_log_dir: 2024-03-24T00:10:48.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.wal_dir: 2024-03-24T00:10:48.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T00:10:48.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T00:10:48.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T00:10:48.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T00:10:48.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T00:10:48.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T00:10:48.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T00:10:48.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T00:10:48.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.write_buffer_manager: 0x55ea699ec1e0 2024-03-24T00:10:48.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T00:10:48.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T00:10:48.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T00:10:48.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T00:10:48.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T00:10:48.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T00:10:48.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T00:10:48.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T00:10:48.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.unordered_write: 0 2024-03-24T00:10:48.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T00:10:48.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T00:10:48.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T00:10:48.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T00:10:48.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.row_cache: None 2024-03-24T00:10:48.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.wal_filter: None 2024-03-24T00:10:48.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T00:10:48.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T00:10:48.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.two_write_queues: 0 2024-03-24T00:10:48.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T00:10:48.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.wal_compression: 0 2024-03-24T00:10:48.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.atomic_flush: 0 2024-03-24T00:10:48.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T00:10:48.813 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T00:10:48.813 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T00:10:48.813 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.log_readahead_size: 0 2024-03-24T00:10:48.813 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T00:10:48.813 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T00:10:48.813 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T00:10:48.813 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T00:10:48.813 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T00:10:48.813 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T00:10:48.813 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T00:10:48.813 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.max_background_jobs: 2 2024-03-24T00:10:48.813 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.max_background_compactions: -1 2024-03-24T00:10:48.813 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.max_subcompactions: 1 2024-03-24T00:10:48.814 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T00:10:48.814 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T00:10:48.814 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T00:10:48.814 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T00:10:48.814 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T00:10:48.814 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T00:10:48.814 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T00:10:48.814 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T00:10:48.814 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.max_open_files: -1 2024-03-24T00:10:48.814 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T00:10:48.814 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T00:10:48.814 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T00:10:48.814 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T00:10:48.815 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.max_background_flushes: -1 2024-03-24T00:10:48.815 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Compression algorithms supported: 2024-03-24T00:10:48.815 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T00:10:48.815 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: kZSTD supported: 0 2024-03-24T00:10:48.815 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: kXpressCompression supported: 0 2024-03-24T00:10:48.815 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T00:10:48.815 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: kZlibCompression supported: 1 2024-03-24T00:10:48.815 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: kSnappyCompression supported: 1 2024-03-24T00:10:48.815 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: kLZ4Compression supported: 1 2024-03-24T00:10:48.815 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: kBZip2Compression supported: 0 2024-03-24T00:10:48.815 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T00:10:48.815 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T00:10:48.815 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-24T00:10:48.815 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T00:10:48.816 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T00:10:48.816 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.merge_operator: 2024-03-24T00:10:48.816 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.compaction_filter: None 2024-03-24T00:10:48.816 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T00:10:48.816 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T00:10:48.816 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T00:10:48.816 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T00:10:48.816 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ea69a19a20) 2024-03-24T00:10:48.816 INFO:journalctl@ceph.mon.a.smithi078.stdout: cache_index_and_filter_blocks: 1 2024-03-24T00:10:48.816 INFO:journalctl@ceph.mon.a.smithi078.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T00:10:48.816 INFO:journalctl@ceph.mon.a.smithi078.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T00:10:48.816 INFO:journalctl@ceph.mon.a.smithi078.stdout: pin_top_level_index_and_filter: 1 2024-03-24T00:10:48.816 INFO:journalctl@ceph.mon.a.smithi078.stdout: index_type: 0 2024-03-24T00:10:48.816 INFO:journalctl@ceph.mon.a.smithi078.stdout: data_block_index_type: 0 2024-03-24T00:10:48.817 INFO:journalctl@ceph.mon.a.smithi078.stdout: index_shortening: 1 2024-03-24T00:10:48.817 INFO:journalctl@ceph.mon.a.smithi078.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T00:10:48.817 INFO:journalctl@ceph.mon.a.smithi078.stdout: checksum: 4 2024-03-24T00:10:48.817 INFO:journalctl@ceph.mon.a.smithi078.stdout: no_block_cache: 0 2024-03-24T00:10:48.817 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache: 0x55ea69a1f090 2024-03-24T00:10:48.817 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache_name: BinnedLRUCache 2024-03-24T00:10:48.817 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache_options: 2024-03-24T00:10:48.817 INFO:journalctl@ceph.mon.a.smithi078.stdout: capacity : 536870912 2024-03-24T00:10:48.817 INFO:journalctl@ceph.mon.a.smithi078.stdout: num_shard_bits : 4 2024-03-24T00:10:48.817 INFO:journalctl@ceph.mon.a.smithi078.stdout: strict_capacity_limit : 0 2024-03-24T00:10:48.817 INFO:journalctl@ceph.mon.a.smithi078.stdout: high_pri_pool_ratio: 0.000 2024-03-24T00:10:48.817 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache_compressed: (nil) 2024-03-24T00:10:48.817 INFO:journalctl@ceph.mon.a.smithi078.stdout: persistent_cache: (nil) 2024-03-24T00:10:48.818 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_size: 4096 2024-03-24T00:10:48.818 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_size_deviation: 10 2024-03-24T00:10:48.818 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_restart_interval: 16 2024-03-24T00:10:48.818 INFO:journalctl@ceph.mon.a.smithi078.stdout: index_block_restart_interval: 1 2024-03-24T00:10:48.818 INFO:journalctl@ceph.mon.a.smithi078.stdout: metadata_block_size: 4096 2024-03-24T00:10:48.818 INFO:journalctl@ceph.mon.a.smithi078.stdout: partition_filters: 0 2024-03-24T00:10:48.818 INFO:journalctl@ceph.mon.a.smithi078.stdout: use_delta_encoding: 1 2024-03-24T00:10:48.818 INFO:journalctl@ceph.mon.a.smithi078.stdout: filter_policy: bloomfilter 2024-03-24T00:10:48.818 INFO:journalctl@ceph.mon.a.smithi078.stdout: whole_key_filtering: 1 2024-03-24T00:10:48.818 INFO:journalctl@ceph.mon.a.smithi078.stdout: verify_compression: 0 2024-03-24T00:10:48.818 INFO:journalctl@ceph.mon.a.smithi078.stdout: read_amp_bytes_per_bit: 0 2024-03-24T00:10:48.818 INFO:journalctl@ceph.mon.a.smithi078.stdout: format_version: 5 2024-03-24T00:10:48.818 INFO:journalctl@ceph.mon.a.smithi078.stdout: enable_index_compression: 1 2024-03-24T00:10:48.818 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_align: 0 2024-03-24T00:10:48.819 INFO:journalctl@ceph.mon.a.smithi078.stdout: max_auto_readahead_size: 262144 2024-03-24T00:10:48.819 INFO:journalctl@ceph.mon.a.smithi078.stdout: prepopulate_block_cache: 0 2024-03-24T00:10:48.819 INFO:journalctl@ceph.mon.a.smithi078.stdout: initial_auto_readahead_size: 8192 2024-03-24T00:10:48.819 INFO:journalctl@ceph.mon.a.smithi078.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T00:10:48.819 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T00:10:48.819 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T00:10:48.819 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.compression: NoCompression 2024-03-24T00:10:48.819 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T00:10:48.819 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T00:10:48.819 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T00:10:48.819 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.num_levels: 7 2024-03-24T00:10:48.819 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T00:10:48.819 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T00:10:48.819 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T00:10:48.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T00:10:48.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T00:10:48.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T00:10:48.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T00:10:48.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T00:10:48.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T00:10:48.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T00:10:48.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T00:10:48.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T00:10:48.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T00:10:48.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T00:10:48.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T00:10:48.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T00:10:48.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T00:10:48.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T00:10:48.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T00:10:48.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T00:10:48.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T00:10:48.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T00:10:48.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T00:10:48.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T00:10:48.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T00:10:48.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T00:10:48.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T00:10:48.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T00:10:48.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T00:10:48.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T00:10:48.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T00:10:48.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T00:10:48.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T00:10:48.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T00:10:48.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T00:10:48.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T00:10:48.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T00:10:48.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T00:10:48.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T00:10:48.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T00:10:48.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T00:10:48.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T00:10:48.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T00:10:48.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T00:10:48.823 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T00:10:48.823 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T00:10:48.823 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T00:10:48.823 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T00:10:48.823 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T00:10:48.823 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T00:10:48.823 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T00:10:48.823 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T00:10:48.823 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T00:10:48.823 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.table_properties_collectors: 2024-03-24T00:10:48.823 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.inplace_update_support: 0 2024-03-24T00:10:48.823 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T00:10:48.823 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T00:10:48.823 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T00:10:48.824 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T00:10:48.824 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.bloom_locality: 0 2024-03-24T00:10:48.824 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.max_successive_merges: 0 2024-03-24T00:10:48.824 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T00:10:48.824 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T00:10:48.824 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T00:10:48.824 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T00:10:48.824 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.ttl: 2592000 2024-03-24T00:10:48.824 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T00:10:48.824 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T00:10:48.824 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T00:10:48.824 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.enable_blob_files: false 2024-03-24T00:10:48.824 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.min_blob_size: 0 2024-03-24T00:10:48.824 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T00:10:48.825 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T00:10:48.825 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T00:10:48.825 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T00:10:48.825 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T00:10:48.825 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T00:10:48.825 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T00:10:48.825 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T00:10:48.825 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-24T00:10:48.825 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-24T00:10:48.825 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 4d917bc8-8de4-451b-9eea-dea8d16e59e4 2024-03-24T00:10:48.825 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711239048753658, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-24T00:10:48.825 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-24T00:10:48.825 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711239048754640, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 94802, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 314, "table_properties": {"data_size": 92790, "index_size": 294, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13640, "raw_average_key_size": 49, "raw_value_size": 85315, "raw_average_value_size": 312, "num_data_blocks": 11, "num_entries": 273, "num_filter_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711239048, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "4d917bc8-8de4-451b-9eea-dea8d16e59e4", "db_session_id": "3V110Q3RVX3C98YBBBSA", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-24T00:10:48.825 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711239048754720, "job": 1, "event": "recovery_finished"} 2024-03-24T00:10:48.826 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-24T00:10:48.826 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T00:10:48.826 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55ea69b12000 2024-03-24T00:10:48.826 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: DB pointer 0x55ea69afc000 2024-03-24T00:10:48.826 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T00:10:48.826 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T00:10:48.826 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** DB Stats ** 2024-03-24T00:10:48.826 INFO:journalctl@ceph.mon.a.smithi078.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T00:10:48.826 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T00:10:48.826 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T00:10:48.826 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T00:10:48.826 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T00:10:48.826 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T00:10:48.826 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T00:10:48.827 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-03-24T00:10:48.827 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** Compaction Stats [default] ** 2024-03-24T00:10:48.827 INFO:journalctl@ceph.mon.a.smithi078.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T00:10:48.827 INFO:journalctl@ceph.mon.a.smithi078.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T00:10:48.827 INFO:journalctl@ceph.mon.a.smithi078.stdout: L0 2/0 94.42 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T00:10:48.827 INFO:journalctl@ceph.mon.a.smithi078.stdout: Sum 2/0 94.42 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T00:10:48.827 INFO:journalctl@ceph.mon.a.smithi078.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T00:10:48.827 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-03-24T00:10:48.827 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** Compaction Stats [default] ** 2024-03-24T00:10:48.827 INFO:journalctl@ceph.mon.a.smithi078.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T00:10:48.827 INFO:journalctl@ceph.mon.a.smithi078.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T00:10:48.827 INFO:journalctl@ceph.mon.a.smithi078.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 144.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T00:10:48.827 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-03-24T00:10:48.827 INFO:journalctl@ceph.mon.a.smithi078.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T00:10:48.828 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-03-24T00:10:48.828 INFO:journalctl@ceph.mon.a.smithi078.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T00:10:48.828 INFO:journalctl@ceph.mon.a.smithi078.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T00:10:48.828 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T00:10:48.828 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T00:10:48.828 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T00:10:48.828 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T00:10:48.828 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative compaction: 0.00 GB write, 30.80 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T00:10:48.828 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval compaction: 0.00 GB write, 30.80 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T00:10:48.828 INFO:journalctl@ceph.mon.a.smithi078.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T00:10:48.828 INFO:journalctl@ceph.mon.a.smithi078.stdout: Block cache BinnedLRUCache@0x55ea69a1f090#2 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-24T00:10:48.828 INFO:journalctl@ceph.mon.a.smithi078.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-24T00:10:48.828 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-03-24T00:10:48.829 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T00:10:48.829 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: starting mon.a rank 0 at public addrs [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] at bind addrs [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid c7e2806a-e972-11ee-95cd-87774f69a715 2024-03-24T00:10:48.829 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: mon.a@-1(???) e1 preinit fsid c7e2806a-e972-11ee-95cd-87774f69a715 2024-03-24T00:10:48.829 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: mon.a@-1(???).mds e1 new map 2024-03-24T00:10:48.829 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: mon.a@-1(???).mds e1 print_map 2024-03-24T00:10:48.829 INFO:journalctl@ceph.mon.a.smithi078.stdout: e1 2024-03-24T00:10:48.829 INFO:journalctl@ceph.mon.a.smithi078.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T00:10:48.829 INFO:journalctl@ceph.mon.a.smithi078.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-24T00:10:48.829 INFO:journalctl@ceph.mon.a.smithi078.stdout: legacy client fscid: -1 2024-03-24T00:10:48.829 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-03-24T00:10:48.829 INFO:journalctl@ceph.mon.a.smithi078.stdout: No filesystems configured 2024-03-24T00:10:48.829 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T00:10:48.829 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T00:10:48.830 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T00:10:48.830 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T00:10:48.830 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-24T00:10:48.830 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 bash[32410]: c766f205ef0efd9d287f7e718d1e87cb3ca7b62fe6851449dbfa54159e05dc51 2024-03-24T00:10:48.830 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 systemd[1]: Started Ceph mon.a for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:10:48.830 INFO:teuthology.orchestra.run.smithi078.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-24T00:10:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T00:10:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: monmap e1: 1 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T00:10:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: fsmap 2024-03-24T00:10:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: osdmap e1: 0 total, 0 up, 0 in 2024-03-24T00:10:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:48 smithi078 ceph-mon[32429]: mgrmap e1: no daemons active 2024-03-24T00:10:49.963 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.233+0000 7f9bfb5c2700 1 Processor -- start 2024-03-24T00:10:49.968 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.234+0000 7f9bfb5c2700 1 -- start start 2024-03-24T00:10:49.968 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.234+0000 7f9bfb5c2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bf4107a20 0x7f9bf4107e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:10:49.968 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.234+0000 7f9bfb5c2700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf4108410 con 0x7f9bf4107a20 2024-03-24T00:10:49.968 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.234+0000 7f9bf935e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bf4107a20 0x7f9bf4107e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:10:49.968 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.234+0000 7f9bf935e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bf4107a20 0x7f9bf4107e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53554/0 (socket says 172.21.15.78:53554) 2024-03-24T00:10:49.968 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.234+0000 7f9bf935e700 1 -- 172.21.15.78:0/3345402188 learned_addr learned my addr 172.21.15.78:0/3345402188 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:10:49.968 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.235+0000 7f9bf935e700 1 -- 172.21.15.78:0/3345402188 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf4108c20 con 0x7f9bf4107a20 2024-03-24T00:10:49.968 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.235+0000 7f9bf935e700 1 --2- 172.21.15.78:0/3345402188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bf4107a20 0x7f9bf4107e40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9bf00098a0 tx=0x7f9bf0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ad9d1ef996697b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:10:49.968 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.235+0000 7f9bebfff700 1 -- 172.21.15.78:0/3345402188 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9bf0004160 con 0x7f9bf4107a20 2024-03-24T00:10:49.968 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.235+0000 7f9bebfff700 1 -- 172.21.15.78:0/3345402188 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9bf0004300 con 0x7f9bf4107a20 2024-03-24T00:10:49.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.235+0000 7f9bebfff700 1 -- 172.21.15.78:0/3345402188 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9bf0011b50 con 0x7f9bf4107a20 2024-03-24T00:10:49.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.235+0000 7f9bfb5c2700 1 -- 172.21.15.78:0/3345402188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bf4107a20 msgr2=0x7f9bf4107e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:10:49.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.235+0000 7f9bfb5c2700 1 --2- 172.21.15.78:0/3345402188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bf4107a20 0x7f9bf4107e40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9bf00098a0 tx=0x7f9bf0009bb0 comp rx=0 tx=0).stop 2024-03-24T00:10:49.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.236+0000 7f9bfb5c2700 1 -- 172.21.15.78:0/3345402188 shutdown_connections 2024-03-24T00:10:49.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.236+0000 7f9bfb5c2700 1 --2- 172.21.15.78:0/3345402188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bf4107a20 0x7f9bf4107e40 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:10:49.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.236+0000 7f9bfb5c2700 1 -- 172.21.15.78:0/3345402188 >> 172.21.15.78:0/3345402188 conn(0x7f9bf41030a0 msgr2=0x7f9bf41054c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:10:49.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.236+0000 7f9bfb5c2700 1 -- 172.21.15.78:0/3345402188 shutdown_connections 2024-03-24T00:10:49.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.236+0000 7f9bfb5c2700 1 -- 172.21.15.78:0/3345402188 wait complete. 2024-03-24T00:10:49.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.236+0000 7f9bfb5c2700 1 Processor -- start 2024-03-24T00:10:49.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.236+0000 7f9bfb5c2700 1 -- start start 2024-03-24T00:10:49.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.236+0000 7f9bfb5c2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bf4107a20 0x7f9bf419b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:10:49.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.236+0000 7f9bfb5c2700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf410bfd0 con 0x7f9bf4107a20 2024-03-24T00:10:49.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.237+0000 7f9bf935e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bf4107a20 0x7f9bf419b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:10:49.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.237+0000 7f9bf935e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bf4107a20 0x7f9bf419b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53570/0 (socket says 172.21.15.78:53570) 2024-03-24T00:10:49.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.237+0000 7f9bf935e700 1 -- 172.21.15.78:0/1865933854 learned_addr learned my addr 172.21.15.78:0/1865933854 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:10:49.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.237+0000 7f9bf935e700 1 -- 172.21.15.78:0/1865933854 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf419bef0 con 0x7f9bf4107a20 2024-03-24T00:10:49.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.237+0000 7f9bf935e700 1 --2- 172.21.15.78:0/1865933854 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bf4107a20 0x7f9bf419b9b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0011670 tx=0x7f9bf0011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:10:49.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.237+0000 7f9bea7fc700 1 -- 172.21.15.78:0/1865933854 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9bf00039c0 con 0x7f9bf4107a20 2024-03-24T00:10:49.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.237+0000 7f9bea7fc700 1 -- 172.21.15.78:0/1865933854 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9bf0011e50 con 0x7f9bf4107a20 2024-03-24T00:10:49.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.237+0000 7f9bfb5c2700 1 -- 172.21.15.78:0/1865933854 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf419c180 con 0x7f9bf4107a20 2024-03-24T00:10:49.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.237+0000 7f9bea7fc700 1 -- 172.21.15.78:0/1865933854 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9bf001a5c0 con 0x7f9bf4107a20 2024-03-24T00:10:49.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.237+0000 7f9bfb5c2700 1 -- 172.21.15.78:0/1865933854 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf419ee70 con 0x7f9bf4107a20 2024-03-24T00:10:49.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.237+0000 7f9bea7fc700 1 -- 172.21.15.78:0/1865933854 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f9bf001ac50 con 0x7f9bf4107a20 2024-03-24T00:10:49.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.237+0000 7f9bea7fc700 1 -- 172.21.15.78:0/1865933854 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9bf0019700 con 0x7f9bf4107a20 2024-03-24T00:10:49.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.238+0000 7f9bfb5c2700 1 -- 172.21.15.78:0/1865933854 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf40637c0 con 0x7f9bf4107a20 2024-03-24T00:10:49.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.240+0000 7f9bea7fc700 1 -- 172.21.15.78:0/1865933854 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f9bf0003b60 con 0x7f9bf4107a20 2024-03-24T00:10:49.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.307+0000 7f9bfb5c2700 1 -- 172.21.15.78:0/1865933854 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f9bf419f430 con 0x7f9bf4107a20 2024-03-24T00:10:49.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.309+0000 7f9bea7fc700 1 -- 172.21.15.78:0/1865933854 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f9bf0019900 con 0x7f9bf4107a20 2024-03-24T00:10:49.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.309+0000 7f9bfb5c2700 1 -- 172.21.15.78:0/1865933854 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bf4107a20 msgr2=0x7f9bf419b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:10:49.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.309+0000 7f9bfb5c2700 1 --2- 172.21.15.78:0/1865933854 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bf4107a20 0x7f9bf419b9b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0011670 tx=0x7f9bf0011b60 comp rx=0 tx=0).stop 2024-03-24T00:10:49.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.309+0000 7f9bfb5c2700 1 -- 172.21.15.78:0/1865933854 shutdown_connections 2024-03-24T00:10:49.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.309+0000 7f9bfb5c2700 1 --2- 172.21.15.78:0/1865933854 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bf4107a20 0x7f9bf419b9b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:10:49.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.309+0000 7f9bfb5c2700 1 -- 172.21.15.78:0/1865933854 >> 172.21.15.78:0/1865933854 conn(0x7f9bf41030a0 msgr2=0x7f9bf4103bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:10:49.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.310+0000 7f9bfb5c2700 1 -- 172.21.15.78:0/1865933854 shutdown_connections 2024-03-24T00:10:49.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:49.310+0000 7f9bfb5c2700 1 -- 172.21.15.78:0/1865933854 wait complete. 2024-03-24T00:10:49.971 INFO:teuthology.orchestra.run.smithi078.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-24T00:10:49.993 INFO:teuthology.orchestra.run.smithi078.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-24T00:10:49.993 INFO:teuthology.orchestra.run.smithi078.stdout:Creating mgr... 2024-03-24T00:10:49.994 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-24T00:10:49.994 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-24T00:10:50.236 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mgr.x 2024-03-24T00:10:50.236 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mgr.x.service: Unit ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mgr.x.service not loaded. 2024-03-24T00:10:50.382 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c7e2806a-e972-11ee-95cd-87774f69a715.target.wants/ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-c7e2806a-e972-11ee-95cd-87774f69a715@.service. 2024-03-24T00:10:50.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:50 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1865933854' entity='client.admin' 2024-03-24T00:10:50.591 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:10:50 smithi078 systemd[1]: Starting Ceph mgr.x for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:10:50.890 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:10:50 smithi078 podman[32643]: 2024-03-24 00:10:50.613659259 +0000 UTC m=+0.111238223 container create f41546bd5b7fa8bf8e03bbf855c6658b6aadab51760348f365de1cdf2dfeb9fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-24T00:10:50.890 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:10:50 smithi078 podman[32643]: 2024-03-24 00:10:50.520142121 +0000 UTC m=+0.017721102 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:10:50.890 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:10:50 smithi078 podman[32643]: 2024-03-24 00:10:50.824014301 +0000 UTC m=+0.321593284 container init f41546bd5b7fa8bf8e03bbf855c6658b6aadab51760348f365de1cdf2dfeb9fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319) 2024-03-24T00:10:50.890 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:10:50 smithi078 podman[32643]: 2024-03-24 00:10:50.826523578 +0000 UTC m=+0.324102543 container start f41546bd5b7fa8bf8e03bbf855c6658b6aadab51760348f365de1cdf2dfeb9fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:10:50.890 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:10:50 smithi078 bash[32643]: f41546bd5b7fa8bf8e03bbf855c6658b6aadab51760348f365de1cdf2dfeb9fc 2024-03-24T00:10:50.903 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T00:10:50.903 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout disabled 2024-03-24T00:10:50.910 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T00:10:50.910 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2024-03-24T00:10:50.910 INFO:teuthology.orchestra.run.smithi078.stdout:firewalld.service is not enabled 2024-03-24T00:10:50.910 INFO:teuthology.orchestra.run.smithi078.stdout:Not possible to enable service . firewalld.service is not available 2024-03-24T00:10:50.919 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T00:10:50.919 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout disabled 2024-03-24T00:10:50.926 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T00:10:50.926 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2024-03-24T00:10:50.926 INFO:teuthology.orchestra.run.smithi078.stdout:firewalld.service is not enabled 2024-03-24T00:10:50.926 INFO:teuthology.orchestra.run.smithi078.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-24T00:10:50.927 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mgr to start... 2024-03-24T00:10:50.927 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mgr... 2024-03-24T00:10:51.226 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:10:50 smithi078 systemd[1]: Started Ceph mgr.x for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:10:51.486 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:51 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3297801427' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T00:10:51.486 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:10:51 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:10:51.387+0000 7f9fe2c20200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T00:10:51.487 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:10:51 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:10:51.486+0000 7f9fe2c20200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T00:10:52.019 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-24T00:10:52.019 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-03-24T00:10:52.019 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsid": "c7e2806a-e972-11ee-95cd-87774f69a715", 2024-03-24T00:10:52.019 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T00:10:52.019 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T00:10:52.019 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T00:10:52.019 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T00:10:52.020 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:10:52.020 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T00:10:52.020 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T00:10:52.020 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 0 2024-03-24T00:10:52.020 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-03-24T00:10:52.020 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T00:10:52.020 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T00:10:52.020 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-03-24T00:10:52.020 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-24T00:10:52.020 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T00:10:52.020 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:10:52.020 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T00:10:52.020 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T00:10:52.021 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:10:52.021 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T00:10:52.021 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:10:52.021 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T00:10:52.021 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T00:10:52.021 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T00:10:52.021 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T00:10:52.021 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T00:10:52.021 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T00:10:52.021 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:10:52.021 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T00:10:52.021 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T00:10:52.021 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T00:10:52.022 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T00:10:52.022 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T00:10:52.022 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T00:10:52.022 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T00:10:52.022 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T00:10:52.022 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T00:10:52.022 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:10:52.022 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T00:10:52.022 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:10:52.022 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T00:10:52.022 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T00:10:52.022 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:10:52.023 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T00:10:52.023 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T00:10:52.023 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T00:10:52.023 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T00:10:52.023 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T00:10:52.023 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T00:10:52.023 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T00:10:52.023 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-03-24T00:10:52.023 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T00:10:52.023 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:10:52.023 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T00:10:52.023 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:10:52.023 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T00:10:44.135408+0000", 2024-03-24T00:10:52.023 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T00:10:52.023 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:10:52.023 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T00:10:52.024 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-03-24T00:10:52.024 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.326+0000 7f87d06cf700 1 Processor -- start 2024-03-24T00:10:52.024 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.327+0000 7f87d06cf700 1 -- start start 2024-03-24T00:10:52.024 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.327+0000 7f87d06cf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c8108490 0x7f87c810a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:10:52.024 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.327+0000 7f87d06cf700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c807ae70 con 0x7f87c8108490 2024-03-24T00:10:52.024 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.327+0000 7f87ce46b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c8108490 0x7f87c810a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:10:52.024 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.327+0000 7f87ce46b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c8108490 0x7f87c810a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53594/0 (socket says 172.21.15.78:53594) 2024-03-24T00:10:52.024 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.327+0000 7f87ce46b700 1 -- 172.21.15.78:0/1051483998 learned_addr learned my addr 172.21.15.78:0/1051483998 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:10:52.024 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.328+0000 7f87ce46b700 1 -- 172.21.15.78:0/1051483998 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c810adc0 con 0x7f87c8108490 2024-03-24T00:10:52.024 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.328+0000 7f87ce46b700 1 --2- 172.21.15.78:0/1051483998 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c8108490 0x7f87c810a880 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f87c40098a0 tx=0x7f87c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdc2a9aaf71a5c44 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:10:52.024 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.328+0000 7f87cd469700 1 -- 172.21.15.78:0/1051483998 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87c4004160 con 0x7f87c8108490 2024-03-24T00:10:52.024 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.328+0000 7f87cd469700 1 -- 172.21.15.78:0/1051483998 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f87c4004300 con 0x7f87c8108490 2024-03-24T00:10:52.024 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.328+0000 7f87cd469700 1 -- 172.21.15.78:0/1051483998 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87c4011b50 con 0x7f87c8108490 2024-03-24T00:10:52.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.329+0000 7f87d06cf700 1 -- 172.21.15.78:0/1051483998 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c8108490 msgr2=0x7f87c810a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:10:52.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.329+0000 7f87d06cf700 1 --2- 172.21.15.78:0/1051483998 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c8108490 0x7f87c810a880 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f87c40098a0 tx=0x7f87c4009bb0 comp rx=0 tx=0).stop 2024-03-24T00:10:52.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.329+0000 7f87d06cf700 1 -- 172.21.15.78:0/1051483998 shutdown_connections 2024-03-24T00:10:52.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.329+0000 7f87d06cf700 1 --2- 172.21.15.78:0/1051483998 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c8108490 0x7f87c810a880 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:10:52.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.329+0000 7f87d06cf700 1 -- 172.21.15.78:0/1051483998 >> 172.21.15.78:0/1051483998 conn(0x7f87c8101f40 msgr2=0x7f87c8104360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:10:52.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.329+0000 7f87d06cf700 1 -- 172.21.15.78:0/1051483998 shutdown_connections 2024-03-24T00:10:52.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.329+0000 7f87d06cf700 1 -- 172.21.15.78:0/1051483998 wait complete. 2024-03-24T00:10:52.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.330+0000 7f87d06cf700 1 Processor -- start 2024-03-24T00:10:52.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.330+0000 7f87d06cf700 1 -- start start 2024-03-24T00:10:52.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.330+0000 7f87d06cf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c8108490 0x7f87c819b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:10:52.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.330+0000 7f87d06cf700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c810bd90 con 0x7f87c8108490 2024-03-24T00:10:52.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.330+0000 7f87ce46b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c8108490 0x7f87c819b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:10:52.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.330+0000 7f87ce46b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c8108490 0x7f87c819b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53608/0 (socket says 172.21.15.78:53608) 2024-03-24T00:10:52.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.330+0000 7f87ce46b700 1 -- 172.21.15.78:0/3297801427 learned_addr learned my addr 172.21.15.78:0/3297801427 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:10:52.025 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.330+0000 7f87ce46b700 1 -- 172.21.15.78:0/3297801427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c819beb0 con 0x7f87c8108490 2024-03-24T00:10:52.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.330+0000 7f87ce46b700 1 --2- 172.21.15.78:0/3297801427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c8108490 0x7f87c819b970 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f87c4009fb0 tx=0x7f87c4011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:10:52.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.331+0000 7f87bf7fe700 1 -- 172.21.15.78:0/3297801427 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87c40038b0 con 0x7f87c8108490 2024-03-24T00:10:52.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.331+0000 7f87bf7fe700 1 -- 172.21.15.78:0/3297801427 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f87c4011e50 con 0x7f87c8108490 2024-03-24T00:10:52.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.331+0000 7f87d06cf700 1 -- 172.21.15.78:0/3297801427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87c819c140 con 0x7f87c8108490 2024-03-24T00:10:52.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.331+0000 7f87bf7fe700 1 -- 172.21.15.78:0/3297801427 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87c4019420 con 0x7f87c8108490 2024-03-24T00:10:52.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.331+0000 7f87d06cf700 1 -- 172.21.15.78:0/3297801427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87c819ee30 con 0x7f87c8108490 2024-03-24T00:10:52.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.331+0000 7f87bf7fe700 1 -- 172.21.15.78:0/3297801427 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f87c4019b30 con 0x7f87c8108490 2024-03-24T00:10:52.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.331+0000 7f87bf7fe700 1 -- 172.21.15.78:0/3297801427 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f87c4024730 con 0x7f87c8108490 2024-03-24T00:10:52.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.332+0000 7f87d06cf700 1 -- 172.21.15.78:0/3297801427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87c8069220 con 0x7f87c8108490 2024-03-24T00:10:52.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.334+0000 7f87bf7fe700 1 -- 172.21.15.78:0/3297801427 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f87c4003a50 con 0x7f87c8108490 2024-03-24T00:10:52.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.406+0000 7f87d06cf700 1 -- 172.21.15.78:0/3297801427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f87c819f3e0 con 0x7f87c8108490 2024-03-24T00:10:52.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.407+0000 7f87bf7fe700 1 -- 172.21.15.78:0/3297801427 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f87c4032030 con 0x7f87c8108490 2024-03-24T00:10:52.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.408+0000 7f87d06cf700 1 -- 172.21.15.78:0/3297801427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c8108490 msgr2=0x7f87c819b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:10:52.026 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.408+0000 7f87d06cf700 1 --2- 172.21.15.78:0/3297801427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c8108490 0x7f87c819b970 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f87c4009fb0 tx=0x7f87c4011bf0 comp rx=0 tx=0).stop 2024-03-24T00:10:52.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.408+0000 7f87d06cf700 1 -- 172.21.15.78:0/3297801427 shutdown_connections 2024-03-24T00:10:52.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.408+0000 7f87d06cf700 1 --2- 172.21.15.78:0/3297801427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87c8108490 0x7f87c819b970 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:10:52.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.408+0000 7f87d06cf700 1 -- 172.21.15.78:0/3297801427 >> 172.21.15.78:0/3297801427 conn(0x7f87c8101f40 msgr2=0x7f87c8102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:10:52.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.408+0000 7f87d06cf700 1 -- 172.21.15.78:0/3297801427 shutdown_connections 2024-03-24T00:10:52.027 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:51.408+0000 7f87d06cf700 1 -- 172.21.15.78:0/3297801427 wait complete. 2024-03-24T00:10:52.027 INFO:teuthology.orchestra.run.smithi078.stdout:mgr not available, waiting (1/15)... 2024-03-24T00:10:52.340 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:10:52 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:10:52.129+0000 7f9fe2c20200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T00:10:52.340 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:10:52 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:10:52.222+0000 7f9fe2c20200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T00:10:52.819 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:10:52 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:10:52.369+0000 7f9fe2c20200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T00:10:52.820 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:10:52 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:10:52.472+0000 7f9fe2c20200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T00:10:52.820 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:10:52 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:10:52.554+0000 7f9fe2c20200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T00:10:52.820 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:10:52 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:10:52.819+0000 7f9fe2c20200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T00:10:53.090 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:10:52 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:10:52.929+0000 7f9fe2c20200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T00:10:53.090 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:10:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:10:53.015+0000 7f9fe2c20200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T00:10:53.590 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:10:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:10:53.109+0000 7f9fe2c20200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T00:10:53.590 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:10:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:10:53.283+0000 7f9fe2c20200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T00:10:54.590 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:10:54 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:10:54.271+0000 7f9fe2c20200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T00:10:54.590 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:10:54 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:10:54.400+0000 7f9fe2c20200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T00:10:55.033 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:10:54 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:10:54.748+0000 7f9fe2c20200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T00:10:55.033 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:10:54 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:10:54.837+0000 7f9fe2c20200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T00:10:55.033 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:10:55 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:10:55.033+0000 7f9fe2c20200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:10:55.284 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:10:55 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:10:55.220+0000 7f9fe2c20200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T00:10:55.590 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:10:55 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:10:55.510+0000 7f9fe2c20200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:10:56.090 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:10:55 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:10:55.773+0000 7f9fe2c20200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T00:10:56.090 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:10:56 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:10:56.010+0000 7f9fe2c20200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T00:10:57.173 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:10:56 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:10:56.856+0000 7f9fe2c20200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T00:10:57.449 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:57 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2746264361' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T00:10:57.761 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:10:57 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:10:57.448+0000 7f9fe2c20200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T00:10:57.761 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:10:57 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:10:57.547+0000 7f9fe2c20200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T00:10:57.761 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:10:57 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:10:57.761+0000 7f9fe2c20200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T00:10:58.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:58 smithi078 ceph-mon[32429]: Activating manager daemon x 2024-03-24T00:10:58.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:58 smithi078 ceph-mon[32429]: mgrmap e2: x(active, starting, since 0.00319238s) 2024-03-24T00:10:58.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:58 smithi078 ceph-mon[32429]: from='mgr.14100 172.21.15.78:0/1296898335' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T00:10:58.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:58 smithi078 ceph-mon[32429]: from='mgr.14100 172.21.15.78:0/1296898335' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T00:10:58.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:58 smithi078 ceph-mon[32429]: from='mgr.14100 172.21.15.78:0/1296898335' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T00:10:58.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:58 smithi078 ceph-mon[32429]: from='mgr.14100 172.21.15.78:0/1296898335' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:10:58.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:58 smithi078 ceph-mon[32429]: from='mgr.14100 172.21.15.78:0/1296898335' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T00:10:58.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:58 smithi078 ceph-mon[32429]: Manager daemon x is now available 2024-03-24T00:10:58.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:58 smithi078 ceph-mon[32429]: from='mgr.14100 172.21.15.78:0/1296898335' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T00:10:58.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:58 smithi078 ceph-mon[32429]: from='mgr.14100 172.21.15.78:0/1296898335' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T00:10:58.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:58 smithi078 ceph-mon[32429]: from='mgr.14100 172.21.15.78:0/1296898335' entity='mgr.x' 2024-03-24T00:10:58.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:58 smithi078 ceph-mon[32429]: from='mgr.14100 172.21.15.78:0/1296898335' entity='mgr.x' 2024-03-24T00:10:58.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:58 smithi078 ceph-mon[32429]: from='mgr.14100 172.21.15.78:0/1296898335' entity='mgr.x' 2024-03-24T00:11:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:10:59 smithi078 ceph-mon[32429]: mgrmap e3: x(active, since 1.00702s) 2024-03-24T00:11:02.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:01 smithi078 ceph-mon[32429]: mgrmap e4: x(active, since 3s) 2024-03-24T00:11:02.626 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-24T00:11:02.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-03-24T00:11:02.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsid": "c7e2806a-e972-11ee-95cd-87774f69a715", 2024-03-24T00:11:02.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T00:11:02.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T00:11:02.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T00:11:02.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T00:11:02.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:11:02.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T00:11:02.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T00:11:02.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 0 2024-03-24T00:11:02.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-03-24T00:11:02.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T00:11:02.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T00:11:02.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-03-24T00:11:02.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-24T00:11:02.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T00:11:02.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:11:02.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T00:11:02.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T00:11:02.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:11:02.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T00:11:02.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:11:02.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T00:11:02.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T00:11:02.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T00:11:02.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T00:11:02.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T00:11:02.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T00:11:02.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:11:02.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T00:11:02.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T00:11:02.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T00:11:02.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T00:11:02.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T00:11:02.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T00:11:02.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T00:11:02.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T00:11:02.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T00:11:02.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:11:02.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T00:11:02.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:11:02.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T00:11:02.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T00:11:02.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:11:02.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T00:11:02.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T00:11:02.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T00:11:02.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T00:11:02.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T00:11:02.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T00:11:02.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T00:11:02.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-03-24T00:11:02.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T00:11:02.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:11:02.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T00:11:02.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:11:02.879 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T00:10:44.135408+0000", 2024-03-24T00:11:02.879 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T00:11:02.879 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:11:02.879 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T00:11:02.879 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-03-24T00:11:02.879 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.173+0000 7fa1f7881700 1 Processor -- start 2024-03-24T00:11:02.879 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.173+0000 7fa1f7881700 1 -- start start 2024-03-24T00:11:02.879 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.173+0000 7fa1f7881700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1f0108490 0x7fa1f010a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:02.879 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.173+0000 7fa1f7881700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f007ae70 con 0x7fa1f0108490 2024-03-24T00:11:02.879 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.173+0000 7fa1f561d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1f0108490 0x7fa1f010a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:02.879 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.173+0000 7fa1f561d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1f0108490 0x7fa1f010a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55182/0 (socket says 172.21.15.78:55182) 2024-03-24T00:11:02.879 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.173+0000 7fa1f561d700 1 -- 172.21.15.78:0/3230926329 learned_addr learned my addr 172.21.15.78:0/3230926329 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:02.879 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.174+0000 7fa1f561d700 1 -- 172.21.15.78:0/3230926329 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f010adc0 con 0x7fa1f0108490 2024-03-24T00:11:02.879 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.174+0000 7fa1f561d700 1 --2- 172.21.15.78:0/3230926329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1f0108490 0x7fa1f010a880 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec0098a0 tx=0x7fa1ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a680bb6401295f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:02.879 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.174+0000 7fa1e7fff700 1 -- 172.21.15.78:0/3230926329 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa1ec004160 con 0x7fa1f0108490 2024-03-24T00:11:02.879 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.174+0000 7fa1e7fff700 1 -- 172.21.15.78:0/3230926329 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa1ec004300 con 0x7fa1f0108490 2024-03-24T00:11:02.880 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.174+0000 7fa1e7fff700 1 -- 172.21.15.78:0/3230926329 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa1ec011b50 con 0x7fa1f0108490 2024-03-24T00:11:02.880 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.175+0000 7fa1f7881700 1 -- 172.21.15.78:0/3230926329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1f0108490 msgr2=0x7fa1f010a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:02.880 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.175+0000 7fa1f7881700 1 --2- 172.21.15.78:0/3230926329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1f0108490 0x7fa1f010a880 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec0098a0 tx=0x7fa1ec009bb0 comp rx=0 tx=0).stop 2024-03-24T00:11:02.880 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.175+0000 7fa1f7881700 1 -- 172.21.15.78:0/3230926329 shutdown_connections 2024-03-24T00:11:02.880 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.175+0000 7fa1f7881700 1 --2- 172.21.15.78:0/3230926329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1f0108490 0x7fa1f010a880 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:02.880 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.175+0000 7fa1f7881700 1 -- 172.21.15.78:0/3230926329 >> 172.21.15.78:0/3230926329 conn(0x7fa1f0101f40 msgr2=0x7fa1f0104360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:02.880 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.175+0000 7fa1f7881700 1 -- 172.21.15.78:0/3230926329 shutdown_connections 2024-03-24T00:11:02.880 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.175+0000 7fa1f7881700 1 -- 172.21.15.78:0/3230926329 wait complete. 2024-03-24T00:11:02.880 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.176+0000 7fa1f7881700 1 Processor -- start 2024-03-24T00:11:02.880 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.176+0000 7fa1f7881700 1 -- start start 2024-03-24T00:11:02.880 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.176+0000 7fa1f7881700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1f0108490 0x7fa1f019b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:02.880 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.176+0000 7fa1f7881700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f010bd90 con 0x7fa1f0108490 2024-03-24T00:11:02.880 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.176+0000 7fa1f561d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1f0108490 0x7fa1f019b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:02.880 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.176+0000 7fa1f561d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1f0108490 0x7fa1f019b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55190/0 (socket says 172.21.15.78:55190) 2024-03-24T00:11:02.880 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.176+0000 7fa1f561d700 1 -- 172.21.15.78:0/2746264361 learned_addr learned my addr 172.21.15.78:0/2746264361 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:02.881 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.177+0000 7fa1f561d700 1 -- 172.21.15.78:0/2746264361 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f019beb0 con 0x7fa1f0108490 2024-03-24T00:11:02.881 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.177+0000 7fa1f561d700 1 --2- 172.21.15.78:0/2746264361 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1f0108490 0x7fa1f019b970 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec009fb0 tx=0x7fa1ec011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:02.881 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.177+0000 7fa1e67fc700 1 -- 172.21.15.78:0/2746264361 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa1ec0038b0 con 0x7fa1f0108490 2024-03-24T00:11:02.881 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.177+0000 7fa1e67fc700 1 -- 172.21.15.78:0/2746264361 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa1ec011e50 con 0x7fa1f0108490 2024-03-24T00:11:02.881 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.177+0000 7fa1f7881700 1 -- 172.21.15.78:0/2746264361 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1f019c140 con 0x7fa1f0108490 2024-03-24T00:11:02.881 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.177+0000 7fa1e67fc700 1 -- 172.21.15.78:0/2746264361 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa1ec019420 con 0x7fa1f0108490 2024-03-24T00:11:02.881 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.177+0000 7fa1f7881700 1 -- 172.21.15.78:0/2746264361 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1f019ee30 con 0x7fa1f0108490 2024-03-24T00:11:02.881 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.177+0000 7fa1e67fc700 1 -- 172.21.15.78:0/2746264361 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fa1ec019ad0 con 0x7fa1f0108490 2024-03-24T00:11:02.881 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.177+0000 7fa1e67fc700 1 -- 172.21.15.78:0/2746264361 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa1ec024980 con 0x7fa1f0108490 2024-03-24T00:11:02.881 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.178+0000 7fa1f7881700 1 -- 172.21.15.78:0/2746264361 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1f0069220 con 0x7fa1f0108490 2024-03-24T00:11:02.881 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.181+0000 7fa1e67fc700 1 -- 172.21.15.78:0/2746264361 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fa1ec043dd0 con 0x7fa1f0108490 2024-03-24T00:11:02.881 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.267+0000 7fa1f7881700 1 -- 172.21.15.78:0/2746264361 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa1f019f380 con 0x7fa1f0108490 2024-03-24T00:11:02.881 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.267+0000 7fa1e67fc700 1 -- 172.21.15.78:0/2746264361 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fa1ec032020 con 0x7fa1f0108490 2024-03-24T00:11:02.881 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.268+0000 7fa1f7881700 1 -- 172.21.15.78:0/2746264361 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1f0108490 msgr2=0x7fa1f019b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:02.881 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.268+0000 7fa1f7881700 1 --2- 172.21.15.78:0/2746264361 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1f0108490 0x7fa1f019b970 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec009fb0 tx=0x7fa1ec011bf0 comp rx=0 tx=0).stop 2024-03-24T00:11:02.881 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.269+0000 7fa1f7881700 1 -- 172.21.15.78:0/2746264361 shutdown_connections 2024-03-24T00:11:02.882 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.269+0000 7fa1f7881700 1 --2- 172.21.15.78:0/2746264361 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1f0108490 0x7fa1f019b970 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:02.882 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.269+0000 7fa1f7881700 1 -- 172.21.15.78:0/2746264361 >> 172.21.15.78:0/2746264361 conn(0x7fa1f0101f40 msgr2=0x7fa1f0102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:02.882 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.269+0000 7fa1f7881700 1 -- 172.21.15.78:0/2746264361 shutdown_connections 2024-03-24T00:11:02.882 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:10:57.269+0000 7fa1f7881700 1 -- 172.21.15.78:0/2746264361 wait complete. 2024-03-24T00:11:02.882 INFO:teuthology.orchestra.run.smithi078.stdout:mgr not available, waiting (2/15)... 2024-03-24T00:11:05.759 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:05 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1811630301' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T00:11:05.958 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-24T00:11:05.959 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-03-24T00:11:05.959 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsid": "c7e2806a-e972-11ee-95cd-87774f69a715", 2024-03-24T00:11:05.959 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T00:11:05.959 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T00:11:05.959 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T00:11:05.959 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T00:11:05.959 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:11:05.959 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T00:11:05.959 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T00:11:05.959 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 0 2024-03-24T00:11:05.959 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-03-24T00:11:05.959 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T00:11:05.959 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T00:11:05.959 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-03-24T00:11:05.960 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_age": 16, 2024-03-24T00:11:05.960 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T00:11:05.960 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:11:05.960 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T00:11:05.960 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T00:11:05.960 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:11:05.960 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T00:11:05.960 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:11:05.960 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T00:11:05.960 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T00:11:05.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T00:11:05.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T00:11:05.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T00:11:05.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T00:11:05.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:11:05.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T00:11:05.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T00:11:05.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T00:11:05.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T00:11:05.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T00:11:05.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T00:11:05.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T00:11:05.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T00:11:05.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T00:11:05.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:11:05.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T00:11:05.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:11:05.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T00:11:05.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T00:11:05.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:11:05.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T00:11:05.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T00:11:05.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T00:11:05.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T00:11:05.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T00:11:05.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T00:11:05.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T00:11:05.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-03-24T00:11:05.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T00:11:05.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:11:05.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T00:11:05.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:11:05.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T00:10:44.135408+0000", 2024-03-24T00:11:05.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T00:11:05.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:11:05.972 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T00:11:05.972 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-03-24T00:11:05.972 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.073+0000 7f48aab62700 1 Processor -- start 2024-03-24T00:11:05.972 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.073+0000 7f48aab62700 1 -- start start 2024-03-24T00:11:05.972 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.073+0000 7f48aab62700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48a407cee0 0x7f48a407b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:05.972 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.073+0000 7f48aab62700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a407b910 con 0x7f48a407cee0 2024-03-24T00:11:05.972 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.073+0000 7f48a88fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48a407cee0 0x7f48a407b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:05.972 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.073+0000 7f48a88fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48a407cee0 0x7f48a407b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44654/0 (socket says 172.21.15.78:44654) 2024-03-24T00:11:05.972 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.073+0000 7f48a88fe700 1 -- 172.21.15.78:0/2973830156 learned_addr learned my addr 172.21.15.78:0/2973830156 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:05.972 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.074+0000 7f48a88fe700 1 -- 172.21.15.78:0/2973830156 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a407ba90 con 0x7f48a407cee0 2024-03-24T00:11:05.972 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.074+0000 7f48a88fe700 1 --2- 172.21.15.78:0/2973830156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48a407cee0 0x7f48a407b340 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f48980098a0 tx=0x7f4898009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=210a93aef6a1a121 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:05.972 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.074+0000 7f48a37fe700 1 -- 172.21.15.78:0/2973830156 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4898004160 con 0x7f48a407cee0 2024-03-24T00:11:05.972 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.074+0000 7f48a37fe700 1 -- 172.21.15.78:0/2973830156 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4898004300 con 0x7f48a407cee0 2024-03-24T00:11:05.972 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.074+0000 7f48a37fe700 1 -- 172.21.15.78:0/2973830156 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4898011b50 con 0x7f48a407cee0 2024-03-24T00:11:05.973 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.075+0000 7f48aab62700 1 -- 172.21.15.78:0/2973830156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48a407cee0 msgr2=0x7f48a407b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:05.973 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.075+0000 7f48aab62700 1 --2- 172.21.15.78:0/2973830156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48a407cee0 0x7f48a407b340 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f48980098a0 tx=0x7f4898009bb0 comp rx=0 tx=0).stop 2024-03-24T00:11:05.973 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.075+0000 7f48aab62700 1 -- 172.21.15.78:0/2973830156 shutdown_connections 2024-03-24T00:11:05.973 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.075+0000 7f48aab62700 1 --2- 172.21.15.78:0/2973830156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48a407cee0 0x7f48a407b340 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:05.973 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.075+0000 7f48aab62700 1 -- 172.21.15.78:0/2973830156 >> 172.21.15.78:0/2973830156 conn(0x7f48a4103040 msgr2=0x7f48a4105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:05.973 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.075+0000 7f48aab62700 1 -- 172.21.15.78:0/2973830156 shutdown_connections 2024-03-24T00:11:05.973 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.075+0000 7f48aab62700 1 -- 172.21.15.78:0/2973830156 wait complete. 2024-03-24T00:11:05.973 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.075+0000 7f48aab62700 1 Processor -- start 2024-03-24T00:11:05.973 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.075+0000 7f48aab62700 1 -- start start 2024-03-24T00:11:05.973 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.075+0000 7f48aab62700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48a407cee0 0x7f48a419dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:05.973 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.075+0000 7f48aab62700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a410e010 con 0x7f48a407cee0 2024-03-24T00:11:05.973 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.075+0000 7f48a88fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48a407cee0 0x7f48a419dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:05.973 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.076+0000 7f48a88fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48a407cee0 0x7f48a419dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44660/0 (socket says 172.21.15.78:44660) 2024-03-24T00:11:05.973 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.076+0000 7f48a88fe700 1 -- 172.21.15.78:0/1811630301 learned_addr learned my addr 172.21.15.78:0/1811630301 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:05.974 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.076+0000 7f48a88fe700 1 -- 172.21.15.78:0/1811630301 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a419e130 con 0x7f48a407cee0 2024-03-24T00:11:05.974 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.076+0000 7f48a88fe700 1 --2- 172.21.15.78:0/1811630301 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48a407cee0 0x7f48a419dbf0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4898009fb0 tx=0x7f4898011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:05.974 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.076+0000 7f48a1ffb700 1 -- 172.21.15.78:0/1811630301 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f48980038b0 con 0x7f48a407cee0 2024-03-24T00:11:05.974 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.076+0000 7f48aab62700 1 -- 172.21.15.78:0/1811630301 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48a419e3c0 con 0x7f48a407cee0 2024-03-24T00:11:05.974 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.076+0000 7f48a1ffb700 1 -- 172.21.15.78:0/1811630301 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4898011e50 con 0x7f48a407cee0 2024-03-24T00:11:05.974 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.076+0000 7f48a1ffb700 1 -- 172.21.15.78:0/1811630301 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4898019550 con 0x7f48a407cee0 2024-03-24T00:11:05.974 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.076+0000 7f48aab62700 1 -- 172.21.15.78:0/1811630301 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48a41a10b0 con 0x7f48a407cee0 2024-03-24T00:11:05.974 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.077+0000 7f48a1ffb700 1 -- 172.21.15.78:0/1811630301 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f4898019850 con 0x7f48a407cee0 2024-03-24T00:11:05.974 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.077+0000 7f48aab62700 1 -- 172.21.15.78:0/1811630301 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48a4069220 con 0x7f48a407cee0 2024-03-24T00:11:05.974 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.077+0000 7f48a1ffb700 1 --2- 172.21.15.78:0/1811630301 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7f4894038e80 0x7f489403b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:05.974 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.077+0000 7f48a1ffb700 1 -- 172.21.15.78:0/1811630301 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f489804d050 con 0x7f48a407cee0 2024-03-24T00:11:05.974 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.078+0000 7f48a3fff700 1 --2- 172.21.15.78:0/1811630301 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7f4894038e80 0x7f489403b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:05.974 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.078+0000 7f48a3fff700 1 --2- 172.21.15.78:0/1811630301 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7f4894038e80 0x7f489403b340 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4890006fd0 tx=0x7f4890006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:05.975 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.083+0000 7f48a1ffb700 1 -- 172.21.15.78:0/1811630301 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f489800bc40 con 0x7f48a407cee0 2024-03-24T00:11:05.975 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.355+0000 7f48aab62700 1 -- 172.21.15.78:0/1811630301 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f48a41a1280 con 0x7f48a407cee0 2024-03-24T00:11:05.975 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.356+0000 7f48a1ffb700 1 -- 172.21.15.78:0/1811630301 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f48a41a1280 con 0x7f48a407cee0 2024-03-24T00:11:05.975 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.358+0000 7f48aab62700 1 -- 172.21.15.78:0/1811630301 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7f4894038e80 msgr2=0x7f489403b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:05.975 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.358+0000 7f48aab62700 1 --2- 172.21.15.78:0/1811630301 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7f4894038e80 0x7f489403b340 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4890006fd0 tx=0x7f4890006e40 comp rx=0 tx=0).stop 2024-03-24T00:11:05.975 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.358+0000 7f48aab62700 1 -- 172.21.15.78:0/1811630301 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48a407cee0 msgr2=0x7f48a419dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:05.975 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.358+0000 7f48aab62700 1 --2- 172.21.15.78:0/1811630301 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48a407cee0 0x7f48a419dbf0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4898009fb0 tx=0x7f4898011bf0 comp rx=0 tx=0).stop 2024-03-24T00:11:05.975 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.358+0000 7f48aab62700 1 -- 172.21.15.78:0/1811630301 shutdown_connections 2024-03-24T00:11:05.975 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.358+0000 7f48aab62700 1 --2- 172.21.15.78:0/1811630301 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7f4894038e80 0x7f489403b340 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:05.975 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.358+0000 7f48aab62700 1 --2- 172.21.15.78:0/1811630301 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48a407cee0 0x7f48a419dbf0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:05.975 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.358+0000 7f48aab62700 1 -- 172.21.15.78:0/1811630301 >> 172.21.15.78:0/1811630301 conn(0x7f48a4103040 msgr2=0x7f48a4103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:05.975 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.358+0000 7f48aab62700 1 -- 172.21.15.78:0/1811630301 shutdown_connections 2024-03-24T00:11:05.975 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:05.358+0000 7f48aab62700 1 -- 172.21.15.78:0/1811630301 wait complete. 2024-03-24T00:11:05.975 INFO:teuthology.orchestra.run.smithi078.stdout:mgr is available 2024-03-24T00:11:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:06 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2813116675' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T00:11:07.106 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-24T00:11:07.106 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [global] 2024-03-24T00:11:07.107 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout fsid = c7e2806a-e972-11ee-95cd-87774f69a715 2024-03-24T00:11:07.107 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-24T00:11:07.107 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-24T00:11:07.107 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-24T00:11:07.107 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-24T00:11:07.107 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-24T00:11:07.107 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-24T00:11:07.107 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-24T00:11:07.108 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-24T00:11:07.108 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [osd] 2024-03-24T00:11:07.108 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-24T00:11:07.108 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-24T00:11:07.108 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-24T00:11:07.108 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-24T00:11:07.108 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-24T00:11:07.108 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.370+0000 7f334c7dc700 1 Processor -- start 2024-03-24T00:11:07.108 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.370+0000 7f334c7dc700 1 -- start start 2024-03-24T00:11:07.109 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.370+0000 7f334c7dc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33441068c0 0x7f3344106ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:07.109 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.370+0000 7f334c7dc700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33441072b0 con 0x7f33441068c0 2024-03-24T00:11:07.109 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.371+0000 7f334a578700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33441068c0 0x7f3344106ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:07.109 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.371+0000 7f334a578700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33441068c0 0x7f3344106ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44662/0 (socket says 172.21.15.78:44662) 2024-03-24T00:11:07.109 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.371+0000 7f334a578700 1 -- 172.21.15.78:0/999495948 learned_addr learned my addr 172.21.15.78:0/999495948 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:07.109 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.371+0000 7f334a578700 1 -- 172.21.15.78:0/999495948 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3344107ac0 con 0x7f33441068c0 2024-03-24T00:11:07.109 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.371+0000 7f334a578700 1 --2- 172.21.15.78:0/999495948 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33441068c0 0x7f3344106ce0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f33400098a0 tx=0x7f3340009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7a9684e075b7e40 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:07.110 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.371+0000 7f3349576700 1 -- 172.21.15.78:0/999495948 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3340004160 con 0x7f33441068c0 2024-03-24T00:11:07.110 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.371+0000 7f3349576700 1 -- 172.21.15.78:0/999495948 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3340004300 con 0x7f33441068c0 2024-03-24T00:11:07.110 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.371+0000 7f3349576700 1 -- 172.21.15.78:0/999495948 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3340011b50 con 0x7f33441068c0 2024-03-24T00:11:07.110 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.372+0000 7f334c7dc700 1 -- 172.21.15.78:0/999495948 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33441068c0 msgr2=0x7f3344106ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:07.110 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.372+0000 7f334c7dc700 1 --2- 172.21.15.78:0/999495948 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33441068c0 0x7f3344106ce0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f33400098a0 tx=0x7f3340009bb0 comp rx=0 tx=0).stop 2024-03-24T00:11:07.110 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.372+0000 7f334c7dc700 1 -- 172.21.15.78:0/999495948 shutdown_connections 2024-03-24T00:11:07.110 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.372+0000 7f334c7dc700 1 --2- 172.21.15.78:0/999495948 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33441068c0 0x7f3344106ce0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:07.110 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.372+0000 7f334c7dc700 1 -- 172.21.15.78:0/999495948 >> 172.21.15.78:0/999495948 conn(0x7f3344101f40 msgr2=0x7f3344104360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:07.111 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.372+0000 7f334c7dc700 1 -- 172.21.15.78:0/999495948 shutdown_connections 2024-03-24T00:11:07.111 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.372+0000 7f334c7dc700 1 -- 172.21.15.78:0/999495948 wait complete. 2024-03-24T00:11:07.111 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.372+0000 7f334c7dc700 1 Processor -- start 2024-03-24T00:11:07.111 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.373+0000 7f334c7dc700 1 -- start start 2024-03-24T00:11:07.111 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.373+0000 7f334c7dc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33441068c0 0x7f33441997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:07.111 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.373+0000 7f334c7dc700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3344079ff0 con 0x7f33441068c0 2024-03-24T00:11:07.111 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.373+0000 7f334a578700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33441068c0 0x7f33441997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:07.112 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.373+0000 7f334a578700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33441068c0 0x7f33441997e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44678/0 (socket says 172.21.15.78:44678) 2024-03-24T00:11:07.112 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.373+0000 7f334a578700 1 -- 172.21.15.78:0/2813116675 learned_addr learned my addr 172.21.15.78:0/2813116675 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:07.112 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.373+0000 7f334a578700 1 -- 172.21.15.78:0/2813116675 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3344199d20 con 0x7f33441068c0 2024-03-24T00:11:07.112 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.373+0000 7f334a578700 1 --2- 172.21.15.78:0/2813116675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33441068c0 0x7f33441997e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3340011850 tx=0x7f3340011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:07.112 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.373+0000 7f333b7fe700 1 -- 172.21.15.78:0/2813116675 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3340003af0 con 0x7f33441068c0 2024-03-24T00:11:07.112 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.373+0000 7f333b7fe700 1 -- 172.21.15.78:0/2813116675 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3340020070 con 0x7f33441068c0 2024-03-24T00:11:07.112 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.373+0000 7f334c7dc700 1 -- 172.21.15.78:0/2813116675 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3344199fb0 con 0x7f33441068c0 2024-03-24T00:11:07.112 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.373+0000 7f333b7fe700 1 -- 172.21.15.78:0/2813116675 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f33400195a0 con 0x7f33441068c0 2024-03-24T00:11:07.112 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.373+0000 7f334c7dc700 1 -- 172.21.15.78:0/2813116675 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f334419cca0 con 0x7f33441068c0 2024-03-24T00:11:07.113 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.374+0000 7f333b7fe700 1 -- 172.21.15.78:0/2813116675 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f3340019860 con 0x7f33441068c0 2024-03-24T00:11:07.113 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.374+0000 7f333b7fe700 1 --2- 172.21.15.78:0/2813116675 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7f333003d330 0x7f333003f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:07.113 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.374+0000 7f334c7dc700 1 -- 172.21.15.78:0/2813116675 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3344069220 con 0x7f33441068c0 2024-03-24T00:11:07.113 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.374+0000 7f333b7fe700 1 -- 172.21.15.78:0/2813116675 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3340011e00 con 0x7f33441068c0 2024-03-24T00:11:07.113 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.374+0000 7f3349d77700 1 --2- 172.21.15.78:0/2813116675 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7f333003d330 0x7f333003f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:07.113 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.375+0000 7f3349d77700 1 --2- 172.21.15.78:0/2813116675 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7f333003d330 0x7f333003f7f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3334006fd0 tx=0x7f3334006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:07.113 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.379+0000 7f333b7fe700 1 -- 172.21.15.78:0/2813116675 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3340023370 con 0x7f33441068c0 2024-03-24T00:11:07.113 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.569+0000 7f334c7dc700 1 -- 172.21.15.78:0/2813116675 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f334419ce70 con 0x7f33441068c0 2024-03-24T00:11:07.114 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.570+0000 7f333b7fe700 1 -- 172.21.15.78:0/2813116675 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f334419ce70 con 0x7f33441068c0 2024-03-24T00:11:07.114 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.572+0000 7f334c7dc700 1 -- 172.21.15.78:0/2813116675 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7f333003d330 msgr2=0x7f333003f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:07.114 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.572+0000 7f334c7dc700 1 --2- 172.21.15.78:0/2813116675 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7f333003d330 0x7f333003f7f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3334006fd0 tx=0x7f3334006e40 comp rx=0 tx=0).stop 2024-03-24T00:11:07.114 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.572+0000 7f334c7dc700 1 -- 172.21.15.78:0/2813116675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33441068c0 msgr2=0x7f33441997e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:07.114 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.572+0000 7f334c7dc700 1 --2- 172.21.15.78:0/2813116675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33441068c0 0x7f33441997e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3340011850 tx=0x7f3340011c10 comp rx=0 tx=0).stop 2024-03-24T00:11:07.114 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.572+0000 7f334c7dc700 1 -- 172.21.15.78:0/2813116675 shutdown_connections 2024-03-24T00:11:07.115 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.572+0000 7f334c7dc700 1 --2- 172.21.15.78:0/2813116675 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7f333003d330 0x7f333003f7f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:07.115 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.572+0000 7f334c7dc700 1 --2- 172.21.15.78:0/2813116675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33441068c0 0x7f33441997e0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:07.115 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.572+0000 7f334c7dc700 1 -- 172.21.15.78:0/2813116675 >> 172.21.15.78:0/2813116675 conn(0x7f3344101f40 msgr2=0x7f334418fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:07.115 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.572+0000 7f334c7dc700 1 -- 172.21.15.78:0/2813116675 shutdown_connections 2024-03-24T00:11:07.115 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:06.572+0000 7f334c7dc700 1 -- 172.21.15.78:0/2813116675 wait complete. 2024-03-24T00:11:07.115 INFO:teuthology.orchestra.run.smithi078.stdout:Enabling cephadm module... 2024-03-24T00:11:08.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:07 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/703130869' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-24T00:11:08.091 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:07 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: ignoring --setuser ceph since I am not root 2024-03-24T00:11:08.091 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:07 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: ignoring --setgroup ceph since I am not root 2024-03-24T00:11:08.840 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:08.390+0000 7fcab881e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T00:11:08.840 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:08.484+0000 7fcab881e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T00:11:09.106 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:08 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/703130869' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-24T00:11:09.106 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:08 smithi078 ceph-mon[32429]: mgrmap e5: x(active, since 10s) 2024-03-24T00:11:09.373 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:09 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:09.122+0000 7fcab881e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T00:11:09.374 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:09 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:09.214+0000 7fcab881e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T00:11:09.374 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:09 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:09.356+0000 7fcab881e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T00:11:09.381 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.529+0000 7f55ec19f700 1 Processor -- start 2024-03-24T00:11:09.381 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.529+0000 7f55ec19f700 1 -- start start 2024-03-24T00:11:09.381 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.529+0000 7f55ec19f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55e41068f0 0x7f55e4106d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:09.381 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.529+0000 7f55ec19f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e41072e0 con 0x7f55e41068f0 2024-03-24T00:11:09.381 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.529+0000 7f55e9f3b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55e41068f0 0x7f55e4106d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:09.381 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.529+0000 7f55e9f3b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55e41068f0 0x7f55e4106d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44682/0 (socket says 172.21.15.78:44682) 2024-03-24T00:11:09.381 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.529+0000 7f55e9f3b700 1 -- 172.21.15.78:0/2412829740 learned_addr learned my addr 172.21.15.78:0/2412829740 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:09.381 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.529+0000 7f55e9f3b700 1 -- 172.21.15.78:0/2412829740 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e4107af0 con 0x7f55e41068f0 2024-03-24T00:11:09.381 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.530+0000 7f55e9f3b700 1 --2- 172.21.15.78:0/2412829740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55e41068f0 0x7f55e4106d10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f55d40098a0 tx=0x7f55d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3292d45dc15b3cb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:09.382 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.530+0000 7f55e8f39700 1 -- 172.21.15.78:0/2412829740 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f55d4004160 con 0x7f55e41068f0 2024-03-24T00:11:09.382 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.530+0000 7f55e8f39700 1 -- 172.21.15.78:0/2412829740 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f55d4004300 con 0x7f55e41068f0 2024-03-24T00:11:09.382 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.530+0000 7f55e8f39700 1 -- 172.21.15.78:0/2412829740 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f55d4011b50 con 0x7f55e41068f0 2024-03-24T00:11:09.382 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.530+0000 7f55ec19f700 1 -- 172.21.15.78:0/2412829740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55e41068f0 msgr2=0x7f55e4106d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:09.382 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.530+0000 7f55ec19f700 1 --2- 172.21.15.78:0/2412829740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55e41068f0 0x7f55e4106d10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f55d40098a0 tx=0x7f55d4009bb0 comp rx=0 tx=0).stop 2024-03-24T00:11:09.382 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.530+0000 7f55ec19f700 1 -- 172.21.15.78:0/2412829740 shutdown_connections 2024-03-24T00:11:09.382 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.530+0000 7f55ec19f700 1 --2- 172.21.15.78:0/2412829740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55e41068f0 0x7f55e4106d10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:09.382 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.530+0000 7f55ec19f700 1 -- 172.21.15.78:0/2412829740 >> 172.21.15.78:0/2412829740 conn(0x7f55e4101f70 msgr2=0x7f55e4104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:09.382 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.530+0000 7f55ec19f700 1 -- 172.21.15.78:0/2412829740 shutdown_connections 2024-03-24T00:11:09.382 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.530+0000 7f55ec19f700 1 -- 172.21.15.78:0/2412829740 wait complete. 2024-03-24T00:11:09.382 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.531+0000 7f55ec19f700 1 Processor -- start 2024-03-24T00:11:09.382 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.531+0000 7f55ec19f700 1 -- start start 2024-03-24T00:11:09.382 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.531+0000 7f55ec19f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55e41068f0 0x7f55e4191aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:09.383 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.531+0000 7f55ec19f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e410aea0 con 0x7f55e41068f0 2024-03-24T00:11:09.383 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.531+0000 7f55e9f3b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55e41068f0 0x7f55e4191aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:09.383 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.531+0000 7f55e9f3b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55e41068f0 0x7f55e4191aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44690/0 (socket says 172.21.15.78:44690) 2024-03-24T00:11:09.383 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.531+0000 7f55e9f3b700 1 -- 172.21.15.78:0/703130869 learned_addr learned my addr 172.21.15.78:0/703130869 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:09.383 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.531+0000 7f55e9f3b700 1 -- 172.21.15.78:0/703130869 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e4191fe0 con 0x7f55e41068f0 2024-03-24T00:11:09.383 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.532+0000 7f55e9f3b700 1 --2- 172.21.15.78:0/703130869 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55e41068f0 0x7f55e4191aa0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f55d4011670 tx=0x7f55d4011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:09.383 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.532+0000 7f55daffd700 1 -- 172.21.15.78:0/703130869 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f55d40039c0 con 0x7f55e41068f0 2024-03-24T00:11:09.383 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.532+0000 7f55ec19f700 1 -- 172.21.15.78:0/703130869 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55e41901d0 con 0x7f55e41068f0 2024-03-24T00:11:09.383 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.532+0000 7f55ec19f700 1 -- 172.21.15.78:0/703130869 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55e4190710 con 0x7f55e41068f0 2024-03-24T00:11:09.383 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.532+0000 7f55daffd700 1 -- 172.21.15.78:0/703130869 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f55d4011e50 con 0x7f55e41068f0 2024-03-24T00:11:09.383 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.532+0000 7f55daffd700 1 -- 172.21.15.78:0/703130869 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f55d401a5c0 con 0x7f55e41068f0 2024-03-24T00:11:09.383 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.532+0000 7f55daffd700 1 -- 172.21.15.78:0/703130869 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f55d401a760 con 0x7f55e41068f0 2024-03-24T00:11:09.383 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.533+0000 7f55daffd700 1 --2- 172.21.15.78:0/703130869 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7f55d0038e80 0x7f55d003b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:09.384 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.533+0000 7f55daffd700 1 -- 172.21.15.78:0/703130869 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f55d404e020 con 0x7f55e41068f0 2024-03-24T00:11:09.384 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.533+0000 7f55e973a700 1 --2- 172.21.15.78:0/703130869 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7f55d0038e80 0x7f55d003b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:09.384 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.533+0000 7f55ec19f700 1 -- 172.21.15.78:0/703130869 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55c8005330 con 0x7f55e41068f0 2024-03-24T00:11:09.384 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.533+0000 7f55e973a700 1 --2- 172.21.15.78:0/703130869 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7f55d0038e80 0x7f55d003b340 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f55e0006fd0 tx=0x7f55e0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:09.384 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.538+0000 7f55daffd700 1 -- 172.21.15.78:0/703130869 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f55d400ba30 con 0x7f55e41068f0 2024-03-24T00:11:09.384 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.786+0000 7f55ec19f700 1 -- 172.21.15.78:0/703130869 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f55c8005620 con 0x7f55e41068f0 2024-03-24T00:11:09.384 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.838+0000 7f55daffd700 1 -- 172.21.15.78:0/703130869 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f55d4003b60 con 0x7f55e41068f0 2024-03-24T00:11:09.384 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.839+0000 7f55daffd700 1 -- 172.21.15.78:0/703130869 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f55d402d770 con 0x7f55e41068f0 2024-03-24T00:11:09.384 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.841+0000 7f55ec19f700 1 -- 172.21.15.78:0/703130869 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7f55d0038e80 msgr2=0x7f55d003b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:09.384 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.841+0000 7f55ec19f700 1 --2- 172.21.15.78:0/703130869 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7f55d0038e80 0x7f55d003b340 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f55e0006fd0 tx=0x7f55e0006e40 comp rx=0 tx=0).stop 2024-03-24T00:11:09.384 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.841+0000 7f55ec19f700 1 -- 172.21.15.78:0/703130869 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55e41068f0 msgr2=0x7f55e4191aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:09.384 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.841+0000 7f55ec19f700 1 --2- 172.21.15.78:0/703130869 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55e41068f0 0x7f55e4191aa0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f55d4011670 tx=0x7f55d4011b60 comp rx=0 tx=0).stop 2024-03-24T00:11:09.384 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.841+0000 7f55ec19f700 1 -- 172.21.15.78:0/703130869 shutdown_connections 2024-03-24T00:11:09.384 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.841+0000 7f55ec19f700 1 --2- 172.21.15.78:0/703130869 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7f55d0038e80 0x7f55d003b340 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:09.385 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.841+0000 7f55ec19f700 1 --2- 172.21.15.78:0/703130869 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55e41068f0 0x7f55e4191aa0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:09.385 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.841+0000 7f55ec19f700 1 -- 172.21.15.78:0/703130869 >> 172.21.15.78:0/703130869 conn(0x7f55e4101f70 msgr2=0x7f55e4102df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:09.385 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.841+0000 7f55ec19f700 1 -- 172.21.15.78:0/703130869 shutdown_connections 2024-03-24T00:11:09.385 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:07.841+0000 7f55ec19f700 1 -- 172.21.15.78:0/703130869 wait complete. 2024-03-24T00:11:09.718 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:09 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:09.467+0000 7fcab881e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T00:11:09.718 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:09 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:09.551+0000 7fcab881e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T00:11:10.017 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:09 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:09.817+0000 7fcab881e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T00:11:10.017 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:09 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:09.931+0000 7fcab881e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T00:11:10.017 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:10 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:10.017+0000 7fcab881e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T00:11:10.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:10 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3958623495' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T00:11:10.290 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:10 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:10.116+0000 7fcab881e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T00:11:10.590 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:10 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:10.290+0000 7fcab881e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T00:11:10.639 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-03-24T00:11:10.639 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-24T00:11:10.639 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T00:11:10.639 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-24T00:11:10.639 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-24T00:11:10.639 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-03-24T00:11:10.639 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.818+0000 7fa1f04dd700 1 Processor -- start 2024-03-24T00:11:10.640 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.818+0000 7fa1f04dd700 1 -- start start 2024-03-24T00:11:10.640 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.818+0000 7fa1f04dd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1e8108460 0x7fa1e810a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:10.640 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.818+0000 7fa1f04dd700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e807ae90 con 0x7fa1e8108460 2024-03-24T00:11:10.640 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.818+0000 7fa1ee279700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1e8108460 0x7fa1e810a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:10.640 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.818+0000 7fa1ee279700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1e8108460 0x7fa1e810a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44714/0 (socket says 172.21.15.78:44714) 2024-03-24T00:11:10.640 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.818+0000 7fa1ee279700 1 -- 172.21.15.78:0/518187368 learned_addr learned my addr 172.21.15.78:0/518187368 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:10.640 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.819+0000 7fa1ee279700 1 -- 172.21.15.78:0/518187368 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e810ad90 con 0x7fa1e8108460 2024-03-24T00:11:10.640 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.819+0000 7fa1ee279700 1 --2- 172.21.15.78:0/518187368 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1e8108460 0x7fa1e810a850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa1e40098a0 tx=0x7fa1e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c26c5a1da5727cf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:10.640 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.819+0000 7fa1ed277700 1 -- 172.21.15.78:0/518187368 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa1e4004160 con 0x7fa1e8108460 2024-03-24T00:11:10.640 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.819+0000 7fa1ed277700 1 -- 172.21.15.78:0/518187368 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa1e4004300 con 0x7fa1e8108460 2024-03-24T00:11:10.640 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.819+0000 7fa1ed277700 1 -- 172.21.15.78:0/518187368 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa1e4011b50 con 0x7fa1e8108460 2024-03-24T00:11:10.640 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.820+0000 7fa1f04dd700 1 -- 172.21.15.78:0/518187368 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1e8108460 msgr2=0x7fa1e810a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:10.640 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.820+0000 7fa1f04dd700 1 --2- 172.21.15.78:0/518187368 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1e8108460 0x7fa1e810a850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa1e40098a0 tx=0x7fa1e4009bb0 comp rx=0 tx=0).stop 2024-03-24T00:11:10.640 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.820+0000 7fa1f04dd700 1 -- 172.21.15.78:0/518187368 shutdown_connections 2024-03-24T00:11:10.641 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.820+0000 7fa1f04dd700 1 --2- 172.21.15.78:0/518187368 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1e8108460 0x7fa1e810a850 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:10.641 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.820+0000 7fa1f04dd700 1 -- 172.21.15.78:0/518187368 >> 172.21.15.78:0/518187368 conn(0x7fa1e8101f10 msgr2=0x7fa1e8104330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:10.641 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.820+0000 7fa1f04dd700 1 -- 172.21.15.78:0/518187368 shutdown_connections 2024-03-24T00:11:10.641 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.820+0000 7fa1f04dd700 1 -- 172.21.15.78:0/518187368 wait complete. 2024-03-24T00:11:10.641 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.820+0000 7fa1f04dd700 1 Processor -- start 2024-03-24T00:11:10.641 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.820+0000 7fa1f04dd700 1 -- start start 2024-03-24T00:11:10.641 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.821+0000 7fa1f04dd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1e8108460 0x7fa1e819b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:10.641 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.821+0000 7fa1f04dd700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e810bdb0 con 0x7fa1e8108460 2024-03-24T00:11:10.641 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.821+0000 7fa1ee279700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1e8108460 0x7fa1e819b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:10.641 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.821+0000 7fa1ee279700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1e8108460 0x7fa1e819b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44718/0 (socket says 172.21.15.78:44718) 2024-03-24T00:11:10.641 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.821+0000 7fa1ee279700 1 -- 172.21.15.78:0/3958623495 learned_addr learned my addr 172.21.15.78:0/3958623495 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:10.641 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.821+0000 7fa1ee279700 1 -- 172.21.15.78:0/3958623495 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e819be60 con 0x7fa1e8108460 2024-03-24T00:11:10.641 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.821+0000 7fa1ee279700 1 --2- 172.21.15.78:0/3958623495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1e8108460 0x7fa1e819b920 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4011670 tx=0x7fa1e4011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:10.641 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.821+0000 7fa1df7fe700 1 -- 172.21.15.78:0/3958623495 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa1e40039c0 con 0x7fa1e8108460 2024-03-24T00:11:10.642 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.821+0000 7fa1df7fe700 1 -- 172.21.15.78:0/3958623495 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa1e4011e50 con 0x7fa1e8108460 2024-03-24T00:11:10.642 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.821+0000 7fa1f04dd700 1 -- 172.21.15.78:0/3958623495 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1e819c0f0 con 0x7fa1e8108460 2024-03-24T00:11:10.642 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.821+0000 7fa1df7fe700 1 -- 172.21.15.78:0/3958623495 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa1e401a5c0 con 0x7fa1e8108460 2024-03-24T00:11:10.642 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.821+0000 7fa1f04dd700 1 -- 172.21.15.78:0/3958623495 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1e819ede0 con 0x7fa1e8108460 2024-03-24T00:11:10.642 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.822+0000 7fa1df7fe700 1 -- 172.21.15.78:0/3958623495 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fa1e401a8c0 con 0x7fa1e8108460 2024-03-24T00:11:10.642 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.822+0000 7fa1df7fe700 1 --2- 172.21.15.78:0/3958623495 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7fa1d4038ed0 0x7fa1d403b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:10.642 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.822+0000 7fa1df7fe700 1 -- 172.21.15.78:0/3958623495 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa1e4024770 con 0x7fa1e8108460 2024-03-24T00:11:10.642 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.822+0000 7fa1eda78700 1 -- 172.21.15.78:0/3958623495 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7fa1d4038ed0 msgr2=0x7fa1d403b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2913847518 2024-03-24T00:11:10.642 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.822+0000 7fa1eda78700 1 --2- 172.21.15.78:0/3958623495 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7fa1d4038ed0 0x7fa1d403b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:11:10.642 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.822+0000 7fa1f04dd700 1 -- 172.21.15.78:0/3958623495 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1e8069220 con 0x7fa1e8108460 2024-03-24T00:11:10.642 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:09.827+0000 7fa1df7fe700 1 -- 172.21.15.78:0/3958623495 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa1e4019c70 con 0x7fa1e8108460 2024-03-24T00:11:10.642 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:10.023+0000 7fa1eda78700 1 -- 172.21.15.78:0/3958623495 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7fa1d4038ed0 msgr2=0x7fa1d403b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2913847518 2024-03-24T00:11:10.643 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:10.023+0000 7fa1eda78700 1 --2- 172.21.15.78:0/3958623495 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7fa1d4038ed0 0x7fa1d403b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:11:10.643 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:10.081+0000 7fa1f04dd700 1 -- 172.21.15.78:0/3958623495 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fa1e819efb0 con 0x7fa1e8108460 2024-03-24T00:11:10.643 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:10.082+0000 7fa1df7fe700 1 -- 172.21.15.78:0/3958623495 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7fa1e400f2e0 con 0x7fa1e8108460 2024-03-24T00:11:10.643 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:10.084+0000 7fa1f04dd700 1 -- 172.21.15.78:0/3958623495 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7fa1d4038ed0 msgr2=0x7fa1d403b390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T00:11:10.643 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:10.084+0000 7fa1f04dd700 1 --2- 172.21.15.78:0/3958623495 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7fa1d4038ed0 0x7fa1d403b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:10.643 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:10.084+0000 7fa1f04dd700 1 -- 172.21.15.78:0/3958623495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1e8108460 msgr2=0x7fa1e819b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:10.643 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:10.084+0000 7fa1f04dd700 1 --2- 172.21.15.78:0/3958623495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1e8108460 0x7fa1e819b920 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4011670 tx=0x7fa1e4011b60 comp rx=0 tx=0).stop 2024-03-24T00:11:10.643 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:10.084+0000 7fa1f04dd700 1 -- 172.21.15.78:0/3958623495 shutdown_connections 2024-03-24T00:11:10.643 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:10.084+0000 7fa1f04dd700 1 --2- 172.21.15.78:0/3958623495 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7fa1d4038ed0 0x7fa1d403b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:10.643 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:10.084+0000 7fa1f04dd700 1 --2- 172.21.15.78:0/3958623495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1e8108460 0x7fa1e819b920 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:10.643 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:10.084+0000 7fa1f04dd700 1 -- 172.21.15.78:0/3958623495 >> 172.21.15.78:0/3958623495 conn(0x7fa1e8101f10 msgr2=0x7fa1e81029f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:10.643 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:10.084+0000 7fa1f04dd700 1 -- 172.21.15.78:0/3958623495 shutdown_connections 2024-03-24T00:11:10.643 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:10.084+0000 7fa1f04dd700 1 -- 172.21.15.78:0/3958623495 wait complete. 2024-03-24T00:11:10.644 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for the mgr to restart... 2024-03-24T00:11:10.644 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mgr epoch 5... 2024-03-24T00:11:11.590 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:11 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:11.259+0000 7fcab881e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T00:11:11.591 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:11 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:11.387+0000 7fcab881e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T00:11:11.994 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:11 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:11.712+0000 7fcab881e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T00:11:11.995 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:11 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:11.801+0000 7fcab881e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T00:11:11.995 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:11 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:11.994+0000 7fcab881e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:11:12.340 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:12 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:12.178+0000 7fcab881e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T00:11:12.719 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:12 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:12.462+0000 7fcab881e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:11:12.719 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:12 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:12.719+0000 7fcab881e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T00:11:13.090 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:12 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:12.954+0000 7fcab881e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T00:11:14.090 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:13.746+0000 7fcab881e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T00:11:14.409 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:14.311+0000 7fcab881e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T00:11:14.409 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:14.409+0000 7fcab881e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T00:11:14.688 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:14.624+0000 7fcab881e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T00:11:15.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:14 smithi078 ceph-mon[32429]: Active manager daemon x restarted 2024-03-24T00:11:15.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:14 smithi078 ceph-mon[32429]: Activating manager daemon x 2024-03-24T00:11:15.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:14 smithi078 ceph-mon[32429]: osdmap e2: 0 total, 0 up, 0 in 2024-03-24T00:11:15.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:14 smithi078 ceph-mon[32429]: mgrmap e6: x(active, starting, since 0.00325702s) 2024-03-24T00:11:15.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:14 smithi078 ceph-mon[32429]: from='mgr.14118 172.21.15.78:0/3702326166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:11:15.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:14 smithi078 ceph-mon[32429]: from='mgr.14118 172.21.15.78:0/3702326166' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T00:11:15.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:14 smithi078 ceph-mon[32429]: from='mgr.14118 172.21.15.78:0/3702326166' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T00:11:15.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:14 smithi078 ceph-mon[32429]: from='mgr.14118 172.21.15.78:0/3702326166' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T00:11:15.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:14 smithi078 ceph-mon[32429]: from='mgr.14118 172.21.15.78:0/3702326166' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T00:11:15.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:14 smithi078 ceph-mon[32429]: Manager daemon x is now available 2024-03-24T00:11:15.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:14 smithi078 ceph-mon[32429]: from='mgr.14118 172.21.15.78:0/3702326166' entity='mgr.x' 2024-03-24T00:11:15.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:14 smithi078 ceph-mon[32429]: from='mgr.14118 172.21.15.78:0/3702326166' entity='mgr.x' 2024-03-24T00:11:15.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:14 smithi078 ceph-mon[32429]: from='mgr.14118 172.21.15.78:0/3702326166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:11:15.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:14 smithi078 ceph-mon[32429]: from='mgr.14118 172.21.15.78:0/3702326166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:11:15.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:14 smithi078 ceph-mon[32429]: from='mgr.14118 172.21.15.78:0/3702326166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:11:15.960 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:15 smithi078 ceph-mon[32429]: Found migration_current of "None". Setting to last migration. 2024-03-24T00:11:15.960 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:15 smithi078 ceph-mon[32429]: from='mgr.14118 172.21.15.78:0/3702326166' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T00:11:15.960 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:15 smithi078 ceph-mon[32429]: from='mgr.14118 172.21.15.78:0/3702326166' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T00:11:15.960 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:15 smithi078 ceph-mon[32429]: from='mgr.14118 172.21.15.78:0/3702326166' entity='mgr.x' 2024-03-24T00:11:15.960 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:15 smithi078 ceph-mon[32429]: from='mgr.14118 172.21.15.78:0/3702326166' entity='mgr.x' 2024-03-24T00:11:15.960 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:15 smithi078 ceph-mon[32429]: mgrmap e7: x(active, since 1.00506s) 2024-03-24T00:11:16.248 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-03-24T00:11:16.248 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-24T00:11:16.248 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-24T00:11:16.248 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-03-24T00:11:16.248 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.132+0000 7f691a0fe700 1 Processor -- start 2024-03-24T00:11:16.248 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.132+0000 7f691a0fe700 1 -- start start 2024-03-24T00:11:16.248 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.132+0000 7f691a0fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6914108450 0x7f691410a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:16.248 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.132+0000 7f691a0fe700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691407aed0 con 0x7f6914108450 2024-03-24T00:11:16.248 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.132+0000 7f69137fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6914108450 0x7f691410a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:16.249 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.132+0000 7f69137fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6914108450 0x7f691410a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44726/0 (socket says 172.21.15.78:44726) 2024-03-24T00:11:16.249 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.132+0000 7f69137fe700 1 -- 172.21.15.78:0/2238642077 learned_addr learned my addr 172.21.15.78:0/2238642077 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:16.249 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.133+0000 7f69137fe700 1 -- 172.21.15.78:0/2238642077 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691410ad80 con 0x7f6914108450 2024-03-24T00:11:16.249 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.133+0000 7f69137fe700 1 --2- 172.21.15.78:0/2238642077 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6914108450 0x7f691410a840 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f69080098a0 tx=0x7f6908009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb1251d8f3318766 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:16.249 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.133+0000 7f69127fc700 1 -- 172.21.15.78:0/2238642077 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6908004160 con 0x7f6914108450 2024-03-24T00:11:16.249 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.133+0000 7f69127fc700 1 -- 172.21.15.78:0/2238642077 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6908004300 con 0x7f6914108450 2024-03-24T00:11:16.249 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.133+0000 7f69127fc700 1 -- 172.21.15.78:0/2238642077 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6908011b50 con 0x7f6914108450 2024-03-24T00:11:16.249 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.133+0000 7f691a0fe700 1 -- 172.21.15.78:0/2238642077 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6914108450 msgr2=0x7f691410a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:16.249 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.134+0000 7f691a0fe700 1 --2- 172.21.15.78:0/2238642077 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6914108450 0x7f691410a840 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f69080098a0 tx=0x7f6908009bb0 comp rx=0 tx=0).stop 2024-03-24T00:11:16.249 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.134+0000 7f691a0fe700 1 -- 172.21.15.78:0/2238642077 shutdown_connections 2024-03-24T00:11:16.249 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.134+0000 7f691a0fe700 1 --2- 172.21.15.78:0/2238642077 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6914108450 0x7f691410a840 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:16.250 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.134+0000 7f691a0fe700 1 -- 172.21.15.78:0/2238642077 >> 172.21.15.78:0/2238642077 conn(0x7f6914101ee0 msgr2=0x7f6914104300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:16.250 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.134+0000 7f691a0fe700 1 -- 172.21.15.78:0/2238642077 shutdown_connections 2024-03-24T00:11:16.250 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.134+0000 7f691a0fe700 1 -- 172.21.15.78:0/2238642077 wait complete. 2024-03-24T00:11:16.250 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.134+0000 7f691a0fe700 1 Processor -- start 2024-03-24T00:11:16.250 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.134+0000 7f691a0fe700 1 -- start start 2024-03-24T00:11:16.250 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.135+0000 7f691a0fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6914108450 0x7f691419ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:16.250 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.135+0000 7f691a0fe700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691410bdb0 con 0x7f6914108450 2024-03-24T00:11:16.250 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.135+0000 7f69137fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6914108450 0x7f691419ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:16.250 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.135+0000 7f69137fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6914108450 0x7f691419ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44734/0 (socket says 172.21.15.78:44734) 2024-03-24T00:11:16.250 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.135+0000 7f69137fe700 1 -- 172.21.15.78:0/2920003835 learned_addr learned my addr 172.21.15.78:0/2920003835 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:16.250 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.135+0000 7f69137fe700 1 -- 172.21.15.78:0/2920003835 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691419bf40 con 0x7f6914108450 2024-03-24T00:11:16.251 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.135+0000 7f69137fe700 1 --2- 172.21.15.78:0/2920003835 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6914108450 0x7f691419ba00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6908011b60 tx=0x7f6908011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:16.251 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.135+0000 7f6910ff9700 1 -- 172.21.15.78:0/2920003835 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6908003970 con 0x7f6914108450 2024-03-24T00:11:16.251 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.135+0000 7f6910ff9700 1 -- 172.21.15.78:0/2920003835 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6908003b10 con 0x7f6914108450 2024-03-24T00:11:16.251 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.135+0000 7f6910ff9700 1 -- 172.21.15.78:0/2920003835 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6908024610 con 0x7f6914108450 2024-03-24T00:11:16.251 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.135+0000 7f691a0fe700 1 -- 172.21.15.78:0/2920003835 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f691419c1d0 con 0x7f6914108450 2024-03-24T00:11:16.251 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.136+0000 7f691a0fe700 1 -- 172.21.15.78:0/2920003835 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f691419eec0 con 0x7f6914108450 2024-03-24T00:11:16.251 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.136+0000 7f6910ff9700 1 -- 172.21.15.78:0/2920003835 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f69080248c0 con 0x7f6914108450 2024-03-24T00:11:16.251 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.137+0000 7f6910ff9700 1 --2- 172.21.15.78:0/2920003835 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7f68fc038f70 0x7f68fc03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:16.251 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.137+0000 7f6910ff9700 1 -- 172.21.15.78:0/2920003835 --> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f68fc03bb40 con 0x7f68fc038f70 2024-03-24T00:11:16.251 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.137+0000 7f6910ff9700 1 -- 172.21.15.78:0/2920003835 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6908011e00 con 0x7f6914108450 2024-03-24T00:11:16.251 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.137+0000 7f6912ffd700 1 -- 172.21.15.78:0/2920003835 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7f68fc038f70 msgr2=0x7f68fc03b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2913847518 2024-03-24T00:11:16.252 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.137+0000 7f6912ffd700 1 --2- 172.21.15.78:0/2920003835 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7f68fc038f70 0x7f68fc03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:11:16.252 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.338+0000 7f6912ffd700 1 -- 172.21.15.78:0/2920003835 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7f68fc038f70 msgr2=0x7f68fc03b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2913847518 2024-03-24T00:11:16.252 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.338+0000 7f6912ffd700 1 --2- 172.21.15.78:0/2920003835 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7f68fc038f70 0x7f68fc03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:11:16.252 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.739+0000 7f6912ffd700 1 -- 172.21.15.78:0/2920003835 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7f68fc038f70 msgr2=0x7f68fc03b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2913847518 2024-03-24T00:11:16.252 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:11.739+0000 7f6912ffd700 1 --2- 172.21.15.78:0/2920003835 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7f68fc038f70 0x7f68fc03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:11:16.252 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:12.540+0000 7f6912ffd700 1 -- 172.21.15.78:0/2920003835 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7f68fc038f70 msgr2=0x7f68fc03b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2913847518 2024-03-24T00:11:16.252 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:12.540+0000 7f6912ffd700 1 --2- 172.21.15.78:0/2920003835 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7f68fc038f70 0x7f68fc03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:11:16.252 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:14.141+0000 7f6912ffd700 1 -- 172.21.15.78:0/2920003835 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7f68fc038f70 msgr2=0x7f68fc03b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2913847518 2024-03-24T00:11:16.252 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:14.141+0000 7f6912ffd700 1 --2- 172.21.15.78:0/2920003835 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7f68fc038f70 0x7f68fc03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T00:11:16.252 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:14.639+0000 7f6910ff9700 1 -- 172.21.15.78:0/2920003835 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mgrmap(e 6) v1 ==== 45873+0+0 (secure 0 0 0) 0x7f6908024bb0 con 0x7f6914108450 2024-03-24T00:11:16.252 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:14.639+0000 7f6910ff9700 1 -- 172.21.15.78:0/2920003835 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7f68fc038f70 msgr2=0x7f68fc03b430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T00:11:16.252 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:14.639+0000 7f6910ff9700 1 --2- 172.21.15.78:0/2920003835 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7f68fc038f70 0x7f68fc03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:16.252 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:15.640+0000 7f6910ff9700 1 -- 172.21.15.78:0/2920003835 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 7) v1 ==== 46000+0+0 (secure 0 0 0) 0x7f690804edc0 con 0x7f6914108450 2024-03-24T00:11:16.253 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:15.640+0000 7f6910ff9700 1 --2- 172.21.15.78:0/2920003835 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f68fc03c860 0x7f68fc03ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:16.253 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:15.640+0000 7f6910ff9700 1 -- 172.21.15.78:0/2920003835 --> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6908010ec0 con 0x7f68fc03c860 2024-03-24T00:11:16.253 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:15.641+0000 7f6912ffd700 1 --2- 172.21.15.78:0/2920003835 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f68fc03c860 0x7f68fc03ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:16.253 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:15.641+0000 7f6912ffd700 1 --2- 172.21.15.78:0/2920003835 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f68fc03c860 0x7f68fc03ec50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6904003a10 tx=0x7f69040092b0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:16.253 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:15.642+0000 7f6910ff9700 1 -- 172.21.15.78:0/2920003835 <== mgr.14118 v2:172.21.15.78:6800/2748920178 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f6908010ec0 con 0x7f68fc03c860 2024-03-24T00:11:16.253 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:15.648+0000 7f691a0fe700 1 -- 172.21.15.78:0/2920003835 --> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f6914069220 con 0x7f68fc03c860 2024-03-24T00:11:16.253 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:15.648+0000 7f6910ff9700 1 -- 172.21.15.78:0/2920003835 <== mgr.14118 v2:172.21.15.78:6800/2748920178 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f6914069220 con 0x7f68fc03c860 2024-03-24T00:11:16.253 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:15.649+0000 7f691a0fe700 1 -- 172.21.15.78:0/2920003835 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f68fc03c860 msgr2=0x7f68fc03ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:16.253 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:15.649+0000 7f691a0fe700 1 --2- 172.21.15.78:0/2920003835 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f68fc03c860 0x7f68fc03ec50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6904003a10 tx=0x7f69040092b0 comp rx=0 tx=0).stop 2024-03-24T00:11:16.253 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:15.649+0000 7f691a0fe700 1 -- 172.21.15.78:0/2920003835 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6914108450 msgr2=0x7f691419ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:16.253 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:15.649+0000 7f691a0fe700 1 --2- 172.21.15.78:0/2920003835 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6914108450 0x7f691419ba00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6908011b60 tx=0x7f6908011c40 comp rx=0 tx=0).stop 2024-03-24T00:11:16.253 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:15.649+0000 7f691a0fe700 1 -- 172.21.15.78:0/2920003835 shutdown_connections 2024-03-24T00:11:16.253 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:15.649+0000 7f691a0fe700 1 --2- 172.21.15.78:0/2920003835 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f68fc03c860 0x7f68fc03ec50 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:16.254 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:15.649+0000 7f691a0fe700 1 --2- 172.21.15.78:0/2920003835 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6914108450 0x7f691419ba00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:16.254 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:15.649+0000 7f691a0fe700 1 --2- 172.21.15.78:0/2920003835 >> [v2:172.21.15.78:6800/2913847518,v1:172.21.15.78:6801/2913847518] conn(0x7f68fc038f70 0x7f68fc03b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:16.254 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:15.649+0000 7f691a0fe700 1 -- 172.21.15.78:0/2920003835 >> 172.21.15.78:0/2920003835 conn(0x7f6914101ee0 msgr2=0x7f6914102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:16.254 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:15.649+0000 7f691a0fe700 1 -- 172.21.15.78:0/2920003835 shutdown_connections 2024-03-24T00:11:16.254 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:15.649+0000 7f691a0fe700 1 -- 172.21.15.78:0/2920003835 wait complete. 2024-03-24T00:11:16.254 INFO:teuthology.orchestra.run.smithi078.stdout:mgr epoch 5 is available 2024-03-24T00:11:16.254 INFO:teuthology.orchestra.run.smithi078.stdout:Setting orchestrator backend to cephadm... 2024-03-24T00:11:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:16 smithi078 ceph-mon[32429]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-24T00:11:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:16 smithi078 ceph-mon[32429]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-24T00:11:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:16 smithi078 ceph-mon[32429]: from='mgr.14118 172.21.15.78:0/3702326166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:11:17.442 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.671+0000 7f63f7a9a700 1 Processor -- start 2024-03-24T00:11:17.442 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.671+0000 7f63f7a9a700 1 -- start start 2024-03-24T00:11:17.442 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.671+0000 7f63f7a9a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63f01084a0 0x7f63f010a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:17.442 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.671+0000 7f63f7a9a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f007aeb0 con 0x7f63f01084a0 2024-03-24T00:11:17.442 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.671+0000 7f63f5836700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63f01084a0 0x7f63f010a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:17.443 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.671+0000 7f63f5836700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63f01084a0 0x7f63f010a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60858/0 (socket says 172.21.15.78:60858) 2024-03-24T00:11:17.443 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.671+0000 7f63f5836700 1 -- 172.21.15.78:0/727901995 learned_addr learned my addr 172.21.15.78:0/727901995 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:17.443 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.672+0000 7f63f5836700 1 -- 172.21.15.78:0/727901995 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f010add0 con 0x7f63f01084a0 2024-03-24T00:11:17.443 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.672+0000 7f63f5836700 1 --2- 172.21.15.78:0/727901995 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63f01084a0 0x7f63f010a890 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f63ec0098a0 tx=0x7f63ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10573bd75cd6e535 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:17.443 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.672+0000 7f63f4834700 1 -- 172.21.15.78:0/727901995 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f63ec004160 con 0x7f63f01084a0 2024-03-24T00:11:17.443 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.672+0000 7f63f4834700 1 -- 172.21.15.78:0/727901995 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f63ec004300 con 0x7f63f01084a0 2024-03-24T00:11:17.443 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.672+0000 7f63f4834700 1 -- 172.21.15.78:0/727901995 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f63ec011b50 con 0x7f63f01084a0 2024-03-24T00:11:17.443 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.672+0000 7f63f7a9a700 1 -- 172.21.15.78:0/727901995 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63f01084a0 msgr2=0x7f63f010a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:17.443 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.672+0000 7f63f7a9a700 1 --2- 172.21.15.78:0/727901995 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63f01084a0 0x7f63f010a890 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f63ec0098a0 tx=0x7f63ec009bb0 comp rx=0 tx=0).stop 2024-03-24T00:11:17.443 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.673+0000 7f63f7a9a700 1 -- 172.21.15.78:0/727901995 shutdown_connections 2024-03-24T00:11:17.443 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.673+0000 7f63f7a9a700 1 --2- 172.21.15.78:0/727901995 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63f01084a0 0x7f63f010a890 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:17.443 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.673+0000 7f63f7a9a700 1 -- 172.21.15.78:0/727901995 >> 172.21.15.78:0/727901995 conn(0x7f63f0101f70 msgr2=0x7f63f0104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:17.443 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.673+0000 7f63f7a9a700 1 -- 172.21.15.78:0/727901995 shutdown_connections 2024-03-24T00:11:17.444 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.673+0000 7f63f7a9a700 1 -- 172.21.15.78:0/727901995 wait complete. 2024-03-24T00:11:17.444 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.673+0000 7f63f7a9a700 1 Processor -- start 2024-03-24T00:11:17.444 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.673+0000 7f63f7a9a700 1 -- start start 2024-03-24T00:11:17.444 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.673+0000 7f63f7a9a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63f01084a0 0x7f63f019b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:17.444 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.673+0000 7f63f7a9a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f010bd90 con 0x7f63f01084a0 2024-03-24T00:11:17.444 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.674+0000 7f63f5836700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63f01084a0 0x7f63f019b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:17.444 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.674+0000 7f63f5836700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63f01084a0 0x7f63f019b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60870/0 (socket says 172.21.15.78:60870) 2024-03-24T00:11:17.444 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.674+0000 7f63f5836700 1 -- 172.21.15.78:0/3608536733 learned_addr learned my addr 172.21.15.78:0/3608536733 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:17.444 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.674+0000 7f63f5836700 1 -- 172.21.15.78:0/3608536733 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f019bed0 con 0x7f63f01084a0 2024-03-24T00:11:17.444 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.674+0000 7f63f5836700 1 --2- 172.21.15.78:0/3608536733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63f01084a0 0x7f63f019b990 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f63ec011b60 tx=0x7f63ec011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:17.444 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.674+0000 7f63e6ffd700 1 -- 172.21.15.78:0/3608536733 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f63ec003970 con 0x7f63f01084a0 2024-03-24T00:11:17.444 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.674+0000 7f63f7a9a700 1 -- 172.21.15.78:0/3608536733 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63f019c1c0 con 0x7f63f01084a0 2024-03-24T00:11:17.444 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.674+0000 7f63e6ffd700 1 -- 172.21.15.78:0/3608536733 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f63ec003b10 con 0x7f63f01084a0 2024-03-24T00:11:17.444 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.674+0000 7f63e6ffd700 1 -- 172.21.15.78:0/3608536733 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f63ec024610 con 0x7f63f01084a0 2024-03-24T00:11:17.444 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.674+0000 7f63f7a9a700 1 -- 172.21.15.78:0/3608536733 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63f019ee50 con 0x7f63f01084a0 2024-03-24T00:11:17.445 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.675+0000 7f63e6ffd700 1 -- 172.21.15.78:0/3608536733 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 7) v1 ==== 46000+0+0 (secure 0 0 0) 0x7f63ec01a430 con 0x7f63f01084a0 2024-03-24T00:11:17.445 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.675+0000 7f63f7a9a700 1 -- 172.21.15.78:0/3608536733 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63f00637c0 con 0x7f63f01084a0 2024-03-24T00:11:17.445 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.675+0000 7f63e6ffd700 1 --2- 172.21.15.78:0/3608536733 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f63dc038e00 0x7f63dc03b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:17.445 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.675+0000 7f63e6ffd700 1 -- 172.21.15.78:0/3608536733 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f63ec011e00 con 0x7f63f01084a0 2024-03-24T00:11:17.445 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.675+0000 7f63f5035700 1 --2- 172.21.15.78:0/3608536733 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f63dc038e00 0x7f63dc03b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:17.445 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.676+0000 7f63f5035700 1 --2- 172.21.15.78:0/3608536733 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f63dc038e00 0x7f63dc03b2c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f63e0006fd0 tx=0x7f63e0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:17.445 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.680+0000 7f63e6ffd700 1 -- 172.21.15.78:0/3608536733 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f63ec02ce20 con 0x7f63f01084a0 2024-03-24T00:11:17.445 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.694+0000 7f63e6ffd700 1 -- 172.21.15.78:0/3608536733 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f63ec012040 con 0x7f63f01084a0 2024-03-24T00:11:17.445 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.892+0000 7f63f7a9a700 1 -- 172.21.15.78:0/3608536733 --> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f63f01031c0 con 0x7f63dc038e00 2024-03-24T00:11:17.445 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.897+0000 7f63e6ffd700 1 -- 172.21.15.78:0/3608536733 <== mgr.14118 v2:172.21.15.78:6800/2748920178 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f63f01031c0 con 0x7f63dc038e00 2024-03-24T00:11:17.445 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.900+0000 7f63f7a9a700 1 -- 172.21.15.78:0/3608536733 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f63dc038e00 msgr2=0x7f63dc03b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:17.445 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.900+0000 7f63f7a9a700 1 --2- 172.21.15.78:0/3608536733 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f63dc038e00 0x7f63dc03b2c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f63e0006fd0 tx=0x7f63e0006e40 comp rx=0 tx=0).stop 2024-03-24T00:11:17.445 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.900+0000 7f63f7a9a700 1 -- 172.21.15.78:0/3608536733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63f01084a0 msgr2=0x7f63f019b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:17.445 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.900+0000 7f63f7a9a700 1 --2- 172.21.15.78:0/3608536733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63f01084a0 0x7f63f019b990 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f63ec011b60 tx=0x7f63ec011c40 comp rx=0 tx=0).stop 2024-03-24T00:11:17.446 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.900+0000 7f63f7a9a700 1 -- 172.21.15.78:0/3608536733 shutdown_connections 2024-03-24T00:11:17.446 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.900+0000 7f63f7a9a700 1 --2- 172.21.15.78:0/3608536733 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f63dc038e00 0x7f63dc03b2c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:17.446 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.900+0000 7f63f7a9a700 1 --2- 172.21.15.78:0/3608536733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63f01084a0 0x7f63f019b990 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:17.446 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.900+0000 7f63f7a9a700 1 -- 172.21.15.78:0/3608536733 >> 172.21.15.78:0/3608536733 conn(0x7f63f0101f70 msgr2=0x7f63f0102ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:17.446 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.900+0000 7f63f7a9a700 1 -- 172.21.15.78:0/3608536733 shutdown_connections 2024-03-24T00:11:17.446 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:16.900+0000 7f63f7a9a700 1 -- 172.21.15.78:0/3608536733 wait complete. 2024-03-24T00:11:17.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:17 smithi078 ceph-mon[32429]: [24/Mar/2024:00:11:16] ENGINE Bus STARTING 2024-03-24T00:11:17.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:17 smithi078 ceph-mon[32429]: [24/Mar/2024:00:11:16] ENGINE Serving on http://172.21.15.78:8765 2024-03-24T00:11:17.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:17 smithi078 ceph-mon[32429]: [24/Mar/2024:00:11:16] ENGINE Serving on https://172.21.15.78:7150 2024-03-24T00:11:17.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:17 smithi078 ceph-mon[32429]: [24/Mar/2024:00:11:16] ENGINE Bus STARTED 2024-03-24T00:11:17.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:17 smithi078 ceph-mon[32429]: mgrmap e8: x(active, since 2s) 2024-03-24T00:11:17.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:17 smithi078 ceph-mon[32429]: from='mgr.14118 172.21.15.78:0/3702326166' entity='mgr.x' 2024-03-24T00:11:17.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:17 smithi078 ceph-mon[32429]: from='mgr.14118 172.21.15.78:0/3702326166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:11:18.673 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-24T00:11:18.674 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.888+0000 7f7c66f0b700 1 Processor -- start 2024-03-24T00:11:18.674 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.888+0000 7f7c66f0b700 1 -- start start 2024-03-24T00:11:18.674 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.888+0000 7f7c66f0b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c60108470 0x7f7c6010a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:18.674 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.888+0000 7f7c66f0b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6007aeb0 con 0x7f7c60108470 2024-03-24T00:11:18.674 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.889+0000 7f7c64ca7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c60108470 0x7f7c6010a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:18.674 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.889+0000 7f7c64ca7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c60108470 0x7f7c6010a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60886/0 (socket says 172.21.15.78:60886) 2024-03-24T00:11:18.674 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.889+0000 7f7c64ca7700 1 -- 172.21.15.78:0/4199201086 learned_addr learned my addr 172.21.15.78:0/4199201086 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:18.674 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.889+0000 7f7c64ca7700 1 -- 172.21.15.78:0/4199201086 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c6010ada0 con 0x7f7c60108470 2024-03-24T00:11:18.674 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.889+0000 7f7c64ca7700 1 --2- 172.21.15.78:0/4199201086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c60108470 0x7f7c6010a860 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7c540098a0 tx=0x7f7c54009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d58d4959076a587 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:18.674 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.889+0000 7f7c5f7fe700 1 -- 172.21.15.78:0/4199201086 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7c54004160 con 0x7f7c60108470 2024-03-24T00:11:18.674 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.889+0000 7f7c5f7fe700 1 -- 172.21.15.78:0/4199201086 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7c54004300 con 0x7f7c60108470 2024-03-24T00:11:18.674 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.889+0000 7f7c5f7fe700 1 -- 172.21.15.78:0/4199201086 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7c54011b50 con 0x7f7c60108470 2024-03-24T00:11:18.674 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.890+0000 7f7c66f0b700 1 -- 172.21.15.78:0/4199201086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c60108470 msgr2=0x7f7c6010a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:18.675 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.890+0000 7f7c66f0b700 1 --2- 172.21.15.78:0/4199201086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c60108470 0x7f7c6010a860 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7c540098a0 tx=0x7f7c54009bb0 comp rx=0 tx=0).stop 2024-03-24T00:11:18.675 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.890+0000 7f7c66f0b700 1 -- 172.21.15.78:0/4199201086 shutdown_connections 2024-03-24T00:11:18.675 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.890+0000 7f7c66f0b700 1 --2- 172.21.15.78:0/4199201086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c60108470 0x7f7c6010a860 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:18.675 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.890+0000 7f7c66f0b700 1 -- 172.21.15.78:0/4199201086 >> 172.21.15.78:0/4199201086 conn(0x7f7c60101ee0 msgr2=0x7f7c60104300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:18.675 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.890+0000 7f7c66f0b700 1 -- 172.21.15.78:0/4199201086 shutdown_connections 2024-03-24T00:11:18.675 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.890+0000 7f7c66f0b700 1 -- 172.21.15.78:0/4199201086 wait complete. 2024-03-24T00:11:18.675 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.890+0000 7f7c66f0b700 1 Processor -- start 2024-03-24T00:11:18.675 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.890+0000 7f7c66f0b700 1 -- start start 2024-03-24T00:11:18.675 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.890+0000 7f7c66f0b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c60108470 0x7f7c6019b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:18.675 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.890+0000 7f7c66f0b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6010bd60 con 0x7f7c60108470 2024-03-24T00:11:18.676 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.891+0000 7f7c64ca7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c60108470 0x7f7c6019b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:18.676 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.891+0000 7f7c64ca7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c60108470 0x7f7c6019b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60894/0 (socket says 172.21.15.78:60894) 2024-03-24T00:11:18.676 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.891+0000 7f7c64ca7700 1 -- 172.21.15.78:0/605477584 learned_addr learned my addr 172.21.15.78:0/605477584 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:18.676 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.891+0000 7f7c64ca7700 1 -- 172.21.15.78:0/605477584 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c6019bef0 con 0x7f7c60108470 2024-03-24T00:11:18.676 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.891+0000 7f7c64ca7700 1 --2- 172.21.15.78:0/605477584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c60108470 0x7f7c6019b9b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7c54009fb0 tx=0x7f7c54011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:18.676 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.891+0000 7f7c5dffb700 1 -- 172.21.15.78:0/605477584 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7c540038b0 con 0x7f7c60108470 2024-03-24T00:11:18.676 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.891+0000 7f7c5dffb700 1 -- 172.21.15.78:0/605477584 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7c54011e50 con 0x7f7c60108470 2024-03-24T00:11:18.676 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.891+0000 7f7c5dffb700 1 -- 172.21.15.78:0/605477584 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7c540193c0 con 0x7f7c60108470 2024-03-24T00:11:18.676 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.891+0000 7f7c66f0b700 1 -- 172.21.15.78:0/605477584 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c6019c1e0 con 0x7f7c60108470 2024-03-24T00:11:18.676 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.891+0000 7f7c66f0b700 1 -- 172.21.15.78:0/605477584 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c6019ee70 con 0x7f7c60108470 2024-03-24T00:11:18.676 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.892+0000 7f7c5dffb700 1 -- 172.21.15.78:0/605477584 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f7c54003a50 con 0x7f7c60108470 2024-03-24T00:11:18.676 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.892+0000 7f7c66f0b700 1 -- 172.21.15.78:0/605477584 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c60069220 con 0x7f7c60108470 2024-03-24T00:11:18.676 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.892+0000 7f7c5dffb700 1 --2- 172.21.15.78:0/605477584 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f7c50038f20 0x7f7c5003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:18.676 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.892+0000 7f7c5dffb700 1 -- 172.21.15.78:0/605477584 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f7c5404d060 con 0x7f7c60108470 2024-03-24T00:11:18.677 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.892+0000 7f7c5ffff700 1 --2- 172.21.15.78:0/605477584 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f7c50038f20 0x7f7c5003b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:18.677 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.893+0000 7f7c5ffff700 1 --2- 172.21.15.78:0/605477584 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f7c50038f20 0x7f7c5003b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c006fd0 tx=0x7f7c4c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:18.677 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:17.897+0000 7f7c5dffb700 1 -- 172.21.15.78:0/605477584 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7c54023390 con 0x7f7c60108470 2024-03-24T00:11:18.677 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:18.091+0000 7f7c66f0b700 1 -- 172.21.15.78:0/605477584 --> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f7c6019f040 con 0x7f7c50038f20 2024-03-24T00:11:18.677 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:18.092+0000 7f7c5dffb700 1 -- 172.21.15.78:0/605477584 <== mgr.14118 v2:172.21.15.78:6800/2748920178 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f7c6019f040 con 0x7f7c50038f20 2024-03-24T00:11:18.677 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:18.094+0000 7f7c66f0b700 1 -- 172.21.15.78:0/605477584 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f7c50038f20 msgr2=0x7f7c5003b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:18.677 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:18.094+0000 7f7c66f0b700 1 --2- 172.21.15.78:0/605477584 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f7c50038f20 0x7f7c5003b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c006fd0 tx=0x7f7c4c006e40 comp rx=0 tx=0).stop 2024-03-24T00:11:18.677 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:18.094+0000 7f7c66f0b700 1 -- 172.21.15.78:0/605477584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c60108470 msgr2=0x7f7c6019b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:18.677 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:18.094+0000 7f7c66f0b700 1 --2- 172.21.15.78:0/605477584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c60108470 0x7f7c6019b9b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7c54009fb0 tx=0x7f7c54011bf0 comp rx=0 tx=0).stop 2024-03-24T00:11:18.677 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:18.094+0000 7f7c66f0b700 1 -- 172.21.15.78:0/605477584 shutdown_connections 2024-03-24T00:11:18.677 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:18.094+0000 7f7c66f0b700 1 --2- 172.21.15.78:0/605477584 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f7c50038f20 0x7f7c5003b3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:18.677 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:18.094+0000 7f7c66f0b700 1 --2- 172.21.15.78:0/605477584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c60108470 0x7f7c6019b9b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:18.677 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:18.094+0000 7f7c66f0b700 1 -- 172.21.15.78:0/605477584 >> 172.21.15.78:0/605477584 conn(0x7f7c60101ee0 msgr2=0x7f7c60102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:18.677 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:18.094+0000 7f7c66f0b700 1 -- 172.21.15.78:0/605477584 shutdown_connections 2024-03-24T00:11:18.677 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:18.094+0000 7f7c66f0b700 1 -- 172.21.15.78:0/605477584 wait complete. 2024-03-24T00:11:18.678 INFO:teuthology.orchestra.run.smithi078.stdout:Generating ssh key... 2024-03-24T00:11:18.968 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:18 smithi078 ceph-mon[32429]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:11:19.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:19 smithi078 ceph-mon[32429]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:11:19.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:19 smithi078 ceph-mon[32429]: from='mgr.14118 172.21.15.78:0/3702326166' entity='mgr.x' 2024-03-24T00:11:19.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:19 smithi078 ceph-mon[32429]: from='mgr.14118 172.21.15.78:0/3702326166' entity='mgr.x' 2024-03-24T00:11:19.841 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: Generating public/private rsa key pair. 2024-03-24T00:11:19.841 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: Your identification has been saved in /tmp/tmp_emz_plp/key. 2024-03-24T00:11:19.841 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: Your public key has been saved in /tmp/tmp_emz_plp/key.pub. 2024-03-24T00:11:19.841 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: The key fingerprint is: 2024-03-24T00:11:19.841 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: SHA256:UWd7cdFNT4tE/OyonY1IKS/RnhHrrFDgN9CMyBkOi38 ceph-c7e2806a-e972-11ee-95cd-87774f69a715 2024-03-24T00:11:19.841 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: The key's randomart image is: 2024-03-24T00:11:19.841 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: +---[RSA 3072]----+ 2024-03-24T00:11:19.841 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: | . . . =o.oB| 2024-03-24T00:11:19.841 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: | . = + +. o.o.+=| 2024-03-24T00:11:19.842 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: | . . = +.o ..+..| 2024-03-24T00:11:19.842 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: | . . o. . . o | 2024-03-24T00:11:19.842 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: | . E .S+. + o | 2024-03-24T00:11:19.842 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: | . oo.* . . | 2024-03-24T00:11:19.842 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: | . O * + | 2024-03-24T00:11:19.842 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: | .. O + . | 2024-03-24T00:11:19.842 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: | .o | 2024-03-24T00:11:19.842 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: +----[SHA256]-----+ 2024-03-24T00:11:20.163 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.067+0000 7f33c2ed0700 1 Processor -- start 2024-03-24T00:11:20.164 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.067+0000 7f33c2ed0700 1 -- start start 2024-03-24T00:11:20.164 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.067+0000 7f33c2ed0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33bc1084b0 0x7f33bc10a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:20.164 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.067+0000 7f33c2ed0700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33bc07ae90 con 0x7f33bc1084b0 2024-03-24T00:11:20.164 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.067+0000 7f33c0c6c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33bc1084b0 0x7f33bc10a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:20.164 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.067+0000 7f33c0c6c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33bc1084b0 0x7f33bc10a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60898/0 (socket says 172.21.15.78:60898) 2024-03-24T00:11:20.164 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.067+0000 7f33c0c6c700 1 -- 172.21.15.78:0/2444583276 learned_addr learned my addr 172.21.15.78:0/2444583276 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:20.164 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.068+0000 7f33c0c6c700 1 -- 172.21.15.78:0/2444583276 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33bc10ade0 con 0x7f33bc1084b0 2024-03-24T00:11:20.164 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.068+0000 7f33c0c6c700 1 --2- 172.21.15.78:0/2444583276 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33bc1084b0 0x7f33bc10a8a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f33b00098a0 tx=0x7f33b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eaa9f736cebfc0f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:20.164 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.068+0000 7f33bb7fe700 1 -- 172.21.15.78:0/2444583276 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f33b0004160 con 0x7f33bc1084b0 2024-03-24T00:11:20.164 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.068+0000 7f33bb7fe700 1 -- 172.21.15.78:0/2444583276 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f33b0004300 con 0x7f33bc1084b0 2024-03-24T00:11:20.164 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.068+0000 7f33bb7fe700 1 -- 172.21.15.78:0/2444583276 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f33b0011b50 con 0x7f33bc1084b0 2024-03-24T00:11:20.164 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.068+0000 7f33c2ed0700 1 -- 172.21.15.78:0/2444583276 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33bc1084b0 msgr2=0x7f33bc10a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:20.164 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.068+0000 7f33c2ed0700 1 --2- 172.21.15.78:0/2444583276 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33bc1084b0 0x7f33bc10a8a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f33b00098a0 tx=0x7f33b0009bb0 comp rx=0 tx=0).stop 2024-03-24T00:11:20.165 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.068+0000 7f33c2ed0700 1 -- 172.21.15.78:0/2444583276 shutdown_connections 2024-03-24T00:11:20.165 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.068+0000 7f33c2ed0700 1 --2- 172.21.15.78:0/2444583276 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33bc1084b0 0x7f33bc10a8a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:20.165 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.068+0000 7f33c2ed0700 1 -- 172.21.15.78:0/2444583276 >> 172.21.15.78:0/2444583276 conn(0x7f33bc101fa0 msgr2=0x7f33bc1043c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:20.165 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.069+0000 7f33c2ed0700 1 -- 172.21.15.78:0/2444583276 shutdown_connections 2024-03-24T00:11:20.165 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.069+0000 7f33c2ed0700 1 -- 172.21.15.78:0/2444583276 wait complete. 2024-03-24T00:11:20.165 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.069+0000 7f33c2ed0700 1 Processor -- start 2024-03-24T00:11:20.165 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.069+0000 7f33c2ed0700 1 -- start start 2024-03-24T00:11:20.165 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.069+0000 7f33c2ed0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33bc1084b0 0x7f33bc107d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:20.165 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.069+0000 7f33c2ed0700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33bc10be00 con 0x7f33bc1084b0 2024-03-24T00:11:20.165 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.069+0000 7f33c0c6c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33bc1084b0 0x7f33bc107d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:20.165 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.069+0000 7f33c0c6c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33bc1084b0 0x7f33bc107d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60912/0 (socket says 172.21.15.78:60912) 2024-03-24T00:11:20.165 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.069+0000 7f33c0c6c700 1 -- 172.21.15.78:0/3399276128 learned_addr learned my addr 172.21.15.78:0/3399276128 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:20.165 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.070+0000 7f33c0c6c700 1 -- 172.21.15.78:0/3399276128 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33bc108270 con 0x7f33bc1084b0 2024-03-24T00:11:20.165 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.070+0000 7f33c0c6c700 1 --2- 172.21.15.78:0/3399276128 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33bc1084b0 0x7f33bc107d30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f33b0009fb0 tx=0x7f33b0011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:20.166 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.070+0000 7f33b9ffb700 1 -- 172.21.15.78:0/3399276128 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f33b00038b0 con 0x7f33bc1084b0 2024-03-24T00:11:20.166 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.070+0000 7f33b9ffb700 1 -- 172.21.15.78:0/3399276128 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f33b0011e50 con 0x7f33bc1084b0 2024-03-24T00:11:20.166 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.070+0000 7f33c2ed0700 1 -- 172.21.15.78:0/3399276128 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33bc106460 con 0x7f33bc1084b0 2024-03-24T00:11:20.166 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.070+0000 7f33b9ffb700 1 -- 172.21.15.78:0/3399276128 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f33b0019420 con 0x7f33bc1084b0 2024-03-24T00:11:20.166 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.070+0000 7f33c2ed0700 1 -- 172.21.15.78:0/3399276128 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33bc1069a0 con 0x7f33bc1084b0 2024-03-24T00:11:20.166 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.071+0000 7f33b9ffb700 1 -- 172.21.15.78:0/3399276128 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f33b0019720 con 0x7f33bc1084b0 2024-03-24T00:11:20.166 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.071+0000 7f33c2ed0700 1 -- 172.21.15.78:0/3399276128 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33bc069220 con 0x7f33bc1084b0 2024-03-24T00:11:20.166 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.071+0000 7f33b9ffb700 1 --2- 172.21.15.78:0/3399276128 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f33ac038ed0 0x7f33ac03b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:20.166 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.071+0000 7f33b9ffb700 1 -- 172.21.15.78:0/3399276128 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f33b004d380 con 0x7f33bc1084b0 2024-03-24T00:11:20.166 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.071+0000 7f33bbfff700 1 --2- 172.21.15.78:0/3399276128 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f33ac038ed0 0x7f33ac03b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:20.166 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.072+0000 7f33bbfff700 1 --2- 172.21.15.78:0/3399276128 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f33ac038ed0 0x7f33ac03b390 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f33a8006fd0 tx=0x7f33a8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:20.166 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.076+0000 7f33b9ffb700 1 -- 172.21.15.78:0/3399276128 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f33b000ba30 con 0x7f33bc1084b0 2024-03-24T00:11:20.167 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.271+0000 7f33c2ed0700 1 -- 172.21.15.78:0/3399276128 --> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f33bc107200 con 0x7f33ac038ed0 2024-03-24T00:11:20.167 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.557+0000 7f33b9ffb700 1 -- 172.21.15.78:0/3399276128 <== mgr.14118 v2:172.21.15.78:6800/2748920178 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f33bc107200 con 0x7f33ac038ed0 2024-03-24T00:11:20.167 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.559+0000 7f33c2ed0700 1 -- 172.21.15.78:0/3399276128 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f33ac038ed0 msgr2=0x7f33ac03b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:20.167 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.559+0000 7f33c2ed0700 1 --2- 172.21.15.78:0/3399276128 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f33ac038ed0 0x7f33ac03b390 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f33a8006fd0 tx=0x7f33a8006e40 comp rx=0 tx=0).stop 2024-03-24T00:11:20.167 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.559+0000 7f33c2ed0700 1 -- 172.21.15.78:0/3399276128 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33bc1084b0 msgr2=0x7f33bc107d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:20.167 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.559+0000 7f33c2ed0700 1 --2- 172.21.15.78:0/3399276128 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33bc1084b0 0x7f33bc107d30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f33b0009fb0 tx=0x7f33b0011bf0 comp rx=0 tx=0).stop 2024-03-24T00:11:20.167 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.560+0000 7f33c2ed0700 1 -- 172.21.15.78:0/3399276128 shutdown_connections 2024-03-24T00:11:20.167 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.560+0000 7f33c2ed0700 1 --2- 172.21.15.78:0/3399276128 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f33ac038ed0 0x7f33ac03b390 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:20.167 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.560+0000 7f33c2ed0700 1 --2- 172.21.15.78:0/3399276128 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33bc1084b0 0x7f33bc107d30 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:20.167 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.560+0000 7f33c2ed0700 1 -- 172.21.15.78:0/3399276128 >> 172.21.15.78:0/3399276128 conn(0x7f33bc101fa0 msgr2=0x7f33bc102ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:20.167 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.560+0000 7f33c2ed0700 1 -- 172.21.15.78:0/3399276128 shutdown_connections 2024-03-24T00:11:20.167 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:19.560+0000 7f33c2ed0700 1 -- 172.21.15.78:0/3399276128 wait complete. 2024-03-24T00:11:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:20 smithi078 ceph-mon[32429]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:11:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:20 smithi078 ceph-mon[32429]: Generating ssh key... 2024-03-24T00:11:21.272 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQjw9fCTKvxEubrnnnAOWAiSBPw7fwWwXV81MRe//tb6Dr2lrzwC5WrQ2TB0e+DPnXDCFTJUKYIczy422eCtWu7rX03OP6LtVtem7IG9Qgdek6LP0++SpexObhyC3oRsw1p/XHGQKt4C1khhOd9QJke2jGSNXDmuPIB/AQq6kuZMGCeWsYtvANDygnQsk1vtuNMmdIq+RFnn3R1cutcHFDRU0/8wdEu3Dt9QoBJ3tag3onZxblPA26dEy8OLaMR4qGBzPv9aUyWxzA+vzMXAVYrksW4QMTnA8vK9VP9mPxCnyImtjhVj5gGFaz3dVWOTTDKrUzIC9VUAirSyo7/0hA9pmCbMbToD8eW85G11eRjMJIEe3IEXKHgKUgknBzCevF4Jh9O2OahmZ/bSik+l7TQwJqPUshMJePIyv7RA+Y1gSTLP+PTBvmMIVfZ39ivoeQ52Cf0qWSBASLH/xWJQnI7b8p+b5SAbYTM+VEB8bLc7kVYu5tZJJRV/C62RRqWAM= ceph-c7e2806a-e972-11ee-95cd-87774f69a715 2024-03-24T00:11:21.272 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.568+0000 7fa5224b4700 1 Processor -- start 2024-03-24T00:11:21.272 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.568+0000 7fa5224b4700 1 -- start start 2024-03-24T00:11:21.272 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.568+0000 7fa5224b4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa51c07cee0 0x7fa51c07b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:21.272 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.568+0000 7fa5224b4700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51c07b910 con 0x7fa51c07cee0 2024-03-24T00:11:21.272 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.568+0000 7fa51bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa51c07cee0 0x7fa51c07b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:21.273 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.568+0000 7fa51bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa51c07cee0 0x7fa51c07b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60920/0 (socket says 172.21.15.78:60920) 2024-03-24T00:11:21.273 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.568+0000 7fa51bfff700 1 -- 172.21.15.78:0/44791078 learned_addr learned my addr 172.21.15.78:0/44791078 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:21.273 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.569+0000 7fa51bfff700 1 -- 172.21.15.78:0/44791078 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa51c07ba90 con 0x7fa51c07cee0 2024-03-24T00:11:21.273 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.569+0000 7fa51bfff700 1 --2- 172.21.15.78:0/44791078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa51c07cee0 0x7fa51c07b340 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa50c0098a0 tx=0x7fa50c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d19d29e503e1b2b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:21.273 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.569+0000 7fa51affd700 1 -- 172.21.15.78:0/44791078 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa50c004160 con 0x7fa51c07cee0 2024-03-24T00:11:21.273 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.569+0000 7fa51affd700 1 -- 172.21.15.78:0/44791078 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa50c004300 con 0x7fa51c07cee0 2024-03-24T00:11:21.273 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.569+0000 7fa51affd700 1 -- 172.21.15.78:0/44791078 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa50c011b50 con 0x7fa51c07cee0 2024-03-24T00:11:21.273 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.570+0000 7fa5224b4700 1 -- 172.21.15.78:0/44791078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa51c07cee0 msgr2=0x7fa51c07b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:21.273 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.570+0000 7fa5224b4700 1 --2- 172.21.15.78:0/44791078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa51c07cee0 0x7fa51c07b340 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa50c0098a0 tx=0x7fa50c009bb0 comp rx=0 tx=0).stop 2024-03-24T00:11:21.273 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.570+0000 7fa5224b4700 1 -- 172.21.15.78:0/44791078 shutdown_connections 2024-03-24T00:11:21.273 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.570+0000 7fa5224b4700 1 --2- 172.21.15.78:0/44791078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa51c07cee0 0x7fa51c07b340 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:21.273 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.570+0000 7fa5224b4700 1 -- 172.21.15.78:0/44791078 >> 172.21.15.78:0/44791078 conn(0x7fa51c103040 msgr2=0x7fa51c105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:21.273 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.570+0000 7fa5224b4700 1 -- 172.21.15.78:0/44791078 shutdown_connections 2024-03-24T00:11:21.273 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.570+0000 7fa5224b4700 1 -- 172.21.15.78:0/44791078 wait complete. 2024-03-24T00:11:21.274 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.570+0000 7fa5224b4700 1 Processor -- start 2024-03-24T00:11:21.274 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.570+0000 7fa5224b4700 1 -- start start 2024-03-24T00:11:21.274 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.571+0000 7fa5224b4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa51c07cee0 0x7fa51c19dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:21.274 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.571+0000 7fa5224b4700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51c10dff0 con 0x7fa51c07cee0 2024-03-24T00:11:21.274 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.571+0000 7fa51bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa51c07cee0 0x7fa51c19dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:21.274 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.571+0000 7fa51bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa51c07cee0 0x7fa51c19dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60934/0 (socket says 172.21.15.78:60934) 2024-03-24T00:11:21.274 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.571+0000 7fa51bfff700 1 -- 172.21.15.78:0/837563574 learned_addr learned my addr 172.21.15.78:0/837563574 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:21.274 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.571+0000 7fa51bfff700 1 -- 172.21.15.78:0/837563574 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa51c19e160 con 0x7fa51c07cee0 2024-03-24T00:11:21.274 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.571+0000 7fa51bfff700 1 --2- 172.21.15.78:0/837563574 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa51c07cee0 0x7fa51c19dc20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa50c009fb0 tx=0x7fa50c011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:21.275 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.571+0000 7fa5197fa700 1 -- 172.21.15.78:0/837563574 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa50c0038b0 con 0x7fa51c07cee0 2024-03-24T00:11:21.275 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.571+0000 7fa5197fa700 1 -- 172.21.15.78:0/837563574 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa50c011e50 con 0x7fa51c07cee0 2024-03-24T00:11:21.275 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.571+0000 7fa5224b4700 1 -- 172.21.15.78:0/837563574 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa51c19e3f0 con 0x7fa51c07cee0 2024-03-24T00:11:21.275 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.571+0000 7fa5197fa700 1 -- 172.21.15.78:0/837563574 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa50c019550 con 0x7fa51c07cee0 2024-03-24T00:11:21.275 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.571+0000 7fa5224b4700 1 -- 172.21.15.78:0/837563574 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa51c1a10e0 con 0x7fa51c07cee0 2024-03-24T00:11:21.275 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.572+0000 7fa5197fa700 1 -- 172.21.15.78:0/837563574 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fa50c019800 con 0x7fa51c07cee0 2024-03-24T00:11:21.275 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.572+0000 7fa5197fa700 1 --2- 172.21.15.78:0/837563574 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7fa504038f20 0x7fa50403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:21.275 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.572+0000 7fa5224b4700 1 -- 172.21.15.78:0/837563574 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa51c069220 con 0x7fa51c07cee0 2024-03-24T00:11:21.275 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.573+0000 7fa5197fa700 1 -- 172.21.15.78:0/837563574 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa50c04d0a0 con 0x7fa51c07cee0 2024-03-24T00:11:21.275 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.573+0000 7fa51b7fe700 1 --2- 172.21.15.78:0/837563574 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7fa504038f20 0x7fa50403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:21.275 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.573+0000 7fa51b7fe700 1 --2- 172.21.15.78:0/837563574 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7fa504038f20 0x7fa50403b3e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa510006fd0 tx=0x7fa510006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:21.275 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.577+0000 7fa5197fa700 1 -- 172.21.15.78:0/837563574 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa50c02d960 con 0x7fa51c07cee0 2024-03-24T00:11:21.275 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.777+0000 7fa5224b4700 1 -- 172.21.15.78:0/837563574 --> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fa51c1042c0 con 0x7fa504038f20 2024-03-24T00:11:21.276 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.778+0000 7fa5197fa700 1 -- 172.21.15.78:0/837563574 <== mgr.14118 v2:172.21.15.78:6800/2748920178 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fa51c1042c0 con 0x7fa504038f20 2024-03-24T00:11:21.276 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.780+0000 7fa5224b4700 1 -- 172.21.15.78:0/837563574 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7fa504038f20 msgr2=0x7fa50403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:21.276 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.780+0000 7fa5224b4700 1 --2- 172.21.15.78:0/837563574 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7fa504038f20 0x7fa50403b3e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa510006fd0 tx=0x7fa510006e40 comp rx=0 tx=0).stop 2024-03-24T00:11:21.276 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.780+0000 7fa5224b4700 1 -- 172.21.15.78:0/837563574 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa51c07cee0 msgr2=0x7fa51c19dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:21.276 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.780+0000 7fa5224b4700 1 --2- 172.21.15.78:0/837563574 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa51c07cee0 0x7fa51c19dc20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa50c009fb0 tx=0x7fa50c011bf0 comp rx=0 tx=0).stop 2024-03-24T00:11:21.276 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.780+0000 7fa5224b4700 1 -- 172.21.15.78:0/837563574 shutdown_connections 2024-03-24T00:11:21.276 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.780+0000 7fa5224b4700 1 --2- 172.21.15.78:0/837563574 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7fa504038f20 0x7fa50403b3e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:21.276 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.780+0000 7fa5224b4700 1 --2- 172.21.15.78:0/837563574 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa51c07cee0 0x7fa51c19dc20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:21.276 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.780+0000 7fa5224b4700 1 -- 172.21.15.78:0/837563574 >> 172.21.15.78:0/837563574 conn(0x7fa51c103040 msgr2=0x7fa51c103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:21.276 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.780+0000 7fa5224b4700 1 -- 172.21.15.78:0/837563574 shutdown_connections 2024-03-24T00:11:21.276 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:20.780+0000 7fa5224b4700 1 -- 172.21.15.78:0/837563574 wait complete. 2024-03-24T00:11:21.276 INFO:teuthology.orchestra.run.smithi078.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-24T00:11:21.277 INFO:teuthology.orchestra.run.smithi078.stdout:Adding key to root@localhost authorized_keys... 2024-03-24T00:11:21.277 INFO:teuthology.orchestra.run.smithi078.stdout:Adding host smithi078... 2024-03-24T00:11:21.930 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:21 smithi078 ceph-mon[32429]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:11:22.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:22 smithi078 ceph-mon[32429]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi078", "addr": "172.21.15.78", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:11:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:23 smithi078 ceph-mon[32429]: Deploying cephadm binary to smithi078 2024-03-24T00:11:26.649 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout Added host 'smithi078' with addr '172.21.15.78' 2024-03-24T00:11:26.649 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.647+0000 7f8e1f76c700 1 Processor -- start 2024-03-24T00:11:26.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.647+0000 7f8e1f76c700 1 -- start start 2024-03-24T00:11:26.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.647+0000 7f8e1f76c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e18106270 0x7f8e18106690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:26.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.648+0000 7f8e1f76c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1807ae90 con 0x7f8e18106270 2024-03-24T00:11:26.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.648+0000 7f8e1d508700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e18106270 0x7f8e18106690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:26.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.648+0000 7f8e1d508700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e18106270 0x7f8e18106690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60946/0 (socket says 172.21.15.78:60946) 2024-03-24T00:11:26.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.648+0000 7f8e1d508700 1 -- 172.21.15.78:0/3140383342 learned_addr learned my addr 172.21.15.78:0/3140383342 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:26.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.648+0000 7f8e1d508700 1 -- 172.21.15.78:0/3140383342 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e18106bd0 con 0x7f8e18106270 2024-03-24T00:11:26.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.648+0000 7f8e1d508700 1 --2- 172.21.15.78:0/3140383342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e18106270 0x7f8e18106690 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8e140098a0 tx=0x7f8e14009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ac5e353a0881fcf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:26.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.648+0000 7f8e0ffff700 1 -- 172.21.15.78:0/3140383342 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e14004160 con 0x7f8e18106270 2024-03-24T00:11:26.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.648+0000 7f8e0ffff700 1 -- 172.21.15.78:0/3140383342 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8e14004300 con 0x7f8e18106270 2024-03-24T00:11:26.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.648+0000 7f8e0ffff700 1 -- 172.21.15.78:0/3140383342 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e14011b00 con 0x7f8e18106270 2024-03-24T00:11:26.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.649+0000 7f8e1f76c700 1 -- 172.21.15.78:0/3140383342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e18106270 msgr2=0x7f8e18106690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:26.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.649+0000 7f8e1f76c700 1 --2- 172.21.15.78:0/3140383342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e18106270 0x7f8e18106690 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8e140098a0 tx=0x7f8e14009bb0 comp rx=0 tx=0).stop 2024-03-24T00:11:26.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.649+0000 7f8e1f76c700 1 -- 172.21.15.78:0/3140383342 shutdown_connections 2024-03-24T00:11:26.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.649+0000 7f8e1f76c700 1 --2- 172.21.15.78:0/3140383342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e18106270 0x7f8e18106690 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:26.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.649+0000 7f8e1f76c700 1 -- 172.21.15.78:0/3140383342 >> 172.21.15.78:0/3140383342 conn(0x7f8e18101f10 msgr2=0x7f8e18104330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:26.653 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.649+0000 7f8e1f76c700 1 -- 172.21.15.78:0/3140383342 shutdown_connections 2024-03-24T00:11:26.653 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.649+0000 7f8e1f76c700 1 -- 172.21.15.78:0/3140383342 wait complete. 2024-03-24T00:11:26.653 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.650+0000 7f8e1f76c700 1 Processor -- start 2024-03-24T00:11:26.653 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.650+0000 7f8e1f76c700 1 -- start start 2024-03-24T00:11:26.653 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.650+0000 7f8e1f76c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e1819ddb0 0x7f8e1819e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:26.653 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.650+0000 7f8e1f76c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e18107c60 con 0x7f8e1819ddb0 2024-03-24T00:11:26.654 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.650+0000 7f8e1d508700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e1819ddb0 0x7f8e1819e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:26.654 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.650+0000 7f8e1d508700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e1819ddb0 0x7f8e1819e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60948/0 (socket says 172.21.15.78:60948) 2024-03-24T00:11:26.654 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.650+0000 7f8e1d508700 1 -- 172.21.15.78:0/1939140465 learned_addr learned my addr 172.21.15.78:0/1939140465 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:26.654 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.650+0000 7f8e1d508700 1 -- 172.21.15.78:0/1939140465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e1819e710 con 0x7f8e1819ddb0 2024-03-24T00:11:26.654 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.650+0000 7f8e1d508700 1 --2- 172.21.15.78:0/1939140465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e1819ddb0 0x7f8e1819e1d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8e14009870 tx=0x7f8e14011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:26.654 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.651+0000 7f8e0e7fc700 1 -- 172.21.15.78:0/1939140465 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e140038b0 con 0x7f8e1819ddb0 2024-03-24T00:11:26.654 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.651+0000 7f8e0e7fc700 1 -- 172.21.15.78:0/1939140465 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8e14003a50 con 0x7f8e1819ddb0 2024-03-24T00:11:26.655 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.651+0000 7f8e1f76c700 1 -- 172.21.15.78:0/1939140465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e1819e9a0 con 0x7f8e1819ddb0 2024-03-24T00:11:26.655 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.651+0000 7f8e0e7fc700 1 -- 172.21.15.78:0/1939140465 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e1401b5f0 con 0x7f8e1819ddb0 2024-03-24T00:11:26.655 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.651+0000 7f8e1f76c700 1 -- 172.21.15.78:0/1939140465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e181a1480 con 0x7f8e1819ddb0 2024-03-24T00:11:26.655 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.651+0000 7f8e0e7fc700 1 -- 172.21.15.78:0/1939140465 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f8e1401b820 con 0x7f8e1819ddb0 2024-03-24T00:11:26.655 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.652+0000 7f8e1f76c700 1 -- 172.21.15.78:0/1939140465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e180637c0 con 0x7f8e1819ddb0 2024-03-24T00:11:26.655 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.652+0000 7f8e0e7fc700 1 --2- 172.21.15.78:0/1939140465 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f8e04038f20 0x7f8e0403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:26.655 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.652+0000 7f8e0e7fc700 1 -- 172.21.15.78:0/1939140465 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f8e1404d5a0 con 0x7f8e1819ddb0 2024-03-24T00:11:26.655 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.652+0000 7f8e1cd07700 1 --2- 172.21.15.78:0/1939140465 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f8e04038f20 0x7f8e0403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:26.655 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.652+0000 7f8e1cd07700 1 --2- 172.21.15.78:0/1939140465 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f8e04038f20 0x7f8e0403b3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8e08006fd0 tx=0x7f8e08006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:26.655 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.657+0000 7f8e0e7fc700 1 -- 172.21.15.78:0/1939140465 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8e14003c00 con 0x7f8e1819ddb0 2024-03-24T00:11:26.655 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:21.860+0000 7f8e1f76c700 1 -- 172.21.15.78:0/1939140465 --> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi078", "addr": "172.21.15.78", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e180625d0 con 0x7f8e04038f20 2024-03-24T00:11:26.656 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:26.107+0000 7f8e0e7fc700 1 -- 172.21.15.78:0/1939140465 <== mgr.14118 v2:172.21.15.78:6800/2748920178 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f8e180625d0 con 0x7f8e04038f20 2024-03-24T00:11:26.656 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:26.110+0000 7f8e1f76c700 1 -- 172.21.15.78:0/1939140465 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f8e04038f20 msgr2=0x7f8e0403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:26.656 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:26.110+0000 7f8e1f76c700 1 --2- 172.21.15.78:0/1939140465 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f8e04038f20 0x7f8e0403b3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8e08006fd0 tx=0x7f8e08006e40 comp rx=0 tx=0).stop 2024-03-24T00:11:26.656 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:26.110+0000 7f8e1f76c700 1 -- 172.21.15.78:0/1939140465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e1819ddb0 msgr2=0x7f8e1819e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:26.656 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:26.110+0000 7f8e1f76c700 1 --2- 172.21.15.78:0/1939140465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e1819ddb0 0x7f8e1819e1d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8e14009870 tx=0x7f8e14011c10 comp rx=0 tx=0).stop 2024-03-24T00:11:26.656 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:26.110+0000 7f8e1f76c700 1 -- 172.21.15.78:0/1939140465 shutdown_connections 2024-03-24T00:11:26.656 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:26.110+0000 7f8e1f76c700 1 --2- 172.21.15.78:0/1939140465 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f8e04038f20 0x7f8e0403b3e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:26.656 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:26.110+0000 7f8e1f76c700 1 --2- 172.21.15.78:0/1939140465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e1819ddb0 0x7f8e1819e1d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:26.656 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:26.110+0000 7f8e1f76c700 1 -- 172.21.15.78:0/1939140465 >> 172.21.15.78:0/1939140465 conn(0x7f8e18101f10 msgr2=0x7f8e18194910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:26.656 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:26.110+0000 7f8e1f76c700 1 -- 172.21.15.78:0/1939140465 shutdown_connections 2024-03-24T00:11:26.656 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:26.110+0000 7f8e1f76c700 1 -- 172.21.15.78:0/1939140465 wait complete. 2024-03-24T00:11:26.657 INFO:teuthology.orchestra.run.smithi078.stdout:Deploying unmanaged mon service... 2024-03-24T00:11:27.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:27 smithi078 ceph-mon[32429]: from='mgr.14118 172.21.15.78:0/3702326166' entity='mgr.x' 2024-03-24T00:11:27.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:27 smithi078 ceph-mon[32429]: Added host smithi078 2024-03-24T00:11:27.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:27 smithi078 ceph-mon[32429]: from='mgr.14118 172.21.15.78:0/3702326166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:11:27.925 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-24T00:11:27.926 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.089+0000 7f14a7d41700 1 Processor -- start 2024-03-24T00:11:27.926 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.090+0000 7f14a7d41700 1 -- start start 2024-03-24T00:11:27.926 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.090+0000 7f14a7d41700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14a0105c80 0x7f14a01060a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:27.926 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.090+0000 7f14a7d41700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a0106670 con 0x7f14a0105c80 2024-03-24T00:11:27.926 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.090+0000 7f14a5add700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14a0105c80 0x7f14a01060a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:27.926 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.090+0000 7f14a5add700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14a0105c80 0x7f14a01060a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52552/0 (socket says 172.21.15.78:52552) 2024-03-24T00:11:27.926 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.090+0000 7f14a5add700 1 -- 172.21.15.78:0/1196798231 learned_addr learned my addr 172.21.15.78:0/1196798231 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:27.926 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.090+0000 7f14a5add700 1 -- 172.21.15.78:0/1196798231 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a01067f0 con 0x7f14a0105c80 2024-03-24T00:11:27.926 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.090+0000 7f14a5add700 1 --2- 172.21.15.78:0/1196798231 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14a0105c80 0x7f14a01060a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f14900098a0 tx=0x7f1490009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e31098daa107b9b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:27.926 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.091+0000 7f14a4adb700 1 -- 172.21.15.78:0/1196798231 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1490004160 con 0x7f14a0105c80 2024-03-24T00:11:27.926 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.091+0000 7f14a4adb700 1 -- 172.21.15.78:0/1196798231 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1490004300 con 0x7f14a0105c80 2024-03-24T00:11:27.927 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.091+0000 7f14a4adb700 1 -- 172.21.15.78:0/1196798231 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1490011b50 con 0x7f14a0105c80 2024-03-24T00:11:27.927 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.091+0000 7f14a7d41700 1 -- 172.21.15.78:0/1196798231 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14a0105c80 msgr2=0x7f14a01060a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:27.927 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.091+0000 7f14a7d41700 1 --2- 172.21.15.78:0/1196798231 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14a0105c80 0x7f14a01060a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f14900098a0 tx=0x7f1490009bb0 comp rx=0 tx=0).stop 2024-03-24T00:11:27.927 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.091+0000 7f14a7d41700 1 -- 172.21.15.78:0/1196798231 shutdown_connections 2024-03-24T00:11:27.927 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.091+0000 7f14a7d41700 1 --2- 172.21.15.78:0/1196798231 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14a0105c80 0x7f14a01060a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:27.927 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.091+0000 7f14a7d41700 1 -- 172.21.15.78:0/1196798231 >> 172.21.15.78:0/1196798231 conn(0x7f14a01016a0 msgr2=0x7f14a0103ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:27.927 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.091+0000 7f14a7d41700 1 -- 172.21.15.78:0/1196798231 shutdown_connections 2024-03-24T00:11:27.927 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.091+0000 7f14a7d41700 1 -- 172.21.15.78:0/1196798231 wait complete. 2024-03-24T00:11:27.927 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.092+0000 7f14a7d41700 1 Processor -- start 2024-03-24T00:11:27.927 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.092+0000 7f14a7d41700 1 -- start start 2024-03-24T00:11:27.927 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.092+0000 7f14a7d41700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14a0105c80 0x7f14a0108580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:27.927 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.092+0000 7f14a7d41700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a010bf30 con 0x7f14a0105c80 2024-03-24T00:11:27.927 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.092+0000 7f14a5add700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14a0105c80 0x7f14a0108580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:27.927 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.092+0000 7f14a5add700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14a0105c80 0x7f14a0108580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52568/0 (socket says 172.21.15.78:52568) 2024-03-24T00:11:27.928 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.092+0000 7f14a5add700 1 -- 172.21.15.78:0/2940939520 learned_addr learned my addr 172.21.15.78:0/2940939520 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:27.928 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.092+0000 7f14a5add700 1 -- 172.21.15.78:0/2940939520 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a0108ac0 con 0x7f14a0105c80 2024-03-24T00:11:27.928 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.092+0000 7f14a5add700 1 --2- 172.21.15.78:0/2940939520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14a0105c80 0x7f14a0108580 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1490011850 tx=0x7f1490011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:27.928 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.093+0000 7f1496ffd700 1 -- 172.21.15.78:0/2940939520 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1490003af0 con 0x7f14a0105c80 2024-03-24T00:11:27.928 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.093+0000 7f1496ffd700 1 -- 172.21.15.78:0/2940939520 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1490020070 con 0x7f14a0105c80 2024-03-24T00:11:27.928 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.093+0000 7f1496ffd700 1 -- 172.21.15.78:0/2940939520 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f14900195a0 con 0x7f14a0105c80 2024-03-24T00:11:27.928 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.093+0000 7f14a7d41700 1 -- 172.21.15.78:0/2940939520 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14a0106cb0 con 0x7f14a0105c80 2024-03-24T00:11:27.928 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.093+0000 7f14a7d41700 1 -- 172.21.15.78:0/2940939520 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14a01071f0 con 0x7f14a0105c80 2024-03-24T00:11:27.928 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.093+0000 7f14a7d41700 1 -- 172.21.15.78:0/2940939520 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14a00637c0 con 0x7f14a0105c80 2024-03-24T00:11:27.928 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.093+0000 7f1496ffd700 1 -- 172.21.15.78:0/2940939520 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f1490019850 con 0x7f14a0105c80 2024-03-24T00:11:27.928 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.094+0000 7f1496ffd700 1 --2- 172.21.15.78:0/2940939520 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f148c038f70 0x7f148c03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:27.928 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.094+0000 7f1496ffd700 1 -- 172.21.15.78:0/2940939520 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f1490011e00 con 0x7f14a0105c80 2024-03-24T00:11:27.928 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.094+0000 7f14a52dc700 1 --2- 172.21.15.78:0/2940939520 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f148c038f70 0x7f148c03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:27.928 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.094+0000 7f14a52dc700 1 --2- 172.21.15.78:0/2940939520 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f148c038f70 0x7f148c03b430 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f149c006fd0 tx=0x7f149c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:27.929 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.099+0000 7f1496ffd700 1 -- 172.21.15.78:0/2940939520 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1490003c90 con 0x7f14a0105c80 2024-03-24T00:11:27.929 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.346+0000 7f14a7d41700 1 -- 172.21.15.78:0/2940939520 --> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f14a0102990 con 0x7f148c038f70 2024-03-24T00:11:27.929 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.353+0000 7f1496ffd700 1 -- 172.21.15.78:0/2940939520 <== mgr.14118 v2:172.21.15.78:6800/2748920178 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f14a0102990 con 0x7f148c038f70 2024-03-24T00:11:27.929 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.355+0000 7f14a7d41700 1 -- 172.21.15.78:0/2940939520 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f148c038f70 msgr2=0x7f148c03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:27.929 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.355+0000 7f14a7d41700 1 --2- 172.21.15.78:0/2940939520 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f148c038f70 0x7f148c03b430 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f149c006fd0 tx=0x7f149c006e40 comp rx=0 tx=0).stop 2024-03-24T00:11:27.929 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.355+0000 7f14a7d41700 1 -- 172.21.15.78:0/2940939520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14a0105c80 msgr2=0x7f14a0108580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:27.929 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.355+0000 7f14a7d41700 1 --2- 172.21.15.78:0/2940939520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14a0105c80 0x7f14a0108580 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1490011850 tx=0x7f1490011c10 comp rx=0 tx=0).stop 2024-03-24T00:11:27.929 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.355+0000 7f14a7d41700 1 -- 172.21.15.78:0/2940939520 shutdown_connections 2024-03-24T00:11:27.929 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.355+0000 7f14a7d41700 1 --2- 172.21.15.78:0/2940939520 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f148c038f70 0x7f148c03b430 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:27.929 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.355+0000 7f14a7d41700 1 --2- 172.21.15.78:0/2940939520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14a0105c80 0x7f14a0108580 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:27.929 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.355+0000 7f14a7d41700 1 -- 172.21.15.78:0/2940939520 >> 172.21.15.78:0/2940939520 conn(0x7f14a01016a0 msgr2=0x7f14a0102280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:27.929 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.355+0000 7f14a7d41700 1 -- 172.21.15.78:0/2940939520 shutdown_connections 2024-03-24T00:11:27.929 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:27.355+0000 7f14a7d41700 1 -- 172.21.15.78:0/2940939520 wait complete. 2024-03-24T00:11:27.929 INFO:teuthology.orchestra.run.smithi078.stdout:Deploying unmanaged mgr service... 2024-03-24T00:11:28.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:28 smithi078 ceph-mon[32429]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:11:28.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:28 smithi078 ceph-mon[32429]: Saving service mon spec with placement count:5 2024-03-24T00:11:28.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:28 smithi078 ceph-mon[32429]: from='mgr.14118 172.21.15.78:0/3702326166' entity='mgr.x' 2024-03-24T00:11:29.117 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-24T00:11:29.117 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.356+0000 7f5831dac700 1 Processor -- start 2024-03-24T00:11:29.118 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.356+0000 7f5831dac700 1 -- start start 2024-03-24T00:11:29.118 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.356+0000 7f5831dac700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f582c108480 0x7f582c10a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:29.118 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.356+0000 7f5831dac700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f582c07ae90 con 0x7f582c108480 2024-03-24T00:11:29.118 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.356+0000 7f582b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f582c108480 0x7f582c10a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:29.118 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.356+0000 7f582b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f582c108480 0x7f582c10a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52570/0 (socket says 172.21.15.78:52570) 2024-03-24T00:11:29.118 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.356+0000 7f582b7fe700 1 -- 172.21.15.78:0/1131507072 learned_addr learned my addr 172.21.15.78:0/1131507072 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:29.118 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.357+0000 7f582b7fe700 1 -- 172.21.15.78:0/1131507072 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f582c10adb0 con 0x7f582c108480 2024-03-24T00:11:29.118 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.357+0000 7f582b7fe700 1 --2- 172.21.15.78:0/1131507072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f582c108480 0x7f582c10a870 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f58200098a0 tx=0x7f5820009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=967b5f72218352da server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:29.118 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.357+0000 7f582a7fc700 1 -- 172.21.15.78:0/1131507072 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5820004160 con 0x7f582c108480 2024-03-24T00:11:29.118 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.357+0000 7f582a7fc700 1 -- 172.21.15.78:0/1131507072 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5820004300 con 0x7f582c108480 2024-03-24T00:11:29.118 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.357+0000 7f582a7fc700 1 -- 172.21.15.78:0/1131507072 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5820011b50 con 0x7f582c108480 2024-03-24T00:11:29.118 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.358+0000 7f5831dac700 1 -- 172.21.15.78:0/1131507072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f582c108480 msgr2=0x7f582c10a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:29.118 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.358+0000 7f5831dac700 1 --2- 172.21.15.78:0/1131507072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f582c108480 0x7f582c10a870 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f58200098a0 tx=0x7f5820009bb0 comp rx=0 tx=0).stop 2024-03-24T00:11:29.119 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.358+0000 7f5831dac700 1 -- 172.21.15.78:0/1131507072 shutdown_connections 2024-03-24T00:11:29.119 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.358+0000 7f5831dac700 1 --2- 172.21.15.78:0/1131507072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f582c108480 0x7f582c10a870 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:29.119 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.358+0000 7f5831dac700 1 -- 172.21.15.78:0/1131507072 >> 172.21.15.78:0/1131507072 conn(0x7f582c101f10 msgr2=0x7f582c104330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:29.119 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.358+0000 7f5831dac700 1 -- 172.21.15.78:0/1131507072 shutdown_connections 2024-03-24T00:11:29.119 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.358+0000 7f5831dac700 1 -- 172.21.15.78:0/1131507072 wait complete. 2024-03-24T00:11:29.119 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.358+0000 7f5831dac700 1 Processor -- start 2024-03-24T00:11:29.119 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.358+0000 7f5831dac700 1 -- start start 2024-03-24T00:11:29.119 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.359+0000 7f5831dac700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f582c108480 0x7f582c107d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:29.119 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.359+0000 7f5831dac700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f582c10bdd0 con 0x7f582c108480 2024-03-24T00:11:29.119 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.359+0000 7f582b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f582c108480 0x7f582c107d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:29.119 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.359+0000 7f582b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f582c108480 0x7f582c107d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52584/0 (socket says 172.21.15.78:52584) 2024-03-24T00:11:29.119 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.359+0000 7f582b7fe700 1 -- 172.21.15.78:0/2562609456 learned_addr learned my addr 172.21.15.78:0/2562609456 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:29.119 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.359+0000 7f582b7fe700 1 -- 172.21.15.78:0/2562609456 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f582c108240 con 0x7f582c108480 2024-03-24T00:11:29.119 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.359+0000 7f582b7fe700 1 --2- 172.21.15.78:0/2562609456 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f582c108480 0x7f582c107d00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5820011b60 tx=0x7f5820011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:29.120 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.359+0000 7f5828ff9700 1 -- 172.21.15.78:0/2562609456 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5820003970 con 0x7f582c108480 2024-03-24T00:11:29.120 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.359+0000 7f5828ff9700 1 -- 172.21.15.78:0/2562609456 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5820003b10 con 0x7f582c108480 2024-03-24T00:11:29.120 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.359+0000 7f5831dac700 1 -- 172.21.15.78:0/2562609456 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f582c106430 con 0x7f582c108480 2024-03-24T00:11:29.120 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.359+0000 7f5828ff9700 1 -- 172.21.15.78:0/2562609456 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5820024610 con 0x7f582c108480 2024-03-24T00:11:29.120 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.359+0000 7f5831dac700 1 -- 172.21.15.78:0/2562609456 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f582c106970 con 0x7f582c108480 2024-03-24T00:11:29.120 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.360+0000 7f5828ff9700 1 -- 172.21.15.78:0/2562609456 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f58200248c0 con 0x7f582c108480 2024-03-24T00:11:29.120 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.361+0000 7f5828ff9700 1 --2- 172.21.15.78:0/2562609456 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f5814038f70 0x7f581403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:29.120 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.361+0000 7f5831dac700 1 -- 172.21.15.78:0/2562609456 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f582c0637c0 con 0x7f582c108480 2024-03-24T00:11:29.120 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.361+0000 7f5828ff9700 1 -- 172.21.15.78:0/2562609456 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f5820011e00 con 0x7f582c108480 2024-03-24T00:11:29.120 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.361+0000 7f582affd700 1 --2- 172.21.15.78:0/2562609456 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f5814038f70 0x7f581403b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:29.120 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.361+0000 7f582affd700 1 --2- 172.21.15.78:0/2562609456 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f5814038f70 0x7f581403b430 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f581c006fd0 tx=0x7f581c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:29.120 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.366+0000 7f5828ff9700 1 -- 172.21.15.78:0/2562609456 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f582000ba00 con 0x7f582c108480 2024-03-24T00:11:29.120 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.584+0000 7f5831dac700 1 -- 172.21.15.78:0/2562609456 --> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f582c1031c0 con 0x7f5814038f70 2024-03-24T00:11:29.120 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.591+0000 7f5828ff9700 1 -- 172.21.15.78:0/2562609456 <== mgr.14118 v2:172.21.15.78:6800/2748920178 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f582c1031c0 con 0x7f5814038f70 2024-03-24T00:11:29.121 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.593+0000 7f5831dac700 1 -- 172.21.15.78:0/2562609456 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f5814038f70 msgr2=0x7f581403b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:29.121 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.593+0000 7f5831dac700 1 --2- 172.21.15.78:0/2562609456 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f5814038f70 0x7f581403b430 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f581c006fd0 tx=0x7f581c006e40 comp rx=0 tx=0).stop 2024-03-24T00:11:29.121 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.593+0000 7f5831dac700 1 -- 172.21.15.78:0/2562609456 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f582c108480 msgr2=0x7f582c107d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:29.121 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.593+0000 7f5831dac700 1 --2- 172.21.15.78:0/2562609456 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f582c108480 0x7f582c107d00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5820011b60 tx=0x7f5820011c40 comp rx=0 tx=0).stop 2024-03-24T00:11:29.121 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.593+0000 7f5831dac700 1 -- 172.21.15.78:0/2562609456 shutdown_connections 2024-03-24T00:11:29.121 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.593+0000 7f5831dac700 1 --2- 172.21.15.78:0/2562609456 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f5814038f70 0x7f581403b430 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:29.121 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.593+0000 7f5831dac700 1 --2- 172.21.15.78:0/2562609456 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f582c108480 0x7f582c107d00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:29.121 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.593+0000 7f5831dac700 1 -- 172.21.15.78:0/2562609456 >> 172.21.15.78:0/2562609456 conn(0x7f582c101f10 msgr2=0x7f582c102ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:29.121 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.594+0000 7f5831dac700 1 -- 172.21.15.78:0/2562609456 shutdown_connections 2024-03-24T00:11:29.121 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:28.594+0000 7f5831dac700 1 -- 172.21.15.78:0/2562609456 wait complete. 2024-03-24T00:11:29.689 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:29 smithi078 ceph-mon[32429]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:11:29.689 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:29 smithi078 ceph-mon[32429]: Saving service mgr spec with placement count:2 2024-03-24T00:11:29.689 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:29 smithi078 ceph-mon[32429]: from='mgr.14118 172.21.15.78:0/3702326166' entity='mgr.x' 2024-03-24T00:11:30.669 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.792+0000 7fa1a93c0700 1 Processor -- start 2024-03-24T00:11:30.670 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.792+0000 7fa1a93c0700 1 -- start start 2024-03-24T00:11:30.670 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.792+0000 7fa1a93c0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1a41062b0 0x7fa1a41066d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:30.670 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.792+0000 7fa1a93c0700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a407ae30 con 0x7fa1a41062b0 2024-03-24T00:11:30.670 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.793+0000 7fa1a2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1a41062b0 0x7fa1a41066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:30.670 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.793+0000 7fa1a2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1a41062b0 0x7fa1a41066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52600/0 (socket says 172.21.15.78:52600) 2024-03-24T00:11:30.670 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.793+0000 7fa1a2ffd700 1 -- 172.21.15.78:0/809891692 learned_addr learned my addr 172.21.15.78:0/809891692 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:30.670 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.793+0000 7fa1a2ffd700 1 -- 172.21.15.78:0/809891692 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a4106c10 con 0x7fa1a41062b0 2024-03-24T00:11:30.670 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.793+0000 7fa1a2ffd700 1 --2- 172.21.15.78:0/809891692 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1a41062b0 0x7fa1a41066d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa1940098a0 tx=0x7fa194009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2463015dae51e626 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:30.670 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.793+0000 7fa1a27fc700 1 -- 172.21.15.78:0/809891692 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa194004160 con 0x7fa1a41062b0 2024-03-24T00:11:30.670 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.793+0000 7fa1a27fc700 1 -- 172.21.15.78:0/809891692 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa194004300 con 0x7fa1a41062b0 2024-03-24T00:11:30.670 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.793+0000 7fa1a27fc700 1 -- 172.21.15.78:0/809891692 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa194011b50 con 0x7fa1a41062b0 2024-03-24T00:11:30.670 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.794+0000 7fa1a93c0700 1 -- 172.21.15.78:0/809891692 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1a41062b0 msgr2=0x7fa1a41066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:30.671 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.794+0000 7fa1a93c0700 1 --2- 172.21.15.78:0/809891692 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1a41062b0 0x7fa1a41066d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa1940098a0 tx=0x7fa194009bb0 comp rx=0 tx=0).stop 2024-03-24T00:11:30.671 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.794+0000 7fa1a93c0700 1 -- 172.21.15.78:0/809891692 shutdown_connections 2024-03-24T00:11:30.671 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.794+0000 7fa1a93c0700 1 --2- 172.21.15.78:0/809891692 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1a41062b0 0x7fa1a41066d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:30.671 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.794+0000 7fa1a93c0700 1 -- 172.21.15.78:0/809891692 >> 172.21.15.78:0/809891692 conn(0x7fa1a4101f10 msgr2=0x7fa1a4104330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:30.671 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.794+0000 7fa1a93c0700 1 -- 172.21.15.78:0/809891692 shutdown_connections 2024-03-24T00:11:30.671 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.794+0000 7fa1a93c0700 1 -- 172.21.15.78:0/809891692 wait complete. 2024-03-24T00:11:30.671 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.794+0000 7fa1a93c0700 1 Processor -- start 2024-03-24T00:11:30.671 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.795+0000 7fa1a93c0700 1 -- start start 2024-03-24T00:11:30.671 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.795+0000 7fa1a93c0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1a41062b0 0x7fa1a419db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:30.671 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.795+0000 7fa1a93c0700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a4107ca0 con 0x7fa1a41062b0 2024-03-24T00:11:30.671 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.795+0000 7fa1a2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1a41062b0 0x7fa1a419db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:30.671 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.795+0000 7fa1a2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1a41062b0 0x7fa1a419db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52604/0 (socket says 172.21.15.78:52604) 2024-03-24T00:11:30.671 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.795+0000 7fa1a2ffd700 1 -- 172.21.15.78:0/3360988155 learned_addr learned my addr 172.21.15.78:0/3360988155 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:30.672 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.795+0000 7fa1a2ffd700 1 -- 172.21.15.78:0/3360988155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a419e090 con 0x7fa1a41062b0 2024-03-24T00:11:30.672 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.795+0000 7fa1a2ffd700 1 --2- 172.21.15.78:0/3360988155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1a41062b0 0x7fa1a419db50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa194011b60 tx=0x7fa194011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:30.672 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.795+0000 7fa1a0ff9700 1 -- 172.21.15.78:0/3360988155 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa194003b20 con 0x7fa1a41062b0 2024-03-24T00:11:30.672 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.795+0000 7fa1a0ff9700 1 -- 172.21.15.78:0/3360988155 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa194003cc0 con 0x7fa1a41062b0 2024-03-24T00:11:30.672 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.795+0000 7fa1a0ff9700 1 -- 172.21.15.78:0/3360988155 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa194024840 con 0x7fa1a41062b0 2024-03-24T00:11:30.672 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.795+0000 7fa1a93c0700 1 -- 172.21.15.78:0/3360988155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1a419e320 con 0x7fa1a41062b0 2024-03-24T00:11:30.672 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.796+0000 7fa1a93c0700 1 -- 172.21.15.78:0/3360988155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a41a1010 con 0x7fa1a41062b0 2024-03-24T00:11:30.672 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.796+0000 7fa1a0ff9700 1 -- 172.21.15.78:0/3360988155 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fa194022020 con 0x7fa1a41062b0 2024-03-24T00:11:30.672 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.797+0000 7fa1a93c0700 1 -- 172.21.15.78:0/3360988155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1a4069220 con 0x7fa1a41062b0 2024-03-24T00:11:30.672 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.797+0000 7fa1a0ff9700 1 --2- 172.21.15.78:0/3360988155 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7fa190038f70 0x7fa19003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:30.672 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.797+0000 7fa1a0ff9700 1 -- 172.21.15.78:0/3360988155 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa194011e00 con 0x7fa1a41062b0 2024-03-24T00:11:30.672 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.797+0000 7fa19a7fc700 1 --2- 172.21.15.78:0/3360988155 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7fa190038f70 0x7fa19003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:30.672 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.797+0000 7fa19a7fc700 1 --2- 172.21.15.78:0/3360988155 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7fa190038f70 0x7fa19003b430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa18c006fd0 tx=0x7fa18c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:30.673 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:29.802+0000 7fa1a0ff9700 1 -- 172.21.15.78:0/3360988155 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa19400ba30 con 0x7fa1a41062b0 2024-03-24T00:11:30.673 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:30.001+0000 7fa1a93c0700 1 -- 172.21.15.78:0/3360988155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fa1a40008d0 con 0x7fa1a41062b0 2024-03-24T00:11:30.673 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:30.006+0000 7fa1a0ff9700 1 -- 172.21.15.78:0/3360988155 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v6)=0 v6) v1 ==== 142+0+0 (secure 0 0 0) 0x7fa19400bc90 con 0x7fa1a41062b0 2024-03-24T00:11:30.673 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:30.008+0000 7fa1a93c0700 1 -- 172.21.15.78:0/3360988155 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7fa190038f70 msgr2=0x7fa19003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:30.673 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:30.008+0000 7fa1a93c0700 1 --2- 172.21.15.78:0/3360988155 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7fa190038f70 0x7fa19003b430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa18c006fd0 tx=0x7fa18c006e40 comp rx=0 tx=0).stop 2024-03-24T00:11:30.673 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:30.008+0000 7fa1a93c0700 1 -- 172.21.15.78:0/3360988155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1a41062b0 msgr2=0x7fa1a419db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:30.673 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:30.008+0000 7fa1a93c0700 1 --2- 172.21.15.78:0/3360988155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1a41062b0 0x7fa1a419db50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa194011b60 tx=0x7fa194011c40 comp rx=0 tx=0).stop 2024-03-24T00:11:30.673 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:30.008+0000 7fa1a93c0700 1 -- 172.21.15.78:0/3360988155 shutdown_connections 2024-03-24T00:11:30.673 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:30.008+0000 7fa1a93c0700 1 --2- 172.21.15.78:0/3360988155 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7fa190038f70 0x7fa19003b430 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:30.673 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:30.008+0000 7fa1a93c0700 1 --2- 172.21.15.78:0/3360988155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1a41062b0 0x7fa1a419db50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:30.673 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:30.008+0000 7fa1a93c0700 1 -- 172.21.15.78:0/3360988155 >> 172.21.15.78:0/3360988155 conn(0x7fa1a4101f10 msgr2=0x7fa1a4194770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:30.673 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:30.009+0000 7fa1a93c0700 1 -- 172.21.15.78:0/3360988155 shutdown_connections 2024-03-24T00:11:30.673 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:30.009+0000 7fa1a93c0700 1 -- 172.21.15.78:0/3360988155 wait complete. 2024-03-24T00:11:31.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:31 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3360988155' entity='client.admin' 2024-03-24T00:11:32.036 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.108+0000 7f0bbd0b0700 1 Processor -- start 2024-03-24T00:11:32.036 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.108+0000 7f0bbd0b0700 1 -- start start 2024-03-24T00:11:32.036 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.108+0000 7f0bbd0b0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bb807ce50 0x7f0bb807b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:32.036 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.108+0000 7f0bbd0b0700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb807b840 con 0x7f0bb807ce50 2024-03-24T00:11:32.036 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.109+0000 7f0bb6d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bb807ce50 0x7f0bb807b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:32.036 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.109+0000 7f0bb6d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bb807ce50 0x7f0bb807b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52606/0 (socket says 172.21.15.78:52606) 2024-03-24T00:11:32.036 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.109+0000 7f0bb6d9d700 1 -- 172.21.15.78:0/3085606002 learned_addr learned my addr 172.21.15.78:0/3085606002 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:32.036 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.109+0000 7f0bb6d9d700 1 -- 172.21.15.78:0/3085606002 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb807b9c0 con 0x7f0bb807ce50 2024-03-24T00:11:32.036 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.109+0000 7f0bb6d9d700 1 --2- 172.21.15.78:0/3085606002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bb807ce50 0x7f0bb807b300 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0ba80098a0 tx=0x7f0ba8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=236dfa3e04d8b790 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:32.036 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.109+0000 7f0bb5d9b700 1 -- 172.21.15.78:0/3085606002 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ba8004160 con 0x7f0bb807ce50 2024-03-24T00:11:32.036 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.109+0000 7f0bb5d9b700 1 -- 172.21.15.78:0/3085606002 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0ba8004300 con 0x7f0bb807ce50 2024-03-24T00:11:32.036 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.109+0000 7f0bb5d9b700 1 -- 172.21.15.78:0/3085606002 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ba8011b50 con 0x7f0bb807ce50 2024-03-24T00:11:32.037 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.110+0000 7f0bbd0b0700 1 -- 172.21.15.78:0/3085606002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bb807ce50 msgr2=0x7f0bb807b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:32.037 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.110+0000 7f0bbd0b0700 1 --2- 172.21.15.78:0/3085606002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bb807ce50 0x7f0bb807b300 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0ba80098a0 tx=0x7f0ba8009bb0 comp rx=0 tx=0).stop 2024-03-24T00:11:32.037 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.110+0000 7f0bbd0b0700 1 -- 172.21.15.78:0/3085606002 shutdown_connections 2024-03-24T00:11:32.037 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.110+0000 7f0bbd0b0700 1 --2- 172.21.15.78:0/3085606002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bb807ce50 0x7f0bb807b300 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:32.037 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.110+0000 7f0bbd0b0700 1 -- 172.21.15.78:0/3085606002 >> 172.21.15.78:0/3085606002 conn(0x7f0bb8103050 msgr2=0x7f0bb8105470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:32.037 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.110+0000 7f0bbd0b0700 1 -- 172.21.15.78:0/3085606002 shutdown_connections 2024-03-24T00:11:32.037 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.110+0000 7f0bbd0b0700 1 -- 172.21.15.78:0/3085606002 wait complete. 2024-03-24T00:11:32.037 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.111+0000 7f0bbd0b0700 1 Processor -- start 2024-03-24T00:11:32.037 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.111+0000 7f0bbd0b0700 1 -- start start 2024-03-24T00:11:32.037 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.111+0000 7f0bbd0b0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bb807ce50 0x7f0bb819dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:32.037 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.111+0000 7f0bbd0b0700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb810e020 con 0x7f0bb807ce50 2024-03-24T00:11:32.037 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.111+0000 7f0bb6d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bb807ce50 0x7f0bb819dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:32.037 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.111+0000 7f0bb6d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bb807ce50 0x7f0bb819dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52616/0 (socket says 172.21.15.78:52616) 2024-03-24T00:11:32.038 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.111+0000 7f0bb6d9d700 1 -- 172.21.15.78:0/2664757651 learned_addr learned my addr 172.21.15.78:0/2664757651 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:32.038 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.111+0000 7f0bb6d9d700 1 -- 172.21.15.78:0/2664757651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb819e170 con 0x7f0bb807ce50 2024-03-24T00:11:32.038 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.112+0000 7f0bb6d9d700 1 --2- 172.21.15.78:0/2664757651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bb807ce50 0x7f0bb819dc30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8009fb0 tx=0x7f0ba8011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:32.038 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.112+0000 7f0b9ffff700 1 -- 172.21.15.78:0/2664757651 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ba80038b0 con 0x7f0bb807ce50 2024-03-24T00:11:32.038 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.112+0000 7f0bbd0b0700 1 -- 172.21.15.78:0/2664757651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bb819e400 con 0x7f0bb807ce50 2024-03-24T00:11:32.038 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.112+0000 7f0b9ffff700 1 -- 172.21.15.78:0/2664757651 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0ba8011e50 con 0x7f0bb807ce50 2024-03-24T00:11:32.038 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.112+0000 7f0b9ffff700 1 -- 172.21.15.78:0/2664757651 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ba8019420 con 0x7f0bb807ce50 2024-03-24T00:11:32.038 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.112+0000 7f0bbd0b0700 1 -- 172.21.15.78:0/2664757651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb81a10f0 con 0x7f0bb807ce50 2024-03-24T00:11:32.038 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.113+0000 7f0bbd0b0700 1 -- 172.21.15.78:0/2664757651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bb80637c0 con 0x7f0bb807ce50 2024-03-24T00:11:32.038 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.113+0000 7f0b9ffff700 1 -- 172.21.15.78:0/2664757651 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f0ba8003a50 con 0x7f0bb807ce50 2024-03-24T00:11:32.038 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.113+0000 7f0b9ffff700 1 --2- 172.21.15.78:0/2664757651 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f0ba0038ed0 0x7f0ba003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:32.038 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.113+0000 7f0b9ffff700 1 -- 172.21.15.78:0/2664757651 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f0ba804d0b0 con 0x7f0bb807ce50 2024-03-24T00:11:32.038 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.113+0000 7f0bb659c700 1 --2- 172.21.15.78:0/2664757651 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f0ba0038ed0 0x7f0ba003b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:32.039 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.114+0000 7f0bb659c700 1 --2- 172.21.15.78:0/2664757651 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f0ba0038ed0 0x7f0ba003b390 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0bac006fd0 tx=0x7f0bac006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:32.039 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.118+0000 7f0b9ffff700 1 -- 172.21.15.78:0/2664757651 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0ba8016260 con 0x7f0bb807ce50 2024-03-24T00:11:32.039 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.321+0000 7f0bbd0b0700 1 -- 172.21.15.78:0/2664757651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f0bb81a17a0 con 0x7f0bb807ce50 2024-03-24T00:11:32.039 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.324+0000 7f0b9ffff700 1 -- 172.21.15.78:0/2664757651 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v7)=0 v7) v1 ==== 130+0+0 (secure 0 0 0) 0x7f0ba8019720 con 0x7f0bb807ce50 2024-03-24T00:11:32.039 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.327+0000 7f0bbd0b0700 1 -- 172.21.15.78:0/2664757651 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f0ba0038ed0 msgr2=0x7f0ba003b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:32.039 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.327+0000 7f0bbd0b0700 1 --2- 172.21.15.78:0/2664757651 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f0ba0038ed0 0x7f0ba003b390 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0bac006fd0 tx=0x7f0bac006e40 comp rx=0 tx=0).stop 2024-03-24T00:11:32.039 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.327+0000 7f0bbd0b0700 1 -- 172.21.15.78:0/2664757651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bb807ce50 msgr2=0x7f0bb819dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:32.039 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.327+0000 7f0bbd0b0700 1 --2- 172.21.15.78:0/2664757651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bb807ce50 0x7f0bb819dc30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8009fb0 tx=0x7f0ba8011bf0 comp rx=0 tx=0).stop 2024-03-24T00:11:32.039 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.327+0000 7f0bbd0b0700 1 -- 172.21.15.78:0/2664757651 shutdown_connections 2024-03-24T00:11:32.039 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.327+0000 7f0bbd0b0700 1 --2- 172.21.15.78:0/2664757651 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f0ba0038ed0 0x7f0ba003b390 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:32.039 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.327+0000 7f0bbd0b0700 1 --2- 172.21.15.78:0/2664757651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bb807ce50 0x7f0bb819dc30 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:32.039 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.327+0000 7f0bbd0b0700 1 -- 172.21.15.78:0/2664757651 >> 172.21.15.78:0/2664757651 conn(0x7f0bb8103050 msgr2=0x7f0bb8103c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:32.039 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.327+0000 7f0bbd0b0700 1 -- 172.21.15.78:0/2664757651 shutdown_connections 2024-03-24T00:11:32.039 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:31.327+0000 7f0bbd0b0700 1 -- 172.21.15.78:0/2664757651 wait complete. 2024-03-24T00:11:32.040 INFO:teuthology.orchestra.run.smithi078.stdout:Enabling the dashboard module... 2024-03-24T00:11:32.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:32 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2664757651' entity='client.admin' 2024-03-24T00:11:32.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:32 smithi078 ceph-mon[32429]: from='mgr.14118 172.21.15.78:0/3702326166' entity='mgr.x' 2024-03-24T00:11:32.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:32 smithi078 ceph-mon[32429]: from='mgr.14118 172.21.15.78:0/3702326166' entity='mgr.x' 2024-03-24T00:11:33.328 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:33 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2186016690' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-24T00:11:33.590 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:33 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: ignoring --setuser ceph since I am not root 2024-03-24T00:11:33.590 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:33 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: ignoring --setgroup ceph since I am not root 2024-03-24T00:11:33.929 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:33 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:33.895+0000 7fd0a2bf9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T00:11:33.938 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.542+0000 7fb15031f700 1 Processor -- start 2024-03-24T00:11:33.938 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.542+0000 7fb15031f700 1 -- start start 2024-03-24T00:11:33.938 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.543+0000 7fb15031f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb148105cf0 0x7fb148106110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:33.938 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.543+0000 7fb15031f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1481066e0 con 0x7fb148105cf0 2024-03-24T00:11:33.938 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.543+0000 7fb14e0bb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb148105cf0 0x7fb148106110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:33.938 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.543+0000 7fb14e0bb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb148105cf0 0x7fb148106110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52626/0 (socket says 172.21.15.78:52626) 2024-03-24T00:11:33.938 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.543+0000 7fb14e0bb700 1 -- 172.21.15.78:0/2438613000 learned_addr learned my addr 172.21.15.78:0/2438613000 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:33.938 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.543+0000 7fb14e0bb700 1 -- 172.21.15.78:0/2438613000 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb148108c80 con 0x7fb148105cf0 2024-03-24T00:11:33.938 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.543+0000 7fb14e0bb700 1 --2- 172.21.15.78:0/2438613000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb148105cf0 0x7fb148106110 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb1380098a0 tx=0x7fb138009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed5a2179bb329f38 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:33.938 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.543+0000 7fb14d0b9700 1 -- 172.21.15.78:0/2438613000 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb138004160 con 0x7fb148105cf0 2024-03-24T00:11:33.939 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.544+0000 7fb14d0b9700 1 -- 172.21.15.78:0/2438613000 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb138004300 con 0x7fb148105cf0 2024-03-24T00:11:33.939 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.544+0000 7fb14d0b9700 1 -- 172.21.15.78:0/2438613000 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb138011b20 con 0x7fb148105cf0 2024-03-24T00:11:33.939 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.544+0000 7fb15031f700 1 -- 172.21.15.78:0/2438613000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb148105cf0 msgr2=0x7fb148106110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:33.939 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.544+0000 7fb15031f700 1 --2- 172.21.15.78:0/2438613000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb148105cf0 0x7fb148106110 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb1380098a0 tx=0x7fb138009bb0 comp rx=0 tx=0).stop 2024-03-24T00:11:33.939 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.544+0000 7fb15031f700 1 -- 172.21.15.78:0/2438613000 shutdown_connections 2024-03-24T00:11:33.939 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.544+0000 7fb15031f700 1 --2- 172.21.15.78:0/2438613000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb148105cf0 0x7fb148106110 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:33.939 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.544+0000 7fb15031f700 1 -- 172.21.15.78:0/2438613000 >> 172.21.15.78:0/2438613000 conn(0x7fb148101640 msgr2=0x7fb148103a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:33.939 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.544+0000 7fb15031f700 1 -- 172.21.15.78:0/2438613000 shutdown_connections 2024-03-24T00:11:33.939 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.544+0000 7fb15031f700 1 -- 172.21.15.78:0/2438613000 wait complete. 2024-03-24T00:11:33.939 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.545+0000 7fb15031f700 1 Processor -- start 2024-03-24T00:11:33.939 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.545+0000 7fb15031f700 1 -- start start 2024-03-24T00:11:33.939 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.545+0000 7fb15031f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb148105cf0 0x7fb14819b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:33.939 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.545+0000 7fb15031f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14810c030 con 0x7fb148105cf0 2024-03-24T00:11:33.939 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.545+0000 7fb14e0bb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb148105cf0 0x7fb14819b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:33.939 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.545+0000 7fb14e0bb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb148105cf0 0x7fb14819b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52636/0 (socket says 172.21.15.78:52636) 2024-03-24T00:11:33.940 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.545+0000 7fb14e0bb700 1 -- 172.21.15.78:0/2186016690 learned_addr learned my addr 172.21.15.78:0/2186016690 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:33.940 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.546+0000 7fb14e0bb700 1 -- 172.21.15.78:0/2186016690 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb14819bf10 con 0x7fb148105cf0 2024-03-24T00:11:33.940 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.546+0000 7fb14e0bb700 1 --2- 172.21.15.78:0/2186016690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb148105cf0 0x7fb14819b9d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb138000c00 tx=0x7fb138011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:33.940 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.546+0000 7fb13f7fe700 1 -- 172.21.15.78:0/2186016690 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb138003b60 con 0x7fb148105cf0 2024-03-24T00:11:33.940 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.546+0000 7fb15031f700 1 -- 172.21.15.78:0/2186016690 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb14819c1a0 con 0x7fb148105cf0 2024-03-24T00:11:33.940 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.546+0000 7fb13f7fe700 1 -- 172.21.15.78:0/2186016690 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb13801a910 con 0x7fb148105cf0 2024-03-24T00:11:33.940 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.546+0000 7fb15031f700 1 -- 172.21.15.78:0/2186016690 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb14819c600 con 0x7fb148105cf0 2024-03-24T00:11:33.940 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.546+0000 7fb13f7fe700 1 -- 172.21.15.78:0/2186016690 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb138024930 con 0x7fb148105cf0 2024-03-24T00:11:33.940 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.547+0000 7fb13f7fe700 1 -- 172.21.15.78:0/2186016690 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fb138022020 con 0x7fb148105cf0 2024-03-24T00:11:33.940 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.547+0000 7fb15031f700 1 -- 172.21.15.78:0/2186016690 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb12c005360 con 0x7fb148105cf0 2024-03-24T00:11:33.940 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.547+0000 7fb13f7fe700 1 --2- 172.21.15.78:0/2186016690 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7fb134038f20 0x7fb13403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:33.940 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.547+0000 7fb14d8ba700 1 --2- 172.21.15.78:0/2186016690 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7fb134038f20 0x7fb13403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:33.940 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.548+0000 7fb13f7fe700 1 -- 172.21.15.78:0/2186016690 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb13804cfe0 con 0x7fb148105cf0 2024-03-24T00:11:33.941 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.548+0000 7fb14d8ba700 1 --2- 172.21.15.78:0/2186016690 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7fb134038f20 0x7fb13403b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb144006fd0 tx=0x7fb144006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:33.941 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.553+0000 7fb13f7fe700 1 -- 172.21.15.78:0/2186016690 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb13800b5a0 con 0x7fb148105cf0 2024-03-24T00:11:33.941 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:32.822+0000 7fb15031f700 1 -- 172.21.15.78:0/2186016690 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fb12c005650 con 0x7fb148105cf0 2024-03-24T00:11:33.941 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:33.327+0000 7fb13f7fe700 1 -- 172.21.15.78:0/2186016690 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fb13802dc90 con 0x7fb148105cf0 2024-03-24T00:11:33.941 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:33.329+0000 7fb13f7fe700 1 -- 172.21.15.78:0/2186016690 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7fb13801aac0 con 0x7fb148105cf0 2024-03-24T00:11:33.941 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:33.330+0000 7fb15031f700 1 -- 172.21.15.78:0/2186016690 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7fb134038f20 msgr2=0x7fb13403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:33.941 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:33.330+0000 7fb15031f700 1 --2- 172.21.15.78:0/2186016690 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7fb134038f20 0x7fb13403b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb144006fd0 tx=0x7fb144006e40 comp rx=0 tx=0).stop 2024-03-24T00:11:33.941 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:33.330+0000 7fb15031f700 1 -- 172.21.15.78:0/2186016690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb148105cf0 msgr2=0x7fb14819b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:33.941 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:33.330+0000 7fb15031f700 1 --2- 172.21.15.78:0/2186016690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb148105cf0 0x7fb14819b9d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb138000c00 tx=0x7fb138011c10 comp rx=0 tx=0).stop 2024-03-24T00:11:33.941 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:33.330+0000 7fb15031f700 1 -- 172.21.15.78:0/2186016690 shutdown_connections 2024-03-24T00:11:33.941 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:33.330+0000 7fb15031f700 1 --2- 172.21.15.78:0/2186016690 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7fb134038f20 0x7fb13403b3e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:33.941 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:33.330+0000 7fb15031f700 1 --2- 172.21.15.78:0/2186016690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb148105cf0 0x7fb14819b9d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:33.941 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:33.330+0000 7fb15031f700 1 -- 172.21.15.78:0/2186016690 >> 172.21.15.78:0/2186016690 conn(0x7fb148101640 msgr2=0x7fb1481021a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:33.941 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:33.330+0000 7fb15031f700 1 -- 172.21.15.78:0/2186016690 shutdown_connections 2024-03-24T00:11:33.942 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:33.330+0000 7fb15031f700 1 -- 172.21.15.78:0/2186016690 wait complete. 2024-03-24T00:11:34.202 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:33 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:33.993+0000 7fd0a2bf9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T00:11:34.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:34 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2186016690' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-24T00:11:34.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:34 smithi078 ceph-mon[32429]: mgrmap e9: x(active, since 18s) 2024-03-24T00:11:34.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:34 smithi078 ceph-mon[32429]: from='mgr.14118 172.21.15.78:0/3702326166' entity='mgr.x' 2024-03-24T00:11:34.840 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:34.655+0000 7fd0a2bf9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T00:11:34.840 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:34.752+0000 7fd0a2bf9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T00:11:35.245 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:34.896+0000 7fd0a2bf9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T00:11:35.245 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:34.991+0000 7fd0a2bf9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T00:11:35.245 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:35 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:35.076+0000 7fd0a2bf9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T00:11:35.253 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-03-24T00:11:35.253 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-24T00:11:35.253 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T00:11:35.253 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-24T00:11:35.254 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-24T00:11:35.254 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-03-24T00:11:35.254 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.382+0000 7f2225b31700 1 Processor -- start 2024-03-24T00:11:35.254 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.382+0000 7f2225b31700 1 -- start start 2024-03-24T00:11:35.254 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.382+0000 7f2225b31700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2220107ec0 0x7f22201082e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:35.254 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.382+0000 7f2225b31700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22201088b0 con 0x7f2220107ec0 2024-03-24T00:11:35.254 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.382+0000 7f221f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2220107ec0 0x7f22201082e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:35.254 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.382+0000 7f221f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2220107ec0 0x7f22201082e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52666/0 (socket says 172.21.15.78:52666) 2024-03-24T00:11:35.254 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.382+0000 7f221f7fe700 1 -- 172.21.15.78:0/42812669 learned_addr learned my addr 172.21.15.78:0/42812669 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:35.254 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.383+0000 7f221f7fe700 1 -- 172.21.15.78:0/42812669 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2220106ad0 con 0x7f2220107ec0 2024-03-24T00:11:35.254 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.383+0000 7f221f7fe700 1 --2- 172.21.15.78:0/42812669 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2220107ec0 0x7f22201082e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f22140098a0 tx=0x7f2214009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41a89a38c1a8c7c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:35.254 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.383+0000 7f221e7fc700 1 -- 172.21.15.78:0/42812669 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2214004160 con 0x7f2220107ec0 2024-03-24T00:11:35.254 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.383+0000 7f221e7fc700 1 -- 172.21.15.78:0/42812669 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2214004300 con 0x7f2220107ec0 2024-03-24T00:11:35.255 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.383+0000 7f221e7fc700 1 -- 172.21.15.78:0/42812669 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2214011b20 con 0x7f2220107ec0 2024-03-24T00:11:35.255 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.384+0000 7f2225b31700 1 -- 172.21.15.78:0/42812669 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2220107ec0 msgr2=0x7f22201082e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:35.255 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.384+0000 7f2225b31700 1 --2- 172.21.15.78:0/42812669 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2220107ec0 0x7f22201082e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f22140098a0 tx=0x7f2214009bb0 comp rx=0 tx=0).stop 2024-03-24T00:11:35.255 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.384+0000 7f2225b31700 1 -- 172.21.15.78:0/42812669 shutdown_connections 2024-03-24T00:11:35.255 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.384+0000 7f2225b31700 1 --2- 172.21.15.78:0/42812669 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2220107ec0 0x7f22201082e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:35.255 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.384+0000 7f2225b31700 1 -- 172.21.15.78:0/42812669 >> 172.21.15.78:0/42812669 conn(0x7f2220101640 msgr2=0x7f2220103a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:35.255 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.384+0000 7f2225b31700 1 -- 172.21.15.78:0/42812669 shutdown_connections 2024-03-24T00:11:35.255 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.384+0000 7f2225b31700 1 -- 172.21.15.78:0/42812669 wait complete. 2024-03-24T00:11:35.255 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.384+0000 7f2225b31700 1 Processor -- start 2024-03-24T00:11:35.255 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.384+0000 7f2225b31700 1 -- start start 2024-03-24T00:11:35.255 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.385+0000 7f2225b31700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2220107ec0 0x7f2220191a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:35.255 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.385+0000 7f2225b31700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222010ae90 con 0x7f2220107ec0 2024-03-24T00:11:35.255 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.385+0000 7f221f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2220107ec0 0x7f2220191a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:35.256 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.385+0000 7f221f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2220107ec0 0x7f2220191a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52682/0 (socket says 172.21.15.78:52682) 2024-03-24T00:11:35.256 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.385+0000 7f221f7fe700 1 -- 172.21.15.78:0/3705559134 learned_addr learned my addr 172.21.15.78:0/3705559134 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:35.256 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.385+0000 7f221f7fe700 1 -- 172.21.15.78:0/3705559134 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2220191f80 con 0x7f2220107ec0 2024-03-24T00:11:35.256 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.385+0000 7f221f7fe700 1 --2- 172.21.15.78:0/3705559134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2220107ec0 0x7f2220191a40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2214000c00 tx=0x7f2214011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:35.256 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.385+0000 7f221cff9700 1 -- 172.21.15.78:0/3705559134 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2214003b60 con 0x7f2220107ec0 2024-03-24T00:11:35.256 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.385+0000 7f2225b31700 1 -- 172.21.15.78:0/3705559134 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22201901c0 con 0x7f2220107ec0 2024-03-24T00:11:35.256 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.385+0000 7f221cff9700 1 -- 172.21.15.78:0/3705559134 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f221401a910 con 0x7f2220107ec0 2024-03-24T00:11:35.256 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.385+0000 7f221cff9700 1 -- 172.21.15.78:0/3705559134 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2214024930 con 0x7f2220107ec0 2024-03-24T00:11:35.256 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.385+0000 7f2225b31700 1 -- 172.21.15.78:0/3705559134 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2220190620 con 0x7f2220107ec0 2024-03-24T00:11:35.256 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.386+0000 7f221cff9700 1 -- 172.21.15.78:0/3705559134 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f2214022020 con 0x7f2220107ec0 2024-03-24T00:11:35.256 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.386+0000 7f2225b31700 1 -- 172.21.15.78:0/3705559134 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2220069220 con 0x7f2220107ec0 2024-03-24T00:11:35.256 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.386+0000 7f221cff9700 1 --2- 172.21.15.78:0/3705559134 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f2208038fd0 0x7f220803b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:35.256 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.387+0000 7f221cff9700 1 -- 172.21.15.78:0/3705559134 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f221404d040 con 0x7f2220107ec0 2024-03-24T00:11:35.256 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.387+0000 7f221effd700 1 -- 172.21.15.78:0/3705559134 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f2208038fd0 msgr2=0x7f220803b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2748920178 2024-03-24T00:11:35.257 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.387+0000 7f221effd700 1 --2- 172.21.15.78:0/3705559134 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f2208038fd0 0x7f220803b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:11:35.257 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.391+0000 7f221cff9700 1 -- 172.21.15.78:0/3705559134 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f22140172b0 con 0x7f2220107ec0 2024-03-24T00:11:35.257 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.588+0000 7f221effd700 1 -- 172.21.15.78:0/3705559134 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f2208038fd0 msgr2=0x7f220803b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2748920178 2024-03-24T00:11:35.257 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.588+0000 7f221effd700 1 --2- 172.21.15.78:0/3705559134 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f2208038fd0 0x7f220803b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:11:35.257 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.652+0000 7f2225b31700 1 -- 172.21.15.78:0/3705559134 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f22201956d0 con 0x7f2220107ec0 2024-03-24T00:11:35.257 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.652+0000 7f221cff9700 1 -- 172.21.15.78:0/3705559134 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f221402c510 con 0x7f2220107ec0 2024-03-24T00:11:35.257 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.654+0000 7f2225b31700 1 -- 172.21.15.78:0/3705559134 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f2208038fd0 msgr2=0x7f220803b490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T00:11:35.257 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.654+0000 7f2225b31700 1 --2- 172.21.15.78:0/3705559134 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f2208038fd0 0x7f220803b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:35.257 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.654+0000 7f2225b31700 1 -- 172.21.15.78:0/3705559134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2220107ec0 msgr2=0x7f2220191a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:35.257 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.654+0000 7f2225b31700 1 --2- 172.21.15.78:0/3705559134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2220107ec0 0x7f2220191a40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2214000c00 tx=0x7f2214011c10 comp rx=0 tx=0).stop 2024-03-24T00:11:35.257 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.654+0000 7f2225b31700 1 -- 172.21.15.78:0/3705559134 shutdown_connections 2024-03-24T00:11:35.257 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.654+0000 7f2225b31700 1 --2- 172.21.15.78:0/3705559134 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f2208038fd0 0x7f220803b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:35.257 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.654+0000 7f2225b31700 1 --2- 172.21.15.78:0/3705559134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2220107ec0 0x7f2220191a40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:35.257 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.654+0000 7f2225b31700 1 -- 172.21.15.78:0/3705559134 >> 172.21.15.78:0/3705559134 conn(0x7f2220101640 msgr2=0x7f2220102530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:35.258 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.655+0000 7f2225b31700 1 -- 172.21.15.78:0/3705559134 shutdown_connections 2024-03-24T00:11:35.258 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:34.655+0000 7f2225b31700 1 -- 172.21.15.78:0/3705559134 wait complete. 2024-03-24T00:11:35.258 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for the mgr to restart... 2024-03-24T00:11:35.258 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mgr epoch 9... 2024-03-24T00:11:35.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:35 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3705559134' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T00:11:35.552 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:35 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:35.347+0000 7fd0a2bf9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T00:11:35.552 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:35 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:35.460+0000 7fd0a2bf9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T00:11:35.552 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:35 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:35.551+0000 7fd0a2bf9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T00:11:35.828 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:35 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:35.648+0000 7fd0a2bf9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T00:11:36.090 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:35 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:35.827+0000 7fd0a2bf9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T00:11:37.090 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:36 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:36.779+0000 7fd0a2bf9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T00:11:37.090 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:36 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:36.908+0000 7fd0a2bf9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T00:11:37.526 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:37 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:37.239+0000 7fd0a2bf9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T00:11:37.526 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:37 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:37.330+0000 7fd0a2bf9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T00:11:37.526 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:37 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:37.525+0000 7fd0a2bf9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:11:37.840 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:37 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:37.709+0000 7fd0a2bf9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T00:11:38.249 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:37 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:37.992+0000 7fd0a2bf9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:11:38.249 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:38 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:38.248+0000 7fd0a2bf9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T00:11:38.590 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:38 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:38.483+0000 7fd0a2bf9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T00:11:39.590 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:39 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:39.286+0000 7fd0a2bf9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T00:11:40.186 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:39 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:39.868+0000 7fd0a2bf9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T00:11:40.186 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:39 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:39.968+0000 7fd0a2bf9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T00:11:40.186 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:11:40 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:11:40.185+0000 7fd0a2bf9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T00:11:40.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:40 smithi078 ceph-mon[32429]: Active manager daemon x restarted 2024-03-24T00:11:40.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:40 smithi078 ceph-mon[32429]: Activating manager daemon x 2024-03-24T00:11:40.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:40 smithi078 ceph-mon[32429]: osdmap e3: 0 total, 0 up, 0 in 2024-03-24T00:11:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:40 smithi078 ceph-mon[32429]: mgrmap e10: x(active, starting, since 0.0034702s) 2024-03-24T00:11:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:11:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T00:11:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T00:11:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T00:11:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T00:11:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:40 smithi078 ceph-mon[32429]: Manager daemon x is now available 2024-03-24T00:11:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:11:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:11:41.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:41 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T00:11:41.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:41 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T00:11:41.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:41 smithi078 ceph-mon[32429]: mgrmap e11: x(active, since 1.00723s) 2024-03-24T00:11:42.138 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-03-24T00:11:42.138 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-24T00:11:42.138 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-24T00:11:42.138 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-03-24T00:11:42.138 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.688+0000 7f5334914700 1 Processor -- start 2024-03-24T00:11:42.138 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.688+0000 7f5334914700 1 -- start start 2024-03-24T00:11:42.138 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.689+0000 7f5334914700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5330108430 0x7f533010a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:42.138 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.689+0000 7f5334914700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533007aed0 con 0x7f5330108430 2024-03-24T00:11:42.138 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.689+0000 7f532e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5330108430 0x7f533010a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:42.138 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.689+0000 7f532e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5330108430 0x7f533010a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33270/0 (socket says 172.21.15.78:33270) 2024-03-24T00:11:42.139 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.689+0000 7f532e59c700 1 -- 172.21.15.78:0/2460458426 learned_addr learned my addr 172.21.15.78:0/2460458426 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:42.139 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.689+0000 7f532e59c700 1 -- 172.21.15.78:0/2460458426 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f533010ad60 con 0x7f5330108430 2024-03-24T00:11:42.139 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.689+0000 7f532e59c700 1 --2- 172.21.15.78:0/2460458426 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5330108430 0x7f533010a820 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f53200098a0 tx=0x7f5320009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5879ec03f705a682 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:42.139 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.689+0000 7f532d59a700 1 -- 172.21.15.78:0/2460458426 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5320004160 con 0x7f5330108430 2024-03-24T00:11:42.139 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.689+0000 7f532d59a700 1 -- 172.21.15.78:0/2460458426 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5320004300 con 0x7f5330108430 2024-03-24T00:11:42.139 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.689+0000 7f532d59a700 1 -- 172.21.15.78:0/2460458426 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5320011b20 con 0x7f5330108430 2024-03-24T00:11:42.139 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.690+0000 7f5334914700 1 -- 172.21.15.78:0/2460458426 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5330108430 msgr2=0x7f533010a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:42.139 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.690+0000 7f5334914700 1 --2- 172.21.15.78:0/2460458426 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5330108430 0x7f533010a820 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f53200098a0 tx=0x7f5320009bb0 comp rx=0 tx=0).stop 2024-03-24T00:11:42.139 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.690+0000 7f5334914700 1 -- 172.21.15.78:0/2460458426 shutdown_connections 2024-03-24T00:11:42.139 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.690+0000 7f5334914700 1 --2- 172.21.15.78:0/2460458426 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5330108430 0x7f533010a820 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:42.139 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.690+0000 7f5334914700 1 -- 172.21.15.78:0/2460458426 >> 172.21.15.78:0/2460458426 conn(0x7f5330101f40 msgr2=0x7f5330104360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:42.139 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.690+0000 7f5334914700 1 -- 172.21.15.78:0/2460458426 shutdown_connections 2024-03-24T00:11:42.139 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.690+0000 7f5334914700 1 -- 172.21.15.78:0/2460458426 wait complete. 2024-03-24T00:11:42.139 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.691+0000 7f5334914700 1 Processor -- start 2024-03-24T00:11:42.139 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.691+0000 7f5334914700 1 -- start start 2024-03-24T00:11:42.140 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.691+0000 7f5334914700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5330108430 0x7f533019b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:42.140 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.691+0000 7f5334914700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533010bdd0 con 0x7f5330108430 2024-03-24T00:11:42.140 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.691+0000 7f532e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5330108430 0x7f533019b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:42.140 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.691+0000 7f532e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5330108430 0x7f533019b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33274/0 (socket says 172.21.15.78:33274) 2024-03-24T00:11:42.140 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.691+0000 7f532e59c700 1 -- 172.21.15.78:0/3083737269 learned_addr learned my addr 172.21.15.78:0/3083737269 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:42.140 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.692+0000 7f532e59c700 1 -- 172.21.15.78:0/3083737269 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f533019be60 con 0x7f5330108430 2024-03-24T00:11:42.140 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.692+0000 7f532e59c700 1 --2- 172.21.15.78:0/3083737269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5330108430 0x7f533019b920 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f53200116a0 tx=0x7f5320011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:42.140 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.692+0000 7f531f7fe700 1 -- 172.21.15.78:0/3083737269 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5320003a40 con 0x7f5330108430 2024-03-24T00:11:42.140 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.692+0000 7f5334914700 1 -- 172.21.15.78:0/3083737269 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f533019c0f0 con 0x7f5330108430 2024-03-24T00:11:42.140 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.692+0000 7f531f7fe700 1 -- 172.21.15.78:0/3083737269 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5320003be0 con 0x7f5330108430 2024-03-24T00:11:42.140 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.692+0000 7f531f7fe700 1 -- 172.21.15.78:0/3083737269 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f532001a750 con 0x7f5330108430 2024-03-24T00:11:42.140 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.692+0000 7f5334914700 1 -- 172.21.15.78:0/3083737269 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f533019c550 con 0x7f5330108430 2024-03-24T00:11:42.140 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.693+0000 7f531f7fe700 1 -- 172.21.15.78:0/3083737269 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f532001aad0 con 0x7f5330108430 2024-03-24T00:11:42.140 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.693+0000 7f531f7fe700 1 --2- 172.21.15.78:0/3083737269 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f5318038f20 0x7f531803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:42.140 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.693+0000 7f531f7fe700 1 -- 172.21.15.78:0/3083737269 --> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f531803baf0 con 0x7f5318038f20 2024-03-24T00:11:42.140 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.693+0000 7f532dd9b700 1 -- 172.21.15.78:0/3083737269 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f5318038f20 msgr2=0x7f531803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2748920178 2024-03-24T00:11:42.141 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.693+0000 7f531f7fe700 1 -- 172.21.15.78:0/3083737269 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f532004d5a0 con 0x7f5330108430 2024-03-24T00:11:42.141 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.693+0000 7f532dd9b700 1 --2- 172.21.15.78:0/3083737269 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f5318038f20 0x7f531803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:11:42.141 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.894+0000 7f532dd9b700 1 -- 172.21.15.78:0/3083737269 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f5318038f20 msgr2=0x7f531803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2748920178 2024-03-24T00:11:42.141 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:35.894+0000 7f532dd9b700 1 --2- 172.21.15.78:0/3083737269 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f5318038f20 0x7f531803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:11:42.141 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:36.295+0000 7f532dd9b700 1 -- 172.21.15.78:0/3083737269 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f5318038f20 msgr2=0x7f531803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2748920178 2024-03-24T00:11:42.141 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:36.295+0000 7f532dd9b700 1 --2- 172.21.15.78:0/3083737269 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f5318038f20 0x7f531803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:11:42.141 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:37.096+0000 7f532dd9b700 1 -- 172.21.15.78:0/3083737269 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f5318038f20 msgr2=0x7f531803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2748920178 2024-03-24T00:11:42.141 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:37.096+0000 7f532dd9b700 1 --2- 172.21.15.78:0/3083737269 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f5318038f20 0x7f531803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:11:42.141 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:38.697+0000 7f532dd9b700 1 -- 172.21.15.78:0/3083737269 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f5318038f20 msgr2=0x7f531803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2748920178 2024-03-24T00:11:42.141 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:38.697+0000 7f532dd9b700 1 --2- 172.21.15.78:0/3083737269 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f5318038f20 0x7f531803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T00:11:42.141 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:40.200+0000 7f531f7fe700 1 -- 172.21.15.78:0/3083737269 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mgrmap(e 10) v1 ==== 45886+0+0 (secure 0 0 0) 0x7f532002d910 con 0x7f5330108430 2024-03-24T00:11:42.141 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:40.200+0000 7f531f7fe700 1 -- 172.21.15.78:0/3083737269 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f5318038f20 msgr2=0x7f531803b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T00:11:42.141 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:40.200+0000 7f531f7fe700 1 --2- 172.21.15.78:0/3083737269 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f5318038f20 0x7f531803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:42.141 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:41.202+0000 7f531f7fe700 1 -- 172.21.15.78:0/3083737269 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 11) v1 ==== 46013+0+0 (secure 0 0 0) 0x7f532004f1b0 con 0x7f5330108430 2024-03-24T00:11:42.141 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:41.202+0000 7f531f7fe700 1 --2- 172.21.15.78:0/3083737269 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f531803c9a0 0x7f531803ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:42.142 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:41.202+0000 7f531f7fe700 1 -- 172.21.15.78:0/3083737269 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f532004cc00 con 0x7f531803c9a0 2024-03-24T00:11:42.142 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:41.202+0000 7f532dd9b700 1 --2- 172.21.15.78:0/3083737269 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f531803c9a0 0x7f531803ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:42.142 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:41.203+0000 7f532dd9b700 1 --2- 172.21.15.78:0/3083737269 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f531803c9a0 0x7f531803ed90 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f5324003a10 tx=0x7f53240092b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:42.142 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:41.203+0000 7f531f7fe700 1 -- 172.21.15.78:0/3083737269 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f532004cc00 con 0x7f531803c9a0 2024-03-24T00:11:42.142 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:41.210+0000 7f5334914700 1 -- 172.21.15.78:0/3083737269 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f5330069220 con 0x7f531803c9a0 2024-03-24T00:11:42.142 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:41.210+0000 7f531f7fe700 1 -- 172.21.15.78:0/3083737269 <== mgr.14150 v2:172.21.15.78:6800/1983101026 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f5330069220 con 0x7f531803c9a0 2024-03-24T00:11:42.142 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:41.211+0000 7f5334914700 1 -- 172.21.15.78:0/3083737269 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f531803c9a0 msgr2=0x7f531803ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:42.142 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:41.211+0000 7f5334914700 1 --2- 172.21.15.78:0/3083737269 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f531803c9a0 0x7f531803ed90 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f5324003a10 tx=0x7f53240092b0 comp rx=0 tx=0).stop 2024-03-24T00:11:42.142 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:41.211+0000 7f5334914700 1 -- 172.21.15.78:0/3083737269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5330108430 msgr2=0x7f533019b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:42.142 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:41.211+0000 7f5334914700 1 --2- 172.21.15.78:0/3083737269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5330108430 0x7f533019b920 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f53200116a0 tx=0x7f5320011ba0 comp rx=0 tx=0).stop 2024-03-24T00:11:42.142 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:41.211+0000 7f5334914700 1 -- 172.21.15.78:0/3083737269 shutdown_connections 2024-03-24T00:11:42.142 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:41.211+0000 7f5334914700 1 --2- 172.21.15.78:0/3083737269 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f531803c9a0 0x7f531803ed90 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:42.142 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:41.211+0000 7f5334914700 1 --2- 172.21.15.78:0/3083737269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5330108430 0x7f533019b920 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:42.142 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:41.211+0000 7f5334914700 1 --2- 172.21.15.78:0/3083737269 >> [v2:172.21.15.78:6800/2748920178,v1:172.21.15.78:6801/2748920178] conn(0x7f5318038f20 0x7f531803b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:42.143 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:41.211+0000 7f5334914700 1 -- 172.21.15.78:0/3083737269 >> 172.21.15.78:0/3083737269 conn(0x7f5330101f40 msgr2=0x7f53301029e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:42.143 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:41.211+0000 7f5334914700 1 -- 172.21.15.78:0/3083737269 shutdown_connections 2024-03-24T00:11:42.143 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:41.211+0000 7f5334914700 1 -- 172.21.15.78:0/3083737269 wait complete. 2024-03-24T00:11:42.143 INFO:teuthology.orchestra.run.smithi078.stdout:mgr epoch 9 is available 2024-03-24T00:11:42.143 INFO:teuthology.orchestra.run.smithi078.stdout:Generating a dashboard self-signed certificate... 2024-03-24T00:11:42.390 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:42 smithi078 ceph-mon[32429]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-24T00:11:42.390 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:42 smithi078 ceph-mon[32429]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-24T00:11:42.390 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:42 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:11:43.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:43 smithi078 ceph-mon[32429]: [24/Mar/2024:00:11:41] ENGINE Bus STARTING 2024-03-24T00:11:43.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:43 smithi078 ceph-mon[32429]: [24/Mar/2024:00:11:41] ENGINE Serving on http://172.21.15.78:8765 2024-03-24T00:11:43.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:43 smithi078 ceph-mon[32429]: [24/Mar/2024:00:11:41] ENGINE Serving on https://172.21.15.78:7150 2024-03-24T00:11:43.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:43 smithi078 ceph-mon[32429]: [24/Mar/2024:00:11:41] ENGINE Bus STARTED 2024-03-24T00:11:43.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:43 smithi078 ceph-mon[32429]: mgrmap e12: x(active, since 2s) 2024-03-24T00:11:43.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:43 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:11:43.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:43 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:11:43.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:43 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:11:43.509 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-24T00:11:43.509 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.582+0000 7f3bf6495700 1 Processor -- start 2024-03-24T00:11:43.509 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.582+0000 7f3bf6495700 1 -- start start 2024-03-24T00:11:43.509 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.582+0000 7f3bf6495700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bf0108510 0x7f3bf010a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:43.509 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.582+0000 7f3bf6495700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf007adf0 con 0x7f3bf0108510 2024-03-24T00:11:43.509 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.583+0000 7f3beffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bf0108510 0x7f3bf010a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:43.509 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.583+0000 7f3beffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bf0108510 0x7f3bf010a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33360/0 (socket says 172.21.15.78:33360) 2024-03-24T00:11:43.509 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.583+0000 7f3beffff700 1 -- 172.21.15.78:0/3873179606 learned_addr learned my addr 172.21.15.78:0/3873179606 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:43.509 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.583+0000 7f3beffff700 1 -- 172.21.15.78:0/3873179606 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf010ae40 con 0x7f3bf0108510 2024-03-24T00:11:43.510 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.583+0000 7f3beffff700 1 --2- 172.21.15.78:0/3873179606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bf0108510 0x7f3bf010a900 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3be40098a0 tx=0x7f3be4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66fc0457bb13b9ac server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:43.510 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.583+0000 7f3beeffd700 1 -- 172.21.15.78:0/3873179606 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3be4004160 con 0x7f3bf0108510 2024-03-24T00:11:43.510 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.583+0000 7f3beeffd700 1 -- 172.21.15.78:0/3873179606 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3be4004300 con 0x7f3bf0108510 2024-03-24T00:11:43.510 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.583+0000 7f3beeffd700 1 -- 172.21.15.78:0/3873179606 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3be4011b20 con 0x7f3bf0108510 2024-03-24T00:11:43.510 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.585+0000 7f3bf6495700 1 -- 172.21.15.78:0/3873179606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bf0108510 msgr2=0x7f3bf010a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:43.510 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.585+0000 7f3bf6495700 1 --2- 172.21.15.78:0/3873179606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bf0108510 0x7f3bf010a900 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3be40098a0 tx=0x7f3be4009bb0 comp rx=0 tx=0).stop 2024-03-24T00:11:43.510 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.585+0000 7f3bf6495700 1 -- 172.21.15.78:0/3873179606 shutdown_connections 2024-03-24T00:11:43.510 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.585+0000 7f3bf6495700 1 --2- 172.21.15.78:0/3873179606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bf0108510 0x7f3bf010a900 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:43.510 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.585+0000 7f3bf6495700 1 -- 172.21.15.78:0/3873179606 >> 172.21.15.78:0/3873179606 conn(0x7f3bf0101f40 msgr2=0x7f3bf0104360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:43.510 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.585+0000 7f3bf6495700 1 -- 172.21.15.78:0/3873179606 shutdown_connections 2024-03-24T00:11:43.510 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.585+0000 7f3bf6495700 1 -- 172.21.15.78:0/3873179606 wait complete. 2024-03-24T00:11:43.510 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.586+0000 7f3bf6495700 1 Processor -- start 2024-03-24T00:11:43.510 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.586+0000 7f3bf6495700 1 -- start start 2024-03-24T00:11:43.511 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.586+0000 7f3bf6495700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bf0108510 0x7f3bf019b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:43.511 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.586+0000 7f3bf6495700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf010bde0 con 0x7f3bf0108510 2024-03-24T00:11:43.511 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.586+0000 7f3beffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bf0108510 0x7f3bf019b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:43.511 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.586+0000 7f3beffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bf0108510 0x7f3bf019b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33374/0 (socket says 172.21.15.78:33374) 2024-03-24T00:11:43.511 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.586+0000 7f3beffff700 1 -- 172.21.15.78:0/45476612 learned_addr learned my addr 172.21.15.78:0/45476612 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:43.511 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.586+0000 7f3beffff700 1 -- 172.21.15.78:0/45476612 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf019be70 con 0x7f3bf0108510 2024-03-24T00:11:43.511 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.586+0000 7f3beffff700 1 --2- 172.21.15.78:0/45476612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bf0108510 0x7f3bf019b930 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3be4011b60 tx=0x7f3be4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:43.511 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.586+0000 7f3bed7fa700 1 -- 172.21.15.78:0/45476612 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3be4003b60 con 0x7f3bf0108510 2024-03-24T00:11:43.511 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.586+0000 7f3bf6495700 1 -- 172.21.15.78:0/45476612 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bf019c100 con 0x7f3bf0108510 2024-03-24T00:11:43.511 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.586+0000 7f3bed7fa700 1 -- 172.21.15.78:0/45476612 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3be401a910 con 0x7f3bf0108510 2024-03-24T00:11:43.511 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.586+0000 7f3bed7fa700 1 -- 172.21.15.78:0/45476612 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3be4024930 con 0x7f3bf0108510 2024-03-24T00:11:43.511 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.586+0000 7f3bf6495700 1 -- 172.21.15.78:0/45476612 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bf019c560 con 0x7f3bf0108510 2024-03-24T00:11:43.511 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.587+0000 7f3bf6495700 1 -- 172.21.15.78:0/45476612 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bf0069220 con 0x7f3bf0108510 2024-03-24T00:11:43.511 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.587+0000 7f3bed7fa700 1 -- 172.21.15.78:0/45476612 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 11) v1 ==== 46013+0+0 (secure 0 0 0) 0x7f3be4022020 con 0x7f3bf0108510 2024-03-24T00:11:43.512 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.587+0000 7f3bed7fa700 1 --2- 172.21.15.78:0/45476612 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3bd8038e50 0x7f3bd803b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:43.512 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.588+0000 7f3bed7fa700 1 -- 172.21.15.78:0/45476612 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3be404d500 con 0x7f3bf0108510 2024-03-24T00:11:43.512 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.588+0000 7f3bef7fe700 1 --2- 172.21.15.78:0/45476612 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3bd8038e50 0x7f3bd803b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:43.512 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.588+0000 7f3bef7fe700 1 --2- 172.21.15.78:0/45476612 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3bd8038e50 0x7f3bd803b310 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3be0006fd0 tx=0x7f3be0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:43.512 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.592+0000 7f3bed7fa700 1 -- 172.21.15.78:0/45476612 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3be402a2a0 con 0x7f3bf0108510 2024-03-24T00:11:43.512 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.625+0000 7f3bed7fa700 1 -- 172.21.15.78:0/45476612 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f3be404c030 con 0x7f3bf0108510 2024-03-24T00:11:43.512 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.805+0000 7f3bf6495700 1 -- 172.21.15.78:0/45476612 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f3bf01030f0 con 0x7f3bd8038e50 2024-03-24T00:11:43.512 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.945+0000 7f3bed7fa700 1 -- 172.21.15.78:0/45476612 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f3bf01030f0 con 0x7f3bd8038e50 2024-03-24T00:11:43.512 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.948+0000 7f3bf6495700 1 -- 172.21.15.78:0/45476612 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3bd8038e50 msgr2=0x7f3bd803b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:43.512 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.948+0000 7f3bf6495700 1 --2- 172.21.15.78:0/45476612 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3bd8038e50 0x7f3bd803b310 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3be0006fd0 tx=0x7f3be0006e40 comp rx=0 tx=0).stop 2024-03-24T00:11:43.512 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.948+0000 7f3bf6495700 1 -- 172.21.15.78:0/45476612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bf0108510 msgr2=0x7f3bf019b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:43.512 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.948+0000 7f3bf6495700 1 --2- 172.21.15.78:0/45476612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bf0108510 0x7f3bf019b930 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3be4011b60 tx=0x7f3be4011c40 comp rx=0 tx=0).stop 2024-03-24T00:11:43.512 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.949+0000 7f3bf6495700 1 -- 172.21.15.78:0/45476612 shutdown_connections 2024-03-24T00:11:43.512 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.949+0000 7f3bf6495700 1 --2- 172.21.15.78:0/45476612 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3bd8038e50 0x7f3bd803b310 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:43.513 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.949+0000 7f3bf6495700 1 --2- 172.21.15.78:0/45476612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bf0108510 0x7f3bf019b930 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:43.513 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.949+0000 7f3bf6495700 1 -- 172.21.15.78:0/45476612 >> 172.21.15.78:0/45476612 conn(0x7f3bf0101f40 msgr2=0x7f3bf01029e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:43.513 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.949+0000 7f3bf6495700 1 -- 172.21.15.78:0/45476612 shutdown_connections 2024-03-24T00:11:43.513 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:42.949+0000 7f3bf6495700 1 -- 172.21.15.78:0/45476612 wait complete. 2024-03-24T00:11:43.513 INFO:teuthology.orchestra.run.smithi078.stdout:Creating initial admin user... 2024-03-24T00:11:44.415 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:44 smithi078 ceph-mon[32429]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:11:45.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:45 smithi078 ceph-mon[32429]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:11:45.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:45 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:11:45.588 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$yeighg9awcxRu.I.EKqeSOxcdBYaN2bYqOStMXGBVA7c2y/FMHmnS", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711239104, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-24T00:11:45.588 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.922+0000 7f2f88fd5700 1 Processor -- start 2024-03-24T00:11:45.588 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.923+0000 7f2f88fd5700 1 -- start start 2024-03-24T00:11:45.588 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.923+0000 7f2f88fd5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f841084a0 0x7f2f8410a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:45.588 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.923+0000 7f2f88fd5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8407ae90 con 0x7f2f841084a0 2024-03-24T00:11:45.588 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.923+0000 7f2f8259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f841084a0 0x7f2f8410a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:45.588 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.923+0000 7f2f8259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f841084a0 0x7f2f8410a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33386/0 (socket says 172.21.15.78:33386) 2024-03-24T00:11:45.588 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.923+0000 7f2f8259c700 1 -- 172.21.15.78:0/2059219647 learned_addr learned my addr 172.21.15.78:0/2059219647 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:45.588 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.923+0000 7f2f8259c700 1 -- 172.21.15.78:0/2059219647 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f8410add0 con 0x7f2f841084a0 2024-03-24T00:11:45.588 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.924+0000 7f2f8259c700 1 --2- 172.21.15.78:0/2059219647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f841084a0 0x7f2f8410a890 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2f740098a0 tx=0x7f2f74009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb4351a2143f9458 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:45.588 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.924+0000 7f2f8159a700 1 -- 172.21.15.78:0/2059219647 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2f74004160 con 0x7f2f841084a0 2024-03-24T00:11:45.588 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.924+0000 7f2f8159a700 1 -- 172.21.15.78:0/2059219647 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2f74004300 con 0x7f2f841084a0 2024-03-24T00:11:45.589 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.924+0000 7f2f8159a700 1 -- 172.21.15.78:0/2059219647 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2f74011b20 con 0x7f2f841084a0 2024-03-24T00:11:45.589 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.924+0000 7f2f88fd5700 1 -- 172.21.15.78:0/2059219647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f841084a0 msgr2=0x7f2f8410a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:45.589 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.924+0000 7f2f88fd5700 1 --2- 172.21.15.78:0/2059219647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f841084a0 0x7f2f8410a890 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2f740098a0 tx=0x7f2f74009bb0 comp rx=0 tx=0).stop 2024-03-24T00:11:45.589 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.924+0000 7f2f88fd5700 1 -- 172.21.15.78:0/2059219647 shutdown_connections 2024-03-24T00:11:45.589 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.924+0000 7f2f88fd5700 1 --2- 172.21.15.78:0/2059219647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f841084a0 0x7f2f8410a890 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:45.589 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.924+0000 7f2f88fd5700 1 -- 172.21.15.78:0/2059219647 >> 172.21.15.78:0/2059219647 conn(0x7f2f84101fb0 msgr2=0x7f2f841043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:45.589 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.924+0000 7f2f88fd5700 1 -- 172.21.15.78:0/2059219647 shutdown_connections 2024-03-24T00:11:45.589 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.924+0000 7f2f88fd5700 1 -- 172.21.15.78:0/2059219647 wait complete. 2024-03-24T00:11:45.589 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.925+0000 7f2f88fd5700 1 Processor -- start 2024-03-24T00:11:45.589 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.925+0000 7f2f88fd5700 1 -- start start 2024-03-24T00:11:45.589 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.925+0000 7f2f88fd5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f841084a0 0x7f2f8419b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:45.589 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.925+0000 7f2f88fd5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8410be30 con 0x7f2f841084a0 2024-03-24T00:11:45.589 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.925+0000 7f2f8259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f841084a0 0x7f2f8419b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:45.589 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.925+0000 7f2f8259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f841084a0 0x7f2f8419b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33392/0 (socket says 172.21.15.78:33392) 2024-03-24T00:11:45.589 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.926+0000 7f2f8259c700 1 -- 172.21.15.78:0/2809221402 learned_addr learned my addr 172.21.15.78:0/2809221402 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:45.590 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.926+0000 7f2f8259c700 1 -- 172.21.15.78:0/2809221402 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f8419be90 con 0x7f2f841084a0 2024-03-24T00:11:45.590 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.926+0000 7f2f8259c700 1 --2- 172.21.15.78:0/2809221402 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f841084a0 0x7f2f8419b950 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2f74011b60 tx=0x7f2f74011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:45.590 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.926+0000 7f2f737fe700 1 -- 172.21.15.78:0/2809221402 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2f74003b60 con 0x7f2f841084a0 2024-03-24T00:11:45.590 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.926+0000 7f2f737fe700 1 -- 172.21.15.78:0/2809221402 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2f7401a910 con 0x7f2f841084a0 2024-03-24T00:11:45.590 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.926+0000 7f2f88fd5700 1 -- 172.21.15.78:0/2809221402 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f8419c120 con 0x7f2f841084a0 2024-03-24T00:11:45.590 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.926+0000 7f2f737fe700 1 -- 172.21.15.78:0/2809221402 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2f74024930 con 0x7f2f841084a0 2024-03-24T00:11:45.590 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.926+0000 7f2f88fd5700 1 -- 172.21.15.78:0/2809221402 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f8419c580 con 0x7f2f841084a0 2024-03-24T00:11:45.590 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.927+0000 7f2f737fe700 1 -- 172.21.15.78:0/2809221402 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f2f74022020 con 0x7f2f841084a0 2024-03-24T00:11:45.590 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.927+0000 7f2f737fe700 1 --2- 172.21.15.78:0/2809221402 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2f6c038fd0 0x7f2f6c03b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:45.590 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.927+0000 7f2f737fe700 1 -- 172.21.15.78:0/2809221402 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2f7404d1e0 con 0x7f2f841084a0 2024-03-24T00:11:45.590 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.927+0000 7f2f88fd5700 1 -- 172.21.15.78:0/2809221402 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f84069220 con 0x7f2f841084a0 2024-03-24T00:11:45.590 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.927+0000 7f2f81d9b700 1 --2- 172.21.15.78:0/2809221402 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2f6c038fd0 0x7f2f6c03b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:45.591 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.928+0000 7f2f81d9b700 1 --2- 172.21.15.78:0/2809221402 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2f6c038fd0 0x7f2f6c03b490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2f78006fd0 tx=0x7f2f78006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:45.591 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:43.932+0000 7f2f737fe700 1 -- 172.21.15.78:0/2809221402 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2f740212c0 con 0x7f2f841084a0 2024-03-24T00:11:45.591 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:44.165+0000 7f2f88fd5700 1 -- 172.21.15.78:0/2809221402 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f2f841031e0 con 0x7f2f6c038fd0 2024-03-24T00:11:45.591 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:44.390+0000 7f2f737fe700 1 -- 172.21.15.78:0/2809221402 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f2f841031e0 con 0x7f2f6c038fd0 2024-03-24T00:11:45.591 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:44.392+0000 7f2f88fd5700 1 -- 172.21.15.78:0/2809221402 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2f6c038fd0 msgr2=0x7f2f6c03b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:45.591 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:44.392+0000 7f2f88fd5700 1 --2- 172.21.15.78:0/2809221402 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2f6c038fd0 0x7f2f6c03b490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2f78006fd0 tx=0x7f2f78006e40 comp rx=0 tx=0).stop 2024-03-24T00:11:45.591 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:44.392+0000 7f2f88fd5700 1 -- 172.21.15.78:0/2809221402 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f841084a0 msgr2=0x7f2f8419b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:45.591 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:44.392+0000 7f2f88fd5700 1 --2- 172.21.15.78:0/2809221402 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f841084a0 0x7f2f8419b950 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2f74011b60 tx=0x7f2f74011c40 comp rx=0 tx=0).stop 2024-03-24T00:11:45.591 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:44.392+0000 7f2f88fd5700 1 -- 172.21.15.78:0/2809221402 shutdown_connections 2024-03-24T00:11:45.591 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:44.392+0000 7f2f88fd5700 1 --2- 172.21.15.78:0/2809221402 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2f6c038fd0 0x7f2f6c03b490 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:45.591 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:44.392+0000 7f2f88fd5700 1 --2- 172.21.15.78:0/2809221402 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f841084a0 0x7f2f8419b950 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:45.591 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:44.392+0000 7f2f88fd5700 1 -- 172.21.15.78:0/2809221402 >> 172.21.15.78:0/2809221402 conn(0x7f2f84101fb0 msgr2=0x7f2f84102ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:45.591 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:44.392+0000 7f2f88fd5700 1 -- 172.21.15.78:0/2809221402 shutdown_connections 2024-03-24T00:11:45.591 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:44.392+0000 7f2f88fd5700 1 -- 172.21.15.78:0/2809221402 wait complete. 2024-03-24T00:11:45.591 INFO:teuthology.orchestra.run.smithi078.stdout:Fetching dashboard port number... 2024-03-24T00:11:46.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:46 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/301225095' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-24T00:11:46.821 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 8443 2024-03-24T00:11:46.821 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.017+0000 7f2e11566700 1 Processor -- start 2024-03-24T00:11:46.821 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.017+0000 7f2e11566700 1 -- start start 2024-03-24T00:11:46.821 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.018+0000 7f2e11566700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e0c107a20 0x7f2e0c107e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:46.821 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.018+0000 7f2e11566700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0c108410 con 0x7f2e0c107a20 2024-03-24T00:11:46.821 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.018+0000 7f2e0affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e0c107a20 0x7f2e0c107e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:46.821 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.018+0000 7f2e0affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e0c107a20 0x7f2e0c107e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50466/0 (socket says 172.21.15.78:50466) 2024-03-24T00:11:46.821 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.018+0000 7f2e0affd700 1 -- 172.21.15.78:0/1537105700 learned_addr learned my addr 172.21.15.78:0/1537105700 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:46.821 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.018+0000 7f2e0affd700 1 -- 172.21.15.78:0/1537105700 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e0c108bd0 con 0x7f2e0c107a20 2024-03-24T00:11:46.822 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.018+0000 7f2e0affd700 1 --2- 172.21.15.78:0/1537105700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e0c107a20 0x7f2e0c107e40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc0098a0 tx=0x7f2dfc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec69024e928c9ad3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:46.822 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.018+0000 7f2e09ffb700 1 -- 172.21.15.78:0/1537105700 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2dfc004160 con 0x7f2e0c107a20 2024-03-24T00:11:46.822 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.018+0000 7f2e09ffb700 1 -- 172.21.15.78:0/1537105700 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2dfc004300 con 0x7f2e0c107a20 2024-03-24T00:11:46.822 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.018+0000 7f2e09ffb700 1 -- 172.21.15.78:0/1537105700 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2dfc011b20 con 0x7f2e0c107a20 2024-03-24T00:11:46.822 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.019+0000 7f2e11566700 1 -- 172.21.15.78:0/1537105700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e0c107a20 msgr2=0x7f2e0c107e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:46.822 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.019+0000 7f2e11566700 1 --2- 172.21.15.78:0/1537105700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e0c107a20 0x7f2e0c107e40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc0098a0 tx=0x7f2dfc009bb0 comp rx=0 tx=0).stop 2024-03-24T00:11:46.822 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.019+0000 7f2e11566700 1 -- 172.21.15.78:0/1537105700 shutdown_connections 2024-03-24T00:11:46.822 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.019+0000 7f2e11566700 1 --2- 172.21.15.78:0/1537105700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e0c107a20 0x7f2e0c107e40 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:46.822 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.019+0000 7f2e11566700 1 -- 172.21.15.78:0/1537105700 >> 172.21.15.78:0/1537105700 conn(0x7f2e0c103020 msgr2=0x7f2e0c105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:46.822 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.019+0000 7f2e11566700 1 -- 172.21.15.78:0/1537105700 shutdown_connections 2024-03-24T00:11:46.823 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.019+0000 7f2e11566700 1 -- 172.21.15.78:0/1537105700 wait complete. 2024-03-24T00:11:46.823 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.019+0000 7f2e11566700 1 Processor -- start 2024-03-24T00:11:46.823 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.020+0000 7f2e11566700 1 -- start start 2024-03-24T00:11:46.823 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.020+0000 7f2e11566700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e0c107a20 0x7f2e0c19b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:46.823 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.020+0000 7f2e11566700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0c10bf80 con 0x7f2e0c107a20 2024-03-24T00:11:46.823 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.020+0000 7f2e0affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e0c107a20 0x7f2e0c19b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:46.823 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.020+0000 7f2e0affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e0c107a20 0x7f2e0c19b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50476/0 (socket says 172.21.15.78:50476) 2024-03-24T00:11:46.823 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.020+0000 7f2e0affd700 1 -- 172.21.15.78:0/301225095 learned_addr learned my addr 172.21.15.78:0/301225095 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:46.823 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.020+0000 7f2e0affd700 1 -- 172.21.15.78:0/301225095 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e0c19bee0 con 0x7f2e0c107a20 2024-03-24T00:11:46.823 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.020+0000 7f2e0affd700 1 --2- 172.21.15.78:0/301225095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e0c107a20 0x7f2e0c19b9a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc0116a0 tx=0x7f2dfc011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:46.823 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.020+0000 7f2e03fff700 1 -- 172.21.15.78:0/301225095 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2dfc003a40 con 0x7f2e0c107a20 2024-03-24T00:11:46.823 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.020+0000 7f2e03fff700 1 -- 172.21.15.78:0/301225095 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2dfc003be0 con 0x7f2e0c107a20 2024-03-24T00:11:46.824 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.020+0000 7f2e11566700 1 -- 172.21.15.78:0/301225095 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e0c19c170 con 0x7f2e0c107a20 2024-03-24T00:11:46.824 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.020+0000 7f2e03fff700 1 -- 172.21.15.78:0/301225095 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2dfc01a720 con 0x7f2e0c107a20 2024-03-24T00:11:46.824 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.021+0000 7f2e11566700 1 -- 172.21.15.78:0/301225095 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e0c19c5d0 con 0x7f2e0c107a20 2024-03-24T00:11:46.824 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.021+0000 7f2e03fff700 1 -- 172.21.15.78:0/301225095 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f2dfc01a9d0 con 0x7f2e0c107a20 2024-03-24T00:11:46.824 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.021+0000 7f2e11566700 1 -- 172.21.15.78:0/301225095 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e0c0637c0 con 0x7f2e0c107a20 2024-03-24T00:11:46.824 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.022+0000 7f2e03fff700 1 --2- 172.21.15.78:0/301225095 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2df8038f80 0x7f2df803b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:46.824 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.022+0000 7f2e03fff700 1 -- 172.21.15.78:0/301225095 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2dfc04d520 con 0x7f2e0c107a20 2024-03-24T00:11:46.824 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.022+0000 7f2e0a7fc700 1 --2- 172.21.15.78:0/301225095 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2df8038f80 0x7f2df803b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:46.824 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.022+0000 7f2e0a7fc700 1 --2- 172.21.15.78:0/301225095 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2df8038f80 0x7f2df803b440 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2df4006fd0 tx=0x7f2df4006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:46.824 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.026+0000 7f2e03fff700 1 -- 172.21.15.78:0/301225095 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2dfc011e20 con 0x7f2e0c107a20 2024-03-24T00:11:46.825 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.224+0000 7f2e11566700 1 -- 172.21.15.78:0/301225095 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f2e0c19efa0 con 0x7f2e0c107a20 2024-03-24T00:11:46.825 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.224+0000 7f2e03fff700 1 -- 172.21.15.78:0/301225095 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f2dfc01adc0 con 0x7f2e0c107a20 2024-03-24T00:11:46.825 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.226+0000 7f2e11566700 1 -- 172.21.15.78:0/301225095 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2df8038f80 msgr2=0x7f2df803b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:46.825 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.226+0000 7f2e11566700 1 --2- 172.21.15.78:0/301225095 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2df8038f80 0x7f2df803b440 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2df4006fd0 tx=0x7f2df4006e40 comp rx=0 tx=0).stop 2024-03-24T00:11:46.825 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.226+0000 7f2e11566700 1 -- 172.21.15.78:0/301225095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e0c107a20 msgr2=0x7f2e0c19b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:46.825 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.226+0000 7f2e11566700 1 --2- 172.21.15.78:0/301225095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e0c107a20 0x7f2e0c19b9a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc0116a0 tx=0x7f2dfc011ba0 comp rx=0 tx=0).stop 2024-03-24T00:11:46.825 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.226+0000 7f2e11566700 1 -- 172.21.15.78:0/301225095 shutdown_connections 2024-03-24T00:11:46.825 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.226+0000 7f2e11566700 1 --2- 172.21.15.78:0/301225095 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2df8038f80 0x7f2df803b440 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:46.825 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.226+0000 7f2e11566700 1 --2- 172.21.15.78:0/301225095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e0c107a20 0x7f2e0c19b9a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:46.825 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.226+0000 7f2e11566700 1 -- 172.21.15.78:0/301225095 >> 172.21.15.78:0/301225095 conn(0x7f2e0c103020 msgr2=0x7f2e0c103b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:46.825 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.226+0000 7f2e11566700 1 -- 172.21.15.78:0/301225095 shutdown_connections 2024-03-24T00:11:46.825 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:46.226+0000 7f2e11566700 1 -- 172.21.15.78:0/301225095 wait complete. 2024-03-24T00:11:46.827 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T00:11:46.827 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout disabled 2024-03-24T00:11:46.834 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T00:11:46.834 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2024-03-24T00:11:46.834 INFO:teuthology.orchestra.run.smithi078.stdout:firewalld.service is not enabled 2024-03-24T00:11:46.835 INFO:teuthology.orchestra.run.smithi078.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-24T00:11:46.836 INFO:teuthology.orchestra.run.smithi078.stdout:Ceph Dashboard is now available at: 2024-03-24T00:11:46.836 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:11:46.836 INFO:teuthology.orchestra.run.smithi078.stdout: URL: https://smithi078.front.sepia.ceph.com:8443/ 2024-03-24T00:11:46.836 INFO:teuthology.orchestra.run.smithi078.stdout: User: admin 2024-03-24T00:11:46.836 INFO:teuthology.orchestra.run.smithi078.stdout: Password: sd11b9ejy9 2024-03-24T00:11:46.836 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:11:46.837 INFO:teuthology.orchestra.run.smithi078.stdout:Saving cluster configuration to /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config directory 2024-03-24T00:11:46.837 INFO:teuthology.orchestra.run.smithi078.stdout:Enabling autotune for osd_memory_target 2024-03-24T00:11:47.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:47 smithi078 ceph-mon[32429]: mgrmap e13: x(active, since 6s) 2024-03-24T00:11:47.968 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.268+0000 7f7101838700 1 Processor -- start 2024-03-24T00:11:47.968 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.268+0000 7f7101838700 1 -- start start 2024-03-24T00:11:47.968 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.269+0000 7f7101838700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70fc108440 0x7f70fc10a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:47.968 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.269+0000 7f7101838700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70fc07aed0 con 0x7f70fc108440 2024-03-24T00:11:47.968 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.269+0000 7f70faffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70fc108440 0x7f70fc10a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:47.968 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.269+0000 7f70faffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70fc108440 0x7f70fc10a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50490/0 (socket says 172.21.15.78:50490) 2024-03-24T00:11:47.968 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.269+0000 7f70faffd700 1 -- 172.21.15.78:0/3285157955 learned_addr learned my addr 172.21.15.78:0/3285157955 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:47.968 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.269+0000 7f70faffd700 1 -- 172.21.15.78:0/3285157955 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70fc10ad70 con 0x7f70fc108440 2024-03-24T00:11:47.968 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.269+0000 7f70faffd700 1 --2- 172.21.15.78:0/3285157955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70fc108440 0x7f70fc10a830 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f70ec0098a0 tx=0x7f70ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45063fc0266efae6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:47.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.269+0000 7f70fa7fc700 1 -- 172.21.15.78:0/3285157955 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f70ec004160 con 0x7f70fc108440 2024-03-24T00:11:47.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.269+0000 7f70fa7fc700 1 -- 172.21.15.78:0/3285157955 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f70ec004300 con 0x7f70fc108440 2024-03-24T00:11:47.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.269+0000 7f70fa7fc700 1 -- 172.21.15.78:0/3285157955 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f70ec011b20 con 0x7f70fc108440 2024-03-24T00:11:47.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.270+0000 7f7101838700 1 -- 172.21.15.78:0/3285157955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70fc108440 msgr2=0x7f70fc10a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:47.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.270+0000 7f7101838700 1 --2- 172.21.15.78:0/3285157955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70fc108440 0x7f70fc10a830 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f70ec0098a0 tx=0x7f70ec009bb0 comp rx=0 tx=0).stop 2024-03-24T00:11:47.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.270+0000 7f7101838700 1 -- 172.21.15.78:0/3285157955 shutdown_connections 2024-03-24T00:11:47.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.270+0000 7f7101838700 1 --2- 172.21.15.78:0/3285157955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70fc108440 0x7f70fc10a830 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:47.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.270+0000 7f7101838700 1 -- 172.21.15.78:0/3285157955 >> 172.21.15.78:0/3285157955 conn(0x7f70fc101f10 msgr2=0x7f70fc104330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:47.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.270+0000 7f7101838700 1 -- 172.21.15.78:0/3285157955 shutdown_connections 2024-03-24T00:11:47.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.270+0000 7f7101838700 1 -- 172.21.15.78:0/3285157955 wait complete. 2024-03-24T00:11:47.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.270+0000 7f7101838700 1 Processor -- start 2024-03-24T00:11:47.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.271+0000 7f7101838700 1 -- start start 2024-03-24T00:11:47.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.271+0000 7f7101838700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70fc108440 0x7f70fc107cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:47.969 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.271+0000 7f7101838700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70fc10bde0 con 0x7f70fc108440 2024-03-24T00:11:47.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.271+0000 7f70faffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70fc108440 0x7f70fc107cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:47.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.271+0000 7f70faffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70fc108440 0x7f70fc107cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50506/0 (socket says 172.21.15.78:50506) 2024-03-24T00:11:47.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.271+0000 7f70faffd700 1 -- 172.21.15.78:0/1180385047 learned_addr learned my addr 172.21.15.78:0/1180385047 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:47.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.271+0000 7f70faffd700 1 -- 172.21.15.78:0/1180385047 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70fc108200 con 0x7f70fc108440 2024-03-24T00:11:47.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.271+0000 7f70faffd700 1 --2- 172.21.15.78:0/1180385047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70fc108440 0x7f70fc107cc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f70ec011b60 tx=0x7f70ec011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:47.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.271+0000 7f7100836700 1 -- 172.21.15.78:0/1180385047 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f70ec003b60 con 0x7f70fc108440 2024-03-24T00:11:47.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.272+0000 7f7100836700 1 -- 172.21.15.78:0/1180385047 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f70ec01a910 con 0x7f70fc108440 2024-03-24T00:11:47.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.272+0000 7f7100836700 1 -- 172.21.15.78:0/1180385047 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f70ec024930 con 0x7f70fc108440 2024-03-24T00:11:47.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.272+0000 7f7101838700 1 -- 172.21.15.78:0/1180385047 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70fc1063f0 con 0x7f70fc108440 2024-03-24T00:11:47.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.272+0000 7f7101838700 1 -- 172.21.15.78:0/1180385047 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70fc1068b0 con 0x7f70fc108440 2024-03-24T00:11:47.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.272+0000 7f7100836700 1 -- 172.21.15.78:0/1180385047 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f70ec022020 con 0x7f70fc108440 2024-03-24T00:11:47.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.272+0000 7f7100836700 1 --2- 172.21.15.78:0/1180385047 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f70e8039070 0x7f70e803b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:47.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.273+0000 7f7100836700 1 -- 172.21.15.78:0/1180385047 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f70ec04d2d0 con 0x7f70fc108440 2024-03-24T00:11:47.970 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.273+0000 7f70f27fc700 1 --2- 172.21.15.78:0/1180385047 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f70e8039070 0x7f70e803b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:47.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.273+0000 7f7101838700 1 -- 172.21.15.78:0/1180385047 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70fc0637c0 con 0x7f70fc108440 2024-03-24T00:11:47.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.273+0000 7f70f27fc700 1 --2- 172.21.15.78:0/1180385047 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f70e8039070 0x7f70e803b530 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f70e4006fd0 tx=0x7f70e4006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:47.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.278+0000 7f7100836700 1 -- 172.21.15.78:0/1180385047 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f70ec00bc70 con 0x7f70fc108440 2024-03-24T00:11:47.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.472+0000 7f7101838700 1 -- 172.21.15.78:0/1180385047 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f70fc106b30 con 0x7f70fc108440 2024-03-24T00:11:47.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.472+0000 7f7100836700 1 -- 172.21.15.78:0/1180385047 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f70ec02d770 con 0x7f70fc108440 2024-03-24T00:11:47.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.474+0000 7f7101838700 1 -- 172.21.15.78:0/1180385047 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f70e8039070 msgr2=0x7f70e803b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:47.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.474+0000 7f7101838700 1 --2- 172.21.15.78:0/1180385047 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f70e8039070 0x7f70e803b530 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f70e4006fd0 tx=0x7f70e4006e40 comp rx=0 tx=0).stop 2024-03-24T00:11:47.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.474+0000 7f7101838700 1 -- 172.21.15.78:0/1180385047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70fc108440 msgr2=0x7f70fc107cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:47.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.474+0000 7f7101838700 1 --2- 172.21.15.78:0/1180385047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70fc108440 0x7f70fc107cc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f70ec011b60 tx=0x7f70ec011c40 comp rx=0 tx=0).stop 2024-03-24T00:11:47.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.474+0000 7f7101838700 1 -- 172.21.15.78:0/1180385047 shutdown_connections 2024-03-24T00:11:47.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.474+0000 7f7101838700 1 --2- 172.21.15.78:0/1180385047 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f70e8039070 0x7f70e803b530 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:47.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.474+0000 7f7101838700 1 --2- 172.21.15.78:0/1180385047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70fc108440 0x7f70fc107cc0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:47.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.474+0000 7f7101838700 1 -- 172.21.15.78:0/1180385047 >> 172.21.15.78:0/1180385047 conn(0x7f70fc101f10 msgr2=0x7f70fc102a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:47.971 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.474+0000 7f7101838700 1 -- 172.21.15.78:0/1180385047 shutdown_connections 2024-03-24T00:11:47.972 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:47.474+0000 7f7101838700 1 -- 172.21.15.78:0/1180385047 wait complete. 2024-03-24T00:11:49.308 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.459+0000 7f257e087700 1 Processor -- start 2024-03-24T00:11:49.308 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.459+0000 7f257e087700 1 -- start start 2024-03-24T00:11:49.308 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.460+0000 7f257e087700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25781068d0 0x7f2578106cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:49.308 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.460+0000 7f257e087700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25781072c0 con 0x7f25781068d0 2024-03-24T00:11:49.308 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.460+0000 7f25777fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25781068d0 0x7f2578106cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:49.308 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.460+0000 7f25777fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25781068d0 0x7f2578106cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50510/0 (socket says 172.21.15.78:50510) 2024-03-24T00:11:49.308 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.460+0000 7f25777fe700 1 -- 172.21.15.78:0/3801736885 learned_addr learned my addr 172.21.15.78:0/3801736885 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:49.309 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.460+0000 7f25777fe700 1 -- 172.21.15.78:0/3801736885 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2578107ad0 con 0x7f25781068d0 2024-03-24T00:11:49.309 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.460+0000 7f25777fe700 1 --2- 172.21.15.78:0/3801736885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25781068d0 0x7f2578106cf0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2568009e80 tx=0x7f256800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf7dbce7d206b863 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:49.309 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.460+0000 7f25767fc700 1 -- 172.21.15.78:0/3801736885 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2568017070 con 0x7f25781068d0 2024-03-24T00:11:49.309 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.460+0000 7f25767fc700 1 -- 172.21.15.78:0/3801736885 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f256800b8c0 con 0x7f25781068d0 2024-03-24T00:11:49.309 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.460+0000 7f25767fc700 1 -- 172.21.15.78:0/3801736885 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f25680045c0 con 0x7f25781068d0 2024-03-24T00:11:49.309 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.461+0000 7f257e087700 1 -- 172.21.15.78:0/3801736885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25781068d0 msgr2=0x7f2578106cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:49.309 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.461+0000 7f257e087700 1 --2- 172.21.15.78:0/3801736885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25781068d0 0x7f2578106cf0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2568009e80 tx=0x7f256800b330 comp rx=0 tx=0).stop 2024-03-24T00:11:49.309 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.461+0000 7f257e087700 1 -- 172.21.15.78:0/3801736885 shutdown_connections 2024-03-24T00:11:49.309 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.461+0000 7f257e087700 1 --2- 172.21.15.78:0/3801736885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25781068d0 0x7f2578106cf0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:49.309 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.461+0000 7f257e087700 1 -- 172.21.15.78:0/3801736885 >> 172.21.15.78:0/3801736885 conn(0x7f2578101f10 msgr2=0x7f2578104330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:49.309 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.461+0000 7f257e087700 1 -- 172.21.15.78:0/3801736885 shutdown_connections 2024-03-24T00:11:49.309 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.461+0000 7f257e087700 1 -- 172.21.15.78:0/3801736885 wait complete. 2024-03-24T00:11:49.309 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.462+0000 7f257e087700 1 Processor -- start 2024-03-24T00:11:49.309 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.462+0000 7f257e087700 1 -- start start 2024-03-24T00:11:49.310 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.462+0000 7f257e087700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25781068d0 0x7f257819a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:49.310 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.462+0000 7f257e087700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257810ae80 con 0x7f25781068d0 2024-03-24T00:11:49.310 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.462+0000 7f25777fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25781068d0 0x7f257819a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:49.310 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.462+0000 7f25777fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25781068d0 0x7f257819a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50526/0 (socket says 172.21.15.78:50526) 2024-03-24T00:11:49.310 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.462+0000 7f25777fe700 1 -- 172.21.15.78:0/1244484903 learned_addr learned my addr 172.21.15.78:0/1244484903 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:49.310 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.462+0000 7f25777fe700 1 -- 172.21.15.78:0/1244484903 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f257819ae00 con 0x7f25781068d0 2024-03-24T00:11:49.310 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.462+0000 7f25777fe700 1 --2- 172.21.15.78:0/1244484903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25781068d0 0x7f257819a8c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2568009530 tx=0x7f2568003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:49.310 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.463+0000 7f2574ff9700 1 -- 172.21.15.78:0/1244484903 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2568020020 con 0x7f25781068d0 2024-03-24T00:11:49.310 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.463+0000 7f2574ff9700 1 -- 172.21.15.78:0/1244484903 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f256800bba0 con 0x7f25781068d0 2024-03-24T00:11:49.310 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.463+0000 7f2574ff9700 1 -- 172.21.15.78:0/1244484903 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2568017040 con 0x7f25781068d0 2024-03-24T00:11:49.310 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.463+0000 7f257e087700 1 -- 172.21.15.78:0/1244484903 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f257819b090 con 0x7f25781068d0 2024-03-24T00:11:49.310 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.463+0000 7f257e087700 1 -- 172.21.15.78:0/1244484903 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f257819dd80 con 0x7f25781068d0 2024-03-24T00:11:49.310 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.463+0000 7f2574ff9700 1 -- 172.21.15.78:0/1244484903 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f2568023020 con 0x7f25781068d0 2024-03-24T00:11:49.311 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.464+0000 7f257e087700 1 -- 172.21.15.78:0/1244484903 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25780637c0 con 0x7f25781068d0 2024-03-24T00:11:49.311 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.464+0000 7f2574ff9700 1 --2- 172.21.15.78:0/1244484903 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2560039070 0x7f256003b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:49.311 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.464+0000 7f2574ff9700 1 -- 172.21.15.78:0/1244484903 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f256800f050 con 0x7f25781068d0 2024-03-24T00:11:49.311 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.464+0000 7f2576ffd700 1 --2- 172.21.15.78:0/1244484903 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2560039070 0x7f256003b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:49.311 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.464+0000 7f2576ffd700 1 --2- 172.21.15.78:0/1244484903 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2560039070 0x7f256003b530 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f256c006fd0 tx=0x7f256c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:49.311 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.469+0000 7f2574ff9700 1 -- 172.21.15.78:0/1244484903 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f256802a020 con 0x7f25781068d0 2024-03-24T00:11:49.311 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.776+0000 7f257e087700 1 -- 172.21.15.78:0/1244484903 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f257819e4e0 con 0x7f25781068d0 2024-03-24T00:11:49.311 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.779+0000 7f2574ff9700 1 -- 172.21.15.78:0/1244484903 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) v1 ==== 153+0+0 (secure 0 0 0) 0x7f2568012430 con 0x7f25781068d0 2024-03-24T00:11:49.311 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.781+0000 7f257e087700 1 -- 172.21.15.78:0/1244484903 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2560039070 msgr2=0x7f256003b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:49.311 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.781+0000 7f257e087700 1 --2- 172.21.15.78:0/1244484903 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2560039070 0x7f256003b530 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f256c006fd0 tx=0x7f256c006e40 comp rx=0 tx=0).stop 2024-03-24T00:11:49.311 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.781+0000 7f257e087700 1 -- 172.21.15.78:0/1244484903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25781068d0 msgr2=0x7f257819a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:49.311 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.781+0000 7f257e087700 1 --2- 172.21.15.78:0/1244484903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25781068d0 0x7f257819a8c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2568009530 tx=0x7f2568003d00 comp rx=0 tx=0).stop 2024-03-24T00:11:49.311 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.781+0000 7f257e087700 1 -- 172.21.15.78:0/1244484903 shutdown_connections 2024-03-24T00:11:49.311 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.781+0000 7f257e087700 1 --2- 172.21.15.78:0/1244484903 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2560039070 0x7f256003b530 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:49.312 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.781+0000 7f257e087700 1 --2- 172.21.15.78:0/1244484903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25781068d0 0x7f257819a8c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:49.312 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.781+0000 7f257e087700 1 -- 172.21.15.78:0/1244484903 >> 172.21.15.78:0/1244484903 conn(0x7f2578101f10 msgr2=0x7f2578102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:49.312 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.781+0000 7f257e087700 1 -- 172.21.15.78:0/1244484903 shutdown_connections 2024-03-24T00:11:49.312 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-24T00:11:48.781+0000 7f257e087700 1 -- 172.21.15.78:0/1244484903 wait complete. 2024-03-24T00:11:49.312 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-24T00:11:49.312 INFO:teuthology.orchestra.run.smithi078.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-24T00:11:49.312 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:11:49.312 INFO:teuthology.orchestra.run.smithi078.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-24T00:11:49.312 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:11:49.312 INFO:teuthology.orchestra.run.smithi078.stdout:Or, if you are only running a single cluster on this host: 2024-03-24T00:11:49.312 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:11:49.312 INFO:teuthology.orchestra.run.smithi078.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-24T00:11:49.312 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:11:49.312 INFO:teuthology.orchestra.run.smithi078.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-24T00:11:49.312 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:11:49.313 INFO:teuthology.orchestra.run.smithi078.stdout: ceph telemetry on 2024-03-24T00:11:49.313 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:11:49.313 INFO:teuthology.orchestra.run.smithi078.stdout:For more information see: 2024-03-24T00:11:49.313 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:11:49.313 INFO:teuthology.orchestra.run.smithi078.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-24T00:11:49.313 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:11:49.313 INFO:teuthology.orchestra.run.smithi078.stdout:Bootstrap complete. 2024-03-24T00:11:49.346 INFO:tasks.cephadm:Fetching config... 2024-03-24T00:11:49.346 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-24T00:11:49.346 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-24T00:11:49.361 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-24T00:11:49.362 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-24T00:11:49.362 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-24T00:11:49.416 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-24T00:11:49.416 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-24T00:11:49.417 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-24T00:11:49.484 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-24T00:11:49.484 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-24T00:11:49.484 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-24T00:11:49.541 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-24T00:11:49.541 DEBUG:teuthology.orchestra.run.smithi078:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQjw9fCTKvxEubrnnnAOWAiSBPw7fwWwXV81MRe//tb6Dr2lrzwC5WrQ2TB0e+DPnXDCFTJUKYIczy422eCtWu7rX03OP6LtVtem7IG9Qgdek6LP0++SpexObhyC3oRsw1p/XHGQKt4C1khhOd9QJke2jGSNXDmuPIB/AQq6kuZMGCeWsYtvANDygnQsk1vtuNMmdIq+RFnn3R1cutcHFDRU0/8wdEu3Dt9QoBJ3tag3onZxblPA26dEy8OLaMR4qGBzPv9aUyWxzA+vzMXAVYrksW4QMTnA8vK9VP9mPxCnyImtjhVj5gGFaz3dVWOTTDKrUzIC9VUAirSyo7/0hA9pmCbMbToD8eW85G11eRjMJIEe3IEXKHgKUgknBzCevF4Jh9O2OahmZ/bSik+l7TQwJqPUshMJePIyv7RA+Y1gSTLP+PTBvmMIVfZ39ivoeQ52Cf0qWSBASLH/xWJQnI7b8p+b5SAbYTM+VEB8bLc7kVYu5tZJJRV/C62RRqWAM= ceph-c7e2806a-e972-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T00:11:49.621 INFO:teuthology.orchestra.run.smithi078.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQjw9fCTKvxEubrnnnAOWAiSBPw7fwWwXV81MRe//tb6Dr2lrzwC5WrQ2TB0e+DPnXDCFTJUKYIczy422eCtWu7rX03OP6LtVtem7IG9Qgdek6LP0++SpexObhyC3oRsw1p/XHGQKt4C1khhOd9QJke2jGSNXDmuPIB/AQq6kuZMGCeWsYtvANDygnQsk1vtuNMmdIq+RFnn3R1cutcHFDRU0/8wdEu3Dt9QoBJ3tag3onZxblPA26dEy8OLaMR4qGBzPv9aUyWxzA+vzMXAVYrksW4QMTnA8vK9VP9mPxCnyImtjhVj5gGFaz3dVWOTTDKrUzIC9VUAirSyo7/0hA9pmCbMbToD8eW85G11eRjMJIEe3IEXKHgKUgknBzCevF4Jh9O2OahmZ/bSik+l7TQwJqPUshMJePIyv7RA+Y1gSTLP+PTBvmMIVfZ39ivoeQ52Cf0qWSBASLH/xWJQnI7b8p+b5SAbYTM+VEB8bLc7kVYu5tZJJRV/C62RRqWAM= ceph-c7e2806a-e972-11ee-95cd-87774f69a715 2024-03-24T00:11:49.640 DEBUG:teuthology.orchestra.run.smithi106:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQjw9fCTKvxEubrnnnAOWAiSBPw7fwWwXV81MRe//tb6Dr2lrzwC5WrQ2TB0e+DPnXDCFTJUKYIczy422eCtWu7rX03OP6LtVtem7IG9Qgdek6LP0++SpexObhyC3oRsw1p/XHGQKt4C1khhOd9QJke2jGSNXDmuPIB/AQq6kuZMGCeWsYtvANDygnQsk1vtuNMmdIq+RFnn3R1cutcHFDRU0/8wdEu3Dt9QoBJ3tag3onZxblPA26dEy8OLaMR4qGBzPv9aUyWxzA+vzMXAVYrksW4QMTnA8vK9VP9mPxCnyImtjhVj5gGFaz3dVWOTTDKrUzIC9VUAirSyo7/0hA9pmCbMbToD8eW85G11eRjMJIEe3IEXKHgKUgknBzCevF4Jh9O2OahmZ/bSik+l7TQwJqPUshMJePIyv7RA+Y1gSTLP+PTBvmMIVfZ39ivoeQ52Cf0qWSBASLH/xWJQnI7b8p+b5SAbYTM+VEB8bLc7kVYu5tZJJRV/C62RRqWAM= ceph-c7e2806a-e972-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T00:11:49.685 INFO:teuthology.orchestra.run.smithi106.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQjw9fCTKvxEubrnnnAOWAiSBPw7fwWwXV81MRe//tb6Dr2lrzwC5WrQ2TB0e+DPnXDCFTJUKYIczy422eCtWu7rX03OP6LtVtem7IG9Qgdek6LP0++SpexObhyC3oRsw1p/XHGQKt4C1khhOd9QJke2jGSNXDmuPIB/AQq6kuZMGCeWsYtvANDygnQsk1vtuNMmdIq+RFnn3R1cutcHFDRU0/8wdEu3Dt9QoBJ3tag3onZxblPA26dEy8OLaMR4qGBzPv9aUyWxzA+vzMXAVYrksW4QMTnA8vK9VP9mPxCnyImtjhVj5gGFaz3dVWOTTDKrUzIC9VUAirSyo7/0hA9pmCbMbToD8eW85G11eRjMJIEe3IEXKHgKUgknBzCevF4Jh9O2OahmZ/bSik+l7TQwJqPUshMJePIyv7RA+Y1gSTLP+PTBvmMIVfZ39ivoeQ52Cf0qWSBASLH/xWJQnI7b8p+b5SAbYTM+VEB8bLc7kVYu5tZJJRV/C62RRqWAM= ceph-c7e2806a-e972-11ee-95cd-87774f69a715 2024-03-24T00:11:49.700 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQjw9fCTKvxEubrnnnAOWAiSBPw7fwWwXV81MRe//tb6Dr2lrzwC5WrQ2TB0e+DPnXDCFTJUKYIczy422eCtWu7rX03OP6LtVtem7IG9Qgdek6LP0++SpexObhyC3oRsw1p/XHGQKt4C1khhOd9QJke2jGSNXDmuPIB/AQq6kuZMGCeWsYtvANDygnQsk1vtuNMmdIq+RFnn3R1cutcHFDRU0/8wdEu3Dt9QoBJ3tag3onZxblPA26dEy8OLaMR4qGBzPv9aUyWxzA+vzMXAVYrksW4QMTnA8vK9VP9mPxCnyImtjhVj5gGFaz3dVWOTTDKrUzIC9VUAirSyo7/0hA9pmCbMbToD8eW85G11eRjMJIEe3IEXKHgKUgknBzCevF4Jh9O2OahmZ/bSik+l7TQwJqPUshMJePIyv7RA+Y1gSTLP+PTBvmMIVfZ39ivoeQ52Cf0qWSBASLH/xWJQnI7b8p+b5SAbYTM+VEB8bLc7kVYu5tZJJRV/C62RRqWAM= ceph-c7e2806a-e972-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T00:11:49.746 INFO:teuthology.orchestra.run.smithi107.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQjw9fCTKvxEubrnnnAOWAiSBPw7fwWwXV81MRe//tb6Dr2lrzwC5WrQ2TB0e+DPnXDCFTJUKYIczy422eCtWu7rX03OP6LtVtem7IG9Qgdek6LP0++SpexObhyC3oRsw1p/XHGQKt4C1khhOd9QJke2jGSNXDmuPIB/AQq6kuZMGCeWsYtvANDygnQsk1vtuNMmdIq+RFnn3R1cutcHFDRU0/8wdEu3Dt9QoBJ3tag3onZxblPA26dEy8OLaMR4qGBzPv9aUyWxzA+vzMXAVYrksW4QMTnA8vK9VP9mPxCnyImtjhVj5gGFaz3dVWOTTDKrUzIC9VUAirSyo7/0hA9pmCbMbToD8eW85G11eRjMJIEe3IEXKHgKUgknBzCevF4Jh9O2OahmZ/bSik+l7TQwJqPUshMJePIyv7RA+Y1gSTLP+PTBvmMIVfZ39ivoeQ52Cf0qWSBASLH/xWJQnI7b8p+b5SAbYTM+VEB8bLc7kVYu5tZJJRV/C62RRqWAM= ceph-c7e2806a-e972-11ee-95cd-87774f69a715 2024-03-24T00:11:49.761 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-24T00:11:50.001 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:11:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:49 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1244484903' entity='client.admin' 2024-03-24T00:11:52.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.128+0000 7f2df9f0b700 1 -- 172.21.15.78:0/3677972215 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df4100ac0 msgr2=0x7f2df4100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:52.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.128+0000 7f2df9f0b700 1 --2- 172.21.15.78:0/3677972215 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df4100ac0 0x7f2df4100ee0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2de4009910 tx=0x7f2de4009c20 comp rx=0 tx=0).stop 2024-03-24T00:11:52.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.128+0000 7f2df9f0b700 1 -- 172.21.15.78:0/3677972215 shutdown_connections 2024-03-24T00:11:52.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.128+0000 7f2df9f0b700 1 --2- 172.21.15.78:0/3677972215 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df4100ac0 0x7f2df4100ee0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:52.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.128+0000 7f2df9f0b700 1 -- 172.21.15.78:0/3677972215 >> 172.21.15.78:0/3677972215 conn(0x7f2df40fc060 msgr2=0x7f2df40fe4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:52.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.128+0000 7f2df9f0b700 1 -- 172.21.15.78:0/3677972215 shutdown_connections 2024-03-24T00:11:52.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.128+0000 7f2df9f0b700 1 -- 172.21.15.78:0/3677972215 wait complete. 2024-03-24T00:11:52.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.129+0000 7f2df9f0b700 1 Processor -- start 2024-03-24T00:11:52.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.129+0000 7f2df9f0b700 1 -- start start 2024-03-24T00:11:52.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.129+0000 7f2df9f0b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df4100ac0 0x7f2df41972f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:52.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.129+0000 7f2df9f0b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df4106b50 con 0x7f2df4100ac0 2024-03-24T00:11:52.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.129+0000 7f2df37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df4100ac0 0x7f2df41972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:52.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.129+0000 7f2df37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df4100ac0 0x7f2df41972f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50550/0 (socket says 172.21.15.78:50550) 2024-03-24T00:11:52.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.129+0000 7f2df37fe700 1 -- 172.21.15.78:0/1062172142 learned_addr learned my addr 172.21.15.78:0/1062172142 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:52.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.129+0000 7f2df37fe700 1 -- 172.21.15.78:0/1062172142 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df4197830 con 0x7f2df4100ac0 2024-03-24T00:11:52.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.129+0000 7f2df37fe700 1 --2- 172.21.15.78:0/1062172142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df4100ac0 0x7f2df41972f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2de4006010 tx=0x7f2de4005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:52.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.130+0000 7f2df0ff9700 1 -- 172.21.15.78:0/1062172142 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2de4027070 con 0x7f2df4100ac0 2024-03-24T00:11:52.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.130+0000 7f2df0ff9700 1 -- 172.21.15.78:0/1062172142 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2de400be40 con 0x7f2df4100ac0 2024-03-24T00:11:52.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.130+0000 7f2df9f0b700 1 -- 172.21.15.78:0/1062172142 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2df4197ac0 con 0x7f2df4100ac0 2024-03-24T00:11:52.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.130+0000 7f2df0ff9700 1 -- 172.21.15.78:0/1062172142 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2de400f460 con 0x7f2df4100ac0 2024-03-24T00:11:52.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.130+0000 7f2df9f0b700 1 -- 172.21.15.78:0/1062172142 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2df419a7b0 con 0x7f2df4100ac0 2024-03-24T00:11:52.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.131+0000 7f2df0ff9700 1 -- 172.21.15.78:0/1062172142 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f2de402a070 con 0x7f2df4100ac0 2024-03-24T00:11:52.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.131+0000 7f2df9f0b700 1 -- 172.21.15.78:0/1062172142 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2df41072d0 con 0x7f2df4100ac0 2024-03-24T00:11:52.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.131+0000 7f2df0ff9700 1 --2- 172.21.15.78:0/1062172142 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2ddc039020 0x7f2ddc03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:52.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.131+0000 7f2df0ff9700 1 -- 172.21.15.78:0/1062172142 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2de401e030 con 0x7f2df4100ac0 2024-03-24T00:11:52.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.131+0000 7f2df2ffd700 1 --2- 172.21.15.78:0/1062172142 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2ddc039020 0x7f2ddc03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:52.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.131+0000 7f2df2ffd700 1 --2- 172.21.15.78:0/1062172142 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2ddc039020 0x7f2ddc03b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2de8006fd0 tx=0x7f2de8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:52.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.136+0000 7f2df0ff9700 1 -- 172.21.15.78:0/1062172142 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2de4013070 con 0x7f2df4100ac0 2024-03-24T00:11:52.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.332+0000 7f2df9f0b700 1 -- 172.21.15.78:0/1062172142 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f2df419abb0 con 0x7f2df4100ac0 2024-03-24T00:11:52.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.335+0000 7f2df0ff9700 1 -- 172.21.15.78:0/1062172142 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f2de40310c0 con 0x7f2df4100ac0 2024-03-24T00:11:52.334 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:52 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:11:52.335 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:52 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:11:52.335 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:52 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi078", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:11:52.335 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:52 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:11:52.335 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:52 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:11:52.335 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:52 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:11:52.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.337+0000 7f2df9f0b700 1 -- 172.21.15.78:0/1062172142 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2ddc039020 msgr2=0x7f2ddc03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:52.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.337+0000 7f2df9f0b700 1 --2- 172.21.15.78:0/1062172142 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2ddc039020 0x7f2ddc03b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2de8006fd0 tx=0x7f2de8006e40 comp rx=0 tx=0).stop 2024-03-24T00:11:52.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.337+0000 7f2df9f0b700 1 -- 172.21.15.78:0/1062172142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df4100ac0 msgr2=0x7f2df41972f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:52.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.337+0000 7f2df9f0b700 1 --2- 172.21.15.78:0/1062172142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df4100ac0 0x7f2df41972f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2de4006010 tx=0x7f2de4005050 comp rx=0 tx=0).stop 2024-03-24T00:11:52.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.337+0000 7f2df9f0b700 1 -- 172.21.15.78:0/1062172142 shutdown_connections 2024-03-24T00:11:52.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.337+0000 7f2df9f0b700 1 --2- 172.21.15.78:0/1062172142 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2ddc039020 0x7f2ddc03b4e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:52.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.337+0000 7f2df9f0b700 1 --2- 172.21.15.78:0/1062172142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df4100ac0 0x7f2df41972f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:52.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.337+0000 7f2df9f0b700 1 -- 172.21.15.78:0/1062172142 >> 172.21.15.78:0/1062172142 conn(0x7f2df40fc060 msgr2=0x7f2df4069580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:52.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.337+0000 7f2df9f0b700 1 -- 172.21.15.78:0/1062172142 shutdown_connections 2024-03-24T00:11:52.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:52.337+0000 7f2df9f0b700 1 -- 172.21.15.78:0/1062172142 wait complete. 2024-03-24T00:11:52.864 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-24T00:11:52.865 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-24T00:11:53.098 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:11:53.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:53 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1062172142' entity='client.admin' 2024-03-24T00:11:53.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:53 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:11:53.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:53 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:11:54.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.436+0000 7f455f55a700 1 -- 172.21.15.78:0/1729145053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4558100280 msgr2=0x7f45581006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:54.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.436+0000 7f455f55a700 1 --2- 172.21.15.78:0/1729145053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4558100280 0x7f45581006a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f455400aa20 tx=0x7f455400ad30 comp rx=0 tx=0).stop 2024-03-24T00:11:54.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.436+0000 7f455f55a700 1 -- 172.21.15.78:0/1729145053 shutdown_connections 2024-03-24T00:11:54.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.436+0000 7f455f55a700 1 --2- 172.21.15.78:0/1729145053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4558100280 0x7f45581006a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:54.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.436+0000 7f455f55a700 1 -- 172.21.15.78:0/1729145053 >> 172.21.15.78:0/1729145053 conn(0x7f45580fb840 msgr2=0x7f45580fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:54.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.436+0000 7f455f55a700 1 -- 172.21.15.78:0/1729145053 shutdown_connections 2024-03-24T00:11:54.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.436+0000 7f455f55a700 1 -- 172.21.15.78:0/1729145053 wait complete. 2024-03-24T00:11:54.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.437+0000 7f455f55a700 1 Processor -- start 2024-03-24T00:11:54.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.437+0000 7f455f55a700 1 -- start start 2024-03-24T00:11:54.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.437+0000 7f455f55a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4558100280 0x7f45581950d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:54.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.437+0000 7f455f55a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4558105a90 con 0x7f4558100280 2024-03-24T00:11:54.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.437+0000 7f455d2f6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4558100280 0x7f45581950d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:54.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.437+0000 7f455d2f6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4558100280 0x7f45581950d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50570/0 (socket says 172.21.15.78:50570) 2024-03-24T00:11:54.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.437+0000 7f455d2f6700 1 -- 172.21.15.78:0/1346894970 learned_addr learned my addr 172.21.15.78:0/1346894970 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:54.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.437+0000 7f455d2f6700 1 -- 172.21.15.78:0/1346894970 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4558195610 con 0x7f4558100280 2024-03-24T00:11:54.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.437+0000 7f455d2f6700 1 --2- 172.21.15.78:0/1346894970 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4558100280 0x7f45581950d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f455400afd0 tx=0x7f4554005980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:54.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.437+0000 7f454e7fc700 1 -- 172.21.15.78:0/1346894970 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4554026070 con 0x7f4558100280 2024-03-24T00:11:54.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.437+0000 7f454e7fc700 1 -- 172.21.15.78:0/1346894970 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4554005560 con 0x7f4558100280 2024-03-24T00:11:54.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.437+0000 7f455f55a700 1 -- 172.21.15.78:0/1346894970 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45581958a0 con 0x7f4558100280 2024-03-24T00:11:54.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.438+0000 7f454e7fc700 1 -- 172.21.15.78:0/1346894970 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f455400e700 con 0x7f4558100280 2024-03-24T00:11:54.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.438+0000 7f455f55a700 1 -- 172.21.15.78:0/1346894970 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4558198590 con 0x7f4558100280 2024-03-24T00:11:54.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.438+0000 7f454e7fc700 1 -- 172.21.15.78:0/1346894970 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f4554029070 con 0x7f4558100280 2024-03-24T00:11:54.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.438+0000 7f455f55a700 1 -- 172.21.15.78:0/1346894970 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4558106210 con 0x7f4558100280 2024-03-24T00:11:54.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.439+0000 7f454e7fc700 1 --2- 172.21.15.78:0/1346894970 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4544039020 0x7f454403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:54.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.439+0000 7f454e7fc700 1 -- 172.21.15.78:0/1346894970 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f455404e100 con 0x7f4558100280 2024-03-24T00:11:54.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.439+0000 7f455caf5700 1 --2- 172.21.15.78:0/1346894970 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4544039020 0x7f454403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:54.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.439+0000 7f455caf5700 1 --2- 172.21.15.78:0/1346894970 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4544039020 0x7f454403b4e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4548006fd0 tx=0x7f4548006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:54.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.444+0000 7f454e7fc700 1 -- 172.21.15.78:0/1346894970 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f455400e920 con 0x7f4558100280 2024-03-24T00:11:54.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.642+0000 7f455f55a700 1 -- 172.21.15.78:0/1346894970 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f455802db40 con 0x7f4544039020 2024-03-24T00:11:54.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.645+0000 7f454e7fc700 1 -- 172.21.15.78:0/1346894970 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f455802db40 con 0x7f4544039020 2024-03-24T00:11:54.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.647+0000 7f455f55a700 1 -- 172.21.15.78:0/1346894970 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4544039020 msgr2=0x7f454403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:54.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.647+0000 7f455f55a700 1 --2- 172.21.15.78:0/1346894970 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4544039020 0x7f454403b4e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4548006fd0 tx=0x7f4548006e40 comp rx=0 tx=0).stop 2024-03-24T00:11:54.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.647+0000 7f455f55a700 1 -- 172.21.15.78:0/1346894970 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4558100280 msgr2=0x7f45581950d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:54.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.647+0000 7f455f55a700 1 --2- 172.21.15.78:0/1346894970 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4558100280 0x7f45581950d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f455400afd0 tx=0x7f4554005980 comp rx=0 tx=0).stop 2024-03-24T00:11:54.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.647+0000 7f455f55a700 1 -- 172.21.15.78:0/1346894970 shutdown_connections 2024-03-24T00:11:54.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.647+0000 7f455f55a700 1 --2- 172.21.15.78:0/1346894970 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4544039020 0x7f454403b4e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:54.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.647+0000 7f455f55a700 1 --2- 172.21.15.78:0/1346894970 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4558100280 0x7f45581950d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:54.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.647+0000 7f455f55a700 1 -- 172.21.15.78:0/1346894970 >> 172.21.15.78:0/1346894970 conn(0x7f45580fb840 msgr2=0x7f45580fc200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:54.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.647+0000 7f455f55a700 1 -- 172.21.15.78:0/1346894970 shutdown_connections 2024-03-24T00:11:54.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:54.647+0000 7f455f55a700 1 -- 172.21.15.78:0/1346894970 wait complete. 2024-03-24T00:11:55.484 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi106 2024-03-24T00:11:55.484 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-24T00:11:55.484 DEBUG:teuthology.orchestra.run.smithi106:> dd of=/etc/ceph/ceph.conf 2024-03-24T00:11:55.503 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-24T00:11:55.503 DEBUG:teuthology.orchestra.run.smithi106:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:11:55.560 INFO:tasks.cephadm:Adding host smithi106 to orchestrator... 2024-03-24T00:11:55.560 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph orch host add smithi106 2024-03-24T00:11:55.792 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:11:55.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:55 smithi078 ceph-mon[32429]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:11:55.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:55 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:11:55.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:55 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:11:55.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:55 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:11:55.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:55 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:11:55.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:55 smithi078 ceph-mon[32429]: Updating smithi078:/etc/ceph/ceph.conf 2024-03-24T00:11:55.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:55 smithi078 ceph-mon[32429]: Updating smithi078:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:11:55.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:55 smithi078 ceph-mon[32429]: Updating smithi078:/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:11:55.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:55 smithi078 ceph-mon[32429]: Updating smithi078:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T00:11:55.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:55 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:11:55.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:55 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:11:55.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:55 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:11:57.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:57.357+0000 7fb2b3840700 1 -- 172.21.15.78:0/986876062 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2ac1003b0 msgr2=0x7fb2ac100790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:57.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:57.357+0000 7fb2b3840700 1 --2- 172.21.15.78:0/986876062 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2ac1003b0 0x7fb2ac100790 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb2a8009910 tx=0x7fb2a8009c20 comp rx=0 tx=0).stop 2024-03-24T00:11:57.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:57.357+0000 7fb2b3840700 1 -- 172.21.15.78:0/986876062 shutdown_connections 2024-03-24T00:11:57.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:57.357+0000 7fb2b3840700 1 --2- 172.21.15.78:0/986876062 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2ac1003b0 0x7fb2ac100790 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:57.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:57.357+0000 7fb2b3840700 1 -- 172.21.15.78:0/986876062 >> 172.21.15.78:0/986876062 conn(0x7fb2ac077140 msgr2=0x7fb2ac077550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:57.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:57.357+0000 7fb2b3840700 1 -- 172.21.15.78:0/986876062 shutdown_connections 2024-03-24T00:11:57.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:57.357+0000 7fb2b3840700 1 -- 172.21.15.78:0/986876062 wait complete. 2024-03-24T00:11:57.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:57.358+0000 7fb2b3840700 1 Processor -- start 2024-03-24T00:11:57.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:57.358+0000 7fb2b3840700 1 -- start start 2024-03-24T00:11:57.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:57.358+0000 7fb2b3840700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2ac1003b0 0x7fb2ac1976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:57.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:57.358+0000 7fb2b3840700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ac10c240 con 0x7fb2ac1003b0 2024-03-24T00:11:57.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:57.358+0000 7fb2b15dc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2ac1003b0 0x7fb2ac1976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:57.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:57.358+0000 7fb2b15dc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2ac1003b0 0x7fb2ac1976a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60270/0 (socket says 172.21.15.78:60270) 2024-03-24T00:11:57.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:57.358+0000 7fb2b15dc700 1 -- 172.21.15.78:0/2706782141 learned_addr learned my addr 172.21.15.78:0/2706782141 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:11:57.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:57.359+0000 7fb2b15dc700 1 -- 172.21.15.78:0/2706782141 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2ac197be0 con 0x7fb2ac1003b0 2024-03-24T00:11:57.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:57.359+0000 7fb2b15dc700 1 --2- 172.21.15.78:0/2706782141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2ac1003b0 0x7fb2ac1976a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb2a8005850 tx=0x7fb2a8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:57.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:57.359+0000 7fb2a27fc700 1 -- 172.21.15.78:0/2706782141 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb2a8026070 con 0x7fb2ac1003b0 2024-03-24T00:11:57.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:57.359+0000 7fb2a27fc700 1 -- 172.21.15.78:0/2706782141 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb2a800bbd0 con 0x7fb2ac1003b0 2024-03-24T00:11:57.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:57.359+0000 7fb2b3840700 1 -- 172.21.15.78:0/2706782141 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2ac197ed0 con 0x7fb2ac1003b0 2024-03-24T00:11:57.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:57.359+0000 7fb2a27fc700 1 -- 172.21.15.78:0/2706782141 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb2a8003970 con 0x7fb2ac1003b0 2024-03-24T00:11:57.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:57.359+0000 7fb2b3840700 1 -- 172.21.15.78:0/2706782141 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2ac19bf70 con 0x7fb2ac1003b0 2024-03-24T00:11:57.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:57.360+0000 7fb2a27fc700 1 -- 172.21.15.78:0/2706782141 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fb2a8029070 con 0x7fb2ac1003b0 2024-03-24T00:11:57.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:57.360+0000 7fb2b3840700 1 -- 172.21.15.78:0/2706782141 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2ac103470 con 0x7fb2ac1003b0 2024-03-24T00:11:57.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:57.360+0000 7fb2a27fc700 1 --2- 172.21.15.78:0/2706782141 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fb298039020 0x7fb29803b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:57.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:57.360+0000 7fb2a27fc700 1 -- 172.21.15.78:0/2706782141 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb2a801e070 con 0x7fb2ac1003b0 2024-03-24T00:11:57.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:57.360+0000 7fb2b0ddb700 1 --2- 172.21.15.78:0/2706782141 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fb298039020 0x7fb29803b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:57.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:57.360+0000 7fb2b0ddb700 1 --2- 172.21.15.78:0/2706782141 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fb298039020 0x7fb29803b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb29c006fd0 tx=0x7fb29c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:57.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:57.365+0000 7fb2a27fc700 1 -- 172.21.15.78:0/2706782141 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb2a8013070 con 0x7fb2ac1003b0 2024-03-24T00:11:57.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:11:57.566+0000 7fb2b3840700 1 -- 172.21.15.78:0/2706782141 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi106", "target": ["mon-mgr", ""]}) v1 -- 0x7fb2ac198a00 con 0x7fb298039020 2024-03-24T00:11:58.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:58 smithi078 ceph-mon[32429]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi106", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:11:58.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:11:58 smithi078 ceph-mon[32429]: Deploying cephadm binary to smithi106 2024-03-24T00:12:01.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:01 smithi078 ceph-mon[32429]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:01.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:01.666+0000 7fb2a27fc700 1 -- 172.21.15.78:0/2706782141 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fb2ac198a00 con 0x7fb298039020 2024-03-24T00:12:01.666 INFO:teuthology.orchestra.run.smithi078.stdout:Added host 'smithi106' with addr '172.21.15.106' 2024-03-24T00:12:01.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:01.668+0000 7fb2b3840700 1 -- 172.21.15.78:0/2706782141 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fb298039020 msgr2=0x7fb29803b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:01.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:01.668+0000 7fb2b3840700 1 --2- 172.21.15.78:0/2706782141 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fb298039020 0x7fb29803b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb29c006fd0 tx=0x7fb29c006e40 comp rx=0 tx=0).stop 2024-03-24T00:12:01.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:01.668+0000 7fb2b3840700 1 -- 172.21.15.78:0/2706782141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2ac1003b0 msgr2=0x7fb2ac1976a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:01.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:01.668+0000 7fb2b3840700 1 --2- 172.21.15.78:0/2706782141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2ac1003b0 0x7fb2ac1976a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb2a8005850 tx=0x7fb2a8005030 comp rx=0 tx=0).stop 2024-03-24T00:12:01.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:01.668+0000 7fb2b3840700 1 -- 172.21.15.78:0/2706782141 shutdown_connections 2024-03-24T00:12:01.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:01.668+0000 7fb2b3840700 1 --2- 172.21.15.78:0/2706782141 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fb298039020 0x7fb29803b4e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:01.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:01.668+0000 7fb2b3840700 1 --2- 172.21.15.78:0/2706782141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2ac1003b0 0x7fb2ac1976a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:01.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:01.668+0000 7fb2b3840700 1 -- 172.21.15.78:0/2706782141 >> 172.21.15.78:0/2706782141 conn(0x7fb2ac077140 msgr2=0x7fb2ac0fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:01.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:01.668+0000 7fb2b3840700 1 -- 172.21.15.78:0/2706782141 shutdown_connections 2024-03-24T00:12:01.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:01.668+0000 7fb2b3840700 1 -- 172.21.15.78:0/2706782141 wait complete. 2024-03-24T00:12:02.273 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-24T00:12:02.508 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:12:02.905 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:02 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:02.906 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:02 smithi078 ceph-mon[32429]: Added host smithi106 2024-03-24T00:12:02.906 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:02 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:12:02.906 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:02 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:02.906 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:02 smithi078 ceph-mon[32429]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:03.961 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:03 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:03.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:03.987+0000 7fca45e09700 1 -- 172.21.15.78:0/882980628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca40106a70 msgr2=0x7fca40106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:03.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:03.987+0000 7fca45e09700 1 --2- 172.21.15.78:0/882980628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca40106a70 0x7fca40106e50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fca30009910 tx=0x7fca30009c20 comp rx=0 tx=0).stop 2024-03-24T00:12:03.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:03.987+0000 7fca45e09700 1 -- 172.21.15.78:0/882980628 shutdown_connections 2024-03-24T00:12:03.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:03.987+0000 7fca45e09700 1 --2- 172.21.15.78:0/882980628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca40106a70 0x7fca40106e50 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:03.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:03.987+0000 7fca45e09700 1 -- 172.21.15.78:0/882980628 >> 172.21.15.78:0/882980628 conn(0x7fca400fc580 msgr2=0x7fca400fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:03.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:03.987+0000 7fca45e09700 1 -- 172.21.15.78:0/882980628 shutdown_connections 2024-03-24T00:12:03.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:03.987+0000 7fca45e09700 1 -- 172.21.15.78:0/882980628 wait complete. 2024-03-24T00:12:03.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:03.988+0000 7fca45e09700 1 Processor -- start 2024-03-24T00:12:03.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:03.988+0000 7fca45e09700 1 -- start start 2024-03-24T00:12:03.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:03.988+0000 7fca45e09700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca40106a70 0x7fca401984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:03.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:03.988+0000 7fca45e09700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca4010b0b0 con 0x7fca40106a70 2024-03-24T00:12:03.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:03.988+0000 7fca3f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca40106a70 0x7fca401984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:03.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:03.988+0000 7fca3f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca40106a70 0x7fca401984c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60288/0 (socket says 172.21.15.78:60288) 2024-03-24T00:12:03.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:03.988+0000 7fca3f7fe700 1 -- 172.21.15.78:0/2585969931 learned_addr learned my addr 172.21.15.78:0/2585969931 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:12:03.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:03.989+0000 7fca3f7fe700 1 -- 172.21.15.78:0/2585969931 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca40198a00 con 0x7fca40106a70 2024-03-24T00:12:03.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:03.989+0000 7fca3f7fe700 1 --2- 172.21.15.78:0/2585969931 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca40106a70 0x7fca401984c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fca30005850 tx=0x7fca30005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:03.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:03.989+0000 7fca3cff9700 1 -- 172.21.15.78:0/2585969931 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fca30026070 con 0x7fca40106a70 2024-03-24T00:12:03.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:03.989+0000 7fca45e09700 1 -- 172.21.15.78:0/2585969931 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca40198cf0 con 0x7fca40106a70 2024-03-24T00:12:03.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:03.989+0000 7fca3cff9700 1 -- 172.21.15.78:0/2585969931 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fca3000bbd0 con 0x7fca40106a70 2024-03-24T00:12:03.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:03.989+0000 7fca3cff9700 1 -- 172.21.15.78:0/2585969931 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fca30003970 con 0x7fca40106a70 2024-03-24T00:12:03.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:03.989+0000 7fca45e09700 1 -- 172.21.15.78:0/2585969931 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca401991b0 con 0x7fca40106a70 2024-03-24T00:12:03.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:03.990+0000 7fca3cff9700 1 -- 172.21.15.78:0/2585969931 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fca30029070 con 0x7fca40106a70 2024-03-24T00:12:03.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:03.990+0000 7fca45e09700 1 -- 172.21.15.78:0/2585969931 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca401021b0 con 0x7fca40106a70 2024-03-24T00:12:03.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:03.990+0000 7fca3cff9700 1 --2- 172.21.15.78:0/2585969931 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fca28038fc0 0x7fca2803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:03.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:03.990+0000 7fca3cff9700 1 -- 172.21.15.78:0/2585969931 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fca3001e070 con 0x7fca40106a70 2024-03-24T00:12:03.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:03.990+0000 7fca3effd700 1 --2- 172.21.15.78:0/2585969931 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fca28038fc0 0x7fca2803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:03.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:03.991+0000 7fca3effd700 1 --2- 172.21.15.78:0/2585969931 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fca28038fc0 0x7fca2803b480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fca34006fd0 tx=0x7fca34006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:03.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:03.995+0000 7fca3cff9700 1 -- 172.21.15.78:0/2585969931 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fca30013070 con 0x7fca40106a70 2024-03-24T00:12:04.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:04.193+0000 7fca45e09700 1 -- 172.21.15.78:0/2585969931 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fca4006a620 con 0x7fca28038fc0 2024-03-24T00:12:04.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:04.194+0000 7fca3cff9700 1 -- 172.21.15.78:0/2585969931 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7fca4006a620 con 0x7fca28038fc0 2024-03-24T00:12:04.194 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:12:04.195 INFO:teuthology.orchestra.run.smithi078.stdout:[{"addr": "172.21.15.78", "hostname": "smithi078", "labels": [], "status": ""}, {"addr": "172.21.15.106", "hostname": "smithi106", "labels": [], "status": ""}] 2024-03-24T00:12:04.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:04.196+0000 7fca45e09700 1 -- 172.21.15.78:0/2585969931 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fca28038fc0 msgr2=0x7fca2803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:04.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:04.197+0000 7fca45e09700 1 --2- 172.21.15.78:0/2585969931 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fca28038fc0 0x7fca2803b480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fca34006fd0 tx=0x7fca34006e40 comp rx=0 tx=0).stop 2024-03-24T00:12:04.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:04.197+0000 7fca45e09700 1 -- 172.21.15.78:0/2585969931 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca40106a70 msgr2=0x7fca401984c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:04.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:04.197+0000 7fca45e09700 1 --2- 172.21.15.78:0/2585969931 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca40106a70 0x7fca401984c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fca30005850 tx=0x7fca30005030 comp rx=0 tx=0).stop 2024-03-24T00:12:04.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:04.197+0000 7fca45e09700 1 -- 172.21.15.78:0/2585969931 shutdown_connections 2024-03-24T00:12:04.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:04.197+0000 7fca45e09700 1 --2- 172.21.15.78:0/2585969931 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fca28038fc0 0x7fca2803b480 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:04.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:04.197+0000 7fca45e09700 1 --2- 172.21.15.78:0/2585969931 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca40106a70 0x7fca401984c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:04.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:04.197+0000 7fca45e09700 1 -- 172.21.15.78:0/2585969931 >> 172.21.15.78:0/2585969931 conn(0x7fca400fc580 msgr2=0x7fca400fd220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:04.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:04.197+0000 7fca45e09700 1 -- 172.21.15.78:0/2585969931 shutdown_connections 2024-03-24T00:12:04.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:04.197+0000 7fca45e09700 1 -- 172.21.15.78:0/2585969931 wait complete. 2024-03-24T00:12:04.751 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi107 2024-03-24T00:12:04.751 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-24T00:12:04.751 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/etc/ceph/ceph.conf 2024-03-24T00:12:04.770 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-24T00:12:04.771 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:12:04.829 INFO:tasks.cephadm:Adding host smithi107 to orchestrator... 2024-03-24T00:12:04.830 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph orch host add smithi107 2024-03-24T00:12:05.066 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:12:05.509 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:05 smithi078 ceph-mon[32429]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:12:05.509 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:05 smithi078 ceph-mon[32429]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:05.509 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:05 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:06.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:06.549+0000 7ffbede27700 1 -- 172.21.15.78:0/3646165298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffbe8100450 msgr2=0x7ffbe8100830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:06.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:06.549+0000 7ffbede27700 1 --2- 172.21.15.78:0/3646165298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffbe8100450 0x7ffbe8100830 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc00aa20 tx=0x7ffbdc00ad30 comp rx=0 tx=0).stop 2024-03-24T00:12:06.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:06.549+0000 7ffbede27700 1 -- 172.21.15.78:0/3646165298 shutdown_connections 2024-03-24T00:12:06.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:06.549+0000 7ffbede27700 1 --2- 172.21.15.78:0/3646165298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffbe8100450 0x7ffbe8100830 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:06.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:06.549+0000 7ffbede27700 1 -- 172.21.15.78:0/3646165298 >> 172.21.15.78:0/3646165298 conn(0x7ffbe80fbce0 msgr2=0x7ffbe80fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:06.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:06.549+0000 7ffbede27700 1 -- 172.21.15.78:0/3646165298 shutdown_connections 2024-03-24T00:12:06.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:06.549+0000 7ffbede27700 1 -- 172.21.15.78:0/3646165298 wait complete. 2024-03-24T00:12:06.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:06.550+0000 7ffbede27700 1 Processor -- start 2024-03-24T00:12:06.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:06.550+0000 7ffbede27700 1 -- start start 2024-03-24T00:12:06.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:06.550+0000 7ffbede27700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffbe8100450 0x7ffbe8197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:06.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:06.550+0000 7ffbede27700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe8109fa0 con 0x7ffbe8100450 2024-03-24T00:12:06.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:06.550+0000 7ffbe77fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffbe8100450 0x7ffbe8197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:06.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:06.550+0000 7ffbe77fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffbe8100450 0x7ffbe8197660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58716/0 (socket says 172.21.15.78:58716) 2024-03-24T00:12:06.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:06.550+0000 7ffbe77fe700 1 -- 172.21.15.78:0/2562301975 learned_addr learned my addr 172.21.15.78:0/2562301975 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:12:06.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:06.550+0000 7ffbe77fe700 1 -- 172.21.15.78:0/2562301975 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe8197ba0 con 0x7ffbe8100450 2024-03-24T00:12:06.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:06.551+0000 7ffbe77fe700 1 --2- 172.21.15.78:0/2562301975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffbe8100450 0x7ffbe8197660 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc004c90 tx=0x7ffbdc004d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:06.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:06.551+0000 7ffbe4ff9700 1 -- 172.21.15.78:0/2562301975 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffbdc005310 con 0x7ffbe8100450 2024-03-24T00:12:06.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:06.551+0000 7ffbe4ff9700 1 -- 172.21.15.78:0/2562301975 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffbdc0054b0 con 0x7ffbe8100450 2024-03-24T00:12:06.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:06.551+0000 7ffbe4ff9700 1 -- 172.21.15.78:0/2562301975 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffbdc00e790 con 0x7ffbe8100450 2024-03-24T00:12:06.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:06.551+0000 7ffbede27700 1 -- 172.21.15.78:0/2562301975 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbe8197e30 con 0x7ffbe8100450 2024-03-24T00:12:06.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:06.551+0000 7ffbede27700 1 -- 172.21.15.78:0/2562301975 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbe819bf30 con 0x7ffbe8100450 2024-03-24T00:12:06.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:06.552+0000 7ffbe4ff9700 1 -- 172.21.15.78:0/2562301975 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7ffbdc00ea40 con 0x7ffbe8100450 2024-03-24T00:12:06.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:06.552+0000 7ffbede27700 1 -- 172.21.15.78:0/2562301975 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbe8101930 con 0x7ffbe8100450 2024-03-24T00:12:06.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:06.552+0000 7ffbe4ff9700 1 --2- 172.21.15.78:0/2562301975 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ffbd0039020 0x7ffbd003b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:06.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:06.552+0000 7ffbe4ff9700 1 -- 172.21.15.78:0/2562301975 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ffbdc01e070 con 0x7ffbe8100450 2024-03-24T00:12:06.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:06.552+0000 7ffbe6ffd700 1 --2- 172.21.15.78:0/2562301975 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ffbd0039020 0x7ffbd003b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:06.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:06.552+0000 7ffbe6ffd700 1 --2- 172.21.15.78:0/2562301975 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ffbd0039020 0x7ffbd003b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8006fd0 tx=0x7ffbd8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:06.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:06.557+0000 7ffbe4ff9700 1 -- 172.21.15.78:0/2562301975 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ffbdc020370 con 0x7ffbe8100450 2024-03-24T00:12:06.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:06.756+0000 7ffbede27700 1 -- 172.21.15.78:0/2562301975 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi107", "target": ["mon-mgr", ""]}) v1 -- 0x7ffbe81989c0 con 0x7ffbd0039020 2024-03-24T00:12:07.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:07 smithi078 ceph-mon[32429]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:08.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:08 smithi078 ceph-mon[32429]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi107", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:12:08.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:08 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:08.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:08 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:08.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:08 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:08.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:08 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:08.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:08 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi106", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:12:08.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:08 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:08.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:08 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:12:08.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:08 smithi078 ceph-mon[32429]: Updating smithi106:/etc/ceph/ceph.conf 2024-03-24T00:12:08.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:08 smithi078 ceph-mon[32429]: Deploying cephadm binary to smithi107 2024-03-24T00:12:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:09 smithi078 ceph-mon[32429]: Updating smithi106:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:12:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:09 smithi078 ceph-mon[32429]: Updating smithi106:/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:12:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:09 smithi078 ceph-mon[32429]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:10.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:10.881+0000 7ffbe4ff9700 1 -- 172.21.15.78:0/2562301975 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7ffbe81989c0 con 0x7ffbd0039020 2024-03-24T00:12:10.881 INFO:teuthology.orchestra.run.smithi078.stdout:Added host 'smithi107' with addr '172.21.15.107' 2024-03-24T00:12:10.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:10.883+0000 7ffbede27700 1 -- 172.21.15.78:0/2562301975 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ffbd0039020 msgr2=0x7ffbd003b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:10.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:10.883+0000 7ffbede27700 1 --2- 172.21.15.78:0/2562301975 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ffbd0039020 0x7ffbd003b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8006fd0 tx=0x7ffbd8006e40 comp rx=0 tx=0).stop 2024-03-24T00:12:10.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:10.883+0000 7ffbede27700 1 -- 172.21.15.78:0/2562301975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffbe8100450 msgr2=0x7ffbe8197660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:10.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:10.883+0000 7ffbede27700 1 --2- 172.21.15.78:0/2562301975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffbe8100450 0x7ffbe8197660 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc004c90 tx=0x7ffbdc004d70 comp rx=0 tx=0).stop 2024-03-24T00:12:10.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:10.883+0000 7ffbede27700 1 -- 172.21.15.78:0/2562301975 shutdown_connections 2024-03-24T00:12:10.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:10.883+0000 7ffbede27700 1 --2- 172.21.15.78:0/2562301975 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ffbd0039020 0x7ffbd003b4e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:10.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:10.883+0000 7ffbede27700 1 --2- 172.21.15.78:0/2562301975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffbe8100450 0x7ffbe8197660 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:10.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:10.883+0000 7ffbede27700 1 -- 172.21.15.78:0/2562301975 >> 172.21.15.78:0/2562301975 conn(0x7ffbe80fbce0 msgr2=0x7ffbe8107fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:10.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:10.883+0000 7ffbede27700 1 -- 172.21.15.78:0/2562301975 shutdown_connections 2024-03-24T00:12:10.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:10.883+0000 7ffbede27700 1 -- 172.21.15.78:0/2562301975 wait complete. 2024-03-24T00:12:11.461 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:11 smithi078 ceph-mon[32429]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:11.461 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:11 smithi078 ceph-mon[32429]: Updating smithi106:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T00:12:11.461 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:11 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:11.461 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:11 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:11.461 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:11 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:11.461 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:11 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:11.461 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:11 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:12:11.489 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-24T00:12:11.724 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:12:12.508 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:12 smithi078 ceph-mon[32429]: Added host smithi107 2024-03-24T00:12:12.509 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:12 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:12.509 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:12 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:13.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.245+0000 7f3911c22700 1 -- 172.21.15.78:0/1983241395 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f390c102d10 msgr2=0x7f390c1030f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:13.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.245+0000 7f3911c22700 1 --2- 172.21.15.78:0/1983241395 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f390c102d10 0x7f390c1030f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f38fc009910 tx=0x7f38fc009c20 comp rx=0 tx=0).stop 2024-03-24T00:12:13.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.245+0000 7f3911c22700 1 -- 172.21.15.78:0/1983241395 shutdown_connections 2024-03-24T00:12:13.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.245+0000 7f3911c22700 1 --2- 172.21.15.78:0/1983241395 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f390c102d10 0x7f390c1030f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:13.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.245+0000 7f3911c22700 1 -- 172.21.15.78:0/1983241395 >> 172.21.15.78:0/1983241395 conn(0x7f390c0fc540 msgr2=0x7f390c0fe960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:13.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.246+0000 7f3911c22700 1 -- 172.21.15.78:0/1983241395 shutdown_connections 2024-03-24T00:12:13.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.246+0000 7f3911c22700 1 -- 172.21.15.78:0/1983241395 wait complete. 2024-03-24T00:12:13.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.246+0000 7f3911c22700 1 Processor -- start 2024-03-24T00:12:13.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.246+0000 7f3911c22700 1 -- start start 2024-03-24T00:12:13.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.246+0000 7f3911c22700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f390c102d10 0x7f390c100970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:13.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.246+0000 7f3911c22700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390c10c200 con 0x7f390c102d10 2024-03-24T00:12:13.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.247+0000 7f390b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f390c102d10 0x7f390c100970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:13.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.247+0000 7f390b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f390c102d10 0x7f390c100970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58726/0 (socket says 172.21.15.78:58726) 2024-03-24T00:12:13.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.247+0000 7f390b7fe700 1 -- 172.21.15.78:0/3706278092 learned_addr learned my addr 172.21.15.78:0/3706278092 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:12:13.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.247+0000 7f390b7fe700 1 -- 172.21.15.78:0/3706278092 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f390c102860 con 0x7f390c102d10 2024-03-24T00:12:13.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.247+0000 7f390b7fe700 1 --2- 172.21.15.78:0/3706278092 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f390c102d10 0x7f390c100970 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f38fc006010 tx=0x7f38fc005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:13.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.247+0000 7f3908ff9700 1 -- 172.21.15.78:0/3706278092 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f38fc026070 con 0x7f390c102d10 2024-03-24T00:12:13.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.247+0000 7f3908ff9700 1 -- 172.21.15.78:0/3706278092 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38fc00bc30 con 0x7f390c102d10 2024-03-24T00:12:13.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.247+0000 7f3911c22700 1 -- 172.21.15.78:0/3706278092 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f390c100eb0 con 0x7f390c102d10 2024-03-24T00:12:13.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.247+0000 7f3908ff9700 1 -- 172.21.15.78:0/3706278092 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f38fc003ab0 con 0x7f390c102d10 2024-03-24T00:12:13.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.247+0000 7f3911c22700 1 -- 172.21.15.78:0/3706278092 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f390c101370 con 0x7f390c102d10 2024-03-24T00:12:13.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.248+0000 7f3911c22700 1 -- 172.21.15.78:0/3706278092 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f390c06a6f0 con 0x7f390c102d10 2024-03-24T00:12:13.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.248+0000 7f3908ff9700 1 -- 172.21.15.78:0/3706278092 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f38fc029070 con 0x7f390c102d10 2024-03-24T00:12:13.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.248+0000 7f3908ff9700 1 --2- 172.21.15.78:0/3706278092 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f38f4039020 0x7f38f403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:13.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.248+0000 7f3908ff9700 1 -- 172.21.15.78:0/3706278092 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f38fc01e070 con 0x7f390c102d10 2024-03-24T00:12:13.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.249+0000 7f390affd700 1 --2- 172.21.15.78:0/3706278092 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f38f4039020 0x7f38f403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:13.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.249+0000 7f390affd700 1 --2- 172.21.15.78:0/3706278092 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f38f4039020 0x7f38f403b4e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3900006fd0 tx=0x7f3900006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:13.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.253+0000 7f3908ff9700 1 -- 172.21.15.78:0/3706278092 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f38fc013070 con 0x7f390c102d10 2024-03-24T00:12:13.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.457+0000 7f3911c22700 1 -- 172.21.15.78:0/3706278092 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f390c101f10 con 0x7f38f4039020 2024-03-24T00:12:13.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.458+0000 7f3908ff9700 1 -- 172.21.15.78:0/3706278092 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+240 (secure 0 0 0) 0x7f390c101f10 con 0x7f38f4039020 2024-03-24T00:12:13.458 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:12:13.458 INFO:teuthology.orchestra.run.smithi078.stdout:[{"addr": "172.21.15.78", "hostname": "smithi078", "labels": [], "status": ""}, {"addr": "172.21.15.106", "hostname": "smithi106", "labels": [], "status": ""}, {"addr": "172.21.15.107", "hostname": "smithi107", "labels": [], "status": ""}] 2024-03-24T00:12:13.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.460+0000 7f3911c22700 1 -- 172.21.15.78:0/3706278092 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f38f4039020 msgr2=0x7f38f403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:13.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.460+0000 7f3911c22700 1 --2- 172.21.15.78:0/3706278092 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f38f4039020 0x7f38f403b4e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3900006fd0 tx=0x7f3900006e40 comp rx=0 tx=0).stop 2024-03-24T00:12:13.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.460+0000 7f3911c22700 1 -- 172.21.15.78:0/3706278092 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f390c102d10 msgr2=0x7f390c100970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:13.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.460+0000 7f3911c22700 1 --2- 172.21.15.78:0/3706278092 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f390c102d10 0x7f390c100970 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f38fc006010 tx=0x7f38fc005050 comp rx=0 tx=0).stop 2024-03-24T00:12:13.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.461+0000 7f3911c22700 1 -- 172.21.15.78:0/3706278092 shutdown_connections 2024-03-24T00:12:13.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.461+0000 7f3911c22700 1 --2- 172.21.15.78:0/3706278092 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f38f4039020 0x7f38f403b4e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:13.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.461+0000 7f3911c22700 1 --2- 172.21.15.78:0/3706278092 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f390c102d10 0x7f390c100970 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:13.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.461+0000 7f3911c22700 1 -- 172.21.15.78:0/3706278092 >> 172.21.15.78:0/3706278092 conn(0x7f390c0fc540 msgr2=0x7f390c103ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:13.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.461+0000 7f3911c22700 1 -- 172.21.15.78:0/3706278092 shutdown_connections 2024-03-24T00:12:13.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:13.461+0000 7f3911c22700 1 -- 172.21.15.78:0/3706278092 wait complete. 2024-03-24T00:12:13.471 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:13 smithi078 ceph-mon[32429]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:13.986 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-24T00:12:13.987 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph osd crush tunables default 2024-03-24T00:12:14.222 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:12:14.652 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:14 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:14.652 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:14 smithi078 ceph-mon[32429]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:12:14.652 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:14 smithi078 ceph-mon[32429]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:15.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:15.745+0000 7f4ad450c700 1 -- 172.21.15.78:0/4109361867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4acc1002e0 msgr2=0x7f4acc1006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:15.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:15.745+0000 7f4ad450c700 1 --2- 172.21.15.78:0/4109361867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4acc1002e0 0x7f4acc1006c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8009910 tx=0x7f4ac8009c20 comp rx=0 tx=0).stop 2024-03-24T00:12:15.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:15.745+0000 7f4ad450c700 1 -- 172.21.15.78:0/4109361867 shutdown_connections 2024-03-24T00:12:15.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:15.746+0000 7f4ad450c700 1 --2- 172.21.15.78:0/4109361867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4acc1002e0 0x7f4acc1006c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:15.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:15.746+0000 7f4ad450c700 1 -- 172.21.15.78:0/4109361867 >> 172.21.15.78:0/4109361867 conn(0x7f4acc0fbce0 msgr2=0x7f4acc0fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:15.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:15.746+0000 7f4ad450c700 1 -- 172.21.15.78:0/4109361867 shutdown_connections 2024-03-24T00:12:15.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:15.746+0000 7f4ad450c700 1 -- 172.21.15.78:0/4109361867 wait complete. 2024-03-24T00:12:15.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:15.746+0000 7f4ad450c700 1 Processor -- start 2024-03-24T00:12:15.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:15.746+0000 7f4ad450c700 1 -- start start 2024-03-24T00:12:15.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:15.746+0000 7f4ad450c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4acc1002e0 0x7f4acc1976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:15.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:15.746+0000 7f4ad450c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4acc10c1a0 con 0x7f4acc1002e0 2024-03-24T00:12:15.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:15.746+0000 7f4ad22a8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4acc1002e0 0x7f4acc1976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:15.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:15.746+0000 7f4ad22a8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4acc1002e0 0x7f4acc1976a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59778/0 (socket says 172.21.15.78:59778) 2024-03-24T00:12:15.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:15.746+0000 7f4ad22a8700 1 -- 172.21.15.78:0/3190464281 learned_addr learned my addr 172.21.15.78:0/3190464281 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:12:15.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:15.747+0000 7f4ad22a8700 1 -- 172.21.15.78:0/3190464281 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4acc197be0 con 0x7f4acc1002e0 2024-03-24T00:12:15.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:15.747+0000 7f4ad22a8700 1 --2- 172.21.15.78:0/3190464281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4acc1002e0 0x7f4acc1976a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8005850 tx=0x7f4ac8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:15.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:15.747+0000 7f4ac37fe700 1 -- 172.21.15.78:0/3190464281 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4ac8026070 con 0x7f4acc1002e0 2024-03-24T00:12:15.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:15.747+0000 7f4ac37fe700 1 -- 172.21.15.78:0/3190464281 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ac800bbd0 con 0x7f4acc1002e0 2024-03-24T00:12:15.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:15.747+0000 7f4ad450c700 1 -- 172.21.15.78:0/3190464281 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4acc197ed0 con 0x7f4acc1002e0 2024-03-24T00:12:15.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:15.747+0000 7f4ac37fe700 1 -- 172.21.15.78:0/3190464281 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4ac8003970 con 0x7f4acc1002e0 2024-03-24T00:12:15.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:15.747+0000 7f4ad450c700 1 -- 172.21.15.78:0/3190464281 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4acc19bf70 con 0x7f4acc1002e0 2024-03-24T00:12:15.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:15.748+0000 7f4ac37fe700 1 -- 172.21.15.78:0/3190464281 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f4ac8029070 con 0x7f4acc1002e0 2024-03-24T00:12:15.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:15.748+0000 7f4ad450c700 1 -- 172.21.15.78:0/3190464281 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4acc1033a0 con 0x7f4acc1002e0 2024-03-24T00:12:15.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:15.748+0000 7f4ac37fe700 1 --2- 172.21.15.78:0/3190464281 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4ab8038fc0 0x7f4ab803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:15.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:15.748+0000 7f4ac37fe700 1 -- 172.21.15.78:0/3190464281 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4ac801e070 con 0x7f4acc1002e0 2024-03-24T00:12:15.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:15.748+0000 7f4ad1aa7700 1 --2- 172.21.15.78:0/3190464281 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4ab8038fc0 0x7f4ab803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:15.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:15.748+0000 7f4ad1aa7700 1 --2- 172.21.15.78:0/3190464281 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4ab8038fc0 0x7f4ab803b480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4abc006fd0 tx=0x7f4abc006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:15.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:15.753+0000 7f4ac37fe700 1 -- 172.21.15.78:0/3190464281 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4ac8013070 con 0x7f4acc1002e0 2024-03-24T00:12:15.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:15.952+0000 7f4ad450c700 1 -- 172.21.15.78:0/3190464281 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f4acc198a00 con 0x7f4acc1002e0 2024-03-24T00:12:16.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:16.003+0000 7f4ac37fe700 1 -- 172.21.15.78:0/3190464281 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f4ac8030090 con 0x7f4acc1002e0 2024-03-24T00:12:16.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:16.005+0000 7f4ad450c700 1 -- 172.21.15.78:0/3190464281 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4ab8038fc0 msgr2=0x7f4ab803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:16.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:16.005+0000 7f4ad450c700 1 --2- 172.21.15.78:0/3190464281 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4ab8038fc0 0x7f4ab803b480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4abc006fd0 tx=0x7f4abc006e40 comp rx=0 tx=0).stop 2024-03-24T00:12:16.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:16.005+0000 7f4ad450c700 1 -- 172.21.15.78:0/3190464281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4acc1002e0 msgr2=0x7f4acc1976a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:16.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:16.005+0000 7f4ad450c700 1 --2- 172.21.15.78:0/3190464281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4acc1002e0 0x7f4acc1976a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8005850 tx=0x7f4ac8005030 comp rx=0 tx=0).stop 2024-03-24T00:12:16.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:16.005+0000 7f4ad450c700 1 -- 172.21.15.78:0/3190464281 shutdown_connections 2024-03-24T00:12:16.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:16.005+0000 7f4ad450c700 1 --2- 172.21.15.78:0/3190464281 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4ab8038fc0 0x7f4ab803b480 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:16.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:16.005+0000 7f4ad450c700 1 --2- 172.21.15.78:0/3190464281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4acc1002e0 0x7f4acc1976a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:16.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:16.005+0000 7f4ad450c700 1 -- 172.21.15.78:0/3190464281 >> 172.21.15.78:0/3190464281 conn(0x7f4acc0fbce0 msgr2=0x7f4acc104b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:16.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:16.005+0000 7f4ad450c700 1 -- 172.21.15.78:0/3190464281 shutdown_connections 2024-03-24T00:12:16.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:16.005+0000 7f4ad450c700 1 -- 172.21.15.78:0/3190464281 wait complete. 2024-03-24T00:12:16.006 INFO:teuthology.orchestra.run.smithi078.stderr:adjusted tunables profile to default 2024-03-24T00:12:16.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:16 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3190464281' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T00:12:16.599 INFO:tasks.cephadm:Adding mon.a on smithi078 2024-03-24T00:12:16.600 INFO:tasks.cephadm:Adding mon.b on smithi106 2024-03-24T00:12:16.600 INFO:tasks.cephadm:Adding mon.c on smithi107 2024-03-24T00:12:16.600 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph orch apply mon '3;smithi078:172.21.15.78=a;smithi106:172.21.15.106=b;smithi107:172.21.15.107=c' 2024-03-24T00:12:16.829 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T00:12:16.877 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T00:12:17.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:17 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3190464281' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T00:12:17.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:17 smithi078 ceph-mon[32429]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T00:12:17.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:17 smithi078 ceph-mon[32429]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:17.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:17 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:17.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:17 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:17.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:17 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:17.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:17 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:17.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:17 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi107", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:12:17.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:17 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:17.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:17 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:12:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:18 smithi078 ceph-mon[32429]: Updating smithi107:/etc/ceph/ceph.conf 2024-03-24T00:12:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:18 smithi078 ceph-mon[32429]: Updating smithi107:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:12:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:18 smithi078 ceph-mon[32429]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:12:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:18 smithi078 ceph-mon[32429]: Updating smithi107:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T00:12:18.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:18 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:18.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:18 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:18.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:18 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:18.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.605+0000 7ff101c14700 1 -- 172.21.15.107:0/2290739165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0fc106a70 msgr2=0x7ff0fc106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:18.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.605+0000 7ff101c14700 1 --2- 172.21.15.107:0/2290739165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0fc106a70 0x7ff0fc106e50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec009910 tx=0x7ff0ec009c20 comp rx=0 tx=0).stop 2024-03-24T00:12:18.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.605+0000 7ff101c14700 1 -- 172.21.15.107:0/2290739165 shutdown_connections 2024-03-24T00:12:18.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.605+0000 7ff101c14700 1 --2- 172.21.15.107:0/2290739165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0fc106a70 0x7ff0fc106e50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:18.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.605+0000 7ff101c14700 1 -- 172.21.15.107:0/2290739165 >> 172.21.15.107:0/2290739165 conn(0x7ff0fc0fc580 msgr2=0x7ff0fc0fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:18.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.605+0000 7ff101c14700 1 -- 172.21.15.107:0/2290739165 shutdown_connections 2024-03-24T00:12:18.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.605+0000 7ff101c14700 1 -- 172.21.15.107:0/2290739165 wait complete. 2024-03-24T00:12:18.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.606+0000 7ff101c14700 1 Processor -- start 2024-03-24T00:12:18.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.606+0000 7ff101c14700 1 -- start start 2024-03-24T00:12:18.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.606+0000 7ff101c14700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0fc106a70 0x7ff0fc0fca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:18.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.606+0000 7ff101c14700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0fc10b0b0 con 0x7ff0fc106a70 2024-03-24T00:12:18.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.606+0000 7ff0fb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0fc106a70 0x7ff0fc0fca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:18.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.606+0000 7ff0fb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0fc106a70 0x7ff0fc0fca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.107:60522/0 (socket says 172.21.15.107:60522) 2024-03-24T00:12:18.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.606+0000 7ff0fb7fe700 1 -- 172.21.15.107:0/3222356236 learned_addr learned my addr 172.21.15.107:0/3222356236 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-03-24T00:12:18.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.607+0000 7ff0fb7fe700 1 -- 172.21.15.107:0/3222356236 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0fc0fe920 con 0x7ff0fc106a70 2024-03-24T00:12:18.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.607+0000 7ff0fb7fe700 1 --2- 172.21.15.107:0/3222356236 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0fc106a70 0x7ff0fc0fca30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec005850 tx=0x7ff0ec005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:18.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.607+0000 7ff0f8ff9700 1 -- 172.21.15.107:0/3222356236 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff0ec026070 con 0x7ff0fc106a70 2024-03-24T00:12:18.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.607+0000 7ff0f8ff9700 1 -- 172.21.15.107:0/3222356236 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0ec00bc30 con 0x7ff0fc106a70 2024-03-24T00:12:18.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.607+0000 7ff101c14700 1 -- 172.21.15.107:0/3222356236 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0fc0fcfd0 con 0x7ff0fc106a70 2024-03-24T00:12:18.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.607+0000 7ff0f8ff9700 1 -- 172.21.15.107:0/3222356236 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff0ec003a20 con 0x7ff0fc106a70 2024-03-24T00:12:18.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.607+0000 7ff101c14700 1 -- 172.21.15.107:0/3222356236 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0fc0fd490 con 0x7ff0fc106a70 2024-03-24T00:12:18.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.608+0000 7ff101c14700 1 -- 172.21.15.107:0/3222356236 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0fc1021b0 con 0x7ff0fc106a70 2024-03-24T00:12:18.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.608+0000 7ff0f8ff9700 1 -- 172.21.15.107:0/3222356236 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7ff0ec029070 con 0x7ff0fc106a70 2024-03-24T00:12:18.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.608+0000 7ff0f8ff9700 1 --2- 172.21.15.107:0/3222356236 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff0e4038fc0 0x7ff0e403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:18.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.608+0000 7ff0f8ff9700 1 -- 172.21.15.107:0/3222356236 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff0ec01e070 con 0x7ff0fc106a70 2024-03-24T00:12:18.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.609+0000 7ff0faffd700 1 --2- 172.21.15.107:0/3222356236 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff0e4038fc0 0x7ff0e403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:18.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.609+0000 7ff0faffd700 1 --2- 172.21.15.107:0/3222356236 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff0e4038fc0 0x7ff0e403b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0006fd0 tx=0x7ff0f0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:18.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.614+0000 7ff0f8ff9700 1 -- 172.21.15.107:0/3222356236 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff0ec013070 con 0x7ff0fc106a70 2024-03-24T00:12:18.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.818+0000 7ff101c14700 1 -- 172.21.15.107:0/3222356236 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi078:172.21.15.78=a;smithi106:172.21.15.106=b;smithi107:172.21.15.107=c", "target": ["mon-mgr", ""]}) v1 -- 0x7ff0fc0fdfd0 con 0x7ff0e4038fc0 2024-03-24T00:12:18.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.825+0000 7ff0f8ff9700 1 -- 172.21.15.107:0/3222356236 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7ff0fc0fdfd0 con 0x7ff0e4038fc0 2024-03-24T00:12:18.825 INFO:teuthology.orchestra.run.smithi107.stdout:Scheduled mon update... 2024-03-24T00:12:18.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.827+0000 7ff101c14700 1 -- 172.21.15.107:0/3222356236 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff0e4038fc0 msgr2=0x7ff0e403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:18.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.827+0000 7ff101c14700 1 --2- 172.21.15.107:0/3222356236 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff0e4038fc0 0x7ff0e403b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0006fd0 tx=0x7ff0f0006e40 comp rx=0 tx=0).stop 2024-03-24T00:12:18.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.827+0000 7ff101c14700 1 -- 172.21.15.107:0/3222356236 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0fc106a70 msgr2=0x7ff0fc0fca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:18.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.827+0000 7ff101c14700 1 --2- 172.21.15.107:0/3222356236 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0fc106a70 0x7ff0fc0fca30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec005850 tx=0x7ff0ec005030 comp rx=0 tx=0).stop 2024-03-24T00:12:18.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.827+0000 7ff101c14700 1 -- 172.21.15.107:0/3222356236 shutdown_connections 2024-03-24T00:12:18.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.827+0000 7ff101c14700 1 --2- 172.21.15.107:0/3222356236 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff0e4038fc0 0x7ff0e403b480 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:18.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.827+0000 7ff101c14700 1 --2- 172.21.15.107:0/3222356236 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0fc106a70 0x7ff0fc0fca30 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:18.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.827+0000 7ff101c14700 1 -- 172.21.15.107:0/3222356236 >> 172.21.15.107:0/3222356236 conn(0x7ff0fc0fc580 msgr2=0x7ff0fc069ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:18.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.828+0000 7ff101c14700 1 -- 172.21.15.107:0/3222356236 shutdown_connections 2024-03-24T00:12:18.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:18.828+0000 7ff101c14700 1 -- 172.21.15.107:0/3222356236 wait complete. 2024-03-24T00:12:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:19 smithi078 ceph-mon[32429]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:19 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:19 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:12:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:19 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:19 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:12:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:19 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:19 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:12:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:19 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:19.343 DEBUG:teuthology.orchestra.run.smithi106:mon.b> sudo journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mon.b.service 2024-03-24T00:12:19.346 DEBUG:teuthology.orchestra.run.smithi107:mon.c> sudo journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mon.c.service 2024-03-24T00:12:19.348 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T00:12:19.348 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T00:12:20.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:20 smithi078 ceph-mon[32429]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi078:172.21.15.78=a;smithi106:172.21.15.106=b;smithi107:172.21.15.107=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:12:20.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:20 smithi078 ceph-mon[32429]: Saving service mon spec with placement smithi078:172.21.15.78=a;smithi106:172.21.15.106=b;smithi107:172.21.15.107=c;count:3 2024-03-24T00:12:20.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:20 smithi078 ceph-mon[32429]: Deploying daemon mon.c on smithi107 2024-03-24T00:12:21.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:21 smithi078 ceph-mon[32429]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:21.503 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:12:22.734 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:22 smithi107 systemd[1]: Starting Ceph mon.c for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:12:23.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.288+0000 7fcdd81c0700 1 -- 172.21.15.107:0/2298806976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcdd0101ed0 msgr2=0x7fcdd01022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:23.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.288+0000 7fcdd81c0700 1 --2- 172.21.15.107:0/2298806976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcdd0101ed0 0x7fcdd01022b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc009910 tx=0x7fcdcc009c20 comp rx=0 tx=0).stop 2024-03-24T00:12:23.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.288+0000 7fcdd81c0700 1 -- 172.21.15.107:0/2298806976 shutdown_connections 2024-03-24T00:12:23.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.288+0000 7fcdd81c0700 1 --2- 172.21.15.107:0/2298806976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcdd0101ed0 0x7fcdd01022b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:23.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.288+0000 7fcdd81c0700 1 -- 172.21.15.107:0/2298806976 >> 172.21.15.107:0/2298806976 conn(0x7fcdd00fd760 msgr2=0x7fcdd00ffb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:23.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.288+0000 7fcdd81c0700 1 -- 172.21.15.107:0/2298806976 shutdown_connections 2024-03-24T00:12:23.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.288+0000 7fcdd81c0700 1 -- 172.21.15.107:0/2298806976 wait complete. 2024-03-24T00:12:23.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.288+0000 7fcdd81c0700 1 Processor -- start 2024-03-24T00:12:23.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.288+0000 7fcdd81c0700 1 -- start start 2024-03-24T00:12:23.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.289+0000 7fcdd81c0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcdd0101ed0 0x7fcdd019d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:23.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.289+0000 7fcdd81c0700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd010c2a0 con 0x7fcdd0101ed0 2024-03-24T00:12:23.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.289+0000 7fcdd5f5c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcdd0101ed0 0x7fcdd019d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:23.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.289+0000 7fcdd5f5c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcdd0101ed0 0x7fcdd019d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.107:60534/0 (socket says 172.21.15.107:60534) 2024-03-24T00:12:23.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.289+0000 7fcdd5f5c700 1 -- 172.21.15.107:0/3759684172 learned_addr learned my addr 172.21.15.107:0/3759684172 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-03-24T00:12:23.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.289+0000 7fcdd5f5c700 1 -- 172.21.15.107:0/3759684172 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd019d960 con 0x7fcdd0101ed0 2024-03-24T00:12:23.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.289+0000 7fcdd5f5c700 1 --2- 172.21.15.107:0/3759684172 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcdd0101ed0 0x7fcdd019d420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc005850 tx=0x7fcdcc005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:23.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.289+0000 7fcdc6ffd700 1 -- 172.21.15.107:0/3759684172 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcdcc026070 con 0x7fcdd0101ed0 2024-03-24T00:12:23.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.290+0000 7fcdd81c0700 1 -- 172.21.15.107:0/3759684172 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdd019dc50 con 0x7fcdd0101ed0 2024-03-24T00:12:23.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.290+0000 7fcdc6ffd700 1 -- 172.21.15.107:0/3759684172 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcdcc00bbd0 con 0x7fcdd0101ed0 2024-03-24T00:12:23.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.290+0000 7fcdc6ffd700 1 -- 172.21.15.107:0/3759684172 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcdcc003970 con 0x7fcdd0101ed0 2024-03-24T00:12:23.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.290+0000 7fcdd81c0700 1 -- 172.21.15.107:0/3759684172 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdd01975e0 con 0x7fcdd0101ed0 2024-03-24T00:12:23.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.290+0000 7fcdc6ffd700 1 -- 172.21.15.107:0/3759684172 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fcdcc029070 con 0x7fcdd0101ed0 2024-03-24T00:12:23.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.290+0000 7fcdd81c0700 1 -- 172.21.15.107:0/3759684172 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdd01033b0 con 0x7fcdd0101ed0 2024-03-24T00:12:23.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.291+0000 7fcdc6ffd700 1 --2- 172.21.15.107:0/3759684172 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fcdbc038fc0 0x7fcdbc03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:23.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.291+0000 7fcdc6ffd700 1 -- 172.21.15.107:0/3759684172 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fcdcc01e070 con 0x7fcdd0101ed0 2024-03-24T00:12:23.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.291+0000 7fcdd575b700 1 --2- 172.21.15.107:0/3759684172 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fcdbc038fc0 0x7fcdbc03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:23.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.291+0000 7fcdd575b700 1 --2- 172.21.15.107:0/3759684172 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fcdbc038fc0 0x7fcdbc03b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0006fd0 tx=0x7fcdc0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:23.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.296+0000 7fcdc6ffd700 1 -- 172.21.15.107:0/3759684172 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcdcc013070 con 0x7fcdd0101ed0 2024-03-24T00:12:23.296 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 podman[36435]: 2024-03-24 00:12:23.082737915 +0000 UTC m=+0.253981910 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:12:23.573 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 podman[36435]: 2024-03-24 00:12:23.388785576 +0000 UTC m=+0.560029543 container create 37d0db3ece2c5741886d1c9dc03a6f86a03ade018620b958b1ecfc944b6cccfd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mon-c, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-24T00:12:23.573 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 podman[36435]: 2024-03-24 00:12:23.57448954 +0000 UTC m=+0.745733516 container init 37d0db3ece2c5741886d1c9dc03a6f86a03ade018620b958b1ecfc944b6cccfd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mon-c, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True) 2024-03-24T00:12:23.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:23 smithi078 ceph-mon[32429]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:23.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.594+0000 7fcdd81c0700 1 -- 172.21.15.107:0/3759684172 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcdd00637c0 con 0x7fcdd0101ed0 2024-03-24T00:12:23.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.595+0000 7fcdc6ffd700 1 -- 172.21.15.107:0/3759684172 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+745 (secure 0 0 0) 0x7fcdcc030090 con 0x7fcdd0101ed0 2024-03-24T00:12:23.595 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-24T00:12:23.595 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":1,"fsid":"c7e2806a-e972-11ee-95cd-87774f69a715","modified":"2024-03-24T00:10:41.133103Z","created":"2024-03-24T00:10:41.133103Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T00:12:23.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.597+0000 7fcdd81c0700 1 -- 172.21.15.107:0/3759684172 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fcdbc038fc0 msgr2=0x7fcdbc03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:23.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.597+0000 7fcdd81c0700 1 --2- 172.21.15.107:0/3759684172 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fcdbc038fc0 0x7fcdbc03b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0006fd0 tx=0x7fcdc0006e40 comp rx=0 tx=0).stop 2024-03-24T00:12:23.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.597+0000 7fcdd81c0700 1 -- 172.21.15.107:0/3759684172 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcdd0101ed0 msgr2=0x7fcdd019d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:23.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.597+0000 7fcdd81c0700 1 --2- 172.21.15.107:0/3759684172 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcdd0101ed0 0x7fcdd019d420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc005850 tx=0x7fcdcc005030 comp rx=0 tx=0).stop 2024-03-24T00:12:23.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.597+0000 7fcdd81c0700 1 -- 172.21.15.107:0/3759684172 shutdown_connections 2024-03-24T00:12:23.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.597+0000 7fcdd81c0700 1 --2- 172.21.15.107:0/3759684172 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fcdbc038fc0 0x7fcdbc03b480 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:23.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.597+0000 7fcdd81c0700 1 --2- 172.21.15.107:0/3759684172 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcdd0101ed0 0x7fcdd019d420 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:23.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.597+0000 7fcdd81c0700 1 -- 172.21.15.107:0/3759684172 >> 172.21.15.107:0/3759684172 conn(0x7fcdd00fd760 msgr2=0x7fcdd01038a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:23.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.597+0000 7fcdd81c0700 1 -- 172.21.15.107:0/3759684172 shutdown_connections 2024-03-24T00:12:23.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:23.597+0000 7fcdd81c0700 1 -- 172.21.15.107:0/3759684172 wait complete. 2024-03-24T00:12:23.598 INFO:teuthology.orchestra.run.smithi107.stderr:dumped monmap epoch 1 2024-03-24T00:12:23.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 podman[36435]: 2024-03-24 00:12:23.577430341 +0000 UTC m=+0.748674320 container start 37d0db3ece2c5741886d1c9dc03a6f86a03ade018620b958b1ecfc944b6cccfd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mon-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-24T00:12:23.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T00:12:23.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev), process ceph-mon, pid 2 2024-03-24T00:12:23.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: pidfile_write: ignore empty --pid-file 2024-03-24T00:12:23.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: load: jerasure load: lrc 2024-03-24T00:12:23.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: RocksDB version: 7.9.2 2024-03-24T00:12:23.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Git sha 0 2024-03-24T00:12:23.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Compile date 2024-03-22 04:58:55 2024-03-24T00:12:23.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: DB SUMMARY 2024-03-24T00:12:23.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: DB Session ID: IQ80Z8B620NJ5SP3RCP5 2024-03-24T00:12:23.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: CURRENT file: CURRENT 2024-03-24T00:12:23.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T00:12:23.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-24T00:12:23.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-24T00:12:23.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-24T00:12:23.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.error_if_exists: 0 2024-03-24T00:12:23.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.create_if_missing: 0 2024-03-24T00:12:23.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.paranoid_checks: 1 2024-03-24T00:12:23.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T00:12:23.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T00:12:23.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T00:12:23.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.env: 0x55a0cc27cb20 2024-03-24T00:12:23.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T00:12:23.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.info_log: 0x55a0cd341920 2024-03-24T00:12:23.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T00:12:23.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.statistics: (nil) 2024-03-24T00:12:23.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.use_fsync: 0 2024-03-24T00:12:23.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.max_log_file_size: 0 2024-03-24T00:12:23.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T00:12:23.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T00:12:23.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T00:12:23.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T00:12:23.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.allow_fallocate: 1 2024-03-24T00:12:23.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T00:12:23.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T00:12:23.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.use_direct_reads: 0 2024-03-24T00:12:23.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T00:12:23.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T00:12:23.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.db_log_dir: 2024-03-24T00:12:23.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.wal_dir: 2024-03-24T00:12:23.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T00:12:23.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T00:12:23.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T00:12:23.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T00:12:23.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T00:12:23.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T00:12:23.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T00:12:23.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T00:12:23.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.write_buffer_manager: 0x55a0cd3141e0 2024-03-24T00:12:23.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T00:12:23.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T00:12:23.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T00:12:23.848 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T00:12:23.848 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T00:12:23.848 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T00:12:23.848 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T00:12:23.848 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T00:12:23.848 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.unordered_write: 0 2024-03-24T00:12:23.848 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T00:12:23.848 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T00:12:23.848 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T00:12:23.848 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T00:12:23.848 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.row_cache: None 2024-03-24T00:12:23.848 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.wal_filter: None 2024-03-24T00:12:23.848 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T00:12:23.848 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T00:12:23.849 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.two_write_queues: 0 2024-03-24T00:12:23.849 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T00:12:23.849 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.wal_compression: 0 2024-03-24T00:12:23.849 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.atomic_flush: 0 2024-03-24T00:12:23.849 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T00:12:23.849 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T00:12:23.850 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T00:12:23.850 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.log_readahead_size: 0 2024-03-24T00:12:23.851 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T00:12:23.851 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T00:12:23.851 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T00:12:23.851 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T00:12:23.851 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T00:12:23.851 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T00:12:23.851 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T00:12:23.851 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.max_background_jobs: 2 2024-03-24T00:12:23.851 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.max_background_compactions: -1 2024-03-24T00:12:23.851 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.max_subcompactions: 1 2024-03-24T00:12:23.851 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T00:12:23.851 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T00:12:23.851 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T00:12:23.851 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T00:12:23.851 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T00:12:23.852 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T00:12:23.852 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T00:12:23.852 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T00:12:23.852 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.max_open_files: -1 2024-03-24T00:12:23.852 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T00:12:23.852 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T00:12:23.852 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T00:12:23.852 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T00:12:23.852 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.max_background_flushes: -1 2024-03-24T00:12:23.852 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Compression algorithms supported: 2024-03-24T00:12:23.852 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T00:12:23.852 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: kZSTD supported: 0 2024-03-24T00:12:23.852 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: kXpressCompression supported: 0 2024-03-24T00:12:23.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T00:12:23.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: kZlibCompression supported: 1 2024-03-24T00:12:23.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: kSnappyCompression supported: 1 2024-03-24T00:12:23.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: kLZ4Compression supported: 1 2024-03-24T00:12:23.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: kBZip2Compression supported: 0 2024-03-24T00:12:23.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T00:12:23.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T00:12:23.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-24T00:12:23.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T00:12:23.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T00:12:23.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.merge_operator: 2024-03-24T00:12:23.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.compaction_filter: None 2024-03-24T00:12:23.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T00:12:23.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T00:12:23.854 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T00:12:23.854 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T00:12:23.854 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a0cd341a20) 2024-03-24T00:12:23.854 INFO:journalctl@ceph.mon.c.smithi107.stdout: cache_index_and_filter_blocks: 1 2024-03-24T00:12:23.854 INFO:journalctl@ceph.mon.c.smithi107.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T00:12:23.854 INFO:journalctl@ceph.mon.c.smithi107.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T00:12:23.854 INFO:journalctl@ceph.mon.c.smithi107.stdout: pin_top_level_index_and_filter: 1 2024-03-24T00:12:23.854 INFO:journalctl@ceph.mon.c.smithi107.stdout: index_type: 0 2024-03-24T00:12:23.854 INFO:journalctl@ceph.mon.c.smithi107.stdout: data_block_index_type: 0 2024-03-24T00:12:23.854 INFO:journalctl@ceph.mon.c.smithi107.stdout: index_shortening: 1 2024-03-24T00:12:23.854 INFO:journalctl@ceph.mon.c.smithi107.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T00:12:23.854 INFO:journalctl@ceph.mon.c.smithi107.stdout: checksum: 4 2024-03-24T00:12:23.854 INFO:journalctl@ceph.mon.c.smithi107.stdout: no_block_cache: 0 2024-03-24T00:12:23.854 INFO:journalctl@ceph.mon.c.smithi107.stdout: block_cache: 0x55a0cd347090 2024-03-24T00:12:23.855 INFO:journalctl@ceph.mon.c.smithi107.stdout: block_cache_name: BinnedLRUCache 2024-03-24T00:12:23.855 INFO:journalctl@ceph.mon.c.smithi107.stdout: block_cache_options: 2024-03-24T00:12:23.855 INFO:journalctl@ceph.mon.c.smithi107.stdout: capacity : 536870912 2024-03-24T00:12:23.855 INFO:journalctl@ceph.mon.c.smithi107.stdout: num_shard_bits : 4 2024-03-24T00:12:23.855 INFO:journalctl@ceph.mon.c.smithi107.stdout: strict_capacity_limit : 0 2024-03-24T00:12:23.855 INFO:journalctl@ceph.mon.c.smithi107.stdout: high_pri_pool_ratio: 0.000 2024-03-24T00:12:23.855 INFO:journalctl@ceph.mon.c.smithi107.stdout: block_cache_compressed: (nil) 2024-03-24T00:12:23.855 INFO:journalctl@ceph.mon.c.smithi107.stdout: persistent_cache: (nil) 2024-03-24T00:12:23.855 INFO:journalctl@ceph.mon.c.smithi107.stdout: block_size: 4096 2024-03-24T00:12:23.855 INFO:journalctl@ceph.mon.c.smithi107.stdout: block_size_deviation: 10 2024-03-24T00:12:23.856 INFO:journalctl@ceph.mon.c.smithi107.stdout: block_restart_interval: 16 2024-03-24T00:12:23.856 INFO:journalctl@ceph.mon.c.smithi107.stdout: index_block_restart_interval: 1 2024-03-24T00:12:23.856 INFO:journalctl@ceph.mon.c.smithi107.stdout: metadata_block_size: 4096 2024-03-24T00:12:23.856 INFO:journalctl@ceph.mon.c.smithi107.stdout: partition_filters: 0 2024-03-24T00:12:23.856 INFO:journalctl@ceph.mon.c.smithi107.stdout: use_delta_encoding: 1 2024-03-24T00:12:23.856 INFO:journalctl@ceph.mon.c.smithi107.stdout: filter_policy: bloomfilter 2024-03-24T00:12:23.856 INFO:journalctl@ceph.mon.c.smithi107.stdout: whole_key_filtering: 1 2024-03-24T00:12:23.856 INFO:journalctl@ceph.mon.c.smithi107.stdout: verify_compression: 0 2024-03-24T00:12:23.856 INFO:journalctl@ceph.mon.c.smithi107.stdout: read_amp_bytes_per_bit: 0 2024-03-24T00:12:23.856 INFO:journalctl@ceph.mon.c.smithi107.stdout: format_version: 5 2024-03-24T00:12:23.856 INFO:journalctl@ceph.mon.c.smithi107.stdout: enable_index_compression: 1 2024-03-24T00:12:23.856 INFO:journalctl@ceph.mon.c.smithi107.stdout: block_align: 0 2024-03-24T00:12:23.856 INFO:journalctl@ceph.mon.c.smithi107.stdout: max_auto_readahead_size: 262144 2024-03-24T00:12:23.856 INFO:journalctl@ceph.mon.c.smithi107.stdout: prepopulate_block_cache: 0 2024-03-24T00:12:23.856 INFO:journalctl@ceph.mon.c.smithi107.stdout: initial_auto_readahead_size: 8192 2024-03-24T00:12:23.857 INFO:journalctl@ceph.mon.c.smithi107.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T00:12:23.857 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T00:12:23.857 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T00:12:23.857 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.compression: NoCompression 2024-03-24T00:12:23.857 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T00:12:23.857 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T00:12:23.857 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T00:12:23.857 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.num_levels: 7 2024-03-24T00:12:23.857 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T00:12:23.857 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T00:12:23.857 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T00:12:23.857 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T00:12:23.857 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T00:12:23.857 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T00:12:23.857 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T00:12:23.858 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T00:12:23.858 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T00:12:23.858 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T00:12:23.858 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T00:12:23.858 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T00:12:23.858 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T00:12:23.858 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T00:12:23.858 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T00:12:23.858 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T00:12:23.858 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T00:12:23.858 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T00:12:23.858 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T00:12:23.858 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T00:12:23.858 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T00:12:23.858 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T00:12:23.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T00:12:23.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T00:12:23.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T00:12:23.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T00:12:23.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T00:12:23.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T00:12:23.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T00:12:23.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T00:12:23.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T00:12:23.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T00:12:23.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T00:12:23.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T00:12:23.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T00:12:23.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T00:12:23.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T00:12:23.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T00:12:23.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T00:12:23.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T00:12:23.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T00:12:23.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T00:12:23.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T00:12:23.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T00:12:23.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T00:12:23.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T00:12:23.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T00:12:23.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T00:12:23.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T00:12:23.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T00:12:23.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T00:12:23.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T00:12:23.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T00:12:23.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.table_properties_collectors: 2024-03-24T00:12:23.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.inplace_update_support: 0 2024-03-24T00:12:23.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T00:12:23.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T00:12:23.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T00:12:23.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T00:12:23.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.bloom_locality: 0 2024-03-24T00:12:23.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.max_successive_merges: 0 2024-03-24T00:12:23.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T00:12:23.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T00:12:23.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T00:12:23.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T00:12:23.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.ttl: 2592000 2024-03-24T00:12:23.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T00:12:23.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T00:12:23.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T00:12:23.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.enable_blob_files: false 2024-03-24T00:12:23.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.min_blob_size: 0 2024-03-24T00:12:23.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T00:12:23.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T00:12:23.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T00:12:23.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T00:12:23.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T00:12:23.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T00:12:23.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T00:12:23.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T00:12:23.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-24T00:12:23.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-24T00:12:23.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: dc711961-338b-4e7a-a366-c0d8f7562d0d 2024-03-24T00:12:23.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711239143607528, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-24T00:12:23.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-24T00:12:23.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711239143607956, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711239143, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "dc711961-338b-4e7a-a366-c0d8f7562d0d", "db_session_id": "IQ80Z8B620NJ5SP3RCP5", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-24T00:12:23.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711239143608045, "job": 1, "event": "recovery_finished"} 2024-03-24T00:12:23.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-24T00:12:23.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T00:12:23.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a0cd43a000 2024-03-24T00:12:23.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: DB pointer 0x55a0cd424000 2024-03-24T00:12:23.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T00:12:23.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T00:12:23.863 INFO:journalctl@ceph.mon.c.smithi107.stdout: ** DB Stats ** 2024-03-24T00:12:23.863 INFO:journalctl@ceph.mon.c.smithi107.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T00:12:23.865 INFO:journalctl@ceph.mon.c.smithi107.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T00:12:23.865 INFO:journalctl@ceph.mon.c.smithi107.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T00:12:23.865 INFO:journalctl@ceph.mon.c.smithi107.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T00:12:23.865 INFO:journalctl@ceph.mon.c.smithi107.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T00:12:23.866 INFO:journalctl@ceph.mon.c.smithi107.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T00:12:23.866 INFO:journalctl@ceph.mon.c.smithi107.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T00:12:23.866 INFO:journalctl@ceph.mon.c.smithi107.stdout: 2024-03-24T00:12:23.866 INFO:journalctl@ceph.mon.c.smithi107.stdout: ** Compaction Stats [default] ** 2024-03-24T00:12:23.866 INFO:journalctl@ceph.mon.c.smithi107.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T00:12:23.866 INFO:journalctl@ceph.mon.c.smithi107.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T00:12:23.866 INFO:journalctl@ceph.mon.c.smithi107.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T00:12:23.866 INFO:journalctl@ceph.mon.c.smithi107.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T00:12:23.866 INFO:journalctl@ceph.mon.c.smithi107.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T00:12:23.866 INFO:journalctl@ceph.mon.c.smithi107.stdout: 2024-03-24T00:12:23.866 INFO:journalctl@ceph.mon.c.smithi107.stdout: ** Compaction Stats [default] ** 2024-03-24T00:12:23.866 INFO:journalctl@ceph.mon.c.smithi107.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T00:12:23.866 INFO:journalctl@ceph.mon.c.smithi107.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T00:12:23.866 INFO:journalctl@ceph.mon.c.smithi107.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T00:12:23.867 INFO:journalctl@ceph.mon.c.smithi107.stdout: 2024-03-24T00:12:23.867 INFO:journalctl@ceph.mon.c.smithi107.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T00:12:23.867 INFO:journalctl@ceph.mon.c.smithi107.stdout: 2024-03-24T00:12:23.867 INFO:journalctl@ceph.mon.c.smithi107.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T00:12:23.867 INFO:journalctl@ceph.mon.c.smithi107.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T00:12:23.867 INFO:journalctl@ceph.mon.c.smithi107.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T00:12:23.867 INFO:journalctl@ceph.mon.c.smithi107.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T00:12:23.867 INFO:journalctl@ceph.mon.c.smithi107.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T00:12:23.867 INFO:journalctl@ceph.mon.c.smithi107.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T00:12:23.867 INFO:journalctl@ceph.mon.c.smithi107.stdout: Cumulative compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T00:12:23.867 INFO:journalctl@ceph.mon.c.smithi107.stdout: Interval compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T00:12:23.868 INFO:journalctl@ceph.mon.c.smithi107.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T00:12:23.868 INFO:journalctl@ceph.mon.c.smithi107.stdout: Block cache BinnedLRUCache@0x55a0cd347090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-24T00:12:23.868 INFO:journalctl@ceph.mon.c.smithi107.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-24T00:12:23.868 INFO:journalctl@ceph.mon.c.smithi107.stdout: 2024-03-24T00:12:23.868 INFO:journalctl@ceph.mon.c.smithi107.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T00:12:23.868 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-24T00:12:23.868 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: using public_addr v2:172.21.15.107:0/0 -> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2024-03-24T00:12:23.868 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: starting mon.c rank -1 at public addrs [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] at bind addrs [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid c7e2806a-e972-11ee-95cd-87774f69a715 2024-03-24T00:12:23.868 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: mon.c@-1(???) e0 preinit fsid c7e2806a-e972-11ee-95cd-87774f69a715 2024-03-24T00:12:23.868 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: mon.c@-1(synchronizing).mds e1 new map 2024-03-24T00:12:23.868 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-24T00:12:23.868 INFO:journalctl@ceph.mon.c.smithi107.stdout: e1 2024-03-24T00:12:23.869 INFO:journalctl@ceph.mon.c.smithi107.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T00:12:23.869 INFO:journalctl@ceph.mon.c.smithi107.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-24T00:12:23.869 INFO:journalctl@ceph.mon.c.smithi107.stdout: legacy client fscid: -1 2024-03-24T00:12:23.869 INFO:journalctl@ceph.mon.c.smithi107.stdout: 2024-03-24T00:12:23.869 INFO:journalctl@ceph.mon.c.smithi107.stdout: No filesystems configured 2024-03-24T00:12:23.869 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-24T00:12:23.869 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-24T00:12:23.869 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-24T00:12:23.869 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-24T00:12:23.869 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-24T00:12:23.869 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-24T00:12:23.869 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T00:12:23.869 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T00:12:23.869 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T00:12:23.870 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T00:12:23.870 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:12:23.870 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:23.870 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:23.870 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:23.870 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi107", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:12:23.870 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:23.870 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:23.870 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:23.870 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:23.870 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi106", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:12:23.870 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:23.870 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:12:23.870 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: Updating smithi106:/etc/ceph/ceph.conf 2024-03-24T00:12:23.870 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: Deploying cephadm binary to smithi107 2024-03-24T00:12:23.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: Updating smithi106:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:12:23.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: Updating smithi106:/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:12:23.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:23.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:23.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: Updating smithi106:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T00:12:23.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:23.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:23.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:23.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:23.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:12:23.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: Added host smithi107 2024-03-24T00:12:23.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:23.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:23.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:23.872 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:23.872 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:12:23.872 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:23.872 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3190464281' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T00:12:23.872 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3190464281' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T00:12:23.872 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T00:12:23.872 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:23.872 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:23.872 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:23.872 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:23.872 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:23.872 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi107", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:12:23.872 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:23.872 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:12:23.872 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: Updating smithi107:/etc/ceph/ceph.conf 2024-03-24T00:12:23.873 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: Updating smithi107:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:12:23.873 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:12:23.873 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: Updating smithi107:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T00:12:23.873 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:23.873 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:23.873 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:23.873 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:23.873 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:23.873 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:12:23.873 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:23.873 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:12:23.873 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:23.873 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:12:23.873 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:23.873 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi078:172.21.15.78=a;smithi106:172.21.15.106=b;smithi107:172.21.15.107=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:12:23.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: Saving service mon spec with placement smithi078:172.21.15.78=a;smithi106:172.21.15.106=b;smithi107:172.21.15.107=c;count:3 2024-03-24T00:12:23.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: Deploying daemon mon.c on smithi107 2024-03-24T00:12:23.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:23.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:23.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 ceph-mon[36485]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-24T00:12:23.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 bash[36435]: 37d0db3ece2c5741886d1c9dc03a6f86a03ade018620b958b1ecfc944b6cccfd 2024-03-24T00:12:23.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:23 smithi107 systemd[1]: Started Ceph mon.c for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:12:25.424 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T00:12:25.424 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T00:12:25.666 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:12:28.997 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:28 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:12:28.997 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:28 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:12:28.997 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:28 smithi078 ceph-mon[32429]: mon.a calling monitor election 2024-03-24T00:12:28.997 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:28 smithi078 ceph-mon[32429]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:28.997 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:28 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:12:28.998 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:28 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:12:28.998 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:28 smithi078 ceph-mon[32429]: mon.c calling monitor election 2024-03-24T00:12:28.998 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:28 smithi078 ceph-mon[32429]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:28.998 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:28 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:12:28.998 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:28 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:12:28.998 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:28 smithi078 ceph-mon[32429]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:28.998 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:28 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:12:28.998 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:28 smithi078 ceph-mon[32429]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T00:12:28.998 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:28 smithi078 ceph-mon[32429]: monmap e2: 2 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],c=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T00:12:28.998 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:28 smithi078 ceph-mon[32429]: fsmap 2024-03-24T00:12:28.998 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:28 smithi078 ceph-mon[32429]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T00:12:28.998 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:28 smithi078 ceph-mon[32429]: mgrmap e13: x(active, since 48s) 2024-03-24T00:12:28.998 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:28 smithi078 ceph-mon[32429]: overall HEALTH_OK 2024-03-24T00:12:28.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:28 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:28.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:28 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:28.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:28 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:28.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:28 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:12:28.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:28 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:28 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:12:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:28 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:12:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:28 smithi107 ceph-mon[36485]: mon.a calling monitor election 2024-03-24T00:12:29.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:28 smithi107 ceph-mon[36485]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:29.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:28 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:12:29.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:28 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:12:29.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:28 smithi107 ceph-mon[36485]: mon.c calling monitor election 2024-03-24T00:12:29.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:28 smithi107 ceph-mon[36485]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:29.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:28 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:12:29.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:28 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:12:29.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:28 smithi107 ceph-mon[36485]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:29.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:28 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:12:29.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:28 smithi107 ceph-mon[36485]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T00:12:29.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:28 smithi107 ceph-mon[36485]: monmap e2: 2 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],c=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T00:12:29.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:28 smithi107 ceph-mon[36485]: fsmap 2024-03-24T00:12:29.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:28 smithi107 ceph-mon[36485]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T00:12:29.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:28 smithi107 ceph-mon[36485]: mgrmap e13: x(active, since 48s) 2024-03-24T00:12:29.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:28 smithi107 ceph-mon[36485]: overall HEALTH_OK 2024-03-24T00:12:29.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:28 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:29.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:28 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:29.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:28 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:29.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:28 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:12:29.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:28 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:30.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:29 smithi078 ceph-mon[32429]: Deploying daemon mon.b on smithi106 2024-03-24T00:12:30.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:29 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:12:30.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.069+0000 7ff17871b700 1 -- 172.21.15.107:0/3447339269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff15c0035c0 msgr2=0x7ff15c005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:30.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.069+0000 7ff17871b700 1 --2- 172.21.15.107:0/3447339269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff15c0035c0 0x7ff15c005a50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff170100a70 tx=0x7ff168009e80 comp rx=0 tx=0).stop 2024-03-24T00:12:30.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.069+0000 7ff17871b700 1 -- 172.21.15.107:0/3447339269 shutdown_connections 2024-03-24T00:12:30.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.069+0000 7ff17871b700 1 --2- 172.21.15.107:0/3447339269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff15c0035c0 0x7ff15c005a50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:30.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.069+0000 7ff17871b700 1 -- 172.21.15.107:0/3447339269 >> 172.21.15.107:0/3447339269 conn(0x7ff1700fc630 msgr2=0x7ff1700fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:30.070 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:29 smithi107 ceph-mon[36485]: Deploying daemon mon.b on smithi106 2024-03-24T00:12:30.070 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:29 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:12:30.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.069+0000 7ff17871b700 1 -- 172.21.15.107:0/3447339269 shutdown_connections 2024-03-24T00:12:30.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.069+0000 7ff17871b700 1 -- 172.21.15.107:0/3447339269 wait complete. 2024-03-24T00:12:30.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.070+0000 7ff17871b700 1 Processor -- start 2024-03-24T00:12:30.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.070+0000 7ff17871b700 1 -- start start 2024-03-24T00:12:30.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.071+0000 7ff17871b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff15c0035c0 0x7ff17019d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:30.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.071+0000 7ff17871b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff17019daa0 0x7ff170197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:30.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.071+0000 7ff17871b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff170108bd0 con 0x7ff15c0035c0 2024-03-24T00:12:30.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.071+0000 7ff17871b700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff170108a50 con 0x7ff17019daa0 2024-03-24T00:12:30.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.071+0000 7ff175cb6700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff17019daa0 0x7ff170197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:30.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.071+0000 7ff175cb6700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff17019daa0 0x7ff170197630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60336/0 (socket says 172.21.15.107:60336) 2024-03-24T00:12:30.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.071+0000 7ff1764b7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff15c0035c0 0x7ff17019d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:30.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.071+0000 7ff175cb6700 1 -- 172.21.15.107:0/1624951005 learned_addr learned my addr 172.21.15.107:0/1624951005 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-03-24T00:12:30.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.071+0000 7ff175cb6700 1 -- 172.21.15.107:0/1624951005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff17019daa0 msgr2=0x7ff170197630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-24T00:12:30.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.071+0000 7ff175cb6700 1 -- 172.21.15.107:0/1624951005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff17019daa0 msgr2=0x7ff170197630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-24T00:12:30.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.071+0000 7ff175cb6700 1 --2- 172.21.15.107:0/1624951005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff17019daa0 0x7ff170197630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T00:12:30.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.071+0000 7ff175cb6700 1 --2- 172.21.15.107:0/1624951005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff17019daa0 0x7ff170197630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:12:30.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.071+0000 7ff1764b7700 1 -- 172.21.15.107:0/1624951005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff17019daa0 msgr2=0x7ff170197630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T00:12:30.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.071+0000 7ff1764b7700 1 --2- 172.21.15.107:0/1624951005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff17019daa0 0x7ff170197630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:30.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.071+0000 7ff1764b7700 1 -- 172.21.15.107:0/1624951005 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff17019e020 con 0x7ff15c0035c0 2024-03-24T00:12:30.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.072+0000 7ff1764b7700 1 --2- 172.21.15.107:0/1624951005 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff15c0035c0 0x7ff17019d560 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff168007590 tx=0x7ff168017890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:30.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.072+0000 7ff1677fe700 1 -- 172.21.15.107:0/1624951005 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7ff168005260 con 0x7ff15c0035c0 2024-03-24T00:12:30.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.072+0000 7ff1677fe700 1 -- 172.21.15.107:0/1624951005 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff168024030 con 0x7ff15c0035c0 2024-03-24T00:12:30.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.072+0000 7ff17871b700 1 -- 172.21.15.107:0/1624951005 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff170197bd0 con 0x7ff15c0035c0 2024-03-24T00:12:30.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.072+0000 7ff1677fe700 1 -- 172.21.15.107:0/1624951005 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7ff168020bf0 con 0x7ff15c0035c0 2024-03-24T00:12:30.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.072+0000 7ff17871b700 1 -- 172.21.15.107:0/1624951005 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff170198160 con 0x7ff15c0035c0 2024-03-24T00:12:30.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.073+0000 7ff1677fe700 1 -- 172.21.15.107:0/1624951005 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7ff16801e070 con 0x7ff15c0035c0 2024-03-24T00:12:30.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.073+0000 7ff17871b700 1 -- 172.21.15.107:0/1624951005 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff170102280 con 0x7ff15c0035c0 2024-03-24T00:12:30.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.073+0000 7ff1677fe700 1 --2- 172.21.15.107:0/1624951005 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff154038db0 0x7ff15403b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:30.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.073+0000 7ff1677fe700 1 -- 172.21.15.107:0/1624951005 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff16801c070 con 0x7ff15c0035c0 2024-03-24T00:12:30.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.074+0000 7ff175cb6700 1 --2- 172.21.15.107:0/1624951005 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff154038db0 0x7ff15403b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:30.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.074+0000 7ff175cb6700 1 --2- 172.21.15.107:0/1624951005 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff154038db0 0x7ff15403b270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff160009180 tx=0x7ff160008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:30.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.079+0000 7ff1677fe700 1 -- 172.21.15.107:0/1624951005 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff16801e360 con 0x7ff15c0035c0 2024-03-24T00:12:30.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.359+0000 7ff17871b700 1 -- 172.21.15.107:0/1624951005 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff1701984d0 con 0x7ff15c0035c0 2024-03-24T00:12:30.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.360+0000 7ff1677fe700 1 -- 172.21.15.107:0/1624951005 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1015 (secure 0 0 0) 0x7ff168028d90 con 0x7ff15c0035c0 2024-03-24T00:12:30.360 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-24T00:12:30.361 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":2,"fsid":"c7e2806a-e972-11ee-95cd-87774f69a715","modified":"2024-03-24T00:12:23.632059Z","created":"2024-03-24T00:10:41.133103Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T00:12:30.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.362+0000 7ff17871b700 1 -- 172.21.15.107:0/1624951005 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff154038db0 msgr2=0x7ff15403b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:30.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.362+0000 7ff17871b700 1 --2- 172.21.15.107:0/1624951005 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff154038db0 0x7ff15403b270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff160009180 tx=0x7ff160008040 comp rx=0 tx=0).stop 2024-03-24T00:12:30.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.362+0000 7ff17871b700 1 -- 172.21.15.107:0/1624951005 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff15c0035c0 msgr2=0x7ff17019d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:30.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.362+0000 7ff17871b700 1 --2- 172.21.15.107:0/1624951005 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff15c0035c0 0x7ff17019d560 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff168007590 tx=0x7ff168017890 comp rx=0 tx=0).stop 2024-03-24T00:12:30.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.362+0000 7ff17871b700 1 -- 172.21.15.107:0/1624951005 shutdown_connections 2024-03-24T00:12:30.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.362+0000 7ff17871b700 1 --2- 172.21.15.107:0/1624951005 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff154038db0 0x7ff15403b270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:30.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.362+0000 7ff17871b700 1 --2- 172.21.15.107:0/1624951005 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff15c0035c0 0x7ff17019d560 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:30.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.362+0000 7ff17871b700 1 --2- 172.21.15.107:0/1624951005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff17019daa0 0x7ff170197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:30.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.362+0000 7ff17871b700 1 -- 172.21.15.107:0/1624951005 >> 172.21.15.107:0/1624951005 conn(0x7ff1700fc630 msgr2=0x7ff1700fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:30.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.363+0000 7ff17871b700 1 -- 172.21.15.107:0/1624951005 shutdown_connections 2024-03-24T00:12:30.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:30.363+0000 7ff17871b700 1 -- 172.21.15.107:0/1624951005 wait complete. 2024-03-24T00:12:30.364 INFO:teuthology.orchestra.run.smithi107.stderr:dumped monmap epoch 2 2024-03-24T00:12:30.880 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:30 smithi107 ceph-mon[36485]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:30.880 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:30 smithi107 ceph-mon[36485]: from='client.? 172.21.15.107:0/1624951005' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T00:12:31.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:30 smithi078 ceph-mon[32429]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:31.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:30 smithi078 ceph-mon[32429]: from='client.? 172.21.15.107:0/1624951005' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T00:12:31.907 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T00:12:31.908 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T00:12:32.150 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:12:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:32 smithi106 systemd[1]: Started Ceph mon.b for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:12:38.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.042+0000 7f2774d9c700 1 -- 172.21.15.107:0/2442796833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27580035c0 msgr2=0x7f2758005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:38.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.042+0000 7f2774d9c700 1 --2- 172.21.15.107:0/2442796833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27580035c0 0x7f2758005a50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2770100ac0 tx=0x7f276000fe60 comp rx=0 tx=0).stop 2024-03-24T00:12:38.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.042+0000 7f2774d9c700 1 -- 172.21.15.107:0/2442796833 shutdown_connections 2024-03-24T00:12:38.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.042+0000 7f2774d9c700 1 --2- 172.21.15.107:0/2442796833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27580035c0 0x7f2758005a50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:38.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.042+0000 7f2774d9c700 1 -- 172.21.15.107:0/2442796833 >> 172.21.15.107:0/2442796833 conn(0x7f27700fc6a0 msgr2=0x7f27700feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:38.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.042+0000 7f2774d9c700 1 -- 172.21.15.107:0/2442796833 shutdown_connections 2024-03-24T00:12:38.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.042+0000 7f2774d9c700 1 -- 172.21.15.107:0/2442796833 wait complete. 2024-03-24T00:12:38.043 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:37 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:12:38.043 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:37 smithi107 ceph-mon[36485]: mon.a calling monitor election 2024-03-24T00:12:38.043 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:37 smithi107 ceph-mon[36485]: mon.c calling monitor election 2024-03-24T00:12:38.043 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:37 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:12:38.044 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:37 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:12:38.044 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:37 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:12:38.044 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:37 smithi107 ceph-mon[36485]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:38.044 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:37 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:12:38.044 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:37 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:12:38.044 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:37 smithi107 ceph-mon[36485]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:38.044 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:37 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:12:38.044 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:37 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:12:38.044 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:37 smithi107 ceph-mon[36485]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T00:12:38.044 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:37 smithi107 ceph-mon[36485]: monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0],c=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T00:12:38.044 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:37 smithi107 ceph-mon[36485]: fsmap 2024-03-24T00:12:38.044 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:37 smithi107 ceph-mon[36485]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T00:12:38.044 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:37 smithi107 ceph-mon[36485]: mgrmap e13: x(active, since 57s) 2024-03-24T00:12:38.044 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:37 smithi107 ceph-mon[36485]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-24T00:12:38.045 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:37 smithi107 ceph-mon[36485]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-24T00:12:38.045 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:37 smithi107 ceph-mon[36485]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-24T00:12:38.045 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:37 smithi107 ceph-mon[36485]: mon.b (rank 2) addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] is down (out of quorum) 2024-03-24T00:12:38.045 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:37 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:38.045 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:37 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:38.045 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:37 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:38.045 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:37 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:38.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.043+0000 7f2774d9c700 1 Processor -- start 2024-03-24T00:12:38.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.043+0000 7f2774d9c700 1 -- start start 2024-03-24T00:12:38.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.043+0000 7f2774d9c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27580035c0 0x7f2770198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:38.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.043+0000 7f2774d9c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2770198cb0 0x7f2770199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:38.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.043+0000 7f2774d9c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27701a17e0 0x7f27701a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:38.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.043+0000 7f2774d9c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2770108bd0 con 0x7f27701a17e0 2024-03-24T00:12:38.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.043+0000 7f2774d9c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2770108a50 con 0x7f27580035c0 2024-03-24T00:12:38.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.043+0000 7f2774d9c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2770108d50 con 0x7f2770198cb0 2024-03-24T00:12:38.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.044+0000 7f276dd9b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2770198cb0 0x7f2770199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:38.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.044+0000 7f276dd9b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2770198cb0 0x7f2770199130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58604/0 (socket says 172.21.15.107:58604) 2024-03-24T00:12:38.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.044+0000 7f276dd9b700 1 -- 172.21.15.107:0/2037819639 learned_addr learned my addr 172.21.15.107:0/2037819639 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-03-24T00:12:38.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.044+0000 7f276ed9d700 1 --2- 172.21.15.107:0/2037819639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27701a17e0 0x7f27701a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:38.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.044+0000 7f276e59c700 1 --2- 172.21.15.107:0/2037819639 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27580035c0 0x7f2770198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:38.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.044+0000 7f276dd9b700 1 -- 172.21.15.107:0/2037819639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2770198cb0 msgr2=0x7f2770199130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-24T00:12:38.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.044+0000 7f276dd9b700 1 -- 172.21.15.107:0/2037819639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2770198cb0 msgr2=0x7f2770199130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-24T00:12:38.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.044+0000 7f276dd9b700 1 --2- 172.21.15.107:0/2037819639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2770198cb0 0x7f2770199130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T00:12:38.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.044+0000 7f276dd9b700 1 --2- 172.21.15.107:0/2037819639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2770198cb0 0x7f2770199130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:12:38.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.044+0000 7f276e59c700 1 -- 172.21.15.107:0/2037819639 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27580035c0 msgr2=0x7f2770198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-24T00:12:38.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.044+0000 7f276e59c700 1 -- 172.21.15.107:0/2037819639 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27580035c0 msgr2=0x7f2770198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-24T00:12:38.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.044+0000 7f276e59c700 1 --2- 172.21.15.107:0/2037819639 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27580035c0 0x7f2770198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T00:12:38.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.044+0000 7f276e59c700 1 --2- 172.21.15.107:0/2037819639 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27580035c0 0x7f2770198770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:12:38.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.044+0000 7f276ed9d700 1 -- 172.21.15.107:0/2037819639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2770198cb0 msgr2=0x7f2770199130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T00:12:38.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.044+0000 7f276ed9d700 1 --2- 172.21.15.107:0/2037819639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2770198cb0 0x7f2770199130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:38.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.044+0000 7f276ed9d700 1 -- 172.21.15.107:0/2037819639 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27580035c0 msgr2=0x7f2770198770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T00:12:38.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.044+0000 7f276ed9d700 1 --2- 172.21.15.107:0/2037819639 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27580035c0 0x7f2770198770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:38.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.044+0000 7f276ed9d700 1 -- 172.21.15.107:0/2037819639 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27701a4410 con 0x7f27701a17e0 2024-03-24T00:12:38.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.044+0000 7f276ed9d700 1 --2- 172.21.15.107:0/2037819639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27701a17e0 0x7f27701a3bd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f276800bd50 tx=0x7f276800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:38.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.044+0000 7f275f7fe700 1 -- 172.21.15.107:0/2037819639 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2768019070 con 0x7f27701a17e0 2024-03-24T00:12:38.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.045+0000 7f2774d9c700 1 -- 172.21.15.107:0/2037819639 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27701a4700 con 0x7f27701a17e0 2024-03-24T00:12:38.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.045+0000 7f275f7fe700 1 -- 172.21.15.107:0/2037819639 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f276800cd10 con 0x7f27701a17e0 2024-03-24T00:12:38.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.045+0000 7f275f7fe700 1 -- 172.21.15.107:0/2037819639 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27680127e0 con 0x7f27701a17e0 2024-03-24T00:12:38.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.045+0000 7f2774d9c700 1 -- 172.21.15.107:0/2037819639 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27701a4c40 con 0x7f27701a17e0 2024-03-24T00:12:38.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.045+0000 7f275f7fe700 1 -- 172.21.15.107:0/2037819639 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f276801c070 con 0x7f27701a17e0 2024-03-24T00:12:38.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.045+0000 7f2774d9c700 1 -- 172.21.15.107:0/2037819639 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27700637c0 con 0x7f27701a17e0 2024-03-24T00:12:38.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.046+0000 7f275f7fe700 1 --2- 172.21.15.107:0/2037819639 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2750039450 0x7f275003b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:38.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.046+0000 7f275f7fe700 1 -- 172.21.15.107:0/2037819639 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f276804b970 con 0x7f27701a17e0 2024-03-24T00:12:38.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.046+0000 7f276e59c700 1 --2- 172.21.15.107:0/2037819639 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2750039450 0x7f275003b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:38.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.046+0000 7f276e59c700 1 --2- 172.21.15.107:0/2037819639 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2750039450 0x7f275003b910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2760020040 tx=0x7f2760001e70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:38.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.051+0000 7f275f7fe700 1 -- 172.21.15.107:0/2037819639 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2768014ab0 con 0x7f27701a17e0 2024-03-24T00:12:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:37 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:12:38.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:37 smithi078 ceph-mon[32429]: mon.a calling monitor election 2024-03-24T00:12:38.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:37 smithi078 ceph-mon[32429]: mon.c calling monitor election 2024-03-24T00:12:38.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:37 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:12:38.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:37 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:12:38.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:37 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:12:38.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:37 smithi078 ceph-mon[32429]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:38.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:37 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:12:38.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:37 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:12:38.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:37 smithi078 ceph-mon[32429]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:38.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:37 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:12:38.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:37 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:12:38.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:37 smithi078 ceph-mon[32429]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T00:12:38.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:37 smithi078 ceph-mon[32429]: monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0],c=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T00:12:38.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:37 smithi078 ceph-mon[32429]: fsmap 2024-03-24T00:12:38.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:37 smithi078 ceph-mon[32429]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T00:12:38.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:37 smithi078 ceph-mon[32429]: mgrmap e13: x(active, since 57s) 2024-03-24T00:12:38.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:37 smithi078 ceph-mon[32429]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-24T00:12:38.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:37 smithi078 ceph-mon[32429]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-24T00:12:38.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:37 smithi078 ceph-mon[32429]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-24T00:12:38.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:37 smithi078 ceph-mon[32429]: mon.b (rank 2) addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] is down (out of quorum) 2024-03-24T00:12:38.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:37 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:38.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:37 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:38.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:37 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:38.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:37 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:38.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.344+0000 7f2774d9c700 1 -- 172.21.15.107:0/2037819639 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f277002da30 con 0x7f27701a17e0 2024-03-24T00:12:38.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.344+0000 7f275f7fe700 1 -- 172.21.15.107:0/2037819639 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1283 (secure 0 0 0) 0x7f276800e090 con 0x7f27701a17e0 2024-03-24T00:12:38.345 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-24T00:12:38.345 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":3,"fsid":"c7e2806a-e972-11ee-95cd-87774f69a715","modified":"2024-03-24T00:12:32.554263Z","created":"2024-03-24T00:10:41.133103Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:3300","nonce":0},{"type":"v1","addr":"172.21.15.106:6789","nonce":0}]},"addr":"172.21.15.106:6789/0","public_addr":"172.21.15.106:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T00:12:38.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.346+0000 7f2774d9c700 1 -- 172.21.15.107:0/2037819639 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2750039450 msgr2=0x7f275003b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:38.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.346+0000 7f2774d9c700 1 --2- 172.21.15.107:0/2037819639 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2750039450 0x7f275003b910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2760020040 tx=0x7f2760001e70 comp rx=0 tx=0).stop 2024-03-24T00:12:38.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.346+0000 7f2774d9c700 1 -- 172.21.15.107:0/2037819639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27701a17e0 msgr2=0x7f27701a3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:38.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.346+0000 7f2774d9c700 1 --2- 172.21.15.107:0/2037819639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27701a17e0 0x7f27701a3bd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f276800bd50 tx=0x7f276800be30 comp rx=0 tx=0).stop 2024-03-24T00:12:38.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.347+0000 7f2774d9c700 1 -- 172.21.15.107:0/2037819639 shutdown_connections 2024-03-24T00:12:38.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.347+0000 7f2774d9c700 1 --2- 172.21.15.107:0/2037819639 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2750039450 0x7f275003b910 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:38.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.347+0000 7f2774d9c700 1 --2- 172.21.15.107:0/2037819639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27701a17e0 0x7f27701a3bd0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:38.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.347+0000 7f2774d9c700 1 --2- 172.21.15.107:0/2037819639 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27580035c0 0x7f2770198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:38.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.347+0000 7f2774d9c700 1 --2- 172.21.15.107:0/2037819639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2770198cb0 0x7f2770199130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:38.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.347+0000 7f2774d9c700 1 -- 172.21.15.107:0/2037819639 >> 172.21.15.107:0/2037819639 conn(0x7f27700fc6a0 msgr2=0x7f27700fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:38.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.347+0000 7f2774d9c700 1 -- 172.21.15.107:0/2037819639 shutdown_connections 2024-03-24T00:12:38.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:38.347+0000 7f2774d9c700 1 -- 172.21.15.107:0/2037819639 wait complete. 2024-03-24T00:12:38.348 INFO:teuthology.orchestra.run.smithi107.stderr:dumped monmap epoch 3 2024-03-24T00:12:38.898 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:38 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:12:38.899 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:38 smithi107 ceph-mon[36485]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:38.899 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:38 smithi107 ceph-mon[36485]: from='client.? 172.21.15.107:0/2037819639' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T00:12:38.899 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:38 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:12:38.900 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-24T00:12:38.900 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph config generate-minimal-conf 2024-03-24T00:12:38.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:38 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:12:38.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:38 smithi078 ceph-mon[32429]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:38.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:38 smithi078 ceph-mon[32429]: from='client.? 172.21.15.107:0/2037819639' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T00:12:38.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:38 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:12:39.154 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:12:39.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:39 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:12:39.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:39 smithi106 ceph-mon[36505]: mon.a calling monitor election 2024-03-24T00:12:39.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:39 smithi106 ceph-mon[36505]: mon.c calling monitor election 2024-03-24T00:12:39.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:39 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:12:39.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:39 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:12:39.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:39 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:12:39.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:39 smithi106 ceph-mon[36505]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:39.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:39 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:12:39.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:39 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:12:39.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:39 smithi106 ceph-mon[36505]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:39.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:39 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:12:39.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:39 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:12:39.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:39 smithi106 ceph-mon[36505]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T00:12:39.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:39 smithi106 ceph-mon[36505]: monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0],c=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T00:12:39.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:39 smithi106 ceph-mon[36505]: fsmap 2024-03-24T00:12:39.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:39 smithi106 ceph-mon[36505]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T00:12:39.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:39 smithi106 ceph-mon[36505]: mgrmap e13: x(active, since 57s) 2024-03-24T00:12:39.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:39 smithi106 ceph-mon[36505]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-24T00:12:39.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:39 smithi106 ceph-mon[36505]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-24T00:12:39.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:39 smithi106 ceph-mon[36505]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-24T00:12:39.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:39 smithi106 ceph-mon[36505]: mon.b (rank 2) addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] is down (out of quorum) 2024-03-24T00:12:39.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:39 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:39.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:39 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:39.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:39 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:39.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:39 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:39.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:39 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:12:39.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:39 smithi106 ceph-mon[36505]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:39.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:39 smithi106 ceph-mon[36505]: from='client.? 172.21.15.107:0/2037819639' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T00:12:39.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:39 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:12:40.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.727+0000 7f67c59f7700 1 -- 172.21.15.78:0/1198589601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67c0106a70 msgr2=0x7f67c0106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:40.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.727+0000 7f67c59f7700 1 --2- 172.21.15.78:0/1198589601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67c0106a70 0x7f67c0106e50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f67b0009910 tx=0x7f67b0009c20 comp rx=0 tx=0).stop 2024-03-24T00:12:40.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.727+0000 7f67c59f7700 1 -- 172.21.15.78:0/1198589601 shutdown_connections 2024-03-24T00:12:40.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.727+0000 7f67c59f7700 1 --2- 172.21.15.78:0/1198589601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67c0106a70 0x7f67c0106e50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:40.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.727+0000 7f67c59f7700 1 -- 172.21.15.78:0/1198589601 >> 172.21.15.78:0/1198589601 conn(0x7f67c00fc580 msgr2=0x7f67c00fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:40.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.727+0000 7f67c59f7700 1 -- 172.21.15.78:0/1198589601 shutdown_connections 2024-03-24T00:12:40.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.727+0000 7f67c59f7700 1 -- 172.21.15.78:0/1198589601 wait complete. 2024-03-24T00:12:40.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.728+0000 7f67c59f7700 1 Processor -- start 2024-03-24T00:12:40.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.728+0000 7f67c59f7700 1 -- start start 2024-03-24T00:12:40.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.728+0000 7f67c59f7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67c0106a70 0x7f67c0198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:40.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.728+0000 7f67c59f7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67c0198c80 0x7f67c0199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:40.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.728+0000 7f67c59f7700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67c01a17b0 0x7f67c01a3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:40.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.728+0000 7f67c59f7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c010b060 con 0x7f67c0106a70 2024-03-24T00:12:40.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.728+0000 7f67c59f7700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c010aee0 con 0x7f67c0198c80 2024-03-24T00:12:40.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.728+0000 7f67c59f7700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c010b1e0 con 0x7f67c01a17b0 2024-03-24T00:12:40.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.728+0000 7f67beffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67c0106a70 0x7f67c0198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:40.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.728+0000 7f67beffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67c0106a70 0x7f67c0198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52770/0 (socket says 172.21.15.78:52770) 2024-03-24T00:12:40.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.728+0000 7f67beffd700 1 -- 172.21.15.78:0/2430974420 learned_addr learned my addr 172.21.15.78:0/2430974420 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:12:40.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.729+0000 7f67bf7fe700 1 --2- 172.21.15.78:0/2430974420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67c01a17b0 0x7f67c01a3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:40.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.729+0000 7f67be7fc700 1 --2- 172.21.15.78:0/2430974420 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67c0198c80 0x7f67c0199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:40.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.729+0000 7f67beffd700 1 -- 172.21.15.78:0/2430974420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67c01a17b0 msgr2=0x7f67c01a3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:40.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.729+0000 7f67beffd700 1 --2- 172.21.15.78:0/2430974420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67c01a17b0 0x7f67c01a3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:40.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.729+0000 7f67beffd700 1 -- 172.21.15.78:0/2430974420 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67c0198c80 msgr2=0x7f67c0199100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:40.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.729+0000 7f67beffd700 1 --2- 172.21.15.78:0/2430974420 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67c0198c80 0x7f67c0199100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:40.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.729+0000 7f67beffd700 1 -- 172.21.15.78:0/2430974420 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c01a4410 con 0x7f67c0106a70 2024-03-24T00:12:40.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.729+0000 7f67beffd700 1 --2- 172.21.15.78:0/2430974420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67c0106a70 0x7f67c0198740 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f67b000bdc0 tx=0x7f67b000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:40.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.729+0000 7f67c49f5700 1 -- 172.21.15.78:0/2430974420 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b0026070 con 0x7f67c0106a70 2024-03-24T00:12:40.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.729+0000 7f67c59f7700 1 -- 172.21.15.78:0/2430974420 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67c01a4700 con 0x7f67c0106a70 2024-03-24T00:12:40.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.729+0000 7f67c49f5700 1 -- 172.21.15.78:0/2430974420 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67b00050d0 con 0x7f67c0106a70 2024-03-24T00:12:40.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.729+0000 7f67c49f5700 1 -- 172.21.15.78:0/2430974420 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b000fdf0 con 0x7f67c0106a70 2024-03-24T00:12:40.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.729+0000 7f67c59f7700 1 -- 172.21.15.78:0/2430974420 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67c01a4c90 con 0x7f67c0106a70 2024-03-24T00:12:40.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.730+0000 7f67c49f5700 1 -- 172.21.15.78:0/2430974420 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f67b0029070 con 0x7f67c0106a70 2024-03-24T00:12:40.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.730+0000 7f67c59f7700 1 -- 172.21.15.78:0/2430974420 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67c00637c0 con 0x7f67c0106a70 2024-03-24T00:12:40.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.730+0000 7f67c49f5700 1 --2- 172.21.15.78:0/2430974420 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f67a8039450 0x7f67a803b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:40.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.730+0000 7f67c49f5700 1 -- 172.21.15.78:0/2430974420 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f67b004cbe0 con 0x7f67c0106a70 2024-03-24T00:12:40.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.731+0000 7f67be7fc700 1 --2- 172.21.15.78:0/2430974420 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f67a8039450 0x7f67a803b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:40.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.731+0000 7f67be7fc700 1 --2- 172.21.15.78:0/2430974420 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f67a8039450 0x7f67a803b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f67b4009850 tx=0x7f67b4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:40.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.737+0000 7f67c49f5700 1 -- 172.21.15.78:0/2430974420 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f67b0013070 con 0x7f67c0106a70 2024-03-24T00:12:40.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: mon.b calling monitor election 2024-03-24T00:12:40.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: Updating smithi078:/etc/ceph/ceph.conf 2024-03-24T00:12:40.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: Updating smithi106:/etc/ceph/ceph.conf 2024-03-24T00:12:40.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: Updating smithi107:/etc/ceph/ceph.conf 2024-03-24T00:12:40.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: mon.b calling monitor election 2024-03-24T00:12:40.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: mon.c calling monitor election 2024-03-24T00:12:40.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: mon.a calling monitor election 2024-03-24T00:12:40.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T00:12:40.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0],c=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T00:12:40.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: fsmap 2024-03-24T00:12:40.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T00:12:40.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: mgrmap e13: x(active, since 59s) 2024-03-24T00:12:40.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-24T00:12:40.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:12:40.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: overall HEALTH_OK 2024-03-24T00:12:40.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: Updating smithi078:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:12:40.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: Updating smithi106:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:12:40.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: Updating smithi107:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:12:40.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T00:12:40.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:12:40.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T00:12:40.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:40.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: Reconfiguring daemon mon.a on smithi078 2024-03-24T00:12:40.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:40.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: mon.b calling monitor election 2024-03-24T00:12:40.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: Updating smithi078:/etc/ceph/ceph.conf 2024-03-24T00:12:40.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: Updating smithi106:/etc/ceph/ceph.conf 2024-03-24T00:12:40.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: Updating smithi107:/etc/ceph/ceph.conf 2024-03-24T00:12:40.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: mon.b calling monitor election 2024-03-24T00:12:40.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: mon.c calling monitor election 2024-03-24T00:12:40.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: mon.a calling monitor election 2024-03-24T00:12:40.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T00:12:40.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0],c=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T00:12:40.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: fsmap 2024-03-24T00:12:40.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T00:12:40.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: mgrmap e13: x(active, since 59s) 2024-03-24T00:12:40.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-24T00:12:40.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:12:40.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: overall HEALTH_OK 2024-03-24T00:12:40.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: Updating smithi078:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:12:40.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: Updating smithi106:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:12:40.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: Updating smithi107:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:12:40.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T00:12:40.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:12:40.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T00:12:40.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:40.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: Reconfiguring daemon mon.a on smithi078 2024-03-24T00:12:40.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:40.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: mon.b calling monitor election 2024-03-24T00:12:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: Updating smithi078:/etc/ceph/ceph.conf 2024-03-24T00:12:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: Updating smithi106:/etc/ceph/ceph.conf 2024-03-24T00:12:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: Updating smithi107:/etc/ceph/ceph.conf 2024-03-24T00:12:40.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: mon.b calling monitor election 2024-03-24T00:12:40.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: mon.c calling monitor election 2024-03-24T00:12:40.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: mon.a calling monitor election 2024-03-24T00:12:40.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T00:12:40.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0],c=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T00:12:40.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: fsmap 2024-03-24T00:12:40.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T00:12:40.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: mgrmap e13: x(active, since 59s) 2024-03-24T00:12:40.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-24T00:12:40.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:12:40.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: overall HEALTH_OK 2024-03-24T00:12:40.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: Updating smithi078:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:12:40.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: Updating smithi106:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:12:40.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: Updating smithi107:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:12:40.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T00:12:40.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:12:40.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T00:12:40.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:40.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: Reconfiguring daemon mon.a on smithi078 2024-03-24T00:12:40.905 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:40.905 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:40.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.945+0000 7f67c59f7700 1 -- 172.21.15.78:0/2430974420 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f67c01a4ef0 con 0x7f67c0106a70 2024-03-24T00:12:40.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.946+0000 7f67c49f5700 1 -- 172.21.15.78:0/2430974420 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+275 (secure 0 0 0) 0x7f67b0030090 con 0x7f67c0106a70 2024-03-24T00:12:40.946 INFO:teuthology.orchestra.run.smithi078.stdout:# minimal ceph.conf for c7e2806a-e972-11ee-95cd-87774f69a715 2024-03-24T00:12:40.946 INFO:teuthology.orchestra.run.smithi078.stdout:[global] 2024-03-24T00:12:40.946 INFO:teuthology.orchestra.run.smithi078.stdout: fsid = c7e2806a-e972-11ee-95cd-87774f69a715 2024-03-24T00:12:40.946 INFO:teuthology.orchestra.run.smithi078.stdout: mon_host = [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2024-03-24T00:12:40.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.948+0000 7f67c59f7700 1 -- 172.21.15.78:0/2430974420 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f67a8039450 msgr2=0x7f67a803b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:40.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.948+0000 7f67c59f7700 1 --2- 172.21.15.78:0/2430974420 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f67a8039450 0x7f67a803b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f67b4009850 tx=0x7f67b4008040 comp rx=0 tx=0).stop 2024-03-24T00:12:40.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.948+0000 7f67c59f7700 1 -- 172.21.15.78:0/2430974420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67c0106a70 msgr2=0x7f67c0198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:40.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.948+0000 7f67c59f7700 1 --2- 172.21.15.78:0/2430974420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67c0106a70 0x7f67c0198740 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f67b000bdc0 tx=0x7f67b000bea0 comp rx=0 tx=0).stop 2024-03-24T00:12:40.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.948+0000 7f67c59f7700 1 -- 172.21.15.78:0/2430974420 shutdown_connections 2024-03-24T00:12:40.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.948+0000 7f67c59f7700 1 --2- 172.21.15.78:0/2430974420 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f67a8039450 0x7f67a803b910 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:40.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.948+0000 7f67c59f7700 1 --2- 172.21.15.78:0/2430974420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67c01a17b0 0x7f67c01a3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:40.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.948+0000 7f67c59f7700 1 --2- 172.21.15.78:0/2430974420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67c0106a70 0x7f67c0198740 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:40.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.948+0000 7f67c59f7700 1 --2- 172.21.15.78:0/2430974420 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67c0198c80 0x7f67c0199100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:40.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.948+0000 7f67c59f7700 1 -- 172.21.15.78:0/2430974420 >> 172.21.15.78:0/2430974420 conn(0x7f67c00fc580 msgr2=0x7f67c0102570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:40.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.948+0000 7f67c59f7700 1 -- 172.21.15.78:0/2430974420 shutdown_connections 2024-03-24T00:12:40.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:40.948+0000 7f67c59f7700 1 -- 172.21.15.78:0/2430974420 wait complete. 2024-03-24T00:12:41.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:41 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:12:41.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:41 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2430974420' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:41.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:41 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:12:41.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:41 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2430974420' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:41.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:41 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:12:41.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:41 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2430974420' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:42.232 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-24T00:12:42.233 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-24T00:12:42.233 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T00:12:42.302 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-24T00:12:42.302 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:12:42.369 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-24T00:12:42.369 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T00:12:42.399 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-24T00:12:42.400 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:12:42.466 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-24T00:12:42.466 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T00:12:42.500 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-24T00:12:42.500 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:12:42.570 INFO:tasks.cephadm:Adding mgr.x on smithi078 2024-03-24T00:12:42.571 INFO:tasks.cephadm:Adding mgr.y on smithi106 2024-03-24T00:12:42.571 INFO:tasks.cephadm:Adding mgr.z on smithi107 2024-03-24T00:12:42.571 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph orch apply mgr '3;smithi078=x;smithi106=y;smithi107=z' 2024-03-24T00:12:42.842 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:12:42.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:42 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:42.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:42 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:42.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:42 smithi106 ceph-mon[36505]: Reconfiguring mon.b (monmap changed)... 2024-03-24T00:12:42.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:42 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:12:42.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:42 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T00:12:42.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:42 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:42.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:42 smithi106 ceph-mon[36505]: Reconfiguring daemon mon.b on smithi106 2024-03-24T00:12:42.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:42 smithi106 ceph-mon[36505]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:43.138 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:42 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:43.139 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:42 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:43.139 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:42 smithi107 ceph-mon[36485]: Reconfiguring mon.b (monmap changed)... 2024-03-24T00:12:43.139 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:42 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:12:43.139 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:42 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T00:12:43.139 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:42 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:43.139 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:42 smithi107 ceph-mon[36485]: Reconfiguring daemon mon.b on smithi106 2024-03-24T00:12:43.139 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:42 smithi107 ceph-mon[36485]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:42 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:42 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:42 smithi078 ceph-mon[32429]: Reconfiguring mon.b (monmap changed)... 2024-03-24T00:12:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:42 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:12:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:42 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T00:12:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:42 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:43.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:42 smithi078 ceph-mon[32429]: Reconfiguring daemon mon.b on smithi106 2024-03-24T00:12:43.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:42 smithi078 ceph-mon[32429]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:44.546 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:44 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:44.546 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:44 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:44.547 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:44 smithi107 ceph-mon[36485]: Reconfiguring mon.c (monmap changed)... 2024-03-24T00:12:44.547 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:44 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:12:44.547 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:44 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T00:12:44.547 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:44 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:44.547 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:44 smithi107 ceph-mon[36485]: Reconfiguring daemon mon.c on smithi107 2024-03-24T00:12:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:44 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:44 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:44 smithi106 ceph-mon[36505]: Reconfiguring mon.c (monmap changed)... 2024-03-24T00:12:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:44 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:12:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:44 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T00:12:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:44 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:44 smithi106 ceph-mon[36505]: Reconfiguring daemon mon.c on smithi107 2024-03-24T00:12:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:44 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:44 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:44 smithi078 ceph-mon[32429]: Reconfiguring mon.c (monmap changed)... 2024-03-24T00:12:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:44 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:12:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:44 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T00:12:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:44 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:44 smithi078 ceph-mon[32429]: Reconfiguring daemon mon.c on smithi107 2024-03-24T00:12:44.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.958+0000 7f0587402700 1 -- 172.21.15.107:0/975843649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05800fff80 msgr2=0x7f05801003a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:44.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.958+0000 7f0587402700 1 --2- 172.21.15.107:0/975843649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05800fff80 0x7f05801003a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0570009910 tx=0x7f0570009c20 comp rx=0 tx=0).stop 2024-03-24T00:12:45.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.958+0000 7f0587402700 1 -- 172.21.15.107:0/975843649 shutdown_connections 2024-03-24T00:12:45.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.958+0000 7f0587402700 1 --2- 172.21.15.107:0/975843649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05800fff80 0x7f05801003a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:45.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.958+0000 7f0587402700 1 -- 172.21.15.107:0/975843649 >> 172.21.15.107:0/975843649 conn(0x7f05800fb8e0 msgr2=0x7f05800fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:45.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.958+0000 7f0587402700 1 -- 172.21.15.107:0/975843649 shutdown_connections 2024-03-24T00:12:45.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.958+0000 7f0587402700 1 -- 172.21.15.107:0/975843649 wait complete. 2024-03-24T00:12:45.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.958+0000 7f0587402700 1 Processor -- start 2024-03-24T00:12:45.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.958+0000 7f0587402700 1 -- start start 2024-03-24T00:12:45.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.958+0000 7f0587402700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05800fff80 0x7f0580197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:45.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.958+0000 7f0587402700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0580197a60 0x7f058019cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:45.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.958+0000 7f0587402700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0580197ee0 0x7f058019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:45.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.958+0000 7f0587402700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0580107d00 con 0x7f05800fff80 2024-03-24T00:12:45.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.958+0000 7f0587402700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0580107b80 con 0x7f0580197ee0 2024-03-24T00:12:45.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.958+0000 7f0587402700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0580107e80 con 0x7f0580197a60 2024-03-24T00:12:45.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.959+0000 7f058499d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0580197a60 0x7f058019cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:45.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.959+0000 7f058519e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05800fff80 0x7f0580197520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:45.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.959+0000 7f058499d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0580197a60 0x7f058019cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58610/0 (socket says 172.21.15.107:58610) 2024-03-24T00:12:45.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.959+0000 7f058499d700 1 -- 172.21.15.107:0/3323887975 learned_addr learned my addr 172.21.15.107:0/3323887975 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-03-24T00:12:45.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.959+0000 7f058519e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05800fff80 0x7f0580197520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.107:41844/0 (socket says 172.21.15.107:41844) 2024-03-24T00:12:45.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.959+0000 7f058599f700 1 --2- 172.21.15.107:0/3323887975 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0580197ee0 0x7f058019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:45.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.959+0000 7f058499d700 1 -- 172.21.15.107:0/3323887975 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0580197ee0 msgr2=0x7f058019f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:45.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.959+0000 7f058499d700 1 --2- 172.21.15.107:0/3323887975 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0580197ee0 0x7f058019f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:45.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.959+0000 7f058499d700 1 -- 172.21.15.107:0/3323887975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05800fff80 msgr2=0x7f0580197520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:45.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.959+0000 7f058499d700 1 --2- 172.21.15.107:0/3323887975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05800fff80 0x7f0580197520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:45.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.959+0000 7f058499d700 1 -- 172.21.15.107:0/3323887975 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f058019f930 con 0x7f0580197a60 2024-03-24T00:12:45.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.959+0000 7f058599f700 1 --2- 172.21.15.107:0/3323887975 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0580197ee0 0x7f058019f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:12:45.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.959+0000 7f058519e700 1 --2- 172.21.15.107:0/3323887975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05800fff80 0x7f0580197520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:12:45.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.959+0000 7f058499d700 1 --2- 172.21.15.107:0/3323887975 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0580197a60 0x7f058019cad0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f057c00b780 tx=0x7f057c00ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:45.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.959+0000 7f05767fc700 1 -- 172.21.15.107:0/3323887975 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057c004230 con 0x7f0580197a60 2024-03-24T00:12:45.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.959+0000 7f0587402700 1 -- 172.21.15.107:0/3323887975 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f058019fc20 con 0x7f0580197a60 2024-03-24T00:12:45.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.959+0000 7f05767fc700 1 -- 172.21.15.107:0/3323887975 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f057c0043d0 con 0x7f0580197a60 2024-03-24T00:12:45.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.959+0000 7f05767fc700 1 -- 172.21.15.107:0/3323887975 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057c011c90 con 0x7f0580197a60 2024-03-24T00:12:45.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.959+0000 7f0587402700 1 -- 172.21.15.107:0/3323887975 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05801a0230 con 0x7f0580197a60 2024-03-24T00:12:45.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.960+0000 7f05767fc700 1 -- 172.21.15.107:0/3323887975 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f057c01a400 con 0x7f0580197a60 2024-03-24T00:12:45.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.960+0000 7f05767fc700 1 --2- 172.21.15.107:0/3323887975 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f056c039450 0x7f056c03b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:45.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.960+0000 7f05767fc700 1 -- 172.21.15.107:0/3323887975 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f057c04bc10 con 0x7f0580197a60 2024-03-24T00:12:45.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.960+0000 7f0587402700 1 -- 172.21.15.107:0/3323887975 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05800637c0 con 0x7f0580197a60 2024-03-24T00:12:45.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.960+0000 7f058519e700 1 --2- 172.21.15.107:0/3323887975 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f056c039450 0x7f056c03b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:45.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.961+0000 7f058519e700 1 --2- 172.21.15.107:0/3323887975 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f056c039450 0x7f056c03b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0570005200 tx=0x7f057001a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:45.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:44.966+0000 7f05767fc700 1 -- 172.21.15.107:0/3323887975 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f057c020500 con 0x7f0580197a60 2024-03-24T00:12:45.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:45.176+0000 7f0587402700 1 -- 172.21.15.107:0/3323887975 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi078=x;smithi106=y;smithi107=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f0580062720 con 0x7f056c039450 2024-03-24T00:12:45.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:45.182+0000 7f05767fc700 1 -- 172.21.15.107:0/3323887975 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f0580062720 con 0x7f056c039450 2024-03-24T00:12:45.383 INFO:teuthology.orchestra.run.smithi107.stdout:Scheduled mgr update... 2024-03-24T00:12:45.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:45.185+0000 7f0587402700 1 -- 172.21.15.107:0/3323887975 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f056c039450 msgr2=0x7f056c03b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:45.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:45.185+0000 7f0587402700 1 --2- 172.21.15.107:0/3323887975 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f056c039450 0x7f056c03b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0570005200 tx=0x7f057001a040 comp rx=0 tx=0).stop 2024-03-24T00:12:45.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:45.185+0000 7f0587402700 1 -- 172.21.15.107:0/3323887975 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0580197a60 msgr2=0x7f058019cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:45.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:45.185+0000 7f0587402700 1 --2- 172.21.15.107:0/3323887975 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0580197a60 0x7f058019cad0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f057c00b780 tx=0x7f057c00ba90 comp rx=0 tx=0).stop 2024-03-24T00:12:45.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:45.185+0000 7f0587402700 1 -- 172.21.15.107:0/3323887975 shutdown_connections 2024-03-24T00:12:45.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:45.185+0000 7f0587402700 1 --2- 172.21.15.107:0/3323887975 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f056c039450 0x7f056c03b910 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:45.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:45.185+0000 7f0587402700 1 --2- 172.21.15.107:0/3323887975 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0580197ee0 0x7f058019f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:45.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:45.185+0000 7f0587402700 1 --2- 172.21.15.107:0/3323887975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05800fff80 0x7f0580197520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:45.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:45.185+0000 7f0587402700 1 --2- 172.21.15.107:0/3323887975 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0580197a60 0x7f058019cad0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:45.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:45.185+0000 7f0587402700 1 -- 172.21.15.107:0/3323887975 >> 172.21.15.107:0/3323887975 conn(0x7f05800fb8e0 msgr2=0x7f05800fc650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:45.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:45.185+0000 7f0587402700 1 -- 172.21.15.107:0/3323887975 shutdown_connections 2024-03-24T00:12:45.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:12:45.185+0000 7f0587402700 1 -- 172.21.15.107:0/3323887975 wait complete. 2024-03-24T00:12:45.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:45 smithi107 ceph-mon[36485]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:45.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:45 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:45.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:45 smithi106 ceph-mon[36505]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:45.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:45 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:45.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:45 smithi078 ceph-mon[32429]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:45.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:45 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:45.962 DEBUG:teuthology.orchestra.run.smithi106:mgr.y> sudo journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mgr.y.service 2024-03-24T00:12:45.965 DEBUG:teuthology.orchestra.run.smithi107:mgr.z> sudo journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mgr.z.service 2024-03-24T00:12:45.967 INFO:tasks.cephadm:Deploying OSDs... 2024-03-24T00:12:45.968 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-24T00:12:45.968 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T00:12:45.983 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T00:12:45.983 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_1 2024-03-24T00:12:46.037 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T00:12:46.037 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:12:46.037 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-24T00:12:46.037 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:12:46.038 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:12:46.038 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-03-24 00:11:50.338644198 +0000 2024-03-24T00:12:46.038 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-03-24 00:06:33.485678127 +0000 2024-03-24T00:12:46.038 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-03-24 00:06:33.485678127 +0000 2024-03-24T00:12:46.038 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-03-24 00:06:33.485678127 +0000 2024-03-24T00:12:46.038 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T00:12:46.103 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-03-24T00:12:46.103 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-03-24T00:12:46.103 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000199467 s, 2.6 MB/s 2024-03-24T00:12:46.104 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T00:12:46.160 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_2 2024-03-24T00:12:46.214 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T00:12:46.215 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:12:46.215 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-24T00:12:46.215 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:12:46.215 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:12:46.215 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-03-24 00:11:50.366643702 +0000 2024-03-24T00:12:46.215 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-03-24 00:06:33.836671826 +0000 2024-03-24T00:12:46.215 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-03-24 00:06:33.836671826 +0000 2024-03-24T00:12:46.215 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-03-24 00:06:33.836671826 +0000 2024-03-24T00:12:46.215 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T00:12:46.280 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-03-24T00:12:46.280 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-03-24T00:12:46.280 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000197777 s, 2.6 MB/s 2024-03-24T00:12:46.281 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T00:12:46.337 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_3 2024-03-24T00:12:46.392 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T00:12:46.392 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:12:46.392 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-24T00:12:46.392 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:12:46.392 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:12:46.392 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-03-24 00:11:50.394643205 +0000 2024-03-24T00:12:46.392 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-03-24 00:06:34.167665884 +0000 2024-03-24T00:12:46.392 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-03-24 00:06:34.167665884 +0000 2024-03-24T00:12:46.392 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-03-24 00:06:34.167665884 +0000 2024-03-24T00:12:46.393 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T00:12:46.458 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-03-24T00:12:46.458 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-03-24T00:12:46.458 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000181407 s, 2.8 MB/s 2024-03-24T00:12:46.459 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T00:12:46.514 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_4 2024-03-24T00:12:46.569 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T00:12:46.569 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:12:46.569 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-24T00:12:46.569 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:12:46.569 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:12:46.569 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-03-24 00:11:50.422642708 +0000 2024-03-24T00:12:46.569 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-03-24 00:06:34.505659816 +0000 2024-03-24T00:12:46.569 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-03-24 00:06:34.505659816 +0000 2024-03-24T00:12:46.570 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-03-24 00:06:34.505659816 +0000 2024-03-24T00:12:46.570 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T00:12:46.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:46 smithi078 ceph-mon[32429]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi078=x;smithi106=y;smithi107=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:12:46.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:46 smithi078 ceph-mon[32429]: Saving service mgr spec with placement smithi078=x;smithi106=y;smithi107=z;count:3 2024-03-24T00:12:46.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:46 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:46.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:46 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:46.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:46 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:12:46.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:46 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:46.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:46 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:12:46.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:46 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:46.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:46 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T00:12:46.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:46 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T00:12:46.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:46 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T00:12:46.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:46 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:46.634 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-03-24T00:12:46.634 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-03-24T00:12:46.635 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000194384 s, 2.6 MB/s 2024-03-24T00:12:46.636 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T00:12:46.638 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:46 smithi107 ceph-mon[36485]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi078=x;smithi106=y;smithi107=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:12:46.638 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:46 smithi107 ceph-mon[36485]: Saving service mgr spec with placement smithi078=x;smithi106=y;smithi107=z;count:3 2024-03-24T00:12:46.638 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:46 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:46.639 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:46 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:46.639 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:46 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:12:46.639 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:46 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:46.639 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:46 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:12:46.639 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:46 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:46.639 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:46 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T00:12:46.639 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:46 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T00:12:46.639 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:46 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T00:12:46.639 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:46 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:46 smithi106 ceph-mon[36505]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi078=x;smithi106=y;smithi107=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:12:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:46 smithi106 ceph-mon[36505]: Saving service mgr spec with placement smithi078=x;smithi106=y;smithi107=z;count:3 2024-03-24T00:12:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:46 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:46 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:46 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:12:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:46 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:46 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:12:46.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:46 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:46.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:46 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T00:12:46.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:46 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T00:12:46.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:46 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T00:12:46.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:46 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:46.691 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-24T00:12:46.691 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T00:12:46.710 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T00:12:46.710 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_1 2024-03-24T00:12:46.768 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T00:12:46.768 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:12:46.768 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-03-24T00:12:46.768 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:12:46.768 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:12:46.768 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-03-24 00:12:32.009370419 +0000 2024-03-24T00:12:46.768 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-03-24 00:06:51.598567713 +0000 2024-03-24T00:12:46.768 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-03-24 00:06:51.598567713 +0000 2024-03-24T00:12:46.768 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: 2024-03-24 00:06:51.598567713 +0000 2024-03-24T00:12:46.769 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T00:12:46.838 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-03-24T00:12:46.838 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-03-24T00:12:46.838 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000288644 s, 1.8 MB/s 2024-03-24T00:12:46.839 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T00:12:46.896 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_2 2024-03-24T00:12:46.952 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T00:12:46.952 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:12:46.952 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d Inode: 680 Links: 1 2024-03-24T00:12:46.953 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:12:46.953 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:12:46.953 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-03-24 00:12:32.009370419 +0000 2024-03-24T00:12:46.953 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-03-24 00:06:51.951559545 +0000 2024-03-24T00:12:46.953 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-03-24 00:06:51.951559545 +0000 2024-03-24T00:12:46.953 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: 2024-03-24 00:06:51.951559545 +0000 2024-03-24T00:12:46.953 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T00:12:47.020 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-03-24T00:12:47.021 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-03-24T00:12:47.021 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.00022316 s, 2.3 MB/s 2024-03-24T00:12:47.022 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T00:12:47.079 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_3 2024-03-24T00:12:47.135 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T00:12:47.135 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:12:47.135 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d Inode: 699 Links: 1 2024-03-24T00:12:47.135 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:12:47.135 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:12:47.135 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-03-24 00:12:32.010370397 +0000 2024-03-24T00:12:47.136 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-03-24 00:06:52.298551514 +0000 2024-03-24T00:12:47.136 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-03-24 00:06:52.298551514 +0000 2024-03-24T00:12:47.136 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: 2024-03-24 00:06:52.298551514 +0000 2024-03-24T00:12:47.136 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T00:12:47.201 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-03-24T00:12:47.202 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-03-24T00:12:47.202 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000225931 s, 2.3 MB/s 2024-03-24T00:12:47.203 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T00:12:47.259 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_4 2024-03-24T00:12:47.314 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T00:12:47.314 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:12:47.314 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d Inode: 717 Links: 1 2024-03-24T00:12:47.314 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:12:47.314 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:12:47.314 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-03-24 00:12:32.010370397 +0000 2024-03-24T00:12:47.314 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-03-24 00:06:52.636543690 +0000 2024-03-24T00:12:47.314 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-03-24 00:06:52.636543690 +0000 2024-03-24T00:12:47.314 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: 2024-03-24 00:06:52.636543690 +0000 2024-03-24T00:12:47.315 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T00:12:47.380 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-03-24T00:12:47.380 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-03-24T00:12:47.380 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000213091 s, 2.4 MB/s 2024-03-24T00:12:47.382 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T00:12:47.438 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-24T00:12:47.439 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T00:12:47.458 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T00:12:47.458 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_1 2024-03-24T00:12:47.514 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T00:12:47.514 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:12:47.514 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 655 Links: 1 2024-03-24T00:12:47.515 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:12:47.515 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:12:47.515 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-03-24 00:12:22.561755406 +0000 2024-03-24T00:12:47.515 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-03-24 00:06:50.966491500 +0000 2024-03-24T00:12:47.515 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-03-24 00:06:50.966491500 +0000 2024-03-24T00:12:47.515 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-03-24 00:06:50.966491500 +0000 2024-03-24T00:12:47.515 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T00:12:47.583 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-03-24T00:12:47.583 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-03-24T00:12:47.583 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000226258 s, 2.3 MB/s 2024-03-24T00:12:47.584 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T00:12:47.641 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_2 2024-03-24T00:12:47.650 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:47 smithi107 ceph-mon[36485]: Deploying daemon mgr.z on smithi107 2024-03-24T00:12:47.650 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:47 smithi107 ceph-mon[36485]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:47.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:47 smithi106 ceph-mon[36505]: Deploying daemon mgr.z on smithi107 2024-03-24T00:12:47.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:47 smithi106 ceph-mon[36505]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:47.665 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T00:12:47.665 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:12:47.665 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 674 Links: 1 2024-03-24T00:12:47.665 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:12:47.665 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:12:47.665 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-03-24 00:12:22.561755406 +0000 2024-03-24T00:12:47.665 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-03-24 00:06:51.304485638 +0000 2024-03-24T00:12:47.665 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-03-24 00:06:51.304485638 +0000 2024-03-24T00:12:47.665 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-03-24 00:06:51.304485638 +0000 2024-03-24T00:12:47.666 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T00:12:47.817 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-03-24T00:12:47.818 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-03-24T00:12:47.818 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000188771 s, 2.7 MB/s 2024-03-24T00:12:47.819 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T00:12:47.834 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_3 2024-03-24T00:12:47.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:47 smithi078 ceph-mon[32429]: Deploying daemon mgr.z on smithi107 2024-03-24T00:12:47.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:47 smithi078 ceph-mon[32429]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:47.888 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T00:12:47.889 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:12:47.889 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 695 Links: 1 2024-03-24T00:12:47.889 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:12:47.889 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:12:47.889 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-03-24 00:12:47.758335048 +0000 2024-03-24T00:12:47.889 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-03-24 00:06:51.633479932 +0000 2024-03-24T00:12:47.889 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-03-24 00:06:51.633479932 +0000 2024-03-24T00:12:47.889 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-03-24 00:06:51.633479932 +0000 2024-03-24T00:12:47.889 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T00:12:48.002 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-03-24T00:12:48.002 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-03-24T00:12:48.002 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000261058 s, 2.0 MB/s 2024-03-24T00:12:48.004 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T00:12:48.019 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_4 2024-03-24T00:12:48.075 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T00:12:48.075 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:12:48.075 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 714 Links: 1 2024-03-24T00:12:48.075 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:12:48.075 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:12:48.075 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-03-24 00:12:47.943331960 +0000 2024-03-24T00:12:48.075 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-03-24 00:06:51.964474191 +0000 2024-03-24T00:12:48.075 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-03-24 00:06:51.964474191 +0000 2024-03-24T00:12:48.075 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-03-24 00:06:51.964474191 +0000 2024-03-24T00:12:48.075 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T00:12:48.142 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-03-24T00:12:48.142 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-03-24T00:12:48.142 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000177895 s, 2.9 MB/s 2024-03-24T00:12:48.143 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T00:12:48.198 INFO:tasks.cephadm:Deploying osd.0 on smithi078 with /dev/vg_nvme/lv_4... 2024-03-24T00:12:48.198 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T00:12:48.446 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:12:49.093 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:12:48 smithi107 systemd[1]: Started Ceph mgr.z for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:12:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:49 smithi078 ceph-mon[32429]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:49 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:49 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:49 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:49 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T00:12:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:49 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T00:12:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:49 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T00:12:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:49 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:49.593 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:12:49 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:12:49.168+0000 7fb77c63d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T00:12:49.593 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:12:49 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:12:49.260+0000 7fb77c63d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T00:12:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:49 smithi107 ceph-mon[36485]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:49 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:49 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:49.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:49 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:49.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:49 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T00:12:49.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:49 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T00:12:49.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:49 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T00:12:49.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:49 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:49.643 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:49 smithi106 ceph-mon[36505]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:49.643 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:49 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:49.643 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:49 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:49.643 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:49 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:49.644 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:49 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T00:12:49.644 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:49 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T00:12:49.644 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:49 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T00:12:49.644 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:49 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:50.218 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:12:49 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:12:49.893+0000 7fb77c63d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T00:12:50.218 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:12:49 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:12:49.984+0000 7fb77c63d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T00:12:50.218 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:12:50 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:12:50.124+0000 7fb77c63d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T00:12:50.305 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T00:12:50.305 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T00:12:50.306 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10+0 records in 2024-03-24T00:12:50.306 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2024-03-24T00:12:50.306 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116928 s, 897 MB/s 2024-03-24T00:12:50.306 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping successful for: 2024-03-24T00:12:50.491 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:50 smithi107 ceph-mon[36485]: Deploying daemon mgr.y on smithi106 2024-03-24T00:12:50.491 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:12:50 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:12:50.218+0000 7fb77c63d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T00:12:50.492 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:12:50 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:12:50.304+0000 7fb77c63d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T00:12:50.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:50 smithi078 ceph-mon[32429]: Deploying daemon mgr.y on smithi106 2024-03-24T00:12:50.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:50 smithi106 ceph-mon[36505]: Deploying daemon mgr.y on smithi106 2024-03-24T00:12:50.778 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:12:50 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:12:50.575+0000 7fb77c63d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T00:12:50.778 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:12:50 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:12:50.689+0000 7fb77c63d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T00:12:50.778 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:12:50 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:12:50.778+0000 7fb77c63d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T00:12:50.883 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi078:vg_nvme/lv_4 2024-03-24T00:12:51.054 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:12:50 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:12:50.875+0000 7fb77c63d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T00:12:51.054 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:12:51 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:12:51.054+0000 7fb77c63d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T00:12:51.119 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:12:51.533 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:51 smithi078 ceph-mon[32429]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:51.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:51 smithi106 ceph-mon[36505]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:51.652 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:12:51 smithi106 systemd[1]: Started Ceph mgr.y for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:12:51.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:51 smithi107 ceph-mon[36485]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:52.152 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:12:51 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:12:51.886+0000 7f123abb1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T00:12:52.152 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:12:51 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:12:51.981+0000 7f123abb1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T00:12:52.343 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:12:52 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:12:52.056+0000 7fb77c63d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T00:12:52.343 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:12:52 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:12:52.200+0000 7fb77c63d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T00:12:52.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:52 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:52.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:52 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:52.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:52 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:52.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:52 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:52.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:52 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:12:52.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:52 smithi078 ceph-mon[32429]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:52.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.626+0000 7f12a563a700 1 -- 172.21.15.78:0/1635859680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12a01022c0 msgr2=0x7f12a0102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:52.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.626+0000 7f12a563a700 1 --2- 172.21.15.78:0/1635859680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12a01022c0 0x7f12a0102720 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f12940099c0 tx=0x7f1294009cd0 comp rx=0 tx=0).stop 2024-03-24T00:12:52.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.626+0000 7f12a563a700 1 -- 172.21.15.78:0/1635859680 shutdown_connections 2024-03-24T00:12:52.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.626+0000 7f12a563a700 1 --2- 172.21.15.78:0/1635859680 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12a0102c60 0x7f12a0109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:52.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.626+0000 7f12a563a700 1 --2- 172.21.15.78:0/1635859680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12a01010c0 0x7f12a01014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:52.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.626+0000 7f12a563a700 1 --2- 172.21.15.78:0/1635859680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12a01022c0 0x7f12a0102720 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:52.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.626+0000 7f12a563a700 1 -- 172.21.15.78:0/1635859680 >> 172.21.15.78:0/1635859680 conn(0x7f12a00fc680 msgr2=0x7f12a00feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:52.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.626+0000 7f12a563a700 1 -- 172.21.15.78:0/1635859680 shutdown_connections 2024-03-24T00:12:52.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.626+0000 7f12a563a700 1 -- 172.21.15.78:0/1635859680 wait complete. 2024-03-24T00:12:52.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.627+0000 7f12a563a700 1 Processor -- start 2024-03-24T00:12:52.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.627+0000 7f12a563a700 1 -- start start 2024-03-24T00:12:52.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.627+0000 7f12a563a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12a01010c0 0x7f12a006b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:52.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.627+0000 7f12a563a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12a01022c0 0x7f12a019d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:52.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.627+0000 7f12a563a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12a0102c60 0x7f12a019f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:52.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.627+0000 7f12a563a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a010cc80 con 0x7f12a0102c60 2024-03-24T00:12:52.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.627+0000 7f12a563a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a010cb00 con 0x7f12a01022c0 2024-03-24T00:12:52.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.627+0000 7f12a563a700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a010ce00 con 0x7f12a01010c0 2024-03-24T00:12:52.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.627+0000 7f129e7fc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12a01022c0 0x7f12a019d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:52.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.627+0000 7f129effd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12a01010c0 0x7f12a006b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:52.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.627+0000 7f129e7fc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12a01022c0 0x7f12a019d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.78:43290/0 (socket says 172.21.15.78:43290) 2024-03-24T00:12:52.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.627+0000 7f129e7fc700 1 -- 172.21.15.78:0/2230445284 learned_addr learned my addr 172.21.15.78:0/2230445284 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:12:52.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.627+0000 7f129f7fe700 1 --2- 172.21.15.78:0/2230445284 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12a0102c60 0x7f12a019f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:52.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.628+0000 7f129effd700 1 -- 172.21.15.78:0/2230445284 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12a01022c0 msgr2=0x7f12a019d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:52.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.628+0000 7f129effd700 1 --2- 172.21.15.78:0/2230445284 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12a01022c0 0x7f12a019d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:52.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.628+0000 7f129effd700 1 -- 172.21.15.78:0/2230445284 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12a0102c60 msgr2=0x7f12a019f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:52.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.628+0000 7f129effd700 1 --2- 172.21.15.78:0/2230445284 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12a0102c60 0x7f12a019f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:52.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.628+0000 7f129effd700 1 -- 172.21.15.78:0/2230445284 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a019fff0 con 0x7f12a01010c0 2024-03-24T00:12:52.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.628+0000 7f129f7fe700 1 --2- 172.21.15.78:0/2230445284 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12a0102c60 0x7f12a019f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:12:52.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.628+0000 7f129effd700 1 --2- 172.21.15.78:0/2230445284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12a01010c0 0x7f12a006b490 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f129000d720 tx=0x7f129000da30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:52.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.628+0000 7f1287fff700 1 -- 172.21.15.78:0/2230445284 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1290014030 con 0x7f12a01010c0 2024-03-24T00:12:52.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.628+0000 7f1287fff700 1 -- 172.21.15.78:0/2230445284 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f129000fb70 con 0x7f12a01010c0 2024-03-24T00:12:52.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.628+0000 7f12a563a700 1 -- 172.21.15.78:0/2230445284 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12a01a02e0 con 0x7f12a01010c0 2024-03-24T00:12:52.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.628+0000 7f1287fff700 1 -- 172.21.15.78:0/2230445284 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1290010e00 con 0x7f12a01010c0 2024-03-24T00:12:52.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.628+0000 7f12a563a700 1 -- 172.21.15.78:0/2230445284 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a01a0820 con 0x7f12a01010c0 2024-03-24T00:12:52.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.629+0000 7f1287fff700 1 -- 172.21.15.78:0/2230445284 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f1290017030 con 0x7f12a01010c0 2024-03-24T00:12:52.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.629+0000 7f1287fff700 1 --2- 172.21.15.78:0/2230445284 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f1288039200 0x7f128803b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:52.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.629+0000 7f1287fff700 1 -- 172.21.15.78:0/2230445284 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f129004bb50 con 0x7f12a01010c0 2024-03-24T00:12:52.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.629+0000 7f12a563a700 1 -- 172.21.15.78:0/2230445284 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12a0068e50 con 0x7f12a01010c0 2024-03-24T00:12:52.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.629+0000 7f129e7fc700 1 --2- 172.21.15.78:0/2230445284 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f1288039200 0x7f128803b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:52.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.630+0000 7f129e7fc700 1 --2- 172.21.15.78:0/2230445284 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f1288039200 0x7f128803b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1294003890 tx=0x7f1294003960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:52.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.634+0000 7f1287fff700 1 -- 172.21.15.78:0/2230445284 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1290020300 con 0x7f12a01010c0 2024-03-24T00:12:52.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:52 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:52.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:52 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:52.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:52 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:52.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:52 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:52.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:52 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:12:52.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:52 smithi106 ceph-mon[36505]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:52.824 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:12:52 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:12:52.535+0000 7fb77c63d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T00:12:52.824 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:12:52 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:12:52.627+0000 7fb77c63d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T00:12:52.824 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:12:52 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:12:52.823+0000 7fb77c63d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:12:52.824 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:52 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:52.824 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:52 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:52.824 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:52 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:52.824 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:52 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:52.825 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:52 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:12:52.825 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:52 smithi107 ceph-mon[36485]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:52.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:52.842+0000 7f12a563a700 1 -- 172.21.15.78:0/2230445284 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f12a00625d0 con 0x7f1288039200 2024-03-24T00:12:52.995 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:12:52 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:12:52.660+0000 7f123abb1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T00:12:52.995 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:12:52 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:12:52.752+0000 7f123abb1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T00:12:52.995 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:12:52 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:12:52.901+0000 7f123abb1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T00:12:52.996 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:12:52 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:12:52.995+0000 7f123abb1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T00:12:53.093 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:12:53 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:12:53.016+0000 7fb77c63d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T00:12:53.335 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:12:53 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:12:53.079+0000 7f123abb1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T00:12:53.336 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:12:53 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:12:53.335+0000 7f123abb1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T00:12:53.553 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:12:53 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:12:53.296+0000 7fb77c63d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:12:53.553 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:12:53 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:12:53.553+0000 7fb77c63d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T00:12:53.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:53 smithi107 ceph-mon[36485]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:12:53.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:53 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:12:53.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:53 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:12:53.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:53 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:53.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:53 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:53.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:53 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:53.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:53 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:53.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:53 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:53.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:53 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:53.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:53 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:12:53.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:53 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:53.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:53 smithi107 ceph-mon[36485]: Reconfiguring mgr.x (unknown last config time)... 2024-03-24T00:12:53.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:53 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T00:12:53.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:53 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T00:12:53.555 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:53 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:53.555 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:53 smithi107 ceph-mon[36485]: Reconfiguring daemon mgr.x on smithi078 2024-03-24T00:12:53.616 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:12:53 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:12:53.440+0000 7f123abb1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T00:12:53.616 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:12:53 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:12:53.523+0000 7f123abb1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T00:12:53.617 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:12:53 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:12:53.615+0000 7f123abb1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T00:12:53.617 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:53 smithi106 ceph-mon[36505]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:12:53.617 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:53 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:12:53.617 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:53 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:12:53.617 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:53 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:53.617 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:53 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:53.617 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:53 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:53.617 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:53 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:53.617 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:53 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:53.617 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:53 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:53.617 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:53 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:12:53.618 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:53 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:53.618 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:53 smithi106 ceph-mon[36505]: Reconfiguring mgr.x (unknown last config time)... 2024-03-24T00:12:53.618 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:53 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T00:12:53.618 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:53 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T00:12:53.618 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:53 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:53.618 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:53 smithi106 ceph-mon[36505]: Reconfiguring daemon mgr.x on smithi078 2024-03-24T00:12:53.735 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:53 smithi078 ceph-mon[32429]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:12:53.735 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:53 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:12:53.735 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:53 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:12:53.735 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:53 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:53.735 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:53 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:53.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:53 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:53.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:53 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:53.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:53 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:53.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:53 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:53.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:53 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:12:53.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:53 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:53.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:53 smithi078 ceph-mon[32429]: Reconfiguring mgr.x (unknown last config time)... 2024-03-24T00:12:53.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:53 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T00:12:53.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:53 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T00:12:53.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:53 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:53.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:53 smithi078 ceph-mon[32429]: Reconfiguring daemon mgr.x on smithi078 2024-03-24T00:12:53.843 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:12:53 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:12:53.784+0000 7fb77c63d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T00:12:53.902 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:12:53 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:12:53.787+0000 7f123abb1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T00:12:54.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:54 smithi078 ceph-mon[32429]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:54 smithi107 ceph-mon[36485]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:54.843 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:12:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:12:54.569+0000 7fb77c63d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T00:12:54.856 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:54 smithi106 ceph-mon[36505]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:54.856 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:12:54 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:12:54.730+0000 7f123abb1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T00:12:54.856 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:12:54 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:12:54.855+0000 7f123abb1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T00:12:55.438 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:12:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:12:55.129+0000 7fb77c63d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T00:12:55.438 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:12:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:12:55.224+0000 7fb77c63d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T00:12:55.438 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:12:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:12:55.437+0000 7fb77c63d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T00:12:55.461 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:12:55 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:12:55.181+0000 7f123abb1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T00:12:55.462 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:12:55 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:12:55.269+0000 7f123abb1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T00:12:55.462 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:12:55 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:12:55.461+0000 7f123abb1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:12:55.902 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:12:55 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:12:55.644+0000 7f123abb1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T00:12:56.176 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:12:55 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:12:55.921+0000 7f123abb1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:12:56.176 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:12:56 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:12:56.175+0000 7f123abb1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T00:12:56.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:56.410+0000 7f1287fff700 1 -- 172.21.15.78:0/2230445284 <== mon.1 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 14) v1 ==== 91698+0+0 (secure 0 0 0) 0x7f1290018c60 con 0x7f12a01010c0 2024-03-24T00:12:56.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:56 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:56.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:56 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:56.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:56 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:56.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:56 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:12:56.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:56 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:56.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:56 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:12:56.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:56 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:56.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:56 smithi078 ceph-mon[32429]: Standby manager daemon z started 2024-03-24T00:12:56.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:56 smithi078 ceph-mon[32429]: from='mgr.? 172.21.15.107:0/3366457209' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T00:12:56.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:56 smithi078 ceph-mon[32429]: from='mgr.? 172.21.15.107:0/3366457209' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T00:12:56.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:56 smithi078 ceph-mon[32429]: from='mgr.? 172.21.15.107:0/3366457209' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T00:12:56.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:56 smithi078 ceph-mon[32429]: from='mgr.? 172.21.15.107:0/3366457209' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T00:12:56.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:56 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:56.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:56 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:56.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:56 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:56.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:56 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:12:56.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:56 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:56.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:56 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:12:56.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:56 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:56.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:56 smithi107 ceph-mon[36485]: Standby manager daemon z started 2024-03-24T00:12:56.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:56 smithi107 ceph-mon[36485]: from='mgr.? 172.21.15.107:0/3366457209' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T00:12:56.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:56 smithi107 ceph-mon[36485]: from='mgr.? 172.21.15.107:0/3366457209' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T00:12:56.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:56 smithi107 ceph-mon[36485]: from='mgr.? 172.21.15.107:0/3366457209' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T00:12:56.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:56 smithi107 ceph-mon[36485]: from='mgr.? 172.21.15.107:0/3366457209' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T00:12:56.652 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:12:56 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:12:56.408+0000 7f123abb1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T00:12:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:56 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:56 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:56 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:56 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:12:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:56 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:56 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:12:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:56 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:12:56.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:56 smithi106 ceph-mon[36505]: Standby manager daemon z started 2024-03-24T00:12:56.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:56 smithi106 ceph-mon[36505]: from='mgr.? 172.21.15.107:0/3366457209' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T00:12:56.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:56 smithi106 ceph-mon[36505]: from='mgr.? 172.21.15.107:0/3366457209' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T00:12:56.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:56 smithi106 ceph-mon[36505]: from='mgr.? 172.21.15.107:0/3366457209' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T00:12:56.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:56 smithi106 ceph-mon[36505]: from='mgr.? 172.21.15.107:0/3366457209' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T00:12:57.652 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:12:57 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:12:57.194+0000 7f123abb1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T00:12:57.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:57 smithi106 ceph-mon[36505]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:57.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:57 smithi106 ceph-mon[36505]: mgrmap e14: x(active, since 76s), standbys: z 2024-03-24T00:12:57.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:57 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T00:12:57.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:57 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/353345221' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b950388a-c437-426b-b703-3f8ae821dd0d"}]: dispatch 2024-03-24T00:12:57.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:57 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/353345221' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b950388a-c437-426b-b703-3f8ae821dd0d"}]': finished 2024-03-24T00:12:57.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:57 smithi106 ceph-mon[36505]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T00:12:57.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:57 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:12:57.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:57 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1653353270' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:12:57.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:57 smithi078 ceph-mon[32429]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:57.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:57 smithi078 ceph-mon[32429]: mgrmap e14: x(active, since 76s), standbys: z 2024-03-24T00:12:57.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:57 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T00:12:57.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:57 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/353345221' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b950388a-c437-426b-b703-3f8ae821dd0d"}]: dispatch 2024-03-24T00:12:57.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:57 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/353345221' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b950388a-c437-426b-b703-3f8ae821dd0d"}]': finished 2024-03-24T00:12:57.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:57 smithi078 ceph-mon[32429]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T00:12:57.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:57 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:12:57.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:57 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1653353270' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:12:57.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:57 smithi107 ceph-mon[36485]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:57.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:57 smithi107 ceph-mon[36485]: mgrmap e14: x(active, since 76s), standbys: z 2024-03-24T00:12:57.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:57 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T00:12:57.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:57 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/353345221' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b950388a-c437-426b-b703-3f8ae821dd0d"}]: dispatch 2024-03-24T00:12:57.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:57 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/353345221' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b950388a-c437-426b-b703-3f8ae821dd0d"}]': finished 2024-03-24T00:12:57.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:57 smithi107 ceph-mon[36485]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T00:12:57.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:57 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:12:57.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:57 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1653353270' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:12:58.055 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:12:57 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:12:57.750+0000 7f123abb1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T00:12:58.055 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:12:57 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:12:57.845+0000 7f123abb1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T00:12:58.055 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:12:58 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:12:58.054+0000 7f123abb1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T00:12:58.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:12:58.418+0000 7f1287fff700 1 -- 172.21.15.78:0/2230445284 <== mon.1 v2:172.21.15.107:3300/0 8 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7f129005d370 con 0x7f12a01010c0 2024-03-24T00:12:58.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:58 smithi078 ceph-mon[32429]: Standby manager daemon y started 2024-03-24T00:12:58.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:58 smithi078 ceph-mon[32429]: from='mgr.? 172.21.15.106:0/3545393394' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T00:12:58.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:58 smithi078 ceph-mon[32429]: from='mgr.? 172.21.15.106:0/3545393394' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T00:12:58.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:58 smithi078 ceph-mon[32429]: from='mgr.? 172.21.15.106:0/3545393394' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T00:12:58.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:58 smithi078 ceph-mon[32429]: from='mgr.? 172.21.15.106:0/3545393394' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T00:12:58.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:58 smithi078 ceph-mon[32429]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:58 smithi107 ceph-mon[36485]: Standby manager daemon y started 2024-03-24T00:12:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:58 smithi107 ceph-mon[36485]: from='mgr.? 172.21.15.106:0/3545393394' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T00:12:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:58 smithi107 ceph-mon[36485]: from='mgr.? 172.21.15.106:0/3545393394' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T00:12:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:58 smithi107 ceph-mon[36485]: from='mgr.? 172.21.15.106:0/3545393394' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T00:12:58.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:58 smithi107 ceph-mon[36485]: from='mgr.? 172.21.15.106:0/3545393394' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T00:12:58.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:58 smithi107 ceph-mon[36485]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:58 smithi106 ceph-mon[36505]: Standby manager daemon y started 2024-03-24T00:12:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:58 smithi106 ceph-mon[36505]: from='mgr.? 172.21.15.106:0/3545393394' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T00:12:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:58 smithi106 ceph-mon[36505]: from='mgr.? 172.21.15.106:0/3545393394' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T00:12:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:58 smithi106 ceph-mon[36505]: from='mgr.? 172.21.15.106:0/3545393394' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T00:12:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:58 smithi106 ceph-mon[36505]: from='mgr.? 172.21.15.106:0/3545393394' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T00:12:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:58 smithi106 ceph-mon[36505]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:12:59.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:59 smithi078 ceph-mon[32429]: mgrmap e15: x(active, since 78s), standbys: y, z 2024-03-24T00:12:59.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:12:59 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T00:12:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:59 smithi107 ceph-mon[36485]: mgrmap e15: x(active, since 78s), standbys: y, z 2024-03-24T00:12:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:12:59 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T00:12:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:59 smithi106 ceph-mon[36505]: mgrmap e15: x(active, since 78s), standbys: y, z 2024-03-24T00:12:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:12:59 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T00:13:00.725 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:00 smithi106 ceph-mon[36505]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:13:00.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:00 smithi078 ceph-mon[32429]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:13:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:00 smithi107 ceph-mon[36485]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:13:03.450 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:03 smithi078 ceph-mon[32429]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:13:03.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:03 smithi107 ceph-mon[36485]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:13:03.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:03 smithi106 ceph-mon[36505]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:13:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:04 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T00:13:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:04 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:13:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:04 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T00:13:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:04 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:13:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:04 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T00:13:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:04 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:13:05.452 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:05 smithi078 ceph-mon[32429]: Deploying daemon osd.0 on smithi078 2024-03-24T00:13:05.452 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:05 smithi078 ceph-mon[32429]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:13:05.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:05 smithi107 ceph-mon[36485]: Deploying daemon osd.0 on smithi078 2024-03-24T00:13:05.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:05 smithi107 ceph-mon[36485]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:13:05.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:05 smithi106 ceph-mon[36505]: Deploying daemon osd.0 on smithi078 2024-03-24T00:13:05.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:05 smithi106 ceph-mon[36505]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:13:07.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:07 smithi078 ceph-mon[32429]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:13:07.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:07 smithi107 ceph-mon[36485]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:13:07.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:07 smithi106 ceph-mon[36505]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:13:09.567 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:09 smithi078 ceph-mon[32429]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:13:09.568 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:09 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:09.568 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:09 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:09.568 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:09 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:13:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:09 smithi107 ceph-mon[36485]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:13:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:09 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:09 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:09 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:13:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:09 smithi106 ceph-mon[36505]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:13:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:09 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:09 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:09 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:13:11.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:11 smithi078 ceph-mon[32429]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:13:11.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:11 smithi107 ceph-mon[36485]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:13:11.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:11 smithi106 ceph-mon[36505]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:13:12.147 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 0 on host 'smithi078' 2024-03-24T00:13:12.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:13:12.146+0000 7f1287fff700 1 -- 172.21.15.78:0/2230445284 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f12a00625d0 con 0x7f1288039200 2024-03-24T00:13:12.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:13:12.149+0000 7f12a563a700 1 -- 172.21.15.78:0/2230445284 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f1288039200 msgr2=0x7f128803b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:12.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:13:12.149+0000 7f12a563a700 1 --2- 172.21.15.78:0/2230445284 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f1288039200 0x7f128803b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1294003890 tx=0x7f1294003960 comp rx=0 tx=0).stop 2024-03-24T00:13:12.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:13:12.149+0000 7f12a563a700 1 -- 172.21.15.78:0/2230445284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12a01010c0 msgr2=0x7f12a006b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:12.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:13:12.149+0000 7f12a563a700 1 --2- 172.21.15.78:0/2230445284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12a01010c0 0x7f12a006b490 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f129000d720 tx=0x7f129000da30 comp rx=0 tx=0).stop 2024-03-24T00:13:12.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:13:12.149+0000 7f12a563a700 1 -- 172.21.15.78:0/2230445284 shutdown_connections 2024-03-24T00:13:12.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:13:12.149+0000 7f12a563a700 1 --2- 172.21.15.78:0/2230445284 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f1288039200 0x7f128803b6c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:12.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:13:12.149+0000 7f12a563a700 1 --2- 172.21.15.78:0/2230445284 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12a0102c60 0x7f12a019f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:12.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:13:12.149+0000 7f12a563a700 1 --2- 172.21.15.78:0/2230445284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12a01010c0 0x7f12a006b490 secure :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f129000d720 tx=0x7f129000da30 comp rx=0 tx=0).stop 2024-03-24T00:13:12.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:13:12.149+0000 7f12a563a700 1 --2- 172.21.15.78:0/2230445284 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12a01022c0 0x7f12a019d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:12.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:13:12.149+0000 7f12a563a700 1 -- 172.21.15.78:0/2230445284 >> 172.21.15.78:0/2230445284 conn(0x7f12a00fc680 msgr2=0x7f12a01055c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:12.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:13:12.149+0000 7f12a563a700 1 -- 172.21.15.78:0/2230445284 shutdown_connections 2024-03-24T00:13:12.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:13:12.149+0000 7f12a563a700 1 -- 172.21.15.78:0/2230445284 wait complete. 2024-03-24T00:13:12.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:12 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:12.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:12 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:12.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:12 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:13:12.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:12 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:13:12.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:12 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:12.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:12 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:12.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:12 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:13:12.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:12 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:12.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:12 smithi078 ceph-mon[32429]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:13:12.730 DEBUG:teuthology.orchestra.run.smithi078:osd.0> sudo journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.0.service 2024-03-24T00:13:12.732 INFO:tasks.cephadm:Deploying osd.1 on smithi106 with /dev/vg_nvme/lv_4... 2024-03-24T00:13:12.732 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T00:13:12.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:12 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:12.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:12 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:12.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:12 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:13:12.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:12 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:13:12.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:12 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:12.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:12 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:12.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:12 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:13:12.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:12 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:12.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:12 smithi106 ceph-mon[36505]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:13:12.964 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 24 00:13:12 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-0[44470]: 2024-03-24T00:13:12.768+0000 7f7031dda7c0 -1 osd.0 0 log_to_monitors true 2024-03-24T00:13:12.973 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T00:13:13.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:12 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:13.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:12 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:13.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:12 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:13:13.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:12 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:13:13.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:12 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:13.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:12 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:13.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:12 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:13:13.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:12 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:13.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:12 smithi107 ceph-mon[36485]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:13:13.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:13 smithi106 ceph-mon[36505]: from='osd.0 [v2:172.21.15.78:6802/1816879397,v1:172.21.15.78:6803/1816879397]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T00:13:13.894 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:13 smithi078 ceph-mon[32429]: from='osd.0 [v2:172.21.15.78:6802/1816879397,v1:172.21.15.78:6803/1816879397]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T00:13:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:13 smithi107 ceph-mon[36485]: from='osd.0 [v2:172.21.15.78:6802/1816879397,v1:172.21.15.78:6803/1816879397]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T00:13:14.808 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T00:13:14.808 INFO:teuthology.orchestra.run.smithi106.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T00:13:14.808 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10+0 records in 2024-03-24T00:13:14.808 INFO:teuthology.orchestra.run.smithi106.stderr:10+0 records out 2024-03-24T00:13:14.808 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113451 s, 924 MB/s 2024-03-24T00:13:14.808 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping successful for: 2024-03-24T00:13:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:14 smithi106 ceph-mon[36505]: from='osd.0 [v2:172.21.15.78:6802/1816879397,v1:172.21.15.78:6803/1816879397]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T00:13:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:14 smithi106 ceph-mon[36505]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T00:13:14.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:14 smithi106 ceph-mon[36505]: from='osd.0 [v2:172.21.15.78:6802/1816879397,v1:172.21.15.78:6803/1816879397]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-24T00:13:14.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:14 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:13:14.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:14 smithi106 ceph-mon[36505]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:13:15.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:14 smithi078 ceph-mon[32429]: from='osd.0 [v2:172.21.15.78:6802/1816879397,v1:172.21.15.78:6803/1816879397]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T00:13:15.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:14 smithi078 ceph-mon[32429]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T00:13:15.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:14 smithi078 ceph-mon[32429]: from='osd.0 [v2:172.21.15.78:6802/1816879397,v1:172.21.15.78:6803/1816879397]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-24T00:13:15.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:14 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:13:15.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:14 smithi078 ceph-mon[32429]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:13:15.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:14 smithi107 ceph-mon[36485]: from='osd.0 [v2:172.21.15.78:6802/1816879397,v1:172.21.15.78:6803/1816879397]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T00:13:15.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:14 smithi107 ceph-mon[36485]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T00:13:15.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:14 smithi107 ceph-mon[36485]: from='osd.0 [v2:172.21.15.78:6802/1816879397,v1:172.21.15.78:6803/1816879397]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-24T00:13:15.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:14 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:13:15.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:14 smithi107 ceph-mon[36485]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:13:15.352 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi106:vg_nvme/lv_4 2024-03-24T00:13:15.542 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 24 00:13:15 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-0[44470]: 2024-03-24T00:13:15.351+0000 7f702743f700 -1 osd.0 0 waiting for initial osdmap 2024-03-24T00:13:15.543 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 24 00:13:15 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-0[44470]: 2024-03-24T00:13:15.355+0000 7f701f22a700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:13:15.589 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T00:13:15.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:15 smithi078 ceph-mon[32429]: from='osd.0 [v2:172.21.15.78:6802/1816879397,v1:172.21.15.78:6803/1816879397]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-24T00:13:15.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:15 smithi078 ceph-mon[32429]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T00:13:15.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:15 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:13:15.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:15 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:13:15.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:15 smithi078 ceph-mon[32429]: from='osd.0 [v2:172.21.15.78:6802/1816879397,v1:172.21.15.78:6803/1816879397]' entity='osd.0' 2024-03-24T00:13:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:15 smithi106 ceph-mon[36505]: from='osd.0 [v2:172.21.15.78:6802/1816879397,v1:172.21.15.78:6803/1816879397]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-24T00:13:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:15 smithi106 ceph-mon[36505]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T00:13:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:15 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:13:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:15 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:13:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:15 smithi106 ceph-mon[36505]: from='osd.0 [v2:172.21.15.78:6802/1816879397,v1:172.21.15.78:6803/1816879397]' entity='osd.0' 2024-03-24T00:13:16.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:15 smithi107 ceph-mon[36485]: from='osd.0 [v2:172.21.15.78:6802/1816879397,v1:172.21.15.78:6803/1816879397]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-24T00:13:16.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:15 smithi107 ceph-mon[36485]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T00:13:16.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:15 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:13:16.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:15 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:13:16.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:15 smithi107 ceph-mon[36485]: from='osd.0 [v2:172.21.15.78:6802/1816879397,v1:172.21.15.78:6803/1816879397]' entity='osd.0' 2024-03-24T00:13:16.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:16 smithi106 ceph-mon[36505]: purged_snaps scrub starts 2024-03-24T00:13:16.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:16 smithi106 ceph-mon[36505]: purged_snaps scrub ok 2024-03-24T00:13:16.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:16 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:13:16.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:16 smithi106 ceph-mon[36505]: Detected new or changed devices on smithi078 2024-03-24T00:13:16.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:16 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:16.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:16 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:16.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:16 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:13:16.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:16 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi078 to 17088M 2024-03-24T00:13:16.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:16 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:16.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:16 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:13:16.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:16 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:13:16.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:16 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:16.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:16 smithi106 ceph-mon[36505]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:13:16.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:16 smithi106 ceph-mon[36505]: osd.0 [v2:172.21.15.78:6802/1816879397,v1:172.21.15.78:6803/1816879397] boot 2024-03-24T00:13:16.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:16 smithi106 ceph-mon[36505]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T00:13:16.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:16 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:13:17.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:16 smithi078 ceph-mon[32429]: purged_snaps scrub starts 2024-03-24T00:13:17.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:16 smithi078 ceph-mon[32429]: purged_snaps scrub ok 2024-03-24T00:13:17.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:16 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:13:17.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:16 smithi078 ceph-mon[32429]: Detected new or changed devices on smithi078 2024-03-24T00:13:17.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:16 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:17.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:16 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:17.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:16 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:13:17.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:16 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi078 to 17088M 2024-03-24T00:13:17.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:16 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:17.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:16 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:13:17.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:16 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:13:17.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:16 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:17.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:16 smithi078 ceph-mon[32429]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:13:17.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:16 smithi078 ceph-mon[32429]: osd.0 [v2:172.21.15.78:6802/1816879397,v1:172.21.15.78:6803/1816879397] boot 2024-03-24T00:13:17.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:16 smithi078 ceph-mon[32429]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T00:13:17.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:16 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:13:17.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:16 smithi107 ceph-mon[36485]: purged_snaps scrub starts 2024-03-24T00:13:17.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:16 smithi107 ceph-mon[36485]: purged_snaps scrub ok 2024-03-24T00:13:17.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:16 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:13:17.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:16 smithi107 ceph-mon[36485]: Detected new or changed devices on smithi078 2024-03-24T00:13:17.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:16 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:17.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:16 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:17.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:16 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:13:17.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:16 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi078 to 17088M 2024-03-24T00:13:17.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:16 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:17.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:16 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:13:17.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:16 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:13:17.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:16 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:17.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:16 smithi107 ceph-mon[36485]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:13:17.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:16 smithi107 ceph-mon[36485]: osd.0 [v2:172.21.15.78:6802/1816879397,v1:172.21.15.78:6803/1816879397] boot 2024-03-24T00:13:17.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:16 smithi107 ceph-mon[36485]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T00:13:17.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:16 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:13:17.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.163+0000 7fc85bc30700 1 -- 172.21.15.106:0/3184332331 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc854102cc0 msgr2=0x7fc8541050b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:17.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.163+0000 7fc85bc30700 1 --2- 172.21.15.106:0/3184332331 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc854102cc0 0x7fc8541050b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc850009990 tx=0x7fc850009ca0 comp rx=0 tx=0).stop 2024-03-24T00:13:17.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.163+0000 7fc85bc30700 1 -- 172.21.15.106:0/3184332331 shutdown_connections 2024-03-24T00:13:17.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.163+0000 7fc85bc30700 1 --2- 172.21.15.106:0/3184332331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc854107f20 0x7fc85410a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:17.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.163+0000 7fc85bc30700 1 --2- 172.21.15.106:0/3184332331 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc854102cc0 0x7fc8541050b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:17.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.163+0000 7fc85bc30700 1 --2- 172.21.15.106:0/3184332331 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8541055f0 0x7fc8541079e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:17.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.163+0000 7fc85bc30700 1 -- 172.21.15.106:0/3184332331 >> 172.21.15.106:0/3184332331 conn(0x7fc8540fc6f0 msgr2=0x7fc8540feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:17.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.163+0000 7fc85bc30700 1 -- 172.21.15.106:0/3184332331 shutdown_connections 2024-03-24T00:13:17.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.163+0000 7fc85bc30700 1 -- 172.21.15.106:0/3184332331 wait complete. 2024-03-24T00:13:17.165 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.163+0000 7fc85bc30700 1 Processor -- start 2024-03-24T00:13:17.165 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.163+0000 7fc85bc30700 1 -- start start 2024-03-24T00:13:17.165 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.164+0000 7fc85bc30700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc854102cc0 0x7fc8541983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:17.165 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.164+0000 7fc85bc30700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc8541055f0 0x7fc854198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:17.165 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.164+0000 7fc85bc30700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc854107f20 0x7fc85419f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:17.165 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.164+0000 7fc85bc30700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85410dda0 con 0x7fc854107f20 2024-03-24T00:13:17.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.164+0000 7fc85bc30700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85410dc20 con 0x7fc8541055f0 2024-03-24T00:13:17.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.164+0000 7fc85bc30700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85410df20 con 0x7fc854102cc0 2024-03-24T00:13:17.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.164+0000 7fc8591cb700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc8541055f0 0x7fc854198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:17.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.164+0000 7fc8591cb700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc8541055f0 0x7fc854198920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:36346/0 (socket says 172.21.15.106:36346) 2024-03-24T00:13:17.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.164+0000 7fc8591cb700 1 -- 172.21.15.106:0/779196581 learned_addr learned my addr 172.21.15.106:0/779196581 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-24T00:13:17.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.164+0000 7fc8599cc700 1 --2- 172.21.15.106:0/779196581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc854102cc0 0x7fc8541983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:17.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.164+0000 7fc85a1cd700 1 --2- 172.21.15.106:0/779196581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc854107f20 0x7fc85419f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:17.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.164+0000 7fc8591cb700 1 -- 172.21.15.106:0/779196581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc854102cc0 msgr2=0x7fc8541983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:17.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.164+0000 7fc8591cb700 1 --2- 172.21.15.106:0/779196581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc854102cc0 0x7fc8541983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:17.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.164+0000 7fc8591cb700 1 -- 172.21.15.106:0/779196581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc854107f20 msgr2=0x7fc85419f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:17.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.164+0000 7fc8591cb700 1 --2- 172.21.15.106:0/779196581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc854107f20 0x7fc85419f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:17.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.164+0000 7fc8591cb700 1 -- 172.21.15.106:0/779196581 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8541a0040 con 0x7fc8541055f0 2024-03-24T00:13:17.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.164+0000 7fc8591cb700 1 --2- 172.21.15.106:0/779196581 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc8541055f0 0x7fc854198920 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc84400d950 tx=0x7fc84400dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:17.167 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.164+0000 7fc84affd700 1 -- 172.21.15.106:0/779196581 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc844016070 con 0x7fc8541055f0 2024-03-24T00:13:17.167 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.164+0000 7fc84affd700 1 -- 172.21.15.106:0/779196581 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc84400fd70 con 0x7fc8541055f0 2024-03-24T00:13:17.167 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.165+0000 7fc84affd700 1 -- 172.21.15.106:0/779196581 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84400e610 con 0x7fc8541055f0 2024-03-24T00:13:17.167 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.165+0000 7fc85bc30700 1 -- 172.21.15.106:0/779196581 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8541a0330 con 0x7fc8541055f0 2024-03-24T00:13:17.167 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.165+0000 7fc85bc30700 1 -- 172.21.15.106:0/779196581 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8541a08c0 con 0x7fc8541055f0 2024-03-24T00:13:17.167 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.165+0000 7fc85bc30700 1 -- 172.21.15.106:0/779196581 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc854068e50 con 0x7fc8541055f0 2024-03-24T00:13:17.168 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.166+0000 7fc84affd700 1 -- 172.21.15.106:0/779196581 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7fc844019070 con 0x7fc8541055f0 2024-03-24T00:13:17.168 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.167+0000 7fc84affd700 1 --2- 172.21.15.106:0/779196581 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fc8400a2880 0x7fc8400a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:17.168 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.167+0000 7fc84affd700 1 -- 172.21.15.106:0/779196581 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1767+0+0 (secure 0 0 0) 0x7fc8440cc9a0 con 0x7fc8541055f0 2024-03-24T00:13:17.168 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.167+0000 7fc8599cc700 1 --2- 172.21.15.106:0/779196581 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fc8400a2880 0x7fc8400a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:17.168 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.168+0000 7fc8599cc700 1 --2- 172.21.15.106:0/779196581 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fc8400a2880 0x7fc8400a4d40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc85000b5c0 tx=0x7fc850003960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:17.171 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.172+0000 7fc84affd700 1 -- 172.21.15.106:0/779196581 <== mon.2 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc84409b4b0 con 0x7fc8541055f0 2024-03-24T00:13:17.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:17.374+0000 7fc85bc30700 1 -- 172.21.15.106:0/779196581 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fc8540625d0 con 0x7fc8400a2880 2024-03-24T00:13:18.245 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:17 smithi106 ceph-mon[36505]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:13:18.245 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:17 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:13:18.245 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:17 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:13:18.246 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:17 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:13:18.246 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:17 smithi106 ceph-mon[36505]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T00:13:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:17 smithi078 ceph-mon[32429]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:13:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:17 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:13:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:17 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:13:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:17 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:13:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:17 smithi078 ceph-mon[32429]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T00:13:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:17 smithi107 ceph-mon[36485]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:13:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:17 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:13:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:17 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:13:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:17 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:13:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:17 smithi107 ceph-mon[36485]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T00:13:19.056 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:18 smithi106 ceph-mon[36505]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:13:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:18 smithi078 ceph-mon[32429]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:13:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:18 smithi107 ceph-mon[36485]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:13:20.187 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:19 smithi106 ceph-mon[36505]: from='client.? 172.21.15.106:0/2246359104' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8d64714-dd65-400f-b3f4-673ab14e1365"}]: dispatch 2024-03-24T00:13:20.187 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:19 smithi106 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8d64714-dd65-400f-b3f4-673ab14e1365"}]: dispatch 2024-03-24T00:13:20.187 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:19 smithi106 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b8d64714-dd65-400f-b3f4-673ab14e1365"}]': finished 2024-03-24T00:13:20.187 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:19 smithi106 ceph-mon[36505]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T00:13:20.187 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:19 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:13:20.244 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:19 smithi107 ceph-mon[36485]: from='client.? 172.21.15.106:0/2246359104' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8d64714-dd65-400f-b3f4-673ab14e1365"}]: dispatch 2024-03-24T00:13:20.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:19 smithi107 ceph-mon[36485]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8d64714-dd65-400f-b3f4-673ab14e1365"}]: dispatch 2024-03-24T00:13:20.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:19 smithi107 ceph-mon[36485]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b8d64714-dd65-400f-b3f4-673ab14e1365"}]': finished 2024-03-24T00:13:20.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:19 smithi107 ceph-mon[36485]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T00:13:20.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:19 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:13:20.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:19 smithi078 ceph-mon[32429]: from='client.? 172.21.15.106:0/2246359104' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8d64714-dd65-400f-b3f4-673ab14e1365"}]: dispatch 2024-03-24T00:13:20.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:19 smithi078 ceph-mon[32429]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8d64714-dd65-400f-b3f4-673ab14e1365"}]: dispatch 2024-03-24T00:13:20.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:19 smithi078 ceph-mon[32429]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b8d64714-dd65-400f-b3f4-673ab14e1365"}]': finished 2024-03-24T00:13:20.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:19 smithi078 ceph-mon[32429]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T00:13:20.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:19 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:13:21.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:20 smithi078 ceph-mon[32429]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:13:21.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:20 smithi078 ceph-mon[32429]: from='client.? 172.21.15.106:0/1132565601' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:13:21.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:20 smithi107 ceph-mon[36485]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:13:21.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:20 smithi107 ceph-mon[36485]: from='client.? 172.21.15.106:0/1132565601' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:13:21.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:20 smithi106 ceph-mon[36505]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:13:21.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:20 smithi106 ceph-mon[36505]: from='client.? 172.21.15.106:0/1132565601' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:13:23.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:23 smithi078 ceph-mon[32429]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:13:23.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:23 smithi107 ceph-mon[36485]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:13:23.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:23 smithi106 ceph-mon[36505]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:13:25.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:25 smithi106 ceph-mon[36505]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:13:25.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:25 smithi078 ceph-mon[32429]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:13:25.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:25 smithi107 ceph-mon[36485]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:13:27.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:27 smithi078 ceph-mon[32429]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:13:27.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:27 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T00:13:27.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:27 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:13:27.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:27 smithi107 ceph-mon[36485]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:13:27.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:27 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T00:13:27.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:27 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:13:27.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:27 smithi106 ceph-mon[36505]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:13:27.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:27 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T00:13:27.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:27 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:13:28.442 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:28 smithi106 ceph-mon[36505]: Deploying daemon osd.1 on smithi106 2024-03-24T00:13:28.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:28 smithi078 ceph-mon[32429]: Deploying daemon osd.1 on smithi106 2024-03-24T00:13:28.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:28 smithi107 ceph-mon[36485]: Deploying daemon osd.1 on smithi106 2024-03-24T00:13:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:29 smithi106 ceph-mon[36505]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:13:29.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:29 smithi078 ceph-mon[32429]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:13:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:29 smithi107 ceph-mon[36485]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:13:31.539 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:31 smithi106 ceph-mon[36505]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:13:31.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:31 smithi078 ceph-mon[32429]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:13:31.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:31 smithi107 ceph-mon[36485]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:13:33.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:33 smithi106 ceph-mon[36505]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:13:33.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:33 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:33.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:33 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:33.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:33 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:13:33.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:33 smithi078 ceph-mon[32429]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:13:33.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:33 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:33.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:33 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:33.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:33 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:13:33.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:33 smithi107 ceph-mon[36485]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:13:33.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:33 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:33.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:33 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:33.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:33 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:13:35.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:35 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:35.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:35 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:35.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:35 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:13:35.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:35 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:13:35.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:35 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:35.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:35 smithi106 ceph-mon[36505]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:13:35.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:35 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:35.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:35 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:35.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:35 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:13:35.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:35 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:13:35.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:35 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:35.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:35 smithi078 ceph-mon[32429]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:13:35.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:35 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:35.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:35 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:35.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:35 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:13:35.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:35 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:13:35.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:35 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:35.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:35 smithi107 ceph-mon[36485]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:13:36.189 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:36.189+0000 7fc84affd700 1 -- 172.21.15.106:0/779196581 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc8540625d0 con 0x7fc8400a2880 2024-03-24T00:13:36.190 INFO:teuthology.orchestra.run.smithi106.stdout:Created osd(s) 1 on host 'smithi106' 2024-03-24T00:13:36.191 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:36.191+0000 7fc85bc30700 1 -- 172.21.15.106:0/779196581 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fc8400a2880 msgr2=0x7fc8400a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:36.191 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:36.191+0000 7fc85bc30700 1 --2- 172.21.15.106:0/779196581 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fc8400a2880 0x7fc8400a4d40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc85000b5c0 tx=0x7fc850003960 comp rx=0 tx=0).stop 2024-03-24T00:13:36.191 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:36.191+0000 7fc85bc30700 1 -- 172.21.15.106:0/779196581 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc8541055f0 msgr2=0x7fc854198920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:36.192 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:36.191+0000 7fc85bc30700 1 --2- 172.21.15.106:0/779196581 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc8541055f0 0x7fc854198920 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc84400d950 tx=0x7fc84400dc60 comp rx=0 tx=0).stop 2024-03-24T00:13:36.192 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:36.191+0000 7fc85bc30700 1 -- 172.21.15.106:0/779196581 shutdown_connections 2024-03-24T00:13:36.192 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:36.191+0000 7fc85bc30700 1 --2- 172.21.15.106:0/779196581 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fc8400a2880 0x7fc8400a4d40 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:36.192 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:36.191+0000 7fc85bc30700 1 --2- 172.21.15.106:0/779196581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc854107f20 0x7fc85419f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:36.192 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:36.191+0000 7fc85bc30700 1 --2- 172.21.15.106:0/779196581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc854102cc0 0x7fc8541983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:36.192 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:36.191+0000 7fc85bc30700 1 --2- 172.21.15.106:0/779196581 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc8541055f0 0x7fc854198920 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:36.192 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:36.191+0000 7fc85bc30700 1 -- 172.21.15.106:0/779196581 >> 172.21.15.106:0/779196581 conn(0x7fc8540fc6f0 msgr2=0x7fc854103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:36.192 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:36.191+0000 7fc85bc30700 1 -- 172.21.15.106:0/779196581 shutdown_connections 2024-03-24T00:13:36.192 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:13:36.191+0000 7fc85bc30700 1 -- 172.21.15.106:0/779196581 wait complete. 2024-03-24T00:13:36.829 DEBUG:teuthology.orchestra.run.smithi106:osd.1> sudo journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.1.service 2024-03-24T00:13:36.831 INFO:tasks.cephadm:Deploying osd.2 on smithi107 with /dev/vg_nvme/lv_4... 2024-03-24T00:13:36.832 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T00:13:37.075 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:13:37.348 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:37 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:37.349 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:37 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:37.349 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:37 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:13:37.349 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:37 smithi106 ceph-mon[36505]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:13:37.349 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:37 smithi106 ceph-mon[36505]: from='osd.1 [v2:172.21.15.106:6800/1647915612,v1:172.21.15.106:6801/1647915612]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T00:13:37.349 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:37 smithi106 ceph-mon[36505]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T00:13:37.406 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:37 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:37.406 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:37 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:37.406 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:37 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:13:37.407 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:37 smithi107 ceph-mon[36485]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:13:37.407 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:37 smithi107 ceph-mon[36485]: from='osd.1 [v2:172.21.15.106:6800/1647915612,v1:172.21.15.106:6801/1647915612]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T00:13:37.407 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:37 smithi107 ceph-mon[36485]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T00:13:37.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:37 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:37.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:37 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:37.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:37 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:13:37.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:37 smithi078 ceph-mon[32429]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:13:37.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:37 smithi078 ceph-mon[32429]: from='osd.1 [v2:172.21.15.106:6800/1647915612,v1:172.21.15.106:6801/1647915612]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T00:13:37.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:37 smithi078 ceph-mon[32429]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T00:13:38.364 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:38 smithi106 ceph-mon[36505]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T00:13:38.364 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:38 smithi106 ceph-mon[36505]: from='osd.1 [v2:172.21.15.106:6800/1647915612,v1:172.21.15.106:6801/1647915612]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-24T00:13:38.364 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:38 smithi106 ceph-mon[36505]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T00:13:38.364 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:38 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:13:38.364 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:38 smithi106 ceph-mon[36505]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-24T00:13:38.386 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:38 smithi107 ceph-mon[36485]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T00:13:38.386 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:38 smithi107 ceph-mon[36485]: from='osd.1 [v2:172.21.15.106:6800/1647915612,v1:172.21.15.106:6801/1647915612]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-24T00:13:38.386 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:38 smithi107 ceph-mon[36485]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T00:13:38.386 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:38 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:13:38.386 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:38 smithi107 ceph-mon[36485]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-24T00:13:38.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:38 smithi078 ceph-mon[32429]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T00:13:38.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:38 smithi078 ceph-mon[32429]: from='osd.1 [v2:172.21.15.106:6800/1647915612,v1:172.21.15.106:6801/1647915612]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-24T00:13:38.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:38 smithi078 ceph-mon[32429]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T00:13:38.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:38 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:13:38.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:38 smithi078 ceph-mon[32429]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-24T00:13:38.938 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T00:13:38.939 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T00:13:38.939 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10+0 records in 2024-03-24T00:13:38.939 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2024-03-24T00:13:38.939 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0118327 s, 886 MB/s 2024-03-24T00:13:38.939 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping successful for: 2024-03-24T00:13:39.169 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 24 00:13:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-1[41094]: 2024-03-24T00:13:38.905+0000 7f461bb9b700 -1 osd.1 0 waiting for initial osdmap 2024-03-24T00:13:39.169 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 24 00:13:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-1[41094]: 2024-03-24T00:13:38.911+0000 7f4613986700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:13:39.465 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:39 smithi107 ceph-mon[36485]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-24T00:13:39.465 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:39 smithi107 ceph-mon[36485]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T00:13:39.465 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:39 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:13:39.465 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:39 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:13:39.465 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:39 smithi107 ceph-mon[36485]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:13:39.465 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:39 smithi107 ceph-mon[36485]: from='osd.1 ' entity='osd.1' 2024-03-24T00:13:39.491 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi107:vg_nvme/lv_4 2024-03-24T00:13:39.550 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:39 smithi106 ceph-mon[36505]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-24T00:13:39.550 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:39 smithi106 ceph-mon[36505]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T00:13:39.550 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:39 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:13:39.550 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:39 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:13:39.550 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:39 smithi106 ceph-mon[36505]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:13:39.550 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:39 smithi106 ceph-mon[36505]: from='osd.1 ' entity='osd.1' 2024-03-24T00:13:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:39 smithi078 ceph-mon[32429]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-24T00:13:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:39 smithi078 ceph-mon[32429]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T00:13:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:39 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:13:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:39 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:13:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:39 smithi078 ceph-mon[32429]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:13:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:39 smithi078 ceph-mon[32429]: from='osd.1 ' entity='osd.1' 2024-03-24T00:13:39.732 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:13:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:40 smithi107 ceph-mon[36485]: purged_snaps scrub starts 2024-03-24T00:13:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:40 smithi107 ceph-mon[36485]: purged_snaps scrub ok 2024-03-24T00:13:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:13:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:40 smithi107 ceph-mon[36485]: osd.1 [v2:172.21.15.106:6800/1647915612,v1:172.21.15.106:6801/1647915612] boot 2024-03-24T00:13:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:40 smithi107 ceph-mon[36485]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T00:13:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:13:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:40.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:40.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:13:40.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:40.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:13:40.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:13:40.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:40.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:40 smithi078 ceph-mon[32429]: purged_snaps scrub starts 2024-03-24T00:13:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:40 smithi078 ceph-mon[32429]: purged_snaps scrub ok 2024-03-24T00:13:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:13:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:40 smithi078 ceph-mon[32429]: osd.1 [v2:172.21.15.106:6800/1647915612,v1:172.21.15.106:6801/1647915612] boot 2024-03-24T00:13:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:40 smithi078 ceph-mon[32429]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T00:13:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:13:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:13:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:13:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:13:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:40.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:40 smithi106 ceph-mon[36505]: purged_snaps scrub starts 2024-03-24T00:13:40.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:40 smithi106 ceph-mon[36505]: purged_snaps scrub ok 2024-03-24T00:13:40.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:13:40.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:40 smithi106 ceph-mon[36505]: osd.1 [v2:172.21.15.106:6800/1647915612,v1:172.21.15.106:6801/1647915612] boot 2024-03-24T00:13:40.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:40 smithi106 ceph-mon[36505]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T00:13:40.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:13:40.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:40.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:40.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:13:40.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:40.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:13:40.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:13:40.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:41.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.259+0000 7f2682fb9700 1 -- 172.21.15.107:0/3428339914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f267c107eb0 msgr2=0x7f267c10a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:41.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.259+0000 7f2682fb9700 1 --2- 172.21.15.107:0/3428339914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f267c107eb0 0x7f267c10a320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2674009990 tx=0x7f2674009ca0 comp rx=0 tx=0).stop 2024-03-24T00:13:41.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.259+0000 7f2682fb9700 1 -- 172.21.15.107:0/3428339914 shutdown_connections 2024-03-24T00:13:41.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.259+0000 7f2682fb9700 1 --2- 172.21.15.107:0/3428339914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f267c107eb0 0x7f267c10a320 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:41.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.259+0000 7f2682fb9700 1 --2- 172.21.15.107:0/3428339914 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f267c102c50 0x7f267c105040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:41.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.259+0000 7f2682fb9700 1 --2- 172.21.15.107:0/3428339914 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f267c105580 0x7f267c107970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.259+0000 7f2682fb9700 1 -- 172.21.15.107:0/3428339914 >> 172.21.15.107:0/3428339914 conn(0x7f267c0fc660 msgr2=0x7f267c0feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.259+0000 7f2682fb9700 1 -- 172.21.15.107:0/3428339914 shutdown_connections 2024-03-24T00:13:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.259+0000 7f2682fb9700 1 -- 172.21.15.107:0/3428339914 wait complete. 2024-03-24T00:13:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.260+0000 7f2682fb9700 1 Processor -- start 2024-03-24T00:13:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.260+0000 7f2682fb9700 1 -- start start 2024-03-24T00:13:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.260+0000 7f2682fb9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f267c102c50 0x7f267c1025b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.260+0000 7f2682fb9700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f267c105580 0x7f267c100c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.260+0000 7f2682fb9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f267c107eb0 0x7f267c101140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.260+0000 7f2682fb9700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f267c10de70 con 0x7f267c107eb0 2024-03-24T00:13:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.260+0000 7f2682fb9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f267c10dcf0 con 0x7f267c102c50 2024-03-24T00:13:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.260+0000 7f2682fb9700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f267c10dff0 con 0x7f267c105580 2024-03-24T00:13:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.260+0000 7f267bfff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f267c105580 0x7f267c100c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.260+0000 7f267bfff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f267c105580 0x7f267c100c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39092/0 (socket says 172.21.15.107:39092) 2024-03-24T00:13:41.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.260+0000 7f267bfff700 1 -- 172.21.15.107:0/525701084 learned_addr learned my addr 172.21.15.107:0/525701084 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-03-24T00:13:41.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.260+0000 7f2680d55700 1 --2- 172.21.15.107:0/525701084 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f267c102c50 0x7f267c1025b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:41.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.260+0000 7f2681556700 1 --2- 172.21.15.107:0/525701084 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f267c107eb0 0x7f267c101140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:41.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.260+0000 7f267bfff700 1 -- 172.21.15.107:0/525701084 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f267c102c50 msgr2=0x7f267c1025b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:41.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.260+0000 7f267bfff700 1 --2- 172.21.15.107:0/525701084 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f267c102c50 0x7f267c1025b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:41.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.260+0000 7f267bfff700 1 -- 172.21.15.107:0/525701084 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f267c107eb0 msgr2=0x7f267c101140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:41.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.260+0000 7f267bfff700 1 --2- 172.21.15.107:0/525701084 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f267c107eb0 0x7f267c101140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:41.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.260+0000 7f267bfff700 1 -- 172.21.15.107:0/525701084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f267c1019f0 con 0x7f267c105580 2024-03-24T00:13:41.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.260+0000 7f2681556700 1 --2- 172.21.15.107:0/525701084 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f267c107eb0 0x7f267c101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:13:41.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.260+0000 7f267bfff700 1 --2- 172.21.15.107:0/525701084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f267c105580 0x7f267c100c00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f266400b530 tx=0x7f266400b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:41.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.261+0000 7f2679ffb700 1 -- 172.21.15.107:0/525701084 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2664015020 con 0x7f267c105580 2024-03-24T00:13:41.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.261+0000 7f2679ffb700 1 -- 172.21.15.107:0/525701084 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2664011d10 con 0x7f267c105580 2024-03-24T00:13:41.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.261+0000 7f2682fb9700 1 -- 172.21.15.107:0/525701084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f267c1a26d0 con 0x7f267c105580 2024-03-24T00:13:41.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.261+0000 7f2679ffb700 1 -- 172.21.15.107:0/525701084 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2664010e50 con 0x7f267c105580 2024-03-24T00:13:41.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.261+0000 7f2682fb9700 1 -- 172.21.15.107:0/525701084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f267c1a2c90 con 0x7f267c105580 2024-03-24T00:13:41.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.262+0000 7f2682fb9700 1 -- 172.21.15.107:0/525701084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f267c068e50 con 0x7f267c105580 2024-03-24T00:13:41.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.263+0000 7f2679ffb700 1 -- 172.21.15.107:0/525701084 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7f2664018020 con 0x7f267c105580 2024-03-24T00:13:41.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.263+0000 7f2679ffb700 1 --2- 172.21.15.107:0/525701084 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f26680a2a20 0x7f26680a4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:41.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.263+0000 7f2679ffb700 1 -- 172.21.15.107:0/525701084 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7f26640cc9e0 con 0x7f267c105580 2024-03-24T00:13:41.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.263+0000 7f2680d55700 1 --2- 172.21.15.107:0/525701084 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f26680a2a20 0x7f26680a4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:41.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.264+0000 7f2680d55700 1 --2- 172.21.15.107:0/525701084 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f26680a2a20 0x7f26680a4ee0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f266c009f20 tx=0x7f266c0094f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:41.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.269+0000 7f2679ffb700 1 -- 172.21.15.107:0/525701084 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f266401f020 con 0x7f267c105580 2024-03-24T00:13:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:41 smithi107 ceph-mon[36485]: Detected new or changed devices on smithi106 2024-03-24T00:13:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:41 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi106 to 17080M 2024-03-24T00:13:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:41 smithi107 ceph-mon[36485]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:13:41.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:41.472+0000 7f2682fb9700 1 -- 172.21.15.107:0/525701084 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f267c0625d0 con 0x7f26680a2a20 2024-03-24T00:13:41.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:41 smithi078 ceph-mon[32429]: Detected new or changed devices on smithi106 2024-03-24T00:13:41.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:41 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi106 to 17080M 2024-03-24T00:13:41.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:41 smithi078 ceph-mon[32429]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:13:41.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:41 smithi106 ceph-mon[36505]: Detected new or changed devices on smithi106 2024-03-24T00:13:41.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:41 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi106 to 17080M 2024-03-24T00:13:41.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:41 smithi106 ceph-mon[36505]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:13:42.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:42 smithi107 ceph-mon[36485]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T00:13:42.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:42 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:13:42.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:42 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:13:42.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:42 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:13:42.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:42 smithi078 ceph-mon[32429]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T00:13:42.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:42 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:13:42.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:42 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:13:42.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:42 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:13:42.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:42 smithi106 ceph-mon[36505]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T00:13:42.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:42 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:13:42.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:42 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:13:42.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:42 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:13:43.281 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:43 smithi107 ceph-mon[36485]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:13:43.281 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:43 smithi107 ceph-mon[36485]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:13:43.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:43 smithi078 ceph-mon[32429]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:13:43.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:43 smithi078 ceph-mon[32429]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:13:43.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:43 smithi106 ceph-mon[36505]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:13:43.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:43 smithi106 ceph-mon[36505]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:13:44.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:44 smithi107 ceph-mon[36485]: from='client.? 172.21.15.107:0/307731667' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fcfc4dd1-aaf8-4131-a023-5e832212158d"}]: dispatch 2024-03-24T00:13:44.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:44 smithi107 ceph-mon[36485]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fcfc4dd1-aaf8-4131-a023-5e832212158d"}]: dispatch 2024-03-24T00:13:44.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:44 smithi107 ceph-mon[36485]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fcfc4dd1-aaf8-4131-a023-5e832212158d"}]': finished 2024-03-24T00:13:44.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:44 smithi107 ceph-mon[36485]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T00:13:44.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:44 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:13:44.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:44 smithi078 ceph-mon[32429]: from='client.? 172.21.15.107:0/307731667' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fcfc4dd1-aaf8-4131-a023-5e832212158d"}]: dispatch 2024-03-24T00:13:44.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:44 smithi078 ceph-mon[32429]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fcfc4dd1-aaf8-4131-a023-5e832212158d"}]: dispatch 2024-03-24T00:13:44.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:44 smithi078 ceph-mon[32429]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fcfc4dd1-aaf8-4131-a023-5e832212158d"}]': finished 2024-03-24T00:13:44.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:44 smithi078 ceph-mon[32429]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T00:13:44.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:44 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:13:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:44 smithi106 ceph-mon[36505]: from='client.? 172.21.15.107:0/307731667' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fcfc4dd1-aaf8-4131-a023-5e832212158d"}]: dispatch 2024-03-24T00:13:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:44 smithi106 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fcfc4dd1-aaf8-4131-a023-5e832212158d"}]: dispatch 2024-03-24T00:13:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:44 smithi106 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fcfc4dd1-aaf8-4131-a023-5e832212158d"}]': finished 2024-03-24T00:13:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:44 smithi106 ceph-mon[36505]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T00:13:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:44 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:13:45.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:45 smithi078 ceph-mon[32429]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:13:45.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:45 smithi078 ceph-mon[32429]: from='client.? 172.21.15.107:0/3734956250' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:13:45.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:45 smithi107 ceph-mon[36485]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:13:45.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:45 smithi107 ceph-mon[36485]: from='client.? 172.21.15.107:0/3734956250' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:13:45.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:45 smithi106 ceph-mon[36505]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:13:45.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:45 smithi106 ceph-mon[36505]: from='client.? 172.21.15.107:0/3734956250' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:13:47.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:47 smithi078 ceph-mon[32429]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:13:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:47 smithi107 ceph-mon[36485]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:13:47.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:47 smithi106 ceph-mon[36505]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:13:49.541 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:49 smithi107 ceph-mon[36485]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:13:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:49 smithi078 ceph-mon[32429]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:13:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:49 smithi106 ceph-mon[36505]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:13:51.405 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:51 smithi107 ceph-mon[36485]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:13:51.405 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:51 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T00:13:51.405 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:51 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:13:51.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:51 smithi078 ceph-mon[32429]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:13:51.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:51 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T00:13:51.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:51 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:13:51.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:51 smithi106 ceph-mon[36505]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:13:51.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:51 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T00:13:51.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:51 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:13:52.336 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:52 smithi107 ceph-mon[36485]: Deploying daemon osd.2 on smithi107 2024-03-24T00:13:52.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:52 smithi078 ceph-mon[32429]: Deploying daemon osd.2 on smithi107 2024-03-24T00:13:52.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:52 smithi106 ceph-mon[36505]: Deploying daemon osd.2 on smithi107 2024-03-24T00:13:53.576 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:53 smithi107 ceph-mon[36485]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:13:53.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:53 smithi078 ceph-mon[32429]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:13:53.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:53 smithi106 ceph-mon[36505]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:13:55.567 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:55 smithi107 ceph-mon[36485]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:13:55.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:55 smithi078 ceph-mon[32429]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:13:55.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:55 smithi106 ceph-mon[36505]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:13:57.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:57 smithi078 ceph-mon[32429]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:13:57.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:57 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:57.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:57 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:57.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:57 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:13:57.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:57 smithi107 ceph-mon[36485]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:13:57.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:57 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:57.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:57 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:57.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:57 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:13:57.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:57 smithi106 ceph-mon[36505]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:13:57.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:57 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:57.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:57 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:57.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:57 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:13:59.665 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:59 smithi078 ceph-mon[32429]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:13:59.665 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:59 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:59.665 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:59 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:59.665 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:59 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:13:59.665 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:59 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:13:59.665 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:13:59 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:59.668 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:59 smithi107 ceph-mon[36485]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:13:59.668 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:59 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:59.668 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:59 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:59.668 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:59 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:13:59.668 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:59 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:13:59.669 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:13:59 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:59.741 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 2 on host 'smithi107' 2024-03-24T00:13:59.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:59.740+0000 7f2679ffb700 1 -- 172.21.15.107:0/525701084 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f267c0625d0 con 0x7f26680a2a20 2024-03-24T00:13:59.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:59.743+0000 7f2682fb9700 1 -- 172.21.15.107:0/525701084 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f26680a2a20 msgr2=0x7f26680a4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:59.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:59.743+0000 7f2682fb9700 1 --2- 172.21.15.107:0/525701084 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f26680a2a20 0x7f26680a4ee0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f266c009f20 tx=0x7f266c0094f0 comp rx=0 tx=0).stop 2024-03-24T00:13:59.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:59.743+0000 7f2682fb9700 1 -- 172.21.15.107:0/525701084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f267c105580 msgr2=0x7f267c100c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:59.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:59.743+0000 7f2682fb9700 1 --2- 172.21.15.107:0/525701084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f267c105580 0x7f267c100c00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f266400b530 tx=0x7f266400b8f0 comp rx=0 tx=0).stop 2024-03-24T00:13:59.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:59.743+0000 7f2682fb9700 1 -- 172.21.15.107:0/525701084 shutdown_connections 2024-03-24T00:13:59.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:59.743+0000 7f2682fb9700 1 --2- 172.21.15.107:0/525701084 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f26680a2a20 0x7f26680a4ee0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:59.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:59.743+0000 7f2682fb9700 1 --2- 172.21.15.107:0/525701084 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f267c107eb0 0x7f267c101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:59.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:59.743+0000 7f2682fb9700 1 --2- 172.21.15.107:0/525701084 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f267c102c50 0x7f267c1025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:59.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:59.743+0000 7f2682fb9700 1 --2- 172.21.15.107:0/525701084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f267c105580 0x7f267c100c00 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:59.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:59.743+0000 7f2682fb9700 1 -- 172.21.15.107:0/525701084 >> 172.21.15.107:0/525701084 conn(0x7f267c0fc660 msgr2=0x7f267c1036b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:59.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:59.743+0000 7f2682fb9700 1 -- 172.21.15.107:0/525701084 shutdown_connections 2024-03-24T00:13:59.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:13:59.743+0000 7f2682fb9700 1 -- 172.21.15.107:0/525701084 wait complete. 2024-03-24T00:13:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:59 smithi106 ceph-mon[36505]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:13:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:59 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:59 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:13:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:59 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:13:59.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:59 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:13:59.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:13:59 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:00.403 DEBUG:teuthology.orchestra.run.smithi107:osd.2> sudo journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.2.service 2024-03-24T00:14:00.405 INFO:tasks.cephadm:Deploying osd.3 on smithi078 with /dev/vg_nvme/lv_3... 2024-03-24T00:14:00.405 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T00:14:00.652 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:14:00.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:00 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:00.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:00 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:00.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:00 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:14:00.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:00 smithi078 ceph-mon[32429]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:14:00.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:00 smithi078 ceph-mon[32429]: from='osd.2 [v2:172.21.15.107:6800/2010770435,v1:172.21.15.107:6801/2010770435]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T00:14:00.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:00 smithi078 ceph-mon[32429]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T00:14:01.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:00 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:01.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:00 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:01.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:00 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:14:01.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:00 smithi107 ceph-mon[36485]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:14:01.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:00 smithi107 ceph-mon[36485]: from='osd.2 [v2:172.21.15.107:6800/2010770435,v1:172.21.15.107:6801/2010770435]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T00:14:01.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:00 smithi107 ceph-mon[36485]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T00:14:01.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:00 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:01.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:00 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:01.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:00 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:14:01.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:00 smithi106 ceph-mon[36505]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:14:01.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:00 smithi106 ceph-mon[36505]: from='osd.2 [v2:172.21.15.107:6800/2010770435,v1:172.21.15.107:6801/2010770435]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T00:14:01.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:00 smithi106 ceph-mon[36505]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T00:14:01.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:01 smithi107 ceph-mon[36485]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T00:14:01.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:01 smithi107 ceph-mon[36485]: from='osd.2 [v2:172.21.15.107:6800/2010770435,v1:172.21.15.107:6801/2010770435]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-24T00:14:01.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:01 smithi107 ceph-mon[36485]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T00:14:01.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:01 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:14:01.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:01 smithi107 ceph-mon[36485]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-24T00:14:01.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:01 smithi078 ceph-mon[32429]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T00:14:01.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:01 smithi078 ceph-mon[32429]: from='osd.2 [v2:172.21.15.107:6800/2010770435,v1:172.21.15.107:6801/2010770435]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-24T00:14:01.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:01 smithi078 ceph-mon[32429]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T00:14:01.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:01 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:14:01.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:01 smithi078 ceph-mon[32429]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-24T00:14:02.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:01 smithi106 ceph-mon[36505]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T00:14:02.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:01 smithi106 ceph-mon[36505]: from='osd.2 [v2:172.21.15.107:6800/2010770435,v1:172.21.15.107:6801/2010770435]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-24T00:14:02.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:01 smithi106 ceph-mon[36505]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T00:14:02.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:01 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:14:02.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:01 smithi106 ceph-mon[36505]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-24T00:14:02.513 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T00:14:02.513 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T00:14:02.513 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10+0 records in 2024-03-24T00:14:02.513 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2024-03-24T00:14:02.513 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115052 s, 911 MB/s 2024-03-24T00:14:02.513 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping successful for: 2024-03-24T00:14:02.729 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 24 00:14:02 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-2[41647]: 2024-03-24T00:14:02.462+0000 7f64401d4700 -1 osd.2 0 waiting for initial osdmap 2024-03-24T00:14:02.729 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 24 00:14:02 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-2[41647]: 2024-03-24T00:14:02.467+0000 7f6437fbf700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:14:03.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:02 smithi078 ceph-mon[32429]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-03-24T00:14:03.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:02 smithi078 ceph-mon[32429]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T00:14:03.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:02 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:14:03.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:02 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:14:03.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:02 smithi078 ceph-mon[32429]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:14:03.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:02 smithi078 ceph-mon[32429]: from='osd.2 ' entity='osd.2' 2024-03-24T00:14:03.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:02 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:14:03.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:02 smithi107 ceph-mon[36485]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-03-24T00:14:03.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:02 smithi107 ceph-mon[36485]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T00:14:03.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:02 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:14:03.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:02 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:14:03.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:02 smithi107 ceph-mon[36485]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:14:03.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:02 smithi107 ceph-mon[36485]: from='osd.2 ' entity='osd.2' 2024-03-24T00:14:03.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:02 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:14:03.145 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi078:vg_nvme/lv_3 2024-03-24T00:14:03.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:02 smithi106 ceph-mon[36505]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-03-24T00:14:03.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:02 smithi106 ceph-mon[36505]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T00:14:03.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:02 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:14:03.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:02 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:14:03.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:02 smithi106 ceph-mon[36505]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:14:03.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:02 smithi106 ceph-mon[36505]: from='osd.2 ' entity='osd.2' 2024-03-24T00:14:03.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:02 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:14:03.382 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:14:03.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:03 smithi078 ceph-mon[32429]: purged_snaps scrub starts 2024-03-24T00:14:03.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:03 smithi078 ceph-mon[32429]: purged_snaps scrub ok 2024-03-24T00:14:03.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:03 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:03.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:03 smithi078 ceph-mon[32429]: osd.2 [v2:172.21.15.107:6800/2010770435,v1:172.21.15.107:6801/2010770435] boot 2024-03-24T00:14:03.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:03 smithi078 ceph-mon[32429]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T00:14:03.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:03 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:14:03.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:03 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:03.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:03 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:14:03.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:03 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:03.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:03 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:14:03.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:03 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:14:03.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:03 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:03 smithi107 ceph-mon[36485]: purged_snaps scrub starts 2024-03-24T00:14:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:03 smithi107 ceph-mon[36485]: purged_snaps scrub ok 2024-03-24T00:14:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:03 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:03 smithi107 ceph-mon[36485]: osd.2 [v2:172.21.15.107:6800/2010770435,v1:172.21.15.107:6801/2010770435] boot 2024-03-24T00:14:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:03 smithi107 ceph-mon[36485]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T00:14:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:03 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:14:04.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:03 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:04.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:03 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:14:04.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:03 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:04.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:03 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:14:04.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:03 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:14:04.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:03 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:03 smithi106 ceph-mon[36505]: purged_snaps scrub starts 2024-03-24T00:14:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:03 smithi106 ceph-mon[36505]: purged_snaps scrub ok 2024-03-24T00:14:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:03 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:03 smithi106 ceph-mon[36505]: osd.2 [v2:172.21.15.107:6800/2010770435,v1:172.21.15.107:6801/2010770435] boot 2024-03-24T00:14:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:03 smithi106 ceph-mon[36505]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T00:14:04.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:03 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:14:04.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:03 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:04.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:03 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:14:04.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:03 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:04.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:03 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:14:04.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:03 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:14:04.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:03 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:04.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.920+0000 7f79fe8a3700 1 -- 172.21.15.78:0/2359195935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79f81033e0 msgr2=0x7f79f8103840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:04.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.920+0000 7f79fe8a3700 1 --2- 172.21.15.78:0/2359195935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79f81033e0 0x7f79f8103840 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f79ec0099c0 tx=0x7f79ec009cd0 comp rx=0 tx=0).stop 2024-03-24T00:14:04.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.920+0000 7f79fe8a3700 1 -- 172.21.15.78:0/2359195935 shutdown_connections 2024-03-24T00:14:04.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.920+0000 7f79fe8a3700 1 --2- 172.21.15.78:0/2359195935 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79f8103d80 0x7f79f810a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:04.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.920+0000 7f79fe8a3700 1 --2- 172.21.15.78:0/2359195935 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79f81021e0 0x7f79f8102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:04.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.920+0000 7f79fe8a3700 1 --2- 172.21.15.78:0/2359195935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79f81033e0 0x7f79f8103840 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:04.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.920+0000 7f79fe8a3700 1 -- 172.21.15.78:0/2359195935 >> 172.21.15.78:0/2359195935 conn(0x7f79f80fd7a0 msgr2=0x7f79f80ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:04.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.920+0000 7f79fe8a3700 1 -- 172.21.15.78:0/2359195935 shutdown_connections 2024-03-24T00:14:04.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.920+0000 7f79fe8a3700 1 -- 172.21.15.78:0/2359195935 wait complete. 2024-03-24T00:14:04.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.921+0000 7f79fe8a3700 1 Processor -- start 2024-03-24T00:14:04.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.921+0000 7f79fe8a3700 1 -- start start 2024-03-24T00:14:04.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.921+0000 7f79fe8a3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79f81021e0 0x7f79f81983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:04.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.921+0000 7f79fe8a3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79f81033e0 0x7f79f8198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:04.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.921+0000 7f79fe8a3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79f8103d80 0x7f79f819f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:04.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.921+0000 7f79fe8a3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f810dde0 con 0x7f79f81033e0 2024-03-24T00:14:04.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.921+0000 7f79fe8a3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f810dc60 con 0x7f79f8103d80 2024-03-24T00:14:04.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.921+0000 7f79fe8a3700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f810df60 con 0x7f79f81021e0 2024-03-24T00:14:04.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.921+0000 7f79f77fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79f81033e0 0x7f79f8198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:04.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.921+0000 7f79f77fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79f81033e0 0x7f79f8198910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50262/0 (socket says 172.21.15.78:50262) 2024-03-24T00:14:04.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.921+0000 7f79f77fe700 1 -- 172.21.15.78:0/2213700120 learned_addr learned my addr 172.21.15.78:0/2213700120 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:14:04.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.921+0000 7f79f7fff700 1 --2- 172.21.15.78:0/2213700120 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79f81021e0 0x7f79f81983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:04.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.922+0000 7f79fce40700 1 --2- 172.21.15.78:0/2213700120 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79f8103d80 0x7f79f819f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:04.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.922+0000 7f79f77fe700 1 -- 172.21.15.78:0/2213700120 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79f81021e0 msgr2=0x7f79f81983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:04.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.922+0000 7f79f77fe700 1 --2- 172.21.15.78:0/2213700120 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79f81021e0 0x7f79f81983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:04.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.922+0000 7f79f77fe700 1 -- 172.21.15.78:0/2213700120 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79f8103d80 msgr2=0x7f79f819f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:04.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.922+0000 7f79f77fe700 1 --2- 172.21.15.78:0/2213700120 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79f8103d80 0x7f79f819f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:04.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.922+0000 7f79f77fe700 1 -- 172.21.15.78:0/2213700120 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f81a0030 con 0x7f79f81033e0 2024-03-24T00:14:04.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.922+0000 7f79f77fe700 1 --2- 172.21.15.78:0/2213700120 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79f81033e0 0x7f79f8198910 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f79ec000c00 tx=0x7f79ec0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:04.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.922+0000 7f79f57fa700 1 -- 172.21.15.78:0/2213700120 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ec027070 con 0x7f79f81033e0 2024-03-24T00:14:04.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.922+0000 7f79f57fa700 1 -- 172.21.15.78:0/2213700120 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f79ec004150 con 0x7f79f81033e0 2024-03-24T00:14:04.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.922+0000 7f79fe8a3700 1 -- 172.21.15.78:0/2213700120 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79f81a02c0 con 0x7f79f81033e0 2024-03-24T00:14:04.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.922+0000 7f79f57fa700 1 -- 172.21.15.78:0/2213700120 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ec00f460 con 0x7f79f81033e0 2024-03-24T00:14:04.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.922+0000 7f79fe8a3700 1 -- 172.21.15.78:0/2213700120 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79f81a0710 con 0x7f79f81033e0 2024-03-24T00:14:04.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.923+0000 7f79fe8a3700 1 -- 172.21.15.78:0/2213700120 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79f8068e50 con 0x7f79f81033e0 2024-03-24T00:14:04.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.924+0000 7f79f57fa700 1 -- 172.21.15.78:0/2213700120 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7f79ec02a070 con 0x7f79f81033e0 2024-03-24T00:14:04.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.925+0000 7f79f57fa700 1 --2- 172.21.15.78:0/2213700120 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f79e00a2800 0x7f79e00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:04.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.925+0000 7f79f7fff700 1 --2- 172.21.15.78:0/2213700120 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f79e00a2800 0x7f79e00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:04.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.925+0000 7f79f57fa700 1 -- 172.21.15.78:0/2213700120 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3226+0+0 (secure 0 0 0) 0x7f79ec0cedc0 con 0x7f79f81033e0 2024-03-24T00:14:04.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.925+0000 7f79f7fff700 1 --2- 172.21.15.78:0/2213700120 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f79e00a2800 0x7f79e00a4cc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f79f8103240 tx=0x7f79e8007400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:04.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:04.929+0000 7f79f57fa700 1 -- 172.21.15.78:0/2213700120 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f79ec031020 con 0x7f79f81033e0 2024-03-24T00:14:05.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:04 smithi078 ceph-mon[32429]: Detected new or changed devices on smithi107 2024-03-24T00:14:05.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:04 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi107 to 17080M 2024-03-24T00:14:05.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:04 smithi078 ceph-mon[32429]: pgmap v80: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:05.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:04 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T00:14:05.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:04 smithi107 ceph-mon[36485]: Detected new or changed devices on smithi107 2024-03-24T00:14:05.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:04 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi107 to 17080M 2024-03-24T00:14:05.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:04 smithi107 ceph-mon[36485]: pgmap v80: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:05.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:04 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T00:14:05.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:05.134+0000 7f79fe8a3700 1 -- 172.21.15.78:0/2213700120 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f79f80625d0 con 0x7f79e00a2800 2024-03-24T00:14:05.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:04 smithi106 ceph-mon[36505]: Detected new or changed devices on smithi107 2024-03-24T00:14:05.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:04 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi107 to 17080M 2024-03-24T00:14:05.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:04 smithi106 ceph-mon[36505]: pgmap v80: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:05.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:04 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T00:14:06.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:05 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T00:14:06.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:05 smithi078 ceph-mon[32429]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T00:14:06.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:05 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T00:14:06.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:05 smithi078 ceph-mon[32429]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:14:06.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:05 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:14:06.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:05 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:14:06.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:05 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:14:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:05 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T00:14:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:05 smithi107 ceph-mon[36485]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T00:14:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:05 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T00:14:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:05 smithi107 ceph-mon[36485]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:14:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:05 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:14:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:05 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:14:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:05 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:14:06.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:05 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T00:14:06.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:05 smithi106 ceph-mon[36505]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T00:14:06.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:05 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T00:14:06.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:05 smithi106 ceph-mon[36505]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:14:06.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:05 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:14:06.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:05 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:14:06.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:05 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:14:06.339 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 24 00:14:06 smithi078 sudo[49321]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T00:14:06.339 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 24 00:14:06 smithi078 sudo[49321]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:14:06.339 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 24 00:14:06 smithi078 sudo[49321]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:14:06.339 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 24 00:14:06 smithi078 sudo[49321]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:14:06.590 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 24 00:14:06 smithi078 sudo[49324]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T00:14:06.591 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 24 00:14:06 smithi078 sudo[49324]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:14:06.591 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 24 00:14:06 smithi078 sudo[49324]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:14:06.591 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 24 00:14:06 smithi078 sudo[49324]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:14:06.864 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 24 00:14:06 smithi106 sudo[45233]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T00:14:06.864 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 24 00:14:06 smithi106 sudo[45233]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:14:06.864 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 24 00:14:06 smithi106 sudo[45233]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:14:06.864 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 24 00:14:06 smithi106 sudo[45233]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:14:06.864 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:06 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T00:14:06.864 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:06 smithi106 ceph-mon[36505]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T00:14:06.864 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:06 smithi106 ceph-mon[36505]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:06.864 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:06 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:06.996 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:06 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T00:14:06.996 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:06 smithi078 ceph-mon[32429]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T00:14:06.996 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:06 smithi078 ceph-mon[32429]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:06.996 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:06 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:06 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T00:14:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:06 smithi107 ceph-mon[36485]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T00:14:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:06 smithi107 ceph-mon[36485]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:06 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:07.152 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 24 00:14:06 smithi106 sudo[45236]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T00:14:07.152 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 24 00:14:06 smithi106 sudo[45236]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:14:07.152 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 24 00:14:06 smithi106 sudo[45236]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:14:07.152 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 24 00:14:06 smithi106 sudo[45236]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:14:07.383 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 24 00:14:07 smithi107 sudo[45463]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T00:14:07.383 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 24 00:14:07 smithi107 sudo[45463]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:14:07.383 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 24 00:14:07 smithi107 sudo[45463]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:14:07.383 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 24 00:14:07 smithi107 sudo[45463]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:14:07.843 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 24 00:14:07 smithi107 sudo[45466]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T00:14:07.843 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 24 00:14:07 smithi107 sudo[45466]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:14:07.843 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 24 00:14:07 smithi107 sudo[45466]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:14:07.843 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 24 00:14:07 smithi107 sudo[45466]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:14:07.912 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:07 smithi078 sudo[49400]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T00:14:07.912 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:07 smithi078 sudo[49400]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:14:07.912 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:07 smithi078 sudo[49400]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:14:07.912 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:07 smithi078 sudo[49400]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:14:07.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:07.927+0000 7f79f57fa700 1 -- 172.21.15.78:0/2213700120 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f79ec099280 con 0x7f79f81033e0 2024-03-24T00:14:08.170 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:07 smithi107 ceph-mon[36485]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T00:14:08.170 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:07 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:08.170 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:07 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:08.170 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:07 smithi107 ceph-mon[36485]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T00:14:08.170 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:08 smithi107 sudo[45469]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T00:14:08.170 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:08 smithi107 sudo[45469]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:14:08.170 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:08 smithi107 sudo[45469]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:14:08.171 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:08 smithi107 sudo[45469]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:14:08.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:07 smithi078 sudo[49403]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T00:14:08.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:07 smithi078 sudo[49403]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:14:08.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:07 smithi078 sudo[49403]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:14:08.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:07 smithi078 sudo[49403]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:14:08.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:07 smithi078 ceph-mon[32429]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T00:14:08.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:07 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:08.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:07 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:08.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:07 smithi078 ceph-mon[32429]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T00:14:08.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:07 smithi106 ceph-mon[36505]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T00:14:08.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:07 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:08.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:07 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:08.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:07 smithi106 ceph-mon[36505]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T00:14:08.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:08 smithi107 sudo[45472]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T00:14:08.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:08 smithi107 sudo[45472]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:14:08.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:08 smithi107 sudo[45472]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:14:08.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:08 smithi107 sudo[45472]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:14:08.895 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:08 smithi106 sudo[45239]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T00:14:08.895 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:08 smithi106 sudo[45239]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:14:08.895 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:08 smithi106 sudo[45239]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:14:08.895 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:08 smithi106 sudo[45239]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:14:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:08 smithi078 ceph-mon[32429]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T00:14:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:08 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:14:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:08 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:14:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:08 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:14:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:08 smithi078 ceph-mon[32429]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T00:14:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:08 smithi078 ceph-mon[32429]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-03-24T00:14:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:08 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:14:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:08 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:14:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:08 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:14:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:08 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2076425875' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "47c718ab-a0f4-4e52-83c5-b89386b262d0"}]: dispatch 2024-03-24T00:14:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:08 smithi078 ceph-mon[32429]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 93 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:08 smithi078 ceph-mon[32429]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T00:14:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:08 smithi078 ceph-mon[32429]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T00:14:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:08 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:14:09.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:08 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:14:09.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:08 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:14:09.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:08 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2076425875' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "47c718ab-a0f4-4e52-83c5-b89386b262d0"}]': finished 2024-03-24T00:14:09.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:08 smithi078 ceph-mon[32429]: osdmap e22: 4 total, 3 up, 4 in 2024-03-24T00:14:09.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:08 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:14:09.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:08 smithi078 ceph-mon[32429]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T00:14:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:08 smithi106 sudo[45242]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T00:14:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:08 smithi106 sudo[45242]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:14:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:08 smithi106 sudo[45242]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:14:09.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:08 smithi106 sudo[45242]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:14:09.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:08 smithi106 ceph-mon[36505]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T00:14:09.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:08 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:14:09.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:08 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:14:09.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:08 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:14:09.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:08 smithi106 ceph-mon[36505]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T00:14:09.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:08 smithi106 ceph-mon[36505]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-03-24T00:14:09.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:08 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:14:09.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:08 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:14:09.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:08 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:14:09.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:08 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2076425875' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "47c718ab-a0f4-4e52-83c5-b89386b262d0"}]: dispatch 2024-03-24T00:14:09.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:08 smithi106 ceph-mon[36505]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 93 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:09.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:08 smithi106 ceph-mon[36505]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T00:14:09.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:08 smithi106 ceph-mon[36505]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T00:14:09.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:08 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:14:09.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:08 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:14:09.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:08 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:14:09.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:08 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2076425875' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "47c718ab-a0f4-4e52-83c5-b89386b262d0"}]': finished 2024-03-24T00:14:09.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:08 smithi106 ceph-mon[36505]: osdmap e22: 4 total, 3 up, 4 in 2024-03-24T00:14:09.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:08 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:14:09.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:08 smithi106 ceph-mon[36505]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T00:14:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:08 smithi107 ceph-mon[36485]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T00:14:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:08 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:14:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:08 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:14:09.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:08 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:14:09.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:08 smithi107 ceph-mon[36485]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T00:14:09.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:08 smithi107 ceph-mon[36485]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-03-24T00:14:09.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:08 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:14:09.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:08 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:14:09.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:08 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:14:09.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:08 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2076425875' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "47c718ab-a0f4-4e52-83c5-b89386b262d0"}]: dispatch 2024-03-24T00:14:09.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:08 smithi107 ceph-mon[36485]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 93 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:09.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:08 smithi107 ceph-mon[36485]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T00:14:09.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:08 smithi107 ceph-mon[36485]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T00:14:09.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:08 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:14:09.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:08 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:14:09.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:08 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:14:09.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:08 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2076425875' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "47c718ab-a0f4-4e52-83c5-b89386b262d0"}]': finished 2024-03-24T00:14:09.345 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:08 smithi107 ceph-mon[36485]: osdmap e22: 4 total, 3 up, 4 in 2024-03-24T00:14:09.345 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:08 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:14:09.345 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:08 smithi107 ceph-mon[36485]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T00:14:10.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:09 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2538727617' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:14:10.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:09 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2538727617' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:14:10.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:09 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2538727617' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:14:11.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:10 smithi078 ceph-mon[32429]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:11.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:10 smithi107 ceph-mon[36485]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:11.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:10 smithi106 ceph-mon[36505]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:13.525 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:13 smithi078 ceph-mon[32429]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:13.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:13 smithi107 ceph-mon[36485]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:13.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:13 smithi106 ceph-mon[36505]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:15.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:15 smithi078 ceph-mon[32429]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:15.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:15 smithi107 ceph-mon[36485]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:15.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:15 smithi106 ceph-mon[36505]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:16.404 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:16 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T00:14:16.404 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:16 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:14:16.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:16 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T00:14:16.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:16 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:14:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:16 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T00:14:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:16 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:14:17.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:17 smithi078 ceph-mon[32429]: Deploying daemon osd.3 on smithi078 2024-03-24T00:14:17.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:17 smithi078 ceph-mon[32429]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:17.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:17 smithi107 ceph-mon[36485]: Deploying daemon osd.3 on smithi078 2024-03-24T00:14:17.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:17 smithi107 ceph-mon[36485]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:17.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:17 smithi106 ceph-mon[36505]: Deploying daemon osd.3 on smithi078 2024-03-24T00:14:17.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:17 smithi106 ceph-mon[36505]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:19.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:19 smithi078 ceph-mon[32429]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:19 smithi107 ceph-mon[36485]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:19 smithi106 ceph-mon[36505]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:21.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:21 smithi078 ceph-mon[32429]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:21.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:21 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:21.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:21 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:21.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:21 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:14:21.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:21 smithi107 ceph-mon[36485]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:21.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:21 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:21.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:21 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:21.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:21 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:14:21.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:21 smithi106 ceph-mon[36505]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:21.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:21 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:21.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:21 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:21.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:21 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:14:23.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:23 smithi078 ceph-mon[32429]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:23.588 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:23 smithi107 ceph-mon[36485]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:23.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:23 smithi106 ceph-mon[36505]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:24.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:24.414+0000 7f79f57fa700 1 -- 172.21.15.78:0/2213700120 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f79f80625d0 con 0x7f79e00a2800 2024-03-24T00:14:24.416 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 3 on host 'smithi078' 2024-03-24T00:14:24.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:24.416+0000 7f79fe8a3700 1 -- 172.21.15.78:0/2213700120 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f79e00a2800 msgr2=0x7f79e00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:24.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:24.416+0000 7f79fe8a3700 1 --2- 172.21.15.78:0/2213700120 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f79e00a2800 0x7f79e00a4cc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f79f8103240 tx=0x7f79e8007400 comp rx=0 tx=0).stop 2024-03-24T00:14:24.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:24.416+0000 7f79fe8a3700 1 -- 172.21.15.78:0/2213700120 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79f81033e0 msgr2=0x7f79f8198910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:24.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:24.416+0000 7f79fe8a3700 1 --2- 172.21.15.78:0/2213700120 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79f81033e0 0x7f79f8198910 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f79ec000c00 tx=0x7f79ec0057c0 comp rx=0 tx=0).stop 2024-03-24T00:14:24.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:24.416+0000 7f79fe8a3700 1 -- 172.21.15.78:0/2213700120 shutdown_connections 2024-03-24T00:14:24.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:24.416+0000 7f79fe8a3700 1 --2- 172.21.15.78:0/2213700120 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f79e00a2800 0x7f79e00a4cc0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:24.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:24.416+0000 7f79fe8a3700 1 --2- 172.21.15.78:0/2213700120 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79f8103d80 0x7f79f819f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:24.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:24.416+0000 7f79fe8a3700 1 --2- 172.21.15.78:0/2213700120 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79f81021e0 0x7f79f81983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:24.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:24.416+0000 7f79fe8a3700 1 --2- 172.21.15.78:0/2213700120 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79f81033e0 0x7f79f8198910 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:24.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:24.417+0000 7f79fe8a3700 1 -- 172.21.15.78:0/2213700120 >> 172.21.15.78:0/2213700120 conn(0x7f79f80fd7a0 msgr2=0x7f79f81066e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:24.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:24.417+0000 7f79fe8a3700 1 -- 172.21.15.78:0/2213700120 shutdown_connections 2024-03-24T00:14:24.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:14:24.417+0000 7f79fe8a3700 1 -- 172.21.15.78:0/2213700120 wait complete. 2024-03-24T00:14:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:24 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:24 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:24 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:14:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:24 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:14:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:24 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:24 smithi106 ceph-mon[36505]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:24.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:24 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:24.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:24 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:24.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:24 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:14:24.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:24 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:24.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:24 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:24.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:24 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:14:24.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:24 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:14:24.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:24 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:24.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:24 smithi078 ceph-mon[32429]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:24.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:24 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:24.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:24 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:24.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:24 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:14:25.028 DEBUG:teuthology.orchestra.run.smithi078:osd.3> sudo journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.3.service 2024-03-24T00:14:25.030 INFO:tasks.cephadm:Deploying osd.4 on smithi106 with /dev/vg_nvme/lv_3... 2024-03-24T00:14:25.030 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T00:14:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:24 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:24 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:24 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:14:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:24 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:14:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:24 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:25.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:24 smithi107 ceph-mon[36485]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:25.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:24 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:25.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:24 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:25.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:24 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:14:25.272 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T00:14:25.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:25 smithi106 ceph-mon[36505]: from='osd.3 [v2:172.21.15.78:6810/1912251346,v1:172.21.15.78:6811/1912251346]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T00:14:25.897 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:25 smithi078 ceph-mon[32429]: from='osd.3 [v2:172.21.15.78:6810/1912251346,v1:172.21.15.78:6811/1912251346]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T00:14:26.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:25 smithi107 ceph-mon[36485]: from='osd.3 [v2:172.21.15.78:6810/1912251346,v1:172.21.15.78:6811/1912251346]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T00:14:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:26 smithi078 ceph-mon[32429]: from='osd.3 [v2:172.21.15.78:6810/1912251346,v1:172.21.15.78:6811/1912251346]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T00:14:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:26 smithi078 ceph-mon[32429]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T00:14:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:26 smithi078 ceph-mon[32429]: from='osd.3 [v2:172.21.15.78:6810/1912251346,v1:172.21.15.78:6811/1912251346]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-24T00:14:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:26 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:14:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:26 smithi078 ceph-mon[32429]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:27.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:26 smithi107 ceph-mon[36485]: from='osd.3 [v2:172.21.15.78:6810/1912251346,v1:172.21.15.78:6811/1912251346]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T00:14:27.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:26 smithi107 ceph-mon[36485]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T00:14:27.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:26 smithi107 ceph-mon[36485]: from='osd.3 [v2:172.21.15.78:6810/1912251346,v1:172.21.15.78:6811/1912251346]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-24T00:14:27.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:26 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:14:27.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:26 smithi107 ceph-mon[36485]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:27.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:26 smithi106 ceph-mon[36505]: from='osd.3 [v2:172.21.15.78:6810/1912251346,v1:172.21.15.78:6811/1912251346]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T00:14:27.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:26 smithi106 ceph-mon[36505]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T00:14:27.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:26 smithi106 ceph-mon[36505]: from='osd.3 [v2:172.21.15.78:6810/1912251346,v1:172.21.15.78:6811/1912251346]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-24T00:14:27.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:26 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:14:27.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:26 smithi106 ceph-mon[36505]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:14:27.230 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T00:14:27.230 INFO:teuthology.orchestra.run.smithi106.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T00:14:27.230 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10+0 records in 2024-03-24T00:14:27.230 INFO:teuthology.orchestra.run.smithi106.stderr:10+0 records out 2024-03-24T00:14:27.230 INFO:teuthology.orchestra.run.smithi106.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114692 s, 914 MB/s 2024-03-24T00:14:27.231 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 2024-03-24T00:14:27.231 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping successful for: 2024-03-24T00:14:27.711 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 24 00:14:27 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-3[51438]: 2024-03-24T00:14:27.392+0000 7f31d5fa3700 -1 osd.3 0 waiting for initial osdmap 2024-03-24T00:14:27.711 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 24 00:14:27 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-3[51438]: 2024-03-24T00:14:27.400+0000 7f31cd58d700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:14:27.822 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:27 smithi106 ceph-mon[36505]: from='osd.3 [v2:172.21.15.78:6810/1912251346,v1:172.21.15.78:6811/1912251346]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-24T00:14:27.822 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:27 smithi106 ceph-mon[36505]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T00:14:27.822 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:27 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:14:27.822 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:27 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:14:27.822 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:27 smithi106 ceph-mon[36505]: from='osd.3 [v2:172.21.15.78:6810/1912251346,v1:172.21.15.78:6811/1912251346]' entity='osd.3' 2024-03-24T00:14:27.822 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:27 smithi106 ceph-mon[36505]: osd.3 [v2:172.21.15.78:6810/1912251346,v1:172.21.15.78:6811/1912251346] boot 2024-03-24T00:14:27.822 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:27 smithi106 ceph-mon[36505]: osdmap e25: 4 total, 4 up, 4 in 2024-03-24T00:14:27.822 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:27 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:14:27.847 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi106:vg_nvme/lv_3 2024-03-24T00:14:28.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:27 smithi078 ceph-mon[32429]: from='osd.3 [v2:172.21.15.78:6810/1912251346,v1:172.21.15.78:6811/1912251346]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-24T00:14:28.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:27 smithi078 ceph-mon[32429]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T00:14:28.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:27 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:14:28.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:27 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:14:28.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:27 smithi078 ceph-mon[32429]: from='osd.3 [v2:172.21.15.78:6810/1912251346,v1:172.21.15.78:6811/1912251346]' entity='osd.3' 2024-03-24T00:14:28.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:27 smithi078 ceph-mon[32429]: osd.3 [v2:172.21.15.78:6810/1912251346,v1:172.21.15.78:6811/1912251346] boot 2024-03-24T00:14:28.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:27 smithi078 ceph-mon[32429]: osdmap e25: 4 total, 4 up, 4 in 2024-03-24T00:14:28.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:27 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:14:28.090 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T00:14:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:27 smithi107 ceph-mon[36485]: from='osd.3 [v2:172.21.15.78:6810/1912251346,v1:172.21.15.78:6811/1912251346]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-24T00:14:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:27 smithi107 ceph-mon[36485]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T00:14:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:27 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:14:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:27 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:14:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:27 smithi107 ceph-mon[36485]: from='osd.3 [v2:172.21.15.78:6810/1912251346,v1:172.21.15.78:6811/1912251346]' entity='osd.3' 2024-03-24T00:14:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:27 smithi107 ceph-mon[36485]: osd.3 [v2:172.21.15.78:6810/1912251346,v1:172.21.15.78:6811/1912251346] boot 2024-03-24T00:14:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:27 smithi107 ceph-mon[36485]: osdmap e25: 4 total, 4 up, 4 in 2024-03-24T00:14:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:27 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:14:28.769 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:28 smithi106 ceph-mon[36505]: purged_snaps scrub starts 2024-03-24T00:14:28.770 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:28 smithi106 ceph-mon[36505]: purged_snaps scrub ok 2024-03-24T00:14:28.770 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:28 smithi106 ceph-mon[36505]: Detected new or changed devices on smithi078 2024-03-24T00:14:28.770 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:28 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:28.770 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:28 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:28.770 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:28 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:14:28.770 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:28 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:14:28.770 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:28 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi078 to 8544M 2024-03-24T00:14:28.770 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:28 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:28.770 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:28 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:14:28.770 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:28 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:14:28.770 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:28 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:28.770 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:28 smithi106 ceph-mon[36505]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:14:28.770 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:28 smithi106 ceph-mon[36505]: osdmap e26: 4 total, 4 up, 4 in 2024-03-24T00:14:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:28 smithi078 ceph-mon[32429]: purged_snaps scrub starts 2024-03-24T00:14:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:28 smithi078 ceph-mon[32429]: purged_snaps scrub ok 2024-03-24T00:14:29.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:28 smithi078 ceph-mon[32429]: Detected new or changed devices on smithi078 2024-03-24T00:14:29.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:28 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:29.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:28 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:29.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:28 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:14:29.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:28 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:14:29.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:28 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi078 to 8544M 2024-03-24T00:14:29.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:28 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:29.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:28 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:14:29.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:28 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:14:29.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:28 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:29.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:28 smithi078 ceph-mon[32429]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:14:29.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:28 smithi078 ceph-mon[32429]: osdmap e26: 4 total, 4 up, 4 in 2024-03-24T00:14:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:28 smithi107 ceph-mon[36485]: purged_snaps scrub starts 2024-03-24T00:14:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:28 smithi107 ceph-mon[36485]: purged_snaps scrub ok 2024-03-24T00:14:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:28 smithi107 ceph-mon[36485]: Detected new or changed devices on smithi078 2024-03-24T00:14:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:28 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:28 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:28 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:14:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:28 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:14:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:28 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi078 to 8544M 2024-03-24T00:14:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:28 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:28 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:14:29.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:28 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:14:29.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:28 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:29.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:28 smithi107 ceph-mon[36485]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:14:29.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:28 smithi107 ceph-mon[36485]: osdmap e26: 4 total, 4 up, 4 in 2024-03-24T00:14:29.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.576+0000 7f4358dd3700 1 -- 172.21.15.106:0/4106167127 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4354103460 msgr2=0x7f43541038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:29.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.576+0000 7f4358dd3700 1 --2- 172.21.15.106:0/4106167127 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4354103460 0x7f43541038c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4348009990 tx=0x7f4348009ca0 comp rx=0 tx=0).stop 2024-03-24T00:14:29.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.576+0000 7f4358dd3700 1 -- 172.21.15.106:0/4106167127 shutdown_connections 2024-03-24T00:14:29.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.576+0000 7f4358dd3700 1 --2- 172.21.15.106:0/4106167127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4354103e00 0x7f435410a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:29.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.576+0000 7f4358dd3700 1 --2- 172.21.15.106:0/4106167127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4354100480 0x7f43541008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:29.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.576+0000 7f4358dd3700 1 --2- 172.21.15.106:0/4106167127 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4354103460 0x7f43541038c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:29.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.576+0000 7f4358dd3700 1 -- 172.21.15.106:0/4106167127 >> 172.21.15.106:0/4106167127 conn(0x7f43540fbe00 msgr2=0x7f43540fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:29.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.576+0000 7f4358dd3700 1 -- 172.21.15.106:0/4106167127 shutdown_connections 2024-03-24T00:14:29.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.576+0000 7f4358dd3700 1 -- 172.21.15.106:0/4106167127 wait complete. 2024-03-24T00:14:29.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.577+0000 7f4358dd3700 1 Processor -- start 2024-03-24T00:14:29.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.577+0000 7f4358dd3700 1 -- start start 2024-03-24T00:14:29.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.577+0000 7f4358dd3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4354100480 0x7f4354198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:29.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.577+0000 7f4358dd3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4354103460 0x7f4354198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:29.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.577+0000 7f4358dd3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4354103e00 0x7f435419f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:29.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.577+0000 7f4358dd3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435410dd80 con 0x7f4354103460 2024-03-24T00:14:29.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.577+0000 7f4358dd3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435410dc00 con 0x7f4354103e00 2024-03-24T00:14:29.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.577+0000 7f4358dd3700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435410df00 con 0x7f4354100480 2024-03-24T00:14:29.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.577+0000 7f4352d9d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4354103e00 0x7f435419f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:29.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.577+0000 7f4352d9d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4354103e00 0x7f435419f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:51746/0 (socket says 172.21.15.106:51746) 2024-03-24T00:14:29.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.577+0000 7f4352d9d700 1 -- 172.21.15.106:0/1967578334 learned_addr learned my addr 172.21.15.106:0/1967578334 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-24T00:14:29.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.577+0000 7f4351d9b700 1 --2- 172.21.15.106:0/1967578334 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4354103460 0x7f4354198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:29.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.577+0000 7f435259c700 1 --2- 172.21.15.106:0/1967578334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4354100480 0x7f4354198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:29.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.577+0000 7f4352d9d700 1 -- 172.21.15.106:0/1967578334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4354100480 msgr2=0x7f4354198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:29.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.577+0000 7f4352d9d700 1 --2- 172.21.15.106:0/1967578334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4354100480 0x7f4354198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:29.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.577+0000 7f4352d9d700 1 -- 172.21.15.106:0/1967578334 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4354103460 msgr2=0x7f4354198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:29.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.577+0000 7f4352d9d700 1 --2- 172.21.15.106:0/1967578334 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4354103460 0x7f4354198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:29.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.577+0000 7f4352d9d700 1 -- 172.21.15.106:0/1967578334 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435419ff90 con 0x7f4354103e00 2024-03-24T00:14:29.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.578+0000 7f4352d9d700 1 --2- 172.21.15.106:0/1967578334 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4354103e00 0x7f435419f8f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f434c00c900 tx=0x7f434c00cc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:29.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.578+0000 7f43437fe700 1 -- 172.21.15.106:0/1967578334 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434c0078d0 con 0x7f4354103e00 2024-03-24T00:14:29.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.578+0000 7f43437fe700 1 -- 172.21.15.106:0/1967578334 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f434c00f450 con 0x7f4354103e00 2024-03-24T00:14:29.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.578+0000 7f4358dd3700 1 -- 172.21.15.106:0/1967578334 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43541a0280 con 0x7f4354103e00 2024-03-24T00:14:29.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.578+0000 7f43437fe700 1 -- 172.21.15.106:0/1967578334 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434c018660 con 0x7f4354103e00 2024-03-24T00:14:29.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.578+0000 7f4358dd3700 1 -- 172.21.15.106:0/1967578334 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43541a07c0 con 0x7f4354103e00 2024-03-24T00:14:29.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.579+0000 7f4358dd3700 1 -- 172.21.15.106:0/1967578334 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4354068e50 con 0x7f4354103e00 2024-03-24T00:14:29.582 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.580+0000 7f43437fe700 1 -- 172.21.15.106:0/1967578334 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f434c00f600 con 0x7f4354103e00 2024-03-24T00:14:29.582 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.580+0000 7f43437fe700 1 --2- 172.21.15.106:0/1967578334 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f433c0a2910 0x7f433c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:29.582 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.580+0000 7f43437fe700 1 -- 172.21.15.106:0/1967578334 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 3698+0+0 (secure 0 0 0) 0x7f434c0cd580 con 0x7f4354103e00 2024-03-24T00:14:29.582 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.581+0000 7f435259c700 1 --2- 172.21.15.106:0/1967578334 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f433c0a2910 0x7f433c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:29.583 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.581+0000 7f435259c700 1 --2- 172.21.15.106:0/1967578334 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f433c0a2910 0x7f433c0a4dd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4354069c40 tx=0x7f43440091c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:29.585 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.585+0000 7f43437fe700 1 -- 172.21.15.106:0/1967578334 <== mon.2 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f434c014030 con 0x7f4354103e00 2024-03-24T00:14:29.793 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:29.792+0000 7f4358dd3700 1 -- 172.21.15.106:0/1967578334 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f43540625d0 con 0x7f433c0a2910 2024-03-24T00:14:30.648 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:30 smithi106 ceph-mon[36505]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T00:14:30.648 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:30 smithi106 ceph-mon[36505]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:14:30.648 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:30 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:14:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:30 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:14:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:30 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:14:30.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:30 smithi106 ceph-mon[36505]: pgmap v102: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 240 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T00:14:31.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:30 smithi078 ceph-mon[32429]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T00:14:31.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:30 smithi078 ceph-mon[32429]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:14:31.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:30 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:14:31.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:30 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:14:31.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:30 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:14:31.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:30 smithi078 ceph-mon[32429]: pgmap v102: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 240 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T00:14:31.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:30 smithi107 ceph-mon[36485]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T00:14:31.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:30 smithi107 ceph-mon[36485]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:14:31.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:30 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:14:31.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:30 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:14:31.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:30 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:14:31.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:30 smithi107 ceph-mon[36485]: pgmap v102: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 240 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T00:14:32.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:31 smithi078 ceph-mon[32429]: osdmap e28: 4 total, 4 up, 4 in 2024-03-24T00:14:32.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:31 smithi107 ceph-mon[36485]: osdmap e28: 4 total, 4 up, 4 in 2024-03-24T00:14:32.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:31 smithi106 ceph-mon[36505]: osdmap e28: 4 total, 4 up, 4 in 2024-03-24T00:14:33.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:32 smithi078 ceph-mon[32429]: osdmap e29: 4 total, 4 up, 4 in 2024-03-24T00:14:33.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:32 smithi078 ceph-mon[32429]: pgmap v105: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 240 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T00:14:33.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:32 smithi078 ceph-mon[32429]: from='client.? 172.21.15.106:0/303937669' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7dad64dd-677e-4c1c-9638-e4f4f2195983"}]: dispatch 2024-03-24T00:14:33.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:32 smithi078 ceph-mon[32429]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7dad64dd-677e-4c1c-9638-e4f4f2195983"}]: dispatch 2024-03-24T00:14:33.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:32 smithi078 ceph-mon[32429]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7dad64dd-677e-4c1c-9638-e4f4f2195983"}]': finished 2024-03-24T00:14:33.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:32 smithi078 ceph-mon[32429]: osdmap e30: 5 total, 4 up, 5 in 2024-03-24T00:14:33.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:32 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:14:33.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:32 smithi107 ceph-mon[36485]: osdmap e29: 4 total, 4 up, 4 in 2024-03-24T00:14:33.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:32 smithi107 ceph-mon[36485]: pgmap v105: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 240 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T00:14:33.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:32 smithi107 ceph-mon[36485]: from='client.? 172.21.15.106:0/303937669' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7dad64dd-677e-4c1c-9638-e4f4f2195983"}]: dispatch 2024-03-24T00:14:33.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:32 smithi107 ceph-mon[36485]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7dad64dd-677e-4c1c-9638-e4f4f2195983"}]: dispatch 2024-03-24T00:14:33.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:32 smithi107 ceph-mon[36485]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7dad64dd-677e-4c1c-9638-e4f4f2195983"}]': finished 2024-03-24T00:14:33.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:32 smithi107 ceph-mon[36485]: osdmap e30: 5 total, 4 up, 5 in 2024-03-24T00:14:33.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:32 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:14:33.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:32 smithi106 ceph-mon[36505]: osdmap e29: 4 total, 4 up, 4 in 2024-03-24T00:14:33.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:32 smithi106 ceph-mon[36505]: pgmap v105: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 240 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T00:14:33.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:32 smithi106 ceph-mon[36505]: from='client.? 172.21.15.106:0/303937669' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7dad64dd-677e-4c1c-9638-e4f4f2195983"}]: dispatch 2024-03-24T00:14:33.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:32 smithi106 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7dad64dd-677e-4c1c-9638-e4f4f2195983"}]: dispatch 2024-03-24T00:14:33.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:32 smithi106 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7dad64dd-677e-4c1c-9638-e4f4f2195983"}]': finished 2024-03-24T00:14:33.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:32 smithi106 ceph-mon[36505]: osdmap e30: 5 total, 4 up, 5 in 2024-03-24T00:14:33.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:32 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:14:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:33 smithi078 ceph-mon[32429]: from='client.? 172.21.15.106:0/152929243' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:14:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:33 smithi107 ceph-mon[36485]: from='client.? 172.21.15.106:0/152929243' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:14:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:33 smithi106 ceph-mon[36505]: from='client.? 172.21.15.106:0/152929243' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:14:35.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:34 smithi078 ceph-mon[32429]: pgmap v107: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-03-24T00:14:35.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:34 smithi107 ceph-mon[36485]: pgmap v107: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-03-24T00:14:35.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:34 smithi106 ceph-mon[36505]: pgmap v107: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-03-24T00:14:37.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:37 smithi078 ceph-mon[32429]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:14:37.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:37 smithi107 ceph-mon[36485]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:14:37.608 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:37 smithi106 ceph-mon[36505]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:14:39.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:39 smithi106 ceph-mon[36505]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:14:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:39 smithi078 ceph-mon[32429]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:14:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:39 smithi107 ceph-mon[36485]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:14:40.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T00:14:40.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:14:40.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T00:14:40.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:14:40.623 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T00:14:40.623 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:14:41.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:41 smithi078 ceph-mon[32429]: Deploying daemon osd.4 on smithi106 2024-03-24T00:14:41.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:41 smithi078 ceph-mon[32429]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:14:41.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:41 smithi107 ceph-mon[36485]: Deploying daemon osd.4 on smithi106 2024-03-24T00:14:41.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:41 smithi107 ceph-mon[36485]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:14:41.613 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:41 smithi106 ceph-mon[36505]: Deploying daemon osd.4 on smithi106 2024-03-24T00:14:41.613 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:41 smithi106 ceph-mon[36505]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:14:43.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:43 smithi106 ceph-mon[36505]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:14:43.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:43 smithi078 ceph-mon[32429]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:14:43.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:43 smithi107 ceph-mon[36485]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:14:45.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:45 smithi078 ceph-mon[32429]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:14:45.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:45 smithi107 ceph-mon[36485]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:14:45.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:45 smithi106 ceph-mon[36505]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:14:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:46 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:46.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:46 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:46.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:46 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:14:46.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:46 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:46.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:46 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:46.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:46 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:14:46.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:46 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:46.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:46 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:46.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:46 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:14:47.450 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:47 smithi106 ceph-mon[36505]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:14:47.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:47 smithi078 ceph-mon[32429]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:14:47.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:47 smithi107 ceph-mon[36485]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:14:48.756 INFO:teuthology.orchestra.run.smithi106.stdout:Created osd(s) 4 on host 'smithi106' 2024-03-24T00:14:48.757 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:48.756+0000 7f43437fe700 1 -- 172.21.15.106:0/1967578334 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f43540625d0 con 0x7f433c0a2910 2024-03-24T00:14:48.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:48.759+0000 7f4358dd3700 1 -- 172.21.15.106:0/1967578334 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f433c0a2910 msgr2=0x7f433c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:48.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:48.759+0000 7f4358dd3700 1 --2- 172.21.15.106:0/1967578334 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f433c0a2910 0x7f433c0a4dd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4354069c40 tx=0x7f43440091c0 comp rx=0 tx=0).stop 2024-03-24T00:14:48.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:48.759+0000 7f4358dd3700 1 -- 172.21.15.106:0/1967578334 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4354103e00 msgr2=0x7f435419f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:48.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:48.759+0000 7f4358dd3700 1 --2- 172.21.15.106:0/1967578334 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4354103e00 0x7f435419f8f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f434c00c900 tx=0x7f434c00cc10 comp rx=0 tx=0).stop 2024-03-24T00:14:48.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:48.759+0000 7f4358dd3700 1 -- 172.21.15.106:0/1967578334 shutdown_connections 2024-03-24T00:14:48.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:48.759+0000 7f4358dd3700 1 --2- 172.21.15.106:0/1967578334 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f433c0a2910 0x7f433c0a4dd0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:48.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:48.759+0000 7f4358dd3700 1 --2- 172.21.15.106:0/1967578334 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4354103e00 0x7f435419f8f0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:48.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:48.759+0000 7f4358dd3700 1 --2- 172.21.15.106:0/1967578334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4354100480 0x7f4354198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:48.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:48.759+0000 7f4358dd3700 1 --2- 172.21.15.106:0/1967578334 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4354103460 0x7f4354198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:48.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:48.759+0000 7f4358dd3700 1 -- 172.21.15.106:0/1967578334 >> 172.21.15.106:0/1967578334 conn(0x7f43540fbe00 msgr2=0x7f4354106700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:48.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:48.759+0000 7f4358dd3700 1 -- 172.21.15.106:0/1967578334 shutdown_connections 2024-03-24T00:14:48.760 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:14:48.759+0000 7f4358dd3700 1 -- 172.21.15.106:0/1967578334 wait complete. 2024-03-24T00:14:49.312 DEBUG:teuthology.orchestra.run.smithi106:osd.4> sudo journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.4.service 2024-03-24T00:14:49.314 INFO:tasks.cephadm:Deploying osd.5 on smithi107 with /dev/vg_nvme/lv_3... 2024-03-24T00:14:49.314 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T00:14:49.561 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:14:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:49 smithi106 ceph-mon[36505]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:14:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:49 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:49 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:49 smithi106 ceph-mon[36505]: from='osd.4 [v2:172.21.15.106:6808/374641205,v1:172.21.15.106:6809/374641205]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T00:14:49.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:49 smithi106 ceph-mon[36505]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T00:14:49.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:49 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:49.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:49 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:49.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:49 smithi107 ceph-mon[36485]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:14:49.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:49 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:49.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:49 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:49.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:49 smithi107 ceph-mon[36485]: from='osd.4 [v2:172.21.15.106:6808/374641205,v1:172.21.15.106:6809/374641205]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T00:14:49.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:49 smithi107 ceph-mon[36485]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T00:14:49.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:49 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:49.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:49 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:49.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:49 smithi078 ceph-mon[32429]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:14:49.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:49 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:49.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:49 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:49.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:49 smithi078 ceph-mon[32429]: from='osd.4 [v2:172.21.15.106:6808/374641205,v1:172.21.15.106:6809/374641205]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T00:14:49.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:49 smithi078 ceph-mon[32429]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T00:14:49.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:49 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:49.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:49 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:50.890 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:50 smithi107 ceph-mon[36485]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T00:14:50.890 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:50 smithi107 ceph-mon[36485]: osdmap e31: 5 total, 4 up, 5 in 2024-03-24T00:14:50.890 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:50 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:14:50.890 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:50 smithi107 ceph-mon[36485]: from='osd.4 [v2:172.21.15.106:6808/374641205,v1:172.21.15.106:6809/374641205]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-24T00:14:50.890 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:50 smithi107 ceph-mon[36485]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-24T00:14:50.890 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:50 smithi107 ceph-mon[36485]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:14:50.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:50 smithi106 ceph-mon[36505]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T00:14:50.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:50 smithi106 ceph-mon[36505]: osdmap e31: 5 total, 4 up, 5 in 2024-03-24T00:14:50.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:50 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:14:50.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:50 smithi106 ceph-mon[36505]: from='osd.4 [v2:172.21.15.106:6808/374641205,v1:172.21.15.106:6809/374641205]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-24T00:14:50.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:50 smithi106 ceph-mon[36505]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-24T00:14:50.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:50 smithi106 ceph-mon[36505]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:14:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:50 smithi078 ceph-mon[32429]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T00:14:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:50 smithi078 ceph-mon[32429]: osdmap e31: 5 total, 4 up, 5 in 2024-03-24T00:14:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:50 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:14:51.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:50 smithi078 ceph-mon[32429]: from='osd.4 [v2:172.21.15.106:6808/374641205,v1:172.21.15.106:6809/374641205]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-24T00:14:51.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:50 smithi078 ceph-mon[32429]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-24T00:14:51.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:50 smithi078 ceph-mon[32429]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:14:51.448 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T00:14:51.448 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T00:14:51.448 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10+0 records in 2024-03-24T00:14:51.448 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2024-03-24T00:14:51.448 INFO:teuthology.orchestra.run.smithi107.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012331 s, 850 MB/s 2024-03-24T00:14:51.448 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping successful for: 2024-03-24T00:14:51.653 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 24 00:14:51 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-4[48051]: 2024-03-24T00:14:51.480+0000 7fc697608700 -1 osd.4 0 waiting for initial osdmap 2024-03-24T00:14:51.653 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 24 00:14:51 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-4[48051]: 2024-03-24T00:14:51.486+0000 7fc68f3f3700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:14:52.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:51 smithi078 ceph-mon[32429]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-24T00:14:52.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:51 smithi078 ceph-mon[32429]: osdmap e32: 5 total, 4 up, 5 in 2024-03-24T00:14:52.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:51 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:14:52.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:51 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:14:52.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:51 smithi078 ceph-mon[32429]: from='osd.4 ' entity='osd.4' 2024-03-24T00:14:52.092 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:51 smithi106 ceph-mon[36505]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-24T00:14:52.093 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:51 smithi106 ceph-mon[36505]: osdmap e32: 5 total, 4 up, 5 in 2024-03-24T00:14:52.093 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:51 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:14:52.093 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:51 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:14:52.093 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:51 smithi106 ceph-mon[36505]: from='osd.4 ' entity='osd.4' 2024-03-24T00:14:52.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:51 smithi107 ceph-mon[36485]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-24T00:14:52.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:51 smithi107 ceph-mon[36485]: osdmap e32: 5 total, 4 up, 5 in 2024-03-24T00:14:52.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:51 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:14:52.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:51 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:14:52.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:51 smithi107 ceph-mon[36485]: from='osd.4 ' entity='osd.4' 2024-03-24T00:14:52.144 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi107:vg_nvme/lv_3 2024-03-24T00:14:52.392 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:14:52.795 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:52 smithi107 ceph-mon[36485]: purged_snaps scrub starts 2024-03-24T00:14:52.795 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:52 smithi107 ceph-mon[36485]: purged_snaps scrub ok 2024-03-24T00:14:52.795 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:52 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:14:52.795 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:52 smithi107 ceph-mon[36485]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:14:52.795 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:52 smithi107 ceph-mon[36485]: osd.4 [v2:172.21.15.106:6808/374641205,v1:172.21.15.106:6809/374641205] boot 2024-03-24T00:14:52.795 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:52 smithi107 ceph-mon[36485]: osdmap e33: 5 total, 5 up, 5 in 2024-03-24T00:14:52.795 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:52 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:14:53.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:52 smithi078 ceph-mon[32429]: purged_snaps scrub starts 2024-03-24T00:14:53.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:52 smithi078 ceph-mon[32429]: purged_snaps scrub ok 2024-03-24T00:14:53.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:52 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:14:53.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:52 smithi078 ceph-mon[32429]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:14:53.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:52 smithi078 ceph-mon[32429]: osd.4 [v2:172.21.15.106:6808/374641205,v1:172.21.15.106:6809/374641205] boot 2024-03-24T00:14:53.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:52 smithi078 ceph-mon[32429]: osdmap e33: 5 total, 5 up, 5 in 2024-03-24T00:14:53.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:52 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:14:53.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:52 smithi106 ceph-mon[36505]: purged_snaps scrub starts 2024-03-24T00:14:53.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:52 smithi106 ceph-mon[36505]: purged_snaps scrub ok 2024-03-24T00:14:53.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:52 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:14:53.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:52 smithi106 ceph-mon[36505]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-24T00:14:53.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:52 smithi106 ceph-mon[36505]: osd.4 [v2:172.21.15.106:6808/374641205,v1:172.21.15.106:6809/374641205] boot 2024-03-24T00:14:53.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:52 smithi106 ceph-mon[36505]: osdmap e33: 5 total, 5 up, 5 in 2024-03-24T00:14:53.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:52 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:14:53.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.912+0000 7fe4cc270700 1 -- 172.21.15.107:0/2981754531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c41033e0 msgr2=0x7fe4c4103840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:53.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.912+0000 7fe4cc270700 1 --2- 172.21.15.107:0/2981754531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c41033e0 0x7fe4c4103840 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe4b4009990 tx=0x7fe4b4009ca0 comp rx=0 tx=0).stop 2024-03-24T00:14:53.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.912+0000 7fe4cc270700 1 -- 172.21.15.107:0/2981754531 shutdown_connections 2024-03-24T00:14:53.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.912+0000 7fe4cc270700 1 --2- 172.21.15.107:0/2981754531 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4c4103d80 0x7fe4c410a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:53.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.912+0000 7fe4cc270700 1 --2- 172.21.15.107:0/2981754531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4c41021e0 0x7fe4c4102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:53.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.912+0000 7fe4cc270700 1 --2- 172.21.15.107:0/2981754531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c41033e0 0x7fe4c4103840 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:53.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.912+0000 7fe4cc270700 1 -- 172.21.15.107:0/2981754531 >> 172.21.15.107:0/2981754531 conn(0x7fe4c40fd740 msgr2=0x7fe4c40ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:53.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.912+0000 7fe4cc270700 1 -- 172.21.15.107:0/2981754531 shutdown_connections 2024-03-24T00:14:53.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.912+0000 7fe4cc270700 1 -- 172.21.15.107:0/2981754531 wait complete. 2024-03-24T00:14:53.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.913+0000 7fe4cc270700 1 Processor -- start 2024-03-24T00:14:53.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.913+0000 7fe4cc270700 1 -- start start 2024-03-24T00:14:53.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.913+0000 7fe4cc270700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4c41021e0 0x7fe4c41983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:53.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.913+0000 7fe4cc270700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c41033e0 0x7fe4c4198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:53.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.913+0000 7fe4cc270700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4c4103d80 0x7fe4c419f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:53.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.913+0000 7fe4cc270700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c410dda0 con 0x7fe4c41021e0 2024-03-24T00:14:53.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.913+0000 7fe4cc270700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c410dc20 con 0x7fe4c4103d80 2024-03-24T00:14:53.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.913+0000 7fe4cc270700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c410df20 con 0x7fe4c41033e0 2024-03-24T00:14:53.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.913+0000 7fe4c980b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c41033e0 0x7fe4c4198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:53.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.913+0000 7fe4c980b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c41033e0 0x7fe4c4198900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55948/0 (socket says 172.21.15.107:55948) 2024-03-24T00:14:53.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.913+0000 7fe4c980b700 1 -- 172.21.15.107:0/1270332245 learned_addr learned my addr 172.21.15.107:0/1270332245 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-03-24T00:14:53.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.913+0000 7fe4ca80d700 1 --2- 172.21.15.107:0/1270332245 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4c4103d80 0x7fe4c419f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:53.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.913+0000 7fe4ca00c700 1 --2- 172.21.15.107:0/1270332245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4c41021e0 0x7fe4c41983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:53.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.913+0000 7fe4c980b700 1 -- 172.21.15.107:0/1270332245 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4c4103d80 msgr2=0x7fe4c419f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:53.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.913+0000 7fe4c980b700 1 --2- 172.21.15.107:0/1270332245 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4c4103d80 0x7fe4c419f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:53.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.913+0000 7fe4c980b700 1 -- 172.21.15.107:0/1270332245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4c41021e0 msgr2=0x7fe4c41983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:53.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.913+0000 7fe4c980b700 1 --2- 172.21.15.107:0/1270332245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4c41021e0 0x7fe4c41983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:53.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.913+0000 7fe4c980b700 1 -- 172.21.15.107:0/1270332245 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c41a0020 con 0x7fe4c41033e0 2024-03-24T00:14:53.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.914+0000 7fe4c980b700 1 --2- 172.21.15.107:0/1270332245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c41033e0 0x7fe4c4198900 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe4b4004790 tx=0x7fe4b4009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:53.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.914+0000 7fe4bb7fe700 1 -- 172.21.15.107:0/1270332245 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b4027070 con 0x7fe4c41033e0 2024-03-24T00:14:53.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.914+0000 7fe4bb7fe700 1 -- 172.21.15.107:0/1270332245 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe4b4004150 con 0x7fe4c41033e0 2024-03-24T00:14:53.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.914+0000 7fe4cc270700 1 -- 172.21.15.107:0/1270332245 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4c41a02b0 con 0x7fe4c41033e0 2024-03-24T00:14:53.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.914+0000 7fe4bb7fe700 1 -- 172.21.15.107:0/1270332245 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b400f460 con 0x7fe4c41033e0 2024-03-24T00:14:53.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.914+0000 7fe4cc270700 1 -- 172.21.15.107:0/1270332245 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4c41a0700 con 0x7fe4c41033e0 2024-03-24T00:14:53.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.915+0000 7fe4cc270700 1 -- 172.21.15.107:0/1270332245 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4c4068e50 con 0x7fe4c41033e0 2024-03-24T00:14:53.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.916+0000 7fe4bb7fe700 1 -- 172.21.15.107:0/1270332245 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe4b402a070 con 0x7fe4c41033e0 2024-03-24T00:14:53.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.916+0000 7fe4bb7fe700 1 --2- 172.21.15.107:0/1270332245 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe4b00a2890 0x7fe4b00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:53.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.916+0000 7fe4bb7fe700 1 -- 172.21.15.107:0/1270332245 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 4101+0+0 (secure 0 0 0) 0x7fe4b40cf050 con 0x7fe4c41033e0 2024-03-24T00:14:53.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.917+0000 7fe4ca00c700 1 --2- 172.21.15.107:0/1270332245 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe4b00a2890 0x7fe4b00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:53.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.917+0000 7fe4ca00c700 1 --2- 172.21.15.107:0/1270332245 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe4b00a2890 0x7fe4b00a4d50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0005950 tx=0x7fe4c00058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:53.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:53.920+0000 7fe4bb7fe700 1 -- 172.21.15.107:0/1270332245 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe4b4031070 con 0x7fe4c41033e0 2024-03-24T00:14:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:53 smithi078 ceph-mon[32429]: Detected new or changed devices on smithi106 2024-03-24T00:14:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:53 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:53 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:54.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:53 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:14:54.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:53 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:14:54.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:53 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi106 to 8540M 2024-03-24T00:14:54.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:53 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:54.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:53 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:14:54.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:53 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:14:54.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:53 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:54.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:53 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:14:54.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:53 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:14:54.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:53 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:14:54.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:53 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:53 smithi107 ceph-mon[36485]: Detected new or changed devices on smithi106 2024-03-24T00:14:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:53 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:53 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:53 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:14:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:53 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:14:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:53 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi106 to 8540M 2024-03-24T00:14:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:53 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:53 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:14:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:53 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:14:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:53 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:54.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:53 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:14:54.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:53 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:14:54.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:53 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:14:54.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:53 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:54.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:14:54.127+0000 7fe4cc270700 1 -- 172.21.15.107:0/1270332245 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fe4c40625d0 con 0x7fe4b00a2890 2024-03-24T00:14:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:53 smithi106 ceph-mon[36505]: Detected new or changed devices on smithi106 2024-03-24T00:14:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:53 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:53 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:53 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:14:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:53 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:14:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:53 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi106 to 8540M 2024-03-24T00:14:54.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:53 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:54.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:53 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:14:54.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:53 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:14:54.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:53 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:54.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:53 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:14:54.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:53 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:14:54.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:53 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:14:54.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:53 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:14:55.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:55 smithi078 ceph-mon[32429]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T00:14:55.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:55 smithi078 ceph-mon[32429]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:14:55.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:55 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:14:55.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:55 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:14:55.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:55 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:14:55.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:55 smithi078 ceph-mon[32429]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:14:55.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:55 smithi107 ceph-mon[36485]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T00:14:55.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:55 smithi107 ceph-mon[36485]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:14:55.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:55 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:14:55.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:55 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:14:55.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:55 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:14:55.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:55 smithi107 ceph-mon[36485]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:14:55.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:55 smithi106 ceph-mon[36505]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T00:14:55.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:55 smithi106 ceph-mon[36505]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:14:55.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:55 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:14:55.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:55 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:14:55.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:55 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:14:55.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:55 smithi106 ceph-mon[36505]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:14:57.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:57 smithi107 ceph-mon[36485]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:14:57.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:57 smithi107 ceph-mon[36485]: from='client.? 172.21.15.107:0/3802813592' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bbd524e5-328d-4309-8877-1149cbe110dd"}]: dispatch 2024-03-24T00:14:57.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:57 smithi107 ceph-mon[36485]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bbd524e5-328d-4309-8877-1149cbe110dd"}]: dispatch 2024-03-24T00:14:57.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:57 smithi107 ceph-mon[36485]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bbd524e5-328d-4309-8877-1149cbe110dd"}]': finished 2024-03-24T00:14:57.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:57 smithi107 ceph-mon[36485]: osdmap e35: 6 total, 5 up, 6 in 2024-03-24T00:14:57.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:57 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:14:57.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:57 smithi078 ceph-mon[32429]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:14:57.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:57 smithi078 ceph-mon[32429]: from='client.? 172.21.15.107:0/3802813592' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bbd524e5-328d-4309-8877-1149cbe110dd"}]: dispatch 2024-03-24T00:14:57.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:57 smithi078 ceph-mon[32429]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bbd524e5-328d-4309-8877-1149cbe110dd"}]: dispatch 2024-03-24T00:14:57.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:57 smithi078 ceph-mon[32429]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bbd524e5-328d-4309-8877-1149cbe110dd"}]': finished 2024-03-24T00:14:57.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:57 smithi078 ceph-mon[32429]: osdmap e35: 6 total, 5 up, 6 in 2024-03-24T00:14:57.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:57 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:14:57.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:57 smithi106 ceph-mon[36505]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:14:57.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:57 smithi106 ceph-mon[36505]: from='client.? 172.21.15.107:0/3802813592' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bbd524e5-328d-4309-8877-1149cbe110dd"}]: dispatch 2024-03-24T00:14:57.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:57 smithi106 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bbd524e5-328d-4309-8877-1149cbe110dd"}]: dispatch 2024-03-24T00:14:57.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:57 smithi106 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bbd524e5-328d-4309-8877-1149cbe110dd"}]': finished 2024-03-24T00:14:57.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:57 smithi106 ceph-mon[36505]: osdmap e35: 6 total, 5 up, 6 in 2024-03-24T00:14:57.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:57 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:14:58.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:58 smithi078 ceph-mon[32429]: from='client.? 172.21.15.107:0/1809644259' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:14:58.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:58 smithi107 ceph-mon[36485]: from='client.? 172.21.15.107:0/1809644259' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:14:58.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:58 smithi106 ceph-mon[36505]: from='client.? 172.21.15.107:0/1809644259' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:14:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:14:59 smithi078 ceph-mon[32429]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:14:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:14:59 smithi107 ceph-mon[36485]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:14:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:14:59 smithi106 ceph-mon[36505]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:15:01.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:01 smithi078 ceph-mon[32429]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:15:01.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:01 smithi107 ceph-mon[36485]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:15:01.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:01 smithi106 ceph-mon[36505]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:15:03.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:03 smithi078 ceph-mon[32429]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:15:03.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:03 smithi107 ceph-mon[36485]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:15:03.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:03 smithi106 ceph-mon[36505]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:15:04.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:04 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T00:15:04.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:04 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:15:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:04 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T00:15:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:04 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:15:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:04 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T00:15:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:04 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:15:05.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:05 smithi078 ceph-mon[32429]: Deploying daemon osd.5 on smithi107 2024-03-24T00:15:05.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:05 smithi078 ceph-mon[32429]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:15:05.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:05 smithi107 ceph-mon[36485]: Deploying daemon osd.5 on smithi107 2024-03-24T00:15:05.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:05 smithi107 ceph-mon[36485]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:15:05.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:05 smithi106 ceph-mon[36505]: Deploying daemon osd.5 on smithi107 2024-03-24T00:15:05.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:05 smithi106 ceph-mon[36505]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:15:07.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:07 smithi078 ceph-mon[32429]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:15:07.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:07 smithi107 ceph-mon[36485]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:15:07.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:07 smithi106 ceph-mon[36505]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:15:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:09 smithi078 ceph-mon[32429]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:15:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:09 smithi107 ceph-mon[36485]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:15:09.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:09 smithi106 ceph-mon[36505]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:15:10.753 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:10 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:10.753 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:10 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:10.753 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:10 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:15:10.754 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:10 smithi078 ceph-mon[32429]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:15:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:10 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:10 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:10 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:15:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:10 smithi107 ceph-mon[36485]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:15:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:10 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:10 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:10 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:15:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:10 smithi106 ceph-mon[36505]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:15:12.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:15:12.993+0000 7fe4bb7fe700 1 -- 172.21.15.107:0/1270332245 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe4c40625d0 con 0x7fe4b00a2890 2024-03-24T00:15:12.994 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 5 on host 'smithi107' 2024-03-24T00:15:12.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:15:12.995+0000 7fe4cc270700 1 -- 172.21.15.107:0/1270332245 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe4b00a2890 msgr2=0x7fe4b00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:12.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:15:12.995+0000 7fe4cc270700 1 --2- 172.21.15.107:0/1270332245 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe4b00a2890 0x7fe4b00a4d50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0005950 tx=0x7fe4c00058e0 comp rx=0 tx=0).stop 2024-03-24T00:15:12.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:15:12.995+0000 7fe4cc270700 1 -- 172.21.15.107:0/1270332245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c41033e0 msgr2=0x7fe4c4198900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:12.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:15:12.995+0000 7fe4cc270700 1 --2- 172.21.15.107:0/1270332245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c41033e0 0x7fe4c4198900 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe4b4004790 tx=0x7fe4b4009580 comp rx=0 tx=0).stop 2024-03-24T00:15:12.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:15:12.995+0000 7fe4cc270700 1 -- 172.21.15.107:0/1270332245 shutdown_connections 2024-03-24T00:15:12.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:15:12.995+0000 7fe4cc270700 1 --2- 172.21.15.107:0/1270332245 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe4b00a2890 0x7fe4b00a4d50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:12.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:15:12.995+0000 7fe4cc270700 1 --2- 172.21.15.107:0/1270332245 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4c4103d80 0x7fe4c419f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:12.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:15:12.995+0000 7fe4cc270700 1 --2- 172.21.15.107:0/1270332245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4c41021e0 0x7fe4c41983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:12.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:15:12.995+0000 7fe4cc270700 1 --2- 172.21.15.107:0/1270332245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c41033e0 0x7fe4c4198900 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:12.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:15:12.995+0000 7fe4cc270700 1 -- 172.21.15.107:0/1270332245 >> 172.21.15.107:0/1270332245 conn(0x7fe4c40fd740 msgr2=0x7fe4c4106730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:12.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:15:12.996+0000 7fe4cc270700 1 -- 172.21.15.107:0/1270332245 shutdown_connections 2024-03-24T00:15:12.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:15:12.996+0000 7fe4cc270700 1 -- 172.21.15.107:0/1270332245 wait complete. 2024-03-24T00:15:13.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:13 smithi078 ceph-mon[32429]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:15:13.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:13 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:13.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:13 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:13.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:13 smithi107 ceph-mon[36485]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:15:13.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:13 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:13.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:13 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:13.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:13 smithi106 ceph-mon[36505]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:15:13.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:13 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:13.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:13 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:13.741 DEBUG:teuthology.orchestra.run.smithi107:osd.5> sudo journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.5.service 2024-03-24T00:15:13.743 INFO:tasks.cephadm:Deploying osd.6 on smithi078 with /dev/vg_nvme/lv_2... 2024-03-24T00:15:13.743 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T00:15:13.992 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:15:14.440 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:14 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:14.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:14 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:14.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:14 smithi078 ceph-mon[32429]: from='osd.5 [v2:172.21.15.107:6808/3272806073,v1:172.21.15.107:6809/3272806073]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T00:15:14.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:14 smithi078 ceph-mon[32429]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T00:15:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:14 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:14 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:14 smithi107 ceph-mon[36485]: from='osd.5 [v2:172.21.15.107:6808/3272806073,v1:172.21.15.107:6809/3272806073]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T00:15:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:14 smithi107 ceph-mon[36485]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T00:15:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:14 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:14 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:14 smithi106 ceph-mon[36505]: from='osd.5 [v2:172.21.15.107:6808/3272806073,v1:172.21.15.107:6809/3272806073]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T00:15:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:14 smithi106 ceph-mon[36505]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T00:15:15.409 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:15 smithi078 ceph-mon[32429]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:15:15.409 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:15 smithi078 ceph-mon[32429]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T00:15:15.409 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:15 smithi078 ceph-mon[32429]: osdmap e36: 6 total, 5 up, 6 in 2024-03-24T00:15:15.409 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:15 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:15:15.409 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:15 smithi078 ceph-mon[32429]: from='osd.5 [v2:172.21.15.107:6808/3272806073,v1:172.21.15.107:6809/3272806073]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-24T00:15:15.409 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:15 smithi078 ceph-mon[32429]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-24T00:15:15.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:15 smithi107 ceph-mon[36485]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:15:15.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:15 smithi107 ceph-mon[36485]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T00:15:15.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:15 smithi107 ceph-mon[36485]: osdmap e36: 6 total, 5 up, 6 in 2024-03-24T00:15:15.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:15 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:15:15.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:15 smithi107 ceph-mon[36485]: from='osd.5 [v2:172.21.15.107:6808/3272806073,v1:172.21.15.107:6809/3272806073]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-24T00:15:15.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:15 smithi107 ceph-mon[36485]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-24T00:15:15.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:15 smithi106 ceph-mon[36505]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:15:15.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:15 smithi106 ceph-mon[36505]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T00:15:15.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:15 smithi106 ceph-mon[36505]: osdmap e36: 6 total, 5 up, 6 in 2024-03-24T00:15:15.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:15 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:15:15.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:15 smithi106 ceph-mon[36505]: from='osd.5 [v2:172.21.15.107:6808/3272806073,v1:172.21.15.107:6809/3272806073]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-24T00:15:15.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:15 smithi106 ceph-mon[36505]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-24T00:15:15.968 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T00:15:15.968 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T00:15:15.968 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10+0 records in 2024-03-24T00:15:15.968 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2024-03-24T00:15:15.968 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115974 s, 904 MB/s 2024-03-24T00:15:15.968 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping successful for: 2024-03-24T00:15:16.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:16 smithi107 ceph-mon[36485]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-03-24T00:15:16.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:16 smithi107 ceph-mon[36485]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T00:15:16.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:16 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:15:16.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:16 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:15:16.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:16 smithi107 ceph-mon[36485]: from='osd.5 ' entity='osd.5' 2024-03-24T00:15:16.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:16 smithi107 ceph-mon[36485]: osd.5 [v2:172.21.15.107:6808/3272806073,v1:172.21.15.107:6809/3272806073] boot 2024-03-24T00:15:16.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:16 smithi107 ceph-mon[36485]: osdmap e38: 6 total, 6 up, 6 in 2024-03-24T00:15:16.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:16 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:15:16.344 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 00:15:16 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5[48372]: 2024-03-24T00:15:16.088+0000 7fd9e4b69700 -1 osd.5 0 waiting for initial osdmap 2024-03-24T00:15:16.344 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 00:15:16 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5[48372]: 2024-03-24T00:15:16.095+0000 7fd9dc954700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:15:16.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:16 smithi078 ceph-mon[32429]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-03-24T00:15:16.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:16 smithi078 ceph-mon[32429]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T00:15:16.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:16 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:15:16.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:16 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:15:16.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:16 smithi078 ceph-mon[32429]: from='osd.5 ' entity='osd.5' 2024-03-24T00:15:16.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:16 smithi078 ceph-mon[32429]: osd.5 [v2:172.21.15.107:6808/3272806073,v1:172.21.15.107:6809/3272806073] boot 2024-03-24T00:15:16.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:16 smithi078 ceph-mon[32429]: osdmap e38: 6 total, 6 up, 6 in 2024-03-24T00:15:16.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:16 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:15:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:16 smithi106 ceph-mon[36505]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-03-24T00:15:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:16 smithi106 ceph-mon[36505]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T00:15:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:16 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:15:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:16 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:15:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:16 smithi106 ceph-mon[36505]: from='osd.5 ' entity='osd.5' 2024-03-24T00:15:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:16 smithi106 ceph-mon[36505]: osd.5 [v2:172.21.15.107:6808/3272806073,v1:172.21.15.107:6809/3272806073] boot 2024-03-24T00:15:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:16 smithi106 ceph-mon[36505]: osdmap e38: 6 total, 6 up, 6 in 2024-03-24T00:15:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:16 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:15:16.658 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi078:vg_nvme/lv_2 2024-03-24T00:15:16.896 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:15:17.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:17 smithi078 ceph-mon[32429]: purged_snaps scrub starts 2024-03-24T00:15:17.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:17 smithi078 ceph-mon[32429]: purged_snaps scrub ok 2024-03-24T00:15:17.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:17 smithi078 ceph-mon[32429]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:15:17.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:17 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:17.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:17 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:17.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:17 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:15:17.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:17 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:15:17.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:17 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:17.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:17 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:15:17.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:17 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:15:17.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:17 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:17.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:17 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:15:17.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:17 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:15:17.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:17 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:15:17.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:17 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:17.594 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:17 smithi078 ceph-mon[32429]: osdmap e39: 6 total, 6 up, 6 in 2024-03-24T00:15:17.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:17 smithi107 ceph-mon[36485]: purged_snaps scrub starts 2024-03-24T00:15:17.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:17 smithi107 ceph-mon[36485]: purged_snaps scrub ok 2024-03-24T00:15:17.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:17 smithi107 ceph-mon[36485]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:15:17.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:17 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:17.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:17 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:17.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:17 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:15:17.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:17 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:15:17.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:17 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:17.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:17 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:15:17.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:17 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:15:17.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:17 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:17.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:17 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:15:17.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:17 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:15:17.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:17 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:15:17.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:17 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:17.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:17 smithi107 ceph-mon[36485]: osdmap e39: 6 total, 6 up, 6 in 2024-03-24T00:15:17.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:17 smithi106 ceph-mon[36505]: purged_snaps scrub starts 2024-03-24T00:15:17.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:17 smithi106 ceph-mon[36505]: purged_snaps scrub ok 2024-03-24T00:15:17.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:17 smithi106 ceph-mon[36505]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:15:17.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:17 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:17.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:17 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:17.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:17 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:15:17.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:17 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:15:17.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:17 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:17.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:17 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:15:17.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:17 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:15:17.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:17 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:17.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:17 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:15:17.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:17 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:15:17.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:17 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:15:17.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:17 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:17.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:17 smithi106 ceph-mon[36505]: osdmap e39: 6 total, 6 up, 6 in 2024-03-24T00:15:18.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.557+0000 7f74ce3da700 1 -- 172.21.15.78:0/3554845576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74c81033b0 msgr2=0x7f74c8103810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:18.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.557+0000 7f74ce3da700 1 --2- 172.21.15.78:0/3554845576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74c81033b0 0x7f74c8103810 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f74bc0099c0 tx=0x7f74bc009cd0 comp rx=0 tx=0).stop 2024-03-24T00:15:18.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.557+0000 7f74ce3da700 1 -- 172.21.15.78:0/3554845576 shutdown_connections 2024-03-24T00:15:18.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.557+0000 7f74ce3da700 1 --2- 172.21.15.78:0/3554845576 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74c8103d50 0x7f74c810a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:18.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.557+0000 7f74ce3da700 1 --2- 172.21.15.78:0/3554845576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74c81021b0 0x7f74c81025d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:18.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.557+0000 7f74ce3da700 1 --2- 172.21.15.78:0/3554845576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74c81033b0 0x7f74c8103810 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:18.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.557+0000 7f74ce3da700 1 -- 172.21.15.78:0/3554845576 >> 172.21.15.78:0/3554845576 conn(0x7f74c80fd770 msgr2=0x7f74c80ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:18.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.557+0000 7f74ce3da700 1 -- 172.21.15.78:0/3554845576 shutdown_connections 2024-03-24T00:15:18.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.557+0000 7f74ce3da700 1 -- 172.21.15.78:0/3554845576 wait complete. 2024-03-24T00:15:18.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.558+0000 7f74ce3da700 1 Processor -- start 2024-03-24T00:15:18.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.558+0000 7f74ce3da700 1 -- start start 2024-03-24T00:15:18.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.558+0000 7f74ce3da700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74c81021b0 0x7f74c81983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:18.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.558+0000 7f74ce3da700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74c81033b0 0x7f74c81988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:18.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.558+0000 7f74ce3da700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74c8103d50 0x7f74c819f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:18.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.558+0000 7f74ce3da700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c810dda0 con 0x7f74c81033b0 2024-03-24T00:15:18.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.558+0000 7f74ce3da700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c810dc20 con 0x7f74c81021b0 2024-03-24T00:15:18.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.558+0000 7f74ce3da700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c810df20 con 0x7f74c8103d50 2024-03-24T00:15:18.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.558+0000 7f74c77fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74c81033b0 0x7f74c81988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:18.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.558+0000 7f74c77fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74c81033b0 0x7f74c81988f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35480/0 (socket says 172.21.15.78:35480) 2024-03-24T00:15:18.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.558+0000 7f74c77fe700 1 -- 172.21.15.78:0/1588718317 learned_addr learned my addr 172.21.15.78:0/1588718317 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:15:18.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.559+0000 7f74c7fff700 1 --2- 172.21.15.78:0/1588718317 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74c81021b0 0x7f74c81983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:18.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.559+0000 7f74cc977700 1 --2- 172.21.15.78:0/1588718317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74c8103d50 0x7f74c819f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:18.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.559+0000 7f74c77fe700 1 -- 172.21.15.78:0/1588718317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74c8103d50 msgr2=0x7f74c819f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:18.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.559+0000 7f74c77fe700 1 --2- 172.21.15.78:0/1588718317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74c8103d50 0x7f74c819f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:18.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.559+0000 7f74c77fe700 1 -- 172.21.15.78:0/1588718317 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74c81021b0 msgr2=0x7f74c81983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:18.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.559+0000 7f74c77fe700 1 --2- 172.21.15.78:0/1588718317 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74c81021b0 0x7f74c81983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:18.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.559+0000 7f74c77fe700 1 -- 172.21.15.78:0/1588718317 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c81a0010 con 0x7f74c81033b0 2024-03-24T00:15:18.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.559+0000 7f74cc977700 1 --2- 172.21.15.78:0/1588718317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74c8103d50 0x7f74c819f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:15:18.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.559+0000 7f74c77fe700 1 --2- 172.21.15.78:0/1588718317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74c81033b0 0x7f74c81988f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f74bc005910 tx=0x7f74bc0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:18.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.559+0000 7f74c57fa700 1 -- 172.21.15.78:0/1588718317 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74bc027070 con 0x7f74c81033b0 2024-03-24T00:15:18.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.559+0000 7f74ce3da700 1 -- 172.21.15.78:0/1588718317 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c81a02a0 con 0x7f74c81033b0 2024-03-24T00:15:18.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.559+0000 7f74c57fa700 1 -- 172.21.15.78:0/1588718317 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f74bc0041d0 con 0x7f74c81033b0 2024-03-24T00:15:18.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.559+0000 7f74c57fa700 1 -- 172.21.15.78:0/1588718317 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74bc00f460 con 0x7f74c81033b0 2024-03-24T00:15:18.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.559+0000 7f74ce3da700 1 -- 172.21.15.78:0/1588718317 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74c81a0750 con 0x7f74c81033b0 2024-03-24T00:15:18.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.560+0000 7f74ce3da700 1 -- 172.21.15.78:0/1588718317 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74c8068e50 con 0x7f74c81033b0 2024-03-24T00:15:18.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.561+0000 7f74c57fa700 1 -- 172.21.15.78:0/1588718317 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f74bc02a070 con 0x7f74c81033b0 2024-03-24T00:15:18.564 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:18 smithi078 ceph-mon[32429]: Detected new or changed devices on smithi107 2024-03-24T00:15:18.564 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:18 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi107 to 8540M 2024-03-24T00:15:18.564 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:18 smithi078 ceph-mon[32429]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T00:15:18.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.562+0000 7f74c57fa700 1 --2- 172.21.15.78:0/1588718317 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f74b00a2890 0x7f74b00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:18.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.562+0000 7f74c7fff700 1 --2- 172.21.15.78:0/1588718317 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f74b00a2890 0x7f74b00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:18.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.562+0000 7f74c57fa700 1 -- 172.21.15.78:0/1588718317 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4562+0+0 (secure 0 0 0) 0x7f74bc0cef70 con 0x7f74c81033b0 2024-03-24T00:15:18.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.562+0000 7f74c7fff700 1 --2- 172.21.15.78:0/1588718317 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f74b00a2890 0x7f74b00a4d50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f74b8005950 tx=0x7f74b800a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:18.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.566+0000 7f74c57fa700 1 -- 172.21.15.78:0/1588718317 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f74bc013070 con 0x7f74c81033b0 2024-03-24T00:15:18.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:18 smithi106 ceph-mon[36505]: Detected new or changed devices on smithi107 2024-03-24T00:15:18.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:18 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi107 to 8540M 2024-03-24T00:15:18.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:18 smithi106 ceph-mon[36505]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T00:15:18.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:18.771+0000 7f74ce3da700 1 -- 172.21.15.78:0/1588718317 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f74c80625d0 con 0x7f74b00a2890 2024-03-24T00:15:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:18 smithi107 ceph-mon[36485]: Detected new or changed devices on smithi107 2024-03-24T00:15:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:18 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi107 to 8540M 2024-03-24T00:15:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:18 smithi107 ceph-mon[36485]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T00:15:19.624 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:19 smithi078 ceph-mon[32429]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:15:19.625 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:19 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:15:19.625 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:19 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:15:19.625 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:19 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:15:19.625 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:19 smithi078 ceph-mon[32429]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T00:15:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:19 smithi106 ceph-mon[36505]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:15:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:19 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:15:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:19 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:15:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:19 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:15:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:19 smithi106 ceph-mon[36505]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T00:15:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:19 smithi107 ceph-mon[36485]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:15:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:19 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:15:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:19 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:15:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:19 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:15:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:19 smithi107 ceph-mon[36485]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T00:15:20.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:20 smithi078 ceph-mon[32429]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:15:20.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:20 smithi078 ceph-mon[32429]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T00:15:20.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:20 smithi106 ceph-mon[36505]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:15:20.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:20 smithi106 ceph-mon[36505]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T00:15:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:20 smithi107 ceph-mon[36485]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:15:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:20 smithi107 ceph-mon[36485]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T00:15:21.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:21 smithi106 ceph-mon[36505]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 536 GiB / 536 GiB avail; 147 KiB/s, 0 objects/s recovering 2024-03-24T00:15:21.744 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:21 smithi107 ceph-mon[36485]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 536 GiB / 536 GiB avail; 147 KiB/s, 0 objects/s recovering 2024-03-24T00:15:21.772 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:21 smithi078 ceph-mon[32429]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 536 GiB / 536 GiB avail; 147 KiB/s, 0 objects/s recovering 2024-03-24T00:15:22.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:22 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3141741373' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5696389a-918f-4ea8-b20d-4e2a0d4cb3bf"}]: dispatch 2024-03-24T00:15:22.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:22 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3141741373' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5696389a-918f-4ea8-b20d-4e2a0d4cb3bf"}]': finished 2024-03-24T00:15:22.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:22 smithi106 ceph-mon[36505]: osdmap e43: 7 total, 6 up, 7 in 2024-03-24T00:15:22.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:22 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:15:22.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:22 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/562753216' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:15:22.758 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:22 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3141741373' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5696389a-918f-4ea8-b20d-4e2a0d4cb3bf"}]: dispatch 2024-03-24T00:15:22.758 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:22 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3141741373' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5696389a-918f-4ea8-b20d-4e2a0d4cb3bf"}]': finished 2024-03-24T00:15:22.758 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:22 smithi107 ceph-mon[36485]: osdmap e43: 7 total, 6 up, 7 in 2024-03-24T00:15:22.758 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:22 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:15:22.758 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:22 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/562753216' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:15:22.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:22 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3141741373' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5696389a-918f-4ea8-b20d-4e2a0d4cb3bf"}]: dispatch 2024-03-24T00:15:22.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:22 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3141741373' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5696389a-918f-4ea8-b20d-4e2a0d4cb3bf"}]': finished 2024-03-24T00:15:22.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:22 smithi078 ceph-mon[32429]: osdmap e43: 7 total, 6 up, 7 in 2024-03-24T00:15:22.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:22 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:15:22.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:22 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/562753216' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:15:23.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:23 smithi106 ceph-mon[36505]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T00:15:23.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:23 smithi078 ceph-mon[32429]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T00:15:23.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:23 smithi107 ceph-mon[36485]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T00:15:25.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:25 smithi106 ceph-mon[36505]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-24T00:15:25.739 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:25 smithi078 ceph-mon[32429]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-24T00:15:25.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:25 smithi107 ceph-mon[36485]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-24T00:15:27.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:27 smithi078 ceph-mon[32429]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-24T00:15:27.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:27 smithi106 ceph-mon[36505]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-24T00:15:27.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:27 smithi107 ceph-mon[36485]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-24T00:15:29.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:29 smithi078 ceph-mon[32429]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-24T00:15:29.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:29 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T00:15:29.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:29 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:15:29.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:29 smithi106 ceph-mon[36505]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-24T00:15:29.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:29 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T00:15:29.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:29 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:15:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:29 smithi107 ceph-mon[36485]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-24T00:15:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:29 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T00:15:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:29 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:15:30.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:30 smithi078 ceph-mon[32429]: Deploying daemon osd.6 on smithi078 2024-03-24T00:15:30.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:30 smithi106 ceph-mon[36505]: Deploying daemon osd.6 on smithi078 2024-03-24T00:15:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:30 smithi107 ceph-mon[36485]: Deploying daemon osd.6 on smithi078 2024-03-24T00:15:31.435 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:31 smithi078 ceph-mon[32429]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:15:31.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:31 smithi106 ceph-mon[36505]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:15:31.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:31 smithi107 ceph-mon[36485]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:15:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:33 smithi078 ceph-mon[32429]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:15:33.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:33 smithi106 ceph-mon[36505]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:15:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:33 smithi107 ceph-mon[36485]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:15:35.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:35 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:35.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:35 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:35.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:35 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:15:35.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:35 smithi106 ceph-mon[36505]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:15:35.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:35 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:35.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:35 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:35.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:35 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:15:35.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:35 smithi078 ceph-mon[32429]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:15:35.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:35 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:35.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:35 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:35.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:35 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:15:35.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:35 smithi107 ceph-mon[36485]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:15:37.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:37 smithi078 ceph-mon[32429]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:15:37.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:37 smithi107 ceph-mon[36485]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:15:37.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:37 smithi106 ceph-mon[36505]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:15:37.897 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 6 on host 'smithi078' 2024-03-24T00:15:37.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:37.897+0000 7f74c57fa700 1 -- 172.21.15.78:0/1588718317 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f74c80625d0 con 0x7f74b00a2890 2024-03-24T00:15:37.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:37.899+0000 7f74ce3da700 1 -- 172.21.15.78:0/1588718317 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f74b00a2890 msgr2=0x7f74b00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:37.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:37.899+0000 7f74ce3da700 1 --2- 172.21.15.78:0/1588718317 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f74b00a2890 0x7f74b00a4d50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f74b8005950 tx=0x7f74b800a300 comp rx=0 tx=0).stop 2024-03-24T00:15:37.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:37.899+0000 7f74ce3da700 1 -- 172.21.15.78:0/1588718317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74c81033b0 msgr2=0x7f74c81988f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:37.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:37.899+0000 7f74ce3da700 1 --2- 172.21.15.78:0/1588718317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74c81033b0 0x7f74c81988f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f74bc005910 tx=0x7f74bc0047b0 comp rx=0 tx=0).stop 2024-03-24T00:15:37.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:37.900+0000 7f74ce3da700 1 -- 172.21.15.78:0/1588718317 shutdown_connections 2024-03-24T00:15:37.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:37.900+0000 7f74ce3da700 1 --2- 172.21.15.78:0/1588718317 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f74b00a2890 0x7f74b00a4d50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:37.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:37.900+0000 7f74ce3da700 1 --2- 172.21.15.78:0/1588718317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74c8103d50 0x7f74c819f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:37.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:37.900+0000 7f74ce3da700 1 --2- 172.21.15.78:0/1588718317 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74c81021b0 0x7f74c81983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:37.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:37.900+0000 7f74ce3da700 1 --2- 172.21.15.78:0/1588718317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74c81033b0 0x7f74c81988f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:37.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:37.900+0000 7f74ce3da700 1 -- 172.21.15.78:0/1588718317 >> 172.21.15.78:0/1588718317 conn(0x7f74c80fd770 msgr2=0x7f74c8106730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:37.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:37.900+0000 7f74ce3da700 1 -- 172.21.15.78:0/1588718317 shutdown_connections 2024-03-24T00:15:37.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:15:37.900+0000 7f74ce3da700 1 -- 172.21.15.78:0/1588718317 wait complete. 2024-03-24T00:15:38.682 DEBUG:teuthology.orchestra.run.smithi078:osd.6> sudo journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.6.service 2024-03-24T00:15:38.684 INFO:tasks.cephadm:Deploying osd.7 on smithi106 with /dev/vg_nvme/lv_2... 2024-03-24T00:15:38.685 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T00:15:38.931 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T00:15:39.052 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:38 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:39.053 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:38 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:39.053 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:38 smithi106 ceph-mon[36505]: from='osd.6 [v2:172.21.15.78:6818/3091148508,v1:172.21.15.78:6819/3091148508]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T00:15:39.053 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:38 smithi106 ceph-mon[36505]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:15:39.053 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:38 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:39.053 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:38 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:38 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:38 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:38 smithi078 ceph-mon[32429]: from='osd.6 [v2:172.21.15.78:6818/3091148508,v1:172.21.15.78:6819/3091148508]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T00:15:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:38 smithi078 ceph-mon[32429]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:15:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:38 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:38 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:38 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:38 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:38 smithi107 ceph-mon[36485]: from='osd.6 [v2:172.21.15.78:6818/3091148508,v1:172.21.15.78:6819/3091148508]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T00:15:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:38 smithi107 ceph-mon[36485]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:15:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:38 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:38 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:40.080 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:39 smithi078 ceph-mon[32429]: from='osd.6 [v2:172.21.15.78:6818/3091148508,v1:172.21.15.78:6819/3091148508]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T00:15:40.080 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:39 smithi078 ceph-mon[32429]: osdmap e44: 7 total, 6 up, 7 in 2024-03-24T00:15:40.080 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:39 smithi078 ceph-mon[32429]: from='osd.6 [v2:172.21.15.78:6818/3091148508,v1:172.21.15.78:6819/3091148508]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-24T00:15:40.080 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:39 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:15:40.080 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 00:15:39 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6[58136]: 2024-03-24T00:15:39.887+0000 7f8a42470700 -1 osd.6 0 waiting for initial osdmap 2024-03-24T00:15:40.080 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 00:15:39 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6[58136]: 2024-03-24T00:15:39.895+0000 7f8a3e263700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:15:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:39 smithi106 ceph-mon[36505]: from='osd.6 [v2:172.21.15.78:6818/3091148508,v1:172.21.15.78:6819/3091148508]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T00:15:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:39 smithi106 ceph-mon[36505]: osdmap e44: 7 total, 6 up, 7 in 2024-03-24T00:15:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:39 smithi106 ceph-mon[36505]: from='osd.6 [v2:172.21.15.78:6818/3091148508,v1:172.21.15.78:6819/3091148508]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-24T00:15:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:39 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:15:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:39 smithi107 ceph-mon[36485]: from='osd.6 [v2:172.21.15.78:6818/3091148508,v1:172.21.15.78:6819/3091148508]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T00:15:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:39 smithi107 ceph-mon[36485]: osdmap e44: 7 total, 6 up, 7 in 2024-03-24T00:15:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:39 smithi107 ceph-mon[36485]: from='osd.6 [v2:172.21.15.78:6818/3091148508,v1:172.21.15.78:6819/3091148508]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-24T00:15:40.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:39 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:15:41.034 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T00:15:41.035 INFO:teuthology.orchestra.run.smithi106.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T00:15:41.035 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10+0 records in 2024-03-24T00:15:41.035 INFO:teuthology.orchestra.run.smithi106.stderr:10+0 records out 2024-03-24T00:15:41.035 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113246 s, 926 MB/s 2024-03-24T00:15:41.035 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping successful for: 2024-03-24T00:15:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:40 smithi078 ceph-mon[32429]: from='osd.6 [v2:172.21.15.78:6818/3091148508,v1:172.21.15.78:6819/3091148508]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-24T00:15:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:40 smithi078 ceph-mon[32429]: osdmap e45: 7 total, 6 up, 7 in 2024-03-24T00:15:41.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:15:41.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:15:41.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:40 smithi078 ceph-mon[32429]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:15:41.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:40 smithi106 ceph-mon[36505]: from='osd.6 [v2:172.21.15.78:6818/3091148508,v1:172.21.15.78:6819/3091148508]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-24T00:15:41.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:40 smithi106 ceph-mon[36505]: osdmap e45: 7 total, 6 up, 7 in 2024-03-24T00:15:41.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:15:41.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:15:41.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:40 smithi106 ceph-mon[36505]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:15:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:40 smithi107 ceph-mon[36485]: from='osd.6 [v2:172.21.15.78:6818/3091148508,v1:172.21.15.78:6819/3091148508]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-24T00:15:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:40 smithi107 ceph-mon[36485]: osdmap e45: 7 total, 6 up, 7 in 2024-03-24T00:15:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:15:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:15:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:40 smithi107 ceph-mon[36485]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:15:41.733 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi106:vg_nvme/lv_2 2024-03-24T00:15:41.973 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T00:15:42.096 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:41 smithi106 ceph-mon[36505]: purged_snaps scrub starts 2024-03-24T00:15:42.096 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:41 smithi106 ceph-mon[36505]: purged_snaps scrub ok 2024-03-24T00:15:42.096 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:41 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:15:42.096 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:41 smithi106 ceph-mon[36505]: osd.6 [v2:172.21.15.78:6818/3091148508,v1:172.21.15.78:6819/3091148508] boot 2024-03-24T00:15:42.096 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:41 smithi106 ceph-mon[36505]: osdmap e46: 7 total, 7 up, 7 in 2024-03-24T00:15:42.096 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:41 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:15:42.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:41 smithi078 ceph-mon[32429]: purged_snaps scrub starts 2024-03-24T00:15:42.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:41 smithi078 ceph-mon[32429]: purged_snaps scrub ok 2024-03-24T00:15:42.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:41 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:15:42.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:41 smithi078 ceph-mon[32429]: osd.6 [v2:172.21.15.78:6818/3091148508,v1:172.21.15.78:6819/3091148508] boot 2024-03-24T00:15:42.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:41 smithi078 ceph-mon[32429]: osdmap e46: 7 total, 7 up, 7 in 2024-03-24T00:15:42.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:41 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:15:42.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:41 smithi107 ceph-mon[36485]: purged_snaps scrub starts 2024-03-24T00:15:42.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:41 smithi107 ceph-mon[36485]: purged_snaps scrub ok 2024-03-24T00:15:42.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:41 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:15:42.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:41 smithi107 ceph-mon[36485]: osd.6 [v2:172.21.15.78:6818/3091148508,v1:172.21.15.78:6819/3091148508] boot 2024-03-24T00:15:42.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:41 smithi107 ceph-mon[36485]: osdmap e46: 7 total, 7 up, 7 in 2024-03-24T00:15:42.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:41 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:15:43.145 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:42 smithi106 ceph-mon[36505]: osdmap e47: 7 total, 7 up, 7 in 2024-03-24T00:15:43.145 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:42 smithi106 ceph-mon[36505]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:15:43.146 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:42 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:43.146 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:42 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:43.146 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:42 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:15:43.146 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:42 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:15:43.146 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:42 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:15:43.146 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:42 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:43.146 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:42 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:15:43.146 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:42 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:15:43.146 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:42 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:43.146 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:42 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:15:43.146 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:42 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:15:43.146 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:42 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:15:43.146 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:42 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:42 smithi078 ceph-mon[32429]: osdmap e47: 7 total, 7 up, 7 in 2024-03-24T00:15:43.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:42 smithi078 ceph-mon[32429]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:15:43.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:42 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:43.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:42 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:43.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:42 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:15:43.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:42 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:15:43.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:42 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:15:43.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:42 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:43.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:42 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:15:43.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:42 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:15:43.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:42 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:43.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:42 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:15:43.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:42 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:15:43.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:42 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:15:43.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:42 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:43.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:42 smithi107 ceph-mon[36485]: osdmap e47: 7 total, 7 up, 7 in 2024-03-24T00:15:43.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:42 smithi107 ceph-mon[36485]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:15:43.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:42 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:43.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:42 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:43.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:42 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:15:43.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:42 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:15:43.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:42 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:15:43.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:42 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:43.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:42 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:15:43.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:42 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:15:43.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:42 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:43.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:42 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:15:43.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:42 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:15:43.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:42 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:15:43.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:42 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:15:43.660 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.659+0000 7f267203c700 1 -- 172.21.15.106:0/3871239769 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f266c102c80 msgr2=0x7f266c109530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:43.660 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.659+0000 7f267203c700 1 --2- 172.21.15.106:0/3871239769 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f266c102c80 0x7f266c109530 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f266400b0d0 tx=0x7f266400b3e0 comp rx=0 tx=0).stop 2024-03-24T00:15:43.660 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.659+0000 7f267203c700 1 -- 172.21.15.106:0/3871239769 shutdown_connections 2024-03-24T00:15:43.660 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.659+0000 7f267203c700 1 --2- 172.21.15.106:0/3871239769 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f266c102c80 0x7f266c109530 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:43.660 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.659+0000 7f267203c700 1 --2- 172.21.15.106:0/3871239769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f266c1010e0 0x7f266c101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:43.660 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.659+0000 7f267203c700 1 --2- 172.21.15.106:0/3871239769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f266c1022e0 0x7f266c102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:43.660 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.659+0000 7f267203c700 1 -- 172.21.15.106:0/3871239769 >> 172.21.15.106:0/3871239769 conn(0x7f266c0fc680 msgr2=0x7f266c0feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:43.660 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.659+0000 7f267203c700 1 -- 172.21.15.106:0/3871239769 shutdown_connections 2024-03-24T00:15:43.660 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.659+0000 7f267203c700 1 -- 172.21.15.106:0/3871239769 wait complete. 2024-03-24T00:15:43.661 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.660+0000 7f267203c700 1 Processor -- start 2024-03-24T00:15:43.661 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.660+0000 7f267203c700 1 -- start start 2024-03-24T00:15:43.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.660+0000 7f267203c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f266c1010e0 0x7f266c1973b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:43.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.660+0000 7f267203c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f266c1022e0 0x7f266c1978f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:43.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.660+0000 7f267203c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f266c102c80 0x7f266c19e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:43.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.660+0000 7f267203c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266c10cdc0 con 0x7f266c1022e0 2024-03-24T00:15:43.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.660+0000 7f267203c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266c10cc40 con 0x7f266c1010e0 2024-03-24T00:15:43.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.660+0000 7f267203c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266c10cf40 con 0x7f266c102c80 2024-03-24T00:15:43.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.661+0000 7f266b7fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f266c1010e0 0x7f266c1973b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:43.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.661+0000 7f266b7fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f266c1010e0 0x7f266c1973b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:44014/0 (socket says 172.21.15.106:44014) 2024-03-24T00:15:43.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.661+0000 7f266b7fe700 1 -- 172.21.15.106:0/3613474183 learned_addr learned my addr 172.21.15.106:0/3613474183 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-24T00:15:43.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.661+0000 7f266affd700 1 --2- 172.21.15.106:0/3613474183 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f266c1022e0 0x7f266c1978f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:43.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.661+0000 7f266bfff700 1 --2- 172.21.15.106:0/3613474183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f266c102c80 0x7f266c19e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:43.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.661+0000 7f266b7fe700 1 -- 172.21.15.106:0/3613474183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f266c102c80 msgr2=0x7f266c19e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:43.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.661+0000 7f266b7fe700 1 --2- 172.21.15.106:0/3613474183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f266c102c80 0x7f266c19e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:43.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.661+0000 7f266b7fe700 1 -- 172.21.15.106:0/3613474183 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f266c1022e0 msgr2=0x7f266c1978f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:43.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.661+0000 7f266b7fe700 1 --2- 172.21.15.106:0/3613474183 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f266c1022e0 0x7f266c1978f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:43.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.661+0000 7f266b7fe700 1 -- 172.21.15.106:0/3613474183 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f266c19f010 con 0x7f266c1010e0 2024-03-24T00:15:43.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.661+0000 7f266affd700 1 --2- 172.21.15.106:0/3613474183 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f266c1022e0 0x7f266c1978f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:15:43.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.661+0000 7f266b7fe700 1 --2- 172.21.15.106:0/3613474183 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f266c1010e0 0x7f266c1973b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f265c00d9b0 tx=0x7f265c00dd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:43.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.661+0000 7f2668ff9700 1 -- 172.21.15.106:0/3613474183 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265c01b070 con 0x7f266c1010e0 2024-03-24T00:15:43.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.661+0000 7f267203c700 1 -- 172.21.15.106:0/3613474183 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f266c06a910 con 0x7f266c1010e0 2024-03-24T00:15:43.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.661+0000 7f2668ff9700 1 -- 172.21.15.106:0/3613474183 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f265c00ecd0 con 0x7f266c1010e0 2024-03-24T00:15:43.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.661+0000 7f2668ff9700 1 -- 172.21.15.106:0/3613474183 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265c0146c0 con 0x7f266c1010e0 2024-03-24T00:15:43.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.661+0000 7f267203c700 1 -- 172.21.15.106:0/3613474183 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f266c06aed0 con 0x7f266c1010e0 2024-03-24T00:15:43.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.662+0000 7f267203c700 1 -- 172.21.15.106:0/3613474183 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f266c068e50 con 0x7f266c1010e0 2024-03-24T00:15:43.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.663+0000 7f2668ff9700 1 -- 172.21.15.106:0/3613474183 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f265c01e070 con 0x7f266c1010e0 2024-03-24T00:15:43.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.664+0000 7f2668ff9700 1 --2- 172.21.15.106:0/3613474183 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f26540a2890 0x7f26540a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:43.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.664+0000 7f2668ff9700 1 -- 172.21.15.106:0/3613474183 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f265c0cf2f0 con 0x7f266c1010e0 2024-03-24T00:15:43.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.664+0000 7f266affd700 1 --2- 172.21.15.106:0/3613474183 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f26540a2890 0x7f26540a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:43.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.664+0000 7f266affd700 1 --2- 172.21.15.106:0/3613474183 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f26540a2890 0x7f26540a4d50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f26600099d0 tx=0x7f2660008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:43.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.668+0000 7f2668ff9700 1 -- 172.21.15.106:0/3613474183 <== mon.2 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f265c09d0a0 con 0x7f266c1010e0 2024-03-24T00:15:43.870 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:15:43.871+0000 7f267203c700 1 -- 172.21.15.106:0/3613474183 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f266c06b190 con 0x7f26540a2890 2024-03-24T00:15:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:43 smithi106 ceph-mon[36505]: Detected new or changed devices on smithi078 2024-03-24T00:15:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:43 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi078 to 5696M 2024-03-24T00:15:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:43 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:15:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:43 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:15:44.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:43 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:15:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:43 smithi078 ceph-mon[32429]: Detected new or changed devices on smithi078 2024-03-24T00:15:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:43 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi078 to 5696M 2024-03-24T00:15:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:43 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:15:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:43 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:15:44.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:43 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:15:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:43 smithi107 ceph-mon[36485]: Detected new or changed devices on smithi078 2024-03-24T00:15:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:43 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi078 to 5696M 2024-03-24T00:15:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:43 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:15:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:43 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:15:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:43 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:15:45.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:44 smithi106 ceph-mon[36505]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:15:45.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:44 smithi106 ceph-mon[36505]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:15:45.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:44 smithi078 ceph-mon[32429]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:15:45.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:44 smithi078 ceph-mon[32429]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:15:45.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:44 smithi107 ceph-mon[36485]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:15:45.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:44 smithi107 ceph-mon[36485]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:15:47.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:47 smithi078 ceph-mon[32429]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:15:47.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:47 smithi078 ceph-mon[32429]: from='client.? 172.21.15.106:0/1577234058' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6b764e8-ed4e-4c9a-a85b-94af6dc6b924"}]: dispatch 2024-03-24T00:15:47.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:47 smithi078 ceph-mon[32429]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6b764e8-ed4e-4c9a-a85b-94af6dc6b924"}]: dispatch 2024-03-24T00:15:47.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:47 smithi078 ceph-mon[32429]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f6b764e8-ed4e-4c9a-a85b-94af6dc6b924"}]': finished 2024-03-24T00:15:47.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:47 smithi078 ceph-mon[32429]: osdmap e48: 8 total, 7 up, 8 in 2024-03-24T00:15:47.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:47 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:15:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:47 smithi107 ceph-mon[36485]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:15:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:47 smithi107 ceph-mon[36485]: from='client.? 172.21.15.106:0/1577234058' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6b764e8-ed4e-4c9a-a85b-94af6dc6b924"}]: dispatch 2024-03-24T00:15:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:47 smithi107 ceph-mon[36485]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6b764e8-ed4e-4c9a-a85b-94af6dc6b924"}]: dispatch 2024-03-24T00:15:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:47 smithi107 ceph-mon[36485]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f6b764e8-ed4e-4c9a-a85b-94af6dc6b924"}]': finished 2024-03-24T00:15:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:47 smithi107 ceph-mon[36485]: osdmap e48: 8 total, 7 up, 8 in 2024-03-24T00:15:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:47 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:15:47.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:47 smithi106 ceph-mon[36505]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:15:47.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:47 smithi106 ceph-mon[36505]: from='client.? 172.21.15.106:0/1577234058' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6b764e8-ed4e-4c9a-a85b-94af6dc6b924"}]: dispatch 2024-03-24T00:15:47.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:47 smithi106 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6b764e8-ed4e-4c9a-a85b-94af6dc6b924"}]: dispatch 2024-03-24T00:15:47.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:47 smithi106 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f6b764e8-ed4e-4c9a-a85b-94af6dc6b924"}]': finished 2024-03-24T00:15:47.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:47 smithi106 ceph-mon[36505]: osdmap e48: 8 total, 7 up, 8 in 2024-03-24T00:15:47.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:47 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:15:48.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:48 smithi078 ceph-mon[32429]: from='client.? 172.21.15.106:0/2418853557' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:15:48.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:48 smithi107 ceph-mon[36485]: from='client.? 172.21.15.106:0/2418853557' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:15:48.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:48 smithi106 ceph-mon[36505]: from='client.? 172.21.15.106:0/2418853557' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:15:49.553 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:49 smithi106 ceph-mon[36505]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:15:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:49 smithi078 ceph-mon[32429]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:15:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:49 smithi107 ceph-mon[36485]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:15:51.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:51 smithi078 ceph-mon[32429]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:15:51.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:51 smithi107 ceph-mon[36485]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:15:51.630 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:51 smithi106 ceph-mon[36505]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:15:53.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:53 smithi078 ceph-mon[32429]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:15:53.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:53 smithi107 ceph-mon[36485]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:15:53.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:53 smithi106 ceph-mon[36505]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:15:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:54 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T00:15:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:54 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:15:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:54 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T00:15:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:54 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:15:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:54 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T00:15:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:54 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:15:55.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:55 smithi078 ceph-mon[32429]: Deploying daemon osd.7 on smithi106 2024-03-24T00:15:55.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:55 smithi078 ceph-mon[32429]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:15:55.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:55 smithi107 ceph-mon[36485]: Deploying daemon osd.7 on smithi106 2024-03-24T00:15:55.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:55 smithi107 ceph-mon[36485]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:15:55.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:55 smithi106 ceph-mon[36505]: Deploying daemon osd.7 on smithi106 2024-03-24T00:15:55.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:55 smithi106 ceph-mon[36505]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:15:57.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:57 smithi106 ceph-mon[36505]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:15:57.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:57 smithi078 ceph-mon[32429]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:15:57.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:57 smithi107 ceph-mon[36485]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:15:59.496 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:15:59 smithi106 ceph-mon[36505]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:15:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:15:59 smithi078 ceph-mon[32429]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:15:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:15:59 smithi107 ceph-mon[36485]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:16:01.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:00 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:01.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:00 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:01.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:00 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:16:01.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:00 smithi106 ceph-mon[36505]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:16:01.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:00 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:01.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:00 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:01.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:00 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:16:01.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:00 smithi078 ceph-mon[32429]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:16:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:00 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:00 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:00 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:16:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:00 smithi107 ceph-mon[36485]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:16:03.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:03 smithi106 ceph-mon[36505]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:16:03.489 INFO:teuthology.orchestra.run.smithi106.stdout:Created osd(s) 7 on host 'smithi106' 2024-03-24T00:16:03.489 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:16:03.488+0000 7f2668ff9700 1 -- 172.21.15.106:0/3613474183 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f266c06b190 con 0x7f26540a2890 2024-03-24T00:16:03.491 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:16:03.490+0000 7f267203c700 1 -- 172.21.15.106:0/3613474183 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f26540a2890 msgr2=0x7f26540a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:03.491 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:16:03.490+0000 7f267203c700 1 --2- 172.21.15.106:0/3613474183 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f26540a2890 0x7f26540a4d50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f26600099d0 tx=0x7f2660008040 comp rx=0 tx=0).stop 2024-03-24T00:16:03.491 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:16:03.490+0000 7f267203c700 1 -- 172.21.15.106:0/3613474183 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f266c1010e0 msgr2=0x7f266c1973b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:03.492 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:16:03.490+0000 7f267203c700 1 --2- 172.21.15.106:0/3613474183 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f266c1010e0 0x7f266c1973b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f265c00d9b0 tx=0x7f265c00dd70 comp rx=0 tx=0).stop 2024-03-24T00:16:03.492 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:16:03.490+0000 7f267203c700 1 -- 172.21.15.106:0/3613474183 shutdown_connections 2024-03-24T00:16:03.492 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:16:03.490+0000 7f267203c700 1 --2- 172.21.15.106:0/3613474183 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f26540a2890 0x7f26540a4d50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:03.492 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:16:03.490+0000 7f267203c700 1 --2- 172.21.15.106:0/3613474183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f266c102c80 0x7f266c19e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:03.492 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:16:03.490+0000 7f267203c700 1 --2- 172.21.15.106:0/3613474183 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f266c1010e0 0x7f266c1973b0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:03.492 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:16:03.490+0000 7f267203c700 1 --2- 172.21.15.106:0/3613474183 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f266c1022e0 0x7f266c1978f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:03.492 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:16:03.490+0000 7f267203c700 1 -- 172.21.15.106:0/3613474183 >> 172.21.15.106:0/3613474183 conn(0x7f266c0fc680 msgr2=0x7f266c1055e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:03.492 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:16:03.491+0000 7f267203c700 1 -- 172.21.15.106:0/3613474183 shutdown_connections 2024-03-24T00:16:03.492 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:16:03.491+0000 7f267203c700 1 -- 172.21.15.106:0/3613474183 wait complete. 2024-03-24T00:16:03.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:03 smithi078 ceph-mon[32429]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:16:03.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:03 smithi107 ceph-mon[36485]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:16:04.133 DEBUG:teuthology.orchestra.run.smithi106:osd.7> sudo journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.7.service 2024-03-24T00:16:04.136 INFO:tasks.cephadm:Deploying osd.8 on smithi107 with /dev/vg_nvme/lv_2... 2024-03-24T00:16:04.136 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T00:16:04.380 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:16:04.816 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:04 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:04.816 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:04 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:04.816 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:04 smithi107 ceph-mon[36485]: from='osd.7 [v2:172.21.15.106:6816/4262103080,v1:172.21.15.106:6817/4262103080]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T00:16:04.816 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:04 smithi107 ceph-mon[36485]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:16:04.816 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:04 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:04.816 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:04 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:04 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:04 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:04 smithi078 ceph-mon[32429]: from='osd.7 [v2:172.21.15.106:6816/4262103080,v1:172.21.15.106:6817/4262103080]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T00:16:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:04 smithi078 ceph-mon[32429]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:16:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:04 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:04 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:04.877 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:04 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:04.877 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:04 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:04.877 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:04 smithi106 ceph-mon[36505]: from='osd.7 [v2:172.21.15.106:6816/4262103080,v1:172.21.15.106:6817/4262103080]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T00:16:04.877 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:04 smithi106 ceph-mon[36505]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:16:04.877 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:04 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:04.877 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:04 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:05.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:05 smithi078 ceph-mon[32429]: from='osd.7 [v2:172.21.15.106:6816/4262103080,v1:172.21.15.106:6817/4262103080]' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T00:16:05.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:05 smithi078 ceph-mon[32429]: osdmap e49: 8 total, 7 up, 8 in 2024-03-24T00:16:05.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:05 smithi078 ceph-mon[32429]: from='osd.7 [v2:172.21.15.106:6816/4262103080,v1:172.21.15.106:6817/4262103080]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-24T00:16:05.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:05 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:16:05.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:05 smithi107 ceph-mon[36485]: from='osd.7 [v2:172.21.15.106:6816/4262103080,v1:172.21.15.106:6817/4262103080]' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T00:16:05.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:05 smithi107 ceph-mon[36485]: osdmap e49: 8 total, 7 up, 8 in 2024-03-24T00:16:05.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:05 smithi107 ceph-mon[36485]: from='osd.7 [v2:172.21.15.106:6816/4262103080,v1:172.21.15.106:6817/4262103080]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-24T00:16:05.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:05 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:16:05.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:05 smithi106 ceph-mon[36505]: from='osd.7 [v2:172.21.15.106:6816/4262103080,v1:172.21.15.106:6817/4262103080]' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T00:16:05.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:05 smithi106 ceph-mon[36505]: osdmap e49: 8 total, 7 up, 8 in 2024-03-24T00:16:05.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:05 smithi106 ceph-mon[36505]: from='osd.7 [v2:172.21.15.106:6816/4262103080,v1:172.21.15.106:6817/4262103080]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-24T00:16:05.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:05 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:16:05.903 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 00:16:05 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T00:16:05.492+0000 7f655d4fe700 -1 osd.7 0 waiting for initial osdmap 2024-03-24T00:16:05.903 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 00:16:05 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T00:16:05.503+0000 7f65592f1700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:16:06.540 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T00:16:06.540 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T00:16:06.540 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10+0 records in 2024-03-24T00:16:06.540 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2024-03-24T00:16:06.540 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115366 s, 909 MB/s 2024-03-24T00:16:06.540 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping successful for: 2024-03-24T00:16:06.835 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:06 smithi107 ceph-mon[36485]: from='osd.7 [v2:172.21.15.106:6816/4262103080,v1:172.21.15.106:6817/4262103080]' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-24T00:16:06.835 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:06 smithi107 ceph-mon[36485]: osdmap e50: 8 total, 7 up, 8 in 2024-03-24T00:16:06.835 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:06 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:16:06.835 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:06 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:16:06.835 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:06 smithi107 ceph-mon[36485]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:16:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:06 smithi078 ceph-mon[32429]: from='osd.7 [v2:172.21.15.106:6816/4262103080,v1:172.21.15.106:6817/4262103080]' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-24T00:16:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:06 smithi078 ceph-mon[32429]: osdmap e50: 8 total, 7 up, 8 in 2024-03-24T00:16:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:06 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:16:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:06 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:16:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:06 smithi078 ceph-mon[32429]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:16:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:06 smithi106 ceph-mon[36505]: from='osd.7 [v2:172.21.15.106:6816/4262103080,v1:172.21.15.106:6817/4262103080]' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-24T00:16:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:06 smithi106 ceph-mon[36505]: osdmap e50: 8 total, 7 up, 8 in 2024-03-24T00:16:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:06 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:16:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:06 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:16:06.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:06 smithi106 ceph-mon[36505]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:16:07.172 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi107:vg_nvme/lv_2 2024-03-24T00:16:07.415 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:16:07.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:07 smithi106 ceph-mon[36505]: purged_snaps scrub starts 2024-03-24T00:16:07.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:07 smithi106 ceph-mon[36505]: purged_snaps scrub ok 2024-03-24T00:16:07.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:07 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:16:07.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:07 smithi106 ceph-mon[36505]: osd.7 [v2:172.21.15.106:6816/4262103080,v1:172.21.15.106:6817/4262103080] boot 2024-03-24T00:16:07.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:07 smithi106 ceph-mon[36505]: osdmap e51: 8 total, 8 up, 8 in 2024-03-24T00:16:07.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:07 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:16:07.790 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:07 smithi107 ceph-mon[36485]: purged_snaps scrub starts 2024-03-24T00:16:07.790 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:07 smithi107 ceph-mon[36485]: purged_snaps scrub ok 2024-03-24T00:16:07.790 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:07 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:16:07.790 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:07 smithi107 ceph-mon[36485]: osd.7 [v2:172.21.15.106:6816/4262103080,v1:172.21.15.106:6817/4262103080] boot 2024-03-24T00:16:07.790 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:07 smithi107 ceph-mon[36485]: osdmap e51: 8 total, 8 up, 8 in 2024-03-24T00:16:07.790 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:07 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:16:07.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:07 smithi078 ceph-mon[32429]: purged_snaps scrub starts 2024-03-24T00:16:07.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:07 smithi078 ceph-mon[32429]: purged_snaps scrub ok 2024-03-24T00:16:07.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:07 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:16:07.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:07 smithi078 ceph-mon[32429]: osd.7 [v2:172.21.15.106:6816/4262103080,v1:172.21.15.106:6817/4262103080] boot 2024-03-24T00:16:07.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:07 smithi078 ceph-mon[32429]: osdmap e51: 8 total, 8 up, 8 in 2024-03-24T00:16:07.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:07 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:16:08.834 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:08 smithi107 ceph-mon[36485]: osdmap e52: 8 total, 8 up, 8 in 2024-03-24T00:16:08.834 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:08 smithi107 ceph-mon[36485]: Detected new or changed devices on smithi106 2024-03-24T00:16:08.834 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:08 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:08.834 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:08 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:08.835 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:08 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:16:08.835 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:08 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:16:08.835 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:08 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:16:08.835 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:08 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi106 to 5693M 2024-03-24T00:16:08.835 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:08 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:08.835 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:08 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:16:08.835 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:08 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:16:08.835 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:08 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:08.836 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:08 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:16:08.836 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:08 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:16:08.836 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:08 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:16:08.836 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:08 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:08.836 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:08 smithi107 ceph-mon[36485]: pgmap v176: 1 pgs: 1 unknown; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:08 smithi078 ceph-mon[32429]: osdmap e52: 8 total, 8 up, 8 in 2024-03-24T00:16:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:08 smithi078 ceph-mon[32429]: Detected new or changed devices on smithi106 2024-03-24T00:16:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:08 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:08 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:08 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:16:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:08 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:16:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:08 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:16:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:08 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi106 to 5693M 2024-03-24T00:16:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:08 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:08 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:16:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:08 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:16:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:08 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:08 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:16:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:08 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:16:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:08 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:16:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:08 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:08 smithi078 ceph-mon[32429]: pgmap v176: 1 pgs: 1 unknown; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:08 smithi106 ceph-mon[36505]: osdmap e52: 8 total, 8 up, 8 in 2024-03-24T00:16:08.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:08 smithi106 ceph-mon[36505]: Detected new or changed devices on smithi106 2024-03-24T00:16:08.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:08 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:08.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:08 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:08.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:08 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:16:08.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:08 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:16:08.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:08 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:16:08.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:08 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi106 to 5693M 2024-03-24T00:16:08.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:08 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:08.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:08 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:16:08.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:08 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:16:08.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:08 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:08.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:08 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:16:08.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:08 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:16:08.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:08 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:16:08.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:08 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:08.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:08 smithi106 ceph-mon[36505]: pgmap v176: 1 pgs: 1 unknown; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:09.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.072+0000 7ff082e0b700 1 -- 172.21.15.107:0/1055880070 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff07c103e40 msgr2=0x7ff07c10a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:09.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.072+0000 7ff082e0b700 1 --2- 172.21.15.107:0/1055880070 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff07c103e40 0x7ff07c10a6f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff07400b0a0 tx=0x7ff07400b3b0 comp rx=0 tx=0).stop 2024-03-24T00:16:09.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.072+0000 7ff082e0b700 1 -- 172.21.15.107:0/1055880070 shutdown_connections 2024-03-24T00:16:09.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.072+0000 7ff082e0b700 1 --2- 172.21.15.107:0/1055880070 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff07c103e40 0x7ff07c10a6f0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:09.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.072+0000 7ff082e0b700 1 --2- 172.21.15.107:0/1055880070 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff07c1004c0 0x7ff07c1008e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:09.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.072+0000 7ff082e0b700 1 --2- 172.21.15.107:0/1055880070 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff07c1034a0 0x7ff07c103900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:09.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.072+0000 7ff082e0b700 1 -- 172.21.15.107:0/1055880070 >> 172.21.15.107:0/1055880070 conn(0x7ff07c0fbe20 msgr2=0x7ff07c0fe260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:09.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.073+0000 7ff082e0b700 1 -- 172.21.15.107:0/1055880070 shutdown_connections 2024-03-24T00:16:09.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.073+0000 7ff082e0b700 1 -- 172.21.15.107:0/1055880070 wait complete. 2024-03-24T00:16:09.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.073+0000 7ff082e0b700 1 Processor -- start 2024-03-24T00:16:09.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.073+0000 7ff082e0b700 1 -- start start 2024-03-24T00:16:09.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.073+0000 7ff082e0b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff07c1004c0 0x7ff07c1983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:09.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.073+0000 7ff082e0b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff07c1034a0 0x7ff07c1988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:09.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.073+0000 7ff082e0b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff07c103e40 0x7ff07c19f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:09.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.073+0000 7ff082e0b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c10de40 con 0x7ff07c103e40 2024-03-24T00:16:09.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.073+0000 7ff082e0b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c10dcc0 con 0x7ff07c1034a0 2024-03-24T00:16:09.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.073+0000 7ff082e0b700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c10dfc0 con 0x7ff07c1004c0 2024-03-24T00:16:09.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.073+0000 7ff080ba7700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff07c1004c0 0x7ff07c1983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:09.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.073+0000 7ff080ba7700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff07c1004c0 0x7ff07c1983a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39780/0 (socket says 172.21.15.107:39780) 2024-03-24T00:16:09.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.073+0000 7ff080ba7700 1 -- 172.21.15.107:0/13945273 learned_addr learned my addr 172.21.15.107:0/13945273 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-03-24T00:16:09.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.074+0000 7ff07bfff700 1 --2- 172.21.15.107:0/13945273 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff07c1034a0 0x7ff07c1988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:09.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.074+0000 7ff0813a8700 1 --2- 172.21.15.107:0/13945273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff07c103e40 0x7ff07c19f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:09.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.074+0000 7ff080ba7700 1 -- 172.21.15.107:0/13945273 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff07c1034a0 msgr2=0x7ff07c1988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:09.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.074+0000 7ff080ba7700 1 --2- 172.21.15.107:0/13945273 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff07c1034a0 0x7ff07c1988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:09.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.074+0000 7ff080ba7700 1 -- 172.21.15.107:0/13945273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff07c103e40 msgr2=0x7ff07c19f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:09.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.074+0000 7ff080ba7700 1 --2- 172.21.15.107:0/13945273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff07c103e40 0x7ff07c19f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:09.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.074+0000 7ff080ba7700 1 -- 172.21.15.107:0/13945273 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff07c1a0000 con 0x7ff07c1004c0 2024-03-24T00:16:09.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.074+0000 7ff07bfff700 1 --2- 172.21.15.107:0/13945273 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff07c1034a0 0x7ff07c1988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:16:09.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.074+0000 7ff080ba7700 1 --2- 172.21.15.107:0/13945273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff07c1004c0 0x7ff07c1983a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff07000c9a0 tx=0x7ff07000cd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:09.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.074+0000 7ff079ffb700 1 -- 172.21.15.107:0/13945273 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff070004220 con 0x7ff07c1004c0 2024-03-24T00:16:09.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.074+0000 7ff079ffb700 1 -- 172.21.15.107:0/13945273 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff070007d40 con 0x7ff07c1004c0 2024-03-24T00:16:09.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.074+0000 7ff082e0b700 1 -- 172.21.15.107:0/13945273 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff07c1a02f0 con 0x7ff07c1004c0 2024-03-24T00:16:09.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.074+0000 7ff079ffb700 1 -- 172.21.15.107:0/13945273 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07000f450 con 0x7ff07c1004c0 2024-03-24T00:16:09.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.074+0000 7ff082e0b700 1 -- 172.21.15.107:0/13945273 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff07c1a0830 con 0x7ff07c1004c0 2024-03-24T00:16:09.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.075+0000 7ff082e0b700 1 -- 172.21.15.107:0/13945273 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff07c068e50 con 0x7ff07c1004c0 2024-03-24T00:16:09.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.076+0000 7ff079ffb700 1 -- 172.21.15.107:0/13945273 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff070007860 con 0x7ff07c1004c0 2024-03-24T00:16:09.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.077+0000 7ff079ffb700 1 --2- 172.21.15.107:0/13945273 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff06c0a2890 0x7ff06c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:09.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.077+0000 7ff079ffb700 1 -- 172.21.15.107:0/13945273 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5397+0+0 (secure 0 0 0) 0x7ff0700cde80 con 0x7ff07c1004c0 2024-03-24T00:16:09.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.077+0000 7ff07bfff700 1 --2- 172.21.15.107:0/13945273 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff06c0a2890 0x7ff06c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:09.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.077+0000 7ff07bfff700 1 --2- 172.21.15.107:0/13945273 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff06c0a2890 0x7ff06c0a4d50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff0680099d0 tx=0x7ff068008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:09.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.080+0000 7ff079ffb700 1 -- 172.21.15.107:0/13945273 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff070014030 con 0x7ff07c1004c0 2024-03-24T00:16:09.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:09.285+0000 7ff082e0b700 1 -- 172.21.15.107:0/13945273 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7ff07c0625d0 con 0x7ff06c0a2890 2024-03-24T00:16:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:09 smithi107 ceph-mon[36485]: osdmap e53: 8 total, 8 up, 8 in 2024-03-24T00:16:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:09 smithi107 ceph-mon[36485]: osdmap e54: 8 total, 8 up, 8 in 2024-03-24T00:16:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:09 smithi107 ceph-mon[36485]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:16:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:09 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:16:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:09 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:16:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:09 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:16:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:09 smithi078 ceph-mon[32429]: osdmap e53: 8 total, 8 up, 8 in 2024-03-24T00:16:09.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:09 smithi078 ceph-mon[32429]: osdmap e54: 8 total, 8 up, 8 in 2024-03-24T00:16:09.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:09 smithi078 ceph-mon[32429]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:16:09.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:09 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:16:09.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:09 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:16:09.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:09 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:16:09.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:09 smithi106 ceph-mon[36505]: osdmap e53: 8 total, 8 up, 8 in 2024-03-24T00:16:09.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:09 smithi106 ceph-mon[36505]: osdmap e54: 8 total, 8 up, 8 in 2024-03-24T00:16:09.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:09 smithi106 ceph-mon[36505]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:16:09.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:09 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:16:09.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:09 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:16:09.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:09 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:16:11.089 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:10 smithi107 ceph-mon[36485]: osdmap e55: 8 total, 8 up, 8 in 2024-03-24T00:16:11.089 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:10 smithi107 ceph-mon[36485]: pgmap v180: 1 pgs: 1 unknown; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:11.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:10 smithi078 ceph-mon[32429]: osdmap e55: 8 total, 8 up, 8 in 2024-03-24T00:16:11.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:10 smithi078 ceph-mon[32429]: pgmap v180: 1 pgs: 1 unknown; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:11.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:10 smithi106 ceph-mon[36505]: osdmap e55: 8 total, 8 up, 8 in 2024-03-24T00:16:11.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:10 smithi106 ceph-mon[36505]: pgmap v180: 1 pgs: 1 unknown; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:12.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:12 smithi078 ceph-mon[32429]: from='client.? 172.21.15.107:0/185430427' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba16039f-8093-4fa0-87ad-e28b7d142b23"}]: dispatch 2024-03-24T00:16:12.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:12 smithi078 ceph-mon[32429]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba16039f-8093-4fa0-87ad-e28b7d142b23"}]: dispatch 2024-03-24T00:16:12.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:12 smithi078 ceph-mon[32429]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba16039f-8093-4fa0-87ad-e28b7d142b23"}]': finished 2024-03-24T00:16:12.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:12 smithi078 ceph-mon[32429]: osdmap e56: 9 total, 8 up, 9 in 2024-03-24T00:16:12.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:12 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:16:12.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:12 smithi107 ceph-mon[36485]: from='client.? 172.21.15.107:0/185430427' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba16039f-8093-4fa0-87ad-e28b7d142b23"}]: dispatch 2024-03-24T00:16:12.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:12 smithi107 ceph-mon[36485]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba16039f-8093-4fa0-87ad-e28b7d142b23"}]: dispatch 2024-03-24T00:16:12.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:12 smithi107 ceph-mon[36485]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba16039f-8093-4fa0-87ad-e28b7d142b23"}]': finished 2024-03-24T00:16:12.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:12 smithi107 ceph-mon[36485]: osdmap e56: 9 total, 8 up, 9 in 2024-03-24T00:16:12.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:12 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:16:12.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:12 smithi106 ceph-mon[36505]: from='client.? 172.21.15.107:0/185430427' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba16039f-8093-4fa0-87ad-e28b7d142b23"}]: dispatch 2024-03-24T00:16:12.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:12 smithi106 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba16039f-8093-4fa0-87ad-e28b7d142b23"}]: dispatch 2024-03-24T00:16:12.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:12 smithi106 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba16039f-8093-4fa0-87ad-e28b7d142b23"}]': finished 2024-03-24T00:16:12.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:12 smithi106 ceph-mon[36505]: osdmap e56: 9 total, 8 up, 9 in 2024-03-24T00:16:12.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:12 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:16:13.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:13 smithi078 ceph-mon[32429]: pgmap v182: 1 pgs: 1 unknown; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:13.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:13 smithi078 ceph-mon[32429]: from='client.? 172.21.15.107:0/1699002385' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:16:13.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:13 smithi107 ceph-mon[36485]: pgmap v182: 1 pgs: 1 unknown; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:13.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:13 smithi107 ceph-mon[36485]: from='client.? 172.21.15.107:0/1699002385' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:16:13.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:13 smithi106 ceph-mon[36505]: pgmap v182: 1 pgs: 1 unknown; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:13.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:13 smithi106 ceph-mon[36505]: from='client.? 172.21.15.107:0/1699002385' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:16:15.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:15 smithi078 ceph-mon[32429]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:15.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:15 smithi107 ceph-mon[36485]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:15.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:15 smithi106 ceph-mon[36505]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:17.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:17 smithi078 ceph-mon[32429]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:17.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:17 smithi107 ceph-mon[36485]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:17.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:17 smithi106 ceph-mon[36505]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:19.436 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:19 smithi107 ceph-mon[36485]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:19.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:19 smithi078 ceph-mon[32429]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:19 smithi106 ceph-mon[36505]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:20.319 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:20 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-24T00:16:20.320 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:20 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:16:20.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:20 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-24T00:16:20.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:20 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:16:20.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:20 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-24T00:16:20.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:20 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:16:21.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:21 smithi107 ceph-mon[36485]: Deploying daemon osd.8 on smithi107 2024-03-24T00:16:21.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:21 smithi107 ceph-mon[36485]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:21.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:21 smithi078 ceph-mon[32429]: Deploying daemon osd.8 on smithi107 2024-03-24T00:16:21.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:21 smithi078 ceph-mon[32429]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:21.648 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:21 smithi106 ceph-mon[36505]: Deploying daemon osd.8 on smithi107 2024-03-24T00:16:21.648 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:21 smithi106 ceph-mon[36505]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:23.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:23 smithi078 ceph-mon[32429]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:23.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:23 smithi107 ceph-mon[36485]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:23.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:23 smithi106 ceph-mon[36505]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:25.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:25 smithi078 ceph-mon[32429]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:25.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:25 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:25.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:25 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:25.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:25 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:16:25.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:25 smithi107 ceph-mon[36485]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:25.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:25 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:25.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:25 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:25.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:25 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:16:25.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:25 smithi106 ceph-mon[36505]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:25.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:25 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:25.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:25 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:25.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:25 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:16:27.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:27 smithi078 ceph-mon[32429]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:27.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:27 smithi107 ceph-mon[36485]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:27.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:27 smithi106 ceph-mon[36505]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:28.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:28.956+0000 7ff079ffb700 1 -- 172.21.15.107:0/13945273 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff07c0625d0 con 0x7ff06c0a2890 2024-03-24T00:16:28.957 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 8 on host 'smithi107' 2024-03-24T00:16:28.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:28.958+0000 7ff082e0b700 1 -- 172.21.15.107:0/13945273 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff06c0a2890 msgr2=0x7ff06c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:28.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:28.958+0000 7ff082e0b700 1 --2- 172.21.15.107:0/13945273 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff06c0a2890 0x7ff06c0a4d50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff0680099d0 tx=0x7ff068008040 comp rx=0 tx=0).stop 2024-03-24T00:16:28.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:28.958+0000 7ff082e0b700 1 -- 172.21.15.107:0/13945273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff07c1004c0 msgr2=0x7ff07c1983a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:28.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:28.958+0000 7ff082e0b700 1 --2- 172.21.15.107:0/13945273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff07c1004c0 0x7ff07c1983a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff07000c9a0 tx=0x7ff07000cd60 comp rx=0 tx=0).stop 2024-03-24T00:16:28.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:28.958+0000 7ff082e0b700 1 -- 172.21.15.107:0/13945273 shutdown_connections 2024-03-24T00:16:28.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:28.958+0000 7ff082e0b700 1 --2- 172.21.15.107:0/13945273 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff06c0a2890 0x7ff06c0a4d50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:28.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:28.958+0000 7ff082e0b700 1 --2- 172.21.15.107:0/13945273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff07c103e40 0x7ff07c19f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:28.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:28.958+0000 7ff082e0b700 1 --2- 172.21.15.107:0/13945273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff07c1004c0 0x7ff07c1983a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:28.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:28.958+0000 7ff082e0b700 1 --2- 172.21.15.107:0/13945273 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff07c1034a0 0x7ff07c1988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:28.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:28.958+0000 7ff082e0b700 1 -- 172.21.15.107:0/13945273 >> 172.21.15.107:0/13945273 conn(0x7ff07c0fbe20 msgr2=0x7ff07c1066d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:28.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:28.958+0000 7ff082e0b700 1 -- 172.21.15.107:0/13945273 shutdown_connections 2024-03-24T00:16:28.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:16:28.959+0000 7ff082e0b700 1 -- 172.21.15.107:0/13945273 wait complete. 2024-03-24T00:16:29.328 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:29 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:29.329 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:29 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:29.329 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:29 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:16:29.329 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:29 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:16:29.329 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:29 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:29.329 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:29 smithi107 ceph-mon[36485]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:29.329 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:29 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:29.329 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:29 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:29.329 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:29 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:16:29.329 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:29 smithi107 ceph-mon[36485]: from='osd.8 [v2:172.21.15.107:6816/4032799356,v1:172.21.15.107:6817/4032799356]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T00:16:29.329 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:29 smithi107 ceph-mon[36485]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T00:16:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:29 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:29 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:29 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:16:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:29 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:16:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:29 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:29 smithi078 ceph-mon[32429]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:29 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:29 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:29 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:16:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:29 smithi078 ceph-mon[32429]: from='osd.8 [v2:172.21.15.107:6816/4032799356,v1:172.21.15.107:6817/4032799356]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T00:16:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:29 smithi078 ceph-mon[32429]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T00:16:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:29 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:29 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:29 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:16:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:29 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:16:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:29 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:29 smithi106 ceph-mon[36505]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:29 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:29.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:29 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:29.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:29 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:16:29.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:29 smithi106 ceph-mon[36505]: from='osd.8 [v2:172.21.15.107:6816/4032799356,v1:172.21.15.107:6817/4032799356]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T00:16:29.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:29 smithi106 ceph-mon[36505]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T00:16:29.694 DEBUG:teuthology.orchestra.run.smithi107:osd.8> sudo journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.8.service 2024-03-24T00:16:29.696 INFO:tasks.cephadm:Deploying osd.9 on smithi078 with /dev/vg_nvme/lv_1... 2024-03-24T00:16:29.696 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-24T00:16:29.950 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:16:31.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:30 smithi078 ceph-mon[32429]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-24T00:16:31.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:30 smithi078 ceph-mon[32429]: osdmap e57: 9 total, 8 up, 9 in 2024-03-24T00:16:31.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:30 smithi078 ceph-mon[32429]: from='osd.8 [v2:172.21.15.107:6816/4032799356,v1:172.21.15.107:6817/4032799356]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-24T00:16:31.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:30 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:16:31.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:30 smithi078 ceph-mon[32429]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-24T00:16:31.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:30 smithi078 ceph-mon[32429]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:31.249 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:30 smithi107 ceph-mon[36485]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-24T00:16:31.249 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:30 smithi107 ceph-mon[36485]: osdmap e57: 9 total, 8 up, 9 in 2024-03-24T00:16:31.249 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:30 smithi107 ceph-mon[36485]: from='osd.8 [v2:172.21.15.107:6816/4032799356,v1:172.21.15.107:6817/4032799356]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-24T00:16:31.249 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:30 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:16:31.249 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:30 smithi107 ceph-mon[36485]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-24T00:16:31.249 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:30 smithi107 ceph-mon[36485]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:31.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:30 smithi106 ceph-mon[36505]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-24T00:16:31.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:30 smithi106 ceph-mon[36505]: osdmap e57: 9 total, 8 up, 9 in 2024-03-24T00:16:31.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:30 smithi106 ceph-mon[36505]: from='osd.8 [v2:172.21.15.107:6816/4032799356,v1:172.21.15.107:6817/4032799356]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-24T00:16:31.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:30 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:16:31.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:30 smithi106 ceph-mon[36505]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-24T00:16:31.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:30 smithi106 ceph-mon[36505]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:16:31.951 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 00:16:31 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T00:16:31.624+0000 7f9d5e9ec700 -1 osd.8 0 waiting for initial osdmap 2024-03-24T00:16:31.951 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 00:16:31 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T00:16:31.636+0000 7f9d567d7700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:16:32.027 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-24T00:16:32.027 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-24T00:16:32.027 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10+0 records in 2024-03-24T00:16:32.027 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2024-03-24T00:16:32.027 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115516 s, 908 MB/s 2024-03-24T00:16:32.027 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping successful for: 2024-03-24T00:16:32.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:31 smithi106 ceph-mon[36505]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-03-24T00:16:32.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:31 smithi106 ceph-mon[36505]: osdmap e58: 9 total, 8 up, 9 in 2024-03-24T00:16:32.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:31 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:16:32.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:31 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:16:32.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:31 smithi106 ceph-mon[36505]: from='osd.8 ' entity='osd.8' 2024-03-24T00:16:32.331 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:31 smithi078 ceph-mon[32429]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-03-24T00:16:32.332 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:31 smithi078 ceph-mon[32429]: osdmap e58: 9 total, 8 up, 9 in 2024-03-24T00:16:32.332 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:31 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:16:32.332 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:31 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:16:32.332 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:31 smithi078 ceph-mon[32429]: from='osd.8 ' entity='osd.8' 2024-03-24T00:16:32.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:31 smithi107 ceph-mon[36485]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-03-24T00:16:32.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:31 smithi107 ceph-mon[36485]: osdmap e58: 9 total, 8 up, 9 in 2024-03-24T00:16:32.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:31 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:16:32.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:31 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:16:32.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:31 smithi107 ceph-mon[36485]: from='osd.8 ' entity='osd.8' 2024-03-24T00:16:32.765 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi078:vg_nvme/lv_1 2024-03-24T00:16:33.024 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:16:33.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:32 smithi078 ceph-mon[32429]: purged_snaps scrub starts 2024-03-24T00:16:33.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:32 smithi078 ceph-mon[32429]: purged_snaps scrub ok 2024-03-24T00:16:33.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:32 smithi078 ceph-mon[32429]: osd.8 [v2:172.21.15.107:6816/4032799356,v1:172.21.15.107:6817/4032799356] boot 2024-03-24T00:16:33.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:32 smithi078 ceph-mon[32429]: osdmap e59: 9 total, 9 up, 9 in 2024-03-24T00:16:33.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:32 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:16:33.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:32 smithi078 ceph-mon[32429]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:16:33.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:32 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:33.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:32 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:33.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:32 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:16:33.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:32 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:16:33.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:32 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:16:33.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:32 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:33.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:32 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:16:33.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:32 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:16:33.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:32 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:33.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:32 smithi078 ceph-mon[32429]: osdmap e60: 9 total, 9 up, 9 in 2024-03-24T00:16:33.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:32 smithi107 ceph-mon[36485]: purged_snaps scrub starts 2024-03-24T00:16:33.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:32 smithi107 ceph-mon[36485]: purged_snaps scrub ok 2024-03-24T00:16:33.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:32 smithi107 ceph-mon[36485]: osd.8 [v2:172.21.15.107:6816/4032799356,v1:172.21.15.107:6817/4032799356] boot 2024-03-24T00:16:33.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:32 smithi107 ceph-mon[36485]: osdmap e59: 9 total, 9 up, 9 in 2024-03-24T00:16:33.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:32 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:16:33.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:32 smithi107 ceph-mon[36485]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:16:33.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:32 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:33.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:32 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:33.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:32 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:16:33.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:32 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:16:33.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:32 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:16:33.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:32 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:33.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:32 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:16:33.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:32 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:16:33.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:32 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:33.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:32 smithi107 ceph-mon[36485]: osdmap e60: 9 total, 9 up, 9 in 2024-03-24T00:16:33.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:32 smithi106 ceph-mon[36505]: purged_snaps scrub starts 2024-03-24T00:16:33.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:32 smithi106 ceph-mon[36505]: purged_snaps scrub ok 2024-03-24T00:16:33.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:32 smithi106 ceph-mon[36505]: osd.8 [v2:172.21.15.107:6816/4032799356,v1:172.21.15.107:6817/4032799356] boot 2024-03-24T00:16:33.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:32 smithi106 ceph-mon[36505]: osdmap e59: 9 total, 9 up, 9 in 2024-03-24T00:16:33.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:32 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:16:33.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:32 smithi106 ceph-mon[36505]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:16:33.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:32 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:33.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:32 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:33.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:32 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:16:33.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:32 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:16:33.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:32 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:16:33.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:32 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:33.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:32 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:16:33.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:32 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:16:33.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:32 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:33.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:32 smithi106 ceph-mon[36505]: osdmap e60: 9 total, 9 up, 9 in 2024-03-24T00:16:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:33 smithi078 ceph-mon[32429]: Detected new or changed devices on smithi107 2024-03-24T00:16:34.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:33 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi107 to 5693M 2024-03-24T00:16:34.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:33 smithi078 ceph-mon[32429]: osdmap e61: 9 total, 9 up, 9 in 2024-03-24T00:16:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:33 smithi107 ceph-mon[36485]: Detected new or changed devices on smithi107 2024-03-24T00:16:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:33 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi107 to 5693M 2024-03-24T00:16:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:33 smithi107 ceph-mon[36485]: osdmap e61: 9 total, 9 up, 9 in 2024-03-24T00:16:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:33 smithi106 ceph-mon[36505]: Detected new or changed devices on smithi107 2024-03-24T00:16:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:33 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi107 to 5693M 2024-03-24T00:16:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:33 smithi106 ceph-mon[36505]: osdmap e61: 9 total, 9 up, 9 in 2024-03-24T00:16:34.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.773+0000 7f17cd3fd700 1 -- 172.21.15.78:0/2616629775 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17c8102ce0 msgr2=0x7f17c81050d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:34.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.773+0000 7f17cd3fd700 1 --2- 172.21.15.78:0/2616629775 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17c8102ce0 0x7f17c81050d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f17b8009990 tx=0x7f17b8009ca0 comp rx=0 tx=0).stop 2024-03-24T00:16:34.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.773+0000 7f17cd3fd700 1 -- 172.21.15.78:0/2616629775 shutdown_connections 2024-03-24T00:16:34.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.773+0000 7f17cd3fd700 1 --2- 172.21.15.78:0/2616629775 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17c8107f40 0x7f17c810a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:34.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.773+0000 7f17cd3fd700 1 --2- 172.21.15.78:0/2616629775 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17c8102ce0 0x7f17c81050d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:34.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.773+0000 7f17cd3fd700 1 --2- 172.21.15.78:0/2616629775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17c8105610 0x7f17c8107a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:34.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.773+0000 7f17cd3fd700 1 -- 172.21.15.78:0/2616629775 >> 172.21.15.78:0/2616629775 conn(0x7f17c80fc690 msgr2=0x7f17c80fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:34.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.773+0000 7f17cd3fd700 1 -- 172.21.15.78:0/2616629775 shutdown_connections 2024-03-24T00:16:34.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.773+0000 7f17cd3fd700 1 -- 172.21.15.78:0/2616629775 wait complete. 2024-03-24T00:16:34.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.774+0000 7f17cd3fd700 1 Processor -- start 2024-03-24T00:16:34.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.774+0000 7f17cd3fd700 1 -- start start 2024-03-24T00:16:34.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.774+0000 7f17cd3fd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17c8102ce0 0x7f17c81983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:34.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.774+0000 7f17cd3fd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17c8105610 0x7f17c8198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:34.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.774+0000 7f17cd3fd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17c8107f40 0x7f17c819f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:34.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.774+0000 7f17cd3fd700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c810de00 con 0x7f17c8107f40 2024-03-24T00:16:34.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.774+0000 7f17cd3fd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c810dc80 con 0x7f17c8102ce0 2024-03-24T00:16:34.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.774+0000 7f17cd3fd700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c810df80 con 0x7f17c8105610 2024-03-24T00:16:34.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.774+0000 7f17c77fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17c8107f40 0x7f17c819f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:34.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.774+0000 7f17c77fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17c8107f40 0x7f17c819f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37778/0 (socket says 172.21.15.78:37778) 2024-03-24T00:16:34.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.774+0000 7f17c77fe700 1 -- 172.21.15.78:0/1868122807 learned_addr learned my addr 172.21.15.78:0/1868122807 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:16:34.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.774+0000 7f17c6ffd700 1 --2- 172.21.15.78:0/1868122807 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17c8102ce0 0x7f17c81983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:34.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.774+0000 7f17be7fc700 1 --2- 172.21.15.78:0/1868122807 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17c8105610 0x7f17c8198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:34.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.775+0000 7f17c77fe700 1 -- 172.21.15.78:0/1868122807 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17c8105610 msgr2=0x7f17c8198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:34.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.775+0000 7f17c77fe700 1 --2- 172.21.15.78:0/1868122807 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17c8105610 0x7f17c8198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:34.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.775+0000 7f17c77fe700 1 -- 172.21.15.78:0/1868122807 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17c8102ce0 msgr2=0x7f17c81983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:34.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.775+0000 7f17c77fe700 1 --2- 172.21.15.78:0/1868122807 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17c8102ce0 0x7f17c81983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:34.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.775+0000 7f17c77fe700 1 -- 172.21.15.78:0/1868122807 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c81a0050 con 0x7f17c8107f40 2024-03-24T00:16:34.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.775+0000 7f17c77fe700 1 --2- 172.21.15.78:0/1868122807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17c8107f40 0x7f17c819f9b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f17c0007ee0 tx=0x7f17c000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:34.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.775+0000 7f17c4ff9700 1 -- 172.21.15.78:0/1868122807 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c0004db0 con 0x7f17c8107f40 2024-03-24T00:16:34.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.775+0000 7f17c4ff9700 1 -- 172.21.15.78:0/1868122807 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f17c0005020 con 0x7f17c8107f40 2024-03-24T00:16:34.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.775+0000 7f17cd3fd700 1 -- 172.21.15.78:0/1868122807 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c81a0340 con 0x7f17c8107f40 2024-03-24T00:16:34.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.775+0000 7f17c4ff9700 1 -- 172.21.15.78:0/1868122807 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c0015660 con 0x7f17c8107f40 2024-03-24T00:16:34.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.775+0000 7f17cd3fd700 1 -- 172.21.15.78:0/1868122807 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c81a0880 con 0x7f17c8107f40 2024-03-24T00:16:34.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.776+0000 7f17cd3fd700 1 -- 172.21.15.78:0/1868122807 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17c8068e50 con 0x7f17c8107f40 2024-03-24T00:16:34.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.777+0000 7f17c4ff9700 1 -- 172.21.15.78:0/1868122807 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f17c0005560 con 0x7f17c8107f40 2024-03-24T00:16:34.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.778+0000 7f17c4ff9700 1 --2- 172.21.15.78:0/1868122807 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f17b40a2910 0x7f17b40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:34.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.778+0000 7f17c6ffd700 1 --2- 172.21.15.78:0/1868122807 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f17b40a2910 0x7f17b40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:34.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.778+0000 7f17c4ff9700 1 -- 172.21.15.78:0/1868122807 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5858+0+0 (secure 0 0 0) 0x7f17c00ce240 con 0x7f17c8107f40 2024-03-24T00:16:34.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.778+0000 7f17c6ffd700 1 --2- 172.21.15.78:0/1868122807 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f17b40a2910 0x7f17b40a4dd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f17b800b2e0 tx=0x7f17b8005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:34.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.781+0000 7f17c4ff9700 1 -- 172.21.15.78:0/1868122807 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f17c0010070 con 0x7f17c8107f40 2024-03-24T00:16:34.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:34.994+0000 7f17cd3fd700 1 -- 172.21.15.78:0/1868122807 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f17c80625d0 con 0x7f17b40a2910 2024-03-24T00:16:35.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:34 smithi078 ceph-mon[32429]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 376 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:16:35.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:34 smithi107 ceph-mon[36485]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 376 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:16:35.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:34 smithi106 ceph-mon[36505]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 376 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:16:36.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:35 smithi078 ceph-mon[32429]: osdmap e62: 9 total, 9 up, 9 in 2024-03-24T00:16:36.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:35 smithi078 ceph-mon[32429]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:16:36.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:35 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:16:36.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:35 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:16:36.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:35 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:16:36.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:35 smithi107 ceph-mon[36485]: osdmap e62: 9 total, 9 up, 9 in 2024-03-24T00:16:36.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:35 smithi107 ceph-mon[36485]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:16:36.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:35 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:16:36.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:35 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:16:36.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:35 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:16:36.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:35 smithi106 ceph-mon[36505]: osdmap e62: 9 total, 9 up, 9 in 2024-03-24T00:16:36.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:35 smithi106 ceph-mon[36505]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:16:36.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:35 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:16:36.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:35 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:16:36.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:35 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:16:37.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:36 smithi078 ceph-mon[32429]: osdmap e63: 9 total, 9 up, 9 in 2024-03-24T00:16:37.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:36 smithi078 ceph-mon[32429]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 376 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:16:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:36 smithi107 ceph-mon[36485]: osdmap e63: 9 total, 9 up, 9 in 2024-03-24T00:16:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:36 smithi107 ceph-mon[36485]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 376 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:16:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:36 smithi106 ceph-mon[36505]: osdmap e63: 9 total, 9 up, 9 in 2024-03-24T00:16:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:36 smithi106 ceph-mon[36505]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 376 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:16:38.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:37 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2648932240' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "651357f2-0669-42e6-aa91-1d5c02d00c5c"}]: dispatch 2024-03-24T00:16:38.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:37 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2648932240' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "651357f2-0669-42e6-aa91-1d5c02d00c5c"}]': finished 2024-03-24T00:16:38.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:37 smithi078 ceph-mon[32429]: osdmap e64: 10 total, 9 up, 10 in 2024-03-24T00:16:38.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:37 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:16:38.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:37 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2648932240' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "651357f2-0669-42e6-aa91-1d5c02d00c5c"}]: dispatch 2024-03-24T00:16:38.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:37 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2648932240' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "651357f2-0669-42e6-aa91-1d5c02d00c5c"}]': finished 2024-03-24T00:16:38.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:37 smithi107 ceph-mon[36485]: osdmap e64: 10 total, 9 up, 10 in 2024-03-24T00:16:38.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:37 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:16:38.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:37 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2648932240' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "651357f2-0669-42e6-aa91-1d5c02d00c5c"}]: dispatch 2024-03-24T00:16:38.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:37 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2648932240' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "651357f2-0669-42e6-aa91-1d5c02d00c5c"}]': finished 2024-03-24T00:16:38.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:37 smithi106 ceph-mon[36505]: osdmap e64: 10 total, 9 up, 10 in 2024-03-24T00:16:38.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:37 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:16:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:38 smithi078 ceph-mon[32429]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-03-24T00:16:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:38 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/769211758' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:16:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:38 smithi107 ceph-mon[36485]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-03-24T00:16:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:38 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/769211758' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:16:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:38 smithi106 ceph-mon[36505]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-03-24T00:16:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:38 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/769211758' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:16:41.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:41 smithi078 ceph-mon[32429]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T00:16:41.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:41 smithi107 ceph-mon[36485]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T00:16:41.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:41 smithi106 ceph-mon[36505]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T00:16:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:43 smithi078 ceph-mon[32429]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-24T00:16:43.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:43 smithi107 ceph-mon[36485]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-24T00:16:43.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:43 smithi106 ceph-mon[36505]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-24T00:16:45.583 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:45 smithi078 ceph-mon[32429]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-24T00:16:45.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:45 smithi107 ceph-mon[36485]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-24T00:16:45.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:45 smithi106 ceph-mon[36505]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-24T00:16:46.482 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:46 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-24T00:16:46.482 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:46 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:16:46.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:46 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-24T00:16:46.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:46 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:16:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:46 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-24T00:16:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:46 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:16:47.556 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:47 smithi078 ceph-mon[32429]: Deploying daemon osd.9 on smithi078 2024-03-24T00:16:47.556 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:47 smithi078 ceph-mon[32429]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T00:16:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:47 smithi107 ceph-mon[36485]: Deploying daemon osd.9 on smithi078 2024-03-24T00:16:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:47 smithi107 ceph-mon[36485]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T00:16:47.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:47 smithi106 ceph-mon[36505]: Deploying daemon osd.9 on smithi078 2024-03-24T00:16:47.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:47 smithi106 ceph-mon[36505]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T00:16:49.492 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:49 smithi078 ceph-mon[32429]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:16:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:49 smithi107 ceph-mon[36485]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:16:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:49 smithi106 ceph-mon[36505]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:16:51.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:51 smithi078 ceph-mon[32429]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:16:51.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:51 smithi107 ceph-mon[36485]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:16:51.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:51 smithi106 ceph-mon[36505]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:16:52.667 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:52 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:52.667 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:52 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:52.667 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:52 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:16:52.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:52 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:52.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:52 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:52.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:52 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:16:52.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:52 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:52.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:52 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:52.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:52 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:16:53.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:53 smithi078 ceph-mon[32429]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:16:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:53 smithi107 ceph-mon[36485]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:16:53.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:53 smithi106 ceph-mon[36505]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:16:54.752 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:54 smithi078 ceph-mon[32429]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:16:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:54 smithi107 ceph-mon[36485]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:16:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:54 smithi106 ceph-mon[36505]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:16:55.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:55.252+0000 7f17c4ff9700 1 -- 172.21.15.78:0/1868122807 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f17c80625d0 con 0x7f17b40a2910 2024-03-24T00:16:55.253 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 9 on host 'smithi078' 2024-03-24T00:16:55.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:55.255+0000 7f17cd3fd700 1 -- 172.21.15.78:0/1868122807 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f17b40a2910 msgr2=0x7f17b40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:55.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:55.255+0000 7f17cd3fd700 1 --2- 172.21.15.78:0/1868122807 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f17b40a2910 0x7f17b40a4dd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f17b800b2e0 tx=0x7f17b8005fd0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:55.255+0000 7f17cd3fd700 1 -- 172.21.15.78:0/1868122807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17c8107f40 msgr2=0x7f17c819f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:55.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:55.255+0000 7f17cd3fd700 1 --2- 172.21.15.78:0/1868122807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17c8107f40 0x7f17c819f9b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f17c0007ee0 tx=0x7f17c000d3b0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:55.255+0000 7f17cd3fd700 1 -- 172.21.15.78:0/1868122807 shutdown_connections 2024-03-24T00:16:55.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:55.255+0000 7f17cd3fd700 1 --2- 172.21.15.78:0/1868122807 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f17b40a2910 0x7f17b40a4dd0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:55.255+0000 7f17cd3fd700 1 --2- 172.21.15.78:0/1868122807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17c8107f40 0x7f17c819f9b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:55.255+0000 7f17cd3fd700 1 --2- 172.21.15.78:0/1868122807 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17c8102ce0 0x7f17c81983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:55.255+0000 7f17cd3fd700 1 --2- 172.21.15.78:0/1868122807 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17c8105610 0x7f17c8198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:55.255+0000 7f17cd3fd700 1 -- 172.21.15.78:0/1868122807 >> 172.21.15.78:0/1868122807 conn(0x7f17c80fc690 msgr2=0x7f17c8103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:55.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:55.255+0000 7f17cd3fd700 1 -- 172.21.15.78:0/1868122807 shutdown_connections 2024-03-24T00:16:55.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:16:55.255+0000 7f17cd3fd700 1 -- 172.21.15.78:0/1868122807 wait complete. 2024-03-24T00:16:55.925 DEBUG:teuthology.orchestra.run.smithi078:osd.9> sudo journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.9.service 2024-03-24T00:16:55.927 INFO:tasks.cephadm:Deploying osd.10 on smithi106 with /dev/vg_nvme/lv_1... 2024-03-24T00:16:55.927 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-24T00:16:56.174 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T00:16:56.533 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:56 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:56.533 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:56 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:56.533 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:56 smithi106 ceph-mon[36505]: from='osd.9 [v2:172.21.15.78:6826/214023431,v1:172.21.15.78:6827/214023431]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-24T00:16:56.533 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:56 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:56.533 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:56 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:56.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:56 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:56.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:56 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:56.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:56 smithi078 ceph-mon[32429]: from='osd.9 [v2:172.21.15.78:6826/214023431,v1:172.21.15.78:6827/214023431]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-24T00:16:56.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:56 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:56.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:56 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:56.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:56 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:56.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:56 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:56.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:56 smithi107 ceph-mon[36485]: from='osd.9 [v2:172.21.15.78:6826/214023431,v1:172.21.15.78:6827/214023431]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-24T00:16:56.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:56 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:56.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:56 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:16:57.488 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:57 smithi106 ceph-mon[36505]: from='osd.9 [v2:172.21.15.78:6826/214023431,v1:172.21.15.78:6827/214023431]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-24T00:16:57.488 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:57 smithi106 ceph-mon[36505]: osdmap e65: 10 total, 9 up, 10 in 2024-03-24T00:16:57.488 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:57 smithi106 ceph-mon[36505]: from='osd.9 [v2:172.21.15.78:6826/214023431,v1:172.21.15.78:6827/214023431]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-24T00:16:57.488 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:57 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:16:57.488 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:57 smithi106 ceph-mon[36505]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:16:57.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:57 smithi078 ceph-mon[32429]: from='osd.9 [v2:172.21.15.78:6826/214023431,v1:172.21.15.78:6827/214023431]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-24T00:16:57.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:57 smithi078 ceph-mon[32429]: osdmap e65: 10 total, 9 up, 10 in 2024-03-24T00:16:57.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:57 smithi078 ceph-mon[32429]: from='osd.9 [v2:172.21.15.78:6826/214023431,v1:172.21.15.78:6827/214023431]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-24T00:16:57.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:57 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:16:57.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:57 smithi078 ceph-mon[32429]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:16:57.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:57 smithi107 ceph-mon[36485]: from='osd.9 [v2:172.21.15.78:6826/214023431,v1:172.21.15.78:6827/214023431]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-24T00:16:57.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:57 smithi107 ceph-mon[36485]: osdmap e65: 10 total, 9 up, 10 in 2024-03-24T00:16:57.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:57 smithi107 ceph-mon[36485]: from='osd.9 [v2:172.21.15.78:6826/214023431,v1:172.21.15.78:6827/214023431]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-24T00:16:57.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:57 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:16:57.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:57 smithi107 ceph-mon[36485]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:16:58.051 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 00:16:57 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T00:16:57.928+0000 7f08a1ccf700 -1 osd.9 0 waiting for initial osdmap 2024-03-24T00:16:58.051 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 00:16:57 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T00:16:57.941+0000 7f0899aba700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:16:58.189 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-24T00:16:58.189 INFO:teuthology.orchestra.run.smithi106.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-24T00:16:58.190 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10+0 records in 2024-03-24T00:16:58.190 INFO:teuthology.orchestra.run.smithi106.stderr:10+0 records out 2024-03-24T00:16:58.190 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111992 s, 936 MB/s 2024-03-24T00:16:58.190 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping successful for: 2024-03-24T00:16:58.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:58 smithi078 ceph-mon[32429]: from='osd.9 [v2:172.21.15.78:6826/214023431,v1:172.21.15.78:6827/214023431]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-24T00:16:58.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:58 smithi078 ceph-mon[32429]: osdmap e66: 10 total, 9 up, 10 in 2024-03-24T00:16:58.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:58 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:16:58.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:58 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:16:58.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:58 smithi078 ceph-mon[32429]: from='osd.9 [v2:172.21.15.78:6826/214023431,v1:172.21.15.78:6827/214023431]' entity='osd.9' 2024-03-24T00:16:58.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:58 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:16:58.499 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:58 smithi106 ceph-mon[36505]: from='osd.9 [v2:172.21.15.78:6826/214023431,v1:172.21.15.78:6827/214023431]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-24T00:16:58.499 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:58 smithi106 ceph-mon[36505]: osdmap e66: 10 total, 9 up, 10 in 2024-03-24T00:16:58.499 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:58 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:16:58.499 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:58 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:16:58.499 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:58 smithi106 ceph-mon[36505]: from='osd.9 [v2:172.21.15.78:6826/214023431,v1:172.21.15.78:6827/214023431]' entity='osd.9' 2024-03-24T00:16:58.499 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:58 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:16:58.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:58 smithi107 ceph-mon[36485]: from='osd.9 [v2:172.21.15.78:6826/214023431,v1:172.21.15.78:6827/214023431]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-24T00:16:58.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:58 smithi107 ceph-mon[36485]: osdmap e66: 10 total, 9 up, 10 in 2024-03-24T00:16:58.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:58 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:16:58.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:58 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:16:58.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:58 smithi107 ceph-mon[36485]: from='osd.9 [v2:172.21.15.78:6826/214023431,v1:172.21.15.78:6827/214023431]' entity='osd.9' 2024-03-24T00:16:58.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:58 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:16:58.987 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi106:vg_nvme/lv_1 2024-03-24T00:16:59.228 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T00:16:59.450 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:59 smithi078 ceph-mon[32429]: purged_snaps scrub starts 2024-03-24T00:16:59.450 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:59 smithi078 ceph-mon[32429]: purged_snaps scrub ok 2024-03-24T00:16:59.450 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:59 smithi078 ceph-mon[32429]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:16:59.450 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:59 smithi078 ceph-mon[32429]: osd.9 [v2:172.21.15.78:6826/214023431,v1:172.21.15.78:6827/214023431] boot 2024-03-24T00:16:59.450 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:59 smithi078 ceph-mon[32429]: osdmap e67: 10 total, 10 up, 10 in 2024-03-24T00:16:59.450 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:16:59 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:16:59.508 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:59 smithi106 ceph-mon[36505]: purged_snaps scrub starts 2024-03-24T00:16:59.508 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:59 smithi106 ceph-mon[36505]: purged_snaps scrub ok 2024-03-24T00:16:59.508 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:59 smithi106 ceph-mon[36505]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:16:59.508 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:59 smithi106 ceph-mon[36505]: osd.9 [v2:172.21.15.78:6826/214023431,v1:172.21.15.78:6827/214023431] boot 2024-03-24T00:16:59.508 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:59 smithi106 ceph-mon[36505]: osdmap e67: 10 total, 10 up, 10 in 2024-03-24T00:16:59.509 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:16:59 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:16:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:59 smithi107 ceph-mon[36485]: purged_snaps scrub starts 2024-03-24T00:16:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:59 smithi107 ceph-mon[36485]: purged_snaps scrub ok 2024-03-24T00:16:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:59 smithi107 ceph-mon[36485]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:16:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:59 smithi107 ceph-mon[36485]: osd.9 [v2:172.21.15.78:6826/214023431,v1:172.21.15.78:6827/214023431] boot 2024-03-24T00:16:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:59 smithi107 ceph-mon[36485]: osdmap e67: 10 total, 10 up, 10 in 2024-03-24T00:16:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:16:59 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:17:00.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.904+0000 7f591e254700 1 -- 172.21.15.106:0/2879106337 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f59181055e0 msgr2=0x7f59181079d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:00.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.904+0000 7f591e254700 1 --2- 172.21.15.106:0/2879106337 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f59181055e0 0x7f59181079d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f590c009990 tx=0x7f590c009ca0 comp rx=0 tx=0).stop 2024-03-24T00:17:00.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.904+0000 7f591e254700 1 -- 172.21.15.106:0/2879106337 shutdown_connections 2024-03-24T00:17:00.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.904+0000 7f591e254700 1 --2- 172.21.15.106:0/2879106337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5918107f10 0x7f591810a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:00.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.904+0000 7f591e254700 1 --2- 172.21.15.106:0/2879106337 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5918102cb0 0x7f59181050a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:00.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.904+0000 7f591e254700 1 --2- 172.21.15.106:0/2879106337 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f59181055e0 0x7f59181079d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:00.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.904+0000 7f591e254700 1 -- 172.21.15.106:0/2879106337 >> 172.21.15.106:0/2879106337 conn(0x7f59180fc680 msgr2=0x7f59180feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:00.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.904+0000 7f591e254700 1 -- 172.21.15.106:0/2879106337 shutdown_connections 2024-03-24T00:17:00.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.904+0000 7f591e254700 1 -- 172.21.15.106:0/2879106337 wait complete. 2024-03-24T00:17:00.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.904+0000 7f591e254700 1 Processor -- start 2024-03-24T00:17:00.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.904+0000 7f591e254700 1 -- start start 2024-03-24T00:17:00.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.905+0000 7f591e254700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5918102cb0 0x7f5918198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:00.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.905+0000 7f591e254700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f59181055e0 0x7f5918198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:00.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.905+0000 7f591e254700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5918107f10 0x7f591819f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:00.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.905+0000 7f591e254700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591810ddd0 con 0x7f5918107f10 2024-03-24T00:17:00.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.905+0000 7f591e254700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591810dc50 con 0x7f59181055e0 2024-03-24T00:17:00.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.905+0000 7f591e254700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591810df50 con 0x7f5918102cb0 2024-03-24T00:17:00.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.905+0000 7f5916ffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f59181055e0 0x7f5918198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:00.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.905+0000 7f5916ffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f59181055e0 0x7f5918198960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:60814/0 (socket says 172.21.15.106:60814) 2024-03-24T00:17:00.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.905+0000 7f5916ffd700 1 -- 172.21.15.106:0/2342879686 learned_addr learned my addr 172.21.15.106:0/2342879686 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-24T00:17:00.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.905+0000 7f5917fff700 1 --2- 172.21.15.106:0/2342879686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5918107f10 0x7f591819f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:00.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.905+0000 7f59177fe700 1 --2- 172.21.15.106:0/2342879686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5918102cb0 0x7f5918198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:00.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.905+0000 7f5916ffd700 1 -- 172.21.15.106:0/2342879686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5918102cb0 msgr2=0x7f5918198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:00.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.905+0000 7f5916ffd700 1 --2- 172.21.15.106:0/2342879686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5918102cb0 0x7f5918198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:00.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.905+0000 7f5916ffd700 1 -- 172.21.15.106:0/2342879686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5918107f10 msgr2=0x7f591819f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:00.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.905+0000 7f5916ffd700 1 --2- 172.21.15.106:0/2342879686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5918107f10 0x7f591819f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:00.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.905+0000 7f5916ffd700 1 -- 172.21.15.106:0/2342879686 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59181a0030 con 0x7f59181055e0 2024-03-24T00:17:00.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.905+0000 7f59177fe700 1 --2- 172.21.15.106:0/2342879686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5918102cb0 0x7f5918198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:17:00.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.905+0000 7f5916ffd700 1 --2- 172.21.15.106:0/2342879686 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f59181055e0 0x7f5918198960 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f590c005910 tx=0x7f590c00bbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:00.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.905+0000 7f5914ff9700 1 -- 172.21.15.106:0/2342879686 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590c026070 con 0x7f59181055e0 2024-03-24T00:17:00.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.905+0000 7f5914ff9700 1 -- 172.21.15.106:0/2342879686 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f590c00f460 con 0x7f59181055e0 2024-03-24T00:17:00.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.905+0000 7f591e254700 1 -- 172.21.15.106:0/2342879686 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59181a02c0 con 0x7f59181055e0 2024-03-24T00:17:00.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.905+0000 7f5914ff9700 1 -- 172.21.15.106:0/2342879686 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590c021670 con 0x7f59181055e0 2024-03-24T00:17:00.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.905+0000 7f591e254700 1 -- 172.21.15.106:0/2342879686 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59181a0770 con 0x7f59181055e0 2024-03-24T00:17:00.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.906+0000 7f591e254700 1 -- 172.21.15.106:0/2342879686 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5918068e50 con 0x7f59181055e0 2024-03-24T00:17:00.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.907+0000 7f5914ff9700 1 -- 172.21.15.106:0/2342879686 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f590c029070 con 0x7f59181055e0 2024-03-24T00:17:00.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.908+0000 7f5914ff9700 1 --2- 172.21.15.106:0/2342879686 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f59000a2910 0x7f59000a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:00.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.908+0000 7f5914ff9700 1 -- 172.21.15.106:0/2342879686 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f590c0cf290 con 0x7f59181055e0 2024-03-24T00:17:00.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.908+0000 7f59177fe700 1 --2- 172.21.15.106:0/2342879686 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f59000a2910 0x7f59000a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:00.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.909+0000 7f59177fe700 1 --2- 172.21.15.106:0/2342879686 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f59000a2910 0x7f59000a4dd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5908005fd0 tx=0x7f5908005ee0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:00.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:00.911+0000 7f5914ff9700 1 -- 172.21.15.106:0/2342879686 <== mon.2 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f590c030080 con 0x7f59181055e0 2024-03-24T00:17:01.114 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:01.114+0000 7f591e254700 1 -- 172.21.15.106:0/2342879686 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5918062690 con 0x7f59000a2910 2024-03-24T00:17:01.114 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:00 smithi106 ceph-mon[36505]: Detected new or changed devices on smithi078 2024-03-24T00:17:01.114 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:00 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:01.114 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:00 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:01.114 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:00 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:01.114 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:00 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:01.114 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:00 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:01.114 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:00 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:01.115 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:00 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi078 to 4272M 2024-03-24T00:17:01.115 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:00 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:01.115 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:00 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:17:01.115 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:00 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:17:01.115 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:00 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:01.115 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:00 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:17:01.115 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:00 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:17:01.115 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:00 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:17:01.115 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:00 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:01.115 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:00 smithi106 ceph-mon[36505]: osdmap e68: 10 total, 10 up, 10 in 2024-03-24T00:17:01.115 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:00 smithi106 ceph-mon[36505]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:01.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:00 smithi078 ceph-mon[32429]: Detected new or changed devices on smithi078 2024-03-24T00:17:01.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:00 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:01.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:00 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:01.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:00 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:01.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:00 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:01.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:00 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:01.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:00 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:01.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:00 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi078 to 4272M 2024-03-24T00:17:01.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:00 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:01.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:00 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:17:01.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:00 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:17:01.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:00 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:01.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:00 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:17:01.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:00 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:17:01.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:00 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:17:01.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:00 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:01.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:00 smithi078 ceph-mon[32429]: osdmap e68: 10 total, 10 up, 10 in 2024-03-24T00:17:01.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:00 smithi078 ceph-mon[32429]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:00 smithi107 ceph-mon[36485]: Detected new or changed devices on smithi078 2024-03-24T00:17:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:00 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:00 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:00 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:00 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:01.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:00 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:01.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:00 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:01.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:00 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi078 to 4272M 2024-03-24T00:17:01.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:00 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:01.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:00 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:17:01.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:00 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:17:01.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:00 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:01.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:00 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:17:01.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:00 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:17:01.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:00 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:17:01.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:00 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:01.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:00 smithi107 ceph-mon[36485]: osdmap e68: 10 total, 10 up, 10 in 2024-03-24T00:17:01.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:00 smithi107 ceph-mon[36485]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:02.052 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:01 smithi106 ceph-mon[36505]: from='client.24323 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:17:02.052 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:01 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:17:02.052 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:01 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:17:02.052 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:01 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:17:02.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:01 smithi078 ceph-mon[32429]: from='client.24323 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:17:02.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:01 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:17:02.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:01 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:17:02.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:01 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:17:02.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:01 smithi107 ceph-mon[36485]: from='client.24323 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:17:02.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:01 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:17:02.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:01 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:17:02.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:01 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:17:03.066 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:02 smithi106 ceph-mon[36505]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:03.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:02 smithi078 ceph-mon[32429]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:03.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:02 smithi107 ceph-mon[36485]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:04.249 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:03 smithi106 ceph-mon[36505]: from='client.? 172.21.15.106:0/3355674389' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2375119d-346e-426c-8a65-b64af71ad63a"}]: dispatch 2024-03-24T00:17:04.249 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:03 smithi106 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2375119d-346e-426c-8a65-b64af71ad63a"}]: dispatch 2024-03-24T00:17:04.250 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:03 smithi106 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2375119d-346e-426c-8a65-b64af71ad63a"}]': finished 2024-03-24T00:17:04.250 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:03 smithi106 ceph-mon[36505]: osdmap e69: 11 total, 10 up, 11 in 2024-03-24T00:17:04.250 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:03 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:17:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:03 smithi078 ceph-mon[32429]: from='client.? 172.21.15.106:0/3355674389' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2375119d-346e-426c-8a65-b64af71ad63a"}]: dispatch 2024-03-24T00:17:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:03 smithi078 ceph-mon[32429]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2375119d-346e-426c-8a65-b64af71ad63a"}]: dispatch 2024-03-24T00:17:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:03 smithi078 ceph-mon[32429]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2375119d-346e-426c-8a65-b64af71ad63a"}]': finished 2024-03-24T00:17:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:03 smithi078 ceph-mon[32429]: osdmap e69: 11 total, 10 up, 11 in 2024-03-24T00:17:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:03 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:17:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:03 smithi107 ceph-mon[36485]: from='client.? 172.21.15.106:0/3355674389' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2375119d-346e-426c-8a65-b64af71ad63a"}]: dispatch 2024-03-24T00:17:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:03 smithi107 ceph-mon[36485]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2375119d-346e-426c-8a65-b64af71ad63a"}]: dispatch 2024-03-24T00:17:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:03 smithi107 ceph-mon[36485]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2375119d-346e-426c-8a65-b64af71ad63a"}]': finished 2024-03-24T00:17:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:03 smithi107 ceph-mon[36485]: osdmap e69: 11 total, 10 up, 11 in 2024-03-24T00:17:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:03 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:17:05.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:04 smithi078 ceph-mon[32429]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:05.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:04 smithi078 ceph-mon[32429]: from='client.? 172.21.15.106:0/1228912920' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:17:05.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:04 smithi107 ceph-mon[36485]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:05.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:04 smithi107 ceph-mon[36485]: from='client.? 172.21.15.106:0/1228912920' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:17:05.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:04 smithi106 ceph-mon[36505]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:05.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:04 smithi106 ceph-mon[36505]: from='client.? 172.21.15.106:0/1228912920' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:17:07.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:07 smithi078 ceph-mon[32429]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:07.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:07 smithi107 ceph-mon[36485]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:07.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:07 smithi106 ceph-mon[36505]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:09.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:09 smithi106 ceph-mon[36505]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:09 smithi078 ceph-mon[32429]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:09 smithi107 ceph-mon[36485]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:11.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:11 smithi106 ceph-mon[36505]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:11.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:11 smithi078 ceph-mon[32429]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:11.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:11 smithi107 ceph-mon[36485]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:12.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:12 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-24T00:17:12.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:12 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:17:12.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:12 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-24T00:17:12.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:12 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:17:12.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:12 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-24T00:17:12.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:12 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:17:13.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:13 smithi106 ceph-mon[36505]: Deploying daemon osd.10 on smithi106 2024-03-24T00:17:13.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:13 smithi106 ceph-mon[36505]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:13.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:13 smithi078 ceph-mon[32429]: Deploying daemon osd.10 on smithi106 2024-03-24T00:17:13.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:13 smithi078 ceph-mon[32429]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:13.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:13 smithi107 ceph-mon[36485]: Deploying daemon osd.10 on smithi106 2024-03-24T00:17:13.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:13 smithi107 ceph-mon[36485]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:15.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:15 smithi078 ceph-mon[32429]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:15.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:15 smithi107 ceph-mon[36485]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:15.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:15 smithi106 ceph-mon[36505]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:17.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:17 smithi078 ceph-mon[32429]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:17.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:17 smithi107 ceph-mon[36485]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:17.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:17 smithi106 ceph-mon[36505]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:18.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:18 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:18.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:18 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:18.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:18 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:17:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:18 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:18 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:18 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:17:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:18 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:18 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:18 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:17:19.708 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:19 smithi106 ceph-mon[36505]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:19.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:19 smithi078 ceph-mon[32429]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:19 smithi107 ceph-mon[36485]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:21.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:21.032+0000 7f5914ff9700 1 -- 172.21.15.106:0/2342879686 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f5918062690 con 0x7f59000a2910 2024-03-24T00:17:21.034 INFO:teuthology.orchestra.run.smithi106.stdout:Created osd(s) 10 on host 'smithi106' 2024-03-24T00:17:21.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:21.036+0000 7f591e254700 1 -- 172.21.15.106:0/2342879686 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f59000a2910 msgr2=0x7f59000a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:21.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:21.036+0000 7f591e254700 1 --2- 172.21.15.106:0/2342879686 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f59000a2910 0x7f59000a4dd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5908005fd0 tx=0x7f5908005ee0 comp rx=0 tx=0).stop 2024-03-24T00:17:21.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:21.036+0000 7f591e254700 1 -- 172.21.15.106:0/2342879686 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f59181055e0 msgr2=0x7f5918198960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:21.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:21.036+0000 7f591e254700 1 --2- 172.21.15.106:0/2342879686 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f59181055e0 0x7f5918198960 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f590c005910 tx=0x7f590c00bbb0 comp rx=0 tx=0).stop 2024-03-24T00:17:21.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:21.036+0000 7f591e254700 1 -- 172.21.15.106:0/2342879686 shutdown_connections 2024-03-24T00:17:21.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:21.036+0000 7f591e254700 1 --2- 172.21.15.106:0/2342879686 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f59000a2910 0x7f59000a4dd0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:21.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:21.036+0000 7f591e254700 1 --2- 172.21.15.106:0/2342879686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5918107f10 0x7f591819f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:21.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:21.036+0000 7f591e254700 1 --2- 172.21.15.106:0/2342879686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5918102cb0 0x7f5918198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:21.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:21.036+0000 7f591e254700 1 --2- 172.21.15.106:0/2342879686 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f59181055e0 0x7f5918198960 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:21.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:21.036+0000 7f591e254700 1 -- 172.21.15.106:0/2342879686 >> 172.21.15.106:0/2342879686 conn(0x7f59180fc680 msgr2=0x7f5918103710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:21.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:21.036+0000 7f591e254700 1 -- 172.21.15.106:0/2342879686 shutdown_connections 2024-03-24T00:17:21.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-24T00:17:21.036+0000 7f591e254700 1 -- 172.21.15.106:0/2342879686 wait complete. 2024-03-24T00:17:21.623 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:21 smithi106 ceph-mon[36505]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:21.624 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:21 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:21.624 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:21 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:21.624 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:21 smithi106 ceph-mon[36505]: from='osd.10 [v2:172.21.15.106:6824/400029804,v1:172.21.15.106:6825/400029804]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T00:17:21.624 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:21 smithi106 ceph-mon[36505]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T00:17:21.766 DEBUG:teuthology.orchestra.run.smithi106:osd.10> sudo journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.10.service 2024-03-24T00:17:21.768 INFO:tasks.cephadm:Deploying osd.11 on smithi107 with /dev/vg_nvme/lv_1... 2024-03-24T00:17:21.769 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-24T00:17:21.794 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:21 smithi107 ceph-mon[36485]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:21.795 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:21 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:21.795 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:21 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:21.795 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:21 smithi107 ceph-mon[36485]: from='osd.10 [v2:172.21.15.106:6824/400029804,v1:172.21.15.106:6825/400029804]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T00:17:21.795 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:21 smithi107 ceph-mon[36485]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T00:17:21.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:21 smithi078 ceph-mon[32429]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:21.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:21 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:21.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:21 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:21.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:21 smithi078 ceph-mon[32429]: from='osd.10 [v2:172.21.15.106:6824/400029804,v1:172.21.15.106:6825/400029804]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T00:17:21.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:21 smithi078 ceph-mon[32429]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T00:17:22.010 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:17:23.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:22 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:23.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:22 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:23.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:22 smithi106 ceph-mon[36505]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-24T00:17:23.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:22 smithi106 ceph-mon[36505]: from='osd.10 [v2:172.21.15.106:6824/400029804,v1:172.21.15.106:6825/400029804]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-24T00:17:23.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:22 smithi106 ceph-mon[36505]: osdmap e70: 11 total, 10 up, 11 in 2024-03-24T00:17:23.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:22 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:17:23.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:22 smithi106 ceph-mon[36505]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-24T00:17:23.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:22 smithi106 ceph-mon[36505]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:23.160 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:22 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:23.160 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:22 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:23.160 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:22 smithi107 ceph-mon[36485]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-24T00:17:23.160 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:22 smithi107 ceph-mon[36485]: from='osd.10 [v2:172.21.15.106:6824/400029804,v1:172.21.15.106:6825/400029804]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-24T00:17:23.160 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:22 smithi107 ceph-mon[36485]: osdmap e70: 11 total, 10 up, 11 in 2024-03-24T00:17:23.161 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:22 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:17:23.161 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:22 smithi107 ceph-mon[36485]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-24T00:17:23.161 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:22 smithi107 ceph-mon[36485]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:23.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:22 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:23.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:22 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:23.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:22 smithi078 ceph-mon[32429]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-24T00:17:23.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:22 smithi078 ceph-mon[32429]: from='osd.10 [v2:172.21.15.106:6824/400029804,v1:172.21.15.106:6825/400029804]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-24T00:17:23.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:22 smithi078 ceph-mon[32429]: osdmap e70: 11 total, 10 up, 11 in 2024-03-24T00:17:23.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:22 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:17:23.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:22 smithi078 ceph-mon[32429]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-24T00:17:23.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:22 smithi078 ceph-mon[32429]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:17:23.745 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 00:17:23 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T00:17:23.732+0000 7f8502d91700 -1 osd.10 0 waiting for initial osdmap 2024-03-24T00:17:23.745 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 00:17:23 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T00:17:23.744+0000 7f84fab7c700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:17:24.075 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-24T00:17:24.075 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-24T00:17:24.075 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10+0 records in 2024-03-24T00:17:24.075 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2024-03-24T00:17:24.075 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011425 s, 918 MB/s 2024-03-24T00:17:24.075 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping successful for: 2024-03-24T00:17:24.098 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:24 smithi107 ceph-mon[36485]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-24T00:17:24.098 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:24 smithi107 ceph-mon[36485]: osdmap e71: 11 total, 10 up, 11 in 2024-03-24T00:17:24.098 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:24 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:17:24.098 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:24 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:17:24.098 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:24 smithi107 ceph-mon[36485]: from='osd.10 ' entity='osd.10' 2024-03-24T00:17:24.098 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:24 smithi107 ceph-mon[36485]: osd.10 [v2:172.21.15.106:6824/400029804,v1:172.21.15.106:6825/400029804] boot 2024-03-24T00:17:24.098 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:24 smithi107 ceph-mon[36485]: osdmap e72: 11 total, 11 up, 11 in 2024-03-24T00:17:24.098 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:24 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:17:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:24 smithi078 ceph-mon[32429]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-24T00:17:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:24 smithi078 ceph-mon[32429]: osdmap e71: 11 total, 10 up, 11 in 2024-03-24T00:17:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:24 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:17:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:24 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:17:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:24 smithi078 ceph-mon[32429]: from='osd.10 ' entity='osd.10' 2024-03-24T00:17:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:24 smithi078 ceph-mon[32429]: osd.10 [v2:172.21.15.106:6824/400029804,v1:172.21.15.106:6825/400029804] boot 2024-03-24T00:17:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:24 smithi078 ceph-mon[32429]: osdmap e72: 11 total, 11 up, 11 in 2024-03-24T00:17:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:24 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:17:24.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:24 smithi106 ceph-mon[36505]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-24T00:17:24.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:24 smithi106 ceph-mon[36505]: osdmap e71: 11 total, 10 up, 11 in 2024-03-24T00:17:24.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:24 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:17:24.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:24 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:17:24.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:24 smithi106 ceph-mon[36505]: from='osd.10 ' entity='osd.10' 2024-03-24T00:17:24.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:24 smithi106 ceph-mon[36505]: osd.10 [v2:172.21.15.106:6824/400029804,v1:172.21.15.106:6825/400029804] boot 2024-03-24T00:17:24.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:24 smithi106 ceph-mon[36505]: osdmap e72: 11 total, 11 up, 11 in 2024-03-24T00:17:24.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:24 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:17:24.813 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi107:vg_nvme/lv_1 2024-03-24T00:17:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:25 smithi107 ceph-mon[36485]: purged_snaps scrub starts 2024-03-24T00:17:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:25 smithi107 ceph-mon[36485]: purged_snaps scrub ok 2024-03-24T00:17:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:25 smithi107 ceph-mon[36485]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 680 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:17:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:25 smithi107 ceph-mon[36485]: osdmap e73: 11 total, 11 up, 11 in 2024-03-24T00:17:25.096 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:17:25.314 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:25 smithi106 ceph-mon[36505]: purged_snaps scrub starts 2024-03-24T00:17:25.314 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:25 smithi106 ceph-mon[36505]: purged_snaps scrub ok 2024-03-24T00:17:25.314 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:25 smithi106 ceph-mon[36505]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 680 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:17:25.314 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:25 smithi106 ceph-mon[36505]: osdmap e73: 11 total, 11 up, 11 in 2024-03-24T00:17:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:25 smithi078 ceph-mon[32429]: purged_snaps scrub starts 2024-03-24T00:17:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:25 smithi078 ceph-mon[32429]: purged_snaps scrub ok 2024-03-24T00:17:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:25 smithi078 ceph-mon[32429]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 680 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:17:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:25 smithi078 ceph-mon[32429]: osdmap e73: 11 total, 11 up, 11 in 2024-03-24T00:17:26.476 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:26 smithi107 ceph-mon[36485]: Detected new or changed devices on smithi106 2024-03-24T00:17:26.477 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:26 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:26.477 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:26 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:26.477 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:26 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:26.477 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:26 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:26.477 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:26 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:26.477 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:26 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:26.477 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:26 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi106 to 4270M 2024-03-24T00:17:26.477 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:26 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:26.477 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:26 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:17:26.477 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:26 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:17:26.478 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:26 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:26.478 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:26 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:17:26.478 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:26 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:17:26.478 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:26 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:17:26.478 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:26 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:26.478 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:26 smithi107 ceph-mon[36485]: osdmap e74: 11 total, 11 up, 11 in 2024-03-24T00:17:26.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:26 smithi106 ceph-mon[36505]: Detected new or changed devices on smithi106 2024-03-24T00:17:26.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:26 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:26.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:26 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:26 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:26 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:26 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:26 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:26 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi106 to 4270M 2024-03-24T00:17:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:26 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:26 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:17:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:26 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:17:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:26 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:26 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:17:26.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:26 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:17:26.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:26 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:17:26.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:26 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:26.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:26 smithi106 ceph-mon[36505]: osdmap e74: 11 total, 11 up, 11 in 2024-03-24T00:17:26.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.712+0000 7f7d9b99e700 1 -- 172.21.15.107:0/3403604883 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d94102c60 msgr2=0x7f7d94109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:26.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.712+0000 7f7d9b99e700 1 --2- 172.21.15.107:0/3403604883 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d94102c60 0x7f7d94109510 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c00b0a0 tx=0x7f7d8c00b3b0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.712+0000 7f7d9b99e700 1 -- 172.21.15.107:0/3403604883 shutdown_connections 2024-03-24T00:17:26.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.712+0000 7f7d9b99e700 1 --2- 172.21.15.107:0/3403604883 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d94102c60 0x7f7d94109510 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.712+0000 7f7d9b99e700 1 --2- 172.21.15.107:0/3403604883 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d941010c0 0x7f7d941014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.712+0000 7f7d9b99e700 1 --2- 172.21.15.107:0/3403604883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d941022c0 0x7f7d94102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.712+0000 7f7d9b99e700 1 -- 172.21.15.107:0/3403604883 >> 172.21.15.107:0/3403604883 conn(0x7f7d940fc680 msgr2=0x7f7d940feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:26.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.712+0000 7f7d9b99e700 1 -- 172.21.15.107:0/3403604883 shutdown_connections 2024-03-24T00:17:26.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.712+0000 7f7d9b99e700 1 -- 172.21.15.107:0/3403604883 wait complete. 2024-03-24T00:17:26.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.712+0000 7f7d9b99e700 1 Processor -- start 2024-03-24T00:17:26.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.713+0000 7f7d9b99e700 1 -- start start 2024-03-24T00:17:26.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.713+0000 7f7d9b99e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d941010c0 0x7f7d94198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:26.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.713+0000 7f7d9b99e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d941022c0 0x7f7d94198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:26.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.713+0000 7f7d9b99e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d94102c60 0x7f7d9419f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:26.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.713+0000 7f7d9b99e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9410ccb0 con 0x7f7d941010c0 2024-03-24T00:17:26.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.713+0000 7f7d9b99e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9410cb30 con 0x7f7d94102c60 2024-03-24T00:17:26.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.713+0000 7f7d9b99e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9410ce30 con 0x7f7d941022c0 2024-03-24T00:17:26.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.713+0000 7f7d98f39700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d941022c0 0x7f7d94198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:26.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.713+0000 7f7d98f39700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d941022c0 0x7f7d94198870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36818/0 (socket says 172.21.15.107:36818) 2024-03-24T00:17:26.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.713+0000 7f7d98f39700 1 -- 172.21.15.107:0/2001765782 learned_addr learned my addr 172.21.15.107:0/2001765782 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-03-24T00:17:26.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.713+0000 7f7d9973a700 1 --2- 172.21.15.107:0/2001765782 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d941010c0 0x7f7d94198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:26.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.713+0000 7f7d99f3b700 1 --2- 172.21.15.107:0/2001765782 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d94102c60 0x7f7d9419f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:26.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.713+0000 7f7d98f39700 1 -- 172.21.15.107:0/2001765782 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d94102c60 msgr2=0x7f7d9419f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:26.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.713+0000 7f7d98f39700 1 --2- 172.21.15.107:0/2001765782 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d94102c60 0x7f7d9419f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.713+0000 7f7d98f39700 1 -- 172.21.15.107:0/2001765782 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d941010c0 msgr2=0x7f7d94198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:26.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.713+0000 7f7d98f39700 1 --2- 172.21.15.107:0/2001765782 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d941010c0 0x7f7d94198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.713+0000 7f7d98f39700 1 -- 172.21.15.107:0/2001765782 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d9419ff90 con 0x7f7d941022c0 2024-03-24T00:17:26.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.713+0000 7f7d98f39700 1 --2- 172.21.15.107:0/2001765782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d941022c0 0x7f7d94198870 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7d8400d9b0 tx=0x7f7d8400dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:26.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.713+0000 7f7d8a7fc700 1 -- 172.21.15.107:0/2001765782 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d84016070 con 0x7f7d941022c0 2024-03-24T00:17:26.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.713+0000 7f7d8a7fc700 1 -- 172.21.15.107:0/2001765782 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d84004d10 con 0x7f7d941022c0 2024-03-24T00:17:26.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.713+0000 7f7d9b99e700 1 -- 172.21.15.107:0/2001765782 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d941a0280 con 0x7f7d941022c0 2024-03-24T00:17:26.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.713+0000 7f7d8a7fc700 1 -- 172.21.15.107:0/2001765782 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d84005530 con 0x7f7d941022c0 2024-03-24T00:17:26.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.714+0000 7f7d9b99e700 1 -- 172.21.15.107:0/2001765782 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d941a07c0 con 0x7f7d941022c0 2024-03-24T00:17:26.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.714+0000 7f7d9b99e700 1 -- 172.21.15.107:0/2001765782 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d94068e50 con 0x7f7d941022c0 2024-03-24T00:17:26.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.715+0000 7f7d8a7fc700 1 -- 172.21.15.107:0/2001765782 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7d84004750 con 0x7f7d941022c0 2024-03-24T00:17:26.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.716+0000 7f7d8a7fc700 1 --2- 172.21.15.107:0/2001765782 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f7d800a2890 0x7f7d800a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:26.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.716+0000 7f7d8a7fc700 1 -- 172.21.15.107:0/2001765782 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6723+0+0 (secure 0 0 0) 0x7f7d840ce6c0 con 0x7f7d941022c0 2024-03-24T00:17:26.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.716+0000 7f7d9973a700 1 --2- 172.21.15.107:0/2001765782 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f7d800a2890 0x7f7d800a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:26.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.717+0000 7f7d9973a700 1 --2- 172.21.15.107:0/2001765782 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f7d800a2890 0x7f7d800a4d50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7d94102120 tx=0x7f7d9000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:26.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.719+0000 7f7d8a7fc700 1 -- 172.21.15.107:0/2001765782 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7d8401f020 con 0x7f7d941022c0 2024-03-24T00:17:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:26 smithi078 ceph-mon[32429]: Detected new or changed devices on smithi106 2024-03-24T00:17:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:26 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:26 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:26 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:26 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:26 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:26 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:26 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi106 to 4270M 2024-03-24T00:17:26.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:26 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:26.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:26 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:17:26.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:26 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:17:26.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:26 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:26.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:26 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:17:26.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:26 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:17:26.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:26 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:17:26.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:26 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:26.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:26 smithi078 ceph-mon[32429]: osdmap e74: 11 total, 11 up, 11 in 2024-03-24T00:17:26.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:26.926+0000 7f7d9b99e700 1 -- 172.21.15.107:0/2001765782 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d940fe750 con 0x7f7d800a2890 2024-03-24T00:17:27.651 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:27 smithi107 ceph-mon[36485]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 680 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:17:27.651 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:27 smithi107 ceph-mon[36485]: osdmap e75: 11 total, 11 up, 11 in 2024-03-24T00:17:27.651 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:27 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:17:27.651 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:27 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:17:27.651 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:27 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:17:27.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:27 smithi106 ceph-mon[36505]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 680 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:17:27.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:27 smithi106 ceph-mon[36505]: osdmap e75: 11 total, 11 up, 11 in 2024-03-24T00:17:27.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:27 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:17:27.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:27 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:17:27.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:27 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:17:27.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:27 smithi078 ceph-mon[32429]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 680 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:17:27.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:27 smithi078 ceph-mon[32429]: osdmap e75: 11 total, 11 up, 11 in 2024-03-24T00:17:27.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:27 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:17:27.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:27 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:17:27.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:27 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:17:28.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:28 smithi106 ceph-mon[36505]: from='client.24334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:17:28.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:28 smithi106 ceph-mon[36505]: osdmap e76: 11 total, 11 up, 11 in 2024-03-24T00:17:28.811 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:28 smithi107 ceph-mon[36485]: from='client.24334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:17:28.811 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:28 smithi107 ceph-mon[36485]: osdmap e76: 11 total, 11 up, 11 in 2024-03-24T00:17:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:28 smithi078 ceph-mon[32429]: from='client.24334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:17:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:28 smithi078 ceph-mon[32429]: osdmap e76: 11 total, 11 up, 11 in 2024-03-24T00:17:29.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:29 smithi078 ceph-mon[32429]: pgmap v240: 1 pgs: 1 peering; 577 KiB data, 680 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T00:17:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:29 smithi107 ceph-mon[36485]: pgmap v240: 1 pgs: 1 peering; 577 KiB data, 680 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T00:17:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:29 smithi106 ceph-mon[36505]: pgmap v240: 1 pgs: 1 peering; 577 KiB data, 680 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T00:17:30.761 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:30 smithi078 ceph-mon[32429]: from='client.? 172.21.15.107:0/1189862022' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6b609d9-dd78-4c01-ae66-ba2a662ab22f"}]: dispatch 2024-03-24T00:17:30.761 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:30 smithi078 ceph-mon[32429]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6b609d9-dd78-4c01-ae66-ba2a662ab22f"}]: dispatch 2024-03-24T00:17:30.761 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:30 smithi078 ceph-mon[32429]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b6b609d9-dd78-4c01-ae66-ba2a662ab22f"}]': finished 2024-03-24T00:17:30.761 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:30 smithi078 ceph-mon[32429]: osdmap e77: 12 total, 11 up, 12 in 2024-03-24T00:17:30.761 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:30 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:17:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:30 smithi107 ceph-mon[36485]: from='client.? 172.21.15.107:0/1189862022' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6b609d9-dd78-4c01-ae66-ba2a662ab22f"}]: dispatch 2024-03-24T00:17:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:30 smithi107 ceph-mon[36485]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6b609d9-dd78-4c01-ae66-ba2a662ab22f"}]: dispatch 2024-03-24T00:17:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:30 smithi107 ceph-mon[36485]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b6b609d9-dd78-4c01-ae66-ba2a662ab22f"}]': finished 2024-03-24T00:17:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:30 smithi107 ceph-mon[36485]: osdmap e77: 12 total, 11 up, 12 in 2024-03-24T00:17:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:30 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:17:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:30 smithi106 ceph-mon[36505]: from='client.? 172.21.15.107:0/1189862022' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6b609d9-dd78-4c01-ae66-ba2a662ab22f"}]: dispatch 2024-03-24T00:17:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:30 smithi106 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6b609d9-dd78-4c01-ae66-ba2a662ab22f"}]: dispatch 2024-03-24T00:17:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:30 smithi106 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b6b609d9-dd78-4c01-ae66-ba2a662ab22f"}]': finished 2024-03-24T00:17:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:30 smithi106 ceph-mon[36505]: osdmap e77: 12 total, 11 up, 12 in 2024-03-24T00:17:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:30 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:17:31.691 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:31 smithi106 ceph-mon[36505]: pgmap v242: 1 pgs: 1 peering; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail; 131 KiB/s, 0 objects/s recovering 2024-03-24T00:17:31.691 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:31 smithi106 ceph-mon[36505]: from='client.? 172.21.15.107:0/1321824575' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:17:31.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:31 smithi078 ceph-mon[32429]: pgmap v242: 1 pgs: 1 peering; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail; 131 KiB/s, 0 objects/s recovering 2024-03-24T00:17:31.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:31 smithi078 ceph-mon[32429]: from='client.? 172.21.15.107:0/1321824575' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:17:31.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:31 smithi107 ceph-mon[36485]: pgmap v242: 1 pgs: 1 peering; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail; 131 KiB/s, 0 objects/s recovering 2024-03-24T00:17:31.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:31 smithi107 ceph-mon[36485]: from='client.? 172.21.15.107:0/1321824575' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:17:33.829 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:33 smithi107 ceph-mon[36485]: pgmap v243: 1 pgs: 1 peering; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T00:17:33.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:33 smithi078 ceph-mon[32429]: pgmap v243: 1 pgs: 1 peering; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T00:17:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:33 smithi106 ceph-mon[36505]: pgmap v243: 1 pgs: 1 peering; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T00:17:35.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:35 smithi107 ceph-mon[36485]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-24T00:17:35.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:35 smithi078 ceph-mon[32429]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-24T00:17:35.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:35 smithi106 ceph-mon[36505]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-24T00:17:37.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:37 smithi078 ceph-mon[32429]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:17:37.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:37 smithi107 ceph-mon[36485]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:17:37.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:37 smithi106 ceph-mon[36505]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:17:38.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:38 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-24T00:17:38.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:38 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:17:38.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:38 smithi107 ceph-mon[36485]: Deploying daemon osd.11 on smithi107 2024-03-24T00:17:38.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:38 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-24T00:17:38.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:38 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:17:38.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:38 smithi078 ceph-mon[32429]: Deploying daemon osd.11 on smithi107 2024-03-24T00:17:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:38 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-24T00:17:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:38 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:17:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:38 smithi106 ceph-mon[36505]: Deploying daemon osd.11 on smithi107 2024-03-24T00:17:39.750 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:39 smithi107 ceph-mon[36485]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:17:39.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:39 smithi078 ceph-mon[32429]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:17:39.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:39 smithi106 ceph-mon[36505]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:17:41.467 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:41 smithi107 ceph-mon[36485]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:17:41.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:41 smithi078 ceph-mon[32429]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:17:41.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:41 smithi106 ceph-mon[36505]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:17:43.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:43 smithi078 ceph-mon[32429]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:17:43.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:43 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:43.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:43 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:43.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:43 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:17:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:43 smithi107 ceph-mon[36485]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:17:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:43 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:43 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:43 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:17:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:43 smithi106 ceph-mon[36505]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:17:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:43 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:43 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:43 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:17:45.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:45 smithi107 ceph-mon[36485]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:17:45.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:45 smithi078 ceph-mon[32429]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:17:45.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:45 smithi106 ceph-mon[36505]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:17:47.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:47.346+0000 7f7d8a7fc700 1 -- 172.21.15.107:0/2001765782 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f7d940fe750 con 0x7f7d800a2890 2024-03-24T00:17:47.346 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 11 on host 'smithi107' 2024-03-24T00:17:47.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:47.348+0000 7f7d9b99e700 1 -- 172.21.15.107:0/2001765782 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f7d800a2890 msgr2=0x7f7d800a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:47.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:47.348+0000 7f7d9b99e700 1 --2- 172.21.15.107:0/2001765782 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f7d800a2890 0x7f7d800a4d50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7d94102120 tx=0x7f7d9000b410 comp rx=0 tx=0).stop 2024-03-24T00:17:47.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:47.348+0000 7f7d9b99e700 1 -- 172.21.15.107:0/2001765782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d941022c0 msgr2=0x7f7d94198870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:47.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:47.348+0000 7f7d9b99e700 1 --2- 172.21.15.107:0/2001765782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d941022c0 0x7f7d94198870 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7d8400d9b0 tx=0x7f7d8400dcc0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:47.348+0000 7f7d9b99e700 1 -- 172.21.15.107:0/2001765782 shutdown_connections 2024-03-24T00:17:47.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:47.348+0000 7f7d9b99e700 1 --2- 172.21.15.107:0/2001765782 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f7d800a2890 0x7f7d800a4d50 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:47.348+0000 7f7d9b99e700 1 --2- 172.21.15.107:0/2001765782 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d94102c60 0x7f7d9419f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:47.348+0000 7f7d9b99e700 1 --2- 172.21.15.107:0/2001765782 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d941010c0 0x7f7d94198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:47.348+0000 7f7d9b99e700 1 --2- 172.21.15.107:0/2001765782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d941022c0 0x7f7d94198870 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:47.348+0000 7f7d9b99e700 1 -- 172.21.15.107:0/2001765782 >> 172.21.15.107:0/2001765782 conn(0x7f7d940fc680 msgr2=0x7f7d940fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:47.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:47.348+0000 7f7d9b99e700 1 -- 172.21.15.107:0/2001765782 shutdown_connections 2024-03-24T00:17:47.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:17:47.348+0000 7f7d9b99e700 1 -- 172.21.15.107:0/2001765782 wait complete. 2024-03-24T00:17:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:47 smithi107 ceph-mon[36485]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:17:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:47 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:47 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:47 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:17:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:47 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:17:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:47 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:47 smithi107 ceph-mon[36485]: from='osd.11 [v2:172.21.15.107:6824/4191916230,v1:172.21.15.107:6825/4191916230]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T00:17:47.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:47 smithi107 ceph-mon[36485]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T00:17:47.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:47 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:47.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:47 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:47.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:47 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:17:47.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:47 smithi078 ceph-mon[32429]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:17:47.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:47 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:47.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:47 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:47.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:47 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:17:47.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:47 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:17:47.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:47 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:47.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:47 smithi078 ceph-mon[32429]: from='osd.11 [v2:172.21.15.107:6824/4191916230,v1:172.21.15.107:6825/4191916230]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T00:17:47.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:47 smithi078 ceph-mon[32429]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T00:17:47.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:47 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:47.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:47 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:47.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:47 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:17:47.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:47 smithi106 ceph-mon[36505]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:17:47.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:47 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:47.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:47 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:47.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:47 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:17:47.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:47 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:17:47.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:47 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:47.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:47 smithi106 ceph-mon[36505]: from='osd.11 [v2:172.21.15.107:6824/4191916230,v1:172.21.15.107:6825/4191916230]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T00:17:47.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:47 smithi106 ceph-mon[36505]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T00:17:47.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:47 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:47.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:47 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:47.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:47 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:17:48.027 DEBUG:teuthology.orchestra.run.smithi107:osd.11> sudo journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.11.service 2024-03-24T00:17:48.029 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-24T00:17:48.030 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-24T00:17:48.284 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:17:48.534 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:48 smithi107 ceph-mon[36485]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-24T00:17:48.534 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:48 smithi107 ceph-mon[36485]: osdmap e78: 12 total, 11 up, 12 in 2024-03-24T00:17:48.534 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:48 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:17:48.534 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:48 smithi107 ceph-mon[36485]: from='osd.11 [v2:172.21.15.107:6824/4191916230,v1:172.21.15.107:6825/4191916230]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-24T00:17:48.534 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:48 smithi107 ceph-mon[36485]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-24T00:17:48.674 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:48 smithi078 ceph-mon[32429]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-24T00:17:48.674 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:48 smithi078 ceph-mon[32429]: osdmap e78: 12 total, 11 up, 12 in 2024-03-24T00:17:48.674 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:48 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:17:48.674 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:48 smithi078 ceph-mon[32429]: from='osd.11 [v2:172.21.15.107:6824/4191916230,v1:172.21.15.107:6825/4191916230]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-24T00:17:48.674 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:48 smithi078 ceph-mon[32429]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-24T00:17:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:48 smithi106 ceph-mon[36505]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-24T00:17:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:48 smithi106 ceph-mon[36505]: osdmap e78: 12 total, 11 up, 12 in 2024-03-24T00:17:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:48 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:17:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:48 smithi106 ceph-mon[36505]: from='osd.11 [v2:172.21.15.107:6824/4191916230,v1:172.21.15.107:6825/4191916230]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-24T00:17:48.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:48 smithi106 ceph-mon[36505]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-24T00:17:49.593 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 00:17:49 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T00:17:49.234+0000 7fb0530e0700 -1 osd.11 0 waiting for initial osdmap 2024-03-24T00:17:49.594 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 00:17:49 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T00:17:49.246+0000 7fb04a6ca700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:17:49.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:49 smithi107 ceph-mon[36485]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:17:49.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:49 smithi107 ceph-mon[36485]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-03-24T00:17:49.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:49 smithi107 ceph-mon[36485]: osdmap e79: 12 total, 11 up, 12 in 2024-03-24T00:17:49.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:49 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:17:49.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:49 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:17:49.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:49 smithi107 ceph-mon[36485]: from='osd.11 ' entity='osd.11' 2024-03-24T00:17:49.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:49 smithi078 ceph-mon[32429]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:17:49.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:49 smithi078 ceph-mon[32429]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-03-24T00:17:49.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:49 smithi078 ceph-mon[32429]: osdmap e79: 12 total, 11 up, 12 in 2024-03-24T00:17:49.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:49 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:17:49.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:49 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:17:49.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:49 smithi078 ceph-mon[32429]: from='osd.11 ' entity='osd.11' 2024-03-24T00:17:49.803 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:49 smithi106 ceph-mon[36505]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:17:49.804 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:49 smithi106 ceph-mon[36505]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-03-24T00:17:49.804 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:49 smithi106 ceph-mon[36505]: osdmap e79: 12 total, 11 up, 12 in 2024-03-24T00:17:49.804 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:49 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:17:49.804 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:49 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:17:49.804 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:49 smithi106 ceph-mon[36505]: from='osd.11 ' entity='osd.11' 2024-03-24T00:17:49.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.965+0000 7fade0c5c700 1 -- 172.21.15.78:0/3058461661 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faddc1033b0 msgr2=0x7faddc103810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:49.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.965+0000 7fade0c5c700 1 --2- 172.21.15.78:0/3058461661 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faddc1033b0 0x7faddc103810 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fadd00099c0 tx=0x7fadd0009cd0 comp rx=0 tx=0).stop 2024-03-24T00:17:49.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.965+0000 7fade0c5c700 1 -- 172.21.15.78:0/3058461661 shutdown_connections 2024-03-24T00:17:49.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.965+0000 7fade0c5c700 1 --2- 172.21.15.78:0/3058461661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faddc103d50 0x7faddc10a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:49.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.965+0000 7fade0c5c700 1 --2- 172.21.15.78:0/3058461661 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faddc1021b0 0x7faddc1025d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:49.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.965+0000 7fade0c5c700 1 --2- 172.21.15.78:0/3058461661 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faddc1033b0 0x7faddc103810 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:49.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.965+0000 7fade0c5c700 1 -- 172.21.15.78:0/3058461661 >> 172.21.15.78:0/3058461661 conn(0x7faddc0fd770 msgr2=0x7faddc0ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:49.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.965+0000 7fade0c5c700 1 -- 172.21.15.78:0/3058461661 shutdown_connections 2024-03-24T00:17:49.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.965+0000 7fade0c5c700 1 -- 172.21.15.78:0/3058461661 wait complete. 2024-03-24T00:17:49.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.966+0000 7fade0c5c700 1 Processor -- start 2024-03-24T00:17:49.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.966+0000 7fade0c5c700 1 -- start start 2024-03-24T00:17:49.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.966+0000 7fade0c5c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faddc1021b0 0x7faddc1983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:49.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.966+0000 7fade0c5c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faddc1033b0 0x7faddc198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:49.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.966+0000 7fade0c5c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faddc103d50 0x7faddc19f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:49.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.966+0000 7fade0c5c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faddc10dd80 con 0x7faddc1021b0 2024-03-24T00:17:49.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.966+0000 7fade0c5c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faddc10dc00 con 0x7faddc1033b0 2024-03-24T00:17:49.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.966+0000 7fade0c5c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faddc10df00 con 0x7faddc103d50 2024-03-24T00:17:49.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.966+0000 7fadda59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faddc1021b0 0x7faddc1983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:49.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.966+0000 7fadda59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faddc1021b0 0x7faddc1983c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33624/0 (socket says 172.21.15.78:33624) 2024-03-24T00:17:49.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.966+0000 7fadda59c700 1 -- 172.21.15.78:0/1056184502 learned_addr learned my addr 172.21.15.78:0/1056184502 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:17:49.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.966+0000 7faddad9d700 1 --2- 172.21.15.78:0/1056184502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faddc103d50 0x7faddc19f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:49.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.966+0000 7fadd9d9b700 1 --2- 172.21.15.78:0/1056184502 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faddc1033b0 0x7faddc198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:49.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.966+0000 7fadda59c700 1 -- 172.21.15.78:0/1056184502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faddc103d50 msgr2=0x7faddc19f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:49.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.966+0000 7fadda59c700 1 --2- 172.21.15.78:0/1056184502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faddc103d50 0x7faddc19f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:49.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.966+0000 7fadda59c700 1 -- 172.21.15.78:0/1056184502 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faddc1033b0 msgr2=0x7faddc198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:49.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.966+0000 7fadda59c700 1 --2- 172.21.15.78:0/1056184502 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faddc1033b0 0x7faddc198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:49.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.966+0000 7fadda59c700 1 -- 172.21.15.78:0/1056184502 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faddc0743c0 con 0x7faddc1021b0 2024-03-24T00:17:49.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.966+0000 7fadd9d9b700 1 --2- 172.21.15.78:0/1056184502 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faddc1033b0 0x7faddc198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:17:49.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.967+0000 7fadda59c700 1 --2- 172.21.15.78:0/1056184502 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faddc1021b0 0x7faddc1983c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fadcc00d9b0 tx=0x7fadcc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:49.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.967+0000 7fadcb7fe700 1 -- 172.21.15.78:0/1056184502 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadcc017030 con 0x7faddc1021b0 2024-03-24T00:17:49.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.967+0000 7fade0c5c700 1 -- 172.21.15.78:0/1056184502 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faddc0746b0 con 0x7faddc1021b0 2024-03-24T00:17:49.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.967+0000 7fadcb7fe700 1 -- 172.21.15.78:0/1056184502 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fadcc004500 con 0x7faddc1021b0 2024-03-24T00:17:49.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.967+0000 7fadcb7fe700 1 -- 172.21.15.78:0/1056184502 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadcc003aa0 con 0x7faddc1021b0 2024-03-24T00:17:49.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.967+0000 7fade0c5c700 1 -- 172.21.15.78:0/1056184502 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faddc074c40 con 0x7faddc1021b0 2024-03-24T00:17:49.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.968+0000 7fade0c5c700 1 -- 172.21.15.78:0/1056184502 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faddc068e50 con 0x7faddc1021b0 2024-03-24T00:17:49.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.969+0000 7fadcb7fe700 1 -- 172.21.15.78:0/1056184502 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fadcc01a070 con 0x7faddc1021b0 2024-03-24T00:17:49.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.969+0000 7fadcb7fe700 1 --2- 172.21.15.78:0/1056184502 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fadc40a2890 0x7fadc40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:49.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.969+0000 7fadd9d9b700 1 --2- 172.21.15.78:0/1056184502 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fadc40a2890 0x7fadc40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:49.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.969+0000 7fadcb7fe700 1 -- 172.21.15.78:0/1056184502 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6847+0+0 (secure 0 0 0) 0x7fadcc0ce1f0 con 0x7faddc1021b0 2024-03-24T00:17:49.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.970+0000 7fadd9d9b700 1 --2- 172.21.15.78:0/1056184502 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fadc40a2890 0x7fadc40a4d50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fadd0000c00 tx=0x7fadd0003960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:49.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:49.973+0000 7fadcb7fe700 1 -- 172.21.15.78:0/1056184502 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fadcc00e350 con 0x7faddc1021b0 2024-03-24T00:17:50.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:50.174+0000 7fade0c5c700 1 -- 172.21.15.78:0/1056184502 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7faddc076370 con 0x7faddc1021b0 2024-03-24T00:17:50.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:50.175+0000 7fadcb7fe700 1 -- 172.21.15.78:0/1056184502 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v79) v1 ==== 74+0+133 (secure 0 0 0) 0x7fadcc09b7b0 con 0x7faddc1021b0 2024-03-24T00:17:50.175 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:17:50.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:50.177+0000 7fade0c5c700 1 -- 172.21.15.78:0/1056184502 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fadc40a2890 msgr2=0x7fadc40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:50.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:50.177+0000 7fade0c5c700 1 --2- 172.21.15.78:0/1056184502 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fadc40a2890 0x7fadc40a4d50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fadd0000c00 tx=0x7fadd0003960 comp rx=0 tx=0).stop 2024-03-24T00:17:50.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:50.177+0000 7fade0c5c700 1 -- 172.21.15.78:0/1056184502 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faddc1021b0 msgr2=0x7faddc1983c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:50.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:50.177+0000 7fade0c5c700 1 --2- 172.21.15.78:0/1056184502 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faddc1021b0 0x7faddc1983c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fadcc00d9b0 tx=0x7fadcc00dcc0 comp rx=0 tx=0).stop 2024-03-24T00:17:50.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:50.177+0000 7fade0c5c700 1 -- 172.21.15.78:0/1056184502 shutdown_connections 2024-03-24T00:17:50.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:50.177+0000 7fade0c5c700 1 --2- 172.21.15.78:0/1056184502 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fadc40a2890 0x7fadc40a4d50 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:50.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:50.177+0000 7fade0c5c700 1 --2- 172.21.15.78:0/1056184502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faddc103d50 0x7faddc19f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:50.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:50.177+0000 7fade0c5c700 1 --2- 172.21.15.78:0/1056184502 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faddc1021b0 0x7faddc1983c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:50.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:50.177+0000 7fade0c5c700 1 --2- 172.21.15.78:0/1056184502 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faddc1033b0 0x7faddc198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:50.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:50.177+0000 7fade0c5c700 1 -- 172.21.15.78:0/1056184502 >> 172.21.15.78:0/1056184502 conn(0x7faddc0fd770 msgr2=0x7faddc106730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:50.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:50.177+0000 7fade0c5c700 1 -- 172.21.15.78:0/1056184502 shutdown_connections 2024-03-24T00:17:50.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:50.177+0000 7fade0c5c700 1 -- 172.21.15.78:0/1056184502 wait complete. 2024-03-24T00:17:50.817 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:50 smithi106 ceph-mon[36505]: purged_snaps scrub starts 2024-03-24T00:17:50.817 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:50 smithi106 ceph-mon[36505]: purged_snaps scrub ok 2024-03-24T00:17:50.817 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:50 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:17:50.817 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:50 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1056184502' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T00:17:50.817 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:50 smithi106 ceph-mon[36505]: osd.11 [v2:172.21.15.107:6824/4191916230,v1:172.21.15.107:6825/4191916230] boot 2024-03-24T00:17:50.817 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:50 smithi106 ceph-mon[36505]: osdmap e80: 12 total, 12 up, 12 in 2024-03-24T00:17:50.818 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:50 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:17:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:50 smithi078 ceph-mon[32429]: purged_snaps scrub starts 2024-03-24T00:17:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:50 smithi078 ceph-mon[32429]: purged_snaps scrub ok 2024-03-24T00:17:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:50 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:17:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:50 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1056184502' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T00:17:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:50 smithi078 ceph-mon[32429]: osd.11 [v2:172.21.15.107:6824/4191916230,v1:172.21.15.107:6825/4191916230] boot 2024-03-24T00:17:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:50 smithi078 ceph-mon[32429]: osdmap e80: 12 total, 12 up, 12 in 2024-03-24T00:17:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:50 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:17:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:50 smithi107 ceph-mon[36485]: purged_snaps scrub starts 2024-03-24T00:17:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:50 smithi107 ceph-mon[36485]: purged_snaps scrub ok 2024-03-24T00:17:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:50 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:17:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:50 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1056184502' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T00:17:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:50 smithi107 ceph-mon[36485]: osd.11 [v2:172.21.15.107:6824/4191916230,v1:172.21.15.107:6825/4191916230] boot 2024-03-24T00:17:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:50 smithi107 ceph-mon[36485]: osdmap e80: 12 total, 12 up, 12 in 2024-03-24T00:17:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:50 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:17:50.979 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":79,"num_osds":12,"num_up_osds":11,"osd_up_since":1711239443,"num_in_osds":12,"osd_in_since":1711239449,"num_remapped_pgs":0} 2024-03-24T00:17:51.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:51 smithi078 ceph-mon[32429]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:17:51.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:51 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:51.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:51 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:51.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:51 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:51.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:51 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:51.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:51 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:51.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:51 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:51.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:51 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:51.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:51 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:17:51.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:51 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:17:51.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:51 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:51.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:51 smithi107 ceph-mon[36485]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:17:51.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:51 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:51.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:51 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:51.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:51 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:51.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:51 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:51.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:51 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:51.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:51 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:51.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:51 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:51.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:51 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:17:51.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:51 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:17:51.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:51 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:51 smithi106 ceph-mon[36505]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:17:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:51 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:51 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:51 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:51 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:51 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:51 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:17:51.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:51 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:51.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:51 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:17:51.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:51 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:17:51.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:51 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:17:51.980 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-24T00:17:52.222 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:17:52.605 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:52 smithi078 ceph-mon[32429]: Detected new or changed devices on smithi107 2024-03-24T00:17:52.606 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:52 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi107 to 4270M 2024-03-24T00:17:52.606 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:52 smithi078 ceph-mon[32429]: osdmap e81: 12 total, 12 up, 12 in 2024-03-24T00:17:52.744 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:52 smithi107 ceph-mon[36485]: Detected new or changed devices on smithi107 2024-03-24T00:17:52.744 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:52 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi107 to 4270M 2024-03-24T00:17:52.744 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:52 smithi107 ceph-mon[36485]: osdmap e81: 12 total, 12 up, 12 in 2024-03-24T00:17:52.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:52 smithi106 ceph-mon[36505]: Detected new or changed devices on smithi107 2024-03-24T00:17:52.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:52 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi107 to 4270M 2024-03-24T00:17:52.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:52 smithi106 ceph-mon[36505]: osdmap e81: 12 total, 12 up, 12 in 2024-03-24T00:17:53.725 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:53 smithi078 ceph-mon[32429]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:17:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:53 smithi107 ceph-mon[36485]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:17:53.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:53 smithi106 ceph-mon[36505]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:17:54.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.119+0000 7f76f8843700 1 -- 172.21.15.78:0/1159678271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76f4107ee0 msgr2=0x7f76f410a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:54.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.119+0000 7f76f8843700 1 --2- 172.21.15.78:0/1159678271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76f4107ee0 0x7f76f410a310 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f76ec00b0a0 tx=0x7f76ec00b3b0 comp rx=0 tx=0).stop 2024-03-24T00:17:54.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.119+0000 7f76f8843700 1 -- 172.21.15.78:0/1159678271 shutdown_connections 2024-03-24T00:17:54.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.119+0000 7f76f8843700 1 --2- 172.21.15.78:0/1159678271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76f4107ee0 0x7f76f410a310 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:54.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.119+0000 7f76f8843700 1 --2- 172.21.15.78:0/1159678271 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76f4102c80 0x7f76f4105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:54.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.119+0000 7f76f8843700 1 --2- 172.21.15.78:0/1159678271 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76f41055b0 0x7f76f41079a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:54.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.119+0000 7f76f8843700 1 -- 172.21.15.78:0/1159678271 >> 172.21.15.78:0/1159678271 conn(0x7f76f40fc650 msgr2=0x7f76f40fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:54.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.119+0000 7f76f8843700 1 -- 172.21.15.78:0/1159678271 shutdown_connections 2024-03-24T00:17:54.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.119+0000 7f76f8843700 1 -- 172.21.15.78:0/1159678271 wait complete. 2024-03-24T00:17:54.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.120+0000 7f76f8843700 1 Processor -- start 2024-03-24T00:17:54.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.120+0000 7f76f8843700 1 -- start start 2024-03-24T00:17:54.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.120+0000 7f76f8843700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76f4102c80 0x7f76f4198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:54.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.120+0000 7f76f8843700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76f41055b0 0x7f76f41988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:54.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.120+0000 7f76f8843700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76f4107ee0 0x7f76f419f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:54.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.120+0000 7f76f8843700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f410dd60 con 0x7f76f4107ee0 2024-03-24T00:17:54.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.120+0000 7f76f8843700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f410dbe0 con 0x7f76f4102c80 2024-03-24T00:17:54.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.120+0000 7f76f8843700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f410dee0 con 0x7f76f41055b0 2024-03-24T00:17:54.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.120+0000 7f76f2d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76f4107ee0 0x7f76f419f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:54.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.120+0000 7f76f2d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76f4107ee0 0x7f76f419f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33652/0 (socket says 172.21.15.78:33652) 2024-03-24T00:17:54.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.120+0000 7f76f2d9d700 1 -- 172.21.15.78:0/2721837232 learned_addr learned my addr 172.21.15.78:0/2721837232 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:17:54.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.120+0000 7f76f1d9b700 1 --2- 172.21.15.78:0/2721837232 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76f41055b0 0x7f76f41988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:54.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.120+0000 7f76f259c700 1 --2- 172.21.15.78:0/2721837232 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76f4102c80 0x7f76f4198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:54.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.120+0000 7f76f2d9d700 1 -- 172.21.15.78:0/2721837232 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76f41055b0 msgr2=0x7f76f41988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:54.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.120+0000 7f76f2d9d700 1 --2- 172.21.15.78:0/2721837232 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76f41055b0 0x7f76f41988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:54.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.120+0000 7f76f2d9d700 1 -- 172.21.15.78:0/2721837232 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76f4102c80 msgr2=0x7f76f4198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:54.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.120+0000 7f76f2d9d700 1 --2- 172.21.15.78:0/2721837232 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76f4102c80 0x7f76f4198380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:54.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.120+0000 7f76f2d9d700 1 -- 172.21.15.78:0/2721837232 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f419ff70 con 0x7f76f4107ee0 2024-03-24T00:17:54.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.120+0000 7f76f2d9d700 1 --2- 172.21.15.78:0/2721837232 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76f4107ee0 0x7f76f419f8d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f76ec00d010 tx=0x7f76ec004d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:54.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.121+0000 7f76e37fe700 1 -- 172.21.15.78:0/2721837232 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ec021070 con 0x7f76f4107ee0 2024-03-24T00:17:54.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.121+0000 7f76f8843700 1 -- 172.21.15.78:0/2721837232 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76f41a0200 con 0x7f76f4107ee0 2024-03-24T00:17:54.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.121+0000 7f76e37fe700 1 -- 172.21.15.78:0/2721837232 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f76ec007cc0 con 0x7f76f4107ee0 2024-03-24T00:17:54.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.121+0000 7f76e37fe700 1 -- 172.21.15.78:0/2721837232 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ec0053a0 con 0x7f76f4107ee0 2024-03-24T00:17:54.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.121+0000 7f76f8843700 1 -- 172.21.15.78:0/2721837232 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76f41a0700 con 0x7f76f4107ee0 2024-03-24T00:17:54.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.122+0000 7f76f8843700 1 -- 172.21.15.78:0/2721837232 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76f4068e50 con 0x7f76f4107ee0 2024-03-24T00:17:54.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.123+0000 7f76e37fe700 1 -- 172.21.15.78:0/2721837232 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f76ec024070 con 0x7f76f4107ee0 2024-03-24T00:17:54.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.124+0000 7f76e37fe700 1 --2- 172.21.15.78:0/2721837232 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f76dc0a2890 0x7f76dc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:54.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.124+0000 7f76e37fe700 1 -- 172.21.15.78:0/2721837232 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f76ec00f020 con 0x7f76f4107ee0 2024-03-24T00:17:54.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.124+0000 7f76f259c700 1 --2- 172.21.15.78:0/2721837232 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f76dc0a2890 0x7f76dc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:54.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.125+0000 7f76f259c700 1 --2- 172.21.15.78:0/2721837232 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f76dc0a2890 0x7f76dc0a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f76e400bbd0 tx=0x7f76e400b5b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:54.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.128+0000 7f76e37fe700 1 -- 172.21.15.78:0/2721837232 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f76ec02b020 con 0x7f76f4107ee0 2024-03-24T00:17:54.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.324+0000 7f76f8843700 1 -- 172.21.15.78:0/2721837232 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f76f402da30 con 0x7f76f4107ee0 2024-03-24T00:17:54.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.324+0000 7f76e37fe700 1 -- 172.21.15.78:0/2721837232 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) v1 ==== 74+0+133 (secure 0 0 0) 0x7f76ec0074e0 con 0x7f76f4107ee0 2024-03-24T00:17:54.324 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:17:54.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.326+0000 7f76f8843700 1 -- 172.21.15.78:0/2721837232 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f76dc0a2890 msgr2=0x7f76dc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:54.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.326+0000 7f76f8843700 1 --2- 172.21.15.78:0/2721837232 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f76dc0a2890 0x7f76dc0a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f76e400bbd0 tx=0x7f76e400b5b0 comp rx=0 tx=0).stop 2024-03-24T00:17:54.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.326+0000 7f76f8843700 1 -- 172.21.15.78:0/2721837232 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76f4107ee0 msgr2=0x7f76f419f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:54.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.326+0000 7f76f8843700 1 --2- 172.21.15.78:0/2721837232 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76f4107ee0 0x7f76f419f8d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f76ec00d010 tx=0x7f76ec004d70 comp rx=0 tx=0).stop 2024-03-24T00:17:54.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.326+0000 7f76f8843700 1 -- 172.21.15.78:0/2721837232 shutdown_connections 2024-03-24T00:17:54.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.326+0000 7f76f8843700 1 --2- 172.21.15.78:0/2721837232 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f76dc0a2890 0x7f76dc0a4d50 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:54.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.326+0000 7f76f8843700 1 --2- 172.21.15.78:0/2721837232 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76f4107ee0 0x7f76f419f8d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:54.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.326+0000 7f76f8843700 1 --2- 172.21.15.78:0/2721837232 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76f4102c80 0x7f76f4198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:54.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.326+0000 7f76f8843700 1 --2- 172.21.15.78:0/2721837232 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76f41055b0 0x7f76f41988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:54.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.326+0000 7f76f8843700 1 -- 172.21.15.78:0/2721837232 >> 172.21.15.78:0/2721837232 conn(0x7f76f40fc650 msgr2=0x7f76f4103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:54.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.326+0000 7f76f8843700 1 -- 172.21.15.78:0/2721837232 shutdown_connections 2024-03-24T00:17:54.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:54.326+0000 7f76f8843700 1 -- 172.21.15.78:0/2721837232 wait complete. 2024-03-24T00:17:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:54 smithi078 ceph-mon[32429]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:17:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:54 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2721837232' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T00:17:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:54 smithi107 ceph-mon[36485]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:17:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:54 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2721837232' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T00:17:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:54 smithi106 ceph-mon[36505]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:17:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:54 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2721837232' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T00:17:55.109 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1711239470,"num_in_osds":12,"osd_in_since":1711239449,"num_remapped_pgs":0} 2024-03-24T00:17:55.110 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T00:17:55.352 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:17:57.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.039+0000 7ff280320700 1 -- 172.21.15.78:0/2913625782 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2781021e0 msgr2=0x7ff278102660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:57.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.039+0000 7ff280320700 1 --2- 172.21.15.78:0/2913625782 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2781021e0 0x7ff278102660 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff2680099c0 tx=0x7ff268009cd0 comp rx=0 tx=0).stop 2024-03-24T00:17:57.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.039+0000 7ff280320700 1 -- 172.21.15.78:0/2913625782 shutdown_connections 2024-03-24T00:17:57.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.039+0000 7ff280320700 1 --2- 172.21.15.78:0/2913625782 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff278102ba0 0x7ff27810efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:57.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.039+0000 7ff280320700 1 --2- 172.21.15.78:0/2913625782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2781081c0 0x7ff2781085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:57.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.039+0000 7ff280320700 1 --2- 172.21.15.78:0/2913625782 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2781021e0 0x7ff278102660 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:57.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.039+0000 7ff280320700 1 -- 172.21.15.78:0/2913625782 >> 172.21.15.78:0/2913625782 conn(0x7ff2780792c0 msgr2=0x7ff2780796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:57.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.040+0000 7ff280320700 1 -- 172.21.15.78:0/2913625782 shutdown_connections 2024-03-24T00:17:57.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.040+0000 7ff280320700 1 -- 172.21.15.78:0/2913625782 wait complete. 2024-03-24T00:17:57.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.040+0000 7ff280320700 1 Processor -- start 2024-03-24T00:17:57.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.040+0000 7ff280320700 1 -- start start 2024-03-24T00:17:57.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.040+0000 7ff280320700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2781021e0 0x7ff2781a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:57.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.040+0000 7ff280320700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff278102ba0 0x7ff2781a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:57.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.040+0000 7ff280320700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2781081c0 0x7ff27819a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:57.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.040+0000 7ff280320700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2781124c0 con 0x7ff278102ba0 2024-03-24T00:17:57.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.040+0000 7ff280320700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff278112340 con 0x7ff2781081c0 2024-03-24T00:17:57.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.040+0000 7ff280320700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff278112640 con 0x7ff2781021e0 2024-03-24T00:17:57.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.040+0000 7ff27d8bb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff278102ba0 0x7ff2781a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:57.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.040+0000 7ff27d8bb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff278102ba0 0x7ff2781a0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40032/0 (socket says 172.21.15.78:40032) 2024-03-24T00:17:57.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.040+0000 7ff27d8bb700 1 -- 172.21.15.78:0/2146711714 learned_addr learned my addr 172.21.15.78:0/2146711714 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:17:57.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.041+0000 7ff27e0bc700 1 --2- 172.21.15.78:0/2146711714 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2781021e0 0x7ff2781a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:57.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.041+0000 7ff27e8bd700 1 --2- 172.21.15.78:0/2146711714 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2781081c0 0x7ff27819a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:57.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.041+0000 7ff27d8bb700 1 -- 172.21.15.78:0/2146711714 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2781021e0 msgr2=0x7ff2781a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:57.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.041+0000 7ff27d8bb700 1 --2- 172.21.15.78:0/2146711714 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2781021e0 0x7ff2781a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:57.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.041+0000 7ff27d8bb700 1 -- 172.21.15.78:0/2146711714 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2781081c0 msgr2=0x7ff27819a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:57.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.041+0000 7ff27d8bb700 1 --2- 172.21.15.78:0/2146711714 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2781081c0 0x7ff27819a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:57.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.041+0000 7ff27d8bb700 1 -- 172.21.15.78:0/2146711714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff27819afd0 con 0x7ff278102ba0 2024-03-24T00:17:57.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.041+0000 7ff27e8bd700 1 --2- 172.21.15.78:0/2146711714 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2781081c0 0x7ff27819a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:17:57.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.041+0000 7ff27d8bb700 1 --2- 172.21.15.78:0/2146711714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff278102ba0 0x7ff2781a0bf0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff268006010 tx=0x7ff2680038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:57.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.041+0000 7ff26f7fe700 1 -- 172.21.15.78:0/2146711714 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff268027070 con 0x7ff278102ba0 2024-03-24T00:17:57.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.041+0000 7ff26f7fe700 1 -- 172.21.15.78:0/2146711714 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff268004050 con 0x7ff278102ba0 2024-03-24T00:17:57.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.041+0000 7ff280320700 1 -- 172.21.15.78:0/2146711714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff27819b260 con 0x7ff278102ba0 2024-03-24T00:17:57.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.041+0000 7ff26f7fe700 1 -- 172.21.15.78:0/2146711714 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff268022e40 con 0x7ff278102ba0 2024-03-24T00:17:57.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.041+0000 7ff280320700 1 -- 172.21.15.78:0/2146711714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2781a7580 con 0x7ff278102ba0 2024-03-24T00:17:57.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.042+0000 7ff280320700 1 -- 172.21.15.78:0/2146711714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff278068e50 con 0x7ff278102ba0 2024-03-24T00:17:57.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.044+0000 7ff26f7fe700 1 -- 172.21.15.78:0/2146711714 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff26802a070 con 0x7ff278102ba0 2024-03-24T00:17:57.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.045+0000 7ff26f7fe700 1 --2- 172.21.15.78:0/2146711714 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff2640a2910 0x7ff2640a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:57.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.045+0000 7ff27e0bc700 1 --2- 172.21.15.78:0/2146711714 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff2640a2910 0x7ff2640a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:57.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.045+0000 7ff26f7fe700 1 -- 172.21.15.78:0/2146711714 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff2680cfc10 con 0x7ff278102ba0 2024-03-24T00:17:57.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.045+0000 7ff27e0bc700 1 --2- 172.21.15.78:0/2146711714 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff2640a2910 0x7ff2640a4dd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff27819b9b0 tx=0x7ff27400a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:57.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.047+0000 7ff26f7fe700 1 -- 172.21.15.78:0/2146711714 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff268031070 con 0x7ff278102ba0 2024-03-24T00:17:57.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.246+0000 7ff280320700 1 -- 172.21.15.78:0/2146711714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2781a7750 con 0x7ff278102ba0 2024-03-24T00:17:57.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.247+0000 7ff26f7fe700 1 -- 172.21.15.78:0/2146711714 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18457 (secure 0 0 0) 0x7ff268013070 con 0x7ff278102ba0 2024-03-24T00:17:57.247 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:17:57.248 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":81,"fsid":"c7e2806a-e972-11ee-95cd-87774f69a715","created":"2024-03-24T00:10:44.134563+0000","modified":"2024-03-24T00:17:51.449181+0000","last_up_change":"2024-03-24T00:17:50.237686+0000","last_in_change":"2024-03-24T00:17:29.788699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:14:04.534391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b950388a-c437-426b-b703-3f8ae821dd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6803","nonce":1816879397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6805","nonce":1816879397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6809","nonce":1816879397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6807","nonce":1816879397}]},"public_addr":"172.21.15.78:6803/1816879397","cluster_addr":"172.21.15.78:6805/1816879397","heartbeat_back_addr":"172.21.15.78:6809/1816879397","heartbeat_front_addr":"172.21.15.78:6807/1816879397","state":["exists","up"]},{"osd":1,"uuid":"b8d64714-dd65-400f-b3f4-673ab14e1365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6801","nonce":1647915612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6803","nonce":1647915612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6807","nonce":1647915612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6805","nonce":1647915612}]},"public_addr":"172.21.15.106:6801/1647915612","cluster_addr":"172.21.15.106:6803/1647915612","heartbeat_back_addr":"172.21.15.106:6807/1647915612","heartbeat_front_addr":"172.21.15.106:6805/1647915612","state":["exists","up"]},{"osd":2,"uuid":"fcfc4dd1-aaf8-4131-a023-5e832212158d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6801","nonce":2010770435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6803","nonce":2010770435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6807","nonce":2010770435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6805","nonce":2010770435}]},"public_addr":"172.21.15.107:6801/2010770435","cluster_addr":"172.21.15.107:6803/2010770435","heartbeat_back_addr":"172.21.15.107:6807/2010770435","heartbeat_front_addr":"172.21.15.107:6805/2010770435","state":["exists","up"]},{"osd":3,"uuid":"47c718ab-a0f4-4e52-83c5-b89386b262d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6811","nonce":1912251346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6813","nonce":1912251346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6817","nonce":1912251346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6815","nonce":1912251346}]},"public_addr":"172.21.15.78:6811/1912251346","cluster_addr":"172.21.15.78:6813/1912251346","heartbeat_back_addr":"172.21.15.78:6817/1912251346","heartbeat_front_addr":"172.21.15.78:6815/1912251346","state":["exists","up"]},{"osd":4,"uuid":"7dad64dd-677e-4c1c-9638-e4f4f2195983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6809","nonce":374641205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6811","nonce":374641205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6815","nonce":374641205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6813","nonce":374641205}]},"public_addr":"172.21.15.106:6809/374641205","cluster_addr":"172.21.15.106:6811/374641205","heartbeat_back_addr":"172.21.15.106:6815/374641205","heartbeat_front_addr":"172.21.15.106:6813/374641205","state":["exists","up"]},{"osd":5,"uuid":"bbd524e5-328d-4309-8877-1149cbe110dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6809","nonce":3272806073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6811","nonce":3272806073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6815","nonce":3272806073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6813","nonce":3272806073}]},"public_addr":"172.21.15.107:6809/3272806073","cluster_addr":"172.21.15.107:6811/3272806073","heartbeat_back_addr":"172.21.15.107:6815/3272806073","heartbeat_front_addr":"172.21.15.107:6813/3272806073","state":["exists","up"]},{"osd":6,"uuid":"5696389a-918f-4ea8-b20d-4e2a0d4cb3bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6819","nonce":3091148508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6821","nonce":3091148508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6825","nonce":3091148508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6823","nonce":3091148508}]},"public_addr":"172.21.15.78:6819/3091148508","cluster_addr":"172.21.15.78:6821/3091148508","heartbeat_back_addr":"172.21.15.78:6825/3091148508","heartbeat_front_addr":"172.21.15.78:6823/3091148508","state":["exists","up"]},{"osd":7,"uuid":"f6b764e8-ed4e-4c9a-a85b-94af6dc6b924","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6817","nonce":4262103080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6819","nonce":4262103080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6823","nonce":4262103080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6821","nonce":4262103080}]},"public_addr":"172.21.15.106:6817/4262103080","cluster_addr":"172.21.15.106:6819/4262103080","heartbeat_back_addr":"172.21.15.106:6823/4262103080","heartbeat_front_addr":"172.21.15.106:6821/4262103080","state":["exists","up"]},{"osd":8,"uuid":"ba16039f-8093-4fa0-87ad-e28b7d142b23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6817","nonce":4032799356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6819","nonce":4032799356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6823","nonce":4032799356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6821","nonce":4032799356}]},"public_addr":"172.21.15.107:6817/4032799356","cluster_addr":"172.21.15.107:6819/4032799356","heartbeat_back_addr":"172.21.15.107:6823/4032799356","heartbeat_front_addr":"172.21.15.107:6821/4032799356","state":["exists","up"]},{"osd":9,"uuid":"651357f2-0669-42e6-aa91-1d5c02d00c5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6827","nonce":214023431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6829","nonce":214023431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6833","nonce":214023431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6831","nonce":214023431}]},"public_addr":"172.21.15.78:6827/214023431","cluster_addr":"172.21.15.78:6829/214023431","heartbeat_back_addr":"172.21.15.78:6833/214023431","heartbeat_front_addr":"172.21.15.78:6831/214023431","state":["exists","up"]},{"osd":10,"uuid":"2375119d-346e-426c-8a65-b64af71ad63a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6825","nonce":400029804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6827","nonce":400029804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6831","nonce":400029804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6829","nonce":400029804}]},"public_addr":"172.21.15.106:6825/400029804","cluster_addr":"172.21.15.106:6827/400029804","heartbeat_back_addr":"172.21.15.106:6831/400029804","heartbeat_front_addr":"172.21.15.106:6829/400029804","state":["exists","up"]},{"osd":11,"uuid":"b6b609d9-dd78-4c01-ae66-ba2a662ab22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6825","nonce":4191916230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6827","nonce":4191916230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6831","nonce":4191916230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6829","nonce":4191916230}]},"public_addr":"172.21.15.107:6825/4191916230","cluster_addr":"172.21.15.107:6827/4191916230","heartbeat_back_addr":"172.21.15.107:6831/4191916230","heartbeat_front_addr":"172.21.15.107:6829/4191916230","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:13.728602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:37.320809+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:01.269452+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:25.758281+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:50.298863+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:14.345219+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:38.980489+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:04.749214+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:29.992673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:56.269202+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:22.270316+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:48.074907+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/1982202541":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/1933054339":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6801/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6800/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/465691596":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6801/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/3962523609":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6800/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/1502752625":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/2236879401":"2024-03-25T00:11:40.199569+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:17:57.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.249+0000 7ff280320700 1 -- 172.21.15.78:0/2146711714 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff2640a2910 msgr2=0x7ff2640a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:57.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.249+0000 7ff280320700 1 --2- 172.21.15.78:0/2146711714 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff2640a2910 0x7ff2640a4dd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff27819b9b0 tx=0x7ff27400a380 comp rx=0 tx=0).stop 2024-03-24T00:17:57.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.249+0000 7ff280320700 1 -- 172.21.15.78:0/2146711714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff278102ba0 msgr2=0x7ff2781a0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:57.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.249+0000 7ff280320700 1 --2- 172.21.15.78:0/2146711714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff278102ba0 0x7ff2781a0bf0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff268006010 tx=0x7ff2680038b0 comp rx=0 tx=0).stop 2024-03-24T00:17:57.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.249+0000 7ff280320700 1 -- 172.21.15.78:0/2146711714 shutdown_connections 2024-03-24T00:17:57.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.249+0000 7ff280320700 1 --2- 172.21.15.78:0/2146711714 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff2640a2910 0x7ff2640a4dd0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:57.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.249+0000 7ff280320700 1 --2- 172.21.15.78:0/2146711714 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2781081c0 0x7ff27819a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:57.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.249+0000 7ff280320700 1 --2- 172.21.15.78:0/2146711714 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2781021e0 0x7ff2781a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:57.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.249+0000 7ff280320700 1 --2- 172.21.15.78:0/2146711714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff278102ba0 0x7ff2781a0bf0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:57.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.249+0000 7ff280320700 1 -- 172.21.15.78:0/2146711714 >> 172.21.15.78:0/2146711714 conn(0x7ff2780792c0 msgr2=0x7ff27810d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:57.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.249+0000 7ff280320700 1 -- 172.21.15.78:0/2146711714 shutdown_connections 2024-03-24T00:17:57.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:57.249+0000 7ff280320700 1 -- 172.21.15.78:0/2146711714 wait complete. 2024-03-24T00:17:57.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:57 smithi078 ceph-mon[32429]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:17:57.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:57 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2146711714' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:17:57.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:57 smithi107 ceph-mon[36485]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:17:57.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:57 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2146711714' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:17:57.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:57 smithi106 ceph-mon[36505]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:17:57.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:57 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2146711714' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:17:57.971 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:14:04.534391+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:17:57.972 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-24T00:17:58.208 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:17:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:17:59 smithi078 ceph-mon[32429]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:17:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:17:59 smithi107 ceph-mon[36485]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:17:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:17:59 smithi106 ceph-mon[36505]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:17:59.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.952+0000 7f72b83a7700 1 -- 172.21.15.78:0/230259986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72b01034a0 msgr2=0x7f72b0103880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:59.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.952+0000 7f72b83a7700 1 --2- 172.21.15.78:0/230259986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72b01034a0 0x7f72b0103880 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f72ac009990 tx=0x7f72ac009ca0 comp rx=0 tx=0).stop 2024-03-24T00:17:59.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.952+0000 7f72b83a7700 1 -- 172.21.15.78:0/230259986 shutdown_connections 2024-03-24T00:17:59.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.952+0000 7f72b83a7700 1 --2- 172.21.15.78:0/230259986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72b010c880 0x7f72b010ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:59.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.952+0000 7f72b83a7700 1 --2- 172.21.15.78:0/230259986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72b01034a0 0x7f72b0103880 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:59.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.952+0000 7f72b83a7700 1 --2- 172.21.15.78:0/230259986 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72b0103dc0 0x7f72b010c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:59.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.952+0000 7f72b83a7700 1 -- 172.21.15.78:0/230259986 >> 172.21.15.78:0/230259986 conn(0x7f72b00781c0 msgr2=0x7f72b00785d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:59.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.952+0000 7f72b83a7700 1 -- 172.21.15.78:0/230259986 shutdown_connections 2024-03-24T00:17:59.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.952+0000 7f72b83a7700 1 -- 172.21.15.78:0/230259986 wait complete. 2024-03-24T00:17:59.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.952+0000 7f72b83a7700 1 Processor -- start 2024-03-24T00:17:59.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.952+0000 7f72b83a7700 1 -- start start 2024-03-24T00:17:59.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.952+0000 7f72b83a7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72b01034a0 0x7f72b0101100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:59.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.952+0000 7f72b83a7700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72b0103dc0 0x7f72b0101640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:59.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.953+0000 7f72b83a7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72b010c880 0x7f72b0101b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:59.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.953+0000 7f72b83a7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b01125f0 con 0x7f72b010c880 2024-03-24T00:17:59.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.953+0000 7f72b83a7700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b0112470 con 0x7f72b01034a0 2024-03-24T00:17:59.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.953+0000 7f72b83a7700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b0112770 con 0x7f72b0103dc0 2024-03-24T00:17:59.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.953+0000 7f72b6944700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72b010c880 0x7f72b0101b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:59.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.953+0000 7f72b6944700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72b010c880 0x7f72b0101b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40046/0 (socket says 172.21.15.78:40046) 2024-03-24T00:17:59.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.953+0000 7f72b6944700 1 -- 172.21.15.78:0/187442364 learned_addr learned my addr 172.21.15.78:0/187442364 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:17:59.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.953+0000 7f72b6143700 1 --2- 172.21.15.78:0/187442364 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72b01034a0 0x7f72b0101100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:59.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.953+0000 7f72b5942700 1 --2- 172.21.15.78:0/187442364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72b0103dc0 0x7f72b0101640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:59.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.953+0000 7f72b6944700 1 -- 172.21.15.78:0/187442364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72b0103dc0 msgr2=0x7f72b0101640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:59.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.953+0000 7f72b6944700 1 --2- 172.21.15.78:0/187442364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72b0103dc0 0x7f72b0101640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:59.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.953+0000 7f72b6944700 1 -- 172.21.15.78:0/187442364 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72b01034a0 msgr2=0x7f72b0101100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:59.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.953+0000 7f72b6944700 1 --2- 172.21.15.78:0/187442364 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72b01034a0 0x7f72b0101100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:59.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.953+0000 7f72b6944700 1 -- 172.21.15.78:0/187442364 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b01a6f80 con 0x7f72b010c880 2024-03-24T00:17:59.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.953+0000 7f72b6143700 1 --2- 172.21.15.78:0/187442364 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72b01034a0 0x7f72b0101100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:17:59.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.953+0000 7f72b6944700 1 --2- 172.21.15.78:0/187442364 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72b010c880 0x7f72b0101b80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f72a800ada0 tx=0x7f72a800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:59.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.953+0000 7f72a77fe700 1 -- 172.21.15.78:0/187442364 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a8017020 con 0x7f72b010c880 2024-03-24T00:17:59.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.953+0000 7f72b83a7700 1 -- 172.21.15.78:0/187442364 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72b01a7270 con 0x7f72b010c880 2024-03-24T00:17:59.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.953+0000 7f72a77fe700 1 -- 172.21.15.78:0/187442364 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72a8007e30 con 0x7f72b010c880 2024-03-24T00:17:59.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.953+0000 7f72a77fe700 1 -- 172.21.15.78:0/187442364 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a8005500 con 0x7f72b010c880 2024-03-24T00:17:59.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.953+0000 7f72b83a7700 1 -- 172.21.15.78:0/187442364 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72b01a7760 con 0x7f72b010c880 2024-03-24T00:17:59.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.954+0000 7f72b83a7700 1 -- 172.21.15.78:0/187442364 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72b0068e50 con 0x7f72b010c880 2024-03-24T00:17:59.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.956+0000 7f72a77fe700 1 -- 172.21.15.78:0/187442364 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f72a8004ad0 con 0x7f72b010c880 2024-03-24T00:17:59.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.957+0000 7f72a77fe700 1 --2- 172.21.15.78:0/187442364 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f729c0a2890 0x7f729c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:59.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.957+0000 7f72b6143700 1 --2- 172.21.15.78:0/187442364 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f729c0a2890 0x7f729c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:59.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.957+0000 7f72a77fe700 1 -- 172.21.15.78:0/187442364 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f72a80ce440 con 0x7f72b010c880 2024-03-24T00:17:59.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.957+0000 7f72b6143700 1 --2- 172.21.15.78:0/187442364 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f729c0a2890 0x7f729c0a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f72ac00b5c0 tx=0x7f72ac005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:59.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:17:59.960+0000 7f72a77fe700 1 -- 172.21.15.78:0/187442364 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f72a8021020 con 0x7f72b010c880 2024-03-24T00:18:00.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:00.157+0000 7f72b83a7700 1 -- 172.21.15.78:0/187442364 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f72b002da30 con 0x7f72b010c880 2024-03-24T00:18:00.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:00.158+0000 7f72a77fe700 1 -- 172.21.15.78:0/187442364 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f72a809ba10 con 0x7f72b010c880 2024-03-24T00:18:00.158 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-24T00:18:00.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:00.160+0000 7f72b83a7700 1 -- 172.21.15.78:0/187442364 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f729c0a2890 msgr2=0x7f729c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:00.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:00.160+0000 7f72b83a7700 1 --2- 172.21.15.78:0/187442364 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f729c0a2890 0x7f729c0a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f72ac00b5c0 tx=0x7f72ac005fb0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:00.160+0000 7f72b83a7700 1 -- 172.21.15.78:0/187442364 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72b010c880 msgr2=0x7f72b0101b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:00.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:00.160+0000 7f72b83a7700 1 --2- 172.21.15.78:0/187442364 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72b010c880 0x7f72b0101b80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f72a800ada0 tx=0x7f72a800c330 comp rx=0 tx=0).stop 2024-03-24T00:18:00.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:00.160+0000 7f72b83a7700 1 -- 172.21.15.78:0/187442364 shutdown_connections 2024-03-24T00:18:00.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:00.160+0000 7f72b83a7700 1 --2- 172.21.15.78:0/187442364 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f729c0a2890 0x7f729c0a4d50 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:00.160+0000 7f72b83a7700 1 --2- 172.21.15.78:0/187442364 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72b010c880 0x7f72b0101b80 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:00.160+0000 7f72b83a7700 1 --2- 172.21.15.78:0/187442364 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72b01034a0 0x7f72b0101100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:00.160+0000 7f72b83a7700 1 --2- 172.21.15.78:0/187442364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72b0103dc0 0x7f72b0101640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:00.160+0000 7f72b83a7700 1 -- 172.21.15.78:0/187442364 >> 172.21.15.78:0/187442364 conn(0x7f72b00781c0 msgr2=0x7f72b00fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:00.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:00.160+0000 7f72b83a7700 1 -- 172.21.15.78:0/187442364 shutdown_connections 2024-03-24T00:18:00.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:00.160+0000 7f72b83a7700 1 -- 172.21.15.78:0/187442364 wait complete. 2024-03-24T00:18:00.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:00 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/187442364' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:18:00.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:00 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/187442364' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:18:00.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:00 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/187442364' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:18:01.005 INFO:tasks.cephadm:Adding mds.a on smithi078 2024-03-24T00:18:01.005 INFO:tasks.cephadm:Adding mds.d on smithi078 2024-03-24T00:18:01.005 INFO:tasks.cephadm:Adding mds.g on smithi078 2024-03-24T00:18:01.005 INFO:tasks.cephadm:Adding mds.j on smithi078 2024-03-24T00:18:01.005 INFO:tasks.cephadm:Adding mds.b on smithi106 2024-03-24T00:18:01.005 INFO:tasks.cephadm:Adding mds.e on smithi106 2024-03-24T00:18:01.005 INFO:tasks.cephadm:Adding mds.h on smithi106 2024-03-24T00:18:01.005 INFO:tasks.cephadm:Adding mds.k on smithi106 2024-03-24T00:18:01.005 INFO:tasks.cephadm:Adding mds.c on smithi107 2024-03-24T00:18:01.005 INFO:tasks.cephadm:Adding mds.f on smithi107 2024-03-24T00:18:01.005 INFO:tasks.cephadm:Adding mds.i on smithi107 2024-03-24T00:18:01.005 INFO:tasks.cephadm:Adding mds.l on smithi107 2024-03-24T00:18:01.006 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph orch apply mds all '12;smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi106=b;smithi106=e;smithi106=h;smithi106=k;smithi107=c;smithi107=f;smithi107=i;smithi107=l' 2024-03-24T00:18:01.255 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:18:01.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:01 smithi078 ceph-mon[32429]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:01.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:01 smithi107 ceph-mon[36485]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:01.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:01 smithi106 ceph-mon[36505]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.932+0000 7fe12d9db700 1 -- 172.21.15.107:0/2625255594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe128103e50 msgr2=0x7fe12810a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.932+0000 7fe12d9db700 1 --2- 172.21.15.107:0/2625255594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe128103e50 0x7fe12810a700 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe12000b0a0 tx=0x7fe12000b3b0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.932+0000 7fe12d9db700 1 -- 172.21.15.107:0/2625255594 shutdown_connections 2024-03-24T00:18:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.932+0000 7fe12d9db700 1 --2- 172.21.15.107:0/2625255594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe128103e50 0x7fe12810a700 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.932+0000 7fe12d9db700 1 --2- 172.21.15.107:0/2625255594 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1281004d0 0x7fe1281008f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.932+0000 7fe12d9db700 1 --2- 172.21.15.107:0/2625255594 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1281034b0 0x7fe128103910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.932+0000 7fe12d9db700 1 -- 172.21.15.107:0/2625255594 >> 172.21.15.107:0/2625255594 conn(0x7fe1280fbe30 msgr2=0x7fe1280fe270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.932+0000 7fe12d9db700 1 -- 172.21.15.107:0/2625255594 shutdown_connections 2024-03-24T00:18:02.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.932+0000 7fe12d9db700 1 -- 172.21.15.107:0/2625255594 wait complete. 2024-03-24T00:18:02.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.933+0000 7fe12d9db700 1 Processor -- start 2024-03-24T00:18:02.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.933+0000 7fe12d9db700 1 -- start start 2024-03-24T00:18:02.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.933+0000 7fe12d9db700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1281004d0 0x7fe1281982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.933+0000 7fe12d9db700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1281034b0 0x7fe128198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.933+0000 7fe12d9db700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe128103e50 0x7fe12819f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.933+0000 7fe12d9db700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12810de00 con 0x7fe128103e50 2024-03-24T00:18:02.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.933+0000 7fe12d9db700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12810dc80 con 0x7fe1281034b0 2024-03-24T00:18:02.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.933+0000 7fe12d9db700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12810df80 con 0x7fe1281004d0 2024-03-24T00:18:02.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.933+0000 7fe126ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1281004d0 0x7fe1281982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.933+0000 7fe126ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1281004d0 0x7fe1281982c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55396/0 (socket says 172.21.15.107:55396) 2024-03-24T00:18:02.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.933+0000 7fe126ffd700 1 -- 172.21.15.107:0/3992168207 learned_addr learned my addr 172.21.15.107:0/3992168207 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-03-24T00:18:02.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.933+0000 7fe11e7fc700 1 --2- 172.21.15.107:0/3992168207 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1281034b0 0x7fe128198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.933+0000 7fe1277fe700 1 --2- 172.21.15.107:0/3992168207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe128103e50 0x7fe12819f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.933+0000 7fe126ffd700 1 -- 172.21.15.107:0/3992168207 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1281034b0 msgr2=0x7fe128198800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:02.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.933+0000 7fe126ffd700 1 --2- 172.21.15.107:0/3992168207 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1281034b0 0x7fe128198800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.933+0000 7fe126ffd700 1 -- 172.21.15.107:0/3992168207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe128103e50 msgr2=0x7fe12819f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:02.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.933+0000 7fe126ffd700 1 --2- 172.21.15.107:0/3992168207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe128103e50 0x7fe12819f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.934+0000 7fe126ffd700 1 -- 172.21.15.107:0/3992168207 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe12819ff20 con 0x7fe1281004d0 2024-03-24T00:18:02.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.934+0000 7fe1277fe700 1 --2- 172.21.15.107:0/3992168207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe128103e50 0x7fe12819f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:18:02.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.934+0000 7fe126ffd700 1 --2- 172.21.15.107:0/3992168207 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1281004d0 0x7fe1281982c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe11800d9b0 tx=0x7fe11800dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:02.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.934+0000 7fe12c9d9700 1 -- 172.21.15.107:0/3992168207 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe118017070 con 0x7fe1281004d0 2024-03-24T00:18:02.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.934+0000 7fe12d9db700 1 -- 172.21.15.107:0/3992168207 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1281a0210 con 0x7fe1281004d0 2024-03-24T00:18:02.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.934+0000 7fe12c9d9700 1 -- 172.21.15.107:0/3992168207 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe118009e10 con 0x7fe1281004d0 2024-03-24T00:18:02.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.934+0000 7fe12c9d9700 1 -- 172.21.15.107:0/3992168207 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11800f620 con 0x7fe1281004d0 2024-03-24T00:18:02.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.934+0000 7fe12d9db700 1 -- 172.21.15.107:0/3992168207 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1281a0750 con 0x7fe1281004d0 2024-03-24T00:18:02.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.935+0000 7fe12d9db700 1 -- 172.21.15.107:0/3992168207 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe128068e50 con 0x7fe1281004d0 2024-03-24T00:18:02.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.936+0000 7fe12c9d9700 1 -- 172.21.15.107:0/3992168207 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe118009930 con 0x7fe1281004d0 2024-03-24T00:18:02.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.936+0000 7fe12c9d9700 1 --2- 172.21.15.107:0/3992168207 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe1140a2890 0x7fe1140a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.937+0000 7fe12c9d9700 1 -- 172.21.15.107:0/3992168207 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe1180ce120 con 0x7fe1281004d0 2024-03-24T00:18:02.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.937+0000 7fe11e7fc700 1 --2- 172.21.15.107:0/3992168207 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe1140a2890 0x7fe1140a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.937+0000 7fe11e7fc700 1 --2- 172.21.15.107:0/3992168207 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe1140a2890 0x7fe1140a4d50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe11000ac50 tx=0x7fe11000a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:02.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:02.940+0000 7fe12c9d9700 1 -- 172.21.15.107:0/3992168207 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe11809b660 con 0x7fe1281004d0 2024-03-24T00:18:03.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:03.165+0000 7fe12d9db700 1 -- 172.21.15.107:0/3992168207 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi106=b;smithi106=e;smithi106=h;smithi106=k;smithi107=c;smithi107=f;smithi107=i;smithi107=l", "target": ["mon-mgr", ""]}) v1 -- 0x7fe128062890 con 0x7fe1140a2890 2024-03-24T00:18:03.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:03.173+0000 7fe12c9d9700 1 -- 172.21.15.107:0/3992168207 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fe128062890 con 0x7fe1140a2890 2024-03-24T00:18:03.173 INFO:teuthology.orchestra.run.smithi107.stdout:Scheduled mds.all update... 2024-03-24T00:18:03.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:03.175+0000 7fe12d9db700 1 -- 172.21.15.107:0/3992168207 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe1140a2890 msgr2=0x7fe1140a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:03.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:03.175+0000 7fe12d9db700 1 --2- 172.21.15.107:0/3992168207 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe1140a2890 0x7fe1140a4d50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe11000ac50 tx=0x7fe11000a380 comp rx=0 tx=0).stop 2024-03-24T00:18:03.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:03.175+0000 7fe12d9db700 1 -- 172.21.15.107:0/3992168207 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1281004d0 msgr2=0x7fe1281982c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:03.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:03.175+0000 7fe12d9db700 1 --2- 172.21.15.107:0/3992168207 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1281004d0 0x7fe1281982c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe11800d9b0 tx=0x7fe11800dcc0 comp rx=0 tx=0).stop 2024-03-24T00:18:03.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:03.175+0000 7fe12d9db700 1 -- 172.21.15.107:0/3992168207 shutdown_connections 2024-03-24T00:18:03.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:03.175+0000 7fe12d9db700 1 --2- 172.21.15.107:0/3992168207 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe1140a2890 0x7fe1140a4d50 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:03.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:03.175+0000 7fe12d9db700 1 --2- 172.21.15.107:0/3992168207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe128103e50 0x7fe12819f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:03.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:03.175+0000 7fe12d9db700 1 --2- 172.21.15.107:0/3992168207 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1281004d0 0x7fe1281982c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:03.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:03.175+0000 7fe12d9db700 1 --2- 172.21.15.107:0/3992168207 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1281034b0 0x7fe128198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:03.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:03.175+0000 7fe12d9db700 1 -- 172.21.15.107:0/3992168207 >> 172.21.15.107:0/3992168207 conn(0x7fe1280fbe30 msgr2=0x7fe1281066e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:03.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:03.176+0000 7fe12d9db700 1 -- 172.21.15.107:0/3992168207 shutdown_connections 2024-03-24T00:18:03.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-24T00:18:03.176+0000 7fe12d9db700 1 -- 172.21.15.107:0/3992168207 wait complete. 2024-03-24T00:18:03.571 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:03 smithi107 ceph-mon[36485]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:03.571 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:03 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:03.571 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:03 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:18:03.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:03 smithi078 ceph-mon[32429]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:03.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:03 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:03.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:03 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:18:03.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:03 smithi106 ceph-mon[36505]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:03.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:03 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:03.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:03 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:18:03.854 DEBUG:teuthology.orchestra.run.smithi078:mds.a> sudo journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.a.service 2024-03-24T00:18:03.857 DEBUG:teuthology.orchestra.run.smithi078:mds.d> sudo journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.d.service 2024-03-24T00:18:03.900 DEBUG:teuthology.orchestra.run.smithi078:mds.g> sudo journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.g.service 2024-03-24T00:18:03.944 DEBUG:teuthology.orchestra.run.smithi078:mds.j> sudo journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.j.service 2024-03-24T00:18:03.987 DEBUG:teuthology.orchestra.run.smithi106:mds.b> sudo journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.b.service 2024-03-24T00:18:03.989 DEBUG:teuthology.orchestra.run.smithi106:mds.e> sudo journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.e.service 2024-03-24T00:18:04.032 DEBUG:teuthology.orchestra.run.smithi106:mds.h> sudo journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.h.service 2024-03-24T00:18:04.076 DEBUG:teuthology.orchestra.run.smithi106:mds.k> sudo journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.k.service 2024-03-24T00:18:04.119 DEBUG:teuthology.orchestra.run.smithi107:mds.c> sudo journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.c.service 2024-03-24T00:18:04.122 DEBUG:teuthology.orchestra.run.smithi107:mds.f> sudo journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.f.service 2024-03-24T00:18:04.165 DEBUG:teuthology.orchestra.run.smithi107:mds.i> sudo journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.i.service 2024-03-24T00:18:04.208 DEBUG:teuthology.orchestra.run.smithi107:mds.l> sudo journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.l.service 2024-03-24T00:18:04.251 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-24T00:18:04.252 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:18:04.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.376+0000 7fcf61fb3640 1 -- 172.21.15.78:0/2987235240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf5c1021c0 msgr2=0x7fcf5c1045b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:04.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.376+0000 7fcf61fb3640 1 --2- 172.21.15.78:0/2987235240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf5c1021c0 0x7fcf5c1045b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c009900 tx=0x7fcf4c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:04.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.376+0000 7fcf61fb3640 1 -- 172.21.15.78:0/2987235240 shutdown_connections 2024-03-24T00:18:04.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.376+0000 7fcf61fb3640 1 --2- 172.21.15.78:0/2987235240 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf5c107420 0x7fcf5c109810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.376+0000 7fcf61fb3640 1 --2- 172.21.15.78:0/2987235240 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcf5c104af0 0x7fcf5c106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.376+0000 7fcf61fb3640 1 --2- 172.21.15.78:0/2987235240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf5c1021c0 0x7fcf5c1045b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.376+0000 7fcf61fb3640 1 -- 172.21.15.78:0/2987235240 >> 172.21.15.78:0/2987235240 conn(0x7fcf5c0fbb30 msgr2=0x7fcf5c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:04.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.376+0000 7fcf61fb3640 1 -- 172.21.15.78:0/2987235240 shutdown_connections 2024-03-24T00:18:04.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.376+0000 7fcf61fb3640 1 -- 172.21.15.78:0/2987235240 wait complete. 2024-03-24T00:18:04.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.376+0000 7fcf61fb3640 1 Processor -- start 2024-03-24T00:18:04.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.377+0000 7fcf61fb3640 1 -- start start 2024-03-24T00:18:04.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.377+0000 7fcf61fb3640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcf5c1021c0 0x7fcf5c101b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.377+0000 7fcf61fb3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf5c104af0 0x7fcf5c100170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.377+0000 7fcf61fb3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf5c107420 0x7fcf5c1006b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.377+0000 7fcf61fb3640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf5c10d360 con 0x7fcf5c107420 2024-03-24T00:18:04.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.377+0000 7fcf61fb3640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf5c10d1e0 con 0x7fcf5c1021c0 2024-03-24T00:18:04.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.377+0000 7fcf61fb3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf5c10d4e0 con 0x7fcf5c104af0 2024-03-24T00:18:04.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.377+0000 7fcf5bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf5c107420 0x7fcf5c1006b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.377+0000 7fcf5bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf5c107420 0x7fcf5c1006b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40052/0 (socket says 172.21.15.78:40052) 2024-03-24T00:18:04.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.377+0000 7fcf5bfff640 1 -- 172.21.15.78:0/3609702840 learned_addr learned my addr 172.21.15.78:0/3609702840 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:04.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.377+0000 7fcf5b7fe640 1 --2- 172.21.15.78:0/3609702840 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcf5c1021c0 0x7fcf5c101b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.377+0000 7fcf5affd640 1 --2- 172.21.15.78:0/3609702840 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf5c104af0 0x7fcf5c100170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.377+0000 7fcf5bfff640 1 -- 172.21.15.78:0/3609702840 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf5c104af0 msgr2=0x7fcf5c100170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:04.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.377+0000 7fcf5bfff640 1 --2- 172.21.15.78:0/3609702840 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf5c104af0 0x7fcf5c100170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.377+0000 7fcf5bfff640 1 -- 172.21.15.78:0/3609702840 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcf5c1021c0 msgr2=0x7fcf5c101b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:04.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.377+0000 7fcf5bfff640 1 --2- 172.21.15.78:0/3609702840 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcf5c1021c0 0x7fcf5c101b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.377+0000 7fcf5bfff640 1 -- 172.21.15.78:0/3609702840 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf5c100f40 con 0x7fcf5c107420 2024-03-24T00:18:04.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.377+0000 7fcf5affd640 1 --2- 172.21.15.78:0/3609702840 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf5c104af0 0x7fcf5c100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:18:04.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.377+0000 7fcf5bfff640 1 --2- 172.21.15.78:0/3609702840 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf5c107420 0x7fcf5c1006b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fcf5400eb10 tx=0x7fcf5400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:04.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.377+0000 7fcf58ff9640 1 -- 172.21.15.78:0/3609702840 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf54004b20 con 0x7fcf5c107420 2024-03-24T00:18:04.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.377+0000 7fcf61fb3640 1 -- 172.21.15.78:0/3609702840 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf5c1a1b20 con 0x7fcf5c107420 2024-03-24T00:18:04.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.377+0000 7fcf58ff9640 1 -- 172.21.15.78:0/3609702840 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf54004cc0 con 0x7fcf5c107420 2024-03-24T00:18:04.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.377+0000 7fcf58ff9640 1 -- 172.21.15.78:0/3609702840 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf54010760 con 0x7fcf5c107420 2024-03-24T00:18:04.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.377+0000 7fcf61fb3640 1 -- 172.21.15.78:0/3609702840 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf5c1a20c0 con 0x7fcf5c107420 2024-03-24T00:18:04.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.378+0000 7fcf61fb3640 1 -- 172.21.15.78:0/3609702840 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf5c0fd620 con 0x7fcf5c107420 2024-03-24T00:18:04.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.379+0000 7fcf58ff9640 1 -- 172.21.15.78:0/3609702840 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fcf54005020 con 0x7fcf5c107420 2024-03-24T00:18:04.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.380+0000 7fcf58ff9640 1 --2- 172.21.15.78:0/3609702840 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fcf440a2800 0x7fcf440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.380+0000 7fcf58ff9640 1 -- 172.21.15.78:0/3609702840 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fcf540ced70 con 0x7fcf5c107420 2024-03-24T00:18:04.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.380+0000 7fcf5b7fe640 1 --2- 172.21.15.78:0/3609702840 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fcf440a2800 0x7fcf440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.380+0000 7fcf5b7fe640 1 --2- 172.21.15.78:0/3609702840 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fcf440a2800 0x7fcf440a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c02c7b0 tx=0x7fcf4c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:04.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.384+0000 7fcf58ff9640 1 -- 172.21.15.78:0/3609702840 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcf5409c2b0 con 0x7fcf5c107420 2024-03-24T00:18:04.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:04 smithi107 ceph-mon[36485]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi106=b;smithi106=e;smithi106=h;smithi106=k;smithi107=c;smithi107=f;smithi107=i;smithi107=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:18:04.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:04 smithi107 ceph-mon[36485]: Saving service mds.all spec with placement smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi106=b;smithi106=e;smithi106=h;smithi106=k;smithi107=c;smithi107=f;smithi107=i;smithi107=l;count:12 2024-03-24T00:18:04.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:04 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:04.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:04 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:18:04.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:04 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:04.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:04 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:04.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:04 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:04.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:04 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:04.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.561+0000 7fcf61fb3640 1 -- 172.21.15.78:0/3609702840 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf5c02d220 con 0x7fcf5c107420 2024-03-24T00:18:04.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.562+0000 7fcf58ff9640 1 -- 172.21.15.78:0/3609702840 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18457 (secure 0 0 0) 0x7fcf5409bbb0 con 0x7fcf5c107420 2024-03-24T00:18:04.563 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:18:04.563 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":81,"fsid":"c7e2806a-e972-11ee-95cd-87774f69a715","created":"2024-03-24T00:10:44.134563+0000","modified":"2024-03-24T00:17:51.449181+0000","last_up_change":"2024-03-24T00:17:50.237686+0000","last_in_change":"2024-03-24T00:17:29.788699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:14:04.534391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b950388a-c437-426b-b703-3f8ae821dd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6803","nonce":1816879397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6805","nonce":1816879397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6809","nonce":1816879397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6807","nonce":1816879397}]},"public_addr":"172.21.15.78:6803/1816879397","cluster_addr":"172.21.15.78:6805/1816879397","heartbeat_back_addr":"172.21.15.78:6809/1816879397","heartbeat_front_addr":"172.21.15.78:6807/1816879397","state":["exists","up"]},{"osd":1,"uuid":"b8d64714-dd65-400f-b3f4-673ab14e1365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6801","nonce":1647915612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6803","nonce":1647915612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6807","nonce":1647915612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6805","nonce":1647915612}]},"public_addr":"172.21.15.106:6801/1647915612","cluster_addr":"172.21.15.106:6803/1647915612","heartbeat_back_addr":"172.21.15.106:6807/1647915612","heartbeat_front_addr":"172.21.15.106:6805/1647915612","state":["exists","up"]},{"osd":2,"uuid":"fcfc4dd1-aaf8-4131-a023-5e832212158d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6801","nonce":2010770435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6803","nonce":2010770435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6807","nonce":2010770435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6805","nonce":2010770435}]},"public_addr":"172.21.15.107:6801/2010770435","cluster_addr":"172.21.15.107:6803/2010770435","heartbeat_back_addr":"172.21.15.107:6807/2010770435","heartbeat_front_addr":"172.21.15.107:6805/2010770435","state":["exists","up"]},{"osd":3,"uuid":"47c718ab-a0f4-4e52-83c5-b89386b262d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6811","nonce":1912251346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6813","nonce":1912251346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6817","nonce":1912251346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6815","nonce":1912251346}]},"public_addr":"172.21.15.78:6811/1912251346","cluster_addr":"172.21.15.78:6813/1912251346","heartbeat_back_addr":"172.21.15.78:6817/1912251346","heartbeat_front_addr":"172.21.15.78:6815/1912251346","state":["exists","up"]},{"osd":4,"uuid":"7dad64dd-677e-4c1c-9638-e4f4f2195983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6809","nonce":374641205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6811","nonce":374641205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6815","nonce":374641205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6813","nonce":374641205}]},"public_addr":"172.21.15.106:6809/374641205","cluster_addr":"172.21.15.106:6811/374641205","heartbeat_back_addr":"172.21.15.106:6815/374641205","heartbeat_front_addr":"172.21.15.106:6813/374641205","state":["exists","up"]},{"osd":5,"uuid":"bbd524e5-328d-4309-8877-1149cbe110dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6809","nonce":3272806073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6811","nonce":3272806073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6815","nonce":3272806073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6813","nonce":3272806073}]},"public_addr":"172.21.15.107:6809/3272806073","cluster_addr":"172.21.15.107:6811/3272806073","heartbeat_back_addr":"172.21.15.107:6815/3272806073","heartbeat_front_addr":"172.21.15.107:6813/3272806073","state":["exists","up"]},{"osd":6,"uuid":"5696389a-918f-4ea8-b20d-4e2a0d4cb3bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6819","nonce":3091148508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6821","nonce":3091148508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6825","nonce":3091148508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6823","nonce":3091148508}]},"public_addr":"172.21.15.78:6819/3091148508","cluster_addr":"172.21.15.78:6821/3091148508","heartbeat_back_addr":"172.21.15.78:6825/3091148508","heartbeat_front_addr":"172.21.15.78:6823/3091148508","state":["exists","up"]},{"osd":7,"uuid":"f6b764e8-ed4e-4c9a-a85b-94af6dc6b924","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6817","nonce":4262103080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6819","nonce":4262103080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6823","nonce":4262103080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6821","nonce":4262103080}]},"public_addr":"172.21.15.106:6817/4262103080","cluster_addr":"172.21.15.106:6819/4262103080","heartbeat_back_addr":"172.21.15.106:6823/4262103080","heartbeat_front_addr":"172.21.15.106:6821/4262103080","state":["exists","up"]},{"osd":8,"uuid":"ba16039f-8093-4fa0-87ad-e28b7d142b23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6817","nonce":4032799356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6819","nonce":4032799356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6823","nonce":4032799356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6821","nonce":4032799356}]},"public_addr":"172.21.15.107:6817/4032799356","cluster_addr":"172.21.15.107:6819/4032799356","heartbeat_back_addr":"172.21.15.107:6823/4032799356","heartbeat_front_addr":"172.21.15.107:6821/4032799356","state":["exists","up"]},{"osd":9,"uuid":"651357f2-0669-42e6-aa91-1d5c02d00c5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6827","nonce":214023431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6829","nonce":214023431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6833","nonce":214023431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6831","nonce":214023431}]},"public_addr":"172.21.15.78:6827/214023431","cluster_addr":"172.21.15.78:6829/214023431","heartbeat_back_addr":"172.21.15.78:6833/214023431","heartbeat_front_addr":"172.21.15.78:6831/214023431","state":["exists","up"]},{"osd":10,"uuid":"2375119d-346e-426c-8a65-b64af71ad63a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6825","nonce":400029804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6827","nonce":400029804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6831","nonce":400029804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6829","nonce":400029804}]},"public_addr":"172.21.15.106:6825/400029804","cluster_addr":"172.21.15.106:6827/400029804","heartbeat_back_addr":"172.21.15.106:6831/400029804","heartbeat_front_addr":"172.21.15.106:6829/400029804","state":["exists","up"]},{"osd":11,"uuid":"b6b609d9-dd78-4c01-ae66-ba2a662ab22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6825","nonce":4191916230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6827","nonce":4191916230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6831","nonce":4191916230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6829","nonce":4191916230}]},"public_addr":"172.21.15.107:6825/4191916230","cluster_addr":"172.21.15.107:6827/4191916230","heartbeat_back_addr":"172.21.15.107:6831/4191916230","heartbeat_front_addr":"172.21.15.107:6829/4191916230","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:13.728602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:37.320809+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:01.269452+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:25.758281+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:50.298863+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:14.345219+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:38.980489+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:04.749214+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:29.992673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:56.269202+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:22.270316+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:48.074907+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/1982202541":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/1933054339":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6801/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6800/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/465691596":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6801/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/3962523609":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6800/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/1502752625":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/2236879401":"2024-03-25T00:11:40.199569+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:18:04.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.564+0000 7fcf61fb3640 1 -- 172.21.15.78:0/3609702840 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fcf440a2800 msgr2=0x7fcf440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:04.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.564+0000 7fcf61fb3640 1 --2- 172.21.15.78:0/3609702840 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fcf440a2800 0x7fcf440a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c02c7b0 tx=0x7fcf4c005fd0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.564+0000 7fcf61fb3640 1 -- 172.21.15.78:0/3609702840 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf5c107420 msgr2=0x7fcf5c1006b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:04.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.564+0000 7fcf61fb3640 1 --2- 172.21.15.78:0/3609702840 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf5c107420 0x7fcf5c1006b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fcf5400eb10 tx=0x7fcf5400c2d0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.564+0000 7fcf61fb3640 1 -- 172.21.15.78:0/3609702840 shutdown_connections 2024-03-24T00:18:04.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.564+0000 7fcf61fb3640 1 --2- 172.21.15.78:0/3609702840 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fcf440a2800 0x7fcf440a4cc0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.564+0000 7fcf61fb3640 1 --2- 172.21.15.78:0/3609702840 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf5c104af0 0x7fcf5c100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.564+0000 7fcf61fb3640 1 --2- 172.21.15.78:0/3609702840 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf5c107420 0x7fcf5c1006b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.564+0000 7fcf61fb3640 1 --2- 172.21.15.78:0/3609702840 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcf5c1021c0 0x7fcf5c101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.564+0000 7fcf61fb3640 1 -- 172.21.15.78:0/3609702840 >> 172.21.15.78:0/3609702840 conn(0x7fcf5c0fbb30 msgr2=0x7fcf5c102b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:04.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.564+0000 7fcf61fb3640 1 -- 172.21.15.78:0/3609702840 shutdown_connections 2024-03-24T00:18:04.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.564+0000 7fcf61fb3640 1 -- 172.21.15.78:0/3609702840 wait complete. 2024-03-24T00:18:04.574 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:04 smithi078 ceph-mon[32429]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi106=b;smithi106=e;smithi106=h;smithi106=k;smithi107=c;smithi107=f;smithi107=i;smithi107=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:18:04.575 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:04 smithi078 ceph-mon[32429]: Saving service mds.all spec with placement smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi106=b;smithi106=e;smithi106=h;smithi106=k;smithi107=c;smithi107=f;smithi107=i;smithi107=l;count:12 2024-03-24T00:18:04.575 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:04 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:04.575 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:04 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:18:04.575 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:04 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:04.575 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:04 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:04.575 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:04 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:04.575 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:04 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:04.576 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:14:04.534391+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:18:04.577 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:18:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:04 smithi106 ceph-mon[36505]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi106=b;smithi106=e;smithi106=h;smithi106=k;smithi107=c;smithi107=f;smithi107=i;smithi107=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:18:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:04 smithi106 ceph-mon[36505]: Saving service mds.all spec with placement smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi106=b;smithi106=e;smithi106=h;smithi106=k;smithi107=c;smithi107=f;smithi107=i;smithi107=l;count:12 2024-03-24T00:18:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:04 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:04 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:18:04.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:04 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:04.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:04 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:04.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:04 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:04.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:04 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:04.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.680+0000 7f2821a9a640 1 -- 172.21.15.78:0/620128640 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f281c101290 msgr2=0x7f281c101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:04.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.680+0000 7f2821a9a640 1 --2- 172.21.15.78:0/620128640 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f281c101290 0x7f281c101710 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2814009ca0 tx=0x7f281402c180 comp rx=0 tx=0).stop 2024-03-24T00:18:04.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.680+0000 7f2821a9a640 1 -- 172.21.15.78:0/620128640 shutdown_connections 2024-03-24T00:18:04.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.680+0000 7f2821a9a640 1 --2- 172.21.15.78:0/620128640 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f281c1008f0 0x7f281c100d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.680+0000 7f2821a9a640 1 --2- 172.21.15.78:0/620128640 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f281c101290 0x7f281c101710 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.680+0000 7f2821a9a640 1 --2- 172.21.15.78:0/620128640 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f281c0fff90 0x7f281c1003b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.680+0000 7f2821a9a640 1 -- 172.21.15.78:0/620128640 >> 172.21.15.78:0/620128640 conn(0x7f281c0fbb30 msgr2=0x7f281c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:04.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.681+0000 7f2821a9a640 1 -- 172.21.15.78:0/620128640 shutdown_connections 2024-03-24T00:18:04.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.681+0000 7f2821a9a640 1 -- 172.21.15.78:0/620128640 wait complete. 2024-03-24T00:18:04.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.681+0000 7f2821a9a640 1 Processor -- start 2024-03-24T00:18:04.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.681+0000 7f2821a9a640 1 -- start start 2024-03-24T00:18:04.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.681+0000 7f2821a9a640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f281c0fff90 0x7f281c199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.681+0000 7f2821a9a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f281c1008f0 0x7f281c19a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.681+0000 7f2821a9a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f281c101290 0x7f281c1a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.681+0000 7f2821a9a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f281c10f520 con 0x7f281c101290 2024-03-24T00:18:04.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.681+0000 7f2821a9a640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f281c10f3a0 con 0x7f281c0fff90 2024-03-24T00:18:04.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.681+0000 7f2821a9a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f281c10f6a0 con 0x7f281c1008f0 2024-03-24T00:18:04.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.681+0000 7f281bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f281c101290 0x7f281c1a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.681+0000 7f281bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f281c101290 0x7f281c1a1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51294/0 (socket says 172.21.15.78:51294) 2024-03-24T00:18:04.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.681+0000 7f281bfff640 1 -- 172.21.15.78:0/2831150371 learned_addr learned my addr 172.21.15.78:0/2831150371 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:04.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.682+0000 7f2812ffd640 1 --2- 172.21.15.78:0/2831150371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f281c1008f0 0x7f281c19a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.682+0000 7f281b7fe640 1 --2- 172.21.15.78:0/2831150371 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f281c0fff90 0x7f281c199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.682+0000 7f281bfff640 1 -- 172.21.15.78:0/2831150371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f281c1008f0 msgr2=0x7f281c19a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:04.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.682+0000 7f281bfff640 1 --2- 172.21.15.78:0/2831150371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f281c1008f0 0x7f281c19a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.682+0000 7f281bfff640 1 -- 172.21.15.78:0/2831150371 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f281c0fff90 msgr2=0x7f281c199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:04.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.682+0000 7f281bfff640 1 --2- 172.21.15.78:0/2831150371 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f281c0fff90 0x7f281c199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.682+0000 7f281bfff640 1 -- 172.21.15.78:0/2831150371 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f281c1a17e0 con 0x7f281c101290 2024-03-24T00:18:04.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.682+0000 7f281bfff640 1 --2- 172.21.15.78:0/2831150371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f281c101290 0x7f281c1a1170 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2814037040 tx=0x7f2814009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:04.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.682+0000 7f28197fa640 1 -- 172.21.15.78:0/2831150371 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2814039040 con 0x7f281c101290 2024-03-24T00:18:04.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.682+0000 7f2821a9a640 1 -- 172.21.15.78:0/2831150371 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f281c1a1a70 con 0x7f281c101290 2024-03-24T00:18:04.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.682+0000 7f28197fa640 1 -- 172.21.15.78:0/2831150371 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2814044070 con 0x7f281c101290 2024-03-24T00:18:04.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.682+0000 7f28197fa640 1 -- 172.21.15.78:0/2831150371 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2814004370 con 0x7f281c101290 2024-03-24T00:18:04.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.682+0000 7f2821a9a640 1 -- 172.21.15.78:0/2831150371 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f281c1a1fb0 con 0x7f281c101290 2024-03-24T00:18:04.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.683+0000 7f2821a9a640 1 -- 172.21.15.78:0/2831150371 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f281c10aed0 con 0x7f281c101290 2024-03-24T00:18:04.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.684+0000 7f28197fa640 1 -- 172.21.15.78:0/2831150371 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2814004510 con 0x7f281c101290 2024-03-24T00:18:04.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.685+0000 7f28197fa640 1 --2- 172.21.15.78:0/2831150371 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f28080a2800 0x7f28080a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.685+0000 7f281b7fe640 1 --2- 172.21.15.78:0/2831150371 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f28080a2800 0x7f28080a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.685+0000 7f28197fa640 1 -- 172.21.15.78:0/2831150371 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f28140f1180 con 0x7f281c101290 2024-03-24T00:18:04.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.685+0000 7f281b7fe640 1 --2- 172.21.15.78:0/2831150371 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f28080a2800 0x7f28080a4cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f280c009c50 tx=0x7f280c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:04.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.688+0000 7f28197fa640 1 -- 172.21.15.78:0/2831150371 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f28140be6c0 con 0x7f281c101290 2024-03-24T00:18:04.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.868+0000 7f2821a9a640 1 -- 172.21.15.78:0/2831150371 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f281c02d070 con 0x7f281c101290 2024-03-24T00:18:04.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.868+0000 7f28197fa640 1 -- 172.21.15.78:0/2831150371 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f28140bdfc0 con 0x7f281c101290 2024-03-24T00:18:04.869 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-24T00:18:04.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.870+0000 7f2821a9a640 1 -- 172.21.15.78:0/2831150371 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f28080a2800 msgr2=0x7f28080a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:04.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.870+0000 7f2821a9a640 1 --2- 172.21.15.78:0/2831150371 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f28080a2800 0x7f28080a4cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f280c009c50 tx=0x7f280c005f90 comp rx=0 tx=0).stop 2024-03-24T00:18:04.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.870+0000 7f2821a9a640 1 -- 172.21.15.78:0/2831150371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f281c101290 msgr2=0x7f281c1a1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:04.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.870+0000 7f2821a9a640 1 --2- 172.21.15.78:0/2831150371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f281c101290 0x7f281c1a1170 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2814037040 tx=0x7f2814009360 comp rx=0 tx=0).stop 2024-03-24T00:18:04.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.871+0000 7f2821a9a640 1 -- 172.21.15.78:0/2831150371 shutdown_connections 2024-03-24T00:18:04.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.871+0000 7f2821a9a640 1 --2- 172.21.15.78:0/2831150371 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f28080a2800 0x7f28080a4cc0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.871+0000 7f2821a9a640 1 --2- 172.21.15.78:0/2831150371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f281c1008f0 0x7f281c19a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.871+0000 7f2821a9a640 1 --2- 172.21.15.78:0/2831150371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f281c101290 0x7f281c1a1170 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.871+0000 7f2821a9a640 1 --2- 172.21.15.78:0/2831150371 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f281c0fff90 0x7f281c199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.871+0000 7f2821a9a640 1 -- 172.21.15.78:0/2831150371 >> 172.21.15.78:0/2831150371 conn(0x7f281c0fbb30 msgr2=0x7f281c0fd290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:04.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.871+0000 7f2821a9a640 1 -- 172.21.15.78:0/2831150371 shutdown_connections 2024-03-24T00:18:04.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.871+0000 7f2821a9a640 1 -- 172.21.15.78:0/2831150371 wait complete. 2024-03-24T00:18:04.883 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:18:04.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.984+0000 7f90b7b77640 1 -- 172.21.15.78:0/1130716266 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90b00ff740 msgr2=0x7f90b00ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:04.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.984+0000 7f90b7b77640 1 --2- 172.21.15.78:0/1130716266 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90b00ff740 0x7f90b00ffbc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f90ac009900 tx=0x7f90ac02bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:04.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.984+0000 7f90b7b77640 1 -- 172.21.15.78:0/1130716266 shutdown_connections 2024-03-24T00:18:04.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.984+0000 7f90b7b77640 1 --2- 172.21.15.78:0/1130716266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90b0100100 0x7f90b0109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.984+0000 7f90b7b77640 1 --2- 172.21.15.78:0/1130716266 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90b00ff740 0x7f90b00ffbc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.984+0000 7f90b7b77640 1 --2- 172.21.15.78:0/1130716266 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f90b0102cd0 0x7f90b01030f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.984+0000 7f90b7b77640 1 -- 172.21.15.78:0/1130716266 >> 172.21.15.78:0/1130716266 conn(0x7f90b00fb2c0 msgr2=0x7f90b00fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:04.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.984+0000 7f90b7b77640 1 -- 172.21.15.78:0/1130716266 shutdown_connections 2024-03-24T00:18:04.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.984+0000 7f90b7b77640 1 -- 172.21.15.78:0/1130716266 wait complete. 2024-03-24T00:18:04.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.984+0000 7f90b7b77640 1 Processor -- start 2024-03-24T00:18:04.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.984+0000 7f90b7b77640 1 -- start start 2024-03-24T00:18:04.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.985+0000 7f90b7b77640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f90b00ff740 0x7f90b01979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.985+0000 7f90b7b77640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90b0100100 0x7f90b0197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.985+0000 7f90b7b77640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90b0102cd0 0x7f90b019ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.985+0000 7f90b7b77640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b010d3d0 con 0x7f90b0100100 2024-03-24T00:18:04.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.985+0000 7f90b7b77640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b010d250 con 0x7f90b00ff740 2024-03-24T00:18:04.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.985+0000 7f90b7b77640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b010d550 con 0x7f90b0102cd0 2024-03-24T00:18:04.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.985+0000 7f90b5112640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90b0100100 0x7f90b0197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.985+0000 7f90b5112640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90b0100100 0x7f90b0197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51312/0 (socket says 172.21.15.78:51312) 2024-03-24T00:18:04.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.985+0000 7f90b5112640 1 -- 172.21.15.78:0/4184292354 learned_addr learned my addr 172.21.15.78:0/4184292354 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:04.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.985+0000 7f90b6114640 1 --2- 172.21.15.78:0/4184292354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90b0102cd0 0x7f90b019ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.985+0000 7f90b5913640 1 --2- 172.21.15.78:0/4184292354 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f90b00ff740 0x7f90b01979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.985+0000 7f90b5112640 1 -- 172.21.15.78:0/4184292354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90b0102cd0 msgr2=0x7f90b019ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:04.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.985+0000 7f90b5112640 1 --2- 172.21.15.78:0/4184292354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90b0102cd0 0x7f90b019ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.985+0000 7f90b5112640 1 -- 172.21.15.78:0/4184292354 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f90b00ff740 msgr2=0x7f90b01979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:04.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.985+0000 7f90b5112640 1 --2- 172.21.15.78:0/4184292354 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f90b00ff740 0x7f90b01979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.985+0000 7f90b5112640 1 -- 172.21.15.78:0/4184292354 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b019f5d0 con 0x7f90b0100100 2024-03-24T00:18:04.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.985+0000 7f90b5913640 1 --2- 172.21.15.78:0/4184292354 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f90b00ff740 0x7f90b01979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:18:04.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.985+0000 7f90b5112640 1 --2- 172.21.15.78:0/4184292354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90b0100100 0x7f90b0197ef0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f90ac009230 tx=0x7f90ac004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:04.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.985+0000 7f90a6ffd640 1 -- 172.21.15.78:0/4184292354 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac041070 con 0x7f90b0100100 2024-03-24T00:18:04.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.985+0000 7f90a6ffd640 1 -- 172.21.15.78:0/4184292354 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f90ac0054e0 con 0x7f90b0100100 2024-03-24T00:18:04.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.985+0000 7f90b7b77640 1 -- 172.21.15.78:0/4184292354 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90b019f860 con 0x7f90b0100100 2024-03-24T00:18:04.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.985+0000 7f90a6ffd640 1 -- 172.21.15.78:0/4184292354 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac0346a0 con 0x7f90b0100100 2024-03-24T00:18:04.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.985+0000 7f90b7b77640 1 -- 172.21.15.78:0/4184292354 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90b019fd50 con 0x7f90b0100100 2024-03-24T00:18:04.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.986+0000 7f90b7b77640 1 -- 172.21.15.78:0/4184292354 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90b0108b90 con 0x7f90b0100100 2024-03-24T00:18:04.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.987+0000 7f90a6ffd640 1 -- 172.21.15.78:0/4184292354 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f90ac044070 con 0x7f90b0100100 2024-03-24T00:18:04.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.988+0000 7f90a6ffd640 1 --2- 172.21.15.78:0/4184292354 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f909c0a2800 0x7f909c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.988+0000 7f90b5913640 1 --2- 172.21.15.78:0/4184292354 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f909c0a2800 0x7f909c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.988+0000 7f90a6ffd640 1 -- 172.21.15.78:0/4184292354 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f90ac0f0720 con 0x7f90b0100100 2024-03-24T00:18:04.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.988+0000 7f90b5913640 1 --2- 172.21.15.78:0/4184292354 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f909c0a2800 0x7f909c0a4cc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f90b0069a50 tx=0x7f90a0008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:04.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:04.991+0000 7f90a6ffd640 1 -- 172.21.15.78:0/4184292354 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f90ac0bdc60 con 0x7f90b0100100 2024-03-24T00:18:05.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:05.208+0000 7f90b7b77640 1 -- 172.21.15.78:0/4184292354 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90b002d230 con 0x7f90b0100100 2024-03-24T00:18:05.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:05.208+0000 7f90a6ffd640 1 -- 172.21.15.78:0/4184292354 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+535 (secure 0 0 0) 0x7f90ac0bd560 con 0x7f90b0100100 2024-03-24T00:18:05.209 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:18:05.209 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-24T00:18:05.210 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 1 2024-03-24T00:18:05.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:05.210+0000 7f90b7b77640 1 -- 172.21.15.78:0/4184292354 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f909c0a2800 msgr2=0x7f909c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:05.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:05.210+0000 7f90b7b77640 1 --2- 172.21.15.78:0/4184292354 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f909c0a2800 0x7f909c0a4cc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f90b0069a50 tx=0x7f90a0008040 comp rx=0 tx=0).stop 2024-03-24T00:18:05.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:05.210+0000 7f90b7b77640 1 -- 172.21.15.78:0/4184292354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90b0100100 msgr2=0x7f90b0197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:05.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:05.210+0000 7f90b7b77640 1 --2- 172.21.15.78:0/4184292354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90b0100100 0x7f90b0197ef0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f90ac009230 tx=0x7f90ac004040 comp rx=0 tx=0).stop 2024-03-24T00:18:05.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:05.210+0000 7f90b7b77640 1 -- 172.21.15.78:0/4184292354 shutdown_connections 2024-03-24T00:18:05.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:05.210+0000 7f90b7b77640 1 --2- 172.21.15.78:0/4184292354 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f909c0a2800 0x7f909c0a4cc0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:05.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:05.210+0000 7f90b7b77640 1 --2- 172.21.15.78:0/4184292354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90b0102cd0 0x7f90b019ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:05.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:05.210+0000 7f90b7b77640 1 --2- 172.21.15.78:0/4184292354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90b0100100 0x7f90b0197ef0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:05.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:05.210+0000 7f90b7b77640 1 --2- 172.21.15.78:0/4184292354 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f90b00ff740 0x7f90b01979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:05.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:05.210+0000 7f90b7b77640 1 -- 172.21.15.78:0/4184292354 >> 172.21.15.78:0/4184292354 conn(0x7f90b00fb2c0 msgr2=0x7f90b0105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:05.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:05.210+0000 7f90b7b77640 1 -- 172.21.15.78:0/4184292354 shutdown_connections 2024-03-24T00:18:05.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:05.210+0000 7f90b7b77640 1 -- 172.21.15.78:0/4184292354 wait complete. 2024-03-24T00:18:05.502 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:05 smithi107 ceph-mon[36485]: Deploying daemon mds.l on smithi107 2024-03-24T00:18:05.502 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:05 smithi107 ceph-mon[36485]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:05.502 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:05 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3609702840' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:18:05.502 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:05 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2831150371' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:18:05.502 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:05 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/4184292354' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:05.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:05 smithi078 ceph-mon[32429]: Deploying daemon mds.l on smithi107 2024-03-24T00:18:05.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:05 smithi078 ceph-mon[32429]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:05.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:05 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3609702840' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:18:05.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:05 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2831150371' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:18:05.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:05 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/4184292354' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:05.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:05 smithi106 ceph-mon[36505]: Deploying daemon mds.l on smithi107 2024-03-24T00:18:05.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:05 smithi106 ceph-mon[36505]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:05.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:05 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3609702840' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:18:05.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:05 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2831150371' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:18:05.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:05 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/4184292354' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:06.093 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:18:05 smithi107 systemd[1]: Starting Ceph mds.l for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:18:06.424 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:18:06 smithi107 podman[66556]: 2024-03-24 00:18:06.077121326 +0000 UTC m=+0.017503697 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:18:06.424 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:18:06 smithi107 podman[66556]: 2024-03-24 00:18:06.188473386 +0000 UTC m=+0.128855745 container create b4c419602016bf59b2129f162e4842086c872fbbb83a4b36ea7d5bc990682587 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:18:06.424 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:18:06 smithi107 podman[66556]: 2024-03-24 00:18:06.424968125 +0000 UTC m=+0.365350486 container init b4c419602016bf59b2129f162e4842086c872fbbb83a4b36ea7d5bc990682587 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-24T00:18:06.843 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:18:06 smithi107 podman[66556]: 2024-03-24 00:18:06.427650798 +0000 UTC m=+0.368033159 container start b4c419602016bf59b2129f162e4842086c872fbbb83a4b36ea7d5bc990682587 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-24T00:18:06.843 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:18:06 smithi107 ceph-mds[66576]: -- 172.21.15.107:0/1026234856 <== mon.1 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c63e887a40 con 0x55c63e85b000 2024-03-24T00:18:06.843 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:18:06 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: starting mds.l at 2024-03-24T00:18:06.843 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:18:06 smithi107 bash[66556]: b4c419602016bf59b2129f162e4842086c872fbbb83a4b36ea7d5bc990682587 2024-03-24T00:18:06.843 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:18:06 smithi107 systemd[1]: Started Ceph mds.l for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:18:07.223 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:18:07.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.335+0000 7f89fb2dc640 1 -- 172.21.15.78:0/4177751083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89f4104ad0 msgr2=0x7f89f4106ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:07.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.335+0000 7f89fb2dc640 1 --2- 172.21.15.78:0/4177751083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89f4104ad0 0x7f89f4106ec0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f89e4009900 tx=0x7f89e402bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:07.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.335+0000 7f89fb2dc640 1 -- 172.21.15.78:0/4177751083 shutdown_connections 2024-03-24T00:18:07.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.335+0000 7f89fb2dc640 1 --2- 172.21.15.78:0/4177751083 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89f4107400 0x7f89f41097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:07.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.335+0000 7f89fb2dc640 1 --2- 172.21.15.78:0/4177751083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89f4104ad0 0x7f89f4106ec0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:07.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.335+0000 7f89fb2dc640 1 --2- 172.21.15.78:0/4177751083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89f41021a0 0x7f89f4104590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:07.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.335+0000 7f89fb2dc640 1 -- 172.21.15.78:0/4177751083 >> 172.21.15.78:0/4177751083 conn(0x7f89f40fbb30 msgr2=0x7f89f40fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:07.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.335+0000 7f89fb2dc640 1 -- 172.21.15.78:0/4177751083 shutdown_connections 2024-03-24T00:18:07.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.335+0000 7f89fb2dc640 1 -- 172.21.15.78:0/4177751083 wait complete. 2024-03-24T00:18:07.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.335+0000 7f89fb2dc640 1 Processor -- start 2024-03-24T00:18:07.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.335+0000 7f89fb2dc640 1 -- start start 2024-03-24T00:18:07.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.335+0000 7f89fb2dc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89f41021a0 0x7f89f4197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:07.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.335+0000 7f89fb2dc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89f4104ad0 0x7f89f4197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:07.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.335+0000 7f89fb2dc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89f4107400 0x7f89f419efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:07.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.335+0000 7f89fb2dc640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f410d310 con 0x7f89f4107400 2024-03-24T00:18:07.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.335+0000 7f89fb2dc640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f410d190 con 0x7f89f4104ad0 2024-03-24T00:18:07.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.335+0000 7f89fb2dc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f410d490 con 0x7f89f41021a0 2024-03-24T00:18:07.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.336+0000 7f89f9879640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89f4107400 0x7f89f419efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:07.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.336+0000 7f89f9879640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89f4107400 0x7f89f419efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51322/0 (socket says 172.21.15.78:51322) 2024-03-24T00:18:07.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.336+0000 7f89f9879640 1 -- 172.21.15.78:0/1689606138 learned_addr learned my addr 172.21.15.78:0/1689606138 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:07.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.336+0000 7f89f8877640 1 --2- 172.21.15.78:0/1689606138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89f4104ad0 0x7f89f4197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:07.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.336+0000 7f89f9078640 1 --2- 172.21.15.78:0/1689606138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89f41021a0 0x7f89f4197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:07.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.336+0000 7f89f9879640 1 -- 172.21.15.78:0/1689606138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89f41021a0 msgr2=0x7f89f4197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:07.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.336+0000 7f89f9879640 1 --2- 172.21.15.78:0/1689606138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89f41021a0 0x7f89f4197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:07.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.336+0000 7f89f9879640 1 -- 172.21.15.78:0/1689606138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89f4104ad0 msgr2=0x7f89f4197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:07.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.336+0000 7f89f9879640 1 --2- 172.21.15.78:0/1689606138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89f4104ad0 0x7f89f4197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:07.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.336+0000 7f89f9879640 1 -- 172.21.15.78:0/1689606138 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f419f650 con 0x7f89f4107400 2024-03-24T00:18:07.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.336+0000 7f89f9879640 1 --2- 172.21.15.78:0/1689606138 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89f4107400 0x7f89f419efe0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f89ec00cad0 tx=0x7f89ec00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:07.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.336+0000 7f89ea7fc640 1 -- 172.21.15.78:0/1689606138 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ec00ae20 con 0x7f89f4107400 2024-03-24T00:18:07.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.336+0000 7f89ea7fc640 1 -- 172.21.15.78:0/1689606138 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f89ec005020 con 0x7f89f4107400 2024-03-24T00:18:07.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.336+0000 7f89fb2dc640 1 -- 172.21.15.78:0/1689606138 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89f419f940 con 0x7f89f4107400 2024-03-24T00:18:07.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.336+0000 7f89ea7fc640 1 -- 172.21.15.78:0/1689606138 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ec00f6f0 con 0x7f89f4107400 2024-03-24T00:18:07.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.336+0000 7f89fb2dc640 1 -- 172.21.15.78:0/1689606138 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89f419fe10 con 0x7f89f4107400 2024-03-24T00:18:07.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.338+0000 7f89fb2dc640 1 -- 172.21.15.78:0/1689606138 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89f40fd620 con 0x7f89f4107400 2024-03-24T00:18:07.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.338+0000 7f89ea7fc640 1 -- 172.21.15.78:0/1689606138 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f89ec0051d0 con 0x7f89f4107400 2024-03-24T00:18:07.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.339+0000 7f89ea7fc640 1 --2- 172.21.15.78:0/1689606138 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f89e00a2800 0x7f89e00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:07.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.339+0000 7f89f9078640 1 --2- 172.21.15.78:0/1689606138 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f89e00a2800 0x7f89e00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:07.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.339+0000 7f89ea7fc640 1 -- 172.21.15.78:0/1689606138 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f89ec0ce5c0 con 0x7f89f4107400 2024-03-24T00:18:07.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.339+0000 7f89f9078640 1 --2- 172.21.15.78:0/1689606138 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f89e00a2800 0x7f89e00a4cc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f89f0005d00 tx=0x7f89f0005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:07.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.343+0000 7f89ea7fc640 1 -- 172.21.15.78:0/1689606138 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f89ec014030 con 0x7f89f4107400 2024-03-24T00:18:07.542 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:07 smithi106 ceph-mon[36505]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:07.542 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:07 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6832/3086318443,v1:172.21.15.107:6833/3086318443] up:boot 2024-03-24T00:18:07.542 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:07 smithi106 ceph-mon[36505]: fsmap 1 up:standby 2024-03-24T00:18:07.542 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:07 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:18:07.542 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:07 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:07.542 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:07 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:07.542 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:07 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:07.543 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:07 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:07.543 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:07 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:07.543 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:07 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:07.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.558+0000 7f89fb2dc640 1 -- 172.21.15.78:0/1689606138 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89f40622a0 con 0x7f89f4107400 2024-03-24T00:18:07.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.559+0000 7f89ea7fc640 1 -- 172.21.15.78:0/1689606138 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1328 (secure 0 0 0) 0x7f89ec09bb00 con 0x7f89f4107400 2024-03-24T00:18:07.560 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:18:07.560 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/3086318443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3086318443},{"type":"v1","addr":"172.21.15.107:6833","nonce":3086318443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-24T00:18:07.560 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 2 2024-03-24T00:18:07.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.561+0000 7f89fb2dc640 1 -- 172.21.15.78:0/1689606138 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f89e00a2800 msgr2=0x7f89e00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:07.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.561+0000 7f89fb2dc640 1 --2- 172.21.15.78:0/1689606138 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f89e00a2800 0x7f89e00a4cc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f89f0005d00 tx=0x7f89f0005c50 comp rx=0 tx=0).stop 2024-03-24T00:18:07.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.561+0000 7f89fb2dc640 1 -- 172.21.15.78:0/1689606138 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89f4107400 msgr2=0x7f89f419efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:07.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.561+0000 7f89fb2dc640 1 --2- 172.21.15.78:0/1689606138 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89f4107400 0x7f89f419efe0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f89ec00cad0 tx=0x7f89ec00cf90 comp rx=0 tx=0).stop 2024-03-24T00:18:07.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.561+0000 7f89fb2dc640 1 -- 172.21.15.78:0/1689606138 shutdown_connections 2024-03-24T00:18:07.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.561+0000 7f89fb2dc640 1 --2- 172.21.15.78:0/1689606138 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f89e00a2800 0x7f89e00a4cc0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:07.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.561+0000 7f89fb2dc640 1 --2- 172.21.15.78:0/1689606138 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89f4107400 0x7f89f419efe0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:07.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.561+0000 7f89fb2dc640 1 --2- 172.21.15.78:0/1689606138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89f4104ad0 0x7f89f4197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:07.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.561+0000 7f89fb2dc640 1 --2- 172.21.15.78:0/1689606138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89f41021a0 0x7f89f4197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:07.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.561+0000 7f89fb2dc640 1 -- 172.21.15.78:0/1689606138 >> 172.21.15.78:0/1689606138 conn(0x7f89f40fbb30 msgr2=0x7f89f4102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:07.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.561+0000 7f89fb2dc640 1 -- 172.21.15.78:0/1689606138 shutdown_connections 2024-03-24T00:18:07.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:07.561+0000 7f89fb2dc640 1 -- 172.21.15.78:0/1689606138 wait complete. 2024-03-24T00:18:07.572 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:07 smithi078 ceph-mon[32429]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:07.572 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:07 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6832/3086318443,v1:172.21.15.107:6833/3086318443] up:boot 2024-03-24T00:18:07.572 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:07 smithi078 ceph-mon[32429]: fsmap 1 up:standby 2024-03-24T00:18:07.572 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:07 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:18:07.572 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:07 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:07.572 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:07 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:07.572 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:07 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:07.572 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:07 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:07.572 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:07 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:07.572 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:07 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:07.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:07 smithi107 ceph-mon[36485]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:07.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:07 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6832/3086318443,v1:172.21.15.107:6833/3086318443] up:boot 2024-03-24T00:18:07.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:07 smithi107 ceph-mon[36485]: fsmap 1 up:standby 2024-03-24T00:18:07.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:07 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:18:07.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:07 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:07.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:07 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:07.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:07 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:07.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:07 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:07.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:07 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:07.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:07 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:08.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:08 smithi106 ceph-mon[36505]: Deploying daemon mds.k on smithi106 2024-03-24T00:18:08.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:08 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1689606138' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:08 smithi078 ceph-mon[32429]: Deploying daemon mds.k on smithi106 2024-03-24T00:18:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:08 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1689606138' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:08 smithi107 ceph-mon[36485]: Deploying daemon mds.k on smithi106 2024-03-24T00:18:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:08 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1689606138' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:09.108 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:18:08 smithi106 systemd[1]: Starting Ceph mds.k for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:18:09.377 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:09 smithi106 ceph-mon[36505]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:09.377 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:18:09 smithi106 podman[66027]: 2024-03-24 00:18:09.010743439 +0000 UTC m=+0.017607413 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:18:09.378 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:18:09 smithi106 podman[66027]: 2024-03-24 00:18:09.133657645 +0000 UTC m=+0.140521613 container create 98bb5f2ad84ddb896d35c650fbdfcc6b9b3f5bb989d008f61c9c311c18922d17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-24T00:18:09.574 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:18:09.604 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:09 smithi078 ceph-mon[32429]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:09.652 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:18:09 smithi106 podman[66027]: 2024-03-24 00:18:09.378358254 +0000 UTC m=+0.385222230 container init 98bb5f2ad84ddb896d35c650fbdfcc6b9b3f5bb989d008f61c9c311c18922d17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0) 2024-03-24T00:18:09.652 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:18:09 smithi106 podman[66027]: 2024-03-24 00:18:09.381149796 +0000 UTC m=+0.388013758 container start 98bb5f2ad84ddb896d35c650fbdfcc6b9b3f5bb989d008f61c9c311c18922d17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-24T00:18:09.652 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:18:09 smithi106 ceph-mds[66046]: -- 172.21.15.106:0/3995184414 <== mon.2 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55dc1d8f9a40 con 0x55dc1d8ccc00 2024-03-24T00:18:09.652 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:18:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: starting mds.k at 2024-03-24T00:18:09.652 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:18:09 smithi106 bash[66027]: 98bb5f2ad84ddb896d35c650fbdfcc6b9b3f5bb989d008f61c9c311c18922d17 2024-03-24T00:18:09.653 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:18:09 smithi106 systemd[1]: Started Ceph mds.k for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:18:09.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.691+0000 7fa7935f4640 1 -- 172.21.15.78:0/2140276392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa78c1005c0 msgr2=0x7fa78c1009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:09.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.691+0000 7fa7935f4640 1 --2- 172.21.15.78:0/2140276392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa78c1005c0 0x7fa78c1009e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa788009900 tx=0x7fa78802bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:09.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.691+0000 7fa7935f4640 1 -- 172.21.15.78:0/2140276392 shutdown_connections 2024-03-24T00:18:09.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.691+0000 7fa7935f4640 1 --2- 172.21.15.78:0/2140276392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa78c1017b0 0x7fa78c101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:09.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.691+0000 7fa7935f4640 1 --2- 172.21.15.78:0/2140276392 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa78c102170 0x7fa78c108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:09.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.691+0000 7fa7935f4640 1 --2- 172.21.15.78:0/2140276392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa78c1005c0 0x7fa78c1009e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:09.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.691+0000 7fa7935f4640 1 -- 172.21.15.78:0/2140276392 >> 172.21.15.78:0/2140276392 conn(0x7fa78c0fbb30 msgr2=0x7fa78c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:09.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.691+0000 7fa7935f4640 1 -- 172.21.15.78:0/2140276392 shutdown_connections 2024-03-24T00:18:09.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.691+0000 7fa7935f4640 1 -- 172.21.15.78:0/2140276392 wait complete. 2024-03-24T00:18:09.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.692+0000 7fa7935f4640 1 Processor -- start 2024-03-24T00:18:09.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.692+0000 7fa7935f4640 1 -- start start 2024-03-24T00:18:09.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.692+0000 7fa7935f4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa78c1005c0 0x7fa78c069370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:09.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.692+0000 7fa7935f4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa78c1017b0 0x7fa78c19c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:09.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.692+0000 7fa7935f4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa78c102170 0x7fa78c19ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:09.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.692+0000 7fa7935f4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78c10c230 con 0x7fa78c102170 2024-03-24T00:18:09.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.692+0000 7fa7935f4640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78c10c0b0 con 0x7fa78c1005c0 2024-03-24T00:18:09.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.692+0000 7fa7935f4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78c10c3b0 con 0x7fa78c1017b0 2024-03-24T00:18:09.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.692+0000 7fa791b91640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa78c102170 0x7fa78c19ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:09.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.692+0000 7fa791b91640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa78c102170 0x7fa78c19ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51346/0 (socket says 172.21.15.78:51346) 2024-03-24T00:18:09.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.692+0000 7fa791b91640 1 -- 172.21.15.78:0/2641300324 learned_addr learned my addr 172.21.15.78:0/2641300324 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:09.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.692+0000 7fa790b8f640 1 --2- 172.21.15.78:0/2641300324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa78c1017b0 0x7fa78c19c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:09.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.692+0000 7fa791390640 1 --2- 172.21.15.78:0/2641300324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa78c1005c0 0x7fa78c069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:09.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.692+0000 7fa791b91640 1 -- 172.21.15.78:0/2641300324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa78c1017b0 msgr2=0x7fa78c19c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:09.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.692+0000 7fa791b91640 1 --2- 172.21.15.78:0/2641300324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa78c1017b0 0x7fa78c19c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:09.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.692+0000 7fa791b91640 1 -- 172.21.15.78:0/2641300324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa78c1005c0 msgr2=0x7fa78c069370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:09.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.692+0000 7fa791b91640 1 --2- 172.21.15.78:0/2641300324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa78c1005c0 0x7fa78c069370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:09.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.692+0000 7fa791b91640 1 -- 172.21.15.78:0/2641300324 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa78c19f680 con 0x7fa78c102170 2024-03-24T00:18:09.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.692+0000 7fa791390640 1 --2- 172.21.15.78:0/2641300324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa78c1005c0 0x7fa78c069370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:18:09.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.693+0000 7fa791b91640 1 --2- 172.21.15.78:0/2641300324 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa78c102170 0x7fa78c19ed70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa78400eb10 tx=0x7fa78400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:09.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.693+0000 7fa7827fc640 1 -- 172.21.15.78:0/2641300324 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa784004b20 con 0x7fa78c102170 2024-03-24T00:18:09.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.693+0000 7fa7935f4640 1 -- 172.21.15.78:0/2641300324 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa78c19f910 con 0x7fa78c102170 2024-03-24T00:18:09.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.693+0000 7fa7827fc640 1 -- 172.21.15.78:0/2641300324 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa784004cc0 con 0x7fa78c102170 2024-03-24T00:18:09.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.693+0000 7fa7827fc640 1 -- 172.21.15.78:0/2641300324 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa784010760 con 0x7fa78c102170 2024-03-24T00:18:09.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.693+0000 7fa7935f4640 1 -- 172.21.15.78:0/2641300324 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa78c19fe10 con 0x7fa78c102170 2024-03-24T00:18:09.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.694+0000 7fa7935f4640 1 -- 172.21.15.78:0/2641300324 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa78c107960 con 0x7fa78c102170 2024-03-24T00:18:09.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.695+0000 7fa7827fc640 1 -- 172.21.15.78:0/2641300324 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa784005000 con 0x7fa78c102170 2024-03-24T00:18:09.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.695+0000 7fa7827fc640 1 --2- 172.21.15.78:0/2641300324 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa7780a2800 0x7fa7780a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:09.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.696+0000 7fa791390640 1 --2- 172.21.15.78:0/2641300324 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa7780a2800 0x7fa7780a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:09.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.696+0000 7fa7827fc640 1 -- 172.21.15.78:0/2641300324 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa7840cee10 con 0x7fa78c102170 2024-03-24T00:18:09.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.696+0000 7fa791390640 1 --2- 172.21.15.78:0/2641300324 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa7780a2800 0x7fa7780a4cc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa78802c7b0 tx=0x7fa788005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:09.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.699+0000 7fa7827fc640 1 -- 172.21.15.78:0/2641300324 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa784053740 con 0x7fa78c102170 2024-03-24T00:18:09.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:09 smithi107 ceph-mon[36485]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:09.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.918+0000 7fa7935f4640 1 -- 172.21.15.78:0/2641300324 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa78c02d070 con 0x7fa78c102170 2024-03-24T00:18:09.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.918+0000 7fa7827fc640 1 -- 172.21.15.78:0/2641300324 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1328 (secure 0 0 0) 0x7fa78409c350 con 0x7fa78c102170 2024-03-24T00:18:09.919 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:18:09.920 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/3086318443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3086318443},{"type":"v1","addr":"172.21.15.107:6833","nonce":3086318443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-24T00:18:09.920 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 2 2024-03-24T00:18:09.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.920+0000 7fa7935f4640 1 -- 172.21.15.78:0/2641300324 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa7780a2800 msgr2=0x7fa7780a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:09.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.920+0000 7fa7935f4640 1 --2- 172.21.15.78:0/2641300324 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa7780a2800 0x7fa7780a4cc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa78802c7b0 tx=0x7fa788005fd0 comp rx=0 tx=0).stop 2024-03-24T00:18:09.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.920+0000 7fa7935f4640 1 -- 172.21.15.78:0/2641300324 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa78c102170 msgr2=0x7fa78c19ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:09.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.920+0000 7fa7935f4640 1 --2- 172.21.15.78:0/2641300324 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa78c102170 0x7fa78c19ed70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa78400eb10 tx=0x7fa78400c2d0 comp rx=0 tx=0).stop 2024-03-24T00:18:09.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.921+0000 7fa7935f4640 1 -- 172.21.15.78:0/2641300324 shutdown_connections 2024-03-24T00:18:09.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.921+0000 7fa7935f4640 1 --2- 172.21.15.78:0/2641300324 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa7780a2800 0x7fa7780a4cc0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:09.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.921+0000 7fa7935f4640 1 --2- 172.21.15.78:0/2641300324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa78c1017b0 0x7fa78c19c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:09.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.921+0000 7fa7935f4640 1 --2- 172.21.15.78:0/2641300324 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa78c102170 0x7fa78c19ed70 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:09.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.921+0000 7fa7935f4640 1 --2- 172.21.15.78:0/2641300324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa78c1005c0 0x7fa78c069370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:09.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.921+0000 7fa7935f4640 1 -- 172.21.15.78:0/2641300324 >> 172.21.15.78:0/2641300324 conn(0x7fa78c0fbb30 msgr2=0x7fa78c104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:09.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.921+0000 7fa7935f4640 1 -- 172.21.15.78:0/2641300324 shutdown_connections 2024-03-24T00:18:09.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:09.921+0000 7fa7935f4640 1 -- 172.21.15.78:0/2641300324 wait complete. 2024-03-24T00:18:10.811 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:10 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:10.811 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:10 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:10.811 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:10 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:10.811 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:10 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:10.811 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:10 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:10.811 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:10 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:10.811 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:10 smithi106 ceph-mon[36505]: Deploying daemon mds.j on smithi078 2024-03-24T00:18:10.811 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:10 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2641300324' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:10.812 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:10 smithi106 ceph-mon[36505]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:10.812 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:10 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6832/2367018868,v1:172.21.15.106:6833/2367018868] up:boot 2024-03-24T00:18:10.812 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:10 smithi106 ceph-mon[36505]: fsmap 2 up:standby 2024-03-24T00:18:10.812 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:10 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T00:18:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:10 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:10 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:10 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:10 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:10 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:10 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:10 smithi078 ceph-mon[32429]: Deploying daemon mds.j on smithi078 2024-03-24T00:18:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:10 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2641300324' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:10 smithi078 ceph-mon[32429]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:10.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:10 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6832/2367018868,v1:172.21.15.106:6833/2367018868] up:boot 2024-03-24T00:18:10.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:10 smithi078 ceph-mon[32429]: fsmap 2 up:standby 2024-03-24T00:18:10.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:10 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T00:18:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:10 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:10 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:10 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:10 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:10 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:10 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:10 smithi107 ceph-mon[36485]: Deploying daemon mds.j on smithi078 2024-03-24T00:18:10.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:10 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2641300324' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:10.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:10 smithi107 ceph-mon[36485]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:10.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:10 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6832/2367018868,v1:172.21.15.106:6833/2367018868] up:boot 2024-03-24T00:18:10.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:10 smithi107 ceph-mon[36485]: fsmap 2 up:standby 2024-03-24T00:18:10.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:10 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T00:18:11.934 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:18:11.960 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:18:11 smithi078 systemd[1]: Starting Ceph mds.j for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:18:12.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.040+0000 7f6678bd3640 1 -- 172.21.15.78:0/2786862634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66741016f0 msgr2=0x7f6674101b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:12.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.040+0000 7f6678bd3640 1 --2- 172.21.15.78:0/2786862634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66741016f0 0x7f6674101b10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6664009900 tx=0x7f666402bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:12.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.040+0000 7f6678bd3640 1 -- 172.21.15.78:0/2786862634 shutdown_connections 2024-03-24T00:18:12.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.040+0000 7f6678bd3640 1 --2- 172.21.15.78:0/2786862634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66741028e0 0x7f6674102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:12.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.040+0000 7f6678bd3640 1 --2- 172.21.15.78:0/2786862634 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66741032a0 0x7f6674109b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:12.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.040+0000 7f6678bd3640 1 --2- 172.21.15.78:0/2786862634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66741016f0 0x7f6674101b10 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:12.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.040+0000 7f6678bd3640 1 -- 172.21.15.78:0/2786862634 >> 172.21.15.78:0/2786862634 conn(0x7f66740fcc40 msgr2=0x7f66740ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:12.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.040+0000 7f6678bd3640 1 -- 172.21.15.78:0/2786862634 shutdown_connections 2024-03-24T00:18:12.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.040+0000 7f6678bd3640 1 -- 172.21.15.78:0/2786862634 wait complete. 2024-03-24T00:18:12.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.040+0000 7f6678bd3640 1 Processor -- start 2024-03-24T00:18:12.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.040+0000 7f6678bd3640 1 -- start start 2024-03-24T00:18:12.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.041+0000 7f6678bd3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66741016f0 0x7f6674075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:12.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.041+0000 7f6678bd3640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66741028e0 0x7f6674073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:12.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.041+0000 7f6678bd3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66741032a0 0x7f6674073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:12.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.041+0000 7f6678bd3640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667410d340 con 0x7f66741032a0 2024-03-24T00:18:12.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.041+0000 7f6678bd3640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667410d1c0 con 0x7f66741028e0 2024-03-24T00:18:12.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.041+0000 7f6678bd3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667410d4c0 con 0x7f66741016f0 2024-03-24T00:18:12.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.041+0000 7f6672d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66741032a0 0x7f6674073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:12.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.041+0000 7f6672d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66741032a0 0x7f6674073fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51356/0 (socket says 172.21.15.78:51356) 2024-03-24T00:18:12.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.041+0000 7f6672d9d640 1 -- 172.21.15.78:0/4221219098 learned_addr learned my addr 172.21.15.78:0/4221219098 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:12.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.041+0000 7f666bfff640 1 --2- 172.21.15.78:0/4221219098 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66741028e0 0x7f6674073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:12.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.041+0000 7f6672d9d640 1 -- 172.21.15.78:0/4221219098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66741016f0 msgr2=0x7f6674075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:12.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.041+0000 7f6672d9d640 1 --2- 172.21.15.78:0/4221219098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66741016f0 0x7f6674075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:12.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.041+0000 7f6672d9d640 1 -- 172.21.15.78:0/4221219098 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66741028e0 msgr2=0x7f6674073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:12.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.041+0000 7f6672d9d640 1 --2- 172.21.15.78:0/4221219098 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66741028e0 0x7f6674073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:12.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.041+0000 7f6672d9d640 1 -- 172.21.15.78:0/4221219098 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6674074510 con 0x7f66741032a0 2024-03-24T00:18:12.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.041+0000 7f6672d9d640 1 --2- 172.21.15.78:0/4221219098 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66741032a0 0x7f6674073fd0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f666c00cad0 tx=0x7f666c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:12.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.041+0000 7f666b7fe640 1 -- 172.21.15.78:0/4221219098 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666c00ae20 con 0x7f66741032a0 2024-03-24T00:18:12.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.041+0000 7f666b7fe640 1 -- 172.21.15.78:0/4221219098 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f666c005020 con 0x7f66741032a0 2024-03-24T00:18:12.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.041+0000 7f6678bd3640 1 -- 172.21.15.78:0/4221219098 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6674074800 con 0x7f66741032a0 2024-03-24T00:18:12.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.041+0000 7f666b7fe640 1 -- 172.21.15.78:0/4221219098 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666c00f6f0 con 0x7f66741032a0 2024-03-24T00:18:12.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.041+0000 7f6678bd3640 1 -- 172.21.15.78:0/4221219098 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66741a64d0 con 0x7f66741032a0 2024-03-24T00:18:12.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.042+0000 7f6678bd3640 1 -- 172.21.15.78:0/4221219098 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6674108a60 con 0x7f66741032a0 2024-03-24T00:18:12.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.043+0000 7f666b7fe640 1 -- 172.21.15.78:0/4221219098 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f666c0051d0 con 0x7f66741032a0 2024-03-24T00:18:12.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.044+0000 7f666b7fe640 1 --2- 172.21.15.78:0/4221219098 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f66600a2800 0x7f66600a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:12.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.044+0000 7f667259c640 1 --2- 172.21.15.78:0/4221219098 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f66600a2800 0x7f66600a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:12.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.044+0000 7f666b7fe640 1 -- 172.21.15.78:0/4221219098 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f666c0ce670 con 0x7f66741032a0 2024-03-24T00:18:12.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.044+0000 7f667259c640 1 --2- 172.21.15.78:0/4221219098 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f66600a2800 0x7f66600a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6664036040 tx=0x7f6664000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:12.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.048+0000 7f666b7fe640 1 -- 172.21.15.78:0/4221219098 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f666c014030 con 0x7f66741032a0 2024-03-24T00:18:12.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.267+0000 7f6678bd3640 1 -- 172.21.15.78:0/4221219098 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f667402d430 con 0x7f66741032a0 2024-03-24T00:18:12.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.267+0000 7f666b7fe640 1 -- 172.21.15.78:0/4221219098 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2122 (secure 0 0 0) 0x7f666c09bbb0 con 0x7f66741032a0 2024-03-24T00:18:12.268 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:18:12.268 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/3086318443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3086318443},{"type":"v1","addr":"172.21.15.107:6833","nonce":3086318443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/2367018868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2367018868},{"type":"v1","addr":"172.21.15.106:6833","nonce":2367018868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-24T00:18:12.269 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 3 2024-03-24T00:18:12.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.269+0000 7f6678bd3640 1 -- 172.21.15.78:0/4221219098 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f66600a2800 msgr2=0x7f66600a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:12.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.269+0000 7f6678bd3640 1 --2- 172.21.15.78:0/4221219098 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f66600a2800 0x7f66600a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6664036040 tx=0x7f6664000bc0 comp rx=0 tx=0).stop 2024-03-24T00:18:12.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.269+0000 7f6678bd3640 1 -- 172.21.15.78:0/4221219098 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66741032a0 msgr2=0x7f6674073fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:12.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.269+0000 7f6678bd3640 1 --2- 172.21.15.78:0/4221219098 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66741032a0 0x7f6674073fd0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f666c00cad0 tx=0x7f666c00cf90 comp rx=0 tx=0).stop 2024-03-24T00:18:12.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.269+0000 7f6678bd3640 1 -- 172.21.15.78:0/4221219098 shutdown_connections 2024-03-24T00:18:12.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.269+0000 7f6678bd3640 1 --2- 172.21.15.78:0/4221219098 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f66600a2800 0x7f66600a4cc0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:12.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.269+0000 7f6678bd3640 1 --2- 172.21.15.78:0/4221219098 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66741032a0 0x7f6674073fd0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:12.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.269+0000 7f6678bd3640 1 --2- 172.21.15.78:0/4221219098 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66741028e0 0x7f6674073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:12.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.269+0000 7f6678bd3640 1 --2- 172.21.15.78:0/4221219098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66741016f0 0x7f6674075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:12.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.269+0000 7f6678bd3640 1 -- 172.21.15.78:0/4221219098 >> 172.21.15.78:0/4221219098 conn(0x7f66740fcc40 msgr2=0x7f6674105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:12.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.270+0000 7f6678bd3640 1 -- 172.21.15.78:0/4221219098 shutdown_connections 2024-03-24T00:18:12.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:12.270+0000 7f6678bd3640 1 -- 172.21.15.78:0/4221219098 wait complete. 2024-03-24T00:18:12.280 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:18:12 smithi078 podman[70366]: 2024-03-24 00:18:11.935197408 +0000 UTC m=+0.017899356 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:18:12.280 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:18:12 smithi078 podman[70366]: 2024-03-24 00:18:12.054032881 +0000 UTC m=+0.136734814 container create 2bae411f70111c67297db78af224b5d61cd9eaca3e674ba15080643f82ffc3ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-24T00:18:12.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:12 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/4221219098' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:12.590 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:18:12 smithi078 podman[70366]: 2024-03-24 00:18:12.302662642 +0000 UTC m=+0.385364586 container init 2bae411f70111c67297db78af224b5d61cd9eaca3e674ba15080643f82ffc3ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:18:12.590 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:18:12 smithi078 podman[70366]: 2024-03-24 00:18:12.305421192 +0000 UTC m=+0.388123124 container start 2bae411f70111c67297db78af224b5d61cd9eaca3e674ba15080643f82ffc3ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-24T00:18:12.590 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:18:12 smithi078 ceph-mds[70426]: -- 172.21.15.78:0/1282631684 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55af9f14da40 con 0x55af9f120c00 2024-03-24T00:18:12.590 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:18:12 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: starting mds.j at 2024-03-24T00:18:12.591 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:18:12 smithi078 bash[70366]: 2bae411f70111c67297db78af224b5d61cd9eaca3e674ba15080643f82ffc3ee 2024-03-24T00:18:12.591 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:18:12 smithi078 systemd[1]: Started Ceph mds.j for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:18:12.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:12 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/4221219098' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:12.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:12 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/4221219098' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:13.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:13 smithi078 ceph-mon[32429]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:13.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:13 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:13.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:13 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:13.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:13 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:13.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:13 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:13.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:13 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:13.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:13 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:13.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:13 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6834/1416685123,v1:172.21.15.78:6835/1416685123] up:boot 2024-03-24T00:18:13.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:13 smithi078 ceph-mon[32429]: fsmap 3 up:standby 2024-03-24T00:18:13.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:13 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:18:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:13 smithi107 ceph-mon[36485]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:13 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:13 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:13 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:13 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:13.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:13 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:13.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:13 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:13.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:13 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6834/1416685123,v1:172.21.15.78:6835/1416685123] up:boot 2024-03-24T00:18:13.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:13 smithi107 ceph-mon[36485]: fsmap 3 up:standby 2024-03-24T00:18:13.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:13 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:18:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:13 smithi106 ceph-mon[36505]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:13 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:13 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:13 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:13 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:13 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:13 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:13.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:13 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6834/1416685123,v1:172.21.15.78:6835/1416685123] up:boot 2024-03-24T00:18:13.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:13 smithi106 ceph-mon[36505]: fsmap 3 up:standby 2024-03-24T00:18:13.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:13 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:18:14.282 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:18:14.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.528+0000 7fa72ebf3640 1 -- 172.21.15.78:0/2973019553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7281033c0 msgr2=0x7fa728109c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:14.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.528+0000 7fa72ebf3640 1 --2- 172.21.15.78:0/2973019553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7281033c0 0x7fa728109c50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa720009cd0 tx=0x7fa72002c0f0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.528+0000 7fa72ebf3640 1 -- 172.21.15.78:0/2973019553 shutdown_connections 2024-03-24T00:18:14.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.528+0000 7fa72ebf3640 1 --2- 172.21.15.78:0/2973019553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7281033c0 0x7fa728109c50 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.528+0000 7fa72ebf3640 1 --2- 172.21.15.78:0/2973019553 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa728102a00 0x7fa728102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.528+0000 7fa72ebf3640 1 --2- 172.21.15.78:0/2973019553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7280ff990 0x7fa7280ffdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.528+0000 7fa72ebf3640 1 -- 172.21.15.78:0/2973019553 >> 172.21.15.78:0/2973019553 conn(0x7fa7280fb2a0 msgr2=0x7fa7280fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:14.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.528+0000 7fa72ebf3640 1 -- 172.21.15.78:0/2973019553 shutdown_connections 2024-03-24T00:18:14.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.528+0000 7fa72ebf3640 1 -- 172.21.15.78:0/2973019553 wait complete. 2024-03-24T00:18:14.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.528+0000 7fa72ebf3640 1 Processor -- start 2024-03-24T00:18:14.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.528+0000 7fa72ebf3640 1 -- start start 2024-03-24T00:18:14.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.529+0000 7fa72ebf3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7280ff990 0x7fa728197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:14.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.529+0000 7fa72ebf3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa728102a00 0x7fa728197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:14.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.529+0000 7fa72ebf3640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7281033c0 0x7fa72819ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:14.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.529+0000 7fa72ebf3640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa72810d310 con 0x7fa7280ff990 2024-03-24T00:18:14.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.529+0000 7fa72ebf3640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa72810d190 con 0x7fa7281033c0 2024-03-24T00:18:14.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.529+0000 7fa72ebf3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa72810d490 con 0x7fa728102a00 2024-03-24T00:18:14.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.529+0000 7fa72c98f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7280ff990 0x7fa728197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:14.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.529+0000 7fa72c98f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7280ff990 0x7fa728197950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50850/0 (socket says 172.21.15.78:50850) 2024-03-24T00:18:14.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.529+0000 7fa72c98f640 1 -- 172.21.15.78:0/914549937 learned_addr learned my addr 172.21.15.78:0/914549937 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:14.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.529+0000 7fa72c98f640 1 -- 172.21.15.78:0/914549937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa728102a00 msgr2=0x7fa728197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:14.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.529+0000 7fa72c98f640 1 --2- 172.21.15.78:0/914549937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa728102a00 0x7fa728197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.529+0000 7fa72d190640 1 --2- 172.21.15.78:0/914549937 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7281033c0 0x7fa72819ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:14.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.529+0000 7fa72c98f640 1 -- 172.21.15.78:0/914549937 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7281033c0 msgr2=0x7fa72819ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:14.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.529+0000 7fa72c98f640 1 --2- 172.21.15.78:0/914549937 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7281033c0 0x7fa72819ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.529+0000 7fa72c98f640 1 -- 172.21.15.78:0/914549937 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa72819f570 con 0x7fa7280ff990 2024-03-24T00:18:14.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.529+0000 7fa72c98f640 1 --2- 172.21.15.78:0/914549937 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7280ff990 0x7fa728197950 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa71c00c970 tx=0x7fa71c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:14.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.529+0000 7fa725ffb640 1 -- 172.21.15.78:0/914549937 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa71c004220 con 0x7fa7280ff990 2024-03-24T00:18:14.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.529+0000 7fa725ffb640 1 -- 172.21.15.78:0/914549937 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa71c0043c0 con 0x7fa7280ff990 2024-03-24T00:18:14.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.529+0000 7fa72ebf3640 1 -- 172.21.15.78:0/914549937 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa72819f860 con 0x7fa7280ff990 2024-03-24T00:18:14.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.529+0000 7fa725ffb640 1 -- 172.21.15.78:0/914549937 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa71c00f450 con 0x7fa7280ff990 2024-03-24T00:18:14.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.529+0000 7fa72ebf3640 1 -- 172.21.15.78:0/914549937 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa72819fe00 con 0x7fa7280ff990 2024-03-24T00:18:14.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.531+0000 7fa72ebf3640 1 -- 172.21.15.78:0/914549937 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa728108ad0 con 0x7fa7280ff990 2024-03-24T00:18:14.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.531+0000 7fa725ffb640 1 -- 172.21.15.78:0/914549937 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa71c0079b0 con 0x7fa7280ff990 2024-03-24T00:18:14.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.532+0000 7fa725ffb640 1 --2- 172.21.15.78:0/914549937 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa7180a2880 0x7fa7180a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:14.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.532+0000 7fa727fff640 1 --2- 172.21.15.78:0/914549937 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa7180a2880 0x7fa7180a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:14.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.532+0000 7fa725ffb640 1 -- 172.21.15.78:0/914549937 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa71c0ce350 con 0x7fa7280ff990 2024-03-24T00:18:14.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.532+0000 7fa727fff640 1 --2- 172.21.15.78:0/914549937 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa7180a2880 0x7fa7180a4d40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa714005fd0 tx=0x7fa714005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:14.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.537+0000 7fa725ffb640 1 -- 172.21.15.78:0/914549937 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa71c014030 con 0x7fa7280ff990 2024-03-24T00:18:14.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.790+0000 7fa72ebf3640 1 -- 172.21.15.78:0/914549937 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa72802d430 con 0x7fa7280ff990 2024-03-24T00:18:14.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.791+0000 7fa725ffb640 1 -- 172.21.15.78:0/914549937 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2913 (secure 0 0 0) 0x7fa71c09b890 con 0x7fa7280ff990 2024-03-24T00:18:14.792 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:18:14.792 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1416685123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1416685123},{"type":"v1","addr":"172.21.15.78:6835","nonce":1416685123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/3086318443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3086318443},{"type":"v1","addr":"172.21.15.107:6833","nonce":3086318443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/2367018868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2367018868},{"type":"v1","addr":"172.21.15.106:6833","nonce":2367018868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-24T00:18:14.792 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 4 2024-03-24T00:18:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.794+0000 7fa72ebf3640 1 -- 172.21.15.78:0/914549937 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa7180a2880 msgr2=0x7fa7180a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.794+0000 7fa72ebf3640 1 --2- 172.21.15.78:0/914549937 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa7180a2880 0x7fa7180a4d40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa714005fd0 tx=0x7fa714005850 comp rx=0 tx=0).stop 2024-03-24T00:18:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.794+0000 7fa72ebf3640 1 -- 172.21.15.78:0/914549937 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7280ff990 msgr2=0x7fa728197950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.794+0000 7fa72ebf3640 1 --2- 172.21.15.78:0/914549937 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7280ff990 0x7fa728197950 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa71c00c970 tx=0x7fa71c00ce30 comp rx=0 tx=0).stop 2024-03-24T00:18:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.794+0000 7fa72ebf3640 1 -- 172.21.15.78:0/914549937 shutdown_connections 2024-03-24T00:18:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.794+0000 7fa72ebf3640 1 --2- 172.21.15.78:0/914549937 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa7180a2880 0x7fa7180a4d40 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.794+0000 7fa72ebf3640 1 --2- 172.21.15.78:0/914549937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa728102a00 0x7fa728197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.794+0000 7fa72ebf3640 1 --2- 172.21.15.78:0/914549937 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7281033c0 0x7fa72819ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.794+0000 7fa72ebf3640 1 --2- 172.21.15.78:0/914549937 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7280ff990 0x7fa728197950 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.794+0000 7fa72ebf3640 1 -- 172.21.15.78:0/914549937 >> 172.21.15.78:0/914549937 conn(0x7fa7280fb2a0 msgr2=0x7fa728105ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.794+0000 7fa72ebf3640 1 -- 172.21.15.78:0/914549937 shutdown_connections 2024-03-24T00:18:14.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:14.794+0000 7fa72ebf3640 1 -- 172.21.15.78:0/914549937 wait complete. 2024-03-24T00:18:14.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:14 smithi078 ceph-mon[32429]: Deploying daemon mds.d on smithi078 2024-03-24T00:18:14.806 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:18:14 smithi078 systemd[1]: Starting Ceph mds.d for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:18:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:14 smithi107 ceph-mon[36485]: Deploying daemon mds.d on smithi078 2024-03-24T00:18:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:14 smithi106 ceph-mon[36505]: Deploying daemon mds.d on smithi078 2024-03-24T00:18:15.068 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:18:14 smithi078 podman[70756]: 2024-03-24 00:18:14.800107688 +0000 UTC m=+0.018679563 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:18:15.068 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:18:14 smithi078 podman[70756]: 2024-03-24 00:18:14.924853536 +0000 UTC m=+0.143425387 container create e84c2439bea2c279fea20cb137e076397e1f0f62f137b2eeddaf2a8b6d394d5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-24T00:18:15.466 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:18:15 smithi078 podman[70756]: 2024-03-24 00:18:15.169123969 +0000 UTC m=+0.387695831 container init e84c2439bea2c279fea20cb137e076397e1f0f62f137b2eeddaf2a8b6d394d5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-24T00:18:15.466 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:18:15 smithi078 podman[70756]: 2024-03-24 00:18:15.171570874 +0000 UTC m=+0.390142729 container start e84c2439bea2c279fea20cb137e076397e1f0f62f137b2eeddaf2a8b6d394d5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-24T00:18:15.466 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:18:15 smithi078 ceph-mds[70779]: -- 172.21.15.78:0/2018382653 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559f26df7a40 con 0x559f26dcac00 2024-03-24T00:18:15.466 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:18:15 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: starting mds.d at 2024-03-24T00:18:15.466 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:18:15 smithi078 bash[70756]: e84c2439bea2c279fea20cb137e076397e1f0f62f137b2eeddaf2a8b6d394d5d 2024-03-24T00:18:15.466 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:18:15 smithi078 systemd[1]: Started Ceph mds.d for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:18:15.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:15 smithi078 ceph-mon[32429]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:15.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:15 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/914549937' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:15.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:15 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:15.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:15 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:15.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:15 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:15.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:15 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:15.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:15 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:15.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:15 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:15 smithi107 ceph-mon[36485]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:15 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/914549937' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:15 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:15 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:15 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:15 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:15 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:15 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:15 smithi106 ceph-mon[36505]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:15 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/914549937' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:15 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:15 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:15 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:15 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:15 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:15.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:15 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:16 smithi106 ceph-mon[36505]: Deploying daemon mds.e on smithi106 2024-03-24T00:18:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:16 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6836/1071274623,v1:172.21.15.78:6837/1071274623] up:boot 2024-03-24T00:18:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:16 smithi106 ceph-mon[36505]: fsmap 4 up:standby 2024-03-24T00:18:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:16 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:18:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:16 smithi106 ceph-mon[36505]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:16.808 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:18:16.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:16 smithi078 ceph-mon[32429]: Deploying daemon mds.e on smithi106 2024-03-24T00:18:16.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:16 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6836/1071274623,v1:172.21.15.78:6837/1071274623] up:boot 2024-03-24T00:18:16.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:16 smithi078 ceph-mon[32429]: fsmap 4 up:standby 2024-03-24T00:18:16.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:16 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:18:16.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:16 smithi078 ceph-mon[32429]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:16 smithi107 ceph-mon[36485]: Deploying daemon mds.e on smithi106 2024-03-24T00:18:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:16 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6836/1071274623,v1:172.21.15.78:6837/1071274623] up:boot 2024-03-24T00:18:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:16 smithi107 ceph-mon[36485]: fsmap 4 up:standby 2024-03-24T00:18:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:16 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:18:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:16 smithi107 ceph-mon[36485]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:16.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.931+0000 7ff980b7e640 1 -- 172.21.15.78:0/53470733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c1021f0 msgr2=0x7ff97c108a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:16.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.931+0000 7ff980b7e640 1 --2- 172.21.15.78:0/53470733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c1021f0 0x7ff97c108a80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff974009ca0 tx=0x7ff97402c0f0 comp rx=0 tx=0).stop 2024-03-24T00:18:16.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.931+0000 7ff980b7e640 1 -- 172.21.15.78:0/53470733 shutdown_connections 2024-03-24T00:18:16.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.931+0000 7ff980b7e640 1 --2- 172.21.15.78:0/53470733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c1021f0 0x7ff97c108a80 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:16.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.931+0000 7ff980b7e640 1 --2- 172.21.15.78:0/53470733 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff97c101830 0x7ff97c101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:16.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.931+0000 7ff980b7e640 1 --2- 172.21.15.78:0/53470733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff97c100640 0x7ff97c100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:16.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.931+0000 7ff980b7e640 1 -- 172.21.15.78:0/53470733 >> 172.21.15.78:0/53470733 conn(0x7ff97c0fbbd0 msgr2=0x7ff97c0fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:16.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.931+0000 7ff980b7e640 1 -- 172.21.15.78:0/53470733 shutdown_connections 2024-03-24T00:18:16.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.931+0000 7ff980b7e640 1 -- 172.21.15.78:0/53470733 wait complete. 2024-03-24T00:18:16.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.931+0000 7ff980b7e640 1 Processor -- start 2024-03-24T00:18:16.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.932+0000 7ff980b7e640 1 -- start start 2024-03-24T00:18:16.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.932+0000 7ff980b7e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff97c100640 0x7ff97c1967e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:16.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.932+0000 7ff980b7e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c101830 0x7ff97c196d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:16.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.932+0000 7ff980b7e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff97c1021f0 0x7ff97c19dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:16.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.932+0000 7ff980b7e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97c10c250 con 0x7ff97c101830 2024-03-24T00:18:16.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.932+0000 7ff980b7e640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97c10c0d0 con 0x7ff97c100640 2024-03-24T00:18:16.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.932+0000 7ff980b7e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97c10c3d0 con 0x7ff97c1021f0 2024-03-24T00:18:16.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.932+0000 7ff979d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c101830 0x7ff97c196d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:16.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.932+0000 7ff979d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c101830 0x7ff97c196d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50880/0 (socket says 172.21.15.78:50880) 2024-03-24T00:18:16.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.932+0000 7ff979d9b640 1 -- 172.21.15.78:0/3234561975 learned_addr learned my addr 172.21.15.78:0/3234561975 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:16.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.932+0000 7ff97ad9d640 1 --2- 172.21.15.78:0/3234561975 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff97c1021f0 0x7ff97c19dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:16.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.932+0000 7ff97a59c640 1 --2- 172.21.15.78:0/3234561975 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff97c100640 0x7ff97c1967e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:16.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.932+0000 7ff979d9b640 1 -- 172.21.15.78:0/3234561975 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff97c1021f0 msgr2=0x7ff97c19dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:16.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.932+0000 7ff979d9b640 1 --2- 172.21.15.78:0/3234561975 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff97c1021f0 0x7ff97c19dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:16.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.932+0000 7ff979d9b640 1 -- 172.21.15.78:0/3234561975 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff97c100640 msgr2=0x7ff97c1967e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:16.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.932+0000 7ff979d9b640 1 --2- 172.21.15.78:0/3234561975 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff97c100640 0x7ff97c1967e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:16.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.932+0000 7ff979d9b640 1 -- 172.21.15.78:0/3234561975 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff97c19e400 con 0x7ff97c101830 2024-03-24T00:18:16.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.932+0000 7ff979d9b640 1 --2- 172.21.15.78:0/3234561975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c101830 0x7ff97c196d20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff97000c920 tx=0x7ff97000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:16.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.933+0000 7ff96b7fe640 1 -- 172.21.15.78:0/3234561975 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff970007c70 con 0x7ff97c101830 2024-03-24T00:18:16.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.933+0000 7ff96b7fe640 1 -- 172.21.15.78:0/3234561975 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff970007e10 con 0x7ff97c101830 2024-03-24T00:18:16.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.933+0000 7ff980b7e640 1 -- 172.21.15.78:0/3234561975 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff97c19e6f0 con 0x7ff97c101830 2024-03-24T00:18:16.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.933+0000 7ff96b7fe640 1 -- 172.21.15.78:0/3234561975 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff970018890 con 0x7ff97c101830 2024-03-24T00:18:16.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.933+0000 7ff980b7e640 1 -- 172.21.15.78:0/3234561975 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff97c19ec40 con 0x7ff97c101830 2024-03-24T00:18:16.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.934+0000 7ff980b7e640 1 -- 172.21.15.78:0/3234561975 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff97c107900 con 0x7ff97c101830 2024-03-24T00:18:16.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.935+0000 7ff96b7fe640 1 -- 172.21.15.78:0/3234561975 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff970018a70 con 0x7ff97c101830 2024-03-24T00:18:16.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.935+0000 7ff96b7fe640 1 --2- 172.21.15.78:0/3234561975 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff9640a2800 0x7ff9640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:16.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.936+0000 7ff97a59c640 1 --2- 172.21.15.78:0/3234561975 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff9640a2800 0x7ff9640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:16.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.936+0000 7ff96b7fe640 1 -- 172.21.15.78:0/3234561975 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff9700ce710 con 0x7ff97c101830 2024-03-24T00:18:16.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.936+0000 7ff97a59c640 1 --2- 172.21.15.78:0/3234561975 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff9640a2800 0x7ff9640a4cc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff96c005fd0 tx=0x7ff96c005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:16.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:16.940+0000 7ff96b7fe640 1 -- 172.21.15.78:0/3234561975 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff970014030 con 0x7ff97c101830 2024-03-24T00:18:17.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:17.156+0000 7ff980b7e640 1 -- 172.21.15.78:0/3234561975 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff97c02d070 con 0x7ff97c101830 2024-03-24T00:18:17.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:17.156+0000 7ff96b7fe640 1 -- 172.21.15.78:0/3234561975 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3704 (secure 0 0 0) 0x7ff97009bc50 con 0x7ff97c101830 2024-03-24T00:18:17.157 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:18:17.157 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1416685123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1416685123},{"type":"v1","addr":"172.21.15.78:6835","nonce":1416685123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1071274623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1071274623},{"type":"v1","addr":"172.21.15.78:6837","nonce":1071274623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/3086318443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3086318443},{"type":"v1","addr":"172.21.15.107:6833","nonce":3086318443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/2367018868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2367018868},{"type":"v1","addr":"172.21.15.106:6833","nonce":2367018868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-24T00:18:17.158 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 5 2024-03-24T00:18:17.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:17.158+0000 7ff980b7e640 1 -- 172.21.15.78:0/3234561975 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff9640a2800 msgr2=0x7ff9640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:17.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:17.158+0000 7ff980b7e640 1 --2- 172.21.15.78:0/3234561975 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff9640a2800 0x7ff9640a4cc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff96c005fd0 tx=0x7ff96c005850 comp rx=0 tx=0).stop 2024-03-24T00:18:17.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:17.158+0000 7ff980b7e640 1 -- 172.21.15.78:0/3234561975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c101830 msgr2=0x7ff97c196d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:17.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:17.158+0000 7ff980b7e640 1 --2- 172.21.15.78:0/3234561975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c101830 0x7ff97c196d20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff97000c920 tx=0x7ff97000cde0 comp rx=0 tx=0).stop 2024-03-24T00:18:17.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:17.158+0000 7ff980b7e640 1 -- 172.21.15.78:0/3234561975 shutdown_connections 2024-03-24T00:18:17.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:17.158+0000 7ff980b7e640 1 --2- 172.21.15.78:0/3234561975 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff9640a2800 0x7ff9640a4cc0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:17.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:17.158+0000 7ff980b7e640 1 --2- 172.21.15.78:0/3234561975 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff97c1021f0 0x7ff97c19dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:17.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:17.158+0000 7ff980b7e640 1 --2- 172.21.15.78:0/3234561975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c101830 0x7ff97c196d20 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:17.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:17.158+0000 7ff980b7e640 1 --2- 172.21.15.78:0/3234561975 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff97c100640 0x7ff97c1967e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:17.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:17.158+0000 7ff980b7e640 1 -- 172.21.15.78:0/3234561975 >> 172.21.15.78:0/3234561975 conn(0x7ff97c0fbbd0 msgr2=0x7ff97c104ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:17.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:17.158+0000 7ff980b7e640 1 -- 172.21.15.78:0/3234561975 shutdown_connections 2024-03-24T00:18:17.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:17.158+0000 7ff980b7e640 1 -- 172.21.15.78:0/3234561975 wait complete. 2024-03-24T00:18:17.806 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:17 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3234561975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:17.806 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 24 00:18:17 smithi106 systemd[1]: Starting Ceph mds.e for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:18:17.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:17 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3234561975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:17.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:17 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3234561975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:18.059 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 24 00:18:17 smithi106 podman[66353]: 2024-03-24 00:18:17.709298726 +0000 UTC m=+0.017353313 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:18:18.059 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 24 00:18:17 smithi106 podman[66353]: 2024-03-24 00:18:17.831787275 +0000 UTC m=+0.139841879 container create d421b8ece3df473359b1be72524413f52fa2d430cea617546e5c6418ab637952 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-e, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:18:18.059 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 24 00:18:18 smithi106 podman[66353]: 2024-03-24 00:18:18.059376936 +0000 UTC m=+0.367431537 container init d421b8ece3df473359b1be72524413f52fa2d430cea617546e5c6418ab637952 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-e, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-24T00:18:18.402 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 24 00:18:18 smithi106 podman[66353]: 2024-03-24 00:18:18.062333684 +0000 UTC m=+0.370388278 container start d421b8ece3df473359b1be72524413f52fa2d430cea617546e5c6418ab637952 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS) 2024-03-24T00:18:18.402 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 24 00:18:18 smithi106 ceph-mds[66372]: -- 172.21.15.106:0/1563541678 <== mon.2 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5601ef977a40 con 0x5601ef94a800 2024-03-24T00:18:18.402 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 24 00:18:18 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-e[66368]: starting mds.e at 2024-03-24T00:18:18.403 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 24 00:18:18 smithi106 bash[66353]: d421b8ece3df473359b1be72524413f52fa2d430cea617546e5c6418ab637952 2024-03-24T00:18:18.403 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 24 00:18:18 smithi106 systemd[1]: Started Ceph mds.e for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:18:19.170 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:18:19.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.286+0000 7f13e5a52640 1 -- 172.21.15.78:0/3282843126 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13e0102a20 msgr2=0x7f13e0102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:19.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.286+0000 7f13e5a52640 1 --2- 172.21.15.78:0/3282843126 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13e0102a20 0x7f13e0102ea0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f13d4009930 tx=0x7f13d402bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:19.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.286+0000 7f13e5a52640 1 -- 172.21.15.78:0/3282843126 shutdown_connections 2024-03-24T00:18:19.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.286+0000 7f13e5a52640 1 --2- 172.21.15.78:0/3282843126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13e01033e0 0x7f13e0109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:19.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.286+0000 7f13e5a52640 1 --2- 172.21.15.78:0/3282843126 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13e0102a20 0x7f13e0102ea0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:19.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.286+0000 7f13e5a52640 1 --2- 172.21.15.78:0/3282843126 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13e00ff9b0 0x7f13e00ffdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:19.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.286+0000 7f13e5a52640 1 -- 172.21.15.78:0/3282843126 >> 172.21.15.78:0/3282843126 conn(0x7f13e00fb2c0 msgr2=0x7f13e00fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:19.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.286+0000 7f13e5a52640 1 -- 172.21.15.78:0/3282843126 shutdown_connections 2024-03-24T00:18:19.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.286+0000 7f13e5a52640 1 -- 172.21.15.78:0/3282843126 wait complete. 2024-03-24T00:18:19.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.286+0000 7f13e5a52640 1 Processor -- start 2024-03-24T00:18:19.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.286+0000 7f13e5a52640 1 -- start start 2024-03-24T00:18:19.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.287+0000 7f13e5a52640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13e00ff9b0 0x7f13e0102390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:19.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.287+0000 7f13e5a52640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13e0102a20 0x7f13e01009d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:19.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.287+0000 7f13e5a52640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13e01033e0 0x7f13e0100f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:19.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.287+0000 7f13e5a52640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e010d360 con 0x7f13e0102a20 2024-03-24T00:18:19.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.287+0000 7f13e5a52640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e010d1e0 con 0x7f13e01033e0 2024-03-24T00:18:19.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.287+0000 7f13e5a52640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e010d4e0 con 0x7f13e00ff9b0 2024-03-24T00:18:19.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.287+0000 7f13de7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13e0102a20 0x7f13e01009d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:19.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.287+0000 7f13de7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13e0102a20 0x7f13e01009d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50910/0 (socket says 172.21.15.78:50910) 2024-03-24T00:18:19.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.287+0000 7f13de7fc640 1 -- 172.21.15.78:0/3538500745 learned_addr learned my addr 172.21.15.78:0/3538500745 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:19.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.287+0000 7f13df7fe640 1 --2- 172.21.15.78:0/3538500745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13e01033e0 0x7f13e0100f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:19.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.287+0000 7f13deffd640 1 --2- 172.21.15.78:0/3538500745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13e00ff9b0 0x7f13e0102390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:19.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.287+0000 7f13de7fc640 1 -- 172.21.15.78:0/3538500745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13e00ff9b0 msgr2=0x7f13e0102390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:19.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.287+0000 7f13de7fc640 1 --2- 172.21.15.78:0/3538500745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13e00ff9b0 0x7f13e0102390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:19.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.287+0000 7f13de7fc640 1 -- 172.21.15.78:0/3538500745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13e01033e0 msgr2=0x7f13e0100f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:19.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.287+0000 7f13de7fc640 1 --2- 172.21.15.78:0/3538500745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13e01033e0 0x7f13e0100f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:19.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.287+0000 7f13de7fc640 1 -- 172.21.15.78:0/3538500745 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e01017a0 con 0x7f13e0102a20 2024-03-24T00:18:19.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.287+0000 7f13de7fc640 1 --2- 172.21.15.78:0/3538500745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13e0102a20 0x7f13e01009d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f13d4000c00 tx=0x7f13d402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:19.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.287+0000 7f13e4a50640 1 -- 172.21.15.78:0/3538500745 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d4041070 con 0x7f13e0102a20 2024-03-24T00:18:19.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.287+0000 7f13e4a50640 1 -- 172.21.15.78:0/3538500745 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f13d4004450 con 0x7f13e0102a20 2024-03-24T00:18:19.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.287+0000 7f13e5a52640 1 -- 172.21.15.78:0/3538500745 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13e01a1a70 con 0x7f13e0102a20 2024-03-24T00:18:19.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.287+0000 7f13e4a50640 1 -- 172.21.15.78:0/3538500745 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d4034440 con 0x7f13e0102a20 2024-03-24T00:18:19.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.287+0000 7f13e5a52640 1 -- 172.21.15.78:0/3538500745 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13e01a1fc0 con 0x7f13e0102a20 2024-03-24T00:18:19.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.289+0000 7f13e5a52640 1 -- 172.21.15.78:0/3538500745 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13e0108b50 con 0x7f13e0102a20 2024-03-24T00:18:19.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.289+0000 7f13e4a50640 1 -- 172.21.15.78:0/3538500745 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f13d4044070 con 0x7f13e0102a20 2024-03-24T00:18:19.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.290+0000 7f13e4a50640 1 --2- 172.21.15.78:0/3538500745 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f13c80a2800 0x7f13c80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:19.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.290+0000 7f13deffd640 1 --2- 172.21.15.78:0/3538500745 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f13c80a2800 0x7f13c80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:19.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.290+0000 7f13e4a50640 1 -- 172.21.15.78:0/3538500745 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f13d40f0110 con 0x7f13e0102a20 2024-03-24T00:18:19.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.291+0000 7f13deffd640 1 --2- 172.21.15.78:0/3538500745 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f13c80a2800 0x7f13c80a4cc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f13e0069a90 tx=0x7f13d0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:19.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.295+0000 7f13e4a50640 1 -- 172.21.15.78:0/3538500745 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f13d40bd6e0 con 0x7f13e0102a20 2024-03-24T00:18:19.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.506+0000 7f13e5a52640 1 -- 172.21.15.78:0/3538500745 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13e002d070 con 0x7f13e0102a20 2024-03-24T00:18:19.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.507+0000 7f13e4a50640 1 -- 172.21.15.78:0/3538500745 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4498 (secure 0 0 0) 0x7f13d40bcfe0 con 0x7f13e0102a20 2024-03-24T00:18:19.507 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:18:19.507 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1416685123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1416685123},{"type":"v1","addr":"172.21.15.78:6835","nonce":1416685123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1071274623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1071274623},{"type":"v1","addr":"172.21.15.78:6837","nonce":1071274623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/3086318443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3086318443},{"type":"v1","addr":"172.21.15.107:6833","nonce":3086318443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/2367018868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2367018868},{"type":"v1","addr":"172.21.15.106:6833","nonce":2367018868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/2231611477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":2231611477},{"type":"v1","addr":"172.21.15.106:6835","nonce":2231611477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-03-24T00:18:19.507 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 6 2024-03-24T00:18:19.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.509+0000 7f13e5a52640 1 -- 172.21.15.78:0/3538500745 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f13c80a2800 msgr2=0x7f13c80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:19.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.509+0000 7f13e5a52640 1 --2- 172.21.15.78:0/3538500745 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f13c80a2800 0x7f13c80a4cc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f13e0069a90 tx=0x7f13d0005f90 comp rx=0 tx=0).stop 2024-03-24T00:18:19.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.509+0000 7f13e5a52640 1 -- 172.21.15.78:0/3538500745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13e0102a20 msgr2=0x7f13e01009d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:19.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.509+0000 7f13e5a52640 1 --2- 172.21.15.78:0/3538500745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13e0102a20 0x7f13e01009d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f13d4000c00 tx=0x7f13d402c640 comp rx=0 tx=0).stop 2024-03-24T00:18:19.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.509+0000 7f13e5a52640 1 -- 172.21.15.78:0/3538500745 shutdown_connections 2024-03-24T00:18:19.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.509+0000 7f13e5a52640 1 --2- 172.21.15.78:0/3538500745 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f13c80a2800 0x7f13c80a4cc0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:19.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.509+0000 7f13e5a52640 1 --2- 172.21.15.78:0/3538500745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13e01033e0 0x7f13e0100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:19.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.509+0000 7f13e5a52640 1 --2- 172.21.15.78:0/3538500745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13e0102a20 0x7f13e01009d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:19.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.509+0000 7f13e5a52640 1 --2- 172.21.15.78:0/3538500745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13e00ff9b0 0x7f13e0102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:19.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.509+0000 7f13e5a52640 1 -- 172.21.15.78:0/3538500745 >> 172.21.15.78:0/3538500745 conn(0x7f13e00fb2c0 msgr2=0x7f13e0105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:19.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.509+0000 7f13e5a52640 1 -- 172.21.15.78:0/3538500745 shutdown_connections 2024-03-24T00:18:19.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:19.509+0000 7f13e5a52640 1 -- 172.21.15.78:0/3538500745 wait complete. 2024-03-24T00:18:19.519 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:19 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:19.519 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:19 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:19.519 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:19 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:19.519 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:19 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:19.519 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:19 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:19.519 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:19 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:19.519 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:19 smithi078 ceph-mon[32429]: Deploying daemon mds.f on smithi107 2024-03-24T00:18:19.519 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:19 smithi078 ceph-mon[32429]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:19.519 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:19 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6834/2231611477,v1:172.21.15.106:6835/2231611477] up:boot 2024-03-24T00:18:19.519 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:19 smithi078 ceph-mon[32429]: fsmap 5 up:standby 2024-03-24T00:18:19.520 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:19 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T00:18:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:19 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:19 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:19 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:19 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:19 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:19 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:19.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:19 smithi107 ceph-mon[36485]: Deploying daemon mds.f on smithi107 2024-03-24T00:18:19.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:19 smithi107 ceph-mon[36485]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:19.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:19 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6834/2231611477,v1:172.21.15.106:6835/2231611477] up:boot 2024-03-24T00:18:19.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:19 smithi107 ceph-mon[36485]: fsmap 5 up:standby 2024-03-24T00:18:19.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:19 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T00:18:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:19 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:19 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:19 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:19 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:19 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:19 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:19 smithi106 ceph-mon[36505]: Deploying daemon mds.f on smithi107 2024-03-24T00:18:19.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:19 smithi106 ceph-mon[36505]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:19.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:19 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6834/2231611477,v1:172.21.15.106:6835/2231611477] up:boot 2024-03-24T00:18:19.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:19 smithi106 ceph-mon[36505]: fsmap 5 up:standby 2024-03-24T00:18:19.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:19 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T00:18:20.556 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:20 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3538500745' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:20.556 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:18:20 smithi107 systemd[1]: Starting Ceph mds.f for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:18:20.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:20 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3538500745' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:20.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:20 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3538500745' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:20.843 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:18:20 smithi107 podman[66869]: 2024-03-24 00:18:20.458967577 +0000 UTC m=+0.018365306 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:18:20.843 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:18:20 smithi107 podman[66869]: 2024-03-24 00:18:20.575894933 +0000 UTC m=+0.135292647 container create bf5d224a6f1e013d539c03a8d089cddc3e20d940dabd9119c540a98a499b2847 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-24T00:18:21.237 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:18:20 smithi107 podman[66869]: 2024-03-24 00:18:20.853443323 +0000 UTC m=+0.412841042 container init bf5d224a6f1e013d539c03a8d089cddc3e20d940dabd9119c540a98a499b2847 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-24T00:18:21.237 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:18:20 smithi107 podman[66869]: 2024-03-24 00:18:20.856222055 +0000 UTC m=+0.415619778 container start bf5d224a6f1e013d539c03a8d089cddc3e20d940dabd9119c540a98a499b2847 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:18:21.237 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:18:20 smithi107 ceph-mds[66888]: -- 172.21.15.107:0/2286618037 <== mon.1 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5567829dfa40 con 0x5567829b3000 2024-03-24T00:18:21.237 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:18:20 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: starting mds.f at 2024-03-24T00:18:21.238 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:18:20 smithi107 bash[66869]: bf5d224a6f1e013d539c03a8d089cddc3e20d940dabd9119c540a98a499b2847 2024-03-24T00:18:21.238 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:18:20 smithi107 systemd[1]: Started Ceph mds.f for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:18:21.522 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:18:21.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:21 smithi078 ceph-mon[32429]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:21.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:21 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:21.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:21 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:21.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:21 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:21.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:21 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:21.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:21 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:21.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:21 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:21.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:21 smithi107 ceph-mon[36485]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:21.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:21 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:21.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:21 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:21.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:21 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:21.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:21 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:21.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:21 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:21.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:21 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:21.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.637+0000 7f3f783d4640 1 -- 172.21.15.78:0/434208092 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f70101900 msgr2=0x7f3f70108190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:21.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.637+0000 7f3f783d4640 1 --2- 172.21.15.78:0/434208092 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f70101900 0x7f3f70108190 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3f68009ca0 tx=0x7f3f6802c0f0 comp rx=0 tx=0).stop 2024-03-24T00:18:21.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.637+0000 7f3f783d4640 1 -- 172.21.15.78:0/434208092 shutdown_connections 2024-03-24T00:18:21.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.637+0000 7f3f783d4640 1 --2- 172.21.15.78:0/434208092 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f70101900 0x7f3f70108190 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:21.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.637+0000 7f3f783d4640 1 --2- 172.21.15.78:0/434208092 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f70100f40 0x7f3f701013c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:21.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.637+0000 7f3f783d4640 1 --2- 172.21.15.78:0/434208092 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f700ffd50 0x7f3f70100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:21.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.637+0000 7f3f783d4640 1 -- 172.21.15.78:0/434208092 >> 172.21.15.78:0/434208092 conn(0x7f3f700fb2c0 msgr2=0x7f3f700fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:21.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.637+0000 7f3f783d4640 1 -- 172.21.15.78:0/434208092 shutdown_connections 2024-03-24T00:18:21.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.637+0000 7f3f783d4640 1 -- 172.21.15.78:0/434208092 wait complete. 2024-03-24T00:18:21.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.638+0000 7f3f783d4640 1 Processor -- start 2024-03-24T00:18:21.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.638+0000 7f3f783d4640 1 -- start start 2024-03-24T00:18:21.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.638+0000 7f3f783d4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f700ffd50 0x7f3f70103db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:21.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.638+0000 7f3f783d4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f70100f40 0x7f3f701023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:21.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.638+0000 7f3f783d4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f70101900 0x7f3f70102930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:21.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.638+0000 7f3f783d4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7010c260 con 0x7f3f70100f40 2024-03-24T00:18:21.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.638+0000 7f3f783d4640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7010c0e0 con 0x7f3f700ffd50 2024-03-24T00:18:21.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.638+0000 7f3f783d4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7010c3e0 con 0x7f3f70101900 2024-03-24T00:18:21.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.638+0000 7f3f7596f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f70100f40 0x7f3f701023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:21.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.638+0000 7f3f7596f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f70100f40 0x7f3f701023f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50914/0 (socket says 172.21.15.78:50914) 2024-03-24T00:18:21.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.638+0000 7f3f7596f640 1 -- 172.21.15.78:0/2370997577 learned_addr learned my addr 172.21.15.78:0/2370997577 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:21.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.638+0000 7f3f76971640 1 --2- 172.21.15.78:0/2370997577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f70101900 0x7f3f70102930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:21.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.638+0000 7f3f7596f640 1 -- 172.21.15.78:0/2370997577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f70101900 msgr2=0x7f3f70102930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:21.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.638+0000 7f3f7596f640 1 --2- 172.21.15.78:0/2370997577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f70101900 0x7f3f70102930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:21.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.638+0000 7f3f76170640 1 --2- 172.21.15.78:0/2370997577 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f700ffd50 0x7f3f70103db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:21.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.638+0000 7f3f7596f640 1 -- 172.21.15.78:0/2370997577 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f700ffd50 msgr2=0x7f3f70103db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:21.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.638+0000 7f3f7596f640 1 --2- 172.21.15.78:0/2370997577 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f700ffd50 0x7f3f70103db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:21.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.638+0000 7f3f7596f640 1 -- 172.21.15.78:0/2370997577 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f701031c0 con 0x7f3f70100f40 2024-03-24T00:18:21.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.638+0000 7f3f76170640 1 --2- 172.21.15.78:0/2370997577 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f700ffd50 0x7f3f70103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:18:21.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.638+0000 7f3f7596f640 1 --2- 172.21.15.78:0/2370997577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f70100f40 0x7f3f701023f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3f6000c890 tx=0x7f3f6000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:21.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.638+0000 7f3f677fe640 1 -- 172.21.15.78:0/2370997577 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f60004220 con 0x7f3f70100f40 2024-03-24T00:18:21.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.639+0000 7f3f783d4640 1 -- 172.21.15.78:0/2370997577 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f701a1a90 con 0x7f3f70100f40 2024-03-24T00:18:21.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.639+0000 7f3f677fe640 1 -- 172.21.15.78:0/2370997577 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3f600043c0 con 0x7f3f70100f40 2024-03-24T00:18:21.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.639+0000 7f3f677fe640 1 -- 172.21.15.78:0/2370997577 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f60004a00 con 0x7f3f70100f40 2024-03-24T00:18:21.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.639+0000 7f3f783d4640 1 -- 172.21.15.78:0/2370997577 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f701a2030 con 0x7f3f70100f40 2024-03-24T00:18:21.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.640+0000 7f3f783d4640 1 -- 172.21.15.78:0/2370997577 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f701a2260 con 0x7f3f70100f40 2024-03-24T00:18:21.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.641+0000 7f3f677fe640 1 -- 172.21.15.78:0/2370997577 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3f600079b0 con 0x7f3f70100f40 2024-03-24T00:18:21.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.641+0000 7f3f677fe640 1 --2- 172.21.15.78:0/2370997577 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3f5c0a2800 0x7f3f5c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:21.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.641+0000 7f3f76170640 1 --2- 172.21.15.78:0/2370997577 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3f5c0a2800 0x7f3f5c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:21.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.641+0000 7f3f677fe640 1 -- 172.21.15.78:0/2370997577 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3f600ce6e0 con 0x7f3f70100f40 2024-03-24T00:18:21.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.642+0000 7f3f76170640 1 --2- 172.21.15.78:0/2370997577 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3f5c0a2800 0x7f3f5c0a4cc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3f70100da0 tx=0x7f3f6c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:21.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.645+0000 7f3f677fe640 1 -- 172.21.15.78:0/2370997577 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3f60014030 con 0x7f3f70100f40 2024-03-24T00:18:21.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:21 smithi106 ceph-mon[36505]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:21.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:21 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:21.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:21 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:21.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:21 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:21.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:21 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:21.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:21 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:21.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:21 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:21.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.860+0000 7f3f783d4640 1 -- 172.21.15.78:0/2370997577 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f701a2430 con 0x7f3f70100f40 2024-03-24T00:18:21.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.861+0000 7f3f677fe640 1 -- 172.21.15.78:0/2370997577 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5289 (secure 0 0 0) 0x7f3f6009bc20 con 0x7f3f70100f40 2024-03-24T00:18:21.861 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:18:21.861 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1416685123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1416685123},{"type":"v1","addr":"172.21.15.78:6835","nonce":1416685123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1071274623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1071274623},{"type":"v1","addr":"172.21.15.78:6837","nonce":1071274623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/3086318443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3086318443},{"type":"v1","addr":"172.21.15.107:6833","nonce":3086318443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/2367018868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2367018868},{"type":"v1","addr":"172.21.15.106:6833","nonce":2367018868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/2231611477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":2231611477},{"type":"v1","addr":"172.21.15.106:6835","nonce":2231611477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/108936439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":108936439},{"type":"v1","addr":"172.21.15.107:6835","nonce":108936439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7}],"filesystems":[]} 2024-03-24T00:18:21.861 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 7 2024-03-24T00:18:21.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.863+0000 7f3f783d4640 1 -- 172.21.15.78:0/2370997577 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3f5c0a2800 msgr2=0x7f3f5c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:21.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.863+0000 7f3f783d4640 1 --2- 172.21.15.78:0/2370997577 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3f5c0a2800 0x7f3f5c0a4cc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3f70100da0 tx=0x7f3f6c005f90 comp rx=0 tx=0).stop 2024-03-24T00:18:21.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.863+0000 7f3f783d4640 1 -- 172.21.15.78:0/2370997577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f70100f40 msgr2=0x7f3f701023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:21.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.863+0000 7f3f783d4640 1 --2- 172.21.15.78:0/2370997577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f70100f40 0x7f3f701023f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3f6000c890 tx=0x7f3f6000cd50 comp rx=0 tx=0).stop 2024-03-24T00:18:21.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.863+0000 7f3f783d4640 1 -- 172.21.15.78:0/2370997577 shutdown_connections 2024-03-24T00:18:21.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.863+0000 7f3f783d4640 1 --2- 172.21.15.78:0/2370997577 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3f5c0a2800 0x7f3f5c0a4cc0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:21.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.863+0000 7f3f783d4640 1 --2- 172.21.15.78:0/2370997577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f70101900 0x7f3f70102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:21.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.863+0000 7f3f783d4640 1 --2- 172.21.15.78:0/2370997577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f70100f40 0x7f3f701023f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:21.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.863+0000 7f3f783d4640 1 --2- 172.21.15.78:0/2370997577 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f700ffd50 0x7f3f70103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:21.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.863+0000 7f3f783d4640 1 -- 172.21.15.78:0/2370997577 >> 172.21.15.78:0/2370997577 conn(0x7f3f700fb2c0 msgr2=0x7f3f70108850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:21.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.863+0000 7f3f783d4640 1 -- 172.21.15.78:0/2370997577 shutdown_connections 2024-03-24T00:18:21.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:21.863+0000 7f3f783d4640 1 -- 172.21.15.78:0/2370997577 wait complete. 2024-03-24T00:18:22.585 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:22 smithi107 ceph-mon[36485]: Deploying daemon mds.c on smithi107 2024-03-24T00:18:22.585 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:22 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6834/108936439,v1:172.21.15.107:6835/108936439] up:boot 2024-03-24T00:18:22.585 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:22 smithi107 ceph-mon[36485]: fsmap 6 up:standby 2024-03-24T00:18:22.585 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:22 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:18:22.585 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:22 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2370997577' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:22.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:22 smithi078 ceph-mon[32429]: Deploying daemon mds.c on smithi107 2024-03-24T00:18:22.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:22 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6834/108936439,v1:172.21.15.107:6835/108936439] up:boot 2024-03-24T00:18:22.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:22 smithi078 ceph-mon[32429]: fsmap 6 up:standby 2024-03-24T00:18:22.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:22 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:18:22.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:22 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2370997577' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:22.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:22 smithi106 ceph-mon[36505]: Deploying daemon mds.c on smithi107 2024-03-24T00:18:22.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:22 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6834/108936439,v1:172.21.15.107:6835/108936439] up:boot 2024-03-24T00:18:22.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:22 smithi106 ceph-mon[36505]: fsmap 6 up:standby 2024-03-24T00:18:22.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:22 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:18:22.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:22 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2370997577' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:23.093 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 24 00:18:23 smithi107 systemd[1]: Starting Ceph mds.c for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:18:23.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:23 smithi107 ceph-mon[36485]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:23.450 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 24 00:18:23 smithi107 podman[67175]: 2024-03-24 00:18:23.187732569 +0000 UTC m=+0.020000933 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:18:23.451 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 24 00:18:23 smithi107 podman[67175]: 2024-03-24 00:18:23.313685047 +0000 UTC m=+0.145953389 container create cedd2902e47d42e03f52b748bd15350a8e09d557a6ae1b5dffae0e11b6f502a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-24T00:18:23.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:23 smithi078 ceph-mon[32429]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:23.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:23 smithi106 ceph-mon[36505]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:23.843 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 24 00:18:23 smithi107 podman[67175]: 2024-03-24 00:18:23.583328166 +0000 UTC m=+0.415596528 container init cedd2902e47d42e03f52b748bd15350a8e09d557a6ae1b5dffae0e11b6f502a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-c, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-24T00:18:23.843 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 24 00:18:23 smithi107 podman[67175]: 2024-03-24 00:18:23.586584129 +0000 UTC m=+0.418852477 container start cedd2902e47d42e03f52b748bd15350a8e09d557a6ae1b5dffae0e11b6f502a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=) 2024-03-24T00:18:23.843 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 24 00:18:23 smithi107 ceph-mds[67204]: -- 172.21.15.107:0/1947738333 <== mon.1 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5595ebcd1a40 con 0x5595ebca4800 2024-03-24T00:18:23.843 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 24 00:18:23 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-c[67198]: starting mds.c at 2024-03-24T00:18:23.844 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 24 00:18:23 smithi107 bash[67175]: cedd2902e47d42e03f52b748bd15350a8e09d557a6ae1b5dffae0e11b6f502a6 2024-03-24T00:18:23.844 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 24 00:18:23 smithi107 systemd[1]: Started Ceph mds.c for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:18:23.874 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:18:23.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.988+0000 7f2ffa681640 1 -- 172.21.15.78:0/3392974532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ff41017b0 msgr2=0x7f2ff4101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:23.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.988+0000 7f2ffa681640 1 --2- 172.21.15.78:0/3392974532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ff41017b0 0x7f2ff4101c30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8009900 tx=0x7f2fe802bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:23.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.988+0000 7f2ffa681640 1 -- 172.21.15.78:0/3392974532 shutdown_connections 2024-03-24T00:18:23.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.988+0000 7f2ffa681640 1 --2- 172.21.15.78:0/3392974532 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ff4102170 0x7f2ff4108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:23.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.988+0000 7f2ffa681640 1 --2- 172.21.15.78:0/3392974532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ff41017b0 0x7f2ff4101c30 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:23.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.988+0000 7f2ffa681640 1 --2- 172.21.15.78:0/3392974532 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ff41005c0 0x7f2ff41009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:23.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.988+0000 7f2ffa681640 1 -- 172.21.15.78:0/3392974532 >> 172.21.15.78:0/3392974532 conn(0x7f2ff40fbb30 msgr2=0x7f2ff40fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:23.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.988+0000 7f2ffa681640 1 -- 172.21.15.78:0/3392974532 shutdown_connections 2024-03-24T00:18:23.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.988+0000 7f2ffa681640 1 -- 172.21.15.78:0/3392974532 wait complete. 2024-03-24T00:18:23.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.988+0000 7f2ffa681640 1 Processor -- start 2024-03-24T00:18:23.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.988+0000 7f2ffa681640 1 -- start start 2024-03-24T00:18:23.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.989+0000 7f2ffa681640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ff41005c0 0x7f2ff4197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:23.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.989+0000 7f2ffa681640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ff41017b0 0x7f2ff4197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:23.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.989+0000 7f2ffa681640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ff4102170 0x7f2ff419efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:23.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.989+0000 7f2ffa681640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff410c260 con 0x7f2ff41017b0 2024-03-24T00:18:23.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.989+0000 7f2ffa681640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff410c0e0 con 0x7f2ff4102170 2024-03-24T00:18:23.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.989+0000 7f2ffa681640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff410c3e0 con 0x7f2ff41005c0 2024-03-24T00:18:23.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.989+0000 7f2ff37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ff41017b0 0x7f2ff4197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:23.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.989+0000 7f2ff37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ff41017b0 0x7f2ff4197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50920/0 (socket says 172.21.15.78:50920) 2024-03-24T00:18:23.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.989+0000 7f2ff37fe640 1 -- 172.21.15.78:0/873079882 learned_addr learned my addr 172.21.15.78:0/873079882 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:23.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.989+0000 7f2ff3fff640 1 --2- 172.21.15.78:0/873079882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ff41005c0 0x7f2ff4197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:23.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.989+0000 7f2ff8c1e640 1 --2- 172.21.15.78:0/873079882 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ff4102170 0x7f2ff419efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:23.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.989+0000 7f2ff37fe640 1 -- 172.21.15.78:0/873079882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ff41005c0 msgr2=0x7f2ff4197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:23.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.989+0000 7f2ff37fe640 1 --2- 172.21.15.78:0/873079882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ff41005c0 0x7f2ff4197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:23.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.989+0000 7f2ff37fe640 1 -- 172.21.15.78:0/873079882 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ff4102170 msgr2=0x7f2ff419efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:23.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.989+0000 7f2ff37fe640 1 --2- 172.21.15.78:0/873079882 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ff4102170 0x7f2ff419efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:23.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.989+0000 7f2ff37fe640 1 -- 172.21.15.78:0/873079882 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff419f650 con 0x7f2ff41017b0 2024-03-24T00:18:23.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.989+0000 7f2ff37fe640 1 --2- 172.21.15.78:0/873079882 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ff41017b0 0x7f2ff4197fc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8009230 tx=0x7f2fe802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:23.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.989+0000 7f2ff17fa640 1 -- 172.21.15.78:0/873079882 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe8041070 con 0x7f2ff41017b0 2024-03-24T00:18:23.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.989+0000 7f2ff17fa640 1 -- 172.21.15.78:0/873079882 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2fe8004450 con 0x7f2ff41017b0 2024-03-24T00:18:23.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.989+0000 7f2ffa681640 1 -- 172.21.15.78:0/873079882 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff419f8e0 con 0x7f2ff41017b0 2024-03-24T00:18:23.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.989+0000 7f2ff17fa640 1 -- 172.21.15.78:0/873079882 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe8003e20 con 0x7f2ff41017b0 2024-03-24T00:18:23.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.990+0000 7f2ffa681640 1 -- 172.21.15.78:0/873079882 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ff419fdd0 con 0x7f2ff41017b0 2024-03-24T00:18:23.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.990+0000 7f2ffa681640 1 -- 172.21.15.78:0/873079882 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ff41a0030 con 0x7f2ff41017b0 2024-03-24T00:18:23.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.992+0000 7f2ff17fa640 1 -- 172.21.15.78:0/873079882 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2fe8044070 con 0x7f2ff41017b0 2024-03-24T00:18:23.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.992+0000 7f2ff17fa640 1 --2- 172.21.15.78:0/873079882 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2fdc0a2800 0x7f2fdc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:23.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.992+0000 7f2ff3fff640 1 --2- 172.21.15.78:0/873079882 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2fdc0a2800 0x7f2fdc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:23.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.992+0000 7f2ff17fa640 1 -- 172.21.15.78:0/873079882 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f2fe80f04c0 con 0x7f2ff41017b0 2024-03-24T00:18:23.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.993+0000 7f2ff3fff640 1 --2- 172.21.15.78:0/873079882 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2fdc0a2800 0x7f2fdc0a4cc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4101610 tx=0x7f2fe4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:23.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:23.996+0000 7f2ff17fa640 1 -- 172.21.15.78:0/873079882 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2fe80bd3b0 con 0x7f2ff41017b0 2024-03-24T00:18:24.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:24.210+0000 7f2ffa681640 1 -- 172.21.15.78:0/873079882 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ff41a0200 con 0x7f2ff41017b0 2024-03-24T00:18:24.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:24.211+0000 7f2ff17fa640 1 -- 172.21.15.78:0/873079882 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5289 (secure 0 0 0) 0x7f2fe80c1450 con 0x7f2ff41017b0 2024-03-24T00:18:24.211 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:18:24.211 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1416685123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1416685123},{"type":"v1","addr":"172.21.15.78:6835","nonce":1416685123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1071274623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1071274623},{"type":"v1","addr":"172.21.15.78:6837","nonce":1071274623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/3086318443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3086318443},{"type":"v1","addr":"172.21.15.107:6833","nonce":3086318443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/2367018868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2367018868},{"type":"v1","addr":"172.21.15.106:6833","nonce":2367018868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/2231611477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":2231611477},{"type":"v1","addr":"172.21.15.106:6835","nonce":2231611477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/108936439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":108936439},{"type":"v1","addr":"172.21.15.107:6835","nonce":108936439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7}],"filesystems":[]} 2024-03-24T00:18:24.211 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 7 2024-03-24T00:18:24.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:24.213+0000 7f2ffa681640 1 -- 172.21.15.78:0/873079882 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2fdc0a2800 msgr2=0x7f2fdc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:24.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:24.213+0000 7f2ffa681640 1 --2- 172.21.15.78:0/873079882 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2fdc0a2800 0x7f2fdc0a4cc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4101610 tx=0x7f2fe4005f90 comp rx=0 tx=0).stop 2024-03-24T00:18:24.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:24.213+0000 7f2ffa681640 1 -- 172.21.15.78:0/873079882 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ff41017b0 msgr2=0x7f2ff4197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:24.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:24.213+0000 7f2ffa681640 1 --2- 172.21.15.78:0/873079882 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ff41017b0 0x7f2ff4197fc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8009230 tx=0x7f2fe802c650 comp rx=0 tx=0).stop 2024-03-24T00:18:24.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:24.213+0000 7f2ffa681640 1 -- 172.21.15.78:0/873079882 shutdown_connections 2024-03-24T00:18:24.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:24.213+0000 7f2ffa681640 1 --2- 172.21.15.78:0/873079882 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2fdc0a2800 0x7f2fdc0a4cc0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:24.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:24.213+0000 7f2ffa681640 1 --2- 172.21.15.78:0/873079882 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ff4102170 0x7f2ff419efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:24.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:24.213+0000 7f2ffa681640 1 --2- 172.21.15.78:0/873079882 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ff41017b0 0x7f2ff4197fc0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:24.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:24.213+0000 7f2ffa681640 1 --2- 172.21.15.78:0/873079882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ff41005c0 0x7f2ff4197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:24.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:24.213+0000 7f2ffa681640 1 -- 172.21.15.78:0/873079882 >> 172.21.15.78:0/873079882 conn(0x7f2ff40fbb30 msgr2=0x7f2ff40fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:24.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:24.213+0000 7f2ffa681640 1 -- 172.21.15.78:0/873079882 shutdown_connections 2024-03-24T00:18:24.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:24.213+0000 7f2ffa681640 1 -- 172.21.15.78:0/873079882 wait complete. 2024-03-24T00:18:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:24 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:24 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:24 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:24.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:24 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:24.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:24 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:24.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:24 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:24.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:24 smithi106 ceph-mon[36505]: Deploying daemon mds.h on smithi106 2024-03-24T00:18:24.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:24 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/873079882' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:24.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:24 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6836/2939571540,v1:172.21.15.107:6837/2939571540] up:boot 2024-03-24T00:18:24.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:24 smithi106 ceph-mon[36505]: fsmap 7 up:standby 2024-03-24T00:18:24.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:24 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T00:18:24.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:24 smithi106 ceph-mon[36505]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:25.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:24 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:25.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:24 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:25.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:24 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:25.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:24 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:25.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:24 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:25.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:24 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:25.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:24 smithi078 ceph-mon[32429]: Deploying daemon mds.h on smithi106 2024-03-24T00:18:25.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:24 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/873079882' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:25.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:24 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6836/2939571540,v1:172.21.15.107:6837/2939571540] up:boot 2024-03-24T00:18:25.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:24 smithi078 ceph-mon[32429]: fsmap 7 up:standby 2024-03-24T00:18:25.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:24 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T00:18:25.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:24 smithi078 ceph-mon[32429]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:24 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:24 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:24 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:24 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:24 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:24 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:24 smithi107 ceph-mon[36485]: Deploying daemon mds.h on smithi106 2024-03-24T00:18:25.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:24 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/873079882' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:25.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:24 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6836/2939571540,v1:172.21.15.107:6837/2939571540] up:boot 2024-03-24T00:18:25.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:24 smithi107 ceph-mon[36485]: fsmap 7 up:standby 2024-03-24T00:18:25.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:24 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T00:18:25.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:24 smithi107 ceph-mon[36485]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:26.152 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 24 00:18:26 smithi106 systemd[1]: Starting Ceph mds.h for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:18:26.224 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:18:26.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.337+0000 7f677d6a1640 1 -- 172.21.15.78:0/987706322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67781005e0 msgr2=0x7f6778100a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:26.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.337+0000 7f677d6a1640 1 --2- 172.21.15.78:0/987706322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67781005e0 0x7f6778100a00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6768009900 tx=0x7f676802bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:26.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.337+0000 7f677d6a1640 1 -- 172.21.15.78:0/987706322 shutdown_connections 2024-03-24T00:18:26.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.337+0000 7f677d6a1640 1 --2- 172.21.15.78:0/987706322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6778102190 0x7f6778108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:26.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.337+0000 7f677d6a1640 1 --2- 172.21.15.78:0/987706322 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67781017d0 0x7f6778101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:26.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.337+0000 7f677d6a1640 1 --2- 172.21.15.78:0/987706322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67781005e0 0x7f6778100a00 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:26.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.337+0000 7f677d6a1640 1 -- 172.21.15.78:0/987706322 >> 172.21.15.78:0/987706322 conn(0x7f67780fbb50 msgr2=0x7f67780fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:26.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.337+0000 7f677d6a1640 1 -- 172.21.15.78:0/987706322 shutdown_connections 2024-03-24T00:18:26.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.337+0000 7f677d6a1640 1 -- 172.21.15.78:0/987706322 wait complete. 2024-03-24T00:18:26.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.338+0000 7f677d6a1640 1 Processor -- start 2024-03-24T00:18:26.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.338+0000 7f677d6a1640 1 -- start start 2024-03-24T00:18:26.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.338+0000 7f677d6a1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67781005e0 0x7f67781968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:26.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.338+0000 7f677d6a1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67781017d0 0x7f6778196e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:26.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.338+0000 7f677d6a1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6778102190 0x7f677819de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:26.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.338+0000 7f677d6a1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677810c2b0 con 0x7f6778102190 2024-03-24T00:18:26.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.338+0000 7f677d6a1640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677810c130 con 0x7f67781005e0 2024-03-24T00:18:26.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.338+0000 7f677d6a1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677810c430 con 0x7f67781017d0 2024-03-24T00:18:26.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.338+0000 7f67777fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6778102190 0x7f677819de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:26.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.338+0000 7f67777fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6778102190 0x7f677819de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56856/0 (socket says 172.21.15.78:56856) 2024-03-24T00:18:26.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.338+0000 7f67777fe640 1 -- 172.21.15.78:0/2437929327 learned_addr learned my addr 172.21.15.78:0/2437929327 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:26.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.338+0000 7f6776ffd640 1 --2- 172.21.15.78:0/2437929327 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67781005e0 0x7f67781968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:26.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.338+0000 7f676ffff640 1 --2- 172.21.15.78:0/2437929327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67781017d0 0x7f6778196e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:26.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.338+0000 7f67777fe640 1 -- 172.21.15.78:0/2437929327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67781017d0 msgr2=0x7f6778196e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:26.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.338+0000 7f67777fe640 1 --2- 172.21.15.78:0/2437929327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67781017d0 0x7f6778196e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:26.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.338+0000 7f67777fe640 1 -- 172.21.15.78:0/2437929327 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67781005e0 msgr2=0x7f67781968c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:26.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.338+0000 7f67777fe640 1 --2- 172.21.15.78:0/2437929327 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67781005e0 0x7f67781968c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:26.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.338+0000 7f67777fe640 1 -- 172.21.15.78:0/2437929327 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f677819e4e0 con 0x7f6778102190 2024-03-24T00:18:26.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.338+0000 7f67777fe640 1 --2- 172.21.15.78:0/2437929327 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6778102190 0x7f677819de70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f677000ebb0 tx=0x7f6770009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:26.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.338+0000 7f6774ff9640 1 -- 172.21.15.78:0/2437929327 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f677000e2e0 con 0x7f6778102190 2024-03-24T00:18:26.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.339+0000 7f677d6a1640 1 -- 172.21.15.78:0/2437929327 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67780699a0 con 0x7f6778102190 2024-03-24T00:18:26.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.339+0000 7f6774ff9640 1 -- 172.21.15.78:0/2437929327 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6770007950 con 0x7f6778102190 2024-03-24T00:18:26.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.339+0000 7f6774ff9640 1 -- 172.21.15.78:0/2437929327 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6770005790 con 0x7f6778102190 2024-03-24T00:18:26.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.339+0000 7f677d6a1640 1 -- 172.21.15.78:0/2437929327 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6778069ef0 con 0x7f6778102190 2024-03-24T00:18:26.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.340+0000 7f677d6a1640 1 -- 172.21.15.78:0/2437929327 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6778067c00 con 0x7f6778102190 2024-03-24T00:18:26.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.341+0000 7f6774ff9640 1 -- 172.21.15.78:0/2437929327 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6770007b00 con 0x7f6778102190 2024-03-24T00:18:26.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.341+0000 7f6774ff9640 1 --2- 172.21.15.78:0/2437929327 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f67640a2800 0x7f67640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:26.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.341+0000 7f6776ffd640 1 --2- 172.21.15.78:0/2437929327 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f67640a2800 0x7f67640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:26.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.341+0000 7f6774ff9640 1 -- 172.21.15.78:0/2437929327 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f67700cff00 con 0x7f6778102190 2024-03-24T00:18:26.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.342+0000 7f6776ffd640 1 --2- 172.21.15.78:0/2437929327 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f67640a2800 0x7f67640a4cc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f676802c240 tx=0x7f676802df90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:26.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.345+0000 7f6774ff9640 1 -- 172.21.15.78:0/2437929327 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f677009d440 con 0x7f6778102190 2024-03-24T00:18:26.506 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 24 00:18:26 smithi106 podman[66660]: 2024-03-24 00:18:26.137790283 +0000 UTC m=+0.017444478 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:18:26.506 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 24 00:18:26 smithi106 podman[66660]: 2024-03-24 00:18:26.254450344 +0000 UTC m=+0.134104542 container create cb8a0c8e7dbcf4a3542c653bab6687bcb58fe96b3fb11bdffd0d5f2671dc80ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-h, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-24T00:18:26.506 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 24 00:18:26 smithi106 podman[66660]: 2024-03-24 00:18:26.50719811 +0000 UTC m=+0.386852293 container init cb8a0c8e7dbcf4a3542c653bab6687bcb58fe96b3fb11bdffd0d5f2671dc80ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-h, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-24T00:18:26.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.557+0000 7f677d6a1640 1 -- 172.21.15.78:0/2437929327 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6778107530 con 0x7f6778102190 2024-03-24T00:18:26.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.558+0000 7f6774ff9640 1 -- 172.21.15.78:0/2437929327 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6083 (secure 0 0 0) 0x7f677009cd40 con 0x7f6778102190 2024-03-24T00:18:26.558 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:18:26.558 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1416685123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1416685123},{"type":"v1","addr":"172.21.15.78:6835","nonce":1416685123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1071274623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1071274623},{"type":"v1","addr":"172.21.15.78:6837","nonce":1071274623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/3086318443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3086318443},{"type":"v1","addr":"172.21.15.107:6833","nonce":3086318443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/2367018868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2367018868},{"type":"v1","addr":"172.21.15.106:6833","nonce":2367018868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/2231611477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":2231611477},{"type":"v1","addr":"172.21.15.106:6835","nonce":2231611477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/108936439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":108936439},{"type":"v1","addr":"172.21.15.107:6835","nonce":108936439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2939571540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2939571540},{"type":"v1","addr":"172.21.15.107:6837","nonce":2939571540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-24T00:18:26.558 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 8 2024-03-24T00:18:26.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.560+0000 7f677d6a1640 1 -- 172.21.15.78:0/2437929327 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f67640a2800 msgr2=0x7f67640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:26.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.560+0000 7f677d6a1640 1 --2- 172.21.15.78:0/2437929327 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f67640a2800 0x7f67640a4cc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f676802c240 tx=0x7f676802df90 comp rx=0 tx=0).stop 2024-03-24T00:18:26.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.560+0000 7f677d6a1640 1 -- 172.21.15.78:0/2437929327 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6778102190 msgr2=0x7f677819de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:26.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.560+0000 7f677d6a1640 1 --2- 172.21.15.78:0/2437929327 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6778102190 0x7f677819de70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f677000ebb0 tx=0x7f6770009f60 comp rx=0 tx=0).stop 2024-03-24T00:18:26.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.560+0000 7f677d6a1640 1 -- 172.21.15.78:0/2437929327 shutdown_connections 2024-03-24T00:18:26.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.560+0000 7f677d6a1640 1 --2- 172.21.15.78:0/2437929327 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f67640a2800 0x7f67640a4cc0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:26.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.560+0000 7f677d6a1640 1 --2- 172.21.15.78:0/2437929327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67781017d0 0x7f6778196e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:26.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.560+0000 7f677d6a1640 1 --2- 172.21.15.78:0/2437929327 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6778102190 0x7f677819de70 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:26.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.560+0000 7f677d6a1640 1 --2- 172.21.15.78:0/2437929327 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67781005e0 0x7f67781968c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:26.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.560+0000 7f677d6a1640 1 -- 172.21.15.78:0/2437929327 >> 172.21.15.78:0/2437929327 conn(0x7f67780fbb50 msgr2=0x7f6778104ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:26.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.560+0000 7f677d6a1640 1 -- 172.21.15.78:0/2437929327 shutdown_connections 2024-03-24T00:18:26.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:26.560+0000 7f677d6a1640 1 -- 172.21.15.78:0/2437929327 wait complete. 2024-03-24T00:18:26.902 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 24 00:18:26 smithi106 podman[66660]: 2024-03-24 00:18:26.510016337 +0000 UTC m=+0.389670510 container start cb8a0c8e7dbcf4a3542c653bab6687bcb58fe96b3fb11bdffd0d5f2671dc80ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-h, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True) 2024-03-24T00:18:26.902 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 24 00:18:26 smithi106 ceph-mds[66679]: -- 172.21.15.106:0/2803113763 <== mon.2 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5645bccb9a40 con 0x5645bcc8c800 2024-03-24T00:18:26.902 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 24 00:18:26 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-h[66675]: starting mds.h at 2024-03-24T00:18:26.902 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 24 00:18:26 smithi106 bash[66660]: cb8a0c8e7dbcf4a3542c653bab6687bcb58fe96b3fb11bdffd0d5f2671dc80ad 2024-03-24T00:18:26.903 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 24 00:18:26 smithi106 systemd[1]: Started Ceph mds.h for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:18:27.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:27 smithi078 ceph-mon[32429]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:27.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:27 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2437929327' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:27.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:27 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:27.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:27 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:27.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:27 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:27.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:27 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:27.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:27 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:27.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:27 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:27.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:27 smithi107 ceph-mon[36485]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:27.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:27 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2437929327' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:27.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:27 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:27.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:27 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:27.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:27 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:27.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:27 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:27.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:27 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:27.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:27 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:27.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:27 smithi106 ceph-mon[36505]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:27.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:27 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2437929327' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:27.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:27 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:27.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:27 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:27.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:27 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:27.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:27 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:27.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:27 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:27.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:27 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:28.572 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:18:28.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:28 smithi078 ceph-mon[32429]: Deploying daemon mds.i on smithi107 2024-03-24T00:18:28.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:28 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6836/2090968633,v1:172.21.15.106:6837/2090968633] up:boot 2024-03-24T00:18:28.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:28 smithi078 ceph-mon[32429]: fsmap 8 up:standby 2024-03-24T00:18:28.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:28 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T00:18:28.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:28 smithi107 ceph-mon[36485]: Deploying daemon mds.i on smithi107 2024-03-24T00:18:28.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:28 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6836/2090968633,v1:172.21.15.106:6837/2090968633] up:boot 2024-03-24T00:18:28.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:28 smithi107 ceph-mon[36485]: fsmap 8 up:standby 2024-03-24T00:18:28.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:28 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T00:18:28.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:28 smithi106 ceph-mon[36505]: Deploying daemon mds.i on smithi107 2024-03-24T00:18:28.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:28 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6836/2090968633,v1:172.21.15.106:6837/2090968633] up:boot 2024-03-24T00:18:28.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:28 smithi106 ceph-mon[36505]: fsmap 8 up:standby 2024-03-24T00:18:28.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:28 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T00:18:28.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.688+0000 7f74fd337640 1 -- 172.21.15.78:0/3942405006 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74f81017b0 msgr2=0x7f74f8101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:28.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.688+0000 7f74fd337640 1 --2- 172.21.15.78:0/3942405006 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74f81017b0 0x7f74f8101c30 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f74e0009900 tx=0x7f74e002bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:28.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.688+0000 7f74fd337640 1 -- 172.21.15.78:0/3942405006 shutdown_connections 2024-03-24T00:18:28.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.688+0000 7f74fd337640 1 --2- 172.21.15.78:0/3942405006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74f8102170 0x7f74f8108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:28.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.688+0000 7f74fd337640 1 --2- 172.21.15.78:0/3942405006 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74f81017b0 0x7f74f8101c30 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:28.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.688+0000 7f74fd337640 1 --2- 172.21.15.78:0/3942405006 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74f81005c0 0x7f74f81009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:28.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.688+0000 7f74fd337640 1 -- 172.21.15.78:0/3942405006 >> 172.21.15.78:0/3942405006 conn(0x7f74f80fbb50 msgr2=0x7f74f80fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:28.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.688+0000 7f74fd337640 1 -- 172.21.15.78:0/3942405006 shutdown_connections 2024-03-24T00:18:28.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.688+0000 7f74fd337640 1 -- 172.21.15.78:0/3942405006 wait complete. 2024-03-24T00:18:28.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.689+0000 7f74fd337640 1 Processor -- start 2024-03-24T00:18:28.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.689+0000 7f74fd337640 1 -- start start 2024-03-24T00:18:28.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.689+0000 7f74fd337640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74f81005c0 0x7f74f806a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:28.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.689+0000 7f74fd337640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74f81017b0 0x7f74f819c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:28.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.689+0000 7f74fd337640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74f8102170 0x7f74f819ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:28.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.689+0000 7f74fd337640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f810c220 con 0x7f74f8102170 2024-03-24T00:18:28.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.689+0000 7f74fd337640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f810c0a0 con 0x7f74f81005c0 2024-03-24T00:18:28.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.689+0000 7f74fd337640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f810c3a0 con 0x7f74f81017b0 2024-03-24T00:18:28.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.689+0000 7f74f77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74f8102170 0x7f74f819ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:28.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.689+0000 7f74f77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74f8102170 0x7f74f819ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56876/0 (socket says 172.21.15.78:56876) 2024-03-24T00:18:28.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.689+0000 7f74f77fe640 1 -- 172.21.15.78:0/1106224526 learned_addr learned my addr 172.21.15.78:0/1106224526 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:28.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.689+0000 7f74effff640 1 --2- 172.21.15.78:0/1106224526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74f81017b0 0x7f74f819c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:28.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.690+0000 7f74f6ffd640 1 --2- 172.21.15.78:0/1106224526 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74f81005c0 0x7f74f806a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:28.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.690+0000 7f74f77fe640 1 -- 172.21.15.78:0/1106224526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74f81017b0 msgr2=0x7f74f819c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:28.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.690+0000 7f74f77fe640 1 --2- 172.21.15.78:0/1106224526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74f81017b0 0x7f74f819c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:28.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.690+0000 7f74f77fe640 1 -- 172.21.15.78:0/1106224526 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74f81005c0 msgr2=0x7f74f806a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:28.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.690+0000 7f74f77fe640 1 --2- 172.21.15.78:0/1106224526 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74f81005c0 0x7f74f806a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:28.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.690+0000 7f74f77fe640 1 -- 172.21.15.78:0/1106224526 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f819f600 con 0x7f74f8102170 2024-03-24T00:18:28.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.690+0000 7f74f6ffd640 1 --2- 172.21.15.78:0/1106224526 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74f81005c0 0x7f74f806a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:18:28.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.690+0000 7f74f77fe640 1 --2- 172.21.15.78:0/1106224526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74f8102170 0x7f74f819ed40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f74f000eaa0 tx=0x7f74f000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:28.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.690+0000 7f74f4ff9640 1 -- 172.21.15.78:0/1106224526 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f000cdb0 con 0x7f74f8102170 2024-03-24T00:18:28.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.690+0000 7f74f4ff9640 1 -- 172.21.15.78:0/1106224526 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f74f0005020 con 0x7f74f8102170 2024-03-24T00:18:28.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.690+0000 7f74f4ff9640 1 -- 172.21.15.78:0/1106224526 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f00116b0 con 0x7f74f8102170 2024-03-24T00:18:28.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.690+0000 7f74fd337640 1 -- 172.21.15.78:0/1106224526 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74f819f8f0 con 0x7f74f8102170 2024-03-24T00:18:28.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.690+0000 7f74fd337640 1 -- 172.21.15.78:0/1106224526 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74f819fe10 con 0x7f74f8102170 2024-03-24T00:18:28.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.691+0000 7f74fd337640 1 -- 172.21.15.78:0/1106224526 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74f8107880 con 0x7f74f8102170 2024-03-24T00:18:28.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.692+0000 7f74f4ff9640 1 -- 172.21.15.78:0/1106224526 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f74f0005560 con 0x7f74f8102170 2024-03-24T00:18:28.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.693+0000 7f74f4ff9640 1 --2- 172.21.15.78:0/1106224526 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f74e40a2880 0x7f74e40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:28.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.693+0000 7f74f6ffd640 1 --2- 172.21.15.78:0/1106224526 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f74e40a2880 0x7f74e40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:28.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.693+0000 7f74f4ff9640 1 -- 172.21.15.78:0/1106224526 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f74f00ceb30 con 0x7f74f8102170 2024-03-24T00:18:28.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.693+0000 7f74f6ffd640 1 --2- 172.21.15.78:0/1106224526 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f74e40a2880 0x7f74e40a4d40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f74f8101610 tx=0x7f74e8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:28.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.696+0000 7f74f4ff9640 1 -- 172.21.15.78:0/1106224526 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f74f0016030 con 0x7f74f8102170 2024-03-24T00:18:28.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.908+0000 7f74fd337640 1 -- 172.21.15.78:0/1106224526 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74f802d070 con 0x7f74f8102170 2024-03-24T00:18:28.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.909+0000 7f74f4ff9640 1 -- 172.21.15.78:0/1106224526 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6877 (secure 0 0 0) 0x7f74f009c0f0 con 0x7f74f8102170 2024-03-24T00:18:28.909 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:18:28.909 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1416685123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1416685123},{"type":"v1","addr":"172.21.15.78:6835","nonce":1416685123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1071274623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1071274623},{"type":"v1","addr":"172.21.15.78:6837","nonce":1071274623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/3086318443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3086318443},{"type":"v1","addr":"172.21.15.107:6833","nonce":3086318443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/2367018868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2367018868},{"type":"v1","addr":"172.21.15.106:6833","nonce":2367018868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/2231611477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":2231611477},{"type":"v1","addr":"172.21.15.106:6835","nonce":2231611477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/108936439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":108936439},{"type":"v1","addr":"172.21.15.107:6835","nonce":108936439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2939571540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2939571540},{"type":"v1","addr":"172.21.15.107:6837","nonce":2939571540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/2090968633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":2090968633},{"type":"v1","addr":"172.21.15.106:6837","nonce":2090968633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-24T00:18:28.909 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 9 2024-03-24T00:18:28.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.911+0000 7f74fd337640 1 -- 172.21.15.78:0/1106224526 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f74e40a2880 msgr2=0x7f74e40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:28.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.911+0000 7f74fd337640 1 --2- 172.21.15.78:0/1106224526 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f74e40a2880 0x7f74e40a4d40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f74f8101610 tx=0x7f74e8005f90 comp rx=0 tx=0).stop 2024-03-24T00:18:28.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.911+0000 7f74fd337640 1 -- 172.21.15.78:0/1106224526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74f8102170 msgr2=0x7f74f819ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:28.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.911+0000 7f74fd337640 1 --2- 172.21.15.78:0/1106224526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74f8102170 0x7f74f819ed40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f74f000eaa0 tx=0x7f74f000ef60 comp rx=0 tx=0).stop 2024-03-24T00:18:28.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.911+0000 7f74fd337640 1 -- 172.21.15.78:0/1106224526 shutdown_connections 2024-03-24T00:18:28.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.911+0000 7f74fd337640 1 --2- 172.21.15.78:0/1106224526 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f74e40a2880 0x7f74e40a4d40 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:28.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.911+0000 7f74fd337640 1 --2- 172.21.15.78:0/1106224526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74f81017b0 0x7f74f819c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:28.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.911+0000 7f74fd337640 1 --2- 172.21.15.78:0/1106224526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74f8102170 0x7f74f819ed40 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:28.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.911+0000 7f74fd337640 1 --2- 172.21.15.78:0/1106224526 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74f81005c0 0x7f74f806a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:28.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.911+0000 7f74fd337640 1 -- 172.21.15.78:0/1106224526 >> 172.21.15.78:0/1106224526 conn(0x7f74f80fbb50 msgr2=0x7f74f8104a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:28.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.911+0000 7f74fd337640 1 -- 172.21.15.78:0/1106224526 shutdown_connections 2024-03-24T00:18:28.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:28.911+0000 7f74fd337640 1 -- 172.21.15.78:0/1106224526 wait complete. 2024-03-24T00:18:29.047 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:18:28 smithi107 systemd[1]: Starting Ceph mds.i for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:18:29.329 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:18:29 smithi107 podman[67496]: 2024-03-24 00:18:28.950743028 +0000 UTC m=+0.017663605 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:18:29.329 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:18:29 smithi107 podman[67496]: 2024-03-24 00:18:29.08848931 +0000 UTC m=+0.155409884 container create 06ef8cdf5fccc67cd8cc7604888e4363b69f84405e2e625e9e31cbf4960161fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-24T00:18:29.330 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:18:29 smithi107 podman[67496]: 2024-03-24 00:18:29.330329014 +0000 UTC m=+0.397249606 container init 06ef8cdf5fccc67cd8cc7604888e4363b69f84405e2e625e9e31cbf4960161fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, ceph=True) 2024-03-24T00:18:29.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:29 smithi078 ceph-mon[32429]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:29.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:29 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1106224526' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:29 smithi107 ceph-mon[36485]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:29 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1106224526' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:29.593 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:18:29 smithi107 podman[67496]: 2024-03-24 00:18:29.333037383 +0000 UTC m=+0.399957958 container start 06ef8cdf5fccc67cd8cc7604888e4363b69f84405e2e625e9e31cbf4960161fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-24T00:18:29.593 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:18:29 smithi107 ceph-mds[67515]: -- 172.21.15.107:0/323321216 <== mon.1 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560d325d1a40 con 0x560d325a4800 2024-03-24T00:18:29.593 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:18:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: starting mds.i at 2024-03-24T00:18:29.593 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:18:29 smithi107 bash[67496]: 06ef8cdf5fccc67cd8cc7604888e4363b69f84405e2e625e9e31cbf4960161fc 2024-03-24T00:18:29.594 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:18:29 smithi107 systemd[1]: Started Ceph mds.i for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:18:29.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:29 smithi106 ceph-mon[36505]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:29.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:29 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1106224526' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:30.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:30 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:30.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:30 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:30.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:30 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:30.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:30 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:30.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:30 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:30.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:30 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:30.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:30 smithi106 ceph-mon[36505]: Deploying daemon mds.g on smithi078 2024-03-24T00:18:30.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:30 smithi106 ceph-mon[36505]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:30.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:30 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6838/117119462,v1:172.21.15.107:6839/117119462] up:boot 2024-03-24T00:18:30.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:30 smithi106 ceph-mon[36505]: fsmap 9 up:standby 2024-03-24T00:18:30.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:30 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T00:18:30.830 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:30 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:30.830 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:30 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:30.830 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:30 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:30.830 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:30 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:30.830 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:30 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:30.831 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:30 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:30.831 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:30 smithi078 ceph-mon[32429]: Deploying daemon mds.g on smithi078 2024-03-24T00:18:30.831 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:30 smithi078 ceph-mon[32429]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:30.831 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:30 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6838/117119462,v1:172.21.15.107:6839/117119462] up:boot 2024-03-24T00:18:30.831 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:30 smithi078 ceph-mon[32429]: fsmap 9 up:standby 2024-03-24T00:18:30.831 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:30 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T00:18:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:30 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:30 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:30 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:30 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:30 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:30 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:30 smithi107 ceph-mon[36485]: Deploying daemon mds.g on smithi078 2024-03-24T00:18:30.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:30 smithi107 ceph-mon[36485]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:30.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:30 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6838/117119462,v1:172.21.15.107:6839/117119462] up:boot 2024-03-24T00:18:30.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:30 smithi107 ceph-mon[36485]: fsmap 9 up:standby 2024-03-24T00:18:30.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:30 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T00:18:30.923 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:18:31.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.030+0000 7ff446790640 1 -- 172.21.15.78:0/2388628930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff440103290 msgr2=0x7ff440109b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:31.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.030+0000 7ff446790640 1 --2- 172.21.15.78:0/2388628930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff440103290 0x7ff440109b20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff438009ca0 tx=0x7ff43802c0f0 comp rx=0 tx=0).stop 2024-03-24T00:18:31.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.030+0000 7ff446790640 1 -- 172.21.15.78:0/2388628930 shutdown_connections 2024-03-24T00:18:31.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.030+0000 7ff446790640 1 --2- 172.21.15.78:0/2388628930 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4401028d0 0x7ff440102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:31.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.030+0000 7ff446790640 1 --2- 172.21.15.78:0/2388628930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff440103290 0x7ff440109b20 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:31.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.030+0000 7ff446790640 1 --2- 172.21.15.78:0/2388628930 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff4401016e0 0x7ff440101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:31.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.030+0000 7ff446790640 1 -- 172.21.15.78:0/2388628930 >> 172.21.15.78:0/2388628930 conn(0x7ff4400fcc50 msgr2=0x7ff4400ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:31.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.030+0000 7ff446790640 1 -- 172.21.15.78:0/2388628930 shutdown_connections 2024-03-24T00:18:31.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.030+0000 7ff446790640 1 -- 172.21.15.78:0/2388628930 wait complete. 2024-03-24T00:18:31.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.030+0000 7ff446790640 1 Processor -- start 2024-03-24T00:18:31.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.030+0000 7ff446790640 1 -- start start 2024-03-24T00:18:31.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.030+0000 7ff446790640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff4401016e0 0x7ff440197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:31.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.030+0000 7ff446790640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4401028d0 0x7ff440197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:31.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.030+0000 7ff446790640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff440103290 0x7ff44019ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:31.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.030+0000 7ff446790640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44010d340 con 0x7ff4401028d0 2024-03-24T00:18:31.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.030+0000 7ff446790640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44010d1c0 con 0x7ff4401016e0 2024-03-24T00:18:31.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.030+0000 7ff446790640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44010d4c0 con 0x7ff440103290 2024-03-24T00:18:31.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.031+0000 7ff43f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4401028d0 0x7ff440197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:31.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.031+0000 7ff43f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4401028d0 0x7ff440197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56884/0 (socket says 172.21.15.78:56884) 2024-03-24T00:18:31.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.031+0000 7ff43f7fe640 1 -- 172.21.15.78:0/1457181383 learned_addr learned my addr 172.21.15.78:0/1457181383 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:31.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.031+0000 7ff43ffff640 1 --2- 172.21.15.78:0/1457181383 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff4401016e0 0x7ff440197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.031+0000 7ff444d2d640 1 --2- 172.21.15.78:0/1457181383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff440103290 0x7ff44019ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.031+0000 7ff43f7fe640 1 -- 172.21.15.78:0/1457181383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff440103290 msgr2=0x7ff44019ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.031+0000 7ff43f7fe640 1 --2- 172.21.15.78:0/1457181383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff440103290 0x7ff44019ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.031+0000 7ff43f7fe640 1 -- 172.21.15.78:0/1457181383 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff4401016e0 msgr2=0x7ff440197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.031+0000 7ff43f7fe640 1 --2- 172.21.15.78:0/1457181383 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff4401016e0 0x7ff440197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.031+0000 7ff43f7fe640 1 -- 172.21.15.78:0/1457181383 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff44019f5a0 con 0x7ff4401028d0 2024-03-24T00:18:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.031+0000 7ff43f7fe640 1 --2- 172.21.15.78:0/1457181383 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4401028d0 0x7ff440197ec0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff43400c890 tx=0x7ff43400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.031+0000 7ff43d7fa640 1 -- 172.21.15.78:0/1457181383 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff434004220 con 0x7ff4401028d0 2024-03-24T00:18:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.031+0000 7ff43d7fa640 1 -- 172.21.15.78:0/1457181383 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4340043c0 con 0x7ff4401028d0 2024-03-24T00:18:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.031+0000 7ff446790640 1 -- 172.21.15.78:0/1457181383 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff44019f890 con 0x7ff4401028d0 2024-03-24T00:18:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.031+0000 7ff43d7fa640 1 -- 172.21.15.78:0/1457181383 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4340049e0 con 0x7ff4401028d0 2024-03-24T00:18:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.031+0000 7ff446790640 1 -- 172.21.15.78:0/1457181383 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff44019fde0 con 0x7ff4401028d0 2024-03-24T00:18:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.032+0000 7ff446790640 1 -- 172.21.15.78:0/1457181383 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4401089a0 con 0x7ff4401028d0 2024-03-24T00:18:31.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.033+0000 7ff43d7fa640 1 -- 172.21.15.78:0/1457181383 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff4340079b0 con 0x7ff4401028d0 2024-03-24T00:18:31.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.034+0000 7ff43d7fa640 1 --2- 172.21.15.78:0/1457181383 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff4280a2800 0x7ff4280a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:31.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.034+0000 7ff43ffff640 1 --2- 172.21.15.78:0/1457181383 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff4280a2800 0x7ff4280a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:31.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.034+0000 7ff43d7fa640 1 -- 172.21.15.78:0/1457181383 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff4340ce680 con 0x7ff4401028d0 2024-03-24T00:18:31.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.034+0000 7ff43ffff640 1 --2- 172.21.15.78:0/1457181383 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff4280a2800 0x7ff4280a4cc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff440102730 tx=0x7ff43000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:31.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.038+0000 7ff43d7fa640 1 -- 172.21.15.78:0/1457181383 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff434014030 con 0x7ff4401028d0 2024-03-24T00:18:31.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.253+0000 7ff446790640 1 -- 172.21.15.78:0/1457181383 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff44002d070 con 0x7ff4401028d0 2024-03-24T00:18:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.254+0000 7ff43d7fa640 1 -- 172.21.15.78:0/1457181383 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7670 (secure 0 0 0) 0x7ff43409bbc0 con 0x7ff4401028d0 2024-03-24T00:18:31.254 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:18:31.254 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1416685123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1416685123},{"type":"v1","addr":"172.21.15.78:6835","nonce":1416685123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1071274623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1071274623},{"type":"v1","addr":"172.21.15.78:6837","nonce":1071274623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/3086318443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3086318443},{"type":"v1","addr":"172.21.15.107:6833","nonce":3086318443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/2367018868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2367018868},{"type":"v1","addr":"172.21.15.106:6833","nonce":2367018868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/2231611477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":2231611477},{"type":"v1","addr":"172.21.15.106:6835","nonce":2231611477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/108936439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":108936439},{"type":"v1","addr":"172.21.15.107:6835","nonce":108936439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2939571540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2939571540},{"type":"v1","addr":"172.21.15.107:6837","nonce":2939571540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/2090968633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":2090968633},{"type":"v1","addr":"172.21.15.106:6837","nonce":2090968633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/117119462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":117119462},{"type":"v1","addr":"172.21.15.107:6839","nonce":117119462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-24T00:18:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 10 2024-03-24T00:18:31.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.256+0000 7ff446790640 1 -- 172.21.15.78:0/1457181383 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff4280a2800 msgr2=0x7ff4280a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:31.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.256+0000 7ff446790640 1 --2- 172.21.15.78:0/1457181383 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff4280a2800 0x7ff4280a4cc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff440102730 tx=0x7ff43000b390 comp rx=0 tx=0).stop 2024-03-24T00:18:31.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.256+0000 7ff446790640 1 -- 172.21.15.78:0/1457181383 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4401028d0 msgr2=0x7ff440197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:31.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.256+0000 7ff446790640 1 --2- 172.21.15.78:0/1457181383 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4401028d0 0x7ff440197ec0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff43400c890 tx=0x7ff43400cd50 comp rx=0 tx=0).stop 2024-03-24T00:18:31.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.256+0000 7ff446790640 1 -- 172.21.15.78:0/1457181383 shutdown_connections 2024-03-24T00:18:31.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.256+0000 7ff446790640 1 --2- 172.21.15.78:0/1457181383 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff4280a2800 0x7ff4280a4cc0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.256+0000 7ff446790640 1 --2- 172.21.15.78:0/1457181383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff440103290 0x7ff44019ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.256+0000 7ff446790640 1 --2- 172.21.15.78:0/1457181383 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4401028d0 0x7ff440197ec0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.256+0000 7ff446790640 1 --2- 172.21.15.78:0/1457181383 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff4401016e0 0x7ff440197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.256+0000 7ff446790640 1 -- 172.21.15.78:0/1457181383 >> 172.21.15.78:0/1457181383 conn(0x7ff4400fcc50 msgr2=0x7ff440105b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.256+0000 7ff446790640 1 -- 172.21.15.78:0/1457181383 shutdown_connections 2024-03-24T00:18:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:31.256+0000 7ff446790640 1 -- 172.21.15.78:0/1457181383 wait complete. 2024-03-24T00:18:31.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:31 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1457181383' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:31.843 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:18:31 smithi078 systemd[1]: Starting Ceph mds.g for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:18:31.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:31 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1457181383' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:31.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:31 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1457181383' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:32.240 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:18:31 smithi078 podman[71367]: 2024-03-24 00:18:31.855814109 +0000 UTC m=+0.017935620 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:18:32.240 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:18:31 smithi078 podman[71367]: 2024-03-24 00:18:31.987934283 +0000 UTC m=+0.150055786 container create fc59ea233761ff068cce1fb220b5da030e9e3fa27580dd222c2a12846ce01223 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-24T00:18:32.240 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:18:32 smithi078 podman[71367]: 2024-03-24 00:18:32.240349175 +0000 UTC m=+0.402470685 container init fc59ea233761ff068cce1fb220b5da030e9e3fa27580dd222c2a12846ce01223 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-24T00:18:32.590 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:18:32 smithi078 podman[71367]: 2024-03-24 00:18:32.243052545 +0000 UTC m=+0.405174057 container start fc59ea233761ff068cce1fb220b5da030e9e3fa27580dd222c2a12846ce01223 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux ) 2024-03-24T00:18:32.590 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:18:32 smithi078 ceph-mds[71386]: -- 172.21.15.78:0/1836093799 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557289db7a40 con 0x557289d8a800 2024-03-24T00:18:32.590 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:18:32 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: starting mds.g at 2024-03-24T00:18:32.591 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:18:32 smithi078 bash[71367]: fc59ea233761ff068cce1fb220b5da030e9e3fa27580dd222c2a12846ce01223 2024-03-24T00:18:32.591 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:18:32 smithi078 systemd[1]: Started Ceph mds.g for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:18:33.267 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:18:33.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.374+0000 7f6aacdb9640 1 -- 172.21.15.78:0/1227547311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6aa8100640 msgr2=0x7f6aa8100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:33.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.374+0000 7f6aacdb9640 1 --2- 172.21.15.78:0/1227547311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6aa8100640 0x7f6aa8100a60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6a98009900 tx=0x7f6a9802bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:33.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.374+0000 7f6aacdb9640 1 -- 172.21.15.78:0/1227547311 shutdown_connections 2024-03-24T00:18:33.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.374+0000 7f6aacdb9640 1 --2- 172.21.15.78:0/1227547311 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6aa81021f0 0x7f6aa8108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:33.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.374+0000 7f6aacdb9640 1 --2- 172.21.15.78:0/1227547311 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6aa8101830 0x7f6aa8101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:33.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.374+0000 7f6aacdb9640 1 --2- 172.21.15.78:0/1227547311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6aa8100640 0x7f6aa8100a60 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:33.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.374+0000 7f6aacdb9640 1 -- 172.21.15.78:0/1227547311 >> 172.21.15.78:0/1227547311 conn(0x7f6aa80fbbd0 msgr2=0x7f6aa80fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:33.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.374+0000 7f6aacdb9640 1 -- 172.21.15.78:0/1227547311 shutdown_connections 2024-03-24T00:18:33.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.374+0000 7f6aacdb9640 1 -- 172.21.15.78:0/1227547311 wait complete. 2024-03-24T00:18:33.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.375+0000 7f6aacdb9640 1 Processor -- start 2024-03-24T00:18:33.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.375+0000 7f6aacdb9640 1 -- start start 2024-03-24T00:18:33.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.375+0000 7f6aacdb9640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6aa8100640 0x7f6aa806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:33.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.375+0000 7f6aacdb9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6aa8101830 0x7f6aa819c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:33.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.375+0000 7f6aacdb9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6aa81021f0 0x7f6aa819edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:33.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.375+0000 7f6aacdb9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa810c2f0 con 0x7f6aa8101830 2024-03-24T00:18:33.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.375+0000 7f6aacdb9640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa810c170 con 0x7f6aa8100640 2024-03-24T00:18:33.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.375+0000 7f6aacdb9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa810c470 con 0x7f6aa81021f0 2024-03-24T00:18:33.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.375+0000 7f6a9ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6aa8101830 0x7f6aa819c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:33.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.375+0000 7f6a9ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6aa8101830 0x7f6aa819c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56920/0 (socket says 172.21.15.78:56920) 2024-03-24T00:18:33.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.375+0000 7f6a9ffff640 1 -- 172.21.15.78:0/3468041075 learned_addr learned my addr 172.21.15.78:0/3468041075 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:33.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.375+0000 7f6aa659c640 1 --2- 172.21.15.78:0/3468041075 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6aa8100640 0x7f6aa806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:33.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.376+0000 7f6a9ffff640 1 -- 172.21.15.78:0/3468041075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6aa81021f0 msgr2=0x7f6aa819edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:33.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.376+0000 7f6a9ffff640 1 --2- 172.21.15.78:0/3468041075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6aa81021f0 0x7f6aa819edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:33.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.376+0000 7f6a9ffff640 1 -- 172.21.15.78:0/3468041075 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6aa8100640 msgr2=0x7f6aa806a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:33.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.376+0000 7f6a9ffff640 1 --2- 172.21.15.78:0/3468041075 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6aa8100640 0x7f6aa806a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:33.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.376+0000 7f6a9ffff640 1 -- 172.21.15.78:0/3468041075 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aa819f6e0 con 0x7f6aa8101830 2024-03-24T00:18:33.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.376+0000 7f6a9ffff640 1 --2- 172.21.15.78:0/3468041075 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6aa8101830 0x7f6aa819c7b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6a9000d950 tx=0x7f6a9000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:33.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.376+0000 7f6a9f7fe640 1 -- 172.21.15.78:0/3468041075 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a90017070 con 0x7f6aa8101830 2024-03-24T00:18:33.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.376+0000 7f6a9f7fe640 1 -- 172.21.15.78:0/3468041075 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a90009c30 con 0x7f6aa8101830 2024-03-24T00:18:33.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.376+0000 7f6aacdb9640 1 -- 172.21.15.78:0/3468041075 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6aa819f970 con 0x7f6aa8101830 2024-03-24T00:18:33.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.376+0000 7f6a9f7fe640 1 -- 172.21.15.78:0/3468041075 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a9000f6b0 con 0x7f6aa8101830 2024-03-24T00:18:33.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.376+0000 7f6aacdb9640 1 -- 172.21.15.78:0/3468041075 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6aa819fec0 con 0x7f6aa8101830 2024-03-24T00:18:33.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.377+0000 7f6aacdb9640 1 -- 172.21.15.78:0/3468041075 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6aa8107960 con 0x7f6aa8101830 2024-03-24T00:18:33.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.378+0000 7f6a9f7fe640 1 -- 172.21.15.78:0/3468041075 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6a900109d0 con 0x7f6aa8101830 2024-03-24T00:18:33.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.378+0000 7f6a9f7fe640 1 --2- 172.21.15.78:0/3468041075 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f6a940a2800 0x7f6a940a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:33.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.379+0000 7f6aa659c640 1 --2- 172.21.15.78:0/3468041075 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f6a940a2800 0x7f6a940a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:33.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.379+0000 7f6a9f7fe640 1 -- 172.21.15.78:0/3468041075 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6a900ce270 con 0x7f6aa8101830 2024-03-24T00:18:33.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.379+0000 7f6aa659c640 1 --2- 172.21.15.78:0/3468041075 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f6a940a2800 0x7f6a940a4cc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6a9802c170 tx=0x7f6a9802df90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:33.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.383+0000 7f6a9f7fe640 1 -- 172.21.15.78:0/3468041075 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6a90009de0 con 0x7f6aa8101830 2024-03-24T00:18:33.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:33 smithi078 ceph-mon[32429]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:33.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:33 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:33.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:33 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:33.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:33 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:33.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:33 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:33.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:33 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:33.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:33 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:33.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:33 smithi078 ceph-mon[32429]: Deploying daemon mds.a on smithi078 2024-03-24T00:18:33.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:33 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6838/2758357551,v1:172.21.15.78:6839/2758357551] up:boot 2024-03-24T00:18:33.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:33 smithi078 ceph-mon[32429]: fsmap 10 up:standby 2024-03-24T00:18:33.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:33 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:18:33.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.599+0000 7f6aacdb9640 1 -- 172.21.15.78:0/3468041075 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6aa802d070 con 0x7f6aa8101830 2024-03-24T00:18:33.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.599+0000 7f6a9f7fe640 1 -- 172.21.15.78:0/3468041075 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8462 (secure 0 0 0) 0x7f6a9009b7b0 con 0x7f6aa8101830 2024-03-24T00:18:33.599 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:18:33.599 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1416685123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1416685123},{"type":"v1","addr":"172.21.15.78:6835","nonce":1416685123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1071274623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1071274623},{"type":"v1","addr":"172.21.15.78:6837","nonce":1071274623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2758357551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2758357551},{"type":"v1","addr":"172.21.15.78:6839","nonce":2758357551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/3086318443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3086318443},{"type":"v1","addr":"172.21.15.107:6833","nonce":3086318443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/2367018868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2367018868},{"type":"v1","addr":"172.21.15.106:6833","nonce":2367018868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/2231611477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":2231611477},{"type":"v1","addr":"172.21.15.106:6835","nonce":2231611477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/108936439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":108936439},{"type":"v1","addr":"172.21.15.107:6835","nonce":108936439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2939571540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2939571540},{"type":"v1","addr":"172.21.15.107:6837","nonce":2939571540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/2090968633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":2090968633},{"type":"v1","addr":"172.21.15.106:6837","nonce":2090968633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/117119462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":117119462},{"type":"v1","addr":"172.21.15.107:6839","nonce":117119462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-24T00:18:33.600 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 11 2024-03-24T00:18:33.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.602+0000 7f6aacdb9640 1 -- 172.21.15.78:0/3468041075 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f6a940a2800 msgr2=0x7f6a940a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:33.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.602+0000 7f6aacdb9640 1 --2- 172.21.15.78:0/3468041075 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f6a940a2800 0x7f6a940a4cc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6a9802c170 tx=0x7f6a9802df90 comp rx=0 tx=0).stop 2024-03-24T00:18:33.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.602+0000 7f6aacdb9640 1 -- 172.21.15.78:0/3468041075 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6aa8101830 msgr2=0x7f6aa819c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:33.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.602+0000 7f6aacdb9640 1 --2- 172.21.15.78:0/3468041075 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6aa8101830 0x7f6aa819c7b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6a9000d950 tx=0x7f6a9000de10 comp rx=0 tx=0).stop 2024-03-24T00:18:33.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.602+0000 7f6aacdb9640 1 -- 172.21.15.78:0/3468041075 shutdown_connections 2024-03-24T00:18:33.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.602+0000 7f6aacdb9640 1 --2- 172.21.15.78:0/3468041075 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f6a940a2800 0x7f6a940a4cc0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:33.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.602+0000 7f6aacdb9640 1 --2- 172.21.15.78:0/3468041075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6aa81021f0 0x7f6aa819edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:33.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.602+0000 7f6aacdb9640 1 --2- 172.21.15.78:0/3468041075 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6aa8101830 0x7f6aa819c7b0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:33.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.602+0000 7f6aacdb9640 1 --2- 172.21.15.78:0/3468041075 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6aa8100640 0x7f6aa806a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:33.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.602+0000 7f6aacdb9640 1 -- 172.21.15.78:0/3468041075 >> 172.21.15.78:0/3468041075 conn(0x7f6aa80fbbd0 msgr2=0x7f6aa8104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:33.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.602+0000 7f6aacdb9640 1 -- 172.21.15.78:0/3468041075 shutdown_connections 2024-03-24T00:18:33.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:33.602+0000 7f6aacdb9640 1 -- 172.21.15.78:0/3468041075 wait complete. 2024-03-24T00:18:33.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:33 smithi106 ceph-mon[36505]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:33.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:33 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:33.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:33 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:33.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:33 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:33.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:33 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:33.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:33 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:33.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:33 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:33.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:33 smithi106 ceph-mon[36505]: Deploying daemon mds.a on smithi078 2024-03-24T00:18:33.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:33 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6838/2758357551,v1:172.21.15.78:6839/2758357551] up:boot 2024-03-24T00:18:33.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:33 smithi106 ceph-mon[36505]: fsmap 10 up:standby 2024-03-24T00:18:33.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:33 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:18:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:33 smithi107 ceph-mon[36485]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:33 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:33 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:33 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:33 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:33.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:33 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:33.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:33 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:33.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:33 smithi107 ceph-mon[36485]: Deploying daemon mds.a on smithi078 2024-03-24T00:18:33.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:33 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6838/2758357551,v1:172.21.15.78:6839/2758357551] up:boot 2024-03-24T00:18:33.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:33 smithi107 ceph-mon[36485]: fsmap 10 up:standby 2024-03-24T00:18:33.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:33 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:18:34.576 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:34 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3468041075' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:34 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3468041075' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:34 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3468041075' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:35.340 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:18:34 smithi078 systemd[1]: Started Ceph mds.a for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:18:35.613 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:18:35.642 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:35 smithi078 ceph-mon[32429]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:35.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:35 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:35.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:35 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:35.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:35 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:35.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:35 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:35.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:35 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:35.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:35 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:35.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:35 smithi106 ceph-mon[36505]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:35.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:35 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:35.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:35 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:35.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:35 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:35.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:35 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:35.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:35 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:35.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:35 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:35.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.725+0000 7ff6594f1640 1 -- 172.21.15.78:0/2849144888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6541021e0 msgr2=0x7ff6541045d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:35.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.725+0000 7ff6594f1640 1 --2- 172.21.15.78:0/2849144888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6541021e0 0x7ff6541045d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff644009ca0 tx=0x7ff64402c0f0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.725+0000 7ff6594f1640 1 -- 172.21.15.78:0/2849144888 shutdown_connections 2024-03-24T00:18:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.725+0000 7ff6594f1640 1 --2- 172.21.15.78:0/2849144888 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff654107440 0x7ff654109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.725+0000 7ff6594f1640 1 --2- 172.21.15.78:0/2849144888 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff654104b10 0x7ff654106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.725+0000 7ff6594f1640 1 --2- 172.21.15.78:0/2849144888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6541021e0 0x7ff6541045d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.725+0000 7ff6594f1640 1 -- 172.21.15.78:0/2849144888 >> 172.21.15.78:0/2849144888 conn(0x7ff6540fbb30 msgr2=0x7ff6540fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.725+0000 7ff6594f1640 1 -- 172.21.15.78:0/2849144888 shutdown_connections 2024-03-24T00:18:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.725+0000 7ff6594f1640 1 -- 172.21.15.78:0/2849144888 wait complete. 2024-03-24T00:18:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.726+0000 7ff6594f1640 1 Processor -- start 2024-03-24T00:18:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.726+0000 7ff6594f1640 1 -- start start 2024-03-24T00:18:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.726+0000 7ff6594f1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff6541021e0 0x7ff654197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.726+0000 7ff6594f1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff654104b10 0x7ff654197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.726+0000 7ff6594f1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff654107440 0x7ff65419f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.726+0000 7ff6594f1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff65410d3b0 con 0x7ff654104b10 2024-03-24T00:18:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.726+0000 7ff6594f1640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff65410d230 con 0x7ff6541021e0 2024-03-24T00:18:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.726+0000 7ff6594f1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff65410d530 con 0x7ff654107440 2024-03-24T00:18:35.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.726+0000 7ff6527fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff654104b10 0x7ff654197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.726+0000 7ff6527fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff654104b10 0x7ff654197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33212/0 (socket says 172.21.15.78:33212) 2024-03-24T00:18:35.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.726+0000 7ff6527fc640 1 -- 172.21.15.78:0/3792269979 learned_addr learned my addr 172.21.15.78:0/3792269979 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:35.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.726+0000 7ff6537fe640 1 --2- 172.21.15.78:0/3792269979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff654107440 0x7ff65419f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.726+0000 7ff652ffd640 1 --2- 172.21.15.78:0/3792269979 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff6541021e0 0x7ff654197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.727+0000 7ff6527fc640 1 -- 172.21.15.78:0/3792269979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff654107440 msgr2=0x7ff65419f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:35.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.727+0000 7ff6527fc640 1 --2- 172.21.15.78:0/3792269979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff654107440 0x7ff65419f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.727+0000 7ff6527fc640 1 -- 172.21.15.78:0/3792269979 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff6541021e0 msgr2=0x7ff654197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:35.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.727+0000 7ff6527fc640 1 --2- 172.21.15.78:0/3792269979 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff6541021e0 0x7ff654197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.727+0000 7ff6527fc640 1 -- 172.21.15.78:0/3792269979 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff65419f670 con 0x7ff654104b10 2024-03-24T00:18:35.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.727+0000 7ff6537fe640 1 --2- 172.21.15.78:0/3792269979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff654107440 0x7ff65419f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:18:35.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.727+0000 7ff6527fc640 1 --2- 172.21.15.78:0/3792269979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff654104b10 0x7ff654197f90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff648005fd0 tx=0x7ff64800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:35.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.727+0000 7ff63bfff640 1 -- 172.21.15.78:0/3792269979 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff648017070 con 0x7ff654104b10 2024-03-24T00:18:35.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.727+0000 7ff6594f1640 1 -- 172.21.15.78:0/3792269979 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff65419f960 con 0x7ff654104b10 2024-03-24T00:18:35.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.727+0000 7ff63bfff640 1 -- 172.21.15.78:0/3792269979 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff648009c30 con 0x7ff654104b10 2024-03-24T00:18:35.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.727+0000 7ff63bfff640 1 -- 172.21.15.78:0/3792269979 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff648010460 con 0x7ff654104b10 2024-03-24T00:18:35.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.727+0000 7ff6594f1640 1 -- 172.21.15.78:0/3792269979 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff65419feb0 con 0x7ff654104b10 2024-03-24T00:18:35.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.728+0000 7ff6594f1640 1 -- 172.21.15.78:0/3792269979 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6540fd620 con 0x7ff654104b10 2024-03-24T00:18:35.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.729+0000 7ff63bfff640 1 -- 172.21.15.78:0/3792269979 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff648003e70 con 0x7ff654104b10 2024-03-24T00:18:35.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.730+0000 7ff63bfff640 1 --2- 172.21.15.78:0/3792269979 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff63c0a2800 0x7ff63c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.730+0000 7ff652ffd640 1 --2- 172.21.15.78:0/3792269979 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff63c0a2800 0x7ff63c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.730+0000 7ff63bfff640 1 -- 172.21.15.78:0/3792269979 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff6480cea50 con 0x7ff654104b10 2024-03-24T00:18:35.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.730+0000 7ff652ffd640 1 --2- 172.21.15.78:0/3792269979 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff63c0a2800 0x7ff63c0a4cc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff644006010 tx=0x7ff644032590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:35.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.733+0000 7ff63bfff640 1 -- 172.21.15.78:0/3792269979 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff648020030 con 0x7ff654104b10 2024-03-24T00:18:35.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:35 smithi107 ceph-mon[36485]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:35.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:35 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:35.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:35 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:35.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:35 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:35.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:35 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:18:35.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:35 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:18:35.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:35 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:35.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.946+0000 7ff6594f1640 1 -- 172.21.15.78:0/3792269979 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff65402d070 con 0x7ff654104b10 2024-03-24T00:18:35.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.947+0000 7ff63bfff640 1 -- 172.21.15.78:0/3792269979 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9254 (secure 0 0 0) 0x7ff64809bf90 con 0x7ff654104b10 2024-03-24T00:18:35.947 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:18:35.947 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1416685123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1416685123},{"type":"v1","addr":"172.21.15.78:6835","nonce":1416685123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1071274623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1071274623},{"type":"v1","addr":"172.21.15.78:6837","nonce":1071274623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2758357551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2758357551},{"type":"v1","addr":"172.21.15.78:6839","nonce":2758357551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/3424505910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":3424505910},{"type":"v1","addr":"172.21.15.78:6841","nonce":3424505910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/3086318443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3086318443},{"type":"v1","addr":"172.21.15.107:6833","nonce":3086318443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/2367018868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2367018868},{"type":"v1","addr":"172.21.15.106:6833","nonce":2367018868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/2231611477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":2231611477},{"type":"v1","addr":"172.21.15.106:6835","nonce":2231611477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/108936439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":108936439},{"type":"v1","addr":"172.21.15.107:6835","nonce":108936439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2939571540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2939571540},{"type":"v1","addr":"172.21.15.107:6837","nonce":2939571540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/2090968633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":2090968633},{"type":"v1","addr":"172.21.15.106:6837","nonce":2090968633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/117119462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":117119462},{"type":"v1","addr":"172.21.15.107:6839","nonce":117119462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-24T00:18:35.947 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 12 2024-03-24T00:18:35.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.949+0000 7ff6594f1640 1 -- 172.21.15.78:0/3792269979 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff63c0a2800 msgr2=0x7ff63c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:35.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.949+0000 7ff6594f1640 1 --2- 172.21.15.78:0/3792269979 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff63c0a2800 0x7ff63c0a4cc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff644006010 tx=0x7ff644032590 comp rx=0 tx=0).stop 2024-03-24T00:18:35.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.949+0000 7ff6594f1640 1 -- 172.21.15.78:0/3792269979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff654104b10 msgr2=0x7ff654197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:35.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.949+0000 7ff6594f1640 1 --2- 172.21.15.78:0/3792269979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff654104b10 0x7ff654197f90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff648005fd0 tx=0x7ff64800dde0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.949+0000 7ff6594f1640 1 -- 172.21.15.78:0/3792269979 shutdown_connections 2024-03-24T00:18:35.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.949+0000 7ff6594f1640 1 --2- 172.21.15.78:0/3792269979 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff63c0a2800 0x7ff63c0a4cc0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.949+0000 7ff6594f1640 1 --2- 172.21.15.78:0/3792269979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff654107440 0x7ff65419f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.949+0000 7ff6594f1640 1 --2- 172.21.15.78:0/3792269979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff654104b10 0x7ff654197f90 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.949+0000 7ff6594f1640 1 --2- 172.21.15.78:0/3792269979 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff6541021e0 0x7ff654197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.949+0000 7ff6594f1640 1 -- 172.21.15.78:0/3792269979 >> 172.21.15.78:0/3792269979 conn(0x7ff6540fbb30 msgr2=0x7ff654102b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:35.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.949+0000 7ff6594f1640 1 -- 172.21.15.78:0/3792269979 shutdown_connections 2024-03-24T00:18:35.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:35.949+0000 7ff6594f1640 1 -- 172.21.15.78:0/3792269979 wait complete. 2024-03-24T00:18:36.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:36 smithi106 ceph-mon[36505]: Deploying daemon mds.b on smithi106 2024-03-24T00:18:36.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:36 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6840/3424505910,v1:172.21.15.78:6841/3424505910] up:boot 2024-03-24T00:18:36.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:36 smithi106 ceph-mon[36505]: fsmap 11 up:standby 2024-03-24T00:18:36.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:36 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T00:18:36.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:36 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3792269979' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:36.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:36 smithi078 ceph-mon[32429]: Deploying daemon mds.b on smithi106 2024-03-24T00:18:36.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:36 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6840/3424505910,v1:172.21.15.78:6841/3424505910] up:boot 2024-03-24T00:18:36.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:36 smithi078 ceph-mon[32429]: fsmap 11 up:standby 2024-03-24T00:18:36.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:36 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T00:18:36.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:36 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3792269979' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:36 smithi107 ceph-mon[36485]: Deploying daemon mds.b on smithi106 2024-03-24T00:18:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:36 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6840/3424505910,v1:172.21.15.78:6841/3424505910] up:boot 2024-03-24T00:18:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:36 smithi107 ceph-mon[36485]: fsmap 11 up:standby 2024-03-24T00:18:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:36 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T00:18:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:36 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3792269979' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:37 smithi106 ceph-mon[36505]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:37.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:37 smithi078 ceph-mon[32429]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:37.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:37 smithi107 ceph-mon[36485]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:37.961 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:18:38.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.081+0000 7f3551e36640 1 -- 172.21.15.78:0/3734596391 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f354c102170 msgr2=0x7f354c108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:38.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.081+0000 7f3551e36640 1 --2- 172.21.15.78:0/3734596391 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f354c102170 0x7f354c108a00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3544009ca0 tx=0x7f354402c0f0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.081+0000 7f3551e36640 1 -- 172.21.15.78:0/3734596391 shutdown_connections 2024-03-24T00:18:38.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.081+0000 7f3551e36640 1 --2- 172.21.15.78:0/3734596391 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f354c102170 0x7f354c108a00 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.081+0000 7f3551e36640 1 --2- 172.21.15.78:0/3734596391 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f354c1017b0 0x7f354c101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.081+0000 7f3551e36640 1 --2- 172.21.15.78:0/3734596391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f354c1005c0 0x7f354c1009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.081+0000 7f3551e36640 1 -- 172.21.15.78:0/3734596391 >> 172.21.15.78:0/3734596391 conn(0x7f354c0fbb30 msgr2=0x7f354c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:38.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.081+0000 7f3551e36640 1 -- 172.21.15.78:0/3734596391 shutdown_connections 2024-03-24T00:18:38.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.082+0000 7f3551e36640 1 -- 172.21.15.78:0/3734596391 wait complete. 2024-03-24T00:18:38.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.082+0000 7f3551e36640 1 Processor -- start 2024-03-24T00:18:38.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.082+0000 7f3551e36640 1 -- start start 2024-03-24T00:18:38.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.082+0000 7f3551e36640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f354c1005c0 0x7f354c1979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:38.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.082+0000 7f3551e36640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f354c1017b0 0x7f354c197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:38.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.082+0000 7f3551e36640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f354c102170 0x7f354c19ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:38.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.082+0000 7f3551e36640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f354c10c220 con 0x7f354c1017b0 2024-03-24T00:18:38.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.082+0000 7f3551e36640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f354c10c0a0 con 0x7f354c102170 2024-03-24T00:18:38.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.082+0000 7f3551e36640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f354c10c3a0 con 0x7f354c1005c0 2024-03-24T00:18:38.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.083+0000 7f354affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f354c1017b0 0x7f354c197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:38.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.083+0000 7f354affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f354c1017b0 0x7f354c197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33238/0 (socket says 172.21.15.78:33238) 2024-03-24T00:18:38.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.083+0000 7f354affd640 1 -- 172.21.15.78:0/476929793 learned_addr learned my addr 172.21.15.78:0/476929793 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:38.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.083+0000 7f354bfff640 1 --2- 172.21.15.78:0/476929793 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f354c102170 0x7f354c19ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:38.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.083+0000 7f354b7fe640 1 --2- 172.21.15.78:0/476929793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f354c1005c0 0x7f354c1979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:38.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.083+0000 7f354affd640 1 -- 172.21.15.78:0/476929793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f354c1005c0 msgr2=0x7f354c1979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:38.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.083+0000 7f354affd640 1 --2- 172.21.15.78:0/476929793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f354c1005c0 0x7f354c1979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.083+0000 7f354affd640 1 -- 172.21.15.78:0/476929793 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f354c102170 msgr2=0x7f354c19ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:38.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.083+0000 7f354affd640 1 --2- 172.21.15.78:0/476929793 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f354c102170 0x7f354c19ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.083+0000 7f354affd640 1 -- 172.21.15.78:0/476929793 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f354c19f5d0 con 0x7f354c1017b0 2024-03-24T00:18:38.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.083+0000 7f354bfff640 1 --2- 172.21.15.78:0/476929793 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f354c102170 0x7f354c19ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:38.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.083+0000 7f354affd640 1 --2- 172.21.15.78:0/476929793 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f354c1017b0 0x7f354c197ef0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f354000e9f0 tx=0x7f354000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:38.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.083+0000 7f3548ff9640 1 -- 172.21.15.78:0/476929793 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354000cd80 con 0x7f354c1017b0 2024-03-24T00:18:38.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.083+0000 7f3548ff9640 1 -- 172.21.15.78:0/476929793 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3540010910 con 0x7f354c1017b0 2024-03-24T00:18:38.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.083+0000 7f3548ff9640 1 -- 172.21.15.78:0/476929793 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3540018990 con 0x7f354c1017b0 2024-03-24T00:18:38.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.083+0000 7f3551e36640 1 -- 172.21.15.78:0/476929793 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f354c19f8c0 con 0x7f354c1017b0 2024-03-24T00:18:38.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.083+0000 7f3551e36640 1 -- 172.21.15.78:0/476929793 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f354c19fe10 con 0x7f354c1017b0 2024-03-24T00:18:38.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.085+0000 7f3551e36640 1 -- 172.21.15.78:0/476929793 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f354c1a00a0 con 0x7f354c1017b0 2024-03-24T00:18:38.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.086+0000 7f3548ff9640 1 -- 172.21.15.78:0/476929793 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3540010ac0 con 0x7f354c1017b0 2024-03-24T00:18:38.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.086+0000 7f3548ff9640 1 --2- 172.21.15.78:0/476929793 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f35340a2800 0x7f35340a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:38.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.087+0000 7f354b7fe640 1 --2- 172.21.15.78:0/476929793 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f35340a2800 0x7f35340a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:38.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.087+0000 7f3548ff9640 1 -- 172.21.15.78:0/476929793 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f35400ce9c0 con 0x7f354c1017b0 2024-03-24T00:18:38.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.087+0000 7f354b7fe640 1 --2- 172.21.15.78:0/476929793 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f35340a2800 0x7f35340a4cc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f353c005fd0 tx=0x7f353c005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:38.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.092+0000 7f3548ff9640 1 -- 172.21.15.78:0/476929793 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f35400d0e20 con 0x7f354c1017b0 2024-03-24T00:18:38.152 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 24 00:18:37 smithi106 systemd[1]: Started Ceph mds.b for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:18:38.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.312+0000 7f3551e36640 1 -- 172.21.15.78:0/476929793 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f354c1a0270 con 0x7f354c1017b0 2024-03-24T00:18:38.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.313+0000 7f3548ff9640 1 -- 172.21.15.78:0/476929793 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9254 (secure 0 0 0) 0x7f354009bf00 con 0x7f354c1017b0 2024-03-24T00:18:38.312 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:18:38.313 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1416685123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1416685123},{"type":"v1","addr":"172.21.15.78:6835","nonce":1416685123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1071274623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1071274623},{"type":"v1","addr":"172.21.15.78:6837","nonce":1071274623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2758357551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2758357551},{"type":"v1","addr":"172.21.15.78:6839","nonce":2758357551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/3424505910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":3424505910},{"type":"v1","addr":"172.21.15.78:6841","nonce":3424505910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/3086318443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3086318443},{"type":"v1","addr":"172.21.15.107:6833","nonce":3086318443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/2367018868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2367018868},{"type":"v1","addr":"172.21.15.106:6833","nonce":2367018868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/2231611477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":2231611477},{"type":"v1","addr":"172.21.15.106:6835","nonce":2231611477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/108936439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":108936439},{"type":"v1","addr":"172.21.15.107:6835","nonce":108936439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2939571540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2939571540},{"type":"v1","addr":"172.21.15.107:6837","nonce":2939571540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/2090968633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":2090968633},{"type":"v1","addr":"172.21.15.106:6837","nonce":2090968633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/117119462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":117119462},{"type":"v1","addr":"172.21.15.107:6839","nonce":117119462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-24T00:18:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 12 2024-03-24T00:18:38.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.315+0000 7f3551e36640 1 -- 172.21.15.78:0/476929793 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f35340a2800 msgr2=0x7f35340a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:38.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.315+0000 7f3551e36640 1 --2- 172.21.15.78:0/476929793 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f35340a2800 0x7f35340a4cc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f353c005fd0 tx=0x7f353c005850 comp rx=0 tx=0).stop 2024-03-24T00:18:38.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.315+0000 7f3551e36640 1 -- 172.21.15.78:0/476929793 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f354c1017b0 msgr2=0x7f354c197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:38.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.315+0000 7f3551e36640 1 --2- 172.21.15.78:0/476929793 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f354c1017b0 0x7f354c197ef0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f354000e9f0 tx=0x7f354000eeb0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.315+0000 7f3551e36640 1 -- 172.21.15.78:0/476929793 shutdown_connections 2024-03-24T00:18:38.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.315+0000 7f3551e36640 1 --2- 172.21.15.78:0/476929793 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f35340a2800 0x7f35340a4cc0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.315+0000 7f3551e36640 1 --2- 172.21.15.78:0/476929793 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f354c102170 0x7f354c19ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.315+0000 7f3551e36640 1 --2- 172.21.15.78:0/476929793 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f354c1017b0 0x7f354c197ef0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.315+0000 7f3551e36640 1 --2- 172.21.15.78:0/476929793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f354c1005c0 0x7f354c1979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.315+0000 7f3551e36640 1 -- 172.21.15.78:0/476929793 >> 172.21.15.78:0/476929793 conn(0x7f354c0fbb30 msgr2=0x7f354c0fd5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:38.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.315+0000 7f3551e36640 1 -- 172.21.15.78:0/476929793 shutdown_connections 2024-03-24T00:18:38.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:38.315+0000 7f3551e36640 1 -- 172.21.15.78:0/476929793 wait complete. 2024-03-24T00:18:39.246 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:38 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:39.246 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:38 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:39.246 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:38 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:39.246 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:38 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:39.246 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:38 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:39.246 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:38 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:18:39.247 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:38 smithi078 ceph-mon[32429]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:39.247 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:38 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/476929793' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:39.247 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:38 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] up:boot 2024-03-24T00:18:39.247 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:38 smithi078 ceph-mon[32429]: fsmap 12 up:standby 2024-03-24T00:18:39.247 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:38 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T00:18:39.250 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:38 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:39.250 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:38 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:39.250 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:38 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:39.250 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:38 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:39.250 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:38 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:39.250 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:38 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:18:39.250 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:38 smithi106 ceph-mon[36505]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:39.250 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:38 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/476929793' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:39.250 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:38 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] up:boot 2024-03-24T00:18:39.250 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:38 smithi106 ceph-mon[36505]: fsmap 12 up:standby 2024-03-24T00:18:39.250 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:38 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T00:18:39.290 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:38 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:39.291 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:38 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:39.291 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:38 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:39.291 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:38 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:39.291 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:38 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:39.291 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:38 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:18:39.291 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:38 smithi107 ceph-mon[36485]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:39.291 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:38 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/476929793' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:39.291 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:38 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] up:boot 2024-03-24T00:18:39.291 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:38 smithi107 ceph-mon[36485]: fsmap 12 up:standby 2024-03-24T00:18:39.291 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:38 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T00:18:40.328 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:18:40.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.432+0000 7fd33ab53640 1 -- 172.21.15.78:0/3807822673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3341017b0 msgr2=0x7fd334101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:40.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.432+0000 7fd33ab53640 1 --2- 172.21.15.78:0/3807822673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3341017b0 0x7fd334101c30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd320009900 tx=0x7fd32002bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:40.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.432+0000 7fd33ab53640 1 -- 172.21.15.78:0/3807822673 shutdown_connections 2024-03-24T00:18:40.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.432+0000 7fd33ab53640 1 --2- 172.21.15.78:0/3807822673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd334102170 0x7fd334108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:40.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.432+0000 7fd33ab53640 1 --2- 172.21.15.78:0/3807822673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3341017b0 0x7fd334101c30 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:40.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.432+0000 7fd33ab53640 1 --2- 172.21.15.78:0/3807822673 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3341005c0 0x7fd3341009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:40.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.432+0000 7fd33ab53640 1 -- 172.21.15.78:0/3807822673 >> 172.21.15.78:0/3807822673 conn(0x7fd3340fbb30 msgr2=0x7fd3340fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:40.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.432+0000 7fd33ab53640 1 -- 172.21.15.78:0/3807822673 shutdown_connections 2024-03-24T00:18:40.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.432+0000 7fd33ab53640 1 -- 172.21.15.78:0/3807822673 wait complete. 2024-03-24T00:18:40.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.433+0000 7fd33ab53640 1 Processor -- start 2024-03-24T00:18:40.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.433+0000 7fd33ab53640 1 -- start start 2024-03-24T00:18:40.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.433+0000 7fd33ab53640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3341005c0 0x7fd33406a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.433+0000 7fd33ab53640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3341017b0 0x7fd33419c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.433+0000 7fd33ab53640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd334102170 0x7fd33419edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.433+0000 7fd33ab53640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33410c2b0 con 0x7fd3341005c0 2024-03-24T00:18:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.433+0000 7fd33ab53640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33410c130 con 0x7fd3341017b0 2024-03-24T00:18:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.433+0000 7fd33ab53640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33410c430 con 0x7fd334102170 2024-03-24T00:18:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.433+0000 7fd3388ef640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3341005c0 0x7fd33406a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.433+0000 7fd3388ef640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3341005c0 0x7fd33406a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33262/0 (socket says 172.21.15.78:33262) 2024-03-24T00:18:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.433+0000 7fd3388ef640 1 -- 172.21.15.78:0/3492782920 learned_addr learned my addr 172.21.15.78:0/3492782920 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.433+0000 7fd3390f0640 1 --2- 172.21.15.78:0/3492782920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd334102170 0x7fd33419edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.433+0000 7fd3388ef640 1 -- 172.21.15.78:0/3492782920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd334102170 msgr2=0x7fd33419edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.433+0000 7fd3388ef640 1 --2- 172.21.15.78:0/3492782920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd334102170 0x7fd33419edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.433+0000 7fd3388ef640 1 -- 172.21.15.78:0/3492782920 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3341017b0 msgr2=0x7fd33419c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.433+0000 7fd333fff640 1 --2- 172.21.15.78:0/3492782920 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3341017b0 0x7fd33419c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.433+0000 7fd3388ef640 1 --2- 172.21.15.78:0/3492782920 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3341017b0 0x7fd33419c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.433+0000 7fd3388ef640 1 -- 172.21.15.78:0/3492782920 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33419f6d0 con 0x7fd3341005c0 2024-03-24T00:18:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.433+0000 7fd333fff640 1 --2- 172.21.15.78:0/3492782920 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3341017b0 0x7fd33419c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:18:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.433+0000 7fd3388ef640 1 --2- 172.21.15.78:0/3492782920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3341005c0 0x7fd33406a880 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd32800b450 tx=0x7fd32800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.434+0000 7fd331ffb640 1 -- 172.21.15.78:0/3492782920 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd328015020 con 0x7fd3341005c0 2024-03-24T00:18:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.434+0000 7fd331ffb640 1 -- 172.21.15.78:0/3492782920 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd328011d70 con 0x7fd3341005c0 2024-03-24T00:18:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.434+0000 7fd33ab53640 1 -- 172.21.15.78:0/3492782920 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd33419f960 con 0x7fd3341005c0 2024-03-24T00:18:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.434+0000 7fd331ffb640 1 -- 172.21.15.78:0/3492782920 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32800f340 con 0x7fd3341005c0 2024-03-24T00:18:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.434+0000 7fd33ab53640 1 -- 172.21.15.78:0/3492782920 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd33419feb0 con 0x7fd3341005c0 2024-03-24T00:18:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.435+0000 7fd33ab53640 1 -- 172.21.15.78:0/3492782920 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3341078e0 con 0x7fd3341005c0 2024-03-24T00:18:40.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.436+0000 7fd331ffb640 1 -- 172.21.15.78:0/3492782920 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd328018020 con 0x7fd3341005c0 2024-03-24T00:18:40.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.436+0000 7fd331ffb640 1 --2- 172.21.15.78:0/3492782920 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd3240a2800 0x7fd3240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:40.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.436+0000 7fd333fff640 1 --2- 172.21.15.78:0/3492782920 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd3240a2800 0x7fd3240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:40.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.436+0000 7fd331ffb640 1 -- 172.21.15.78:0/3492782920 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd3280ce020 con 0x7fd3341005c0 2024-03-24T00:18:40.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.437+0000 7fd333fff640 1 --2- 172.21.15.78:0/3492782920 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd3240a2800 0x7fd3240a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd320036040 tx=0x7fd320000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:40.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.441+0000 7fd331ffb640 1 -- 172.21.15.78:0/3492782920 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd32801f020 con 0x7fd3341005c0 2024-03-24T00:18:40.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.668+0000 7fd33ab53640 1 -- 172.21.15.78:0/3492782920 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd33402d220 con 0x7fd3341005c0 2024-03-24T00:18:40.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.669+0000 7fd331ffb640 1 -- 172.21.15.78:0/3492782920 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+10049 (secure 0 0 0) 0x7fd32800d040 con 0x7fd3341005c0 2024-03-24T00:18:40.669 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:18:40.669 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1416685123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1416685123},{"type":"v1","addr":"172.21.15.78:6835","nonce":1416685123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1071274623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1071274623},{"type":"v1","addr":"172.21.15.78:6837","nonce":1071274623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2758357551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2758357551},{"type":"v1","addr":"172.21.15.78:6839","nonce":2758357551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/3424505910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":3424505910},{"type":"v1","addr":"172.21.15.78:6841","nonce":3424505910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/3086318443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3086318443},{"type":"v1","addr":"172.21.15.107:6833","nonce":3086318443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/2367018868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2367018868},{"type":"v1","addr":"172.21.15.106:6833","nonce":2367018868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/2231611477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":2231611477},{"type":"v1","addr":"172.21.15.106:6835","nonce":2231611477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/108936439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":108936439},{"type":"v1","addr":"172.21.15.107:6835","nonce":108936439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2939571540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2939571540},{"type":"v1","addr":"172.21.15.107:6837","nonce":2939571540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/2090968633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":2090968633},{"type":"v1","addr":"172.21.15.106:6837","nonce":2090968633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/117119462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":117119462},{"type":"v1","addr":"172.21.15.107:6839","nonce":117119462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24500,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6839/2482071304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2482071304},{"type":"v1","addr":"172.21.15.106:6839","nonce":2482071304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[]} 2024-03-24T00:18:40.669 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 13 2024-03-24T00:18:40.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.671+0000 7fd33ab53640 1 -- 172.21.15.78:0/3492782920 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd3240a2800 msgr2=0x7fd3240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:40.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.671+0000 7fd33ab53640 1 --2- 172.21.15.78:0/3492782920 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd3240a2800 0x7fd3240a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd320036040 tx=0x7fd320000bc0 comp rx=0 tx=0).stop 2024-03-24T00:18:40.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.671+0000 7fd33ab53640 1 -- 172.21.15.78:0/3492782920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3341005c0 msgr2=0x7fd33406a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:40.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.671+0000 7fd33ab53640 1 --2- 172.21.15.78:0/3492782920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3341005c0 0x7fd33406a880 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd32800b450 tx=0x7fd32800b910 comp rx=0 tx=0).stop 2024-03-24T00:18:40.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.671+0000 7fd33ab53640 1 -- 172.21.15.78:0/3492782920 shutdown_connections 2024-03-24T00:18:40.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.671+0000 7fd33ab53640 1 --2- 172.21.15.78:0/3492782920 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd3240a2800 0x7fd3240a4cc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:40.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.671+0000 7fd33ab53640 1 --2- 172.21.15.78:0/3492782920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd334102170 0x7fd33419edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:40.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.671+0000 7fd33ab53640 1 --2- 172.21.15.78:0/3492782920 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3341017b0 0x7fd33419c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:40.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.671+0000 7fd33ab53640 1 --2- 172.21.15.78:0/3492782920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3341005c0 0x7fd33406a880 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:40.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.671+0000 7fd33ab53640 1 -- 172.21.15.78:0/3492782920 >> 172.21.15.78:0/3492782920 conn(0x7fd3340fbb30 msgr2=0x7fd334104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:40.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.671+0000 7fd33ab53640 1 -- 172.21.15.78:0/3492782920 shutdown_connections 2024-03-24T00:18:40.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.671+0000 7fd33ab53640 1 -- 172.21.15.78:0/3492782920 wait complete. 2024-03-24T00:18:40.683 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:18:40.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.787+0000 7f5f62937640 1 -- 172.21.15.78:0/1036662918 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f5c1028c0 msgr2=0x7f5f5c102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:40.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.787+0000 7f5f62937640 1 --2- 172.21.15.78:0/1036662918 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f5c1028c0 0x7f5f5c102d40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5f50009900 tx=0x7f5f5002bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:40.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.787+0000 7f5f62937640 1 -- 172.21.15.78:0/1036662918 shutdown_connections 2024-03-24T00:18:40.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.787+0000 7f5f62937640 1 --2- 172.21.15.78:0/1036662918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f5c103280 0x7f5f5c109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:40.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.787+0000 7f5f62937640 1 --2- 172.21.15.78:0/1036662918 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f5c1028c0 0x7f5f5c102d40 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:40.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.787+0000 7f5f62937640 1 --2- 172.21.15.78:0/1036662918 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f5c1016d0 0x7f5f5c101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:40.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.787+0000 7f5f62937640 1 -- 172.21.15.78:0/1036662918 >> 172.21.15.78:0/1036662918 conn(0x7f5f5c0fcc40 msgr2=0x7f5f5c0ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:40.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.787+0000 7f5f62937640 1 -- 172.21.15.78:0/1036662918 shutdown_connections 2024-03-24T00:18:40.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.787+0000 7f5f62937640 1 -- 172.21.15.78:0/1036662918 wait complete. 2024-03-24T00:18:40.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.787+0000 7f5f62937640 1 Processor -- start 2024-03-24T00:18:40.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.788+0000 7f5f62937640 1 -- start start 2024-03-24T00:18:40.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.788+0000 7f5f62937640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f5c1016d0 0x7f5f5c197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:40.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.788+0000 7f5f62937640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f5c1028c0 0x7f5f5c197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:40.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.788+0000 7f5f62937640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f5c103280 0x7f5f5c19efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:40.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.788+0000 7f5f62937640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5c10d330 con 0x7f5f5c1016d0 2024-03-24T00:18:40.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.788+0000 7f5f62937640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5c10d1b0 con 0x7f5f5c1028c0 2024-03-24T00:18:40.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.788+0000 7f5f62937640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5c10d4b0 con 0x7f5f5c103280 2024-03-24T00:18:40.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.788+0000 7f5f5bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f5c1016d0 0x7f5f5c197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:40.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.788+0000 7f5f5bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f5c1016d0 0x7f5f5c197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33288/0 (socket says 172.21.15.78:33288) 2024-03-24T00:18:40.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.788+0000 7f5f5bfff640 1 -- 172.21.15.78:0/1876462742 learned_addr learned my addr 172.21.15.78:0/1876462742 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:40.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.788+0000 7f5f5b7fe640 1 --2- 172.21.15.78:0/1876462742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f5c1028c0 0x7f5f5c197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:40.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.788+0000 7f5f60ed4640 1 --2- 172.21.15.78:0/1876462742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f5c103280 0x7f5f5c19efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:40.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.788+0000 7f5f5bfff640 1 -- 172.21.15.78:0/1876462742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f5c103280 msgr2=0x7f5f5c19efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:40.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.788+0000 7f5f5bfff640 1 --2- 172.21.15.78:0/1876462742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f5c103280 0x7f5f5c19efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:40.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.788+0000 7f5f5bfff640 1 -- 172.21.15.78:0/1876462742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f5c1028c0 msgr2=0x7f5f5c197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:40.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.788+0000 7f5f5bfff640 1 --2- 172.21.15.78:0/1876462742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f5c1028c0 0x7f5f5c197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:40.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.788+0000 7f5f5bfff640 1 -- 172.21.15.78:0/1876462742 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f5c19f620 con 0x7f5f5c1016d0 2024-03-24T00:18:40.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.788+0000 7f5f5bfff640 1 --2- 172.21.15.78:0/1876462742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f5c1016d0 0x7f5f5c197a00 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c00d770 tx=0x7f5f4c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:40.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.788+0000 7f5f597fa640 1 -- 172.21.15.78:0/1876462742 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c017070 con 0x7f5f5c1016d0 2024-03-24T00:18:40.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.789+0000 7f5f62937640 1 -- 172.21.15.78:0/1876462742 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f5c19f910 con 0x7f5f5c1016d0 2024-03-24T00:18:40.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.789+0000 7f5f597fa640 1 -- 172.21.15.78:0/1876462742 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f4c009c30 con 0x7f5f5c1016d0 2024-03-24T00:18:40.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.789+0000 7f5f597fa640 1 -- 172.21.15.78:0/1876462742 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c003dd0 con 0x7f5f5c1016d0 2024-03-24T00:18:40.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.789+0000 7f5f62937640 1 -- 172.21.15.78:0/1876462742 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f5c19fe30 con 0x7f5f5c1016d0 2024-03-24T00:18:40.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.790+0000 7f5f62937640 1 -- 172.21.15.78:0/1876462742 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f5c108a40 con 0x7f5f5c1016d0 2024-03-24T00:18:40.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.791+0000 7f5f597fa640 1 -- 172.21.15.78:0/1876462742 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5f4c003fb0 con 0x7f5f5c1016d0 2024-03-24T00:18:40.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.791+0000 7f5f597fa640 1 --2- 172.21.15.78:0/1876462742 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f5f440a2800 0x7f5f440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:40.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.791+0000 7f5f5b7fe640 1 --2- 172.21.15.78:0/1876462742 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f5f440a2800 0x7f5f440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:40.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.791+0000 7f5f597fa640 1 -- 172.21.15.78:0/1876462742 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f5f4c0cde90 con 0x7f5f5c1016d0 2024-03-24T00:18:40.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.792+0000 7f5f5b7fe640 1 --2- 172.21.15.78:0/1876462742 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f5f440a2800 0x7f5f440a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5f5002c7b0 tx=0x7f5f50005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:40.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.796+0000 7f5f597fa640 1 -- 172.21.15.78:0/1876462742 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5f4c009de0 con 0x7f5f5c1016d0 2024-03-24T00:18:40.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.981+0000 7f5f62937640 1 -- 172.21.15.78:0/1876462742 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f5c0622a0 con 0x7f5f5c1016d0 2024-03-24T00:18:40.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.982+0000 7f5f597fa640 1 -- 172.21.15.78:0/1876462742 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18457 (secure 0 0 0) 0x7f5f4c09ad80 con 0x7f5f5c1016d0 2024-03-24T00:18:40.982 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:18:40.982 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":81,"fsid":"c7e2806a-e972-11ee-95cd-87774f69a715","created":"2024-03-24T00:10:44.134563+0000","modified":"2024-03-24T00:17:51.449181+0000","last_up_change":"2024-03-24T00:17:50.237686+0000","last_in_change":"2024-03-24T00:17:29.788699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:14:04.534391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b950388a-c437-426b-b703-3f8ae821dd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6803","nonce":1816879397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6805","nonce":1816879397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6809","nonce":1816879397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6807","nonce":1816879397}]},"public_addr":"172.21.15.78:6803/1816879397","cluster_addr":"172.21.15.78:6805/1816879397","heartbeat_back_addr":"172.21.15.78:6809/1816879397","heartbeat_front_addr":"172.21.15.78:6807/1816879397","state":["exists","up"]},{"osd":1,"uuid":"b8d64714-dd65-400f-b3f4-673ab14e1365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6801","nonce":1647915612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6803","nonce":1647915612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6807","nonce":1647915612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6805","nonce":1647915612}]},"public_addr":"172.21.15.106:6801/1647915612","cluster_addr":"172.21.15.106:6803/1647915612","heartbeat_back_addr":"172.21.15.106:6807/1647915612","heartbeat_front_addr":"172.21.15.106:6805/1647915612","state":["exists","up"]},{"osd":2,"uuid":"fcfc4dd1-aaf8-4131-a023-5e832212158d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6801","nonce":2010770435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6803","nonce":2010770435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6807","nonce":2010770435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6805","nonce":2010770435}]},"public_addr":"172.21.15.107:6801/2010770435","cluster_addr":"172.21.15.107:6803/2010770435","heartbeat_back_addr":"172.21.15.107:6807/2010770435","heartbeat_front_addr":"172.21.15.107:6805/2010770435","state":["exists","up"]},{"osd":3,"uuid":"47c718ab-a0f4-4e52-83c5-b89386b262d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6811","nonce":1912251346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6813","nonce":1912251346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6817","nonce":1912251346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6815","nonce":1912251346}]},"public_addr":"172.21.15.78:6811/1912251346","cluster_addr":"172.21.15.78:6813/1912251346","heartbeat_back_addr":"172.21.15.78:6817/1912251346","heartbeat_front_addr":"172.21.15.78:6815/1912251346","state":["exists","up"]},{"osd":4,"uuid":"7dad64dd-677e-4c1c-9638-e4f4f2195983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6809","nonce":374641205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6811","nonce":374641205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6815","nonce":374641205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6813","nonce":374641205}]},"public_addr":"172.21.15.106:6809/374641205","cluster_addr":"172.21.15.106:6811/374641205","heartbeat_back_addr":"172.21.15.106:6815/374641205","heartbeat_front_addr":"172.21.15.106:6813/374641205","state":["exists","up"]},{"osd":5,"uuid":"bbd524e5-328d-4309-8877-1149cbe110dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6809","nonce":3272806073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6811","nonce":3272806073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6815","nonce":3272806073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6813","nonce":3272806073}]},"public_addr":"172.21.15.107:6809/3272806073","cluster_addr":"172.21.15.107:6811/3272806073","heartbeat_back_addr":"172.21.15.107:6815/3272806073","heartbeat_front_addr":"172.21.15.107:6813/3272806073","state":["exists","up"]},{"osd":6,"uuid":"5696389a-918f-4ea8-b20d-4e2a0d4cb3bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6819","nonce":3091148508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6821","nonce":3091148508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6825","nonce":3091148508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6823","nonce":3091148508}]},"public_addr":"172.21.15.78:6819/3091148508","cluster_addr":"172.21.15.78:6821/3091148508","heartbeat_back_addr":"172.21.15.78:6825/3091148508","heartbeat_front_addr":"172.21.15.78:6823/3091148508","state":["exists","up"]},{"osd":7,"uuid":"f6b764e8-ed4e-4c9a-a85b-94af6dc6b924","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6817","nonce":4262103080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6819","nonce":4262103080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6823","nonce":4262103080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6821","nonce":4262103080}]},"public_addr":"172.21.15.106:6817/4262103080","cluster_addr":"172.21.15.106:6819/4262103080","heartbeat_back_addr":"172.21.15.106:6823/4262103080","heartbeat_front_addr":"172.21.15.106:6821/4262103080","state":["exists","up"]},{"osd":8,"uuid":"ba16039f-8093-4fa0-87ad-e28b7d142b23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6817","nonce":4032799356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6819","nonce":4032799356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6823","nonce":4032799356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6821","nonce":4032799356}]},"public_addr":"172.21.15.107:6817/4032799356","cluster_addr":"172.21.15.107:6819/4032799356","heartbeat_back_addr":"172.21.15.107:6823/4032799356","heartbeat_front_addr":"172.21.15.107:6821/4032799356","state":["exists","up"]},{"osd":9,"uuid":"651357f2-0669-42e6-aa91-1d5c02d00c5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6827","nonce":214023431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6829","nonce":214023431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6833","nonce":214023431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6831","nonce":214023431}]},"public_addr":"172.21.15.78:6827/214023431","cluster_addr":"172.21.15.78:6829/214023431","heartbeat_back_addr":"172.21.15.78:6833/214023431","heartbeat_front_addr":"172.21.15.78:6831/214023431","state":["exists","up"]},{"osd":10,"uuid":"2375119d-346e-426c-8a65-b64af71ad63a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6825","nonce":400029804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6827","nonce":400029804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6831","nonce":400029804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6829","nonce":400029804}]},"public_addr":"172.21.15.106:6825/400029804","cluster_addr":"172.21.15.106:6827/400029804","heartbeat_back_addr":"172.21.15.106:6831/400029804","heartbeat_front_addr":"172.21.15.106:6829/400029804","state":["exists","up"]},{"osd":11,"uuid":"b6b609d9-dd78-4c01-ae66-ba2a662ab22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6825","nonce":4191916230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6827","nonce":4191916230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6831","nonce":4191916230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6829","nonce":4191916230}]},"public_addr":"172.21.15.107:6825/4191916230","cluster_addr":"172.21.15.107:6827/4191916230","heartbeat_back_addr":"172.21.15.107:6831/4191916230","heartbeat_front_addr":"172.21.15.107:6829/4191916230","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:13.728602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:37.320809+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:01.269452+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:25.758281+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:50.298863+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:14.345219+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:38.980489+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:04.749214+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:29.992673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:56.269202+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:22.270316+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:48.074907+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/1982202541":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/1933054339":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6801/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6800/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/465691596":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6801/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/3962523609":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6800/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/1502752625":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/2236879401":"2024-03-25T00:11:40.199569+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:18:40.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.984+0000 7f5f62937640 1 -- 172.21.15.78:0/1876462742 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f5f440a2800 msgr2=0x7f5f440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:40.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.984+0000 7f5f62937640 1 --2- 172.21.15.78:0/1876462742 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f5f440a2800 0x7f5f440a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5f5002c7b0 tx=0x7f5f50005fd0 comp rx=0 tx=0).stop 2024-03-24T00:18:40.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.984+0000 7f5f62937640 1 -- 172.21.15.78:0/1876462742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f5c1016d0 msgr2=0x7f5f5c197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:40.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.984+0000 7f5f62937640 1 --2- 172.21.15.78:0/1876462742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f5c1016d0 0x7f5f5c197a00 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c00d770 tx=0x7f5f4c00dc30 comp rx=0 tx=0).stop 2024-03-24T00:18:40.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.984+0000 7f5f62937640 1 -- 172.21.15.78:0/1876462742 shutdown_connections 2024-03-24T00:18:40.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.984+0000 7f5f62937640 1 --2- 172.21.15.78:0/1876462742 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f5f440a2800 0x7f5f440a4cc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:40.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.984+0000 7f5f62937640 1 --2- 172.21.15.78:0/1876462742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f5c103280 0x7f5f5c19efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:40.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.984+0000 7f5f62937640 1 --2- 172.21.15.78:0/1876462742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f5c1028c0 0x7f5f5c197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:40.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.984+0000 7f5f62937640 1 --2- 172.21.15.78:0/1876462742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f5c1016d0 0x7f5f5c197a00 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:40.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.984+0000 7f5f62937640 1 -- 172.21.15.78:0/1876462742 >> 172.21.15.78:0/1876462742 conn(0x7f5f5c0fcc40 msgr2=0x7f5f5c105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:40.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.984+0000 7f5f62937640 1 -- 172.21.15.78:0/1876462742 shutdown_connections 2024-03-24T00:18:40.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:40.984+0000 7f5f62937640 1 -- 172.21.15.78:0/1876462742 wait complete. 2024-03-24T00:18:40.994 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:40.994 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:40.994 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:40.994 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:40.994 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:40.994 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:40.995 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:40 smithi078 ceph-mon[32429]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:40.995 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:40.995 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:40.995 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:18:40.995 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:40.995 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:18:40.995 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:40 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3492782920' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:40.996 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:14:04.534391+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:18:40.997 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:18:41.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.106+0000 7fbb168e8640 1 -- 172.21.15.78:0/429019082 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb101073c0 msgr2=0x7fbb10109810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:41.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.106+0000 7fbb168e8640 1 --2- 172.21.15.78:0/429019082 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb101073c0 0x7fbb10109810 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fbb08009ca0 tx=0x7fbb0802c0f0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.106+0000 7fbb168e8640 1 -- 172.21.15.78:0/429019082 shutdown_connections 2024-03-24T00:18:41.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.106+0000 7fbb168e8640 1 --2- 172.21.15.78:0/429019082 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb10104a90 0x7fbb10106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.106+0000 7fbb168e8640 1 --2- 172.21.15.78:0/429019082 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb101073c0 0x7fbb10109810 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.106+0000 7fbb168e8640 1 --2- 172.21.15.78:0/429019082 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb10102160 0x7fbb10104550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.106+0000 7fbb168e8640 1 -- 172.21.15.78:0/429019082 >> 172.21.15.78:0/429019082 conn(0x7fbb100fbb50 msgr2=0x7fbb100fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:41.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.106+0000 7fbb168e8640 1 -- 172.21.15.78:0/429019082 shutdown_connections 2024-03-24T00:18:41.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.107+0000 7fbb168e8640 1 -- 172.21.15.78:0/429019082 wait complete. 2024-03-24T00:18:41.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.107+0000 7fbb168e8640 1 Processor -- start 2024-03-24T00:18:41.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.107+0000 7fbb168e8640 1 -- start start 2024-03-24T00:18:41.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.107+0000 7fbb168e8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb10102160 0x7fbb10197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:41.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.107+0000 7fbb168e8640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb10104a90 0x7fbb10197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:41.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.107+0000 7fbb168e8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb101073c0 0x7fbb1019efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:41.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.107+0000 7fbb168e8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1010d380 con 0x7fbb101073c0 2024-03-24T00:18:41.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.107+0000 7fbb168e8640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1010d200 con 0x7fbb10104a90 2024-03-24T00:18:41.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.107+0000 7fbb168e8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1010d500 con 0x7fbb10102160 2024-03-24T00:18:41.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.107+0000 7fbb14e85640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb101073c0 0x7fbb1019efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:41.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.108+0000 7fbb0ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb10102160 0x7fbb10197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:41.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.108+0000 7fbb14e85640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb101073c0 0x7fbb1019efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33306/0 (socket says 172.21.15.78:33306) 2024-03-24T00:18:41.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.108+0000 7fbb0f7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb10104a90 0x7fbb10197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:41.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.108+0000 7fbb14e85640 1 -- 172.21.15.78:0/1970620054 learned_addr learned my addr 172.21.15.78:0/1970620054 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:41.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.108+0000 7fbb0ffff640 1 -- 172.21.15.78:0/1970620054 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb10104a90 msgr2=0x7fbb10197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:41.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.108+0000 7fbb0ffff640 1 --2- 172.21.15.78:0/1970620054 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb10104a90 0x7fbb10197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.108+0000 7fbb0ffff640 1 -- 172.21.15.78:0/1970620054 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb101073c0 msgr2=0x7fbb1019efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:41.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.108+0000 7fbb0ffff640 1 --2- 172.21.15.78:0/1970620054 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb101073c0 0x7fbb1019efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.108+0000 7fbb0ffff640 1 -- 172.21.15.78:0/1970620054 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb1019f640 con 0x7fbb10102160 2024-03-24T00:18:41.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.108+0000 7fbb0f7fe640 1 --2- 172.21.15.78:0/1970620054 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb10104a90 0x7fbb10197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:41.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.108+0000 7fbb0ffff640 1 --2- 172.21.15.78:0/1970620054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb10102160 0x7fbb10197a20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbb0000e9f0 tx=0x7fbb0000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:41.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.108+0000 7fbb0d7fa640 1 -- 172.21.15.78:0/1970620054 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0000ccc0 con 0x7fbb10102160 2024-03-24T00:18:41.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.108+0000 7fbb168e8640 1 -- 172.21.15.78:0/1970620054 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb1019f930 con 0x7fbb10102160 2024-03-24T00:18:41.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.108+0000 7fbb0d7fa640 1 -- 172.21.15.78:0/1970620054 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb00010910 con 0x7fbb10102160 2024-03-24T00:18:41.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.108+0000 7fbb0d7fa640 1 -- 172.21.15.78:0/1970620054 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb00018960 con 0x7fbb10102160 2024-03-24T00:18:41.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.108+0000 7fbb168e8640 1 -- 172.21.15.78:0/1970620054 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb1019fe80 con 0x7fbb10102160 2024-03-24T00:18:41.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.109+0000 7fbb168e8640 1 -- 172.21.15.78:0/1970620054 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb100fd640 con 0x7fbb10102160 2024-03-24T00:18:41.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.110+0000 7fbb0d7fa640 1 -- 172.21.15.78:0/1970620054 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fbb00010ac0 con 0x7fbb10102160 2024-03-24T00:18:41.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.111+0000 7fbb0d7fa640 1 --2- 172.21.15.78:0/1970620054 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fbaf80a2800 0x7fbaf80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:41.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.111+0000 7fbb0d7fa640 1 -- 172.21.15.78:0/1970620054 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fbb000ce9f0 con 0x7fbb10102160 2024-03-24T00:18:41.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.111+0000 7fbb0f7fe640 1 --2- 172.21.15.78:0/1970620054 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fbaf80a2800 0x7fbaf80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:41.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.111+0000 7fbb0f7fe640 1 --2- 172.21.15.78:0/1970620054 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fbaf80a2800 0x7fbaf80a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbb10198f30 tx=0x7fbb04005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:41.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.115+0000 7fbb0d7fa640 1 -- 172.21.15.78:0/1970620054 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbb0009b840 con 0x7fbb10102160 2024-03-24T00:18:41.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:41.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:41.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:41.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:41.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:41.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:41.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:40 smithi106 ceph-mon[36505]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:41.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:41.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:41.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:18:41.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:41.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:18:41.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:40 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3492782920' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:41.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.307+0000 7fbb168e8640 1 -- 172.21.15.78:0/1970620054 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbb1002d070 con 0x7fbb10102160 2024-03-24T00:18:41.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.307+0000 7fbb0d7fa640 1 -- 172.21.15.78:0/1970620054 <== mon.1 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbb0009f8e0 con 0x7fbb10102160 2024-03-24T00:18:41.307 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-24T00:18:41.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.309+0000 7fbb168e8640 1 -- 172.21.15.78:0/1970620054 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fbaf80a2800 msgr2=0x7fbaf80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:41.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.309+0000 7fbb168e8640 1 --2- 172.21.15.78:0/1970620054 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fbaf80a2800 0x7fbaf80a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbb10198f30 tx=0x7fbb04005f90 comp rx=0 tx=0).stop 2024-03-24T00:18:41.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.309+0000 7fbb168e8640 1 -- 172.21.15.78:0/1970620054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb10102160 msgr2=0x7fbb10197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:41.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.309+0000 7fbb168e8640 1 --2- 172.21.15.78:0/1970620054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb10102160 0x7fbb10197a20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbb0000e9f0 tx=0x7fbb0000eeb0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.309+0000 7fbb168e8640 1 -- 172.21.15.78:0/1970620054 shutdown_connections 2024-03-24T00:18:41.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.309+0000 7fbb168e8640 1 --2- 172.21.15.78:0/1970620054 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fbaf80a2800 0x7fbaf80a4cc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.309+0000 7fbb168e8640 1 --2- 172.21.15.78:0/1970620054 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb101073c0 0x7fbb1019efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.309+0000 7fbb168e8640 1 --2- 172.21.15.78:0/1970620054 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb10104a90 0x7fbb10197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.309+0000 7fbb168e8640 1 --2- 172.21.15.78:0/1970620054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb10102160 0x7fbb10197a20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.309+0000 7fbb168e8640 1 -- 172.21.15.78:0/1970620054 >> 172.21.15.78:0/1970620054 conn(0x7fbb100fbb50 msgr2=0x7fbb10102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:41.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.309+0000 7fbb168e8640 1 -- 172.21.15.78:0/1970620054 shutdown_connections 2024-03-24T00:18:41.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.309+0000 7fbb168e8640 1 -- 172.21.15.78:0/1970620054 wait complete. 2024-03-24T00:18:41.320 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-24T00:18:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:41.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:40 smithi107 ceph-mon[36485]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:41.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:41.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:41.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:18:41.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:41.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:18:41.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:40 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3492782920' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:41.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.422+0000 7f85c1b87640 1 -- 172.21.15.78:0/1715952482 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85bc103270 msgr2=0x7f85bc109b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:41.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.422+0000 7f85c1b87640 1 --2- 172.21.15.78:0/1715952482 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85bc103270 0x7f85bc109b00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f85b4009ca0 tx=0x7f85b402c0f0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.422+0000 7f85c1b87640 1 -- 172.21.15.78:0/1715952482 shutdown_connections 2024-03-24T00:18:41.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.422+0000 7f85c1b87640 1 --2- 172.21.15.78:0/1715952482 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85bc103270 0x7f85bc109b00 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.422+0000 7f85c1b87640 1 --2- 172.21.15.78:0/1715952482 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85bc1028b0 0x7f85bc102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.422+0000 7f85c1b87640 1 --2- 172.21.15.78:0/1715952482 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85bc1016c0 0x7f85bc101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.422+0000 7f85c1b87640 1 -- 172.21.15.78:0/1715952482 >> 172.21.15.78:0/1715952482 conn(0x7f85bc0fcc50 msgr2=0x7f85bc0ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:41.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.422+0000 7f85c1b87640 1 -- 172.21.15.78:0/1715952482 shutdown_connections 2024-03-24T00:18:41.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.422+0000 7f85c1b87640 1 -- 172.21.15.78:0/1715952482 wait complete. 2024-03-24T00:18:41.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.423+0000 7f85c1b87640 1 Processor -- start 2024-03-24T00:18:41.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.423+0000 7f85c1b87640 1 -- start start 2024-03-24T00:18:41.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.423+0000 7f85c1b87640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85bc1016c0 0x7f85bc197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:41.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.423+0000 7f85c1b87640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85bc1028b0 0x7f85bc197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:41.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.423+0000 7f85c1b87640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85bc103270 0x7f85bc19ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:41.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.423+0000 7f85c1b87640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc10d340 con 0x7f85bc103270 2024-03-24T00:18:41.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.423+0000 7f85c1b87640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc10d1c0 con 0x7f85bc1028b0 2024-03-24T00:18:41.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.423+0000 7f85c1b87640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc10d4c0 con 0x7f85bc1016c0 2024-03-24T00:18:41.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.423+0000 7f85bbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85bc103270 0x7f85bc19ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:41.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.423+0000 7f85bbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85bc103270 0x7f85bc19ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33318/0 (socket says 172.21.15.78:33318) 2024-03-24T00:18:41.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.423+0000 7f85bbfff640 1 -- 172.21.15.78:0/499352348 learned_addr learned my addr 172.21.15.78:0/499352348 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:41.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.423+0000 7f85b2ffd640 1 --2- 172.21.15.78:0/499352348 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85bc1028b0 0x7f85bc197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:41.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.423+0000 7f85bb7fe640 1 --2- 172.21.15.78:0/499352348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85bc1016c0 0x7f85bc197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:41.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.423+0000 7f85bbfff640 1 -- 172.21.15.78:0/499352348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85bc1016c0 msgr2=0x7f85bc197960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:41.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.423+0000 7f85bbfff640 1 --2- 172.21.15.78:0/499352348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85bc1016c0 0x7f85bc197960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.423+0000 7f85bbfff640 1 -- 172.21.15.78:0/499352348 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85bc1028b0 msgr2=0x7f85bc197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:41.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.423+0000 7f85bbfff640 1 --2- 172.21.15.78:0/499352348 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85bc1028b0 0x7f85bc197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.424+0000 7f85bbfff640 1 -- 172.21.15.78:0/499352348 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85bc19f580 con 0x7f85bc103270 2024-03-24T00:18:41.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.424+0000 7f85bb7fe640 1 --2- 172.21.15.78:0/499352348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85bc1016c0 0x7f85bc197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:18:41.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.424+0000 7f85bbfff640 1 --2- 172.21.15.78:0/499352348 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85bc103270 0x7f85bc19ef10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f85b4037040 tx=0x7f85b4032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:41.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.424+0000 7f85b97fa640 1 -- 172.21.15.78:0/499352348 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b4041020 con 0x7f85bc103270 2024-03-24T00:18:41.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.424+0000 7f85c1b87640 1 -- 172.21.15.78:0/499352348 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85bc19f810 con 0x7f85bc103270 2024-03-24T00:18:41.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.424+0000 7f85b97fa640 1 -- 172.21.15.78:0/499352348 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f85b4034cd0 con 0x7f85bc103270 2024-03-24T00:18:41.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.424+0000 7f85b97fa640 1 -- 172.21.15.78:0/499352348 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b4039040 con 0x7f85bc103270 2024-03-24T00:18:41.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.424+0000 7f85c1b87640 1 -- 172.21.15.78:0/499352348 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85bc19fd00 con 0x7f85bc103270 2024-03-24T00:18:41.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.425+0000 7f85c1b87640 1 -- 172.21.15.78:0/499352348 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85bc1089e0 con 0x7f85bc103270 2024-03-24T00:18:41.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.426+0000 7f85b97fa640 1 -- 172.21.15.78:0/499352348 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f85b4044020 con 0x7f85bc103270 2024-03-24T00:18:41.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.426+0000 7f85b97fa640 1 --2- 172.21.15.78:0/499352348 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f85a80a2800 0x7f85a80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:41.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.426+0000 7f85bb7fe640 1 --2- 172.21.15.78:0/499352348 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f85a80a2800 0x7f85a80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:41.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.426+0000 7f85b97fa640 1 -- 172.21.15.78:0/499352348 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f85b40f0510 con 0x7f85bc103270 2024-03-24T00:18:41.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.427+0000 7f85bb7fe640 1 --2- 172.21.15.78:0/499352348 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f85a80a2800 0x7f85a80a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f85bc102710 tx=0x7f85ac005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:41.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.431+0000 7f85b97fa640 1 -- 172.21.15.78:0/499352348 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f85b404b020 con 0x7f85bc103270 2024-03-24T00:18:41.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.617+0000 7f85c1b87640 1 -- 172.21.15.78:0/499352348 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f85bc0622a0 con 0x7f85bc103270 2024-03-24T00:18:41.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.617+0000 7f85b97fa640 1 -- 172.21.15.78:0/499352348 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) v1 ==== 84+0+65 (secure 0 0 0) 0x7f85b40bdae0 con 0x7f85bc103270 2024-03-24T00:18:41.617 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:18:41.617 INFO:teuthology.orchestra.run.smithi078.stdout:[ 2024-03-24T00:18:41.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T00:18:41.618 INFO:teuthology.orchestra.run.smithi078.stdout: "poolnum": 1, 2024-03-24T00:18:41.618 INFO:teuthology.orchestra.run.smithi078.stdout: "poolname": ".mgr" 2024-03-24T00:18:41.618 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T00:18:41.618 INFO:teuthology.orchestra.run.smithi078.stdout:] 2024-03-24T00:18:41.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.619+0000 7f85c1b87640 1 -- 172.21.15.78:0/499352348 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f85a80a2800 msgr2=0x7f85a80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:41.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.619+0000 7f85c1b87640 1 --2- 172.21.15.78:0/499352348 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f85a80a2800 0x7f85a80a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f85bc102710 tx=0x7f85ac005f90 comp rx=0 tx=0).stop 2024-03-24T00:18:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.619+0000 7f85c1b87640 1 -- 172.21.15.78:0/499352348 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85bc103270 msgr2=0x7f85bc19ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.619+0000 7f85c1b87640 1 --2- 172.21.15.78:0/499352348 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85bc103270 0x7f85bc19ef10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f85b4037040 tx=0x7f85b4032f40 comp rx=0 tx=0).stop 2024-03-24T00:18:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.620+0000 7f85c1b87640 1 -- 172.21.15.78:0/499352348 shutdown_connections 2024-03-24T00:18:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.620+0000 7f85c1b87640 1 --2- 172.21.15.78:0/499352348 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f85a80a2800 0x7f85a80a4cc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.620+0000 7f85c1b87640 1 --2- 172.21.15.78:0/499352348 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85bc103270 0x7f85bc19ef10 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.620+0000 7f85c1b87640 1 --2- 172.21.15.78:0/499352348 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85bc1028b0 0x7f85bc197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.620+0000 7f85c1b87640 1 --2- 172.21.15.78:0/499352348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85bc1016c0 0x7f85bc197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.620+0000 7f85c1b87640 1 -- 172.21.15.78:0/499352348 >> 172.21.15.78:0/499352348 conn(0x7f85bc0fcc50 msgr2=0x7f85bc105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.620+0000 7f85c1b87640 1 -- 172.21.15.78:0/499352348 shutdown_connections 2024-03-24T00:18:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.620+0000 7f85c1b87640 1 -- 172.21.15.78:0/499352348 wait complete. 2024-03-24T00:18:41.630 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-24T00:18:41.630 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-24T00:18:41.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.737+0000 7f859f9e8640 1 -- 172.21.15.78:0/3957411438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8598102150 msgr2=0x7f85981089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:41.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.737+0000 7f859f9e8640 1 --2- 172.21.15.78:0/3957411438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8598102150 0x7f85981089e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8590009ca0 tx=0x7f859002c0f0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.737+0000 7f859f9e8640 1 -- 172.21.15.78:0/3957411438 shutdown_connections 2024-03-24T00:18:41.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.737+0000 7f859f9e8640 1 --2- 172.21.15.78:0/3957411438 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8598101790 0x7f8598101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.738+0000 7f859f9e8640 1 --2- 172.21.15.78:0/3957411438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8598102150 0x7f85981089e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.738+0000 7f859f9e8640 1 --2- 172.21.15.78:0/3957411438 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85981005a0 0x7f85981009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.738+0000 7f859f9e8640 1 -- 172.21.15.78:0/3957411438 >> 172.21.15.78:0/3957411438 conn(0x7f85980fbb30 msgr2=0x7f85980fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:41.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.738+0000 7f859f9e8640 1 -- 172.21.15.78:0/3957411438 shutdown_connections 2024-03-24T00:18:41.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.738+0000 7f859f9e8640 1 -- 172.21.15.78:0/3957411438 wait complete. 2024-03-24T00:18:41.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.738+0000 7f859f9e8640 1 Processor -- start 2024-03-24T00:18:41.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.738+0000 7f859f9e8640 1 -- start start 2024-03-24T00:18:41.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.738+0000 7f859f9e8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85981005a0 0x7f85980693d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:41.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.738+0000 7f859f9e8640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8598101790 0x7f859819c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:41.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.738+0000 7f859f9e8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8598102150 0x7f859819ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:41.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.738+0000 7f859f9e8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859810c230 con 0x7f85981005a0 2024-03-24T00:18:41.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.738+0000 7f859f9e8640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859810c0b0 con 0x7f8598101790 2024-03-24T00:18:41.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.738+0000 7f859f9e8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859810c3b0 con 0x7f8598102150 2024-03-24T00:18:41.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.738+0000 7f859d784640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85981005a0 0x7f85980693d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:41.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.738+0000 7f859d784640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85981005a0 0x7f85980693d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33328/0 (socket says 172.21.15.78:33328) 2024-03-24T00:18:41.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.738+0000 7f859d784640 1 -- 172.21.15.78:0/1063056248 learned_addr learned my addr 172.21.15.78:0/1063056248 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:41.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.739+0000 7f859df85640 1 --2- 172.21.15.78:0/1063056248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8598102150 0x7f859819ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:41.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.739+0000 7f859cf83640 1 --2- 172.21.15.78:0/1063056248 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8598101790 0x7f859819c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:41.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.739+0000 7f859d784640 1 -- 172.21.15.78:0/1063056248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8598102150 msgr2=0x7f859819ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:41.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.739+0000 7f859d784640 1 --2- 172.21.15.78:0/1063056248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8598102150 0x7f859819ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.739+0000 7f859d784640 1 -- 172.21.15.78:0/1063056248 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8598101790 msgr2=0x7f859819c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:41.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.739+0000 7f859d784640 1 --2- 172.21.15.78:0/1063056248 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8598101790 0x7f859819c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.739+0000 7f859d784640 1 -- 172.21.15.78:0/1063056248 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f859819f5a0 con 0x7f85981005a0 2024-03-24T00:18:41.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.739+0000 7f859cf83640 1 --2- 172.21.15.78:0/1063056248 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8598101790 0x7f859819c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:18:41.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.739+0000 7f859d784640 1 --2- 172.21.15.78:0/1063056248 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85981005a0 0x7f85980693d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f859400d950 tx=0x7f859400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:41.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.739+0000 7f858e7fc640 1 -- 172.21.15.78:0/1063056248 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8594017020 con 0x7f85981005a0 2024-03-24T00:18:41.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.739+0000 7f859f9e8640 1 -- 172.21.15.78:0/1063056248 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f859819f890 con 0x7f85981005a0 2024-03-24T00:18:41.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.739+0000 7f858e7fc640 1 -- 172.21.15.78:0/1063056248 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8594009c30 con 0x7f85981005a0 2024-03-24T00:18:41.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.739+0000 7f858e7fc640 1 -- 172.21.15.78:0/1063056248 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859400f740 con 0x7f85981005a0 2024-03-24T00:18:41.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.739+0000 7f859f9e8640 1 -- 172.21.15.78:0/1063056248 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f859819fde0 con 0x7f85981005a0 2024-03-24T00:18:41.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.740+0000 7f859f9e8640 1 -- 172.21.15.78:0/1063056248 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85981078c0 con 0x7f85981005a0 2024-03-24T00:18:41.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.741+0000 7f858e7fc640 1 -- 172.21.15.78:0/1063056248 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f859401a020 con 0x7f85981005a0 2024-03-24T00:18:41.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.741+0000 7f858e7fc640 1 --2- 172.21.15.78:0/1063056248 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f85840a2800 0x7f85840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:41.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.742+0000 7f859cf83640 1 --2- 172.21.15.78:0/1063056248 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f85840a2800 0x7f85840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:41.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.742+0000 7f858e7fc640 1 -- 172.21.15.78:0/1063056248 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f85940ce090 con 0x7f85981005a0 2024-03-24T00:18:41.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.742+0000 7f859cf83640 1 --2- 172.21.15.78:0/1063056248 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f85840a2800 0x7f85840a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8588005fd0 tx=0x7f85880094d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:41.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.746+0000 7f858e7fc640 1 -- 172.21.15.78:0/1063056248 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8594021070 con 0x7f85981005a0 2024-03-24T00:18:41.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:41.927+0000 7f859f9e8640 1 -- 172.21.15.78:0/1063056248 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f859802d070 con 0x7f85981005a0 2024-03-24T00:18:42.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.231+0000 7f858e7fc640 1 -- 172.21.15.78:0/1063056248 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8594009de0 con 0x7f85981005a0 2024-03-24T00:18:42.231 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' created 2024-03-24T00:18:42.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.233+0000 7f859f9e8640 1 -- 172.21.15.78:0/1063056248 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f85840a2800 msgr2=0x7f85840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:42.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.233+0000 7f859f9e8640 1 --2- 172.21.15.78:0/1063056248 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f85840a2800 0x7f85840a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8588005fd0 tx=0x7f85880094d0 comp rx=0 tx=0).stop 2024-03-24T00:18:42.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.233+0000 7f859f9e8640 1 -- 172.21.15.78:0/1063056248 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85981005a0 msgr2=0x7f85980693d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:42.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.233+0000 7f859f9e8640 1 --2- 172.21.15.78:0/1063056248 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85981005a0 0x7f85980693d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f859400d950 tx=0x7f859400de10 comp rx=0 tx=0).stop 2024-03-24T00:18:42.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.233+0000 7f859f9e8640 1 -- 172.21.15.78:0/1063056248 shutdown_connections 2024-03-24T00:18:42.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.233+0000 7f859f9e8640 1 --2- 172.21.15.78:0/1063056248 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f85840a2800 0x7f85840a4cc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:42.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.233+0000 7f859f9e8640 1 --2- 172.21.15.78:0/1063056248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8598102150 0x7f859819ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:42.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.233+0000 7f859f9e8640 1 --2- 172.21.15.78:0/1063056248 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8598101790 0x7f859819c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:42.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.233+0000 7f859f9e8640 1 --2- 172.21.15.78:0/1063056248 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85981005a0 0x7f85980693d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:42.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.233+0000 7f859f9e8640 1 -- 172.21.15.78:0/1063056248 >> 172.21.15.78:0/1063056248 conn(0x7f85980fbb30 msgr2=0x7f8598104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:42.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.234+0000 7f859f9e8640 1 -- 172.21.15.78:0/1063056248 shutdown_connections 2024-03-24T00:18:42.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.234+0000 7f859f9e8640 1 -- 172.21.15.78:0/1063056248 wait complete. 2024-03-24T00:18:42.249 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-24T00:18:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.372+0000 7f71a3434640 1 -- 172.21.15.78:0/327354337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f719c1029f0 msgr2=0x7f719c102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.372+0000 7f71a3434640 1 --2- 172.21.15.78:0/327354337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f719c1029f0 0x7f719c102e70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7198009900 tx=0x7f719802bc80 comp rx=0 tx=0).stop 2024-03-24T00:18:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.372+0000 7f71a3434640 1 -- 172.21.15.78:0/327354337 shutdown_connections 2024-03-24T00:18:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.372+0000 7f71a3434640 1 --2- 172.21.15.78:0/327354337 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f719c1033b0 0x7f719c109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.372+0000 7f71a3434640 1 --2- 172.21.15.78:0/327354337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f719c1029f0 0x7f719c102e70 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.372+0000 7f71a3434640 1 --2- 172.21.15.78:0/327354337 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f719c0ff980 0x7f719c0ffda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.372+0000 7f71a3434640 1 -- 172.21.15.78:0/327354337 >> 172.21.15.78:0/327354337 conn(0x7f719c0fb2d0 msgr2=0x7f719c0fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.372+0000 7f71a3434640 1 -- 172.21.15.78:0/327354337 shutdown_connections 2024-03-24T00:18:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.372+0000 7f71a3434640 1 -- 172.21.15.78:0/327354337 wait complete. 2024-03-24T00:18:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.372+0000 7f71a3434640 1 Processor -- start 2024-03-24T00:18:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.372+0000 7f71a3434640 1 -- start start 2024-03-24T00:18:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.372+0000 7f71a3434640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f719c0ff980 0x7f719c197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.372+0000 7f71a3434640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f719c1029f0 0x7f719c197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.372+0000 7f71a3434640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f719c1033b0 0x7f719c19efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.373+0000 7f71a3434640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719c10d3b0 con 0x7f719c1029f0 2024-03-24T00:18:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.373+0000 7f71a3434640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719c10d230 con 0x7f719c1033b0 2024-03-24T00:18:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.373+0000 7f71a3434640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719c10d530 con 0x7f719c0ff980 2024-03-24T00:18:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.373+0000 7f71a09cf640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f719c1029f0 0x7f719c197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.373+0000 7f71a09cf640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f719c1029f0 0x7f719c197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33354/0 (socket says 172.21.15.78:33354) 2024-03-24T00:18:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.373+0000 7f71a09cf640 1 -- 172.21.15.78:0/3227238172 learned_addr learned my addr 172.21.15.78:0/3227238172 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.373+0000 7f71a11d0640 1 --2- 172.21.15.78:0/3227238172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f719c0ff980 0x7f719c197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.373+0000 7f71a09cf640 1 -- 172.21.15.78:0/3227238172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f719c0ff980 msgr2=0x7f719c197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.373+0000 7f71a09cf640 1 --2- 172.21.15.78:0/3227238172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f719c0ff980 0x7f719c197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.373+0000 7f71a09cf640 1 -- 172.21.15.78:0/3227238172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f719c1033b0 msgr2=0x7f719c19efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.373+0000 7f71a09cf640 1 --2- 172.21.15.78:0/3227238172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f719c1033b0 0x7f719c19efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:42.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.373+0000 7f71a09cf640 1 -- 172.21.15.78:0/3227238172 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f719c19f640 con 0x7f719c1029f0 2024-03-24T00:18:42.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.373+0000 7f71a09cf640 1 --2- 172.21.15.78:0/3227238172 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f719c1029f0 0x7f719c197f60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7198009430 tx=0x7f7198005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:42.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.373+0000 7f71927fc640 1 -- 172.21.15.78:0/3227238172 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7198043020 con 0x7f719c1029f0 2024-03-24T00:18:42.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.373+0000 7f71927fc640 1 -- 172.21.15.78:0/3227238172 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7198004410 con 0x7f719c1029f0 2024-03-24T00:18:42.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.373+0000 7f71927fc640 1 -- 172.21.15.78:0/3227238172 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7198003e50 con 0x7f719c1029f0 2024-03-24T00:18:42.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.373+0000 7f71a3434640 1 -- 172.21.15.78:0/3227238172 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f719c19f8d0 con 0x7f719c1029f0 2024-03-24T00:18:42.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.373+0000 7f71a3434640 1 -- 172.21.15.78:0/3227238172 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f719c19fdd0 con 0x7f719c1029f0 2024-03-24T00:18:42.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.374+0000 7f71a3434640 1 -- 172.21.15.78:0/3227238172 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f719c108ba0 con 0x7f719c1029f0 2024-03-24T00:18:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.375+0000 7f71927fc640 1 -- 172.21.15.78:0/3227238172 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7198046020 con 0x7f719c1029f0 2024-03-24T00:18:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.376+0000 7f71927fc640 1 --2- 172.21.15.78:0/3227238172 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f71880a29a0 0x7f71880a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.376+0000 7f71a11d0640 1 --2- 172.21.15.78:0/3227238172 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f71880a29a0 0x7f71880a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.376+0000 7f71927fc640 1 -- 172.21.15.78:0/3227238172 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f71980f0bf0 con 0x7f719c1029f0 2024-03-24T00:18:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.376+0000 7f71a11d0640 1 --2- 172.21.15.78:0/3227238172 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f71880a29a0 0x7f71880a4e60 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f718c005d00 tx=0x7f718c005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:42.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.380+0000 7f71927fc640 1 -- 172.21.15.78:0/3227238172 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f719804d070 con 0x7f719c1029f0 2024-03-24T00:18:42.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:42.564+0000 7f71a3434640 1 -- 172.21.15.78:0/3227238172 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f719c02d070 con 0x7f719c1029f0 2024-03-24T00:18:42.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:42 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1876462742' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:18:42.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:42 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1970620054' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:18:42.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:42 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/499352348' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-24T00:18:42.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:42 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1876462742' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:18:42.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:42 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1970620054' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:18:42.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:42 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/499352348' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-24T00:18:42.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:42 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1876462742' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:18:42.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:42 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1970620054' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:18:42.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:42 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/499352348' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-24T00:18:43.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.239+0000 7f71927fc640 1 -- 172.21.15.78:0/3227238172 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) v1 ==== 171+0+0 (secure 0 0 0) 0x7f71980473b0 con 0x7f719c1029f0 2024-03-24T00:18:43.239 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' created 2024-03-24T00:18:43.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.241+0000 7f71a3434640 1 -- 172.21.15.78:0/3227238172 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f71880a29a0 msgr2=0x7f71880a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:43.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.241+0000 7f71a3434640 1 --2- 172.21.15.78:0/3227238172 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f71880a29a0 0x7f71880a4e60 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f718c005d00 tx=0x7f718c005c50 comp rx=0 tx=0).stop 2024-03-24T00:18:43.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.241+0000 7f71a3434640 1 -- 172.21.15.78:0/3227238172 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f719c1029f0 msgr2=0x7f719c197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:43.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.241+0000 7f71a3434640 1 --2- 172.21.15.78:0/3227238172 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f719c1029f0 0x7f719c197f60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7198009430 tx=0x7f7198005dd0 comp rx=0 tx=0).stop 2024-03-24T00:18:43.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.241+0000 7f71a3434640 1 -- 172.21.15.78:0/3227238172 shutdown_connections 2024-03-24T00:18:43.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.241+0000 7f71a3434640 1 --2- 172.21.15.78:0/3227238172 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f71880a29a0 0x7f71880a4e60 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:43.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.241+0000 7f71a3434640 1 --2- 172.21.15.78:0/3227238172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f719c1033b0 0x7f719c19efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:43.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.241+0000 7f71a3434640 1 --2- 172.21.15.78:0/3227238172 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f719c1029f0 0x7f719c197f60 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:43.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.241+0000 7f71a3434640 1 --2- 172.21.15.78:0/3227238172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f719c0ff980 0x7f719c197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:43.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.241+0000 7f71a3434640 1 -- 172.21.15.78:0/3227238172 >> 172.21.15.78:0/3227238172 conn(0x7f719c0fb2d0 msgr2=0x7f719c105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:43.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.241+0000 7f71a3434640 1 -- 172.21.15.78:0/3227238172 shutdown_connections 2024-03-24T00:18:43.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.241+0000 7f71a3434640 1 -- 172.21.15.78:0/3227238172 wait complete. 2024-03-24T00:18:43.259 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-24T00:18:43.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.366+0000 7f3b573a9640 1 -- 172.21.15.78:0/2288401110 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b501005c0 msgr2=0x7f3b501009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:43.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.366+0000 7f3b573a9640 1 --2- 172.21.15.78:0/2288401110 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b501005c0 0x7f3b501009e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c009900 tx=0x7f3b4c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:43.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.366+0000 7f3b573a9640 1 -- 172.21.15.78:0/2288401110 shutdown_connections 2024-03-24T00:18:43.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.366+0000 7f3b573a9640 1 --2- 172.21.15.78:0/2288401110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b501017b0 0x7f3b50101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:43.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.366+0000 7f3b573a9640 1 --2- 172.21.15.78:0/2288401110 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b50102170 0x7f3b50108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:43.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.366+0000 7f3b573a9640 1 --2- 172.21.15.78:0/2288401110 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b501005c0 0x7f3b501009e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:43.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.366+0000 7f3b573a9640 1 -- 172.21.15.78:0/2288401110 >> 172.21.15.78:0/2288401110 conn(0x7f3b500fbb50 msgr2=0x7f3b500fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:43.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.366+0000 7f3b573a9640 1 -- 172.21.15.78:0/2288401110 shutdown_connections 2024-03-24T00:18:43.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.366+0000 7f3b573a9640 1 -- 172.21.15.78:0/2288401110 wait complete. 2024-03-24T00:18:43.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.366+0000 7f3b573a9640 1 Processor -- start 2024-03-24T00:18:43.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.367+0000 7f3b573a9640 1 -- start start 2024-03-24T00:18:43.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.367+0000 7f3b573a9640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b501005c0 0x7f3b501979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:43.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.367+0000 7f3b573a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b501017b0 0x7f3b50197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:43.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.367+0000 7f3b573a9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b50102170 0x7f3b5019ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:43.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.367+0000 7f3b573a9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5010d360 con 0x7f3b50102170 2024-03-24T00:18:43.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.367+0000 7f3b573a9640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5010d1e0 con 0x7f3b501005c0 2024-03-24T00:18:43.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.367+0000 7f3b573a9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5010d4e0 con 0x7f3b501017b0 2024-03-24T00:18:43.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.367+0000 7f3b55946640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b50102170 0x7f3b5019ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:43.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.367+0000 7f3b55946640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b50102170 0x7f3b5019ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33378/0 (socket says 172.21.15.78:33378) 2024-03-24T00:18:43.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.367+0000 7f3b55946640 1 -- 172.21.15.78:0/3744748533 learned_addr learned my addr 172.21.15.78:0/3744748533 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:43.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.367+0000 7f3b54944640 1 --2- 172.21.15.78:0/3744748533 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b501017b0 0x7f3b50197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:43.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.367+0000 7f3b55145640 1 --2- 172.21.15.78:0/3744748533 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b501005c0 0x7f3b501979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:43.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.367+0000 7f3b55946640 1 -- 172.21.15.78:0/3744748533 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b501017b0 msgr2=0x7f3b50197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:43.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.367+0000 7f3b55946640 1 --2- 172.21.15.78:0/3744748533 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b501017b0 0x7f3b50197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:43.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.367+0000 7f3b55946640 1 -- 172.21.15.78:0/3744748533 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b501005c0 msgr2=0x7f3b501979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:43.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.367+0000 7f3b55946640 1 --2- 172.21.15.78:0/3744748533 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b501005c0 0x7f3b501979c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:43.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.367+0000 7f3b55946640 1 -- 172.21.15.78:0/3744748533 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b5019f5e0 con 0x7f3b50102170 2024-03-24T00:18:43.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.367+0000 7f3b54944640 1 --2- 172.21.15.78:0/3744748533 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b501017b0 0x7f3b50197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:18:43.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.367+0000 7f3b55946640 1 --2- 172.21.15.78:0/3744748533 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b50102170 0x7f3b5019ef70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3b4800cce0 tx=0x7f3b4800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:43.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.367+0000 7f3b467fc640 1 -- 172.21.15.78:0/3744748533 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4800f450 con 0x7f3b50102170 2024-03-24T00:18:43.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.367+0000 7f3b573a9640 1 -- 172.21.15.78:0/3744748533 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b5019f8d0 con 0x7f3b50102170 2024-03-24T00:18:43.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.367+0000 7f3b467fc640 1 -- 172.21.15.78:0/3744748533 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3b4800fad0 con 0x7f3b50102170 2024-03-24T00:18:43.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.367+0000 7f3b467fc640 1 -- 172.21.15.78:0/3744748533 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b48018c10 con 0x7f3b50102170 2024-03-24T00:18:43.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.367+0000 7f3b573a9640 1 -- 172.21.15.78:0/3744748533 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b5019fe20 con 0x7f3b50102170 2024-03-24T00:18:43.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.368+0000 7f3b573a9640 1 -- 172.21.15.78:0/3744748533 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b50107960 con 0x7f3b50102170 2024-03-24T00:18:43.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.370+0000 7f3b467fc640 1 -- 172.21.15.78:0/3744748533 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3b4800f5f0 con 0x7f3b50102170 2024-03-24T00:18:43.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.370+0000 7f3b467fc640 1 --2- 172.21.15.78:0/3744748533 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3b3c0a2800 0x7f3b3c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:43.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.370+0000 7f3b55145640 1 --2- 172.21.15.78:0/3744748533 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3b3c0a2800 0x7f3b3c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:43.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.370+0000 7f3b467fc640 1 -- 172.21.15.78:0/3744748533 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f3b480cf0a0 con 0x7f3b50102170 2024-03-24T00:18:43.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.371+0000 7f3b55145640 1 --2- 172.21.15.78:0/3744748533 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3b3c0a2800 0x7f3b3c0a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c036040 tx=0x7f3b4c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:43.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.374+0000 7f3b467fc640 1 -- 172.21.15.78:0/3744748533 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3b48014030 con 0x7f3b50102170 2024-03-24T00:18:43.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:43.590+0000 7f3b573a9640 1 -- 172.21.15.78:0/3744748533 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f3b5002d070 con 0x7f3b50102170 2024-03-24T00:18:43.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:43 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1063056248' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-24T00:18:43.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:43 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1063056248' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-24T00:18:43.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:43 smithi078 ceph-mon[32429]: osdmap e82: 12 total, 12 up, 12 in 2024-03-24T00:18:43.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:43 smithi078 ceph-mon[32429]: pgmap v283: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:43.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:43 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:43.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:43 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:43.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:43 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:43.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:43 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:43.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:43 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:43.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:43 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:43.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:43 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:43.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:43 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:18:43.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:43 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:43.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:43 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3227238172' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-24T00:18:43.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:43 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1063056248' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-24T00:18:43.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:43 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1063056248' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-24T00:18:43.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:43 smithi107 ceph-mon[36485]: osdmap e82: 12 total, 12 up, 12 in 2024-03-24T00:18:43.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:43 smithi107 ceph-mon[36485]: pgmap v283: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:43.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:43 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:43.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:43 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:43.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:43 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:43.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:43 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:43.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:43 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:43.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:43 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:43.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:43 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:43.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:43 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:18:43.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:43 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:43.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:43 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3227238172' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-24T00:18:43.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:43 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1063056248' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-24T00:18:43.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:43 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1063056248' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-24T00:18:43.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:43 smithi106 ceph-mon[36505]: osdmap e82: 12 total, 12 up, 12 in 2024-03-24T00:18:43.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:43 smithi106 ceph-mon[36505]: pgmap v283: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:43.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:43 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:43.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:43 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:43.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:43 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:43.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:43 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:43.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:43 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:43.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:43 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:43.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:43 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:43.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:43 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:18:43.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:43 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:18:43.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:43 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3227238172' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-24T00:18:44.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.240+0000 7f3b467fc640 1 -- 172.21.15.78:0/3744748533 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f3b4809c380 con 0x7f3b50102170 2024-03-24T00:18:44.240 INFO:teuthology.orchestra.run.smithi078.stderr:new fs with metadata pool 2 and data pool 3 2024-03-24T00:18:44.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.242+0000 7f3b573a9640 1 -- 172.21.15.78:0/3744748533 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3b3c0a2800 msgr2=0x7f3b3c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:44.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.242+0000 7f3b573a9640 1 --2- 172.21.15.78:0/3744748533 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3b3c0a2800 0x7f3b3c0a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c036040 tx=0x7f3b4c005fd0 comp rx=0 tx=0).stop 2024-03-24T00:18:44.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.242+0000 7f3b573a9640 1 -- 172.21.15.78:0/3744748533 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b50102170 msgr2=0x7f3b5019ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:44.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.242+0000 7f3b573a9640 1 --2- 172.21.15.78:0/3744748533 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b50102170 0x7f3b5019ef70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3b4800cce0 tx=0x7f3b4800a620 comp rx=0 tx=0).stop 2024-03-24T00:18:44.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.242+0000 7f3b573a9640 1 -- 172.21.15.78:0/3744748533 shutdown_connections 2024-03-24T00:18:44.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.242+0000 7f3b573a9640 1 --2- 172.21.15.78:0/3744748533 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3b3c0a2800 0x7f3b3c0a4cc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:44.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.242+0000 7f3b573a9640 1 --2- 172.21.15.78:0/3744748533 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b501017b0 0x7f3b50197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:44.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.242+0000 7f3b573a9640 1 --2- 172.21.15.78:0/3744748533 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b50102170 0x7f3b5019ef70 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:44.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.242+0000 7f3b573a9640 1 --2- 172.21.15.78:0/3744748533 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b501005c0 0x7f3b501979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:44.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.242+0000 7f3b573a9640 1 -- 172.21.15.78:0/3744748533 >> 172.21.15.78:0/3744748533 conn(0x7f3b500fbb50 msgr2=0x7f3b50104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:44.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.242+0000 7f3b573a9640 1 -- 172.21.15.78:0/3744748533 shutdown_connections 2024-03-24T00:18:44.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.242+0000 7f3b573a9640 1 -- 172.21.15.78:0/3744748533 wait complete. 2024-03-24T00:18:44.256 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-24T00:18:44.256 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-24T00:18:44.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.363+0000 7fb93af77640 1 -- 172.21.15.78:0/989060274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb934101790 msgr2=0x7fb934101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:44.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.363+0000 7fb93af77640 1 --2- 172.21.15.78:0/989060274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb934101790 0x7fb934101c10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb920009900 tx=0x7fb92002bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:44.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.363+0000 7fb93af77640 1 -- 172.21.15.78:0/989060274 shutdown_connections 2024-03-24T00:18:44.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.363+0000 7fb93af77640 1 --2- 172.21.15.78:0/989060274 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb934102150 0x7fb9341089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:44.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.363+0000 7fb93af77640 1 --2- 172.21.15.78:0/989060274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb934101790 0x7fb934101c10 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:44.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.363+0000 7fb93af77640 1 --2- 172.21.15.78:0/989060274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9341005a0 0x7fb9341009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:44.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.363+0000 7fb93af77640 1 -- 172.21.15.78:0/989060274 >> 172.21.15.78:0/989060274 conn(0x7fb9340fbb30 msgr2=0x7fb9340fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:44.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.363+0000 7fb93af77640 1 -- 172.21.15.78:0/989060274 shutdown_connections 2024-03-24T00:18:44.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.363+0000 7fb93af77640 1 -- 172.21.15.78:0/989060274 wait complete. 2024-03-24T00:18:44.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.364+0000 7fb93af77640 1 Processor -- start 2024-03-24T00:18:44.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.364+0000 7fb93af77640 1 -- start start 2024-03-24T00:18:44.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.364+0000 7fb93af77640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9341005a0 0x7fb934069370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:44.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.364+0000 7fb93af77640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb934101790 0x7fb93419c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:44.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.364+0000 7fb93af77640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb934102150 0x7fb93419ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:44.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.364+0000 7fb93af77640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb93410c230 con 0x7fb934102150 2024-03-24T00:18:44.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.364+0000 7fb93af77640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb93410c0b0 con 0x7fb9341005a0 2024-03-24T00:18:44.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.364+0000 7fb93af77640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb93410c3b0 con 0x7fb934101790 2024-03-24T00:18:44.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.364+0000 7fb939514640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb934102150 0x7fb93419ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:44.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.364+0000 7fb939514640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb934102150 0x7fb93419ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33400/0 (socket says 172.21.15.78:33400) 2024-03-24T00:18:44.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.364+0000 7fb939514640 1 -- 172.21.15.78:0/3113904182 learned_addr learned my addr 172.21.15.78:0/3113904182 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:44.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.364+0000 7fb938d13640 1 --2- 172.21.15.78:0/3113904182 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9341005a0 0x7fb934069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:44.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.364+0000 7fb933fff640 1 --2- 172.21.15.78:0/3113904182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb934101790 0x7fb93419c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:44.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.364+0000 7fb939514640 1 -- 172.21.15.78:0/3113904182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb934101790 msgr2=0x7fb93419c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:44.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.364+0000 7fb939514640 1 --2- 172.21.15.78:0/3113904182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb934101790 0x7fb93419c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:44.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.364+0000 7fb939514640 1 -- 172.21.15.78:0/3113904182 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9341005a0 msgr2=0x7fb934069370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:44.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.364+0000 7fb939514640 1 --2- 172.21.15.78:0/3113904182 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9341005a0 0x7fb934069370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:44.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.364+0000 7fb939514640 1 -- 172.21.15.78:0/3113904182 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb93419f5e0 con 0x7fb934102150 2024-03-24T00:18:44.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.364+0000 7fb939514640 1 --2- 172.21.15.78:0/3113904182 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb934102150 0x7fb93419ecd0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb92c00cce0 tx=0x7fb92c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:44.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.364+0000 7fb931ffb640 1 -- 172.21.15.78:0/3113904182 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb92c00f450 con 0x7fb934102150 2024-03-24T00:18:44.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.364+0000 7fb931ffb640 1 -- 172.21.15.78:0/3113904182 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb92c00fad0 con 0x7fb934102150 2024-03-24T00:18:44.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.364+0000 7fb931ffb640 1 -- 172.21.15.78:0/3113904182 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb92c018c10 con 0x7fb934102150 2024-03-24T00:18:44.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.364+0000 7fb93af77640 1 -- 172.21.15.78:0/3113904182 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb93419f870 con 0x7fb934102150 2024-03-24T00:18:44.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.365+0000 7fb93af77640 1 -- 172.21.15.78:0/3113904182 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb93419fd70 con 0x7fb934102150 2024-03-24T00:18:44.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.366+0000 7fb93af77640 1 -- 172.21.15.78:0/3113904182 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9341078c0 con 0x7fb934102150 2024-03-24T00:18:44.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.367+0000 7fb931ffb640 1 -- 172.21.15.78:0/3113904182 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb92c00f5f0 con 0x7fb934102150 2024-03-24T00:18:44.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.367+0000 7fb931ffb640 1 --2- 172.21.15.78:0/3113904182 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fb9240a2800 0x7fb9240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:44.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.367+0000 7fb938d13640 1 --2- 172.21.15.78:0/3113904182 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fb9240a2800 0x7fb9240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:44.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.367+0000 7fb931ffb640 1 -- 172.21.15.78:0/3113904182 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fb92c0cf010 con 0x7fb934102150 2024-03-24T00:18:44.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.368+0000 7fb938d13640 1 --2- 172.21.15.78:0/3113904182 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fb9240a2800 0x7fb9240a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb928005fd0 tx=0x7fb92800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:44.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.372+0000 7fb931ffb640 1 -- 172.21.15.78:0/3113904182 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb92c014030 con 0x7fb934102150 2024-03-24T00:18:44.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.556+0000 7fb93af77640 1 -- 172.21.15.78:0/3113904182 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7fb93402d270 con 0x7fb934102150 2024-03-24T00:18:44.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:44 smithi078 ceph-mon[32429]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T00:18:44.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:44 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3227238172' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-24T00:18:44.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:44 smithi078 ceph-mon[32429]: osdmap e83: 12 total, 12 up, 12 in 2024-03-24T00:18:44.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:44 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3744748533' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-24T00:18:44.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:44 smithi078 ceph-mon[32429]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-24T00:18:44.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:44 smithi078 ceph-mon[32429]: osdmap e84: 12 total, 12 up, 12 in 2024-03-24T00:18:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:44 smithi107 ceph-mon[36485]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T00:18:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:44 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3227238172' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-24T00:18:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:44 smithi107 ceph-mon[36485]: osdmap e83: 12 total, 12 up, 12 in 2024-03-24T00:18:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:44 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3744748533' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-24T00:18:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:44 smithi107 ceph-mon[36485]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-24T00:18:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:44 smithi107 ceph-mon[36485]: osdmap e84: 12 total, 12 up, 12 in 2024-03-24T00:18:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:44 smithi106 ceph-mon[36505]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T00:18:44.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:44 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3227238172' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-24T00:18:44.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:44 smithi106 ceph-mon[36505]: osdmap e83: 12 total, 12 up, 12 in 2024-03-24T00:18:44.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:44 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3744748533' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-24T00:18:44.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:44 smithi106 ceph-mon[36505]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-24T00:18:44.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:44 smithi106 ceph-mon[36505]: osdmap e84: 12 total, 12 up, 12 in 2024-03-24T00:18:44.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.867+0000 7fb931ffb640 1 -- 172.21.15.78:0/3113904182 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) v1 ==== 157+0+0 (secure 0 0 0) 0x7fb92c09c200 con 0x7fb934102150 2024-03-24T00:18:44.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.869+0000 7fb93af77640 1 -- 172.21.15.78:0/3113904182 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fb9240a2800 msgr2=0x7fb9240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:44.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.869+0000 7fb93af77640 1 --2- 172.21.15.78:0/3113904182 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fb9240a2800 0x7fb9240a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb928005fd0 tx=0x7fb92800b390 comp rx=0 tx=0).stop 2024-03-24T00:18:44.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.870+0000 7fb93af77640 1 -- 172.21.15.78:0/3113904182 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb934102150 msgr2=0x7fb93419ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:44.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.870+0000 7fb93af77640 1 --2- 172.21.15.78:0/3113904182 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb934102150 0x7fb93419ecd0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb92c00cce0 tx=0x7fb92c00a620 comp rx=0 tx=0).stop 2024-03-24T00:18:44.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.870+0000 7fb93af77640 1 -- 172.21.15.78:0/3113904182 shutdown_connections 2024-03-24T00:18:44.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.870+0000 7fb93af77640 1 --2- 172.21.15.78:0/3113904182 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fb9240a2800 0x7fb9240a4cc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:44.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.870+0000 7fb93af77640 1 --2- 172.21.15.78:0/3113904182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb934101790 0x7fb93419c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:44.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.870+0000 7fb93af77640 1 --2- 172.21.15.78:0/3113904182 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb934102150 0x7fb93419ecd0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:44.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.870+0000 7fb93af77640 1 --2- 172.21.15.78:0/3113904182 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9341005a0 0x7fb934069370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:44.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.870+0000 7fb93af77640 1 -- 172.21.15.78:0/3113904182 >> 172.21.15.78:0/3113904182 conn(0x7fb9340fbb30 msgr2=0x7fb934104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:44.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.870+0000 7fb93af77640 1 -- 172.21.15.78:0/3113904182 shutdown_connections 2024-03-24T00:18:44.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.870+0000 7fb93af77640 1 -- 172.21.15.78:0/3113904182 wait complete. 2024-03-24T00:18:44.881 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-24T00:18:44.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.990+0000 7f79d8c75640 1 -- 172.21.15.78:0/1688714167 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79d4104b30 msgr2=0x7f79d4106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:44.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.990+0000 7f79d8c75640 1 --2- 172.21.15.78:0/1688714167 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79d4104b30 0x7f79d4106f20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f79bc009900 tx=0x7f79bc02bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:44.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.990+0000 7f79d8c75640 1 -- 172.21.15.78:0/1688714167 shutdown_connections 2024-03-24T00:18:44.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.990+0000 7f79d8c75640 1 --2- 172.21.15.78:0/1688714167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79d4107460 0x7f79d4109850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:44.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.990+0000 7f79d8c75640 1 --2- 172.21.15.78:0/1688714167 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79d4104b30 0x7f79d4106f20 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:44.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.990+0000 7f79d8c75640 1 --2- 172.21.15.78:0/1688714167 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79d4102200 0x7f79d41045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:44.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.990+0000 7f79d8c75640 1 -- 172.21.15.78:0/1688714167 >> 172.21.15.78:0/1688714167 conn(0x7f79d40fbb30 msgr2=0x7f79d40fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:44.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.990+0000 7f79d8c75640 1 -- 172.21.15.78:0/1688714167 shutdown_connections 2024-03-24T00:18:44.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.990+0000 7f79d8c75640 1 -- 172.21.15.78:0/1688714167 wait complete. 2024-03-24T00:18:44.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.991+0000 7f79d8c75640 1 Processor -- start 2024-03-24T00:18:44.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.991+0000 7f79d8c75640 1 -- start start 2024-03-24T00:18:44.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.991+0000 7f79d8c75640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79d4102200 0x7f79d4197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:44.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.991+0000 7f79d8c75640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79d4104b30 0x7f79d4197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:44.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.991+0000 7f79d8c75640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79d4107460 0x7f79d419eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:44.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.991+0000 7f79d8c75640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d410d3a0 con 0x7f79d4107460 2024-03-24T00:18:44.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.991+0000 7f79d8c75640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d410d220 con 0x7f79d4102200 2024-03-24T00:18:44.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.991+0000 7f79d8c75640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d410d520 con 0x7f79d4104b30 2024-03-24T00:18:44.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.991+0000 7f79d2d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79d4107460 0x7f79d419eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:44.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.991+0000 7f79d2d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79d4107460 0x7f79d419eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43172/0 (socket says 172.21.15.78:43172) 2024-03-24T00:18:44.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.991+0000 7f79d2d9d640 1 -- 172.21.15.78:0/3688776082 learned_addr learned my addr 172.21.15.78:0/3688776082 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:44.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.991+0000 7f79cbfff640 1 --2- 172.21.15.78:0/3688776082 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79d4104b30 0x7f79d4197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:44.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.992+0000 7f79d259c640 1 --2- 172.21.15.78:0/3688776082 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79d4102200 0x7f79d4197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:44.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.992+0000 7f79d2d9d640 1 -- 172.21.15.78:0/3688776082 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79d4104b30 msgr2=0x7f79d4197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:44.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.992+0000 7f79d2d9d640 1 --2- 172.21.15.78:0/3688776082 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79d4104b30 0x7f79d4197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:44.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.992+0000 7f79d2d9d640 1 -- 172.21.15.78:0/3688776082 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79d4102200 msgr2=0x7f79d4197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:44.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.992+0000 7f79d2d9d640 1 --2- 172.21.15.78:0/3688776082 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79d4102200 0x7f79d4197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:44.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.992+0000 7f79d2d9d640 1 -- 172.21.15.78:0/3688776082 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d419f660 con 0x7f79d4107460 2024-03-24T00:18:44.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.992+0000 7f79d2d9d640 1 --2- 172.21.15.78:0/3688776082 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79d4107460 0x7f79d419eff0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f79cc00cce0 tx=0x7f79cc00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:44.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.992+0000 7f79cb7fe640 1 -- 172.21.15.78:0/3688776082 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79cc00f450 con 0x7f79d4107460 2024-03-24T00:18:44.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.992+0000 7f79d8c75640 1 -- 172.21.15.78:0/3688776082 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79d419f950 con 0x7f79d4107460 2024-03-24T00:18:44.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.992+0000 7f79cb7fe640 1 -- 172.21.15.78:0/3688776082 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f79cc00fad0 con 0x7f79d4107460 2024-03-24T00:18:44.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.992+0000 7f79cb7fe640 1 -- 172.21.15.78:0/3688776082 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79cc018c10 con 0x7f79d4107460 2024-03-24T00:18:44.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.992+0000 7f79d8c75640 1 -- 172.21.15.78:0/3688776082 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79d419fe70 con 0x7f79d4107460 2024-03-24T00:18:44.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.993+0000 7f79d8c75640 1 -- 172.21.15.78:0/3688776082 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79d40fd620 con 0x7f79d4107460 2024-03-24T00:18:44.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.994+0000 7f79cb7fe640 1 -- 172.21.15.78:0/3688776082 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f79cc00f5f0 con 0x7f79d4107460 2024-03-24T00:18:44.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.995+0000 7f79cb7fe640 1 --2- 172.21.15.78:0/3688776082 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f79c00a2800 0x7f79c00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:44.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.995+0000 7f79d259c640 1 --2- 172.21.15.78:0/3688776082 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f79c00a2800 0x7f79c00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:44.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.995+0000 7f79cb7fe640 1 -- 172.21.15.78:0/3688776082 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f79cc0cf250 con 0x7f79d4107460 2024-03-24T00:18:44.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.995+0000 7f79d259c640 1 --2- 172.21.15.78:0/3688776082 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f79c00a2800 0x7f79c00a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f79c4009890 tx=0x7f79c4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:44.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:44.999+0000 7f79cb7fe640 1 -- 172.21.15.78:0/3688776082 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f79cc014030 con 0x7f79d4107460 2024-03-24T00:18:45.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:45.172+0000 7f79d8c75640 1 -- 172.21.15.78:0/3688776082 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f79d40622a0 con 0x7f79d4107460 2024-03-24T00:18:45.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:45 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3744748533' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-24T00:18:45.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:45 smithi078 ceph-mon[32429]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-24T00:18:45.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:45 smithi078 ceph-mon[32429]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-24T00:18:45.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:45 smithi078 ceph-mon[32429]: pgmap v286: 97 pgs: 12 creating+peering, 69 unknown, 16 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:45.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:45 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3113904182' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T00:18:45.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:45 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3113904182' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-24T00:18:45.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:45 smithi078 ceph-mon[32429]: osdmap e85: 12 total, 12 up, 12 in 2024-03-24T00:18:45.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:45 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3688776082' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-24T00:18:45.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:45 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3744748533' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-24T00:18:45.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:45 smithi107 ceph-mon[36485]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-24T00:18:45.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:45 smithi107 ceph-mon[36485]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-24T00:18:45.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:45 smithi107 ceph-mon[36485]: pgmap v286: 97 pgs: 12 creating+peering, 69 unknown, 16 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:45.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:45 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3113904182' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T00:18:45.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:45 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3113904182' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-24T00:18:45.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:45 smithi107 ceph-mon[36485]: osdmap e85: 12 total, 12 up, 12 in 2024-03-24T00:18:45.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:45 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3688776082' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-24T00:18:45.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:45 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3744748533' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-24T00:18:45.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:45 smithi106 ceph-mon[36505]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-24T00:18:45.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:45 smithi106 ceph-mon[36505]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-24T00:18:45.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:45 smithi106 ceph-mon[36505]: pgmap v286: 97 pgs: 12 creating+peering, 69 unknown, 16 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:45.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:45 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3113904182' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T00:18:45.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:45 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3113904182' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-24T00:18:45.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:45 smithi106 ceph-mon[36505]: osdmap e85: 12 total, 12 up, 12 in 2024-03-24T00:18:45.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:45 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3688776082' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-24T00:18:46.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:46 smithi078 ceph-mon[32429]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T00:18:46.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:46 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:18:46.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:46 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] up:active 2024-03-24T00:18:46.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:46 smithi078 ceph-mon[32429]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:18:46.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:46 smithi078 ceph-mon[32429]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:18:46.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:46 smithi078 ceph-mon[32429]: osdmap e86: 12 total, 12 up, 12 in 2024-03-24T00:18:46.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:46 smithi107 ceph-mon[36485]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T00:18:46.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:46 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:18:46.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:46 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] up:active 2024-03-24T00:18:46.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:46 smithi107 ceph-mon[36485]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:18:46.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:46 smithi107 ceph-mon[36485]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:18:46.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:46 smithi107 ceph-mon[36485]: osdmap e86: 12 total, 12 up, 12 in 2024-03-24T00:18:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:46 smithi106 ceph-mon[36505]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T00:18:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:46 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:18:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:46 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] up:active 2024-03-24T00:18:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:46 smithi106 ceph-mon[36505]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:18:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:46 smithi106 ceph-mon[36505]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:18:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:46 smithi106 ceph-mon[36505]: osdmap e86: 12 total, 12 up, 12 in 2024-03-24T00:18:46.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:46.871+0000 7f79cb7fe640 1 -- 172.21.15.78:0/3688776082 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) v1 ==== 228+0+0 (secure 0 0 0) 0x7f79cc09c2c0 con 0x7f79d4107460 2024-03-24T00:18:46.871 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data_ec' created 2024-03-24T00:18:46.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:46.874+0000 7f79d8c75640 1 -- 172.21.15.78:0/3688776082 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f79c00a2800 msgr2=0x7f79c00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:46.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:46.874+0000 7f79d8c75640 1 --2- 172.21.15.78:0/3688776082 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f79c00a2800 0x7f79c00a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f79c4009890 tx=0x7f79c4008040 comp rx=0 tx=0).stop 2024-03-24T00:18:46.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:46.874+0000 7f79d8c75640 1 -- 172.21.15.78:0/3688776082 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79d4107460 msgr2=0x7f79d419eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:46.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:46.874+0000 7f79d8c75640 1 --2- 172.21.15.78:0/3688776082 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79d4107460 0x7f79d419eff0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f79cc00cce0 tx=0x7f79cc00a620 comp rx=0 tx=0).stop 2024-03-24T00:18:46.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:46.874+0000 7f79d8c75640 1 -- 172.21.15.78:0/3688776082 shutdown_connections 2024-03-24T00:18:46.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:46.874+0000 7f79d8c75640 1 --2- 172.21.15.78:0/3688776082 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f79c00a2800 0x7f79c00a4cc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:46.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:46.874+0000 7f79d8c75640 1 --2- 172.21.15.78:0/3688776082 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79d4104b30 0x7f79d4197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:46.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:46.875+0000 7f79d8c75640 1 --2- 172.21.15.78:0/3688776082 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79d4107460 0x7f79d419eff0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:46.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:46.875+0000 7f79d8c75640 1 --2- 172.21.15.78:0/3688776082 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79d4102200 0x7f79d4197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:46.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:46.875+0000 7f79d8c75640 1 -- 172.21.15.78:0/3688776082 >> 172.21.15.78:0/3688776082 conn(0x7f79d40fbb30 msgr2=0x7f79d4102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:46.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:46.875+0000 7f79d8c75640 1 -- 172.21.15.78:0/3688776082 shutdown_connections 2024-03-24T00:18:46.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:46.875+0000 7f79d8c75640 1 -- 172.21.15.78:0/3688776082 wait complete. 2024-03-24T00:18:46.891 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-24T00:18:47.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.000+0000 7fe14bd1b640 1 -- 172.21.15.78:0/4000723383 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe144102160 msgr2=0x7fe1441089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:47.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.000+0000 7fe14bd1b640 1 --2- 172.21.15.78:0/4000723383 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe144102160 0x7fe1441089f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe13c009ca0 tx=0x7fe13c02c0f0 comp rx=0 tx=0).stop 2024-03-24T00:18:47.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.001+0000 7fe14bd1b640 1 -- 172.21.15.78:0/4000723383 shutdown_connections 2024-03-24T00:18:47.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.001+0000 7fe14bd1b640 1 --2- 172.21.15.78:0/4000723383 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe144102160 0x7fe1441089f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:47.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.001+0000 7fe14bd1b640 1 --2- 172.21.15.78:0/4000723383 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1441017a0 0x7fe144101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:47.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.001+0000 7fe14bd1b640 1 --2- 172.21.15.78:0/4000723383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1441005b0 0x7fe1441009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:47.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.001+0000 7fe14bd1b640 1 -- 172.21.15.78:0/4000723383 >> 172.21.15.78:0/4000723383 conn(0x7fe1440fbb40 msgr2=0x7fe1440fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:47.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.001+0000 7fe14bd1b640 1 -- 172.21.15.78:0/4000723383 shutdown_connections 2024-03-24T00:18:47.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.001+0000 7fe14bd1b640 1 -- 172.21.15.78:0/4000723383 wait complete. 2024-03-24T00:18:47.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.001+0000 7fe14bd1b640 1 Processor -- start 2024-03-24T00:18:47.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.001+0000 7fe14bd1b640 1 -- start start 2024-03-24T00:18:47.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.001+0000 7fe14bd1b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1441005b0 0x7fe14406a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:47.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.001+0000 7fe14bd1b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1441017a0 0x7fe14419c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:47.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.001+0000 7fe14bd1b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe144102160 0x7fe14419ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:47.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.001+0000 7fe14bd1b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe14410c230 con 0x7fe144102160 2024-03-24T00:18:47.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.001+0000 7fe14bd1b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe14410c0b0 con 0x7fe1441005b0 2024-03-24T00:18:47.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.001+0000 7fe14bd1b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe14410c3b0 con 0x7fe1441017a0 2024-03-24T00:18:47.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.001+0000 7fe14a2b8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe144102160 0x7fe14419ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:47.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.001+0000 7fe14a2b8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe144102160 0x7fe14419ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43198/0 (socket says 172.21.15.78:43198) 2024-03-24T00:18:47.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.001+0000 7fe14a2b8640 1 -- 172.21.15.78:0/1198515679 learned_addr learned my addr 172.21.15.78:0/1198515679 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:47.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.001+0000 7fe149ab7640 1 --2- 172.21.15.78:0/1198515679 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1441005b0 0x7fe14406a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:47.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.002+0000 7fe14a2b8640 1 -- 172.21.15.78:0/1198515679 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1441017a0 msgr2=0x7fe14419c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:47.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.002+0000 7fe14a2b8640 1 --2- 172.21.15.78:0/1198515679 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1441017a0 0x7fe14419c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:47.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.002+0000 7fe14a2b8640 1 -- 172.21.15.78:0/1198515679 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1441005b0 msgr2=0x7fe14406a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:47.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.002+0000 7fe14a2b8640 1 --2- 172.21.15.78:0/1198515679 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1441005b0 0x7fe14406a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:47.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.002+0000 7fe14a2b8640 1 -- 172.21.15.78:0/1198515679 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe14419f5a0 con 0x7fe144102160 2024-03-24T00:18:47.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.002+0000 7fe14a2b8640 1 --2- 172.21.15.78:0/1198515679 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe144102160 0x7fe14419ece0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe13c037040 tx=0x7fe13c032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:47.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.002+0000 7fe13affd640 1 -- 172.21.15.78:0/1198515679 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe13c041020 con 0x7fe144102160 2024-03-24T00:18:47.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.002+0000 7fe14bd1b640 1 -- 172.21.15.78:0/1198515679 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe14419f830 con 0x7fe144102160 2024-03-24T00:18:47.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.002+0000 7fe13affd640 1 -- 172.21.15.78:0/1198515679 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe13c034d90 con 0x7fe144102160 2024-03-24T00:18:47.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.002+0000 7fe13affd640 1 -- 172.21.15.78:0/1198515679 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe13c039040 con 0x7fe144102160 2024-03-24T00:18:47.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.002+0000 7fe14bd1b640 1 -- 172.21.15.78:0/1198515679 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe14419fd70 con 0x7fe144102160 2024-03-24T00:18:47.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.003+0000 7fe14bd1b640 1 -- 172.21.15.78:0/1198515679 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1441078d0 con 0x7fe144102160 2024-03-24T00:18:47.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.004+0000 7fe13affd640 1 -- 172.21.15.78:0/1198515679 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe13c044020 con 0x7fe144102160 2024-03-24T00:18:47.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.005+0000 7fe13affd640 1 --2- 172.21.15.78:0/1198515679 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe1300a2800 0x7fe1300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:47.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.005+0000 7fe149ab7640 1 --2- 172.21.15.78:0/1198515679 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe1300a2800 0x7fe1300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:47.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.005+0000 7fe13affd640 1 -- 172.21.15.78:0/1198515679 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8770+0+0 (secure 0 0 0) 0x7fe13c0f1020 con 0x7fe144102160 2024-03-24T00:18:47.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.005+0000 7fe149ab7640 1 --2- 172.21.15.78:0/1198515679 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe1300a2800 0x7fe1300a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe140005fd0 tx=0x7fe14000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:47.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.009+0000 7fe13affd640 1 -- 172.21.15.78:0/1198515679 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe13c04b020 con 0x7fe144102160 2024-03-24T00:18:47.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.183+0000 7fe14bd1b640 1 -- 172.21.15.78:0/1198515679 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7fe1440622a0 con 0x7fe144102160 2024-03-24T00:18:47.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:47 smithi078 ceph-mon[32429]: pgmap v289: 97 pgs: 12 creating+peering, 55 unknown, 30 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:47.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:47 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3688776082' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-24T00:18:47.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:47 smithi078 ceph-mon[32429]: osdmap e87: 12 total, 12 up, 12 in 2024-03-24T00:18:47.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:47 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1198515679' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-24T00:18:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:47 smithi107 ceph-mon[36485]: pgmap v289: 97 pgs: 12 creating+peering, 55 unknown, 30 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:47 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3688776082' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-24T00:18:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:47 smithi107 ceph-mon[36485]: osdmap e87: 12 total, 12 up, 12 in 2024-03-24T00:18:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:47 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1198515679' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-24T00:18:47.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:47 smithi106 ceph-mon[36505]: pgmap v289: 97 pgs: 12 creating+peering, 55 unknown, 30 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:18:47.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:47 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3688776082' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-24T00:18:47.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:47 smithi106 ceph-mon[36505]: osdmap e87: 12 total, 12 up, 12 in 2024-03-24T00:18:47.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:47 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1198515679' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-24T00:18:47.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.873+0000 7fe13affd640 1 -- 172.21.15.78:0/1198515679 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) v1 ==== 169+0+0 (secure 0 0 0) 0x7fe13c0bdea0 con 0x7fe144102160 2024-03-24T00:18:47.873 INFO:teuthology.orchestra.run.smithi078.stderr:set pool 4 allow_ec_overwrites to true 2024-03-24T00:18:47.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.875+0000 7fe14bd1b640 1 -- 172.21.15.78:0/1198515679 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe1300a2800 msgr2=0x7fe1300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:47.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.875+0000 7fe14bd1b640 1 --2- 172.21.15.78:0/1198515679 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe1300a2800 0x7fe1300a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe140005fd0 tx=0x7fe14000b390 comp rx=0 tx=0).stop 2024-03-24T00:18:47.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.875+0000 7fe14bd1b640 1 -- 172.21.15.78:0/1198515679 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe144102160 msgr2=0x7fe14419ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:47.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.875+0000 7fe14bd1b640 1 --2- 172.21.15.78:0/1198515679 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe144102160 0x7fe14419ece0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe13c037040 tx=0x7fe13c032f20 comp rx=0 tx=0).stop 2024-03-24T00:18:47.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.875+0000 7fe14bd1b640 1 -- 172.21.15.78:0/1198515679 shutdown_connections 2024-03-24T00:18:47.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.875+0000 7fe14bd1b640 1 --2- 172.21.15.78:0/1198515679 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe1300a2800 0x7fe1300a4cc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:47.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.875+0000 7fe14bd1b640 1 --2- 172.21.15.78:0/1198515679 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1441017a0 0x7fe14419c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:47.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.875+0000 7fe14bd1b640 1 --2- 172.21.15.78:0/1198515679 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe144102160 0x7fe14419ece0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:47.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.875+0000 7fe14bd1b640 1 --2- 172.21.15.78:0/1198515679 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1441005b0 0x7fe14406a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:47.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.875+0000 7fe14bd1b640 1 -- 172.21.15.78:0/1198515679 >> 172.21.15.78:0/1198515679 conn(0x7fe1440fbb40 msgr2=0x7fe144104aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:47.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.876+0000 7fe14bd1b640 1 -- 172.21.15.78:0/1198515679 shutdown_connections 2024-03-24T00:18:47.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:47.876+0000 7fe14bd1b640 1 -- 172.21.15.78:0/1198515679 wait complete. 2024-03-24T00:18:47.895 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-24T00:18:48.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.002+0000 7fa365b3b640 1 -- 172.21.15.78:0/1803053102 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa360102140 msgr2=0x7fa360104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:48.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.002+0000 7fa365b3b640 1 --2- 172.21.15.78:0/1803053102 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa360102140 0x7fa360104530 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa350009900 tx=0x7fa35002bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:48.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.002+0000 7fa365b3b640 1 -- 172.21.15.78:0/1803053102 shutdown_connections 2024-03-24T00:18:48.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.002+0000 7fa365b3b640 1 --2- 172.21.15.78:0/1803053102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3601073a0 0x7fa3601097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:48.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.002+0000 7fa365b3b640 1 --2- 172.21.15.78:0/1803053102 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa360104a70 0x7fa360106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:48.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.002+0000 7fa365b3b640 1 --2- 172.21.15.78:0/1803053102 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa360102140 0x7fa360104530 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:48.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.002+0000 7fa365b3b640 1 -- 172.21.15.78:0/1803053102 >> 172.21.15.78:0/1803053102 conn(0x7fa3600fbb50 msgr2=0x7fa3600fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:48.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.002+0000 7fa365b3b640 1 -- 172.21.15.78:0/1803053102 shutdown_connections 2024-03-24T00:18:48.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.002+0000 7fa365b3b640 1 -- 172.21.15.78:0/1803053102 wait complete. 2024-03-24T00:18:48.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.003+0000 7fa365b3b640 1 Processor -- start 2024-03-24T00:18:48.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.003+0000 7fa365b3b640 1 -- start start 2024-03-24T00:18:48.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.003+0000 7fa365b3b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa360102140 0x7fa360101ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:48.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.003+0000 7fa365b3b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa360104a70 0x7fa3601000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:48.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.003+0000 7fa365b3b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3601073a0 0x7fa360100630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:48.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.003+0000 7fa365b3b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36010d370 con 0x7fa3601073a0 2024-03-24T00:18:48.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.003+0000 7fa365b3b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36010d1f0 con 0x7fa360104a70 2024-03-24T00:18:48.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.003+0000 7fa365b3b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36010d4f0 con 0x7fa360102140 2024-03-24T00:18:48.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.003+0000 7fa35ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3601073a0 0x7fa360100630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:48.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.003+0000 7fa35ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3601073a0 0x7fa360100630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43204/0 (socket says 172.21.15.78:43204) 2024-03-24T00:18:48.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.003+0000 7fa35ffff640 1 -- 172.21.15.78:0/3254320105 learned_addr learned my addr 172.21.15.78:0/3254320105 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:48.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.003+0000 7fa35effd640 1 --2- 172.21.15.78:0/3254320105 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa360104a70 0x7fa3601000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:48.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.003+0000 7fa35f7fe640 1 --2- 172.21.15.78:0/3254320105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa360102140 0x7fa360101ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:48.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.003+0000 7fa35ffff640 1 -- 172.21.15.78:0/3254320105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa360102140 msgr2=0x7fa360101ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:48.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.003+0000 7fa35ffff640 1 --2- 172.21.15.78:0/3254320105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa360102140 0x7fa360101ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:48.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.003+0000 7fa35ffff640 1 -- 172.21.15.78:0/3254320105 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa360104a70 msgr2=0x7fa3601000f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:48.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.003+0000 7fa35ffff640 1 --2- 172.21.15.78:0/3254320105 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa360104a70 0x7fa3601000f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:48.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.003+0000 7fa35ffff640 1 -- 172.21.15.78:0/3254320105 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa360100ec0 con 0x7fa3601073a0 2024-03-24T00:18:48.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.003+0000 7fa35f7fe640 1 --2- 172.21.15.78:0/3254320105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa360102140 0x7fa360101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:18:48.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.004+0000 7fa35ffff640 1 --2- 172.21.15.78:0/3254320105 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3601073a0 0x7fa360100630 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa35800eaa0 tx=0x7fa35800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:48.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.004+0000 7fa35cff9640 1 -- 172.21.15.78:0/3254320105 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa35800cdb0 con 0x7fa3601073a0 2024-03-24T00:18:48.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.004+0000 7fa35cff9640 1 -- 172.21.15.78:0/3254320105 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa358005020 con 0x7fa3601073a0 2024-03-24T00:18:48.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.004+0000 7fa365b3b640 1 -- 172.21.15.78:0/3254320105 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3601a1b40 con 0x7fa3601073a0 2024-03-24T00:18:48.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.004+0000 7fa35cff9640 1 -- 172.21.15.78:0/3254320105 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3580116b0 con 0x7fa3601073a0 2024-03-24T00:18:48.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.004+0000 7fa365b3b640 1 -- 172.21.15.78:0/3254320105 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3601a2090 con 0x7fa3601073a0 2024-03-24T00:18:48.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.005+0000 7fa365b3b640 1 -- 172.21.15.78:0/3254320105 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3600fd640 con 0x7fa3601073a0 2024-03-24T00:18:48.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.006+0000 7fa35cff9640 1 -- 172.21.15.78:0/3254320105 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa358005560 con 0x7fa3601073a0 2024-03-24T00:18:48.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.006+0000 7fa35cff9640 1 --2- 172.21.15.78:0/3254320105 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa3480a2800 0x7fa3480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:48.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.006+0000 7fa35f7fe640 1 --2- 172.21.15.78:0/3254320105 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa3480a2800 0x7fa3480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:48.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.006+0000 7fa35cff9640 1 -- 172.21.15.78:0/3254320105 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8770+0+0 (secure 0 0 0) 0x7fa3580cef30 con 0x7fa3601073a0 2024-03-24T00:18:48.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.007+0000 7fa35f7fe640 1 --2- 172.21.15.78:0/3254320105 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa3480a2800 0x7fa3480a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa350006010 tx=0x7fa35002c8a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:48.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.011+0000 7fa35cff9640 1 -- 172.21.15.78:0/3254320105 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa358016030 con 0x7fa3601073a0 2024-03-24T00:18:48.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.223+0000 7fa365b3b640 1 -- 172.21.15.78:0/3254320105 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7fa360062340 con 0x7fa3601073a0 2024-03-24T00:18:48.652 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 24 00:18:48 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-b[66985]: 2024-03-24T00:18:48.392+0000 7f978f4ee700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:18:48.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.879+0000 7fa35cff9640 1 -- 172.21.15.78:0/3254320105 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) v1 ==== 137+0+0 (secure 0 0 0) 0x7fa3580bb030 con 0x7fa3601073a0 2024-03-24T00:18:48.879 INFO:teuthology.orchestra.run.smithi078.stderr:added data pool 4 to fsmap 2024-03-24T00:18:48.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.882+0000 7fa365b3b640 1 -- 172.21.15.78:0/3254320105 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa3480a2800 msgr2=0x7fa3480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:48.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.882+0000 7fa365b3b640 1 --2- 172.21.15.78:0/3254320105 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa3480a2800 0x7fa3480a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa350006010 tx=0x7fa35002c8a0 comp rx=0 tx=0).stop 2024-03-24T00:18:48.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.882+0000 7fa365b3b640 1 -- 172.21.15.78:0/3254320105 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3601073a0 msgr2=0x7fa360100630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:48.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.882+0000 7fa365b3b640 1 --2- 172.21.15.78:0/3254320105 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3601073a0 0x7fa360100630 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa35800eaa0 tx=0x7fa35800ef60 comp rx=0 tx=0).stop 2024-03-24T00:18:48.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.882+0000 7fa365b3b640 1 -- 172.21.15.78:0/3254320105 shutdown_connections 2024-03-24T00:18:48.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.882+0000 7fa365b3b640 1 --2- 172.21.15.78:0/3254320105 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa3480a2800 0x7fa3480a4cc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:48.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.882+0000 7fa365b3b640 1 --2- 172.21.15.78:0/3254320105 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3601073a0 0x7fa360100630 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:48.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.882+0000 7fa365b3b640 1 --2- 172.21.15.78:0/3254320105 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa360104a70 0x7fa3601000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:48.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.882+0000 7fa365b3b640 1 --2- 172.21.15.78:0/3254320105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa360102140 0x7fa360101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:48.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.882+0000 7fa365b3b640 1 -- 172.21.15.78:0/3254320105 >> 172.21.15.78:0/3254320105 conn(0x7fa3600fbb50 msgr2=0x7fa360102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:48.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.882+0000 7fa365b3b640 1 -- 172.21.15.78:0/3254320105 shutdown_connections 2024-03-24T00:18:48.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.882+0000 7fa365b3b640 1 -- 172.21.15.78:0/3254320105 wait complete. 2024-03-24T00:18:48.893 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:18:48.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.995+0000 7fd121b1b640 1 -- 172.21.15.78:0/3465692928 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd11c104a70 msgr2=0x7fd11c106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:48.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.995+0000 7fd121b1b640 1 --2- 172.21.15.78:0/3465692928 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd11c104a70 0x7fd11c106e60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd104009900 tx=0x7fd10402bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:48.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.996+0000 7fd121b1b640 1 -- 172.21.15.78:0/3465692928 shutdown_connections 2024-03-24T00:18:48.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.996+0000 7fd121b1b640 1 --2- 172.21.15.78:0/3465692928 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd11c1073a0 0x7fd11c1097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:48.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.996+0000 7fd121b1b640 1 --2- 172.21.15.78:0/3465692928 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd11c104a70 0x7fd11c106e60 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:48.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.996+0000 7fd121b1b640 1 --2- 172.21.15.78:0/3465692928 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd11c102140 0x7fd11c104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:48.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.996+0000 7fd121b1b640 1 -- 172.21.15.78:0/3465692928 >> 172.21.15.78:0/3465692928 conn(0x7fd11c0fbb30 msgr2=0x7fd11c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:48.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.996+0000 7fd121b1b640 1 -- 172.21.15.78:0/3465692928 shutdown_connections 2024-03-24T00:18:48.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.996+0000 7fd121b1b640 1 -- 172.21.15.78:0/3465692928 wait complete. 2024-03-24T00:18:48.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.996+0000 7fd121b1b640 1 Processor -- start 2024-03-24T00:18:48.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.996+0000 7fd121b1b640 1 -- start start 2024-03-24T00:18:48.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.996+0000 7fd121b1b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd11c102140 0x7fd11c197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:49.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.996+0000 7fd121b1b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd11c104a70 0x7fd11c197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:49.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.996+0000 7fd121b1b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd11c1073a0 0x7fd11c19efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:49.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.996+0000 7fd121b1b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd11c10d350 con 0x7fd11c102140 2024-03-24T00:18:49.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.996+0000 7fd121b1b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd11c10d1d0 con 0x7fd11c1073a0 2024-03-24T00:18:49.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.996+0000 7fd121b1b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd11c10d4d0 con 0x7fd11c104a70 2024-03-24T00:18:49.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.997+0000 7fd11b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd11c102140 0x7fd11c197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:49.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.997+0000 7fd11b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd11c102140 0x7fd11c197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43236/0 (socket says 172.21.15.78:43236) 2024-03-24T00:18:49.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.997+0000 7fd11b7fe640 1 -- 172.21.15.78:0/432182890 learned_addr learned my addr 172.21.15.78:0/432182890 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:49.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.997+0000 7fd11bfff640 1 --2- 172.21.15.78:0/432182890 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd11c1073a0 0x7fd11c19efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:49.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.997+0000 7fd112ffd640 1 --2- 172.21.15.78:0/432182890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd11c104a70 0x7fd11c197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:49.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.997+0000 7fd11b7fe640 1 -- 172.21.15.78:0/432182890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd11c104a70 msgr2=0x7fd11c197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:49.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.997+0000 7fd11b7fe640 1 --2- 172.21.15.78:0/432182890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd11c104a70 0x7fd11c197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:49.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.997+0000 7fd11b7fe640 1 -- 172.21.15.78:0/432182890 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd11c1073a0 msgr2=0x7fd11c19efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:49.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.997+0000 7fd11b7fe640 1 --2- 172.21.15.78:0/432182890 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd11c1073a0 0x7fd11c19efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:49.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.997+0000 7fd11b7fe640 1 -- 172.21.15.78:0/432182890 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd11c19f640 con 0x7fd11c102140 2024-03-24T00:18:49.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.997+0000 7fd11b7fe640 1 --2- 172.21.15.78:0/432182890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd11c102140 0x7fd11c197a20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd10c00d770 tx=0x7fd10c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:49.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.997+0000 7fd1197fa640 1 -- 172.21.15.78:0/432182890 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10c017070 con 0x7fd11c102140 2024-03-24T00:18:49.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.997+0000 7fd1197fa640 1 -- 172.21.15.78:0/432182890 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd10c005500 con 0x7fd11c102140 2024-03-24T00:18:49.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.997+0000 7fd121b1b640 1 -- 172.21.15.78:0/432182890 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd11c19f930 con 0x7fd11c102140 2024-03-24T00:18:49.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.997+0000 7fd1197fa640 1 -- 172.21.15.78:0/432182890 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10c004e00 con 0x7fd11c102140 2024-03-24T00:18:49.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.997+0000 7fd121b1b640 1 -- 172.21.15.78:0/432182890 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd11c19fe80 con 0x7fd11c102140 2024-03-24T00:18:49.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.998+0000 7fd121b1b640 1 -- 172.21.15.78:0/432182890 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd11c0fd620 con 0x7fd11c102140 2024-03-24T00:18:49.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:48.999+0000 7fd1197fa640 1 -- 172.21.15.78:0/432182890 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd10c004fa0 con 0x7fd11c102140 2024-03-24T00:18:49.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.000+0000 7fd1197fa640 1 --2- 172.21.15.78:0/432182890 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd0fc0a2880 0x7fd0fc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:49.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.000+0000 7fd112ffd640 1 --2- 172.21.15.78:0/432182890 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd0fc0a2880 0x7fd0fc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:49.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.000+0000 7fd1197fa640 1 -- 172.21.15.78:0/432182890 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fd10c0ceaa0 con 0x7fd11c102140 2024-03-24T00:18:49.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.000+0000 7fd112ffd640 1 --2- 172.21.15.78:0/432182890 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd0fc0a2880 0x7fd0fc0a4d40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd10402c8e0 tx=0x7fd104005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:49.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.003+0000 7fd1197fa640 1 -- 172.21.15.78:0/432182890 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd10c09b980 con 0x7fd11c102140 2024-03-24T00:18:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:48 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1198515679' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-24T00:18:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:48 smithi106 ceph-mon[36505]: osdmap e88: 12 total, 12 up, 12 in 2024-03-24T00:18:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:48 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3254320105' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-24T00:18:49.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:48 smithi106 ceph-mon[36505]: pgmap v292: 129 pgs: 18 creating+peering, 18 unknown, 93 active+clean; 577 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 7 op/s 2024-03-24T00:18:49.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:48 smithi106 ceph-mon[36505]: osdmap e89: 12 total, 12 up, 12 in 2024-03-24T00:18:49.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.215+0000 7fd121b1b640 1 -- 172.21.15.78:0/432182890 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd11c02d2e0 con 0x7fd11c102140 2024-03-24T00:18:49.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.216+0000 7fd1197fa640 1 -- 172.21.15.78:0/432182890 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+11333 (secure 0 0 0) 0x7fd10c09b280 con 0x7fd11c102140 2024-03-24T00:18:49.215 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:18:49.216 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1416685123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1416685123},{"type":"v1","addr":"172.21.15.78:6835","nonce":1416685123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1071274623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1071274623},{"type":"v1","addr":"172.21.15.78:6837","nonce":1071274623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2758357551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2758357551},{"type":"v1","addr":"172.21.15.78:6839","nonce":2758357551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/3424505910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":3424505910},{"type":"v1","addr":"172.21.15.78:6841","nonce":3424505910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/3086318443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3086318443},{"type":"v1","addr":"172.21.15.107:6833","nonce":3086318443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/2367018868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2367018868},{"type":"v1","addr":"172.21.15.106:6833","nonce":2367018868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/2231611477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":2231611477},{"type":"v1","addr":"172.21.15.106:6835","nonce":2231611477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/108936439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":108936439},{"type":"v1","addr":"172.21.15.107:6835","nonce":108936439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2939571540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2939571540},{"type":"v1","addr":"172.21.15.107:6837","nonce":2939571540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/2090968633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":2090968633},{"type":"v1","addr":"172.21.15.106:6837","nonce":2090968633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/117119462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":117119462},{"type":"v1","addr":"172.21.15.107:6839","nonce":117119462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:18:43.592215+0000","modified":"2024-03-24T00:18:48.224616+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6839/2482071304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2482071304},{"type":"v1","addr":"172.21.15.106:6839","nonce":2482071304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24500,"qdb_cluster":[24500]},"id":1}]} 2024-03-24T00:18:49.216 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 17 2024-03-24T00:18:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.218+0000 7fd121b1b640 1 -- 172.21.15.78:0/432182890 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd0fc0a2880 msgr2=0x7fd0fc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.218+0000 7fd121b1b640 1 --2- 172.21.15.78:0/432182890 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd0fc0a2880 0x7fd0fc0a4d40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd10402c8e0 tx=0x7fd104005fd0 comp rx=0 tx=0).stop 2024-03-24T00:18:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.218+0000 7fd121b1b640 1 -- 172.21.15.78:0/432182890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd11c102140 msgr2=0x7fd11c197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.218+0000 7fd121b1b640 1 --2- 172.21.15.78:0/432182890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd11c102140 0x7fd11c197a20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd10c00d770 tx=0x7fd10c00dc30 comp rx=0 tx=0).stop 2024-03-24T00:18:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.218+0000 7fd121b1b640 1 -- 172.21.15.78:0/432182890 shutdown_connections 2024-03-24T00:18:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.218+0000 7fd121b1b640 1 --2- 172.21.15.78:0/432182890 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd0fc0a2880 0x7fd0fc0a4d40 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.218+0000 7fd121b1b640 1 --2- 172.21.15.78:0/432182890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd11c104a70 0x7fd11c197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.218+0000 7fd121b1b640 1 --2- 172.21.15.78:0/432182890 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd11c1073a0 0x7fd11c19efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.218+0000 7fd121b1b640 1 --2- 172.21.15.78:0/432182890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd11c102140 0x7fd11c197a20 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.218+0000 7fd121b1b640 1 -- 172.21.15.78:0/432182890 >> 172.21.15.78:0/432182890 conn(0x7fd11c0fbb30 msgr2=0x7fd11c102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.218+0000 7fd121b1b640 1 -- 172.21.15.78:0/432182890 shutdown_connections 2024-03-24T00:18:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.218+0000 7fd121b1b640 1 -- 172.21.15.78:0/432182890 wait complete. 2024-03-24T00:18:49.227 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:48 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1198515679' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-24T00:18:49.227 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:48 smithi078 ceph-mon[32429]: osdmap e88: 12 total, 12 up, 12 in 2024-03-24T00:18:49.227 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:48 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3254320105' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-24T00:18:49.227 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:48 smithi078 ceph-mon[32429]: pgmap v292: 129 pgs: 18 creating+peering, 18 unknown, 93 active+clean; 577 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 7 op/s 2024-03-24T00:18:49.228 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:48 smithi078 ceph-mon[32429]: osdmap e89: 12 total, 12 up, 12 in 2024-03-24T00:18:49.229 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:18:49.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.330+0000 7f1f9e0b8640 1 -- 172.21.15.78:0/3527589956 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f981005c0 msgr2=0x7f1f981009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:49.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.330+0000 7f1f9e0b8640 1 --2- 172.21.15.78:0/3527589956 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f981005c0 0x7f1f981009e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1f88009900 tx=0x7f1f8802bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:49.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.330+0000 7f1f9e0b8640 1 -- 172.21.15.78:0/3527589956 shutdown_connections 2024-03-24T00:18:49.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.330+0000 7f1f9e0b8640 1 --2- 172.21.15.78:0/3527589956 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f98102170 0x7f1f98108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:49.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.330+0000 7f1f9e0b8640 1 --2- 172.21.15.78:0/3527589956 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1f981017b0 0x7f1f98101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:49.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.330+0000 7f1f9e0b8640 1 --2- 172.21.15.78:0/3527589956 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f981005c0 0x7f1f981009e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:49.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.330+0000 7f1f9e0b8640 1 -- 172.21.15.78:0/3527589956 >> 172.21.15.78:0/3527589956 conn(0x7f1f980fbb30 msgr2=0x7f1f980fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:49.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.330+0000 7f1f9e0b8640 1 -- 172.21.15.78:0/3527589956 shutdown_connections 2024-03-24T00:18:49.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.330+0000 7f1f9e0b8640 1 -- 172.21.15.78:0/3527589956 wait complete. 2024-03-24T00:18:49.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.331+0000 7f1f9e0b8640 1 Processor -- start 2024-03-24T00:18:49.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.331+0000 7f1f9e0b8640 1 -- start start 2024-03-24T00:18:49.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.331+0000 7f1f9e0b8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f981005c0 0x7f1f98199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:49.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.331+0000 7f1f9e0b8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f981017b0 0x7f1f9819a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:49.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.331+0000 7f1f9e0b8640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1f98102170 0x7f1f981a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:49.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.331+0000 7f1f9e0b8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9810d360 con 0x7f1f981005c0 2024-03-24T00:18:49.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.331+0000 7f1f9e0b8640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9810d1e0 con 0x7f1f98102170 2024-03-24T00:18:49.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.331+0000 7f1f9e0b8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9810d4e0 con 0x7f1f981017b0 2024-03-24T00:18:49.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.331+0000 7f1f977fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f981005c0 0x7f1f98199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:49.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.331+0000 7f1f977fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f981005c0 0x7f1f98199c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43252/0 (socket says 172.21.15.78:43252) 2024-03-24T00:18:49.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.331+0000 7f1f977fe640 1 -- 172.21.15.78:0/398697371 learned_addr learned my addr 172.21.15.78:0/398697371 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:49.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.331+0000 7f1f97fff640 1 --2- 172.21.15.78:0/398697371 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1f98102170 0x7f1f981a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:49.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.332+0000 7f1f977fe640 1 -- 172.21.15.78:0/398697371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f981017b0 msgr2=0x7f1f9819a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:49.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.332+0000 7f1f977fe640 1 --2- 172.21.15.78:0/398697371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f981017b0 0x7f1f9819a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:49.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.332+0000 7f1f977fe640 1 -- 172.21.15.78:0/398697371 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1f98102170 msgr2=0x7f1f981a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:49.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.332+0000 7f1f977fe640 1 --2- 172.21.15.78:0/398697371 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1f98102170 0x7f1f981a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:49.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.332+0000 7f1f977fe640 1 -- 172.21.15.78:0/398697371 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f981a1850 con 0x7f1f981005c0 2024-03-24T00:18:49.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.332+0000 7f1f977fe640 1 --2- 172.21.15.78:0/398697371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f981005c0 0x7f1f98199c40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1f88000c00 tx=0x7f1f88005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:49.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.332+0000 7f1f94ff9640 1 -- 172.21.15.78:0/398697371 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f88041070 con 0x7f1f981005c0 2024-03-24T00:18:49.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.332+0000 7f1f9e0b8640 1 -- 172.21.15.78:0/398697371 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f981a1ae0 con 0x7f1f981005c0 2024-03-24T00:18:49.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.332+0000 7f1f94ff9640 1 -- 172.21.15.78:0/398697371 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f880044d0 con 0x7f1f981005c0 2024-03-24T00:18:49.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.332+0000 7f1f94ff9640 1 -- 172.21.15.78:0/398697371 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f88034440 con 0x7f1f981005c0 2024-03-24T00:18:49.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.332+0000 7f1f9e0b8640 1 -- 172.21.15.78:0/398697371 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f981a1fd0 con 0x7f1f981005c0 2024-03-24T00:18:49.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.333+0000 7f1f9e0b8640 1 -- 172.21.15.78:0/398697371 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f981a2200 con 0x7f1f981005c0 2024-03-24T00:18:49.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.334+0000 7f1f94ff9640 1 -- 172.21.15.78:0/398697371 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1f88044070 con 0x7f1f981005c0 2024-03-24T00:18:49.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.335+0000 7f1f94ff9640 1 --2- 172.21.15.78:0/398697371 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f1f800a2800 0x7f1f800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:49.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.335+0000 7f1f96ffd640 1 --2- 172.21.15.78:0/398697371 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f1f800a2800 0x7f1f800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:49.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.335+0000 7f1f94ff9640 1 -- 172.21.15.78:0/398697371 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f1f880f08d0 con 0x7f1f981005c0 2024-03-24T00:18:49.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.335+0000 7f1f96ffd640 1 --2- 172.21.15.78:0/398697371 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f1f800a2800 0x7f1f800a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c005d00 tx=0x7f1f8c005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:49.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.339+0000 7f1f94ff9640 1 -- 172.21.15.78:0/398697371 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1f880bd730 con 0x7f1f981005c0 2024-03-24T00:18:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:48 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1198515679' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-24T00:18:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:48 smithi107 ceph-mon[36485]: osdmap e88: 12 total, 12 up, 12 in 2024-03-24T00:18:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:48 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3254320105' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-24T00:18:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:48 smithi107 ceph-mon[36485]: pgmap v292: 129 pgs: 18 creating+peering, 18 unknown, 93 active+clean; 577 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 7 op/s 2024-03-24T00:18:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:48 smithi107 ceph-mon[36485]: osdmap e89: 12 total, 12 up, 12 in 2024-03-24T00:18:49.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.517+0000 7f1f9e0b8640 1 -- 172.21.15.78:0/398697371 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f981a29a0 con 0x7f1f981005c0 2024-03-24T00:18:49.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.518+0000 7f1f94ff9640 1 -- 172.21.15.78:0/398697371 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24129 (secure 0 0 0) 0x7f1f880bd030 con 0x7f1f981005c0 2024-03-24T00:18:49.519 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:18:49.519 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":89,"fsid":"c7e2806a-e972-11ee-95cd-87774f69a715","created":"2024-03-24T00:10:44.134563+0000","modified":"2024-03-24T00:18:48.224440+0000","last_up_change":"2024-03-24T00:17:50.237686+0000","last_in_change":"2024-03-24T00:17:29.788699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:14:04.534391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:18:42.178418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:18:42.800424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:18:46.112800+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b950388a-c437-426b-b703-3f8ae821dd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6803","nonce":1816879397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6805","nonce":1816879397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6809","nonce":1816879397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6807","nonce":1816879397}]},"public_addr":"172.21.15.78:6803/1816879397","cluster_addr":"172.21.15.78:6805/1816879397","heartbeat_back_addr":"172.21.15.78:6809/1816879397","heartbeat_front_addr":"172.21.15.78:6807/1816879397","state":["exists","up"]},{"osd":1,"uuid":"b8d64714-dd65-400f-b3f4-673ab14e1365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6801","nonce":1647915612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6803","nonce":1647915612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6807","nonce":1647915612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6805","nonce":1647915612}]},"public_addr":"172.21.15.106:6801/1647915612","cluster_addr":"172.21.15.106:6803/1647915612","heartbeat_back_addr":"172.21.15.106:6807/1647915612","heartbeat_front_addr":"172.21.15.106:6805/1647915612","state":["exists","up"]},{"osd":2,"uuid":"fcfc4dd1-aaf8-4131-a023-5e832212158d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6801","nonce":2010770435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6803","nonce":2010770435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6807","nonce":2010770435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6805","nonce":2010770435}]},"public_addr":"172.21.15.107:6801/2010770435","cluster_addr":"172.21.15.107:6803/2010770435","heartbeat_back_addr":"172.21.15.107:6807/2010770435","heartbeat_front_addr":"172.21.15.107:6805/2010770435","state":["exists","up"]},{"osd":3,"uuid":"47c718ab-a0f4-4e52-83c5-b89386b262d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6811","nonce":1912251346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6813","nonce":1912251346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6817","nonce":1912251346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6815","nonce":1912251346}]},"public_addr":"172.21.15.78:6811/1912251346","cluster_addr":"172.21.15.78:6813/1912251346","heartbeat_back_addr":"172.21.15.78:6817/1912251346","heartbeat_front_addr":"172.21.15.78:6815/1912251346","state":["exists","up"]},{"osd":4,"uuid":"7dad64dd-677e-4c1c-9638-e4f4f2195983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6809","nonce":374641205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6811","nonce":374641205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6815","nonce":374641205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6813","nonce":374641205}]},"public_addr":"172.21.15.106:6809/374641205","cluster_addr":"172.21.15.106:6811/374641205","heartbeat_back_addr":"172.21.15.106:6815/374641205","heartbeat_front_addr":"172.21.15.106:6813/374641205","state":["exists","up"]},{"osd":5,"uuid":"bbd524e5-328d-4309-8877-1149cbe110dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6809","nonce":3272806073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6811","nonce":3272806073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6815","nonce":3272806073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6813","nonce":3272806073}]},"public_addr":"172.21.15.107:6809/3272806073","cluster_addr":"172.21.15.107:6811/3272806073","heartbeat_back_addr":"172.21.15.107:6815/3272806073","heartbeat_front_addr":"172.21.15.107:6813/3272806073","state":["exists","up"]},{"osd":6,"uuid":"5696389a-918f-4ea8-b20d-4e2a0d4cb3bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6819","nonce":3091148508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6821","nonce":3091148508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6825","nonce":3091148508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6823","nonce":3091148508}]},"public_addr":"172.21.15.78:6819/3091148508","cluster_addr":"172.21.15.78:6821/3091148508","heartbeat_back_addr":"172.21.15.78:6825/3091148508","heartbeat_front_addr":"172.21.15.78:6823/3091148508","state":["exists","up"]},{"osd":7,"uuid":"f6b764e8-ed4e-4c9a-a85b-94af6dc6b924","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6817","nonce":4262103080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6819","nonce":4262103080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6823","nonce":4262103080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6821","nonce":4262103080}]},"public_addr":"172.21.15.106:6817/4262103080","cluster_addr":"172.21.15.106:6819/4262103080","heartbeat_back_addr":"172.21.15.106:6823/4262103080","heartbeat_front_addr":"172.21.15.106:6821/4262103080","state":["exists","up"]},{"osd":8,"uuid":"ba16039f-8093-4fa0-87ad-e28b7d142b23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6817","nonce":4032799356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6819","nonce":4032799356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6823","nonce":4032799356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6821","nonce":4032799356}]},"public_addr":"172.21.15.107:6817/4032799356","cluster_addr":"172.21.15.107:6819/4032799356","heartbeat_back_addr":"172.21.15.107:6823/4032799356","heartbeat_front_addr":"172.21.15.107:6821/4032799356","state":["exists","up"]},{"osd":9,"uuid":"651357f2-0669-42e6-aa91-1d5c02d00c5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6827","nonce":214023431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6829","nonce":214023431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6833","nonce":214023431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6831","nonce":214023431}]},"public_addr":"172.21.15.78:6827/214023431","cluster_addr":"172.21.15.78:6829/214023431","heartbeat_back_addr":"172.21.15.78:6833/214023431","heartbeat_front_addr":"172.21.15.78:6831/214023431","state":["exists","up"]},{"osd":10,"uuid":"2375119d-346e-426c-8a65-b64af71ad63a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6825","nonce":400029804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6827","nonce":400029804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6831","nonce":400029804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6829","nonce":400029804}]},"public_addr":"172.21.15.106:6825/400029804","cluster_addr":"172.21.15.106:6827/400029804","heartbeat_back_addr":"172.21.15.106:6831/400029804","heartbeat_front_addr":"172.21.15.106:6829/400029804","state":["exists","up"]},{"osd":11,"uuid":"b6b609d9-dd78-4c01-ae66-ba2a662ab22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6825","nonce":4191916230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6827","nonce":4191916230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6831","nonce":4191916230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6829","nonce":4191916230}]},"public_addr":"172.21.15.107:6825/4191916230","cluster_addr":"172.21.15.107:6827/4191916230","heartbeat_back_addr":"172.21.15.107:6831/4191916230","heartbeat_front_addr":"172.21.15.107:6829/4191916230","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:13.728602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:37.320809+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:01.269452+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:25.758281+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:50.298863+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:14.345219+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:38.980489+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:04.749214+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:29.992673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:56.269202+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:22.270316+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:48.074907+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/1982202541":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/1933054339":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6801/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6800/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/465691596":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6801/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/3962523609":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6800/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/1502752625":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/2236879401":"2024-03-25T00:11:40.199569+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:18:49.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.521+0000 7f1f9e0b8640 1 -- 172.21.15.78:0/398697371 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f1f800a2800 msgr2=0x7f1f800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:49.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.521+0000 7f1f9e0b8640 1 --2- 172.21.15.78:0/398697371 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f1f800a2800 0x7f1f800a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c005d00 tx=0x7f1f8c005c50 comp rx=0 tx=0).stop 2024-03-24T00:18:49.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.521+0000 7f1f9e0b8640 1 -- 172.21.15.78:0/398697371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f981005c0 msgr2=0x7f1f98199c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:49.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.521+0000 7f1f9e0b8640 1 --2- 172.21.15.78:0/398697371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f981005c0 0x7f1f98199c40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1f88000c00 tx=0x7f1f88005d50 comp rx=0 tx=0).stop 2024-03-24T00:18:49.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.521+0000 7f1f9e0b8640 1 -- 172.21.15.78:0/398697371 shutdown_connections 2024-03-24T00:18:49.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.521+0000 7f1f9e0b8640 1 --2- 172.21.15.78:0/398697371 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f1f800a2800 0x7f1f800a4cc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:49.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.521+0000 7f1f9e0b8640 1 --2- 172.21.15.78:0/398697371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f981017b0 0x7f1f9819a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:49.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.521+0000 7f1f9e0b8640 1 --2- 172.21.15.78:0/398697371 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1f98102170 0x7f1f981a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:49.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.521+0000 7f1f9e0b8640 1 --2- 172.21.15.78:0/398697371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f981005c0 0x7f1f98199c40 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:49.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.521+0000 7f1f9e0b8640 1 -- 172.21.15.78:0/398697371 >> 172.21.15.78:0/398697371 conn(0x7f1f980fbb30 msgr2=0x7f1f981093c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:49.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.521+0000 7f1f9e0b8640 1 -- 172.21.15.78:0/398697371 shutdown_connections 2024-03-24T00:18:49.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.521+0000 7f1f9e0b8640 1 -- 172.21.15.78:0/398697371 wait complete. 2024-03-24T00:18:49.532 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:18:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.633+0000 7fa7487c6640 1 -- 172.21.15.78:0/168571763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa740102cb0 msgr2=0x7fa7401030d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.633+0000 7fa7487c6640 1 --2- 172.21.15.78:0/168571763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa740102cb0 0x7fa7401030d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa73c009900 tx=0x7fa73c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.634+0000 7fa7487c6640 1 -- 172.21.15.78:0/168571763 shutdown_connections 2024-03-24T00:18:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.634+0000 7fa7487c6640 1 --2- 172.21.15.78:0/168571763 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7401000e0 0x7fa740109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.634+0000 7fa7487c6640 1 --2- 172.21.15.78:0/168571763 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7400ff720 0x7fa7400ffba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.634+0000 7fa7487c6640 1 --2- 172.21.15.78:0/168571763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa740102cb0 0x7fa7401030d0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.634+0000 7fa7487c6640 1 -- 172.21.15.78:0/168571763 >> 172.21.15.78:0/168571763 conn(0x7fa7400fb2c0 msgr2=0x7fa7400fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.634+0000 7fa7487c6640 1 -- 172.21.15.78:0/168571763 shutdown_connections 2024-03-24T00:18:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.634+0000 7fa7487c6640 1 -- 172.21.15.78:0/168571763 wait complete. 2024-03-24T00:18:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.634+0000 7fa7487c6640 1 Processor -- start 2024-03-24T00:18:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.634+0000 7fa7487c6640 1 -- start start 2024-03-24T00:18:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.634+0000 7fa7487c6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7400ff720 0x7fa740197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.634+0000 7fa7487c6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7401000e0 0x7fa740197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.634+0000 7fa7487c6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa740102cb0 0x7fa74019efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.634+0000 7fa7487c6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74010d3b0 con 0x7fa7401000e0 2024-03-24T00:18:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.634+0000 7fa7487c6640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74010d230 con 0x7fa740102cb0 2024-03-24T00:18:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.634+0000 7fa7487c6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74010d530 con 0x7fa7400ff720 2024-03-24T00:18:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.634+0000 7fa745d61640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7401000e0 0x7fa740197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.634+0000 7fa745d61640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7401000e0 0x7fa740197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43280/0 (socket says 172.21.15.78:43280) 2024-03-24T00:18:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.634+0000 7fa745d61640 1 -- 172.21.15.78:0/1061183090 learned_addr learned my addr 172.21.15.78:0/1061183090 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.635+0000 7fa746d63640 1 --2- 172.21.15.78:0/1061183090 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa740102cb0 0x7fa74019efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.635+0000 7fa746562640 1 --2- 172.21.15.78:0/1061183090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7400ff720 0x7fa740197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.635+0000 7fa745d61640 1 -- 172.21.15.78:0/1061183090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7400ff720 msgr2=0x7fa740197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.635+0000 7fa745d61640 1 --2- 172.21.15.78:0/1061183090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7400ff720 0x7fa740197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.635+0000 7fa745d61640 1 -- 172.21.15.78:0/1061183090 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa740102cb0 msgr2=0x7fa74019efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.635+0000 7fa745d61640 1 --2- 172.21.15.78:0/1061183090 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa740102cb0 0x7fa74019efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.635+0000 7fa745d61640 1 -- 172.21.15.78:0/1061183090 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa74019f620 con 0x7fa7401000e0 2024-03-24T00:18:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.635+0000 7fa746562640 1 --2- 172.21.15.78:0/1061183090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7400ff720 0x7fa740197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:18:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.635+0000 7fa745d61640 1 --2- 172.21.15.78:0/1061183090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7401000e0 0x7fa740197f40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa73000e9f0 tx=0x7fa73000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.635+0000 7fa7377fe640 1 -- 172.21.15.78:0/1061183090 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73000cd80 con 0x7fa7401000e0 2024-03-24T00:18:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.635+0000 7fa7487c6640 1 -- 172.21.15.78:0/1061183090 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa74019f910 con 0x7fa7401000e0 2024-03-24T00:18:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.635+0000 7fa7377fe640 1 -- 172.21.15.78:0/1061183090 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa730010910 con 0x7fa7401000e0 2024-03-24T00:18:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.635+0000 7fa7377fe640 1 -- 172.21.15.78:0/1061183090 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7300188f0 con 0x7fa7401000e0 2024-03-24T00:18:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.635+0000 7fa7487c6640 1 -- 172.21.15.78:0/1061183090 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa74019feb0 con 0x7fa7401000e0 2024-03-24T00:18:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.636+0000 7fa7487c6640 1 -- 172.21.15.78:0/1061183090 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa740108b70 con 0x7fa7401000e0 2024-03-24T00:18:49.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.637+0000 7fa7377fe640 1 -- 172.21.15.78:0/1061183090 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa730010ac0 con 0x7fa7401000e0 2024-03-24T00:18:49.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.638+0000 7fa7377fe640 1 --2- 172.21.15.78:0/1061183090 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa72c0a2800 0x7fa72c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:49.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.638+0000 7fa746562640 1 --2- 172.21.15.78:0/1061183090 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa72c0a2800 0x7fa72c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:49.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.638+0000 7fa7377fe640 1 -- 172.21.15.78:0/1061183090 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fa7300cf4c0 con 0x7fa7401000e0 2024-03-24T00:18:49.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.638+0000 7fa746562640 1 --2- 172.21.15.78:0/1061183090 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa72c0a2800 0x7fa72c0a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa73c000c00 tx=0x7fa73c02cf90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:49.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.641+0000 7fa7377fe640 1 -- 172.21.15.78:0/1061183090 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa73009c320 con 0x7fa7401000e0 2024-03-24T00:18:49.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.816+0000 7fa7487c6640 1 -- 172.21.15.78:0/1061183090 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa74002d290 con 0x7fa7401000e0 2024-03-24T00:18:49.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.817+0000 7fa7377fe640 1 -- 172.21.15.78:0/1061183090 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24129 (secure 0 0 0) 0x7fa730086090 con 0x7fa7401000e0 2024-03-24T00:18:49.817 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:18:49.818 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":89,"fsid":"c7e2806a-e972-11ee-95cd-87774f69a715","created":"2024-03-24T00:10:44.134563+0000","modified":"2024-03-24T00:18:48.224440+0000","last_up_change":"2024-03-24T00:17:50.237686+0000","last_in_change":"2024-03-24T00:17:29.788699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:14:04.534391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:18:42.178418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:18:42.800424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:18:46.112800+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b950388a-c437-426b-b703-3f8ae821dd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6803","nonce":1816879397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6805","nonce":1816879397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6809","nonce":1816879397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6807","nonce":1816879397}]},"public_addr":"172.21.15.78:6803/1816879397","cluster_addr":"172.21.15.78:6805/1816879397","heartbeat_back_addr":"172.21.15.78:6809/1816879397","heartbeat_front_addr":"172.21.15.78:6807/1816879397","state":["exists","up"]},{"osd":1,"uuid":"b8d64714-dd65-400f-b3f4-673ab14e1365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6801","nonce":1647915612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6803","nonce":1647915612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6807","nonce":1647915612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6805","nonce":1647915612}]},"public_addr":"172.21.15.106:6801/1647915612","cluster_addr":"172.21.15.106:6803/1647915612","heartbeat_back_addr":"172.21.15.106:6807/1647915612","heartbeat_front_addr":"172.21.15.106:6805/1647915612","state":["exists","up"]},{"osd":2,"uuid":"fcfc4dd1-aaf8-4131-a023-5e832212158d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6801","nonce":2010770435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6803","nonce":2010770435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6807","nonce":2010770435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6805","nonce":2010770435}]},"public_addr":"172.21.15.107:6801/2010770435","cluster_addr":"172.21.15.107:6803/2010770435","heartbeat_back_addr":"172.21.15.107:6807/2010770435","heartbeat_front_addr":"172.21.15.107:6805/2010770435","state":["exists","up"]},{"osd":3,"uuid":"47c718ab-a0f4-4e52-83c5-b89386b262d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6811","nonce":1912251346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6813","nonce":1912251346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6817","nonce":1912251346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6815","nonce":1912251346}]},"public_addr":"172.21.15.78:6811/1912251346","cluster_addr":"172.21.15.78:6813/1912251346","heartbeat_back_addr":"172.21.15.78:6817/1912251346","heartbeat_front_addr":"172.21.15.78:6815/1912251346","state":["exists","up"]},{"osd":4,"uuid":"7dad64dd-677e-4c1c-9638-e4f4f2195983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6809","nonce":374641205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6811","nonce":374641205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6815","nonce":374641205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6813","nonce":374641205}]},"public_addr":"172.21.15.106:6809/374641205","cluster_addr":"172.21.15.106:6811/374641205","heartbeat_back_addr":"172.21.15.106:6815/374641205","heartbeat_front_addr":"172.21.15.106:6813/374641205","state":["exists","up"]},{"osd":5,"uuid":"bbd524e5-328d-4309-8877-1149cbe110dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6809","nonce":3272806073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6811","nonce":3272806073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6815","nonce":3272806073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6813","nonce":3272806073}]},"public_addr":"172.21.15.107:6809/3272806073","cluster_addr":"172.21.15.107:6811/3272806073","heartbeat_back_addr":"172.21.15.107:6815/3272806073","heartbeat_front_addr":"172.21.15.107:6813/3272806073","state":["exists","up"]},{"osd":6,"uuid":"5696389a-918f-4ea8-b20d-4e2a0d4cb3bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6819","nonce":3091148508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6821","nonce":3091148508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6825","nonce":3091148508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6823","nonce":3091148508}]},"public_addr":"172.21.15.78:6819/3091148508","cluster_addr":"172.21.15.78:6821/3091148508","heartbeat_back_addr":"172.21.15.78:6825/3091148508","heartbeat_front_addr":"172.21.15.78:6823/3091148508","state":["exists","up"]},{"osd":7,"uuid":"f6b764e8-ed4e-4c9a-a85b-94af6dc6b924","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6817","nonce":4262103080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6819","nonce":4262103080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6823","nonce":4262103080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6821","nonce":4262103080}]},"public_addr":"172.21.15.106:6817/4262103080","cluster_addr":"172.21.15.106:6819/4262103080","heartbeat_back_addr":"172.21.15.106:6823/4262103080","heartbeat_front_addr":"172.21.15.106:6821/4262103080","state":["exists","up"]},{"osd":8,"uuid":"ba16039f-8093-4fa0-87ad-e28b7d142b23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6817","nonce":4032799356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6819","nonce":4032799356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6823","nonce":4032799356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6821","nonce":4032799356}]},"public_addr":"172.21.15.107:6817/4032799356","cluster_addr":"172.21.15.107:6819/4032799356","heartbeat_back_addr":"172.21.15.107:6823/4032799356","heartbeat_front_addr":"172.21.15.107:6821/4032799356","state":["exists","up"]},{"osd":9,"uuid":"651357f2-0669-42e6-aa91-1d5c02d00c5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6827","nonce":214023431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6829","nonce":214023431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6833","nonce":214023431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6831","nonce":214023431}]},"public_addr":"172.21.15.78:6827/214023431","cluster_addr":"172.21.15.78:6829/214023431","heartbeat_back_addr":"172.21.15.78:6833/214023431","heartbeat_front_addr":"172.21.15.78:6831/214023431","state":["exists","up"]},{"osd":10,"uuid":"2375119d-346e-426c-8a65-b64af71ad63a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6825","nonce":400029804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6827","nonce":400029804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6831","nonce":400029804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6829","nonce":400029804}]},"public_addr":"172.21.15.106:6825/400029804","cluster_addr":"172.21.15.106:6827/400029804","heartbeat_back_addr":"172.21.15.106:6831/400029804","heartbeat_front_addr":"172.21.15.106:6829/400029804","state":["exists","up"]},{"osd":11,"uuid":"b6b609d9-dd78-4c01-ae66-ba2a662ab22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6825","nonce":4191916230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6827","nonce":4191916230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6831","nonce":4191916230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6829","nonce":4191916230}]},"public_addr":"172.21.15.107:6825/4191916230","cluster_addr":"172.21.15.107:6827/4191916230","heartbeat_back_addr":"172.21.15.107:6831/4191916230","heartbeat_front_addr":"172.21.15.107:6829/4191916230","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:13.728602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:37.320809+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:01.269452+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:25.758281+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:50.298863+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:14.345219+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:38.980489+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:04.749214+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:29.992673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:56.269202+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:22.270316+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:48.074907+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/1982202541":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/1933054339":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6801/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6800/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/465691596":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6801/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/3962523609":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6800/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/1502752625":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/2236879401":"2024-03-25T00:11:40.199569+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:18:49.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.819+0000 7fa7487c6640 1 -- 172.21.15.78:0/1061183090 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa72c0a2800 msgr2=0x7fa72c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:49.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.819+0000 7fa7487c6640 1 --2- 172.21.15.78:0/1061183090 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa72c0a2800 0x7fa72c0a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa73c000c00 tx=0x7fa73c02cf90 comp rx=0 tx=0).stop 2024-03-24T00:18:49.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.819+0000 7fa7487c6640 1 -- 172.21.15.78:0/1061183090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7401000e0 msgr2=0x7fa740197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:49.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.819+0000 7fa7487c6640 1 --2- 172.21.15.78:0/1061183090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7401000e0 0x7fa740197f40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa73000e9f0 tx=0x7fa73000eeb0 comp rx=0 tx=0).stop 2024-03-24T00:18:49.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.819+0000 7fa7487c6640 1 -- 172.21.15.78:0/1061183090 shutdown_connections 2024-03-24T00:18:49.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.819+0000 7fa7487c6640 1 --2- 172.21.15.78:0/1061183090 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa72c0a2800 0x7fa72c0a4cc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:49.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.819+0000 7fa7487c6640 1 --2- 172.21.15.78:0/1061183090 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa740102cb0 0x7fa74019efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:49.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.819+0000 7fa7487c6640 1 --2- 172.21.15.78:0/1061183090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7401000e0 0x7fa740197f40 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:49.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.819+0000 7fa7487c6640 1 --2- 172.21.15.78:0/1061183090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7400ff720 0x7fa740197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:49.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.819+0000 7fa7487c6640 1 -- 172.21.15.78:0/1061183090 >> 172.21.15.78:0/1061183090 conn(0x7fa7400fb2c0 msgr2=0x7fa740105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:49.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.819+0000 7fa7487c6640 1 -- 172.21.15.78:0/1061183090 shutdown_connections 2024-03-24T00:18:49.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.819+0000 7fa7487c6640 1 -- 172.21.15.78:0/1061183090 wait complete. 2024-03-24T00:18:49.846 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:18:49.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.949+0000 7f503bf53640 1 -- 172.21.15.78:0/2548060650 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5034100f60 msgr2=0x7f50341013e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:49.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.949+0000 7f503bf53640 1 --2- 172.21.15.78:0/2548060650 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5034100f60 0x7f50341013e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5030009900 tx=0x7f503002bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:49.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.949+0000 7f503bf53640 1 -- 172.21.15.78:0/2548060650 shutdown_connections 2024-03-24T00:18:49.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.949+0000 7f503bf53640 1 --2- 172.21.15.78:0/2548060650 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5034101920 0x7f50341081b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:49.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.949+0000 7f503bf53640 1 --2- 172.21.15.78:0/2548060650 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5034100f60 0x7f50341013e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:49.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.949+0000 7f503bf53640 1 --2- 172.21.15.78:0/2548060650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50340ffd70 0x7f5034100190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:49.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.949+0000 7f503bf53640 1 -- 172.21.15.78:0/2548060650 >> 172.21.15.78:0/2548060650 conn(0x7f50340fb2e0 msgr2=0x7f50340fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:49.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.949+0000 7f503bf53640 1 -- 172.21.15.78:0/2548060650 shutdown_connections 2024-03-24T00:18:49.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.949+0000 7f503bf53640 1 -- 172.21.15.78:0/2548060650 wait complete. 2024-03-24T00:18:49.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.949+0000 7f503bf53640 1 Processor -- start 2024-03-24T00:18:49.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.949+0000 7f503bf53640 1 -- start start 2024-03-24T00:18:49.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.949+0000 7f503bf53640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f50340ffd70 0x7f5034197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:49.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.949+0000 7f503bf53640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5034100f60 0x7f5034197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:49.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.949+0000 7f503bf53640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5034101920 0x7f503419efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:49.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.949+0000 7f503bf53640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503410c280 con 0x7f5034101920 2024-03-24T00:18:49.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.949+0000 7f503bf53640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503410c100 con 0x7f50340ffd70 2024-03-24T00:18:49.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.949+0000 7f503bf53640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503410c400 con 0x7f5034100f60 2024-03-24T00:18:49.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.950+0000 7f503a4f0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5034101920 0x7f503419efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:49.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.950+0000 7f5039cef640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f50340ffd70 0x7f5034197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:49.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.950+0000 7f503a4f0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5034101920 0x7f503419efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43296/0 (socket says 172.21.15.78:43296) 2024-03-24T00:18:49.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.950+0000 7f503a4f0640 1 -- 172.21.15.78:0/4089936624 learned_addr learned my addr 172.21.15.78:0/4089936624 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:49.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.950+0000 7f50394ee640 1 --2- 172.21.15.78:0/4089936624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5034100f60 0x7f5034197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:49.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.950+0000 7f503a4f0640 1 -- 172.21.15.78:0/4089936624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5034100f60 msgr2=0x7f5034197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:49.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.950+0000 7f503a4f0640 1 --2- 172.21.15.78:0/4089936624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5034100f60 0x7f5034197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:49.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.950+0000 7f503a4f0640 1 -- 172.21.15.78:0/4089936624 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f50340ffd70 msgr2=0x7f5034197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:49.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.950+0000 7f503a4f0640 1 --2- 172.21.15.78:0/4089936624 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f50340ffd70 0x7f5034197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:49.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.950+0000 7f503a4f0640 1 -- 172.21.15.78:0/4089936624 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f503419f650 con 0x7f5034101920 2024-03-24T00:18:49.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.950+0000 7f503a4f0640 1 --2- 172.21.15.78:0/4089936624 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5034101920 0x7f503419efe0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f502c00eb10 tx=0x7f502c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:49.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.950+0000 7f5026ffd640 1 -- 172.21.15.78:0/4089936624 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f502c004b20 con 0x7f5034101920 2024-03-24T00:18:49.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.950+0000 7f5026ffd640 1 -- 172.21.15.78:0/4089936624 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f502c004cc0 con 0x7f5034101920 2024-03-24T00:18:49.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.950+0000 7f503bf53640 1 -- 172.21.15.78:0/4089936624 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f503419f940 con 0x7f5034101920 2024-03-24T00:18:49.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.950+0000 7f5026ffd640 1 -- 172.21.15.78:0/4089936624 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f502c010760 con 0x7f5034101920 2024-03-24T00:18:49.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.950+0000 7f503bf53640 1 -- 172.21.15.78:0/4089936624 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f503419fe40 con 0x7f5034101920 2024-03-24T00:18:49.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.951+0000 7f503bf53640 1 -- 172.21.15.78:0/4089936624 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50341a0070 con 0x7f5034101920 2024-03-24T00:18:49.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.952+0000 7f5026ffd640 1 -- 172.21.15.78:0/4089936624 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f502c005000 con 0x7f5034101920 2024-03-24T00:18:49.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.953+0000 7f5026ffd640 1 --2- 172.21.15.78:0/4089936624 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f50200a2800 0x7f50200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:49.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.953+0000 7f5039cef640 1 --2- 172.21.15.78:0/4089936624 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f50200a2800 0x7f50200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:49.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.953+0000 7f5026ffd640 1 -- 172.21.15.78:0/4089936624 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f502c0cf4f0 con 0x7f5034101920 2024-03-24T00:18:49.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.953+0000 7f5039cef640 1 --2- 172.21.15.78:0/4089936624 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f50200a2800 0x7f50200a4cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5028005fd0 tx=0x7f502800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:49.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:49.956+0000 7f5026ffd640 1 -- 172.21.15.78:0/4089936624 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f502c053740 con 0x7f5034101920 2024-03-24T00:18:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:49 smithi078 ceph-mon[32429]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T00:18:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:49 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3254320105' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-24T00:18:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:49 smithi078 ceph-mon[32429]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:18:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:49 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/432182890' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:49 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/398697371' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:18:50.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:49 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1061183090' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:18:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:49 smithi106 ceph-mon[36505]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T00:18:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:49 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3254320105' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-24T00:18:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:49 smithi106 ceph-mon[36505]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:18:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:49 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/432182890' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:49 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/398697371' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:18:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:49 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1061183090' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:18:50.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.169+0000 7f503bf53640 1 -- 172.21.15.78:0/4089936624 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50341a0240 con 0x7f5034101920 2024-03-24T00:18:50.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.169+0000 7f5026ffd640 1 -- 172.21.15.78:0/4089936624 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+11333 (secure 0 0 0) 0x7f502c09c350 con 0x7f5034101920 2024-03-24T00:18:50.169 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:18:50.169 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1416685123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1416685123},{"type":"v1","addr":"172.21.15.78:6835","nonce":1416685123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1071274623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1071274623},{"type":"v1","addr":"172.21.15.78:6837","nonce":1071274623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2758357551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2758357551},{"type":"v1","addr":"172.21.15.78:6839","nonce":2758357551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/3424505910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":3424505910},{"type":"v1","addr":"172.21.15.78:6841","nonce":3424505910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/3086318443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3086318443},{"type":"v1","addr":"172.21.15.107:6833","nonce":3086318443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/2367018868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2367018868},{"type":"v1","addr":"172.21.15.106:6833","nonce":2367018868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/2231611477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":2231611477},{"type":"v1","addr":"172.21.15.106:6835","nonce":2231611477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/108936439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":108936439},{"type":"v1","addr":"172.21.15.107:6835","nonce":108936439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2939571540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2939571540},{"type":"v1","addr":"172.21.15.107:6837","nonce":2939571540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/2090968633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":2090968633},{"type":"v1","addr":"172.21.15.106:6837","nonce":2090968633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/117119462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":117119462},{"type":"v1","addr":"172.21.15.107:6839","nonce":117119462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:18:43.592215+0000","modified":"2024-03-24T00:18:48.224616+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6839/2482071304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2482071304},{"type":"v1","addr":"172.21.15.106:6839","nonce":2482071304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24500,"qdb_cluster":[24500]},"id":1}]} 2024-03-24T00:18:50.169 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 17 2024-03-24T00:18:50.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.171+0000 7f503bf53640 1 -- 172.21.15.78:0/4089936624 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f50200a2800 msgr2=0x7f50200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:50.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.171+0000 7f503bf53640 1 --2- 172.21.15.78:0/4089936624 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f50200a2800 0x7f50200a4cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5028005fd0 tx=0x7f502800b390 comp rx=0 tx=0).stop 2024-03-24T00:18:50.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.171+0000 7f503bf53640 1 -- 172.21.15.78:0/4089936624 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5034101920 msgr2=0x7f503419efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:50.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.171+0000 7f503bf53640 1 --2- 172.21.15.78:0/4089936624 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5034101920 0x7f503419efe0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f502c00eb10 tx=0x7f502c00c2d0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.171+0000 7f503bf53640 1 -- 172.21.15.78:0/4089936624 shutdown_connections 2024-03-24T00:18:50.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.171+0000 7f503bf53640 1 --2- 172.21.15.78:0/4089936624 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f50200a2800 0x7f50200a4cc0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.171+0000 7f503bf53640 1 --2- 172.21.15.78:0/4089936624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5034100f60 0x7f5034197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.171+0000 7f503bf53640 1 --2- 172.21.15.78:0/4089936624 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5034101920 0x7f503419efe0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.171+0000 7f503bf53640 1 --2- 172.21.15.78:0/4089936624 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f50340ffd70 0x7f5034197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.171+0000 7f503bf53640 1 -- 172.21.15.78:0/4089936624 >> 172.21.15.78:0/4089936624 conn(0x7f50340fb2e0 msgr2=0x7f5034108b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:50.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.171+0000 7f503bf53640 1 -- 172.21.15.78:0/4089936624 shutdown_connections 2024-03-24T00:18:50.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.171+0000 7f503bf53640 1 -- 172.21.15.78:0/4089936624 wait complete. 2024-03-24T00:18:50.182 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:18:50.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.284+0000 7ff6636a1640 1 -- 172.21.15.78:0/4237073666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff65c0ff760 msgr2=0x7ff65c0ffbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:50.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.284+0000 7ff6636a1640 1 --2- 172.21.15.78:0/4237073666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff65c0ff760 0x7ff65c0ffbe0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff650009900 tx=0x7ff65002bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:50.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.284+0000 7ff6636a1640 1 -- 172.21.15.78:0/4237073666 shutdown_connections 2024-03-24T00:18:50.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.284+0000 7ff6636a1640 1 --2- 172.21.15.78:0/4237073666 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff65c100120 0x7ff65c109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.284+0000 7ff6636a1640 1 --2- 172.21.15.78:0/4237073666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff65c0ff760 0x7ff65c0ffbe0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.284+0000 7ff6636a1640 1 --2- 172.21.15.78:0/4237073666 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff65c102cf0 0x7ff65c103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.284+0000 7ff6636a1640 1 -- 172.21.15.78:0/4237073666 >> 172.21.15.78:0/4237073666 conn(0x7ff65c0fb2e0 msgr2=0x7ff65c0fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:50.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.284+0000 7ff6636a1640 1 -- 172.21.15.78:0/4237073666 shutdown_connections 2024-03-24T00:18:50.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.284+0000 7ff6636a1640 1 -- 172.21.15.78:0/4237073666 wait complete. 2024-03-24T00:18:50.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.284+0000 7ff6636a1640 1 Processor -- start 2024-03-24T00:18:50.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.285+0000 7ff6636a1640 1 -- start start 2024-03-24T00:18:50.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.285+0000 7ff6636a1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff65c0ff760 0x7ff65c197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:50.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.285+0000 7ff6636a1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff65c100120 0x7ff65c197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:50.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.285+0000 7ff6636a1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff65c102cf0 0x7ff65c19eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:50.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.285+0000 7ff6636a1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff65c10d370 con 0x7ff65c100120 2024-03-24T00:18:50.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.285+0000 7ff6636a1640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff65c10d1f0 con 0x7ff65c102cf0 2024-03-24T00:18:50.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.285+0000 7ff6636a1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff65c10d4f0 con 0x7ff65c0ff760 2024-03-24T00:18:50.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.285+0000 7ff660c3c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff65c100120 0x7ff65c197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:50.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.285+0000 7ff660c3c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff65c100120 0x7ff65c197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43316/0 (socket says 172.21.15.78:43316) 2024-03-24T00:18:50.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.285+0000 7ff660c3c640 1 -- 172.21.15.78:0/3306264362 learned_addr learned my addr 172.21.15.78:0/3306264362 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:50.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.285+0000 7ff661c3e640 1 --2- 172.21.15.78:0/3306264362 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff65c102cf0 0x7ff65c19eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:50.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.285+0000 7ff66143d640 1 --2- 172.21.15.78:0/3306264362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff65c0ff760 0x7ff65c197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:50.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.285+0000 7ff660c3c640 1 -- 172.21.15.78:0/3306264362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff65c0ff760 msgr2=0x7ff65c197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:50.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.285+0000 7ff660c3c640 1 --2- 172.21.15.78:0/3306264362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff65c0ff760 0x7ff65c197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.285+0000 7ff660c3c640 1 -- 172.21.15.78:0/3306264362 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff65c102cf0 msgr2=0x7ff65c19eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:50.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.285+0000 7ff660c3c640 1 --2- 172.21.15.78:0/3306264362 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff65c102cf0 0x7ff65c19eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.285+0000 7ff660c3c640 1 -- 172.21.15.78:0/3306264362 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff65c19f660 con 0x7ff65c100120 2024-03-24T00:18:50.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.285+0000 7ff66143d640 1 --2- 172.21.15.78:0/3306264362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff65c0ff760 0x7ff65c197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:18:50.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.285+0000 7ff660c3c640 1 --2- 172.21.15.78:0/3306264362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff65c100120 0x7ff65c197f80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff650009230 tx=0x7ff650005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:50.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.285+0000 7ff64e7fc640 1 -- 172.21.15.78:0/3306264362 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff650041070 con 0x7ff65c100120 2024-03-24T00:18:50.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.285+0000 7ff64e7fc640 1 -- 172.21.15.78:0/3306264362 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff6500054e0 con 0x7ff65c100120 2024-03-24T00:18:50.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.285+0000 7ff6636a1640 1 -- 172.21.15.78:0/3306264362 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff65c19f8f0 con 0x7ff65c100120 2024-03-24T00:18:50.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.285+0000 7ff64e7fc640 1 -- 172.21.15.78:0/3306264362 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff650034440 con 0x7ff65c100120 2024-03-24T00:18:50.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.285+0000 7ff6636a1640 1 -- 172.21.15.78:0/3306264362 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff65c19fde0 con 0x7ff65c100120 2024-03-24T00:18:50.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.286+0000 7ff6636a1640 1 -- 172.21.15.78:0/3306264362 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff65c108c00 con 0x7ff65c100120 2024-03-24T00:18:50.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.287+0000 7ff64e7fc640 1 -- 172.21.15.78:0/3306264362 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff650044070 con 0x7ff65c100120 2024-03-24T00:18:50.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.288+0000 7ff64e7fc640 1 --2- 172.21.15.78:0/3306264362 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff6480a2800 0x7ff6480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:50.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.288+0000 7ff66143d640 1 --2- 172.21.15.78:0/3306264362 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff6480a2800 0x7ff6480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:50.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.288+0000 7ff64e7fc640 1 -- 172.21.15.78:0/3306264362 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7ff6500f09b0 con 0x7ff65c100120 2024-03-24T00:18:50.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.288+0000 7ff66143d640 1 --2- 172.21.15.78:0/3306264362 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff6480a2800 0x7ff6480a4cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff6580058c0 tx=0x7ff658005cd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:50.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.292+0000 7ff64e7fc640 1 -- 172.21.15.78:0/3306264362 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff6500bd810 con 0x7ff65c100120 2024-03-24T00:18:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:49 smithi107 ceph-mon[36485]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T00:18:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:49 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3254320105' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-24T00:18:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:49 smithi107 ceph-mon[36485]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:18:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:49 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/432182890' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:49 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/398697371' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:18:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:49 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1061183090' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:18:50.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.469+0000 7ff6636a1640 1 -- 172.21.15.78:0/3306264362 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff65c0622a0 con 0x7ff65c100120 2024-03-24T00:18:50.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.471+0000 7ff64e7fc640 1 -- 172.21.15.78:0/3306264362 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24129 (secure 0 0 0) 0x7ff6500bd110 con 0x7ff65c100120 2024-03-24T00:18:50.471 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:18:50.471 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":89,"fsid":"c7e2806a-e972-11ee-95cd-87774f69a715","created":"2024-03-24T00:10:44.134563+0000","modified":"2024-03-24T00:18:48.224440+0000","last_up_change":"2024-03-24T00:17:50.237686+0000","last_in_change":"2024-03-24T00:17:29.788699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:14:04.534391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:18:42.178418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:18:42.800424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:18:46.112800+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b950388a-c437-426b-b703-3f8ae821dd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6803","nonce":1816879397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6805","nonce":1816879397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6809","nonce":1816879397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6807","nonce":1816879397}]},"public_addr":"172.21.15.78:6803/1816879397","cluster_addr":"172.21.15.78:6805/1816879397","heartbeat_back_addr":"172.21.15.78:6809/1816879397","heartbeat_front_addr":"172.21.15.78:6807/1816879397","state":["exists","up"]},{"osd":1,"uuid":"b8d64714-dd65-400f-b3f4-673ab14e1365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6801","nonce":1647915612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6803","nonce":1647915612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6807","nonce":1647915612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6805","nonce":1647915612}]},"public_addr":"172.21.15.106:6801/1647915612","cluster_addr":"172.21.15.106:6803/1647915612","heartbeat_back_addr":"172.21.15.106:6807/1647915612","heartbeat_front_addr":"172.21.15.106:6805/1647915612","state":["exists","up"]},{"osd":2,"uuid":"fcfc4dd1-aaf8-4131-a023-5e832212158d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6801","nonce":2010770435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6803","nonce":2010770435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6807","nonce":2010770435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6805","nonce":2010770435}]},"public_addr":"172.21.15.107:6801/2010770435","cluster_addr":"172.21.15.107:6803/2010770435","heartbeat_back_addr":"172.21.15.107:6807/2010770435","heartbeat_front_addr":"172.21.15.107:6805/2010770435","state":["exists","up"]},{"osd":3,"uuid":"47c718ab-a0f4-4e52-83c5-b89386b262d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6811","nonce":1912251346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6813","nonce":1912251346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6817","nonce":1912251346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6815","nonce":1912251346}]},"public_addr":"172.21.15.78:6811/1912251346","cluster_addr":"172.21.15.78:6813/1912251346","heartbeat_back_addr":"172.21.15.78:6817/1912251346","heartbeat_front_addr":"172.21.15.78:6815/1912251346","state":["exists","up"]},{"osd":4,"uuid":"7dad64dd-677e-4c1c-9638-e4f4f2195983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6809","nonce":374641205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6811","nonce":374641205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6815","nonce":374641205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6813","nonce":374641205}]},"public_addr":"172.21.15.106:6809/374641205","cluster_addr":"172.21.15.106:6811/374641205","heartbeat_back_addr":"172.21.15.106:6815/374641205","heartbeat_front_addr":"172.21.15.106:6813/374641205","state":["exists","up"]},{"osd":5,"uuid":"bbd524e5-328d-4309-8877-1149cbe110dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6809","nonce":3272806073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6811","nonce":3272806073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6815","nonce":3272806073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6813","nonce":3272806073}]},"public_addr":"172.21.15.107:6809/3272806073","cluster_addr":"172.21.15.107:6811/3272806073","heartbeat_back_addr":"172.21.15.107:6815/3272806073","heartbeat_front_addr":"172.21.15.107:6813/3272806073","state":["exists","up"]},{"osd":6,"uuid":"5696389a-918f-4ea8-b20d-4e2a0d4cb3bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6819","nonce":3091148508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6821","nonce":3091148508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6825","nonce":3091148508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6823","nonce":3091148508}]},"public_addr":"172.21.15.78:6819/3091148508","cluster_addr":"172.21.15.78:6821/3091148508","heartbeat_back_addr":"172.21.15.78:6825/3091148508","heartbeat_front_addr":"172.21.15.78:6823/3091148508","state":["exists","up"]},{"osd":7,"uuid":"f6b764e8-ed4e-4c9a-a85b-94af6dc6b924","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6817","nonce":4262103080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6819","nonce":4262103080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6823","nonce":4262103080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6821","nonce":4262103080}]},"public_addr":"172.21.15.106:6817/4262103080","cluster_addr":"172.21.15.106:6819/4262103080","heartbeat_back_addr":"172.21.15.106:6823/4262103080","heartbeat_front_addr":"172.21.15.106:6821/4262103080","state":["exists","up"]},{"osd":8,"uuid":"ba16039f-8093-4fa0-87ad-e28b7d142b23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6817","nonce":4032799356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6819","nonce":4032799356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6823","nonce":4032799356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6821","nonce":4032799356}]},"public_addr":"172.21.15.107:6817/4032799356","cluster_addr":"172.21.15.107:6819/4032799356","heartbeat_back_addr":"172.21.15.107:6823/4032799356","heartbeat_front_addr":"172.21.15.107:6821/4032799356","state":["exists","up"]},{"osd":9,"uuid":"651357f2-0669-42e6-aa91-1d5c02d00c5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6827","nonce":214023431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6829","nonce":214023431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6833","nonce":214023431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6831","nonce":214023431}]},"public_addr":"172.21.15.78:6827/214023431","cluster_addr":"172.21.15.78:6829/214023431","heartbeat_back_addr":"172.21.15.78:6833/214023431","heartbeat_front_addr":"172.21.15.78:6831/214023431","state":["exists","up"]},{"osd":10,"uuid":"2375119d-346e-426c-8a65-b64af71ad63a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6825","nonce":400029804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6827","nonce":400029804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6831","nonce":400029804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6829","nonce":400029804}]},"public_addr":"172.21.15.106:6825/400029804","cluster_addr":"172.21.15.106:6827/400029804","heartbeat_back_addr":"172.21.15.106:6831/400029804","heartbeat_front_addr":"172.21.15.106:6829/400029804","state":["exists","up"]},{"osd":11,"uuid":"b6b609d9-dd78-4c01-ae66-ba2a662ab22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6825","nonce":4191916230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6827","nonce":4191916230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6831","nonce":4191916230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6829","nonce":4191916230}]},"public_addr":"172.21.15.107:6825/4191916230","cluster_addr":"172.21.15.107:6827/4191916230","heartbeat_back_addr":"172.21.15.107:6831/4191916230","heartbeat_front_addr":"172.21.15.107:6829/4191916230","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:13.728602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:37.320809+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:01.269452+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:25.758281+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:50.298863+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:14.345219+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:38.980489+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:04.749214+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:29.992673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:56.269202+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:22.270316+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:48.074907+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/1982202541":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/1933054339":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6801/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6800/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/465691596":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6801/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/3962523609":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6800/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/1502752625":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/2236879401":"2024-03-25T00:11:40.199569+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:18:50.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.473+0000 7ff6636a1640 1 -- 172.21.15.78:0/3306264362 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff6480a2800 msgr2=0x7ff6480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:50.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.473+0000 7ff6636a1640 1 --2- 172.21.15.78:0/3306264362 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff6480a2800 0x7ff6480a4cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff6580058c0 tx=0x7ff658005cd0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.473+0000 7ff6636a1640 1 -- 172.21.15.78:0/3306264362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff65c100120 msgr2=0x7ff65c197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:50.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.473+0000 7ff6636a1640 1 --2- 172.21.15.78:0/3306264362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff65c100120 0x7ff65c197f80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff650009230 tx=0x7ff650005dd0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.473+0000 7ff6636a1640 1 -- 172.21.15.78:0/3306264362 shutdown_connections 2024-03-24T00:18:50.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.473+0000 7ff6636a1640 1 --2- 172.21.15.78:0/3306264362 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff6480a2800 0x7ff6480a4cc0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.473+0000 7ff6636a1640 1 --2- 172.21.15.78:0/3306264362 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff65c102cf0 0x7ff65c19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.473+0000 7ff6636a1640 1 --2- 172.21.15.78:0/3306264362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff65c100120 0x7ff65c197f80 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.473+0000 7ff6636a1640 1 --2- 172.21.15.78:0/3306264362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff65c0ff760 0x7ff65c197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.473+0000 7ff6636a1640 1 -- 172.21.15.78:0/3306264362 >> 172.21.15.78:0/3306264362 conn(0x7ff65c0fb2e0 msgr2=0x7ff65c105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:50.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.473+0000 7ff6636a1640 1 -- 172.21.15.78:0/3306264362 shutdown_connections 2024-03-24T00:18:50.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.473+0000 7ff6636a1640 1 -- 172.21.15.78:0/3306264362 wait complete. 2024-03-24T00:18:50.484 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:18:43.592215+0000', 'modified': '2024-03-24T00:18:48.224616+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24500}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24500': {'gid': 24500, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.106:6839/2482071304', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6838', 'nonce': 2482071304}, {'type': 'v1', 'addr': '172.21.15.106:6839', 'nonce': 2482071304}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24500, 'qdb_cluster': [24500]} 2024-03-24T00:18:50.484 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T00:18:50.484 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-24T00:18:50.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.587+0000 7f3bbbf52640 1 -- 172.21.15.78:0/2202407531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bb4101790 msgr2=0x7f3bb4101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:50.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.587+0000 7f3bbbf52640 1 --2- 172.21.15.78:0/2202407531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bb4101790 0x7f3bb4101c10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8009900 tx=0x7f3ba802bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:50.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.588+0000 7f3bbbf52640 1 -- 172.21.15.78:0/2202407531 shutdown_connections 2024-03-24T00:18:50.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.588+0000 7f3bbbf52640 1 --2- 172.21.15.78:0/2202407531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bb4102150 0x7f3bb41089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.588+0000 7f3bbbf52640 1 --2- 172.21.15.78:0/2202407531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bb4101790 0x7f3bb4101c10 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.588+0000 7f3bbbf52640 1 --2- 172.21.15.78:0/2202407531 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3bb41005a0 0x7f3bb41009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.588+0000 7f3bbbf52640 1 -- 172.21.15.78:0/2202407531 >> 172.21.15.78:0/2202407531 conn(0x7f3bb40fbb30 msgr2=0x7f3bb40fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:50.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.588+0000 7f3bbbf52640 1 -- 172.21.15.78:0/2202407531 shutdown_connections 2024-03-24T00:18:50.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.588+0000 7f3bbbf52640 1 -- 172.21.15.78:0/2202407531 wait complete. 2024-03-24T00:18:50.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.588+0000 7f3bbbf52640 1 Processor -- start 2024-03-24T00:18:50.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.588+0000 7f3bbbf52640 1 -- start start 2024-03-24T00:18:50.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.588+0000 7f3bbbf52640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bb41005a0 0x7f3bb4197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:50.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.588+0000 7f3bbbf52640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bb4101790 0x7f3bb4197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:50.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.588+0000 7f3bbbf52640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3bb4102150 0x7f3bb419efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:50.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.588+0000 7f3bbbf52640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb410c230 con 0x7f3bb41005a0 2024-03-24T00:18:50.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.588+0000 7f3bbbf52640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb410c0b0 con 0x7f3bb4102150 2024-03-24T00:18:50.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.588+0000 7f3bbbf52640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb410c3b0 con 0x7f3bb4101790 2024-03-24T00:18:50.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.588+0000 7f3bb9cee640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bb41005a0 0x7f3bb4197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:50.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.588+0000 7f3bb9cee640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bb41005a0 0x7f3bb4197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43340/0 (socket says 172.21.15.78:43340) 2024-03-24T00:18:50.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.588+0000 7f3bb9cee640 1 -- 172.21.15.78:0/3815617548 learned_addr learned my addr 172.21.15.78:0/3815617548 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:50.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.589+0000 7f3bba4ef640 1 --2- 172.21.15.78:0/3815617548 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3bb4102150 0x7f3bb419efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:50.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.589+0000 7f3bb94ed640 1 --2- 172.21.15.78:0/3815617548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bb4101790 0x7f3bb4197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:50.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.589+0000 7f3bb9cee640 1 -- 172.21.15.78:0/3815617548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bb4101790 msgr2=0x7f3bb4197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:50.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.589+0000 7f3bb9cee640 1 --2- 172.21.15.78:0/3815617548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bb4101790 0x7f3bb4197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.589+0000 7f3bb9cee640 1 -- 172.21.15.78:0/3815617548 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3bb4102150 msgr2=0x7f3bb419efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:50.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.589+0000 7f3bb9cee640 1 --2- 172.21.15.78:0/3815617548 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3bb4102150 0x7f3bb419efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.589+0000 7f3bb9cee640 1 -- 172.21.15.78:0/3815617548 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb419f630 con 0x7f3bb41005a0 2024-03-24T00:18:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.589+0000 7f3bb94ed640 1 --2- 172.21.15.78:0/3815617548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bb4101790 0x7f3bb4197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:18:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.589+0000 7f3bb9cee640 1 --2- 172.21.15.78:0/3815617548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bb41005a0 0x7f3bb4197a10 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3bac00ebc0 tx=0x7f3bac00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.589+0000 7f3bb2ffd640 1 -- 172.21.15.78:0/3815617548 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bac003b70 con 0x7f3bb41005a0 2024-03-24T00:18:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.589+0000 7f3bbbf52640 1 -- 172.21.15.78:0/3815617548 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bb419f920 con 0x7f3bb41005a0 2024-03-24T00:18:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.589+0000 7f3bb2ffd640 1 -- 172.21.15.78:0/3815617548 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3bac003d10 con 0x7f3bb41005a0 2024-03-24T00:18:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.589+0000 7f3bb2ffd640 1 -- 172.21.15.78:0/3815617548 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bac019980 con 0x7f3bb41005a0 2024-03-24T00:18:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.589+0000 7f3bbbf52640 1 -- 172.21.15.78:0/3815617548 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bb419fe70 con 0x7f3bb41005a0 2024-03-24T00:18:50.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.591+0000 7f3bb2ffd640 1 -- 172.21.15.78:0/3815617548 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3bac019b20 con 0x7f3bb41005a0 2024-03-24T00:18:50.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.591+0000 7f3bbec10740 1 Processor -- start 2024-03-24T00:18:50.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.592+0000 7f3bbec10740 1 -- start start 2024-03-24T00:18:50.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.592+0000 7f3bbec10740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d06c42740 0x563d06c42b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:50.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.592+0000 7f3bbec10740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d06c43130 0x563d06c47c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:50.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.592+0000 7f3bbec10740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d06c482a0 0x563d06c4a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:50.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.592+0000 7f3bbec10740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d06ba7c20 con 0x563d06c482a0 2024-03-24T00:18:50.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.592+0000 7f3bbec10740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d06b50a40 con 0x563d06c43130 2024-03-24T00:18:50.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.592+0000 7f3bbec10740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d06ba3cc0 con 0x563d06c42740 2024-03-24T00:18:50.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.592+0000 7f3bb2ffd640 1 --2- 172.21.15.78:0/3815617548 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3b9c0a2800 0x7f3b9c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:50.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.592+0000 7f3bb94ed640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d06c482a0 0x563d06c4a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:50.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.592+0000 7f3bb9cee640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d06c43130 0x563d06c47c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:50.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.592+0000 7f3bb94ed640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d06c482a0 0x563d06c4a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43342/0 (socket says 172.21.15.78:43342) 2024-03-24T00:18:50.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.592+0000 7f3bb94ed640 1 -- 172.21.15.78:0/2080457541 learned_addr learned my addr 172.21.15.78:0/2080457541 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:50.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.592+0000 7f3bba4ef640 1 --2- 172.21.15.78:0/2080457541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d06c42740 0x563d06c42b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:50.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.592+0000 7f3bba4ef640 1 --2- 172.21.15.78:0/3815617548 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3b9c0a2800 0x7f3b9c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:50.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.592+0000 7f3bb2ffd640 1 -- 172.21.15.78:0/3815617548 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f3bac0cf590 con 0x7f3bb41005a0 2024-03-24T00:18:50.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.592+0000 7f3bb94ed640 1 -- 172.21.15.78:0/2080457541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d06c42740 msgr2=0x563d06c42b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:50.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.592+0000 7f3bb94ed640 1 --2- 172.21.15.78:0/2080457541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d06c42740 0x563d06c42b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.592+0000 7f3bb94ed640 1 -- 172.21.15.78:0/2080457541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d06c43130 msgr2=0x563d06c47c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:50.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.592+0000 7f3bb94ed640 1 --2- 172.21.15.78:0/2080457541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d06c43130 0x563d06c47c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.592+0000 7f3bb94ed640 1 -- 172.21.15.78:0/2080457541 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d06b72f90 con 0x563d06c482a0 2024-03-24T00:18:50.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.592+0000 7f3bb9cee640 1 --2- 172.21.15.78:0/2080457541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d06c43130 0x563d06c47c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:50.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.593+0000 7f3bba4ef640 1 --2- 172.21.15.78:0/3815617548 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3b9c0a2800 0x7f3b9c0a4cc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x563d06c43990 tx=0x7f3ba4005f60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:50.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.593+0000 7f3bb94ed640 1 --2- 172.21.15.78:0/2080457541 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d06c482a0 0x563d06c4a730 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8006010 tx=0x7f3ba80095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7102a1030328d008 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:50.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.593+0000 7f3b9bfff640 1 -- 172.21.15.78:0/2080457541 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba802cc40 con 0x563d06c482a0 2024-03-24T00:18:50.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.593+0000 7f3b9bfff640 1 -- 172.21.15.78:0/2080457541 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3ba8005040 con 0x563d06c482a0 2024-03-24T00:18:50.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.593+0000 7f3b9bfff640 1 -- 172.21.15.78:0/2080457541 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba8004a40 con 0x563d06c482a0 2024-03-24T00:18:50.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.593+0000 7f3bbec10740 1 -- 172.21.15.78:0/2080457541 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d06c482a0 msgr2=0x563d06c4a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:50.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.593+0000 7f3bbec10740 1 --2- 172.21.15.78:0/2080457541 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d06c482a0 0x563d06c4a730 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8006010 tx=0x7f3ba80095e0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.593+0000 7f3bbec10740 1 -- 172.21.15.78:0/2080457541 shutdown_connections 2024-03-24T00:18:50.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.593+0000 7f3bbec10740 1 --2- 172.21.15.78:0/2080457541 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d06c482a0 0x563d06c4a730 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.593+0000 7f3bbec10740 1 --2- 172.21.15.78:0/2080457541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d06c43130 0x563d06c47c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.593+0000 7f3bbec10740 1 --2- 172.21.15.78:0/2080457541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d06c42740 0x563d06c42b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.593+0000 7f3bbec10740 1 -- 172.21.15.78:0/2080457541 >> 172.21.15.78:0/2080457541 conn(0x563d06c0aa60 msgr2=0x563d06c3e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:50.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.594+0000 7f3bbec10740 1 -- 172.21.15.78:0/2080457541 shutdown_connections 2024-03-24T00:18:50.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.594+0000 7f3bbec10740 1 -- 172.21.15.78:0/2080457541 wait complete. 2024-03-24T00:18:50.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.594+0000 7f3bbec10740 1 -- start start 2024-03-24T00:18:50.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.594+0000 7f3bbec10740 1 Processor -- start 2024-03-24T00:18:50.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.594+0000 7f3bbec10740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d06c42740 0x563d06c4cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:50.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.594+0000 7f3bbec10740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d06c43130 0x563d06c4d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:50.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.594+0000 7f3bbec10740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d06c482a0 0x563d06c54110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:50.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.594+0000 7f3bbec10740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d06c4b1f0 con 0x563d06c482a0 2024-03-24T00:18:50.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.594+0000 7f3bbec10740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d06b63e90 con 0x563d06c42740 2024-03-24T00:18:50.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.594+0000 7f3bbec10740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d06c4b370 con 0x563d06c43130 2024-03-24T00:18:50.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.594+0000 7f3bbec10740 10 client.0 _ll_register_callbacks cb 0x563d068bbac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:50.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.594+0000 7f3bba4ef640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d06c482a0 0x563d06c54110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:50.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.594+0000 7f3bba4ef640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d06c482a0 0x563d06c54110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43346/0 (socket says 172.21.15.78:43346) 2024-03-24T00:18:50.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.594+0000 7f3bba4ef640 1 -- 172.21.15.78:0/4171517314 learned_addr learned my addr 172.21.15.78:0/4171517314 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:50.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.594+0000 7f3bb9cee640 1 --2- 172.21.15.78:0/4171517314 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d06c42740 0x563d06c4cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:50.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.595+0000 7f3bba4ef640 1 -- 172.21.15.78:0/4171517314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d06c43130 msgr2=0x563d06c4d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:50.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.595+0000 7f3bba4ef640 1 --2- 172.21.15.78:0/4171517314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d06c43130 0x563d06c4d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.595+0000 7f3bba4ef640 1 -- 172.21.15.78:0/4171517314 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d06c42740 msgr2=0x563d06c4cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:50.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.595+0000 7f3bba4ef640 1 --2- 172.21.15.78:0/4171517314 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d06c42740 0x563d06c4cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.595+0000 7f3bba4ef640 1 -- 172.21.15.78:0/4171517314 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563d06b709d0 con 0x563d06c482a0 2024-03-24T00:18:50.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.595+0000 7f3bba4ef640 1 --2- 172.21.15.78:0/4171517314 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d06c482a0 0x563d06c54110 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4011510 tx=0x7f3ba40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57cd21d5d99daa85 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:50.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.595+0000 7f3b997fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-24T00:18:50.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.595+0000 7f3b997fa640 1 -- 172.21.15.78:0/4171517314 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba400de70 con 0x563d06c482a0 2024-03-24T00:18:50.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.595+0000 7f3bbec10740 1 -- 172.21.15.78:0/4171517314 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3ba8005040 con 0x563d06c482a0 2024-03-24T00:18:50.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.595+0000 7f3b997fa640 1 -- 172.21.15.78:0/4171517314 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f3ba4005020 con 0x563d06c482a0 2024-03-24T00:18:50.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.595+0000 7f3b997fa640 1 -- 172.21.15.78:0/4171517314 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3ba4032050 con 0x563d06c482a0 2024-03-24T00:18:50.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.595+0000 7f3bbec10740 10 client.14769 fetch_fsmap learned FSMap version 17 2024-03-24T00:18:50.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.595+0000 7f3bbec10740 1 -- 172.21.15.78:0/4171517314 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x563d068bc2d0 con 0x563d06c482a0 2024-03-24T00:18:50.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.595+0000 7f3b997fa640 1 -- 172.21.15.78:0/4171517314 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3ba8005040 con 0x563d06c482a0 2024-03-24T00:18:50.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.595+0000 7f3b997fa640 1 -- 172.21.15.78:0/4171517314 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 17) v1 ==== 7922+0+0 (secure 0 0 0) 0x7f3ba4026030 con 0x563d06c482a0 2024-03-24T00:18:50.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.595+0000 7f3bbec10740 10 client.14769 fetch_fsmap finished waiting for FSMap version 17 2024-03-24T00:18:50.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.595+0000 7f3bbec10740 10 client.14769 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:18:50.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.595+0000 7f3bbec10740 20 client.14769 populate_metadata read hostname 'smithi078' 2024-03-24T00:18:50.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.596+0000 7f3bbec10740 1 --2- 172.21.15.78:0/4171517314 >> [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] conn(0x563d06c59470 0x563d06c58450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:50.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.596+0000 7f3bbec10740 4 client.14769 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:18:50.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.596+0000 7f3bbec10740 1 -- 172.21.15.78:0/4171517314 --> [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563d06bb5160 con 0x563d06c59470 2024-03-24T00:18:50.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.596+0000 7f3bb9cee640 1 --2- 172.21.15.78:0/4171517314 >> [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] conn(0x563d06c59470 0x563d06c58450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:50.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.596+0000 7f3bb9cee640 1 --2- 172.21.15.78:0/4171517314 >> [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] conn(0x563d06c59470 0x563d06c58450 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3fc2e717aa98d9a4 server_cookie=337f99ea2ea5f3c3 in_seq=0 out_seq=0 2024-03-24T00:18:50.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.596+0000 7f3b997fa640 10 client.14769 ms_handle_connect on v2:172.21.15.106:6838/2482071304 2024-03-24T00:18:50.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.597+0000 7f3b997fa640 1 -- 172.21.15.78:0/4171517314 <== mds.0 v2:172.21.15.106:6838/2482071304 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563d06bb5160 con 0x563d06c59470 2024-03-24T00:18:50.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.597+0000 7f3b997fa640 10 client.14769 handle_command_reply: tid=0 2024-03-24T00:18:50.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.597+0000 7f3bbec10740 1 client.14769 shutdown 2024-03-24T00:18:50.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.597+0000 7f3bbec10740 1 -- 172.21.15.78:0/4171517314 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d06c482a0 msgr2=0x563d06c54110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:50.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.597+0000 7f3bbec10740 1 --2- 172.21.15.78:0/4171517314 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d06c482a0 0x563d06c54110 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4011510 tx=0x7f3ba40046c0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.597+0000 7f3bbec10740 1 -- 172.21.15.78:0/4171517314 shutdown_connections 2024-03-24T00:18:50.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.597+0000 7f3bbec10740 1 --2- 172.21.15.78:0/4171517314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d06c43130 0x563d06c4d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.597+0000 7f3bbec10740 1 --2- 172.21.15.78:0/4171517314 >> [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] conn(0x563d06c59470 0x563d06c58450 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.597+0000 7f3bbec10740 1 --2- 172.21.15.78:0/4171517314 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d06c482a0 0x563d06c54110 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.597+0000 7f3bbec10740 1 --2- 172.21.15.78:0/4171517314 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d06c42740 0x563d06c4cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.597+0000 7f3b997fa640 0 client.14769 ms_handle_reset on v2:172.21.15.106:6838/2482071304 2024-03-24T00:18:50.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.597+0000 7f3bbec10740 1 -- 172.21.15.78:0/4171517314 >> 172.21.15.78:0/4171517314 conn(0x563d06c0aa60 msgr2=0x563d06c3d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:50.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.597+0000 7f3bbec10740 1 -- 172.21.15.78:0/4171517314 shutdown_connections 2024-03-24T00:18:50.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.597+0000 7f3bbec10740 1 -- 172.21.15.78:0/4171517314 wait complete. 2024-03-24T00:18:50.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.597+0000 7f3bbec10740 20 client.14769 trim_cache size 0 max 16384 2024-03-24T00:18:50.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.614+0000 7f3bbec10740 1 Processor -- start 2024-03-24T00:18:50.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.614+0000 7f3bbec10740 1 -- start start 2024-03-24T00:18:50.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.614+0000 7f3bbec10740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d06c42740 0x563d06c445d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:50.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.614+0000 7f3bbec10740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d06c43130 0x563d06c44b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:50.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.614+0000 7f3bbec10740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d06c482a0 0x563d06c540d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:50.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.614+0000 7f3bbec10740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d06c58c90 con 0x563d06c42740 2024-03-24T00:18:50.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.614+0000 7f3bbec10740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d06c592f0 con 0x563d06c482a0 2024-03-24T00:18:50.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.614+0000 7f3bbec10740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d06c59950 con 0x563d06c43130 2024-03-24T00:18:50.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.614+0000 7f3bb9cee640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d06c42740 0x563d06c445d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:50.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.614+0000 7f3bb9cee640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d06c42740 0x563d06c445d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43362/0 (socket says 172.21.15.78:43362) 2024-03-24T00:18:50.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.614+0000 7f3bb9cee640 1 -- 172.21.15.78:0/3812795049 learned_addr learned my addr 172.21.15.78:0/3812795049 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:50.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.614+0000 7f3bba4ef640 1 --2- 172.21.15.78:0/3812795049 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d06c482a0 0x563d06c540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:50.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.614+0000 7f3bb9cee640 1 -- 172.21.15.78:0/3812795049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d06c43130 msgr2=0x563d06c44b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:50.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.614+0000 7f3bb9cee640 1 --2- 172.21.15.78:0/3812795049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d06c43130 0x563d06c44b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.614+0000 7f3bb9cee640 1 -- 172.21.15.78:0/3812795049 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d06c482a0 msgr2=0x563d06c540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:50.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.614+0000 7f3bb9cee640 1 --2- 172.21.15.78:0/3812795049 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d06c482a0 0x563d06c540d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.614+0000 7f3bb9cee640 1 -- 172.21.15.78:0/3812795049 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d069866b0 con 0x563d06c42740 2024-03-24T00:18:50.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.614+0000 7f3bba4ef640 1 --2- 172.21.15.78:0/3812795049 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d06c482a0 0x563d06c540d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:18:50.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.614+0000 7f3bb9cee640 1 --2- 172.21.15.78:0/3812795049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d06c42740 0x563d06c445d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3bac0049d0 tx=0x7f3bac0c7040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=282736bdac5977c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:50.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.614+0000 7f3b9b7fe640 1 -- 172.21.15.78:0/3812795049 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bac0b3070 con 0x563d06c42740 2024-03-24T00:18:50.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.614+0000 7f3b9b7fe640 1 -- 172.21.15.78:0/3812795049 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3bac09bc90 con 0x563d06c42740 2024-03-24T00:18:50.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.614+0000 7f3b9b7fe640 1 -- 172.21.15.78:0/3812795049 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bac098300 con 0x563d06c42740 2024-03-24T00:18:50.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.615+0000 7f3bbec10740 1 -- 172.21.15.78:0/3812795049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d06c42740 msgr2=0x563d06c445d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:50.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.615+0000 7f3bbec10740 1 --2- 172.21.15.78:0/3812795049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d06c42740 0x563d06c445d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3bac0049d0 tx=0x7f3bac0c7040 comp rx=0 tx=0).stop 2024-03-24T00:18:50.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.615+0000 7f3bbec10740 1 -- 172.21.15.78:0/3812795049 shutdown_connections 2024-03-24T00:18:50.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.615+0000 7f3bbec10740 1 --2- 172.21.15.78:0/3812795049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d06c43130 0x563d06c44b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.615+0000 7f3bbec10740 1 --2- 172.21.15.78:0/3812795049 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d06c482a0 0x563d06c540d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.615+0000 7f3bbec10740 1 --2- 172.21.15.78:0/3812795049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d06c42740 0x563d06c445d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.615+0000 7f3bbec10740 1 -- 172.21.15.78:0/3812795049 >> 172.21.15.78:0/3812795049 conn(0x563d06c0aa60 msgr2=0x563d06c4cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:50.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.615+0000 7f3bbec10740 1 -- 172.21.15.78:0/3812795049 shutdown_connections 2024-03-24T00:18:50.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.615+0000 7f3bbec10740 1 -- 172.21.15.78:0/3812795049 wait complete. 2024-03-24T00:18:50.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.615+0000 7f3bbec10740 1 -- start start 2024-03-24T00:18:50.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.615+0000 7f3bbec10740 1 Processor -- start 2024-03-24T00:18:50.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.616+0000 7f3bbec10740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d06c42740 0x563d06c50330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:50.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.616+0000 7f3bbec10740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d06c43130 0x563d06c50870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:50.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.616+0000 7f3bbec10740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d06c482a0 0x563d06c53890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:50.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.616+0000 7f3bbec10740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d06c5a610 con 0x563d06c42740 2024-03-24T00:18:50.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.616+0000 7f3bbec10740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d06c59fb0 con 0x563d06c482a0 2024-03-24T00:18:50.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.616+0000 7f3bbec10740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d06c5ac70 con 0x563d06c43130 2024-03-24T00:18:50.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.616+0000 7f3bbec10740 10 client.0 _ll_register_callbacks cb 0x7f3ba400a940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:50.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.616+0000 7f3bb9cee640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d06c42740 0x563d06c50330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:50.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.616+0000 7f3bb9cee640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d06c42740 0x563d06c50330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43372/0 (socket says 172.21.15.78:43372) 2024-03-24T00:18:50.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.616+0000 7f3bb9cee640 1 -- 172.21.15.78:0/898571425 learned_addr learned my addr 172.21.15.78:0/898571425 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:50.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.616+0000 7f3bba4ef640 1 --2- 172.21.15.78:0/898571425 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d06c482a0 0x563d06c53890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:50.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.616+0000 7f3bb9cee640 1 -- 172.21.15.78:0/898571425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d06c43130 msgr2=0x563d06c50870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:50.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.616+0000 7f3bb9cee640 1 --2- 172.21.15.78:0/898571425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d06c43130 0x563d06c50870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.616+0000 7f3bb9cee640 1 -- 172.21.15.78:0/898571425 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d06c482a0 msgr2=0x563d06c53890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:50.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.616+0000 7f3bb9cee640 1 --2- 172.21.15.78:0/898571425 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d06c482a0 0x563d06c53890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.616+0000 7f3bb9cee640 1 -- 172.21.15.78:0/898571425 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563d06c53e80 con 0x563d06c42740 2024-03-24T00:18:50.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.616+0000 7f3bb9cee640 1 --2- 172.21.15.78:0/898571425 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d06c42740 0x563d06c50330 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3bac004950 tx=0x7f3bac00eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a82b842ae150038 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:50.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.616+0000 7f3b997fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-24T00:18:50.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.616+0000 7f3b997fa640 1 -- 172.21.15.78:0/898571425 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bac0b3040 con 0x563d06c42740 2024-03-24T00:18:50.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.616+0000 7f3b997fa640 1 -- 172.21.15.78:0/898571425 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f3bac0a3fa0 con 0x563d06c42740 2024-03-24T00:18:50.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.616+0000 7f3bbec10740 1 -- 172.21.15.78:0/898571425 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3bac09bc90 con 0x563d06c42740 2024-03-24T00:18:50.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.616+0000 7f3b997fa640 1 -- 172.21.15.78:0/898571425 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3bac019e10 con 0x563d06c42740 2024-03-24T00:18:50.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.616+0000 7f3bbec10740 10 client.14775 fetch_fsmap learned FSMap version 17 2024-03-24T00:18:50.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.616+0000 7f3bbec10740 1 -- 172.21.15.78:0/898571425 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x563d06cb4f90 con 0x563d06c42740 2024-03-24T00:18:50.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.616+0000 7f3b997fa640 1 -- 172.21.15.78:0/898571425 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3bac09bc90 con 0x563d06c42740 2024-03-24T00:18:50.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.617+0000 7f3b997fa640 1 -- 172.21.15.78:0/898571425 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 17) v1 ==== 7922+0+0 (secure 0 0 0) 0x7f3bac004b60 con 0x563d06c42740 2024-03-24T00:18:50.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.617+0000 7f3bbec10740 10 client.14775 fetch_fsmap finished waiting for FSMap version 17 2024-03-24T00:18:50.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.617+0000 7f3bbec10740 10 client.14775 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:18:50.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.617+0000 7f3bbec10740 20 client.14775 populate_metadata read hostname 'smithi078' 2024-03-24T00:18:50.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.617+0000 7f3bbec10740 1 --2- 172.21.15.78:0/898571425 >> [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] conn(0x563d06cb9150 0x563d06cb5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:50.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.617+0000 7f3bbec10740 4 client.14775 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:18:50.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.617+0000 7f3bbec10740 1 -- 172.21.15.78:0/898571425 --> [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563d06baf4b0 con 0x563d06cb9150 2024-03-24T00:18:50.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.617+0000 7f3bb94ed640 1 --2- 172.21.15.78:0/898571425 >> [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] conn(0x563d06cb9150 0x563d06cb5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:50.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.617+0000 7f3b997fa640 10 client.14775 ms_handle_connect on v2:172.21.15.106:6838/2482071304 2024-03-24T00:18:50.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.617+0000 7f3bb94ed640 1 --2- 172.21.15.78:0/898571425 >> [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] conn(0x563d06cb9150 0x563d06cb5160 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1af5716763fb1bb9 server_cookie=3079f59aebf004d4 in_seq=0 out_seq=0 2024-03-24T00:18:50.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.617+0000 7f3b997fa640 1 -- 172.21.15.78:0/898571425 <== mds.0 v2:172.21.15.106:6838/2482071304 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x563d06baf4b0 con 0x563d06cb9150 2024-03-24T00:18:50.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.617+0000 7f3b997fa640 10 client.14775 handle_command_reply: tid=0 2024-03-24T00:18:50.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.617+0000 7f3bbec10740 1 client.14775 shutdown 2024-03-24T00:18:50.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.618+0000 7f3bbec10740 1 -- 172.21.15.78:0/898571425 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d06c42740 msgr2=0x563d06c50330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:50.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.618+0000 7f3bbec10740 1 --2- 172.21.15.78:0/898571425 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d06c42740 0x563d06c50330 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3bac004950 tx=0x7f3bac00eec0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.618+0000 7f3bbec10740 1 -- 172.21.15.78:0/898571425 shutdown_connections 2024-03-24T00:18:50.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.618+0000 7f3bbec10740 1 --2- 172.21.15.78:0/898571425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d06c43130 0x563d06c50870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.618+0000 7f3bbec10740 1 --2- 172.21.15.78:0/898571425 >> [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] conn(0x563d06cb9150 0x563d06cb5160 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.618+0000 7f3bbec10740 1 --2- 172.21.15.78:0/898571425 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d06c482a0 0x563d06c53890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.618+0000 7f3bbec10740 1 --2- 172.21.15.78:0/898571425 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563d06c42740 0x563d06c50330 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.618+0000 7f3b997fa640 0 client.14775 ms_handle_reset on v2:172.21.15.106:6838/2482071304 2024-03-24T00:18:50.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.618+0000 7f3bbec10740 1 -- 172.21.15.78:0/898571425 >> 172.21.15.78:0/898571425 conn(0x563d06c0aa60 msgr2=0x563d06c445d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:50.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.618+0000 7f3bbec10740 1 -- 172.21.15.78:0/898571425 shutdown_connections 2024-03-24T00:18:50.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.618+0000 7f3bbec10740 1 -- 172.21.15.78:0/898571425 wait complete. 2024-03-24T00:18:50.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.618+0000 7f3bbec10740 20 client.14775 trim_cache size 0 max 16384 2024-03-24T00:18:50.625 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-24T00:18:50.625 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "c7e2806a-e972-11ee-95cd-87774f69a715", 2024-03-24T00:18:50.625 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-03-24T00:18:50.625 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24500, 2024-03-24T00:18:50.626 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-03-24T00:18:50.626 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-03-24T00:18:50.626 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-03-24T00:18:50.626 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 6.3761071720000002, 2024-03-24T00:18:50.626 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 17, 2024-03-24T00:18:50.626 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 89, 2024-03-24T00:18:50.626 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 89, 2024-03-24T00:18:50.626 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 12.751513139 2024-03-24T00:18:50.626 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-24T00:18:50.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.618+0000 7f3bbbf52640 1 -- 172.21.15.78:0/3815617548 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3b9c0a2800 msgr2=0x7f3b9c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:50.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.618+0000 7f3bbbf52640 1 --2- 172.21.15.78:0/3815617548 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3b9c0a2800 0x7f3b9c0a4cc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x563d06c43990 tx=0x7f3ba4005f60 comp rx=0 tx=0).stop 2024-03-24T00:18:50.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.618+0000 7f3bbbf52640 1 -- 172.21.15.78:0/3815617548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bb41005a0 msgr2=0x7f3bb4197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:50.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.618+0000 7f3bbbf52640 1 --2- 172.21.15.78:0/3815617548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bb41005a0 0x7f3bb4197a10 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3bac00ebc0 tx=0x7f3bac00c6a0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.618+0000 7f3bbbf52640 1 -- 172.21.15.78:0/3815617548 shutdown_connections 2024-03-24T00:18:50.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.618+0000 7f3bbbf52640 1 --2- 172.21.15.78:0/3815617548 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3b9c0a2800 0x7f3b9c0a4cc0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.618+0000 7f3bbbf52640 1 --2- 172.21.15.78:0/3815617548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bb4101790 0x7f3bb4197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.618+0000 7f3bbbf52640 1 --2- 172.21.15.78:0/3815617548 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3bb4102150 0x7f3bb419efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.618+0000 7f3bbbf52640 1 --2- 172.21.15.78:0/3815617548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bb41005a0 0x7f3bb4197a10 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:50.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.618+0000 7f3bbbf52640 1 -- 172.21.15.78:0/3815617548 >> 172.21.15.78:0/3815617548 conn(0x7f3bb40fbb30 msgr2=0x7f3bb40fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:50.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.618+0000 7f3bbbf52640 1 -- 172.21.15.78:0/3815617548 shutdown_connections 2024-03-24T00:18:50.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:50.618+0000 7f3bbbf52640 1 -- 172.21.15.78:0/3815617548 wait complete. 2024-03-24T00:18:50.629 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-24T00:18:50.629 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-24T00:18:50.629 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph fs ls 2024-03-24T00:18:50.866 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:18:51.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:50 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/4089936624' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:51.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:50 smithi106 ceph-mon[36505]: pgmap v294: 129 pgs: 14 creating+peering, 9 unknown, 106 active+clean; 579 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 10 op/s 2024-03-24T00:18:51.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:50 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3306264362' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:18:51.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:50 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/4089936624' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:51.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:50 smithi078 ceph-mon[32429]: pgmap v294: 129 pgs: 14 creating+peering, 9 unknown, 106 active+clean; 579 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 10 op/s 2024-03-24T00:18:51.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:50 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3306264362' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:18:51.244 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:50 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/4089936624' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:51.244 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:50 smithi107 ceph-mon[36485]: pgmap v294: 129 pgs: 14 creating+peering, 9 unknown, 106 active+clean; 579 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 10 op/s 2024-03-24T00:18:51.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:50 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3306264362' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:18:52.135 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:51 smithi078 ceph-mon[32429]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T00:18:52.135 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:51 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:18:52.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:51 smithi106 ceph-mon[36505]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T00:18:52.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:51 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:18:52.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:51 smithi107 ceph-mon[36485]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T00:18:52.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:51 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:18:52.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.495+0000 7fbeeabe3700 1 -- 172.21.15.78:0/850017636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbee4101320 msgr2=0x7fbee4101700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:52.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.495+0000 7fbeeabe3700 1 --2- 172.21.15.78:0/850017636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbee4101320 0x7fbee4101700 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbed80099c0 tx=0x7fbed8009cd0 comp rx=0 tx=0).stop 2024-03-24T00:18:52.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.495+0000 7fbeeabe3700 1 -- 172.21.15.78:0/850017636 shutdown_connections 2024-03-24T00:18:52.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.495+0000 7fbeeabe3700 1 --2- 172.21.15.78:0/850017636 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbee4106370 0x7fbee410df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:52.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.495+0000 7fbeeabe3700 1 --2- 172.21.15.78:0/850017636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbee4101320 0x7fbee4101700 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:52.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.495+0000 7fbeeabe3700 1 --2- 172.21.15.78:0/850017636 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbee4101cd0 0x7fbee4105c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:52.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.495+0000 7fbeeabe3700 1 -- 172.21.15.78:0/850017636 >> 172.21.15.78:0/850017636 conn(0x7fbee40770e0 msgr2=0x7fbee40774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:52.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.495+0000 7fbeeabe3700 1 -- 172.21.15.78:0/850017636 shutdown_connections 2024-03-24T00:18:52.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.495+0000 7fbeeabe3700 1 -- 172.21.15.78:0/850017636 wait complete. 2024-03-24T00:18:52.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.496+0000 7fbeeabe3700 1 Processor -- start 2024-03-24T00:18:52.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.496+0000 7fbeeabe3700 1 -- start start 2024-03-24T00:18:52.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.496+0000 7fbeeabe3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbee4101320 0x7fbee419e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:52.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.496+0000 7fbeeabe3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbee4101cd0 0x7fbee419eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:52.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.496+0000 7fbeeabe3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbee4106370 0x7fbee41987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:52.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.496+0000 7fbeeabe3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee4111380 con 0x7fbee4101320 2024-03-24T00:18:52.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.496+0000 7fbeeabe3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee4111200 con 0x7fbee4106370 2024-03-24T00:18:52.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.496+0000 7fbeeabe3700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee4111500 con 0x7fbee4101cd0 2024-03-24T00:18:52.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.496+0000 7fbee897f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbee4101320 0x7fbee419e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:52.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.496+0000 7fbee897f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbee4101320 0x7fbee419e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43392/0 (socket says 172.21.15.78:43392) 2024-03-24T00:18:52.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.496+0000 7fbee897f700 1 -- 172.21.15.78:0/3027054152 learned_addr learned my addr 172.21.15.78:0/3027054152 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:52.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.496+0000 7fbee9180700 1 --2- 172.21.15.78:0/3027054152 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbee4106370 0x7fbee41987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:52.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.496+0000 7fbee3fff700 1 --2- 172.21.15.78:0/3027054152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbee4101cd0 0x7fbee419eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:52.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.496+0000 7fbee897f700 1 -- 172.21.15.78:0/3027054152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbee4101cd0 msgr2=0x7fbee419eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:52.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.496+0000 7fbee897f700 1 --2- 172.21.15.78:0/3027054152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbee4101cd0 0x7fbee419eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:52.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.496+0000 7fbee897f700 1 -- 172.21.15.78:0/3027054152 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbee4106370 msgr2=0x7fbee41987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:52.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.496+0000 7fbee897f700 1 --2- 172.21.15.78:0/3027054152 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbee4106370 0x7fbee41987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:52.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.496+0000 7fbee897f700 1 -- 172.21.15.78:0/3027054152 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee4198fa0 con 0x7fbee4101320 2024-03-24T00:18:52.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.497+0000 7fbee897f700 1 --2- 172.21.15.78:0/3027054152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbee4101320 0x7fbee419e640 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbed80059d0 tx=0x7fbed800fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:52.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.497+0000 7fbee1ffb700 1 -- 172.21.15.78:0/3027054152 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed8027070 con 0x7fbee4101320 2024-03-24T00:18:52.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.497+0000 7fbee1ffb700 1 -- 172.21.15.78:0/3027054152 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbed8003680 con 0x7fbee4101320 2024-03-24T00:18:52.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.497+0000 7fbee1ffb700 1 -- 172.21.15.78:0/3027054152 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed80173f0 con 0x7fbee4101320 2024-03-24T00:18:52.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.497+0000 7fbeeabe3700 1 -- 172.21.15.78:0/3027054152 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbee4199230 con 0x7fbee4101320 2024-03-24T00:18:52.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.497+0000 7fbeeabe3700 1 -- 172.21.15.78:0/3027054152 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbee41a54a0 con 0x7fbee4101320 2024-03-24T00:18:52.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.498+0000 7fbeeabe3700 1 -- 172.21.15.78:0/3027054152 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbee4102770 con 0x7fbee4101320 2024-03-24T00:18:52.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.499+0000 7fbee1ffb700 1 -- 172.21.15.78:0/3027054152 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fbed802a070 con 0x7fbee4101320 2024-03-24T00:18:52.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.499+0000 7fbee1ffb700 1 --2- 172.21.15.78:0/3027054152 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fbed40a2910 0x7fbed40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:52.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.500+0000 7fbee3fff700 1 --2- 172.21.15.78:0/3027054152 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fbed40a2910 0x7fbed40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:52.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.500+0000 7fbee1ffb700 1 -- 172.21.15.78:0/3027054152 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fbed80cf770 con 0x7fbee4101320 2024-03-24T00:18:52.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.500+0000 7fbee3fff700 1 --2- 172.21.15.78:0/3027054152 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fbed40a2910 0x7fbed40a4dd0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbed0009680 tx=0x7fbed0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:52.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.503+0000 7fbee1ffb700 1 -- 172.21.15.78:0/3027054152 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbed8013070 con 0x7fbee4101320 2024-03-24T00:18:52.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.742+0000 7fbeeabe3700 1 -- 172.21.15.78:0/3027054152 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fbee402da30 con 0x7fbee4101320 2024-03-24T00:18:52.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.743+0000 7fbee1ffb700 1 -- 172.21.15.78:0/3027054152 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+88 (secure 0 0 0) 0x7fbed809c660 con 0x7fbee4101320 2024-03-24T00:18:52.743 INFO:teuthology.orchestra.run.smithi078.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-24T00:18:52.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.745+0000 7fbeeabe3700 1 -- 172.21.15.78:0/3027054152 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fbed40a2910 msgr2=0x7fbed40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:52.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.745+0000 7fbeeabe3700 1 --2- 172.21.15.78:0/3027054152 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fbed40a2910 0x7fbed40a4dd0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbed0009680 tx=0x7fbed0009260 comp rx=0 tx=0).stop 2024-03-24T00:18:52.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.745+0000 7fbeeabe3700 1 -- 172.21.15.78:0/3027054152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbee4101320 msgr2=0x7fbee419e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:52.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.745+0000 7fbeeabe3700 1 --2- 172.21.15.78:0/3027054152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbee4101320 0x7fbee419e640 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbed80059d0 tx=0x7fbed800fa00 comp rx=0 tx=0).stop 2024-03-24T00:18:52.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.745+0000 7fbeeabe3700 1 -- 172.21.15.78:0/3027054152 shutdown_connections 2024-03-24T00:18:52.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.745+0000 7fbeeabe3700 1 --2- 172.21.15.78:0/3027054152 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fbed40a2910 0x7fbed40a4dd0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:52.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.745+0000 7fbeeabe3700 1 --2- 172.21.15.78:0/3027054152 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbee4106370 0x7fbee41987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:52.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.745+0000 7fbeeabe3700 1 --2- 172.21.15.78:0/3027054152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbee4101320 0x7fbee419e640 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:52.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.745+0000 7fbeeabe3700 1 --2- 172.21.15.78:0/3027054152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbee4101cd0 0x7fbee419eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:52.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.745+0000 7fbeeabe3700 1 -- 172.21.15.78:0/3027054152 >> 172.21.15.78:0/3027054152 conn(0x7fbee40770e0 msgr2=0x7fbee406aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:52.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.745+0000 7fbeeabe3700 1 -- 172.21.15.78:0/3027054152 shutdown_connections 2024-03-24T00:18:52.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:52.745+0000 7fbeeabe3700 1 -- 172.21.15.78:0/3027054152 wait complete. 2024-03-24T00:18:53.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:52 smithi078 ceph-mon[32429]: pgmap v295: 129 pgs: 10 creating+peering, 119 active+clean; 579 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 7 op/s 2024-03-24T00:18:53.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:52 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3027054152' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T00:18:53.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:52 smithi106 ceph-mon[36505]: pgmap v295: 129 pgs: 10 creating+peering, 119 active+clean; 579 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 7 op/s 2024-03-24T00:18:53.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:52 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3027054152' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T00:18:53.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:52 smithi107 ceph-mon[36485]: pgmap v295: 129 pgs: 10 creating+peering, 119 active+clean; 579 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 7 op/s 2024-03-24T00:18:53.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:52 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3027054152' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T00:18:53.435 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-24T00:18:53.435 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:18:53.435 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi078.front.sepia.ceph.com 2024-03-24T00:18:53.435 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-24T00:18:53.435 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-24T00:18:53.435 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-24T00:18:53.435 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T00:18:53.435 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:18:53.435 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-24T00:18:53.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:18:53.436 DEBUG:teuthology.orchestra.run.smithi078:> ip addr 2024-03-24T00:18:53.452 INFO:teuthology.orchestra.run.smithi078.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:18:53.452 INFO:teuthology.orchestra.run.smithi078.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:18:53.452 INFO:teuthology.orchestra.run.smithi078.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:18:53.452 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:18:53.452 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 ::1/128 scope host 2024-03-24T00:18:53.452 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:18:53.452 INFO:teuthology.orchestra.run.smithi078.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:18:53.452 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:22 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:18:53.452 INFO:teuthology.orchestra.run.smithi078.stdout: altname enp7s0f0 2024-03-24T00:18:53.452 INFO:teuthology.orchestra.run.smithi078.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:18:53.453 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:23 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:18:53.453 INFO:teuthology.orchestra.run.smithi078.stdout: altname enp7s0f1 2024-03-24T00:18:53.453 INFO:teuthology.orchestra.run.smithi078.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:18:53.453 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dc brd ff:ff:ff:ff:ff:ff 2024-03-24T00:18:53.453 INFO:teuthology.orchestra.run.smithi078.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:18:53.453 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dd brd ff:ff:ff:ff:ff:ff 2024-03-24T00:18:53.453 INFO:teuthology.orchestra.run.smithi078.stdout: inet 172.21.15.78/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T00:18:53.453 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft 42220sec preferred_lft 42220sec 2024-03-24T00:18:53.453 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 fe80::ec4:7aff:fe8f:d4dd/64 scope link 2024-03-24T00:18:53.453 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:18:53.454 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-24T00:18:53.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:18:53.454 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2024-03-24T00:18:53.454 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-03-24T00:18:53.454 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link add name ceph-brx type bridge 2024-03-24T00:18:53.454 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip addr flush dev ceph-brx 2024-03-24T00:18:53.454 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set ceph-brx up 2024-03-24T00:18:53.454 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-24T00:18:53.455 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-03-24T00:18:53.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:18:53.628 DEBUG:teuthology.orchestra.run.smithi078:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:18:53.655 INFO:teuthology.orchestra.run.smithi078.stdout:1 2024-03-24T00:18:53.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:18:53.657 DEBUG:teuthology.orchestra.run.smithi078:> route 2024-03-24T00:18:53.737 INFO:teuthology.orchestra.run.smithi078.stdout:Kernel IP routing table 2024-03-24T00:18:53.737 INFO:teuthology.orchestra.run.smithi078.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:18:53.737 INFO:teuthology.orchestra.run.smithi078.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T00:18:53.737 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T00:18:53.737 INFO:teuthology.orchestra.run.smithi078.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:18:53.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:18:53.738 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2024-03-24T00:18:53.738 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-03-24T00:18:53.738 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T00:18:53.738 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T00:18:53.738 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T00:18:53.738 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-03-24T00:18:53.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:18:54.047 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2024-03-24T00:18:54.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:18:54.063 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list-id 2024-03-24T00:18:54.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:18:54.119 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2024-03-24T00:18:54.119 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-03-24T00:18:54.119 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-24T00:18:54.119 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-24T00:18:54.119 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-03-24T00:18:54.201 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-24T00:18:54.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:18:54.202 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2024-03-24T00:18:54.202 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-03-24T00:18:54.202 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-24T00:18:54.202 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-24T00:18:54.202 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-24T00:18:54.202 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-24T00:18:54.202 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-24T00:18:54.202 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-03-24T00:18:54.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:18:54.340 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2024-03-24T00:18:54.340 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-03-24T00:18:54.340 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set brx.0 up 2024-03-24T00:18:54.341 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set dev brx.0 master ceph-brx 2024-03-24T00:18:54.341 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-03-24T00:18:54.382 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-24T00:18:54.382 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:18:54.382 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-24T00:18:54.437 INFO:teuthology.orchestra.run.smithi078.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-24T00:18:54.437 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:18:54.437 DEBUG:teuthology.orchestra.run.smithi078:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-24T00:18:54.492 DEBUG:teuthology.orchestra.run.smithi078:> sudo modprobe fuse 2024-03-24T00:18:54.560 DEBUG:teuthology.orchestra.run.smithi078:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T00:18:54.616 INFO:teuthology.orchestra.run.smithi078.stdout:/proc 2024-03-24T00:18:54.616 INFO:teuthology.orchestra.run.smithi078.stdout:/sys 2024-03-24T00:18:54.616 INFO:teuthology.orchestra.run.smithi078.stdout:/dev 2024-03-24T00:18:54.616 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/security 2024-03-24T00:18:54.616 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/shm 2024-03-24T00:18:54.616 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/pts 2024-03-24T00:18:54.616 INFO:teuthology.orchestra.run.smithi078.stdout:/run 2024-03-24T00:18:54.616 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/cgroup 2024-03-24T00:18:54.616 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/pstore 2024-03-24T00:18:54.616 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/bpf 2024-03-24T00:18:54.616 INFO:teuthology.orchestra.run.smithi078.stdout:/ 2024-03-24T00:18:54.616 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/selinux 2024-03-24T00:18:54.617 INFO:teuthology.orchestra.run.smithi078.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T00:18:54.617 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/hugepages 2024-03-24T00:18:54.617 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/mqueue 2024-03-24T00:18:54.617 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/debug 2024-03-24T00:18:54.617 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/tracing 2024-03-24T00:18:54.617 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T00:18:54.617 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/fuse/connections 2024-03-24T00:18:54.617 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/config 2024-03-24T00:18:54.617 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T00:18:54.617 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T00:18:54.617 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T00:18:54.617 INFO:teuthology.orchestra.run.smithi078.stdout:/run/user/1000 2024-03-24T00:18:54.617 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph 2024-03-24T00:18:54.618 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay 2024-03-24T00:18:54.618 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/e7b1b482a113a0ff7e53cc7c3e69a311ae73cfa73ed294530d851ffffd03a9fb/merged 2024-03-24T00:18:54.618 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/836075a9eb57d2f287a1c08cae3365e912ef285bd576dfbd4529bb8a00fc0094/merged 2024-03-24T00:18:54.618 INFO:teuthology.orchestra.run.smithi078.stdout:/run/user/0 2024-03-24T00:18:54.618 INFO:teuthology.orchestra.run.smithi078.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T00:18:54.618 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/660b0fa5df42f8f1ac2a83b5e91aacb4a0bf276fd34107ccd3e5c06e96ea805a/merged 2024-03-24T00:18:54.618 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/d13b30b2053acb039309feee811247995f10a8aa81af3835ac2444d9f005f3af/merged 2024-03-24T00:18:54.618 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/31caf2291fe46f3f576a490ad8794999585c7ea587c14858a05884722f024f7f/merged 2024-03-24T00:18:54.618 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/a3bbbb6b63f54fc44b0914f7a5bc8fdaa4b2744163a16d501d6a5b8eb8e9ab09/merged 2024-03-24T00:18:54.618 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/6a70077972f19ca3cdf2f99f3fbd7961e51c47926225fe825de5eba90afb5418/merged 2024-03-24T00:18:54.618 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/1f1d4aa56bf70592b232d9557850fc3458cad2d1be32f455592819f306160161/merged 2024-03-24T00:18:54.618 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/49ce3f61f5372aeb32a24e41da513ceb8967856187e6ea3fe0b7cd6ce582db1f/merged 2024-03-24T00:18:54.618 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/1b8395c1d9180ac4a0b4d41009157f82c0b345c1b93f89ab738696ab91ae93d2/merged 2024-03-24T00:18:54.618 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns 2024-03-24T00:18:54.619 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-24T00:18:54.619 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-24T00:18:54.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:18:54.620 DEBUG:teuthology.orchestra.run.smithi078:> ls /sys/fs/fuse/connections 2024-03-24T00:18:54.674 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-24T00:18:54.674 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-24T00:18:54.718 DEBUG:teuthology.orchestra.run.smithi078:> sudo modprobe fuse 2024-03-24T00:18:54.744 DEBUG:teuthology.orchestra.run.smithi078:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T00:18:54.799 INFO:teuthology.orchestra.run.smithi078.stdout:/proc 2024-03-24T00:18:54.800 INFO:teuthology.orchestra.run.smithi078.stdout:/sys 2024-03-24T00:18:54.800 INFO:teuthology.orchestra.run.smithi078.stdout:/dev 2024-03-24T00:18:54.800 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/security 2024-03-24T00:18:54.800 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/shm 2024-03-24T00:18:54.800 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/pts 2024-03-24T00:18:54.800 INFO:teuthology.orchestra.run.smithi078.stdout:/run 2024-03-24T00:18:54.800 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/cgroup 2024-03-24T00:18:54.800 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/pstore 2024-03-24T00:18:54.800 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/bpf 2024-03-24T00:18:54.800 INFO:teuthology.orchestra.run.smithi078.stdout:/ 2024-03-24T00:18:54.800 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/selinux 2024-03-24T00:18:54.800 INFO:teuthology.orchestra.run.smithi078.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T00:18:54.801 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/hugepages 2024-03-24T00:18:54.801 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/mqueue 2024-03-24T00:18:54.801 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/debug 2024-03-24T00:18:54.801 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/tracing 2024-03-24T00:18:54.801 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T00:18:54.801 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/fuse/connections 2024-03-24T00:18:54.801 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/config 2024-03-24T00:18:54.801 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T00:18:54.801 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T00:18:54.801 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T00:18:54.801 INFO:teuthology.orchestra.run.smithi078.stdout:/run/user/1000 2024-03-24T00:18:54.801 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph 2024-03-24T00:18:54.801 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay 2024-03-24T00:18:54.801 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/e7b1b482a113a0ff7e53cc7c3e69a311ae73cfa73ed294530d851ffffd03a9fb/merged 2024-03-24T00:18:54.801 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/836075a9eb57d2f287a1c08cae3365e912ef285bd576dfbd4529bb8a00fc0094/merged 2024-03-24T00:18:54.802 INFO:teuthology.orchestra.run.smithi078.stdout:/run/user/0 2024-03-24T00:18:54.802 INFO:teuthology.orchestra.run.smithi078.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T00:18:54.802 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/660b0fa5df42f8f1ac2a83b5e91aacb4a0bf276fd34107ccd3e5c06e96ea805a/merged 2024-03-24T00:18:54.802 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/d13b30b2053acb039309feee811247995f10a8aa81af3835ac2444d9f005f3af/merged 2024-03-24T00:18:54.802 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/31caf2291fe46f3f576a490ad8794999585c7ea587c14858a05884722f024f7f/merged 2024-03-24T00:18:54.802 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/a3bbbb6b63f54fc44b0914f7a5bc8fdaa4b2744163a16d501d6a5b8eb8e9ab09/merged 2024-03-24T00:18:54.802 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/6a70077972f19ca3cdf2f99f3fbd7961e51c47926225fe825de5eba90afb5418/merged 2024-03-24T00:18:54.802 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/1f1d4aa56bf70592b232d9557850fc3458cad2d1be32f455592819f306160161/merged 2024-03-24T00:18:54.802 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/49ce3f61f5372aeb32a24e41da513ceb8967856187e6ea3fe0b7cd6ce582db1f/merged 2024-03-24T00:18:54.802 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/1b8395c1d9180ac4a0b4d41009157f82c0b345c1b93f89ab738696ab91ae93d2/merged 2024-03-24T00:18:54.802 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns 2024-03-24T00:18:54.802 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-24T00:18:54.802 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-24T00:18:54.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:18:54.803 DEBUG:teuthology.orchestra.run.smithi078:> ls /sys/fs/fuse/connections 2024-03-24T00:18:54.817 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi078.stderr:2024-03-24T00:18:54.817+0000 7fbb7ac3a480 -1 init, newargv = 0x563ee8592d70 newargc=15 2024-03-24T00:18:54.817 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi078.stderr:ceph-fuse[73883]: starting ceph client 2024-03-24T00:18:54.822 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi078.stderr:ceph-fuse[73883]: starting fuse 2024-03-24T00:18:54.831 INFO:teuthology.orchestra.run.smithi078.stdout:119 2024-03-24T00:18:54.831 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-03-24T00:18:54.831 DEBUG:teuthology.orchestra.run.smithi078:> sudo stdin-killer -- python3 -c ' 2024-03-24T00:18:54.832 DEBUG:teuthology.orchestra.run.smithi078:> import glob 2024-03-24T00:18:54.832 DEBUG:teuthology.orchestra.run.smithi078:> import re 2024-03-24T00:18:54.832 DEBUG:teuthology.orchestra.run.smithi078:> import os 2024-03-24T00:18:54.832 DEBUG:teuthology.orchestra.run.smithi078:> import subprocess 2024-03-24T00:18:54.832 DEBUG:teuthology.orchestra.run.smithi078:> 2024-03-24T00:18:54.832 DEBUG:teuthology.orchestra.run.smithi078:> def _find_admin_socket(client_name): 2024-03-24T00:18:54.832 DEBUG:teuthology.orchestra.run.smithi078:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-24T00:18:54.832 DEBUG:teuthology.orchestra.run.smithi078:> files = glob.glob(asok_path) 2024-03-24T00:18:54.832 DEBUG:teuthology.orchestra.run.smithi078:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-24T00:18:54.832 DEBUG:teuthology.orchestra.run.smithi078:> 2024-03-24T00:18:54.832 DEBUG:teuthology.orchestra.run.smithi078:> # Given a non-glob path, it better be there 2024-03-24T00:18:54.832 DEBUG:teuthology.orchestra.run.smithi078:> if "*" not in asok_path: 2024-03-24T00:18:54.832 DEBUG:teuthology.orchestra.run.smithi078:> assert(len(files) == 1) 2024-03-24T00:18:54.832 DEBUG:teuthology.orchestra.run.smithi078:> return files[0] 2024-03-24T00:18:54.832 DEBUG:teuthology.orchestra.run.smithi078:> 2024-03-24T00:18:54.832 DEBUG:teuthology.orchestra.run.smithi078:> for f in files: 2024-03-24T00:18:54.833 DEBUG:teuthology.orchestra.run.smithi078:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T00:18:54.833 DEBUG:teuthology.orchestra.run.smithi078:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T00:18:54.833 DEBUG:teuthology.orchestra.run.smithi078:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T00:18:54.833 DEBUG:teuthology.orchestra.run.smithi078:> contents = proc_f.read() 2024-03-24T00:18:54.833 DEBUG:teuthology.orchestra.run.smithi078:> if mountpoint in contents: 2024-03-24T00:18:54.833 DEBUG:teuthology.orchestra.run.smithi078:> return f 2024-03-24T00:18:54.833 DEBUG:teuthology.orchestra.run.smithi078:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T00:18:54.833 DEBUG:teuthology.orchestra.run.smithi078:> 2024-03-24T00:18:54.833 DEBUG:teuthology.orchestra.run.smithi078:> print(_find_admin_socket("client.admin")) 2024-03-24T00:18:54.833 DEBUG:teuthology.orchestra.run.smithi078:> ' 2024-03-24T00:18:54.952 INFO:teuthology.orchestra.run.smithi078.stdout:/var/run/ceph/ceph-client.admin.73883.asok 2024-03-24T00:18:54.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:18:54.962 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.73883.asok 2024-03-24T00:18:54.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:18:54.963 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.73883.asok status 2024-03-24T00:18:55.097 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-24T00:18:55.097 INFO:teuthology.orchestra.run.smithi078.stdout: "metadata": { 2024-03-24T00:18:55.097 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-24T00:18:55.098 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T00:18:55.098 INFO:teuthology.orchestra.run.smithi078.stdout: "entity_id": "admin", 2024-03-24T00:18:55.098 INFO:teuthology.orchestra.run.smithi078.stdout: "hostname": "smithi078", 2024-03-24T00:18:55.098 INFO:teuthology.orchestra.run.smithi078.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-24T00:18:55.098 INFO:teuthology.orchestra.run.smithi078.stdout: "pid": "73883", 2024-03-24T00:18:55.098 INFO:teuthology.orchestra.run.smithi078.stdout: "root": "/" 2024-03-24T00:18:55.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T00:18:55.098 INFO:teuthology.orchestra.run.smithi078.stdout: "dentry_count": 0, 2024-03-24T00:18:55.098 INFO:teuthology.orchestra.run.smithi078.stdout: "dentry_pinned_count": 0, 2024-03-24T00:18:55.098 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 14787, 2024-03-24T00:18:55.098 INFO:teuthology.orchestra.run.smithi078.stdout: "inst": { 2024-03-24T00:18:55.098 INFO:teuthology.orchestra.run.smithi078.stdout: "name": { 2024-03-24T00:18:55.098 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "client", 2024-03-24T00:18:55.098 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 14787 2024-03-24T00:18:55.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T00:18:55.099 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": { 2024-03-24T00:18:55.099 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-03-24T00:18:55.099 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "192.168.0.1:0", 2024-03-24T00:18:55.099 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1642865574 2024-03-24T00:18:55.099 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T00:18:55.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T00:18:55.099 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": { 2024-03-24T00:18:55.099 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-03-24T00:18:55.099 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "192.168.0.1:0", 2024-03-24T00:18:55.099 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1642865574 2024-03-24T00:18:55.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T00:18:55.099 INFO:teuthology.orchestra.run.smithi078.stdout: "inst_str": "client.14787 192.168.0.1:0/1642865574", 2024-03-24T00:18:55.099 INFO:teuthology.orchestra.run.smithi078.stdout: "addr_str": "192.168.0.1:0/1642865574", 2024-03-24T00:18:55.099 INFO:teuthology.orchestra.run.smithi078.stdout: "inode_count": 1, 2024-03-24T00:18:55.099 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_epoch": 17, 2024-03-24T00:18:55.100 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_epoch": 89, 2024-03-24T00:18:55.100 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_epoch_barrier": 0, 2024-03-24T00:18:55.100 INFO:teuthology.orchestra.run.smithi078.stdout: "blocklisted": false, 2024-03-24T00:18:55.100 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs" 2024-03-24T00:18:55.100 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-24T00:18:55.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:18:55.106 DEBUG:teuthology.orchestra.run.smithi078:> stat --file-system '--printf=%T 2024-03-24T00:18:55.106 DEBUG:teuthology.orchestra.run.smithi078:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-24T00:18:55.121 INFO:teuthology.orchestra.run.smithi078.stdout:fuseblk 2024-03-24T00:18:55.121 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-24T00:18:55.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:18:55.121 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-24T00:18:55.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:18:55.190 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-24T00:18:55.264 INFO:teuthology.orchestra.run.smithi078.stdout:# file: . 2024-03-24T00:18:55.264 INFO:teuthology.orchestra.run.smithi078.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-24T00:18:55.265 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:18:55.265 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-24T00:18:55.265 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:18:55.322 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:18:55.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.465+0000 7f203328a640 1 -- 172.21.15.78:0/464716336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f202c1005c0 msgr2=0x7f202c1009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:55.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.465+0000 7f203328a640 1 --2- 172.21.15.78:0/464716336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f202c1005c0 0x7f202c1009e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2020009900 tx=0x7f202002bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:55.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.465+0000 7f203328a640 1 -- 172.21.15.78:0/464716336 shutdown_connections 2024-03-24T00:18:55.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.465+0000 7f203328a640 1 --2- 172.21.15.78:0/464716336 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f202c1017b0 0x7f202c101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:55.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.465+0000 7f203328a640 1 --2- 172.21.15.78:0/464716336 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f202c102170 0x7f202c108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:55.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.465+0000 7f203328a640 1 --2- 172.21.15.78:0/464716336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f202c1005c0 0x7f202c1009e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:55.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.465+0000 7f203328a640 1 -- 172.21.15.78:0/464716336 >> 172.21.15.78:0/464716336 conn(0x7f202c0fbb30 msgr2=0x7f202c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:55.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.465+0000 7f203328a640 1 -- 172.21.15.78:0/464716336 shutdown_connections 2024-03-24T00:18:55.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.465+0000 7f203328a640 1 -- 172.21.15.78:0/464716336 wait complete. 2024-03-24T00:18:55.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.466+0000 7f203328a640 1 Processor -- start 2024-03-24T00:18:55.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.466+0000 7f203328a640 1 -- start start 2024-03-24T00:18:55.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.466+0000 7f203328a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f202c1005c0 0x7f202c197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:55.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.466+0000 7f203328a640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f202c1017b0 0x7f202c197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:55.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.466+0000 7f203328a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f202c102170 0x7f202c19efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:55.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.466+0000 7f203328a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f202c10d360 con 0x7f202c102170 2024-03-24T00:18:55.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.466+0000 7f203328a640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f202c10d1e0 con 0x7f202c1017b0 2024-03-24T00:18:55.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.466+0000 7f203328a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f202c10d4e0 con 0x7f202c1005c0 2024-03-24T00:18:55.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.466+0000 7f2031827640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f202c102170 0x7f202c19efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:55.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.466+0000 7f2030825640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f202c1017b0 0x7f202c197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:55.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.466+0000 7f2031827640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f202c102170 0x7f202c19efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34758/0 (socket says 172.21.15.78:34758) 2024-03-24T00:18:55.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.466+0000 7f2031827640 1 -- 172.21.15.78:0/3966808636 learned_addr learned my addr 172.21.15.78:0/3966808636 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:55.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.466+0000 7f2031026640 1 --2- 172.21.15.78:0/3966808636 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f202c1005c0 0x7f202c197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:55.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.466+0000 7f2031827640 1 -- 172.21.15.78:0/3966808636 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f202c1005c0 msgr2=0x7f202c197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:55.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.466+0000 7f2031827640 1 --2- 172.21.15.78:0/3966808636 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f202c1005c0 0x7f202c197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:55.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.466+0000 7f2031827640 1 -- 172.21.15.78:0/3966808636 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f202c1017b0 msgr2=0x7f202c197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:55.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.466+0000 7f2031827640 1 --2- 172.21.15.78:0/3966808636 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f202c1017b0 0x7f202c197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:55.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.466+0000 7f2031827640 1 -- 172.21.15.78:0/3966808636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f202c19f630 con 0x7f202c102170 2024-03-24T00:18:55.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.466+0000 7f2031827640 1 --2- 172.21.15.78:0/3966808636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f202c102170 0x7f202c19efc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f202400eb10 tx=0x7f202400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:55.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.466+0000 7f201e7fc640 1 -- 172.21.15.78:0/3966808636 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2024004b20 con 0x7f202c102170 2024-03-24T00:18:55.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.466+0000 7f201e7fc640 1 -- 172.21.15.78:0/3966808636 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2024004cc0 con 0x7f202c102170 2024-03-24T00:18:55.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.467+0000 7f203328a640 1 -- 172.21.15.78:0/3966808636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f202c19f920 con 0x7f202c102170 2024-03-24T00:18:55.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.467+0000 7f201e7fc640 1 -- 172.21.15.78:0/3966808636 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2024010760 con 0x7f202c102170 2024-03-24T00:18:55.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.467+0000 7f203328a640 1 -- 172.21.15.78:0/3966808636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f202c19fe20 con 0x7f202c102170 2024-03-24T00:18:55.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.467+0000 7f203328a640 1 -- 172.21.15.78:0/3966808636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f202c107930 con 0x7f202c102170 2024-03-24T00:18:55.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.469+0000 7f201e7fc640 1 -- 172.21.15.78:0/3966808636 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2024010900 con 0x7f202c102170 2024-03-24T00:18:55.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.469+0000 7f201e7fc640 1 --2- 172.21.15.78:0/3966808636 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f20180a2800 0x7f20180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:55.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.469+0000 7f2031026640 1 --2- 172.21.15.78:0/3966808636 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f20180a2800 0x7f20180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:55.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.469+0000 7f201e7fc640 1 -- 172.21.15.78:0/3966808636 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f20240cf440 con 0x7f202c102170 2024-03-24T00:18:55.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.470+0000 7f2031026640 1 --2- 172.21.15.78:0/3966808636 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f20180a2800 0x7f20180a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2020000c00 tx=0x7f2020005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:55.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.473+0000 7f201e7fc640 1 -- 172.21.15.78:0/3966808636 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2024053740 con 0x7f202c102170 2024-03-24T00:18:55.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:55 smithi078 ceph-mon[32429]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 5 op/s 2024-03-24T00:18:55.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:55 smithi107 ceph-mon[36485]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 5 op/s 2024-03-24T00:18:55.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.650+0000 7f203328a640 1 -- 172.21.15.78:0/3966808636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f202c02d1f0 con 0x7f202c102170 2024-03-24T00:18:55.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.651+0000 7f201e7fc640 1 -- 172.21.15.78:0/3966808636 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24129 (secure 0 0 0) 0x7f202409c2a0 con 0x7f202c102170 2024-03-24T00:18:55.651 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:18:55.652 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":89,"fsid":"c7e2806a-e972-11ee-95cd-87774f69a715","created":"2024-03-24T00:10:44.134563+0000","modified":"2024-03-24T00:18:48.224440+0000","last_up_change":"2024-03-24T00:17:50.237686+0000","last_in_change":"2024-03-24T00:17:29.788699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:14:04.534391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:18:42.178418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:18:42.800424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:18:46.112800+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b950388a-c437-426b-b703-3f8ae821dd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6803","nonce":1816879397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6805","nonce":1816879397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6809","nonce":1816879397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6807","nonce":1816879397}]},"public_addr":"172.21.15.78:6803/1816879397","cluster_addr":"172.21.15.78:6805/1816879397","heartbeat_back_addr":"172.21.15.78:6809/1816879397","heartbeat_front_addr":"172.21.15.78:6807/1816879397","state":["exists","up"]},{"osd":1,"uuid":"b8d64714-dd65-400f-b3f4-673ab14e1365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6801","nonce":1647915612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6803","nonce":1647915612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6807","nonce":1647915612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6805","nonce":1647915612}]},"public_addr":"172.21.15.106:6801/1647915612","cluster_addr":"172.21.15.106:6803/1647915612","heartbeat_back_addr":"172.21.15.106:6807/1647915612","heartbeat_front_addr":"172.21.15.106:6805/1647915612","state":["exists","up"]},{"osd":2,"uuid":"fcfc4dd1-aaf8-4131-a023-5e832212158d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6801","nonce":2010770435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6803","nonce":2010770435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6807","nonce":2010770435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6805","nonce":2010770435}]},"public_addr":"172.21.15.107:6801/2010770435","cluster_addr":"172.21.15.107:6803/2010770435","heartbeat_back_addr":"172.21.15.107:6807/2010770435","heartbeat_front_addr":"172.21.15.107:6805/2010770435","state":["exists","up"]},{"osd":3,"uuid":"47c718ab-a0f4-4e52-83c5-b89386b262d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6811","nonce":1912251346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6813","nonce":1912251346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6817","nonce":1912251346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6815","nonce":1912251346}]},"public_addr":"172.21.15.78:6811/1912251346","cluster_addr":"172.21.15.78:6813/1912251346","heartbeat_back_addr":"172.21.15.78:6817/1912251346","heartbeat_front_addr":"172.21.15.78:6815/1912251346","state":["exists","up"]},{"osd":4,"uuid":"7dad64dd-677e-4c1c-9638-e4f4f2195983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6809","nonce":374641205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6811","nonce":374641205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6815","nonce":374641205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6813","nonce":374641205}]},"public_addr":"172.21.15.106:6809/374641205","cluster_addr":"172.21.15.106:6811/374641205","heartbeat_back_addr":"172.21.15.106:6815/374641205","heartbeat_front_addr":"172.21.15.106:6813/374641205","state":["exists","up"]},{"osd":5,"uuid":"bbd524e5-328d-4309-8877-1149cbe110dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6809","nonce":3272806073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6811","nonce":3272806073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6815","nonce":3272806073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6813","nonce":3272806073}]},"public_addr":"172.21.15.107:6809/3272806073","cluster_addr":"172.21.15.107:6811/3272806073","heartbeat_back_addr":"172.21.15.107:6815/3272806073","heartbeat_front_addr":"172.21.15.107:6813/3272806073","state":["exists","up"]},{"osd":6,"uuid":"5696389a-918f-4ea8-b20d-4e2a0d4cb3bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6819","nonce":3091148508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6821","nonce":3091148508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6825","nonce":3091148508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6823","nonce":3091148508}]},"public_addr":"172.21.15.78:6819/3091148508","cluster_addr":"172.21.15.78:6821/3091148508","heartbeat_back_addr":"172.21.15.78:6825/3091148508","heartbeat_front_addr":"172.21.15.78:6823/3091148508","state":["exists","up"]},{"osd":7,"uuid":"f6b764e8-ed4e-4c9a-a85b-94af6dc6b924","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6817","nonce":4262103080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6819","nonce":4262103080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6823","nonce":4262103080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6821","nonce":4262103080}]},"public_addr":"172.21.15.106:6817/4262103080","cluster_addr":"172.21.15.106:6819/4262103080","heartbeat_back_addr":"172.21.15.106:6823/4262103080","heartbeat_front_addr":"172.21.15.106:6821/4262103080","state":["exists","up"]},{"osd":8,"uuid":"ba16039f-8093-4fa0-87ad-e28b7d142b23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6817","nonce":4032799356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6819","nonce":4032799356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6823","nonce":4032799356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6821","nonce":4032799356}]},"public_addr":"172.21.15.107:6817/4032799356","cluster_addr":"172.21.15.107:6819/4032799356","heartbeat_back_addr":"172.21.15.107:6823/4032799356","heartbeat_front_addr":"172.21.15.107:6821/4032799356","state":["exists","up"]},{"osd":9,"uuid":"651357f2-0669-42e6-aa91-1d5c02d00c5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6827","nonce":214023431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6829","nonce":214023431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6833","nonce":214023431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6831","nonce":214023431}]},"public_addr":"172.21.15.78:6827/214023431","cluster_addr":"172.21.15.78:6829/214023431","heartbeat_back_addr":"172.21.15.78:6833/214023431","heartbeat_front_addr":"172.21.15.78:6831/214023431","state":["exists","up"]},{"osd":10,"uuid":"2375119d-346e-426c-8a65-b64af71ad63a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6825","nonce":400029804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6827","nonce":400029804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6831","nonce":400029804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6829","nonce":400029804}]},"public_addr":"172.21.15.106:6825/400029804","cluster_addr":"172.21.15.106:6827/400029804","heartbeat_back_addr":"172.21.15.106:6831/400029804","heartbeat_front_addr":"172.21.15.106:6829/400029804","state":["exists","up"]},{"osd":11,"uuid":"b6b609d9-dd78-4c01-ae66-ba2a662ab22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6825","nonce":4191916230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6827","nonce":4191916230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6831","nonce":4191916230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6829","nonce":4191916230}]},"public_addr":"172.21.15.107:6825/4191916230","cluster_addr":"172.21.15.107:6827/4191916230","heartbeat_back_addr":"172.21.15.107:6831/4191916230","heartbeat_front_addr":"172.21.15.107:6829/4191916230","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:13.728602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:37.320809+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:01.269452+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:25.758281+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:50.298863+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:14.345219+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:38.980489+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:04.749214+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:29.992673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:56.269202+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:22.270316+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:48.074907+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/1982202541":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/1933054339":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6801/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6800/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/465691596":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6801/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/3962523609":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6800/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/1502752625":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/2236879401":"2024-03-25T00:11:40.199569+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:18:55.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:55 smithi106 ceph-mon[36505]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 5 op/s 2024-03-24T00:18:55.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.653+0000 7f203328a640 1 -- 172.21.15.78:0/3966808636 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f20180a2800 msgr2=0x7f20180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:55.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.653+0000 7f203328a640 1 --2- 172.21.15.78:0/3966808636 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f20180a2800 0x7f20180a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2020000c00 tx=0x7f2020005fd0 comp rx=0 tx=0).stop 2024-03-24T00:18:55.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.653+0000 7f203328a640 1 -- 172.21.15.78:0/3966808636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f202c102170 msgr2=0x7f202c19efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:55.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.653+0000 7f203328a640 1 --2- 172.21.15.78:0/3966808636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f202c102170 0x7f202c19efc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f202400eb10 tx=0x7f202400c2d0 comp rx=0 tx=0).stop 2024-03-24T00:18:55.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.653+0000 7f203328a640 1 -- 172.21.15.78:0/3966808636 shutdown_connections 2024-03-24T00:18:55.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.653+0000 7f203328a640 1 --2- 172.21.15.78:0/3966808636 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f20180a2800 0x7f20180a4cc0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:55.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.653+0000 7f203328a640 1 --2- 172.21.15.78:0/3966808636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f202c102170 0x7f202c19efc0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:55.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.653+0000 7f203328a640 1 --2- 172.21.15.78:0/3966808636 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f202c1017b0 0x7f202c197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:55.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.653+0000 7f203328a640 1 --2- 172.21.15.78:0/3966808636 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f202c1005c0 0x7f202c197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:55.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.653+0000 7f203328a640 1 -- 172.21.15.78:0/3966808636 >> 172.21.15.78:0/3966808636 conn(0x7f202c0fbb30 msgr2=0x7f202c104b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:55.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.654+0000 7f203328a640 1 -- 172.21.15.78:0/3966808636 shutdown_connections 2024-03-24T00:18:55.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.654+0000 7f203328a640 1 -- 172.21.15.78:0/3966808636 wait complete. 2024-03-24T00:18:55.666 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:14:04.534391+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:18:42.178418+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:18:42.800424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T00:18:46.112800+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T00:18:55.666 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:18:55.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.771+0000 7f6108227640 1 -- 172.21.15.78:0/3928770238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61001005a0 msgr2=0x7f61001009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:55.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.771+0000 7f6108227640 1 --2- 172.21.15.78:0/3928770238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61001005a0 0x7f61001009c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f60f0009900 tx=0x7f60f002bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:55.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.771+0000 7f6108227640 1 -- 172.21.15.78:0/3928770238 shutdown_connections 2024-03-24T00:18:55.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.771+0000 7f6108227640 1 --2- 172.21.15.78:0/3928770238 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6100101790 0x7f6100101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:55.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.771+0000 7f6108227640 1 --2- 172.21.15.78:0/3928770238 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6100102150 0x7f61001089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:55.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.771+0000 7f6108227640 1 --2- 172.21.15.78:0/3928770238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61001005a0 0x7f61001009c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:55.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.771+0000 7f6108227640 1 -- 172.21.15.78:0/3928770238 >> 172.21.15.78:0/3928770238 conn(0x7f61000fbb30 msgr2=0x7f61000fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:55.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.771+0000 7f6108227640 1 -- 172.21.15.78:0/3928770238 shutdown_connections 2024-03-24T00:18:55.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.771+0000 7f6108227640 1 -- 172.21.15.78:0/3928770238 wait complete. 2024-03-24T00:18:55.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.771+0000 7f6108227640 1 Processor -- start 2024-03-24T00:18:55.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.771+0000 7f6108227640 1 -- start start 2024-03-24T00:18:55.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.771+0000 7f6108227640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61001005a0 0x7f6100196940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:55.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.771+0000 7f6108227640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6100101790 0x7f6100196e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:55.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.771+0000 7f6108227640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6100102150 0x7f610019def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:55.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.771+0000 7f6108227640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f610010c200 con 0x7f6100101790 2024-03-24T00:18:55.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.771+0000 7f6108227640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f610010c080 con 0x7f6100102150 2024-03-24T00:18:55.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.771+0000 7f6108227640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f610010c380 con 0x7f61001005a0 2024-03-24T00:18:55.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.772+0000 7f61057c2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6100101790 0x7f6100196e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:55.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.772+0000 7f61057c2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6100101790 0x7f6100196e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34780/0 (socket says 172.21.15.78:34780) 2024-03-24T00:18:55.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.772+0000 7f61057c2640 1 -- 172.21.15.78:0/2630689208 learned_addr learned my addr 172.21.15.78:0/2630689208 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:55.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.772+0000 7f61067c4640 1 --2- 172.21.15.78:0/2630689208 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6100102150 0x7f610019def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:55.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.772+0000 7f6105fc3640 1 --2- 172.21.15.78:0/2630689208 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61001005a0 0x7f6100196940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:55.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.772+0000 7f61057c2640 1 -- 172.21.15.78:0/2630689208 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61001005a0 msgr2=0x7f6100196940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:55.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.772+0000 7f61057c2640 1 --2- 172.21.15.78:0/2630689208 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61001005a0 0x7f6100196940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:55.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.772+0000 7f61057c2640 1 -- 172.21.15.78:0/2630689208 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6100102150 msgr2=0x7f610019def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:55.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.772+0000 7f61057c2640 1 --2- 172.21.15.78:0/2630689208 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6100102150 0x7f610019def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:55.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.772+0000 7f61057c2640 1 -- 172.21.15.78:0/2630689208 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f610019e560 con 0x7f6100101790 2024-03-24T00:18:55.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.772+0000 7f61057c2640 1 --2- 172.21.15.78:0/2630689208 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6100101790 0x7f6100196e80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f60fc00d950 tx=0x7f60fc00dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:55.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.772+0000 7f60f6ffd640 1 -- 172.21.15.78:0/2630689208 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60fc017030 con 0x7f6100101790 2024-03-24T00:18:55.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.772+0000 7f60f6ffd640 1 -- 172.21.15.78:0/2630689208 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60fc004020 con 0x7f6100101790 2024-03-24T00:18:55.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.772+0000 7f6108227640 1 -- 172.21.15.78:0/2630689208 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f610019e850 con 0x7f6100101790 2024-03-24T00:18:55.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.772+0000 7f60f6ffd640 1 -- 172.21.15.78:0/2630689208 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60fc005650 con 0x7f6100101790 2024-03-24T00:18:55.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.772+0000 7f6108227640 1 -- 172.21.15.78:0/2630689208 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f610019edf0 con 0x7f6100101790 2024-03-24T00:18:55.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.773+0000 7f6108227640 1 -- 172.21.15.78:0/2630689208 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61001079e0 con 0x7f6100101790 2024-03-24T00:18:55.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.774+0000 7f60f6ffd640 1 -- 172.21.15.78:0/2630689208 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f60fc0041d0 con 0x7f6100101790 2024-03-24T00:18:55.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.775+0000 7f60f6ffd640 1 --2- 172.21.15.78:0/2630689208 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f60ec0a2880 0x7f60ec0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:55.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.775+0000 7f60f6ffd640 1 -- 172.21.15.78:0/2630689208 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f60fc0ced20 con 0x7f6100101790 2024-03-24T00:18:55.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.775+0000 7f6105fc3640 1 --2- 172.21.15.78:0/2630689208 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f60ec0a2880 0x7f60ec0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:55.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.775+0000 7f6105fc3640 1 --2- 172.21.15.78:0/2630689208 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f60ec0a2880 0x7f60ec0a4d40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f60f002c7b0 tx=0x7f60f0005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:55.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.778+0000 7f60f6ffd640 1 -- 172.21.15.78:0/2630689208 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f60fc00e350 con 0x7f6100101790 2024-03-24T00:18:55.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.960+0000 7f6108227640 1 -- 172.21.15.78:0/2630689208 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f610002d200 con 0x7f6100101790 2024-03-24T00:18:55.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.960+0000 7f60f6ffd640 1 -- 172.21.15.78:0/2630689208 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f60fc09bc00 con 0x7f6100101790 2024-03-24T00:18:55.960 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-24T00:18:55.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.962+0000 7f6108227640 1 -- 172.21.15.78:0/2630689208 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f60ec0a2880 msgr2=0x7f60ec0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:55.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.962+0000 7f6108227640 1 --2- 172.21.15.78:0/2630689208 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f60ec0a2880 0x7f60ec0a4d40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f60f002c7b0 tx=0x7f60f0005fd0 comp rx=0 tx=0).stop 2024-03-24T00:18:55.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.962+0000 7f6108227640 1 -- 172.21.15.78:0/2630689208 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6100101790 msgr2=0x7f6100196e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:55.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.962+0000 7f6108227640 1 --2- 172.21.15.78:0/2630689208 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6100101790 0x7f6100196e80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f60fc00d950 tx=0x7f60fc00dd50 comp rx=0 tx=0).stop 2024-03-24T00:18:55.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.962+0000 7f6108227640 1 -- 172.21.15.78:0/2630689208 shutdown_connections 2024-03-24T00:18:55.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.962+0000 7f6108227640 1 --2- 172.21.15.78:0/2630689208 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f60ec0a2880 0x7f60ec0a4d40 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:55.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.962+0000 7f6108227640 1 --2- 172.21.15.78:0/2630689208 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6100102150 0x7f610019def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:55.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.962+0000 7f6108227640 1 --2- 172.21.15.78:0/2630689208 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6100101790 0x7f6100196e80 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:55.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.962+0000 7f6108227640 1 --2- 172.21.15.78:0/2630689208 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61001005a0 0x7f6100196940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:55.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.962+0000 7f6108227640 1 -- 172.21.15.78:0/2630689208 >> 172.21.15.78:0/2630689208 conn(0x7f61000fbb30 msgr2=0x7f6100104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:55.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.962+0000 7f6108227640 1 -- 172.21.15.78:0/2630689208 shutdown_connections 2024-03-24T00:18:55.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:55.962+0000 7f6108227640 1 -- 172.21.15.78:0/2630689208 wait complete. 2024-03-24T00:18:55.972 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:18:56.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.078+0000 7f81a5e59640 1 -- 172.21.15.78:0/2496258513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81a01016b0 msgr2=0x7f81a0101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:56.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.078+0000 7f81a5e59640 1 --2- 172.21.15.78:0/2496258513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81a01016b0 0x7f81a0101ad0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f8194009900 tx=0x7f819402bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:56.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.078+0000 7f81a5e59640 1 -- 172.21.15.78:0/2496258513 shutdown_connections 2024-03-24T00:18:56.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.078+0000 7f81a5e59640 1 --2- 172.21.15.78:0/2496258513 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81a01028a0 0x7f81a0102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:56.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.078+0000 7f81a5e59640 1 --2- 172.21.15.78:0/2496258513 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f81a0103260 0x7f81a0109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:56.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.078+0000 7f81a5e59640 1 --2- 172.21.15.78:0/2496258513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81a01016b0 0x7f81a0101ad0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:56.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.078+0000 7f81a5e59640 1 -- 172.21.15.78:0/2496258513 >> 172.21.15.78:0/2496258513 conn(0x7f81a00fcc40 msgr2=0x7f81a00ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:56.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.078+0000 7f81a5e59640 1 -- 172.21.15.78:0/2496258513 shutdown_connections 2024-03-24T00:18:56.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.078+0000 7f81a5e59640 1 -- 172.21.15.78:0/2496258513 wait complete. 2024-03-24T00:18:56.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.079+0000 7f81a5e59640 1 Processor -- start 2024-03-24T00:18:56.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.079+0000 7f81a5e59640 1 -- start start 2024-03-24T00:18:56.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.079+0000 7f81a5e59640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81a01016b0 0x7f81a0197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:56.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.079+0000 7f81a5e59640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81a01028a0 0x7f81a0197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:56.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.079+0000 7f81a5e59640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f81a0103260 0x7f81a019f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:56.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.079+0000 7f81a5e59640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a010d330 con 0x7f81a01016b0 2024-03-24T00:18:56.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.079+0000 7f81a5e59640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a010d1b0 con 0x7f81a0103260 2024-03-24T00:18:56.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.079+0000 7f81a5e59640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a010d4b0 con 0x7f81a01028a0 2024-03-24T00:18:56.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.079+0000 7f819f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81a01016b0 0x7f81a0197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:56.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.079+0000 7f819f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81a01016b0 0x7f81a0197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34790/0 (socket says 172.21.15.78:34790) 2024-03-24T00:18:56.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.079+0000 7f819f7fe640 1 -- 172.21.15.78:0/1044211560 learned_addr learned my addr 172.21.15.78:0/1044211560 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:56.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.079+0000 7f819ffff640 1 --2- 172.21.15.78:0/1044211560 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f81a0103260 0x7f81a019f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:56.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.079+0000 7f819effd640 1 --2- 172.21.15.78:0/1044211560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81a01028a0 0x7f81a0197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:56.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.079+0000 7f819f7fe640 1 -- 172.21.15.78:0/1044211560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81a01028a0 msgr2=0x7f81a0197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:56.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.079+0000 7f819f7fe640 1 --2- 172.21.15.78:0/1044211560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81a01028a0 0x7f81a0197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:56.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.079+0000 7f819f7fe640 1 -- 172.21.15.78:0/1044211560 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f81a0103260 msgr2=0x7f81a019f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:56.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.079+0000 7f819f7fe640 1 --2- 172.21.15.78:0/1044211560 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f81a0103260 0x7f81a019f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:56.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.079+0000 7f819f7fe640 1 -- 172.21.15.78:0/1044211560 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a019f670 con 0x7f81a01016b0 2024-03-24T00:18:56.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.079+0000 7f819effd640 1 --2- 172.21.15.78:0/1044211560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81a01028a0 0x7f81a0197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:18:56.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.079+0000 7f819f7fe640 1 --2- 172.21.15.78:0/1044211560 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81a01016b0 0x7f81a0197a50 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8194000c00 tx=0x7f8194005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:56.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.080+0000 7f819cff9640 1 -- 172.21.15.78:0/1044211560 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8194041070 con 0x7f81a01016b0 2024-03-24T00:18:56.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.080+0000 7f819cff9640 1 -- 172.21.15.78:0/1044211560 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f81940044d0 con 0x7f81a01016b0 2024-03-24T00:18:56.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.080+0000 7f81a5e59640 1 -- 172.21.15.78:0/1044211560 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81a019f900 con 0x7f81a01016b0 2024-03-24T00:18:56.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.080+0000 7f819cff9640 1 -- 172.21.15.78:0/1044211560 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8194034440 con 0x7f81a01016b0 2024-03-24T00:18:56.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.080+0000 7f81a5e59640 1 -- 172.21.15.78:0/1044211560 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81a019fe40 con 0x7f81a01016b0 2024-03-24T00:18:56.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.081+0000 7f81a5e59640 1 -- 172.21.15.78:0/1044211560 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81a0108af0 con 0x7f81a01016b0 2024-03-24T00:18:56.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.082+0000 7f819cff9640 1 -- 172.21.15.78:0/1044211560 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8194044070 con 0x7f81a01016b0 2024-03-24T00:18:56.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.082+0000 7f819cff9640 1 --2- 172.21.15.78:0/1044211560 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f81880a2880 0x7f81880a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:56.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.082+0000 7f819effd640 1 --2- 172.21.15.78:0/1044211560 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f81880a2880 0x7f81880a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:56.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.082+0000 7f819cff9640 1 -- 172.21.15.78:0/1044211560 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f81940f07c0 con 0x7f81a01016b0 2024-03-24T00:18:56.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.083+0000 7f819effd640 1 --2- 172.21.15.78:0/1044211560 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f81880a2880 0x7f81880a4d40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f81900058c0 tx=0x7f8190005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:56.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.086+0000 7f819cff9640 1 -- 172.21.15.78:0/1044211560 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f81940bd620 con 0x7f81a01016b0 2024-03-24T00:18:56.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.260+0000 7f81a5e59640 1 -- 172.21.15.78:0/1044211560 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f81a00622d0 con 0x7f81a01016b0 2024-03-24T00:18:56.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.261+0000 7f819cff9640 1 -- 172.21.15.78:0/1044211560 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f81940bcf20 con 0x7f81a01016b0 2024-03-24T00:18:56.261 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-24T00:18:56.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.263+0000 7f81a5e59640 1 -- 172.21.15.78:0/1044211560 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f81880a2880 msgr2=0x7f81880a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:56.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.263+0000 7f81a5e59640 1 --2- 172.21.15.78:0/1044211560 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f81880a2880 0x7f81880a4d40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f81900058c0 tx=0x7f8190005850 comp rx=0 tx=0).stop 2024-03-24T00:18:56.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.263+0000 7f81a5e59640 1 -- 172.21.15.78:0/1044211560 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81a01016b0 msgr2=0x7f81a0197a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:56.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.263+0000 7f81a5e59640 1 --2- 172.21.15.78:0/1044211560 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81a01016b0 0x7f81a0197a50 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8194000c00 tx=0x7f8194005d50 comp rx=0 tx=0).stop 2024-03-24T00:18:56.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.263+0000 7f81a5e59640 1 -- 172.21.15.78:0/1044211560 shutdown_connections 2024-03-24T00:18:56.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.263+0000 7f81a5e59640 1 --2- 172.21.15.78:0/1044211560 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f81880a2880 0x7f81880a4d40 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:56.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.263+0000 7f81a5e59640 1 --2- 172.21.15.78:0/1044211560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81a01028a0 0x7f81a0197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:56.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.263+0000 7f81a5e59640 1 --2- 172.21.15.78:0/1044211560 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f81a0103260 0x7f81a019f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:56.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.263+0000 7f81a5e59640 1 --2- 172.21.15.78:0/1044211560 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81a01016b0 0x7f81a0197a50 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:56.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.263+0000 7f81a5e59640 1 -- 172.21.15.78:0/1044211560 >> 172.21.15.78:0/1044211560 conn(0x7f81a00fcc40 msgr2=0x7f81a0105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:56.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.263+0000 7f81a5e59640 1 -- 172.21.15.78:0/1044211560 shutdown_connections 2024-03-24T00:18:56.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.263+0000 7f81a5e59640 1 -- 172.21.15.78:0/1044211560 wait complete. 2024-03-24T00:18:56.273 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:18:56.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.379+0000 7f4568359640 1 -- 172.21.15.78:0/3052641340 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45600ff990 msgr2=0x7f45600ffdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:56.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.379+0000 7f4568359640 1 --2- 172.21.15.78:0/3052641340 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45600ff990 0x7f45600ffdb0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f455c009930 tx=0x7f455c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:56.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.379+0000 7f4568359640 1 -- 172.21.15.78:0/3052641340 shutdown_connections 2024-03-24T00:18:56.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.379+0000 7f4568359640 1 --2- 172.21.15.78:0/3052641340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4560102960 0x7f4560102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:56.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.379+0000 7f4568359640 1 --2- 172.21.15.78:0/3052641340 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4560103320 0x7f4560109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:56.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.379+0000 7f4568359640 1 --2- 172.21.15.78:0/3052641340 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45600ff990 0x7f45600ffdb0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:56.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.379+0000 7f4568359640 1 -- 172.21.15.78:0/3052641340 >> 172.21.15.78:0/3052641340 conn(0x7f45600fb2a0 msgr2=0x7f45600fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:56.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.379+0000 7f4568359640 1 -- 172.21.15.78:0/3052641340 shutdown_connections 2024-03-24T00:18:56.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.379+0000 7f4568359640 1 -- 172.21.15.78:0/3052641340 wait complete. 2024-03-24T00:18:56.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.379+0000 7f4568359640 1 Processor -- start 2024-03-24T00:18:56.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.379+0000 7f4568359640 1 -- start start 2024-03-24T00:18:56.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.379+0000 7f4568359640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45600ff990 0x7f4560197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:56.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.379+0000 7f4568359640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4560102960 0x7f4560197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:56.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.380+0000 7f4568359640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4560103320 0x7f456019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:56.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.380+0000 7f4568359640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f456010d370 con 0x7f4560102960 2024-03-24T00:18:56.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.380+0000 7f4568359640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f456010d1f0 con 0x7f4560103320 2024-03-24T00:18:56.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.380+0000 7f4568359640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f456010d4f0 con 0x7f45600ff990 2024-03-24T00:18:56.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.380+0000 7f45658f4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4560102960 0x7f4560197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:56.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.380+0000 7f45658f4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4560102960 0x7f4560197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34806/0 (socket says 172.21.15.78:34806) 2024-03-24T00:18:56.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.380+0000 7f45658f4640 1 -- 172.21.15.78:0/4272981375 learned_addr learned my addr 172.21.15.78:0/4272981375 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:56.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.380+0000 7f45668f6640 1 --2- 172.21.15.78:0/4272981375 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4560103320 0x7f456019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:56.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.380+0000 7f45660f5640 1 --2- 172.21.15.78:0/4272981375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45600ff990 0x7f4560197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:56.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.380+0000 7f45658f4640 1 -- 172.21.15.78:0/4272981375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45600ff990 msgr2=0x7f4560197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:56.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.380+0000 7f45658f4640 1 --2- 172.21.15.78:0/4272981375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45600ff990 0x7f4560197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:56.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.380+0000 7f45658f4640 1 -- 172.21.15.78:0/4272981375 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4560103320 msgr2=0x7f456019eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:56.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.380+0000 7f45658f4640 1 --2- 172.21.15.78:0/4272981375 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4560103320 0x7f456019eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:56.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.380+0000 7f45658f4640 1 -- 172.21.15.78:0/4272981375 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f456019f660 con 0x7f4560102960 2024-03-24T00:18:56.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.380+0000 7f45668f6640 1 --2- 172.21.15.78:0/4272981375 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4560103320 0x7f456019eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:56.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.380+0000 7f45660f5640 1 --2- 172.21.15.78:0/4272981375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45600ff990 0x7f4560197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:18:56.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.380+0000 7f45658f4640 1 --2- 172.21.15.78:0/4272981375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4560102960 0x7f4560197f80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f455000d770 tx=0x7f455000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:56.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.380+0000 7f45577fe640 1 -- 172.21.15.78:0/4272981375 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4550017070 con 0x7f4560102960 2024-03-24T00:18:56.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.380+0000 7f45577fe640 1 -- 172.21.15.78:0/4272981375 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4550005500 con 0x7f4560102960 2024-03-24T00:18:56.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.380+0000 7f45577fe640 1 -- 172.21.15.78:0/4272981375 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4550004e00 con 0x7f4560102960 2024-03-24T00:18:56.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.380+0000 7f4568359640 1 -- 172.21.15.78:0/4272981375 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f456019f950 con 0x7f4560102960 2024-03-24T00:18:56.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.380+0000 7f4568359640 1 -- 172.21.15.78:0/4272981375 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f456019fe50 con 0x7f4560102960 2024-03-24T00:18:56.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.381+0000 7f4568359640 1 -- 172.21.15.78:0/4272981375 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4560108ae0 con 0x7f4560102960 2024-03-24T00:18:56.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.382+0000 7f45577fe640 1 -- 172.21.15.78:0/4272981375 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4550004fa0 con 0x7f4560102960 2024-03-24T00:18:56.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.383+0000 7f45577fe640 1 --2- 172.21.15.78:0/4272981375 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f454c0a2800 0x7f454c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:56.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.383+0000 7f45660f5640 1 --2- 172.21.15.78:0/4272981375 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f454c0a2800 0x7f454c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:56.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.383+0000 7f45577fe640 1 -- 172.21.15.78:0/4272981375 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f45500ce6e0 con 0x7f4560102960 2024-03-24T00:18:56.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.384+0000 7f45660f5640 1 --2- 172.21.15.78:0/4272981375 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f454c0a2800 0x7f454c0a4cc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f455c0096c0 tx=0x7f455c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:56.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.388+0000 7f45577fe640 1 -- 172.21.15.78:0/4272981375 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f455009b540 con 0x7f4560102960 2024-03-24T00:18:56.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.562+0000 7f4568359640 1 -- 172.21.15.78:0/4272981375 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f45600622a0 con 0x7f4560102960 2024-03-24T00:18:56.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.563+0000 7f45577fe640 1 -- 172.21.15.78:0/4272981375 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f455009ae40 con 0x7f4560102960 2024-03-24T00:18:56.563 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-03-24T00:18:56.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.565+0000 7f4568359640 1 -- 172.21.15.78:0/4272981375 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f454c0a2800 msgr2=0x7f454c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:56.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.565+0000 7f4568359640 1 --2- 172.21.15.78:0/4272981375 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f454c0a2800 0x7f454c0a4cc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f455c0096c0 tx=0x7f455c005fd0 comp rx=0 tx=0).stop 2024-03-24T00:18:56.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.565+0000 7f4568359640 1 -- 172.21.15.78:0/4272981375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4560102960 msgr2=0x7f4560197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:56.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.565+0000 7f4568359640 1 --2- 172.21.15.78:0/4272981375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4560102960 0x7f4560197f80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f455000d770 tx=0x7f455000dc30 comp rx=0 tx=0).stop 2024-03-24T00:18:56.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.565+0000 7f4568359640 1 -- 172.21.15.78:0/4272981375 shutdown_connections 2024-03-24T00:18:56.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.565+0000 7f4568359640 1 --2- 172.21.15.78:0/4272981375 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f454c0a2800 0x7f454c0a4cc0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:56.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.565+0000 7f4568359640 1 --2- 172.21.15.78:0/4272981375 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4560103320 0x7f456019eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:56.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.565+0000 7f4568359640 1 --2- 172.21.15.78:0/4272981375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4560102960 0x7f4560197f80 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:56.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.565+0000 7f4568359640 1 --2- 172.21.15.78:0/4272981375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45600ff990 0x7f4560197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:56.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.565+0000 7f4568359640 1 -- 172.21.15.78:0/4272981375 >> 172.21.15.78:0/4272981375 conn(0x7f45600fb2a0 msgr2=0x7f4560105cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:56.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.565+0000 7f4568359640 1 -- 172.21.15.78:0/4272981375 shutdown_connections 2024-03-24T00:18:56.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.565+0000 7f4568359640 1 -- 172.21.15.78:0/4272981375 wait complete. 2024-03-24T00:18:56.574 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:56 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3966808636' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:18:56.574 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:56 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2630689208' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:18:56.575 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:56 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1044211560' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:18:56.575 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T00:18:56.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:56 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3966808636' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:18:56.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:56 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2630689208' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:18:56.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:56 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1044211560' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:18:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:56 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3966808636' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:18:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:56 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2630689208' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:18:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:56 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1044211560' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:18:56.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.679+0000 7f33c5322640 1 -- 172.21.15.78:0/1708360933 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33c0102150 msgr2=0x7f33c01089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:56.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.679+0000 7f33c5322640 1 --2- 172.21.15.78:0/1708360933 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33c0102150 0x7f33c01089e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f33b0009900 tx=0x7f33b002bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:56.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.679+0000 7f33c5322640 1 -- 172.21.15.78:0/1708360933 shutdown_connections 2024-03-24T00:18:56.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.679+0000 7f33c5322640 1 --2- 172.21.15.78:0/1708360933 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33c0102150 0x7f33c01089e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:56.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.679+0000 7f33c5322640 1 --2- 172.21.15.78:0/1708360933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33c0101790 0x7f33c0101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:56.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.679+0000 7f33c5322640 1 --2- 172.21.15.78:0/1708360933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33c01005a0 0x7f33c01009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:56.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.679+0000 7f33c5322640 1 -- 172.21.15.78:0/1708360933 >> 172.21.15.78:0/1708360933 conn(0x7f33c00fbb30 msgr2=0x7f33c00fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:56.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.679+0000 7f33c5322640 1 -- 172.21.15.78:0/1708360933 shutdown_connections 2024-03-24T00:18:56.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.679+0000 7f33c5322640 1 -- 172.21.15.78:0/1708360933 wait complete. 2024-03-24T00:18:56.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.680+0000 7f33c5322640 1 Processor -- start 2024-03-24T00:18:56.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.680+0000 7f33c5322640 1 -- start start 2024-03-24T00:18:56.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.680+0000 7f33c5322640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33c01005a0 0x7f33c0069390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:56.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.680+0000 7f33c5322640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33c0101790 0x7f33c019c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:56.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.680+0000 7f33c5322640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33c0102150 0x7f33c019ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:56.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.680+0000 7f33c5322640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c010c230 con 0x7f33c01005a0 2024-03-24T00:18:56.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.680+0000 7f33c5322640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c010c0b0 con 0x7f33c0101790 2024-03-24T00:18:56.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.680+0000 7f33c5322640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c010c3b0 con 0x7f33c0102150 2024-03-24T00:18:56.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.680+0000 7f33beffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33c01005a0 0x7f33c0069390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:56.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.680+0000 7f33beffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33c01005a0 0x7f33c0069390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34822/0 (socket says 172.21.15.78:34822) 2024-03-24T00:18:56.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.680+0000 7f33beffd640 1 -- 172.21.15.78:0/1870013182 learned_addr learned my addr 172.21.15.78:0/1870013182 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:56.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.680+0000 7f33be7fc640 1 --2- 172.21.15.78:0/1870013182 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33c0101790 0x7f33c019c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:56.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.681+0000 7f33bf7fe640 1 --2- 172.21.15.78:0/1870013182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33c0102150 0x7f33c019ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:56.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.681+0000 7f33beffd640 1 -- 172.21.15.78:0/1870013182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33c0102150 msgr2=0x7f33c019ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:56.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.681+0000 7f33beffd640 1 --2- 172.21.15.78:0/1870013182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33c0102150 0x7f33c019ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:56.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.681+0000 7f33beffd640 1 -- 172.21.15.78:0/1870013182 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33c0101790 msgr2=0x7f33c019c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:56.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.681+0000 7f33beffd640 1 --2- 172.21.15.78:0/1870013182 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33c0101790 0x7f33c019c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:56.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.681+0000 7f33beffd640 1 -- 172.21.15.78:0/1870013182 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c019f590 con 0x7f33c01005a0 2024-03-24T00:18:56.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.681+0000 7f33bf7fe640 1 --2- 172.21.15.78:0/1870013182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33c0102150 0x7f33c019ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:18:56.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.681+0000 7f33beffd640 1 --2- 172.21.15.78:0/1870013182 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33c01005a0 0x7f33c0069390 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f33b800d950 tx=0x7f33b800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:56.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.681+0000 7f33a7fff640 1 -- 172.21.15.78:0/1870013182 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b8017070 con 0x7f33c01005a0 2024-03-24T00:18:56.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.681+0000 7f33a7fff640 1 -- 172.21.15.78:0/1870013182 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33b8009c30 con 0x7f33c01005a0 2024-03-24T00:18:56.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.681+0000 7f33c5322640 1 -- 172.21.15.78:0/1870013182 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33c019f880 con 0x7f33c01005a0 2024-03-24T00:18:56.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.681+0000 7f33a7fff640 1 -- 172.21.15.78:0/1870013182 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b800f600 con 0x7f33c01005a0 2024-03-24T00:18:56.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.681+0000 7f33c5322640 1 -- 172.21.15.78:0/1870013182 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33c019fd80 con 0x7f33c01005a0 2024-03-24T00:18:56.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.682+0000 7f33c5322640 1 -- 172.21.15.78:0/1870013182 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33c01078c0 con 0x7f33c01005a0 2024-03-24T00:18:56.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.683+0000 7f33a7fff640 1 -- 172.21.15.78:0/1870013182 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f33b801a070 con 0x7f33c01005a0 2024-03-24T00:18:56.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.683+0000 7f33a7fff640 1 --2- 172.21.15.78:0/1870013182 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f33a80a2800 0x7f33a80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:56.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.684+0000 7f33be7fc640 1 --2- 172.21.15.78:0/1870013182 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f33a80a2800 0x7f33a80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:56.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.684+0000 7f33a7fff640 1 -- 172.21.15.78:0/1870013182 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f33b80ce700 con 0x7f33c01005a0 2024-03-24T00:18:56.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.684+0000 7f33be7fc640 1 --2- 172.21.15.78:0/1870013182 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f33a80a2800 0x7f33a80a4cc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f33b40099a0 tx=0x7f33b4009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:56.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.688+0000 7f33a7fff640 1 -- 172.21.15.78:0/1870013182 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f33b8009de0 con 0x7f33c01005a0 2024-03-24T00:18:56.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.862+0000 7f33c5322640 1 -- 172.21.15.78:0/1870013182 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f33c00622a0 con 0x7f33c01005a0 2024-03-24T00:18:56.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.863+0000 7f33a7fff640 1 -- 172.21.15.78:0/1870013182 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7f33b809b560 con 0x7f33c01005a0 2024-03-24T00:18:56.863 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-24T00:18:56.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.865+0000 7f33c5322640 1 -- 172.21.15.78:0/1870013182 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f33a80a2800 msgr2=0x7f33a80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:56.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.865+0000 7f33c5322640 1 --2- 172.21.15.78:0/1870013182 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f33a80a2800 0x7f33a80a4cc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f33b40099a0 tx=0x7f33b4009500 comp rx=0 tx=0).stop 2024-03-24T00:18:56.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.865+0000 7f33c5322640 1 -- 172.21.15.78:0/1870013182 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33c01005a0 msgr2=0x7f33c0069390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:56.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.865+0000 7f33c5322640 1 --2- 172.21.15.78:0/1870013182 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33c01005a0 0x7f33c0069390 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f33b800d950 tx=0x7f33b800de10 comp rx=0 tx=0).stop 2024-03-24T00:18:56.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.865+0000 7f33c5322640 1 -- 172.21.15.78:0/1870013182 shutdown_connections 2024-03-24T00:18:56.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.865+0000 7f33c5322640 1 --2- 172.21.15.78:0/1870013182 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f33a80a2800 0x7f33a80a4cc0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:56.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.865+0000 7f33c5322640 1 --2- 172.21.15.78:0/1870013182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33c0102150 0x7f33c019ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:56.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.865+0000 7f33c5322640 1 --2- 172.21.15.78:0/1870013182 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33c0101790 0x7f33c019c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:56.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.865+0000 7f33c5322640 1 --2- 172.21.15.78:0/1870013182 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33c01005a0 0x7f33c0069390 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:56.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.865+0000 7f33c5322640 1 -- 172.21.15.78:0/1870013182 >> 172.21.15.78:0/1870013182 conn(0x7f33c00fbb30 msgr2=0x7f33c0104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:56.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.865+0000 7f33c5322640 1 -- 172.21.15.78:0/1870013182 shutdown_connections 2024-03-24T00:18:56.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.865+0000 7f33c5322640 1 -- 172.21.15.78:0/1870013182 wait complete. 2024-03-24T00:18:56.876 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:18:56.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.977+0000 7f2a826dc640 1 -- 172.21.15.78:0/4091401221 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a7c101b50 msgr2=0x7f2a7c101f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:56.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.977+0000 7f2a826dc640 1 --2- 172.21.15.78:0/4091401221 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a7c101b50 0x7f2a7c101f70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c009860 tx=0x7f2a6c02bb70 comp rx=0 tx=0).stop 2024-03-24T00:18:56.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.977+0000 7f2a826dc640 1 -- 172.21.15.78:0/4091401221 shutdown_connections 2024-03-24T00:18:56.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.977+0000 7f2a826dc640 1 --2- 172.21.15.78:0/4091401221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a7c101240 0x7f2a7c108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:56.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.977+0000 7f2a826dc640 1 --2- 172.21.15.78:0/4091401221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a7c100880 0x7f2a7c100d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:56.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.977+0000 7f2a826dc640 1 --2- 172.21.15.78:0/4091401221 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a7c101b50 0x7f2a7c101f70 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:56.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.977+0000 7f2a826dc640 1 -- 172.21.15.78:0/4091401221 >> 172.21.15.78:0/4091401221 conn(0x7f2a7c0fb2b0 msgr2=0x7f2a7c0fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:56.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.977+0000 7f2a826dc640 1 -- 172.21.15.78:0/4091401221 shutdown_connections 2024-03-24T00:18:56.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.977+0000 7f2a826dc640 1 -- 172.21.15.78:0/4091401221 wait complete. 2024-03-24T00:18:56.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.977+0000 7f2a826dc640 1 Processor -- start 2024-03-24T00:18:56.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.977+0000 7f2a826dc640 1 -- start start 2024-03-24T00:18:56.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.978+0000 7f2a826dc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a7c100880 0x7f2a7c1968f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:56.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.978+0000 7f2a826dc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a7c101240 0x7f2a7c196e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:56.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.978+0000 7f2a826dc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a7c101b50 0x7f2a7c19dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:56.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.978+0000 7f2a826dc640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7c10c260 con 0x7f2a7c101240 2024-03-24T00:18:56.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.978+0000 7f2a826dc640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7c10c0e0 con 0x7f2a7c100880 2024-03-24T00:18:56.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.978+0000 7f2a826dc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7c10c3e0 con 0x7f2a7c101b50 2024-03-24T00:18:56.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.978+0000 7f2a7b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a7c101240 0x7f2a7c196e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:56.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.978+0000 7f2a7b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a7c101240 0x7f2a7c196e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34846/0 (socket says 172.21.15.78:34846) 2024-03-24T00:18:56.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.978+0000 7f2a7b7fe640 1 -- 172.21.15.78:0/3245274730 learned_addr learned my addr 172.21.15.78:0/3245274730 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:56.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.978+0000 7f2a7bfff640 1 --2- 172.21.15.78:0/3245274730 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a7c100880 0x7f2a7c1968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:56.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.978+0000 7f2a80c79640 1 --2- 172.21.15.78:0/3245274730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a7c101b50 0x7f2a7c19dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:56.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.978+0000 7f2a7b7fe640 1 -- 172.21.15.78:0/3245274730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a7c101b50 msgr2=0x7f2a7c19dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:56.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.978+0000 7f2a7b7fe640 1 --2- 172.21.15.78:0/3245274730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a7c101b50 0x7f2a7c19dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:56.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.978+0000 7f2a7b7fe640 1 -- 172.21.15.78:0/3245274730 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a7c100880 msgr2=0x7f2a7c1968f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:56.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.978+0000 7f2a7b7fe640 1 --2- 172.21.15.78:0/3245274730 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a7c100880 0x7f2a7c1968f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:56.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.978+0000 7f2a7b7fe640 1 -- 172.21.15.78:0/3245274730 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a7c19e510 con 0x7f2a7c101240 2024-03-24T00:18:56.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.978+0000 7f2a7bfff640 1 --2- 172.21.15.78:0/3245274730 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a7c100880 0x7f2a7c1968f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:56.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.978+0000 7f2a7b7fe640 1 --2- 172.21.15.78:0/3245274730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a7c101240 0x7f2a7c196e30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2a7000e9c0 tx=0x7f2a7000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:56.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.978+0000 7f2a797fa640 1 -- 172.21.15.78:0/3245274730 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7000cde0 con 0x7f2a7c101240 2024-03-24T00:18:56.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.979+0000 7f2a826dc640 1 -- 172.21.15.78:0/3245274730 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a7c19e800 con 0x7f2a7c101240 2024-03-24T00:18:56.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.979+0000 7f2a797fa640 1 -- 172.21.15.78:0/3245274730 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2a700035f0 con 0x7f2a7c101240 2024-03-24T00:18:56.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.979+0000 7f2a797fa640 1 -- 172.21.15.78:0/3245274730 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a70010650 con 0x7f2a7c101240 2024-03-24T00:18:56.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.979+0000 7f2a826dc640 1 -- 172.21.15.78:0/3245274730 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a7c19eda0 con 0x7f2a7c101240 2024-03-24T00:18:56.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.979+0000 7f2a826dc640 1 -- 172.21.15.78:0/3245274730 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a7c107a90 con 0x7f2a7c101240 2024-03-24T00:18:56.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.980+0000 7f2a797fa640 1 -- 172.21.15.78:0/3245274730 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2a700037a0 con 0x7f2a7c101240 2024-03-24T00:18:56.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.981+0000 7f2a797fa640 1 --2- 172.21.15.78:0/3245274730 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2a640a2800 0x7f2a640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:56.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.981+0000 7f2a7bfff640 1 --2- 172.21.15.78:0/3245274730 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2a640a2800 0x7f2a640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:56.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.981+0000 7f2a797fa640 1 -- 172.21.15.78:0/3245274730 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f2a700cf050 con 0x7f2a7c101240 2024-03-24T00:18:56.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.982+0000 7f2a7bfff640 1 --2- 172.21.15.78:0/3245274730 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2a640a2800 0x7f2a640a4cc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c009830 tx=0x7f2a6c02e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:56.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:56.985+0000 7f2a797fa640 1 -- 172.21.15.78:0/3245274730 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2a7009beb0 con 0x7f2a7c101240 2024-03-24T00:18:57.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.175+0000 7f2a826dc640 1 -- 172.21.15.78:0/3245274730 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f2a7c0622a0 con 0x7f2a7c101240 2024-03-24T00:18:57.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.175+0000 7f2a797fa640 1 -- 172.21.15.78:0/3245274730 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) v1 ==== 81+0+591 (secure 0 0 0) 0x7f2a7009b7b0 con 0x7f2a7c101240 2024-03-24T00:18:57.175 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1982202541 2024-03-25T00:11:40.199569+0000 2024-03-24T00:18:57.175 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1933054339 2024-03-25T00:11:40.199569+0000 2024-03-24T00:18:57.176 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/2748920178 2024-03-25T00:11:40.199569+0000 2024-03-24T00:18:57.176 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/2748920178 2024-03-25T00:11:40.199569+0000 2024-03-24T00:18:57.176 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/465691596 2024-03-25T00:11:14.637881+0000 2024-03-24T00:18:57.176 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/2913847518 2024-03-25T00:11:14.637881+0000 2024-03-24T00:18:57.176 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/3962523609 2024-03-25T00:11:14.637881+0000 2024-03-24T00:18:57.176 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/2913847518 2024-03-25T00:11:14.637881+0000 2024-03-24T00:18:57.176 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1502752625 2024-03-25T00:11:14.637881+0000 2024-03-24T00:18:57.176 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/2236879401 2024-03-25T00:11:40.199569+0000 2024-03-24T00:18:57.176 INFO:teuthology.orchestra.run.smithi078.stderr:listed 10 entries 2024-03-24T00:18:57.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.177+0000 7f2a826dc640 1 -- 172.21.15.78:0/3245274730 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2a640a2800 msgr2=0x7f2a640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:57.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.177+0000 7f2a826dc640 1 --2- 172.21.15.78:0/3245274730 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2a640a2800 0x7f2a640a4cc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c009830 tx=0x7f2a6c02e040 comp rx=0 tx=0).stop 2024-03-24T00:18:57.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.177+0000 7f2a826dc640 1 -- 172.21.15.78:0/3245274730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a7c101240 msgr2=0x7f2a7c196e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:57.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.177+0000 7f2a826dc640 1 --2- 172.21.15.78:0/3245274730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a7c101240 0x7f2a7c196e30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2a7000e9c0 tx=0x7f2a7000ee80 comp rx=0 tx=0).stop 2024-03-24T00:18:57.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.177+0000 7f2a826dc640 1 -- 172.21.15.78:0/3245274730 shutdown_connections 2024-03-24T00:18:57.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.177+0000 7f2a826dc640 1 --2- 172.21.15.78:0/3245274730 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2a640a2800 0x7f2a640a4cc0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:57.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.177+0000 7f2a826dc640 1 --2- 172.21.15.78:0/3245274730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a7c101b50 0x7f2a7c19dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:57.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.177+0000 7f2a826dc640 1 --2- 172.21.15.78:0/3245274730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a7c101240 0x7f2a7c196e30 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:57.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.177+0000 7f2a826dc640 1 --2- 172.21.15.78:0/3245274730 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a7c100880 0x7f2a7c1968f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:57.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.177+0000 7f2a826dc640 1 -- 172.21.15.78:0/3245274730 >> 172.21.15.78:0/3245274730 conn(0x7f2a7c0fb2b0 msgr2=0x7f2a7c104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:57.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.177+0000 7f2a826dc640 1 -- 172.21.15.78:0/3245274730 shutdown_connections 2024-03-24T00:18:57.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.177+0000 7f2a826dc640 1 -- 172.21.15.78:0/3245274730 wait complete. 2024-03-24T00:18:57.188 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-24T00:18:57.188 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:18:57.205 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:18:57.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.346+0000 7fbe40eac640 1 -- 172.21.15.78:0/2444370659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe3c104ad0 msgr2=0x7fbe3c106ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:57.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.346+0000 7fbe40eac640 1 --2- 172.21.15.78:0/2444370659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe3c104ad0 0x7fbe3c106ec0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbe24009900 tx=0x7fbe2402bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:57.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.346+0000 7fbe40eac640 1 -- 172.21.15.78:0/2444370659 shutdown_connections 2024-03-24T00:18:57.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.346+0000 7fbe40eac640 1 --2- 172.21.15.78:0/2444370659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe3c107400 0x7fbe3c1097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:57.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.346+0000 7fbe40eac640 1 --2- 172.21.15.78:0/2444370659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe3c104ad0 0x7fbe3c106ec0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:57.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.346+0000 7fbe40eac640 1 --2- 172.21.15.78:0/2444370659 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbe3c1021a0 0x7fbe3c104590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:57.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.347+0000 7fbe40eac640 1 -- 172.21.15.78:0/2444370659 >> 172.21.15.78:0/2444370659 conn(0x7fbe3c0fbb30 msgr2=0x7fbe3c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:57.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.347+0000 7fbe40eac640 1 -- 172.21.15.78:0/2444370659 shutdown_connections 2024-03-24T00:18:57.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.347+0000 7fbe40eac640 1 -- 172.21.15.78:0/2444370659 wait complete. 2024-03-24T00:18:57.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.347+0000 7fbe40eac640 1 Processor -- start 2024-03-24T00:18:57.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.347+0000 7fbe40eac640 1 -- start start 2024-03-24T00:18:57.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.347+0000 7fbe40eac640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbe3c1021a0 0x7fbe3c197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:57.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.347+0000 7fbe40eac640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe3c104ad0 0x7fbe3c197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:57.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.347+0000 7fbe40eac640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe3c107400 0x7fbe3c19efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:57.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.347+0000 7fbe40eac640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3c10d340 con 0x7fbe3c104ad0 2024-03-24T00:18:57.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.347+0000 7fbe40eac640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3c10d1c0 con 0x7fbe3c1021a0 2024-03-24T00:18:57.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.347+0000 7fbe40eac640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3c10d4c0 con 0x7fbe3c107400 2024-03-24T00:18:57.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.348+0000 7fbe39d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe3c104ad0 0x7fbe3c197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:57.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.348+0000 7fbe3a59c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbe3c1021a0 0x7fbe3c197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:57.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.348+0000 7fbe39d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe3c104ad0 0x7fbe3c197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34870/0 (socket says 172.21.15.78:34870) 2024-03-24T00:18:57.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.348+0000 7fbe3a59c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbe3c1021a0 0x7fbe3c197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.78:46744/0 (socket says 172.21.15.78:46744) 2024-03-24T00:18:57.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.348+0000 7fbe39d9b640 1 -- 172.21.15.78:0/1025257820 learned_addr learned my addr 172.21.15.78:0/1025257820 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:57.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.348+0000 7fbe3ad9d640 1 --2- 172.21.15.78:0/1025257820 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe3c107400 0x7fbe3c19efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:57.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.348+0000 7fbe39d9b640 1 -- 172.21.15.78:0/1025257820 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe3c107400 msgr2=0x7fbe3c19efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:57.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.348+0000 7fbe39d9b640 1 --2- 172.21.15.78:0/1025257820 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe3c107400 0x7fbe3c19efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:57.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.348+0000 7fbe39d9b640 1 -- 172.21.15.78:0/1025257820 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbe3c1021a0 msgr2=0x7fbe3c197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:57.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.348+0000 7fbe39d9b640 1 --2- 172.21.15.78:0/1025257820 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbe3c1021a0 0x7fbe3c197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:57.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.348+0000 7fbe39d9b640 1 -- 172.21.15.78:0/1025257820 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe3c19f630 con 0x7fbe3c104ad0 2024-03-24T00:18:57.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.348+0000 7fbe3ad9d640 1 --2- 172.21.15.78:0/1025257820 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe3c107400 0x7fbe3c19efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:18:57.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.348+0000 7fbe3a59c640 1 --2- 172.21.15.78:0/1025257820 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbe3c1021a0 0x7fbe3c197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:57.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.348+0000 7fbe39d9b640 1 --2- 172.21.15.78:0/1025257820 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe3c104ad0 0x7fbe3c197f50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fbe24000c00 tx=0x7fbe24005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:57.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.348+0000 7fbe337fe640 1 -- 172.21.15.78:0/1025257820 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe24041070 con 0x7fbe3c104ad0 2024-03-24T00:18:57.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.348+0000 7fbe337fe640 1 -- 172.21.15.78:0/1025257820 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe240044d0 con 0x7fbe3c104ad0 2024-03-24T00:18:57.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.348+0000 7fbe40eac640 1 -- 172.21.15.78:0/1025257820 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe3c19f8c0 con 0x7fbe3c104ad0 2024-03-24T00:18:57.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.348+0000 7fbe337fe640 1 -- 172.21.15.78:0/1025257820 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe24034440 con 0x7fbe3c104ad0 2024-03-24T00:18:57.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.348+0000 7fbe40eac640 1 -- 172.21.15.78:0/1025257820 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe3c19fe00 con 0x7fbe3c104ad0 2024-03-24T00:18:57.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.349+0000 7fbe40eac640 1 -- 172.21.15.78:0/1025257820 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe3c0fd620 con 0x7fbe3c104ad0 2024-03-24T00:18:57.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.350+0000 7fbe337fe640 1 -- 172.21.15.78:0/1025257820 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fbe24044070 con 0x7fbe3c104ad0 2024-03-24T00:18:57.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.351+0000 7fbe337fe640 1 --2- 172.21.15.78:0/1025257820 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fbe280a2880 0x7fbe280a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:57.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.351+0000 7fbe3a59c640 1 --2- 172.21.15.78:0/1025257820 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fbe280a2880 0x7fbe280a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:57.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.351+0000 7fbe337fe640 1 -- 172.21.15.78:0/1025257820 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fbe240f07c0 con 0x7fbe3c104ad0 2024-03-24T00:18:57.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.351+0000 7fbe3a59c640 1 --2- 172.21.15.78:0/1025257820 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fbe280a2880 0x7fbe280a4d40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbe34005fd0 tx=0x7fbe34009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:57.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.355+0000 7fbe337fe640 1 -- 172.21.15.78:0/1025257820 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbe240bd620 con 0x7fbe3c104ad0 2024-03-24T00:18:57.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.531+0000 7fbe40eac640 1 -- 172.21.15.78:0/1025257820 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe3c0622a0 con 0x7fbe3c104ad0 2024-03-24T00:18:57.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.532+0000 7fbe337fe640 1 -- 172.21.15.78:0/1025257820 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24129 (secure 0 0 0) 0x7fbe240bcf20 con 0x7fbe3c104ad0 2024-03-24T00:18:57.533 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:18:57.533 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":89,"fsid":"c7e2806a-e972-11ee-95cd-87774f69a715","created":"2024-03-24T00:10:44.134563+0000","modified":"2024-03-24T00:18:48.224440+0000","last_up_change":"2024-03-24T00:17:50.237686+0000","last_in_change":"2024-03-24T00:17:29.788699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:14:04.534391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:18:42.178418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:18:42.800424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:18:46.112800+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b950388a-c437-426b-b703-3f8ae821dd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6803","nonce":1816879397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6805","nonce":1816879397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6809","nonce":1816879397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6807","nonce":1816879397}]},"public_addr":"172.21.15.78:6803/1816879397","cluster_addr":"172.21.15.78:6805/1816879397","heartbeat_back_addr":"172.21.15.78:6809/1816879397","heartbeat_front_addr":"172.21.15.78:6807/1816879397","state":["exists","up"]},{"osd":1,"uuid":"b8d64714-dd65-400f-b3f4-673ab14e1365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6801","nonce":1647915612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6803","nonce":1647915612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6807","nonce":1647915612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6805","nonce":1647915612}]},"public_addr":"172.21.15.106:6801/1647915612","cluster_addr":"172.21.15.106:6803/1647915612","heartbeat_back_addr":"172.21.15.106:6807/1647915612","heartbeat_front_addr":"172.21.15.106:6805/1647915612","state":["exists","up"]},{"osd":2,"uuid":"fcfc4dd1-aaf8-4131-a023-5e832212158d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6801","nonce":2010770435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6803","nonce":2010770435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6807","nonce":2010770435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6805","nonce":2010770435}]},"public_addr":"172.21.15.107:6801/2010770435","cluster_addr":"172.21.15.107:6803/2010770435","heartbeat_back_addr":"172.21.15.107:6807/2010770435","heartbeat_front_addr":"172.21.15.107:6805/2010770435","state":["exists","up"]},{"osd":3,"uuid":"47c718ab-a0f4-4e52-83c5-b89386b262d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6811","nonce":1912251346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6813","nonce":1912251346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6817","nonce":1912251346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6815","nonce":1912251346}]},"public_addr":"172.21.15.78:6811/1912251346","cluster_addr":"172.21.15.78:6813/1912251346","heartbeat_back_addr":"172.21.15.78:6817/1912251346","heartbeat_front_addr":"172.21.15.78:6815/1912251346","state":["exists","up"]},{"osd":4,"uuid":"7dad64dd-677e-4c1c-9638-e4f4f2195983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6809","nonce":374641205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6811","nonce":374641205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6815","nonce":374641205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6813","nonce":374641205}]},"public_addr":"172.21.15.106:6809/374641205","cluster_addr":"172.21.15.106:6811/374641205","heartbeat_back_addr":"172.21.15.106:6815/374641205","heartbeat_front_addr":"172.21.15.106:6813/374641205","state":["exists","up"]},{"osd":5,"uuid":"bbd524e5-328d-4309-8877-1149cbe110dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6809","nonce":3272806073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6811","nonce":3272806073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6815","nonce":3272806073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6813","nonce":3272806073}]},"public_addr":"172.21.15.107:6809/3272806073","cluster_addr":"172.21.15.107:6811/3272806073","heartbeat_back_addr":"172.21.15.107:6815/3272806073","heartbeat_front_addr":"172.21.15.107:6813/3272806073","state":["exists","up"]},{"osd":6,"uuid":"5696389a-918f-4ea8-b20d-4e2a0d4cb3bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6819","nonce":3091148508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6821","nonce":3091148508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6825","nonce":3091148508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6823","nonce":3091148508}]},"public_addr":"172.21.15.78:6819/3091148508","cluster_addr":"172.21.15.78:6821/3091148508","heartbeat_back_addr":"172.21.15.78:6825/3091148508","heartbeat_front_addr":"172.21.15.78:6823/3091148508","state":["exists","up"]},{"osd":7,"uuid":"f6b764e8-ed4e-4c9a-a85b-94af6dc6b924","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6817","nonce":4262103080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6819","nonce":4262103080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6823","nonce":4262103080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6821","nonce":4262103080}]},"public_addr":"172.21.15.106:6817/4262103080","cluster_addr":"172.21.15.106:6819/4262103080","heartbeat_back_addr":"172.21.15.106:6823/4262103080","heartbeat_front_addr":"172.21.15.106:6821/4262103080","state":["exists","up"]},{"osd":8,"uuid":"ba16039f-8093-4fa0-87ad-e28b7d142b23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6817","nonce":4032799356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6819","nonce":4032799356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6823","nonce":4032799356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6821","nonce":4032799356}]},"public_addr":"172.21.15.107:6817/4032799356","cluster_addr":"172.21.15.107:6819/4032799356","heartbeat_back_addr":"172.21.15.107:6823/4032799356","heartbeat_front_addr":"172.21.15.107:6821/4032799356","state":["exists","up"]},{"osd":9,"uuid":"651357f2-0669-42e6-aa91-1d5c02d00c5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6827","nonce":214023431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6829","nonce":214023431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6833","nonce":214023431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6831","nonce":214023431}]},"public_addr":"172.21.15.78:6827/214023431","cluster_addr":"172.21.15.78:6829/214023431","heartbeat_back_addr":"172.21.15.78:6833/214023431","heartbeat_front_addr":"172.21.15.78:6831/214023431","state":["exists","up"]},{"osd":10,"uuid":"2375119d-346e-426c-8a65-b64af71ad63a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6825","nonce":400029804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6827","nonce":400029804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6831","nonce":400029804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6829","nonce":400029804}]},"public_addr":"172.21.15.106:6825/400029804","cluster_addr":"172.21.15.106:6827/400029804","heartbeat_back_addr":"172.21.15.106:6831/400029804","heartbeat_front_addr":"172.21.15.106:6829/400029804","state":["exists","up"]},{"osd":11,"uuid":"b6b609d9-dd78-4c01-ae66-ba2a662ab22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6825","nonce":4191916230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6827","nonce":4191916230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6831","nonce":4191916230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6829","nonce":4191916230}]},"public_addr":"172.21.15.107:6825/4191916230","cluster_addr":"172.21.15.107:6827/4191916230","heartbeat_back_addr":"172.21.15.107:6831/4191916230","heartbeat_front_addr":"172.21.15.107:6829/4191916230","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:13.728602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:37.320809+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:01.269452+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:25.758281+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:50.298863+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:14.345219+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:38.980489+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:04.749214+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:29.992673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:56.269202+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:22.270316+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:48.074907+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/1982202541":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/1933054339":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6801/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6800/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/465691596":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6801/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/3962523609":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6800/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/1502752625":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/2236879401":"2024-03-25T00:11:40.199569+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:18:57.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.535+0000 7fbe40eac640 1 -- 172.21.15.78:0/1025257820 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fbe280a2880 msgr2=0x7fbe280a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:57.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.535+0000 7fbe40eac640 1 --2- 172.21.15.78:0/1025257820 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fbe280a2880 0x7fbe280a4d40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbe34005fd0 tx=0x7fbe34009500 comp rx=0 tx=0).stop 2024-03-24T00:18:57.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.535+0000 7fbe40eac640 1 -- 172.21.15.78:0/1025257820 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe3c104ad0 msgr2=0x7fbe3c197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:57.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.535+0000 7fbe40eac640 1 --2- 172.21.15.78:0/1025257820 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe3c104ad0 0x7fbe3c197f50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fbe24000c00 tx=0x7fbe24005d50 comp rx=0 tx=0).stop 2024-03-24T00:18:57.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.535+0000 7fbe40eac640 1 -- 172.21.15.78:0/1025257820 shutdown_connections 2024-03-24T00:18:57.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.535+0000 7fbe40eac640 1 --2- 172.21.15.78:0/1025257820 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fbe280a2880 0x7fbe280a4d40 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:57.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.535+0000 7fbe40eac640 1 --2- 172.21.15.78:0/1025257820 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe3c107400 0x7fbe3c19efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:57.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.535+0000 7fbe40eac640 1 --2- 172.21.15.78:0/1025257820 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe3c104ad0 0x7fbe3c197f50 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:57.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.535+0000 7fbe40eac640 1 --2- 172.21.15.78:0/1025257820 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbe3c1021a0 0x7fbe3c197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:57.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.535+0000 7fbe40eac640 1 -- 172.21.15.78:0/1025257820 >> 172.21.15.78:0/1025257820 conn(0x7fbe3c0fbb30 msgr2=0x7fbe3c102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:57.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.535+0000 7fbe40eac640 1 -- 172.21.15.78:0/1025257820 shutdown_connections 2024-03-24T00:18:57.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.535+0000 7fbe40eac640 1 -- 172.21.15.78:0/1025257820 wait complete. 2024-03-24T00:18:57.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:57 smithi078 ceph-mon[32429]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-24T00:18:57.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:57 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/4272981375' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:18:57.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:57 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1870013182' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T00:18:57.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:57 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3245274730' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T00:18:57.546 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:14:04.534391+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:18:42.178418+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:18:42.800424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T00:18:46.112800+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T00:18:57.546 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:18:57.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:57 smithi107 ceph-mon[36485]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-24T00:18:57.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:57 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/4272981375' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:18:57.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:57 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1870013182' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T00:18:57.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:57 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3245274730' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T00:18:57.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.650+0000 7f4f021b9640 1 -- 172.21.15.78:0/2127836565 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4efc1016d0 msgr2=0x7f4efc101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:57.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.650+0000 7f4f021b9640 1 --2- 172.21.15.78:0/2127836565 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4efc1016d0 0x7f4efc101af0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4eec009900 tx=0x7f4eec02bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:57.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.650+0000 7f4f021b9640 1 -- 172.21.15.78:0/2127836565 shutdown_connections 2024-03-24T00:18:57.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.650+0000 7f4f021b9640 1 --2- 172.21.15.78:0/2127836565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4efc1028c0 0x7f4efc102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:57.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.650+0000 7f4f021b9640 1 --2- 172.21.15.78:0/2127836565 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4efc103280 0x7f4efc109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:57.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.650+0000 7f4f021b9640 1 --2- 172.21.15.78:0/2127836565 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4efc1016d0 0x7f4efc101af0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:57.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.650+0000 7f4f021b9640 1 -- 172.21.15.78:0/2127836565 >> 172.21.15.78:0/2127836565 conn(0x7f4efc0fcc40 msgr2=0x7f4efc0ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:57.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.650+0000 7f4f021b9640 1 -- 172.21.15.78:0/2127836565 shutdown_connections 2024-03-24T00:18:57.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.651+0000 7f4f021b9640 1 -- 172.21.15.78:0/2127836565 wait complete. 2024-03-24T00:18:57.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.651+0000 7f4f021b9640 1 Processor -- start 2024-03-24T00:18:57.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.651+0000 7f4f021b9640 1 -- start start 2024-03-24T00:18:57.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.651+0000 7f4f021b9640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4efc1016d0 0x7f4efc197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:57.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.651+0000 7f4f021b9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4efc1028c0 0x7f4efc197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:57.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.651+0000 7f4f021b9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4efc103280 0x7f4efc19f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:57.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.651+0000 7f4f021b9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4efc10d340 con 0x7f4efc1028c0 2024-03-24T00:18:57.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.651+0000 7f4f021b9640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4efc10d1c0 con 0x7f4efc1016d0 2024-03-24T00:18:57.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.651+0000 7f4f021b9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4efc10d4c0 con 0x7f4efc103280 2024-03-24T00:18:57.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.651+0000 7f4efaffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4efc1028c0 0x7f4efc197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:57.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.651+0000 7f4efb7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4efc1016d0 0x7f4efc197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:57.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.651+0000 7f4efaffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4efc1028c0 0x7f4efc197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34884/0 (socket says 172.21.15.78:34884) 2024-03-24T00:18:57.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.651+0000 7f4efaffd640 1 -- 172.21.15.78:0/3998413172 learned_addr learned my addr 172.21.15.78:0/3998413172 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:57.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.652+0000 7f4efaffd640 1 -- 172.21.15.78:0/3998413172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4efc103280 msgr2=0x7f4efc19f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:57.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.652+0000 7f4efaffd640 1 --2- 172.21.15.78:0/3998413172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4efc103280 0x7f4efc19f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:57.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.652+0000 7f4efaffd640 1 -- 172.21.15.78:0/3998413172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4efc1016d0 msgr2=0x7f4efc197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:57.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.652+0000 7f4efaffd640 1 --2- 172.21.15.78:0/3998413172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4efc1016d0 0x7f4efc197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:57.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.652+0000 7f4efaffd640 1 -- 172.21.15.78:0/3998413172 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4efc19f670 con 0x7f4efc1028c0 2024-03-24T00:18:57.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.652+0000 7f4efaffd640 1 --2- 172.21.15.78:0/3998413172 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4efc1028c0 0x7f4efc197f90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4ef000d770 tx=0x7f4ef000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:57.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.652+0000 7f4ef8ff9640 1 -- 172.21.15.78:0/3998413172 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef0017070 con 0x7f4efc1028c0 2024-03-24T00:18:57.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.652+0000 7f4f021b9640 1 -- 172.21.15.78:0/3998413172 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4efc19f960 con 0x7f4efc1028c0 2024-03-24T00:18:57.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.652+0000 7f4ef8ff9640 1 -- 172.21.15.78:0/3998413172 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ef0005500 con 0x7f4efc1028c0 2024-03-24T00:18:57.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.652+0000 7f4ef8ff9640 1 -- 172.21.15.78:0/3998413172 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef0004e00 con 0x7f4efc1028c0 2024-03-24T00:18:57.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.652+0000 7f4f021b9640 1 -- 172.21.15.78:0/3998413172 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4efc19feb0 con 0x7f4efc1028c0 2024-03-24T00:18:57.655 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:57 smithi106 ceph-mon[36505]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-24T00:18:57.655 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:57 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/4272981375' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:18:57.655 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:57 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1870013182' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T00:18:57.655 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:57 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3245274730' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T00:18:57.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.653+0000 7f4f021b9640 1 -- 172.21.15.78:0/3998413172 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4efc108ae0 con 0x7f4efc1028c0 2024-03-24T00:18:57.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.654+0000 7f4ef8ff9640 1 -- 172.21.15.78:0/3998413172 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4ef0004fa0 con 0x7f4efc1028c0 2024-03-24T00:18:57.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.655+0000 7f4ef8ff9640 1 --2- 172.21.15.78:0/3998413172 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4ee40a2800 0x7f4ee40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:57.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.655+0000 7f4efb7fe640 1 --2- 172.21.15.78:0/3998413172 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4ee40a2800 0x7f4ee40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:57.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.655+0000 7f4ef8ff9640 1 -- 172.21.15.78:0/3998413172 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f4ef00ce6e0 con 0x7f4efc1028c0 2024-03-24T00:18:57.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.655+0000 7f4efb7fe640 1 --2- 172.21.15.78:0/3998413172 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4ee40a2800 0x7f4ee40a4cc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4eec000c00 tx=0x7f4eec005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:57.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.660+0000 7f4ef8ff9640 1 -- 172.21.15.78:0/3998413172 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4ef009b540 con 0x7f4efc1028c0 2024-03-24T00:18:57.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.835+0000 7f4f021b9640 1 -- 172.21.15.78:0/3998413172 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4efc0622a0 con 0x7f4efc1028c0 2024-03-24T00:18:57.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.835+0000 7f4ef8ff9640 1 -- 172.21.15.78:0/3998413172 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4ef009ae40 con 0x7f4efc1028c0 2024-03-24T00:18:57.836 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-24T00:18:57.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.837+0000 7f4f021b9640 1 -- 172.21.15.78:0/3998413172 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4ee40a2800 msgr2=0x7f4ee40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:57.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.837+0000 7f4f021b9640 1 --2- 172.21.15.78:0/3998413172 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4ee40a2800 0x7f4ee40a4cc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4eec000c00 tx=0x7f4eec005fd0 comp rx=0 tx=0).stop 2024-03-24T00:18:57.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.837+0000 7f4f021b9640 1 -- 172.21.15.78:0/3998413172 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4efc1028c0 msgr2=0x7f4efc197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:57.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.837+0000 7f4f021b9640 1 --2- 172.21.15.78:0/3998413172 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4efc1028c0 0x7f4efc197f90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4ef000d770 tx=0x7f4ef000dc30 comp rx=0 tx=0).stop 2024-03-24T00:18:57.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.837+0000 7f4f021b9640 1 -- 172.21.15.78:0/3998413172 shutdown_connections 2024-03-24T00:18:57.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.837+0000 7f4f021b9640 1 --2- 172.21.15.78:0/3998413172 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4ee40a2800 0x7f4ee40a4cc0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:57.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.837+0000 7f4f021b9640 1 --2- 172.21.15.78:0/3998413172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4efc103280 0x7f4efc19f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:57.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.837+0000 7f4f021b9640 1 --2- 172.21.15.78:0/3998413172 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4efc1028c0 0x7f4efc197f90 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:57.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.837+0000 7f4f021b9640 1 --2- 172.21.15.78:0/3998413172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4efc1016d0 0x7f4efc197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:57.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.837+0000 7f4f021b9640 1 -- 172.21.15.78:0/3998413172 >> 172.21.15.78:0/3998413172 conn(0x7f4efc0fcc40 msgr2=0x7f4efc105cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:57.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.838+0000 7f4f021b9640 1 -- 172.21.15.78:0/3998413172 shutdown_connections 2024-03-24T00:18:57.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.838+0000 7f4f021b9640 1 -- 172.21.15.78:0/3998413172 wait complete. 2024-03-24T00:18:57.848 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:18:57.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.952+0000 7f5707fd4640 1 -- 172.21.15.78:0/410714336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5700102160 msgr2=0x7f57001089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:57.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.952+0000 7f5707fd4640 1 --2- 172.21.15.78:0/410714336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5700102160 0x7f57001089f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f56fc009900 tx=0x7f56fc02bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:57.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.952+0000 7f5707fd4640 1 -- 172.21.15.78:0/410714336 shutdown_connections 2024-03-24T00:18:57.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.952+0000 7f5707fd4640 1 --2- 172.21.15.78:0/410714336 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57001017a0 0x7f5700101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:57.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.952+0000 7f5707fd4640 1 --2- 172.21.15.78:0/410714336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5700102160 0x7f57001089f0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:57.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.952+0000 7f5707fd4640 1 --2- 172.21.15.78:0/410714336 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f57001005b0 0x7f57001009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:57.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.952+0000 7f5707fd4640 1 -- 172.21.15.78:0/410714336 >> 172.21.15.78:0/410714336 conn(0x7f57000fbb40 msgr2=0x7f57000fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:57.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.952+0000 7f5707fd4640 1 -- 172.21.15.78:0/410714336 shutdown_connections 2024-03-24T00:18:57.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.952+0000 7f5707fd4640 1 -- 172.21.15.78:0/410714336 wait complete. 2024-03-24T00:18:57.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.953+0000 7f5707fd4640 1 Processor -- start 2024-03-24T00:18:57.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.953+0000 7f5707fd4640 1 -- start start 2024-03-24T00:18:57.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.953+0000 7f5707fd4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57001005b0 0x7f570006a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:57.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.953+0000 7f5707fd4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f57001017a0 0x7f570019c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:57.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.953+0000 7f5707fd4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5700102160 0x7f570019ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:57.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.953+0000 7f5707fd4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f570010c290 con 0x7f57001005b0 2024-03-24T00:18:57.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.953+0000 7f5707fd4640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f570010c110 con 0x7f57001017a0 2024-03-24T00:18:57.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.953+0000 7f5707fd4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f570010c410 con 0x7f5700102160 2024-03-24T00:18:57.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.953+0000 7f5705d70640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57001005b0 0x7f570006a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:57.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.953+0000 7f5705d70640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57001005b0 0x7f570006a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34904/0 (socket says 172.21.15.78:34904) 2024-03-24T00:18:57.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.953+0000 7f5705d70640 1 -- 172.21.15.78:0/822381528 learned_addr learned my addr 172.21.15.78:0/822381528 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:57.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.953+0000 7f570556f640 1 --2- 172.21.15.78:0/822381528 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f57001017a0 0x7f570019c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:57.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.953+0000 7f5706571640 1 --2- 172.21.15.78:0/822381528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5700102160 0x7f570019ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:57.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.953+0000 7f5705d70640 1 -- 172.21.15.78:0/822381528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5700102160 msgr2=0x7f570019ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:57.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.953+0000 7f5705d70640 1 --2- 172.21.15.78:0/822381528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5700102160 0x7f570019ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:57.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.953+0000 7f5705d70640 1 -- 172.21.15.78:0/822381528 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f57001017a0 msgr2=0x7f570019c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:57.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.953+0000 7f5705d70640 1 --2- 172.21.15.78:0/822381528 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f57001017a0 0x7f570019c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:57.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.953+0000 7f5705d70640 1 -- 172.21.15.78:0/822381528 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f570019f650 con 0x7f57001005b0 2024-03-24T00:18:57.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.954+0000 7f5705d70640 1 --2- 172.21.15.78:0/822381528 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57001005b0 0x7f570006a7e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f56f8009d20 tx=0x7f56f800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:57.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.954+0000 7f56f6ffd640 1 -- 172.21.15.78:0/822381528 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f8010630 con 0x7f57001005b0 2024-03-24T00:18:57.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.954+0000 7f56f6ffd640 1 -- 172.21.15.78:0/822381528 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f56f8010cb0 con 0x7f57001005b0 2024-03-24T00:18:57.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.954+0000 7f5707fd4640 1 -- 172.21.15.78:0/822381528 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f570019f8e0 con 0x7f57001005b0 2024-03-24T00:18:57.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.954+0000 7f56f6ffd640 1 -- 172.21.15.78:0/822381528 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f8018ce0 con 0x7f57001005b0 2024-03-24T00:18:57.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.954+0000 7f5707fd4640 1 -- 172.21.15.78:0/822381528 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f570019fe30 con 0x7f57001005b0 2024-03-24T00:18:57.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.955+0000 7f5707fd4640 1 -- 172.21.15.78:0/822381528 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5700107950 con 0x7f57001005b0 2024-03-24T00:18:57.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.956+0000 7f56f6ffd640 1 -- 172.21.15.78:0/822381528 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f56f801f020 con 0x7f57001005b0 2024-03-24T00:18:57.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.956+0000 7f56f6ffd640 1 --2- 172.21.15.78:0/822381528 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f56ec0a2800 0x7f56ec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:57.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.957+0000 7f570556f640 1 --2- 172.21.15.78:0/822381528 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f56ec0a2800 0x7f56ec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:57.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.957+0000 7f56f6ffd640 1 -- 172.21.15.78:0/822381528 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f56f80cf0e0 con 0x7f57001005b0 2024-03-24T00:18:57.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.957+0000 7f570556f640 1 --2- 172.21.15.78:0/822381528 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f56ec0a2800 0x7f56ec0a4cc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f570006a550 tx=0x7f56f0009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:57.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:57.961+0000 7f56f6ffd640 1 -- 172.21.15.78:0/822381528 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f56f809bf40 con 0x7f57001005b0 2024-03-24T00:18:58.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.138+0000 7f5707fd4640 1 -- 172.21.15.78:0/822381528 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5700062340 con 0x7f57001005b0 2024-03-24T00:18:58.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.138+0000 7f56f6ffd640 1 -- 172.21.15.78:0/822381528 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f56f809b840 con 0x7f57001005b0 2024-03-24T00:18:58.139 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-24T00:18:58.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.140+0000 7f5707fd4640 1 -- 172.21.15.78:0/822381528 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f56ec0a2800 msgr2=0x7f56ec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:58.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.140+0000 7f5707fd4640 1 --2- 172.21.15.78:0/822381528 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f56ec0a2800 0x7f56ec0a4cc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f570006a550 tx=0x7f56f0009110 comp rx=0 tx=0).stop 2024-03-24T00:18:58.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.140+0000 7f5707fd4640 1 -- 172.21.15.78:0/822381528 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57001005b0 msgr2=0x7f570006a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:58.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.140+0000 7f5707fd4640 1 --2- 172.21.15.78:0/822381528 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57001005b0 0x7f570006a7e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f56f8009d20 tx=0x7f56f800c6a0 comp rx=0 tx=0).stop 2024-03-24T00:18:58.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.140+0000 7f5707fd4640 1 -- 172.21.15.78:0/822381528 shutdown_connections 2024-03-24T00:18:58.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.140+0000 7f5707fd4640 1 --2- 172.21.15.78:0/822381528 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f56ec0a2800 0x7f56ec0a4cc0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:58.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.140+0000 7f5707fd4640 1 --2- 172.21.15.78:0/822381528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5700102160 0x7f570019ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:58.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.140+0000 7f5707fd4640 1 --2- 172.21.15.78:0/822381528 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f57001017a0 0x7f570019c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:58.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.140+0000 7f5707fd4640 1 --2- 172.21.15.78:0/822381528 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57001005b0 0x7f570006a7e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:58.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.140+0000 7f5707fd4640 1 -- 172.21.15.78:0/822381528 >> 172.21.15.78:0/822381528 conn(0x7f57000fbb40 msgr2=0x7f5700104b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:58.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.141+0000 7f5707fd4640 1 -- 172.21.15.78:0/822381528 shutdown_connections 2024-03-24T00:18:58.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.141+0000 7f5707fd4640 1 -- 172.21.15.78:0/822381528 wait complete. 2024-03-24T00:18:58.151 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:18:58.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.254+0000 7f9bf30a1640 1 -- 172.21.15.78:0/2496308039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bec104ac0 msgr2=0x7f9bec106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:58.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.254+0000 7f9bf30a1640 1 --2- 172.21.15.78:0/2496308039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bec104ac0 0x7f9bec106eb0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8009900 tx=0x7f9bd802bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:58.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.254+0000 7f9bf30a1640 1 -- 172.21.15.78:0/2496308039 shutdown_connections 2024-03-24T00:18:58.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.254+0000 7f9bf30a1640 1 --2- 172.21.15.78:0/2496308039 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bec1073f0 0x7f9bec1097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:58.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.254+0000 7f9bf30a1640 1 --2- 172.21.15.78:0/2496308039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bec104ac0 0x7f9bec106eb0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:58.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.254+0000 7f9bf30a1640 1 --2- 172.21.15.78:0/2496308039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bec102190 0x7f9bec104580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:58.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.254+0000 7f9bf30a1640 1 -- 172.21.15.78:0/2496308039 >> 172.21.15.78:0/2496308039 conn(0x7f9bec0fbb40 msgr2=0x7f9bec0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:58.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.254+0000 7f9bf30a1640 1 -- 172.21.15.78:0/2496308039 shutdown_connections 2024-03-24T00:18:58.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.254+0000 7f9bf30a1640 1 -- 172.21.15.78:0/2496308039 wait complete. 2024-03-24T00:18:58.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.255+0000 7f9bf30a1640 1 Processor -- start 2024-03-24T00:18:58.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.255+0000 7f9bf30a1640 1 -- start start 2024-03-24T00:18:58.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.255+0000 7f9bf30a1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bec102190 0x7f9bec197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:58.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.255+0000 7f9bf30a1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bec104ac0 0x7f9bec197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:58.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.255+0000 7f9bf30a1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bec1073f0 0x7f9bec19eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:58.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.255+0000 7f9bf30a1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec10d300 con 0x7f9bec104ac0 2024-03-24T00:18:58.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.255+0000 7f9bf30a1640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec10d180 con 0x7f9bec1073f0 2024-03-24T00:18:58.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.255+0000 7f9bf30a1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec10d480 con 0x7f9bec102190 2024-03-24T00:18:58.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.255+0000 7f9bebfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bec104ac0 0x7f9bec197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:58.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.255+0000 7f9bebfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bec104ac0 0x7f9bec197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34924/0 (socket says 172.21.15.78:34924) 2024-03-24T00:18:58.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.255+0000 7f9bebfff640 1 -- 172.21.15.78:0/3329873862 learned_addr learned my addr 172.21.15.78:0/3329873862 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:58.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.255+0000 7f9bf163e640 1 --2- 172.21.15.78:0/3329873862 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bec1073f0 0x7f9bec19eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:58.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.255+0000 7f9bf0e3d640 1 --2- 172.21.15.78:0/3329873862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bec102190 0x7f9bec197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:58.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.255+0000 7f9bebfff640 1 -- 172.21.15.78:0/3329873862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bec102190 msgr2=0x7f9bec197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:58.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.255+0000 7f9bebfff640 1 --2- 172.21.15.78:0/3329873862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bec102190 0x7f9bec197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:58.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.255+0000 7f9bebfff640 1 -- 172.21.15.78:0/3329873862 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bec1073f0 msgr2=0x7f9bec19eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:58.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.255+0000 7f9bebfff640 1 --2- 172.21.15.78:0/3329873862 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bec1073f0 0x7f9bec19eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:58.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.255+0000 7f9bebfff640 1 -- 172.21.15.78:0/3329873862 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bec19f660 con 0x7f9bec104ac0 2024-03-24T00:18:58.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.255+0000 7f9bf0e3d640 1 --2- 172.21.15.78:0/3329873862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bec102190 0x7f9bec197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:18:58.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.255+0000 7f9bebfff640 1 --2- 172.21.15.78:0/3329873862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bec104ac0 0x7f9bec197fd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8000c00 tx=0x7f9bd8005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:58.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.255+0000 7f9be9ffb640 1 -- 172.21.15.78:0/3329873862 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd8041070 con 0x7f9bec104ac0 2024-03-24T00:18:58.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.256+0000 7f9bf30a1640 1 -- 172.21.15.78:0/3329873862 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bec19f8f0 con 0x7f9bec104ac0 2024-03-24T00:18:58.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.256+0000 7f9be9ffb640 1 -- 172.21.15.78:0/3329873862 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9bd80044d0 con 0x7f9bec104ac0 2024-03-24T00:18:58.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.256+0000 7f9be9ffb640 1 -- 172.21.15.78:0/3329873862 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd8034440 con 0x7f9bec104ac0 2024-03-24T00:18:58.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.256+0000 7f9bf30a1640 1 -- 172.21.15.78:0/3329873862 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bec19fe30 con 0x7f9bec104ac0 2024-03-24T00:18:58.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.257+0000 7f9bf30a1640 1 -- 172.21.15.78:0/3329873862 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bec0fd630 con 0x7f9bec104ac0 2024-03-24T00:18:58.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.258+0000 7f9be9ffb640 1 -- 172.21.15.78:0/3329873862 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9bd8044070 con 0x7f9bec104ac0 2024-03-24T00:18:58.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.258+0000 7f9be9ffb640 1 --2- 172.21.15.78:0/3329873862 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f9bdc0a2800 0x7f9bdc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:58.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.258+0000 7f9bf0e3d640 1 --2- 172.21.15.78:0/3329873862 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f9bdc0a2800 0x7f9bdc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:58.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.258+0000 7f9be9ffb640 1 -- 172.21.15.78:0/3329873862 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f9bd80f0a70 con 0x7f9bec104ac0 2024-03-24T00:18:58.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.259+0000 7f9bf0e3d640 1 --2- 172.21.15.78:0/3329873862 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f9bdc0a2800 0x7f9bdc0a4cc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9be00058c0 tx=0x7f9be0005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:58.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.262+0000 7f9be9ffb640 1 -- 172.21.15.78:0/3329873862 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9bd80bd8d0 con 0x7f9bec104ac0 2024-03-24T00:18:58.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.439+0000 7f9bf30a1640 1 -- 172.21.15.78:0/3329873862 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9bec02d070 con 0x7f9bec104ac0 2024-03-24T00:18:58.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.439+0000 7f9be9ffb640 1 -- 172.21.15.78:0/3329873862 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9bd80bd1d0 con 0x7f9bec104ac0 2024-03-24T00:18:58.440 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-03-24T00:18:58.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.441+0000 7f9bf30a1640 1 -- 172.21.15.78:0/3329873862 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f9bdc0a2800 msgr2=0x7f9bdc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:58.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.441+0000 7f9bf30a1640 1 --2- 172.21.15.78:0/3329873862 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f9bdc0a2800 0x7f9bdc0a4cc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9be00058c0 tx=0x7f9be0005850 comp rx=0 tx=0).stop 2024-03-24T00:18:58.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.441+0000 7f9bf30a1640 1 -- 172.21.15.78:0/3329873862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bec104ac0 msgr2=0x7f9bec197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:58.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.441+0000 7f9bf30a1640 1 --2- 172.21.15.78:0/3329873862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bec104ac0 0x7f9bec197fd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8000c00 tx=0x7f9bd8005d50 comp rx=0 tx=0).stop 2024-03-24T00:18:58.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.441+0000 7f9bf30a1640 1 -- 172.21.15.78:0/3329873862 shutdown_connections 2024-03-24T00:18:58.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.441+0000 7f9bf30a1640 1 --2- 172.21.15.78:0/3329873862 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f9bdc0a2800 0x7f9bdc0a4cc0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:58.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.441+0000 7f9bf30a1640 1 --2- 172.21.15.78:0/3329873862 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bec1073f0 0x7f9bec19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:58.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.441+0000 7f9bf30a1640 1 --2- 172.21.15.78:0/3329873862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bec104ac0 0x7f9bec197fd0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:58.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.441+0000 7f9bf30a1640 1 --2- 172.21.15.78:0/3329873862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bec102190 0x7f9bec197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:58.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.441+0000 7f9bf30a1640 1 -- 172.21.15.78:0/3329873862 >> 172.21.15.78:0/3329873862 conn(0x7f9bec0fbb40 msgr2=0x7f9bec102bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:58.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.441+0000 7f9bf30a1640 1 -- 172.21.15.78:0/3329873862 shutdown_connections 2024-03-24T00:18:58.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.441+0000 7f9bf30a1640 1 -- 172.21.15.78:0/3329873862 wait complete. 2024-03-24T00:18:58.451 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:58 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1025257820' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:18:58.452 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:58 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3998413172' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:18:58.452 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:58 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/822381528' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:18:58.453 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T00:18:58.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.556+0000 7f903389a640 1 -- 172.21.15.78:0/380952109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f902c101790 msgr2=0x7f902c101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:58.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.556+0000 7f903389a640 1 --2- 172.21.15.78:0/380952109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f902c101790 0x7f902c101c10 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9020009900 tx=0x7f902002bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:58.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.556+0000 7f903389a640 1 -- 172.21.15.78:0/380952109 shutdown_connections 2024-03-24T00:18:58.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.556+0000 7f903389a640 1 --2- 172.21.15.78:0/380952109 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f902c102150 0x7f902c1089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:58.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.556+0000 7f903389a640 1 --2- 172.21.15.78:0/380952109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f902c101790 0x7f902c101c10 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:58.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.556+0000 7f903389a640 1 --2- 172.21.15.78:0/380952109 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f902c1005a0 0x7f902c1009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:58.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.556+0000 7f903389a640 1 -- 172.21.15.78:0/380952109 >> 172.21.15.78:0/380952109 conn(0x7f902c0fbb30 msgr2=0x7f902c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:58.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.556+0000 7f903389a640 1 -- 172.21.15.78:0/380952109 shutdown_connections 2024-03-24T00:18:58.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.556+0000 7f903389a640 1 -- 172.21.15.78:0/380952109 wait complete. 2024-03-24T00:18:58.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.557+0000 7f903389a640 1 Processor -- start 2024-03-24T00:18:58.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.557+0000 7f903389a640 1 -- start start 2024-03-24T00:18:58.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.557+0000 7f903389a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f902c1005a0 0x7f902c197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:58.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.557+0000 7f903389a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f902c101790 0x7f902c197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:58.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.557+0000 7f903389a640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f902c102150 0x7f902c19ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:58.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.557+0000 7f903389a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902c10c230 con 0x7f902c101790 2024-03-24T00:18:58.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.557+0000 7f903389a640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902c10c0b0 con 0x7f902c102150 2024-03-24T00:18:58.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.557+0000 7f903389a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902c10c3b0 con 0x7f902c1005a0 2024-03-24T00:18:58.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.557+0000 7f9030e35640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f902c101790 0x7f902c197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:58.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.557+0000 7f9030e35640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f902c101790 0x7f902c197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34946/0 (socket says 172.21.15.78:34946) 2024-03-24T00:18:58.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.557+0000 7f9030e35640 1 -- 172.21.15.78:0/561297507 learned_addr learned my addr 172.21.15.78:0/561297507 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:58.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.557+0000 7f9031e37640 1 --2- 172.21.15.78:0/561297507 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f902c102150 0x7f902c19ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:58.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.557+0000 7f9031636640 1 --2- 172.21.15.78:0/561297507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f902c1005a0 0x7f902c197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:58.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.557+0000 7f9030e35640 1 -- 172.21.15.78:0/561297507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f902c1005a0 msgr2=0x7f902c197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:58.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.557+0000 7f9030e35640 1 --2- 172.21.15.78:0/561297507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f902c1005a0 0x7f902c197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:58.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.558+0000 7f9030e35640 1 -- 172.21.15.78:0/561297507 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f902c102150 msgr2=0x7f902c19ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:58.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.558+0000 7f9030e35640 1 --2- 172.21.15.78:0/561297507 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f902c102150 0x7f902c19ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:58.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.558+0000 7f9030e35640 1 -- 172.21.15.78:0/561297507 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f902c19f5e0 con 0x7f902c101790 2024-03-24T00:18:58.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.558+0000 7f9030e35640 1 --2- 172.21.15.78:0/561297507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f902c101790 0x7f902c197f50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9020009230 tx=0x7f9020005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:58.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.558+0000 7f902a7fc640 1 -- 172.21.15.78:0/561297507 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9020041070 con 0x7f902c101790 2024-03-24T00:18:58.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.558+0000 7f902a7fc640 1 -- 172.21.15.78:0/561297507 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f90200044d0 con 0x7f902c101790 2024-03-24T00:18:58.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.558+0000 7f902a7fc640 1 -- 172.21.15.78:0/561297507 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9020034440 con 0x7f902c101790 2024-03-24T00:18:58.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.558+0000 7f903389a640 1 -- 172.21.15.78:0/561297507 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f902c19f870 con 0x7f902c101790 2024-03-24T00:18:58.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.558+0000 7f903389a640 1 -- 172.21.15.78:0/561297507 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f902c19fd60 con 0x7f902c101790 2024-03-24T00:18:58.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.559+0000 7f903389a640 1 -- 172.21.15.78:0/561297507 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f902c19ffc0 con 0x7f902c101790 2024-03-24T00:18:58.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.560+0000 7f902a7fc640 1 -- 172.21.15.78:0/561297507 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9020044070 con 0x7f902c101790 2024-03-24T00:18:58.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.560+0000 7f902a7fc640 1 --2- 172.21.15.78:0/561297507 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f90140a2800 0x7f90140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:58.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.560+0000 7f9031636640 1 --2- 172.21.15.78:0/561297507 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f90140a2800 0x7f90140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:58.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.561+0000 7f902a7fc640 1 -- 172.21.15.78:0/561297507 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f90200f0870 con 0x7f902c101790 2024-03-24T00:18:58.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.561+0000 7f9031636640 1 --2- 172.21.15.78:0/561297507 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f90140a2800 0x7f90140a4cc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f902c1015f0 tx=0x7f901c009680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:58.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.565+0000 7f902a7fc640 1 -- 172.21.15.78:0/561297507 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f90200bd080 con 0x7f902c101790 2024-03-24T00:18:58.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:58 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1025257820' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:18:58.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:58 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3998413172' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:18:58.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:58 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/822381528' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:18:58.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:58 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1025257820' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:18:58.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:58 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3998413172' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:18:58.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:58 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/822381528' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:18:58.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.737+0000 7f903389a640 1 -- 172.21.15.78:0/561297507 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f902c1a0190 con 0x7f902c101790 2024-03-24T00:18:58.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.737+0000 7f902a7fc640 1 -- 172.21.15.78:0/561297507 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7f90200c1120 con 0x7f902c101790 2024-03-24T00:18:58.738 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-24T00:18:58.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.739+0000 7f903389a640 1 -- 172.21.15.78:0/561297507 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f90140a2800 msgr2=0x7f90140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:58.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.739+0000 7f903389a640 1 --2- 172.21.15.78:0/561297507 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f90140a2800 0x7f90140a4cc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f902c1015f0 tx=0x7f901c009680 comp rx=0 tx=0).stop 2024-03-24T00:18:58.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.739+0000 7f903389a640 1 -- 172.21.15.78:0/561297507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f902c101790 msgr2=0x7f902c197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:58.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.739+0000 7f903389a640 1 --2- 172.21.15.78:0/561297507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f902c101790 0x7f902c197f50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9020009230 tx=0x7f9020005d50 comp rx=0 tx=0).stop 2024-03-24T00:18:58.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.739+0000 7f903389a640 1 -- 172.21.15.78:0/561297507 shutdown_connections 2024-03-24T00:18:58.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.739+0000 7f903389a640 1 --2- 172.21.15.78:0/561297507 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f90140a2800 0x7f90140a4cc0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:58.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.739+0000 7f903389a640 1 --2- 172.21.15.78:0/561297507 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f902c102150 0x7f902c19ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:58.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.739+0000 7f903389a640 1 --2- 172.21.15.78:0/561297507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f902c101790 0x7f902c197f50 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:58.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.739+0000 7f903389a640 1 --2- 172.21.15.78:0/561297507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f902c1005a0 0x7f902c197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:58.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.739+0000 7f903389a640 1 -- 172.21.15.78:0/561297507 >> 172.21.15.78:0/561297507 conn(0x7f902c0fbb30 msgr2=0x7f902c0fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:58.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.739+0000 7f903389a640 1 -- 172.21.15.78:0/561297507 shutdown_connections 2024-03-24T00:18:58.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.739+0000 7f903389a640 1 -- 172.21.15.78:0/561297507 wait complete. 2024-03-24T00:18:58.751 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:18:58.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.853+0000 7fc141b1b640 1 -- 172.21.15.78:0/3085370814 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc13c107380 msgr2=0x7fc13c1097d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:58.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.853+0000 7fc141b1b640 1 --2- 172.21.15.78:0/3085370814 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc13c107380 0x7fc13c1097d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc134009ca0 tx=0x7fc13402c0f0 comp rx=0 tx=0).stop 2024-03-24T00:18:58.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.853+0000 7fc141b1b640 1 -- 172.21.15.78:0/3085370814 shutdown_connections 2024-03-24T00:18:58.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.853+0000 7fc141b1b640 1 --2- 172.21.15.78:0/3085370814 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc13c104a50 0x7fc13c106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:58.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.853+0000 7fc141b1b640 1 --2- 172.21.15.78:0/3085370814 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc13c107380 0x7fc13c1097d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:58.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.853+0000 7fc141b1b640 1 --2- 172.21.15.78:0/3085370814 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc13c102120 0x7fc13c104510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:58.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.853+0000 7fc141b1b640 1 -- 172.21.15.78:0/3085370814 >> 172.21.15.78:0/3085370814 conn(0x7fc13c0fbb30 msgr2=0x7fc13c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:58.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.853+0000 7fc141b1b640 1 -- 172.21.15.78:0/3085370814 shutdown_connections 2024-03-24T00:18:58.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.853+0000 7fc141b1b640 1 -- 172.21.15.78:0/3085370814 wait complete. 2024-03-24T00:18:58.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.853+0000 7fc141b1b640 1 Processor -- start 2024-03-24T00:18:58.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.853+0000 7fc141b1b640 1 -- start start 2024-03-24T00:18:58.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.854+0000 7fc141b1b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc13c102120 0x7fc13c197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:58.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.854+0000 7fc141b1b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc13c104a50 0x7fc13c197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:58.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.854+0000 7fc141b1b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc13c107380 0x7fc13c19ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:58.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.854+0000 7fc141b1b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c10d330 con 0x7fc13c102120 2024-03-24T00:18:58.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.854+0000 7fc141b1b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c10d1b0 con 0x7fc13c107380 2024-03-24T00:18:58.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.854+0000 7fc141b1b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c10d4b0 con 0x7fc13c104a50 2024-03-24T00:18:58.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.854+0000 7fc13b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc13c102120 0x7fc13c197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:58.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.854+0000 7fc13b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc13c102120 0x7fc13c197980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34970/0 (socket says 172.21.15.78:34970) 2024-03-24T00:18:58.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.854+0000 7fc13b7fe640 1 -- 172.21.15.78:0/3108630146 learned_addr learned my addr 172.21.15.78:0/3108630146 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:18:58.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.854+0000 7fc13bfff640 1 --2- 172.21.15.78:0/3108630146 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc13c107380 0x7fc13c19ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:58.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.854+0000 7fc13b7fe640 1 -- 172.21.15.78:0/3108630146 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc13c104a50 msgr2=0x7fc13c197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:58.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.854+0000 7fc13affd640 1 --2- 172.21.15.78:0/3108630146 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc13c104a50 0x7fc13c197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:58.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.854+0000 7fc13b7fe640 1 --2- 172.21.15.78:0/3108630146 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc13c104a50 0x7fc13c197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:58.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.854+0000 7fc13b7fe640 1 -- 172.21.15.78:0/3108630146 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc13c107380 msgr2=0x7fc13c19ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:58.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.854+0000 7fc13b7fe640 1 --2- 172.21.15.78:0/3108630146 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc13c107380 0x7fc13c19ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:58.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.854+0000 7fc13b7fe640 1 -- 172.21.15.78:0/3108630146 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc13c19f5a0 con 0x7fc13c102120 2024-03-24T00:18:58.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.854+0000 7fc13b7fe640 1 --2- 172.21.15.78:0/3108630146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc13c102120 0x7fc13c197980 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc12c00e9f0 tx=0x7fc12c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:58.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.854+0000 7fc138ff9640 1 -- 172.21.15.78:0/3108630146 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c00cd80 con 0x7fc13c102120 2024-03-24T00:18:58.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.854+0000 7fc141b1b640 1 -- 172.21.15.78:0/3108630146 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc13c19f890 con 0x7fc13c102120 2024-03-24T00:18:58.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.854+0000 7fc138ff9640 1 -- 172.21.15.78:0/3108630146 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc12c010910 con 0x7fc13c102120 2024-03-24T00:18:58.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.854+0000 7fc141b1b640 1 -- 172.21.15.78:0/3108630146 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc13c19fe30 con 0x7fc13c102120 2024-03-24T00:18:58.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.854+0000 7fc138ff9640 1 -- 172.21.15.78:0/3108630146 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c018910 con 0x7fc13c102120 2024-03-24T00:18:58.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.856+0000 7fc141b1b640 1 -- 172.21.15.78:0/3108630146 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc13c0fd620 con 0x7fc13c102120 2024-03-24T00:18:58.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.856+0000 7fc138ff9640 1 -- 172.21.15.78:0/3108630146 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc12c010ac0 con 0x7fc13c102120 2024-03-24T00:18:58.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.857+0000 7fc138ff9640 1 --2- 172.21.15.78:0/3108630146 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fc1240a2800 0x7fc1240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:58.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.857+0000 7fc13affd640 1 --2- 172.21.15.78:0/3108630146 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fc1240a2800 0x7fc1240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:58.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.857+0000 7fc138ff9640 1 -- 172.21.15.78:0/3108630146 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fc12c0cf0b0 con 0x7fc13c102120 2024-03-24T00:18:58.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.857+0000 7fc13affd640 1 --2- 172.21.15.78:0/3108630146 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fc1240a2800 0x7fc1240a4cc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc130005fd0 tx=0x7fc130005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:58.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:58.861+0000 7fc138ff9640 1 -- 172.21.15.78:0/3108630146 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc12c09bf10 con 0x7fc13c102120 2024-03-24T00:18:59.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:59.046+0000 7fc141b1b640 1 -- 172.21.15.78:0/3108630146 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc13c0622a0 con 0x7fc13c102120 2024-03-24T00:18:59.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:59.047+0000 7fc138ff9640 1 -- 172.21.15.78:0/3108630146 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) v1 ==== 81+0+591 (secure 0 0 0) 0x7fc12c09b810 con 0x7fc13c102120 2024-03-24T00:18:59.048 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1982202541 2024-03-25T00:11:40.199569+0000 2024-03-24T00:18:59.048 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1933054339 2024-03-25T00:11:40.199569+0000 2024-03-24T00:18:59.048 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/2748920178 2024-03-25T00:11:40.199569+0000 2024-03-24T00:18:59.048 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/2748920178 2024-03-25T00:11:40.199569+0000 2024-03-24T00:18:59.048 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/465691596 2024-03-25T00:11:14.637881+0000 2024-03-24T00:18:59.048 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/2913847518 2024-03-25T00:11:14.637881+0000 2024-03-24T00:18:59.048 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/3962523609 2024-03-25T00:11:14.637881+0000 2024-03-24T00:18:59.048 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/2913847518 2024-03-25T00:11:14.637881+0000 2024-03-24T00:18:59.048 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1502752625 2024-03-25T00:11:14.637881+0000 2024-03-24T00:18:59.048 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/2236879401 2024-03-25T00:11:40.199569+0000 2024-03-24T00:18:59.049 INFO:teuthology.orchestra.run.smithi078.stderr:listed 10 entries 2024-03-24T00:18:59.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:59.049+0000 7fc141b1b640 1 -- 172.21.15.78:0/3108630146 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fc1240a2800 msgr2=0x7fc1240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:59.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:59.049+0000 7fc141b1b640 1 --2- 172.21.15.78:0/3108630146 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fc1240a2800 0x7fc1240a4cc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc130005fd0 tx=0x7fc130005850 comp rx=0 tx=0).stop 2024-03-24T00:18:59.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:59.049+0000 7fc141b1b640 1 -- 172.21.15.78:0/3108630146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc13c102120 msgr2=0x7fc13c197980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:59.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:59.049+0000 7fc141b1b640 1 --2- 172.21.15.78:0/3108630146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc13c102120 0x7fc13c197980 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc12c00e9f0 tx=0x7fc12c00eeb0 comp rx=0 tx=0).stop 2024-03-24T00:18:59.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:59.049+0000 7fc141b1b640 1 -- 172.21.15.78:0/3108630146 shutdown_connections 2024-03-24T00:18:59.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:59.049+0000 7fc141b1b640 1 --2- 172.21.15.78:0/3108630146 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fc1240a2800 0x7fc1240a4cc0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:59.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:59.049+0000 7fc141b1b640 1 --2- 172.21.15.78:0/3108630146 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc13c104a50 0x7fc13c197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:59.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:59.049+0000 7fc141b1b640 1 --2- 172.21.15.78:0/3108630146 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc13c107380 0x7fc13c19ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:59.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:59.049+0000 7fc141b1b640 1 --2- 172.21.15.78:0/3108630146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc13c102120 0x7fc13c197980 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:59.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:59.049+0000 7fc141b1b640 1 -- 172.21.15.78:0/3108630146 >> 172.21.15.78:0/3108630146 conn(0x7fc13c0fbb30 msgr2=0x7fc13c102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:59.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:59.049+0000 7fc141b1b640 1 -- 172.21.15.78:0/3108630146 shutdown_connections 2024-03-24T00:18:59.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:18:59.049+0000 7fc141b1b640 1 -- 172.21.15.78:0/3108630146 wait complete. 2024-03-24T00:18:59.060 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi078.front.sepia.ceph.com... 2024-03-24T00:18:59.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:18:59.061 DEBUG:teuthology.orchestra.run.smithi078:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-24T00:18:59.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi078.stderr:ceph-fuse[73883]: fuse finished with error 0 and tester_r 0 2024-03-24T00:18:59.089 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:18:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:59 smithi078 ceph-mon[32429]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 1 op/s 2024-03-24T00:18:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:59 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3329873862' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:18:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:59 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/561297507' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T00:18:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:18:59 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3108630146' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T00:18:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:59 smithi106 ceph-mon[36505]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 1 op/s 2024-03-24T00:18:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:59 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3329873862' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:18:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:59 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/561297507' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T00:18:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:18:59 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3108630146' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T00:18:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:59 smithi107 ceph-mon[36485]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 1 op/s 2024-03-24T00:18:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:59 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3329873862' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:18:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:59 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/561297507' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T00:18:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:18:59 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3108630146' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T00:19:01.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:01 smithi106 ceph-mon[36505]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 628 B/s wr, 1 op/s 2024-03-24T00:19:01.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:01 smithi078 ceph-mon[32429]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 628 B/s wr, 1 op/s 2024-03-24T00:19:01.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:01 smithi107 ceph-mon[36485]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 628 B/s wr, 1 op/s 2024-03-24T00:19:03.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:03 smithi106 ceph-mon[36505]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-24T00:19:03.674 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:03 smithi078 ceph-mon[32429]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-24T00:19:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:03 smithi107 ceph-mon[36485]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-24T00:19:05.191 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi078.front.sepia.ceph.com 2024-03-24T00:19:05.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:19:05.191 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-24T00:19:05.223 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:19:05.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.378+0000 7f250ffc6640 1 -- 172.21.15.78:0/856117343 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2508100f40 msgr2=0x7f25081013c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:05.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.378+0000 7f250ffc6640 1 --2- 172.21.15.78:0/856117343 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2508100f40 0x7f25081013c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f24fc009900 tx=0x7f24fc02bc30 comp rx=0 tx=0).stop 2024-03-24T00:19:05.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.378+0000 7f250ffc6640 1 -- 172.21.15.78:0/856117343 shutdown_connections 2024-03-24T00:19:05.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.378+0000 7f250ffc6640 1 --2- 172.21.15.78:0/856117343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2508101900 0x7f2508108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:05.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.378+0000 7f250ffc6640 1 --2- 172.21.15.78:0/856117343 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2508100f40 0x7f25081013c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:05.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.378+0000 7f250ffc6640 1 --2- 172.21.15.78:0/856117343 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f25080ffd50 0x7f2508100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:05.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.378+0000 7f250ffc6640 1 -- 172.21.15.78:0/856117343 >> 172.21.15.78:0/856117343 conn(0x7f25080fb2c0 msgr2=0x7f25080fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:05.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.378+0000 7f250ffc6640 1 -- 172.21.15.78:0/856117343 shutdown_connections 2024-03-24T00:19:05.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.378+0000 7f250ffc6640 1 -- 172.21.15.78:0/856117343 wait complete. 2024-03-24T00:19:05.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.378+0000 7f250ffc6640 1 Processor -- start 2024-03-24T00:19:05.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.379+0000 7f250ffc6640 1 -- start start 2024-03-24T00:19:05.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.379+0000 7f250ffc6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25080ffd50 0x7f2508195780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:05.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.379+0000 7f250ffc6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2508100f40 0x7f2508195cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:05.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.379+0000 7f250ffc6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2508101900 0x7f250819cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:05.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.379+0000 7f250ffc6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250810b160 con 0x7f2508101900 2024-03-24T00:19:05.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.379+0000 7f250ffc6640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250810afe0 con 0x7f2508100f40 2024-03-24T00:19:05.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.379+0000 7f250ffc6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250810b2e0 con 0x7f25080ffd50 2024-03-24T00:19:05.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.379+0000 7f250e563640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2508101900 0x7f250819cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:05.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.379+0000 7f250e563640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2508101900 0x7f250819cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58152/0 (socket says 172.21.15.78:58152) 2024-03-24T00:19:05.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.379+0000 7f250e563640 1 -- 172.21.15.78:0/2659051951 learned_addr learned my addr 172.21.15.78:0/2659051951 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:19:05.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.379+0000 7f250dd62640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25080ffd50 0x7f2508195780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:05.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.379+0000 7f250d561640 1 --2- 172.21.15.78:0/2659051951 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2508100f40 0x7f2508195cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:05.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.379+0000 7f250d561640 1 -- 172.21.15.78:0/2659051951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25080ffd50 msgr2=0x7f2508195780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:05.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.379+0000 7f250d561640 1 --2- 172.21.15.78:0/2659051951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25080ffd50 0x7f2508195780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:05.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.379+0000 7f250d561640 1 -- 172.21.15.78:0/2659051951 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2508101900 msgr2=0x7f250819cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:05.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.379+0000 7f250d561640 1 --2- 172.21.15.78:0/2659051951 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2508101900 0x7f250819cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:05.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.379+0000 7f250d561640 1 -- 172.21.15.78:0/2659051951 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f250819d3a0 con 0x7f2508100f40 2024-03-24T00:19:05.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.379+0000 7f250e563640 1 --2- 172.21.15.78:0/2659051951 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2508101900 0x7f250819cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:05.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.379+0000 7f250d561640 1 --2- 172.21.15.78:0/2659051951 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2508100f40 0x7f2508195cc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f24fc009230 tx=0x7f24fc02c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:05.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.380+0000 7f2506ffd640 1 -- 172.21.15.78:0/2659051951 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24fc041070 con 0x7f2508100f40 2024-03-24T00:19:05.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.380+0000 7f2506ffd640 1 -- 172.21.15.78:0/2659051951 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f24fc004450 con 0x7f2508100f40 2024-03-24T00:19:05.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.380+0000 7f2506ffd640 1 -- 172.21.15.78:0/2659051951 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24fc034440 con 0x7f2508100f40 2024-03-24T00:19:05.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.380+0000 7f250ffc6640 1 -- 172.21.15.78:0/2659051951 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f250819d630 con 0x7f2508100f40 2024-03-24T00:19:05.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.380+0000 7f250ffc6640 1 -- 172.21.15.78:0/2659051951 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f250819db70 con 0x7f2508100f40 2024-03-24T00:19:05.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.381+0000 7f250ffc6640 1 -- 172.21.15.78:0/2659051951 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2508107010 con 0x7f2508100f40 2024-03-24T00:19:05.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.381+0000 7f2506ffd640 1 -- 172.21.15.78:0/2659051951 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f24fc044070 con 0x7f2508100f40 2024-03-24T00:19:05.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.382+0000 7f2506ffd640 1 --2- 172.21.15.78:0/2659051951 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f24f00a2880 0x7f24f00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:05.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.382+0000 7f250dd62640 1 --2- 172.21.15.78:0/2659051951 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f24f00a2880 0x7f24f00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:05.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.382+0000 7f2506ffd640 1 -- 172.21.15.78:0/2659051951 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f24fc0f0750 con 0x7f2508100f40 2024-03-24T00:19:05.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.382+0000 7f250dd62640 1 --2- 172.21.15.78:0/2659051951 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f24f00a2880 0x7f24f00a4d40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2508100da0 tx=0x7f2500011040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:05.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.387+0000 7f2506ffd640 1 -- 172.21.15.78:0/2659051951 <== mon.2 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f24fc0bcf60 con 0x7f2508100f40 2024-03-24T00:19:05.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.564+0000 7f250ffc6640 1 -- 172.21.15.78:0/2659051951 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25080622a0 con 0x7f2508100f40 2024-03-24T00:19:05.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.567+0000 7f2506ffd640 1 -- 172.21.15.78:0/2659051951 <== mon.2 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24129 (secure 0 0 0) 0x7f24fc0c1000 con 0x7f2508100f40 2024-03-24T00:19:05.567 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:19:05.567 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":89,"fsid":"c7e2806a-e972-11ee-95cd-87774f69a715","created":"2024-03-24T00:10:44.134563+0000","modified":"2024-03-24T00:18:48.224440+0000","last_up_change":"2024-03-24T00:17:50.237686+0000","last_in_change":"2024-03-24T00:17:29.788699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:14:04.534391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:18:42.178418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:18:42.800424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:18:46.112800+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b950388a-c437-426b-b703-3f8ae821dd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6803","nonce":1816879397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6805","nonce":1816879397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6809","nonce":1816879397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6807","nonce":1816879397}]},"public_addr":"172.21.15.78:6803/1816879397","cluster_addr":"172.21.15.78:6805/1816879397","heartbeat_back_addr":"172.21.15.78:6809/1816879397","heartbeat_front_addr":"172.21.15.78:6807/1816879397","state":["exists","up"]},{"osd":1,"uuid":"b8d64714-dd65-400f-b3f4-673ab14e1365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6801","nonce":1647915612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6803","nonce":1647915612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6807","nonce":1647915612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6805","nonce":1647915612}]},"public_addr":"172.21.15.106:6801/1647915612","cluster_addr":"172.21.15.106:6803/1647915612","heartbeat_back_addr":"172.21.15.106:6807/1647915612","heartbeat_front_addr":"172.21.15.106:6805/1647915612","state":["exists","up"]},{"osd":2,"uuid":"fcfc4dd1-aaf8-4131-a023-5e832212158d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6801","nonce":2010770435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6803","nonce":2010770435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6807","nonce":2010770435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6805","nonce":2010770435}]},"public_addr":"172.21.15.107:6801/2010770435","cluster_addr":"172.21.15.107:6803/2010770435","heartbeat_back_addr":"172.21.15.107:6807/2010770435","heartbeat_front_addr":"172.21.15.107:6805/2010770435","state":["exists","up"]},{"osd":3,"uuid":"47c718ab-a0f4-4e52-83c5-b89386b262d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6811","nonce":1912251346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6813","nonce":1912251346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6817","nonce":1912251346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6815","nonce":1912251346}]},"public_addr":"172.21.15.78:6811/1912251346","cluster_addr":"172.21.15.78:6813/1912251346","heartbeat_back_addr":"172.21.15.78:6817/1912251346","heartbeat_front_addr":"172.21.15.78:6815/1912251346","state":["exists","up"]},{"osd":4,"uuid":"7dad64dd-677e-4c1c-9638-e4f4f2195983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6809","nonce":374641205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6811","nonce":374641205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6815","nonce":374641205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6813","nonce":374641205}]},"public_addr":"172.21.15.106:6809/374641205","cluster_addr":"172.21.15.106:6811/374641205","heartbeat_back_addr":"172.21.15.106:6815/374641205","heartbeat_front_addr":"172.21.15.106:6813/374641205","state":["exists","up"]},{"osd":5,"uuid":"bbd524e5-328d-4309-8877-1149cbe110dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6809","nonce":3272806073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6811","nonce":3272806073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6815","nonce":3272806073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6813","nonce":3272806073}]},"public_addr":"172.21.15.107:6809/3272806073","cluster_addr":"172.21.15.107:6811/3272806073","heartbeat_back_addr":"172.21.15.107:6815/3272806073","heartbeat_front_addr":"172.21.15.107:6813/3272806073","state":["exists","up"]},{"osd":6,"uuid":"5696389a-918f-4ea8-b20d-4e2a0d4cb3bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6819","nonce":3091148508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6821","nonce":3091148508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6825","nonce":3091148508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6823","nonce":3091148508}]},"public_addr":"172.21.15.78:6819/3091148508","cluster_addr":"172.21.15.78:6821/3091148508","heartbeat_back_addr":"172.21.15.78:6825/3091148508","heartbeat_front_addr":"172.21.15.78:6823/3091148508","state":["exists","up"]},{"osd":7,"uuid":"f6b764e8-ed4e-4c9a-a85b-94af6dc6b924","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6817","nonce":4262103080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6819","nonce":4262103080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6823","nonce":4262103080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6821","nonce":4262103080}]},"public_addr":"172.21.15.106:6817/4262103080","cluster_addr":"172.21.15.106:6819/4262103080","heartbeat_back_addr":"172.21.15.106:6823/4262103080","heartbeat_front_addr":"172.21.15.106:6821/4262103080","state":["exists","up"]},{"osd":8,"uuid":"ba16039f-8093-4fa0-87ad-e28b7d142b23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6817","nonce":4032799356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6819","nonce":4032799356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6823","nonce":4032799356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6821","nonce":4032799356}]},"public_addr":"172.21.15.107:6817/4032799356","cluster_addr":"172.21.15.107:6819/4032799356","heartbeat_back_addr":"172.21.15.107:6823/4032799356","heartbeat_front_addr":"172.21.15.107:6821/4032799356","state":["exists","up"]},{"osd":9,"uuid":"651357f2-0669-42e6-aa91-1d5c02d00c5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6827","nonce":214023431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6829","nonce":214023431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6833","nonce":214023431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6831","nonce":214023431}]},"public_addr":"172.21.15.78:6827/214023431","cluster_addr":"172.21.15.78:6829/214023431","heartbeat_back_addr":"172.21.15.78:6833/214023431","heartbeat_front_addr":"172.21.15.78:6831/214023431","state":["exists","up"]},{"osd":10,"uuid":"2375119d-346e-426c-8a65-b64af71ad63a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6825","nonce":400029804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6827","nonce":400029804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6831","nonce":400029804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6829","nonce":400029804}]},"public_addr":"172.21.15.106:6825/400029804","cluster_addr":"172.21.15.106:6827/400029804","heartbeat_back_addr":"172.21.15.106:6831/400029804","heartbeat_front_addr":"172.21.15.106:6829/400029804","state":["exists","up"]},{"osd":11,"uuid":"b6b609d9-dd78-4c01-ae66-ba2a662ab22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6825","nonce":4191916230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6827","nonce":4191916230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6831","nonce":4191916230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6829","nonce":4191916230}]},"public_addr":"172.21.15.107:6825/4191916230","cluster_addr":"172.21.15.107:6827/4191916230","heartbeat_back_addr":"172.21.15.107:6831/4191916230","heartbeat_front_addr":"172.21.15.107:6829/4191916230","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:13.728602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:37.320809+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:01.269452+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:25.758281+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:50.298863+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:14.345219+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:38.980489+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:04.749214+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:29.992673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:56.269202+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:22.270316+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:48.074907+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/1982202541":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/1933054339":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6801/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6800/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/465691596":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6801/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/3962523609":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6800/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/1502752625":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/2236879401":"2024-03-25T00:11:40.199569+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:19:05.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.568+0000 7f250ffc6640 1 -- 172.21.15.78:0/2659051951 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f24f00a2880 msgr2=0x7f24f00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:05.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.568+0000 7f250ffc6640 1 --2- 172.21.15.78:0/2659051951 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f24f00a2880 0x7f24f00a4d40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2508100da0 tx=0x7f2500011040 comp rx=0 tx=0).stop 2024-03-24T00:19:05.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.568+0000 7f250ffc6640 1 -- 172.21.15.78:0/2659051951 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2508100f40 msgr2=0x7f2508195cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:05.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.568+0000 7f250ffc6640 1 --2- 172.21.15.78:0/2659051951 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2508100f40 0x7f2508195cc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f24fc009230 tx=0x7f24fc02c6c0 comp rx=0 tx=0).stop 2024-03-24T00:19:05.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.568+0000 7f250ffc6640 1 -- 172.21.15.78:0/2659051951 shutdown_connections 2024-03-24T00:19:05.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.568+0000 7f250ffc6640 1 --2- 172.21.15.78:0/2659051951 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f24f00a2880 0x7f24f00a4d40 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:05.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.568+0000 7f250ffc6640 1 --2- 172.21.15.78:0/2659051951 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2508101900 0x7f250819cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:05.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.568+0000 7f250ffc6640 1 --2- 172.21.15.78:0/2659051951 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2508100f40 0x7f2508195cc0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:05.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.568+0000 7f250ffc6640 1 --2- 172.21.15.78:0/2659051951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25080ffd50 0x7f2508195780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:05.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.568+0000 7f250ffc6640 1 -- 172.21.15.78:0/2659051951 >> 172.21.15.78:0/2659051951 conn(0x7f25080fb2c0 msgr2=0x7f25081041e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:05.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.568+0000 7f250ffc6640 1 -- 172.21.15.78:0/2659051951 shutdown_connections 2024-03-24T00:19:05.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.568+0000 7f250ffc6640 1 -- 172.21.15.78:0/2659051951 wait complete. 2024-03-24T00:19:05.578 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:05 smithi078 ceph-mon[32429]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-24T00:19:05.579 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:19:05.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:05 smithi106 ceph-mon[36505]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-24T00:19:05.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.680+0000 7ff7df8a1640 1 -- 172.21.15.78:0/905866517 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7d81017d0 msgr2=0x7ff7d8101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:05.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.680+0000 7ff7df8a1640 1 --2- 172.21.15.78:0/905866517 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7d81017d0 0x7ff7d8101c50 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff7d4009900 tx=0x7ff7d402bc30 comp rx=0 tx=0).stop 2024-03-24T00:19:05.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.680+0000 7ff7df8a1640 1 -- 172.21.15.78:0/905866517 shutdown_connections 2024-03-24T00:19:05.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.680+0000 7ff7df8a1640 1 --2- 172.21.15.78:0/905866517 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7d8102190 0x7ff7d8108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:05.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.680+0000 7ff7df8a1640 1 --2- 172.21.15.78:0/905866517 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7d81017d0 0x7ff7d8101c50 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:05.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.680+0000 7ff7df8a1640 1 --2- 172.21.15.78:0/905866517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff7d81005e0 0x7ff7d8100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:05.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.680+0000 7ff7df8a1640 1 -- 172.21.15.78:0/905866517 >> 172.21.15.78:0/905866517 conn(0x7ff7d80fbb50 msgr2=0x7ff7d80fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:05.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.680+0000 7ff7df8a1640 1 -- 172.21.15.78:0/905866517 shutdown_connections 2024-03-24T00:19:05.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.680+0000 7ff7df8a1640 1 -- 172.21.15.78:0/905866517 wait complete. 2024-03-24T00:19:05.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.680+0000 7ff7df8a1640 1 Processor -- start 2024-03-24T00:19:05.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.680+0000 7ff7df8a1640 1 -- start start 2024-03-24T00:19:05.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.681+0000 7ff7df8a1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff7d81005e0 0x7ff7d81968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:05.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.681+0000 7ff7df8a1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7d81017d0 0x7ff7d8196de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:05.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.681+0000 7ff7df8a1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7d8102190 0x7ff7d819de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:05.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.681+0000 7ff7df8a1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d810c260 con 0x7ff7d81017d0 2024-03-24T00:19:05.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.681+0000 7ff7df8a1640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d810c0e0 con 0x7ff7d81005e0 2024-03-24T00:19:05.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.681+0000 7ff7df8a1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d810c3e0 con 0x7ff7d8102190 2024-03-24T00:19:05.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.681+0000 7ff7dce3c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7d81017d0 0x7ff7d8196de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:05.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.681+0000 7ff7dce3c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7d81017d0 0x7ff7d8196de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58174/0 (socket says 172.21.15.78:58174) 2024-03-24T00:19:05.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.681+0000 7ff7dce3c640 1 -- 172.21.15.78:0/4149511573 learned_addr learned my addr 172.21.15.78:0/4149511573 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:19:05.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.681+0000 7ff7dd63d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff7d81005e0 0x7ff7d81968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:05.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.681+0000 7ff7dde3e640 1 --2- 172.21.15.78:0/4149511573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7d8102190 0x7ff7d819de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:05.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.681+0000 7ff7dce3c640 1 -- 172.21.15.78:0/4149511573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7d8102190 msgr2=0x7ff7d819de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:05.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.681+0000 7ff7dce3c640 1 --2- 172.21.15.78:0/4149511573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7d8102190 0x7ff7d819de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:05.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.681+0000 7ff7dce3c640 1 -- 172.21.15.78:0/4149511573 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff7d81005e0 msgr2=0x7ff7d81968a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:05.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.681+0000 7ff7dce3c640 1 --2- 172.21.15.78:0/4149511573 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff7d81005e0 0x7ff7d81968a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:05.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.681+0000 7ff7dce3c640 1 -- 172.21.15.78:0/4149511573 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d819e4c0 con 0x7ff7d81017d0 2024-03-24T00:19:05.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.681+0000 7ff7dd63d640 1 --2- 172.21.15.78:0/4149511573 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff7d81005e0 0x7ff7d81968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:05.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.681+0000 7ff7dce3c640 1 --2- 172.21.15.78:0/4149511573 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7d81017d0 0x7ff7d8196de0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff7d4009230 tx=0x7ff7d402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:05.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.681+0000 7ff7ca7fc640 1 -- 172.21.15.78:0/4149511573 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d4041070 con 0x7ff7d81017d0 2024-03-24T00:19:05.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.681+0000 7ff7ca7fc640 1 -- 172.21.15.78:0/4149511573 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff7d4004450 con 0x7ff7d81017d0 2024-03-24T00:19:05.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.681+0000 7ff7df8a1640 1 -- 172.21.15.78:0/4149511573 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d819e750 con 0x7ff7d81017d0 2024-03-24T00:19:05.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.681+0000 7ff7ca7fc640 1 -- 172.21.15.78:0/4149511573 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d4003e20 con 0x7ff7d81017d0 2024-03-24T00:19:05.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.681+0000 7ff7df8a1640 1 -- 172.21.15.78:0/4149511573 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d819ec90 con 0x7ff7d81017d0 2024-03-24T00:19:05.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.682+0000 7ff7df8a1640 1 -- 172.21.15.78:0/4149511573 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d8107900 con 0x7ff7d81017d0 2024-03-24T00:19:05.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.683+0000 7ff7ca7fc640 1 -- 172.21.15.78:0/4149511573 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff7d4044070 con 0x7ff7d81017d0 2024-03-24T00:19:05.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.684+0000 7ff7ca7fc640 1 --2- 172.21.15.78:0/4149511573 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff7c40a2880 0x7ff7c40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:05.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.684+0000 7ff7dd63d640 1 --2- 172.21.15.78:0/4149511573 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff7c40a2880 0x7ff7c40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:05.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.684+0000 7ff7ca7fc640 1 -- 172.21.15.78:0/4149511573 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7ff7d40f0860 con 0x7ff7d81017d0 2024-03-24T00:19:05.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.684+0000 7ff7dd63d640 1 --2- 172.21.15.78:0/4149511573 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff7c40a2880 0x7ff7c40a4d40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8101630 tx=0x7ff7cc005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:05.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.689+0000 7ff7ca7fc640 1 -- 172.21.15.78:0/4149511573 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff7d40bd6c0 con 0x7ff7d81017d0 2024-03-24T00:19:05.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:05 smithi107 ceph-mon[36485]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-24T00:19:05.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.865+0000 7ff7df8a1640 1 -- 172.21.15.78:0/4149511573 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7d80622a0 con 0x7ff7d81017d0 2024-03-24T00:19:05.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.866+0000 7ff7ca7fc640 1 -- 172.21.15.78:0/4149511573 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24129 (secure 0 0 0) 0x7ff7d40bcfc0 con 0x7ff7d81017d0 2024-03-24T00:19:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:19:05.868 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":89,"fsid":"c7e2806a-e972-11ee-95cd-87774f69a715","created":"2024-03-24T00:10:44.134563+0000","modified":"2024-03-24T00:18:48.224440+0000","last_up_change":"2024-03-24T00:17:50.237686+0000","last_in_change":"2024-03-24T00:17:29.788699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:14:04.534391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:18:42.178418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:18:42.800424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:18:46.112800+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b950388a-c437-426b-b703-3f8ae821dd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6803","nonce":1816879397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6805","nonce":1816879397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6809","nonce":1816879397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6807","nonce":1816879397}]},"public_addr":"172.21.15.78:6803/1816879397","cluster_addr":"172.21.15.78:6805/1816879397","heartbeat_back_addr":"172.21.15.78:6809/1816879397","heartbeat_front_addr":"172.21.15.78:6807/1816879397","state":["exists","up"]},{"osd":1,"uuid":"b8d64714-dd65-400f-b3f4-673ab14e1365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6801","nonce":1647915612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6803","nonce":1647915612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6807","nonce":1647915612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6805","nonce":1647915612}]},"public_addr":"172.21.15.106:6801/1647915612","cluster_addr":"172.21.15.106:6803/1647915612","heartbeat_back_addr":"172.21.15.106:6807/1647915612","heartbeat_front_addr":"172.21.15.106:6805/1647915612","state":["exists","up"]},{"osd":2,"uuid":"fcfc4dd1-aaf8-4131-a023-5e832212158d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6801","nonce":2010770435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6803","nonce":2010770435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6807","nonce":2010770435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6805","nonce":2010770435}]},"public_addr":"172.21.15.107:6801/2010770435","cluster_addr":"172.21.15.107:6803/2010770435","heartbeat_back_addr":"172.21.15.107:6807/2010770435","heartbeat_front_addr":"172.21.15.107:6805/2010770435","state":["exists","up"]},{"osd":3,"uuid":"47c718ab-a0f4-4e52-83c5-b89386b262d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6811","nonce":1912251346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6813","nonce":1912251346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6817","nonce":1912251346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6815","nonce":1912251346}]},"public_addr":"172.21.15.78:6811/1912251346","cluster_addr":"172.21.15.78:6813/1912251346","heartbeat_back_addr":"172.21.15.78:6817/1912251346","heartbeat_front_addr":"172.21.15.78:6815/1912251346","state":["exists","up"]},{"osd":4,"uuid":"7dad64dd-677e-4c1c-9638-e4f4f2195983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6809","nonce":374641205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6811","nonce":374641205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6815","nonce":374641205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6813","nonce":374641205}]},"public_addr":"172.21.15.106:6809/374641205","cluster_addr":"172.21.15.106:6811/374641205","heartbeat_back_addr":"172.21.15.106:6815/374641205","heartbeat_front_addr":"172.21.15.106:6813/374641205","state":["exists","up"]},{"osd":5,"uuid":"bbd524e5-328d-4309-8877-1149cbe110dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6809","nonce":3272806073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6811","nonce":3272806073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6815","nonce":3272806073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6813","nonce":3272806073}]},"public_addr":"172.21.15.107:6809/3272806073","cluster_addr":"172.21.15.107:6811/3272806073","heartbeat_back_addr":"172.21.15.107:6815/3272806073","heartbeat_front_addr":"172.21.15.107:6813/3272806073","state":["exists","up"]},{"osd":6,"uuid":"5696389a-918f-4ea8-b20d-4e2a0d4cb3bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6819","nonce":3091148508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6821","nonce":3091148508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6825","nonce":3091148508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6823","nonce":3091148508}]},"public_addr":"172.21.15.78:6819/3091148508","cluster_addr":"172.21.15.78:6821/3091148508","heartbeat_back_addr":"172.21.15.78:6825/3091148508","heartbeat_front_addr":"172.21.15.78:6823/3091148508","state":["exists","up"]},{"osd":7,"uuid":"f6b764e8-ed4e-4c9a-a85b-94af6dc6b924","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6817","nonce":4262103080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6819","nonce":4262103080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6823","nonce":4262103080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6821","nonce":4262103080}]},"public_addr":"172.21.15.106:6817/4262103080","cluster_addr":"172.21.15.106:6819/4262103080","heartbeat_back_addr":"172.21.15.106:6823/4262103080","heartbeat_front_addr":"172.21.15.106:6821/4262103080","state":["exists","up"]},{"osd":8,"uuid":"ba16039f-8093-4fa0-87ad-e28b7d142b23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6817","nonce":4032799356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6819","nonce":4032799356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6823","nonce":4032799356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6821","nonce":4032799356}]},"public_addr":"172.21.15.107:6817/4032799356","cluster_addr":"172.21.15.107:6819/4032799356","heartbeat_back_addr":"172.21.15.107:6823/4032799356","heartbeat_front_addr":"172.21.15.107:6821/4032799356","state":["exists","up"]},{"osd":9,"uuid":"651357f2-0669-42e6-aa91-1d5c02d00c5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6827","nonce":214023431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6829","nonce":214023431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6833","nonce":214023431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6831","nonce":214023431}]},"public_addr":"172.21.15.78:6827/214023431","cluster_addr":"172.21.15.78:6829/214023431","heartbeat_back_addr":"172.21.15.78:6833/214023431","heartbeat_front_addr":"172.21.15.78:6831/214023431","state":["exists","up"]},{"osd":10,"uuid":"2375119d-346e-426c-8a65-b64af71ad63a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6825","nonce":400029804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6827","nonce":400029804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6831","nonce":400029804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6829","nonce":400029804}]},"public_addr":"172.21.15.106:6825/400029804","cluster_addr":"172.21.15.106:6827/400029804","heartbeat_back_addr":"172.21.15.106:6831/400029804","heartbeat_front_addr":"172.21.15.106:6829/400029804","state":["exists","up"]},{"osd":11,"uuid":"b6b609d9-dd78-4c01-ae66-ba2a662ab22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6825","nonce":4191916230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6827","nonce":4191916230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6831","nonce":4191916230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6829","nonce":4191916230}]},"public_addr":"172.21.15.107:6825/4191916230","cluster_addr":"172.21.15.107:6827/4191916230","heartbeat_back_addr":"172.21.15.107:6831/4191916230","heartbeat_front_addr":"172.21.15.107:6829/4191916230","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:13.728602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:37.320809+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:01.269452+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:25.758281+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:50.298863+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:14.345219+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:38.980489+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:04.749214+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:29.992673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:56.269202+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:22.270316+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:48.074907+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/1982202541":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/1933054339":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6801/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6800/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/465691596":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6801/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/3962523609":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6800/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/1502752625":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/2236879401":"2024-03-25T00:11:40.199569+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:19:05.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.868+0000 7ff7df8a1640 1 -- 172.21.15.78:0/4149511573 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff7c40a2880 msgr2=0x7ff7c40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:05.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.868+0000 7ff7df8a1640 1 --2- 172.21.15.78:0/4149511573 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff7c40a2880 0x7ff7c40a4d40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8101630 tx=0x7ff7cc005f90 comp rx=0 tx=0).stop 2024-03-24T00:19:05.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.868+0000 7ff7df8a1640 1 -- 172.21.15.78:0/4149511573 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7d81017d0 msgr2=0x7ff7d8196de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:05.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.868+0000 7ff7df8a1640 1 --2- 172.21.15.78:0/4149511573 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7d81017d0 0x7ff7d8196de0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff7d4009230 tx=0x7ff7d402c650 comp rx=0 tx=0).stop 2024-03-24T00:19:05.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.868+0000 7ff7df8a1640 1 -- 172.21.15.78:0/4149511573 shutdown_connections 2024-03-24T00:19:05.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.868+0000 7ff7df8a1640 1 --2- 172.21.15.78:0/4149511573 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ff7c40a2880 0x7ff7c40a4d40 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:05.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.868+0000 7ff7df8a1640 1 --2- 172.21.15.78:0/4149511573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7d8102190 0x7ff7d819de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:05.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.868+0000 7ff7df8a1640 1 --2- 172.21.15.78:0/4149511573 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7d81017d0 0x7ff7d8196de0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:05.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.868+0000 7ff7df8a1640 1 --2- 172.21.15.78:0/4149511573 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff7d81005e0 0x7ff7d81968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:05.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.868+0000 7ff7df8a1640 1 -- 172.21.15.78:0/4149511573 >> 172.21.15.78:0/4149511573 conn(0x7ff7d80fbb50 msgr2=0x7ff7d8104ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:05.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.868+0000 7ff7df8a1640 1 -- 172.21.15.78:0/4149511573 shutdown_connections 2024-03-24T00:19:05.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.868+0000 7ff7df8a1640 1 -- 172.21.15.78:0/4149511573 wait complete. 2024-03-24T00:19:05.880 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-24T00:19:05.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.984+0000 7fd3036c6640 1 -- 172.21.15.78:0/3912017545 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2fc103310 msgr2=0x7fd2fc109ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:05.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.984+0000 7fd3036c6640 1 --2- 172.21.15.78:0/3912017545 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2fc103310 0x7fd2fc109ba0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4009cd0 tx=0x7fd2f402c0f0 comp rx=0 tx=0).stop 2024-03-24T00:19:05.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.984+0000 7fd3036c6640 1 -- 172.21.15.78:0/3912017545 shutdown_connections 2024-03-24T00:19:05.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.984+0000 7fd3036c6640 1 --2- 172.21.15.78:0/3912017545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2fc102950 0x7fd2fc102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:05.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.984+0000 7fd3036c6640 1 --2- 172.21.15.78:0/3912017545 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2fc103310 0x7fd2fc109ba0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:05.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.984+0000 7fd3036c6640 1 --2- 172.21.15.78:0/3912017545 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2fc0ff980 0x7fd2fc0ffda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:05.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.984+0000 7fd3036c6640 1 -- 172.21.15.78:0/3912017545 >> 172.21.15.78:0/3912017545 conn(0x7fd2fc0fb2b0 msgr2=0x7fd2fc0fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:05.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.984+0000 7fd3036c6640 1 -- 172.21.15.78:0/3912017545 shutdown_connections 2024-03-24T00:19:05.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.984+0000 7fd3036c6640 1 -- 172.21.15.78:0/3912017545 wait complete. 2024-03-24T00:19:05.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.984+0000 7fd3036c6640 1 Processor -- start 2024-03-24T00:19:05.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.984+0000 7fd3036c6640 1 -- start start 2024-03-24T00:19:05.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.985+0000 7fd3036c6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2fc0ff980 0x7fd2fc1022c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:05.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.985+0000 7fd3036c6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2fc102950 0x7fd2fc100900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:05.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.985+0000 7fd3036c6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2fc103310 0x7fd2fc100e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:05.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.985+0000 7fd3036c6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc10d360 con 0x7fd2fc102950 2024-03-24T00:19:05.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.985+0000 7fd3036c6640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc10d1e0 con 0x7fd2fc103310 2024-03-24T00:19:05.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.985+0000 7fd3036c6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc10d4e0 con 0x7fd2fc0ff980 2024-03-24T00:19:05.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.985+0000 7fd300c61640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2fc102950 0x7fd2fc100900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:05.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.985+0000 7fd300c61640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2fc102950 0x7fd2fc100900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58188/0 (socket says 172.21.15.78:58188) 2024-03-24T00:19:05.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.985+0000 7fd300c61640 1 -- 172.21.15.78:0/2648084952 learned_addr learned my addr 172.21.15.78:0/2648084952 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:19:05.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.985+0000 7fd301462640 1 --2- 172.21.15.78:0/2648084952 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2fc0ff980 0x7fd2fc1022c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:05.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.985+0000 7fd301c63640 1 --2- 172.21.15.78:0/2648084952 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2fc103310 0x7fd2fc100e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:05.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.985+0000 7fd300c61640 1 -- 172.21.15.78:0/2648084952 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2fc0ff980 msgr2=0x7fd2fc1022c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:05.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.985+0000 7fd300c61640 1 --2- 172.21.15.78:0/2648084952 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2fc0ff980 0x7fd2fc1022c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:05.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.985+0000 7fd300c61640 1 -- 172.21.15.78:0/2648084952 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2fc103310 msgr2=0x7fd2fc100e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:05.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.985+0000 7fd300c61640 1 --2- 172.21.15.78:0/2648084952 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2fc103310 0x7fd2fc100e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:05.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.985+0000 7fd300c61640 1 -- 172.21.15.78:0/2648084952 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2fc1016d0 con 0x7fd2fc102950 2024-03-24T00:19:05.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.985+0000 7fd300c61640 1 --2- 172.21.15.78:0/2648084952 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2fc102950 0x7fd2fc100900 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec00d770 tx=0x7fd2ec00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:05.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.985+0000 7fd2f27fc640 1 -- 172.21.15.78:0/2648084952 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec017070 con 0x7fd2fc102950 2024-03-24T00:19:05.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.985+0000 7fd2f27fc640 1 -- 172.21.15.78:0/2648084952 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd2ec005500 con 0x7fd2fc102950 2024-03-24T00:19:05.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.985+0000 7fd3036c6640 1 -- 172.21.15.78:0/2648084952 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2fc1a1a90 con 0x7fd2fc102950 2024-03-24T00:19:05.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.985+0000 7fd2f27fc640 1 -- 172.21.15.78:0/2648084952 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec004e00 con 0x7fd2fc102950 2024-03-24T00:19:05.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.985+0000 7fd3036c6640 1 -- 172.21.15.78:0/2648084952 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2fc1a1f90 con 0x7fd2fc102950 2024-03-24T00:19:05.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.986+0000 7fd3036c6640 1 -- 172.21.15.78:0/2648084952 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2fc108a20 con 0x7fd2fc102950 2024-03-24T00:19:05.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.987+0000 7fd2f27fc640 1 -- 172.21.15.78:0/2648084952 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd2ec005000 con 0x7fd2fc102950 2024-03-24T00:19:05.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.988+0000 7fd2f27fc640 1 --2- 172.21.15.78:0/2648084952 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd2e80a2800 0x7fd2e80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:05.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.988+0000 7fd301462640 1 --2- 172.21.15.78:0/2648084952 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd2e80a2800 0x7fd2e80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:05.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.988+0000 7fd2f27fc640 1 -- 172.21.15.78:0/2648084952 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fd2ec0ce630 con 0x7fd2fc102950 2024-03-24T00:19:05.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.988+0000 7fd301462640 1 --2- 172.21.15.78:0/2648084952 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd2e80a2800 0x7fd2e80a4cc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc069b50 tx=0x7fd2f8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:05.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:05.992+0000 7fd2f27fc640 1 -- 172.21.15.78:0/2648084952 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd2ec09ae40 con 0x7fd2fc102950 2024-03-24T00:19:06.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.202+0000 7fd3036c6640 1 -- 172.21.15.78:0/2648084952 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fd2fc0622a0 con 0x7fd2fc102950 2024-03-24T00:19:06.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.359+0000 7fd2f27fc640 1 -- 172.21.15.78:0/2648084952 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd2ec09eee0 con 0x7fd2fc102950 2024-03-24T00:19:06.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.362+0000 7fd3036c6640 1 -- 172.21.15.78:0/2648084952 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd2e80a2800 msgr2=0x7fd2e80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:06.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.362+0000 7fd3036c6640 1 --2- 172.21.15.78:0/2648084952 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd2e80a2800 0x7fd2e80a4cc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc069b50 tx=0x7fd2f8005f90 comp rx=0 tx=0).stop 2024-03-24T00:19:06.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.362+0000 7fd3036c6640 1 -- 172.21.15.78:0/2648084952 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2fc102950 msgr2=0x7fd2fc100900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:06.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.362+0000 7fd3036c6640 1 --2- 172.21.15.78:0/2648084952 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2fc102950 0x7fd2fc100900 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec00d770 tx=0x7fd2ec00dc30 comp rx=0 tx=0).stop 2024-03-24T00:19:06.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.362+0000 7fd3036c6640 1 -- 172.21.15.78:0/2648084952 shutdown_connections 2024-03-24T00:19:06.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.362+0000 7fd3036c6640 1 --2- 172.21.15.78:0/2648084952 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd2e80a2800 0x7fd2e80a4cc0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:06.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.362+0000 7fd3036c6640 1 --2- 172.21.15.78:0/2648084952 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2fc103310 0x7fd2fc100e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:06.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.362+0000 7fd3036c6640 1 --2- 172.21.15.78:0/2648084952 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2fc102950 0x7fd2fc100900 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:06.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.362+0000 7fd3036c6640 1 --2- 172.21.15.78:0/2648084952 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2fc0ff980 0x7fd2fc1022c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:06.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.362+0000 7fd3036c6640 1 -- 172.21.15.78:0/2648084952 >> 172.21.15.78:0/2648084952 conn(0x7fd2fc0fb2b0 msgr2=0x7fd2fc105bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:06.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.362+0000 7fd3036c6640 1 -- 172.21.15.78:0/2648084952 shutdown_connections 2024-03-24T00:19:06.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.362+0000 7fd3036c6640 1 -- 172.21.15.78:0/2648084952 wait complete. 2024-03-24T00:19:06.373 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-03-24T00:19:06.373 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-24T00:19:06.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.477+0000 7fae12881640 1 -- 172.21.15.78:0/1853460350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae0c104ab0 msgr2=0x7fae0c106ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:06.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.477+0000 7fae12881640 1 --2- 172.21.15.78:0/1853460350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae0c104ab0 0x7fae0c106ea0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fae00009900 tx=0x7fae0002bc30 comp rx=0 tx=0).stop 2024-03-24T00:19:06.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.477+0000 7fae12881640 1 -- 172.21.15.78:0/1853460350 shutdown_connections 2024-03-24T00:19:06.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.477+0000 7fae12881640 1 --2- 172.21.15.78:0/1853460350 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae0c1073e0 0x7fae0c1097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:06.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.477+0000 7fae12881640 1 --2- 172.21.15.78:0/1853460350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae0c104ab0 0x7fae0c106ea0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:06.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.477+0000 7fae12881640 1 --2- 172.21.15.78:0/1853460350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae0c102180 0x7fae0c104570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:06.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.477+0000 7fae12881640 1 -- 172.21.15.78:0/1853460350 >> 172.21.15.78:0/1853460350 conn(0x7fae0c0fbb10 msgr2=0x7fae0c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:06.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.477+0000 7fae12881640 1 -- 172.21.15.78:0/1853460350 shutdown_connections 2024-03-24T00:19:06.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.477+0000 7fae12881640 1 -- 172.21.15.78:0/1853460350 wait complete. 2024-03-24T00:19:06.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.477+0000 7fae12881640 1 Processor -- start 2024-03-24T00:19:06.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.477+0000 7fae12881640 1 -- start start 2024-03-24T00:19:06.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.478+0000 7fae12881640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae0c102180 0x7fae0c197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:06.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.478+0000 7fae12881640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae0c104ab0 0x7fae0c197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:06.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.478+0000 7fae12881640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae0c1073e0 0x7fae0c19eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:06.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.478+0000 7fae12881640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0c10d370 con 0x7fae0c104ab0 2024-03-24T00:19:06.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.478+0000 7fae12881640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0c10d1f0 con 0x7fae0c1073e0 2024-03-24T00:19:06.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.478+0000 7fae12881640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0c10d4f0 con 0x7fae0c102180 2024-03-24T00:19:06.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.478+0000 7fae0b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae0c104ab0 0x7fae0c197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:06.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.478+0000 7fae0b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae0c104ab0 0x7fae0c197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58208/0 (socket says 172.21.15.78:58208) 2024-03-24T00:19:06.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.478+0000 7fae0b7fe640 1 -- 172.21.15.78:0/3014717271 learned_addr learned my addr 172.21.15.78:0/3014717271 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:19:06.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.478+0000 7fae10e1e640 1 --2- 172.21.15.78:0/3014717271 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae0c1073e0 0x7fae0c19eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:06.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.478+0000 7fae0bfff640 1 --2- 172.21.15.78:0/3014717271 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae0c102180 0x7fae0c197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:06.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.478+0000 7fae0b7fe640 1 -- 172.21.15.78:0/3014717271 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae0c102180 msgr2=0x7fae0c197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:06.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.478+0000 7fae0b7fe640 1 --2- 172.21.15.78:0/3014717271 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae0c102180 0x7fae0c197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:06.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.478+0000 7fae0b7fe640 1 -- 172.21.15.78:0/3014717271 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae0c1073e0 msgr2=0x7fae0c19eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:06.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.478+0000 7fae0b7fe640 1 --2- 172.21.15.78:0/3014717271 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae0c1073e0 0x7fae0c19eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:06.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.478+0000 7fae0b7fe640 1 -- 172.21.15.78:0/3014717271 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae0c19f660 con 0x7fae0c104ab0 2024-03-24T00:19:06.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.478+0000 7fae0b7fe640 1 --2- 172.21.15.78:0/3014717271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae0c104ab0 0x7fae0c197f80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fae00000c00 tx=0x7fae00005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:06.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.478+0000 7fae097fa640 1 -- 172.21.15.78:0/3014717271 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae00041070 con 0x7fae0c104ab0 2024-03-24T00:19:06.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.478+0000 7fae097fa640 1 -- 172.21.15.78:0/3014717271 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fae000044d0 con 0x7fae0c104ab0 2024-03-24T00:19:06.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.478+0000 7fae12881640 1 -- 172.21.15.78:0/3014717271 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae0c19f8f0 con 0x7fae0c104ab0 2024-03-24T00:19:06.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.478+0000 7fae097fa640 1 -- 172.21.15.78:0/3014717271 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae00034440 con 0x7fae0c104ab0 2024-03-24T00:19:06.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.479+0000 7fae12881640 1 -- 172.21.15.78:0/3014717271 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae0c19fde0 con 0x7fae0c104ab0 2024-03-24T00:19:06.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.480+0000 7fae12881640 1 -- 172.21.15.78:0/3014717271 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae0c0fd600 con 0x7fae0c104ab0 2024-03-24T00:19:06.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.481+0000 7fae097fa640 1 -- 172.21.15.78:0/3014717271 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fae00044070 con 0x7fae0c104ab0 2024-03-24T00:19:06.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.481+0000 7fae097fa640 1 --2- 172.21.15.78:0/3014717271 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fadf40a2800 0x7fadf40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:06.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.481+0000 7fae0bfff640 1 --2- 172.21.15.78:0/3014717271 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fadf40a2800 0x7fadf40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:06.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.481+0000 7fae097fa640 1 -- 172.21.15.78:0/3014717271 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fae000f08d0 con 0x7fae0c104ab0 2024-03-24T00:19:06.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.482+0000 7fae0bfff640 1 --2- 172.21.15.78:0/3014717271 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fadf40a2800 0x7fadf40a4cc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fadfc009890 tx=0x7fadfc008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:06.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.486+0000 7fae097fa640 1 -- 172.21.15.78:0/3014717271 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fae000bd730 con 0x7fae0c104ab0 2024-03-24T00:19:06.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:06 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2659051951' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:19:06.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:06 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/4149511573' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:19:06.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:06 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2648084952' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-24T00:19:06.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:06.698+0000 7fae12881640 1 -- 172.21.15.78:0/3014717271 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7fae0c0622a0 con 0x7fae0c104ab0 2024-03-24T00:19:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:06 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2659051951' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:19:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:06 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/4149511573' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:19:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:06 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2648084952' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-24T00:19:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:06 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2659051951' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:19:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:06 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/4149511573' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:19:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:06 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2648084952' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-24T00:19:07.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.361+0000 7fae097fa640 1 -- 172.21.15.78:0/3014717271 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) v1 ==== 105+0+0 (secure 0 0 0) 0x7fae000bd030 con 0x7fae0c104ab0 2024-03-24T00:19:07.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.363+0000 7fae12881640 1 -- 172.21.15.78:0/3014717271 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fadf40a2800 msgr2=0x7fadf40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:07.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.363+0000 7fae12881640 1 --2- 172.21.15.78:0/3014717271 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fadf40a2800 0x7fadf40a4cc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fadfc009890 tx=0x7fadfc008040 comp rx=0 tx=0).stop 2024-03-24T00:19:07.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.363+0000 7fae12881640 1 -- 172.21.15.78:0/3014717271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae0c104ab0 msgr2=0x7fae0c197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:07.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.363+0000 7fae12881640 1 --2- 172.21.15.78:0/3014717271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae0c104ab0 0x7fae0c197f80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fae00000c00 tx=0x7fae00005d50 comp rx=0 tx=0).stop 2024-03-24T00:19:07.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.364+0000 7fae12881640 1 -- 172.21.15.78:0/3014717271 shutdown_connections 2024-03-24T00:19:07.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.364+0000 7fae12881640 1 --2- 172.21.15.78:0/3014717271 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fadf40a2800 0x7fadf40a4cc0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:07.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.364+0000 7fae12881640 1 --2- 172.21.15.78:0/3014717271 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae0c1073e0 0x7fae0c19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:07.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.364+0000 7fae12881640 1 --2- 172.21.15.78:0/3014717271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae0c104ab0 0x7fae0c197f80 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:07.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.364+0000 7fae12881640 1 --2- 172.21.15.78:0/3014717271 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae0c102180 0x7fae0c197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:07.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.364+0000 7fae12881640 1 -- 172.21.15.78:0/3014717271 >> 172.21.15.78:0/3014717271 conn(0x7fae0c0fbb10 msgr2=0x7fae0c102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:07.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.364+0000 7fae12881640 1 -- 172.21.15.78:0/3014717271 shutdown_connections 2024-03-24T00:19:07.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.364+0000 7fae12881640 1 -- 172.21.15.78:0/3014717271 wait complete. 2024-03-24T00:19:07.379 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-24T00:19:07.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.485+0000 7fb7d04c0640 1 -- 172.21.15.78:0/1525135702 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7c81005a0 msgr2=0x7fb7c81009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:07.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.485+0000 7fb7d04c0640 1 --2- 172.21.15.78:0/1525135702 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7c81005a0 0x7fb7c81009c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4009900 tx=0x7fb7c402bc30 comp rx=0 tx=0).stop 2024-03-24T00:19:07.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.486+0000 7fb7d04c0640 1 -- 172.21.15.78:0/1525135702 shutdown_connections 2024-03-24T00:19:07.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.486+0000 7fb7d04c0640 1 --2- 172.21.15.78:0/1525135702 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7c8101790 0x7fb7c8101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:07.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.486+0000 7fb7d04c0640 1 --2- 172.21.15.78:0/1525135702 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7c8102150 0x7fb7c81089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:07.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.486+0000 7fb7d04c0640 1 --2- 172.21.15.78:0/1525135702 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7c81005a0 0x7fb7c81009c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:07.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.486+0000 7fb7d04c0640 1 -- 172.21.15.78:0/1525135702 >> 172.21.15.78:0/1525135702 conn(0x7fb7c80fbb10 msgr2=0x7fb7c80fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:07.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.486+0000 7fb7d04c0640 1 -- 172.21.15.78:0/1525135702 shutdown_connections 2024-03-24T00:19:07.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.486+0000 7fb7d04c0640 1 -- 172.21.15.78:0/1525135702 wait complete. 2024-03-24T00:19:07.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.486+0000 7fb7d04c0640 1 Processor -- start 2024-03-24T00:19:07.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.486+0000 7fb7d04c0640 1 -- start start 2024-03-24T00:19:07.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.486+0000 7fb7d04c0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7c81005a0 0x7fb7c8069390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:07.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.486+0000 7fb7d04c0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7c8101790 0x7fb7c819c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:07.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.486+0000 7fb7d04c0640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7c8102150 0x7fb7c819ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:07.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.486+0000 7fb7d04c0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c810c230 con 0x7fb7c8101790 2024-03-24T00:19:07.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.486+0000 7fb7d04c0640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c810c0b0 con 0x7fb7c8102150 2024-03-24T00:19:07.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.486+0000 7fb7d04c0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c810c3b0 con 0x7fb7c81005a0 2024-03-24T00:19:07.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.486+0000 7fb7cda5b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7c8101790 0x7fb7c819c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:07.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.486+0000 7fb7cda5b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7c8101790 0x7fb7c819c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58234/0 (socket says 172.21.15.78:58234) 2024-03-24T00:19:07.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.486+0000 7fb7cda5b640 1 -- 172.21.15.78:0/4176436863 learned_addr learned my addr 172.21.15.78:0/4176436863 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:19:07.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.486+0000 7fb7cea5d640 1 --2- 172.21.15.78:0/4176436863 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7c8102150 0x7fb7c819ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:07.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.487+0000 7fb7ce25c640 1 --2- 172.21.15.78:0/4176436863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7c81005a0 0x7fb7c8069390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:07.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.487+0000 7fb7cda5b640 1 -- 172.21.15.78:0/4176436863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7c81005a0 msgr2=0x7fb7c8069390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:07.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.487+0000 7fb7cda5b640 1 --2- 172.21.15.78:0/4176436863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7c81005a0 0x7fb7c8069390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:07.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.487+0000 7fb7cda5b640 1 -- 172.21.15.78:0/4176436863 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7c8102150 msgr2=0x7fb7c819ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:07.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.487+0000 7fb7cda5b640 1 --2- 172.21.15.78:0/4176436863 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7c8102150 0x7fb7c819ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:07.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.487+0000 7fb7cda5b640 1 -- 172.21.15.78:0/4176436863 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c819f600 con 0x7fb7c8101790 2024-03-24T00:19:07.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.487+0000 7fb7cda5b640 1 --2- 172.21.15.78:0/4176436863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7c8101790 0x7fb7c819c700 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00d770 tx=0x7fb7bc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:07.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.487+0000 7fb7bb7fe640 1 -- 172.21.15.78:0/4176436863 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc017070 con 0x7fb7c8101790 2024-03-24T00:19:07.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.487+0000 7fb7d04c0640 1 -- 172.21.15.78:0/4176436863 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7c819f8f0 con 0x7fb7c8101790 2024-03-24T00:19:07.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.487+0000 7fb7bb7fe640 1 -- 172.21.15.78:0/4176436863 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7bc005500 con 0x7fb7c8101790 2024-03-24T00:19:07.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.487+0000 7fb7bb7fe640 1 -- 172.21.15.78:0/4176436863 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc004e00 con 0x7fb7c8101790 2024-03-24T00:19:07.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.487+0000 7fb7d04c0640 1 -- 172.21.15.78:0/4176436863 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7c819fe40 con 0x7fb7c8101790 2024-03-24T00:19:07.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.488+0000 7fb7d04c0640 1 -- 172.21.15.78:0/4176436863 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7c8107990 con 0x7fb7c8101790 2024-03-24T00:19:07.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.489+0000 7fb7bb7fe640 1 -- 172.21.15.78:0/4176436863 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb7bc004fa0 con 0x7fb7c8101790 2024-03-24T00:19:07.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.489+0000 7fb7bb7fe640 1 --2- 172.21.15.78:0/4176436863 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fb7b40a2800 0x7fb7b40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:07.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.490+0000 7fb7bb7fe640 1 -- 172.21.15.78:0/4176436863 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb7bc0ce6d0 con 0x7fb7c8101790 2024-03-24T00:19:07.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.490+0000 7fb7ce25c640 1 --2- 172.21.15.78:0/4176436863 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fb7b40a2800 0x7fb7b40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:07.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.490+0000 7fb7ce25c640 1 --2- 172.21.15.78:0/4176436863 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fb7b40a2800 0x7fb7b40a4cc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4036040 tx=0x7fb7c4005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:07.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.494+0000 7fb7bb7fe640 1 -- 172.21.15.78:0/4176436863 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb7bc09b530 con 0x7fb7c8101790 2024-03-24T00:19:07.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:07 smithi106 ceph-mon[36505]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-24T00:19:07.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:07 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2648084952' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-24T00:19:07.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:07 smithi106 ceph-mon[36505]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:19:07.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:07 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3014717271' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-24T00:19:07.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:07.704+0000 7fb7d04c0640 1 -- 172.21.15.78:0/4176436863 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7fb7c80622a0 con 0x7fb7c8101790 2024-03-24T00:19:07.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:07 smithi078 ceph-mon[32429]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-24T00:19:07.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:07 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2648084952' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-24T00:19:07.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:07 smithi078 ceph-mon[32429]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:19:07.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:07 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3014717271' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-24T00:19:07.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:07 smithi107 ceph-mon[36485]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-24T00:19:07.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:07 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2648084952' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-24T00:19:07.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:07 smithi107 ceph-mon[36485]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:19:07.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:07 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3014717271' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-24T00:19:08.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.373+0000 7fb7bb7fe640 1 -- 172.21.15.78:0/4176436863 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) v1 ==== 121+0+0 (secure 0 0 0) 0x7fb7bc09ae30 con 0x7fb7c8101790 2024-03-24T00:19:08.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.375+0000 7fb7d04c0640 1 -- 172.21.15.78:0/4176436863 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fb7b40a2800 msgr2=0x7fb7b40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:08.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.375+0000 7fb7d04c0640 1 --2- 172.21.15.78:0/4176436863 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fb7b40a2800 0x7fb7b40a4cc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4036040 tx=0x7fb7c4005fd0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.375+0000 7fb7d04c0640 1 -- 172.21.15.78:0/4176436863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7c8101790 msgr2=0x7fb7c819c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:08.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.375+0000 7fb7d04c0640 1 --2- 172.21.15.78:0/4176436863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7c8101790 0x7fb7c819c700 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00d770 tx=0x7fb7bc00dc30 comp rx=0 tx=0).stop 2024-03-24T00:19:08.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.376+0000 7fb7d04c0640 1 -- 172.21.15.78:0/4176436863 shutdown_connections 2024-03-24T00:19:08.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.376+0000 7fb7d04c0640 1 --2- 172.21.15.78:0/4176436863 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fb7b40a2800 0x7fb7b40a4cc0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.376+0000 7fb7d04c0640 1 --2- 172.21.15.78:0/4176436863 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7c8102150 0x7fb7c819ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.376+0000 7fb7d04c0640 1 --2- 172.21.15.78:0/4176436863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7c8101790 0x7fb7c819c700 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.376+0000 7fb7d04c0640 1 --2- 172.21.15.78:0/4176436863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7c81005a0 0x7fb7c8069390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.376+0000 7fb7d04c0640 1 -- 172.21.15.78:0/4176436863 >> 172.21.15.78:0/4176436863 conn(0x7fb7c80fbb10 msgr2=0x7fb7c8104b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:08.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.376+0000 7fb7d04c0640 1 -- 172.21.15.78:0/4176436863 shutdown_connections 2024-03-24T00:19:08.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.376+0000 7fb7d04c0640 1 -- 172.21.15.78:0/4176436863 wait complete. 2024-03-24T00:19:08.390 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-24T00:19:08.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.494+0000 7fe4de7b2640 1 -- 172.21.15.78:0/3565683191 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4d81017b0 msgr2=0x7fe4d8101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:08.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.494+0000 7fe4de7b2640 1 --2- 172.21.15.78:0/3565683191 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4d81017b0 0x7fe4d8101c30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc009900 tx=0x7fe4cc02bc30 comp rx=0 tx=0).stop 2024-03-24T00:19:08.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.494+0000 7fe4de7b2640 1 -- 172.21.15.78:0/3565683191 shutdown_connections 2024-03-24T00:19:08.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.494+0000 7fe4de7b2640 1 --2- 172.21.15.78:0/3565683191 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4d8102170 0x7fe4d8108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.494+0000 7fe4de7b2640 1 --2- 172.21.15.78:0/3565683191 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4d81017b0 0x7fe4d8101c30 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.494+0000 7fe4de7b2640 1 --2- 172.21.15.78:0/3565683191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4d81005c0 0x7fe4d81009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.494+0000 7fe4de7b2640 1 -- 172.21.15.78:0/3565683191 >> 172.21.15.78:0/3565683191 conn(0x7fe4d80fbb30 msgr2=0x7fe4d80fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:08.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.494+0000 7fe4de7b2640 1 -- 172.21.15.78:0/3565683191 shutdown_connections 2024-03-24T00:19:08.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.494+0000 7fe4de7b2640 1 -- 172.21.15.78:0/3565683191 wait complete. 2024-03-24T00:19:08.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.494+0000 7fe4de7b2640 1 Processor -- start 2024-03-24T00:19:08.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.495+0000 7fe4de7b2640 1 -- start start 2024-03-24T00:19:08.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.495+0000 7fe4de7b2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4d81005c0 0x7fe4d806a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.495+0000 7fe4de7b2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4d81017b0 0x7fe4d819c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.495+0000 7fe4de7b2640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4d8102170 0x7fe4d819ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.495+0000 7fe4de7b2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d810c240 con 0x7fe4d81017b0 2024-03-24T00:19:08.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.495+0000 7fe4de7b2640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d810c0c0 con 0x7fe4d8102170 2024-03-24T00:19:08.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.495+0000 7fe4de7b2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d810c3c0 con 0x7fe4d81005c0 2024-03-24T00:19:08.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.495+0000 7fe4d77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4d81017b0 0x7fe4d819c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.495+0000 7fe4d77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4d81017b0 0x7fe4d819c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58252/0 (socket says 172.21.15.78:58252) 2024-03-24T00:19:08.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.495+0000 7fe4d7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4d81005c0 0x7fe4d806a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.495+0000 7fe4d77fe640 1 -- 172.21.15.78:0/2344476509 learned_addr learned my addr 172.21.15.78:0/2344476509 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:19:08.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.495+0000 7fe4dcd4f640 1 --2- 172.21.15.78:0/2344476509 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4d8102170 0x7fe4d819ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.495+0000 7fe4d77fe640 1 -- 172.21.15.78:0/2344476509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4d81005c0 msgr2=0x7fe4d806a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:08.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.495+0000 7fe4d77fe640 1 --2- 172.21.15.78:0/2344476509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4d81005c0 0x7fe4d806a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.495+0000 7fe4d77fe640 1 -- 172.21.15.78:0/2344476509 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4d8102170 msgr2=0x7fe4d819ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:08.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.495+0000 7fe4d77fe640 1 --2- 172.21.15.78:0/2344476509 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4d8102170 0x7fe4d819ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.495+0000 7fe4d77fe640 1 -- 172.21.15.78:0/2344476509 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d819f6f0 con 0x7fe4d81017b0 2024-03-24T00:19:08.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.495+0000 7fe4d77fe640 1 --2- 172.21.15.78:0/2344476509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4d81017b0 0x7fe4d819c7c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc005ab0 tx=0x7fe4cc0049c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:08.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.496+0000 7fe4d57fa640 1 -- 172.21.15.78:0/2344476509 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc041070 con 0x7fe4d81017b0 2024-03-24T00:19:08.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.496+0000 7fe4de7b2640 1 -- 172.21.15.78:0/2344476509 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4d819f920 con 0x7fe4d81017b0 2024-03-24T00:19:08.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.496+0000 7fe4d57fa640 1 -- 172.21.15.78:0/2344476509 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe4cc004450 con 0x7fe4d81017b0 2024-03-24T00:19:08.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.496+0000 7fe4d57fa640 1 -- 172.21.15.78:0/2344476509 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc003e50 con 0x7fe4d81017b0 2024-03-24T00:19:08.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.496+0000 7fe4de7b2640 1 -- 172.21.15.78:0/2344476509 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4d819fe60 con 0x7fe4d81017b0 2024-03-24T00:19:08.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.497+0000 7fe4de7b2640 1 -- 172.21.15.78:0/2344476509 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4d8107a00 con 0x7fe4d81017b0 2024-03-24T00:19:08.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.498+0000 7fe4d57fa640 1 -- 172.21.15.78:0/2344476509 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe4cc044070 con 0x7fe4d81017b0 2024-03-24T00:19:08.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.498+0000 7fe4d57fa640 1 --2- 172.21.15.78:0/2344476509 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe4c00a29a0 0x7fe4c00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.498+0000 7fe4d57fa640 1 -- 172.21.15.78:0/2344476509 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fe4cc0f07c0 con 0x7fe4d81017b0 2024-03-24T00:19:08.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.498+0000 7fe4d7fff640 1 --2- 172.21.15.78:0/2344476509 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe4c00a29a0 0x7fe4c00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.499+0000 7fe4d7fff640 1 --2- 172.21.15.78:0/2344476509 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe4c00a29a0 0x7fe4c00a4e60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe4d8101610 tx=0x7fe4c8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:08.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.503+0000 7fe4d57fa640 1 -- 172.21.15.78:0/2344476509 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe4cc0bcfd0 con 0x7fe4d81017b0 2024-03-24T00:19:08.652 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 24 00:19:08 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-b[66985]: 2024-03-24T00:19:08.394+0000 7f978f4ee700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-24T00:19:08.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:08 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-24T00:19:08.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:08 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3014717271' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-24T00:19:08.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:08 smithi106 ceph-mon[36505]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-24T00:19:08.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:08 smithi106 ceph-mon[36505]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:19:08.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:08 smithi106 ceph-mon[36505]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-24T00:19:08.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:08 smithi106 ceph-mon[36505]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-24T00:19:08.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:08 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/4176436863' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-24T00:19:08.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.715+0000 7fe4de7b2640 1 -- 172.21.15.78:0/2344476509 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7fe4d80622a0 con 0x7fe4d81017b0 2024-03-24T00:19:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:08 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-24T00:19:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:08 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3014717271' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-24T00:19:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:08 smithi078 ceph-mon[32429]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-24T00:19:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:08 smithi078 ceph-mon[32429]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:19:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:08 smithi078 ceph-mon[32429]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-24T00:19:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:08 smithi078 ceph-mon[32429]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-24T00:19:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:08 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/4176436863' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-24T00:19:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:08 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-24T00:19:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:08 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3014717271' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-24T00:19:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:08 smithi107 ceph-mon[36485]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-24T00:19:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:08 smithi107 ceph-mon[36485]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:19:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:08 smithi107 ceph-mon[36485]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-24T00:19:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:08 smithi107 ceph-mon[36485]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-24T00:19:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:08 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/4176436863' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-24T00:19:08.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.871+0000 7fe4d57fa640 1 -- 172.21.15.78:0/2344476509 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) v1 ==== 115+0+0 (secure 0 0 0) 0x7fe4cc0c1070 con 0x7fe4d81017b0 2024-03-24T00:19:08.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.873+0000 7fe4de7b2640 1 -- 172.21.15.78:0/2344476509 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe4c00a29a0 msgr2=0x7fe4c00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:08.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.873+0000 7fe4de7b2640 1 --2- 172.21.15.78:0/2344476509 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe4c00a29a0 0x7fe4c00a4e60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe4d8101610 tx=0x7fe4c8005f90 comp rx=0 tx=0).stop 2024-03-24T00:19:08.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.873+0000 7fe4de7b2640 1 -- 172.21.15.78:0/2344476509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4d81017b0 msgr2=0x7fe4d819c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:08.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.873+0000 7fe4de7b2640 1 --2- 172.21.15.78:0/2344476509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4d81017b0 0x7fe4d819c7c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc005ab0 tx=0x7fe4cc0049c0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.873+0000 7fe4de7b2640 1 -- 172.21.15.78:0/2344476509 shutdown_connections 2024-03-24T00:19:08.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.873+0000 7fe4de7b2640 1 --2- 172.21.15.78:0/2344476509 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe4c00a29a0 0x7fe4c00a4e60 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.873+0000 7fe4de7b2640 1 --2- 172.21.15.78:0/2344476509 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4d8102170 0x7fe4d819ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.873+0000 7fe4de7b2640 1 --2- 172.21.15.78:0/2344476509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4d81017b0 0x7fe4d819c7c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.873+0000 7fe4de7b2640 1 --2- 172.21.15.78:0/2344476509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4d81005c0 0x7fe4d806a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.873+0000 7fe4de7b2640 1 -- 172.21.15.78:0/2344476509 >> 172.21.15.78:0/2344476509 conn(0x7fe4d80fbb30 msgr2=0x7fe4d8104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:08.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.873+0000 7fe4de7b2640 1 -- 172.21.15.78:0/2344476509 shutdown_connections 2024-03-24T00:19:08.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.873+0000 7fe4de7b2640 1 -- 172.21.15.78:0/2344476509 wait complete. 2024-03-24T00:19:08.886 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-03-24T00:19:08.886 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-24T00:19:08.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.990+0000 7f19d0d89640 1 -- 172.21.15.78:0/1966014815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19cc102150 msgr2=0x7f19cc1089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:08.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.990+0000 7f19d0d89640 1 --2- 172.21.15.78:0/1966014815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19cc102150 0x7f19cc1089e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f19bc009900 tx=0x7f19bc02bc30 comp rx=0 tx=0).stop 2024-03-24T00:19:08.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.990+0000 7f19d0d89640 1 -- 172.21.15.78:0/1966014815 shutdown_connections 2024-03-24T00:19:08.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.990+0000 7f19d0d89640 1 --2- 172.21.15.78:0/1966014815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19cc102150 0x7f19cc1089e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.990+0000 7f19d0d89640 1 --2- 172.21.15.78:0/1966014815 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19cc101790 0x7f19cc101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.990+0000 7f19d0d89640 1 --2- 172.21.15.78:0/1966014815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19cc1005a0 0x7f19cc1009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.990+0000 7f19d0d89640 1 -- 172.21.15.78:0/1966014815 >> 172.21.15.78:0/1966014815 conn(0x7f19cc0fbb30 msgr2=0x7f19cc0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:08.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.990+0000 7f19d0d89640 1 -- 172.21.15.78:0/1966014815 shutdown_connections 2024-03-24T00:19:08.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.990+0000 7f19d0d89640 1 -- 172.21.15.78:0/1966014815 wait complete. 2024-03-24T00:19:08.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.991+0000 7f19d0d89640 1 Processor -- start 2024-03-24T00:19:08.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.991+0000 7f19d0d89640 1 -- start start 2024-03-24T00:19:08.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.991+0000 7f19d0d89640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19cc1005a0 0x7f19cc069370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.991+0000 7f19d0d89640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19cc101790 0x7f19cc19c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.991+0000 7f19d0d89640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19cc102150 0x7f19cc19ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.991+0000 7f19d0d89640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19cc10c230 con 0x7f19cc102150 2024-03-24T00:19:08.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.991+0000 7f19d0d89640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19cc10c0b0 con 0x7f19cc101790 2024-03-24T00:19:08.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.991+0000 7f19d0d89640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19cc10c3b0 con 0x7f19cc1005a0 2024-03-24T00:19:08.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.991+0000 7f19cad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19cc102150 0x7f19cc19ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.991+0000 7f19cad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19cc102150 0x7f19cc19ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58268/0 (socket says 172.21.15.78:58268) 2024-03-24T00:19:08.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.991+0000 7f19cad9d640 1 -- 172.21.15.78:0/2994029893 learned_addr learned my addr 172.21.15.78:0/2994029893 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:19:08.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.991+0000 7f19c9d9b640 1 --2- 172.21.15.78:0/2994029893 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19cc101790 0x7f19cc19c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.991+0000 7f19ca59c640 1 --2- 172.21.15.78:0/2994029893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19cc1005a0 0x7f19cc069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.992+0000 7f19cad9d640 1 -- 172.21.15.78:0/2994029893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19cc1005a0 msgr2=0x7f19cc069370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:08.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.992+0000 7f19cad9d640 1 --2- 172.21.15.78:0/2994029893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19cc1005a0 0x7f19cc069370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.992+0000 7f19cad9d640 1 -- 172.21.15.78:0/2994029893 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19cc101790 msgr2=0x7f19cc19c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:08.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.992+0000 7f19cad9d640 1 --2- 172.21.15.78:0/2994029893 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19cc101790 0x7f19cc19c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.992+0000 7f19cad9d640 1 -- 172.21.15.78:0/2994029893 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19cc19f630 con 0x7f19cc102150 2024-03-24T00:19:08.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.992+0000 7f19c9d9b640 1 --2- 172.21.15.78:0/2994029893 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19cc101790 0x7f19cc19c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:08.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.992+0000 7f19cad9d640 1 --2- 172.21.15.78:0/2994029893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19cc102150 0x7f19cc19ed20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f19bc005ab0 tx=0x7f19bc02c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:08.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.992+0000 7f19bb7fe640 1 -- 172.21.15.78:0/2994029893 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19bc041070 con 0x7f19cc102150 2024-03-24T00:19:08.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.992+0000 7f19bb7fe640 1 -- 172.21.15.78:0/2994029893 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f19bc004450 con 0x7f19cc102150 2024-03-24T00:19:08.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.992+0000 7f19d0d89640 1 -- 172.21.15.78:0/2994029893 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19cc19f860 con 0x7f19cc102150 2024-03-24T00:19:08.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.992+0000 7f19bb7fe640 1 -- 172.21.15.78:0/2994029893 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19bc003e60 con 0x7f19cc102150 2024-03-24T00:19:08.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.992+0000 7f19d0d89640 1 -- 172.21.15.78:0/2994029893 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19cc19fd50 con 0x7f19cc102150 2024-03-24T00:19:08.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.993+0000 7f19d0d89640 1 -- 172.21.15.78:0/2994029893 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19cc1078c0 con 0x7f19cc102150 2024-03-24T00:19:08.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.994+0000 7f19bb7fe640 1 -- 172.21.15.78:0/2994029893 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f19bc044070 con 0x7f19cc102150 2024-03-24T00:19:08.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.994+0000 7f19bb7fe640 1 --2- 172.21.15.78:0/2994029893 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f19b40a2800 0x7f19b40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.995+0000 7f19ca59c640 1 --2- 172.21.15.78:0/2994029893 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f19b40a2800 0x7f19b40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.995+0000 7f19bb7fe640 1 -- 172.21.15.78:0/2994029893 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f19bc0f08c0 con 0x7f19cc102150 2024-03-24T00:19:08.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.995+0000 7f19ca59c640 1 --2- 172.21.15.78:0/2994029893 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f19b40a2800 0x7f19b40a4cc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f19cc1015f0 tx=0x7f19c4009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:08.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:08.999+0000 7f19bb7fe640 1 -- 172.21.15.78:0/2994029893 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f19bc0bd0d0 con 0x7f19cc102150 2024-03-24T00:19:09.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.199+0000 7f19d0d89640 1 -- 172.21.15.78:0/2994029893 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f19cc061ae0 con 0x7f19b40a2800 2024-03-24T00:19:09.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.219+0000 7f19bb7fe640 1 -- 172.21.15.78:0/2994029893 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f19cc061ae0 con 0x7f19b40a2800 2024-03-24T00:19:09.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.221+0000 7f19d0d89640 1 -- 172.21.15.78:0/2994029893 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f19b40a2800 msgr2=0x7f19b40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:09.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.221+0000 7f19d0d89640 1 --2- 172.21.15.78:0/2994029893 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f19b40a2800 0x7f19b40a4cc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f19cc1015f0 tx=0x7f19c4009500 comp rx=0 tx=0).stop 2024-03-24T00:19:09.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.221+0000 7f19d0d89640 1 -- 172.21.15.78:0/2994029893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19cc102150 msgr2=0x7f19cc19ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:09.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.221+0000 7f19d0d89640 1 --2- 172.21.15.78:0/2994029893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19cc102150 0x7f19cc19ed20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f19bc005ab0 tx=0x7f19bc02c6c0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.221+0000 7f19d0d89640 1 -- 172.21.15.78:0/2994029893 shutdown_connections 2024-03-24T00:19:09.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.221+0000 7f19d0d89640 1 --2- 172.21.15.78:0/2994029893 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f19b40a2800 0x7f19b40a4cc0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.221+0000 7f19d0d89640 1 --2- 172.21.15.78:0/2994029893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19cc102150 0x7f19cc19ed20 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.221+0000 7f19d0d89640 1 --2- 172.21.15.78:0/2994029893 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19cc101790 0x7f19cc19c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.221+0000 7f19d0d89640 1 --2- 172.21.15.78:0/2994029893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19cc1005a0 0x7f19cc069370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.221+0000 7f19d0d89640 1 -- 172.21.15.78:0/2994029893 >> 172.21.15.78:0/2994029893 conn(0x7f19cc0fbb30 msgr2=0x7f19cc104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:09.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.221+0000 7f19d0d89640 1 -- 172.21.15.78:0/2994029893 shutdown_connections 2024-03-24T00:19:09.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.221+0000 7f19d0d89640 1 -- 172.21.15.78:0/2994029893 wait complete. 2024-03-24T00:19:09.232 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-24T00:19:09.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.339+0000 7f52b5f77640 1 -- 172.21.15.78:0/1634201468 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52b0102150 msgr2=0x7f52b01089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:09.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.339+0000 7f52b5f77640 1 --2- 172.21.15.78:0/1634201468 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52b0102150 0x7f52b01089e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f52a8009ca0 tx=0x7f52a802c0f0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.339+0000 7f52b5f77640 1 -- 172.21.15.78:0/1634201468 shutdown_connections 2024-03-24T00:19:09.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.339+0000 7f52b5f77640 1 --2- 172.21.15.78:0/1634201468 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52b0101790 0x7f52b0101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.339+0000 7f52b5f77640 1 --2- 172.21.15.78:0/1634201468 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52b0102150 0x7f52b01089e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.339+0000 7f52b5f77640 1 --2- 172.21.15.78:0/1634201468 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52b01005a0 0x7f52b01009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.339+0000 7f52b5f77640 1 -- 172.21.15.78:0/1634201468 >> 172.21.15.78:0/1634201468 conn(0x7f52b00fbb30 msgr2=0x7f52b00fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:09.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.339+0000 7f52b5f77640 1 -- 172.21.15.78:0/1634201468 shutdown_connections 2024-03-24T00:19:09.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.339+0000 7f52b5f77640 1 -- 172.21.15.78:0/1634201468 wait complete. 2024-03-24T00:19:09.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.340+0000 7f52b5f77640 1 Processor -- start 2024-03-24T00:19:09.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.340+0000 7f52b5f77640 1 -- start start 2024-03-24T00:19:09.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.340+0000 7f52b5f77640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52b01005a0 0x7f52b006a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.340+0000 7f52b5f77640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52b0101790 0x7f52b019c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.340+0000 7f52b5f77640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52b0102150 0x7f52b019ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.340+0000 7f52b5f77640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b010c230 con 0x7f52b0101790 2024-03-24T00:19:09.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.340+0000 7f52b5f77640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b010c0b0 con 0x7f52b0102150 2024-03-24T00:19:09.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.340+0000 7f52b5f77640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b010c3b0 con 0x7f52b01005a0 2024-03-24T00:19:09.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.340+0000 7f52aeffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52b0101790 0x7f52b019c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.340+0000 7f52aeffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52b0101790 0x7f52b019c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58322/0 (socket says 172.21.15.78:58322) 2024-03-24T00:19:09.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.340+0000 7f52aeffd640 1 -- 172.21.15.78:0/389256695 learned_addr learned my addr 172.21.15.78:0/389256695 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:19:09.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.340+0000 7f52af7fe640 1 --2- 172.21.15.78:0/389256695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52b01005a0 0x7f52b006a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.340+0000 7f52affff640 1 --2- 172.21.15.78:0/389256695 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52b0102150 0x7f52b019ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.341+0000 7f52af7fe640 1 -- 172.21.15.78:0/389256695 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52b0102150 msgr2=0x7f52b019ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:09.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.341+0000 7f52af7fe640 1 --2- 172.21.15.78:0/389256695 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52b0102150 0x7f52b019ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.341+0000 7f52af7fe640 1 -- 172.21.15.78:0/389256695 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52b0101790 msgr2=0x7f52b019c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:09.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.341+0000 7f52af7fe640 1 --2- 172.21.15.78:0/389256695 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52b0101790 0x7f52b019c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.341+0000 7f52af7fe640 1 -- 172.21.15.78:0/389256695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b019f580 con 0x7f52b01005a0 2024-03-24T00:19:09.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.341+0000 7f52aeffd640 1 --2- 172.21.15.78:0/389256695 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52b0101790 0x7f52b019c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:09.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.341+0000 7f52af7fe640 1 --2- 172.21.15.78:0/389256695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52b01005a0 0x7f52b006a8a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f52a000e9c0 tx=0x7f52a000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:09.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.341+0000 7f52acff9640 1 -- 172.21.15.78:0/389256695 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a000cd80 con 0x7f52b01005a0 2024-03-24T00:19:09.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.341+0000 7f52b5f77640 1 -- 172.21.15.78:0/389256695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52b019f870 con 0x7f52b01005a0 2024-03-24T00:19:09.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.341+0000 7f52acff9640 1 -- 172.21.15.78:0/389256695 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52a0010910 con 0x7f52b01005a0 2024-03-24T00:19:09.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.341+0000 7f52acff9640 1 -- 172.21.15.78:0/389256695 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a0018980 con 0x7f52b01005a0 2024-03-24T00:19:09.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.341+0000 7f52b5f77640 1 -- 172.21.15.78:0/389256695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52b019fdc0 con 0x7f52b01005a0 2024-03-24T00:19:09.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.342+0000 7f52b5f77640 1 -- 172.21.15.78:0/389256695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52b0107860 con 0x7f52b01005a0 2024-03-24T00:19:09.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.343+0000 7f52acff9640 1 -- 172.21.15.78:0/389256695 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f52a001f020 con 0x7f52b01005a0 2024-03-24T00:19:09.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.344+0000 7f52acff9640 1 --2- 172.21.15.78:0/389256695 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f52980a2800 0x7f52980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.344+0000 7f52aeffd640 1 --2- 172.21.15.78:0/389256695 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f52980a2800 0x7f52980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.344+0000 7f52acff9640 1 -- 172.21.15.78:0/389256695 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f52a00cedb0 con 0x7f52b01005a0 2024-03-24T00:19:09.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.344+0000 7f52aeffd640 1 --2- 172.21.15.78:0/389256695 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f52980a2800 0x7f52980a4cc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f52b006a610 tx=0x7f52a4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:09.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.347+0000 7f52acff9640 1 -- 172.21.15.78:0/389256695 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f52a009bc10 con 0x7f52b01005a0 2024-03-24T00:19:09.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.551+0000 7f52b5f77640 1 -- 172.21.15.78:0/389256695 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f52b0061b70 con 0x7f52980a2800 2024-03-24T00:19:09.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.571+0000 7f52acff9640 1 -- 172.21.15.78:0/389256695 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f52b0061b70 con 0x7f52980a2800 2024-03-24T00:19:09.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.574+0000 7f52b5f77640 1 -- 172.21.15.78:0/389256695 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f52980a2800 msgr2=0x7f52980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:09.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.574+0000 7f52b5f77640 1 --2- 172.21.15.78:0/389256695 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f52980a2800 0x7f52980a4cc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f52b006a610 tx=0x7f52a4005f90 comp rx=0 tx=0).stop 2024-03-24T00:19:09.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.574+0000 7f52b5f77640 1 -- 172.21.15.78:0/389256695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52b01005a0 msgr2=0x7f52b006a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:09.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.574+0000 7f52b5f77640 1 --2- 172.21.15.78:0/389256695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52b01005a0 0x7f52b006a8a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f52a000e9c0 tx=0x7f52a000ee80 comp rx=0 tx=0).stop 2024-03-24T00:19:09.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.574+0000 7f52b5f77640 1 -- 172.21.15.78:0/389256695 shutdown_connections 2024-03-24T00:19:09.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.574+0000 7f52b5f77640 1 --2- 172.21.15.78:0/389256695 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f52980a2800 0x7f52980a4cc0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.574+0000 7f52b5f77640 1 --2- 172.21.15.78:0/389256695 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52b0102150 0x7f52b019ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.574+0000 7f52b5f77640 1 --2- 172.21.15.78:0/389256695 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52b0101790 0x7f52b019c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.574+0000 7f52b5f77640 1 --2- 172.21.15.78:0/389256695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52b01005a0 0x7f52b006a8a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.574+0000 7f52b5f77640 1 -- 172.21.15.78:0/389256695 >> 172.21.15.78:0/389256695 conn(0x7f52b00fbb30 msgr2=0x7f52b0104a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:09.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.574+0000 7f52b5f77640 1 -- 172.21.15.78:0/389256695 shutdown_connections 2024-03-24T00:19:09.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.574+0000 7f52b5f77640 1 -- 172.21.15.78:0/389256695 wait complete. 2024-03-24T00:19:09.584 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:19:09 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:19:09.203+0000 7fd0050d0700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:19:09.584 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:19:09 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:19:09.203+0000 7fd0050d0700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:19:09.585 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:19:09 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:19:09.203+0000 7fd0050d0700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:19:09.585 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:19:09 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:19:09.203+0000 7fd0050d0700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:19:09.585 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:19:09 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:19:09.203+0000 7fd0050d0700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:19:09.585 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:09 smithi078 ceph-mon[32429]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s wr, 1 op/s 2024-03-24T00:19:09.585 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:09 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/4176436863' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-24T00:19:09.585 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:09 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6838/117119462,v1:172.21.15.107:6839/117119462] up:active 2024-03-24T00:19:09.585 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:09 smithi078 ceph-mon[32429]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-24T00:19:09.585 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:09 smithi078 ceph-mon[32429]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-24T00:19:09.585 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:09 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:19:09.586 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:09 smithi078 ceph-mon[32429]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-24T00:19:09.586 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:09 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-24T00:19:09.586 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:09 smithi078 ceph-mon[32429]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-24T00:19:09.586 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:09 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2344476509' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-24T00:19:09.586 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:09 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2344476509' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-24T00:19:09.586 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:09 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6836/2090968633,v1:172.21.15.106:6837/2090968633] up:active 2024-03-24T00:19:09.586 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:09 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:19:09.587 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-24T00:19:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:09 smithi106 ceph-mon[36505]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s wr, 1 op/s 2024-03-24T00:19:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:09 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/4176436863' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-24T00:19:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:09 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6838/117119462,v1:172.21.15.107:6839/117119462] up:active 2024-03-24T00:19:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:09 smithi106 ceph-mon[36505]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-24T00:19:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:09 smithi106 ceph-mon[36505]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-24T00:19:09.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:09 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:19:09.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:09 smithi106 ceph-mon[36505]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-24T00:19:09.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:09 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-24T00:19:09.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:09 smithi106 ceph-mon[36505]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-24T00:19:09.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:09 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2344476509' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-24T00:19:09.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:09 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2344476509' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-24T00:19:09.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:09 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6836/2090968633,v1:172.21.15.106:6837/2090968633] up:active 2024-03-24T00:19:09.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:09 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:19:09.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.692+0000 7f4e07875640 1 -- 172.21.15.78:0/4221651650 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e00104a60 msgr2=0x7f4e00106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:09.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.692+0000 7f4e07875640 1 --2- 172.21.15.78:0/4221651650 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e00104a60 0x7f4e00106e50 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4df0009900 tx=0x7f4df002bc30 comp rx=0 tx=0).stop 2024-03-24T00:19:09.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.692+0000 7f4e07875640 1 -- 172.21.15.78:0/4221651650 shutdown_connections 2024-03-24T00:19:09.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.692+0000 7f4e07875640 1 --2- 172.21.15.78:0/4221651650 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4e00107390 0x7f4e001097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.692+0000 7f4e07875640 1 --2- 172.21.15.78:0/4221651650 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e00104a60 0x7f4e00106e50 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.692+0000 7f4e07875640 1 --2- 172.21.15.78:0/4221651650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e00102130 0x7f4e00104520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.692+0000 7f4e07875640 1 -- 172.21.15.78:0/4221651650 >> 172.21.15.78:0/4221651650 conn(0x7f4e000fbb40 msgr2=0x7f4e000fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:09.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.692+0000 7f4e07875640 1 -- 172.21.15.78:0/4221651650 shutdown_connections 2024-03-24T00:19:09.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.692+0000 7f4e07875640 1 -- 172.21.15.78:0/4221651650 wait complete. 2024-03-24T00:19:09.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.693+0000 7f4e07875640 1 Processor -- start 2024-03-24T00:19:09.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.693+0000 7f4e07875640 1 -- start start 2024-03-24T00:19:09.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.693+0000 7f4e07875640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e00102130 0x7f4e00101aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.693+0000 7f4e07875640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4e00104a60 0x7f4e001000e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.693+0000 7f4e07875640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e00107390 0x7f4e00100620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.693+0000 7f4e07875640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0010d340 con 0x7f4e00102130 2024-03-24T00:19:09.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.693+0000 7f4e07875640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0010d1c0 con 0x7f4e00104a60 2024-03-24T00:19:09.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.693+0000 7f4e07875640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0010d4c0 con 0x7f4e00107390 2024-03-24T00:19:09.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.693+0000 7f4e05611640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e00102130 0x7f4e00101aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.693+0000 7f4e05611640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e00102130 0x7f4e00101aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58340/0 (socket says 172.21.15.78:58340) 2024-03-24T00:19:09.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.693+0000 7f4e05611640 1 -- 172.21.15.78:0/3770888883 learned_addr learned my addr 172.21.15.78:0/3770888883 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:19:09.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.693+0000 7f4e05e12640 1 --2- 172.21.15.78:0/3770888883 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e00107390 0x7f4e00100620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.693+0000 7f4e04e10640 1 --2- 172.21.15.78:0/3770888883 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4e00104a60 0x7f4e001000e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.693+0000 7f4e05611640 1 -- 172.21.15.78:0/3770888883 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e00107390 msgr2=0x7f4e00100620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:09.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.693+0000 7f4e05611640 1 --2- 172.21.15.78:0/3770888883 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e00107390 0x7f4e00100620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.693+0000 7f4e05611640 1 -- 172.21.15.78:0/3770888883 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4e00104a60 msgr2=0x7f4e001000e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:09.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.693+0000 7f4e05611640 1 --2- 172.21.15.78:0/3770888883 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4e00104a60 0x7f4e001000e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.693+0000 7f4e05611640 1 -- 172.21.15.78:0/3770888883 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e00100eb0 con 0x7f4e00102130 2024-03-24T00:19:09.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.693+0000 7f4e05611640 1 --2- 172.21.15.78:0/3770888883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e00102130 0x7f4e00101aa0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc00e9f0 tx=0x7f4dfc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:09.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.694+0000 7f4df67fc640 1 -- 172.21.15.78:0/3770888883 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc00cd80 con 0x7f4e00102130 2024-03-24T00:19:09.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.694+0000 7f4df67fc640 1 -- 172.21.15.78:0/3770888883 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4dfc005500 con 0x7f4e00102130 2024-03-24T00:19:09.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.694+0000 7f4e07875640 1 -- 172.21.15.78:0/3770888883 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e001a1b30 con 0x7f4e00102130 2024-03-24T00:19:09.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.694+0000 7f4df67fc640 1 -- 172.21.15.78:0/3770888883 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc010690 con 0x7f4e00102130 2024-03-24T00:19:09.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.694+0000 7f4e07875640 1 -- 172.21.15.78:0/3770888883 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e001a2050 con 0x7f4e00102130 2024-03-24T00:19:09.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.695+0000 7f4e07875640 1 -- 172.21.15.78:0/3770888883 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e000688e0 con 0x7f4e00102130 2024-03-24T00:19:09.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.696+0000 7f4df67fc640 1 -- 172.21.15.78:0/3770888883 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4dfc0046c0 con 0x7f4e00102130 2024-03-24T00:19:09.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.696+0000 7f4df67fc640 1 --2- 172.21.15.78:0/3770888883 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4dec0a2880 0x7f4dec0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.696+0000 7f4df67fc640 1 -- 172.21.15.78:0/3770888883 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f4dfc0cf260 con 0x7f4e00102130 2024-03-24T00:19:09.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.696+0000 7f4e04e10640 1 --2- 172.21.15.78:0/3770888883 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4dec0a2880 0x7f4dec0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.697+0000 7f4e04e10640 1 --2- 172.21.15.78:0/3770888883 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4dec0a2880 0x7f4dec0a4d40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4df0000c00 tx=0x7f4df0005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:09.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.700+0000 7f4df67fc640 1 -- 172.21.15.78:0/3770888883 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4dfc053740 con 0x7f4e00102130 2024-03-24T00:19:09.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:09 smithi107 ceph-mon[36485]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s wr, 1 op/s 2024-03-24T00:19:09.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:09 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/4176436863' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-24T00:19:09.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:09 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6838/117119462,v1:172.21.15.107:6839/117119462] up:active 2024-03-24T00:19:09.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:09 smithi107 ceph-mon[36485]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-24T00:19:09.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:09 smithi107 ceph-mon[36485]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-24T00:19:09.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:09 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:19:09.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:09 smithi107 ceph-mon[36485]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-24T00:19:09.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:09 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-24T00:19:09.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:09 smithi107 ceph-mon[36485]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-24T00:19:09.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:09 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2344476509' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-24T00:19:09.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:09 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2344476509' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-24T00:19:09.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:09 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6836/2090968633,v1:172.21.15.106:6837/2090968633] up:active 2024-03-24T00:19:09.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:09 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:19:09.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.907+0000 7f4e07875640 1 -- 172.21.15.78:0/3770888883 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e00061b70 con 0x7f4dec0a2880 2024-03-24T00:19:09.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.913+0000 7f4df67fc640 1 -- 172.21.15.78:0/3770888883 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f4e00061b70 con 0x7f4dec0a2880 2024-03-24T00:19:09.914 INFO:teuthology.orchestra.run.smithi078.stdout:/volumes/qa/sv_0/9ade28e8-4076-48aa-b881-0fb7491fbb55 2024-03-24T00:19:09.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.915+0000 7f4e07875640 1 -- 172.21.15.78:0/3770888883 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4dec0a2880 msgr2=0x7f4dec0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:09.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.915+0000 7f4e07875640 1 --2- 172.21.15.78:0/3770888883 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4dec0a2880 0x7f4dec0a4d40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4df0000c00 tx=0x7f4df0005fd0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.915+0000 7f4e07875640 1 -- 172.21.15.78:0/3770888883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e00102130 msgr2=0x7f4e00101aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:09.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.915+0000 7f4e07875640 1 --2- 172.21.15.78:0/3770888883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e00102130 0x7f4e00101aa0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc00e9f0 tx=0x7f4dfc00edf0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.915+0000 7f4e07875640 1 -- 172.21.15.78:0/3770888883 shutdown_connections 2024-03-24T00:19:09.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.915+0000 7f4e07875640 1 --2- 172.21.15.78:0/3770888883 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4dec0a2880 0x7f4dec0a4d40 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.915+0000 7f4e07875640 1 --2- 172.21.15.78:0/3770888883 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e00107390 0x7f4e00100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.915+0000 7f4e07875640 1 --2- 172.21.15.78:0/3770888883 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4e00104a60 0x7f4e001000e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.915+0000 7f4e07875640 1 --2- 172.21.15.78:0/3770888883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e00102130 0x7f4e00101aa0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.915+0000 7f4e07875640 1 -- 172.21.15.78:0/3770888883 >> 172.21.15.78:0/3770888883 conn(0x7f4e000fbb40 msgr2=0x7f4e00102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:09.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.915+0000 7f4e07875640 1 -- 172.21.15.78:0/3770888883 shutdown_connections 2024-03-24T00:19:09.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:09.915+0000 7f4e07875640 1 -- 172.21.15.78:0/3770888883 wait complete. 2024-03-24T00:19:09.927 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-24T00:19:10.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.032+0000 7f0bcca6c640 1 -- 172.21.15.78:0/1943633766 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bc81005a0 msgr2=0x7f0bc81009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.032+0000 7f0bcca6c640 1 --2- 172.21.15.78:0/1943633766 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bc81005a0 0x7f0bc81009c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8009900 tx=0x7f0bb802bc30 comp rx=0 tx=0).stop 2024-03-24T00:19:10.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.033+0000 7f0bcca6c640 1 -- 172.21.15.78:0/1943633766 shutdown_connections 2024-03-24T00:19:10.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.033+0000 7f0bcca6c640 1 --2- 172.21.15.78:0/1943633766 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bc8101790 0x7f0bc8101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.033+0000 7f0bcca6c640 1 --2- 172.21.15.78:0/1943633766 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0bc8102150 0x7f0bc81089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.033+0000 7f0bcca6c640 1 --2- 172.21.15.78:0/1943633766 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bc81005a0 0x7f0bc81009c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.033+0000 7f0bcca6c640 1 -- 172.21.15.78:0/1943633766 >> 172.21.15.78:0/1943633766 conn(0x7f0bc80fbb30 msgr2=0x7f0bc80fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:10.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.033+0000 7f0bcca6c640 1 -- 172.21.15.78:0/1943633766 shutdown_connections 2024-03-24T00:19:10.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.033+0000 7f0bcca6c640 1 -- 172.21.15.78:0/1943633766 wait complete. 2024-03-24T00:19:10.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.033+0000 7f0bcca6c640 1 Processor -- start 2024-03-24T00:19:10.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.033+0000 7f0bcca6c640 1 -- start start 2024-03-24T00:19:10.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.034+0000 7f0bcca6c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bc81005a0 0x7f0bc8197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.034+0000 7f0bcca6c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0bc8101790 0x7f0bc8197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.034+0000 7f0bcca6c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bc8102150 0x7f0bc819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.034+0000 7f0bcca6c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc810d340 con 0x7f0bc8102150 2024-03-24T00:19:10.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.034+0000 7f0bcca6c640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc810d1c0 con 0x7f0bc8101790 2024-03-24T00:19:10.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.034+0000 7f0bcca6c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc810d4c0 con 0x7f0bc81005a0 2024-03-24T00:19:10.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.034+0000 7f0bc6d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bc8102150 0x7f0bc819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.034+0000 7f0bc6d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bc8102150 0x7f0bc819efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58358/0 (socket says 172.21.15.78:58358) 2024-03-24T00:19:10.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.034+0000 7f0bc6d9d640 1 -- 172.21.15.78:0/3695303644 learned_addr learned my addr 172.21.15.78:0/3695303644 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:19:10.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.034+0000 7f0bc659c640 1 --2- 172.21.15.78:0/3695303644 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bc81005a0 0x7f0bc8197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.034+0000 7f0bc5d9b640 1 --2- 172.21.15.78:0/3695303644 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0bc8101790 0x7f0bc8197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.034+0000 7f0bc6d9d640 1 -- 172.21.15.78:0/3695303644 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bc81005a0 msgr2=0x7f0bc8197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.034+0000 7f0bc6d9d640 1 --2- 172.21.15.78:0/3695303644 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bc81005a0 0x7f0bc8197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.034+0000 7f0bc6d9d640 1 -- 172.21.15.78:0/3695303644 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0bc8101790 msgr2=0x7f0bc8197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.034+0000 7f0bc6d9d640 1 --2- 172.21.15.78:0/3695303644 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0bc8101790 0x7f0bc8197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.034+0000 7f0bc6d9d640 1 -- 172.21.15.78:0/3695303644 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc819f620 con 0x7f0bc8102150 2024-03-24T00:19:10.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.034+0000 7f0bc6d9d640 1 --2- 172.21.15.78:0/3695303644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bc8102150 0x7f0bc819efb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0bc000eb10 tx=0x7f0bc000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:10.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.034+0000 7f0bb77fe640 1 -- 172.21.15.78:0/3695303644 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc0004b20 con 0x7f0bc8102150 2024-03-24T00:19:10.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.034+0000 7f0bb77fe640 1 -- 172.21.15.78:0/3695303644 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0bc0004cc0 con 0x7f0bc8102150 2024-03-24T00:19:10.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.034+0000 7f0bcca6c640 1 -- 172.21.15.78:0/3695303644 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc819f910 con 0x7f0bc8102150 2024-03-24T00:19:10.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.034+0000 7f0bb77fe640 1 -- 172.21.15.78:0/3695303644 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc0010760 con 0x7f0bc8102150 2024-03-24T00:19:10.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.034+0000 7f0bcca6c640 1 -- 172.21.15.78:0/3695303644 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc819fe10 con 0x7f0bc8102150 2024-03-24T00:19:10.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.035+0000 7f0bcca6c640 1 -- 172.21.15.78:0/3695303644 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bc81078e0 con 0x7f0bc8102150 2024-03-24T00:19:10.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.036+0000 7f0bb77fe640 1 -- 172.21.15.78:0/3695303644 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0bc0005020 con 0x7f0bc8102150 2024-03-24T00:19:10.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.037+0000 7f0bb77fe640 1 --2- 172.21.15.78:0/3695303644 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f0bb00a2800 0x7f0bb00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.037+0000 7f0bc659c640 1 --2- 172.21.15.78:0/3695303644 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f0bb00a2800 0x7f0bb00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.037+0000 7f0bb77fe640 1 -- 172.21.15.78:0/3695303644 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f0bc00cf210 con 0x7f0bc8102150 2024-03-24T00:19:10.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.038+0000 7f0bc659c640 1 --2- 172.21.15.78:0/3695303644 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f0bb00a2800 0x7f0bb00a4cc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8000c00 tx=0x7f0bb8005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:10.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.042+0000 7f0bb77fe640 1 -- 172.21.15.78:0/3695303644 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0bc009c070 con 0x7f0bc8102150 2024-03-24T00:19:10.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.245+0000 7f0bcca6c640 1 -- 172.21.15.78:0/3695303644 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f0bc8061ae0 con 0x7f0bb00a2800 2024-03-24T00:19:10.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.269+0000 7f0bb77fe640 1 -- 172.21.15.78:0/3695303644 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0bc8061ae0 con 0x7f0bb00a2800 2024-03-24T00:19:10.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.271+0000 7f0bcca6c640 1 -- 172.21.15.78:0/3695303644 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f0bb00a2800 msgr2=0x7f0bb00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.271+0000 7f0bcca6c640 1 --2- 172.21.15.78:0/3695303644 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f0bb00a2800 0x7f0bb00a4cc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8000c00 tx=0x7f0bb8005fd0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.271+0000 7f0bcca6c640 1 -- 172.21.15.78:0/3695303644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bc8102150 msgr2=0x7f0bc819efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.271+0000 7f0bcca6c640 1 --2- 172.21.15.78:0/3695303644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bc8102150 0x7f0bc819efb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0bc000eb10 tx=0x7f0bc000c2d0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.271+0000 7f0bcca6c640 1 -- 172.21.15.78:0/3695303644 shutdown_connections 2024-03-24T00:19:10.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.271+0000 7f0bcca6c640 1 --2- 172.21.15.78:0/3695303644 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f0bb00a2800 0x7f0bb00a4cc0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.271+0000 7f0bcca6c640 1 --2- 172.21.15.78:0/3695303644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bc8102150 0x7f0bc819efb0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.271+0000 7f0bcca6c640 1 --2- 172.21.15.78:0/3695303644 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0bc8101790 0x7f0bc8197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.271+0000 7f0bcca6c640 1 --2- 172.21.15.78:0/3695303644 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bc81005a0 0x7f0bc8197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.271+0000 7f0bcca6c640 1 -- 172.21.15.78:0/3695303644 >> 172.21.15.78:0/3695303644 conn(0x7f0bc80fbb30 msgr2=0x7f0bc8104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:10.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.272+0000 7f0bcca6c640 1 -- 172.21.15.78:0/3695303644 shutdown_connections 2024-03-24T00:19:10.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.272+0000 7f0bcca6c640 1 -- 172.21.15.78:0/3695303644 wait complete. 2024-03-24T00:19:10.283 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-24T00:19:10.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.394+0000 7fec5980b640 1 -- 172.21.15.78:0/2003912102 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec541005c0 msgr2=0x7fec541009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.394+0000 7fec5980b640 1 --2- 172.21.15.78:0/2003912102 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec541005c0 0x7fec541009e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fec44009900 tx=0x7fec4402bc30 comp rx=0 tx=0).stop 2024-03-24T00:19:10.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.394+0000 7fec5980b640 1 -- 172.21.15.78:0/2003912102 shutdown_connections 2024-03-24T00:19:10.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.394+0000 7fec5980b640 1 --2- 172.21.15.78:0/2003912102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec54102170 0x7fec54108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.394+0000 7fec5980b640 1 --2- 172.21.15.78:0/2003912102 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fec541017b0 0x7fec54101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.394+0000 7fec5980b640 1 --2- 172.21.15.78:0/2003912102 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec541005c0 0x7fec541009e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.394+0000 7fec5980b640 1 -- 172.21.15.78:0/2003912102 >> 172.21.15.78:0/2003912102 conn(0x7fec540fbb50 msgr2=0x7fec540fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:10.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.394+0000 7fec5980b640 1 -- 172.21.15.78:0/2003912102 shutdown_connections 2024-03-24T00:19:10.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.394+0000 7fec5980b640 1 -- 172.21.15.78:0/2003912102 wait complete. 2024-03-24T00:19:10.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.395+0000 7fec5980b640 1 Processor -- start 2024-03-24T00:19:10.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.395+0000 7fec5980b640 1 -- start start 2024-03-24T00:19:10.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.395+0000 7fec5980b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec541005c0 0x7fec5406a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.395+0000 7fec5980b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec541017b0 0x7fec5419c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.395+0000 7fec5980b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fec54102170 0x7fec5419ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.395+0000 7fec5980b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5410c230 con 0x7fec541017b0 2024-03-24T00:19:10.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.395+0000 7fec5980b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5410c0b0 con 0x7fec54102170 2024-03-24T00:19:10.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.395+0000 7fec5980b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5410c3b0 con 0x7fec541005c0 2024-03-24T00:19:10.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.395+0000 7fec4bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec541017b0 0x7fec5419c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.395+0000 7fec4bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec541017b0 0x7fec5419c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58370/0 (socket says 172.21.15.78:58370) 2024-03-24T00:19:10.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.395+0000 7fec4bfff640 1 -- 172.21.15.78:0/2465647223 learned_addr learned my addr 172.21.15.78:0/2465647223 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:19:10.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.395+0000 7fec52ffd640 1 --2- 172.21.15.78:0/2465647223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec541005c0 0x7fec5406a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.395+0000 7fec537fe640 1 --2- 172.21.15.78:0/2465647223 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fec54102170 0x7fec5419ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.395+0000 7fec4bfff640 1 -- 172.21.15.78:0/2465647223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec541005c0 msgr2=0x7fec5406a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.395+0000 7fec4bfff640 1 --2- 172.21.15.78:0/2465647223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec541005c0 0x7fec5406a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.395+0000 7fec4bfff640 1 -- 172.21.15.78:0/2465647223 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fec54102170 msgr2=0x7fec5419ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.395+0000 7fec4bfff640 1 --2- 172.21.15.78:0/2465647223 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fec54102170 0x7fec5419ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.395+0000 7fec4bfff640 1 -- 172.21.15.78:0/2465647223 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec5419f630 con 0x7fec541017b0 2024-03-24T00:19:10.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.395+0000 7fec4bfff640 1 --2- 172.21.15.78:0/2465647223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec541017b0 0x7fec5419c730 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fec3c00e9f0 tx=0x7fec3c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:10.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.396+0000 7fec58809640 1 -- 172.21.15.78:0/2465647223 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec3c00cd80 con 0x7fec541017b0 2024-03-24T00:19:10.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.396+0000 7fec5980b640 1 -- 172.21.15.78:0/2465647223 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec5419f920 con 0x7fec541017b0 2024-03-24T00:19:10.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.396+0000 7fec58809640 1 -- 172.21.15.78:0/2465647223 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec3c010910 con 0x7fec541017b0 2024-03-24T00:19:10.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.396+0000 7fec58809640 1 -- 172.21.15.78:0/2465647223 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec3c0188f0 con 0x7fec541017b0 2024-03-24T00:19:10.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.396+0000 7fec5980b640 1 -- 172.21.15.78:0/2465647223 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec5419fec0 con 0x7fec541017b0 2024-03-24T00:19:10.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.397+0000 7fec5980b640 1 -- 172.21.15.78:0/2465647223 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec54107960 con 0x7fec541017b0 2024-03-24T00:19:10.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.398+0000 7fec58809640 1 -- 172.21.15.78:0/2465647223 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fec3c010ac0 con 0x7fec541017b0 2024-03-24T00:19:10.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.398+0000 7fec58809640 1 --2- 172.21.15.78:0/2465647223 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fec400a2800 0x7fec400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.398+0000 7fec52ffd640 1 --2- 172.21.15.78:0/2465647223 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fec400a2800 0x7fec400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.398+0000 7fec58809640 1 -- 172.21.15.78:0/2465647223 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fec3c0cf410 con 0x7fec541017b0 2024-03-24T00:19:10.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.398+0000 7fec52ffd640 1 --2- 172.21.15.78:0/2465647223 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fec400a2800 0x7fec400a4cc0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fec44000c00 tx=0x7fec44005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:10.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.402+0000 7fec58809640 1 -- 172.21.15.78:0/2465647223 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fec3c09bc20 con 0x7fec541017b0 2024-03-24T00:19:10.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:10 smithi078 ceph-mon[32429]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:19:10.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.609+0000 7fec5980b640 1 -- 172.21.15.78:0/2465647223 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fec54061b70 con 0x7fec400a2800 2024-03-24T00:19:10.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.614+0000 7fec58809640 1 -- 172.21.15.78:0/2465647223 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7fec54061b70 con 0x7fec400a2800 2024-03-24T00:19:10.614 INFO:teuthology.orchestra.run.smithi078.stdout:/volumes/qa/sv_1/9775ff22-04a1-4999-bfcb-98d63ec721c1 2024-03-24T00:19:10.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.616+0000 7fec5980b640 1 -- 172.21.15.78:0/2465647223 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fec400a2800 msgr2=0x7fec400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.616+0000 7fec5980b640 1 --2- 172.21.15.78:0/2465647223 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fec400a2800 0x7fec400a4cc0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fec44000c00 tx=0x7fec44005fd0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.616+0000 7fec5980b640 1 -- 172.21.15.78:0/2465647223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec541017b0 msgr2=0x7fec5419c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.616+0000 7fec5980b640 1 --2- 172.21.15.78:0/2465647223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec541017b0 0x7fec5419c730 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fec3c00e9f0 tx=0x7fec3c00eeb0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.616+0000 7fec5980b640 1 -- 172.21.15.78:0/2465647223 shutdown_connections 2024-03-24T00:19:10.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.616+0000 7fec5980b640 1 --2- 172.21.15.78:0/2465647223 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fec400a2800 0x7fec400a4cc0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.616+0000 7fec5980b640 1 --2- 172.21.15.78:0/2465647223 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fec54102170 0x7fec5419ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.616+0000 7fec5980b640 1 --2- 172.21.15.78:0/2465647223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec541017b0 0x7fec5419c730 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.616+0000 7fec5980b640 1 --2- 172.21.15.78:0/2465647223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec541005c0 0x7fec5406a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.617+0000 7fec5980b640 1 -- 172.21.15.78:0/2465647223 >> 172.21.15.78:0/2465647223 conn(0x7fec540fbb50 msgr2=0x7fec54104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:10.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.617+0000 7fec5980b640 1 -- 172.21.15.78:0/2465647223 shutdown_connections 2024-03-24T00:19:10.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.617+0000 7fec5980b640 1 -- 172.21.15.78:0/2465647223 wait complete. 2024-03-24T00:19:10.627 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:19:10.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:10 smithi106 ceph-mon[36505]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:19:10.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.729+0000 7fe75fd87640 1 -- 172.21.15.78:0/107041493 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7581017d0 msgr2=0x7fe758101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.729+0000 7fe75fd87640 1 --2- 172.21.15.78:0/107041493 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7581017d0 0x7fe758101c50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe754009860 tx=0x7fe75402bb70 comp rx=0 tx=0).stop 2024-03-24T00:19:10.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.729+0000 7fe75fd87640 1 -- 172.21.15.78:0/107041493 shutdown_connections 2024-03-24T00:19:10.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.729+0000 7fe75fd87640 1 --2- 172.21.15.78:0/107041493 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7581017d0 0x7fe758101c50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.729+0000 7fe75fd87640 1 --2- 172.21.15.78:0/107041493 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe758102190 0x7fe758108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.729+0000 7fe75fd87640 1 --2- 172.21.15.78:0/107041493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7581005e0 0x7fe758100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.729+0000 7fe75fd87640 1 -- 172.21.15.78:0/107041493 >> 172.21.15.78:0/107041493 conn(0x7fe7580fbb50 msgr2=0x7fe7580fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:10.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.729+0000 7fe75fd87640 1 -- 172.21.15.78:0/107041493 shutdown_connections 2024-03-24T00:19:10.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.729+0000 7fe75fd87640 1 -- 172.21.15.78:0/107041493 wait complete. 2024-03-24T00:19:10.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.729+0000 7fe75fd87640 1 Processor -- start 2024-03-24T00:19:10.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.730+0000 7fe75fd87640 1 -- start start 2024-03-24T00:19:10.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.730+0000 7fe75fd87640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7581005e0 0x7fe75806a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.730+0000 7fe75fd87640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe7581017d0 0x7fe75819c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.730+0000 7fe75fd87640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe758102190 0x7fe75819ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.730+0000 7fe75fd87640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75810c260 con 0x7fe758102190 2024-03-24T00:19:10.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.730+0000 7fe75fd87640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75810c0e0 con 0x7fe7581017d0 2024-03-24T00:19:10.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.730+0000 7fe75fd87640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75810c3e0 con 0x7fe7581005e0 2024-03-24T00:19:10.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.730+0000 7fe75e324640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe758102190 0x7fe75819ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.730+0000 7fe75e324640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe758102190 0x7fe75819ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58386/0 (socket says 172.21.15.78:58386) 2024-03-24T00:19:10.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.730+0000 7fe75db23640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7581005e0 0x7fe75806a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.730+0000 7fe75e324640 1 -- 172.21.15.78:0/2733468502 learned_addr learned my addr 172.21.15.78:0/2733468502 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:19:10.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.730+0000 7fe75d322640 1 --2- 172.21.15.78:0/2733468502 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe7581017d0 0x7fe75819c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.730+0000 7fe75e324640 1 -- 172.21.15.78:0/2733468502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7581005e0 msgr2=0x7fe75806a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.730+0000 7fe75e324640 1 --2- 172.21.15.78:0/2733468502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7581005e0 0x7fe75806a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.730+0000 7fe75e324640 1 -- 172.21.15.78:0/2733468502 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe7581017d0 msgr2=0x7fe75819c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.730+0000 7fe75e324640 1 --2- 172.21.15.78:0/2733468502 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe7581017d0 0x7fe75819c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.730+0000 7fe75e324640 1 -- 172.21.15.78:0/2733468502 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75819f5e0 con 0x7fe758102190 2024-03-24T00:19:10.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.730+0000 7fe75e324640 1 --2- 172.21.15.78:0/2733468502 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe758102190 0x7fe75819ed20 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe75000eb10 tx=0x7fe75000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:10.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.730+0000 7fe74effd640 1 -- 172.21.15.78:0/2733468502 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75000ce20 con 0x7fe758102190 2024-03-24T00:19:10.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.730+0000 7fe74effd640 1 -- 172.21.15.78:0/2733468502 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe750005020 con 0x7fe758102190 2024-03-24T00:19:10.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.730+0000 7fe75fd87640 1 -- 172.21.15.78:0/2733468502 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe75819f8d0 con 0x7fe758102190 2024-03-24T00:19:10.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.730+0000 7fe74effd640 1 -- 172.21.15.78:0/2733468502 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe750010690 con 0x7fe758102190 2024-03-24T00:19:10.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.730+0000 7fe75fd87640 1 -- 172.21.15.78:0/2733468502 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe75819fdf0 con 0x7fe758102190 2024-03-24T00:19:10.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.731+0000 7fe75fd87640 1 -- 172.21.15.78:0/2733468502 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe758107900 con 0x7fe758102190 2024-03-24T00:19:10.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.732+0000 7fe74effd640 1 -- 172.21.15.78:0/2733468502 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe750005540 con 0x7fe758102190 2024-03-24T00:19:10.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.733+0000 7fe74effd640 1 --2- 172.21.15.78:0/2733468502 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe7440a2800 0x7fe7440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.733+0000 7fe75db23640 1 --2- 172.21.15.78:0/2733468502 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe7440a2800 0x7fe7440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.733+0000 7fe74effd640 1 -- 172.21.15.78:0/2733468502 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fe7500cf390 con 0x7fe758102190 2024-03-24T00:19:10.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.733+0000 7fe75db23640 1 --2- 172.21.15.78:0/2733468502 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe7440a2800 0x7fe7440a4cc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe758101630 tx=0x7fe748009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:10.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.737+0000 7fe74effd640 1 -- 172.21.15.78:0/2733468502 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe75009c1f0 con 0x7fe758102190 2024-03-24T00:19:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:10 smithi107 ceph-mon[36485]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:19:10.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.949+0000 7fe75fd87640 1 -- 172.21.15.78:0/2733468502 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7580622a0 con 0x7fe758102190 2024-03-24T00:19:10.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.949+0000 7fe74effd640 1 -- 172.21.15.78:0/2733468502 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11406 (secure 0 0 0) 0x7fe75009baf0 con 0x7fe758102190 2024-03-24T00:19:10.949 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:19:10.949 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1416685123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1416685123},{"type":"v1","addr":"172.21.15.78:6835","nonce":1416685123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1071274623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1071274623},{"type":"v1","addr":"172.21.15.78:6837","nonce":1071274623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2758357551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2758357551},{"type":"v1","addr":"172.21.15.78:6839","nonce":2758357551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/3424505910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":3424505910},{"type":"v1","addr":"172.21.15.78:6841","nonce":3424505910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/3086318443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3086318443},{"type":"v1","addr":"172.21.15.107:6833","nonce":3086318443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/2367018868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2367018868},{"type":"v1","addr":"172.21.15.106:6833","nonce":2367018868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:18:43.592215+0000","modified":"2024-03-24T00:19:08.871384+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24500,"mds_1":24481,"mds_2":24464},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/2231611477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":2231611477},{"type":"v1","addr":"172.21.15.106:6835","nonce":2231611477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6835/108936439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":108936439},{"type":"v1","addr":"172.21.15.107:6835","nonce":108936439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6837/2939571540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2939571540},{"type":"v1","addr":"172.21.15.107:6837","nonce":2939571540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24464":{"gid":24464,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.106:6837/2090968633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":2090968633},{"type":"v1","addr":"172.21.15.106:6837","nonce":2090968633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.107:6839/117119462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":117119462},{"type":"v1","addr":"172.21.15.107:6839","nonce":117119462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6839/2482071304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2482071304},{"type":"v1","addr":"172.21.15.106:6839","nonce":2482071304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24481,24500,24464]},"id":1}]} 2024-03-24T00:19:10.950 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 23 2024-03-24T00:19:10.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.951+0000 7fe75fd87640 1 -- 172.21.15.78:0/2733468502 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe7440a2800 msgr2=0x7fe7440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.951+0000 7fe75fd87640 1 --2- 172.21.15.78:0/2733468502 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe7440a2800 0x7fe7440a4cc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe758101630 tx=0x7fe748009260 comp rx=0 tx=0).stop 2024-03-24T00:19:10.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.951+0000 7fe75fd87640 1 -- 172.21.15.78:0/2733468502 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe758102190 msgr2=0x7fe75819ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.951+0000 7fe75fd87640 1 --2- 172.21.15.78:0/2733468502 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe758102190 0x7fe75819ed20 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe75000eb10 tx=0x7fe75000ef10 comp rx=0 tx=0).stop 2024-03-24T00:19:10.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.951+0000 7fe75fd87640 1 -- 172.21.15.78:0/2733468502 shutdown_connections 2024-03-24T00:19:10.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.951+0000 7fe75fd87640 1 --2- 172.21.15.78:0/2733468502 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe7440a2800 0x7fe7440a4cc0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.951+0000 7fe75fd87640 1 --2- 172.21.15.78:0/2733468502 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe758102190 0x7fe75819ed20 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.951+0000 7fe75fd87640 1 --2- 172.21.15.78:0/2733468502 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe7581017d0 0x7fe75819c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.951+0000 7fe75fd87640 1 --2- 172.21.15.78:0/2733468502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7581005e0 0x7fe75806a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.951+0000 7fe75fd87640 1 -- 172.21.15.78:0/2733468502 >> 172.21.15.78:0/2733468502 conn(0x7fe7580fbb50 msgr2=0x7fe758104ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:10.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.951+0000 7fe75fd87640 1 -- 172.21.15.78:0/2733468502 shutdown_connections 2024-03-24T00:19:10.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:10.951+0000 7fe75fd87640 1 -- 172.21.15.78:0/2733468502 wait complete. 2024-03-24T00:19:10.963 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:19:11.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.065+0000 7f597568a640 1 -- 172.21.15.78:0/3243947432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5970102200 msgr2=0x7f59701045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:11.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.065+0000 7f597568a640 1 --2- 172.21.15.78:0/3243947432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5970102200 0x7f59701045f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5960009900 tx=0x7f596002bc30 comp rx=0 tx=0).stop 2024-03-24T00:19:11.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.065+0000 7f597568a640 1 -- 172.21.15.78:0/3243947432 shutdown_connections 2024-03-24T00:19:11.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.065+0000 7f597568a640 1 --2- 172.21.15.78:0/3243947432 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5970107460 0x7f5970109850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.065+0000 7f597568a640 1 --2- 172.21.15.78:0/3243947432 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5970104b30 0x7f5970106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.065+0000 7f597568a640 1 --2- 172.21.15.78:0/3243947432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5970102200 0x7f59701045f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.065+0000 7f597568a640 1 -- 172.21.15.78:0/3243947432 >> 172.21.15.78:0/3243947432 conn(0x7f59700fbb50 msgr2=0x7f59700fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:11.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.065+0000 7f597568a640 1 -- 172.21.15.78:0/3243947432 shutdown_connections 2024-03-24T00:19:11.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.065+0000 7f597568a640 1 -- 172.21.15.78:0/3243947432 wait complete. 2024-03-24T00:19:11.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.066+0000 7f597568a640 1 Processor -- start 2024-03-24T00:19:11.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.066+0000 7f597568a640 1 -- start start 2024-03-24T00:19:11.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.066+0000 7f597568a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5970102200 0x7f59701979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:11.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.066+0000 7f597568a640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5970104b30 0x7f5970197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:11.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.066+0000 7f597568a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5970107460 0x7f597019efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:11.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.066+0000 7f597568a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597010d370 con 0x7f5970102200 2024-03-24T00:19:11.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.066+0000 7f597568a640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597010d1f0 con 0x7f5970104b30 2024-03-24T00:19:11.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.066+0000 7f597568a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597010d4f0 con 0x7f5970107460 2024-03-24T00:19:11.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.066+0000 7f596effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5970102200 0x7f59701979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:11.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.066+0000 7f596effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5970102200 0x7f59701979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58404/0 (socket says 172.21.15.78:58404) 2024-03-24T00:19:11.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.066+0000 7f596effd640 1 -- 172.21.15.78:0/2734379494 learned_addr learned my addr 172.21.15.78:0/2734379494 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:19:11.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.066+0000 7f596e7fc640 1 --2- 172.21.15.78:0/2734379494 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5970104b30 0x7f5970197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:11.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.066+0000 7f596f7fe640 1 --2- 172.21.15.78:0/2734379494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5970107460 0x7f597019efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:11.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.067+0000 7f596effd640 1 -- 172.21.15.78:0/2734379494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5970107460 msgr2=0x7f597019efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:11.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.067+0000 7f596effd640 1 --2- 172.21.15.78:0/2734379494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5970107460 0x7f597019efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.067+0000 7f596effd640 1 -- 172.21.15.78:0/2734379494 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5970104b30 msgr2=0x7f5970197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:11.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.067+0000 7f596effd640 1 --2- 172.21.15.78:0/2734379494 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5970104b30 0x7f5970197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.067+0000 7f596effd640 1 -- 172.21.15.78:0/2734379494 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f597019f610 con 0x7f5970102200 2024-03-24T00:19:11.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.067+0000 7f596e7fc640 1 --2- 172.21.15.78:0/2734379494 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5970104b30 0x7f5970197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:11.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.067+0000 7f596effd640 1 --2- 172.21.15.78:0/2734379494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5970102200 0x7f59701979f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5960009230 tx=0x7f596002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:11.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.067+0000 7f5957fff640 1 -- 172.21.15.78:0/2734379494 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5960041070 con 0x7f5970102200 2024-03-24T00:19:11.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.067+0000 7f5957fff640 1 -- 172.21.15.78:0/2734379494 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5960004450 con 0x7f5970102200 2024-03-24T00:19:11.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.067+0000 7f597568a640 1 -- 172.21.15.78:0/2734379494 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f597019f8a0 con 0x7f5970102200 2024-03-24T00:19:11.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.067+0000 7f5957fff640 1 -- 172.21.15.78:0/2734379494 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5960003e20 con 0x7f5970102200 2024-03-24T00:19:11.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.067+0000 7f597568a640 1 -- 172.21.15.78:0/2734379494 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f597019fd90 con 0x7f5970102200 2024-03-24T00:19:11.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.068+0000 7f597568a640 1 -- 172.21.15.78:0/2734379494 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59700fd640 con 0x7f5970102200 2024-03-24T00:19:11.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.069+0000 7f5957fff640 1 -- 172.21.15.78:0/2734379494 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5960044070 con 0x7f5970102200 2024-03-24T00:19:11.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.069+0000 7f5957fff640 1 --2- 172.21.15.78:0/2734379494 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f59580a2800 0x7f59580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:11.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.070+0000 7f596e7fc640 1 --2- 172.21.15.78:0/2734379494 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f59580a2800 0x7f59580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:11.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.070+0000 7f5957fff640 1 -- 172.21.15.78:0/2734379494 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f59600f0840 con 0x7f5970102200 2024-03-24T00:19:11.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.070+0000 7f596e7fc640 1 --2- 172.21.15.78:0/2734379494 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f59580a2800 0x7f59580a4cc0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f59640095f0 tx=0x7f5964005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:11.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.074+0000 7f5957fff640 1 -- 172.21.15.78:0/2734379494 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f59600bd050 con 0x7f5970102200 2024-03-24T00:19:11.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.251+0000 7f597568a640 1 -- 172.21.15.78:0/2734379494 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59700622d0 con 0x7f5970102200 2024-03-24T00:19:11.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.252+0000 7f5957fff640 1 -- 172.21.15.78:0/2734379494 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24129 (secure 0 0 0) 0x7f59600c10f0 con 0x7f5970102200 2024-03-24T00:19:11.256 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:19:11.256 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":89,"fsid":"c7e2806a-e972-11ee-95cd-87774f69a715","created":"2024-03-24T00:10:44.134563+0000","modified":"2024-03-24T00:18:48.224440+0000","last_up_change":"2024-03-24T00:17:50.237686+0000","last_in_change":"2024-03-24T00:17:29.788699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:14:04.534391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:18:42.178418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:18:42.800424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:18:46.112800+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b950388a-c437-426b-b703-3f8ae821dd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6803","nonce":1816879397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6805","nonce":1816879397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6809","nonce":1816879397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6807","nonce":1816879397}]},"public_addr":"172.21.15.78:6803/1816879397","cluster_addr":"172.21.15.78:6805/1816879397","heartbeat_back_addr":"172.21.15.78:6809/1816879397","heartbeat_front_addr":"172.21.15.78:6807/1816879397","state":["exists","up"]},{"osd":1,"uuid":"b8d64714-dd65-400f-b3f4-673ab14e1365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6801","nonce":1647915612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6803","nonce":1647915612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6807","nonce":1647915612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6805","nonce":1647915612}]},"public_addr":"172.21.15.106:6801/1647915612","cluster_addr":"172.21.15.106:6803/1647915612","heartbeat_back_addr":"172.21.15.106:6807/1647915612","heartbeat_front_addr":"172.21.15.106:6805/1647915612","state":["exists","up"]},{"osd":2,"uuid":"fcfc4dd1-aaf8-4131-a023-5e832212158d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6801","nonce":2010770435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6803","nonce":2010770435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6807","nonce":2010770435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6805","nonce":2010770435}]},"public_addr":"172.21.15.107:6801/2010770435","cluster_addr":"172.21.15.107:6803/2010770435","heartbeat_back_addr":"172.21.15.107:6807/2010770435","heartbeat_front_addr":"172.21.15.107:6805/2010770435","state":["exists","up"]},{"osd":3,"uuid":"47c718ab-a0f4-4e52-83c5-b89386b262d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6811","nonce":1912251346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6813","nonce":1912251346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6817","nonce":1912251346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6815","nonce":1912251346}]},"public_addr":"172.21.15.78:6811/1912251346","cluster_addr":"172.21.15.78:6813/1912251346","heartbeat_back_addr":"172.21.15.78:6817/1912251346","heartbeat_front_addr":"172.21.15.78:6815/1912251346","state":["exists","up"]},{"osd":4,"uuid":"7dad64dd-677e-4c1c-9638-e4f4f2195983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6809","nonce":374641205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6811","nonce":374641205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6815","nonce":374641205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6813","nonce":374641205}]},"public_addr":"172.21.15.106:6809/374641205","cluster_addr":"172.21.15.106:6811/374641205","heartbeat_back_addr":"172.21.15.106:6815/374641205","heartbeat_front_addr":"172.21.15.106:6813/374641205","state":["exists","up"]},{"osd":5,"uuid":"bbd524e5-328d-4309-8877-1149cbe110dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6809","nonce":3272806073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6811","nonce":3272806073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6815","nonce":3272806073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6813","nonce":3272806073}]},"public_addr":"172.21.15.107:6809/3272806073","cluster_addr":"172.21.15.107:6811/3272806073","heartbeat_back_addr":"172.21.15.107:6815/3272806073","heartbeat_front_addr":"172.21.15.107:6813/3272806073","state":["exists","up"]},{"osd":6,"uuid":"5696389a-918f-4ea8-b20d-4e2a0d4cb3bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6819","nonce":3091148508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6821","nonce":3091148508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6825","nonce":3091148508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6823","nonce":3091148508}]},"public_addr":"172.21.15.78:6819/3091148508","cluster_addr":"172.21.15.78:6821/3091148508","heartbeat_back_addr":"172.21.15.78:6825/3091148508","heartbeat_front_addr":"172.21.15.78:6823/3091148508","state":["exists","up"]},{"osd":7,"uuid":"f6b764e8-ed4e-4c9a-a85b-94af6dc6b924","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6817","nonce":4262103080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6819","nonce":4262103080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6823","nonce":4262103080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6821","nonce":4262103080}]},"public_addr":"172.21.15.106:6817/4262103080","cluster_addr":"172.21.15.106:6819/4262103080","heartbeat_back_addr":"172.21.15.106:6823/4262103080","heartbeat_front_addr":"172.21.15.106:6821/4262103080","state":["exists","up"]},{"osd":8,"uuid":"ba16039f-8093-4fa0-87ad-e28b7d142b23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6817","nonce":4032799356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6819","nonce":4032799356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6823","nonce":4032799356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6821","nonce":4032799356}]},"public_addr":"172.21.15.107:6817/4032799356","cluster_addr":"172.21.15.107:6819/4032799356","heartbeat_back_addr":"172.21.15.107:6823/4032799356","heartbeat_front_addr":"172.21.15.107:6821/4032799356","state":["exists","up"]},{"osd":9,"uuid":"651357f2-0669-42e6-aa91-1d5c02d00c5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6827","nonce":214023431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6829","nonce":214023431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6833","nonce":214023431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6831","nonce":214023431}]},"public_addr":"172.21.15.78:6827/214023431","cluster_addr":"172.21.15.78:6829/214023431","heartbeat_back_addr":"172.21.15.78:6833/214023431","heartbeat_front_addr":"172.21.15.78:6831/214023431","state":["exists","up"]},{"osd":10,"uuid":"2375119d-346e-426c-8a65-b64af71ad63a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6825","nonce":400029804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6827","nonce":400029804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6831","nonce":400029804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6829","nonce":400029804}]},"public_addr":"172.21.15.106:6825/400029804","cluster_addr":"172.21.15.106:6827/400029804","heartbeat_back_addr":"172.21.15.106:6831/400029804","heartbeat_front_addr":"172.21.15.106:6829/400029804","state":["exists","up"]},{"osd":11,"uuid":"b6b609d9-dd78-4c01-ae66-ba2a662ab22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6825","nonce":4191916230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6827","nonce":4191916230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6831","nonce":4191916230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6829","nonce":4191916230}]},"public_addr":"172.21.15.107:6825/4191916230","cluster_addr":"172.21.15.107:6827/4191916230","heartbeat_back_addr":"172.21.15.107:6831/4191916230","heartbeat_front_addr":"172.21.15.107:6829/4191916230","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:13.728602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:37.320809+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:01.269452+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:25.758281+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:50.298863+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:14.345219+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:38.980489+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:04.749214+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:29.992673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:56.269202+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:22.270316+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:48.074907+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/1982202541":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/1933054339":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6801/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6800/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/465691596":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6801/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/3962523609":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6800/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/1502752625":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/2236879401":"2024-03-25T00:11:40.199569+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:19:11.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.254+0000 7f597568a640 1 -- 172.21.15.78:0/2734379494 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f59580a2800 msgr2=0x7f59580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:11.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.254+0000 7f597568a640 1 --2- 172.21.15.78:0/2734379494 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f59580a2800 0x7f59580a4cc0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f59640095f0 tx=0x7f5964005f90 comp rx=0 tx=0).stop 2024-03-24T00:19:11.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.254+0000 7f597568a640 1 -- 172.21.15.78:0/2734379494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5970102200 msgr2=0x7f59701979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:11.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.254+0000 7f597568a640 1 --2- 172.21.15.78:0/2734379494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5970102200 0x7f59701979f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5960009230 tx=0x7f596002c650 comp rx=0 tx=0).stop 2024-03-24T00:19:11.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.254+0000 7f597568a640 1 -- 172.21.15.78:0/2734379494 shutdown_connections 2024-03-24T00:19:11.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.254+0000 7f597568a640 1 --2- 172.21.15.78:0/2734379494 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f59580a2800 0x7f59580a4cc0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.254+0000 7f597568a640 1 --2- 172.21.15.78:0/2734379494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5970107460 0x7f597019efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.254+0000 7f597568a640 1 --2- 172.21.15.78:0/2734379494 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5970104b30 0x7f5970197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.254+0000 7f597568a640 1 --2- 172.21.15.78:0/2734379494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5970102200 0x7f59701979f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.254+0000 7f597568a640 1 -- 172.21.15.78:0/2734379494 >> 172.21.15.78:0/2734379494 conn(0x7f59700fbb50 msgr2=0x7f5970102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:11.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.254+0000 7f597568a640 1 -- 172.21.15.78:0/2734379494 shutdown_connections 2024-03-24T00:19:11.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.254+0000 7f597568a640 1 -- 172.21.15.78:0/2734379494 wait complete. 2024-03-24T00:19:11.266 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-24T00:19:11.266 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:19:11.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.368+0000 7f7d171c4640 1 -- 172.21.15.78:0/3653188372 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d10104a70 msgr2=0x7f7d10106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:11.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.368+0000 7f7d171c4640 1 --2- 172.21.15.78:0/3653188372 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d10104a70 0x7f7d10106e60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7d04009900 tx=0x7f7d0402bc30 comp rx=0 tx=0).stop 2024-03-24T00:19:11.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.368+0000 7f7d171c4640 1 -- 172.21.15.78:0/3653188372 shutdown_connections 2024-03-24T00:19:11.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.368+0000 7f7d171c4640 1 --2- 172.21.15.78:0/3653188372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d101073a0 0x7f7d101097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.368+0000 7f7d171c4640 1 --2- 172.21.15.78:0/3653188372 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d10104a70 0x7f7d10106e60 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.368+0000 7f7d171c4640 1 --2- 172.21.15.78:0/3653188372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d10102140 0x7f7d10104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.368+0000 7f7d171c4640 1 -- 172.21.15.78:0/3653188372 >> 172.21.15.78:0/3653188372 conn(0x7f7d100fbb30 msgr2=0x7f7d100fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:11.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.369+0000 7f7d171c4640 1 -- 172.21.15.78:0/3653188372 shutdown_connections 2024-03-24T00:19:11.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.369+0000 7f7d171c4640 1 -- 172.21.15.78:0/3653188372 wait complete. 2024-03-24T00:19:11.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.369+0000 7f7d171c4640 1 Processor -- start 2024-03-24T00:19:11.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.369+0000 7f7d171c4640 1 -- start start 2024-03-24T00:19:11.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.369+0000 7f7d171c4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d10102140 0x7f7d10197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:11.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.369+0000 7f7d171c4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d10104a70 0x7f7d10197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:11.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.369+0000 7f7d171c4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d101073a0 0x7f7d1019efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:11.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.369+0000 7f7d171c4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1010d350 con 0x7f7d10102140 2024-03-24T00:19:11.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.369+0000 7f7d171c4640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1010d1d0 con 0x7f7d101073a0 2024-03-24T00:19:11.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.369+0000 7f7d171c4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1010d4d0 con 0x7f7d10104a70 2024-03-24T00:19:11.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.369+0000 7f7d14f60640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d10102140 0x7f7d10197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:11.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.369+0000 7f7d14f60640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d10102140 0x7f7d10197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58418/0 (socket says 172.21.15.78:58418) 2024-03-24T00:19:11.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.369+0000 7f7d14f60640 1 -- 172.21.15.78:0/651746524 learned_addr learned my addr 172.21.15.78:0/651746524 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:19:11.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.369+0000 7f7d0ffff640 1 --2- 172.21.15.78:0/651746524 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d10104a70 0x7f7d10197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:11.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.369+0000 7f7d15761640 1 --2- 172.21.15.78:0/651746524 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d101073a0 0x7f7d1019efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:11.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.370+0000 7f7d14f60640 1 -- 172.21.15.78:0/651746524 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d10104a70 msgr2=0x7f7d10197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:11.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.370+0000 7f7d14f60640 1 --2- 172.21.15.78:0/651746524 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d10104a70 0x7f7d10197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.370+0000 7f7d14f60640 1 -- 172.21.15.78:0/651746524 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d101073a0 msgr2=0x7f7d1019efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:11.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.370+0000 7f7d14f60640 1 --2- 172.21.15.78:0/651746524 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d101073a0 0x7f7d1019efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.370+0000 7f7d14f60640 1 -- 172.21.15.78:0/651746524 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d1019f640 con 0x7f7d10102140 2024-03-24T00:19:11.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.370+0000 7f7d14f60640 1 --2- 172.21.15.78:0/651746524 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d10102140 0x7f7d10197a20 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7d0000e9f0 tx=0x7f7d0000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:11.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.370+0000 7f7d0dffb640 1 -- 172.21.15.78:0/651746524 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0000cd80 con 0x7f7d10102140 2024-03-24T00:19:11.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.370+0000 7f7d171c4640 1 -- 172.21.15.78:0/651746524 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d1019f930 con 0x7f7d10102140 2024-03-24T00:19:11.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.370+0000 7f7d0dffb640 1 -- 172.21.15.78:0/651746524 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d00005500 con 0x7f7d10102140 2024-03-24T00:19:11.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.370+0000 7f7d0dffb640 1 -- 172.21.15.78:0/651746524 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d00018630 con 0x7f7d10102140 2024-03-24T00:19:11.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.370+0000 7f7d171c4640 1 -- 172.21.15.78:0/651746524 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d1019fe00 con 0x7f7d10102140 2024-03-24T00:19:11.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.371+0000 7f7d171c4640 1 -- 172.21.15.78:0/651746524 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d100fd620 con 0x7f7d10102140 2024-03-24T00:19:11.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.372+0000 7f7d0dffb640 1 -- 172.21.15.78:0/651746524 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7d000046c0 con 0x7f7d10102140 2024-03-24T00:19:11.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.372+0000 7f7d0dffb640 1 --2- 172.21.15.78:0/651746524 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f7cf80a2800 0x7f7cf80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:11.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.373+0000 7f7d0ffff640 1 --2- 172.21.15.78:0/651746524 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f7cf80a2800 0x7f7cf80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:11.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.373+0000 7f7d0dffb640 1 -- 172.21.15.78:0/651746524 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f7d000cf1a0 con 0x7f7d10102140 2024-03-24T00:19:11.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.373+0000 7f7d0ffff640 1 --2- 172.21.15.78:0/651746524 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f7cf80a2800 0x7f7cf80a4cc0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7d04000c00 tx=0x7f7d04005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:11.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.377+0000 7f7d0dffb640 1 -- 172.21.15.78:0/651746524 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7d0009c000 con 0x7f7d10102140 2024-03-24T00:19:11.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.395+0000 7f7d0dffb640 1 -- 172.21.15.78:0/651746524 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f7d00097f70 con 0x7f7d10102140 2024-03-24T00:19:11.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.553+0000 7f7d171c4640 1 -- 172.21.15.78:0/651746524 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7d10061ae0 con 0x7f7cf80a2800 2024-03-24T00:19:11.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.563+0000 7f7d0dffb640 1 -- 172.21.15.78:0/651746524 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340247 (secure 0 0 0) 0x7f7d10061ae0 con 0x7f7cf80a2800 2024-03-24T00:19:11.567 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:19:11.569 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-03-24T00:19:11.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.566+0000 7f7d171c4640 1 -- 172.21.15.78:0/651746524 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f7cf80a2800 msgr2=0x7f7cf80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:11.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.566+0000 7f7d171c4640 1 --2- 172.21.15.78:0/651746524 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f7cf80a2800 0x7f7cf80a4cc0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7d04000c00 tx=0x7f7d04005fd0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.566+0000 7f7d171c4640 1 -- 172.21.15.78:0/651746524 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d10102140 msgr2=0x7f7d10197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:11.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.566+0000 7f7d171c4640 1 --2- 172.21.15.78:0/651746524 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d10102140 0x7f7d10197a20 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7d0000e9f0 tx=0x7f7d0000edf0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.566+0000 7f7d171c4640 1 -- 172.21.15.78:0/651746524 shutdown_connections 2024-03-24T00:19:11.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.566+0000 7f7d171c4640 1 --2- 172.21.15.78:0/651746524 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f7cf80a2800 0x7f7cf80a4cc0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.566+0000 7f7d171c4640 1 --2- 172.21.15.78:0/651746524 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d10104a70 0x7f7d10197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.566+0000 7f7d171c4640 1 --2- 172.21.15.78:0/651746524 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d101073a0 0x7f7d1019efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.566+0000 7f7d171c4640 1 --2- 172.21.15.78:0/651746524 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d10102140 0x7f7d10197a20 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.566+0000 7f7d171c4640 1 -- 172.21.15.78:0/651746524 >> 172.21.15.78:0/651746524 conn(0x7f7d100fbb30 msgr2=0x7f7d10102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:11.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.567+0000 7f7d171c4640 1 -- 172.21.15.78:0/651746524 shutdown_connections 2024-03-24T00:19:11.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.567+0000 7f7d171c4640 1 -- 172.21.15.78:0/651746524 wait complete. 2024-03-24T00:19:11.576 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:11 smithi078 ceph-mon[32429]: from='client.24670 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:19:11.576 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:11 smithi078 ceph-mon[32429]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:19:11.577 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:11 smithi078 ceph-mon[32429]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:19:11.577 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:11 smithi078 ceph-mon[32429]: pgmap v304: 129 pgs: 129 active+clean; 582 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-24T00:19:11.577 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:11 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2733468502' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:19:11.577 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:11 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2734379494' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:19:11.578 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2024-03-24T00:19:10.284693+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595770,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":142,"num_read_kb":127,"num_write":274,"num_write_kb":2965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":212,"ondisk_log_size":212,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":408360,"kb_used_data":13424,"kb_used_omap":18,"kb_used_meta":394797,"kb_avail":1124533464,"statfs":{"total":1151940427776,"available":1151522267136,"internally_reserved":0,"allocated":13746176,"data_stored":7533235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":404272444},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2992,"num_objects":18,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":6,"num_read_kb":8,"num_write":48,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003518"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.319135+0000","last_change":"2024-03-24T00:18:43.890863+0000","last_active":"2024-03-24T00:18:51.319135+0000","last_peered":"2024-03-24T00:18:51.319135+0000","last_clean":"2024-03-24T00:18:51.319135+0000","last_became_active":"2024-03-24T00:18:43.890503+0000","last_became_peered":"2024-03-24T00:18:43.890503+0000","last_unstale":"2024-03-24T00:18:51.319135+0000","last_undegraded":"2024-03-24T00:18:51.319135+0000","last_fullsized":"2024-03-24T00:18:51.319135+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:33:10.518835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.401790+0000","last_change":"2024-03-24T00:18:43.882039+0000","last_active":"2024-03-24T00:18:51.401790+0000","last_peered":"2024-03-24T00:18:51.401790+0000","last_clean":"2024-03-24T00:18:51.401790+0000","last_became_active":"2024-03-24T00:18:43.881704+0000","last_became_peered":"2024-03-24T00:18:43.881704+0000","last_unstale":"2024-03-24T00:18:51.401790+0000","last_undegraded":"2024-03-24T00:18:51.401790+0000","last_fullsized":"2024-03-24T00:18:51.401790+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:04:46.386032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.401666+0000","last_change":"2024-03-24T00:18:43.888649+0000","last_active":"2024-03-24T00:18:51.401666+0000","last_peered":"2024-03-24T00:18:51.401666+0000","last_clean":"2024-03-24T00:18:51.401666+0000","last_became_active":"2024-03-24T00:18:43.888339+0000","last_became_peered":"2024-03-24T00:18:43.888339+0000","last_unstale":"2024-03-24T00:18:51.401666+0000","last_undegraded":"2024-03-24T00:18:51.401666+0000","last_fullsized":"2024-03-24T00:18:51.401666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:28:35.787820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.627849+0000","last_change":"2024-03-24T00:18:43.893347+0000","last_active":"2024-03-24T00:18:51.627849+0000","last_peered":"2024-03-24T00:18:51.627849+0000","last_clean":"2024-03-24T00:18:51.627849+0000","last_became_active":"2024-03-24T00:18:43.893030+0000","last_became_peered":"2024-03-24T00:18:43.893030+0000","last_unstale":"2024-03-24T00:18:51.627849+0000","last_undegraded":"2024-03-24T00:18:51.627849+0000","last_fullsized":"2024-03-24T00:18:51.627849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:51:06.288550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.374069+0000","last_change":"2024-03-24T00:18:43.890677+0000","last_active":"2024-03-24T00:18:51.374069+0000","last_peered":"2024-03-24T00:18:51.374069+0000","last_clean":"2024-03-24T00:18:51.374069+0000","last_became_active":"2024-03-24T00:18:43.887161+0000","last_became_peered":"2024-03-24T00:18:43.887161+0000","last_unstale":"2024-03-24T00:18:51.374069+0000","last_undegraded":"2024-03-24T00:18:51.374069+0000","last_fullsized":"2024-03-24T00:18:51.374069+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:09:32.160568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.870048+0000","last_change":"2024-03-24T00:18:43.887293+0000","last_active":"2024-03-24T00:18:48.870048+0000","last_peered":"2024-03-24T00:18:48.870048+0000","last_clean":"2024-03-24T00:18:48.870048+0000","last_became_active":"2024-03-24T00:18:43.886961+0000","last_became_peered":"2024-03-24T00:18:43.886961+0000","last_unstale":"2024-03-24T00:18:48.870048+0000","last_undegraded":"2024-03-24T00:18:48.870048+0000","last_fullsized":"2024-03-24T00:18:48.870048+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:48:14.312376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.195112+0000","last_change":"2024-03-24T00:18:43.876014+0000","last_active":"2024-03-24T00:18:51.195112+0000","last_peered":"2024-03-24T00:18:51.195112+0000","last_clean":"2024-03-24T00:18:51.195112+0000","last_became_active":"2024-03-24T00:18:43.875619+0000","last_became_peered":"2024-03-24T00:18:43.875619+0000","last_unstale":"2024-03-24T00:18:51.195112+0000","last_undegraded":"2024-03-24T00:18:51.195112+0000","last_fullsized":"2024-03-24T00:18:51.195112+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:10:26.707681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.392797+0000","last_change":"2024-03-24T00:18:43.894851+0000","last_active":"2024-03-24T00:18:51.392797+0000","last_peered":"2024-03-24T00:18:51.392797+0000","last_clean":"2024-03-24T00:18:51.392797+0000","last_became_active":"2024-03-24T00:18:43.894364+0000","last_became_peered":"2024-03-24T00:18:43.894364+0000","last_unstale":"2024-03-24T00:18:51.392797+0000","last_undegraded":"2024-03-24T00:18:51.392797+0000","last_fullsized":"2024-03-24T00:18:51.392797+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:52:01.538415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:49.904156+0000","last_change":"2024-03-24T00:18:43.898453+0000","last_active":"2024-03-24T00:18:49.904156+0000","last_peered":"2024-03-24T00:18:49.904156+0000","last_clean":"2024-03-24T00:18:49.904156+0000","last_became_active":"2024-03-24T00:18:43.898118+0000","last_became_peered":"2024-03-24T00:18:43.898118+0000","last_unstale":"2024-03-24T00:18:49.904156+0000","last_undegraded":"2024-03-24T00:18:49.904156+0000","last_fullsized":"2024-03-24T00:18:49.904156+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:18:00.478010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.870346+0000","last_change":"2024-03-24T00:18:43.894133+0000","last_active":"2024-03-24T00:18:48.870346+0000","last_peered":"2024-03-24T00:18:48.870346+0000","last_clean":"2024-03-24T00:18:48.870346+0000","last_became_active":"2024-03-24T00:18:43.893777+0000","last_became_peered":"2024-03-24T00:18:43.893777+0000","last_unstale":"2024-03-24T00:18:48.870346+0000","last_undegraded":"2024-03-24T00:18:48.870346+0000","last_fullsized":"2024-03-24T00:18:48.870346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:46:33.029934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.195505+0000","last_change":"2024-03-24T00:18:43.888024+0000","last_active":"2024-03-24T00:18:51.195505+0000","last_peered":"2024-03-24T00:18:51.195505+0000","last_clean":"2024-03-24T00:18:51.195505+0000","last_became_active":"2024-03-24T00:18:43.887430+0000","last_became_peered":"2024-03-24T00:18:43.887430+0000","last_unstale":"2024-03-24T00:18:51.195505+0000","last_undegraded":"2024-03-24T00:18:51.195505+0000","last_fullsized":"2024-03-24T00:18:51.195505+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:54:29.299788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.627764+0000","last_change":"2024-03-24T00:18:43.889240+0000","last_active":"2024-03-24T00:18:51.627764+0000","last_peered":"2024-03-24T00:18:51.627764+0000","last_clean":"2024-03-24T00:18:51.627764+0000","last_became_active":"2024-03-24T00:18:43.888861+0000","last_became_peered":"2024-03-24T00:18:43.888861+0000","last_unstale":"2024-03-24T00:18:51.627764+0000","last_undegraded":"2024-03-24T00:18:51.627764+0000","last_fullsized":"2024-03-24T00:18:51.627764+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:41:56.637532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.446207+0000","last_change":"2024-03-24T00:18:43.873750+0000","last_active":"2024-03-24T00:18:51.446207+0000","last_peered":"2024-03-24T00:18:51.446207+0000","last_clean":"2024-03-24T00:18:51.446207+0000","last_became_active":"2024-03-24T00:18:43.873406+0000","last_became_peered":"2024-03-24T00:18:43.873406+0000","last_unstale":"2024-03-24T00:18:51.446207+0000","last_undegraded":"2024-03-24T00:18:51.446207+0000","last_fullsized":"2024-03-24T00:18:51.446207+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:53.262272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.401556+0000","last_change":"2024-03-24T00:18:43.883717+0000","last_active":"2024-03-24T00:18:51.401556+0000","last_peered":"2024-03-24T00:18:51.401556+0000","last_clean":"2024-03-24T00:18:51.401556+0000","last_became_active":"2024-03-24T00:18:43.883329+0000","last_became_peered":"2024-03-24T00:18:43.883329+0000","last_unstale":"2024-03-24T00:18:51.401556+0000","last_undegraded":"2024-03-24T00:18:51.401556+0000","last_fullsized":"2024-03-24T00:18:51.401556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:26:15.127054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.401516+0000","last_change":"2024-03-24T00:18:43.890831+0000","last_active":"2024-03-24T00:18:51.401516+0000","last_peered":"2024-03-24T00:18:51.401516+0000","last_clean":"2024-03-24T00:18:51.401516+0000","last_became_active":"2024-03-24T00:18:43.890448+0000","last_became_peered":"2024-03-24T00:18:43.890448+0000","last_unstale":"2024-03-24T00:18:51.401516+0000","last_undegraded":"2024-03-24T00:18:51.401516+0000","last_fullsized":"2024-03-24T00:18:51.401516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:56:13.222271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:49.903663+0000","last_change":"2024-03-24T00:18:43.890309+0000","last_active":"2024-03-24T00:18:49.903663+0000","last_peered":"2024-03-24T00:18:49.903663+0000","last_clean":"2024-03-24T00:18:49.903663+0000","last_became_active":"2024-03-24T00:18:43.889138+0000","last_became_peered":"2024-03-24T00:18:43.889138+0000","last_unstale":"2024-03-24T00:18:49.903663+0000","last_undegraded":"2024-03-24T00:18:49.903663+0000","last_fullsized":"2024-03-24T00:18:49.903663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:13:52.480938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.318987+0000","last_change":"2024-03-24T00:18:43.891633+0000","last_active":"2024-03-24T00:18:51.318987+0000","last_peered":"2024-03-24T00:18:51.318987+0000","last_clean":"2024-03-24T00:18:51.318987+0000","last_became_active":"2024-03-24T00:18:43.891303+0000","last_became_peered":"2024-03-24T00:18:43.891303+0000","last_unstale":"2024-03-24T00:18:51.318987+0000","last_undegraded":"2024-03-24T00:18:51.318987+0000","last_fullsized":"2024-03-24T00:18:51.318987+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:34:01.874142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.869726+0000","last_change":"2024-03-24T00:18:43.887695+0000","last_active":"2024-03-24T00:18:48.869726+0000","last_peered":"2024-03-24T00:18:48.869726+0000","last_clean":"2024-03-24T00:18:48.869726+0000","last_became_active":"2024-03-24T00:18:43.887359+0000","last_became_peered":"2024-03-24T00:18:43.887359+0000","last_unstale":"2024-03-24T00:18:48.869726+0000","last_undegraded":"2024-03-24T00:18:48.869726+0000","last_fullsized":"2024-03-24T00:18:48.869726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:00:50.951362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.872796+0000","last_change":"2024-03-24T00:18:43.895671+0000","last_active":"2024-03-24T00:18:48.872796+0000","last_peered":"2024-03-24T00:18:48.872796+0000","last_clean":"2024-03-24T00:18:48.872796+0000","last_became_active":"2024-03-24T00:18:43.895267+0000","last_became_peered":"2024-03-24T00:18:43.895267+0000","last_unstale":"2024-03-24T00:18:48.872796+0000","last_undegraded":"2024-03-24T00:18:48.872796+0000","last_fullsized":"2024-03-24T00:18:48.872796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:24:48.014768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.872588+0000","last_change":"2024-03-24T00:18:47.896879+0000","last_active":"2024-03-24T00:18:48.872588+0000","last_peered":"2024-03-24T00:18:48.872588+0000","last_clean":"2024-03-24T00:18:48.872588+0000","last_became_active":"2024-03-24T00:18:47.896517+0000","last_became_peered":"2024-03-24T00:18:47.896517+0000","last_unstale":"2024-03-24T00:18:48.872588+0000","last_undegraded":"2024-03-24T00:18:48.872588+0000","last_fullsized":"2024-03-24T00:18:48.872588+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:28:34.331404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:19:08.388688+0000","last_change":"2024-03-24T00:18:43.257103+0000","last_active":"2024-03-24T00:19:08.388688+0000","last_peered":"2024-03-24T00:19:08.388688+0000","last_clean":"2024-03-24T00:19:08.388688+0000","last_became_active":"2024-03-24T00:18:43.256774+0000","last_became_peered":"2024-03-24T00:18:43.256774+0000","last_unstale":"2024-03-24T00:19:08.388688+0000","last_undegraded":"2024-03-24T00:19:08.388688+0000","last_fullsized":"2024-03-24T00:19:08.388688+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:50:05.350855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.393311+0000","last_change":"2024-03-24T00:18:43.892643+0000","last_active":"2024-03-24T00:18:51.393311+0000","last_peered":"2024-03-24T00:18:51.393311+0000","last_clean":"2024-03-24T00:18:51.393311+0000","last_became_active":"2024-03-24T00:18:43.892317+0000","last_became_peered":"2024-03-24T00:18:43.892317+0000","last_unstale":"2024-03-24T00:18:51.393311+0000","last_undegraded":"2024-03-24T00:18:51.393311+0000","last_fullsized":"2024-03-24T00:18:51.393311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:26:28.374587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.627424+0000","last_change":"2024-03-24T00:18:47.895307+0000","last_active":"2024-03-24T00:18:51.627424+0000","last_peered":"2024-03-24T00:18:51.627424+0000","last_clean":"2024-03-24T00:18:51.627424+0000","last_became_active":"2024-03-24T00:18:47.894988+0000","last_became_peered":"2024-03-24T00:18:47.894988+0000","last_unstale":"2024-03-24T00:18:51.627424+0000","last_undegraded":"2024-03-24T00:18:51.627424+0000","last_fullsized":"2024-03-24T00:18:51.627424+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:48:41.068076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.393462+0000","last_change":"2024-03-24T00:18:43.255861+0000","last_active":"2024-03-24T00:18:51.393462+0000","last_peered":"2024-03-24T00:18:51.393462+0000","last_clean":"2024-03-24T00:18:51.393462+0000","last_became_active":"2024-03-24T00:18:43.255529+0000","last_became_peered":"2024-03-24T00:18:43.255529+0000","last_unstale":"2024-03-24T00:18:51.393462+0000","last_undegraded":"2024-03-24T00:18:51.393462+0000","last_fullsized":"2024-03-24T00:18:51.393462+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:43:19.617673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.401851+0000","last_change":"2024-03-24T00:18:43.886651+0000","last_active":"2024-03-24T00:18:51.401851+0000","last_peered":"2024-03-24T00:18:51.401851+0000","last_clean":"2024-03-24T00:18:51.401851+0000","last_became_active":"2024-03-24T00:18:43.886341+0000","last_became_peered":"2024-03-24T00:18:43.886341+0000","last_unstale":"2024-03-24T00:18:51.401851+0000","last_undegraded":"2024-03-24T00:18:51.401851+0000","last_fullsized":"2024-03-24T00:18:51.401851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:14:11.993717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.627639+0000","last_change":"2024-03-24T00:18:47.897123+0000","last_active":"2024-03-24T00:18:51.627639+0000","last_peered":"2024-03-24T00:18:51.627639+0000","last_clean":"2024-03-24T00:18:51.627639+0000","last_became_active":"2024-03-24T00:18:47.896762+0000","last_became_peered":"2024-03-24T00:18:47.896762+0000","last_unstale":"2024-03-24T00:18:51.627639+0000","last_undegraded":"2024-03-24T00:18:51.627639+0000","last_fullsized":"2024-03-24T00:18:51.627639+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:45:50.639038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.374091+0000","last_change":"2024-03-24T00:18:43.263895+0000","last_active":"2024-03-24T00:18:51.374091+0000","last_peered":"2024-03-24T00:18:51.374091+0000","last_clean":"2024-03-24T00:18:51.374091+0000","last_became_active":"2024-03-24T00:18:43.263557+0000","last_became_peered":"2024-03-24T00:18:43.263557+0000","last_unstale":"2024-03-24T00:18:51.374091+0000","last_undegraded":"2024-03-24T00:18:51.374091+0000","last_fullsized":"2024-03-24T00:18:51.374091+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:44:28.763293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:49.903928+0000","last_change":"2024-03-24T00:18:43.888820+0000","last_active":"2024-03-24T00:18:49.903928+0000","last_peered":"2024-03-24T00:18:49.903928+0000","last_clean":"2024-03-24T00:18:49.903928+0000","last_became_active":"2024-03-24T00:18:43.888382+0000","last_became_peered":"2024-03-24T00:18:43.888382+0000","last_unstale":"2024-03-24T00:18:49.903928+0000","last_undegraded":"2024-03-24T00:18:49.903928+0000","last_fullsized":"2024-03-24T00:18:49.903928+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:04.097113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.627580+0000","last_change":"2024-03-24T00:18:47.896443+0000","last_active":"2024-03-24T00:18:51.627580+0000","last_peered":"2024-03-24T00:18:51.627580+0000","last_clean":"2024-03-24T00:18:51.627580+0000","last_became_active":"2024-03-24T00:18:47.895792+0000","last_became_peered":"2024-03-24T00:18:47.895792+0000","last_unstale":"2024-03-24T00:18:51.627580+0000","last_undegraded":"2024-03-24T00:18:51.627580+0000","last_fullsized":"2024-03-24T00:18:51.627580+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:59:42.360902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.373915+0000","last_change":"2024-03-24T00:18:43.261085+0000","last_active":"2024-03-24T00:18:51.373915+0000","last_peered":"2024-03-24T00:18:51.373915+0000","last_clean":"2024-03-24T00:18:51.373915+0000","last_became_active":"2024-03-24T00:18:43.260724+0000","last_became_peered":"2024-03-24T00:18:43.260724+0000","last_unstale":"2024-03-24T00:18:51.373915+0000","last_undegraded":"2024-03-24T00:18:51.373915+0000","last_fullsized":"2024-03-24T00:18:51.373915+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:05:20.992776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.622130+0000","last_change":"2024-03-24T00:18:43.892891+0000","last_active":"2024-03-24T00:18:51.622130+0000","last_peered":"2024-03-24T00:18:51.622130+0000","last_clean":"2024-03-24T00:18:51.622130+0000","last_became_active":"2024-03-24T00:18:43.892430+0000","last_became_peered":"2024-03-24T00:18:43.892430+0000","last_unstale":"2024-03-24T00:18:51.622130+0000","last_undegraded":"2024-03-24T00:18:51.622130+0000","last_fullsized":"2024-03-24T00:18:51.622130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:30:24.926031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.873061+0000","last_change":"2024-03-24T00:18:47.889119+0000","last_active":"2024-03-24T00:18:48.873061+0000","last_peered":"2024-03-24T00:18:48.873061+0000","last_clean":"2024-03-24T00:18:48.873061+0000","last_became_active":"2024-03-24T00:18:47.888620+0000","last_became_peered":"2024-03-24T00:18:47.888620+0000","last_unstale":"2024-03-24T00:18:48.873061+0000","last_undegraded":"2024-03-24T00:18:48.873061+0000","last_fullsized":"2024-03-24T00:18:48.873061+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:51.483286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.621605+0000","last_change":"2024-03-24T00:18:43.251824+0000","last_active":"2024-03-24T00:18:51.621605+0000","last_peered":"2024-03-24T00:18:51.621605+0000","last_clean":"2024-03-24T00:18:51.621605+0000","last_became_active":"2024-03-24T00:18:43.251458+0000","last_became_peered":"2024-03-24T00:18:43.251458+0000","last_unstale":"2024-03-24T00:18:51.621605+0000","last_undegraded":"2024-03-24T00:18:51.621605+0000","last_fullsized":"2024-03-24T00:18:51.621605+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:13:02.303670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.374289+0000","last_change":"2024-03-24T00:18:43.891289+0000","last_active":"2024-03-24T00:18:51.374289+0000","last_peered":"2024-03-24T00:18:51.374289+0000","last_clean":"2024-03-24T00:18:51.374289+0000","last_became_active":"2024-03-24T00:18:43.890725+0000","last_became_peered":"2024-03-24T00:18:43.890725+0000","last_unstale":"2024-03-24T00:18:51.374289+0000","last_undegraded":"2024-03-24T00:18:51.374289+0000","last_fullsized":"2024-03-24T00:18:51.374289+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:37:41.096520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.872677+0000","last_change":"2024-03-24T00:18:47.896175+0000","last_active":"2024-03-24T00:18:48.872677+0000","last_peered":"2024-03-24T00:18:48.872677+0000","last_clean":"2024-03-24T00:18:48.872677+0000","last_became_active":"2024-03-24T00:18:47.895723+0000","last_became_peered":"2024-03-24T00:18:47.895723+0000","last_unstale":"2024-03-24T00:18:48.872677+0000","last_undegraded":"2024-03-24T00:18:48.872677+0000","last_fullsized":"2024-03-24T00:18:48.872677+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:41:49.167532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:19:08.387233+0000","last_change":"2024-03-24T00:18:43.258986+0000","last_active":"2024-03-24T00:19:08.387233+0000","last_peered":"2024-03-24T00:19:08.387233+0000","last_clean":"2024-03-24T00:19:08.387233+0000","last_became_active":"2024-03-24T00:18:43.258654+0000","last_became_peered":"2024-03-24T00:18:43.258654+0000","last_unstale":"2024-03-24T00:19:08.387233+0000","last_undegraded":"2024-03-24T00:19:08.387233+0000","last_fullsized":"2024-03-24T00:19:08.387233+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:34:44.339991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.622197+0000","last_change":"2024-03-24T00:18:43.885852+0000","last_active":"2024-03-24T00:18:51.622197+0000","last_peered":"2024-03-24T00:18:51.622197+0000","last_clean":"2024-03-24T00:18:51.622197+0000","last_became_active":"2024-03-24T00:18:43.885538+0000","last_became_peered":"2024-03-24T00:18:43.885538+0000","last_unstale":"2024-03-24T00:18:51.622197+0000","last_undegraded":"2024-03-24T00:18:51.622197+0000","last_fullsized":"2024-03-24T00:18:51.622197+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:42:02.879892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.318919+0000","last_change":"2024-03-24T00:18:47.893230+0000","last_active":"2024-03-24T00:18:51.318919+0000","last_peered":"2024-03-24T00:18:51.318919+0000","last_clean":"2024-03-24T00:18:51.318919+0000","last_became_active":"2024-03-24T00:18:47.892522+0000","last_became_peered":"2024-03-24T00:18:47.892522+0000","last_unstale":"2024-03-24T00:18:51.318919+0000","last_undegraded":"2024-03-24T00:18:51.318919+0000","last_fullsized":"2024-03-24T00:18:51.318919+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:10:11.768154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.445893+0000","last_change":"2024-03-24T00:18:43.262255+0000","last_active":"2024-03-24T00:18:51.445893+0000","last_peered":"2024-03-24T00:18:51.445893+0000","last_clean":"2024-03-24T00:18:51.445893+0000","last_became_active":"2024-03-24T00:18:43.261940+0000","last_became_peered":"2024-03-24T00:18:43.261940+0000","last_unstale":"2024-03-24T00:18:51.445893+0000","last_undegraded":"2024-03-24T00:18:51.445893+0000","last_fullsized":"2024-03-24T00:18:51.445893+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:58:48.789845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.393102+0000","last_change":"2024-03-24T00:18:43.883046+0000","last_active":"2024-03-24T00:18:51.393102+0000","last_peered":"2024-03-24T00:18:51.393102+0000","last_clean":"2024-03-24T00:18:51.393102+0000","last_became_active":"2024-03-24T00:18:43.881405+0000","last_became_peered":"2024-03-24T00:18:43.881405+0000","last_unstale":"2024-03-24T00:18:51.393102+0000","last_undegraded":"2024-03-24T00:18:51.393102+0000","last_fullsized":"2024-03-24T00:18:51.393102+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:21:00.047019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.401934+0000","last_change":"2024-03-24T00:18:47.897630+0000","last_active":"2024-03-24T00:18:51.401934+0000","last_peered":"2024-03-24T00:18:51.401934+0000","last_clean":"2024-03-24T00:18:51.401934+0000","last_became_active":"2024-03-24T00:18:47.897211+0000","last_became_peered":"2024-03-24T00:18:47.897211+0000","last_unstale":"2024-03-24T00:18:51.401934+0000","last_undegraded":"2024-03-24T00:18:51.401934+0000","last_fullsized":"2024-03-24T00:18:51.401934+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:22:48.762870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.621544+0000","last_change":"2024-03-24T00:18:43.262134+0000","last_active":"2024-03-24T00:18:51.621544+0000","last_peered":"2024-03-24T00:18:51.621544+0000","last_clean":"2024-03-24T00:18:51.621544+0000","last_became_active":"2024-03-24T00:18:43.261783+0000","last_became_peered":"2024-03-24T00:18:43.261783+0000","last_unstale":"2024-03-24T00:18:51.621544+0000","last_undegraded":"2024-03-24T00:18:51.621544+0000","last_fullsized":"2024-03-24T00:18:51.621544+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:16:31.060204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.869298+0000","last_change":"2024-03-24T00:18:43.880532+0000","last_active":"2024-03-24T00:18:48.869298+0000","last_peered":"2024-03-24T00:18:48.869298+0000","last_clean":"2024-03-24T00:18:48.869298+0000","last_became_active":"2024-03-24T00:18:43.880198+0000","last_became_peered":"2024-03-24T00:18:43.880198+0000","last_unstale":"2024-03-24T00:18:48.869298+0000","last_undegraded":"2024-03-24T00:18:48.869298+0000","last_fullsized":"2024-03-24T00:18:48.869298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:33:16.710842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.393392+0000","last_change":"2024-03-24T00:18:47.893607+0000","last_active":"2024-03-24T00:18:51.393392+0000","last_peered":"2024-03-24T00:18:51.393392+0000","last_clean":"2024-03-24T00:18:51.393392+0000","last_became_active":"2024-03-24T00:18:47.893026+0000","last_became_peered":"2024-03-24T00:18:47.893026+0000","last_unstale":"2024-03-24T00:18:51.393392+0000","last_undegraded":"2024-03-24T00:18:51.393392+0000","last_fullsized":"2024-03-24T00:18:51.393392+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:37:58.994146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:19:07.379066+0000","last_change":"2024-03-24T00:18:43.257520+0000","last_active":"2024-03-24T00:19:07.379066+0000","last_peered":"2024-03-24T00:19:07.379066+0000","last_clean":"2024-03-24T00:19:07.379066+0000","last_became_active":"2024-03-24T00:18:43.257175+0000","last_became_peered":"2024-03-24T00:18:43.257175+0000","last_unstale":"2024-03-24T00:19:07.379066+0000","last_undegraded":"2024-03-24T00:19:07.379066+0000","last_fullsized":"2024-03-24T00:19:07.379066+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:52:45.244983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:49.904115+0000","last_change":"2024-03-24T00:18:43.887739+0000","last_active":"2024-03-24T00:18:49.904115+0000","last_peered":"2024-03-24T00:18:49.904115+0000","last_clean":"2024-03-24T00:18:49.904115+0000","last_became_active":"2024-03-24T00:18:43.887379+0000","last_became_peered":"2024-03-24T00:18:43.887379+0000","last_unstale":"2024-03-24T00:18:49.904115+0000","last_undegraded":"2024-03-24T00:18:49.904115+0000","last_fullsized":"2024-03-24T00:18:49.904115+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:10:50.384759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.869337+0000","last_change":"2024-03-24T00:18:47.890827+0000","last_active":"2024-03-24T00:18:48.869337+0000","last_peered":"2024-03-24T00:18:48.869337+0000","last_clean":"2024-03-24T00:18:48.869337+0000","last_became_active":"2024-03-24T00:18:47.890399+0000","last_became_peered":"2024-03-24T00:18:47.890399+0000","last_unstale":"2024-03-24T00:18:48.869337+0000","last_undegraded":"2024-03-24T00:18:48.869337+0000","last_fullsized":"2024-03-24T00:18:48.869337+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:46:54.575468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:19:07.379186+0000","last_change":"2024-03-24T00:18:43.248132+0000","last_active":"2024-03-24T00:19:07.379186+0000","last_peered":"2024-03-24T00:19:07.379186+0000","last_clean":"2024-03-24T00:19:07.379186+0000","last_became_active":"2024-03-24T00:18:43.247772+0000","last_became_peered":"2024-03-24T00:18:43.247772+0000","last_unstale":"2024-03-24T00:19:07.379186+0000","last_undegraded":"2024-03-24T00:19:07.379186+0000","last_fullsized":"2024-03-24T00:19:07.379186+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:09.330155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.401794+0000","last_change":"2024-03-24T00:18:43.889920+0000","last_active":"2024-03-24T00:18:51.401794+0000","last_peered":"2024-03-24T00:18:51.401794+0000","last_clean":"2024-03-24T00:18:51.401794+0000","last_became_active":"2024-03-24T00:18:43.889593+0000","last_became_peered":"2024-03-24T00:18:43.889593+0000","last_unstale":"2024-03-24T00:18:51.401794+0000","last_undegraded":"2024-03-24T00:18:51.401794+0000","last_fullsized":"2024-03-24T00:18:51.401794+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:48:41.101019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.627755+0000","last_change":"2024-03-24T00:18:47.893133+0000","last_active":"2024-03-24T00:18:51.627755+0000","last_peered":"2024-03-24T00:18:51.627755+0000","last_clean":"2024-03-24T00:18:51.627755+0000","last_became_active":"2024-03-24T00:18:47.892731+0000","last_became_peered":"2024-03-24T00:18:47.892731+0000","last_unstale":"2024-03-24T00:18:51.627755+0000","last_undegraded":"2024-03-24T00:18:51.627755+0000","last_fullsized":"2024-03-24T00:18:51.627755+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:51:09.539852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.621608+0000","last_change":"2024-03-24T00:18:43.257697+0000","last_active":"2024-03-24T00:18:51.621608+0000","last_peered":"2024-03-24T00:18:51.621608+0000","last_clean":"2024-03-24T00:18:51.621608+0000","last_became_active":"2024-03-24T00:18:43.257334+0000","last_became_peered":"2024-03-24T00:18:43.257334+0000","last_unstale":"2024-03-24T00:18:51.621608+0000","last_undegraded":"2024-03-24T00:18:51.621608+0000","last_fullsized":"2024-03-24T00:18:51.621608+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:35:22.817064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:49.904020+0000","last_change":"2024-03-24T00:18:43.896238+0000","last_active":"2024-03-24T00:18:49.904020+0000","last_peered":"2024-03-24T00:18:49.904020+0000","last_clean":"2024-03-24T00:18:49.904020+0000","last_became_active":"2024-03-24T00:18:43.895792+0000","last_became_peered":"2024-03-24T00:18:43.895792+0000","last_unstale":"2024-03-24T00:18:49.904020+0000","last_undegraded":"2024-03-24T00:18:49.904020+0000","last_fullsized":"2024-03-24T00:18:49.904020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:55:11.865771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.869305+0000","last_change":"2024-03-24T00:18:47.891407+0000","last_active":"2024-03-24T00:18:48.869305+0000","last_peered":"2024-03-24T00:18:48.869305+0000","last_clean":"2024-03-24T00:18:48.869305+0000","last_became_active":"2024-03-24T00:18:47.891042+0000","last_became_peered":"2024-03-24T00:18:47.891042+0000","last_unstale":"2024-03-24T00:18:48.869305+0000","last_undegraded":"2024-03-24T00:18:48.869305+0000","last_fullsized":"2024-03-24T00:18:48.869305+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:14:02.939199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'11","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:19:08.880371+0000","last_change":"2024-03-24T00:18:43.258742+0000","last_active":"2024-03-24T00:19:08.880371+0000","last_peered":"2024-03-24T00:19:08.880371+0000","last_clean":"2024-03-24T00:19:08.880371+0000","last_became_active":"2024-03-24T00:18:43.258412+0000","last_became_peered":"2024-03-24T00:18:43.258412+0000","last_unstale":"2024-03-24T00:19:08.880371+0000","last_undegraded":"2024-03-24T00:19:08.880371+0000","last_fullsized":"2024-03-24T00:19:08.880371+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:30:49.364147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":12,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.374582+0000","last_change":"2024-03-24T00:18:43.893740+0000","last_active":"2024-03-24T00:18:51.374582+0000","last_peered":"2024-03-24T00:18:51.374582+0000","last_clean":"2024-03-24T00:18:51.374582+0000","last_became_active":"2024-03-24T00:18:43.893277+0000","last_became_peered":"2024-03-24T00:18:43.893277+0000","last_unstale":"2024-03-24T00:18:51.374582+0000","last_undegraded":"2024-03-24T00:18:51.374582+0000","last_fullsized":"2024-03-24T00:18:51.374582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:21:47.040629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.622387+0000","last_change":"2024-03-24T00:18:47.898769+0000","last_active":"2024-03-24T00:18:51.622387+0000","last_peered":"2024-03-24T00:18:51.622387+0000","last_clean":"2024-03-24T00:18:51.622387+0000","last_became_active":"2024-03-24T00:18:47.898214+0000","last_became_peered":"2024-03-24T00:18:47.898214+0000","last_unstale":"2024-03-24T00:18:51.622387+0000","last_undegraded":"2024-03-24T00:18:51.622387+0000","last_fullsized":"2024-03-24T00:18:51.622387+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:24:42.022429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:19:08.387449+0000","last_change":"2024-03-24T00:18:43.262532+0000","last_active":"2024-03-24T00:19:08.387449+0000","last_peered":"2024-03-24T00:19:08.387449+0000","last_clean":"2024-03-24T00:19:08.387449+0000","last_became_active":"2024-03-24T00:18:43.262211+0000","last_became_peered":"2024-03-24T00:18:43.262211+0000","last_unstale":"2024-03-24T00:19:08.387449+0000","last_undegraded":"2024-03-24T00:19:08.387449+0000","last_fullsized":"2024-03-24T00:19:08.387449+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:23:22.862495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.446229+0000","last_change":"2024-03-24T00:18:43.881691+0000","last_active":"2024-03-24T00:18:51.446229+0000","last_peered":"2024-03-24T00:18:51.446229+0000","last_clean":"2024-03-24T00:18:51.446229+0000","last_became_active":"2024-03-24T00:18:43.881386+0000","last_became_peered":"2024-03-24T00:18:43.881386+0000","last_unstale":"2024-03-24T00:18:51.446229+0000","last_undegraded":"2024-03-24T00:18:51.446229+0000","last_fullsized":"2024-03-24T00:18:51.446229+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:33:24.808315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.318722+0000","last_change":"2024-03-24T00:18:47.883340+0000","last_active":"2024-03-24T00:18:51.318722+0000","last_peered":"2024-03-24T00:18:51.318722+0000","last_clean":"2024-03-24T00:18:51.318722+0000","last_became_active":"2024-03-24T00:18:47.882934+0000","last_became_peered":"2024-03-24T00:18:47.882934+0000","last_unstale":"2024-03-24T00:18:51.318722+0000","last_undegraded":"2024-03-24T00:18:51.318722+0000","last_fullsized":"2024-03-24T00:18:51.318722+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:30:56.659709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.621538+0000","last_change":"2024-03-24T00:18:43.253444+0000","last_active":"2024-03-24T00:18:51.621538+0000","last_peered":"2024-03-24T00:18:51.621538+0000","last_clean":"2024-03-24T00:18:51.621538+0000","last_became_active":"2024-03-24T00:18:43.252121+0000","last_became_peered":"2024-03-24T00:18:43.252121+0000","last_unstale":"2024-03-24T00:18:51.621538+0000","last_undegraded":"2024-03-24T00:18:51.621538+0000","last_fullsized":"2024-03-24T00:18:51.621538+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:44:48.663995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.195190+0000","last_change":"2024-03-24T00:18:43.890286+0000","last_active":"2024-03-24T00:18:51.195190+0000","last_peered":"2024-03-24T00:18:51.195190+0000","last_clean":"2024-03-24T00:18:51.195190+0000","last_became_active":"2024-03-24T00:18:43.889918+0000","last_became_peered":"2024-03-24T00:18:43.889918+0000","last_unstale":"2024-03-24T00:18:51.195190+0000","last_undegraded":"2024-03-24T00:18:51.195190+0000","last_fullsized":"2024-03-24T00:18:51.195190+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:13:50.804306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.869709+0000","last_change":"2024-03-24T00:18:47.897513+0000","last_active":"2024-03-24T00:18:48.869709+0000","last_peered":"2024-03-24T00:18:48.869709+0000","last_clean":"2024-03-24T00:18:48.869709+0000","last_became_active":"2024-03-24T00:18:47.896606+0000","last_became_peered":"2024-03-24T00:18:47.896606+0000","last_unstale":"2024-03-24T00:18:48.869709+0000","last_undegraded":"2024-03-24T00:18:48.869709+0000","last_fullsized":"2024-03-24T00:18:48.869709+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:09:09.389437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.627220+0000","last_change":"2024-03-24T00:18:43.251682+0000","last_active":"2024-03-24T00:18:51.627220+0000","last_peered":"2024-03-24T00:18:51.627220+0000","last_clean":"2024-03-24T00:18:51.627220+0000","last_became_active":"2024-03-24T00:18:43.250953+0000","last_became_peered":"2024-03-24T00:18:43.250953+0000","last_unstale":"2024-03-24T00:18:51.627220+0000","last_undegraded":"2024-03-24T00:18:51.627220+0000","last_fullsized":"2024-03-24T00:18:51.627220+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:22:35.218781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.195197+0000","last_change":"2024-03-24T00:18:43.876334+0000","last_active":"2024-03-24T00:18:51.195197+0000","last_peered":"2024-03-24T00:18:51.195197+0000","last_clean":"2024-03-24T00:18:51.195197+0000","last_became_active":"2024-03-24T00:18:43.875397+0000","last_became_peered":"2024-03-24T00:18:43.875397+0000","last_unstale":"2024-03-24T00:18:51.195197+0000","last_undegraded":"2024-03-24T00:18:51.195197+0000","last_fullsized":"2024-03-24T00:18:51.195197+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:40:20.474055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.446561+0000","last_change":"2024-03-24T00:18:47.897488+0000","last_active":"2024-03-24T00:18:51.446561+0000","last_peered":"2024-03-24T00:18:51.446561+0000","last_clean":"2024-03-24T00:18:51.446561+0000","last_became_active":"2024-03-24T00:18:47.897042+0000","last_became_peered":"2024-03-24T00:18:47.897042+0000","last_unstale":"2024-03-24T00:18:51.446561+0000","last_undegraded":"2024-03-24T00:18:51.446561+0000","last_fullsized":"2024-03-24T00:18:51.446561+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:23:13.071695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.373968+0000","last_change":"2024-03-24T00:18:43.255278+0000","last_active":"2024-03-24T00:18:51.373968+0000","last_peered":"2024-03-24T00:18:51.373968+0000","last_clean":"2024-03-24T00:18:51.373968+0000","last_became_active":"2024-03-24T00:18:43.254943+0000","last_became_peered":"2024-03-24T00:18:43.254943+0000","last_unstale":"2024-03-24T00:18:51.373968+0000","last_undegraded":"2024-03-24T00:18:51.373968+0000","last_fullsized":"2024-03-24T00:18:51.373968+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:09:56.755303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.621966+0000","last_change":"2024-03-24T00:18:43.892620+0000","last_active":"2024-03-24T00:18:51.621966+0000","last_peered":"2024-03-24T00:18:51.621966+0000","last_clean":"2024-03-24T00:18:51.621966+0000","last_became_active":"2024-03-24T00:18:43.892033+0000","last_became_peered":"2024-03-24T00:18:43.892033+0000","last_unstale":"2024-03-24T00:18:51.621966+0000","last_undegraded":"2024-03-24T00:18:51.621966+0000","last_fullsized":"2024-03-24T00:18:51.621966+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:30:19.472861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.622125+0000","last_change":"2024-03-24T00:18:47.895623+0000","last_active":"2024-03-24T00:18:51.622125+0000","last_peered":"2024-03-24T00:18:51.622125+0000","last_clean":"2024-03-24T00:18:51.622125+0000","last_became_active":"2024-03-24T00:18:47.895242+0000","last_became_peered":"2024-03-24T00:18:47.895242+0000","last_unstale":"2024-03-24T00:18:51.622125+0000","last_undegraded":"2024-03-24T00:18:51.622125+0000","last_fullsized":"2024-03-24T00:18:51.622125+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:50:32.775988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.374566+0000","last_change":"2024-03-24T00:18:43.265575+0000","last_active":"2024-03-24T00:18:51.374566+0000","last_peered":"2024-03-24T00:18:51.374566+0000","last_clean":"2024-03-24T00:18:51.374566+0000","last_became_active":"2024-03-24T00:18:43.265265+0000","last_became_peered":"2024-03-24T00:18:43.265265+0000","last_unstale":"2024-03-24T00:18:51.374566+0000","last_undegraded":"2024-03-24T00:18:51.374566+0000","last_fullsized":"2024-03-24T00:18:51.374566+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:15:46.566812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.318780+0000","last_change":"2024-03-24T00:18:43.898574+0000","last_active":"2024-03-24T00:18:51.318780+0000","last_peered":"2024-03-24T00:18:51.318780+0000","last_clean":"2024-03-24T00:18:51.318780+0000","last_became_active":"2024-03-24T00:18:43.898153+0000","last_became_peered":"2024-03-24T00:18:43.898153+0000","last_unstale":"2024-03-24T00:18:51.318780+0000","last_undegraded":"2024-03-24T00:18:51.318780+0000","last_fullsized":"2024-03-24T00:18:51.318780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:50:15.317257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.374214+0000","last_change":"2024-03-24T00:18:47.896336+0000","last_active":"2024-03-24T00:18:51.374214+0000","last_peered":"2024-03-24T00:18:51.374214+0000","last_clean":"2024-03-24T00:18:51.374214+0000","last_became_active":"2024-03-24T00:18:47.895939+0000","last_became_peered":"2024-03-24T00:18:47.895939+0000","last_unstale":"2024-03-24T00:18:51.374214+0000","last_undegraded":"2024-03-24T00:18:51.374214+0000","last_fullsized":"2024-03-24T00:18:51.374214+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:54:03.195827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.869425+0000","last_change":"2024-03-24T00:18:43.253081+0000","last_active":"2024-03-24T00:18:48.869425+0000","last_peered":"2024-03-24T00:18:48.869425+0000","last_clean":"2024-03-24T00:18:48.869425+0000","last_became_active":"2024-03-24T00:18:43.252415+0000","last_became_peered":"2024-03-24T00:18:43.252415+0000","last_unstale":"2024-03-24T00:18:48.869425+0000","last_undegraded":"2024-03-24T00:18:48.869425+0000","last_fullsized":"2024-03-24T00:18:48.869425+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:06:45.200233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.870227+0000","last_change":"2024-03-24T00:18:43.885820+0000","last_active":"2024-03-24T00:18:48.870227+0000","last_peered":"2024-03-24T00:18:48.870227+0000","last_clean":"2024-03-24T00:18:48.870227+0000","last_became_active":"2024-03-24T00:18:43.885485+0000","last_became_peered":"2024-03-24T00:18:43.885485+0000","last_unstale":"2024-03-24T00:18:48.870227+0000","last_undegraded":"2024-03-24T00:18:48.870227+0000","last_fullsized":"2024-03-24T00:18:48.870227+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:09:12.286000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.446402+0000","last_change":"2024-03-24T00:18:47.899054+0000","last_active":"2024-03-24T00:18:51.446402+0000","last_peered":"2024-03-24T00:18:51.446402+0000","last_clean":"2024-03-24T00:18:51.446402+0000","last_became_active":"2024-03-24T00:18:47.898412+0000","last_became_peered":"2024-03-24T00:18:47.898412+0000","last_unstale":"2024-03-24T00:18:51.446402+0000","last_undegraded":"2024-03-24T00:18:51.446402+0000","last_fullsized":"2024-03-24T00:18:51.446402+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:17:08.273323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.195367+0000","last_change":"2024-03-24T00:18:43.251402+0000","last_active":"2024-03-24T00:18:51.195367+0000","last_peered":"2024-03-24T00:18:51.195367+0000","last_clean":"2024-03-24T00:18:51.195367+0000","last_became_active":"2024-03-24T00:18:43.251055+0000","last_became_peered":"2024-03-24T00:18:43.251055+0000","last_unstale":"2024-03-24T00:18:51.195367+0000","last_undegraded":"2024-03-24T00:18:51.195367+0000","last_fullsized":"2024-03-24T00:18:51.195367+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:07:30.966065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.870122+0000","last_change":"2024-03-24T00:18:43.886594+0000","last_active":"2024-03-24T00:18:48.870122+0000","last_peered":"2024-03-24T00:18:48.870122+0000","last_clean":"2024-03-24T00:18:48.870122+0000","last_became_active":"2024-03-24T00:18:43.886251+0000","last_became_peered":"2024-03-24T00:18:43.886251+0000","last_unstale":"2024-03-24T00:18:48.870122+0000","last_undegraded":"2024-03-24T00:18:48.870122+0000","last_fullsized":"2024-03-24T00:18:48.870122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:55:24.662318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.446529+0000","last_change":"2024-03-24T00:18:47.901352+0000","last_active":"2024-03-24T00:18:51.446529+0000","last_peered":"2024-03-24T00:18:51.446529+0000","last_clean":"2024-03-24T00:18:51.446529+0000","last_became_active":"2024-03-24T00:18:47.900943+0000","last_became_peered":"2024-03-24T00:18:47.900943+0000","last_unstale":"2024-03-24T00:18:51.446529+0000","last_undegraded":"2024-03-24T00:18:51.446529+0000","last_fullsized":"2024-03-24T00:18:51.446529+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:25:29.626277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.622218+0000","last_change":"2024-03-24T00:18:43.885235+0000","last_active":"2024-03-24T00:18:51.622218+0000","last_peered":"2024-03-24T00:18:51.622218+0000","last_clean":"2024-03-24T00:18:51.622218+0000","last_became_active":"2024-03-24T00:18:43.884903+0000","last_became_peered":"2024-03-24T00:18:43.884903+0000","last_unstale":"2024-03-24T00:18:51.622218+0000","last_undegraded":"2024-03-24T00:18:51.622218+0000","last_fullsized":"2024-03-24T00:18:51.622218+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:01:31.929097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.446045+0000","last_change":"2024-03-24T00:18:43.263213+0000","last_active":"2024-03-24T00:18:51.446045+0000","last_peered":"2024-03-24T00:18:51.446045+0000","last_clean":"2024-03-24T00:18:51.446045+0000","last_became_active":"2024-03-24T00:18:43.262668+0000","last_became_peered":"2024-03-24T00:18:43.262668+0000","last_unstale":"2024-03-24T00:18:51.446045+0000","last_undegraded":"2024-03-24T00:18:51.446045+0000","last_fullsized":"2024-03-24T00:18:51.446045+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:18:15.518187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.195369+0000","last_change":"2024-03-24T00:18:47.891607+0000","last_active":"2024-03-24T00:18:51.195369+0000","last_peered":"2024-03-24T00:18:51.195369+0000","last_clean":"2024-03-24T00:18:51.195369+0000","last_became_active":"2024-03-24T00:18:47.891241+0000","last_became_peered":"2024-03-24T00:18:47.891241+0000","last_unstale":"2024-03-24T00:18:51.195369+0000","last_undegraded":"2024-03-24T00:18:51.195369+0000","last_fullsized":"2024-03-24T00:18:51.195369+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:14:29.593767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.393140+0000","last_change":"2024-03-24T00:18:43.894492+0000","last_active":"2024-03-24T00:18:51.393140+0000","last_peered":"2024-03-24T00:18:51.393140+0000","last_clean":"2024-03-24T00:18:51.393140+0000","last_became_active":"2024-03-24T00:18:43.893104+0000","last_became_peered":"2024-03-24T00:18:43.893104+0000","last_unstale":"2024-03-24T00:18:51.393140+0000","last_undegraded":"2024-03-24T00:18:51.393140+0000","last_fullsized":"2024-03-24T00:18:51.393140+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:26:32.130449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:19:07.377260+0000","last_change":"2024-03-24T00:18:43.251344+0000","last_active":"2024-03-24T00:19:07.377260+0000","last_peered":"2024-03-24T00:19:07.377260+0000","last_clean":"2024-03-24T00:19:07.377260+0000","last_became_active":"2024-03-24T00:18:43.250961+0000","last_became_peered":"2024-03-24T00:18:43.250961+0000","last_unstale":"2024-03-24T00:19:07.377260+0000","last_undegraded":"2024-03-24T00:19:07.377260+0000","last_fullsized":"2024-03-24T00:19:07.377260+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:13:38.423465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":356,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.195291+0000","last_change":"2024-03-24T00:17:27.864318+0000","last_active":"2024-03-24T00:18:51.195291+0000","last_peered":"2024-03-24T00:18:51.195291+0000","last_clean":"2024-03-24T00:18:51.195291+0000","last_became_active":"2024-03-24T00:17:27.863875+0000","last_became_peered":"2024-03-24T00:17:27.863875+0000","last_unstale":"2024-03-24T00:18:51.195291+0000","last_undegraded":"2024-03-24T00:18:51.195291+0000","last_fullsized":"2024-03-24T00:18:51.195291+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:14:04.731181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:14:04.731181+0000","last_clean_scrub_stamp":"2024-03-24T00:14:04.731181+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:53:25.036280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.622516+0000","last_change":"2024-03-24T00:18:47.895344+0000","last_active":"2024-03-24T00:18:51.622516+0000","last_peered":"2024-03-24T00:18:51.622516+0000","last_clean":"2024-03-24T00:18:51.622516+0000","last_became_active":"2024-03-24T00:18:47.894694+0000","last_became_peered":"2024-03-24T00:18:47.894694+0000","last_unstale":"2024-03-24T00:18:51.622516+0000","last_undegraded":"2024-03-24T00:18:51.622516+0000","last_fullsized":"2024-03-24T00:18:51.622516+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:16.969143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.446016+0000","last_change":"2024-03-24T00:18:43.890236+0000","last_active":"2024-03-24T00:18:51.446016+0000","last_peered":"2024-03-24T00:18:51.446016+0000","last_clean":"2024-03-24T00:18:51.446016+0000","last_became_active":"2024-03-24T00:18:43.889896+0000","last_became_peered":"2024-03-24T00:18:43.889896+0000","last_unstale":"2024-03-24T00:18:51.446016+0000","last_undegraded":"2024-03-24T00:18:51.446016+0000","last_fullsized":"2024-03-24T00:18:51.446016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:13:19.443445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.195149+0000","last_change":"2024-03-24T00:18:43.260722+0000","last_active":"2024-03-24T00:18:51.195149+0000","last_peered":"2024-03-24T00:18:51.195149+0000","last_clean":"2024-03-24T00:18:51.195149+0000","last_became_active":"2024-03-24T00:18:43.260244+0000","last_became_peered":"2024-03-24T00:18:43.260244+0000","last_unstale":"2024-03-24T00:18:51.195149+0000","last_undegraded":"2024-03-24T00:18:51.195149+0000","last_fullsized":"2024-03-24T00:18:51.195149+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:37:37.838046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.318984+0000","last_change":"2024-03-24T00:18:47.891168+0000","last_active":"2024-03-24T00:18:51.318984+0000","last_peered":"2024-03-24T00:18:51.318984+0000","last_clean":"2024-03-24T00:18:51.318984+0000","last_became_active":"2024-03-24T00:18:47.890713+0000","last_became_peered":"2024-03-24T00:18:47.890713+0000","last_unstale":"2024-03-24T00:18:51.318984+0000","last_undegraded":"2024-03-24T00:18:51.318984+0000","last_fullsized":"2024-03-24T00:18:51.318984+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:20:30.858381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.374297+0000","last_change":"2024-03-24T00:18:43.256638+0000","last_active":"2024-03-24T00:18:51.374297+0000","last_peered":"2024-03-24T00:18:51.374297+0000","last_clean":"2024-03-24T00:18:51.374297+0000","last_became_active":"2024-03-24T00:18:43.256118+0000","last_became_peered":"2024-03-24T00:18:43.256118+0000","last_unstale":"2024-03-24T00:18:51.374297+0000","last_undegraded":"2024-03-24T00:18:51.374297+0000","last_fullsized":"2024-03-24T00:18:51.374297+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:40:05.523388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.446497+0000","last_change":"2024-03-24T00:18:43.885293+0000","last_active":"2024-03-24T00:18:51.446497+0000","last_peered":"2024-03-24T00:18:51.446497+0000","last_clean":"2024-03-24T00:18:51.446497+0000","last_became_active":"2024-03-24T00:18:43.884989+0000","last_became_peered":"2024-03-24T00:18:43.884989+0000","last_unstale":"2024-03-24T00:18:51.446497+0000","last_undegraded":"2024-03-24T00:18:51.446497+0000","last_fullsized":"2024-03-24T00:18:51.446497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:32:01.519131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.393102+0000","last_change":"2024-03-24T00:18:47.894253+0000","last_active":"2024-03-24T00:18:51.393102+0000","last_peered":"2024-03-24T00:18:51.393102+0000","last_clean":"2024-03-24T00:18:51.393102+0000","last_became_active":"2024-03-24T00:18:47.892654+0000","last_became_peered":"2024-03-24T00:18:47.892654+0000","last_unstale":"2024-03-24T00:18:51.393102+0000","last_undegraded":"2024-03-24T00:18:51.393102+0000","last_fullsized":"2024-03-24T00:18:51.393102+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:59:43.400810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:19:08.387581+0000","last_change":"2024-03-24T00:18:43.253074+0000","last_active":"2024-03-24T00:19:08.387581+0000","last_peered":"2024-03-24T00:19:08.387581+0000","last_clean":"2024-03-24T00:19:08.387581+0000","last_became_active":"2024-03-24T00:18:43.252382+0000","last_became_peered":"2024-03-24T00:18:43.252382+0000","last_unstale":"2024-03-24T00:19:08.387581+0000","last_undegraded":"2024-03-24T00:19:08.387581+0000","last_fullsized":"2024-03-24T00:19:08.387581+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:23:05.853912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.393418+0000","last_change":"2024-03-24T00:18:43.883654+0000","last_active":"2024-03-24T00:18:51.393418+0000","last_peered":"2024-03-24T00:18:51.393418+0000","last_clean":"2024-03-24T00:18:51.393418+0000","last_became_active":"2024-03-24T00:18:43.883335+0000","last_became_peered":"2024-03-24T00:18:43.883335+0000","last_unstale":"2024-03-24T00:18:51.393418+0000","last_undegraded":"2024-03-24T00:18:51.393418+0000","last_fullsized":"2024-03-24T00:18:51.393418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:42:10.856037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.872320+0000","last_change":"2024-03-24T00:18:47.888562+0000","last_active":"2024-03-24T00:18:48.872320+0000","last_peered":"2024-03-24T00:18:48.872320+0000","last_clean":"2024-03-24T00:18:48.872320+0000","last_became_active":"2024-03-24T00:18:47.888074+0000","last_became_peered":"2024-03-24T00:18:47.888074+0000","last_unstale":"2024-03-24T00:18:48.872320+0000","last_undegraded":"2024-03-24T00:18:48.872320+0000","last_fullsized":"2024-03-24T00:18:48.872320+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:01:04.617098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:19:07.378657+0000","last_change":"2024-03-24T00:18:43.261668+0000","last_active":"2024-03-24T00:19:07.378657+0000","last_peered":"2024-03-24T00:19:07.378657+0000","last_clean":"2024-03-24T00:19:07.378657+0000","last_became_active":"2024-03-24T00:18:43.261279+0000","last_became_peered":"2024-03-24T00:18:43.261279+0000","last_unstale":"2024-03-24T00:19:07.378657+0000","last_undegraded":"2024-03-24T00:19:07.378657+0000","last_fullsized":"2024-03-24T00:19:07.378657+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:36:05.390006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.401988+0000","last_change":"2024-03-24T00:18:43.889854+0000","last_active":"2024-03-24T00:18:51.401988+0000","last_peered":"2024-03-24T00:18:51.401988+0000","last_clean":"2024-03-24T00:18:51.401988+0000","last_became_active":"2024-03-24T00:18:43.889540+0000","last_became_peered":"2024-03-24T00:18:43.889540+0000","last_unstale":"2024-03-24T00:18:51.401988+0000","last_undegraded":"2024-03-24T00:18:51.401988+0000","last_fullsized":"2024-03-24T00:18:51.401988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:33:58.060564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.627764+0000","last_change":"2024-03-24T00:18:47.895495+0000","last_active":"2024-03-24T00:18:51.627764+0000","last_peered":"2024-03-24T00:18:51.627764+0000","last_clean":"2024-03-24T00:18:51.627764+0000","last_became_active":"2024-03-24T00:18:47.895158+0000","last_became_peered":"2024-03-24T00:18:47.895158+0000","last_unstale":"2024-03-24T00:18:51.627764+0000","last_undegraded":"2024-03-24T00:18:51.627764+0000","last_fullsized":"2024-03-24T00:18:51.627764+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:22:02.132419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:19:07.378431+0000","last_change":"2024-03-24T00:18:43.253063+0000","last_active":"2024-03-24T00:19:07.378431+0000","last_peered":"2024-03-24T00:19:07.378431+0000","last_clean":"2024-03-24T00:19:07.378431+0000","last_became_active":"2024-03-24T00:18:43.252534+0000","last_became_peered":"2024-03-24T00:18:43.252534+0000","last_unstale":"2024-03-24T00:19:07.378431+0000","last_undegraded":"2024-03-24T00:19:07.378431+0000","last_fullsized":"2024-03-24T00:19:07.378431+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:50:59.512780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.374063+0000","last_change":"2024-03-24T00:18:43.885574+0000","last_active":"2024-03-24T00:18:51.374063+0000","last_peered":"2024-03-24T00:18:51.374063+0000","last_clean":"2024-03-24T00:18:51.374063+0000","last_became_active":"2024-03-24T00:18:43.885257+0000","last_became_peered":"2024-03-24T00:18:43.885257+0000","last_unstale":"2024-03-24T00:18:51.374063+0000","last_undegraded":"2024-03-24T00:18:51.374063+0000","last_fullsized":"2024-03-24T00:18:51.374063+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:10.868436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.869745+0000","last_change":"2024-03-24T00:18:47.892739+0000","last_active":"2024-03-24T00:18:48.869745+0000","last_peered":"2024-03-24T00:18:48.869745+0000","last_clean":"2024-03-24T00:18:48.869745+0000","last_became_active":"2024-03-24T00:18:47.892291+0000","last_became_peered":"2024-03-24T00:18:47.892291+0000","last_unstale":"2024-03-24T00:18:48.869745+0000","last_undegraded":"2024-03-24T00:18:48.869745+0000","last_fullsized":"2024-03-24T00:18:48.869745+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:06:16.027968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:19:07.378287+0000","last_change":"2024-03-24T00:18:43.263856+0000","last_active":"2024-03-24T00:19:07.378287+0000","last_peered":"2024-03-24T00:19:07.378287+0000","last_clean":"2024-03-24T00:19:07.378287+0000","last_became_active":"2024-03-24T00:18:43.263032+0000","last_became_peered":"2024-03-24T00:18:43.263032+0000","last_unstale":"2024-03-24T00:19:07.378287+0000","last_undegraded":"2024-03-24T00:19:07.378287+0000","last_fullsized":"2024-03-24T00:19:07.378287+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:25:03.239696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.318699+0000","last_change":"2024-03-24T00:18:43.884674+0000","last_active":"2024-03-24T00:18:51.318699+0000","last_peered":"2024-03-24T00:18:51.318699+0000","last_clean":"2024-03-24T00:18:51.318699+0000","last_became_active":"2024-03-24T00:18:43.884346+0000","last_became_peered":"2024-03-24T00:18:43.884346+0000","last_unstale":"2024-03-24T00:18:51.318699+0000","last_undegraded":"2024-03-24T00:18:51.318699+0000","last_fullsized":"2024-03-24T00:18:51.318699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:58:31.474515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.627685+0000","last_change":"2024-03-24T00:18:47.892472+0000","last_active":"2024-03-24T00:18:51.627685+0000","last_peered":"2024-03-24T00:18:51.627685+0000","last_clean":"2024-03-24T00:18:51.627685+0000","last_became_active":"2024-03-24T00:18:47.892082+0000","last_became_peered":"2024-03-24T00:18:47.892082+0000","last_unstale":"2024-03-24T00:18:51.627685+0000","last_undegraded":"2024-03-24T00:18:51.627685+0000","last_fullsized":"2024-03-24T00:18:51.627685+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:26:01.084042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.872533+0000","last_change":"2024-03-24T00:18:43.263780+0000","last_active":"2024-03-24T00:18:48.872533+0000","last_peered":"2024-03-24T00:18:48.872533+0000","last_clean":"2024-03-24T00:18:48.872533+0000","last_became_active":"2024-03-24T00:18:43.263446+0000","last_became_peered":"2024-03-24T00:18:43.263446+0000","last_unstale":"2024-03-24T00:18:48.872533+0000","last_undegraded":"2024-03-24T00:18:48.872533+0000","last_fullsized":"2024-03-24T00:18:48.872533+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:09:09.821842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.393112+0000","last_change":"2024-03-24T00:18:43.888516+0000","last_active":"2024-03-24T00:18:51.393112+0000","last_peered":"2024-03-24T00:18:51.393112+0000","last_clean":"2024-03-24T00:18:51.393112+0000","last_became_active":"2024-03-24T00:18:43.888190+0000","last_became_peered":"2024-03-24T00:18:43.888190+0000","last_unstale":"2024-03-24T00:18:51.393112+0000","last_undegraded":"2024-03-24T00:18:51.393112+0000","last_fullsized":"2024-03-24T00:18:51.393112+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:53:35.685128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.627721+0000","last_change":"2024-03-24T00:18:47.897115+0000","last_active":"2024-03-24T00:18:51.627721+0000","last_peered":"2024-03-24T00:18:51.627721+0000","last_clean":"2024-03-24T00:18:51.627721+0000","last_became_active":"2024-03-24T00:18:47.896722+0000","last_became_peered":"2024-03-24T00:18:47.896722+0000","last_unstale":"2024-03-24T00:18:51.627721+0000","last_undegraded":"2024-03-24T00:18:51.627721+0000","last_fullsized":"2024-03-24T00:18:51.627721+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:44:40.541618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.373874+0000","last_change":"2024-03-24T00:18:43.263459+0000","last_active":"2024-03-24T00:18:51.373874+0000","last_peered":"2024-03-24T00:18:51.373874+0000","last_clean":"2024-03-24T00:18:51.373874+0000","last_became_active":"2024-03-24T00:18:43.263019+0000","last_became_peered":"2024-03-24T00:18:43.263019+0000","last_unstale":"2024-03-24T00:18:51.373874+0000","last_undegraded":"2024-03-24T00:18:51.373874+0000","last_fullsized":"2024-03-24T00:18:51.373874+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:10:57.171625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.195283+0000","last_change":"2024-03-24T00:18:43.886547+0000","last_active":"2024-03-24T00:18:51.195283+0000","last_peered":"2024-03-24T00:18:51.195283+0000","last_clean":"2024-03-24T00:18:51.195283+0000","last_became_active":"2024-03-24T00:18:43.886217+0000","last_became_peered":"2024-03-24T00:18:43.886217+0000","last_unstale":"2024-03-24T00:18:51.195283+0000","last_undegraded":"2024-03-24T00:18:51.195283+0000","last_fullsized":"2024-03-24T00:18:51.195283+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:59:33.576577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:49.903810+0000","last_change":"2024-03-24T00:18:47.894246+0000","last_active":"2024-03-24T00:18:49.903810+0000","last_peered":"2024-03-24T00:18:49.903810+0000","last_clean":"2024-03-24T00:18:49.903810+0000","last_became_active":"2024-03-24T00:18:47.893837+0000","last_became_peered":"2024-03-24T00:18:47.893837+0000","last_unstale":"2024-03-24T00:18:49.903810+0000","last_undegraded":"2024-03-24T00:18:49.903810+0000","last_fullsized":"2024-03-24T00:18:49.903810+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:33:06.044355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:19:08.387741+0000","last_change":"2024-03-24T00:18:43.256808+0000","last_active":"2024-03-24T00:19:08.387741+0000","last_peered":"2024-03-24T00:19:08.387741+0000","last_clean":"2024-03-24T00:19:08.387741+0000","last_became_active":"2024-03-24T00:18:43.256397+0000","last_became_peered":"2024-03-24T00:18:43.256397+0000","last_unstale":"2024-03-24T00:19:08.387741+0000","last_undegraded":"2024-03-24T00:19:08.387741+0000","last_fullsized":"2024-03-24T00:19:08.387741+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:36:13.505016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.195501+0000","last_change":"2024-03-24T00:18:43.887754+0000","last_active":"2024-03-24T00:18:51.195501+0000","last_peered":"2024-03-24T00:18:51.195501+0000","last_clean":"2024-03-24T00:18:51.195501+0000","last_became_active":"2024-03-24T00:18:43.886978+0000","last_became_peered":"2024-03-24T00:18:43.886978+0000","last_unstale":"2024-03-24T00:18:51.195501+0000","last_undegraded":"2024-03-24T00:18:51.195501+0000","last_fullsized":"2024-03-24T00:18:51.195501+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:43:18.186682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.401869+0000","last_change":"2024-03-24T00:18:47.887576+0000","last_active":"2024-03-24T00:18:51.401869+0000","last_peered":"2024-03-24T00:18:51.401869+0000","last_clean":"2024-03-24T00:18:51.401869+0000","last_became_active":"2024-03-24T00:18:47.887190+0000","last_became_peered":"2024-03-24T00:18:47.887190+0000","last_unstale":"2024-03-24T00:18:51.401869+0000","last_undegraded":"2024-03-24T00:18:51.401869+0000","last_fullsized":"2024-03-24T00:18:51.401869+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:10:10.396432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.621693+0000","last_change":"2024-03-24T00:18:43.259927+0000","last_active":"2024-03-24T00:18:51.621693+0000","last_peered":"2024-03-24T00:18:51.621693+0000","last_clean":"2024-03-24T00:18:51.621693+0000","last_became_active":"2024-03-24T00:18:43.259587+0000","last_became_peered":"2024-03-24T00:18:43.259587+0000","last_unstale":"2024-03-24T00:18:51.621693+0000","last_undegraded":"2024-03-24T00:18:51.621693+0000","last_fullsized":"2024-03-24T00:18:51.621693+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:26:06.380010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.195384+0000","last_change":"2024-03-24T00:18:43.897041+0000","last_active":"2024-03-24T00:18:51.195384+0000","last_peered":"2024-03-24T00:18:51.195384+0000","last_clean":"2024-03-24T00:18:51.195384+0000","last_became_active":"2024-03-24T00:18:43.896766+0000","last_became_peered":"2024-03-24T00:18:43.896766+0000","last_unstale":"2024-03-24T00:18:51.195384+0000","last_undegraded":"2024-03-24T00:18:51.195384+0000","last_fullsized":"2024-03-24T00:18:51.195384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:27:59.577589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.621814+0000","last_change":"2024-03-24T00:18:47.893146+0000","last_active":"2024-03-24T00:18:51.621814+0000","last_peered":"2024-03-24T00:18:51.621814+0000","last_clean":"2024-03-24T00:18:51.621814+0000","last_became_active":"2024-03-24T00:18:47.892150+0000","last_became_peered":"2024-03-24T00:18:47.892150+0000","last_unstale":"2024-03-24T00:18:51.621814+0000","last_undegraded":"2024-03-24T00:18:51.621814+0000","last_fullsized":"2024-03-24T00:18:51.621814+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:28:13.887194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.445937+0000","last_change":"2024-03-24T00:18:43.263972+0000","last_active":"2024-03-24T00:18:51.445937+0000","last_peered":"2024-03-24T00:18:51.445937+0000","last_clean":"2024-03-24T00:18:51.445937+0000","last_became_active":"2024-03-24T00:18:43.263577+0000","last_became_peered":"2024-03-24T00:18:43.263577+0000","last_unstale":"2024-03-24T00:18:51.445937+0000","last_undegraded":"2024-03-24T00:18:51.445937+0000","last_fullsized":"2024-03-24T00:18:51.445937+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:55:36.725420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.195619+0000","last_change":"2024-03-24T00:18:43.894722+0000","last_active":"2024-03-24T00:18:51.195619+0000","last_peered":"2024-03-24T00:18:51.195619+0000","last_clean":"2024-03-24T00:18:51.195619+0000","last_became_active":"2024-03-24T00:18:43.894359+0000","last_became_peered":"2024-03-24T00:18:43.894359+0000","last_unstale":"2024-03-24T00:18:51.195619+0000","last_undegraded":"2024-03-24T00:18:51.195619+0000","last_fullsized":"2024-03-24T00:18:51.195619+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:35:23.974163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.318815+0000","last_change":"2024-03-24T00:18:43.892749+0000","last_active":"2024-03-24T00:18:51.318815+0000","last_peered":"2024-03-24T00:18:51.318815+0000","last_clean":"2024-03-24T00:18:51.318815+0000","last_became_active":"2024-03-24T00:18:43.892014+0000","last_became_peered":"2024-03-24T00:18:43.892014+0000","last_unstale":"2024-03-24T00:18:51.318815+0000","last_undegraded":"2024-03-24T00:18:51.318815+0000","last_fullsized":"2024-03-24T00:18:51.318815+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:53:14.592384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.872878+0000","last_change":"2024-03-24T00:18:43.899540+0000","last_active":"2024-03-24T00:18:48.872878+0000","last_peered":"2024-03-24T00:18:48.872878+0000","last_clean":"2024-03-24T00:18:48.872878+0000","last_became_active":"2024-03-24T00:18:43.899206+0000","last_became_peered":"2024-03-24T00:18:43.899206+0000","last_unstale":"2024-03-24T00:18:48.872878+0000","last_undegraded":"2024-03-24T00:18:48.872878+0000","last_fullsized":"2024-03-24T00:18:48.872878+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:51:17.214705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.872328+0000","last_change":"2024-03-24T00:18:43.896254+0000","last_active":"2024-03-24T00:18:48.872328+0000","last_peered":"2024-03-24T00:18:48.872328+0000","last_clean":"2024-03-24T00:18:48.872328+0000","last_became_active":"2024-03-24T00:18:43.895889+0000","last_became_peered":"2024-03-24T00:18:43.895889+0000","last_unstale":"2024-03-24T00:18:48.872328+0000","last_undegraded":"2024-03-24T00:18:48.872328+0000","last_fullsized":"2024-03-24T00:18:48.872328+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:07:46.403117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.401814+0000","last_change":"2024-03-24T00:18:43.890813+0000","last_active":"2024-03-24T00:18:51.401814+0000","last_peered":"2024-03-24T00:18:51.401814+0000","last_clean":"2024-03-24T00:18:51.401814+0000","last_became_active":"2024-03-24T00:18:43.890407+0000","last_became_peered":"2024-03-24T00:18:43.890407+0000","last_unstale":"2024-03-24T00:18:51.401814+0000","last_undegraded":"2024-03-24T00:18:51.401814+0000","last_fullsized":"2024-03-24T00:18:51.401814+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:57:45.146055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.446399+0000","last_change":"2024-03-24T00:18:43.890280+0000","last_active":"2024-03-24T00:18:51.446399+0000","last_peered":"2024-03-24T00:18:51.446399+0000","last_clean":"2024-03-24T00:18:51.446399+0000","last_became_active":"2024-03-24T00:18:43.889888+0000","last_became_peered":"2024-03-24T00:18:43.889888+0000","last_unstale":"2024-03-24T00:18:51.446399+0000","last_undegraded":"2024-03-24T00:18:51.446399+0000","last_fullsized":"2024-03-24T00:18:51.446399+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:49:17.010306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.318973+0000","last_change":"2024-03-24T00:18:43.898453+0000","last_active":"2024-03-24T00:18:51.318973+0000","last_peered":"2024-03-24T00:18:51.318973+0000","last_clean":"2024-03-24T00:18:51.318973+0000","last_became_active":"2024-03-24T00:18:43.898081+0000","last_became_peered":"2024-03-24T00:18:43.898081+0000","last_unstale":"2024-03-24T00:18:51.318973+0000","last_undegraded":"2024-03-24T00:18:51.318973+0000","last_fullsized":"2024-03-24T00:18:51.318973+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:03:50.856755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.872863+0000","last_change":"2024-03-24T00:18:43.894667+0000","last_active":"2024-03-24T00:18:48.872863+0000","last_peered":"2024-03-24T00:18:48.872863+0000","last_clean":"2024-03-24T00:18:48.872863+0000","last_became_active":"2024-03-24T00:18:43.893869+0000","last_became_peered":"2024-03-24T00:18:43.893869+0000","last_unstale":"2024-03-24T00:18:48.872863+0000","last_undegraded":"2024-03-24T00:18:48.872863+0000","last_fullsized":"2024-03-24T00:18:48.872863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:51:43.273867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.393526+0000","last_change":"2024-03-24T00:18:43.893201+0000","last_active":"2024-03-24T00:18:51.393526+0000","last_peered":"2024-03-24T00:18:51.393526+0000","last_clean":"2024-03-24T00:18:51.393526+0000","last_became_active":"2024-03-24T00:18:43.892866+0000","last_became_peered":"2024-03-24T00:18:43.892866+0000","last_unstale":"2024-03-24T00:18:51.393526+0000","last_undegraded":"2024-03-24T00:18:51.393526+0000","last_fullsized":"2024-03-24T00:18:51.393526+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:03:16.108806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.621817+0000","last_change":"2024-03-24T00:18:43.882211+0000","last_active":"2024-03-24T00:18:51.621817+0000","last_peered":"2024-03-24T00:18:51.621817+0000","last_clean":"2024-03-24T00:18:51.621817+0000","last_became_active":"2024-03-24T00:18:43.881857+0000","last_became_peered":"2024-03-24T00:18:43.881857+0000","last_unstale":"2024-03-24T00:18:51.621817+0000","last_undegraded":"2024-03-24T00:18:51.621817+0000","last_fullsized":"2024-03-24T00:18:51.621817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:36:45.911853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.393331+0000","last_change":"2024-03-24T00:18:43.892599+0000","last_active":"2024-03-24T00:18:51.393331+0000","last_peered":"2024-03-24T00:18:51.393331+0000","last_clean":"2024-03-24T00:18:51.393331+0000","last_became_active":"2024-03-24T00:18:43.891966+0000","last_became_peered":"2024-03-24T00:18:43.891966+0000","last_unstale":"2024-03-24T00:18:51.393331+0000","last_undegraded":"2024-03-24T00:18:51.393331+0000","last_fullsized":"2024-03-24T00:18:51.393331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:22:03.486466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:49.903899+0000","last_change":"2024-03-24T00:18:43.892410+0000","last_active":"2024-03-24T00:18:49.903899+0000","last_peered":"2024-03-24T00:18:49.903899+0000","last_clean":"2024-03-24T00:18:49.903899+0000","last_became_active":"2024-03-24T00:18:43.891140+0000","last_became_peered":"2024-03-24T00:18:43.891140+0000","last_unstale":"2024-03-24T00:18:49.903899+0000","last_undegraded":"2024-03-24T00:18:49.903899+0000","last_fullsized":"2024-03-24T00:18:49.903899+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:03:14.252863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.446266+0000","last_change":"2024-03-24T00:18:43.876508+0000","last_active":"2024-03-24T00:18:51.446266+0000","last_peered":"2024-03-24T00:18:51.446266+0000","last_clean":"2024-03-24T00:18:51.446266+0000","last_became_active":"2024-03-24T00:18:43.876204+0000","last_became_peered":"2024-03-24T00:18:43.876204+0000","last_unstale":"2024-03-24T00:18:51.446266+0000","last_undegraded":"2024-03-24T00:18:51.446266+0000","last_fullsized":"2024-03-24T00:18:51.446266+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:35.358964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.318709+0000","last_change":"2024-03-24T00:18:43.887855+0000","last_active":"2024-03-24T00:18:51.318709+0000","last_peered":"2024-03-24T00:18:51.318709+0000","last_clean":"2024-03-24T00:18:51.318709+0000","last_became_active":"2024-03-24T00:18:43.887520+0000","last_became_peered":"2024-03-24T00:18:43.887520+0000","last_unstale":"2024-03-24T00:18:51.318709+0000","last_undegraded":"2024-03-24T00:18:51.318709+0000","last_fullsized":"2024-03-24T00:18:51.318709+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:17:15.930626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5402,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":6,"num_read_kb":8,"num_write":93,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":36867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383698,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":983040,"data_stored":477215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645335,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36912,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708240,"statfs":{"total":95995035648,"available":95957237760,"internally_reserved":0,"allocated":1613824,"data_stored":1072381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808861,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":1003520,"data_stored":478033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070497,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":991232,"data_stored":477819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332135,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28568,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716584,"statfs":{"total":95995035648,"available":95965782016,"internally_reserved":0,"allocated":1589248,"data_stored":1068787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495659,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":983040,"data_stored":477215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":500356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36304,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708848,"statfs":{"total":95995035648,"available":95957860352,"internally_reserved":0,"allocated":987136,"data_stored":477305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411392,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":481713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182458,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":995328,"data_stored":477409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920822,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36304,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708848,"statfs":{"total":95995035648,"available":95957860352,"internally_reserved":0,"allocated":987136,"data_stored":477237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757296,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36892,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708260,"statfs":{"total":95995035648,"available":95957258240,"internally_reserved":0,"allocated":1589248,"data_stored":1067765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":23141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:19:11.584 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:19:11.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:11 smithi106 ceph-mon[36505]: from='client.24670 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:19:11.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:11 smithi106 ceph-mon[36505]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:19:11.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:11 smithi106 ceph-mon[36505]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:19:11.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:11 smithi106 ceph-mon[36505]: pgmap v304: 129 pgs: 129 active+clean; 582 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-24T00:19:11.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:11 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2733468502' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:19:11.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:11 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2734379494' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:19:11.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.720+0000 7f5f8b75b640 1 -- 172.21.15.78:0/2291210096 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f84100f60 msgr2=0x7f5f841013e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:11.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.720+0000 7f5f8b75b640 1 --2- 172.21.15.78:0/2291210096 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f84100f60 0x7f5f841013e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f5f74009900 tx=0x7f5f7402bc30 comp rx=0 tx=0).stop 2024-03-24T00:19:11.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.720+0000 7f5f8b75b640 1 -- 172.21.15.78:0/2291210096 shutdown_connections 2024-03-24T00:19:11.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.720+0000 7f5f8b75b640 1 --2- 172.21.15.78:0/2291210096 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f84101920 0x7f5f841081b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.720+0000 7f5f8b75b640 1 --2- 172.21.15.78:0/2291210096 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f84100f60 0x7f5f841013e0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.720+0000 7f5f8b75b640 1 --2- 172.21.15.78:0/2291210096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f840ffd70 0x7f5f84100190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.720+0000 7f5f8b75b640 1 -- 172.21.15.78:0/2291210096 >> 172.21.15.78:0/2291210096 conn(0x7f5f840fb2e0 msgr2=0x7f5f840fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:11.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.720+0000 7f5f8b75b640 1 -- 172.21.15.78:0/2291210096 shutdown_connections 2024-03-24T00:19:11.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.720+0000 7f5f8b75b640 1 -- 172.21.15.78:0/2291210096 wait complete. 2024-03-24T00:19:11.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.721+0000 7f5f8b75b640 1 Processor -- start 2024-03-24T00:19:11.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.721+0000 7f5f8b75b640 1 -- start start 2024-03-24T00:19:11.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.721+0000 7f5f8b75b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f840ffd70 0x7f5f84197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:11.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.721+0000 7f5f8b75b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f84100f60 0x7f5f84198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:11.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.721+0000 7f5f8b75b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f84101920 0x7f5f8419f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:11.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.721+0000 7f5f8b75b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8410b9f0 con 0x7f5f84100f60 2024-03-24T00:19:11.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.721+0000 7f5f8b75b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8410b870 con 0x7f5f84101920 2024-03-24T00:19:11.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.721+0000 7f5f8b75b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8410bb70 con 0x7f5f840ffd70 2024-03-24T00:19:11.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.721+0000 7f5f88cf6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f84100f60 0x7f5f84198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:11.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.721+0000 7f5f88cf6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f84100f60 0x7f5f84198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58422/0 (socket says 172.21.15.78:58422) 2024-03-24T00:19:11.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.721+0000 7f5f88cf6640 1 -- 172.21.15.78:0/3254201716 learned_addr learned my addr 172.21.15.78:0/3254201716 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:19:11.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.721+0000 7f5f89cf8640 1 --2- 172.21.15.78:0/3254201716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f84101920 0x7f5f8419f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:11.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.721+0000 7f5f894f7640 1 --2- 172.21.15.78:0/3254201716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f840ffd70 0x7f5f84197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:11.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.721+0000 7f5f88cf6640 1 -- 172.21.15.78:0/3254201716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f840ffd70 msgr2=0x7f5f84197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:11.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.721+0000 7f5f88cf6640 1 --2- 172.21.15.78:0/3254201716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f840ffd70 0x7f5f84197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.721+0000 7f5f88cf6640 1 -- 172.21.15.78:0/3254201716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f84101920 msgr2=0x7f5f8419f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:11.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.721+0000 7f5f88cf6640 1 --2- 172.21.15.78:0/3254201716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f84101920 0x7f5f8419f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.721+0000 7f5f88cf6640 1 -- 172.21.15.78:0/3254201716 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f8419f6a0 con 0x7f5f84100f60 2024-03-24T00:19:11.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.721+0000 7f5f89cf8640 1 --2- 172.21.15.78:0/3254201716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f84101920 0x7f5f8419f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:11.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.721+0000 7f5f88cf6640 1 --2- 172.21.15.78:0/3254201716 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f84100f60 0x7f5f84198010 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f5f74009230 tx=0x7f5f7402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:11.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.722+0000 7f5f7a7fc640 1 -- 172.21.15.78:0/3254201716 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f74041070 con 0x7f5f84100f60 2024-03-24T00:19:11.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.722+0000 7f5f7a7fc640 1 -- 172.21.15.78:0/3254201716 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f74004450 con 0x7f5f84100f60 2024-03-24T00:19:11.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.722+0000 7f5f7a7fc640 1 -- 172.21.15.78:0/3254201716 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f74003e20 con 0x7f5f84100f60 2024-03-24T00:19:11.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.722+0000 7f5f8b75b640 1 -- 172.21.15.78:0/3254201716 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f8419f930 con 0x7f5f84100f60 2024-03-24T00:19:11.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.722+0000 7f5f8b75b640 1 -- 172.21.15.78:0/3254201716 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f8419fe20 con 0x7f5f84100f60 2024-03-24T00:19:11.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.723+0000 7f5f8b75b640 1 -- 172.21.15.78:0/3254201716 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f841a0050 con 0x7f5f84100f60 2024-03-24T00:19:11.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.724+0000 7f5f7a7fc640 1 -- 172.21.15.78:0/3254201716 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f5f74044070 con 0x7f5f84100f60 2024-03-24T00:19:11.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.724+0000 7f5f7a7fc640 1 --2- 172.21.15.78:0/3254201716 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f5f700a2890 0x7f5f700a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:11.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.725+0000 7f5f894f7640 1 --2- 172.21.15.78:0/3254201716 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f5f700a2890 0x7f5f700a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:11.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.725+0000 7f5f7a7fc640 1 -- 172.21.15.78:0/3254201716 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f5f740f08a0 con 0x7f5f84100f60 2024-03-24T00:19:11.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.725+0000 7f5f894f7640 1 --2- 172.21.15.78:0/3254201716 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f5f700a2890 0x7f5f700a4d50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5f84100dc0 tx=0x7f5f80005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:11.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.729+0000 7f5f7a7fc640 1 -- 172.21.15.78:0/3254201716 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5f740bd060 con 0x7f5f84100f60 2024-03-24T00:19:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:11 smithi107 ceph-mon[36485]: from='client.24670 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:19:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:11 smithi107 ceph-mon[36485]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:19:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:11 smithi107 ceph-mon[36485]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:19:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:11 smithi107 ceph-mon[36485]: pgmap v304: 129 pgs: 129 active+clean; 582 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-24T00:19:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:11 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2733468502' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:19:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:11 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2734379494' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:19:11.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.903+0000 7f5f8b75b640 1 -- 172.21.15.78:0/3254201716 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5f841073b0 con 0x7f5f700a2890 2024-03-24T00:19:11.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.913+0000 7f5f7a7fc640 1 -- 172.21.15.78:0/3254201716 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340247 (secure 0 0 0) 0x7f5f841073b0 con 0x7f5f700a2890 2024-03-24T00:19:11.916 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:19:11.918 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-03-24T00:19:11.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.916+0000 7f5f8b75b640 1 -- 172.21.15.78:0/3254201716 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f5f700a2890 msgr2=0x7f5f700a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:11.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.916+0000 7f5f8b75b640 1 --2- 172.21.15.78:0/3254201716 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f5f700a2890 0x7f5f700a4d50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5f84100dc0 tx=0x7f5f80005f90 comp rx=0 tx=0).stop 2024-03-24T00:19:11.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.916+0000 7f5f8b75b640 1 -- 172.21.15.78:0/3254201716 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f84100f60 msgr2=0x7f5f84198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:11.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.916+0000 7f5f8b75b640 1 --2- 172.21.15.78:0/3254201716 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f84100f60 0x7f5f84198010 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f5f74009230 tx=0x7f5f7402c650 comp rx=0 tx=0).stop 2024-03-24T00:19:11.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.916+0000 7f5f8b75b640 1 -- 172.21.15.78:0/3254201716 shutdown_connections 2024-03-24T00:19:11.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.916+0000 7f5f8b75b640 1 --2- 172.21.15.78:0/3254201716 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f5f700a2890 0x7f5f700a4d50 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.916+0000 7f5f8b75b640 1 --2- 172.21.15.78:0/3254201716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f84101920 0x7f5f8419f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.916+0000 7f5f8b75b640 1 --2- 172.21.15.78:0/3254201716 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f84100f60 0x7f5f84198010 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.916+0000 7f5f8b75b640 1 --2- 172.21.15.78:0/3254201716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f840ffd70 0x7f5f84197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.916+0000 7f5f8b75b640 1 -- 172.21.15.78:0/3254201716 >> 172.21.15.78:0/3254201716 conn(0x7f5f840fb2e0 msgr2=0x7f5f840fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:11.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.916+0000 7f5f8b75b640 1 -- 172.21.15.78:0/3254201716 shutdown_connections 2024-03-24T00:19:11.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:11.916+0000 7f5f8b75b640 1 -- 172.21.15.78:0/3254201716 wait complete. 2024-03-24T00:19:11.927 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2024-03-24T00:19:10.284693+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595770,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":142,"num_read_kb":127,"num_write":274,"num_write_kb":2965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":212,"ondisk_log_size":212,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":408360,"kb_used_data":13424,"kb_used_omap":18,"kb_used_meta":394797,"kb_avail":1124533464,"statfs":{"total":1151940427776,"available":1151522267136,"internally_reserved":0,"allocated":13746176,"data_stored":7533235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":404272444},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2992,"num_objects":18,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":6,"num_read_kb":8,"num_write":48,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003518"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.319135+0000","last_change":"2024-03-24T00:18:43.890863+0000","last_active":"2024-03-24T00:18:51.319135+0000","last_peered":"2024-03-24T00:18:51.319135+0000","last_clean":"2024-03-24T00:18:51.319135+0000","last_became_active":"2024-03-24T00:18:43.890503+0000","last_became_peered":"2024-03-24T00:18:43.890503+0000","last_unstale":"2024-03-24T00:18:51.319135+0000","last_undegraded":"2024-03-24T00:18:51.319135+0000","last_fullsized":"2024-03-24T00:18:51.319135+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:33:10.518835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.401790+0000","last_change":"2024-03-24T00:18:43.882039+0000","last_active":"2024-03-24T00:18:51.401790+0000","last_peered":"2024-03-24T00:18:51.401790+0000","last_clean":"2024-03-24T00:18:51.401790+0000","last_became_active":"2024-03-24T00:18:43.881704+0000","last_became_peered":"2024-03-24T00:18:43.881704+0000","last_unstale":"2024-03-24T00:18:51.401790+0000","last_undegraded":"2024-03-24T00:18:51.401790+0000","last_fullsized":"2024-03-24T00:18:51.401790+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:04:46.386032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.401666+0000","last_change":"2024-03-24T00:18:43.888649+0000","last_active":"2024-03-24T00:18:51.401666+0000","last_peered":"2024-03-24T00:18:51.401666+0000","last_clean":"2024-03-24T00:18:51.401666+0000","last_became_active":"2024-03-24T00:18:43.888339+0000","last_became_peered":"2024-03-24T00:18:43.888339+0000","last_unstale":"2024-03-24T00:18:51.401666+0000","last_undegraded":"2024-03-24T00:18:51.401666+0000","last_fullsized":"2024-03-24T00:18:51.401666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:28:35.787820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.627849+0000","last_change":"2024-03-24T00:18:43.893347+0000","last_active":"2024-03-24T00:18:51.627849+0000","last_peered":"2024-03-24T00:18:51.627849+0000","last_clean":"2024-03-24T00:18:51.627849+0000","last_became_active":"2024-03-24T00:18:43.893030+0000","last_became_peered":"2024-03-24T00:18:43.893030+0000","last_unstale":"2024-03-24T00:18:51.627849+0000","last_undegraded":"2024-03-24T00:18:51.627849+0000","last_fullsized":"2024-03-24T00:18:51.627849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:51:06.288550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.374069+0000","last_change":"2024-03-24T00:18:43.890677+0000","last_active":"2024-03-24T00:18:51.374069+0000","last_peered":"2024-03-24T00:18:51.374069+0000","last_clean":"2024-03-24T00:18:51.374069+0000","last_became_active":"2024-03-24T00:18:43.887161+0000","last_became_peered":"2024-03-24T00:18:43.887161+0000","last_unstale":"2024-03-24T00:18:51.374069+0000","last_undegraded":"2024-03-24T00:18:51.374069+0000","last_fullsized":"2024-03-24T00:18:51.374069+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:09:32.160568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.870048+0000","last_change":"2024-03-24T00:18:43.887293+0000","last_active":"2024-03-24T00:18:48.870048+0000","last_peered":"2024-03-24T00:18:48.870048+0000","last_clean":"2024-03-24T00:18:48.870048+0000","last_became_active":"2024-03-24T00:18:43.886961+0000","last_became_peered":"2024-03-24T00:18:43.886961+0000","last_unstale":"2024-03-24T00:18:48.870048+0000","last_undegraded":"2024-03-24T00:18:48.870048+0000","last_fullsized":"2024-03-24T00:18:48.870048+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:48:14.312376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.195112+0000","last_change":"2024-03-24T00:18:43.876014+0000","last_active":"2024-03-24T00:18:51.195112+0000","last_peered":"2024-03-24T00:18:51.195112+0000","last_clean":"2024-03-24T00:18:51.195112+0000","last_became_active":"2024-03-24T00:18:43.875619+0000","last_became_peered":"2024-03-24T00:18:43.875619+0000","last_unstale":"2024-03-24T00:18:51.195112+0000","last_undegraded":"2024-03-24T00:18:51.195112+0000","last_fullsized":"2024-03-24T00:18:51.195112+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:10:26.707681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.392797+0000","last_change":"2024-03-24T00:18:43.894851+0000","last_active":"2024-03-24T00:18:51.392797+0000","last_peered":"2024-03-24T00:18:51.392797+0000","last_clean":"2024-03-24T00:18:51.392797+0000","last_became_active":"2024-03-24T00:18:43.894364+0000","last_became_peered":"2024-03-24T00:18:43.894364+0000","last_unstale":"2024-03-24T00:18:51.392797+0000","last_undegraded":"2024-03-24T00:18:51.392797+0000","last_fullsized":"2024-03-24T00:18:51.392797+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:52:01.538415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:49.904156+0000","last_change":"2024-03-24T00:18:43.898453+0000","last_active":"2024-03-24T00:18:49.904156+0000","last_peered":"2024-03-24T00:18:49.904156+0000","last_clean":"2024-03-24T00:18:49.904156+0000","last_became_active":"2024-03-24T00:18:43.898118+0000","last_became_peered":"2024-03-24T00:18:43.898118+0000","last_unstale":"2024-03-24T00:18:49.904156+0000","last_undegraded":"2024-03-24T00:18:49.904156+0000","last_fullsized":"2024-03-24T00:18:49.904156+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:18:00.478010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.870346+0000","last_change":"2024-03-24T00:18:43.894133+0000","last_active":"2024-03-24T00:18:48.870346+0000","last_peered":"2024-03-24T00:18:48.870346+0000","last_clean":"2024-03-24T00:18:48.870346+0000","last_became_active":"2024-03-24T00:18:43.893777+0000","last_became_peered":"2024-03-24T00:18:43.893777+0000","last_unstale":"2024-03-24T00:18:48.870346+0000","last_undegraded":"2024-03-24T00:18:48.870346+0000","last_fullsized":"2024-03-24T00:18:48.870346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:46:33.029934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.195505+0000","last_change":"2024-03-24T00:18:43.888024+0000","last_active":"2024-03-24T00:18:51.195505+0000","last_peered":"2024-03-24T00:18:51.195505+0000","last_clean":"2024-03-24T00:18:51.195505+0000","last_became_active":"2024-03-24T00:18:43.887430+0000","last_became_peered":"2024-03-24T00:18:43.887430+0000","last_unstale":"2024-03-24T00:18:51.195505+0000","last_undegraded":"2024-03-24T00:18:51.195505+0000","last_fullsized":"2024-03-24T00:18:51.195505+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:54:29.299788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.627764+0000","last_change":"2024-03-24T00:18:43.889240+0000","last_active":"2024-03-24T00:18:51.627764+0000","last_peered":"2024-03-24T00:18:51.627764+0000","last_clean":"2024-03-24T00:18:51.627764+0000","last_became_active":"2024-03-24T00:18:43.888861+0000","last_became_peered":"2024-03-24T00:18:43.888861+0000","last_unstale":"2024-03-24T00:18:51.627764+0000","last_undegraded":"2024-03-24T00:18:51.627764+0000","last_fullsized":"2024-03-24T00:18:51.627764+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:41:56.637532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.446207+0000","last_change":"2024-03-24T00:18:43.873750+0000","last_active":"2024-03-24T00:18:51.446207+0000","last_peered":"2024-03-24T00:18:51.446207+0000","last_clean":"2024-03-24T00:18:51.446207+0000","last_became_active":"2024-03-24T00:18:43.873406+0000","last_became_peered":"2024-03-24T00:18:43.873406+0000","last_unstale":"2024-03-24T00:18:51.446207+0000","last_undegraded":"2024-03-24T00:18:51.446207+0000","last_fullsized":"2024-03-24T00:18:51.446207+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:53.262272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.401556+0000","last_change":"2024-03-24T00:18:43.883717+0000","last_active":"2024-03-24T00:18:51.401556+0000","last_peered":"2024-03-24T00:18:51.401556+0000","last_clean":"2024-03-24T00:18:51.401556+0000","last_became_active":"2024-03-24T00:18:43.883329+0000","last_became_peered":"2024-03-24T00:18:43.883329+0000","last_unstale":"2024-03-24T00:18:51.401556+0000","last_undegraded":"2024-03-24T00:18:51.401556+0000","last_fullsized":"2024-03-24T00:18:51.401556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:26:15.127054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.401516+0000","last_change":"2024-03-24T00:18:43.890831+0000","last_active":"2024-03-24T00:18:51.401516+0000","last_peered":"2024-03-24T00:18:51.401516+0000","last_clean":"2024-03-24T00:18:51.401516+0000","last_became_active":"2024-03-24T00:18:43.890448+0000","last_became_peered":"2024-03-24T00:18:43.890448+0000","last_unstale":"2024-03-24T00:18:51.401516+0000","last_undegraded":"2024-03-24T00:18:51.401516+0000","last_fullsized":"2024-03-24T00:18:51.401516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:56:13.222271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:49.903663+0000","last_change":"2024-03-24T00:18:43.890309+0000","last_active":"2024-03-24T00:18:49.903663+0000","last_peered":"2024-03-24T00:18:49.903663+0000","last_clean":"2024-03-24T00:18:49.903663+0000","last_became_active":"2024-03-24T00:18:43.889138+0000","last_became_peered":"2024-03-24T00:18:43.889138+0000","last_unstale":"2024-03-24T00:18:49.903663+0000","last_undegraded":"2024-03-24T00:18:49.903663+0000","last_fullsized":"2024-03-24T00:18:49.903663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:13:52.480938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.318987+0000","last_change":"2024-03-24T00:18:43.891633+0000","last_active":"2024-03-24T00:18:51.318987+0000","last_peered":"2024-03-24T00:18:51.318987+0000","last_clean":"2024-03-24T00:18:51.318987+0000","last_became_active":"2024-03-24T00:18:43.891303+0000","last_became_peered":"2024-03-24T00:18:43.891303+0000","last_unstale":"2024-03-24T00:18:51.318987+0000","last_undegraded":"2024-03-24T00:18:51.318987+0000","last_fullsized":"2024-03-24T00:18:51.318987+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:34:01.874142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.869726+0000","last_change":"2024-03-24T00:18:43.887695+0000","last_active":"2024-03-24T00:18:48.869726+0000","last_peered":"2024-03-24T00:18:48.869726+0000","last_clean":"2024-03-24T00:18:48.869726+0000","last_became_active":"2024-03-24T00:18:43.887359+0000","last_became_peered":"2024-03-24T00:18:43.887359+0000","last_unstale":"2024-03-24T00:18:48.869726+0000","last_undegraded":"2024-03-24T00:18:48.869726+0000","last_fullsized":"2024-03-24T00:18:48.869726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:00:50.951362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.872796+0000","last_change":"2024-03-24T00:18:43.895671+0000","last_active":"2024-03-24T00:18:48.872796+0000","last_peered":"2024-03-24T00:18:48.872796+0000","last_clean":"2024-03-24T00:18:48.872796+0000","last_became_active":"2024-03-24T00:18:43.895267+0000","last_became_peered":"2024-03-24T00:18:43.895267+0000","last_unstale":"2024-03-24T00:18:48.872796+0000","last_undegraded":"2024-03-24T00:18:48.872796+0000","last_fullsized":"2024-03-24T00:18:48.872796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:24:48.014768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.872588+0000","last_change":"2024-03-24T00:18:47.896879+0000","last_active":"2024-03-24T00:18:48.872588+0000","last_peered":"2024-03-24T00:18:48.872588+0000","last_clean":"2024-03-24T00:18:48.872588+0000","last_became_active":"2024-03-24T00:18:47.896517+0000","last_became_peered":"2024-03-24T00:18:47.896517+0000","last_unstale":"2024-03-24T00:18:48.872588+0000","last_undegraded":"2024-03-24T00:18:48.872588+0000","last_fullsized":"2024-03-24T00:18:48.872588+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:28:34.331404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:19:08.388688+0000","last_change":"2024-03-24T00:18:43.257103+0000","last_active":"2024-03-24T00:19:08.388688+0000","last_peered":"2024-03-24T00:19:08.388688+0000","last_clean":"2024-03-24T00:19:08.388688+0000","last_became_active":"2024-03-24T00:18:43.256774+0000","last_became_peered":"2024-03-24T00:18:43.256774+0000","last_unstale":"2024-03-24T00:19:08.388688+0000","last_undegraded":"2024-03-24T00:19:08.388688+0000","last_fullsized":"2024-03-24T00:19:08.388688+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:50:05.350855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.393311+0000","last_change":"2024-03-24T00:18:43.892643+0000","last_active":"2024-03-24T00:18:51.393311+0000","last_peered":"2024-03-24T00:18:51.393311+0000","last_clean":"2024-03-24T00:18:51.393311+0000","last_became_active":"2024-03-24T00:18:43.892317+0000","last_became_peered":"2024-03-24T00:18:43.892317+0000","last_unstale":"2024-03-24T00:18:51.393311+0000","last_undegraded":"2024-03-24T00:18:51.393311+0000","last_fullsized":"2024-03-24T00:18:51.393311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:26:28.374587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.627424+0000","last_change":"2024-03-24T00:18:47.895307+0000","last_active":"2024-03-24T00:18:51.627424+0000","last_peered":"2024-03-24T00:18:51.627424+0000","last_clean":"2024-03-24T00:18:51.627424+0000","last_became_active":"2024-03-24T00:18:47.894988+0000","last_became_peered":"2024-03-24T00:18:47.894988+0000","last_unstale":"2024-03-24T00:18:51.627424+0000","last_undegraded":"2024-03-24T00:18:51.627424+0000","last_fullsized":"2024-03-24T00:18:51.627424+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:48:41.068076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.393462+0000","last_change":"2024-03-24T00:18:43.255861+0000","last_active":"2024-03-24T00:18:51.393462+0000","last_peered":"2024-03-24T00:18:51.393462+0000","last_clean":"2024-03-24T00:18:51.393462+0000","last_became_active":"2024-03-24T00:18:43.255529+0000","last_became_peered":"2024-03-24T00:18:43.255529+0000","last_unstale":"2024-03-24T00:18:51.393462+0000","last_undegraded":"2024-03-24T00:18:51.393462+0000","last_fullsized":"2024-03-24T00:18:51.393462+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:43:19.617673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.401851+0000","last_change":"2024-03-24T00:18:43.886651+0000","last_active":"2024-03-24T00:18:51.401851+0000","last_peered":"2024-03-24T00:18:51.401851+0000","last_clean":"2024-03-24T00:18:51.401851+0000","last_became_active":"2024-03-24T00:18:43.886341+0000","last_became_peered":"2024-03-24T00:18:43.886341+0000","last_unstale":"2024-03-24T00:18:51.401851+0000","last_undegraded":"2024-03-24T00:18:51.401851+0000","last_fullsized":"2024-03-24T00:18:51.401851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:14:11.993717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.627639+0000","last_change":"2024-03-24T00:18:47.897123+0000","last_active":"2024-03-24T00:18:51.627639+0000","last_peered":"2024-03-24T00:18:51.627639+0000","last_clean":"2024-03-24T00:18:51.627639+0000","last_became_active":"2024-03-24T00:18:47.896762+0000","last_became_peered":"2024-03-24T00:18:47.896762+0000","last_unstale":"2024-03-24T00:18:51.627639+0000","last_undegraded":"2024-03-24T00:18:51.627639+0000","last_fullsized":"2024-03-24T00:18:51.627639+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:45:50.639038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.374091+0000","last_change":"2024-03-24T00:18:43.263895+0000","last_active":"2024-03-24T00:18:51.374091+0000","last_peered":"2024-03-24T00:18:51.374091+0000","last_clean":"2024-03-24T00:18:51.374091+0000","last_became_active":"2024-03-24T00:18:43.263557+0000","last_became_peered":"2024-03-24T00:18:43.263557+0000","last_unstale":"2024-03-24T00:18:51.374091+0000","last_undegraded":"2024-03-24T00:18:51.374091+0000","last_fullsized":"2024-03-24T00:18:51.374091+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:44:28.763293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:49.903928+0000","last_change":"2024-03-24T00:18:43.888820+0000","last_active":"2024-03-24T00:18:49.903928+0000","last_peered":"2024-03-24T00:18:49.903928+0000","last_clean":"2024-03-24T00:18:49.903928+0000","last_became_active":"2024-03-24T00:18:43.888382+0000","last_became_peered":"2024-03-24T00:18:43.888382+0000","last_unstale":"2024-03-24T00:18:49.903928+0000","last_undegraded":"2024-03-24T00:18:49.903928+0000","last_fullsized":"2024-03-24T00:18:49.903928+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:04.097113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.627580+0000","last_change":"2024-03-24T00:18:47.896443+0000","last_active":"2024-03-24T00:18:51.627580+0000","last_peered":"2024-03-24T00:18:51.627580+0000","last_clean":"2024-03-24T00:18:51.627580+0000","last_became_active":"2024-03-24T00:18:47.895792+0000","last_became_peered":"2024-03-24T00:18:47.895792+0000","last_unstale":"2024-03-24T00:18:51.627580+0000","last_undegraded":"2024-03-24T00:18:51.627580+0000","last_fullsized":"2024-03-24T00:18:51.627580+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:59:42.360902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.373915+0000","last_change":"2024-03-24T00:18:43.261085+0000","last_active":"2024-03-24T00:18:51.373915+0000","last_peered":"2024-03-24T00:18:51.373915+0000","last_clean":"2024-03-24T00:18:51.373915+0000","last_became_active":"2024-03-24T00:18:43.260724+0000","last_became_peered":"2024-03-24T00:18:43.260724+0000","last_unstale":"2024-03-24T00:18:51.373915+0000","last_undegraded":"2024-03-24T00:18:51.373915+0000","last_fullsized":"2024-03-24T00:18:51.373915+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:05:20.992776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.622130+0000","last_change":"2024-03-24T00:18:43.892891+0000","last_active":"2024-03-24T00:18:51.622130+0000","last_peered":"2024-03-24T00:18:51.622130+0000","last_clean":"2024-03-24T00:18:51.622130+0000","last_became_active":"2024-03-24T00:18:43.892430+0000","last_became_peered":"2024-03-24T00:18:43.892430+0000","last_unstale":"2024-03-24T00:18:51.622130+0000","last_undegraded":"2024-03-24T00:18:51.622130+0000","last_fullsized":"2024-03-24T00:18:51.622130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:30:24.926031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.873061+0000","last_change":"2024-03-24T00:18:47.889119+0000","last_active":"2024-03-24T00:18:48.873061+0000","last_peered":"2024-03-24T00:18:48.873061+0000","last_clean":"2024-03-24T00:18:48.873061+0000","last_became_active":"2024-03-24T00:18:47.888620+0000","last_became_peered":"2024-03-24T00:18:47.888620+0000","last_unstale":"2024-03-24T00:18:48.873061+0000","last_undegraded":"2024-03-24T00:18:48.873061+0000","last_fullsized":"2024-03-24T00:18:48.873061+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:51.483286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.621605+0000","last_change":"2024-03-24T00:18:43.251824+0000","last_active":"2024-03-24T00:18:51.621605+0000","last_peered":"2024-03-24T00:18:51.621605+0000","last_clean":"2024-03-24T00:18:51.621605+0000","last_became_active":"2024-03-24T00:18:43.251458+0000","last_became_peered":"2024-03-24T00:18:43.251458+0000","last_unstale":"2024-03-24T00:18:51.621605+0000","last_undegraded":"2024-03-24T00:18:51.621605+0000","last_fullsized":"2024-03-24T00:18:51.621605+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:13:02.303670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.374289+0000","last_change":"2024-03-24T00:18:43.891289+0000","last_active":"2024-03-24T00:18:51.374289+0000","last_peered":"2024-03-24T00:18:51.374289+0000","last_clean":"2024-03-24T00:18:51.374289+0000","last_became_active":"2024-03-24T00:18:43.890725+0000","last_became_peered":"2024-03-24T00:18:43.890725+0000","last_unstale":"2024-03-24T00:18:51.374289+0000","last_undegraded":"2024-03-24T00:18:51.374289+0000","last_fullsized":"2024-03-24T00:18:51.374289+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:37:41.096520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.872677+0000","last_change":"2024-03-24T00:18:47.896175+0000","last_active":"2024-03-24T00:18:48.872677+0000","last_peered":"2024-03-24T00:18:48.872677+0000","last_clean":"2024-03-24T00:18:48.872677+0000","last_became_active":"2024-03-24T00:18:47.895723+0000","last_became_peered":"2024-03-24T00:18:47.895723+0000","last_unstale":"2024-03-24T00:18:48.872677+0000","last_undegraded":"2024-03-24T00:18:48.872677+0000","last_fullsized":"2024-03-24T00:18:48.872677+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:41:49.167532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:19:08.387233+0000","last_change":"2024-03-24T00:18:43.258986+0000","last_active":"2024-03-24T00:19:08.387233+0000","last_peered":"2024-03-24T00:19:08.387233+0000","last_clean":"2024-03-24T00:19:08.387233+0000","last_became_active":"2024-03-24T00:18:43.258654+0000","last_became_peered":"2024-03-24T00:18:43.258654+0000","last_unstale":"2024-03-24T00:19:08.387233+0000","last_undegraded":"2024-03-24T00:19:08.387233+0000","last_fullsized":"2024-03-24T00:19:08.387233+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:34:44.339991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.622197+0000","last_change":"2024-03-24T00:18:43.885852+0000","last_active":"2024-03-24T00:18:51.622197+0000","last_peered":"2024-03-24T00:18:51.622197+0000","last_clean":"2024-03-24T00:18:51.622197+0000","last_became_active":"2024-03-24T00:18:43.885538+0000","last_became_peered":"2024-03-24T00:18:43.885538+0000","last_unstale":"2024-03-24T00:18:51.622197+0000","last_undegraded":"2024-03-24T00:18:51.622197+0000","last_fullsized":"2024-03-24T00:18:51.622197+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:42:02.879892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.318919+0000","last_change":"2024-03-24T00:18:47.893230+0000","last_active":"2024-03-24T00:18:51.318919+0000","last_peered":"2024-03-24T00:18:51.318919+0000","last_clean":"2024-03-24T00:18:51.318919+0000","last_became_active":"2024-03-24T00:18:47.892522+0000","last_became_peered":"2024-03-24T00:18:47.892522+0000","last_unstale":"2024-03-24T00:18:51.318919+0000","last_undegraded":"2024-03-24T00:18:51.318919+0000","last_fullsized":"2024-03-24T00:18:51.318919+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:10:11.768154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.445893+0000","last_change":"2024-03-24T00:18:43.262255+0000","last_active":"2024-03-24T00:18:51.445893+0000","last_peered":"2024-03-24T00:18:51.445893+0000","last_clean":"2024-03-24T00:18:51.445893+0000","last_became_active":"2024-03-24T00:18:43.261940+0000","last_became_peered":"2024-03-24T00:18:43.261940+0000","last_unstale":"2024-03-24T00:18:51.445893+0000","last_undegraded":"2024-03-24T00:18:51.445893+0000","last_fullsized":"2024-03-24T00:18:51.445893+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:58:48.789845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.393102+0000","last_change":"2024-03-24T00:18:43.883046+0000","last_active":"2024-03-24T00:18:51.393102+0000","last_peered":"2024-03-24T00:18:51.393102+0000","last_clean":"2024-03-24T00:18:51.393102+0000","last_became_active":"2024-03-24T00:18:43.881405+0000","last_became_peered":"2024-03-24T00:18:43.881405+0000","last_unstale":"2024-03-24T00:18:51.393102+0000","last_undegraded":"2024-03-24T00:18:51.393102+0000","last_fullsized":"2024-03-24T00:18:51.393102+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:21:00.047019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.401934+0000","last_change":"2024-03-24T00:18:47.897630+0000","last_active":"2024-03-24T00:18:51.401934+0000","last_peered":"2024-03-24T00:18:51.401934+0000","last_clean":"2024-03-24T00:18:51.401934+0000","last_became_active":"2024-03-24T00:18:47.897211+0000","last_became_peered":"2024-03-24T00:18:47.897211+0000","last_unstale":"2024-03-24T00:18:51.401934+0000","last_undegraded":"2024-03-24T00:18:51.401934+0000","last_fullsized":"2024-03-24T00:18:51.401934+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:22:48.762870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.621544+0000","last_change":"2024-03-24T00:18:43.262134+0000","last_active":"2024-03-24T00:18:51.621544+0000","last_peered":"2024-03-24T00:18:51.621544+0000","last_clean":"2024-03-24T00:18:51.621544+0000","last_became_active":"2024-03-24T00:18:43.261783+0000","last_became_peered":"2024-03-24T00:18:43.261783+0000","last_unstale":"2024-03-24T00:18:51.621544+0000","last_undegraded":"2024-03-24T00:18:51.621544+0000","last_fullsized":"2024-03-24T00:18:51.621544+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:16:31.060204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.869298+0000","last_change":"2024-03-24T00:18:43.880532+0000","last_active":"2024-03-24T00:18:48.869298+0000","last_peered":"2024-03-24T00:18:48.869298+0000","last_clean":"2024-03-24T00:18:48.869298+0000","last_became_active":"2024-03-24T00:18:43.880198+0000","last_became_peered":"2024-03-24T00:18:43.880198+0000","last_unstale":"2024-03-24T00:18:48.869298+0000","last_undegraded":"2024-03-24T00:18:48.869298+0000","last_fullsized":"2024-03-24T00:18:48.869298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:33:16.710842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.393392+0000","last_change":"2024-03-24T00:18:47.893607+0000","last_active":"2024-03-24T00:18:51.393392+0000","last_peered":"2024-03-24T00:18:51.393392+0000","last_clean":"2024-03-24T00:18:51.393392+0000","last_became_active":"2024-03-24T00:18:47.893026+0000","last_became_peered":"2024-03-24T00:18:47.893026+0000","last_unstale":"2024-03-24T00:18:51.393392+0000","last_undegraded":"2024-03-24T00:18:51.393392+0000","last_fullsized":"2024-03-24T00:18:51.393392+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:37:58.994146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:19:07.379066+0000","last_change":"2024-03-24T00:18:43.257520+0000","last_active":"2024-03-24T00:19:07.379066+0000","last_peered":"2024-03-24T00:19:07.379066+0000","last_clean":"2024-03-24T00:19:07.379066+0000","last_became_active":"2024-03-24T00:18:43.257175+0000","last_became_peered":"2024-03-24T00:18:43.257175+0000","last_unstale":"2024-03-24T00:19:07.379066+0000","last_undegraded":"2024-03-24T00:19:07.379066+0000","last_fullsized":"2024-03-24T00:19:07.379066+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:52:45.244983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:49.904115+0000","last_change":"2024-03-24T00:18:43.887739+0000","last_active":"2024-03-24T00:18:49.904115+0000","last_peered":"2024-03-24T00:18:49.904115+0000","last_clean":"2024-03-24T00:18:49.904115+0000","last_became_active":"2024-03-24T00:18:43.887379+0000","last_became_peered":"2024-03-24T00:18:43.887379+0000","last_unstale":"2024-03-24T00:18:49.904115+0000","last_undegraded":"2024-03-24T00:18:49.904115+0000","last_fullsized":"2024-03-24T00:18:49.904115+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:10:50.384759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.869337+0000","last_change":"2024-03-24T00:18:47.890827+0000","last_active":"2024-03-24T00:18:48.869337+0000","last_peered":"2024-03-24T00:18:48.869337+0000","last_clean":"2024-03-24T00:18:48.869337+0000","last_became_active":"2024-03-24T00:18:47.890399+0000","last_became_peered":"2024-03-24T00:18:47.890399+0000","last_unstale":"2024-03-24T00:18:48.869337+0000","last_undegraded":"2024-03-24T00:18:48.869337+0000","last_fullsized":"2024-03-24T00:18:48.869337+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:46:54.575468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:19:07.379186+0000","last_change":"2024-03-24T00:18:43.248132+0000","last_active":"2024-03-24T00:19:07.379186+0000","last_peered":"2024-03-24T00:19:07.379186+0000","last_clean":"2024-03-24T00:19:07.379186+0000","last_became_active":"2024-03-24T00:18:43.247772+0000","last_became_peered":"2024-03-24T00:18:43.247772+0000","last_unstale":"2024-03-24T00:19:07.379186+0000","last_undegraded":"2024-03-24T00:19:07.379186+0000","last_fullsized":"2024-03-24T00:19:07.379186+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:09.330155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.401794+0000","last_change":"2024-03-24T00:18:43.889920+0000","last_active":"2024-03-24T00:18:51.401794+0000","last_peered":"2024-03-24T00:18:51.401794+0000","last_clean":"2024-03-24T00:18:51.401794+0000","last_became_active":"2024-03-24T00:18:43.889593+0000","last_became_peered":"2024-03-24T00:18:43.889593+0000","last_unstale":"2024-03-24T00:18:51.401794+0000","last_undegraded":"2024-03-24T00:18:51.401794+0000","last_fullsized":"2024-03-24T00:18:51.401794+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:48:41.101019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.627755+0000","last_change":"2024-03-24T00:18:47.893133+0000","last_active":"2024-03-24T00:18:51.627755+0000","last_peered":"2024-03-24T00:18:51.627755+0000","last_clean":"2024-03-24T00:18:51.627755+0000","last_became_active":"2024-03-24T00:18:47.892731+0000","last_became_peered":"2024-03-24T00:18:47.892731+0000","last_unstale":"2024-03-24T00:18:51.627755+0000","last_undegraded":"2024-03-24T00:18:51.627755+0000","last_fullsized":"2024-03-24T00:18:51.627755+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:51:09.539852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.621608+0000","last_change":"2024-03-24T00:18:43.257697+0000","last_active":"2024-03-24T00:18:51.621608+0000","last_peered":"2024-03-24T00:18:51.621608+0000","last_clean":"2024-03-24T00:18:51.621608+0000","last_became_active":"2024-03-24T00:18:43.257334+0000","last_became_peered":"2024-03-24T00:18:43.257334+0000","last_unstale":"2024-03-24T00:18:51.621608+0000","last_undegraded":"2024-03-24T00:18:51.621608+0000","last_fullsized":"2024-03-24T00:18:51.621608+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:35:22.817064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:49.904020+0000","last_change":"2024-03-24T00:18:43.896238+0000","last_active":"2024-03-24T00:18:49.904020+0000","last_peered":"2024-03-24T00:18:49.904020+0000","last_clean":"2024-03-24T00:18:49.904020+0000","last_became_active":"2024-03-24T00:18:43.895792+0000","last_became_peered":"2024-03-24T00:18:43.895792+0000","last_unstale":"2024-03-24T00:18:49.904020+0000","last_undegraded":"2024-03-24T00:18:49.904020+0000","last_fullsized":"2024-03-24T00:18:49.904020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:55:11.865771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.869305+0000","last_change":"2024-03-24T00:18:47.891407+0000","last_active":"2024-03-24T00:18:48.869305+0000","last_peered":"2024-03-24T00:18:48.869305+0000","last_clean":"2024-03-24T00:18:48.869305+0000","last_became_active":"2024-03-24T00:18:47.891042+0000","last_became_peered":"2024-03-24T00:18:47.891042+0000","last_unstale":"2024-03-24T00:18:48.869305+0000","last_undegraded":"2024-03-24T00:18:48.869305+0000","last_fullsized":"2024-03-24T00:18:48.869305+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:14:02.939199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'11","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:19:08.880371+0000","last_change":"2024-03-24T00:18:43.258742+0000","last_active":"2024-03-24T00:19:08.880371+0000","last_peered":"2024-03-24T00:19:08.880371+0000","last_clean":"2024-03-24T00:19:08.880371+0000","last_became_active":"2024-03-24T00:18:43.258412+0000","last_became_peered":"2024-03-24T00:18:43.258412+0000","last_unstale":"2024-03-24T00:19:08.880371+0000","last_undegraded":"2024-03-24T00:19:08.880371+0000","last_fullsized":"2024-03-24T00:19:08.880371+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:30:49.364147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":12,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.374582+0000","last_change":"2024-03-24T00:18:43.893740+0000","last_active":"2024-03-24T00:18:51.374582+0000","last_peered":"2024-03-24T00:18:51.374582+0000","last_clean":"2024-03-24T00:18:51.374582+0000","last_became_active":"2024-03-24T00:18:43.893277+0000","last_became_peered":"2024-03-24T00:18:43.893277+0000","last_unstale":"2024-03-24T00:18:51.374582+0000","last_undegraded":"2024-03-24T00:18:51.374582+0000","last_fullsized":"2024-03-24T00:18:51.374582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:21:47.040629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.622387+0000","last_change":"2024-03-24T00:18:47.898769+0000","last_active":"2024-03-24T00:18:51.622387+0000","last_peered":"2024-03-24T00:18:51.622387+0000","last_clean":"2024-03-24T00:18:51.622387+0000","last_became_active":"2024-03-24T00:18:47.898214+0000","last_became_peered":"2024-03-24T00:18:47.898214+0000","last_unstale":"2024-03-24T00:18:51.622387+0000","last_undegraded":"2024-03-24T00:18:51.622387+0000","last_fullsized":"2024-03-24T00:18:51.622387+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:24:42.022429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:19:08.387449+0000","last_change":"2024-03-24T00:18:43.262532+0000","last_active":"2024-03-24T00:19:08.387449+0000","last_peered":"2024-03-24T00:19:08.387449+0000","last_clean":"2024-03-24T00:19:08.387449+0000","last_became_active":"2024-03-24T00:18:43.262211+0000","last_became_peered":"2024-03-24T00:18:43.262211+0000","last_unstale":"2024-03-24T00:19:08.387449+0000","last_undegraded":"2024-03-24T00:19:08.387449+0000","last_fullsized":"2024-03-24T00:19:08.387449+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:23:22.862495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.446229+0000","last_change":"2024-03-24T00:18:43.881691+0000","last_active":"2024-03-24T00:18:51.446229+0000","last_peered":"2024-03-24T00:18:51.446229+0000","last_clean":"2024-03-24T00:18:51.446229+0000","last_became_active":"2024-03-24T00:18:43.881386+0000","last_became_peered":"2024-03-24T00:18:43.881386+0000","last_unstale":"2024-03-24T00:18:51.446229+0000","last_undegraded":"2024-03-24T00:18:51.446229+0000","last_fullsized":"2024-03-24T00:18:51.446229+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:33:24.808315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.318722+0000","last_change":"2024-03-24T00:18:47.883340+0000","last_active":"2024-03-24T00:18:51.318722+0000","last_peered":"2024-03-24T00:18:51.318722+0000","last_clean":"2024-03-24T00:18:51.318722+0000","last_became_active":"2024-03-24T00:18:47.882934+0000","last_became_peered":"2024-03-24T00:18:47.882934+0000","last_unstale":"2024-03-24T00:18:51.318722+0000","last_undegraded":"2024-03-24T00:18:51.318722+0000","last_fullsized":"2024-03-24T00:18:51.318722+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:30:56.659709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.621538+0000","last_change":"2024-03-24T00:18:43.253444+0000","last_active":"2024-03-24T00:18:51.621538+0000","last_peered":"2024-03-24T00:18:51.621538+0000","last_clean":"2024-03-24T00:18:51.621538+0000","last_became_active":"2024-03-24T00:18:43.252121+0000","last_became_peered":"2024-03-24T00:18:43.252121+0000","last_unstale":"2024-03-24T00:18:51.621538+0000","last_undegraded":"2024-03-24T00:18:51.621538+0000","last_fullsized":"2024-03-24T00:18:51.621538+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:44:48.663995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.195190+0000","last_change":"2024-03-24T00:18:43.890286+0000","last_active":"2024-03-24T00:18:51.195190+0000","last_peered":"2024-03-24T00:18:51.195190+0000","last_clean":"2024-03-24T00:18:51.195190+0000","last_became_active":"2024-03-24T00:18:43.889918+0000","last_became_peered":"2024-03-24T00:18:43.889918+0000","last_unstale":"2024-03-24T00:18:51.195190+0000","last_undegraded":"2024-03-24T00:18:51.195190+0000","last_fullsized":"2024-03-24T00:18:51.195190+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:13:50.804306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.869709+0000","last_change":"2024-03-24T00:18:47.897513+0000","last_active":"2024-03-24T00:18:48.869709+0000","last_peered":"2024-03-24T00:18:48.869709+0000","last_clean":"2024-03-24T00:18:48.869709+0000","last_became_active":"2024-03-24T00:18:47.896606+0000","last_became_peered":"2024-03-24T00:18:47.896606+0000","last_unstale":"2024-03-24T00:18:48.869709+0000","last_undegraded":"2024-03-24T00:18:48.869709+0000","last_fullsized":"2024-03-24T00:18:48.869709+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:09:09.389437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.627220+0000","last_change":"2024-03-24T00:18:43.251682+0000","last_active":"2024-03-24T00:18:51.627220+0000","last_peered":"2024-03-24T00:18:51.627220+0000","last_clean":"2024-03-24T00:18:51.627220+0000","last_became_active":"2024-03-24T00:18:43.250953+0000","last_became_peered":"2024-03-24T00:18:43.250953+0000","last_unstale":"2024-03-24T00:18:51.627220+0000","last_undegraded":"2024-03-24T00:18:51.627220+0000","last_fullsized":"2024-03-24T00:18:51.627220+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:22:35.218781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.195197+0000","last_change":"2024-03-24T00:18:43.876334+0000","last_active":"2024-03-24T00:18:51.195197+0000","last_peered":"2024-03-24T00:18:51.195197+0000","last_clean":"2024-03-24T00:18:51.195197+0000","last_became_active":"2024-03-24T00:18:43.875397+0000","last_became_peered":"2024-03-24T00:18:43.875397+0000","last_unstale":"2024-03-24T00:18:51.195197+0000","last_undegraded":"2024-03-24T00:18:51.195197+0000","last_fullsized":"2024-03-24T00:18:51.195197+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:40:20.474055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.446561+0000","last_change":"2024-03-24T00:18:47.897488+0000","last_active":"2024-03-24T00:18:51.446561+0000","last_peered":"2024-03-24T00:18:51.446561+0000","last_clean":"2024-03-24T00:18:51.446561+0000","last_became_active":"2024-03-24T00:18:47.897042+0000","last_became_peered":"2024-03-24T00:18:47.897042+0000","last_unstale":"2024-03-24T00:18:51.446561+0000","last_undegraded":"2024-03-24T00:18:51.446561+0000","last_fullsized":"2024-03-24T00:18:51.446561+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:23:13.071695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.373968+0000","last_change":"2024-03-24T00:18:43.255278+0000","last_active":"2024-03-24T00:18:51.373968+0000","last_peered":"2024-03-24T00:18:51.373968+0000","last_clean":"2024-03-24T00:18:51.373968+0000","last_became_active":"2024-03-24T00:18:43.254943+0000","last_became_peered":"2024-03-24T00:18:43.254943+0000","last_unstale":"2024-03-24T00:18:51.373968+0000","last_undegraded":"2024-03-24T00:18:51.373968+0000","last_fullsized":"2024-03-24T00:18:51.373968+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:09:56.755303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.621966+0000","last_change":"2024-03-24T00:18:43.892620+0000","last_active":"2024-03-24T00:18:51.621966+0000","last_peered":"2024-03-24T00:18:51.621966+0000","last_clean":"2024-03-24T00:18:51.621966+0000","last_became_active":"2024-03-24T00:18:43.892033+0000","last_became_peered":"2024-03-24T00:18:43.892033+0000","last_unstale":"2024-03-24T00:18:51.621966+0000","last_undegraded":"2024-03-24T00:18:51.621966+0000","last_fullsized":"2024-03-24T00:18:51.621966+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:30:19.472861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.622125+0000","last_change":"2024-03-24T00:18:47.895623+0000","last_active":"2024-03-24T00:18:51.622125+0000","last_peered":"2024-03-24T00:18:51.622125+0000","last_clean":"2024-03-24T00:18:51.622125+0000","last_became_active":"2024-03-24T00:18:47.895242+0000","last_became_peered":"2024-03-24T00:18:47.895242+0000","last_unstale":"2024-03-24T00:18:51.622125+0000","last_undegraded":"2024-03-24T00:18:51.622125+0000","last_fullsized":"2024-03-24T00:18:51.622125+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:50:32.775988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.374566+0000","last_change":"2024-03-24T00:18:43.265575+0000","last_active":"2024-03-24T00:18:51.374566+0000","last_peered":"2024-03-24T00:18:51.374566+0000","last_clean":"2024-03-24T00:18:51.374566+0000","last_became_active":"2024-03-24T00:18:43.265265+0000","last_became_peered":"2024-03-24T00:18:43.265265+0000","last_unstale":"2024-03-24T00:18:51.374566+0000","last_undegraded":"2024-03-24T00:18:51.374566+0000","last_fullsized":"2024-03-24T00:18:51.374566+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:15:46.566812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.318780+0000","last_change":"2024-03-24T00:18:43.898574+0000","last_active":"2024-03-24T00:18:51.318780+0000","last_peered":"2024-03-24T00:18:51.318780+0000","last_clean":"2024-03-24T00:18:51.318780+0000","last_became_active":"2024-03-24T00:18:43.898153+0000","last_became_peered":"2024-03-24T00:18:43.898153+0000","last_unstale":"2024-03-24T00:18:51.318780+0000","last_undegraded":"2024-03-24T00:18:51.318780+0000","last_fullsized":"2024-03-24T00:18:51.318780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:50:15.317257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.374214+0000","last_change":"2024-03-24T00:18:47.896336+0000","last_active":"2024-03-24T00:18:51.374214+0000","last_peered":"2024-03-24T00:18:51.374214+0000","last_clean":"2024-03-24T00:18:51.374214+0000","last_became_active":"2024-03-24T00:18:47.895939+0000","last_became_peered":"2024-03-24T00:18:47.895939+0000","last_unstale":"2024-03-24T00:18:51.374214+0000","last_undegraded":"2024-03-24T00:18:51.374214+0000","last_fullsized":"2024-03-24T00:18:51.374214+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:54:03.195827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.869425+0000","last_change":"2024-03-24T00:18:43.253081+0000","last_active":"2024-03-24T00:18:48.869425+0000","last_peered":"2024-03-24T00:18:48.869425+0000","last_clean":"2024-03-24T00:18:48.869425+0000","last_became_active":"2024-03-24T00:18:43.252415+0000","last_became_peered":"2024-03-24T00:18:43.252415+0000","last_unstale":"2024-03-24T00:18:48.869425+0000","last_undegraded":"2024-03-24T00:18:48.869425+0000","last_fullsized":"2024-03-24T00:18:48.869425+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:06:45.200233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.870227+0000","last_change":"2024-03-24T00:18:43.885820+0000","last_active":"2024-03-24T00:18:48.870227+0000","last_peered":"2024-03-24T00:18:48.870227+0000","last_clean":"2024-03-24T00:18:48.870227+0000","last_became_active":"2024-03-24T00:18:43.885485+0000","last_became_peered":"2024-03-24T00:18:43.885485+0000","last_unstale":"2024-03-24T00:18:48.870227+0000","last_undegraded":"2024-03-24T00:18:48.870227+0000","last_fullsized":"2024-03-24T00:18:48.870227+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:09:12.286000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.446402+0000","last_change":"2024-03-24T00:18:47.899054+0000","last_active":"2024-03-24T00:18:51.446402+0000","last_peered":"2024-03-24T00:18:51.446402+0000","last_clean":"2024-03-24T00:18:51.446402+0000","last_became_active":"2024-03-24T00:18:47.898412+0000","last_became_peered":"2024-03-24T00:18:47.898412+0000","last_unstale":"2024-03-24T00:18:51.446402+0000","last_undegraded":"2024-03-24T00:18:51.446402+0000","last_fullsized":"2024-03-24T00:18:51.446402+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:17:08.273323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.195367+0000","last_change":"2024-03-24T00:18:43.251402+0000","last_active":"2024-03-24T00:18:51.195367+0000","last_peered":"2024-03-24T00:18:51.195367+0000","last_clean":"2024-03-24T00:18:51.195367+0000","last_became_active":"2024-03-24T00:18:43.251055+0000","last_became_peered":"2024-03-24T00:18:43.251055+0000","last_unstale":"2024-03-24T00:18:51.195367+0000","last_undegraded":"2024-03-24T00:18:51.195367+0000","last_fullsized":"2024-03-24T00:18:51.195367+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:07:30.966065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.870122+0000","last_change":"2024-03-24T00:18:43.886594+0000","last_active":"2024-03-24T00:18:48.870122+0000","last_peered":"2024-03-24T00:18:48.870122+0000","last_clean":"2024-03-24T00:18:48.870122+0000","last_became_active":"2024-03-24T00:18:43.886251+0000","last_became_peered":"2024-03-24T00:18:43.886251+0000","last_unstale":"2024-03-24T00:18:48.870122+0000","last_undegraded":"2024-03-24T00:18:48.870122+0000","last_fullsized":"2024-03-24T00:18:48.870122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:55:24.662318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.446529+0000","last_change":"2024-03-24T00:18:47.901352+0000","last_active":"2024-03-24T00:18:51.446529+0000","last_peered":"2024-03-24T00:18:51.446529+0000","last_clean":"2024-03-24T00:18:51.446529+0000","last_became_active":"2024-03-24T00:18:47.900943+0000","last_became_peered":"2024-03-24T00:18:47.900943+0000","last_unstale":"2024-03-24T00:18:51.446529+0000","last_undegraded":"2024-03-24T00:18:51.446529+0000","last_fullsized":"2024-03-24T00:18:51.446529+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:25:29.626277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.622218+0000","last_change":"2024-03-24T00:18:43.885235+0000","last_active":"2024-03-24T00:18:51.622218+0000","last_peered":"2024-03-24T00:18:51.622218+0000","last_clean":"2024-03-24T00:18:51.622218+0000","last_became_active":"2024-03-24T00:18:43.884903+0000","last_became_peered":"2024-03-24T00:18:43.884903+0000","last_unstale":"2024-03-24T00:18:51.622218+0000","last_undegraded":"2024-03-24T00:18:51.622218+0000","last_fullsized":"2024-03-24T00:18:51.622218+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:01:31.929097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.446045+0000","last_change":"2024-03-24T00:18:43.263213+0000","last_active":"2024-03-24T00:18:51.446045+0000","last_peered":"2024-03-24T00:18:51.446045+0000","last_clean":"2024-03-24T00:18:51.446045+0000","last_became_active":"2024-03-24T00:18:43.262668+0000","last_became_peered":"2024-03-24T00:18:43.262668+0000","last_unstale":"2024-03-24T00:18:51.446045+0000","last_undegraded":"2024-03-24T00:18:51.446045+0000","last_fullsized":"2024-03-24T00:18:51.446045+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:18:15.518187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.195369+0000","last_change":"2024-03-24T00:18:47.891607+0000","last_active":"2024-03-24T00:18:51.195369+0000","last_peered":"2024-03-24T00:18:51.195369+0000","last_clean":"2024-03-24T00:18:51.195369+0000","last_became_active":"2024-03-24T00:18:47.891241+0000","last_became_peered":"2024-03-24T00:18:47.891241+0000","last_unstale":"2024-03-24T00:18:51.195369+0000","last_undegraded":"2024-03-24T00:18:51.195369+0000","last_fullsized":"2024-03-24T00:18:51.195369+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:14:29.593767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.393140+0000","last_change":"2024-03-24T00:18:43.894492+0000","last_active":"2024-03-24T00:18:51.393140+0000","last_peered":"2024-03-24T00:18:51.393140+0000","last_clean":"2024-03-24T00:18:51.393140+0000","last_became_active":"2024-03-24T00:18:43.893104+0000","last_became_peered":"2024-03-24T00:18:43.893104+0000","last_unstale":"2024-03-24T00:18:51.393140+0000","last_undegraded":"2024-03-24T00:18:51.393140+0000","last_fullsized":"2024-03-24T00:18:51.393140+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:26:32.130449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:19:07.377260+0000","last_change":"2024-03-24T00:18:43.251344+0000","last_active":"2024-03-24T00:19:07.377260+0000","last_peered":"2024-03-24T00:19:07.377260+0000","last_clean":"2024-03-24T00:19:07.377260+0000","last_became_active":"2024-03-24T00:18:43.250961+0000","last_became_peered":"2024-03-24T00:18:43.250961+0000","last_unstale":"2024-03-24T00:19:07.377260+0000","last_undegraded":"2024-03-24T00:19:07.377260+0000","last_fullsized":"2024-03-24T00:19:07.377260+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:13:38.423465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":356,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.195291+0000","last_change":"2024-03-24T00:17:27.864318+0000","last_active":"2024-03-24T00:18:51.195291+0000","last_peered":"2024-03-24T00:18:51.195291+0000","last_clean":"2024-03-24T00:18:51.195291+0000","last_became_active":"2024-03-24T00:17:27.863875+0000","last_became_peered":"2024-03-24T00:17:27.863875+0000","last_unstale":"2024-03-24T00:18:51.195291+0000","last_undegraded":"2024-03-24T00:18:51.195291+0000","last_fullsized":"2024-03-24T00:18:51.195291+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:14:04.731181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:14:04.731181+0000","last_clean_scrub_stamp":"2024-03-24T00:14:04.731181+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:53:25.036280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.622516+0000","last_change":"2024-03-24T00:18:47.895344+0000","last_active":"2024-03-24T00:18:51.622516+0000","last_peered":"2024-03-24T00:18:51.622516+0000","last_clean":"2024-03-24T00:18:51.622516+0000","last_became_active":"2024-03-24T00:18:47.894694+0000","last_became_peered":"2024-03-24T00:18:47.894694+0000","last_unstale":"2024-03-24T00:18:51.622516+0000","last_undegraded":"2024-03-24T00:18:51.622516+0000","last_fullsized":"2024-03-24T00:18:51.622516+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:16.969143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.446016+0000","last_change":"2024-03-24T00:18:43.890236+0000","last_active":"2024-03-24T00:18:51.446016+0000","last_peered":"2024-03-24T00:18:51.446016+0000","last_clean":"2024-03-24T00:18:51.446016+0000","last_became_active":"2024-03-24T00:18:43.889896+0000","last_became_peered":"2024-03-24T00:18:43.889896+0000","last_unstale":"2024-03-24T00:18:51.446016+0000","last_undegraded":"2024-03-24T00:18:51.446016+0000","last_fullsized":"2024-03-24T00:18:51.446016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:13:19.443445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.195149+0000","last_change":"2024-03-24T00:18:43.260722+0000","last_active":"2024-03-24T00:18:51.195149+0000","last_peered":"2024-03-24T00:18:51.195149+0000","last_clean":"2024-03-24T00:18:51.195149+0000","last_became_active":"2024-03-24T00:18:43.260244+0000","last_became_peered":"2024-03-24T00:18:43.260244+0000","last_unstale":"2024-03-24T00:18:51.195149+0000","last_undegraded":"2024-03-24T00:18:51.195149+0000","last_fullsized":"2024-03-24T00:18:51.195149+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:37:37.838046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.318984+0000","last_change":"2024-03-24T00:18:47.891168+0000","last_active":"2024-03-24T00:18:51.318984+0000","last_peered":"2024-03-24T00:18:51.318984+0000","last_clean":"2024-03-24T00:18:51.318984+0000","last_became_active":"2024-03-24T00:18:47.890713+0000","last_became_peered":"2024-03-24T00:18:47.890713+0000","last_unstale":"2024-03-24T00:18:51.318984+0000","last_undegraded":"2024-03-24T00:18:51.318984+0000","last_fullsized":"2024-03-24T00:18:51.318984+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:20:30.858381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.374297+0000","last_change":"2024-03-24T00:18:43.256638+0000","last_active":"2024-03-24T00:18:51.374297+0000","last_peered":"2024-03-24T00:18:51.374297+0000","last_clean":"2024-03-24T00:18:51.374297+0000","last_became_active":"2024-03-24T00:18:43.256118+0000","last_became_peered":"2024-03-24T00:18:43.256118+0000","last_unstale":"2024-03-24T00:18:51.374297+0000","last_undegraded":"2024-03-24T00:18:51.374297+0000","last_fullsized":"2024-03-24T00:18:51.374297+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:40:05.523388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.446497+0000","last_change":"2024-03-24T00:18:43.885293+0000","last_active":"2024-03-24T00:18:51.446497+0000","last_peered":"2024-03-24T00:18:51.446497+0000","last_clean":"2024-03-24T00:18:51.446497+0000","last_became_active":"2024-03-24T00:18:43.884989+0000","last_became_peered":"2024-03-24T00:18:43.884989+0000","last_unstale":"2024-03-24T00:18:51.446497+0000","last_undegraded":"2024-03-24T00:18:51.446497+0000","last_fullsized":"2024-03-24T00:18:51.446497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:32:01.519131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.393102+0000","last_change":"2024-03-24T00:18:47.894253+0000","last_active":"2024-03-24T00:18:51.393102+0000","last_peered":"2024-03-24T00:18:51.393102+0000","last_clean":"2024-03-24T00:18:51.393102+0000","last_became_active":"2024-03-24T00:18:47.892654+0000","last_became_peered":"2024-03-24T00:18:47.892654+0000","last_unstale":"2024-03-24T00:18:51.393102+0000","last_undegraded":"2024-03-24T00:18:51.393102+0000","last_fullsized":"2024-03-24T00:18:51.393102+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:59:43.400810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:19:08.387581+0000","last_change":"2024-03-24T00:18:43.253074+0000","last_active":"2024-03-24T00:19:08.387581+0000","last_peered":"2024-03-24T00:19:08.387581+0000","last_clean":"2024-03-24T00:19:08.387581+0000","last_became_active":"2024-03-24T00:18:43.252382+0000","last_became_peered":"2024-03-24T00:18:43.252382+0000","last_unstale":"2024-03-24T00:19:08.387581+0000","last_undegraded":"2024-03-24T00:19:08.387581+0000","last_fullsized":"2024-03-24T00:19:08.387581+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:23:05.853912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.393418+0000","last_change":"2024-03-24T00:18:43.883654+0000","last_active":"2024-03-24T00:18:51.393418+0000","last_peered":"2024-03-24T00:18:51.393418+0000","last_clean":"2024-03-24T00:18:51.393418+0000","last_became_active":"2024-03-24T00:18:43.883335+0000","last_became_peered":"2024-03-24T00:18:43.883335+0000","last_unstale":"2024-03-24T00:18:51.393418+0000","last_undegraded":"2024-03-24T00:18:51.393418+0000","last_fullsized":"2024-03-24T00:18:51.393418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:42:10.856037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.872320+0000","last_change":"2024-03-24T00:18:47.888562+0000","last_active":"2024-03-24T00:18:48.872320+0000","last_peered":"2024-03-24T00:18:48.872320+0000","last_clean":"2024-03-24T00:18:48.872320+0000","last_became_active":"2024-03-24T00:18:47.888074+0000","last_became_peered":"2024-03-24T00:18:47.888074+0000","last_unstale":"2024-03-24T00:18:48.872320+0000","last_undegraded":"2024-03-24T00:18:48.872320+0000","last_fullsized":"2024-03-24T00:18:48.872320+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:01:04.617098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:19:07.378657+0000","last_change":"2024-03-24T00:18:43.261668+0000","last_active":"2024-03-24T00:19:07.378657+0000","last_peered":"2024-03-24T00:19:07.378657+0000","last_clean":"2024-03-24T00:19:07.378657+0000","last_became_active":"2024-03-24T00:18:43.261279+0000","last_became_peered":"2024-03-24T00:18:43.261279+0000","last_unstale":"2024-03-24T00:19:07.378657+0000","last_undegraded":"2024-03-24T00:19:07.378657+0000","last_fullsized":"2024-03-24T00:19:07.378657+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:36:05.390006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.401988+0000","last_change":"2024-03-24T00:18:43.889854+0000","last_active":"2024-03-24T00:18:51.401988+0000","last_peered":"2024-03-24T00:18:51.401988+0000","last_clean":"2024-03-24T00:18:51.401988+0000","last_became_active":"2024-03-24T00:18:43.889540+0000","last_became_peered":"2024-03-24T00:18:43.889540+0000","last_unstale":"2024-03-24T00:18:51.401988+0000","last_undegraded":"2024-03-24T00:18:51.401988+0000","last_fullsized":"2024-03-24T00:18:51.401988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:33:58.060564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.627764+0000","last_change":"2024-03-24T00:18:47.895495+0000","last_active":"2024-03-24T00:18:51.627764+0000","last_peered":"2024-03-24T00:18:51.627764+0000","last_clean":"2024-03-24T00:18:51.627764+0000","last_became_active":"2024-03-24T00:18:47.895158+0000","last_became_peered":"2024-03-24T00:18:47.895158+0000","last_unstale":"2024-03-24T00:18:51.627764+0000","last_undegraded":"2024-03-24T00:18:51.627764+0000","last_fullsized":"2024-03-24T00:18:51.627764+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:22:02.132419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:19:07.378431+0000","last_change":"2024-03-24T00:18:43.253063+0000","last_active":"2024-03-24T00:19:07.378431+0000","last_peered":"2024-03-24T00:19:07.378431+0000","last_clean":"2024-03-24T00:19:07.378431+0000","last_became_active":"2024-03-24T00:18:43.252534+0000","last_became_peered":"2024-03-24T00:18:43.252534+0000","last_unstale":"2024-03-24T00:19:07.378431+0000","last_undegraded":"2024-03-24T00:19:07.378431+0000","last_fullsized":"2024-03-24T00:19:07.378431+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:50:59.512780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.374063+0000","last_change":"2024-03-24T00:18:43.885574+0000","last_active":"2024-03-24T00:18:51.374063+0000","last_peered":"2024-03-24T00:18:51.374063+0000","last_clean":"2024-03-24T00:18:51.374063+0000","last_became_active":"2024-03-24T00:18:43.885257+0000","last_became_peered":"2024-03-24T00:18:43.885257+0000","last_unstale":"2024-03-24T00:18:51.374063+0000","last_undegraded":"2024-03-24T00:18:51.374063+0000","last_fullsized":"2024-03-24T00:18:51.374063+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:10.868436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.869745+0000","last_change":"2024-03-24T00:18:47.892739+0000","last_active":"2024-03-24T00:18:48.869745+0000","last_peered":"2024-03-24T00:18:48.869745+0000","last_clean":"2024-03-24T00:18:48.869745+0000","last_became_active":"2024-03-24T00:18:47.892291+0000","last_became_peered":"2024-03-24T00:18:47.892291+0000","last_unstale":"2024-03-24T00:18:48.869745+0000","last_undegraded":"2024-03-24T00:18:48.869745+0000","last_fullsized":"2024-03-24T00:18:48.869745+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:06:16.027968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:19:07.378287+0000","last_change":"2024-03-24T00:18:43.263856+0000","last_active":"2024-03-24T00:19:07.378287+0000","last_peered":"2024-03-24T00:19:07.378287+0000","last_clean":"2024-03-24T00:19:07.378287+0000","last_became_active":"2024-03-24T00:18:43.263032+0000","last_became_peered":"2024-03-24T00:18:43.263032+0000","last_unstale":"2024-03-24T00:19:07.378287+0000","last_undegraded":"2024-03-24T00:19:07.378287+0000","last_fullsized":"2024-03-24T00:19:07.378287+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:25:03.239696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.318699+0000","last_change":"2024-03-24T00:18:43.884674+0000","last_active":"2024-03-24T00:18:51.318699+0000","last_peered":"2024-03-24T00:18:51.318699+0000","last_clean":"2024-03-24T00:18:51.318699+0000","last_became_active":"2024-03-24T00:18:43.884346+0000","last_became_peered":"2024-03-24T00:18:43.884346+0000","last_unstale":"2024-03-24T00:18:51.318699+0000","last_undegraded":"2024-03-24T00:18:51.318699+0000","last_fullsized":"2024-03-24T00:18:51.318699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:58:31.474515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.627685+0000","last_change":"2024-03-24T00:18:47.892472+0000","last_active":"2024-03-24T00:18:51.627685+0000","last_peered":"2024-03-24T00:18:51.627685+0000","last_clean":"2024-03-24T00:18:51.627685+0000","last_became_active":"2024-03-24T00:18:47.892082+0000","last_became_peered":"2024-03-24T00:18:47.892082+0000","last_unstale":"2024-03-24T00:18:51.627685+0000","last_undegraded":"2024-03-24T00:18:51.627685+0000","last_fullsized":"2024-03-24T00:18:51.627685+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:26:01.084042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.872533+0000","last_change":"2024-03-24T00:18:43.263780+0000","last_active":"2024-03-24T00:18:48.872533+0000","last_peered":"2024-03-24T00:18:48.872533+0000","last_clean":"2024-03-24T00:18:48.872533+0000","last_became_active":"2024-03-24T00:18:43.263446+0000","last_became_peered":"2024-03-24T00:18:43.263446+0000","last_unstale":"2024-03-24T00:18:48.872533+0000","last_undegraded":"2024-03-24T00:18:48.872533+0000","last_fullsized":"2024-03-24T00:18:48.872533+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:09:09.821842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.393112+0000","last_change":"2024-03-24T00:18:43.888516+0000","last_active":"2024-03-24T00:18:51.393112+0000","last_peered":"2024-03-24T00:18:51.393112+0000","last_clean":"2024-03-24T00:18:51.393112+0000","last_became_active":"2024-03-24T00:18:43.888190+0000","last_became_peered":"2024-03-24T00:18:43.888190+0000","last_unstale":"2024-03-24T00:18:51.393112+0000","last_undegraded":"2024-03-24T00:18:51.393112+0000","last_fullsized":"2024-03-24T00:18:51.393112+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:53:35.685128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.627721+0000","last_change":"2024-03-24T00:18:47.897115+0000","last_active":"2024-03-24T00:18:51.627721+0000","last_peered":"2024-03-24T00:18:51.627721+0000","last_clean":"2024-03-24T00:18:51.627721+0000","last_became_active":"2024-03-24T00:18:47.896722+0000","last_became_peered":"2024-03-24T00:18:47.896722+0000","last_unstale":"2024-03-24T00:18:51.627721+0000","last_undegraded":"2024-03-24T00:18:51.627721+0000","last_fullsized":"2024-03-24T00:18:51.627721+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:44:40.541618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.373874+0000","last_change":"2024-03-24T00:18:43.263459+0000","last_active":"2024-03-24T00:18:51.373874+0000","last_peered":"2024-03-24T00:18:51.373874+0000","last_clean":"2024-03-24T00:18:51.373874+0000","last_became_active":"2024-03-24T00:18:43.263019+0000","last_became_peered":"2024-03-24T00:18:43.263019+0000","last_unstale":"2024-03-24T00:18:51.373874+0000","last_undegraded":"2024-03-24T00:18:51.373874+0000","last_fullsized":"2024-03-24T00:18:51.373874+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:10:57.171625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.195283+0000","last_change":"2024-03-24T00:18:43.886547+0000","last_active":"2024-03-24T00:18:51.195283+0000","last_peered":"2024-03-24T00:18:51.195283+0000","last_clean":"2024-03-24T00:18:51.195283+0000","last_became_active":"2024-03-24T00:18:43.886217+0000","last_became_peered":"2024-03-24T00:18:43.886217+0000","last_unstale":"2024-03-24T00:18:51.195283+0000","last_undegraded":"2024-03-24T00:18:51.195283+0000","last_fullsized":"2024-03-24T00:18:51.195283+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:59:33.576577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:49.903810+0000","last_change":"2024-03-24T00:18:47.894246+0000","last_active":"2024-03-24T00:18:49.903810+0000","last_peered":"2024-03-24T00:18:49.903810+0000","last_clean":"2024-03-24T00:18:49.903810+0000","last_became_active":"2024-03-24T00:18:47.893837+0000","last_became_peered":"2024-03-24T00:18:47.893837+0000","last_unstale":"2024-03-24T00:18:49.903810+0000","last_undegraded":"2024-03-24T00:18:49.903810+0000","last_fullsized":"2024-03-24T00:18:49.903810+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:33:06.044355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:19:08.387741+0000","last_change":"2024-03-24T00:18:43.256808+0000","last_active":"2024-03-24T00:19:08.387741+0000","last_peered":"2024-03-24T00:19:08.387741+0000","last_clean":"2024-03-24T00:19:08.387741+0000","last_became_active":"2024-03-24T00:18:43.256397+0000","last_became_peered":"2024-03-24T00:18:43.256397+0000","last_unstale":"2024-03-24T00:19:08.387741+0000","last_undegraded":"2024-03-24T00:19:08.387741+0000","last_fullsized":"2024-03-24T00:19:08.387741+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:36:13.505016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.195501+0000","last_change":"2024-03-24T00:18:43.887754+0000","last_active":"2024-03-24T00:18:51.195501+0000","last_peered":"2024-03-24T00:18:51.195501+0000","last_clean":"2024-03-24T00:18:51.195501+0000","last_became_active":"2024-03-24T00:18:43.886978+0000","last_became_peered":"2024-03-24T00:18:43.886978+0000","last_unstale":"2024-03-24T00:18:51.195501+0000","last_undegraded":"2024-03-24T00:18:51.195501+0000","last_fullsized":"2024-03-24T00:18:51.195501+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:43:18.186682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.401869+0000","last_change":"2024-03-24T00:18:47.887576+0000","last_active":"2024-03-24T00:18:51.401869+0000","last_peered":"2024-03-24T00:18:51.401869+0000","last_clean":"2024-03-24T00:18:51.401869+0000","last_became_active":"2024-03-24T00:18:47.887190+0000","last_became_peered":"2024-03-24T00:18:47.887190+0000","last_unstale":"2024-03-24T00:18:51.401869+0000","last_undegraded":"2024-03-24T00:18:51.401869+0000","last_fullsized":"2024-03-24T00:18:51.401869+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:10:10.396432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.621693+0000","last_change":"2024-03-24T00:18:43.259927+0000","last_active":"2024-03-24T00:18:51.621693+0000","last_peered":"2024-03-24T00:18:51.621693+0000","last_clean":"2024-03-24T00:18:51.621693+0000","last_became_active":"2024-03-24T00:18:43.259587+0000","last_became_peered":"2024-03-24T00:18:43.259587+0000","last_unstale":"2024-03-24T00:18:51.621693+0000","last_undegraded":"2024-03-24T00:18:51.621693+0000","last_fullsized":"2024-03-24T00:18:51.621693+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:26:06.380010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.195384+0000","last_change":"2024-03-24T00:18:43.897041+0000","last_active":"2024-03-24T00:18:51.195384+0000","last_peered":"2024-03-24T00:18:51.195384+0000","last_clean":"2024-03-24T00:18:51.195384+0000","last_became_active":"2024-03-24T00:18:43.896766+0000","last_became_peered":"2024-03-24T00:18:43.896766+0000","last_unstale":"2024-03-24T00:18:51.195384+0000","last_undegraded":"2024-03-24T00:18:51.195384+0000","last_fullsized":"2024-03-24T00:18:51.195384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:27:59.577589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.621814+0000","last_change":"2024-03-24T00:18:47.893146+0000","last_active":"2024-03-24T00:18:51.621814+0000","last_peered":"2024-03-24T00:18:51.621814+0000","last_clean":"2024-03-24T00:18:51.621814+0000","last_became_active":"2024-03-24T00:18:47.892150+0000","last_became_peered":"2024-03-24T00:18:47.892150+0000","last_unstale":"2024-03-24T00:18:51.621814+0000","last_undegraded":"2024-03-24T00:18:51.621814+0000","last_fullsized":"2024-03-24T00:18:51.621814+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:28:13.887194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.445937+0000","last_change":"2024-03-24T00:18:43.263972+0000","last_active":"2024-03-24T00:18:51.445937+0000","last_peered":"2024-03-24T00:18:51.445937+0000","last_clean":"2024-03-24T00:18:51.445937+0000","last_became_active":"2024-03-24T00:18:43.263577+0000","last_became_peered":"2024-03-24T00:18:43.263577+0000","last_unstale":"2024-03-24T00:18:51.445937+0000","last_undegraded":"2024-03-24T00:18:51.445937+0000","last_fullsized":"2024-03-24T00:18:51.445937+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:55:36.725420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.195619+0000","last_change":"2024-03-24T00:18:43.894722+0000","last_active":"2024-03-24T00:18:51.195619+0000","last_peered":"2024-03-24T00:18:51.195619+0000","last_clean":"2024-03-24T00:18:51.195619+0000","last_became_active":"2024-03-24T00:18:43.894359+0000","last_became_peered":"2024-03-24T00:18:43.894359+0000","last_unstale":"2024-03-24T00:18:51.195619+0000","last_undegraded":"2024-03-24T00:18:51.195619+0000","last_fullsized":"2024-03-24T00:18:51.195619+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:35:23.974163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.318815+0000","last_change":"2024-03-24T00:18:43.892749+0000","last_active":"2024-03-24T00:18:51.318815+0000","last_peered":"2024-03-24T00:18:51.318815+0000","last_clean":"2024-03-24T00:18:51.318815+0000","last_became_active":"2024-03-24T00:18:43.892014+0000","last_became_peered":"2024-03-24T00:18:43.892014+0000","last_unstale":"2024-03-24T00:18:51.318815+0000","last_undegraded":"2024-03-24T00:18:51.318815+0000","last_fullsized":"2024-03-24T00:18:51.318815+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:53:14.592384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.872878+0000","last_change":"2024-03-24T00:18:43.899540+0000","last_active":"2024-03-24T00:18:48.872878+0000","last_peered":"2024-03-24T00:18:48.872878+0000","last_clean":"2024-03-24T00:18:48.872878+0000","last_became_active":"2024-03-24T00:18:43.899206+0000","last_became_peered":"2024-03-24T00:18:43.899206+0000","last_unstale":"2024-03-24T00:18:48.872878+0000","last_undegraded":"2024-03-24T00:18:48.872878+0000","last_fullsized":"2024-03-24T00:18:48.872878+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:51:17.214705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.872328+0000","last_change":"2024-03-24T00:18:43.896254+0000","last_active":"2024-03-24T00:18:48.872328+0000","last_peered":"2024-03-24T00:18:48.872328+0000","last_clean":"2024-03-24T00:18:48.872328+0000","last_became_active":"2024-03-24T00:18:43.895889+0000","last_became_peered":"2024-03-24T00:18:43.895889+0000","last_unstale":"2024-03-24T00:18:48.872328+0000","last_undegraded":"2024-03-24T00:18:48.872328+0000","last_fullsized":"2024-03-24T00:18:48.872328+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:07:46.403117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.401814+0000","last_change":"2024-03-24T00:18:43.890813+0000","last_active":"2024-03-24T00:18:51.401814+0000","last_peered":"2024-03-24T00:18:51.401814+0000","last_clean":"2024-03-24T00:18:51.401814+0000","last_became_active":"2024-03-24T00:18:43.890407+0000","last_became_peered":"2024-03-24T00:18:43.890407+0000","last_unstale":"2024-03-24T00:18:51.401814+0000","last_undegraded":"2024-03-24T00:18:51.401814+0000","last_fullsized":"2024-03-24T00:18:51.401814+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:57:45.146055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.446399+0000","last_change":"2024-03-24T00:18:43.890280+0000","last_active":"2024-03-24T00:18:51.446399+0000","last_peered":"2024-03-24T00:18:51.446399+0000","last_clean":"2024-03-24T00:18:51.446399+0000","last_became_active":"2024-03-24T00:18:43.889888+0000","last_became_peered":"2024-03-24T00:18:43.889888+0000","last_unstale":"2024-03-24T00:18:51.446399+0000","last_undegraded":"2024-03-24T00:18:51.446399+0000","last_fullsized":"2024-03-24T00:18:51.446399+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:49:17.010306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.318973+0000","last_change":"2024-03-24T00:18:43.898453+0000","last_active":"2024-03-24T00:18:51.318973+0000","last_peered":"2024-03-24T00:18:51.318973+0000","last_clean":"2024-03-24T00:18:51.318973+0000","last_became_active":"2024-03-24T00:18:43.898081+0000","last_became_peered":"2024-03-24T00:18:43.898081+0000","last_unstale":"2024-03-24T00:18:51.318973+0000","last_undegraded":"2024-03-24T00:18:51.318973+0000","last_fullsized":"2024-03-24T00:18:51.318973+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:03:50.856755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:48.872863+0000","last_change":"2024-03-24T00:18:43.894667+0000","last_active":"2024-03-24T00:18:48.872863+0000","last_peered":"2024-03-24T00:18:48.872863+0000","last_clean":"2024-03-24T00:18:48.872863+0000","last_became_active":"2024-03-24T00:18:43.893869+0000","last_became_peered":"2024-03-24T00:18:43.893869+0000","last_unstale":"2024-03-24T00:18:48.872863+0000","last_undegraded":"2024-03-24T00:18:48.872863+0000","last_fullsized":"2024-03-24T00:18:48.872863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:51:43.273867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.393526+0000","last_change":"2024-03-24T00:18:43.893201+0000","last_active":"2024-03-24T00:18:51.393526+0000","last_peered":"2024-03-24T00:18:51.393526+0000","last_clean":"2024-03-24T00:18:51.393526+0000","last_became_active":"2024-03-24T00:18:43.892866+0000","last_became_peered":"2024-03-24T00:18:43.892866+0000","last_unstale":"2024-03-24T00:18:51.393526+0000","last_undegraded":"2024-03-24T00:18:51.393526+0000","last_fullsized":"2024-03-24T00:18:51.393526+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:03:16.108806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.621817+0000","last_change":"2024-03-24T00:18:43.882211+0000","last_active":"2024-03-24T00:18:51.621817+0000","last_peered":"2024-03-24T00:18:51.621817+0000","last_clean":"2024-03-24T00:18:51.621817+0000","last_became_active":"2024-03-24T00:18:43.881857+0000","last_became_peered":"2024-03-24T00:18:43.881857+0000","last_unstale":"2024-03-24T00:18:51.621817+0000","last_undegraded":"2024-03-24T00:18:51.621817+0000","last_fullsized":"2024-03-24T00:18:51.621817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:36:45.911853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.393331+0000","last_change":"2024-03-24T00:18:43.892599+0000","last_active":"2024-03-24T00:18:51.393331+0000","last_peered":"2024-03-24T00:18:51.393331+0000","last_clean":"2024-03-24T00:18:51.393331+0000","last_became_active":"2024-03-24T00:18:43.891966+0000","last_became_peered":"2024-03-24T00:18:43.891966+0000","last_unstale":"2024-03-24T00:18:51.393331+0000","last_undegraded":"2024-03-24T00:18:51.393331+0000","last_fullsized":"2024-03-24T00:18:51.393331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:22:03.486466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:49.903899+0000","last_change":"2024-03-24T00:18:43.892410+0000","last_active":"2024-03-24T00:18:49.903899+0000","last_peered":"2024-03-24T00:18:49.903899+0000","last_clean":"2024-03-24T00:18:49.903899+0000","last_became_active":"2024-03-24T00:18:43.891140+0000","last_became_peered":"2024-03-24T00:18:43.891140+0000","last_unstale":"2024-03-24T00:18:49.903899+0000","last_undegraded":"2024-03-24T00:18:49.903899+0000","last_fullsized":"2024-03-24T00:18:49.903899+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:03:14.252863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.446266+0000","last_change":"2024-03-24T00:18:43.876508+0000","last_active":"2024-03-24T00:18:51.446266+0000","last_peered":"2024-03-24T00:18:51.446266+0000","last_clean":"2024-03-24T00:18:51.446266+0000","last_became_active":"2024-03-24T00:18:43.876204+0000","last_became_peered":"2024-03-24T00:18:43.876204+0000","last_unstale":"2024-03-24T00:18:51.446266+0000","last_undegraded":"2024-03-24T00:18:51.446266+0000","last_fullsized":"2024-03-24T00:18:51.446266+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:35.358964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:18:51.318709+0000","last_change":"2024-03-24T00:18:43.887855+0000","last_active":"2024-03-24T00:18:51.318709+0000","last_peered":"2024-03-24T00:18:51.318709+0000","last_clean":"2024-03-24T00:18:51.318709+0000","last_became_active":"2024-03-24T00:18:43.887520+0000","last_became_peered":"2024-03-24T00:18:43.887520+0000","last_unstale":"2024-03-24T00:18:51.318709+0000","last_undegraded":"2024-03-24T00:18:51.318709+0000","last_fullsized":"2024-03-24T00:18:51.318709+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:17:15.930626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5402,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":6,"num_read_kb":8,"num_write":93,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":36867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383698,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":983040,"data_stored":477215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645335,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36912,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708240,"statfs":{"total":95995035648,"available":95957237760,"internally_reserved":0,"allocated":1613824,"data_stored":1072381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808861,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":1003520,"data_stored":478033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070497,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":991232,"data_stored":477819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332135,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28568,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716584,"statfs":{"total":95995035648,"available":95965782016,"internally_reserved":0,"allocated":1589248,"data_stored":1068787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495659,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":983040,"data_stored":477215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":500356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36304,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708848,"statfs":{"total":95995035648,"available":95957860352,"internally_reserved":0,"allocated":987136,"data_stored":477305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411392,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":481713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182458,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":995328,"data_stored":477409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920822,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36304,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708848,"statfs":{"total":95995035648,"available":95957860352,"internally_reserved":0,"allocated":987136,"data_stored":477237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757296,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36892,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708260,"statfs":{"total":95995035648,"available":95957258240,"internally_reserved":0,"allocated":1589248,"data_stored":1067765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":23141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:19:11.932 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-24T00:19:11.932 INFO:tasks.cephadm:Setting up client nodes... 2024-03-24T00:19:11.933 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-24T00:19:12.208 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:19:12.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:12 smithi078 ceph-mon[32429]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:19:12.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:12 smithi078 ceph-mon[32429]: mgrmap e17: x(active, since 7m), standbys: y, z 2024-03-24T00:19:12.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:12 smithi106 ceph-mon[36505]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:19:12.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:12 smithi106 ceph-mon[36505]: mgrmap e17: x(active, since 7m), standbys: y, z 2024-03-24T00:19:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:12 smithi107 ceph-mon[36485]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:19:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:12 smithi107 ceph-mon[36485]: mgrmap e17: x(active, since 7m), standbys: y, z 2024-03-24T00:19:13.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:13 smithi078 ceph-mon[32429]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:19:13.624 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:13 smithi078 ceph-mon[32429]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:19:13.624 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:13 smithi078 ceph-mon[32429]: pgmap v305: 129 pgs: 129 active+clean; 584 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-03-24T00:19:13.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:13 smithi106 ceph-mon[36505]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:19:13.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:13 smithi106 ceph-mon[36505]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:19:13.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:13 smithi106 ceph-mon[36505]: pgmap v305: 129 pgs: 129 active+clean; 584 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-03-24T00:19:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:13 smithi107 ceph-mon[36485]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:19:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:13 smithi107 ceph-mon[36485]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:19:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:13 smithi107 ceph-mon[36485]: pgmap v305: 129 pgs: 129 active+clean; 584 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-03-24T00:19:14.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.028+0000 7f2342bae700 1 -- 172.21.15.78:0/2225491055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f233c102d30 msgr2=0x7f233c105120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:14.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.028+0000 7f2342bae700 1 --2- 172.21.15.78:0/2225491055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f233c102d30 0x7f233c105120 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f233800b0a0 tx=0x7f233800b3b0 comp rx=0 tx=0).stop 2024-03-24T00:19:14.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.028+0000 7f2342bae700 1 -- 172.21.15.78:0/2225491055 shutdown_connections 2024-03-24T00:19:14.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.028+0000 7f2342bae700 1 --2- 172.21.15.78:0/2225491055 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f233c107f90 0x7f233c10a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:14.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.028+0000 7f2342bae700 1 --2- 172.21.15.78:0/2225491055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f233c102d30 0x7f233c105120 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:14.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.028+0000 7f2342bae700 1 --2- 172.21.15.78:0/2225491055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f233c105660 0x7f233c107a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.028+0000 7f2342bae700 1 -- 172.21.15.78:0/2225491055 >> 172.21.15.78:0/2225491055 conn(0x7f233c0fc680 msgr2=0x7f233c0feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.028+0000 7f2342bae700 1 -- 172.21.15.78:0/2225491055 shutdown_connections 2024-03-24T00:19:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.028+0000 7f2342bae700 1 -- 172.21.15.78:0/2225491055 wait complete. 2024-03-24T00:19:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.028+0000 7f2342bae700 1 Processor -- start 2024-03-24T00:19:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.029+0000 7f2342bae700 1 -- start start 2024-03-24T00:19:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.029+0000 7f2342bae700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f233c102d30 0x7f233c19a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.029+0000 7f2342bae700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f233c105660 0x7f233c19aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.029+0000 7f2342bae700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f233c107f90 0x7f233c1a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.029+0000 7f2342bae700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f233c10ddb0 con 0x7f233c102d30 2024-03-24T00:19:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.029+0000 7f2342bae700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f233c10dc30 con 0x7f233c105660 2024-03-24T00:19:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.029+0000 7f2342bae700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f233c10df30 con 0x7f233c107f90 2024-03-24T00:19:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.029+0000 7f234094a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f233c102d30 0x7f233c19a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.029+0000 7f234094a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f233c102d30 0x7f233c19a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58450/0 (socket says 172.21.15.78:58450) 2024-03-24T00:19:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.029+0000 7f234094a700 1 -- 172.21.15.78:0/2457212574 learned_addr learned my addr 172.21.15.78:0/2457212574 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:19:14.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.029+0000 7f2333fff700 1 --2- 172.21.15.78:0/2457212574 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f233c105660 0x7f233c19aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:14.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.029+0000 7f234114b700 1 --2- 172.21.15.78:0/2457212574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f233c107f90 0x7f233c1a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:14.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.029+0000 7f234094a700 1 -- 172.21.15.78:0/2457212574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f233c107f90 msgr2=0x7f233c1a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:14.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.029+0000 7f234094a700 1 --2- 172.21.15.78:0/2457212574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f233c107f90 0x7f233c1a1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:14.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.029+0000 7f234094a700 1 -- 172.21.15.78:0/2457212574 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f233c105660 msgr2=0x7f233c19aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:14.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.029+0000 7f234094a700 1 --2- 172.21.15.78:0/2457212574 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f233c105660 0x7f233c19aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:14.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.029+0000 7f234094a700 1 -- 172.21.15.78:0/2457212574 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f233c1a21b0 con 0x7f233c102d30 2024-03-24T00:19:14.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.029+0000 7f2333fff700 1 --2- 172.21.15.78:0/2457212574 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f233c105660 0x7f233c19aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:14.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.029+0000 7f234094a700 1 --2- 172.21.15.78:0/2457212574 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f233c102d30 0x7f233c19a550 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2338007fa0 tx=0x7f2338004e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:14.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.029+0000 7f2331ffb700 1 -- 172.21.15.78:0/2457212574 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2338021070 con 0x7f233c102d30 2024-03-24T00:19:14.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.029+0000 7f2331ffb700 1 -- 172.21.15.78:0/2457212574 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2338007c80 con 0x7f233c102d30 2024-03-24T00:19:14.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.029+0000 7f2342bae700 1 -- 172.21.15.78:0/2457212574 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f233c1a2440 con 0x7f233c102d30 2024-03-24T00:19:14.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.030+0000 7f2331ffb700 1 -- 172.21.15.78:0/2457212574 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f233801d450 con 0x7f233c102d30 2024-03-24T00:19:14.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.030+0000 7f2342bae700 1 -- 172.21.15.78:0/2457212574 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f233c1a2940 con 0x7f233c102d30 2024-03-24T00:19:14.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.031+0000 7f2342bae700 1 -- 172.21.15.78:0/2457212574 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f233c0506a0 con 0x7f233c102d30 2024-03-24T00:19:14.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.032+0000 7f2331ffb700 1 -- 172.21.15.78:0/2457212574 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f23380041b0 con 0x7f233c102d30 2024-03-24T00:19:14.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.032+0000 7f2331ffb700 1 --2- 172.21.15.78:0/2457212574 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f23240a2a50 0x7f23240a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:14.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.033+0000 7f2333fff700 1 --2- 172.21.15.78:0/2457212574 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f23240a2a50 0x7f23240a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:14.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.033+0000 7f2331ffb700 1 -- 172.21.15.78:0/2457212574 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f233800f020 con 0x7f233c102d30 2024-03-24T00:19:14.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.033+0000 7f2333fff700 1 --2- 172.21.15.78:0/2457212574 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f23240a2a50 0x7f23240a4f10 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f232c010420 tx=0x7f232c0073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:14.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.037+0000 7f2331ffb700 1 -- 172.21.15.78:0/2457212574 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f233802a020 con 0x7f233c102d30 2024-03-24T00:19:14.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.313+0000 7f2342bae700 1 -- 172.21.15.78:0/2457212574 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f233c02da80 con 0x7f233c102d30 2024-03-24T00:19:14.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.316+0000 7f2331ffb700 1 -- 172.21.15.78:0/2457212574 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) v1 ==== 170+0+59 (secure 0 0 0) 0x7f233809d100 con 0x7f233c102d30 2024-03-24T00:19:14.316 INFO:teuthology.orchestra.run.smithi078.stdout:[client.0] 2024-03-24T00:19:14.316 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQCCcf9lq8vGEhAA1UWhOe/tCK6EP816Q1kOmQ== 2024-03-24T00:19:14.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.318+0000 7f2342bae700 1 -- 172.21.15.78:0/2457212574 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f23240a2a50 msgr2=0x7f23240a4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:14.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.318+0000 7f2342bae700 1 --2- 172.21.15.78:0/2457212574 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f23240a2a50 0x7f23240a4f10 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f232c010420 tx=0x7f232c0073d0 comp rx=0 tx=0).stop 2024-03-24T00:19:14.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.318+0000 7f2342bae700 1 -- 172.21.15.78:0/2457212574 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f233c102d30 msgr2=0x7f233c19a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:14.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.318+0000 7f2342bae700 1 --2- 172.21.15.78:0/2457212574 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f233c102d30 0x7f233c19a550 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2338007fa0 tx=0x7f2338004e40 comp rx=0 tx=0).stop 2024-03-24T00:19:14.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.318+0000 7f2342bae700 1 -- 172.21.15.78:0/2457212574 shutdown_connections 2024-03-24T00:19:14.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.318+0000 7f2342bae700 1 --2- 172.21.15.78:0/2457212574 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f23240a2a50 0x7f23240a4f10 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:14.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.318+0000 7f2342bae700 1 --2- 172.21.15.78:0/2457212574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f233c107f90 0x7f233c1a1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:14.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.318+0000 7f2342bae700 1 --2- 172.21.15.78:0/2457212574 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f233c102d30 0x7f233c19a550 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:14.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.318+0000 7f2342bae700 1 --2- 172.21.15.78:0/2457212574 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f233c105660 0x7f233c19aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:14.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.318+0000 7f2342bae700 1 -- 172.21.15.78:0/2457212574 >> 172.21.15.78:0/2457212574 conn(0x7f233c0fc680 msgr2=0x7f233c102460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:14.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.318+0000 7f2342bae700 1 -- 172.21.15.78:0/2457212574 shutdown_connections 2024-03-24T00:19:14.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:14.318+0000 7f2342bae700 1 -- 172.21.15.78:0/2457212574 wait complete. 2024-03-24T00:19:14.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:14 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2457212574' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T00:19:14.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:14 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2457212574' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T00:19:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:14 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2457212574' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T00:19:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:14 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2457212574' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T00:19:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:14 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2457212574' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T00:19:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:14 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2457212574' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T00:19:15.072 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-24T00:19:15.072 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-24T00:19:15.072 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-24T00:19:15.115 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-24T00:19:15.115 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-24T00:19:15.115 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph mgr dump --format=json 2024-03-24T00:19:15.403 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:19:15.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:15 smithi106 ceph-mon[36505]: pgmap v306: 129 pgs: 129 active+clean; 625 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 6.6 KiB/s wr, 13 op/s 2024-03-24T00:19:15.659 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:15 smithi078 ceph-mon[32429]: pgmap v306: 129 pgs: 129 active+clean; 625 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 6.6 KiB/s wr, 13 op/s 2024-03-24T00:19:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:15 smithi107 ceph-mon[36485]: pgmap v306: 129 pgs: 129 active+clean; 625 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 6.6 KiB/s wr, 13 op/s 2024-03-24T00:19:17.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.419+0000 7f66f3f80700 1 -- 172.21.15.78:0/3391230299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66ec102bd0 msgr2=0x7f66ec10f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:17.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.419+0000 7f66f3f80700 1 --2- 172.21.15.78:0/3391230299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66ec102bd0 0x7f66ec10f030 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f66dc009990 tx=0x7f66dc009ca0 comp rx=0 tx=0).stop 2024-03-24T00:19:17.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.419+0000 7f66f3f80700 1 -- 172.21.15.78:0/3391230299 shutdown_connections 2024-03-24T00:19:17.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.419+0000 7f66f3f80700 1 --2- 172.21.15.78:0/3391230299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66ec102bd0 0x7f66ec10f030 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:17.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.419+0000 7f66f3f80700 1 --2- 172.21.15.78:0/3391230299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66ec1082f0 0x7f66ec1086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:17.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.419+0000 7f66f3f80700 1 --2- 172.21.15.78:0/3391230299 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66ec102210 0x7f66ec102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:17.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.419+0000 7f66f3f80700 1 -- 172.21.15.78:0/3391230299 >> 172.21.15.78:0/3391230299 conn(0x7f66ec079250 msgr2=0x7f66ec079660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:17.420 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:17 smithi078 ceph-mon[32429]: pgmap v307: 129 pgs: 129 active+clean; 625 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 6.5 KiB/s wr, 13 op/s 2024-03-24T00:19:17.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.419+0000 7f66f3f80700 1 -- 172.21.15.78:0/3391230299 shutdown_connections 2024-03-24T00:19:17.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.420+0000 7f66f3f80700 1 -- 172.21.15.78:0/3391230299 wait complete. 2024-03-24T00:19:17.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.420+0000 7f66f3f80700 1 Processor -- start 2024-03-24T00:19:17.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.420+0000 7f66f3f80700 1 -- start start 2024-03-24T00:19:17.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.421+0000 7f66f3f80700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66ec102210 0x7f66ec19e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:17.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.421+0000 7f66f3f80700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66ec102bd0 0x7f66ec19eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:17.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.421+0000 7f66f3f80700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66ec1082f0 0x7f66ec198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:17.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.421+0000 7f66f3f80700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec1124c0 con 0x7f66ec102210 2024-03-24T00:19:17.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.421+0000 7f66f3f80700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec112340 con 0x7f66ec1082f0 2024-03-24T00:19:17.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.421+0000 7f66f3f80700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec112640 con 0x7f66ec102bd0 2024-03-24T00:19:17.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.421+0000 7f66f1d1c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66ec102210 0x7f66ec19e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:17.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.421+0000 7f66f1d1c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66ec102210 0x7f66ec19e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33614/0 (socket says 172.21.15.78:33614) 2024-03-24T00:19:17.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.421+0000 7f66f1d1c700 1 -- 172.21.15.78:0/3359817014 learned_addr learned my addr 172.21.15.78:0/3359817014 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:19:17.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.421+0000 7f66f151b700 1 --2- 172.21.15.78:0/3359817014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66ec102bd0 0x7f66ec19eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:17.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.421+0000 7f66f251d700 1 --2- 172.21.15.78:0/3359817014 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66ec1082f0 0x7f66ec198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:17.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.421+0000 7f66f1d1c700 1 -- 172.21.15.78:0/3359817014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66ec102bd0 msgr2=0x7f66ec19eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:17.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.421+0000 7f66f1d1c700 1 --2- 172.21.15.78:0/3359817014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66ec102bd0 0x7f66ec19eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:17.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.421+0000 7f66f1d1c700 1 -- 172.21.15.78:0/3359817014 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66ec1082f0 msgr2=0x7f66ec198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:17.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.421+0000 7f66f1d1c700 1 --2- 172.21.15.78:0/3359817014 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66ec1082f0 0x7f66ec198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:17.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.421+0000 7f66f1d1c700 1 -- 172.21.15.78:0/3359817014 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ec198fe0 con 0x7f66ec102210 2024-03-24T00:19:17.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.421+0000 7f66f1d1c700 1 --2- 172.21.15.78:0/3359817014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66ec102210 0x7f66ec19e5e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f66e400ef30 tx=0x7f66e400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:17.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.422+0000 7f66eaffd700 1 -- 172.21.15.78:0/3359817014 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e40092e0 con 0x7f66ec102210 2024-03-24T00:19:17.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.422+0000 7f66f3f80700 1 -- 172.21.15.78:0/3359817014 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66ec199270 con 0x7f66ec102210 2024-03-24T00:19:17.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.422+0000 7f66f3f80700 1 -- 172.21.15.78:0/3359817014 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66ec1a5560 con 0x7f66ec102210 2024-03-24T00:19:17.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.422+0000 7f66eaffd700 1 -- 172.21.15.78:0/3359817014 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f66e4004bb0 con 0x7f66ec102210 2024-03-24T00:19:17.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.422+0000 7f66eaffd700 1 -- 172.21.15.78:0/3359817014 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e4004280 con 0x7f66ec102210 2024-03-24T00:19:17.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.423+0000 7f66f3f80700 1 -- 172.21.15.78:0/3359817014 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66ec1038d0 con 0x7f66ec102210 2024-03-24T00:19:17.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.424+0000 7f66eaffd700 1 -- 172.21.15.78:0/3359817014 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f66e401b030 con 0x7f66ec102210 2024-03-24T00:19:17.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.424+0000 7f66eaffd700 1 --2- 172.21.15.78:0/3359817014 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f66d40a2800 0x7f66d40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:17.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.425+0000 7f66f151b700 1 --2- 172.21.15.78:0/3359817014 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f66d40a2800 0x7f66d40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:17.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.425+0000 7f66eaffd700 1 -- 172.21.15.78:0/3359817014 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f66e40cf1a0 con 0x7f66ec102210 2024-03-24T00:19:17.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.425+0000 7f66f151b700 1 --2- 172.21.15.78:0/3359817014 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f66d40a2800 0x7f66d40a4cc0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f66ec199e70 tx=0x7f66e00073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:17.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.428+0000 7f66eaffd700 1 -- 172.21.15.78:0/3359817014 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f66e4022020 con 0x7f66ec102210 2024-03-24T00:19:17.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:17 smithi106 ceph-mon[36505]: pgmap v307: 129 pgs: 129 active+clean; 625 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 6.5 KiB/s wr, 13 op/s 2024-03-24T00:19:17.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.682+0000 7f66f3f80700 1 -- 172.21.15.78:0/3359817014 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f66ec068e50 con 0x7f66ec102210 2024-03-24T00:19:17.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.686+0000 7f66eaffd700 1 -- 172.21.15.78:0/3359817014 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+262659 (secure 0 0 0) 0x7f66e4010040 con 0x7f66ec102210 2024-03-24T00:19:17.689 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:19:17.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.689+0000 7f66f3f80700 1 -- 172.21.15.78:0/3359817014 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f66d40a2800 msgr2=0x7f66d40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:17.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.689+0000 7f66f3f80700 1 --2- 172.21.15.78:0/3359817014 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f66d40a2800 0x7f66d40a4cc0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f66ec199e70 tx=0x7f66e00073d0 comp rx=0 tx=0).stop 2024-03-24T00:19:17.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.690+0000 7f66f3f80700 1 -- 172.21.15.78:0/3359817014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66ec102210 msgr2=0x7f66ec19e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:17.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.690+0000 7f66f3f80700 1 --2- 172.21.15.78:0/3359817014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66ec102210 0x7f66ec19e5e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f66e400ef30 tx=0x7f66e400c3a0 comp rx=0 tx=0).stop 2024-03-24T00:19:17.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.690+0000 7f66f3f80700 1 -- 172.21.15.78:0/3359817014 shutdown_connections 2024-03-24T00:19:17.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.690+0000 7f66f3f80700 1 --2- 172.21.15.78:0/3359817014 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f66d40a2800 0x7f66d40a4cc0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:17.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.690+0000 7f66f3f80700 1 --2- 172.21.15.78:0/3359817014 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66ec1082f0 0x7f66ec198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:17.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.690+0000 7f66f3f80700 1 --2- 172.21.15.78:0/3359817014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66ec102210 0x7f66ec19e5e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:17.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.690+0000 7f66f3f80700 1 --2- 172.21.15.78:0/3359817014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66ec102bd0 0x7f66ec19eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:17.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.690+0000 7f66f3f80700 1 -- 172.21.15.78:0/3359817014 >> 172.21.15.78:0/3359817014 conn(0x7f66ec079250 msgr2=0x7f66ec100d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:17.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.690+0000 7f66f3f80700 1 -- 172.21.15.78:0/3359817014 shutdown_connections 2024-03-24T00:19:17.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:17.690+0000 7f66f3f80700 1 -- 172.21.15.78:0/3359817014 wait complete. 2024-03-24T00:19:17.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:17 smithi107 ceph-mon[36485]: pgmap v307: 129 pgs: 129 active+clean; 625 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 6.5 KiB/s wr, 13 op/s 2024-03-24T00:19:18.432 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":1983101026},{"type":"v1","addr":"172.21.15.78:6801","nonce":1983101026}]},"active_addr":"172.21.15.78:6801/1983101026","active_change":"2024-03-24T00:11:40.199719+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24107,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.78:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":442826396}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":1685151467}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":359613096}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":1510740971}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":2161490061}]}]} 2024-03-24T00:19:18.437 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-24T00:19:18.437 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-24T00:19:18.437 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T00:19:18.677 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:19:18.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:18 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3359817014' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T00:19:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:18 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3359817014' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T00:19:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:18 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3359817014' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T00:19:19.802 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:19 smithi106 ceph-mon[36505]: pgmap v308: 129 pgs: 129 active+clean; 625 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.5 KiB/s wr, 14 op/s 2024-03-24T00:19:19.823 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:19 smithi078 ceph-mon[32429]: pgmap v308: 129 pgs: 129 active+clean; 625 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.5 KiB/s wr, 14 op/s 2024-03-24T00:19:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:19 smithi107 ceph-mon[36485]: pgmap v308: 129 pgs: 129 active+clean; 625 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.5 KiB/s wr, 14 op/s 2024-03-24T00:19:20.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.467+0000 7eff23cc6700 1 -- 172.21.15.78:0/124589793 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff1c112e20 msgr2=0x7eff1c115210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:20.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.467+0000 7eff23cc6700 1 --2- 172.21.15.78:0/124589793 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff1c112e20 0x7eff1c115210 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7eff1400b0a0 tx=0x7eff1400b3b0 comp rx=0 tx=0).stop 2024-03-24T00:19:20.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.468+0000 7eff23cc6700 1 -- 172.21.15.78:0/124589793 shutdown_connections 2024-03-24T00:19:20.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.468+0000 7eff23cc6700 1 --2- 172.21.15.78:0/124589793 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff1c112e20 0x7eff1c115210 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:20.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.468+0000 7eff23cc6700 1 --2- 172.21.15.78:0/124589793 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff1c0761c0 0x7eff1c0743c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:20.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.468+0000 7eff23cc6700 1 --2- 172.21.15.78:0/124589793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff1c074900 0x7eff1c074d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:20.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.468+0000 7eff23cc6700 1 -- 172.21.15.78:0/124589793 >> 172.21.15.78:0/124589793 conn(0x7eff1c0792c0 msgr2=0x7eff1c0796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:20.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.468+0000 7eff23cc6700 1 -- 172.21.15.78:0/124589793 shutdown_connections 2024-03-24T00:19:20.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.468+0000 7eff23cc6700 1 -- 172.21.15.78:0/124589793 wait complete. 2024-03-24T00:19:20.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.468+0000 7eff23cc6700 1 Processor -- start 2024-03-24T00:19:20.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.468+0000 7eff23cc6700 1 -- start start 2024-03-24T00:19:20.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.468+0000 7eff23cc6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff1c074900 0x7eff1c19eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:20.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.468+0000 7eff23cc6700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff1c0761c0 0x7eff1c19f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:20.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.468+0000 7eff23cc6700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff1c112e20 0x7eff1c1a3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:20.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.468+0000 7eff23cc6700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1c118a60 con 0x7eff1c074900 2024-03-24T00:19:20.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.468+0000 7eff23cc6700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1c1188e0 con 0x7eff1c112e20 2024-03-24T00:19:20.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.468+0000 7eff23cc6700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1c118be0 con 0x7eff1c0761c0 2024-03-24T00:19:20.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.469+0000 7eff21a62700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff1c074900 0x7eff1c19eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:20.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.469+0000 7eff21261700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff1c0761c0 0x7eff1c19f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:20.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.469+0000 7eff21261700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff1c0761c0 0x7eff1c19f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.78:39656/0 (socket says 172.21.15.78:39656) 2024-03-24T00:19:20.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.469+0000 7eff21a62700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff1c074900 0x7eff1c19eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33634/0 (socket says 172.21.15.78:33634) 2024-03-24T00:19:20.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.469+0000 7eff21261700 1 -- 172.21.15.78:0/2417903981 learned_addr learned my addr 172.21.15.78:0/2417903981 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:19:20.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.469+0000 7eff22263700 1 --2- 172.21.15.78:0/2417903981 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff1c112e20 0x7eff1c1a3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:20.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.469+0000 7eff21261700 1 -- 172.21.15.78:0/2417903981 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff1c112e20 msgr2=0x7eff1c1a3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:20.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.469+0000 7eff21261700 1 --2- 172.21.15.78:0/2417903981 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff1c112e20 0x7eff1c1a3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:20.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.469+0000 7eff21261700 1 -- 172.21.15.78:0/2417903981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff1c074900 msgr2=0x7eff1c19eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:20.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.469+0000 7eff21261700 1 --2- 172.21.15.78:0/2417903981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff1c074900 0x7eff1c19eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:20.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.469+0000 7eff21261700 1 -- 172.21.15.78:0/2417903981 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff1c1a3ec0 con 0x7eff1c0761c0 2024-03-24T00:19:20.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.469+0000 7eff21261700 1 --2- 172.21.15.78:0/2417903981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff1c0761c0 0x7eff1c19f3f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7eff0c00e9a0 tx=0x7eff0c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:20.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.469+0000 7eff12ffd700 1 -- 172.21.15.78:0/2417903981 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff0c00ca10 con 0x7eff1c0761c0 2024-03-24T00:19:20.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.469+0000 7eff12ffd700 1 -- 172.21.15.78:0/2417903981 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff0c00cbb0 con 0x7eff1c0761c0 2024-03-24T00:19:20.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.469+0000 7eff12ffd700 1 -- 172.21.15.78:0/2417903981 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff0c018680 con 0x7eff1c0761c0 2024-03-24T00:19:20.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.469+0000 7eff23cc6700 1 -- 172.21.15.78:0/2417903981 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff1c1a41b0 con 0x7eff1c0761c0 2024-03-24T00:19:20.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.469+0000 7eff23cc6700 1 -- 172.21.15.78:0/2417903981 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff1c1aba30 con 0x7eff1c0761c0 2024-03-24T00:19:20.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.470+0000 7eff23cc6700 1 -- 172.21.15.78:0/2417903981 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff1c068e50 con 0x7eff1c0761c0 2024-03-24T00:19:20.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.471+0000 7eff12ffd700 1 -- 172.21.15.78:0/2417903981 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7eff0c010430 con 0x7eff1c0761c0 2024-03-24T00:19:20.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.472+0000 7eff12ffd700 1 --2- 172.21.15.78:0/2417903981 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7eff080a2920 0x7eff080a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:20.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.472+0000 7eff21a62700 1 --2- 172.21.15.78:0/2417903981 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7eff080a2920 0x7eff080a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:20.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.472+0000 7eff12ffd700 1 -- 172.21.15.78:0/2417903981 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7eff0c0cf0d0 con 0x7eff1c0761c0 2024-03-24T00:19:20.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.472+0000 7eff21a62700 1 --2- 172.21.15.78:0/2417903981 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7eff080a2920 0x7eff080a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7eff18009be0 tx=0x7eff18009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:20.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.476+0000 7eff12ffd700 1 -- 172.21.15.78:0/2417903981 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7eff0c09bd70 con 0x7eff1c0761c0 2024-03-24T00:19:20.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.672+0000 7eff23cc6700 1 -- 172.21.15.78:0/2417903981 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff1c1abc30 con 0x7eff1c0761c0 2024-03-24T00:19:20.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.673+0000 7eff12ffd700 1 -- 172.21.15.78:0/2417903981 <== mon.1 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24129 (secure 0 0 0) 0x7eff0c09b670 con 0x7eff1c0761c0 2024-03-24T00:19:20.674 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:19:20.674 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":89,"fsid":"c7e2806a-e972-11ee-95cd-87774f69a715","created":"2024-03-24T00:10:44.134563+0000","modified":"2024-03-24T00:18:48.224440+0000","last_up_change":"2024-03-24T00:17:50.237686+0000","last_in_change":"2024-03-24T00:17:29.788699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:14:04.534391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:18:42.178418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:18:42.800424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:18:46.112800+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b950388a-c437-426b-b703-3f8ae821dd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6803","nonce":1816879397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6805","nonce":1816879397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6809","nonce":1816879397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6807","nonce":1816879397}]},"public_addr":"172.21.15.78:6803/1816879397","cluster_addr":"172.21.15.78:6805/1816879397","heartbeat_back_addr":"172.21.15.78:6809/1816879397","heartbeat_front_addr":"172.21.15.78:6807/1816879397","state":["exists","up"]},{"osd":1,"uuid":"b8d64714-dd65-400f-b3f4-673ab14e1365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6801","nonce":1647915612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6803","nonce":1647915612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6807","nonce":1647915612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6805","nonce":1647915612}]},"public_addr":"172.21.15.106:6801/1647915612","cluster_addr":"172.21.15.106:6803/1647915612","heartbeat_back_addr":"172.21.15.106:6807/1647915612","heartbeat_front_addr":"172.21.15.106:6805/1647915612","state":["exists","up"]},{"osd":2,"uuid":"fcfc4dd1-aaf8-4131-a023-5e832212158d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6801","nonce":2010770435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6803","nonce":2010770435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6807","nonce":2010770435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6805","nonce":2010770435}]},"public_addr":"172.21.15.107:6801/2010770435","cluster_addr":"172.21.15.107:6803/2010770435","heartbeat_back_addr":"172.21.15.107:6807/2010770435","heartbeat_front_addr":"172.21.15.107:6805/2010770435","state":["exists","up"]},{"osd":3,"uuid":"47c718ab-a0f4-4e52-83c5-b89386b262d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6811","nonce":1912251346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6813","nonce":1912251346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6817","nonce":1912251346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6815","nonce":1912251346}]},"public_addr":"172.21.15.78:6811/1912251346","cluster_addr":"172.21.15.78:6813/1912251346","heartbeat_back_addr":"172.21.15.78:6817/1912251346","heartbeat_front_addr":"172.21.15.78:6815/1912251346","state":["exists","up"]},{"osd":4,"uuid":"7dad64dd-677e-4c1c-9638-e4f4f2195983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6809","nonce":374641205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6811","nonce":374641205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6815","nonce":374641205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6813","nonce":374641205}]},"public_addr":"172.21.15.106:6809/374641205","cluster_addr":"172.21.15.106:6811/374641205","heartbeat_back_addr":"172.21.15.106:6815/374641205","heartbeat_front_addr":"172.21.15.106:6813/374641205","state":["exists","up"]},{"osd":5,"uuid":"bbd524e5-328d-4309-8877-1149cbe110dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6809","nonce":3272806073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6811","nonce":3272806073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6815","nonce":3272806073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6813","nonce":3272806073}]},"public_addr":"172.21.15.107:6809/3272806073","cluster_addr":"172.21.15.107:6811/3272806073","heartbeat_back_addr":"172.21.15.107:6815/3272806073","heartbeat_front_addr":"172.21.15.107:6813/3272806073","state":["exists","up"]},{"osd":6,"uuid":"5696389a-918f-4ea8-b20d-4e2a0d4cb3bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6819","nonce":3091148508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6821","nonce":3091148508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6825","nonce":3091148508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6823","nonce":3091148508}]},"public_addr":"172.21.15.78:6819/3091148508","cluster_addr":"172.21.15.78:6821/3091148508","heartbeat_back_addr":"172.21.15.78:6825/3091148508","heartbeat_front_addr":"172.21.15.78:6823/3091148508","state":["exists","up"]},{"osd":7,"uuid":"f6b764e8-ed4e-4c9a-a85b-94af6dc6b924","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6817","nonce":4262103080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6819","nonce":4262103080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6823","nonce":4262103080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6821","nonce":4262103080}]},"public_addr":"172.21.15.106:6817/4262103080","cluster_addr":"172.21.15.106:6819/4262103080","heartbeat_back_addr":"172.21.15.106:6823/4262103080","heartbeat_front_addr":"172.21.15.106:6821/4262103080","state":["exists","up"]},{"osd":8,"uuid":"ba16039f-8093-4fa0-87ad-e28b7d142b23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6817","nonce":4032799356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6819","nonce":4032799356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6823","nonce":4032799356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6821","nonce":4032799356}]},"public_addr":"172.21.15.107:6817/4032799356","cluster_addr":"172.21.15.107:6819/4032799356","heartbeat_back_addr":"172.21.15.107:6823/4032799356","heartbeat_front_addr":"172.21.15.107:6821/4032799356","state":["exists","up"]},{"osd":9,"uuid":"651357f2-0669-42e6-aa91-1d5c02d00c5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6827","nonce":214023431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6829","nonce":214023431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6833","nonce":214023431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6831","nonce":214023431}]},"public_addr":"172.21.15.78:6827/214023431","cluster_addr":"172.21.15.78:6829/214023431","heartbeat_back_addr":"172.21.15.78:6833/214023431","heartbeat_front_addr":"172.21.15.78:6831/214023431","state":["exists","up"]},{"osd":10,"uuid":"2375119d-346e-426c-8a65-b64af71ad63a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6825","nonce":400029804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6827","nonce":400029804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6831","nonce":400029804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6829","nonce":400029804}]},"public_addr":"172.21.15.106:6825/400029804","cluster_addr":"172.21.15.106:6827/400029804","heartbeat_back_addr":"172.21.15.106:6831/400029804","heartbeat_front_addr":"172.21.15.106:6829/400029804","state":["exists","up"]},{"osd":11,"uuid":"b6b609d9-dd78-4c01-ae66-ba2a662ab22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6825","nonce":4191916230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6827","nonce":4191916230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6831","nonce":4191916230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6829","nonce":4191916230}]},"public_addr":"172.21.15.107:6825/4191916230","cluster_addr":"172.21.15.107:6827/4191916230","heartbeat_back_addr":"172.21.15.107:6831/4191916230","heartbeat_front_addr":"172.21.15.107:6829/4191916230","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:13.728602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:37.320809+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:01.269452+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:25.758281+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:50.298863+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:14.345219+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:38.980489+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:04.749214+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:29.992673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:56.269202+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:22.270316+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:48.074907+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/1982202541":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/1933054339":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6801/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6800/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/465691596":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6801/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/3962523609":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6800/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/1502752625":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/2236879401":"2024-03-25T00:11:40.199569+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:19:20.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.675+0000 7eff23cc6700 1 -- 172.21.15.78:0/2417903981 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7eff080a2920 msgr2=0x7eff080a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:20.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.675+0000 7eff23cc6700 1 --2- 172.21.15.78:0/2417903981 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7eff080a2920 0x7eff080a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7eff18009be0 tx=0x7eff18009260 comp rx=0 tx=0).stop 2024-03-24T00:19:20.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.675+0000 7eff23cc6700 1 -- 172.21.15.78:0/2417903981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff1c0761c0 msgr2=0x7eff1c19f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:20.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.675+0000 7eff23cc6700 1 --2- 172.21.15.78:0/2417903981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff1c0761c0 0x7eff1c19f3f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7eff0c00e9a0 tx=0x7eff0c00ecb0 comp rx=0 tx=0).stop 2024-03-24T00:19:20.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.676+0000 7eff23cc6700 1 -- 172.21.15.78:0/2417903981 shutdown_connections 2024-03-24T00:19:20.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.676+0000 7eff23cc6700 1 --2- 172.21.15.78:0/2417903981 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7eff080a2920 0x7eff080a4de0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:20.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.676+0000 7eff23cc6700 1 --2- 172.21.15.78:0/2417903981 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff1c112e20 0x7eff1c1a3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:20.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.676+0000 7eff23cc6700 1 --2- 172.21.15.78:0/2417903981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff1c074900 0x7eff1c19eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:20.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.676+0000 7eff23cc6700 1 --2- 172.21.15.78:0/2417903981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff1c0761c0 0x7eff1c19f3f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:20.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.676+0000 7eff23cc6700 1 -- 172.21.15.78:0/2417903981 >> 172.21.15.78:0/2417903981 conn(0x7eff1c0792c0 msgr2=0x7eff1c111b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:20.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.676+0000 7eff23cc6700 1 -- 172.21.15.78:0/2417903981 shutdown_connections 2024-03-24T00:19:20.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:20.676+0000 7eff23cc6700 1 -- 172.21.15.78:0/2417903981 wait complete. 2024-03-24T00:19:21.441 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-24T00:19:21.441 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T00:19:21.682 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:19:21.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:21 smithi078 ceph-mon[32429]: pgmap v309: 129 pgs: 129 active+clean; 640 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.3 KiB/s wr, 15 op/s 2024-03-24T00:19:21.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:21 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2417903981' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:19:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:21 smithi107 ceph-mon[36485]: pgmap v309: 129 pgs: 129 active+clean; 640 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.3 KiB/s wr, 15 op/s 2024-03-24T00:19:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:21 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2417903981' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:19:21.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:21 smithi106 ceph-mon[36505]: pgmap v309: 129 pgs: 129 active+clean; 640 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.3 KiB/s wr, 15 op/s 2024-03-24T00:19:21.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:21 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2417903981' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:19:23.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.378+0000 7f0f27345700 1 -- 172.21.15.78:0/2903810226 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f20107560 msgr2=0x7f0f2010f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:23.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.378+0000 7f0f27345700 1 --2- 172.21.15.78:0/2903810226 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f20107560 0x7f0f2010f170 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0f1800b0d0 tx=0x7f0f1800b3e0 comp rx=0 tx=0).stop 2024-03-24T00:19:23.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.379+0000 7f0f27345700 1 -- 172.21.15.78:0/2903810226 shutdown_connections 2024-03-24T00:19:23.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.379+0000 7f0f27345700 1 --2- 172.21.15.78:0/2903810226 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f20107560 0x7f0f2010f170 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:23.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.379+0000 7f0f27345700 1 --2- 172.21.15.78:0/2903810226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f20100d90 0x7f0f20101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:23.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.379+0000 7f0f27345700 1 --2- 172.21.15.78:0/2903810226 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0f20101740 0x7f0f20106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:23.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.379+0000 7f0f27345700 1 -- 172.21.15.78:0/2903810226 >> 172.21.15.78:0/2903810226 conn(0x7f0f20076410 msgr2=0x7f0f20076820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:23.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.379+0000 7f0f27345700 1 -- 172.21.15.78:0/2903810226 shutdown_connections 2024-03-24T00:19:23.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.379+0000 7f0f27345700 1 -- 172.21.15.78:0/2903810226 wait complete. 2024-03-24T00:19:23.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.379+0000 7f0f27345700 1 Processor -- start 2024-03-24T00:19:23.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.380+0000 7f0f27345700 1 -- start start 2024-03-24T00:19:23.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.380+0000 7f0f27345700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f20100d90 0x7f0f2019e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:23.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.380+0000 7f0f27345700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f20101740 0x7f0f2019eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:23.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.380+0000 7f0f27345700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0f20107560 0x7f0f201987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:23.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.380+0000 7f0f27345700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f201125a0 con 0x7f0f20101740 2024-03-24T00:19:23.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.380+0000 7f0f27345700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f20112420 con 0x7f0f20107560 2024-03-24T00:19:23.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.380+0000 7f0f27345700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f20112720 con 0x7f0f20100d90 2024-03-24T00:19:23.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.380+0000 7f0f248e0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f20101740 0x7f0f2019eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:23.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.380+0000 7f0f248e0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f20101740 0x7f0f2019eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33646/0 (socket says 172.21.15.78:33646) 2024-03-24T00:19:23.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.380+0000 7f0f248e0700 1 -- 172.21.15.78:0/3933089981 learned_addr learned my addr 172.21.15.78:0/3933089981 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:19:23.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.380+0000 7f0f250e1700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f20100d90 0x7f0f2019e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:23.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.380+0000 7f0f258e2700 1 --2- 172.21.15.78:0/3933089981 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0f20107560 0x7f0f201987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:23.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.380+0000 7f0f248e0700 1 -- 172.21.15.78:0/3933089981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f20100d90 msgr2=0x7f0f2019e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:23.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.380+0000 7f0f248e0700 1 --2- 172.21.15.78:0/3933089981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f20100d90 0x7f0f2019e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:23.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.380+0000 7f0f248e0700 1 -- 172.21.15.78:0/3933089981 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0f20107560 msgr2=0x7f0f201987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:23.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.380+0000 7f0f248e0700 1 --2- 172.21.15.78:0/3933089981 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0f20107560 0x7f0f201987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:23.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.380+0000 7f0f248e0700 1 -- 172.21.15.78:0/3933089981 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f20198fc0 con 0x7f0f20101740 2024-03-24T00:19:23.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.380+0000 7f0f250e1700 1 --2- 172.21.15.78:0/3933089981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f20100d90 0x7f0f2019e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:23.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.381+0000 7f0f248e0700 1 --2- 172.21.15.78:0/3933089981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f20101740 0x7f0f2019eba0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0f1000e970 tx=0x7f0f1000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:23.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.381+0000 7f0f167fc700 1 -- 172.21.15.78:0/3933089981 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1000cbd0 con 0x7f0f20101740 2024-03-24T00:19:23.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.381+0000 7f0f167fc700 1 -- 172.21.15.78:0/3933089981 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f1000cd70 con 0x7f0f20101740 2024-03-24T00:19:23.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.381+0000 7f0f27345700 1 -- 172.21.15.78:0/3933089981 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f201992b0 con 0x7f0f20101740 2024-03-24T00:19:23.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.381+0000 7f0f27345700 1 -- 172.21.15.78:0/3933089981 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f20101c40 con 0x7f0f20101740 2024-03-24T00:19:23.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.381+0000 7f0f167fc700 1 -- 172.21.15.78:0/3933089981 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f100056b0 con 0x7f0f20101740 2024-03-24T00:19:23.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.382+0000 7f0f27345700 1 -- 172.21.15.78:0/3933089981 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f04005330 con 0x7f0f20101740 2024-03-24T00:19:23.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.383+0000 7f0f167fc700 1 -- 172.21.15.78:0/3933089981 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f0f10004750 con 0x7f0f20101740 2024-03-24T00:19:23.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.383+0000 7f0f167fc700 1 --2- 172.21.15.78:0/3933089981 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f0f0c0a2920 0x7f0f0c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:23.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.384+0000 7f0f250e1700 1 --2- 172.21.15.78:0/3933089981 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f0f0c0a2920 0x7f0f0c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:23.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.384+0000 7f0f167fc700 1 -- 172.21.15.78:0/3933089981 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f0f100cf0e0 con 0x7f0f20101740 2024-03-24T00:19:23.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.384+0000 7f0f250e1700 1 --2- 172.21.15.78:0/3933089981 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f0f0c0a2920 0x7f0f0c0a4de0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c009730 tx=0x7f0f1c012040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:23.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.388+0000 7f0f167fc700 1 -- 172.21.15.78:0/3933089981 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0f1009bde0 con 0x7f0f20101740 2024-03-24T00:19:23.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.584+0000 7f0f27345700 1 -- 172.21.15.78:0/3933089981 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f040058f0 con 0x7f0f20101740 2024-03-24T00:19:23.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.586+0000 7f0f167fc700 1 -- 172.21.15.78:0/3933089981 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24129 (secure 0 0 0) 0x7f0f1009b6e0 con 0x7f0f20101740 2024-03-24T00:19:23.586 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:19:23.586 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":89,"fsid":"c7e2806a-e972-11ee-95cd-87774f69a715","created":"2024-03-24T00:10:44.134563+0000","modified":"2024-03-24T00:18:48.224440+0000","last_up_change":"2024-03-24T00:17:50.237686+0000","last_in_change":"2024-03-24T00:17:29.788699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:14:04.534391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:18:42.178418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:18:42.800424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:18:46.112800+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b950388a-c437-426b-b703-3f8ae821dd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6803","nonce":1816879397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6805","nonce":1816879397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6809","nonce":1816879397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6807","nonce":1816879397}]},"public_addr":"172.21.15.78:6803/1816879397","cluster_addr":"172.21.15.78:6805/1816879397","heartbeat_back_addr":"172.21.15.78:6809/1816879397","heartbeat_front_addr":"172.21.15.78:6807/1816879397","state":["exists","up"]},{"osd":1,"uuid":"b8d64714-dd65-400f-b3f4-673ab14e1365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6801","nonce":1647915612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6803","nonce":1647915612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6807","nonce":1647915612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6805","nonce":1647915612}]},"public_addr":"172.21.15.106:6801/1647915612","cluster_addr":"172.21.15.106:6803/1647915612","heartbeat_back_addr":"172.21.15.106:6807/1647915612","heartbeat_front_addr":"172.21.15.106:6805/1647915612","state":["exists","up"]},{"osd":2,"uuid":"fcfc4dd1-aaf8-4131-a023-5e832212158d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6801","nonce":2010770435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6803","nonce":2010770435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6807","nonce":2010770435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6805","nonce":2010770435}]},"public_addr":"172.21.15.107:6801/2010770435","cluster_addr":"172.21.15.107:6803/2010770435","heartbeat_back_addr":"172.21.15.107:6807/2010770435","heartbeat_front_addr":"172.21.15.107:6805/2010770435","state":["exists","up"]},{"osd":3,"uuid":"47c718ab-a0f4-4e52-83c5-b89386b262d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6811","nonce":1912251346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6813","nonce":1912251346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6817","nonce":1912251346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6815","nonce":1912251346}]},"public_addr":"172.21.15.78:6811/1912251346","cluster_addr":"172.21.15.78:6813/1912251346","heartbeat_back_addr":"172.21.15.78:6817/1912251346","heartbeat_front_addr":"172.21.15.78:6815/1912251346","state":["exists","up"]},{"osd":4,"uuid":"7dad64dd-677e-4c1c-9638-e4f4f2195983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6809","nonce":374641205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6811","nonce":374641205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6815","nonce":374641205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6813","nonce":374641205}]},"public_addr":"172.21.15.106:6809/374641205","cluster_addr":"172.21.15.106:6811/374641205","heartbeat_back_addr":"172.21.15.106:6815/374641205","heartbeat_front_addr":"172.21.15.106:6813/374641205","state":["exists","up"]},{"osd":5,"uuid":"bbd524e5-328d-4309-8877-1149cbe110dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6809","nonce":3272806073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6811","nonce":3272806073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6815","nonce":3272806073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6813","nonce":3272806073}]},"public_addr":"172.21.15.107:6809/3272806073","cluster_addr":"172.21.15.107:6811/3272806073","heartbeat_back_addr":"172.21.15.107:6815/3272806073","heartbeat_front_addr":"172.21.15.107:6813/3272806073","state":["exists","up"]},{"osd":6,"uuid":"5696389a-918f-4ea8-b20d-4e2a0d4cb3bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6819","nonce":3091148508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6821","nonce":3091148508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6825","nonce":3091148508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6823","nonce":3091148508}]},"public_addr":"172.21.15.78:6819/3091148508","cluster_addr":"172.21.15.78:6821/3091148508","heartbeat_back_addr":"172.21.15.78:6825/3091148508","heartbeat_front_addr":"172.21.15.78:6823/3091148508","state":["exists","up"]},{"osd":7,"uuid":"f6b764e8-ed4e-4c9a-a85b-94af6dc6b924","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6817","nonce":4262103080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6819","nonce":4262103080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6823","nonce":4262103080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6821","nonce":4262103080}]},"public_addr":"172.21.15.106:6817/4262103080","cluster_addr":"172.21.15.106:6819/4262103080","heartbeat_back_addr":"172.21.15.106:6823/4262103080","heartbeat_front_addr":"172.21.15.106:6821/4262103080","state":["exists","up"]},{"osd":8,"uuid":"ba16039f-8093-4fa0-87ad-e28b7d142b23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6817","nonce":4032799356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6819","nonce":4032799356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6823","nonce":4032799356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6821","nonce":4032799356}]},"public_addr":"172.21.15.107:6817/4032799356","cluster_addr":"172.21.15.107:6819/4032799356","heartbeat_back_addr":"172.21.15.107:6823/4032799356","heartbeat_front_addr":"172.21.15.107:6821/4032799356","state":["exists","up"]},{"osd":9,"uuid":"651357f2-0669-42e6-aa91-1d5c02d00c5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6827","nonce":214023431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6829","nonce":214023431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6833","nonce":214023431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6831","nonce":214023431}]},"public_addr":"172.21.15.78:6827/214023431","cluster_addr":"172.21.15.78:6829/214023431","heartbeat_back_addr":"172.21.15.78:6833/214023431","heartbeat_front_addr":"172.21.15.78:6831/214023431","state":["exists","up"]},{"osd":10,"uuid":"2375119d-346e-426c-8a65-b64af71ad63a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6825","nonce":400029804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6827","nonce":400029804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6831","nonce":400029804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6829","nonce":400029804}]},"public_addr":"172.21.15.106:6825/400029804","cluster_addr":"172.21.15.106:6827/400029804","heartbeat_back_addr":"172.21.15.106:6831/400029804","heartbeat_front_addr":"172.21.15.106:6829/400029804","state":["exists","up"]},{"osd":11,"uuid":"b6b609d9-dd78-4c01-ae66-ba2a662ab22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6825","nonce":4191916230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6827","nonce":4191916230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6831","nonce":4191916230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6829","nonce":4191916230}]},"public_addr":"172.21.15.107:6825/4191916230","cluster_addr":"172.21.15.107:6827/4191916230","heartbeat_back_addr":"172.21.15.107:6831/4191916230","heartbeat_front_addr":"172.21.15.107:6829/4191916230","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:13.728602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:37.320809+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:01.269452+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:25.758281+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:50.298863+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:14.345219+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:38.980489+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:04.749214+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:29.992673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:56.269202+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:22.270316+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:48.074907+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/1982202541":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/1933054339":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6801/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6800/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/465691596":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6801/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/3962523609":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6800/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/1502752625":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/2236879401":"2024-03-25T00:11:40.199569+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:19:23.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.588+0000 7f0f27345700 1 -- 172.21.15.78:0/3933089981 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f0f0c0a2920 msgr2=0x7f0f0c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:23.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.588+0000 7f0f27345700 1 --2- 172.21.15.78:0/3933089981 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f0f0c0a2920 0x7f0f0c0a4de0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c009730 tx=0x7f0f1c012040 comp rx=0 tx=0).stop 2024-03-24T00:19:23.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.588+0000 7f0f27345700 1 -- 172.21.15.78:0/3933089981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f20101740 msgr2=0x7f0f2019eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:23.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.588+0000 7f0f27345700 1 --2- 172.21.15.78:0/3933089981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f20101740 0x7f0f2019eba0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0f1000e970 tx=0x7f0f1000ec80 comp rx=0 tx=0).stop 2024-03-24T00:19:23.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.588+0000 7f0f27345700 1 -- 172.21.15.78:0/3933089981 shutdown_connections 2024-03-24T00:19:23.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.588+0000 7f0f27345700 1 --2- 172.21.15.78:0/3933089981 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f0f0c0a2920 0x7f0f0c0a4de0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:23.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.588+0000 7f0f27345700 1 --2- 172.21.15.78:0/3933089981 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0f20107560 0x7f0f201987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:23.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.588+0000 7f0f27345700 1 --2- 172.21.15.78:0/3933089981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f20100d90 0x7f0f2019e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:23.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.588+0000 7f0f27345700 1 --2- 172.21.15.78:0/3933089981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f20101740 0x7f0f2019eba0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:23.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.588+0000 7f0f27345700 1 -- 172.21.15.78:0/3933089981 >> 172.21.15.78:0/3933089981 conn(0x7f0f20076410 msgr2=0x7f0f200feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:23.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.588+0000 7f0f27345700 1 -- 172.21.15.78:0/3933089981 shutdown_connections 2024-03-24T00:19:23.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:23.588+0000 7f0f27345700 1 -- 172.21.15.78:0/3933089981 wait complete. 2024-03-24T00:19:23.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:23 smithi078 ceph-mon[32429]: pgmap v310: 129 pgs: 129 active+clean; 640 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 7.2 KiB/s wr, 12 op/s 2024-03-24T00:19:23.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:23 smithi107 ceph-mon[36485]: pgmap v310: 129 pgs: 129 active+clean; 640 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 7.2 KiB/s wr, 12 op/s 2024-03-24T00:19:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:23 smithi106 ceph-mon[36505]: pgmap v310: 129 pgs: 129 active+clean; 640 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 7.2 KiB/s wr, 12 op/s 2024-03-24T00:19:24.353 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-24T00:19:24.354 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-24T00:19:24.354 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-24T00:19:24.355 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-24T00:19:24.355 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-24T00:19:24.355 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-24T00:19:24.356 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-24T00:19:24.356 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-24T00:19:24.357 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-24T00:19:24.357 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-24T00:19:24.357 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-24T00:19:24.358 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-24T00:19:24.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:24 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3933089981' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:19:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:24 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3933089981' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:19:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:24 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3933089981' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:19:25.014 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:19:25.139 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:19:25.140 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:19:25.140 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:19:25.141 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:19:25.381 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:19:25.382 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:19:25.382 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:19:25.383 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:19:25.384 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:19:25.384 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:19:25.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:25 smithi078 ceph-mon[32429]: pgmap v311: 129 pgs: 129 active+clean; 640 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.0 KiB/s wr, 10 op/s 2024-03-24T00:19:25.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:25 smithi107 ceph-mon[36485]: pgmap v311: 129 pgs: 129 active+clean; 640 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.0 KiB/s wr, 10 op/s 2024-03-24T00:19:25.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:25 smithi106 ceph-mon[36505]: pgmap v311: 129 pgs: 129 active+clean; 640 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.0 KiB/s wr, 10 op/s 2024-03-24T00:19:27.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:27 smithi078 ceph-mon[32429]: pgmap v312: 129 pgs: 129 active+clean; 640 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-24T00:19:27.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:27 smithi107 ceph-mon[36485]: pgmap v312: 129 pgs: 129 active+clean; 640 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-24T00:19:27.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:27 smithi106 ceph-mon[36505]: pgmap v312: 129 pgs: 129 active+clean; 640 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-24T00:19:29.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:29 smithi078 ceph-mon[32429]: pgmap v313: 129 pgs: 129 active+clean; 640 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-24T00:19:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:29 smithi107 ceph-mon[36485]: pgmap v313: 129 pgs: 129 active+clean; 640 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-24T00:19:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:29 smithi106 ceph-mon[36505]: pgmap v313: 129 pgs: 129 active+clean; 640 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-24T00:19:31.451 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:31 smithi078 ceph-mon[32429]: pgmap v314: 129 pgs: 129 active+clean; 641 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-24T00:19:31.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:31 smithi107 ceph-mon[36485]: pgmap v314: 129 pgs: 129 active+clean; 641 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-24T00:19:31.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:31 smithi106 ceph-mon[36505]: pgmap v314: 129 pgs: 129 active+clean; 641 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-24T00:19:33.686 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:33 smithi078 ceph-mon[32429]: pgmap v315: 129 pgs: 129 active+clean; 641 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-24T00:19:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:33 smithi107 ceph-mon[36485]: pgmap v315: 129 pgs: 129 active+clean; 641 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-24T00:19:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:33 smithi106 ceph-mon[36505]: pgmap v315: 129 pgs: 129 active+clean; 641 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-24T00:19:34.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.573+0000 7f4800ba1700 1 -- 172.21.15.78:0/793116975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47fc1011d0 msgr2=0x7f47fc1015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:34.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.573+0000 7f4800ba1700 1 --2- 172.21.15.78:0/793116975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47fc1011d0 0x7f47fc1015b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f47f40099c0 tx=0x7f47f4009cd0 comp rx=0 tx=0).stop 2024-03-24T00:19:34.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.573+0000 7f4800ba1700 1 -- 172.21.15.78:0/793116975 shutdown_connections 2024-03-24T00:19:34.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.573+0000 7f4800ba1700 1 --2- 172.21.15.78:0/793116975 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47fc1062b0 0x7f47fc10de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:34.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.573+0000 7f4800ba1700 1 --2- 172.21.15.78:0/793116975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47fc1011d0 0x7f47fc1015b0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:34.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.573+0000 7f4800ba1700 1 --2- 172.21.15.78:0/793116975 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47fc101b80 0x7f47fc105bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:34.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.573+0000 7f4800ba1700 1 -- 172.21.15.78:0/793116975 >> 172.21.15.78:0/793116975 conn(0x7f47fc0fca60 msgr2=0x7f47fc0fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:34.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.573+0000 7f4800ba1700 1 -- 172.21.15.78:0/793116975 shutdown_connections 2024-03-24T00:19:34.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.573+0000 7f4800ba1700 1 -- 172.21.15.78:0/793116975 wait complete. 2024-03-24T00:19:34.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.574+0000 7f4800ba1700 1 Processor -- start 2024-03-24T00:19:34.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.574+0000 7f4800ba1700 1 -- start start 2024-03-24T00:19:34.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.574+0000 7f4800ba1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47fc1011d0 0x7f47fc19e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:34.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.574+0000 7f4800ba1700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47fc101b80 0x7f47fc19e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:34.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.574+0000 7f4800ba1700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47fc1062b0 0x7f47fc198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:34.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.574+0000 7f4800ba1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47fc1112b0 con 0x7f47fc1011d0 2024-03-24T00:19:34.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.574+0000 7f4800ba1700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47fc111130 con 0x7f47fc101b80 2024-03-24T00:19:34.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.574+0000 7f4800ba1700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47fc111430 con 0x7f47fc1062b0 2024-03-24T00:19:34.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.574+0000 7f47fb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47fc1011d0 0x7f47fc19e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:34.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.574+0000 7f47fb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47fc1011d0 0x7f47fc19e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43052/0 (socket says 172.21.15.78:43052) 2024-03-24T00:19:34.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.574+0000 7f47fb7fe700 1 -- 172.21.15.78:0/309211580 learned_addr learned my addr 172.21.15.78:0/309211580 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:19:34.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.574+0000 7f47faffd700 1 --2- 172.21.15.78:0/309211580 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47fc101b80 0x7f47fc19e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:34.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.574+0000 7f47fbfff700 1 --2- 172.21.15.78:0/309211580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47fc1062b0 0x7f47fc198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:34.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.575+0000 7f47fb7fe700 1 -- 172.21.15.78:0/309211580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47fc1062b0 msgr2=0x7f47fc198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:34.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.575+0000 7f47fb7fe700 1 --2- 172.21.15.78:0/309211580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47fc1062b0 0x7f47fc198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:34.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.575+0000 7f47fb7fe700 1 -- 172.21.15.78:0/309211580 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47fc101b80 msgr2=0x7f47fc19e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:34.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.575+0000 7f47fb7fe700 1 --2- 172.21.15.78:0/309211580 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47fc101b80 0x7f47fc19e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:34.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.575+0000 7f47fb7fe700 1 -- 172.21.15.78:0/309211580 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47fc198d60 con 0x7f47fc1011d0 2024-03-24T00:19:34.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.575+0000 7f47faffd700 1 --2- 172.21.15.78:0/309211580 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47fc101b80 0x7f47fc19e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:34.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.575+0000 7f47fb7fe700 1 --2- 172.21.15.78:0/309211580 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47fc1011d0 0x7f47fc19e440 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f47f4006010 tx=0x7f47f400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:34.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.575+0000 7f47f8ff9700 1 -- 172.21.15.78:0/309211580 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f4003680 con 0x7f47fc1011d0 2024-03-24T00:19:34.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.575+0000 7f47f8ff9700 1 -- 172.21.15.78:0/309211580 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f47f40050d0 con 0x7f47fc1011d0 2024-03-24T00:19:34.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.575+0000 7f4800ba1700 1 -- 172.21.15.78:0/309211580 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47fc198ff0 con 0x7f47fc1011d0 2024-03-24T00:19:34.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.575+0000 7f47f8ff9700 1 -- 172.21.15.78:0/309211580 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f4003db0 con 0x7f47fc1011d0 2024-03-24T00:19:34.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.575+0000 7f4800ba1700 1 -- 172.21.15.78:0/309211580 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47fc1a5310 con 0x7f47fc1011d0 2024-03-24T00:19:34.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.576+0000 7f4800ba1700 1 -- 172.21.15.78:0/309211580 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f47fc062920 con 0x7f47fc1011d0 2024-03-24T00:19:34.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.577+0000 7f47f8ff9700 1 -- 172.21.15.78:0/309211580 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f47f4005280 con 0x7f47fc1011d0 2024-03-24T00:19:34.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.578+0000 7f47f8ff9700 1 --2- 172.21.15.78:0/309211580 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f47e40a2ac0 0x7f47e40a4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:34.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.578+0000 7f47faffd700 1 --2- 172.21.15.78:0/309211580 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f47e40a2ac0 0x7f47e40a4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:34.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.578+0000 7f47f8ff9700 1 -- 172.21.15.78:0/309211580 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f47f40cfec0 con 0x7f47fc1011d0 2024-03-24T00:19:34.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.578+0000 7f47f8ff9700 1 --2- 172.21.15.78:0/309211580 >> [v2:172.21.15.78:6826/214023431,v1:172.21.15.78:6827/214023431] conn(0x7f47e40ad780 0x7f47e40afbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:34.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.578+0000 7f47f8ff9700 1 -- 172.21.15.78:0/309211580 --> [v2:172.21.15.78:6826/214023431,v1:172.21.15.78:6827/214023431] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47f4003680 con 0x7f47e40ad780 2024-03-24T00:19:34.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.578+0000 7f47f8ff9700 1 -- 172.21.15.78:0/309211580 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f47f40d0310 con 0x7f47fc1011d0 2024-03-24T00:19:34.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.578+0000 7f47fbfff700 1 --2- 172.21.15.78:0/309211580 >> [v2:172.21.15.78:6826/214023431,v1:172.21.15.78:6827/214023431] conn(0x7f47e40ad780 0x7f47e40afbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:34.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.578+0000 7f47faffd700 1 --2- 172.21.15.78:0/309211580 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f47e40a2ac0 0x7f47e40a4f80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f47fc107910 tx=0x7f47f000b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:34.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.578+0000 7f47fbfff700 1 --2- 172.21.15.78:0/309211580 >> [v2:172.21.15.78:6826/214023431,v1:172.21.15.78:6827/214023431] conn(0x7f47e40ad780 0x7f47e40afbc0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:34.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.579+0000 7f47f8ff9700 1 -- 172.21.15.78:0/309211580 <== osd.9 v2:172.21.15.78:6826/214023431 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f47f4003680 con 0x7f47e40ad780 2024-03-24T00:19:34.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.595+0000 7f4800ba1700 1 -- 172.21.15.78:0/309211580 --> [v2:172.21.15.78:6826/214023431,v1:172.21.15.78:6827/214023431] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f47fc062da0 con 0x7f47e40ad780 2024-03-24T00:19:34.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.596+0000 7f47f8ff9700 1 -- 172.21.15.78:0/309211580 <== osd.9 v2:172.21.15.78:6826/214023431 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f47fc062da0 con 0x7f47e40ad780 2024-03-24T00:19:34.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.596+0000 7f4800ba1700 1 -- 172.21.15.78:0/309211580 >> [v2:172.21.15.78:6826/214023431,v1:172.21.15.78:6827/214023431] conn(0x7f47e40ad780 msgr2=0x7f47e40afbc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:34.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.596+0000 7f4800ba1700 1 --2- 172.21.15.78:0/309211580 >> [v2:172.21.15.78:6826/214023431,v1:172.21.15.78:6827/214023431] conn(0x7f47e40ad780 0x7f47e40afbc0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:34.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.596+0000 7f4800ba1700 1 -- 172.21.15.78:0/309211580 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f47e40a2ac0 msgr2=0x7f47e40a4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:34.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.596+0000 7f4800ba1700 1 --2- 172.21.15.78:0/309211580 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f47e40a2ac0 0x7f47e40a4f80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f47fc107910 tx=0x7f47f000b040 comp rx=0 tx=0).stop 2024-03-24T00:19:34.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.596+0000 7f4800ba1700 1 -- 172.21.15.78:0/309211580 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47fc1011d0 msgr2=0x7f47fc19e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:34.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.596+0000 7f4800ba1700 1 --2- 172.21.15.78:0/309211580 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47fc1011d0 0x7f47fc19e440 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f47f4006010 tx=0x7f47f400f690 comp rx=0 tx=0).stop 2024-03-24T00:19:34.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.596+0000 7f47fbfff700 1 -- 172.21.15.78:0/309211580 reap_dead start 2024-03-24T00:19:34.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.596+0000 7f4800ba1700 1 -- 172.21.15.78:0/309211580 shutdown_connections 2024-03-24T00:19:34.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.596+0000 7f4800ba1700 1 -- 172.21.15.78:0/309211580 >> 172.21.15.78:0/309211580 conn(0x7f47fc0fca60 msgr2=0x7f47fc06a9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:34.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.596+0000 7f4800ba1700 1 -- 172.21.15.78:0/309211580 shutdown_connections 2024-03-24T00:19:34.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:32.596+0000 7f4800ba1700 1 -- 172.21.15.78:0/309211580 wait complete. 2024-03-24T00:19:35.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.271+0000 7fc3c8051700 1 -- 172.21.15.78:0/1830242273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3c0100ad0 msgr2=0x7fc3c0100f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:35.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.271+0000 7fc3c8051700 1 --2- 172.21.15.78:0/1830242273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3c0100ad0 0x7fc3c0100f50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc3b000aaa0 tx=0x7fc3b000adb0 comp rx=0 tx=0).stop 2024-03-24T00:19:35.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.271+0000 7fc3c8051700 1 -- 172.21.15.78:0/1830242273 shutdown_connections 2024-03-24T00:19:35.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.271+0000 7fc3c8051700 1 --2- 172.21.15.78:0/1830242273 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3c0101490 0x7fc3c010d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:35.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.271+0000 7fc3c8051700 1 --2- 172.21.15.78:0/1830242273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3c01068b0 0x7fc3c0106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:35.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.271+0000 7fc3c8051700 1 --2- 172.21.15.78:0/1830242273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3c0100ad0 0x7fc3c0100f50 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:35.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.271+0000 7fc3c8051700 1 -- 172.21.15.78:0/1830242273 >> 172.21.15.78:0/1830242273 conn(0x7fc3c00763f0 msgr2=0x7fc3c0076800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:35.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.271+0000 7fc3c8051700 1 -- 172.21.15.78:0/1830242273 shutdown_connections 2024-03-24T00:19:35.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.271+0000 7fc3c8051700 1 -- 172.21.15.78:0/1830242273 wait complete. 2024-03-24T00:19:35.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.272+0000 7fc3c8051700 1 Processor -- start 2024-03-24T00:19:35.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.272+0000 7fc3c8051700 1 -- start start 2024-03-24T00:19:35.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.272+0000 7fc3c8051700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3c0100ad0 0x7fc3c019e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:35.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.272+0000 7fc3c8051700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3c0101490 0x7fc3c019eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:35.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.272+0000 7fc3c8051700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3c01068b0 0x7fc3c01986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:35.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.272+0000 7fc3c8051700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c01102c0 con 0x7fc3c0100ad0 2024-03-24T00:19:35.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.272+0000 7fc3c8051700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c0110140 con 0x7fc3c0101490 2024-03-24T00:19:35.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.272+0000 7fc3c8051700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c0110440 con 0x7fc3c01068b0 2024-03-24T00:19:35.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.272+0000 7fc3c5ded700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3c0100ad0 0x7fc3c019e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:35.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.272+0000 7fc3c5ded700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3c0100ad0 0x7fc3c019e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34342/0 (socket says 172.21.15.78:34342) 2024-03-24T00:19:35.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.272+0000 7fc3c5ded700 1 -- 172.21.15.78:0/1732290140 learned_addr learned my addr 172.21.15.78:0/1732290140 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:19:35.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.272+0000 7fc3c65ee700 1 --2- 172.21.15.78:0/1732290140 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3c01068b0 0x7fc3c01986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:35.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.272+0000 7fc3c55ec700 1 --2- 172.21.15.78:0/1732290140 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3c0101490 0x7fc3c019eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:35.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.272+0000 7fc3c5ded700 1 -- 172.21.15.78:0/1732290140 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3c01068b0 msgr2=0x7fc3c01986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:35.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.272+0000 7fc3c5ded700 1 --2- 172.21.15.78:0/1732290140 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3c01068b0 0x7fc3c01986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:35.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.272+0000 7fc3c5ded700 1 -- 172.21.15.78:0/1732290140 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3c0101490 msgr2=0x7fc3c019eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:35.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.272+0000 7fc3c5ded700 1 --2- 172.21.15.78:0/1732290140 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3c0101490 0x7fc3c019eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:35.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.272+0000 7fc3c5ded700 1 -- 172.21.15.78:0/1732290140 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c0198fd0 con 0x7fc3c0100ad0 2024-03-24T00:19:35.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.272+0000 7fc3c5ded700 1 --2- 172.21.15.78:0/1732290140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3c0100ad0 0x7fc3c019e620 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc00e9c0 tx=0x7fc3bc00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:35.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.273+0000 7fc3b6ffd700 1 -- 172.21.15.78:0/1732290140 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc00cbd0 con 0x7fc3c0100ad0 2024-03-24T00:19:35.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.273+0000 7fc3c8051700 1 -- 172.21.15.78:0/1732290140 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3c01992c0 con 0x7fc3c0100ad0 2024-03-24T00:19:35.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.273+0000 7fc3b6ffd700 1 -- 172.21.15.78:0/1732290140 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc3bc00ee60 con 0x7fc3c0100ad0 2024-03-24T00:19:35.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.273+0000 7fc3b6ffd700 1 -- 172.21.15.78:0/1732290140 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc005630 con 0x7fc3c0100ad0 2024-03-24T00:19:35.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.273+0000 7fc3c8051700 1 -- 172.21.15.78:0/1732290140 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3c01a55a0 con 0x7fc3c0100ad0 2024-03-24T00:19:35.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.274+0000 7fc3c8051700 1 -- 172.21.15.78:0/1732290140 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc3c0062920 con 0x7fc3c0100ad0 2024-03-24T00:19:35.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.275+0000 7fc3b6ffd700 1 -- 172.21.15.78:0/1732290140 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fc3bc004750 con 0x7fc3c0100ad0 2024-03-24T00:19:35.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.275+0000 7fc3b6ffd700 1 --2- 172.21.15.78:0/1732290140 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fc3ac0a2920 0x7fc3ac0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:35.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.275+0000 7fc3b6ffd700 1 -- 172.21.15.78:0/1732290140 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fc3bc0cf4d0 con 0x7fc3c0100ad0 2024-03-24T00:19:35.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.275+0000 7fc3c55ec700 1 --2- 172.21.15.78:0/1732290140 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fc3ac0a2920 0x7fc3ac0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:35.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.276+0000 7fc3b6ffd700 1 --2- 172.21.15.78:0/1732290140 >> [v2:172.21.15.106:6824/400029804,v1:172.21.15.106:6825/400029804] conn(0x7fc3ac0ad5e0 0x7fc3ac0afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:35.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.276+0000 7fc3b6ffd700 1 -- 172.21.15.78:0/1732290140 --> [v2:172.21.15.106:6824/400029804,v1:172.21.15.106:6825/400029804] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc3bc01fe50 con 0x7fc3ac0ad5e0 2024-03-24T00:19:35.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.276+0000 7fc3b6ffd700 1 -- 172.21.15.78:0/1732290140 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc3bc0cf920 con 0x7fc3c0100ad0 2024-03-24T00:19:35.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.276+0000 7fc3c65ee700 1 --2- 172.21.15.78:0/1732290140 >> [v2:172.21.15.106:6824/400029804,v1:172.21.15.106:6825/400029804] conn(0x7fc3ac0ad5e0 0x7fc3ac0afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:35.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.276+0000 7fc3c55ec700 1 --2- 172.21.15.78:0/1732290140 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fc3ac0a2920 0x7fc3ac0a4de0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc3b000c6d0 tx=0x7fc3b0005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:35.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.276+0000 7fc3c65ee700 1 --2- 172.21.15.78:0/1732290140 >> [v2:172.21.15.106:6824/400029804,v1:172.21.15.106:6825/400029804] conn(0x7fc3ac0ad5e0 0x7fc3ac0afa20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:35.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.277+0000 7fc3b6ffd700 1 -- 172.21.15.78:0/1732290140 <== osd.10 v2:172.21.15.106:6824/400029804 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc3bc01fe50 con 0x7fc3ac0ad5e0 2024-03-24T00:19:35.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.293+0000 7fc3c8051700 1 -- 172.21.15.78:0/1732290140 --> [v2:172.21.15.106:6824/400029804,v1:172.21.15.106:6825/400029804] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc3c0062da0 con 0x7fc3ac0ad5e0 2024-03-24T00:19:35.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.293+0000 7fc3b6ffd700 1 -- 172.21.15.78:0/1732290140 <== osd.10 v2:172.21.15.106:6824/400029804 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fc3c0062da0 con 0x7fc3ac0ad5e0 2024-03-24T00:19:35.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.294+0000 7fc3c8051700 1 -- 172.21.15.78:0/1732290140 >> [v2:172.21.15.106:6824/400029804,v1:172.21.15.106:6825/400029804] conn(0x7fc3ac0ad5e0 msgr2=0x7fc3ac0afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:35.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.294+0000 7fc3c8051700 1 --2- 172.21.15.78:0/1732290140 >> [v2:172.21.15.106:6824/400029804,v1:172.21.15.106:6825/400029804] conn(0x7fc3ac0ad5e0 0x7fc3ac0afa20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:35.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.294+0000 7fc3c8051700 1 -- 172.21.15.78:0/1732290140 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fc3ac0a2920 msgr2=0x7fc3ac0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:35.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.294+0000 7fc3c8051700 1 --2- 172.21.15.78:0/1732290140 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fc3ac0a2920 0x7fc3ac0a4de0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc3b000c6d0 tx=0x7fc3b0005960 comp rx=0 tx=0).stop 2024-03-24T00:19:35.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.294+0000 7fc3c8051700 1 -- 172.21.15.78:0/1732290140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3c0100ad0 msgr2=0x7fc3c019e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:35.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.294+0000 7fc3c8051700 1 --2- 172.21.15.78:0/1732290140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3c0100ad0 0x7fc3c019e620 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc00e9c0 tx=0x7fc3bc00ecd0 comp rx=0 tx=0).stop 2024-03-24T00:19:35.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.294+0000 7fc3c65ee700 1 -- 172.21.15.78:0/1732290140 reap_dead start 2024-03-24T00:19:35.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.294+0000 7fc3c8051700 1 -- 172.21.15.78:0/1732290140 shutdown_connections 2024-03-24T00:19:35.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.294+0000 7fc3c8051700 1 -- 172.21.15.78:0/1732290140 >> 172.21.15.78:0/1732290140 conn(0x7fc3c00763f0 msgr2=0x7fc3c00fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:35.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.294+0000 7fc3c8051700 1 -- 172.21.15.78:0/1732290140 shutdown_connections 2024-03-24T00:19:35.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:35.294+0000 7fc3c8051700 1 -- 172.21.15.78:0/1732290140 wait complete. 2024-03-24T00:19:35.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:35 smithi078 ceph-mon[32429]: pgmap v316: 129 pgs: 129 active+clean; 641 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-24T00:19:35.675 INFO:teuthology.orchestra.run.smithi078.stdout:287762808866 2024-03-24T00:19:35.676 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-24T00:19:35.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:35 smithi107 ceph-mon[36485]: pgmap v316: 129 pgs: 129 active+clean; 641 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-24T00:19:35.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:35 smithi106 ceph-mon[36505]: pgmap v316: 129 pgs: 129 active+clean; 641 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-24T00:19:36.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.206+0000 7f71318f5700 1 -- 172.21.15.78:0/146649101 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f712c102d80 msgr2=0x7f712c106dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:36.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.206+0000 7f71318f5700 1 --2- 172.21.15.78:0/146649101 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f712c102d80 0x7f712c106dd0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f71140099c0 tx=0x7f7114009cd0 comp rx=0 tx=0).stop 2024-03-24T00:19:36.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.206+0000 7f71318f5700 1 -- 172.21.15.78:0/146649101 shutdown_connections 2024-03-24T00:19:36.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.206+0000 7f71318f5700 1 --2- 172.21.15.78:0/146649101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f712c1074b0 0x7f712c10f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:36.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.206+0000 7f71318f5700 1 --2- 172.21.15.78:0/146649101 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f712c1023d0 0x7f712c1027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:36.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.206+0000 7f71318f5700 1 --2- 172.21.15.78:0/146649101 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f712c102d80 0x7f712c106dd0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:36.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.206+0000 7f71318f5700 1 -- 172.21.15.78:0/146649101 >> 172.21.15.78:0/146649101 conn(0x7f712c0fdc40 msgr2=0x7f712c100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:36.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.206+0000 7f71318f5700 1 -- 172.21.15.78:0/146649101 shutdown_connections 2024-03-24T00:19:36.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.206+0000 7f71318f5700 1 -- 172.21.15.78:0/146649101 wait complete. 2024-03-24T00:19:36.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.206+0000 7f71318f5700 1 Processor -- start 2024-03-24T00:19:36.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.207+0000 7f71318f5700 1 -- start start 2024-03-24T00:19:36.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.207+0000 7f71318f5700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f712c1023d0 0x7f712c19e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:36.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.207+0000 7f71318f5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f712c102d80 0x7f712c19ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:36.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.207+0000 7f71318f5700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f712c1074b0 0x7f712c198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:36.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.207+0000 7f71318f5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712c112490 con 0x7f712c102d80 2024-03-24T00:19:36.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.207+0000 7f71318f5700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712c112310 con 0x7f712c1074b0 2024-03-24T00:19:36.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.207+0000 7f71318f5700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712c112610 con 0x7f712c1023d0 2024-03-24T00:19:36.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.207+0000 7f71227fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f712c102d80 0x7f712c19ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:36.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.207+0000 7f712affd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f712c1023d0 0x7f712c19e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:36.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.207+0000 7f71227fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f712c102d80 0x7f712c19ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34350/0 (socket says 172.21.15.78:34350) 2024-03-24T00:19:36.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.207+0000 7f71227fc700 1 -- 172.21.15.78:0/2321592627 learned_addr learned my addr 172.21.15.78:0/2321592627 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:19:36.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.207+0000 7f712b7fe700 1 --2- 172.21.15.78:0/2321592627 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f712c1074b0 0x7f712c198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:36.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.207+0000 7f71227fc700 1 -- 172.21.15.78:0/2321592627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f712c1023d0 msgr2=0x7f712c19e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:36.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.207+0000 7f71227fc700 1 --2- 172.21.15.78:0/2321592627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f712c1023d0 0x7f712c19e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:36.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.207+0000 7f71227fc700 1 -- 172.21.15.78:0/2321592627 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f712c1074b0 msgr2=0x7f712c198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:36.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.207+0000 7f71227fc700 1 --2- 172.21.15.78:0/2321592627 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f712c1074b0 0x7f712c198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:36.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.207+0000 7f71227fc700 1 -- 172.21.15.78:0/2321592627 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f712c199050 con 0x7f712c102d80 2024-03-24T00:19:36.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.207+0000 7f712affd700 1 --2- 172.21.15.78:0/2321592627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f712c1023d0 0x7f712c19e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:36.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.208+0000 7f71227fc700 1 --2- 172.21.15.78:0/2321592627 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f712c102d80 0x7f712c19ebe0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f711400f690 tx=0x7f7114005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:36.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.208+0000 7f71308f3700 1 -- 172.21.15.78:0/2321592627 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7114027070 con 0x7f712c102d80 2024-03-24T00:19:36.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.208+0000 7f71308f3700 1 -- 172.21.15.78:0/2321592627 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7114003680 con 0x7f712c102d80 2024-03-24T00:19:36.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.208+0000 7f71318f5700 1 -- 172.21.15.78:0/2321592627 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f712c1992e0 con 0x7f712c102d80 2024-03-24T00:19:36.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.208+0000 7f71308f3700 1 -- 172.21.15.78:0/2321592627 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71140173f0 con 0x7f712c102d80 2024-03-24T00:19:36.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.208+0000 7f71318f5700 1 -- 172.21.15.78:0/2321592627 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f712c1a5450 con 0x7f712c102d80 2024-03-24T00:19:36.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.209+0000 7f71318f5700 1 -- 172.21.15.78:0/2321592627 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f712c0628e0 con 0x7f712c102d80 2024-03-24T00:19:36.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.210+0000 7f71308f3700 1 -- 172.21.15.78:0/2321592627 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f711402a070 con 0x7f712c102d80 2024-03-24T00:19:36.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.210+0000 7f71308f3700 1 --2- 172.21.15.78:0/2321592627 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f71180a2920 0x7f71180a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:36.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.210+0000 7f712affd700 1 --2- 172.21.15.78:0/2321592627 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f71180a2920 0x7f71180a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:36.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.210+0000 7f71308f3700 1 -- 172.21.15.78:0/2321592627 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f71140d0440 con 0x7f712c102d80 2024-03-24T00:19:36.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.211+0000 7f71308f3700 1 --2- 172.21.15.78:0/2321592627 >> [v2:172.21.15.107:6800/2010770435,v1:172.21.15.107:6801/2010770435] conn(0x7f71180ad5e0 0x7f71180afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:36.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.211+0000 7f71308f3700 1 -- 172.21.15.78:0/2321592627 --> [v2:172.21.15.107:6800/2010770435,v1:172.21.15.107:6801/2010770435] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7114010ec0 con 0x7f71180ad5e0 2024-03-24T00:19:36.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.211+0000 7f71308f3700 1 -- 172.21.15.78:0/2321592627 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f71140d0890 con 0x7f712c102d80 2024-03-24T00:19:36.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.211+0000 7f712affd700 1 --2- 172.21.15.78:0/2321592627 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f71180a2920 0x7f71180a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f711c011490 tx=0x7f711c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:36.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.211+0000 7f712b7fe700 1 --2- 172.21.15.78:0/2321592627 >> [v2:172.21.15.107:6800/2010770435,v1:172.21.15.107:6801/2010770435] conn(0x7f71180ad5e0 0x7f71180afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:36.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.211+0000 7f712b7fe700 1 --2- 172.21.15.78:0/2321592627 >> [v2:172.21.15.107:6800/2010770435,v1:172.21.15.107:6801/2010770435] conn(0x7f71180ad5e0 0x7f71180afa20 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:36.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.212+0000 7f71308f3700 1 -- 172.21.15.78:0/2321592627 <== osd.2 v2:172.21.15.107:6800/2010770435 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7114010ec0 con 0x7f71180ad5e0 2024-03-24T00:19:36.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.228+0000 7f71318f5700 1 -- 172.21.15.78:0/2321592627 --> [v2:172.21.15.107:6800/2010770435,v1:172.21.15.107:6801/2010770435] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f712c062d60 con 0x7f71180ad5e0 2024-03-24T00:19:36.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.228+0000 7f71308f3700 1 -- 172.21.15.78:0/2321592627 <== osd.2 v2:172.21.15.107:6800/2010770435 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f712c062d60 con 0x7f71180ad5e0 2024-03-24T00:19:36.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.229+0000 7f71318f5700 1 -- 172.21.15.78:0/2321592627 >> [v2:172.21.15.107:6800/2010770435,v1:172.21.15.107:6801/2010770435] conn(0x7f71180ad5e0 msgr2=0x7f71180afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:36.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.229+0000 7f71318f5700 1 --2- 172.21.15.78:0/2321592627 >> [v2:172.21.15.107:6800/2010770435,v1:172.21.15.107:6801/2010770435] conn(0x7f71180ad5e0 0x7f71180afa20 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:36.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.229+0000 7f71318f5700 1 -- 172.21.15.78:0/2321592627 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f71180a2920 msgr2=0x7f71180a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:36.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.229+0000 7f71318f5700 1 --2- 172.21.15.78:0/2321592627 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f71180a2920 0x7f71180a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f711c011490 tx=0x7f711c009260 comp rx=0 tx=0).stop 2024-03-24T00:19:36.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.229+0000 7f71318f5700 1 -- 172.21.15.78:0/2321592627 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f712c102d80 msgr2=0x7f712c19ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:36.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.229+0000 7f71318f5700 1 --2- 172.21.15.78:0/2321592627 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f712c102d80 0x7f712c19ebe0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f711400f690 tx=0x7f7114005f10 comp rx=0 tx=0).stop 2024-03-24T00:19:36.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.229+0000 7f712b7fe700 1 -- 172.21.15.78:0/2321592627 reap_dead start 2024-03-24T00:19:36.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.229+0000 7f71318f5700 1 -- 172.21.15.78:0/2321592627 shutdown_connections 2024-03-24T00:19:36.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.229+0000 7f71318f5700 1 -- 172.21.15.78:0/2321592627 >> 172.21.15.78:0/2321592627 conn(0x7f712c0fdc40 msgr2=0x7f712c0fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:36.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.229+0000 7f71318f5700 1 -- 172.21.15.78:0/2321592627 shutdown_connections 2024-03-24T00:19:36.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:36.229+0000 7f71318f5700 1 -- 172.21.15.78:0/2321592627 wait complete. 2024-03-24T00:19:37.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.414+0000 7f5f1519e700 1 -- 172.21.15.78:0/2468337314 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f10102240 msgr2=0x7f5f101026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:37.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.414+0000 7f5f1519e700 1 --2- 172.21.15.78:0/2468337314 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f10102240 0x7f5f101026c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5f04009990 tx=0x7f5f04009ca0 comp rx=0 tx=0).stop 2024-03-24T00:19:37.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.414+0000 7f5f1519e700 1 -- 172.21.15.78:0/2468337314 shutdown_connections 2024-03-24T00:19:37.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.414+0000 7f5f1519e700 1 --2- 172.21.15.78:0/2468337314 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f10102c00 0x7f5f1010eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:37.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.414+0000 7f5f1519e700 1 --2- 172.21.15.78:0/2468337314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f101082b0 0x7f5f10108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:37.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.414+0000 7f5f1519e700 1 --2- 172.21.15.78:0/2468337314 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f10102240 0x7f5f101026c0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:37.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.414+0000 7f5f1519e700 1 -- 172.21.15.78:0/2468337314 >> 172.21.15.78:0/2468337314 conn(0x7f5f10079260 msgr2=0x7f5f10079670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:37.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.414+0000 7f5f1519e700 1 -- 172.21.15.78:0/2468337314 shutdown_connections 2024-03-24T00:19:37.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.414+0000 7f5f1519e700 1 -- 172.21.15.78:0/2468337314 wait complete. 2024-03-24T00:19:37.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.415+0000 7f5f1519e700 1 Processor -- start 2024-03-24T00:19:37.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.415+0000 7f5f1519e700 1 -- start start 2024-03-24T00:19:37.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.416+0000 7f5f1519e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f10102240 0x7f5f1019e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:37.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.416+0000 7f5f1519e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f10102c00 0x7f5f1019eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:37.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.416+0000 7f5f1519e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f101082b0 0x7f5f10198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:37.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.416+0000 7f5f1519e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f101124f0 con 0x7f5f101082b0 2024-03-24T00:19:37.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.416+0000 7f5f1519e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f10112370 con 0x7f5f10102240 2024-03-24T00:19:37.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.416+0000 7f5f1519e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f10112670 con 0x7f5f10102c00 2024-03-24T00:19:37.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.416+0000 7f5f0f59e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f101082b0 0x7f5f10198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:37.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.416+0000 7f5f0ed9d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f10102240 0x7f5f1019e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:37.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.416+0000 7f5f0f59e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f101082b0 0x7f5f10198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34372/0 (socket says 172.21.15.78:34372) 2024-03-24T00:19:37.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.416+0000 7f5f0f59e700 1 -- 172.21.15.78:0/2355602877 learned_addr learned my addr 172.21.15.78:0/2355602877 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:19:37.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.416+0000 7f5f0e59c700 1 --2- 172.21.15.78:0/2355602877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f10102c00 0x7f5f1019eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:37.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.416+0000 7f5f0ed9d700 1 -- 172.21.15.78:0/2355602877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f10102c00 msgr2=0x7f5f1019eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:37.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.416+0000 7f5f0ed9d700 1 --2- 172.21.15.78:0/2355602877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f10102c00 0x7f5f1019eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:37.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.416+0000 7f5f0ed9d700 1 -- 172.21.15.78:0/2355602877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f101082b0 msgr2=0x7f5f10198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:37.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.416+0000 7f5f0ed9d700 1 --2- 172.21.15.78:0/2355602877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f101082b0 0x7f5f10198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:37.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.416+0000 7f5f0ed9d700 1 -- 172.21.15.78:0/2355602877 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f10199050 con 0x7f5f10102240 2024-03-24T00:19:37.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.416+0000 7f5f0e59c700 1 --2- 172.21.15.78:0/2355602877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f10102c00 0x7f5f1019eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:37.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.416+0000 7f5f0f59e700 1 --2- 172.21.15.78:0/2355602877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f101082b0 0x7f5f10198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:37.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.416+0000 7f5f0ed9d700 1 --2- 172.21.15.78:0/2355602877 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f10102240 0x7f5f1019e650 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5f0000e9f0 tx=0x7f5f0000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:37.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.416+0000 7f5ef7fff700 1 -- 172.21.15.78:0/2355602877 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f00004220 con 0x7f5f10102240 2024-03-24T00:19:37.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.416+0000 7f5f1519e700 1 -- 172.21.15.78:0/2355602877 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f10199340 con 0x7f5f10102240 2024-03-24T00:19:37.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.416+0000 7f5ef7fff700 1 -- 172.21.15.78:0/2355602877 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f00004d10 con 0x7f5f10102240 2024-03-24T00:19:37.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.416+0000 7f5ef7fff700 1 -- 172.21.15.78:0/2355602877 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f00005020 con 0x7f5f10102240 2024-03-24T00:19:37.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.417+0000 7f5f1519e700 1 -- 172.21.15.78:0/2355602877 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f101a55d0 con 0x7f5f10102240 2024-03-24T00:19:37.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.417+0000 7f5f1519e700 1 -- 172.21.15.78:0/2355602877 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5f100628a0 con 0x7f5f10102240 2024-03-24T00:19:37.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.419+0000 7f5ef7fff700 1 -- 172.21.15.78:0/2355602877 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f5f00004750 con 0x7f5f10102240 2024-03-24T00:19:37.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.419+0000 7f5ef7fff700 1 --2- 172.21.15.78:0/2355602877 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f5ef80a2920 0x7f5ef80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:37.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.419+0000 7f5f0e59c700 1 --2- 172.21.15.78:0/2355602877 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f5ef80a2920 0x7f5ef80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:37.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.419+0000 7f5ef7fff700 1 -- 172.21.15.78:0/2355602877 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f5f000cefe0 con 0x7f5f10102240 2024-03-24T00:19:37.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.420+0000 7f5ef7fff700 1 --2- 172.21.15.78:0/2355602877 >> [v2:172.21.15.106:6800/1647915612,v1:172.21.15.106:6801/1647915612] conn(0x7f5ef80ad5e0 0x7f5ef80afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:37.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.420+0000 7f5ef7fff700 1 -- 172.21.15.78:0/2355602877 --> [v2:172.21.15.106:6800/1647915612,v1:172.21.15.106:6801/1647915612] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5f00012ec0 con 0x7f5ef80ad5e0 2024-03-24T00:19:37.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.420+0000 7f5ef7fff700 1 -- 172.21.15.78:0/2355602877 <== mon.2 v2:172.21.15.106:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5f000cf390 con 0x7f5f10102240 2024-03-24T00:19:37.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.420+0000 7f5f0e59c700 1 --2- 172.21.15.78:0/2355602877 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f5ef80a2920 0x7f5ef80a4de0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5f0400b2e0 tx=0x7f5f04005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:37.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.420+0000 7f5f0f59e700 1 --2- 172.21.15.78:0/2355602877 >> [v2:172.21.15.106:6800/1647915612,v1:172.21.15.106:6801/1647915612] conn(0x7f5ef80ad5e0 0x7f5ef80afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:37.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.420+0000 7f5f0f59e700 1 --2- 172.21.15.78:0/2355602877 >> [v2:172.21.15.106:6800/1647915612,v1:172.21.15.106:6801/1647915612] conn(0x7f5ef80ad5e0 0x7f5ef80afa20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:37.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.421+0000 7f5ef7fff700 1 -- 172.21.15.78:0/2355602877 <== osd.1 v2:172.21.15.106:6800/1647915612 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5f00012ec0 con 0x7f5ef80ad5e0 2024-03-24T00:19:37.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.437+0000 7f5f1519e700 1 -- 172.21.15.78:0/2355602877 --> [v2:172.21.15.106:6800/1647915612,v1:172.21.15.106:6801/1647915612] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5f10062d50 con 0x7f5ef80ad5e0 2024-03-24T00:19:37.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.438+0000 7f5ef7fff700 1 -- 172.21.15.78:0/2355602877 <== osd.1 v2:172.21.15.106:6800/1647915612 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5f10062d50 con 0x7f5ef80ad5e0 2024-03-24T00:19:37.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.438+0000 7f5f1519e700 1 -- 172.21.15.78:0/2355602877 >> [v2:172.21.15.106:6800/1647915612,v1:172.21.15.106:6801/1647915612] conn(0x7f5ef80ad5e0 msgr2=0x7f5ef80afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.438+0000 7f5f1519e700 1 --2- 172.21.15.78:0/2355602877 >> [v2:172.21.15.106:6800/1647915612,v1:172.21.15.106:6801/1647915612] conn(0x7f5ef80ad5e0 0x7f5ef80afa20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.438+0000 7f5f1519e700 1 -- 172.21.15.78:0/2355602877 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f5ef80a2920 msgr2=0x7f5ef80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.438+0000 7f5f1519e700 1 --2- 172.21.15.78:0/2355602877 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f5ef80a2920 0x7f5ef80a4de0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5f0400b2e0 tx=0x7f5f04005960 comp rx=0 tx=0).stop 2024-03-24T00:19:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.438+0000 7f5f1519e700 1 -- 172.21.15.78:0/2355602877 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f10102240 msgr2=0x7f5f1019e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.438+0000 7f5f1519e700 1 --2- 172.21.15.78:0/2355602877 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f10102240 0x7f5f1019e650 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5f0000e9f0 tx=0x7f5f0000ed00 comp rx=0 tx=0).stop 2024-03-24T00:19:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.438+0000 7f5f0f59e700 1 -- 172.21.15.78:0/2355602877 reap_dead start 2024-03-24T00:19:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.438+0000 7f5f1519e700 1 -- 172.21.15.78:0/2355602877 shutdown_connections 2024-03-24T00:19:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.438+0000 7f5f1519e700 1 -- 172.21.15.78:0/2355602877 >> 172.21.15.78:0/2355602877 conn(0x7f5f10079260 msgr2=0x7f5f10100ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.438+0000 7f5f1519e700 1 -- 172.21.15.78:0/2355602877 shutdown_connections 2024-03-24T00:19:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:37.438+0000 7f5f1519e700 1 -- 172.21.15.78:0/2355602877 wait complete. 2024-03-24T00:19:37.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:37 smithi078 ceph-mon[32429]: pgmap v317: 129 pgs: 129 active+clean; 641 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-24T00:19:37.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:37 smithi107 ceph-mon[36485]: pgmap v317: 129 pgs: 129 active+clean; 641 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-24T00:19:37.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:37 smithi106 ceph-mon[36505]: pgmap v317: 129 pgs: 129 active+clean; 641 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-24T00:19:38.907 INFO:teuthology.orchestra.run.smithi078.stdout:309237645341 2024-03-24T00:19:38.907 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-24T00:19:39.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.249+0000 7fda8c484700 1 -- 172.21.15.78:0/3298574228 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda841081c0 msgr2=0x7fda841085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:39.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.249+0000 7fda8c484700 1 --2- 172.21.15.78:0/3298574228 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda841081c0 0x7fda841085a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fda800099c0 tx=0x7fda80009cd0 comp rx=0 tx=0).stop 2024-03-24T00:19:39.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.249+0000 7fda8c484700 1 -- 172.21.15.78:0/3298574228 shutdown_connections 2024-03-24T00:19:39.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.249+0000 7fda8c484700 1 --2- 172.21.15.78:0/3298574228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda84102ba0 0x7fda8410efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:39.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.249+0000 7fda8c484700 1 --2- 172.21.15.78:0/3298574228 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda841081c0 0x7fda841085a0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:39.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.249+0000 7fda8c484700 1 --2- 172.21.15.78:0/3298574228 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda841021e0 0x7fda84102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:39.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.249+0000 7fda8c484700 1 -- 172.21.15.78:0/3298574228 >> 172.21.15.78:0/3298574228 conn(0x7fda840792c0 msgr2=0x7fda840796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:39.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.249+0000 7fda8c484700 1 -- 172.21.15.78:0/3298574228 shutdown_connections 2024-03-24T00:19:39.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.249+0000 7fda8c484700 1 -- 172.21.15.78:0/3298574228 wait complete. 2024-03-24T00:19:39.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.249+0000 7fda8c484700 1 Processor -- start 2024-03-24T00:19:39.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.250+0000 7fda8c484700 1 -- start start 2024-03-24T00:19:39.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.250+0000 7fda8c484700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda841021e0 0x7fda8419e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:39.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.250+0000 7fda8c484700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda84102ba0 0x7fda8419eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:39.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.250+0000 7fda8c484700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda841081c0 0x7fda84198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:39.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.250+0000 7fda8c484700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda84112440 con 0x7fda841081c0 2024-03-24T00:19:39.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.250+0000 7fda8c484700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda841122c0 con 0x7fda84102ba0 2024-03-24T00:19:39.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.250+0000 7fda8c484700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda841125c0 con 0x7fda841021e0 2024-03-24T00:19:39.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.250+0000 7fda8aa21700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda841081c0 0x7fda84198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:39.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.250+0000 7fda8a220700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda841021e0 0x7fda8419e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:39.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.250+0000 7fda8aa21700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda841081c0 0x7fda84198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34390/0 (socket says 172.21.15.78:34390) 2024-03-24T00:19:39.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.250+0000 7fda8a220700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda841021e0 0x7fda8419e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.78:56872/0 (socket says 172.21.15.78:56872) 2024-03-24T00:19:39.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.250+0000 7fda8aa21700 1 -- 172.21.15.78:0/2239687453 learned_addr learned my addr 172.21.15.78:0/2239687453 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:19:39.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.250+0000 7fda89a1f700 1 --2- 172.21.15.78:0/2239687453 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda84102ba0 0x7fda8419eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:39.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.250+0000 7fda8aa21700 1 -- 172.21.15.78:0/2239687453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda841021e0 msgr2=0x7fda8419e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:39.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.250+0000 7fda8aa21700 1 --2- 172.21.15.78:0/2239687453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda841021e0 0x7fda8419e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:39.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.250+0000 7fda8aa21700 1 -- 172.21.15.78:0/2239687453 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda84102ba0 msgr2=0x7fda8419eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:39.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.250+0000 7fda8aa21700 1 --2- 172.21.15.78:0/2239687453 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda84102ba0 0x7fda8419eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:39.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.250+0000 7fda8aa21700 1 -- 172.21.15.78:0/2239687453 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda84198f60 con 0x7fda841081c0 2024-03-24T00:19:39.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.250+0000 7fda8a220700 1 --2- 172.21.15.78:0/2239687453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda841021e0 0x7fda8419e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:39.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.250+0000 7fda89a1f700 1 --2- 172.21.15.78:0/2239687453 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda84102ba0 0x7fda8419eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:39.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.251+0000 7fda8aa21700 1 --2- 172.21.15.78:0/2239687453 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda841081c0 0x7fda84198680 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fda7c00ef30 tx=0x7fda7c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:39.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.251+0000 7fda7b7fe700 1 -- 172.21.15.78:0/2239687453 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7c0092e0 con 0x7fda841081c0 2024-03-24T00:19:39.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.251+0000 7fda8c484700 1 -- 172.21.15.78:0/2239687453 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda841991f0 con 0x7fda841081c0 2024-03-24T00:19:39.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.251+0000 7fda8c484700 1 -- 172.21.15.78:0/2239687453 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda841a5530 con 0x7fda841081c0 2024-03-24T00:19:39.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.251+0000 7fda7b7fe700 1 -- 172.21.15.78:0/2239687453 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda7c004bb0 con 0x7fda841081c0 2024-03-24T00:19:39.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.251+0000 7fda7b7fe700 1 -- 172.21.15.78:0/2239687453 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7c004280 con 0x7fda841081c0 2024-03-24T00:19:39.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.252+0000 7fda8c484700 1 -- 172.21.15.78:0/2239687453 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fda68001030 con 0x7fda841081c0 2024-03-24T00:19:39.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.253+0000 7fda7b7fe700 1 -- 172.21.15.78:0/2239687453 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fda7c01b030 con 0x7fda841081c0 2024-03-24T00:19:39.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.254+0000 7fda7b7fe700 1 --2- 172.21.15.78:0/2239687453 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fda700a29a0 0x7fda700a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:39.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.254+0000 7fda8a220700 1 --2- 172.21.15.78:0/2239687453 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fda700a29a0 0x7fda700a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:39.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.254+0000 7fda7b7fe700 1 -- 172.21.15.78:0/2239687453 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fda7c0ce780 con 0x7fda841081c0 2024-03-24T00:19:39.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.254+0000 7fda7b7fe700 1 --2- 172.21.15.78:0/2239687453 >> [v2:172.21.15.107:6816/4032799356,v1:172.21.15.107:6817/4032799356] conn(0x7fda700ad660 0x7fda700afaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:39.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.254+0000 7fda7b7fe700 1 -- 172.21.15.78:0/2239687453 --> [v2:172.21.15.107:6816/4032799356,v1:172.21.15.107:6817/4032799356] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda7c016e50 con 0x7fda700ad660 2024-03-24T00:19:39.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.255+0000 7fda7b7fe700 1 -- 172.21.15.78:0/2239687453 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda7c0d5050 con 0x7fda841081c0 2024-03-24T00:19:39.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.255+0000 7fda8a220700 1 --2- 172.21.15.78:0/2239687453 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fda700a29a0 0x7fda700a4e60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fda80009990 tx=0x7fda80005a90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:39.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.255+0000 7fda89a1f700 1 --2- 172.21.15.78:0/2239687453 >> [v2:172.21.15.107:6816/4032799356,v1:172.21.15.107:6817/4032799356] conn(0x7fda700ad660 0x7fda700afaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:39.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.255+0000 7fda89a1f700 1 --2- 172.21.15.78:0/2239687453 >> [v2:172.21.15.107:6816/4032799356,v1:172.21.15.107:6817/4032799356] conn(0x7fda700ad660 0x7fda700afaa0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:39.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.256+0000 7fda7b7fe700 1 -- 172.21.15.78:0/2239687453 <== osd.8 v2:172.21.15.107:6816/4032799356 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fda7c016e50 con 0x7fda700ad660 2024-03-24T00:19:39.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.273+0000 7fda8c484700 1 -- 172.21.15.78:0/2239687453 --> [v2:172.21.15.107:6816/4032799356,v1:172.21.15.107:6817/4032799356] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fda68002e60 con 0x7fda700ad660 2024-03-24T00:19:39.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.273+0000 7fda7b7fe700 1 -- 172.21.15.78:0/2239687453 <== osd.8 v2:172.21.15.107:6816/4032799356 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fda68002e60 con 0x7fda700ad660 2024-03-24T00:19:39.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.273+0000 7fda8c484700 1 -- 172.21.15.78:0/2239687453 >> [v2:172.21.15.107:6816/4032799356,v1:172.21.15.107:6817/4032799356] conn(0x7fda700ad660 msgr2=0x7fda700afaa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:39.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.273+0000 7fda8c484700 1 --2- 172.21.15.78:0/2239687453 >> [v2:172.21.15.107:6816/4032799356,v1:172.21.15.107:6817/4032799356] conn(0x7fda700ad660 0x7fda700afaa0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:39.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.274+0000 7fda8c484700 1 -- 172.21.15.78:0/2239687453 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fda700a29a0 msgr2=0x7fda700a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:39.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.274+0000 7fda8c484700 1 --2- 172.21.15.78:0/2239687453 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fda700a29a0 0x7fda700a4e60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fda80009990 tx=0x7fda80005a90 comp rx=0 tx=0).stop 2024-03-24T00:19:39.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.274+0000 7fda8c484700 1 -- 172.21.15.78:0/2239687453 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda841081c0 msgr2=0x7fda84198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:39.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.274+0000 7fda8c484700 1 --2- 172.21.15.78:0/2239687453 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda841081c0 0x7fda84198680 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fda7c00ef30 tx=0x7fda7c00c3a0 comp rx=0 tx=0).stop 2024-03-24T00:19:39.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.274+0000 7fda8aa21700 1 -- 172.21.15.78:0/2239687453 reap_dead start 2024-03-24T00:19:39.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.274+0000 7fda8c484700 1 -- 172.21.15.78:0/2239687453 shutdown_connections 2024-03-24T00:19:39.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.274+0000 7fda8c484700 1 -- 172.21.15.78:0/2239687453 >> 172.21.15.78:0/2239687453 conn(0x7fda840792c0 msgr2=0x7fda84100da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:39.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.274+0000 7fda8c484700 1 -- 172.21.15.78:0/2239687453 shutdown_connections 2024-03-24T00:19:39.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:39.274+0000 7fda8c484700 1 -- 172.21.15.78:0/2239687453 wait complete. 2024-03-24T00:19:39.772 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:39 smithi078 ceph-mon[32429]: pgmap v318: 129 pgs: 129 active+clean; 641 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-24T00:19:39.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:39 smithi107 ceph-mon[36485]: pgmap v318: 129 pgs: 129 active+clean; 641 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-24T00:19:39.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:39 smithi106 ceph-mon[36505]: pgmap v318: 129 pgs: 129 active+clean; 641 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-24T00:19:40.050 INFO:teuthology.orchestra.run.smithi078.stdout:55834574923 2024-03-24T00:19:40.050 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-24T00:19:40.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.644+0000 7f01f548f700 1 -- 172.21.15.78:0/3034860874 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01f01063a0 msgr2=0x7f01f010df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:40.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.644+0000 7f01f548f700 1 --2- 172.21.15.78:0/3034860874 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01f01063a0 0x7f01f010df70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f01e0009990 tx=0x7f01e0009ca0 comp rx=0 tx=0).stop 2024-03-24T00:19:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.644+0000 7f01f548f700 1 -- 172.21.15.78:0/3034860874 shutdown_connections 2024-03-24T00:19:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.644+0000 7f01f548f700 1 --2- 172.21.15.78:0/3034860874 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01f01063a0 0x7f01f010df70 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.644+0000 7f01f548f700 1 --2- 172.21.15.78:0/3034860874 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f01f01012e0 0x7f01f01016c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.644+0000 7f01f548f700 1 --2- 172.21.15.78:0/3034860874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01f0101c90 0x7f01f0105cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.644+0000 7f01f548f700 1 -- 172.21.15.78:0/3034860874 >> 172.21.15.78:0/3034860874 conn(0x7f01f0078000 msgr2=0x7f01f0078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.644+0000 7f01f548f700 1 -- 172.21.15.78:0/3034860874 shutdown_connections 2024-03-24T00:19:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.644+0000 7f01f548f700 1 -- 172.21.15.78:0/3034860874 wait complete. 2024-03-24T00:19:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.645+0000 7f01f548f700 1 Processor -- start 2024-03-24T00:19:40.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.645+0000 7f01f548f700 1 -- start start 2024-03-24T00:19:40.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.645+0000 7f01f548f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01f01012e0 0x7f01f019e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:40.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.645+0000 7f01f548f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01f0101c90 0x7f01f019eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:40.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.645+0000 7f01f548f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f01f01063a0 0x7f01f01986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:40.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.645+0000 7f01f548f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f0111380 con 0x7f01f01012e0 2024-03-24T00:19:40.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.645+0000 7f01f548f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f0111200 con 0x7f01f01063a0 2024-03-24T00:19:40.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.645+0000 7f01f548f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f0111500 con 0x7f01f0101c90 2024-03-24T00:19:40.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.645+0000 7f01eeffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01f01012e0 0x7f01f019e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:40.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.645+0000 7f01eeffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01f01012e0 0x7f01f019e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34398/0 (socket says 172.21.15.78:34398) 2024-03-24T00:19:40.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.645+0000 7f01eeffd700 1 -- 172.21.15.78:0/2061191135 learned_addr learned my addr 172.21.15.78:0/2061191135 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:19:40.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.645+0000 7f01ef7fe700 1 --2- 172.21.15.78:0/2061191135 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f01f01063a0 0x7f01f01986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:40.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.646+0000 7f01ee7fc700 1 --2- 172.21.15.78:0/2061191135 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01f0101c90 0x7f01f019eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:40.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.646+0000 7f01eeffd700 1 -- 172.21.15.78:0/2061191135 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01f0101c90 msgr2=0x7f01f019eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:40.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.646+0000 7f01eeffd700 1 --2- 172.21.15.78:0/2061191135 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01f0101c90 0x7f01f019eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:40.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.646+0000 7f01eeffd700 1 -- 172.21.15.78:0/2061191135 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f01f01063a0 msgr2=0x7f01f01986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:40.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.646+0000 7f01eeffd700 1 --2- 172.21.15.78:0/2061191135 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f01f01063a0 0x7f01f01986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:40.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.646+0000 7f01eeffd700 1 -- 172.21.15.78:0/2061191135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f0198fa0 con 0x7f01f01012e0 2024-03-24T00:19:40.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.646+0000 7f01ef7fe700 1 --2- 172.21.15.78:0/2061191135 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f01f01063a0 0x7f01f01986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:40.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.646+0000 7f01eeffd700 1 --2- 172.21.15.78:0/2061191135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01f01012e0 0x7f01f019e5f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f01e800ef10 tx=0x7f01e800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:40.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.646+0000 7f01d7fff700 1 -- 172.21.15.78:0/2061191135 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e800c9c0 con 0x7f01f01012e0 2024-03-24T00:19:40.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.646+0000 7f01d7fff700 1 -- 172.21.15.78:0/2061191135 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f01e8019070 con 0x7f01f01012e0 2024-03-24T00:19:40.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.646+0000 7f01f548f700 1 -- 172.21.15.78:0/2061191135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01f0199230 con 0x7f01f01012e0 2024-03-24T00:19:40.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.646+0000 7f01d7fff700 1 -- 172.21.15.78:0/2061191135 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e8007e60 con 0x7f01f01012e0 2024-03-24T00:19:40.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.646+0000 7f01f548f700 1 -- 172.21.15.78:0/2061191135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01f0102e60 con 0x7f01f01012e0 2024-03-24T00:19:40.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.647+0000 7f01f548f700 1 -- 172.21.15.78:0/2061191135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f01f0062920 con 0x7f01f01012e0 2024-03-24T00:19:40.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.648+0000 7f01d7fff700 1 -- 172.21.15.78:0/2061191135 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f01e8012050 con 0x7f01f01012e0 2024-03-24T00:19:40.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.649+0000 7f01d7fff700 1 --2- 172.21.15.78:0/2061191135 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f01d80a2920 0x7f01d80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:40.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.649+0000 7f01ee7fc700 1 --2- 172.21.15.78:0/2061191135 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f01d80a2920 0x7f01d80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:40.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.649+0000 7f01d7fff700 1 -- 172.21.15.78:0/2061191135 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f01e80cf9c0 con 0x7f01f01012e0 2024-03-24T00:19:40.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.649+0000 7f01d7fff700 1 --2- 172.21.15.78:0/2061191135 >> [v2:172.21.15.78:6802/1816879397,v1:172.21.15.78:6803/1816879397] conn(0x7f01d80ada70 0x7f01d80aff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:40.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.649+0000 7f01d7fff700 1 -- 172.21.15.78:0/2061191135 --> [v2:172.21.15.78:6802/1816879397,v1:172.21.15.78:6803/1816879397] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f01e800be50 con 0x7f01d80ada70 2024-03-24T00:19:40.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.649+0000 7f01d7fff700 1 -- 172.21.15.78:0/2061191135 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7f01e80cfe10 con 0x7f01f01012e0 2024-03-24T00:19:40.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.649+0000 7f01ee7fc700 1 --2- 172.21.15.78:0/2061191135 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f01d80a2920 0x7f01d80a4de0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f01f0199e80 tx=0x7f01e400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:40.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.649+0000 7f01ef7fe700 1 --2- 172.21.15.78:0/2061191135 >> [v2:172.21.15.78:6802/1816879397,v1:172.21.15.78:6803/1816879397] conn(0x7f01d80ada70 0x7f01d80aff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:40.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.650+0000 7f01ef7fe700 1 --2- 172.21.15.78:0/2061191135 >> [v2:172.21.15.78:6802/1816879397,v1:172.21.15.78:6803/1816879397] conn(0x7f01d80ada70 0x7f01d80aff10 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:40.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.650+0000 7f01d7fff700 1 -- 172.21.15.78:0/2061191135 <== osd.0 v2:172.21.15.78:6802/1816879397 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f01e800be50 con 0x7f01d80ada70 2024-03-24T00:19:40.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.668+0000 7f01f548f700 1 -- 172.21.15.78:0/2061191135 --> [v2:172.21.15.78:6802/1816879397,v1:172.21.15.78:6803/1816879397] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f01f0062da0 con 0x7f01d80ada70 2024-03-24T00:19:40.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.669+0000 7f01d7fff700 1 -- 172.21.15.78:0/2061191135 <== osd.0 v2:172.21.15.78:6802/1816879397 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f01f0062da0 con 0x7f01d80ada70 2024-03-24T00:19:40.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.669+0000 7f01f548f700 1 -- 172.21.15.78:0/2061191135 >> [v2:172.21.15.78:6802/1816879397,v1:172.21.15.78:6803/1816879397] conn(0x7f01d80ada70 msgr2=0x7f01d80aff10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:40.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.669+0000 7f01f548f700 1 --2- 172.21.15.78:0/2061191135 >> [v2:172.21.15.78:6802/1816879397,v1:172.21.15.78:6803/1816879397] conn(0x7f01d80ada70 0x7f01d80aff10 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:40.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.669+0000 7f01f548f700 1 -- 172.21.15.78:0/2061191135 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f01d80a2920 msgr2=0x7f01d80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:40.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.669+0000 7f01f548f700 1 --2- 172.21.15.78:0/2061191135 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f01d80a2920 0x7f01d80a4de0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f01f0199e80 tx=0x7f01e400b410 comp rx=0 tx=0).stop 2024-03-24T00:19:40.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.669+0000 7f01f548f700 1 -- 172.21.15.78:0/2061191135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01f01012e0 msgr2=0x7f01f019e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:40.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.669+0000 7f01f548f700 1 --2- 172.21.15.78:0/2061191135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01f01012e0 0x7f01f019e5f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f01e800ef10 tx=0x7f01e800c3a0 comp rx=0 tx=0).stop 2024-03-24T00:19:40.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.669+0000 7f01ef7fe700 1 -- 172.21.15.78:0/2061191135 reap_dead start 2024-03-24T00:19:40.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.669+0000 7f01f548f700 1 -- 172.21.15.78:0/2061191135 shutdown_connections 2024-03-24T00:19:40.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.669+0000 7f01f548f700 1 -- 172.21.15.78:0/2061191135 >> 172.21.15.78:0/2061191135 conn(0x7f01f0078000 msgr2=0x7f01f006ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:40.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.669+0000 7f01f548f700 1 -- 172.21.15.78:0/2061191135 shutdown_connections 2024-03-24T00:19:40.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:40.669+0000 7f01f548f700 1 -- 172.21.15.78:0/2061191135 wait complete. 2024-03-24T00:19:40.755 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-03-24T00:19:40.755 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-03-24T00:19:40.755 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-03-24T00:19:40.755 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]: dispatch 2024-03-24T00:19:40.755 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2024-03-24T00:19:40.755 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-03-24T00:19:40.755 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-03-24T00:19:40.755 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-03-24T00:19:40.756 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2024-03-24T00:19:40.756 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:40 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2024-03-24T00:19:40.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-03-24T00:19:40.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-03-24T00:19:40.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-03-24T00:19:40.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]: dispatch 2024-03-24T00:19:40.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2024-03-24T00:19:40.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-03-24T00:19:40.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-03-24T00:19:40.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-03-24T00:19:40.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2024-03-24T00:19:40.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:40 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2024-03-24T00:19:40.898 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:19:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-03-24T00:19:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-03-24T00:19:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-03-24T00:19:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]: dispatch 2024-03-24T00:19:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2024-03-24T00:19:40.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-03-24T00:19:40.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-03-24T00:19:40.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-03-24T00:19:40.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2024-03-24T00:19:40.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:40 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2024-03-24T00:19:41.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:41 smithi078 ceph-mon[32429]: pgmap v319: 129 pgs: 129 active+clean; 641 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-24T00:19:41.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:41 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-03-24T00:19:41.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:41 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-03-24T00:19:41.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:41 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-03-24T00:19:41.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:41 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]': finished 2024-03-24T00:19:41.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:41 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-03-24T00:19:41.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:41 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-03-24T00:19:41.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:41 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-03-24T00:19:41.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:41 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-03-24T00:19:41.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:41 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2024-03-24T00:19:41.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:41 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2024-03-24T00:19:41.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:41 smithi078 ceph-mon[32429]: osdmap e90: 12 total, 12 up, 12 in 2024-03-24T00:19:41.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:41 smithi107 ceph-mon[36485]: pgmap v319: 129 pgs: 129 active+clean; 641 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-24T00:19:41.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:41 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-03-24T00:19:41.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:41 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-03-24T00:19:41.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:41 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-03-24T00:19:41.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:41 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]': finished 2024-03-24T00:19:41.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:41 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-03-24T00:19:41.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:41 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-03-24T00:19:41.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:41 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-03-24T00:19:41.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:41 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-03-24T00:19:41.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:41 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2024-03-24T00:19:41.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:41 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2024-03-24T00:19:41.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:41 smithi107 ceph-mon[36485]: osdmap e90: 12 total, 12 up, 12 in 2024-03-24T00:19:41.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:41 smithi106 ceph-mon[36505]: pgmap v319: 129 pgs: 129 active+clean; 641 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-24T00:19:41.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:41 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-03-24T00:19:41.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:41 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-03-24T00:19:41.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:41 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-03-24T00:19:41.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:41 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]': finished 2024-03-24T00:19:41.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:41 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-03-24T00:19:41.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:41 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-03-24T00:19:41.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:41 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-03-24T00:19:41.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:41 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-03-24T00:19:41.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:41 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2024-03-24T00:19:41.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:41 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2024-03-24T00:19:41.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:41 smithi106 ceph-mon[36505]: osdmap e90: 12 total, 12 up, 12 in 2024-03-24T00:19:42.490 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:42 smithi078 ceph-mon[32429]: osdmap e91: 12 total, 12 up, 12 in 2024-03-24T00:19:42.516 INFO:teuthology.orchestra.run.smithi078.stdout:253403070504 2024-03-24T00:19:42.517 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-24T00:19:42.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:42 smithi107 ceph-mon[36485]: osdmap e91: 12 total, 12 up, 12 in 2024-03-24T00:19:42.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:42 smithi106 ceph-mon[36505]: osdmap e91: 12 total, 12 up, 12 in 2024-03-24T00:19:42.921 INFO:teuthology.orchestra.run.smithi078.stdout:34359738448 2024-03-24T00:19:42.921 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-24T00:19:43.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.620+0000 7f47ea9f5700 1 -- 172.21.15.78:0/2721546126 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47e4101cd0 msgr2=0x7f47e4105c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:43.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.620+0000 7f47ea9f5700 1 --2- 172.21.15.78:0/2721546126 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47e4101cd0 0x7f47e4105c90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f47d80099c0 tx=0x7f47d8009cd0 comp rx=0 tx=0).stop 2024-03-24T00:19:43.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.620+0000 7f47ea9f5700 1 -- 172.21.15.78:0/2721546126 shutdown_connections 2024-03-24T00:19:43.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.620+0000 7f47ea9f5700 1 --2- 172.21.15.78:0/2721546126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47e4106370 0x7f47e410df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:43.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.620+0000 7f47ea9f5700 1 --2- 172.21.15.78:0/2721546126 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47e4101320 0x7f47e4101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:43.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.620+0000 7f47ea9f5700 1 --2- 172.21.15.78:0/2721546126 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47e4101cd0 0x7f47e4105c90 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:43.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.620+0000 7f47ea9f5700 1 -- 172.21.15.78:0/2721546126 >> 172.21.15.78:0/2721546126 conn(0x7f47e40770e0 msgr2=0x7f47e40774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:43.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.620+0000 7f47ea9f5700 1 -- 172.21.15.78:0/2721546126 shutdown_connections 2024-03-24T00:19:43.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.620+0000 7f47ea9f5700 1 -- 172.21.15.78:0/2721546126 wait complete. 2024-03-24T00:19:43.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.621+0000 7f47ea9f5700 1 Processor -- start 2024-03-24T00:19:43.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.621+0000 7f47ea9f5700 1 -- start start 2024-03-24T00:19:43.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.621+0000 7f47ea9f5700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47e4101320 0x7f47e4199770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:43.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.621+0000 7f47ea9f5700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47e4101cd0 0x7f47e4199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:43.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.621+0000 7f47ea9f5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47e4106370 0x7f47e41a47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:43.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.621+0000 7f47ea9f5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e4111330 con 0x7f47e4106370 2024-03-24T00:19:43.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.621+0000 7f47ea9f5700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e41111b0 con 0x7f47e4101cd0 2024-03-24T00:19:43.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.621+0000 7f47ea9f5700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e41114b0 con 0x7f47e4101320 2024-03-24T00:19:43.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.622+0000 7f47e8f92700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47e4106370 0x7f47e41a47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:43.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.622+0000 7f47e8f92700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47e4106370 0x7f47e41a47f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34422/0 (socket says 172.21.15.78:34422) 2024-03-24T00:19:43.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.622+0000 7f47e8f92700 1 -- 172.21.15.78:0/1822829807 learned_addr learned my addr 172.21.15.78:0/1822829807 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:19:43.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.622+0000 7f47e3fff700 1 --2- 172.21.15.78:0/1822829807 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47e4101320 0x7f47e4199770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:43.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.622+0000 7f47e37fe700 1 --2- 172.21.15.78:0/1822829807 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47e4101cd0 0x7f47e4199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:43.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.622+0000 7f47e3fff700 1 -- 172.21.15.78:0/1822829807 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47e4101cd0 msgr2=0x7f47e4199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:43.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.622+0000 7f47e3fff700 1 --2- 172.21.15.78:0/1822829807 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47e4101cd0 0x7f47e4199cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:43.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.622+0000 7f47e3fff700 1 -- 172.21.15.78:0/1822829807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47e4106370 msgr2=0x7f47e41a47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:43.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.622+0000 7f47e3fff700 1 --2- 172.21.15.78:0/1822829807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47e4106370 0x7f47e41a47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:43.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.622+0000 7f47e3fff700 1 -- 172.21.15.78:0/1822829807 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e41a4d30 con 0x7f47e4101320 2024-03-24T00:19:43.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.622+0000 7f47e8f92700 1 --2- 172.21.15.78:0/1822829807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47e4106370 0x7f47e41a47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:43.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.622+0000 7f47e3fff700 1 --2- 172.21.15.78:0/1822829807 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47e4101320 0x7f47e4199770 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f47d400d950 tx=0x7f47d400dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:43.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.622+0000 7f47e17fa700 1 -- 172.21.15.78:0/1822829807 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d4016070 con 0x7f47e4101320 2024-03-24T00:19:43.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.622+0000 7f47e17fa700 1 -- 172.21.15.78:0/1822829807 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f47d400fd70 con 0x7f47e4101320 2024-03-24T00:19:43.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.622+0000 7f47ea9f5700 1 -- 172.21.15.78:0/1822829807 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47e41a4f60 con 0x7f47e4101320 2024-03-24T00:19:43.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.622+0000 7f47e17fa700 1 -- 172.21.15.78:0/1822829807 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d400e3b0 con 0x7f47e4101320 2024-03-24T00:19:43.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.622+0000 7f47ea9f5700 1 -- 172.21.15.78:0/1822829807 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47e41a5360 con 0x7f47e4101320 2024-03-24T00:19:43.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.623+0000 7f47ea9f5700 1 -- 172.21.15.78:0/1822829807 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f47e40628a0 con 0x7f47e4101320 2024-03-24T00:19:43.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.624+0000 7f47e17fa700 1 -- 172.21.15.78:0/1822829807 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f47d4019070 con 0x7f47e4101320 2024-03-24T00:19:43.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.625+0000 7f47e17fa700 1 --2- 172.21.15.78:0/1822829807 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f47cc0a2920 0x7f47cc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:43.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.625+0000 7f47e37fe700 1 --2- 172.21.15.78:0/1822829807 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f47cc0a2920 0x7f47cc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:43.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.625+0000 7f47e17fa700 1 -- 172.21.15.78:0/1822829807 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f47d40cedd0 con 0x7f47e4101320 2024-03-24T00:19:43.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.625+0000 7f47e17fa700 1 --2- 172.21.15.78:0/1822829807 >> [v2:172.21.15.107:6824/4191916230,v1:172.21.15.107:6825/4191916230] conn(0x7f47cc0ada80 0x7f47cc0aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:43.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.625+0000 7f47e17fa700 1 -- 172.21.15.78:0/1822829807 --> [v2:172.21.15.107:6824/4191916230,v1:172.21.15.107:6825/4191916230] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47d4013ec0 con 0x7f47cc0ada80 2024-03-24T00:19:43.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.625+0000 7f47e17fa700 1 -- 172.21.15.78:0/1822829807 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f47d40cf180 con 0x7f47e4101320 2024-03-24T00:19:43.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.626+0000 7f47e37fe700 1 --2- 172.21.15.78:0/1822829807 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f47cc0a2920 0x7f47cc0a4de0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f47d8000c00 tx=0x7f47d8005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:43.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.626+0000 7f47e8f92700 1 --2- 172.21.15.78:0/1822829807 >> [v2:172.21.15.107:6824/4191916230,v1:172.21.15.107:6825/4191916230] conn(0x7f47cc0ada80 0x7f47cc0aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:43.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.626+0000 7f47e8f92700 1 --2- 172.21.15.78:0/1822829807 >> [v2:172.21.15.107:6824/4191916230,v1:172.21.15.107:6825/4191916230] conn(0x7f47cc0ada80 0x7f47cc0aff20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:43.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.627+0000 7f47e17fa700 1 -- 172.21.15.78:0/1822829807 <== osd.11 v2:172.21.15.107:6824/4191916230 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f47d4013ec0 con 0x7f47cc0ada80 2024-03-24T00:19:43.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.643+0000 7f47ea9f5700 1 -- 172.21.15.78:0/1822829807 --> [v2:172.21.15.107:6824/4191916230,v1:172.21.15.107:6825/4191916230] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f47e4062d20 con 0x7f47cc0ada80 2024-03-24T00:19:43.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.644+0000 7f47e17fa700 1 -- 172.21.15.78:0/1822829807 <== osd.11 v2:172.21.15.107:6824/4191916230 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f47e4062d20 con 0x7f47cc0ada80 2024-03-24T00:19:43.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.644+0000 7f47ea9f5700 1 -- 172.21.15.78:0/1822829807 >> [v2:172.21.15.107:6824/4191916230,v1:172.21.15.107:6825/4191916230] conn(0x7f47cc0ada80 msgr2=0x7f47cc0aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:43.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.644+0000 7f47ea9f5700 1 --2- 172.21.15.78:0/1822829807 >> [v2:172.21.15.107:6824/4191916230,v1:172.21.15.107:6825/4191916230] conn(0x7f47cc0ada80 0x7f47cc0aff20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:43.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.644+0000 7f47ea9f5700 1 -- 172.21.15.78:0/1822829807 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f47cc0a2920 msgr2=0x7f47cc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:43.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.644+0000 7f47ea9f5700 1 --2- 172.21.15.78:0/1822829807 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f47cc0a2920 0x7f47cc0a4de0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f47d8000c00 tx=0x7f47d8005fb0 comp rx=0 tx=0).stop 2024-03-24T00:19:43.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.644+0000 7f47ea9f5700 1 -- 172.21.15.78:0/1822829807 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47e4101320 msgr2=0x7f47e4199770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:43.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.644+0000 7f47ea9f5700 1 --2- 172.21.15.78:0/1822829807 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47e4101320 0x7f47e4199770 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f47d400d950 tx=0x7f47d400dc60 comp rx=0 tx=0).stop 2024-03-24T00:19:43.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.644+0000 7f47e8f92700 1 -- 172.21.15.78:0/1822829807 reap_dead start 2024-03-24T00:19:43.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.644+0000 7f47ea9f5700 1 -- 172.21.15.78:0/1822829807 shutdown_connections 2024-03-24T00:19:43.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.644+0000 7f47ea9f5700 1 -- 172.21.15.78:0/1822829807 >> 172.21.15.78:0/1822829807 conn(0x7f47e40770e0 msgr2=0x7f47e40ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:43.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.644+0000 7f47ea9f5700 1 -- 172.21.15.78:0/1822829807 shutdown_connections 2024-03-24T00:19:43.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:43.644+0000 7f47ea9f5700 1 -- 172.21.15.78:0/1822829807 wait complete. 2024-03-24T00:19:43.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:43 smithi106 ceph-mon[36505]: pgmap v322: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:19:43.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:43 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:19:43.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:43 smithi078 ceph-mon[32429]: pgmap v322: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:19:43.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:43 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:19:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:43 smithi107 ceph-mon[36485]: pgmap v322: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:19:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:43 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:19:44.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.067+0000 7fdd1c487700 1 -- 172.21.15.78:0/1323467469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd141074a0 msgr2=0x7fdd1410f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:44.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.067+0000 7fdd1c487700 1 --2- 172.21.15.78:0/1323467469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd141074a0 0x7fdd1410f050 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c00b0a0 tx=0x7fdd0c00b3b0 comp rx=0 tx=0).stop 2024-03-24T00:19:44.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.067+0000 7fdd1c487700 1 -- 172.21.15.78:0/1323467469 shutdown_connections 2024-03-24T00:19:44.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.067+0000 7fdd1c487700 1 --2- 172.21.15.78:0/1323467469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd141074a0 0x7fdd1410f050 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:44.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.067+0000 7fdd1c487700 1 --2- 172.21.15.78:0/1323467469 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd1406a930 0x7fdd1406ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:44.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.067+0000 7fdd1c487700 1 --2- 172.21.15.78:0/1323467469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd1406b2e0 0x7fdd14106dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:44.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.067+0000 7fdd1c487700 1 -- 172.21.15.78:0/1323467469 >> 172.21.15.78:0/1323467469 conn(0x7fdd140770e0 msgr2=0x7fdd140774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:44.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.067+0000 7fdd1c487700 1 -- 172.21.15.78:0/1323467469 shutdown_connections 2024-03-24T00:19:44.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.067+0000 7fdd1c487700 1 -- 172.21.15.78:0/1323467469 wait complete. 2024-03-24T00:19:44.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.068+0000 7fdd1c487700 1 Processor -- start 2024-03-24T00:19:44.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.068+0000 7fdd1c487700 1 -- start start 2024-03-24T00:19:44.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.068+0000 7fdd1c487700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd1406a930 0x7fdd14101190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:44.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.068+0000 7fdd1c487700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd1406b2e0 0x7fdd141016d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:44.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.068+0000 7fdd1c487700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd141074a0 0x7fdd14101c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:44.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.068+0000 7fdd1c487700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd141124b0 con 0x7fdd1406a930 2024-03-24T00:19:44.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.068+0000 7fdd1c487700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd14112330 con 0x7fdd141074a0 2024-03-24T00:19:44.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.068+0000 7fdd1c487700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd14112630 con 0x7fdd1406b2e0 2024-03-24T00:19:44.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.068+0000 7fdd1a223700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd1406a930 0x7fdd14101190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:44.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.068+0000 7fdd1a223700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd1406a930 0x7fdd14101190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34442/0 (socket says 172.21.15.78:34442) 2024-03-24T00:19:44.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.068+0000 7fdd1a223700 1 -- 172.21.15.78:0/3902363282 learned_addr learned my addr 172.21.15.78:0/3902363282 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:19:44.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.068+0000 7fdd19a22700 1 --2- 172.21.15.78:0/3902363282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd1406b2e0 0x7fdd141016d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:44.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.068+0000 7fdd1aa24700 1 --2- 172.21.15.78:0/3902363282 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd141074a0 0x7fdd14101c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:44.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.068+0000 7fdd1a223700 1 -- 172.21.15.78:0/3902363282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd1406b2e0 msgr2=0x7fdd141016d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:44.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.068+0000 7fdd1a223700 1 --2- 172.21.15.78:0/3902363282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd1406b2e0 0x7fdd141016d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:44.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.068+0000 7fdd1a223700 1 -- 172.21.15.78:0/3902363282 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd141074a0 msgr2=0x7fdd14101c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:44.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.068+0000 7fdd1a223700 1 --2- 172.21.15.78:0/3902363282 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd141074a0 0x7fdd14101c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:44.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.068+0000 7fdd1a223700 1 -- 172.21.15.78:0/3902363282 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd141a6da0 con 0x7fdd1406a930 2024-03-24T00:19:44.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.069+0000 7fdd1a223700 1 --2- 172.21.15.78:0/3902363282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd1406a930 0x7fdd14101190 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fdd1000e9a0 tx=0x7fdd1000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:44.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.069+0000 7fdd0b7fe700 1 -- 172.21.15.78:0/3902363282 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1000ca10 con 0x7fdd1406a930 2024-03-24T00:19:44.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.069+0000 7fdd0b7fe700 1 -- 172.21.15.78:0/3902363282 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd1000cbb0 con 0x7fdd1406a930 2024-03-24T00:19:44.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.069+0000 7fdd0b7fe700 1 -- 172.21.15.78:0/3902363282 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd10018680 con 0x7fdd1406a930 2024-03-24T00:19:44.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.069+0000 7fdd1c487700 1 -- 172.21.15.78:0/3902363282 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd141a7090 con 0x7fdd1406a930 2024-03-24T00:19:44.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.069+0000 7fdd1c487700 1 -- 172.21.15.78:0/3902363282 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd141a75d0 con 0x7fdd1406a930 2024-03-24T00:19:44.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.070+0000 7fdd1c487700 1 -- 172.21.15.78:0/3902363282 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdd14062920 con 0x7fdd1406a930 2024-03-24T00:19:44.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.071+0000 7fdd0b7fe700 1 -- 172.21.15.78:0/3902363282 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fdd10010430 con 0x7fdd1406a930 2024-03-24T00:19:44.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.072+0000 7fdd0b7fe700 1 --2- 172.21.15.78:0/3902363282 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fdd000a2920 0x7fdd000a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:44.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.072+0000 7fdd19a22700 1 --2- 172.21.15.78:0/3902363282 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fdd000a2920 0x7fdd000a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:44.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.072+0000 7fdd0b7fe700 1 -- 172.21.15.78:0/3902363282 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fdd100cfce0 con 0x7fdd1406a930 2024-03-24T00:19:44.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.072+0000 7fdd0b7fe700 1 --2- 172.21.15.78:0/3902363282 >> [v2:172.21.15.106:6808/374641205,v1:172.21.15.106:6809/374641205] conn(0x7fdd000ada80 0x7fdd000aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:44.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.072+0000 7fdd0b7fe700 1 -- 172.21.15.78:0/3902363282 --> [v2:172.21.15.106:6808/374641205,v1:172.21.15.106:6809/374641205] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdd1001ae50 con 0x7fdd000ada80 2024-03-24T00:19:44.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.072+0000 7fdd0b7fe700 1 -- 172.21.15.78:0/3902363282 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd100d0130 con 0x7fdd1406a930 2024-03-24T00:19:44.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.073+0000 7fdd1aa24700 1 --2- 172.21.15.78:0/3902363282 >> [v2:172.21.15.106:6808/374641205,v1:172.21.15.106:6809/374641205] conn(0x7fdd000ada80 0x7fdd000aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:44.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.073+0000 7fdd19a22700 1 --2- 172.21.15.78:0/3902363282 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fdd000a2920 0x7fdd000a4de0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdd14108790 tx=0x7fdd04009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:44.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.073+0000 7fdd1aa24700 1 --2- 172.21.15.78:0/3902363282 >> [v2:172.21.15.106:6808/374641205,v1:172.21.15.106:6809/374641205] conn(0x7fdd000ada80 0x7fdd000aff20 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:44.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.074+0000 7fdd0b7fe700 1 -- 172.21.15.78:0/3902363282 <== osd.4 v2:172.21.15.106:6808/374641205 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdd1001ae50 con 0x7fdd000ada80 2024-03-24T00:19:44.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.091+0000 7fdd1c487700 1 -- 172.21.15.78:0/3902363282 --> [v2:172.21.15.106:6808/374641205,v1:172.21.15.106:6809/374641205] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdd14062b70 con 0x7fdd000ada80 2024-03-24T00:19:44.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.092+0000 7fdd0b7fe700 1 -- 172.21.15.78:0/3902363282 <== osd.4 v2:172.21.15.106:6808/374641205 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fdd14062b70 con 0x7fdd000ada80 2024-03-24T00:19:44.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.092+0000 7fdd1c487700 1 -- 172.21.15.78:0/3902363282 >> [v2:172.21.15.106:6808/374641205,v1:172.21.15.106:6809/374641205] conn(0x7fdd000ada80 msgr2=0x7fdd000aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:44.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.092+0000 7fdd1c487700 1 --2- 172.21.15.78:0/3902363282 >> [v2:172.21.15.106:6808/374641205,v1:172.21.15.106:6809/374641205] conn(0x7fdd000ada80 0x7fdd000aff20 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:44.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.092+0000 7fdd1c487700 1 -- 172.21.15.78:0/3902363282 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fdd000a2920 msgr2=0x7fdd000a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:44.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.092+0000 7fdd1c487700 1 --2- 172.21.15.78:0/3902363282 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fdd000a2920 0x7fdd000a4de0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdd14108790 tx=0x7fdd04009260 comp rx=0 tx=0).stop 2024-03-24T00:19:44.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.092+0000 7fdd1c487700 1 -- 172.21.15.78:0/3902363282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd1406a930 msgr2=0x7fdd14101190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:44.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.092+0000 7fdd1c487700 1 --2- 172.21.15.78:0/3902363282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd1406a930 0x7fdd14101190 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fdd1000e9a0 tx=0x7fdd1000ecb0 comp rx=0 tx=0).stop 2024-03-24T00:19:44.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.092+0000 7fdd1aa24700 1 -- 172.21.15.78:0/3902363282 reap_dead start 2024-03-24T00:19:44.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.092+0000 7fdd1c487700 1 -- 172.21.15.78:0/3902363282 shutdown_connections 2024-03-24T00:19:44.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.092+0000 7fdd1c487700 1 -- 172.21.15.78:0/3902363282 >> 172.21.15.78:0/3902363282 conn(0x7fdd140770e0 msgr2=0x7fdd140ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:44.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.092+0000 7fdd1c487700 1 -- 172.21.15.78:0/3902363282 shutdown_connections 2024-03-24T00:19:44.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:44.092+0000 7fdd1c487700 1 -- 172.21.15.78:0/3902363282 wait complete. 2024-03-24T00:19:45.565 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:45 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:19:45.565 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:45 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:19:45.565 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:45 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:19:45.565 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:45 smithi078 ceph-mon[32429]: pgmap v323: 129 pgs: 1 peering, 128 active+clean; 641 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 7.6 KiB/s, 2 keys/s, 1 objects/s recovering 2024-03-24T00:19:45.565 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:45 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:19:45.593 INFO:teuthology.orchestra.run.smithi078.stdout:77309411398 2024-03-24T00:19:45.593 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-24T00:19:45.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:45 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:19:45.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:45 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:19:45.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:45 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:19:45.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:45 smithi107 ceph-mon[36485]: pgmap v323: 129 pgs: 1 peering, 128 active+clean; 641 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 7.6 KiB/s, 2 keys/s, 1 objects/s recovering 2024-03-24T00:19:45.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:45 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:19:45.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:45 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:19:45.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:45 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:19:45.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:45 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:19:45.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:45 smithi106 ceph-mon[36505]: pgmap v323: 129 pgs: 1 peering, 128 active+clean; 641 KiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 7.6 KiB/s, 2 keys/s, 1 objects/s recovering 2024-03-24T00:19:45.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:45 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:19:46.271 INFO:teuthology.orchestra.run.smithi078.stdout:343597383706 2024-03-24T00:19:46.272 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-24T00:19:46.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.453+0000 7f8822132700 1 -- 172.21.15.78:0/2257145415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f881c101370 msgr2=0x7f881c1017f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:46.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.453+0000 7f8822132700 1 --2- 172.21.15.78:0/2257145415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f881c101370 0x7f881c1017f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f88100099c0 tx=0x7f8810009cd0 comp rx=0 tx=0).stop 2024-03-24T00:19:46.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.454+0000 7f8822132700 1 -- 172.21.15.78:0/2257145415 shutdown_connections 2024-03-24T00:19:46.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.454+0000 7f8822132700 1 --2- 172.21.15.78:0/2257145415 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f881c101d30 0x7f881c10df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:46.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.454+0000 7f8822132700 1 --2- 172.21.15.78:0/2257145415 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f881c1071c0 0x7f881c1075a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:46.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.454+0000 7f8822132700 1 --2- 172.21.15.78:0/2257145415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f881c101370 0x7f881c1017f0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:46.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.454+0000 7f8822132700 1 -- 172.21.15.78:0/2257145415 >> 172.21.15.78:0/2257145415 conn(0x7f881c077f50 msgr2=0x7f881c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:46.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.454+0000 7f8822132700 1 -- 172.21.15.78:0/2257145415 shutdown_connections 2024-03-24T00:19:46.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.454+0000 7f8822132700 1 -- 172.21.15.78:0/2257145415 wait complete. 2024-03-24T00:19:46.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.454+0000 7f8822132700 1 Processor -- start 2024-03-24T00:19:46.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.454+0000 7f8822132700 1 -- start start 2024-03-24T00:19:46.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.455+0000 7f8822132700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f881c101370 0x7f881c19e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:46.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.455+0000 7f8822132700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f881c101d30 0x7f881c19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:46.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.455+0000 7f8822132700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f881c1071c0 0x7f881c198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:46.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.455+0000 7f8822132700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f881c1113c0 con 0x7f881c1071c0 2024-03-24T00:19:46.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.455+0000 7f8822132700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f881c111240 con 0x7f881c101370 2024-03-24T00:19:46.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.455+0000 7f8822132700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f881c111540 con 0x7f881c101d30 2024-03-24T00:19:46.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.455+0000 7f881bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f881c1071c0 0x7f881c198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:46.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.455+0000 7f881bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f881c1071c0 0x7f881c198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57042/0 (socket says 172.21.15.78:57042) 2024-03-24T00:19:46.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.455+0000 7f881bfff700 1 -- 172.21.15.78:0/446756388 learned_addr learned my addr 172.21.15.78:0/446756388 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:19:46.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.455+0000 7f881b7fe700 1 --2- 172.21.15.78:0/446756388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f881c101370 0x7f881c19e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:46.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.455+0000 7f881affd700 1 --2- 172.21.15.78:0/446756388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f881c101d30 0x7f881c19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:46.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.455+0000 7f881affd700 1 -- 172.21.15.78:0/446756388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f881c101370 msgr2=0x7f881c19e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:46.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.455+0000 7f881affd700 1 --2- 172.21.15.78:0/446756388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f881c101370 0x7f881c19e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:46.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.455+0000 7f881affd700 1 -- 172.21.15.78:0/446756388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f881c1071c0 msgr2=0x7f881c198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:46.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.455+0000 7f881affd700 1 --2- 172.21.15.78:0/446756388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f881c1071c0 0x7f881c198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:46.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.455+0000 7f881affd700 1 -- 172.21.15.78:0/446756388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f881c199000 con 0x7f881c101d30 2024-03-24T00:19:46.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.455+0000 7f881bfff700 1 --2- 172.21.15.78:0/446756388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f881c1071c0 0x7f881c198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:46.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.455+0000 7f881b7fe700 1 --2- 172.21.15.78:0/446756388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f881c101370 0x7f881c19e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:19:46.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.455+0000 7f881affd700 1 --2- 172.21.15.78:0/446756388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f881c101d30 0x7f881c19eb90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f88100098e0 tx=0x7f8810005f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:46.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.455+0000 7f8818ff9700 1 -- 172.21.15.78:0/446756388 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8810027070 con 0x7f881c101d30 2024-03-24T00:19:46.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.456+0000 7f8818ff9700 1 -- 172.21.15.78:0/446756388 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8810003680 con 0x7f881c101d30 2024-03-24T00:19:46.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.456+0000 7f8822132700 1 -- 172.21.15.78:0/446756388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f881c199290 con 0x7f881c101d30 2024-03-24T00:19:46.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.456+0000 7f8818ff9700 1 -- 172.21.15.78:0/446756388 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88100173f0 con 0x7f881c101d30 2024-03-24T00:19:46.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.456+0000 7f8822132700 1 -- 172.21.15.78:0/446756388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f881c1a55d0 con 0x7f881c101d30 2024-03-24T00:19:46.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.457+0000 7f8822132700 1 -- 172.21.15.78:0/446756388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f881c106140 con 0x7f881c101d30 2024-03-24T00:19:46.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.458+0000 7f8818ff9700 1 -- 172.21.15.78:0/446756388 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f881002a070 con 0x7f881c101d30 2024-03-24T00:19:46.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.458+0000 7f8818ff9700 1 --2- 172.21.15.78:0/446756388 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f88040a29a0 0x7f88040a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:46.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.458+0000 7f8818ff9700 1 -- 172.21.15.78:0/446756388 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f88100cfd20 con 0x7f881c101d30 2024-03-24T00:19:46.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.458+0000 7f881b7fe700 1 --2- 172.21.15.78:0/446756388 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f88040a29a0 0x7f88040a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:46.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.459+0000 7f8818ff9700 1 --2- 172.21.15.78:0/446756388 >> [v2:172.21.15.78:6810/1912251346,v1:172.21.15.78:6811/1912251346] conn(0x7f88040adb00 0x7f88040affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:46.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.459+0000 7f8818ff9700 1 -- 172.21.15.78:0/446756388 --> [v2:172.21.15.78:6810/1912251346,v1:172.21.15.78:6811/1912251346] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8810001d30 con 0x7f88040adb00 2024-03-24T00:19:46.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.459+0000 7f8818ff9700 1 -- 172.21.15.78:0/446756388 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f88100d0160 con 0x7f881c101d30 2024-03-24T00:19:46.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.459+0000 7f881bfff700 1 --2- 172.21.15.78:0/446756388 >> [v2:172.21.15.78:6810/1912251346,v1:172.21.15.78:6811/1912251346] conn(0x7f88040adb00 0x7f88040affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:46.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.459+0000 7f881b7fe700 1 --2- 172.21.15.78:0/446756388 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f88040a29a0 0x7f88040a4e60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f880c009be0 tx=0x7f880c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:46.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.459+0000 7f881bfff700 1 --2- 172.21.15.78:0/446756388 >> [v2:172.21.15.78:6810/1912251346,v1:172.21.15.78:6811/1912251346] conn(0x7f88040adb00 0x7f88040affa0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:46.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.460+0000 7f8818ff9700 1 -- 172.21.15.78:0/446756388 <== osd.3 v2:172.21.15.78:6810/1912251346 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8810001d30 con 0x7f88040adb00 2024-03-24T00:19:46.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.476+0000 7f8822132700 1 -- 172.21.15.78:0/446756388 --> [v2:172.21.15.78:6810/1912251346,v1:172.21.15.78:6811/1912251346] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f881c1027c0 con 0x7f88040adb00 2024-03-24T00:19:46.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.476+0000 7f8818ff9700 1 -- 172.21.15.78:0/446756388 <== osd.3 v2:172.21.15.78:6810/1912251346 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f881c1027c0 con 0x7f88040adb00 2024-03-24T00:19:46.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.476+0000 7f8822132700 1 -- 172.21.15.78:0/446756388 >> [v2:172.21.15.78:6810/1912251346,v1:172.21.15.78:6811/1912251346] conn(0x7f88040adb00 msgr2=0x7f88040affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:46.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.476+0000 7f8822132700 1 --2- 172.21.15.78:0/446756388 >> [v2:172.21.15.78:6810/1912251346,v1:172.21.15.78:6811/1912251346] conn(0x7f88040adb00 0x7f88040affa0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:46.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.476+0000 7f8822132700 1 -- 172.21.15.78:0/446756388 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f88040a29a0 msgr2=0x7f88040a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:46.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.476+0000 7f8822132700 1 --2- 172.21.15.78:0/446756388 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f88040a29a0 0x7f88040a4e60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f880c009be0 tx=0x7f880c009260 comp rx=0 tx=0).stop 2024-03-24T00:19:46.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.476+0000 7f8822132700 1 -- 172.21.15.78:0/446756388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f881c101d30 msgr2=0x7f881c19eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:46.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.476+0000 7f8822132700 1 --2- 172.21.15.78:0/446756388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f881c101d30 0x7f881c19eb90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f88100098e0 tx=0x7f8810005f10 comp rx=0 tx=0).stop 2024-03-24T00:19:46.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.476+0000 7f881bfff700 1 -- 172.21.15.78:0/446756388 reap_dead start 2024-03-24T00:19:46.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.476+0000 7f8822132700 1 -- 172.21.15.78:0/446756388 shutdown_connections 2024-03-24T00:19:46.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.476+0000 7f8822132700 1 -- 172.21.15.78:0/446756388 >> 172.21.15.78:0/446756388 conn(0x7f881c077f50 msgr2=0x7f881c06aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:46.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.477+0000 7f8822132700 1 -- 172.21.15.78:0/446756388 shutdown_connections 2024-03-24T00:19:46.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:46.477+0000 7f8822132700 1 -- 172.21.15.78:0/446756388 wait complete. 2024-03-24T00:19:47.431 INFO:teuthology.orchestra.run.smithi078.stdout:107374182466 2024-03-24T00:19:47.431 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-24T00:19:47.456 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:47 smithi078 ceph-mon[32429]: pgmap v324: 129 pgs: 1 peering, 128 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s; 7.6 KiB/s, 2 keys/s, 1 objects/s recovering 2024-03-24T00:19:47.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.610+0000 7f12183ea700 1 -- 172.21.15.78:0/3651736852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12101081e0 msgr2=0x7f12101085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:47.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.610+0000 7f12183ea700 1 --2- 172.21.15.78:0/3651736852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12101081e0 0x7f12101085c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f120c0099c0 tx=0x7f120c009cd0 comp rx=0 tx=0).stop 2024-03-24T00:19:47.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.610+0000 7f12183ea700 1 -- 172.21.15.78:0/3651736852 shutdown_connections 2024-03-24T00:19:47.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.610+0000 7f12183ea700 1 --2- 172.21.15.78:0/3651736852 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f121006a7e0 0x7f121010f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:47.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.610+0000 7f12183ea700 1 --2- 172.21.15.78:0/3651736852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12101081e0 0x7f12101085c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:47.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.610+0000 7f12183ea700 1 --2- 172.21.15.78:0/3651736852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1210069e40 0x7f121006a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:47.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.610+0000 7f12183ea700 1 -- 172.21.15.78:0/3651736852 >> 172.21.15.78:0/3651736852 conn(0x7f1210077fc0 msgr2=0x7f12100783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:47.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.610+0000 7f12183ea700 1 -- 172.21.15.78:0/3651736852 shutdown_connections 2024-03-24T00:19:47.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.610+0000 7f12183ea700 1 -- 172.21.15.78:0/3651736852 wait complete. 2024-03-24T00:19:47.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.611+0000 7f12183ea700 1 Processor -- start 2024-03-24T00:19:47.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.611+0000 7f12183ea700 1 -- start start 2024-03-24T00:19:47.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.611+0000 7f12183ea700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1210069e40 0x7f12101a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:47.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.611+0000 7f12183ea700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f121006a7e0 0x7f12101a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:47.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.611+0000 7f12183ea700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12101081e0 0x7f121019a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:47.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.611+0000 7f12183ea700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1210112490 con 0x7f1210069e40 2024-03-24T00:19:47.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.611+0000 7f12183ea700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1210112310 con 0x7f121006a7e0 2024-03-24T00:19:47.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.611+0000 7f12183ea700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1210112610 con 0x7f12101081e0 2024-03-24T00:19:47.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.611+0000 7f1216186700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1210069e40 0x7f12101a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:47.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.611+0000 7f1216186700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1210069e40 0x7f12101a0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57068/0 (socket says 172.21.15.78:57068) 2024-03-24T00:19:47.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.611+0000 7f1216186700 1 -- 172.21.15.78:0/1432115603 learned_addr learned my addr 172.21.15.78:0/1432115603 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:19:47.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.611+0000 7f1216987700 1 --2- 172.21.15.78:0/1432115603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12101081e0 0x7f121019a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:47.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.611+0000 7f1215985700 1 --2- 172.21.15.78:0/1432115603 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f121006a7e0 0x7f12101a0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:47.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.611+0000 7f1216186700 1 -- 172.21.15.78:0/1432115603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12101081e0 msgr2=0x7f121019a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:47.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.611+0000 7f1216186700 1 --2- 172.21.15.78:0/1432115603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12101081e0 0x7f121019a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:47.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.611+0000 7f1216186700 1 -- 172.21.15.78:0/1432115603 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f121006a7e0 msgr2=0x7f12101a0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:47.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.611+0000 7f1216186700 1 --2- 172.21.15.78:0/1432115603 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f121006a7e0 0x7f12101a0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:47.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.611+0000 7f1216186700 1 -- 172.21.15.78:0/1432115603 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f121019b0b0 con 0x7f1210069e40 2024-03-24T00:19:47.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.612+0000 7f1216186700 1 --2- 172.21.15.78:0/1432115603 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1210069e40 0x7f12101a0790 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f120c000c00 tx=0x7f120c00fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:47.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.612+0000 7f1216987700 1 --2- 172.21.15.78:0/1432115603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12101081e0 0x7f121019a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:47.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.612+0000 7f12077fe700 1 -- 172.21.15.78:0/1432115603 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f120c027070 con 0x7f1210069e40 2024-03-24T00:19:47.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.612+0000 7f12077fe700 1 -- 172.21.15.78:0/1432115603 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f120c00be80 con 0x7f1210069e40 2024-03-24T00:19:47.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.612+0000 7f12183ea700 1 -- 172.21.15.78:0/1432115603 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f121019b340 con 0x7f1210069e40 2024-03-24T00:19:47.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.612+0000 7f12077fe700 1 -- 172.21.15.78:0/1432115603 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f120c0173f0 con 0x7f1210069e40 2024-03-24T00:19:47.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.612+0000 7f12183ea700 1 -- 172.21.15.78:0/1432115603 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12101a7660 con 0x7f1210069e40 2024-03-24T00:19:47.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.613+0000 7f12183ea700 1 -- 172.21.15.78:0/1432115603 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f121010ddd0 con 0x7f1210069e40 2024-03-24T00:19:47.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.614+0000 7f12077fe700 1 -- 172.21.15.78:0/1432115603 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f120c02a070 con 0x7f1210069e40 2024-03-24T00:19:47.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.615+0000 7f12077fe700 1 --2- 172.21.15.78:0/1432115603 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f11fc0a2920 0x7f11fc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:47.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.615+0000 7f1215985700 1 --2- 172.21.15.78:0/1432115603 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f11fc0a2920 0x7f11fc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:47.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.615+0000 7f12077fe700 1 -- 172.21.15.78:0/1432115603 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f120c0d0150 con 0x7f1210069e40 2024-03-24T00:19:47.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.615+0000 7f12077fe700 1 --2- 172.21.15.78:0/1432115603 >> [v2:172.21.15.107:6808/3272806073,v1:172.21.15.107:6809/3272806073] conn(0x7f11fc0ada00 0x7f11fc0afea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:47.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.615+0000 7f12077fe700 1 -- 172.21.15.78:0/1432115603 --> [v2:172.21.15.107:6808/3272806073,v1:172.21.15.107:6809/3272806073] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f120c018e50 con 0x7f11fc0ada00 2024-03-24T00:19:47.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.615+0000 7f12077fe700 1 -- 172.21.15.78:0/1432115603 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f120c0d05a0 con 0x7f1210069e40 2024-03-24T00:19:47.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.615+0000 7f1216987700 1 --2- 172.21.15.78:0/1432115603 >> [v2:172.21.15.107:6808/3272806073,v1:172.21.15.107:6809/3272806073] conn(0x7f11fc0ada00 0x7f11fc0afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:47.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.615+0000 7f1215985700 1 --2- 172.21.15.78:0/1432115603 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f11fc0a2920 0x7f11fc0a4de0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f12101030a0 tx=0x7f1200009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:47.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.616+0000 7f1216987700 1 --2- 172.21.15.78:0/1432115603 >> [v2:172.21.15.107:6808/3272806073,v1:172.21.15.107:6809/3272806073] conn(0x7f11fc0ada00 0x7f11fc0afea0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:47.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.616+0000 7f12077fe700 1 -- 172.21.15.78:0/1432115603 <== osd.5 v2:172.21.15.107:6808/3272806073 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f120c018e50 con 0x7f11fc0ada00 2024-03-24T00:19:47.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.645+0000 7f12183ea700 1 -- 172.21.15.78:0/1432115603 --> [v2:172.21.15.107:6808/3272806073,v1:172.21.15.107:6809/3272806073] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1210103890 con 0x7f11fc0ada00 2024-03-24T00:19:47.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.645+0000 7f12077fe700 1 -- 172.21.15.78:0/1432115603 <== osd.5 v2:172.21.15.107:6808/3272806073 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f1210103890 con 0x7f11fc0ada00 2024-03-24T00:19:47.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.645+0000 7f12183ea700 1 -- 172.21.15.78:0/1432115603 >> [v2:172.21.15.107:6808/3272806073,v1:172.21.15.107:6809/3272806073] conn(0x7f11fc0ada00 msgr2=0x7f11fc0afea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:47.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.645+0000 7f12183ea700 1 --2- 172.21.15.78:0/1432115603 >> [v2:172.21.15.107:6808/3272806073,v1:172.21.15.107:6809/3272806073] conn(0x7f11fc0ada00 0x7f11fc0afea0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:47.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.646+0000 7f12183ea700 1 -- 172.21.15.78:0/1432115603 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f11fc0a2920 msgr2=0x7f11fc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:47.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.646+0000 7f12183ea700 1 --2- 172.21.15.78:0/1432115603 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f11fc0a2920 0x7f11fc0a4de0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f12101030a0 tx=0x7f1200009260 comp rx=0 tx=0).stop 2024-03-24T00:19:47.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.646+0000 7f12183ea700 1 -- 172.21.15.78:0/1432115603 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1210069e40 msgr2=0x7f12101a0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:47.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.646+0000 7f12183ea700 1 --2- 172.21.15.78:0/1432115603 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1210069e40 0x7f12101a0790 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f120c000c00 tx=0x7f120c00fa00 comp rx=0 tx=0).stop 2024-03-24T00:19:47.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.646+0000 7f1216987700 1 -- 172.21.15.78:0/1432115603 reap_dead start 2024-03-24T00:19:47.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.646+0000 7f12183ea700 1 -- 172.21.15.78:0/1432115603 shutdown_connections 2024-03-24T00:19:47.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.646+0000 7f12183ea700 1 -- 172.21.15.78:0/1432115603 >> 172.21.15.78:0/1432115603 conn(0x7f1210077fc0 msgr2=0x7f121010d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:47.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.646+0000 7f12183ea700 1 -- 172.21.15.78:0/1432115603 shutdown_connections 2024-03-24T00:19:47.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:47.646+0000 7f12183ea700 1 -- 172.21.15.78:0/1432115603 wait complete. 2024-03-24T00:19:47.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:47 smithi106 ceph-mon[36505]: pgmap v324: 129 pgs: 1 peering, 128 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s; 7.6 KiB/s, 2 keys/s, 1 objects/s recovering 2024-03-24T00:19:47.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:47 smithi107 ceph-mon[36485]: pgmap v324: 129 pgs: 1 peering, 128 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s; 7.6 KiB/s, 2 keys/s, 1 objects/s recovering 2024-03-24T00:19:48.094 INFO:teuthology.orchestra.run.smithi078.stdout:141733920830 2024-03-24T00:19:48.095 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-24T00:19:48.972 INFO:teuthology.orchestra.run.smithi078.stdout:163208757305 2024-03-24T00:19:48.972 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-24T00:19:49.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.393+0000 7f8b162aa700 1 -- 172.21.15.78:0/1573708590 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b101081c0 msgr2=0x7f8b101085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:49.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.393+0000 7f8b162aa700 1 --2- 172.21.15.78:0/1573708590 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b101081c0 0x7f8b101085a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8af80099c0 tx=0x7f8af8009cd0 comp rx=0 tx=0).stop 2024-03-24T00:19:49.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.393+0000 7f8b162aa700 1 -- 172.21.15.78:0/1573708590 shutdown_connections 2024-03-24T00:19:49.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.393+0000 7f8b162aa700 1 --2- 172.21.15.78:0/1573708590 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b10102ba0 0x7f8b1010efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:49.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.393+0000 7f8b162aa700 1 --2- 172.21.15.78:0/1573708590 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b101081c0 0x7f8b101085a0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:49.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.393+0000 7f8b162aa700 1 --2- 172.21.15.78:0/1573708590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b101021e0 0x7f8b10102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:49.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.393+0000 7f8b162aa700 1 -- 172.21.15.78:0/1573708590 >> 172.21.15.78:0/1573708590 conn(0x7f8b100792c0 msgr2=0x7f8b100796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:49.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.393+0000 7f8b162aa700 1 -- 172.21.15.78:0/1573708590 shutdown_connections 2024-03-24T00:19:49.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.393+0000 7f8b162aa700 1 -- 172.21.15.78:0/1573708590 wait complete. 2024-03-24T00:19:49.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.394+0000 7f8b162aa700 1 Processor -- start 2024-03-24T00:19:49.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.394+0000 7f8b162aa700 1 -- start start 2024-03-24T00:19:49.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.394+0000 7f8b162aa700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b101021e0 0x7f8b1019e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:49.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.394+0000 7f8b162aa700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b10102ba0 0x7f8b1019ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:49.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.394+0000 7f8b162aa700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b101081c0 0x7f8b10198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:49.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.394+0000 7f8b162aa700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b10112470 con 0x7f8b101081c0 2024-03-24T00:19:49.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.394+0000 7f8b162aa700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b101122f0 con 0x7f8b101021e0 2024-03-24T00:19:49.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.394+0000 7f8b162aa700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b101125f0 con 0x7f8b10102ba0 2024-03-24T00:19:49.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.394+0000 7f8b0ffff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b101021e0 0x7f8b1019e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:49.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.394+0000 7f8b0f7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b10102ba0 0x7f8b1019ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:49.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.394+0000 7f8b14847700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b101081c0 0x7f8b10198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:49.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.394+0000 7f8b0ffff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b101021e0 0x7f8b1019e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.78:55624/0 (socket says 172.21.15.78:55624) 2024-03-24T00:19:49.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.394+0000 7f8b0f7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b10102ba0 0x7f8b1019ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.78:52492/0 (socket says 172.21.15.78:52492) 2024-03-24T00:19:49.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.394+0000 7f8b0ffff700 1 -- 172.21.15.78:0/4125800309 learned_addr learned my addr 172.21.15.78:0/4125800309 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:19:49.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.394+0000 7f8b14847700 1 -- 172.21.15.78:0/4125800309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b10102ba0 msgr2=0x7f8b1019ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:49.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.394+0000 7f8b14847700 1 --2- 172.21.15.78:0/4125800309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b10102ba0 0x7f8b1019ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:49.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.394+0000 7f8b14847700 1 -- 172.21.15.78:0/4125800309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b101021e0 msgr2=0x7f8b1019e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:49.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.394+0000 7f8b14847700 1 --2- 172.21.15.78:0/4125800309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b101021e0 0x7f8b1019e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:49.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.394+0000 7f8b14847700 1 -- 172.21.15.78:0/4125800309 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b10199030 con 0x7f8b101081c0 2024-03-24T00:19:49.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.394+0000 7f8b0f7fe700 1 --2- 172.21.15.78:0/4125800309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b10102ba0 0x7f8b1019ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:49.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.394+0000 7f8b0ffff700 1 --2- 172.21.15.78:0/4125800309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b101021e0 0x7f8b1019e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:49.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.395+0000 7f8b14847700 1 --2- 172.21.15.78:0/4125800309 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b101081c0 0x7f8b10198750 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8b0800efc0 tx=0x7f8b0800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:49.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.395+0000 7f8b0d7fa700 1 -- 172.21.15.78:0/4125800309 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b080092e0 con 0x7f8b101081c0 2024-03-24T00:19:49.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.395+0000 7f8b162aa700 1 -- 172.21.15.78:0/4125800309 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b101992c0 con 0x7f8b101081c0 2024-03-24T00:19:49.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.395+0000 7f8b0d7fa700 1 -- 172.21.15.78:0/4125800309 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8b08004bb0 con 0x7f8b101081c0 2024-03-24T00:19:49.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.395+0000 7f8b0d7fa700 1 -- 172.21.15.78:0/4125800309 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b08004280 con 0x7f8b101081c0 2024-03-24T00:19:49.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.395+0000 7f8b162aa700 1 -- 172.21.15.78:0/4125800309 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b101a5600 con 0x7f8b101081c0 2024-03-24T00:19:49.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.396+0000 7f8b162aa700 1 -- 172.21.15.78:0/4125800309 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8b100628a0 con 0x7f8b101081c0 2024-03-24T00:19:49.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.397+0000 7f8b0d7fa700 1 -- 172.21.15.78:0/4125800309 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f8b0801b030 con 0x7f8b101081c0 2024-03-24T00:19:49.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.397+0000 7f8b0d7fa700 1 --2- 172.21.15.78:0/4125800309 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f8afc0a2920 0x7f8afc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:49.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.397+0000 7f8b0ffff700 1 --2- 172.21.15.78:0/4125800309 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f8afc0a2920 0x7f8afc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:49.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.398+0000 7f8b0d7fa700 1 -- 172.21.15.78:0/4125800309 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f8b080cf390 con 0x7f8b101081c0 2024-03-24T00:19:49.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.398+0000 7f8b0d7fa700 1 --2- 172.21.15.78:0/4125800309 >> [v2:172.21.15.106:6816/4262103080,v1:172.21.15.106:6817/4262103080] conn(0x7f8afc0ada80 0x7f8afc0aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:49.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.398+0000 7f8b0d7fa700 1 -- 172.21.15.78:0/4125800309 --> [v2:172.21.15.106:6816/4262103080,v1:172.21.15.106:6817/4262103080] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8b0801fe50 con 0x7f8afc0ada80 2024-03-24T00:19:49.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.398+0000 7f8b0d7fa700 1 -- 172.21.15.78:0/4125800309 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b080cf7e0 con 0x7f8b101081c0 2024-03-24T00:19:49.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.398+0000 7f8b0ffff700 1 --2- 172.21.15.78:0/4125800309 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f8afc0a2920 0x7f8afc0a4de0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8af8009990 tx=0x7f8af8005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:49.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.398+0000 7f8b0f7fe700 1 --2- 172.21.15.78:0/4125800309 >> [v2:172.21.15.106:6816/4262103080,v1:172.21.15.106:6817/4262103080] conn(0x7f8afc0ada80 0x7f8afc0aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:49.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.398+0000 7f8b0f7fe700 1 --2- 172.21.15.78:0/4125800309 >> [v2:172.21.15.106:6816/4262103080,v1:172.21.15.106:6817/4262103080] conn(0x7f8afc0ada80 0x7f8afc0aff20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:49.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.399+0000 7f8b0d7fa700 1 -- 172.21.15.78:0/4125800309 <== osd.7 v2:172.21.15.106:6816/4262103080 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8b0801fe50 con 0x7f8afc0ada80 2024-03-24T00:19:49.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.415+0000 7f8b162aa700 1 -- 172.21.15.78:0/4125800309 --> [v2:172.21.15.106:6816/4262103080,v1:172.21.15.106:6817/4262103080] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8b10062d20 con 0x7f8afc0ada80 2024-03-24T00:19:49.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.415+0000 7f8b0d7fa700 1 -- 172.21.15.78:0/4125800309 <== osd.7 v2:172.21.15.106:6816/4262103080 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f8b10062d20 con 0x7f8afc0ada80 2024-03-24T00:19:49.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.416+0000 7f8b162aa700 1 -- 172.21.15.78:0/4125800309 >> [v2:172.21.15.106:6816/4262103080,v1:172.21.15.106:6817/4262103080] conn(0x7f8afc0ada80 msgr2=0x7f8afc0aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:49.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.416+0000 7f8b162aa700 1 --2- 172.21.15.78:0/4125800309 >> [v2:172.21.15.106:6816/4262103080,v1:172.21.15.106:6817/4262103080] conn(0x7f8afc0ada80 0x7f8afc0aff20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:49.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.416+0000 7f8b162aa700 1 -- 172.21.15.78:0/4125800309 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f8afc0a2920 msgr2=0x7f8afc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:49.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.416+0000 7f8b162aa700 1 --2- 172.21.15.78:0/4125800309 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f8afc0a2920 0x7f8afc0a4de0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8af8009990 tx=0x7f8af8005960 comp rx=0 tx=0).stop 2024-03-24T00:19:49.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.416+0000 7f8b162aa700 1 -- 172.21.15.78:0/4125800309 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b101081c0 msgr2=0x7f8b10198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:49.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.416+0000 7f8b162aa700 1 --2- 172.21.15.78:0/4125800309 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b101081c0 0x7f8b10198750 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8b0800efc0 tx=0x7f8b0800c3a0 comp rx=0 tx=0).stop 2024-03-24T00:19:49.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.416+0000 7f8b14847700 1 -- 172.21.15.78:0/4125800309 reap_dead start 2024-03-24T00:19:49.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.416+0000 7f8b162aa700 1 -- 172.21.15.78:0/4125800309 shutdown_connections 2024-03-24T00:19:49.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.416+0000 7f8b162aa700 1 -- 172.21.15.78:0/4125800309 >> 172.21.15.78:0/4125800309 conn(0x7f8b100792c0 msgr2=0x7f8b10100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:49.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.416+0000 7f8b162aa700 1 -- 172.21.15.78:0/4125800309 shutdown_connections 2024-03-24T00:19:49.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:48.416+0000 7f8b162aa700 1 -- 172.21.15.78:0/4125800309 wait complete. 2024-03-24T00:19:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:49 smithi106 ceph-mon[36505]: pgmap v325: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s; 7.6 KiB/s, 2 keys/s, 1 objects/s recovering 2024-03-24T00:19:49.702 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:49 smithi078 ceph-mon[32429]: pgmap v325: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s; 7.6 KiB/s, 2 keys/s, 1 objects/s recovering 2024-03-24T00:19:49.728 INFO:teuthology.orchestra.run.smithi078.stdout:219043332144 2024-03-24T00:19:49.728 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-24T00:19:49.762 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:19:49.769 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:19:49.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:49 smithi107 ceph-mon[36485]: pgmap v325: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s; 7.6 KiB/s, 2 keys/s, 1 objects/s recovering 2024-03-24T00:19:51.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.265+0000 7f0c52c40700 1 -- 172.21.15.78:0/771173959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c4c0761a0 msgr2=0x7f0c4c0743c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:51.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.265+0000 7f0c52c40700 1 --2- 172.21.15.78:0/771173959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c4c0761a0 0x7f0c4c0743c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f0c40009990 tx=0x7f0c40009ca0 comp rx=0 tx=0).stop 2024-03-24T00:19:51.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.265+0000 7f0c52c40700 1 -- 172.21.15.78:0/771173959 shutdown_connections 2024-03-24T00:19:51.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.265+0000 7f0c52c40700 1 --2- 172.21.15.78:0/771173959 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c4c112e50 0x7f0c4c115240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:51.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.265+0000 7f0c52c40700 1 --2- 172.21.15.78:0/771173959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c4c0761a0 0x7f0c4c0743c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:51.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.265+0000 7f0c52c40700 1 --2- 172.21.15.78:0/771173959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c4c074900 0x7f0c4c074d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:51.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.265+0000 7f0c52c40700 1 -- 172.21.15.78:0/771173959 >> 172.21.15.78:0/771173959 conn(0x7f0c4c079250 msgr2=0x7f0c4c079660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:51.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.265+0000 7f0c52c40700 1 -- 172.21.15.78:0/771173959 shutdown_connections 2024-03-24T00:19:51.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.265+0000 7f0c52c40700 1 -- 172.21.15.78:0/771173959 wait complete. 2024-03-24T00:19:51.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.266+0000 7f0c52c40700 1 Processor -- start 2024-03-24T00:19:51.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.266+0000 7f0c52c40700 1 -- start start 2024-03-24T00:19:51.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.266+0000 7f0c52c40700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c4c074900 0x7f0c4c19eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:51.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.266+0000 7f0c52c40700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c4c0761a0 0x7f0c4c19f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:51.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.266+0000 7f0c52c40700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c4c112e50 0x7f0c4c1a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:51.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.266+0000 7f0c52c40700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4c118a50 con 0x7f0c4c0761a0 2024-03-24T00:19:51.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.266+0000 7f0c52c40700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4c1188d0 con 0x7f0c4c074900 2024-03-24T00:19:51.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.266+0000 7f0c52c40700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4c118bd0 con 0x7f0c4c112e50 2024-03-24T00:19:51.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.266+0000 7f0c4bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c4c0761a0 0x7f0c4c19f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:51.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.266+0000 7f0c509dc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c4c074900 0x7f0c4c19eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:51.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.266+0000 7f0c4bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c4c0761a0 0x7f0c4c19f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57090/0 (socket says 172.21.15.78:57090) 2024-03-24T00:19:51.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.266+0000 7f0c4bfff700 1 -- 172.21.15.78:0/6461830 learned_addr learned my addr 172.21.15.78:0/6461830 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:19:51.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.266+0000 7f0c511dd700 1 --2- 172.21.15.78:0/6461830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c4c112e50 0x7f0c4c1a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:51.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.267+0000 7f0c4bfff700 1 -- 172.21.15.78:0/6461830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c4c112e50 msgr2=0x7f0c4c1a3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:51.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.267+0000 7f0c4bfff700 1 --2- 172.21.15.78:0/6461830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c4c112e50 0x7f0c4c1a3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:51.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.267+0000 7f0c4bfff700 1 -- 172.21.15.78:0/6461830 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c4c074900 msgr2=0x7f0c4c19eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:51.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.267+0000 7f0c4bfff700 1 --2- 172.21.15.78:0/6461830 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c4c074900 0x7f0c4c19eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:51.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.267+0000 7f0c4bfff700 1 -- 172.21.15.78:0/6461830 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c4c1a3eb0 con 0x7f0c4c0761a0 2024-03-24T00:19:51.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.267+0000 7f0c4bfff700 1 --2- 172.21.15.78:0/6461830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c4c0761a0 0x7f0c4c19f3e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0c3800e9a0 tx=0x7f0c3800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:51.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.267+0000 7f0c49ffb700 1 -- 172.21.15.78:0/6461830 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3800c9c0 con 0x7f0c4c0761a0 2024-03-24T00:19:51.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.267+0000 7f0c49ffb700 1 -- 172.21.15.78:0/6461830 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c3800cb60 con 0x7f0c4c0761a0 2024-03-24T00:19:51.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.267+0000 7f0c52c40700 1 -- 172.21.15.78:0/6461830 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c4c1a41a0 con 0x7f0c4c0761a0 2024-03-24T00:19:51.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.267+0000 7f0c49ffb700 1 -- 172.21.15.78:0/6461830 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c38018660 con 0x7f0c4c0761a0 2024-03-24T00:19:51.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.267+0000 7f0c52c40700 1 -- 172.21.15.78:0/6461830 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c4c1aba20 con 0x7f0c4c0761a0 2024-03-24T00:19:51.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.268+0000 7f0c52c40700 1 -- 172.21.15.78:0/6461830 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0c4c1125d0 con 0x7f0c4c0761a0 2024-03-24T00:19:51.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.269+0000 7f0c49ffb700 1 -- 172.21.15.78:0/6461830 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f0c3800cd10 con 0x7f0c4c0761a0 2024-03-24T00:19:51.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.270+0000 7f0c49ffb700 1 --2- 172.21.15.78:0/6461830 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f0c3c0a29a0 0x7f0c3c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:51.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.270+0000 7f0c509dc700 1 --2- 172.21.15.78:0/6461830 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f0c3c0a29a0 0x7f0c3c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:51.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.270+0000 7f0c49ffb700 1 -- 172.21.15.78:0/6461830 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f0c380cf4e0 con 0x7f0c4c0761a0 2024-03-24T00:19:51.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.270+0000 7f0c49ffb700 1 --2- 172.21.15.78:0/6461830 >> [v2:172.21.15.78:6818/3091148508,v1:172.21.15.78:6819/3091148508] conn(0x7f0c3c0adb00 0x7f0c3c0affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:51.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.270+0000 7f0c49ffb700 1 -- 172.21.15.78:0/6461830 --> [v2:172.21.15.78:6818/3091148508,v1:172.21.15.78:6819/3091148508] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0c3801bec0 con 0x7f0c3c0adb00 2024-03-24T00:19:51.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.270+0000 7f0c49ffb700 1 -- 172.21.15.78:0/6461830 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c380cf930 con 0x7f0c4c0761a0 2024-03-24T00:19:51.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.270+0000 7f0c511dd700 1 --2- 172.21.15.78:0/6461830 >> [v2:172.21.15.78:6818/3091148508,v1:172.21.15.78:6819/3091148508] conn(0x7f0c3c0adb00 0x7f0c3c0affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:51.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.270+0000 7f0c509dc700 1 --2- 172.21.15.78:0/6461830 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f0c3c0a29a0 0x7f0c3c0a4e60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0c4000b2e0 tx=0x7f0c40005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:51.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.271+0000 7f0c511dd700 1 --2- 172.21.15.78:0/6461830 >> [v2:172.21.15.78:6818/3091148508,v1:172.21.15.78:6819/3091148508] conn(0x7f0c3c0adb00 0x7f0c3c0affa0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:51.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.271+0000 7f0c49ffb700 1 -- 172.21.15.78:0/6461830 <== osd.6 v2:172.21.15.78:6818/3091148508 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0c3801bec0 con 0x7f0c3c0adb00 2024-03-24T00:19:51.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.288+0000 7f0c52c40700 1 -- 172.21.15.78:0/6461830 --> [v2:172.21.15.78:6818/3091148508,v1:172.21.15.78:6819/3091148508] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0c4c075a50 con 0x7f0c3c0adb00 2024-03-24T00:19:51.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.288+0000 7f0c49ffb700 1 -- 172.21.15.78:0/6461830 <== osd.6 v2:172.21.15.78:6818/3091148508 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0c4c075a50 con 0x7f0c3c0adb00 2024-03-24T00:19:51.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.288+0000 7f0c52c40700 1 -- 172.21.15.78:0/6461830 >> [v2:172.21.15.78:6818/3091148508,v1:172.21.15.78:6819/3091148508] conn(0x7f0c3c0adb00 msgr2=0x7f0c3c0affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:51.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.288+0000 7f0c52c40700 1 --2- 172.21.15.78:0/6461830 >> [v2:172.21.15.78:6818/3091148508,v1:172.21.15.78:6819/3091148508] conn(0x7f0c3c0adb00 0x7f0c3c0affa0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:51.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.289+0000 7f0c52c40700 1 -- 172.21.15.78:0/6461830 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f0c3c0a29a0 msgr2=0x7f0c3c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:51.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.289+0000 7f0c52c40700 1 --2- 172.21.15.78:0/6461830 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f0c3c0a29a0 0x7f0c3c0a4e60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0c4000b2e0 tx=0x7f0c40005960 comp rx=0 tx=0).stop 2024-03-24T00:19:51.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.289+0000 7f0c52c40700 1 -- 172.21.15.78:0/6461830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c4c0761a0 msgr2=0x7f0c4c19f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:51.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.289+0000 7f0c52c40700 1 --2- 172.21.15.78:0/6461830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c4c0761a0 0x7f0c4c19f3e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0c3800e9a0 tx=0x7f0c3800ecb0 comp rx=0 tx=0).stop 2024-03-24T00:19:51.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.289+0000 7f0c511dd700 1 -- 172.21.15.78:0/6461830 reap_dead start 2024-03-24T00:19:51.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.289+0000 7f0c52c40700 1 -- 172.21.15.78:0/6461830 shutdown_connections 2024-03-24T00:19:51.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.289+0000 7f0c52c40700 1 -- 172.21.15.78:0/6461830 >> 172.21.15.78:0/6461830 conn(0x7f0c4c079250 msgr2=0x7f0c4c111b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:51.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.289+0000 7f0c52c40700 1 -- 172.21.15.78:0/6461830 shutdown_connections 2024-03-24T00:19:51.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:51.289+0000 7f0c52c40700 1 -- 172.21.15.78:0/6461830 wait complete. 2024-03-24T00:19:51.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:51 smithi078 ceph-mon[32429]: pgmap v326: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 6.2 KiB/s, 1 keys/s, 1 objects/s recovering 2024-03-24T00:19:51.592 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:19:51.593 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:19:51.593 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:19:51.593 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:19:51.598 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:19:51.645 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:51 smithi107 ceph-mon[36485]: pgmap v326: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 6.2 KiB/s, 1 keys/s, 1 objects/s recovering 2024-03-24T00:19:51.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:51 smithi106 ceph-mon[36505]: pgmap v326: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 6.2 KiB/s, 1 keys/s, 1 objects/s recovering 2024-03-24T00:19:53.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:53 smithi106 ceph-mon[36505]: pgmap v327: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 5.6 KiB/s, 1 keys/s, 1 objects/s recovering 2024-03-24T00:19:53.672 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:53 smithi078 ceph-mon[32429]: pgmap v327: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 5.6 KiB/s, 1 keys/s, 1 objects/s recovering 2024-03-24T00:19:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:53 smithi107 ceph-mon[36485]: pgmap v327: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 5.6 KiB/s, 1 keys/s, 1 objects/s recovering 2024-03-24T00:19:55.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:55 smithi078 ceph-mon[32429]: pgmap v328: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s; 5.1 KiB/s, 1 keys/s, 1 objects/s recovering 2024-03-24T00:19:55.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.619+0000 7f8ed7e5e700 1 -- 172.21.15.78:0/1197594600 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ed0108220 msgr2=0x7f8ed0108600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.619+0000 7f8ed7e5e700 1 --2- 172.21.15.78:0/1197594600 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ed0108220 0x7f8ed0108600 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8ec00099c0 tx=0x7f8ec0009cd0 comp rx=0 tx=0).stop 2024-03-24T00:19:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.619+0000 7f8ed7e5e700 1 -- 172.21.15.78:0/1197594600 shutdown_connections 2024-03-24T00:19:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.619+0000 7f8ed7e5e700 1 --2- 172.21.15.78:0/1197594600 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ed0102bd0 0x7f8ed010f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.619+0000 7f8ed7e5e700 1 --2- 172.21.15.78:0/1197594600 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ed0108220 0x7f8ed0108600 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.619+0000 7f8ed7e5e700 1 --2- 172.21.15.78:0/1197594600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ed0102210 0x7f8ed0102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.619+0000 7f8ed7e5e700 1 -- 172.21.15.78:0/1197594600 >> 172.21.15.78:0/1197594600 conn(0x7f8ed00792f0 msgr2=0x7f8ed0079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.619+0000 7f8ed7e5e700 1 -- 172.21.15.78:0/1197594600 shutdown_connections 2024-03-24T00:19:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.619+0000 7f8ed7e5e700 1 -- 172.21.15.78:0/1197594600 wait complete. 2024-03-24T00:19:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.620+0000 7f8ed7e5e700 1 Processor -- start 2024-03-24T00:19:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.620+0000 7f8ed7e5e700 1 -- start start 2024-03-24T00:19:55.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.620+0000 7f8ed7e5e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ed0102210 0x7f8ed0074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:55.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.620+0000 7f8ed7e5e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ed0102bd0 0x7f8ed00749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:55.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.620+0000 7f8ed7e5e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ed0108220 0x7f8ed0074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:55.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.620+0000 7f8ed7e5e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed01124a0 con 0x7f8ed0102bd0 2024-03-24T00:19:55.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.620+0000 7f8ed7e5e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed0112320 con 0x7f8ed0108220 2024-03-24T00:19:55.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.620+0000 7f8ed7e5e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed0112620 con 0x7f8ed0102210 2024-03-24T00:19:55.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.620+0000 7f8ed53f9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ed0102bd0 0x7f8ed00749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:55.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.620+0000 7f8ed53f9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ed0102bd0 0x7f8ed00749d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60822/0 (socket says 172.21.15.78:60822) 2024-03-24T00:19:55.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.620+0000 7f8ed53f9700 1 -- 172.21.15.78:0/2797730287 learned_addr learned my addr 172.21.15.78:0/2797730287 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:19:55.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.620+0000 7f8ed63fb700 1 --2- 172.21.15.78:0/2797730287 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ed0108220 0x7f8ed0074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:55.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.620+0000 7f8ed5bfa700 1 --2- 172.21.15.78:0/2797730287 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ed0102210 0x7f8ed0074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:55.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.621+0000 7f8ed53f9700 1 -- 172.21.15.78:0/2797730287 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ed0102210 msgr2=0x7f8ed0074490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:55.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.621+0000 7f8ed53f9700 1 --2- 172.21.15.78:0/2797730287 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ed0102210 0x7f8ed0074490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:55.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.621+0000 7f8ed53f9700 1 -- 172.21.15.78:0/2797730287 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ed0108220 msgr2=0x7f8ed0074f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:55.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.621+0000 7f8ed53f9700 1 --2- 172.21.15.78:0/2797730287 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ed0108220 0x7f8ed0074f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:55.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.621+0000 7f8ed53f9700 1 -- 172.21.15.78:0/2797730287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed01a6d80 con 0x7f8ed0102bd0 2024-03-24T00:19:55.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.621+0000 7f8ed63fb700 1 --2- 172.21.15.78:0/2797730287 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ed0108220 0x7f8ed0074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:55.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.621+0000 7f8ed53f9700 1 --2- 172.21.15.78:0/2797730287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ed0102bd0 0x7f8ed00749d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc00e9a0 tx=0x7f8ecc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:55.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.621+0000 7f8ec6ffd700 1 -- 172.21.15.78:0/2797730287 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc00ca10 con 0x7f8ed0102bd0 2024-03-24T00:19:55.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.621+0000 7f8ec6ffd700 1 -- 172.21.15.78:0/2797730287 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ecc00cbb0 con 0x7f8ed0102bd0 2024-03-24T00:19:55.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.621+0000 7f8ed7e5e700 1 -- 172.21.15.78:0/2797730287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed01a7070 con 0x7f8ed0102bd0 2024-03-24T00:19:55.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.621+0000 7f8ec6ffd700 1 -- 172.21.15.78:0/2797730287 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc018680 con 0x7f8ed0102bd0 2024-03-24T00:19:55.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.621+0000 7f8ed7e5e700 1 -- 172.21.15.78:0/2797730287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed01a75b0 con 0x7f8ed0102bd0 2024-03-24T00:19:55.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.622+0000 7f8ed7e5e700 1 -- 172.21.15.78:0/2797730287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ed0068e50 con 0x7f8ed0102bd0 2024-03-24T00:19:55.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.623+0000 7f8ec6ffd700 1 -- 172.21.15.78:0/2797730287 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f8ecc010430 con 0x7f8ed0102bd0 2024-03-24T00:19:55.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.624+0000 7f8ec6ffd700 1 --2- 172.21.15.78:0/2797730287 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f8ebc0a2920 0x7f8ebc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:55.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.624+0000 7f8ed5bfa700 1 --2- 172.21.15.78:0/2797730287 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f8ebc0a2920 0x7f8ebc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:55.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.624+0000 7f8ec6ffd700 1 -- 172.21.15.78:0/2797730287 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f8ecc0cf600 con 0x7f8ed0102bd0 2024-03-24T00:19:55.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.624+0000 7f8ed5bfa700 1 --2- 172.21.15.78:0/2797730287 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f8ebc0a2920 0x7f8ebc0a4de0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8ec0000c00 tx=0x7f8ec0005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:55.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.627+0000 7f8ec6ffd700 1 -- 172.21.15.78:0/2797730287 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8ecc09c1e0 con 0x7f8ed0102bd0 2024-03-24T00:19:55.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:55 smithi106 ceph-mon[36505]: pgmap v328: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s; 5.1 KiB/s, 1 keys/s, 1 objects/s recovering 2024-03-24T00:19:55.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.832+0000 7f8ed7e5e700 1 -- 172.21.15.78:0/2797730287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f8ed002da30 con 0x7f8ed0102bd0 2024-03-24T00:19:55.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.832+0000 7f8ec6ffd700 1 -- 172.21.15.78:0/2797730287 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f8ecc09bae0 con 0x7f8ed0102bd0 2024-03-24T00:19:55.833 INFO:teuthology.orchestra.run.smithi078.stdout:253403070507 2024-03-24T00:19:55.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.834+0000 7f8ed7e5e700 1 -- 172.21.15.78:0/2797730287 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f8ebc0a2920 msgr2=0x7f8ebc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:55.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.834+0000 7f8ed7e5e700 1 --2- 172.21.15.78:0/2797730287 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f8ebc0a2920 0x7f8ebc0a4de0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8ec0000c00 tx=0x7f8ec0005fb0 comp rx=0 tx=0).stop 2024-03-24T00:19:55.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.834+0000 7f8ed7e5e700 1 -- 172.21.15.78:0/2797730287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ed0102bd0 msgr2=0x7f8ed00749d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:55.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.834+0000 7f8ed7e5e700 1 --2- 172.21.15.78:0/2797730287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ed0102bd0 0x7f8ed00749d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc00e9a0 tx=0x7f8ecc00ecb0 comp rx=0 tx=0).stop 2024-03-24T00:19:55.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.834+0000 7f8ed7e5e700 1 -- 172.21.15.78:0/2797730287 shutdown_connections 2024-03-24T00:19:55.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.834+0000 7f8ed7e5e700 1 --2- 172.21.15.78:0/2797730287 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f8ebc0a2920 0x7f8ebc0a4de0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:55.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.834+0000 7f8ed7e5e700 1 --2- 172.21.15.78:0/2797730287 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ed0108220 0x7f8ed0074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:55.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.834+0000 7f8ed7e5e700 1 --2- 172.21.15.78:0/2797730287 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ed0102210 0x7f8ed0074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:55.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.834+0000 7f8ed7e5e700 1 --2- 172.21.15.78:0/2797730287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ed0102bd0 0x7f8ed00749d0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:55.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.834+0000 7f8ed7e5e700 1 -- 172.21.15.78:0/2797730287 >> 172.21.15.78:0/2797730287 conn(0x7f8ed00792f0 msgr2=0x7f8ed0100d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:55.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.834+0000 7f8ed7e5e700 1 -- 172.21.15.78:0/2797730287 shutdown_connections 2024-03-24T00:19:55.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.834+0000 7f8ed7e5e700 1 -- 172.21.15.78:0/2797730287 wait complete. 2024-03-24T00:19:55.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:55 smithi107 ceph-mon[36485]: pgmap v328: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s; 5.1 KiB/s, 1 keys/s, 1 objects/s recovering 2024-03-24T00:19:56.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.977+0000 7fa5fde53700 1 -- 172.21.15.78:0/2042833291 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5f8101320 msgr2=0x7fa5f8101700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:56.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.977+0000 7fa5fde53700 1 --2- 172.21.15.78:0/2042833291 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5f8101320 0x7fa5f8101700 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa5e80099c0 tx=0x7fa5e8009cd0 comp rx=0 tx=0).stop 2024-03-24T00:19:56.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.977+0000 7fa5fde53700 1 -- 172.21.15.78:0/2042833291 shutdown_connections 2024-03-24T00:19:56.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.977+0000 7fa5fde53700 1 --2- 172.21.15.78:0/2042833291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5f8106370 0x7fa5f810df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:56.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.977+0000 7fa5fde53700 1 --2- 172.21.15.78:0/2042833291 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5f8101320 0x7fa5f8101700 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:56.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.977+0000 7fa5fde53700 1 --2- 172.21.15.78:0/2042833291 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5f8101cd0 0x7fa5f8105c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:56.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.977+0000 7fa5fde53700 1 -- 172.21.15.78:0/2042833291 >> 172.21.15.78:0/2042833291 conn(0x7fa5f80770e0 msgr2=0x7fa5f80774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:56.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.977+0000 7fa5fde53700 1 -- 172.21.15.78:0/2042833291 shutdown_connections 2024-03-24T00:19:56.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.977+0000 7fa5fde53700 1 -- 172.21.15.78:0/2042833291 wait complete. 2024-03-24T00:19:56.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.978+0000 7fa5fde53700 1 Processor -- start 2024-03-24T00:19:56.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.978+0000 7fa5fde53700 1 -- start start 2024-03-24T00:19:56.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.978+0000 7fa5fde53700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5f8101320 0x7fa5f8102d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:56.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.978+0000 7fa5fde53700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5f8101cd0 0x7fa5f8103270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:56.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.978+0000 7fa5fde53700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5f8106370 0x7fa5f81037b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:56.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.978+0000 7fa5fde53700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f8111380 con 0x7fa5f8101320 2024-03-24T00:19:56.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.978+0000 7fa5fde53700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f8111200 con 0x7fa5f8106370 2024-03-24T00:19:56.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.978+0000 7fa5fde53700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f8111500 con 0x7fa5f8101cd0 2024-03-24T00:19:56.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.978+0000 7fa5f77fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5f8101320 0x7fa5f8102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:56.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.978+0000 7fa5f77fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5f8101320 0x7fa5f8102d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60840/0 (socket says 172.21.15.78:60840) 2024-03-24T00:19:56.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.978+0000 7fa5f6ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5f8101cd0 0x7fa5f8103270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:56.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.978+0000 7fa5f77fe700 1 -- 172.21.15.78:0/1605368195 learned_addr learned my addr 172.21.15.78:0/1605368195 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:19:56.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.978+0000 7fa5f7fff700 1 --2- 172.21.15.78:0/1605368195 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5f8106370 0x7fa5f81037b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:56.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.978+0000 7fa5f77fe700 1 -- 172.21.15.78:0/1605368195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5f8101cd0 msgr2=0x7fa5f8103270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:56.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.978+0000 7fa5f77fe700 1 --2- 172.21.15.78:0/1605368195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5f8101cd0 0x7fa5f8103270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:56.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.978+0000 7fa5f77fe700 1 -- 172.21.15.78:0/1605368195 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5f8106370 msgr2=0x7fa5f81037b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:56.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.978+0000 7fa5f77fe700 1 --2- 172.21.15.78:0/1605368195 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5f8106370 0x7fa5f81037b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:56.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.978+0000 7fa5f77fe700 1 -- 172.21.15.78:0/1605368195 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f81a6eb0 con 0x7fa5f8101320 2024-03-24T00:19:56.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.978+0000 7fa5f6ffd700 1 --2- 172.21.15.78:0/1605368195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5f8101cd0 0x7fa5f8103270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:56.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.979+0000 7fa5f77fe700 1 --2- 172.21.15.78:0/1605368195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5f8101320 0x7fa5f8102d30 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa5e8006010 tx=0x7fa5e800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:56.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.979+0000 7fa5f4ff9700 1 -- 172.21.15.78:0/1605368195 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e8003680 con 0x7fa5f8101320 2024-03-24T00:19:56.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.979+0000 7fa5fde53700 1 -- 172.21.15.78:0/1605368195 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5f81a7140 con 0x7fa5f8101320 2024-03-24T00:19:56.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.979+0000 7fa5f4ff9700 1 -- 172.21.15.78:0/1605368195 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5e80050d0 con 0x7fa5f8101320 2024-03-24T00:19:56.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.979+0000 7fa5f4ff9700 1 -- 172.21.15.78:0/1605368195 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e8003db0 con 0x7fa5f8101320 2024-03-24T00:19:56.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.979+0000 7fa5fde53700 1 -- 172.21.15.78:0/1605368195 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5f81a7590 con 0x7fa5f8101320 2024-03-24T00:19:56.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.980+0000 7fa5fde53700 1 -- 172.21.15.78:0/1605368195 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5f8068e50 con 0x7fa5f8101320 2024-03-24T00:19:56.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.981+0000 7fa5f4ff9700 1 -- 172.21.15.78:0/1605368195 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fa5e8005280 con 0x7fa5f8101320 2024-03-24T00:19:56.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.982+0000 7fa5f4ff9700 1 --2- 172.21.15.78:0/1605368195 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa5e00a2ac0 0x7fa5e00a4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:56.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.982+0000 7fa5f6ffd700 1 --2- 172.21.15.78:0/1605368195 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa5e00a2ac0 0x7fa5e00a4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:56.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.982+0000 7fa5f4ff9700 1 -- 172.21.15.78:0/1605368195 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fa5e80cfe10 con 0x7fa5f8101320 2024-03-24T00:19:56.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.982+0000 7fa5f6ffd700 1 --2- 172.21.15.78:0/1605368195 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa5e00a2ac0 0x7fa5e00a4f80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa5f81077b0 tx=0x7fa5ec009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:56.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:55.985+0000 7fa5f4ff9700 1 -- 172.21.15.78:0/1605368195 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa5e805d740 con 0x7fa5f8101320 2024-03-24T00:19:56.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:56.185+0000 7fa5fde53700 1 -- 172.21.15.78:0/1605368195 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7fa5f81a7970 con 0x7fa5f8101320 2024-03-24T00:19:56.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:56.185+0000 7fa5f4ff9700 1 -- 172.21.15.78:0/1605368195 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fa5e8013070 con 0x7fa5f8101320 2024-03-24T00:19:56.186 INFO:teuthology.orchestra.run.smithi078.stdout:343597383708 2024-03-24T00:19:56.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:56.187+0000 7fa5fde53700 1 -- 172.21.15.78:0/1605368195 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa5e00a2ac0 msgr2=0x7fa5e00a4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:56.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:56.187+0000 7fa5fde53700 1 --2- 172.21.15.78:0/1605368195 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa5e00a2ac0 0x7fa5e00a4f80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa5f81077b0 tx=0x7fa5ec009260 comp rx=0 tx=0).stop 2024-03-24T00:19:56.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:56.187+0000 7fa5fde53700 1 -- 172.21.15.78:0/1605368195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5f8101320 msgr2=0x7fa5f8102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:56.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:56.187+0000 7fa5fde53700 1 --2- 172.21.15.78:0/1605368195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5f8101320 0x7fa5f8102d30 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa5e8006010 tx=0x7fa5e800f690 comp rx=0 tx=0).stop 2024-03-24T00:19:56.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:56.187+0000 7fa5fde53700 1 -- 172.21.15.78:0/1605368195 shutdown_connections 2024-03-24T00:19:56.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:56.187+0000 7fa5fde53700 1 --2- 172.21.15.78:0/1605368195 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa5e00a2ac0 0x7fa5e00a4f80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:56.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:56.187+0000 7fa5fde53700 1 --2- 172.21.15.78:0/1605368195 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5f8106370 0x7fa5f81037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:56.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:56.187+0000 7fa5fde53700 1 --2- 172.21.15.78:0/1605368195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5f8101320 0x7fa5f8102d30 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:56.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:56.187+0000 7fa5fde53700 1 --2- 172.21.15.78:0/1605368195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5f8101cd0 0x7fa5f8103270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:56.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:56.187+0000 7fa5fde53700 1 -- 172.21.15.78:0/1605368195 >> 172.21.15.78:0/1605368195 conn(0x7fa5f80770e0 msgr2=0x7fa5f8100880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:56.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:56.187+0000 7fa5fde53700 1 -- 172.21.15.78:0/1605368195 shutdown_connections 2024-03-24T00:19:56.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:19:56.187+0000 7fa5fde53700 1 -- 172.21.15.78:0/1605368195 wait complete. 2024-03-24T00:19:56.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:56 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2797730287' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-24T00:19:56.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:56 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1605368195' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-24T00:19:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:56 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2797730287' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-24T00:19:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:56 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1605368195' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-24T00:19:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:56 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2797730287' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-24T00:19:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:56 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1605368195' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-24T00:19:56.923 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070504 got 253403070507 for osd.8 2024-03-24T00:19:56.923 DEBUG:teuthology.parallel:result is None 2024-03-24T00:19:57.309 INFO:teuthology.orchestra.run.smithi078.stdout:197568495668 2024-03-24T00:19:57.310 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-24T00:19:57.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:57 smithi078 ceph-mon[32429]: pgmap v329: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:19:57.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:57 smithi106 ceph-mon[36505]: pgmap v329: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:19:57.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:57 smithi107 ceph-mon[36485]: pgmap v329: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:19:59.579 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:19:59 smithi078 ceph-mon[32429]: pgmap v330: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:19:59.608 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383706 got 343597383708 for osd.11 2024-03-24T00:19:59.608 DEBUG:teuthology.parallel:result is None 2024-03-24T00:19:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:19:59 smithi106 ceph-mon[36505]: pgmap v330: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:19:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:19:59 smithi107 ceph-mon[36485]: pgmap v330: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:20:00.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.033+0000 7f3992b85700 1 -- 172.21.15.78:0/963359730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f398c107520 msgr2=0x7f398c10f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:00.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.033+0000 7f3992b85700 1 --2- 172.21.15.78:0/963359730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f398c107520 0x7f398c10f0a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f398400b0d0 tx=0x7f398400b3e0 comp rx=0 tx=0).stop 2024-03-24T00:20:00.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.033+0000 7f3992b85700 1 -- 172.21.15.78:0/963359730 shutdown_connections 2024-03-24T00:20:00.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.033+0000 7f3992b85700 1 --2- 172.21.15.78:0/963359730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f398c107520 0x7f398c10f0a0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:00.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.033+0000 7f3992b85700 1 --2- 172.21.15.78:0/963359730 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f398c102470 0x7f398c102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:00.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.033+0000 7f3992b85700 1 --2- 172.21.15.78:0/963359730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f398c102e20 0x7f398c106df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:00.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.033+0000 7f3992b85700 1 -- 172.21.15.78:0/963359730 >> 172.21.15.78:0/963359730 conn(0x7f398c0792b0 msgr2=0x7f398c0796c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:00.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.033+0000 7f3992b85700 1 -- 172.21.15.78:0/963359730 shutdown_connections 2024-03-24T00:20:00.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.033+0000 7f3992b85700 1 -- 172.21.15.78:0/963359730 wait complete. 2024-03-24T00:20:00.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.034+0000 7f3992b85700 1 Processor -- start 2024-03-24T00:20:00.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.034+0000 7f3992b85700 1 -- start start 2024-03-24T00:20:00.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.034+0000 7f3992b85700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f398c102470 0x7f398c19e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:00.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.034+0000 7f3992b85700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f398c102e20 0x7f398c19eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:00.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.034+0000 7f3992b85700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f398c107520 0x7f398c1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:00.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.034+0000 7f3992b85700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398c1124c0 con 0x7f398c102470 2024-03-24T00:20:00.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.034+0000 7f3992b85700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398c112340 con 0x7f398c102e20 2024-03-24T00:20:00.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.034+0000 7f3992b85700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398c112640 con 0x7f398c107520 2024-03-24T00:20:00.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.034+0000 7f3990921700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f398c102470 0x7f398c19e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:00.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.034+0000 7f3990921700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f398c102470 0x7f398c19e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60860/0 (socket says 172.21.15.78:60860) 2024-03-24T00:20:00.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.034+0000 7f3990921700 1 -- 172.21.15.78:0/2906621328 learned_addr learned my addr 172.21.15.78:0/2906621328 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:00.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.034+0000 7f398bfff700 1 --2- 172.21.15.78:0/2906621328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f398c102e20 0x7f398c19eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:00.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.034+0000 7f3991122700 1 --2- 172.21.15.78:0/2906621328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f398c107520 0x7f398c1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:00.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.035+0000 7f3990921700 1 -- 172.21.15.78:0/2906621328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f398c107520 msgr2=0x7f398c1986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:00.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.035+0000 7f3990921700 1 --2- 172.21.15.78:0/2906621328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f398c107520 0x7f398c1986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:00.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.035+0000 7f3990921700 1 -- 172.21.15.78:0/2906621328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f398c102e20 msgr2=0x7f398c19eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:00.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.035+0000 7f3990921700 1 --2- 172.21.15.78:0/2906621328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f398c102e20 0x7f398c19eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:00.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.035+0000 7f3990921700 1 -- 172.21.15.78:0/2906621328 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f398c198fa0 con 0x7f398c102470 2024-03-24T00:20:00.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.035+0000 7f398bfff700 1 --2- 172.21.15.78:0/2906621328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f398c102e20 0x7f398c19eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:00.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.035+0000 7f3990921700 1 --2- 172.21.15.78:0/2906621328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f398c102470 0x7f398c19e5f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f398000e9a0 tx=0x7f398000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:00.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.035+0000 7f3989ffb700 1 -- 172.21.15.78:0/2906621328 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398000ca10 con 0x7f398c102470 2024-03-24T00:20:00.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.035+0000 7f3992b85700 1 -- 172.21.15.78:0/2906621328 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f398c199290 con 0x7f398c102470 2024-03-24T00:20:00.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.035+0000 7f3989ffb700 1 -- 172.21.15.78:0/2906621328 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f398000cbb0 con 0x7f398c102470 2024-03-24T00:20:00.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.035+0000 7f3989ffb700 1 -- 172.21.15.78:0/2906621328 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3980018680 con 0x7f398c102470 2024-03-24T00:20:00.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.035+0000 7f3992b85700 1 -- 172.21.15.78:0/2906621328 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f398c1a5570 con 0x7f398c102470 2024-03-24T00:20:00.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.036+0000 7f3992b85700 1 -- 172.21.15.78:0/2906621328 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f398c1038d0 con 0x7f398c102470 2024-03-24T00:20:00.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.038+0000 7f3989ffb700 1 -- 172.21.15.78:0/2906621328 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f3980010430 con 0x7f398c102470 2024-03-24T00:20:00.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.038+0000 7f3989ffb700 1 --2- 172.21.15.78:0/2906621328 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f397c0a2920 0x7f397c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:00.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.039+0000 7f3989ffb700 1 -- 172.21.15.78:0/2906621328 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f39800cf620 con 0x7f398c102470 2024-03-24T00:20:00.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.039+0000 7f398bfff700 1 --2- 172.21.15.78:0/2906621328 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f397c0a2920 0x7f397c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:00.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.040+0000 7f398bfff700 1 --2- 172.21.15.78:0/2906621328 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f397c0a2920 0x7f397c0a4de0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f398c199e80 tx=0x7f3978009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:00.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.043+0000 7f3989ffb700 1 -- 172.21.15.78:0/2906621328 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f398009c200 con 0x7f398c102470 2024-03-24T00:20:00.254 INFO:teuthology.orchestra.run.smithi078.stdout:77309411402 2024-03-24T00:20:00.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.253+0000 7f3992b85700 1 -- 172.21.15.78:0/2906621328 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f398c1994c0 con 0x7f398c102470 2024-03-24T00:20:00.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.253+0000 7f3989ffb700 1 -- 172.21.15.78:0/2906621328 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f398009bb00 con 0x7f398c102470 2024-03-24T00:20:00.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.255+0000 7f3992b85700 1 -- 172.21.15.78:0/2906621328 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f397c0a2920 msgr2=0x7f397c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:00.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.255+0000 7f3992b85700 1 --2- 172.21.15.78:0/2906621328 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f397c0a2920 0x7f397c0a4de0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f398c199e80 tx=0x7f3978009260 comp rx=0 tx=0).stop 2024-03-24T00:20:00.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.255+0000 7f3992b85700 1 -- 172.21.15.78:0/2906621328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f398c102470 msgr2=0x7f398c19e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:00.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.255+0000 7f3992b85700 1 --2- 172.21.15.78:0/2906621328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f398c102470 0x7f398c19e5f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f398000e9a0 tx=0x7f398000ecb0 comp rx=0 tx=0).stop 2024-03-24T00:20:00.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.255+0000 7f3992b85700 1 -- 172.21.15.78:0/2906621328 shutdown_connections 2024-03-24T00:20:00.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.255+0000 7f3992b85700 1 --2- 172.21.15.78:0/2906621328 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f397c0a2920 0x7f397c0a4de0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:00.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.255+0000 7f3992b85700 1 --2- 172.21.15.78:0/2906621328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f398c107520 0x7f398c1986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:00.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.255+0000 7f3992b85700 1 --2- 172.21.15.78:0/2906621328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f398c102470 0x7f398c19e5f0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:00.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.255+0000 7f3992b85700 1 --2- 172.21.15.78:0/2906621328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f398c102e20 0x7f398c19eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:00.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.255+0000 7f3992b85700 1 -- 172.21.15.78:0/2906621328 >> 172.21.15.78:0/2906621328 conn(0x7f398c0792b0 msgr2=0x7f398c100220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:00.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.256+0000 7f3992b85700 1 -- 172.21.15.78:0/2906621328 shutdown_connections 2024-03-24T00:20:00.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:00.256+0000 7f3992b85700 1 -- 172.21.15.78:0/2906621328 wait complete. 2024-03-24T00:20:00.525 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:00 smithi078 ceph-mon[32429]: overall HEALTH_OK 2024-03-24T00:20:00.525 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:00 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2906621328' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T00:20:00.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:00 smithi106 ceph-mon[36505]: overall HEALTH_OK 2024-03-24T00:20:00.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:00 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2906621328' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T00:20:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:00 smithi107 ceph-mon[36485]: overall HEALTH_OK 2024-03-24T00:20:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:00 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2906621328' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T00:20:01.199 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:20:01.199 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:20:01.200 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:20:01.200 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:20:01.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:01 smithi106 ceph-mon[36505]: pgmap v331: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:20:01.771 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:01 smithi078 ceph-mon[32429]: pgmap v331: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:20:01.796 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411398 got 77309411402 for osd.2 2024-03-24T00:20:01.796 DEBUG:teuthology.parallel:result is None 2024-03-24T00:20:01.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:01 smithi107 ceph-mon[36485]: pgmap v331: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:20:02.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.435+0000 7f67f3b04700 1 -- 172.21.15.78:0/988641726 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67ec101a30 msgr2=0x7f67ec10df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:02.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.435+0000 7f67f3b04700 1 --2- 172.21.15.78:0/988641726 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67ec101a30 0x7f67ec10df10 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f67e400b0a0 tx=0x7f67e400b3b0 comp rx=0 tx=0).stop 2024-03-24T00:20:02.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.435+0000 7f67f3b04700 1 -- 172.21.15.78:0/988641726 shutdown_connections 2024-03-24T00:20:02.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.435+0000 7f67f3b04700 1 --2- 172.21.15.78:0/988641726 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67ec101a30 0x7f67ec10df10 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:02.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.435+0000 7f67f3b04700 1 --2- 172.21.15.78:0/988641726 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67ec1070d0 0x7f67ec1074b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:02.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.435+0000 7f67f3b04700 1 --2- 172.21.15.78:0/988641726 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67ec101090 0x7f67ec1014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:02.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.435+0000 7f67f3b04700 1 -- 172.21.15.78:0/988641726 >> 172.21.15.78:0/988641726 conn(0x7f67ec077fc0 msgr2=0x7f67ec0783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:02.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.435+0000 7f67f3b04700 1 -- 172.21.15.78:0/988641726 shutdown_connections 2024-03-24T00:20:02.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.435+0000 7f67f3b04700 1 -- 172.21.15.78:0/988641726 wait complete. 2024-03-24T00:20:02.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.436+0000 7f67f3b04700 1 Processor -- start 2024-03-24T00:20:02.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.436+0000 7f67f3b04700 1 -- start start 2024-03-24T00:20:02.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.436+0000 7f67f3b04700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67ec101090 0x7f67ec102d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:02.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.436+0000 7f67f3b04700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67ec101a30 0x7f67ec103280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:02.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.436+0000 7f67f3b04700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67ec1070d0 0x7f67ec1037c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:02.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.436+0000 7f67f3b04700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67ec111350 con 0x7f67ec101a30 2024-03-24T00:20:02.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.436+0000 7f67f3b04700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67ec1111d0 con 0x7f67ec101090 2024-03-24T00:20:02.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.436+0000 7f67f3b04700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67ec1114d0 con 0x7f67ec1070d0 2024-03-24T00:20:02.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.436+0000 7f67f109f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67ec101a30 0x7f67ec103280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:02.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.436+0000 7f67f109f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67ec101a30 0x7f67ec103280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60880/0 (socket says 172.21.15.78:60880) 2024-03-24T00:20:02.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.436+0000 7f67f109f700 1 -- 172.21.15.78:0/1768981885 learned_addr learned my addr 172.21.15.78:0/1768981885 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:02.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.436+0000 7f67f18a0700 1 --2- 172.21.15.78:0/1768981885 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67ec101090 0x7f67ec102d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:02.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.436+0000 7f67f20a1700 1 --2- 172.21.15.78:0/1768981885 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67ec1070d0 0x7f67ec1037c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:02.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.437+0000 7f67f18a0700 1 -- 172.21.15.78:0/1768981885 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67ec1070d0 msgr2=0x7f67ec1037c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:02.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.437+0000 7f67f18a0700 1 --2- 172.21.15.78:0/1768981885 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67ec1070d0 0x7f67ec1037c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:02.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.437+0000 7f67f18a0700 1 -- 172.21.15.78:0/1768981885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67ec101a30 msgr2=0x7f67ec103280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:02.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.437+0000 7f67f18a0700 1 --2- 172.21.15.78:0/1768981885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67ec101a30 0x7f67ec103280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:02.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.437+0000 7f67f18a0700 1 -- 172.21.15.78:0/1768981885 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67ec1a6e80 con 0x7f67ec101090 2024-03-24T00:20:02.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.437+0000 7f67f109f700 1 --2- 172.21.15.78:0/1768981885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67ec101a30 0x7f67ec103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:02.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.437+0000 7f67f18a0700 1 --2- 172.21.15.78:0/1768981885 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67ec101090 0x7f67ec102d40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f67dc00e9f0 tx=0x7f67dc00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:02.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.437+0000 7f67e2ffd700 1 -- 172.21.15.78:0/1768981885 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67dc00cb70 con 0x7f67ec101090 2024-03-24T00:20:02.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.437+0000 7f67f3b04700 1 -- 172.21.15.78:0/1768981885 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67ec1a7170 con 0x7f67ec101090 2024-03-24T00:20:02.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.437+0000 7f67e2ffd700 1 -- 172.21.15.78:0/1768981885 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67dc00ee90 con 0x7f67ec101090 2024-03-24T00:20:02.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.437+0000 7f67e2ffd700 1 -- 172.21.15.78:0/1768981885 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67dc003dc0 con 0x7f67ec101090 2024-03-24T00:20:02.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.437+0000 7f67f3b04700 1 -- 172.21.15.78:0/1768981885 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67ec1a76b0 con 0x7f67ec101090 2024-03-24T00:20:02.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.438+0000 7f67f3b04700 1 -- 172.21.15.78:0/1768981885 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67ec068e50 con 0x7f67ec101090 2024-03-24T00:20:02.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.439+0000 7f67e2ffd700 1 -- 172.21.15.78:0/1768981885 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f67dc004000 con 0x7f67ec101090 2024-03-24T00:20:02.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.439+0000 7f67e2ffd700 1 --2- 172.21.15.78:0/1768981885 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f67d80a2920 0x7f67d80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:02.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.439+0000 7f67f109f700 1 --2- 172.21.15.78:0/1768981885 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f67d80a2920 0x7f67d80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:02.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.440+0000 7f67e2ffd700 1 -- 172.21.15.78:0/1768981885 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f67dc0cf0b0 con 0x7f67ec101090 2024-03-24T00:20:02.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.440+0000 7f67f109f700 1 --2- 172.21.15.78:0/1768981885 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f67d80a2920 0x7f67d80a4de0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f67ec0785c0 tx=0x7f67e8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:02.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.444+0000 7f67e2ffd700 1 -- 172.21.15.78:0/1768981885 <== mon.2 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f67dc09bc30 con 0x7f67ec101090 2024-03-24T00:20:02.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.642+0000 7f67f3b04700 1 -- 172.21.15.78:0/1768981885 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f67ec1a7970 con 0x7f67ec101090 2024-03-24T00:20:02.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.642+0000 7f67e2ffd700 1 -- 172.21.15.78:0/1768981885 <== mon.2 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f67dc09b530 con 0x7f67ec101090 2024-03-24T00:20:02.643 INFO:teuthology.orchestra.run.smithi078.stdout:141733920833 2024-03-24T00:20:02.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.644+0000 7f67f3b04700 1 -- 172.21.15.78:0/1768981885 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f67d80a2920 msgr2=0x7f67d80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:02.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.644+0000 7f67f3b04700 1 --2- 172.21.15.78:0/1768981885 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f67d80a2920 0x7f67d80a4de0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f67ec0785c0 tx=0x7f67e8009260 comp rx=0 tx=0).stop 2024-03-24T00:20:02.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.644+0000 7f67f3b04700 1 -- 172.21.15.78:0/1768981885 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67ec101090 msgr2=0x7f67ec102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:02.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.644+0000 7f67f3b04700 1 --2- 172.21.15.78:0/1768981885 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67ec101090 0x7f67ec102d40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f67dc00e9f0 tx=0x7f67dc00ed00 comp rx=0 tx=0).stop 2024-03-24T00:20:02.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.644+0000 7f67f3b04700 1 -- 172.21.15.78:0/1768981885 shutdown_connections 2024-03-24T00:20:02.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.644+0000 7f67f3b04700 1 --2- 172.21.15.78:0/1768981885 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f67d80a2920 0x7f67d80a4de0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:02.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.644+0000 7f67f3b04700 1 --2- 172.21.15.78:0/1768981885 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67ec1070d0 0x7f67ec1037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:02.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.644+0000 7f67f3b04700 1 --2- 172.21.15.78:0/1768981885 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67ec101090 0x7f67ec102d40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:02.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.644+0000 7f67f3b04700 1 --2- 172.21.15.78:0/1768981885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67ec101a30 0x7f67ec103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:02.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.644+0000 7f67f3b04700 1 -- 172.21.15.78:0/1768981885 >> 172.21.15.78:0/1768981885 conn(0x7f67ec077fc0 msgr2=0x7f67ec100450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:02.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.644+0000 7f67f3b04700 1 -- 172.21.15.78:0/1768981885 shutdown_connections 2024-03-24T00:20:02.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:02.644+0000 7f67f3b04700 1 -- 172.21.15.78:0/1768981885 wait complete. 2024-03-24T00:20:03.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.097+0000 7fe58a046700 1 -- 172.21.15.78:0/55321486 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe584100ac0 msgr2=0x7fe584100f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:03.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.097+0000 7fe58a046700 1 --2- 172.21.15.78:0/55321486 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe584100ac0 0x7fe584100f40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe578009990 tx=0x7fe578009ca0 comp rx=0 tx=0).stop 2024-03-24T00:20:03.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.097+0000 7fe58a046700 1 -- 172.21.15.78:0/55321486 shutdown_connections 2024-03-24T00:20:03.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.097+0000 7fe58a046700 1 --2- 172.21.15.78:0/55321486 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe584101480 0x7fe58410d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:03.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.097+0000 7fe58a046700 1 --2- 172.21.15.78:0/55321486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5841068a0 0x7fe584106c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:03.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.097+0000 7fe58a046700 1 --2- 172.21.15.78:0/55321486 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe584100ac0 0x7fe584100f40 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:03.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.097+0000 7fe58a046700 1 -- 172.21.15.78:0/55321486 >> 172.21.15.78:0/55321486 conn(0x7fe5840763c0 msgr2=0x7fe5840767d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:03.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.097+0000 7fe58a046700 1 -- 172.21.15.78:0/55321486 shutdown_connections 2024-03-24T00:20:03.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.097+0000 7fe58a046700 1 -- 172.21.15.78:0/55321486 wait complete. 2024-03-24T00:20:03.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.098+0000 7fe58a046700 1 Processor -- start 2024-03-24T00:20:03.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.098+0000 7fe58a046700 1 -- start start 2024-03-24T00:20:03.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.098+0000 7fe58a046700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe584100ac0 0x7fe584196770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:03.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.098+0000 7fe58a046700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe584101480 0x7fe584196cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:03.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.099+0000 7fe58a046700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5841068a0 0x7fe58419aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:03.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.099+0000 7fe58a046700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5841102e0 con 0x7fe584101480 2024-03-24T00:20:03.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.099+0000 7fe58a046700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe584110160 con 0x7fe5841068a0 2024-03-24T00:20:03.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.099+0000 7fe58a046700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe584110460 con 0x7fe584100ac0 2024-03-24T00:20:03.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.099+0000 7fe582ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe584101480 0x7fe584196cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:03.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.099+0000 7fe582ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe584101480 0x7fe584196cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60892/0 (socket says 172.21.15.78:60892) 2024-03-24T00:20:03.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.099+0000 7fe5837fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe584100ac0 0x7fe584196770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:03.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.099+0000 7fe583fff700 1 --2- 172.21.15.78:0/3741190062 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5841068a0 0x7fe58419aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:03.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.099+0000 7fe582ffd700 1 -- 172.21.15.78:0/3741190062 learned_addr learned my addr 172.21.15.78:0/3741190062 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:03.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.099+0000 7fe582ffd700 1 -- 172.21.15.78:0/3741190062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe584100ac0 msgr2=0x7fe584196770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:03.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.099+0000 7fe582ffd700 1 --2- 172.21.15.78:0/3741190062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe584100ac0 0x7fe584196770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:03.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.099+0000 7fe582ffd700 1 -- 172.21.15.78:0/3741190062 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5841068a0 msgr2=0x7fe58419aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:03.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.099+0000 7fe582ffd700 1 --2- 172.21.15.78:0/3741190062 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5841068a0 0x7fe58419aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:03.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.099+0000 7fe582ffd700 1 -- 172.21.15.78:0/3741190062 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe58419b730 con 0x7fe584101480 2024-03-24T00:20:03.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.099+0000 7fe5837fe700 1 --2- 172.21.15.78:0/3741190062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe584100ac0 0x7fe584196770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:03.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.099+0000 7fe583fff700 1 --2- 172.21.15.78:0/3741190062 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5841068a0 0x7fe58419aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:03.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.099+0000 7fe582ffd700 1 --2- 172.21.15.78:0/3741190062 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe584101480 0x7fe584196cb0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe57800bae0 tx=0x7fe57800bb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:03.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.099+0000 7fe580ff9700 1 -- 172.21.15.78:0/3741190062 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe578026070 con 0x7fe584101480 2024-03-24T00:20:03.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.099+0000 7fe58a046700 1 -- 172.21.15.78:0/3741190062 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe58419b9c0 con 0x7fe584101480 2024-03-24T00:20:03.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.099+0000 7fe580ff9700 1 -- 172.21.15.78:0/3741190062 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe578003d70 con 0x7fe584101480 2024-03-24T00:20:03.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.099+0000 7fe580ff9700 1 -- 172.21.15.78:0/3741190062 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe578021410 con 0x7fe584101480 2024-03-24T00:20:03.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.099+0000 7fe58a046700 1 -- 172.21.15.78:0/3741190062 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5841a32a0 con 0x7fe584101480 2024-03-24T00:20:03.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.100+0000 7fe58a046700 1 -- 172.21.15.78:0/3741190062 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe584068e50 con 0x7fe584101480 2024-03-24T00:20:03.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.101+0000 7fe580ff9700 1 -- 172.21.15.78:0/3741190062 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fe578029070 con 0x7fe584101480 2024-03-24T00:20:03.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.102+0000 7fe580ff9700 1 --2- 172.21.15.78:0/3741190062 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe56c0a2920 0x7fe56c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:03.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.102+0000 7fe5837fe700 1 --2- 172.21.15.78:0/3741190062 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe56c0a2920 0x7fe56c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:03.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.102+0000 7fe580ff9700 1 -- 172.21.15.78:0/3741190062 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fe5780d02c0 con 0x7fe584101480 2024-03-24T00:20:03.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.103+0000 7fe5837fe700 1 --2- 172.21.15.78:0/3741190062 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe56c0a2920 0x7fe56c0a4de0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe57c009680 tx=0x7fe57c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:03.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.106+0000 7fe580ff9700 1 -- 172.21.15.78:0/3741190062 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe578030080 con 0x7fe584101480 2024-03-24T00:20:03.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.307+0000 7fe58a046700 1 -- 172.21.15.78:0/3741190062 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7fe58402da30 con 0x7fe584101480 2024-03-24T00:20:03.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.307+0000 7fe580ff9700 1 -- 172.21.15.78:0/3741190062 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fe578013070 con 0x7fe584101480 2024-03-24T00:20:03.308 INFO:teuthology.orchestra.run.smithi078.stdout:287762808872 2024-03-24T00:20:03.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.309+0000 7fe58a046700 1 -- 172.21.15.78:0/3741190062 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe56c0a2920 msgr2=0x7fe56c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:03.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.309+0000 7fe58a046700 1 --2- 172.21.15.78:0/3741190062 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe56c0a2920 0x7fe56c0a4de0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe57c009680 tx=0x7fe57c009260 comp rx=0 tx=0).stop 2024-03-24T00:20:03.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.309+0000 7fe58a046700 1 -- 172.21.15.78:0/3741190062 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe584101480 msgr2=0x7fe584196cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:03.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.309+0000 7fe58a046700 1 --2- 172.21.15.78:0/3741190062 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe584101480 0x7fe584196cb0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe57800bae0 tx=0x7fe57800bb10 comp rx=0 tx=0).stop 2024-03-24T00:20:03.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.309+0000 7fe58a046700 1 -- 172.21.15.78:0/3741190062 shutdown_connections 2024-03-24T00:20:03.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.309+0000 7fe58a046700 1 --2- 172.21.15.78:0/3741190062 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe56c0a2920 0x7fe56c0a4de0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:03.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.309+0000 7fe58a046700 1 --2- 172.21.15.78:0/3741190062 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5841068a0 0x7fe58419aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:03.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.309+0000 7fe58a046700 1 --2- 172.21.15.78:0/3741190062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe584100ac0 0x7fe584196770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:03.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.309+0000 7fe58a046700 1 --2- 172.21.15.78:0/3741190062 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe584101480 0x7fe584196cb0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:03.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.309+0000 7fe58a046700 1 -- 172.21.15.78:0/3741190062 >> 172.21.15.78:0/3741190062 conn(0x7fe5840763c0 msgr2=0x7fe5840febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:03.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.309+0000 7fe58a046700 1 -- 172.21.15.78:0/3741190062 shutdown_connections 2024-03-24T00:20:03.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:03.309+0000 7fe58a046700 1 -- 172.21.15.78:0/3741190062 wait complete. 2024-03-24T00:20:03.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:03 smithi078 ceph-mon[32429]: pgmap v332: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:20:03.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:03 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1768981885' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T00:20:03.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:03 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3741190062' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-24T00:20:03.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:03 smithi106 ceph-mon[36505]: pgmap v332: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:20:03.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:03 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1768981885' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T00:20:03.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:03 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3741190062' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-24T00:20:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:03 smithi107 ceph-mon[36485]: pgmap v332: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:20:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:03 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1768981885' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T00:20:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:03 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3741190062' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-24T00:20:03.935 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920830 got 141733920833 for osd.4 2024-03-24T00:20:03.936 DEBUG:teuthology.parallel:result is None 2024-03-24T00:20:05.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.562+0000 7f6033d43700 1 -- 172.21.15.78:0/1022031476 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f602c101cd0 msgr2=0x7f602c105cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:05.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.562+0000 7f6033d43700 1 --2- 172.21.15.78:0/1022031476 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f602c101cd0 0x7f602c105cd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f6028009990 tx=0x7f6028009ca0 comp rx=0 tx=0).stop 2024-03-24T00:20:05.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.562+0000 7f6033d43700 1 -- 172.21.15.78:0/1022031476 shutdown_connections 2024-03-24T00:20:05.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.562+0000 7f6033d43700 1 --2- 172.21.15.78:0/1022031476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f602c1063b0 0x7f602c10df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:05.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.562+0000 7f6033d43700 1 --2- 172.21.15.78:0/1022031476 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f602c101320 0x7f602c101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:05.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.562+0000 7f6033d43700 1 --2- 172.21.15.78:0/1022031476 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f602c101cd0 0x7f602c105cd0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:05.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.562+0000 7f6033d43700 1 -- 172.21.15.78:0/1022031476 >> 172.21.15.78:0/1022031476 conn(0x7f602c0770b0 msgr2=0x7f602c0774c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:05.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.562+0000 7f6033d43700 1 -- 172.21.15.78:0/1022031476 shutdown_connections 2024-03-24T00:20:05.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.562+0000 7f6033d43700 1 -- 172.21.15.78:0/1022031476 wait complete. 2024-03-24T00:20:05.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.563+0000 7f6033d43700 1 Processor -- start 2024-03-24T00:20:05.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.563+0000 7f6033d43700 1 -- start start 2024-03-24T00:20:05.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.563+0000 7f6033d43700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f602c101320 0x7f602c199770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:05.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.563+0000 7f6033d43700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f602c101cd0 0x7f602c199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:05.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.563+0000 7f6033d43700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f602c1063b0 0x7f602c1a47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:05.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.563+0000 7f6033d43700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602c111370 con 0x7f602c101320 2024-03-24T00:20:05.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.563+0000 7f6033d43700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602c1111f0 con 0x7f602c101cd0 2024-03-24T00:20:05.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.563+0000 7f6033d43700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602c1114f0 con 0x7f602c1063b0 2024-03-24T00:20:05.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.563+0000 7f6031adf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f602c101320 0x7f602c199770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:05.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.563+0000 7f6031adf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f602c101320 0x7f602c199770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52502/0 (socket says 172.21.15.78:52502) 2024-03-24T00:20:05.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.563+0000 7f6031adf700 1 -- 172.21.15.78:0/392588865 learned_addr learned my addr 172.21.15.78:0/392588865 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:05.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.563+0000 7f60312de700 1 --2- 172.21.15.78:0/392588865 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f602c101cd0 0x7f602c199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:05.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.563+0000 7f60322e0700 1 --2- 172.21.15.78:0/392588865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f602c1063b0 0x7f602c1a47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:05.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.564+0000 7f6031adf700 1 -- 172.21.15.78:0/392588865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f602c1063b0 msgr2=0x7f602c1a47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:05.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.564+0000 7f6031adf700 1 --2- 172.21.15.78:0/392588865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f602c1063b0 0x7f602c1a47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:05.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.564+0000 7f6031adf700 1 -- 172.21.15.78:0/392588865 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f602c101cd0 msgr2=0x7f602c199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:05.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.564+0000 7f6031adf700 1 --2- 172.21.15.78:0/392588865 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f602c101cd0 0x7f602c199cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:05.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.564+0000 7f6031adf700 1 -- 172.21.15.78:0/392588865 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f602c1a4d30 con 0x7f602c101320 2024-03-24T00:20:05.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.564+0000 7f6031adf700 1 --2- 172.21.15.78:0/392588865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f602c101320 0x7f602c199770 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f601c00e9a0 tx=0x7f601c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:05.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.564+0000 7f6022ffd700 1 -- 172.21.15.78:0/392588865 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601c00ca10 con 0x7f602c101320 2024-03-24T00:20:05.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.564+0000 7f6022ffd700 1 -- 172.21.15.78:0/392588865 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f601c00cbb0 con 0x7f602c101320 2024-03-24T00:20:05.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.564+0000 7f6033d43700 1 -- 172.21.15.78:0/392588865 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f602c1a4f60 con 0x7f602c101320 2024-03-24T00:20:05.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.564+0000 7f6022ffd700 1 -- 172.21.15.78:0/392588865 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601c018680 con 0x7f602c101320 2024-03-24T00:20:05.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.564+0000 7f6033d43700 1 -- 172.21.15.78:0/392588865 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f602c1a53c0 con 0x7f602c101320 2024-03-24T00:20:05.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.565+0000 7f6033d43700 1 -- 172.21.15.78:0/392588865 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f602c1027b0 con 0x7f602c101320 2024-03-24T00:20:05.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.567+0000 7f6022ffd700 1 -- 172.21.15.78:0/392588865 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f601c010430 con 0x7f602c101320 2024-03-24T00:20:05.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.567+0000 7f6022ffd700 1 --2- 172.21.15.78:0/392588865 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f60180a29a0 0x7f60180a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:05.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.567+0000 7f60312de700 1 --2- 172.21.15.78:0/392588865 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f60180a29a0 0x7f60180a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:05.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.567+0000 7f6022ffd700 1 -- 172.21.15.78:0/392588865 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f601c0cf510 con 0x7f602c101320 2024-03-24T00:20:05.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.568+0000 7f60312de700 1 --2- 172.21.15.78:0/392588865 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f60180a29a0 0x7f60180a4e60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f602c06b3f0 tx=0x7f602800b280 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:05.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.571+0000 7f6022ffd700 1 -- 172.21.15.78:0/392588865 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f601c09c0f0 con 0x7f602c101320 2024-03-24T00:20:05.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:05 smithi106 ceph-mon[36505]: pgmap v333: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:20:05.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:05 smithi078 ceph-mon[32429]: pgmap v333: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:20:05.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.771+0000 7f6033d43700 1 -- 172.21.15.78:0/392588865 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f602c02dbf0 con 0x7f602c101320 2024-03-24T00:20:05.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.772+0000 7f6022ffd700 1 -- 172.21.15.78:0/392588865 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f601c09b9f0 con 0x7f602c101320 2024-03-24T00:20:05.773 INFO:teuthology.orchestra.run.smithi078.stdout:34359738453 2024-03-24T00:20:05.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.774+0000 7f6033d43700 1 -- 172.21.15.78:0/392588865 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f60180a29a0 msgr2=0x7f60180a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:05.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.774+0000 7f6033d43700 1 --2- 172.21.15.78:0/392588865 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f60180a29a0 0x7f60180a4e60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f602c06b3f0 tx=0x7f602800b280 comp rx=0 tx=0).stop 2024-03-24T00:20:05.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.774+0000 7f6033d43700 1 -- 172.21.15.78:0/392588865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f602c101320 msgr2=0x7f602c199770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:05.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.774+0000 7f6033d43700 1 --2- 172.21.15.78:0/392588865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f602c101320 0x7f602c199770 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f601c00e9a0 tx=0x7f601c00ecb0 comp rx=0 tx=0).stop 2024-03-24T00:20:05.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.774+0000 7f6033d43700 1 -- 172.21.15.78:0/392588865 shutdown_connections 2024-03-24T00:20:05.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.774+0000 7f6033d43700 1 --2- 172.21.15.78:0/392588865 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f60180a29a0 0x7f60180a4e60 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:05.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.774+0000 7f6033d43700 1 --2- 172.21.15.78:0/392588865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f602c1063b0 0x7f602c1a47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:05.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.774+0000 7f6033d43700 1 --2- 172.21.15.78:0/392588865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f602c101320 0x7f602c199770 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:05.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.774+0000 7f6033d43700 1 --2- 172.21.15.78:0/392588865 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f602c101cd0 0x7f602c199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:05.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.774+0000 7f6033d43700 1 -- 172.21.15.78:0/392588865 >> 172.21.15.78:0/392588865 conn(0x7f602c0770b0 msgr2=0x7f602c100850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:05.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.774+0000 7f6033d43700 1 -- 172.21.15.78:0/392588865 shutdown_connections 2024-03-24T00:20:05.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:05.774+0000 7f6033d43700 1 -- 172.21.15.78:0/392588865 wait complete. 2024-03-24T00:20:05.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:05 smithi107 ceph-mon[36485]: pgmap v333: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:20:06.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:06 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/392588865' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T00:20:06.669 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:06 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/392588865' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T00:20:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:06 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/392588865' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T00:20:07.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:07 smithi106 ceph-mon[36505]: pgmap v334: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:20:07.703 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:07 smithi078 ceph-mon[32429]: pgmap v334: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:20:07.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:07 smithi107 ceph-mon[36485]: pgmap v334: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:20:08.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.509+0000 7f263de97700 1 -- 172.21.15.78:0/2677080467 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2638106370 msgr2=0x7f263810df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:08.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.509+0000 7f263de97700 1 --2- 172.21.15.78:0/2677080467 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2638106370 0x7f263810df40 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f263000b0d0 tx=0x7f263000b3e0 comp rx=0 tx=0).stop 2024-03-24T00:20:08.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.509+0000 7f263de97700 1 -- 172.21.15.78:0/2677080467 shutdown_connections 2024-03-24T00:20:08.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.509+0000 7f263de97700 1 --2- 172.21.15.78:0/2677080467 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2638106370 0x7f263810df40 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:08.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.509+0000 7f263de97700 1 --2- 172.21.15.78:0/2677080467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2638101320 0x7f2638101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:08.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.509+0000 7f263de97700 1 --2- 172.21.15.78:0/2677080467 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2638101cd0 0x7f2638105c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:08.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.509+0000 7f263de97700 1 -- 172.21.15.78:0/2677080467 >> 172.21.15.78:0/2677080467 conn(0x7f26380770e0 msgr2=0x7f26380774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:08.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.509+0000 7f263de97700 1 -- 172.21.15.78:0/2677080467 shutdown_connections 2024-03-24T00:20:08.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.509+0000 7f263de97700 1 -- 172.21.15.78:0/2677080467 wait complete. 2024-03-24T00:20:08.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.510+0000 7f263de97700 1 Processor -- start 2024-03-24T00:20:08.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.510+0000 7f263de97700 1 -- start start 2024-03-24T00:20:08.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.510+0000 7f263de97700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2638101320 0x7f26381996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:08.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.510+0000 7f263de97700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2638101cd0 0x7f2638199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:08.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.510+0000 7f263de97700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2638106370 0x7f26381a4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:08.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.510+0000 7f263de97700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2638111380 con 0x7f2638101320 2024-03-24T00:20:08.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.510+0000 7f263de97700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2638111200 con 0x7f2638106370 2024-03-24T00:20:08.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.510+0000 7f263de97700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2638111500 con 0x7f2638101cd0 2024-03-24T00:20:08.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.510+0000 7f26377fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2638101320 0x7f26381996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:08.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.510+0000 7f2636ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2638101cd0 0x7f2638199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:08.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.510+0000 7f2637fff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2638106370 0x7f26381a4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:08.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.510+0000 7f26377fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2638101320 0x7f26381996e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52516/0 (socket says 172.21.15.78:52516) 2024-03-24T00:20:08.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.510+0000 7f26377fe700 1 -- 172.21.15.78:0/1069246014 learned_addr learned my addr 172.21.15.78:0/1069246014 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:08.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.510+0000 7f2637fff700 1 -- 172.21.15.78:0/1069246014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2638101cd0 msgr2=0x7f2638199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:08.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.510+0000 7f2637fff700 1 --2- 172.21.15.78:0/1069246014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2638101cd0 0x7f2638199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:08.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.510+0000 7f2637fff700 1 -- 172.21.15.78:0/1069246014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2638101320 msgr2=0x7f26381996e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:08.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.510+0000 7f2637fff700 1 --2- 172.21.15.78:0/1069246014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2638101320 0x7f26381996e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:08.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.510+0000 7f2637fff700 1 -- 172.21.15.78:0/1069246014 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26381a4ca0 con 0x7f2638106370 2024-03-24T00:20:08.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.511+0000 7f2637fff700 1 --2- 172.21.15.78:0/1069246014 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2638106370 0x7f26381a4760 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f263000b770 tx=0x7f2630007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:08.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.511+0000 7f2634ff9700 1 -- 172.21.15.78:0/1069246014 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2630021020 con 0x7f2638106370 2024-03-24T00:20:08.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.511+0000 7f2634ff9700 1 -- 172.21.15.78:0/1069246014 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2630004c50 con 0x7f2638106370 2024-03-24T00:20:08.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.511+0000 7f263de97700 1 -- 172.21.15.78:0/1069246014 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26381a4ed0 con 0x7f2638106370 2024-03-24T00:20:08.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.511+0000 7f2634ff9700 1 -- 172.21.15.78:0/1069246014 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26300052b0 con 0x7f2638106370 2024-03-24T00:20:08.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.511+0000 7f263de97700 1 -- 172.21.15.78:0/1069246014 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26381a5330 con 0x7f2638106370 2024-03-24T00:20:08.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.512+0000 7f263de97700 1 -- 172.21.15.78:0/1069246014 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2638068e50 con 0x7f2638106370 2024-03-24T00:20:08.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.513+0000 7f2634ff9700 1 -- 172.21.15.78:0/1069246014 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f2630024020 con 0x7f2638106370 2024-03-24T00:20:08.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.513+0000 7f2634ff9700 1 --2- 172.21.15.78:0/1069246014 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f26200a2920 0x7f26200a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:08.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.513+0000 7f26377fe700 1 --2- 172.21.15.78:0/1069246014 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f26200a2920 0x7f26200a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:08.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.513+0000 7f2634ff9700 1 -- 172.21.15.78:0/1069246014 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f26300cffb0 con 0x7f2638106370 2024-03-24T00:20:08.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.514+0000 7f26377fe700 1 --2- 172.21.15.78:0/1069246014 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f26200a2920 0x7f26200a4de0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2628009d60 tx=0x7f2628009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:08.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.518+0000 7f2634ff9700 1 -- 172.21.15.78:0/1069246014 <== mon.2 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f263002b020 con 0x7f2638106370 2024-03-24T00:20:08.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.718+0000 7f263de97700 1 -- 172.21.15.78:0/1069246014 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f263802da30 con 0x7f2638106370 2024-03-24T00:20:08.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.719+0000 7f2634ff9700 1 -- 172.21.15.78:0/1069246014 <== mon.2 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f263001de20 con 0x7f2638106370 2024-03-24T00:20:08.718 INFO:teuthology.orchestra.run.smithi078.stdout:55834574929 2024-03-24T00:20:08.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.721+0000 7f263de97700 1 -- 172.21.15.78:0/1069246014 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f26200a2920 msgr2=0x7f26200a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:08.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.721+0000 7f263de97700 1 --2- 172.21.15.78:0/1069246014 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f26200a2920 0x7f26200a4de0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2628009d60 tx=0x7f2628009260 comp rx=0 tx=0).stop 2024-03-24T00:20:08.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.721+0000 7f263de97700 1 -- 172.21.15.78:0/1069246014 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2638106370 msgr2=0x7f26381a4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:08.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.721+0000 7f263de97700 1 --2- 172.21.15.78:0/1069246014 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2638106370 0x7f26381a4760 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f263000b770 tx=0x7f2630007a20 comp rx=0 tx=0).stop 2024-03-24T00:20:08.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.721+0000 7f263de97700 1 -- 172.21.15.78:0/1069246014 shutdown_connections 2024-03-24T00:20:08.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.721+0000 7f263de97700 1 --2- 172.21.15.78:0/1069246014 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f26200a2920 0x7f26200a4de0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:08.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.721+0000 7f263de97700 1 --2- 172.21.15.78:0/1069246014 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2638106370 0x7f26381a4760 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:08.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.721+0000 7f263de97700 1 --2- 172.21.15.78:0/1069246014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2638101320 0x7f26381996e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:08.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.721+0000 7f263de97700 1 --2- 172.21.15.78:0/1069246014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2638101cd0 0x7f2638199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:08.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.721+0000 7f263de97700 1 -- 172.21.15.78:0/1069246014 >> 172.21.15.78:0/1069246014 conn(0x7f26380770e0 msgr2=0x7f26380ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:08.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.721+0000 7f263de97700 1 -- 172.21.15.78:0/1069246014 shutdown_connections 2024-03-24T00:20:08.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:08.721+0000 7f263de97700 1 -- 172.21.15.78:0/1069246014 wait complete. 2024-03-24T00:20:09.383 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808866 got 287762808872 for osd.9 2024-03-24T00:20:09.383 DEBUG:teuthology.parallel:result is None 2024-03-24T00:20:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:09 smithi106 ceph-mon[36505]: pgmap v335: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:20:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:09 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1069246014' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T00:20:09.726 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:09 smithi078 ceph-mon[32429]: pgmap v335: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:20:09.726 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:09 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1069246014' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T00:20:09.752 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574923 got 55834574929 for osd.1 2024-03-24T00:20:09.752 DEBUG:teuthology.parallel:result is None 2024-03-24T00:20:09.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:09 smithi107 ceph-mon[36485]: pgmap v335: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:20:09.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:09 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1069246014' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T00:20:10.508 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738448 got 34359738453 for osd.0 2024-03-24T00:20:10.508 DEBUG:teuthology.parallel:result is None 2024-03-24T00:20:11.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.120+0000 7f9ae8fd3700 1 -- 172.21.15.78:0/1019999735 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ae4106370 msgr2=0x7f9ae410df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:11.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.120+0000 7f9ae8fd3700 1 --2- 172.21.15.78:0/1019999735 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ae4106370 0x7f9ae410df40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9adc00b0d0 tx=0x7f9adc00b3e0 comp rx=0 tx=0).stop 2024-03-24T00:20:11.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.120+0000 7f9ae8fd3700 1 -- 172.21.15.78:0/1019999735 shutdown_connections 2024-03-24T00:20:11.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.120+0000 7f9ae8fd3700 1 --2- 172.21.15.78:0/1019999735 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ae4106370 0x7f9ae410df40 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:11.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.120+0000 7f9ae8fd3700 1 --2- 172.21.15.78:0/1019999735 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9ae4101320 0x7f9ae4101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:11.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.120+0000 7f9ae8fd3700 1 --2- 172.21.15.78:0/1019999735 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ae4101cd0 0x7f9ae4105c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:11.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.120+0000 7f9ae8fd3700 1 -- 172.21.15.78:0/1019999735 >> 172.21.15.78:0/1019999735 conn(0x7f9ae40770e0 msgr2=0x7f9ae40774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:11.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.120+0000 7f9ae8fd3700 1 -- 172.21.15.78:0/1019999735 shutdown_connections 2024-03-24T00:20:11.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.120+0000 7f9ae8fd3700 1 -- 172.21.15.78:0/1019999735 wait complete. 2024-03-24T00:20:11.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.121+0000 7f9ae8fd3700 1 Processor -- start 2024-03-24T00:20:11.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.121+0000 7f9ae8fd3700 1 -- start start 2024-03-24T00:20:11.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.121+0000 7f9ae8fd3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ae4101320 0x7f9ae419e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:11.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.121+0000 7f9ae8fd3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9ae4101cd0 0x7f9ae419eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:11.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.121+0000 7f9ae8fd3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ae4106370 0x7f9ae4198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:11.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.121+0000 7f9ae8fd3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae4111360 con 0x7f9ae4101320 2024-03-24T00:20:11.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.121+0000 7f9ae8fd3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae41111e0 con 0x7f9ae4101cd0 2024-03-24T00:20:11.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.121+0000 7f9ae8fd3700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae41114e0 con 0x7f9ae4106370 2024-03-24T00:20:11.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.121+0000 7f9ae259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ae4101320 0x7f9ae419e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:11.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.121+0000 7f9ae259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ae4101320 0x7f9ae419e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52524/0 (socket says 172.21.15.78:52524) 2024-03-24T00:20:11.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.121+0000 7f9ae2d9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ae4106370 0x7f9ae4198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:11.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.121+0000 7f9ae259c700 1 -- 172.21.15.78:0/1394623807 learned_addr learned my addr 172.21.15.78:0/1394623807 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:11.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.121+0000 7f9ae1d9b700 1 --2- 172.21.15.78:0/1394623807 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9ae4101cd0 0x7f9ae419eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:11.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.121+0000 7f9ae259c700 1 -- 172.21.15.78:0/1394623807 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ae4106370 msgr2=0x7f9ae4198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:11.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.121+0000 7f9ae259c700 1 --2- 172.21.15.78:0/1394623807 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ae4106370 0x7f9ae4198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:11.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.121+0000 7f9ae259c700 1 -- 172.21.15.78:0/1394623807 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9ae4101cd0 msgr2=0x7f9ae419eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:11.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.121+0000 7f9ae259c700 1 --2- 172.21.15.78:0/1394623807 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9ae4101cd0 0x7f9ae419eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:11.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.121+0000 7f9ae259c700 1 -- 172.21.15.78:0/1394623807 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae4198f50 con 0x7f9ae4101320 2024-03-24T00:20:11.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.122+0000 7f9ae259c700 1 --2- 172.21.15.78:0/1394623807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ae4101320 0x7f9ae419e5a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9ad400eb30 tx=0x7f9ad400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:11.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.122+0000 7f9ad37fe700 1 -- 172.21.15.78:0/1394623807 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad400cbd0 con 0x7f9ae4101320 2024-03-24T00:20:11.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.122+0000 7f9ae8fd3700 1 -- 172.21.15.78:0/1394623807 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ae4199240 con 0x7f9ae4101320 2024-03-24T00:20:11.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.122+0000 7f9ad37fe700 1 -- 172.21.15.78:0/1394623807 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ad4004500 con 0x7f9ae4101320 2024-03-24T00:20:11.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.122+0000 7f9ad37fe700 1 -- 172.21.15.78:0/1394623807 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad4010430 con 0x7f9ae4101320 2024-03-24T00:20:11.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.122+0000 7f9ae8fd3700 1 -- 172.21.15.78:0/1394623807 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ae41a5520 con 0x7f9ae4101320 2024-03-24T00:20:11.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.123+0000 7f9ae8fd3700 1 -- 172.21.15.78:0/1394623807 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ae4102770 con 0x7f9ae4101320 2024-03-24T00:20:11.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.125+0000 7f9ad37fe700 1 -- 172.21.15.78:0/1394623807 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f9ad4003f00 con 0x7f9ae4101320 2024-03-24T00:20:11.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.125+0000 7f9ad37fe700 1 --2- 172.21.15.78:0/1394623807 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f9acc0a29a0 0x7f9acc0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:11.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.125+0000 7f9ae1d9b700 1 --2- 172.21.15.78:0/1394623807 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f9acc0a29a0 0x7f9acc0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:11.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.125+0000 7f9ad37fe700 1 -- 172.21.15.78:0/1394623807 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f9ad40cf2b0 con 0x7f9ae4101320 2024-03-24T00:20:11.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.126+0000 7f9ae1d9b700 1 --2- 172.21.15.78:0/1394623807 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f9acc0a29a0 0x7f9acc0a4e60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8009680 tx=0x7f9ad8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:11.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.129+0000 7f9ad37fe700 1 -- 172.21.15.78:0/1394623807 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9ad401f340 con 0x7f9ae4101320 2024-03-24T00:20:11.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.334+0000 7f9ae8fd3700 1 -- 172.21.15.78:0/1394623807 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7f9ae41a56f0 con 0x7f9ae4101320 2024-03-24T00:20:11.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.335+0000 7f9ad37fe700 1 -- 172.21.15.78:0/1394623807 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f9ad409bf20 con 0x7f9ae4101320 2024-03-24T00:20:11.335 INFO:teuthology.orchestra.run.smithi078.stdout:309237645348 2024-03-24T00:20:11.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.337+0000 7f9ae8fd3700 1 -- 172.21.15.78:0/1394623807 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f9acc0a29a0 msgr2=0x7f9acc0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:11.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.337+0000 7f9ae8fd3700 1 --2- 172.21.15.78:0/1394623807 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f9acc0a29a0 0x7f9acc0a4e60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8009680 tx=0x7f9ad8009260 comp rx=0 tx=0).stop 2024-03-24T00:20:11.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.337+0000 7f9ae8fd3700 1 -- 172.21.15.78:0/1394623807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ae4101320 msgr2=0x7f9ae419e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:11.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.337+0000 7f9ae8fd3700 1 --2- 172.21.15.78:0/1394623807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ae4101320 0x7f9ae419e5a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9ad400eb30 tx=0x7f9ad400ee40 comp rx=0 tx=0).stop 2024-03-24T00:20:11.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.337+0000 7f9ae8fd3700 1 -- 172.21.15.78:0/1394623807 shutdown_connections 2024-03-24T00:20:11.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.337+0000 7f9ae8fd3700 1 --2- 172.21.15.78:0/1394623807 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f9acc0a29a0 0x7f9acc0a4e60 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:11.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.337+0000 7f9ae8fd3700 1 --2- 172.21.15.78:0/1394623807 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ae4106370 0x7f9ae4198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:11.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.337+0000 7f9ae8fd3700 1 --2- 172.21.15.78:0/1394623807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ae4101320 0x7f9ae419e5a0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:11.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.337+0000 7f9ae8fd3700 1 --2- 172.21.15.78:0/1394623807 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9ae4101cd0 0x7f9ae419eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:11.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.337+0000 7f9ae8fd3700 1 -- 172.21.15.78:0/1394623807 >> 172.21.15.78:0/1394623807 conn(0x7f9ae40770e0 msgr2=0x7f9ae4104ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:11.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.337+0000 7f9ae8fd3700 1 -- 172.21.15.78:0/1394623807 shutdown_connections 2024-03-24T00:20:11.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:11.337+0000 7f9ae8fd3700 1 -- 172.21.15.78:0/1394623807 wait complete. 2024-03-24T00:20:11.639 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:11 smithi078 ceph-mon[32429]: pgmap v336: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:20:11.639 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:11 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1394623807' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-24T00:20:11.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:11 smithi106 ceph-mon[36505]: pgmap v336: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:20:11.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:11 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1394623807' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-24T00:20:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:11 smithi107 ceph-mon[36485]: pgmap v336: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:20:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:11 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1394623807' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-24T00:20:11.847 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:20:12.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.735+0000 7fd4b3c07700 1 -- 172.21.15.78:0/687779752 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4ac102ba0 msgr2=0x7fd4ac10efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:12.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.735+0000 7fd4b3c07700 1 --2- 172.21.15.78:0/687779752 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4ac102ba0 0x7fd4ac10efe0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd4a400b0d0 tx=0x7fd4a400b3e0 comp rx=0 tx=0).stop 2024-03-24T00:20:12.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.735+0000 7fd4b3c07700 1 -- 172.21.15.78:0/687779752 shutdown_connections 2024-03-24T00:20:12.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.735+0000 7fd4b3c07700 1 --2- 172.21.15.78:0/687779752 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4ac102ba0 0x7fd4ac10efe0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:12.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.735+0000 7fd4b3c07700 1 --2- 172.21.15.78:0/687779752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4ac1081c0 0x7fd4ac1085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:12.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.735+0000 7fd4b3c07700 1 --2- 172.21.15.78:0/687779752 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4ac1021e0 0x7fd4ac102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:12.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.735+0000 7fd4b3c07700 1 -- 172.21.15.78:0/687779752 >> 172.21.15.78:0/687779752 conn(0x7fd4ac0792c0 msgr2=0x7fd4ac0796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:12.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.735+0000 7fd4b3c07700 1 -- 172.21.15.78:0/687779752 shutdown_connections 2024-03-24T00:20:12.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.735+0000 7fd4b3c07700 1 -- 172.21.15.78:0/687779752 wait complete. 2024-03-24T00:20:12.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.736+0000 7fd4b3c07700 1 Processor -- start 2024-03-24T00:20:12.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.736+0000 7fd4b3c07700 1 -- start start 2024-03-24T00:20:12.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.736+0000 7fd4b3c07700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4ac1021e0 0x7fd4ac19e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:12.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.736+0000 7fd4b3c07700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4ac102ba0 0x7fd4ac19eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:12.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.736+0000 7fd4b3c07700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4ac1081c0 0x7fd4ac1986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:12.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.736+0000 7fd4b3c07700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac112460 con 0x7fd4ac102ba0 2024-03-24T00:20:12.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.736+0000 7fd4b3c07700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac1122e0 con 0x7fd4ac1081c0 2024-03-24T00:20:12.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.736+0000 7fd4b3c07700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac1125e0 con 0x7fd4ac1021e0 2024-03-24T00:20:12.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.736+0000 7fd4b11a2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4ac102ba0 0x7fd4ac19eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:12.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.736+0000 7fd4b11a2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4ac102ba0 0x7fd4ac19eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52542/0 (socket says 172.21.15.78:52542) 2024-03-24T00:20:12.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.736+0000 7fd4b11a2700 1 -- 172.21.15.78:0/2629515229 learned_addr learned my addr 172.21.15.78:0/2629515229 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:12.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.736+0000 7fd4b19a3700 1 --2- 172.21.15.78:0/2629515229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4ac1021e0 0x7fd4ac19e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:12.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.736+0000 7fd4b21a4700 1 --2- 172.21.15.78:0/2629515229 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4ac1081c0 0x7fd4ac1986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:12.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.736+0000 7fd4b19a3700 1 -- 172.21.15.78:0/2629515229 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4ac1081c0 msgr2=0x7fd4ac1986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:12.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.736+0000 7fd4b19a3700 1 --2- 172.21.15.78:0/2629515229 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4ac1081c0 0x7fd4ac1986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:12.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.736+0000 7fd4b19a3700 1 -- 172.21.15.78:0/2629515229 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4ac102ba0 msgr2=0x7fd4ac19eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:12.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.736+0000 7fd4b19a3700 1 --2- 172.21.15.78:0/2629515229 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4ac102ba0 0x7fd4ac19eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:12.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.736+0000 7fd4b19a3700 1 -- 172.21.15.78:0/2629515229 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ac198f80 con 0x7fd4ac1021e0 2024-03-24T00:20:12.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.736+0000 7fd4b11a2700 1 --2- 172.21.15.78:0/2629515229 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4ac102ba0 0x7fd4ac19eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:12.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.736+0000 7fd4b21a4700 1 --2- 172.21.15.78:0/2629515229 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4ac1081c0 0x7fd4ac1986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:12.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.736+0000 7fd4b19a3700 1 --2- 172.21.15.78:0/2629515229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4ac1021e0 0x7fd4ac19e580 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd4a800e9f0 tx=0x7fd4a800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:12.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.737+0000 7fd4a2ffd700 1 -- 172.21.15.78:0/2629515229 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a8004220 con 0x7fd4ac1021e0 2024-03-24T00:20:12.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.737+0000 7fd4b3c07700 1 -- 172.21.15.78:0/2629515229 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4ac199270 con 0x7fd4ac1021e0 2024-03-24T00:20:12.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.737+0000 7fd4b3c07700 1 -- 172.21.15.78:0/2629515229 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4ac1a5500 con 0x7fd4ac1021e0 2024-03-24T00:20:12.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.737+0000 7fd4a2ffd700 1 -- 172.21.15.78:0/2629515229 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd4a8004d10 con 0x7fd4ac1021e0 2024-03-24T00:20:12.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.737+0000 7fd4a2ffd700 1 -- 172.21.15.78:0/2629515229 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a8005020 con 0x7fd4ac1021e0 2024-03-24T00:20:12.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.737+0000 7fd4b3c07700 1 -- 172.21.15.78:0/2629515229 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4ac103870 con 0x7fd4ac1021e0 2024-03-24T00:20:12.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.739+0000 7fd4a2ffd700 1 -- 172.21.15.78:0/2629515229 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fd4a8004750 con 0x7fd4ac1021e0 2024-03-24T00:20:12.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.739+0000 7fd4a2ffd700 1 --2- 172.21.15.78:0/2629515229 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd4980a2920 0x7fd4980a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:12.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.739+0000 7fd4b11a2700 1 --2- 172.21.15.78:0/2629515229 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd4980a2920 0x7fd4980a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:12.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.739+0000 7fd4a2ffd700 1 -- 172.21.15.78:0/2629515229 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fd4a80cede0 con 0x7fd4ac1021e0 2024-03-24T00:20:12.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.740+0000 7fd4b11a2700 1 --2- 172.21.15.78:0/2629515229 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd4980a2920 0x7fd4980a4de0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac199e10 tx=0x7fd49c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:12.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.744+0000 7fd4a2ffd700 1 -- 172.21.15.78:0/2629515229 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd4a809b9c0 con 0x7fd4ac1021e0 2024-03-24T00:20:12.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.918+0000 7fcfe1afd700 1 -- 172.21.15.78:0/3569879272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfdc101a30 msgr2=0x7fcfdc10df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:12.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.918+0000 7fcfe1afd700 1 --2- 172.21.15.78:0/3569879272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfdc101a30 0x7fcfdc10df10 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcfd400b0a0 tx=0x7fcfd400b3b0 comp rx=0 tx=0).stop 2024-03-24T00:20:12.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.918+0000 7fcfe1afd700 1 -- 172.21.15.78:0/3569879272 shutdown_connections 2024-03-24T00:20:12.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.918+0000 7fcfe1afd700 1 --2- 172.21.15.78:0/3569879272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfdc101a30 0x7fcfdc10df10 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:12.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.918+0000 7fcfe1afd700 1 --2- 172.21.15.78:0/3569879272 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfdc1070d0 0x7fcfdc1074b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:12.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.918+0000 7fcfe1afd700 1 --2- 172.21.15.78:0/3569879272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfdc101090 0x7fcfdc1014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:12.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.918+0000 7fcfe1afd700 1 -- 172.21.15.78:0/3569879272 >> 172.21.15.78:0/3569879272 conn(0x7fcfdc077fc0 msgr2=0x7fcfdc0783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:12.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.918+0000 7fcfe1afd700 1 -- 172.21.15.78:0/3569879272 shutdown_connections 2024-03-24T00:20:12.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.918+0000 7fcfe1afd700 1 -- 172.21.15.78:0/3569879272 wait complete. 2024-03-24T00:20:12.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.919+0000 7fcfe1afd700 1 Processor -- start 2024-03-24T00:20:12.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.919+0000 7fcfe1afd700 1 -- start start 2024-03-24T00:20:12.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.919+0000 7fcfe1afd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfdc101090 0x7fcfdc19e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:12.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.919+0000 7fcfe1afd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfdc101a30 0x7fcfdc19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:12.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.919+0000 7fcfe1afd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfdc1070d0 0x7fcfdc198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:12.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.919+0000 7fcfe1afd700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfdc111350 con 0x7fcfdc1070d0 2024-03-24T00:20:12.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.919+0000 7fcfe1afd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfdc1111d0 con 0x7fcfdc101a30 2024-03-24T00:20:12.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.919+0000 7fcfe1afd700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfdc1114d0 con 0x7fcfdc101090 2024-03-24T00:20:12.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.919+0000 7fcfdbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfdc1070d0 0x7fcfdc198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:12.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.919+0000 7fcfdb7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfdc101090 0x7fcfdc19e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:12.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.919+0000 7fcfdaffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfdc101a30 0x7fcfdc19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:12.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.920+0000 7fcfdbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfdc1070d0 0x7fcfdc198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52556/0 (socket says 172.21.15.78:52556) 2024-03-24T00:20:12.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.920+0000 7fcfdbfff700 1 -- 172.21.15.78:0/667295005 learned_addr learned my addr 172.21.15.78:0/667295005 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:12.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.920+0000 7fcfdbfff700 1 -- 172.21.15.78:0/667295005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfdc101090 msgr2=0x7fcfdc19e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:12.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.920+0000 7fcfdbfff700 1 --2- 172.21.15.78:0/667295005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfdc101090 0x7fcfdc19e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:12.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.920+0000 7fcfdbfff700 1 -- 172.21.15.78:0/667295005 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfdc101a30 msgr2=0x7fcfdc19eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:12.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.920+0000 7fcfdbfff700 1 --2- 172.21.15.78:0/667295005 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfdc101a30 0x7fcfdc19eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:12.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.920+0000 7fcfdbfff700 1 -- 172.21.15.78:0/667295005 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfdc199000 con 0x7fcfdc1070d0 2024-03-24T00:20:12.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.920+0000 7fcfdbfff700 1 --2- 172.21.15.78:0/667295005 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfdc1070d0 0x7fcfdc198720 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fcfd400b780 tx=0x7fcfd4004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:12.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.920+0000 7fcfd8ff9700 1 -- 172.21.15.78:0/667295005 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd4021070 con 0x7fcfdc1070d0 2024-03-24T00:20:12.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.920+0000 7fcfe1afd700 1 -- 172.21.15.78:0/667295005 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfdc199290 con 0x7fcfdc1070d0 2024-03-24T00:20:12.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.920+0000 7fcfd8ff9700 1 -- 172.21.15.78:0/667295005 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfd4007b80 con 0x7fcfdc1070d0 2024-03-24T00:20:12.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.920+0000 7fcfd8ff9700 1 -- 172.21.15.78:0/667295005 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd40052a0 con 0x7fcfdc1070d0 2024-03-24T00:20:12.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.920+0000 7fcfe1afd700 1 -- 172.21.15.78:0/667295005 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfdc1a55d0 con 0x7fcfdc1070d0 2024-03-24T00:20:12.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.921+0000 7fcfe1afd700 1 -- 172.21.15.78:0/667295005 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfdc199600 con 0x7fcfdc1070d0 2024-03-24T00:20:12.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.923+0000 7fcfd8ff9700 1 -- 172.21.15.78:0/667295005 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fcfd4024070 con 0x7fcfdc1070d0 2024-03-24T00:20:12.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.924+0000 7fcfd8ff9700 1 --2- 172.21.15.78:0/667295005 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fcfc40a2920 0x7fcfc40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:12.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.924+0000 7fcfdb7fe700 1 --2- 172.21.15.78:0/667295005 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fcfc40a2920 0x7fcfc40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:12.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.924+0000 7fcfd8ff9700 1 -- 172.21.15.78:0/667295005 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fcfd400f020 con 0x7fcfdc1070d0 2024-03-24T00:20:12.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.924+0000 7fcfdb7fe700 1 --2- 172.21.15.78:0/667295005 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fcfc40a2920 0x7fcfc40a4de0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0009be0 tx=0x7fcfd0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:12.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.927+0000 7fcfd8ff9700 1 -- 172.21.15.78:0/667295005 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcfdc199600 con 0x7fcfdc1070d0 2024-03-24T00:20:12.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.948+0000 7fd4b3c07700 1 -- 172.21.15.78:0/2629515229 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fd4ac02da30 con 0x7fd4ac1021e0 2024-03-24T00:20:12.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.948+0000 7fd4a2ffd700 1 -- 172.21.15.78:0/2629515229 <== mon.1 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fd4a809b2c0 con 0x7fd4ac1021e0 2024-03-24T00:20:12.948 INFO:teuthology.orchestra.run.smithi078.stdout:163208757310 2024-03-24T00:20:12.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.951+0000 7fd4b3c07700 1 -- 172.21.15.78:0/2629515229 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd4980a2920 msgr2=0x7fd4980a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:12.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.951+0000 7fd4b3c07700 1 --2- 172.21.15.78:0/2629515229 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd4980a2920 0x7fd4980a4de0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac199e10 tx=0x7fd49c009260 comp rx=0 tx=0).stop 2024-03-24T00:20:12.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.951+0000 7fd4b3c07700 1 -- 172.21.15.78:0/2629515229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4ac1021e0 msgr2=0x7fd4ac19e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:12.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.951+0000 7fd4b3c07700 1 --2- 172.21.15.78:0/2629515229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4ac1021e0 0x7fd4ac19e580 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd4a800e9f0 tx=0x7fd4a800ed00 comp rx=0 tx=0).stop 2024-03-24T00:20:12.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.951+0000 7fd4b3c07700 1 -- 172.21.15.78:0/2629515229 shutdown_connections 2024-03-24T00:20:12.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.951+0000 7fd4b3c07700 1 --2- 172.21.15.78:0/2629515229 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd4980a2920 0x7fd4980a4de0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:12.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.951+0000 7fd4b3c07700 1 --2- 172.21.15.78:0/2629515229 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4ac1081c0 0x7fd4ac1986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:12.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.951+0000 7fd4b3c07700 1 --2- 172.21.15.78:0/2629515229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4ac1021e0 0x7fd4ac19e580 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:12.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.951+0000 7fd4b3c07700 1 --2- 172.21.15.78:0/2629515229 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4ac102ba0 0x7fd4ac19eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:12.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.951+0000 7fd4b3c07700 1 -- 172.21.15.78:0/2629515229 >> 172.21.15.78:0/2629515229 conn(0x7fd4ac0792c0 msgr2=0x7fd4ac100de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:12.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.951+0000 7fd4b3c07700 1 -- 172.21.15.78:0/2629515229 shutdown_connections 2024-03-24T00:20:12.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:12.951+0000 7fd4b3c07700 1 -- 172.21.15.78:0/2629515229 wait complete. 2024-03-24T00:20:13.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.136+0000 7fcfe1afd700 1 -- 172.21.15.78:0/667295005 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fcfdc1a57a0 con 0x7fcfdc1070d0 2024-03-24T00:20:13.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.137+0000 7fcfd8ff9700 1 -- 172.21.15.78:0/667295005 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fcfd402b020 con 0x7fcfdc1070d0 2024-03-24T00:20:13.137 INFO:teuthology.orchestra.run.smithi078.stdout:107374182471 2024-03-24T00:20:13.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.139+0000 7fcfe1afd700 1 -- 172.21.15.78:0/667295005 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fcfc40a2920 msgr2=0x7fcfc40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:13.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.139+0000 7fcfe1afd700 1 --2- 172.21.15.78:0/667295005 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fcfc40a2920 0x7fcfc40a4de0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0009be0 tx=0x7fcfd0009260 comp rx=0 tx=0).stop 2024-03-24T00:20:13.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.139+0000 7fcfe1afd700 1 -- 172.21.15.78:0/667295005 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfdc1070d0 msgr2=0x7fcfdc198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:13.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.139+0000 7fcfe1afd700 1 --2- 172.21.15.78:0/667295005 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfdc1070d0 0x7fcfdc198720 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fcfd400b780 tx=0x7fcfd4004d00 comp rx=0 tx=0).stop 2024-03-24T00:20:13.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.140+0000 7fcfe1afd700 1 -- 172.21.15.78:0/667295005 shutdown_connections 2024-03-24T00:20:13.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.140+0000 7fcfe1afd700 1 --2- 172.21.15.78:0/667295005 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fcfc40a2920 0x7fcfc40a4de0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:13.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.140+0000 7fcfe1afd700 1 --2- 172.21.15.78:0/667295005 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfdc1070d0 0x7fcfdc198720 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:13.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.140+0000 7fcfe1afd700 1 --2- 172.21.15.78:0/667295005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfdc101090 0x7fcfdc19e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:13.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.140+0000 7fcfe1afd700 1 --2- 172.21.15.78:0/667295005 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfdc101a30 0x7fcfdc19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:13.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.140+0000 7fcfe1afd700 1 -- 172.21.15.78:0/667295005 >> 172.21.15.78:0/667295005 conn(0x7fcfdc077fc0 msgr2=0x7fcfdc0fe980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:13.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.140+0000 7fcfe1afd700 1 -- 172.21.15.78:0/667295005 shutdown_connections 2024-03-24T00:20:13.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.140+0000 7fcfe1afd700 1 -- 172.21.15.78:0/667295005 wait complete. 2024-03-24T00:20:13.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.137+0000 7f45a3b3c700 1 -- 172.21.15.78:0/906474895 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f459c101090 msgr2=0x7f459c1014f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:13.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.137+0000 7f45a3b3c700 1 --2- 172.21.15.78:0/906474895 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f459c101090 0x7f459c1014f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f458c009990 tx=0x7f458c009ca0 comp rx=0 tx=0).stop 2024-03-24T00:20:13.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.137+0000 7f45a3b3c700 1 -- 172.21.15.78:0/906474895 shutdown_connections 2024-03-24T00:20:13.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.137+0000 7f45a3b3c700 1 --2- 172.21.15.78:0/906474895 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f459c101a30 0x7f459c10df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:13.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.137+0000 7f45a3b3c700 1 --2- 172.21.15.78:0/906474895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f459c1070d0 0x7f459c1074b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:13.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.137+0000 7f45a3b3c700 1 --2- 172.21.15.78:0/906474895 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f459c101090 0x7f459c1014f0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:13.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.137+0000 7f45a3b3c700 1 -- 172.21.15.78:0/906474895 >> 172.21.15.78:0/906474895 conn(0x7f459c077fc0 msgr2=0x7f459c0783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:13.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.138+0000 7f45a3b3c700 1 -- 172.21.15.78:0/906474895 shutdown_connections 2024-03-24T00:20:13.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.138+0000 7f45a3b3c700 1 -- 172.21.15.78:0/906474895 wait complete. 2024-03-24T00:20:13.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.138+0000 7f45a3b3c700 1 Processor -- start 2024-03-24T00:20:13.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.138+0000 7f45a3b3c700 1 -- start start 2024-03-24T00:20:13.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.138+0000 7f45a3b3c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f459c101090 0x7f459c19e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:13.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.138+0000 7f45a3b3c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f459c101a30 0x7f459c19eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:13.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.138+0000 7f45a3b3c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f459c1070d0 0x7f459c198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:13.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.138+0000 7f45a3b3c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459c111380 con 0x7f459c1070d0 2024-03-24T00:20:13.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.138+0000 7f45a3b3c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459c111200 con 0x7f459c101a30 2024-03-24T00:20:13.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.138+0000 7f45a3b3c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459c111500 con 0x7f459c101090 2024-03-24T00:20:13.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.139+0000 7f45a18d8700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f459c101090 0x7f459c19e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:13.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.139+0000 7f45a10d7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f459c101a30 0x7f459c19eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:13.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.139+0000 7f45a20d9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f459c1070d0 0x7f459c198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:13.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.139+0000 7f45a10d7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f459c101a30 0x7f459c19eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.78:46006/0 (socket says 172.21.15.78:46006) 2024-03-24T00:20:13.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.139+0000 7f45a18d8700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f459c101090 0x7f459c19e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.78:40426/0 (socket says 172.21.15.78:40426) 2024-03-24T00:20:13.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.139+0000 7f45a10d7700 1 -- 172.21.15.78:0/1212366158 learned_addr learned my addr 172.21.15.78:0/1212366158 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:13.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.139+0000 7f45a10d7700 1 -- 172.21.15.78:0/1212366158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f459c101090 msgr2=0x7f459c19e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:13.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.139+0000 7f45a10d7700 1 --2- 172.21.15.78:0/1212366158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f459c101090 0x7f459c19e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:13.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.139+0000 7f45a10d7700 1 -- 172.21.15.78:0/1212366158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f459c1070d0 msgr2=0x7f459c198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:13.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.139+0000 7f45a10d7700 1 --2- 172.21.15.78:0/1212366158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f459c1070d0 0x7f459c198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:13.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.139+0000 7f45a10d7700 1 -- 172.21.15.78:0/1212366158 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459c198f70 con 0x7f459c101a30 2024-03-24T00:20:13.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.139+0000 7f45a10d7700 1 --2- 172.21.15.78:0/1212366158 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f459c101a30 0x7f459c19eb50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f458c000c00 tx=0x7f458c00bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:13.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.139+0000 7f4592ffd700 1 -- 172.21.15.78:0/1212366158 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458c02b070 con 0x7f459c101a30 2024-03-24T00:20:13.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.139+0000 7f4592ffd700 1 -- 172.21.15.78:0/1212366158 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f458c0056d0 con 0x7f459c101a30 2024-03-24T00:20:13.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.139+0000 7f45a3b3c700 1 -- 172.21.15.78:0/1212366158 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f459c199200 con 0x7f459c101a30 2024-03-24T00:20:13.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.139+0000 7f4592ffd700 1 -- 172.21.15.78:0/1212366158 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458c0266d0 con 0x7f459c101a30 2024-03-24T00:20:13.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.139+0000 7f45a3b3c700 1 -- 172.21.15.78:0/1212366158 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f459c107ee0 con 0x7f459c101a30 2024-03-24T00:20:13.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.140+0000 7f45a3b3c700 1 -- 172.21.15.78:0/1212366158 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f459c102780 con 0x7f459c101a30 2024-03-24T00:20:13.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.142+0000 7f4592ffd700 1 -- 172.21.15.78:0/1212366158 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f458c02e070 con 0x7f459c101a30 2024-03-24T00:20:13.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.142+0000 7f4592ffd700 1 --2- 172.21.15.78:0/1212366158 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f45880a2920 0x7f45880a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:13.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.143+0000 7f45a18d8700 1 --2- 172.21.15.78:0/1212366158 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f45880a2920 0x7f45880a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:13.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.143+0000 7f4592ffd700 1 -- 172.21.15.78:0/1212366158 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f458c0d45e0 con 0x7f459c101a30 2024-03-24T00:20:13.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.143+0000 7f45a18d8700 1 --2- 172.21.15.78:0/1212366158 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f45880a2920 0x7f45880a4de0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4598009be0 tx=0x7f4598009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:13.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.145+0000 7f4592ffd700 1 -- 172.21.15.78:0/1212366158 <== mon.2 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f458c013070 con 0x7f459c101a30 2024-03-24T00:20:13.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.347+0000 7f45a3b3c700 1 -- 172.21.15.78:0/1212366158 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f459c199cc0 con 0x7f459c101a30 2024-03-24T00:20:13.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.348+0000 7f4592ffd700 1 -- 172.21.15.78:0/1212366158 <== mon.2 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f458c0a1250 con 0x7f459c101a30 2024-03-24T00:20:13.348 INFO:teuthology.orchestra.run.smithi078.stdout:219043332148 2024-03-24T00:20:13.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.350+0000 7f45a3b3c700 1 -- 172.21.15.78:0/1212366158 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f45880a2920 msgr2=0x7f45880a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:13.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.350+0000 7f45a3b3c700 1 --2- 172.21.15.78:0/1212366158 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f45880a2920 0x7f45880a4de0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4598009be0 tx=0x7f4598009260 comp rx=0 tx=0).stop 2024-03-24T00:20:13.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.350+0000 7f45a3b3c700 1 -- 172.21.15.78:0/1212366158 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f459c101a30 msgr2=0x7f459c19eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:13.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.350+0000 7f45a3b3c700 1 --2- 172.21.15.78:0/1212366158 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f459c101a30 0x7f459c19eb50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f458c000c00 tx=0x7f458c00bab0 comp rx=0 tx=0).stop 2024-03-24T00:20:13.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.350+0000 7f45a3b3c700 1 -- 172.21.15.78:0/1212366158 shutdown_connections 2024-03-24T00:20:13.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.350+0000 7f45a3b3c700 1 --2- 172.21.15.78:0/1212366158 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f45880a2920 0x7f45880a4de0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:13.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.350+0000 7f45a3b3c700 1 --2- 172.21.15.78:0/1212366158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f459c1070d0 0x7f459c198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:13.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.350+0000 7f45a3b3c700 1 --2- 172.21.15.78:0/1212366158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f459c101090 0x7f459c19e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:13.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.350+0000 7f45a3b3c700 1 --2- 172.21.15.78:0/1212366158 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f459c101a30 0x7f459c19eb50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:13.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.350+0000 7f45a3b3c700 1 -- 172.21.15.78:0/1212366158 >> 172.21.15.78:0/1212366158 conn(0x7f459c077fc0 msgr2=0x7f459c0fea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:13.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.350+0000 7f45a3b3c700 1 -- 172.21.15.78:0/1212366158 shutdown_connections 2024-03-24T00:20:13.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:13.350+0000 7f45a3b3c700 1 -- 172.21.15.78:0/1212366158 wait complete. 2024-03-24T00:20:13.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:13 smithi106 ceph-mon[36505]: pgmap v337: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:20:13.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:13 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2629515229' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T00:20:13.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:13 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/667295005' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T00:20:13.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:13 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1212366158' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T00:20:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:13 smithi078 ceph-mon[32429]: pgmap v337: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:20:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:13 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2629515229' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T00:20:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:13 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/667295005' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T00:20:13.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:13 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1212366158' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T00:20:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:13 smithi107 ceph-mon[36485]: pgmap v337: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:20:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:13 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2629515229' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T00:20:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:13 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/667295005' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T00:20:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:13 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1212366158' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T00:20:13.884 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182466 got 107374182471 for osd.3 2024-03-24T00:20:13.884 DEBUG:teuthology.parallel:result is None 2024-03-24T00:20:14.932 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645341 got 309237645348 for osd.10 2024-03-24T00:20:14.932 DEBUG:teuthology.parallel:result is None 2024-03-24T00:20:15.474 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:15 smithi078 ceph-mon[32429]: pgmap v338: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:20:15.502 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332144 got 219043332148 for osd.7 2024-03-24T00:20:15.502 DEBUG:teuthology.parallel:result is None 2024-03-24T00:20:15.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:15 smithi106 ceph-mon[36505]: pgmap v338: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:20:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:15 smithi107 ceph-mon[36485]: pgmap v338: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:20:16.620 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757305 got 163208757310 for osd.5 2024-03-24T00:20:16.620 DEBUG:teuthology.parallel:result is None 2024-03-24T00:20:17.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.026+0000 7f2f3e65f700 1 -- 172.21.15.78:0/1304681008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f38102bd0 msgr2=0x7f2f3810f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:17.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.026+0000 7f2f3e65f700 1 --2- 172.21.15.78:0/1304681008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f38102bd0 0x7f2f3810f040 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2f3000b0d0 tx=0x7f2f3000b3e0 comp rx=0 tx=0).stop 2024-03-24T00:20:17.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.026+0000 7f2f3e65f700 1 -- 172.21.15.78:0/1304681008 shutdown_connections 2024-03-24T00:20:17.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.026+0000 7f2f3e65f700 1 --2- 172.21.15.78:0/1304681008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f38102bd0 0x7f2f3810f040 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:17.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.026+0000 7f2f3e65f700 1 --2- 172.21.15.78:0/1304681008 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2f38108220 0x7f2f38108600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:17.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.026+0000 7f2f3e65f700 1 --2- 172.21.15.78:0/1304681008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f38102210 0x7f2f38102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:17.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.026+0000 7f2f3e65f700 1 -- 172.21.15.78:0/1304681008 >> 172.21.15.78:0/1304681008 conn(0x7f2f380792f0 msgr2=0x7f2f38079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:17.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.026+0000 7f2f3e65f700 1 -- 172.21.15.78:0/1304681008 shutdown_connections 2024-03-24T00:20:17.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.026+0000 7f2f3e65f700 1 -- 172.21.15.78:0/1304681008 wait complete. 2024-03-24T00:20:17.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.027+0000 7f2f3e65f700 1 Processor -- start 2024-03-24T00:20:17.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.027+0000 7f2f3e65f700 1 -- start start 2024-03-24T00:20:17.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.027+0000 7f2f3e65f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f38102210 0x7f2f3819e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:17.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.027+0000 7f2f3e65f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2f38102bd0 0x7f2f3819eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:17.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.027+0000 7f2f3e65f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f38108220 0x7f2f38198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:17.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.027+0000 7f2f3e65f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f381124d0 con 0x7f2f38102210 2024-03-24T00:20:17.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.027+0000 7f2f3e65f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f38112350 con 0x7f2f38102bd0 2024-03-24T00:20:17.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.027+0000 7f2f3e65f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f38112650 con 0x7f2f38108220 2024-03-24T00:20:17.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.027+0000 7f2f37fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f38102210 0x7f2f3819e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:17.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.027+0000 7f2f37fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f38102210 0x7f2f3819e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41468/0 (socket says 172.21.15.78:41468) 2024-03-24T00:20:17.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.027+0000 7f2f37fff700 1 -- 172.21.15.78:0/1933530027 learned_addr learned my addr 172.21.15.78:0/1933530027 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:17.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.027+0000 7f2f377fe700 1 --2- 172.21.15.78:0/1933530027 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2f38102bd0 0x7f2f3819eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:17.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.027+0000 7f2f3cbfc700 1 --2- 172.21.15.78:0/1933530027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f38108220 0x7f2f38198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:17.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.027+0000 7f2f37fff700 1 -- 172.21.15.78:0/1933530027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f38108220 msgr2=0x7f2f38198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:17.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.027+0000 7f2f37fff700 1 --2- 172.21.15.78:0/1933530027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f38108220 0x7f2f38198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:17.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.027+0000 7f2f37fff700 1 -- 172.21.15.78:0/1933530027 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2f38102bd0 msgr2=0x7f2f3819eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:17.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.027+0000 7f2f37fff700 1 --2- 172.21.15.78:0/1933530027 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2f38102bd0 0x7f2f3819eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:17.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.027+0000 7f2f37fff700 1 -- 172.21.15.78:0/1933530027 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f38198f30 con 0x7f2f38102210 2024-03-24T00:20:17.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.027+0000 7f2f377fe700 1 --2- 172.21.15.78:0/1933530027 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2f38102bd0 0x7f2f3819eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:17.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.028+0000 7f2f37fff700 1 --2- 172.21.15.78:0/1933530027 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f38102210 0x7f2f3819e580 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2f2800e9a0 tx=0x7f2f2800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:17.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.028+0000 7f2f357fa700 1 -- 172.21.15.78:0/1933530027 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2800ca10 con 0x7f2f38102210 2024-03-24T00:20:17.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.028+0000 7f2f3e65f700 1 -- 172.21.15.78:0/1933530027 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f38199220 con 0x7f2f38102210 2024-03-24T00:20:17.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.028+0000 7f2f357fa700 1 -- 172.21.15.78:0/1933530027 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2f2800cbb0 con 0x7f2f38102210 2024-03-24T00:20:17.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.028+0000 7f2f357fa700 1 -- 172.21.15.78:0/1933530027 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f28018680 con 0x7f2f38102210 2024-03-24T00:20:17.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.028+0000 7f2f3e65f700 1 -- 172.21.15.78:0/1933530027 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f381a5500 con 0x7f2f38102210 2024-03-24T00:20:17.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.029+0000 7f2f3e65f700 1 -- 172.21.15.78:0/1933530027 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f381038d0 con 0x7f2f38102210 2024-03-24T00:20:17.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.030+0000 7f2f357fa700 1 -- 172.21.15.78:0/1933530027 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f2f28010430 con 0x7f2f38102210 2024-03-24T00:20:17.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.030+0000 7f2f357fa700 1 --2- 172.21.15.78:0/1933530027 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2f200a2920 0x7f2f200a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:17.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.031+0000 7f2f377fe700 1 --2- 172.21.15.78:0/1933530027 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2f200a2920 0x7f2f200a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:17.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.031+0000 7f2f357fa700 1 -- 172.21.15.78:0/1933530027 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f2f280cf620 con 0x7f2f38102210 2024-03-24T00:20:17.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.031+0000 7f2f377fe700 1 --2- 172.21.15.78:0/1933530027 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2f200a2920 0x7f2f200a4de0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2f38199e10 tx=0x7f2f2c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:17.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.034+0000 7f2f357fa700 1 -- 172.21.15.78:0/1933530027 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2f2809c200 con 0x7f2f38102210 2024-03-24T00:20:17.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.260+0000 7f2f3e65f700 1 -- 172.21.15.78:0/1933530027 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f2f381a56d0 con 0x7f2f38102210 2024-03-24T00:20:17.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.260+0000 7f2f357fa700 1 -- 172.21.15.78:0/1933530027 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f2f2809bb00 con 0x7f2f38102210 2024-03-24T00:20:17.260 INFO:teuthology.orchestra.run.smithi078.stdout:197568495673 2024-03-24T00:20:17.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.263+0000 7f2f3e65f700 1 -- 172.21.15.78:0/1933530027 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2f200a2920 msgr2=0x7f2f200a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:17.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.263+0000 7f2f3e65f700 1 --2- 172.21.15.78:0/1933530027 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2f200a2920 0x7f2f200a4de0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2f38199e10 tx=0x7f2f2c009260 comp rx=0 tx=0).stop 2024-03-24T00:20:17.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.263+0000 7f2f3e65f700 1 -- 172.21.15.78:0/1933530027 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f38102210 msgr2=0x7f2f3819e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:17.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.263+0000 7f2f3e65f700 1 --2- 172.21.15.78:0/1933530027 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f38102210 0x7f2f3819e580 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2f2800e9a0 tx=0x7f2f2800ecb0 comp rx=0 tx=0).stop 2024-03-24T00:20:17.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.263+0000 7f2f3e65f700 1 -- 172.21.15.78:0/1933530027 shutdown_connections 2024-03-24T00:20:17.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.263+0000 7f2f3e65f700 1 --2- 172.21.15.78:0/1933530027 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f2f200a2920 0x7f2f200a4de0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:17.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.263+0000 7f2f3e65f700 1 --2- 172.21.15.78:0/1933530027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f38108220 0x7f2f38198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:17.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.263+0000 7f2f3e65f700 1 --2- 172.21.15.78:0/1933530027 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f38102210 0x7f2f3819e580 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:17.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.263+0000 7f2f3e65f700 1 --2- 172.21.15.78:0/1933530027 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2f38102bd0 0x7f2f3819eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:17.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.263+0000 7f2f3e65f700 1 -- 172.21.15.78:0/1933530027 >> 172.21.15.78:0/1933530027 conn(0x7f2f380792f0 msgr2=0x7f2f38100b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:17.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.263+0000 7f2f3e65f700 1 -- 172.21.15.78:0/1933530027 shutdown_connections 2024-03-24T00:20:17.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:17.263+0000 7f2f3e65f700 1 -- 172.21.15.78:0/1933530027 wait complete. 2024-03-24T00:20:17.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:17 smithi078 ceph-mon[32429]: pgmap v339: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:20:17.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:17 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:20:17.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:17 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:20:17.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:17 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1933530027' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T00:20:17.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:17 smithi106 ceph-mon[36505]: pgmap v339: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:20:17.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:17 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:20:17.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:17 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:20:17.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:17 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1933530027' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T00:20:17.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:17 smithi107 ceph-mon[36485]: pgmap v339: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:20:17.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:17 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:20:17.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:17 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:20:17.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:17 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1933530027' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T00:20:18.183 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495668 got 197568495673 for osd.6 2024-03-24T00:20:18.183 DEBUG:teuthology.parallel:result is None 2024-03-24T00:20:18.183 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-24T00:20:18.183 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T00:20:18.444 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:20:19.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:19 smithi078 ceph-mon[32429]: pgmap v340: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:20:19.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:19 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:20:19.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:19 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:20:19.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:19 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:20:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:19 smithi106 ceph-mon[36505]: pgmap v340: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:20:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:19 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:20:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:19 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:20:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:19 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:20:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:19 smithi107 ceph-mon[36485]: pgmap v340: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:20:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:19 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:20:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:19 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:20:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:19 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' 2024-03-24T00:20:20.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.161+0000 7fb8f5c45700 1 -- 172.21.15.78:0/4107690555 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8f0100d90 msgr2=0x7fb8f0101170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:20.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.161+0000 7fb8f5c45700 1 --2- 172.21.15.78:0/4107690555 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8f0100d90 0x7fb8f0101170 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb8e0009990 tx=0x7fb8e0009ca0 comp rx=0 tx=0).stop 2024-03-24T00:20:20.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.161+0000 7fb8f5c45700 1 -- 172.21.15.78:0/4107690555 shutdown_connections 2024-03-24T00:20:20.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.161+0000 7fb8f5c45700 1 --2- 172.21.15.78:0/4107690555 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8f0105c20 0x7fb8f010d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:20.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.161+0000 7fb8f5c45700 1 --2- 172.21.15.78:0/4107690555 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8f0100d90 0x7fb8f0101170 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:20.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.161+0000 7fb8f5c45700 1 --2- 172.21.15.78:0/4107690555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f0101740 0x7fb8f01054f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:20.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.161+0000 7fb8f5c45700 1 -- 172.21.15.78:0/4107690555 >> 172.21.15.78:0/4107690555 conn(0x7fb8f0076410 msgr2=0x7fb8f0076820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:20.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.161+0000 7fb8f5c45700 1 -- 172.21.15.78:0/4107690555 shutdown_connections 2024-03-24T00:20:20.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.161+0000 7fb8f5c45700 1 -- 172.21.15.78:0/4107690555 wait complete. 2024-03-24T00:20:20.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.161+0000 7fb8f5c45700 1 Processor -- start 2024-03-24T00:20:20.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.161+0000 7fb8f5c45700 1 -- start start 2024-03-24T00:20:20.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.162+0000 7fb8f5c45700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f0100d90 0x7fb8f019e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:20.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.162+0000 7fb8f5c45700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8f0101740 0x7fb8f019ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:20.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.162+0000 7fb8f5c45700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8f0105c20 0x7fb8f0198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:20.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.162+0000 7fb8f5c45700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f0111430 con 0x7fb8f0105c20 2024-03-24T00:20:20.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.162+0000 7fb8f5c45700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f01112b0 con 0x7fb8f0101740 2024-03-24T00:20:20.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.162+0000 7fb8f5c45700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f01115b0 con 0x7fb8f0100d90 2024-03-24T00:20:20.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.162+0000 7fb8eeffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8f0101740 0x7fb8f019ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:20.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.162+0000 7fb8effff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8f0105c20 0x7fb8f0198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:20.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.162+0000 7fb8ef7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f0100d90 0x7fb8f019e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:20.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.162+0000 7fb8eeffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8f0101740 0x7fb8f019ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.78:46802/0 (socket says 172.21.15.78:46802) 2024-03-24T00:20:20.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.162+0000 7fb8eeffd700 1 -- 172.21.15.78:0/300436382 learned_addr learned my addr 172.21.15.78:0/300436382 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:20.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.162+0000 7fb8effff700 1 -- 172.21.15.78:0/300436382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f0100d90 msgr2=0x7fb8f019e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:20.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.162+0000 7fb8effff700 1 --2- 172.21.15.78:0/300436382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f0100d90 0x7fb8f019e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:20.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.162+0000 7fb8effff700 1 -- 172.21.15.78:0/300436382 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8f0101740 msgr2=0x7fb8f019ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:20.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.162+0000 7fb8effff700 1 --2- 172.21.15.78:0/300436382 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8f0101740 0x7fb8f019ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:20.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.162+0000 7fb8effff700 1 -- 172.21.15.78:0/300436382 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f0199000 con 0x7fb8f0105c20 2024-03-24T00:20:20.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.162+0000 7fb8effff700 1 --2- 172.21.15.78:0/300436382 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8f0105c20 0x7fb8f0198820 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb8e800efc0 tx=0x7fb8e800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:20.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.162+0000 7fb8ecff9700 1 -- 172.21.15.78:0/300436382 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e80092e0 con 0x7fb8f0105c20 2024-03-24T00:20:20.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.163+0000 7fb8ecff9700 1 -- 172.21.15.78:0/300436382 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb8e8004bb0 con 0x7fb8f0105c20 2024-03-24T00:20:20.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.163+0000 7fb8f5c45700 1 -- 172.21.15.78:0/300436382 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8f0199290 con 0x7fb8f0105c20 2024-03-24T00:20:20.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.163+0000 7fb8ecff9700 1 -- 172.21.15.78:0/300436382 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e8004280 con 0x7fb8f0105c20 2024-03-24T00:20:20.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.163+0000 7fb8f5c45700 1 -- 172.21.15.78:0/300436382 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8f01a5500 con 0x7fb8f0105c20 2024-03-24T00:20:20.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.163+0000 7fb8f5c45700 1 -- 172.21.15.78:0/300436382 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8f0101fa0 con 0x7fb8f0105c20 2024-03-24T00:20:20.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.165+0000 7fb8ecff9700 1 -- 172.21.15.78:0/300436382 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fb8e801b030 con 0x7fb8f0105c20 2024-03-24T00:20:20.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.165+0000 7fb8ecff9700 1 --2- 172.21.15.78:0/300436382 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fb8d80a2920 0x7fb8d80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:20.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.166+0000 7fb8ef7fe700 1 --2- 172.21.15.78:0/300436382 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fb8d80a2920 0x7fb8d80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:20.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.166+0000 7fb8ecff9700 1 -- 172.21.15.78:0/300436382 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fb8e80cf330 con 0x7fb8f0105c20 2024-03-24T00:20:20.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.166+0000 7fb8ef7fe700 1 --2- 172.21.15.78:0/300436382 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fb8d80a2920 0x7fb8d80a4de0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb8e000b2e0 tx=0x7fb8e0005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:20.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.169+0000 7fb8ecff9700 1 -- 172.21.15.78:0/300436382 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb8e8022020 con 0x7fb8f0105c20 2024-03-24T00:20:20.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.364+0000 7fb8f5c45700 1 -- 172.21.15.78:0/300436382 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb8f00625d0 con 0x7fb8d80a2920 2024-03-24T00:20:20.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.373+0000 7fb8ecff9700 1 -- 172.21.15.78:0/300436382 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340377 (secure 0 0 0) 0x7fb8e00791b0 con 0x7fb8d80a2920 2024-03-24T00:20:20.378 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:20:20.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.378+0000 7fb8f5c45700 1 -- 172.21.15.78:0/300436382 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fb8d80a2920 msgr2=0x7fb8d80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:20.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.378+0000 7fb8f5c45700 1 --2- 172.21.15.78:0/300436382 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fb8d80a2920 0x7fb8d80a4de0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb8e000b2e0 tx=0x7fb8e0005960 comp rx=0 tx=0).stop 2024-03-24T00:20:20.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.378+0000 7fb8f5c45700 1 -- 172.21.15.78:0/300436382 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8f0105c20 msgr2=0x7fb8f0198820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:20.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.378+0000 7fb8f5c45700 1 --2- 172.21.15.78:0/300436382 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8f0105c20 0x7fb8f0198820 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb8e800efc0 tx=0x7fb8e800c3a0 comp rx=0 tx=0).stop 2024-03-24T00:20:20.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.378+0000 7fb8f5c45700 1 -- 172.21.15.78:0/300436382 shutdown_connections 2024-03-24T00:20:20.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.378+0000 7fb8f5c45700 1 --2- 172.21.15.78:0/300436382 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fb8d80a2920 0x7fb8d80a4de0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:20.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.378+0000 7fb8f5c45700 1 --2- 172.21.15.78:0/300436382 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8f0105c20 0x7fb8f0198820 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:20.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.378+0000 7fb8f5c45700 1 --2- 172.21.15.78:0/300436382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f0100d90 0x7fb8f019e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:20.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.378+0000 7fb8f5c45700 1 --2- 172.21.15.78:0/300436382 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8f0101740 0x7fb8f019ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:20.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.378+0000 7fb8f5c45700 1 -- 172.21.15.78:0/300436382 >> 172.21.15.78:0/300436382 conn(0x7fb8f0076410 msgr2=0x7fb8f010e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:20.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.378+0000 7fb8f5c45700 1 -- 172.21.15.78:0/300436382 shutdown_connections 2024-03-24T00:20:20.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:20.378+0000 7fb8f5c45700 1 -- 172.21.15.78:0/300436382 wait complete. 2024-03-24T00:20:20.383 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-03-24T00:20:21.117 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":341,"stamp":"2024-03-24T00:20:20.305318+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656378,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":576,"num_read_kb":410,"num_write":344,"num_write_kb":3054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1243127,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":278,"ondisk_log_size":278,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":413248,"kb_used_data":14084,"kb_used_omap":18,"kb_used_meta":398957,"kb_avail":1124528576,"statfs":{"total":1151940427776,"available":1151517261824,"internally_reserved":0,"allocated":14422016,"data_stored":7955036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":408532284},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":75,"num_read_kb":40,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003673"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.448551+0000","last_change":"2024-03-24T00:18:43.890863+0000","last_active":"2024-03-24T00:19:41.448551+0000","last_peered":"2024-03-24T00:19:41.448551+0000","last_clean":"2024-03-24T00:19:41.448551+0000","last_became_active":"2024-03-24T00:18:43.890503+0000","last_became_peered":"2024-03-24T00:18:43.890503+0000","last_unstale":"2024-03-24T00:19:41.448551+0000","last_undegraded":"2024-03-24T00:19:41.448551+0000","last_fullsized":"2024-03-24T00:19:41.448551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:33:10.518835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.467919+0000","last_change":"2024-03-24T00:18:43.882039+0000","last_active":"2024-03-24T00:19:41.467919+0000","last_peered":"2024-03-24T00:19:41.467919+0000","last_clean":"2024-03-24T00:19:41.467919+0000","last_became_active":"2024-03-24T00:18:43.881704+0000","last_became_peered":"2024-03-24T00:18:43.881704+0000","last_unstale":"2024-03-24T00:19:41.467919+0000","last_undegraded":"2024-03-24T00:19:41.467919+0000","last_fullsized":"2024-03-24T00:19:41.467919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:04:46.386032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.467770+0000","last_change":"2024-03-24T00:18:43.888649+0000","last_active":"2024-03-24T00:19:41.467770+0000","last_peered":"2024-03-24T00:19:41.467770+0000","last_clean":"2024-03-24T00:19:41.467770+0000","last_became_active":"2024-03-24T00:18:43.888339+0000","last_became_peered":"2024-03-24T00:18:43.888339+0000","last_unstale":"2024-03-24T00:19:41.467770+0000","last_undegraded":"2024-03-24T00:19:41.467770+0000","last_fullsized":"2024-03-24T00:19:41.467770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:28:35.787820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.452431+0000","last_change":"2024-03-24T00:19:41.451897+0000","last_active":"2024-03-24T00:19:41.452431+0000","last_peered":"2024-03-24T00:19:41.452431+0000","last_clean":"2024-03-24T00:19:41.452431+0000","last_became_active":"2024-03-24T00:19:41.451597+0000","last_became_peered":"2024-03-24T00:19:41.451597+0000","last_unstale":"2024-03-24T00:19:41.452431+0000","last_undegraded":"2024-03-24T00:19:41.452431+0000","last_fullsized":"2024-03-24T00:19:41.452431+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:10:32.717345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450240+0000","last_change":"2024-03-24T00:18:43.890677+0000","last_active":"2024-03-24T00:19:41.450240+0000","last_peered":"2024-03-24T00:19:41.450240+0000","last_clean":"2024-03-24T00:19:41.450240+0000","last_became_active":"2024-03-24T00:18:43.887161+0000","last_became_peered":"2024-03-24T00:18:43.887161+0000","last_unstale":"2024-03-24T00:19:41.450240+0000","last_undegraded":"2024-03-24T00:19:41.450240+0000","last_fullsized":"2024-03-24T00:19:41.450240+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:09:32.160568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.446421+0000","last_change":"2024-03-24T00:18:43.887293+0000","last_active":"2024-03-24T00:19:41.446421+0000","last_peered":"2024-03-24T00:19:41.446421+0000","last_clean":"2024-03-24T00:19:41.446421+0000","last_became_active":"2024-03-24T00:18:43.886961+0000","last_became_peered":"2024-03-24T00:18:43.886961+0000","last_unstale":"2024-03-24T00:19:41.446421+0000","last_undegraded":"2024-03-24T00:19:41.446421+0000","last_fullsized":"2024-03-24T00:19:41.446421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:48:14.312376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.453058+0000","last_change":"2024-03-24T00:18:43.876014+0000","last_active":"2024-03-24T00:19:41.453058+0000","last_peered":"2024-03-24T00:19:41.453058+0000","last_clean":"2024-03-24T00:19:41.453058+0000","last_became_active":"2024-03-24T00:18:43.875619+0000","last_became_peered":"2024-03-24T00:18:43.875619+0000","last_unstale":"2024-03-24T00:19:41.453058+0000","last_undegraded":"2024-03-24T00:19:41.453058+0000","last_fullsized":"2024-03-24T00:19:41.453058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:10:26.707681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450799+0000","last_change":"2024-03-24T00:18:43.894851+0000","last_active":"2024-03-24T00:19:41.450799+0000","last_peered":"2024-03-24T00:19:41.450799+0000","last_clean":"2024-03-24T00:19:41.450799+0000","last_became_active":"2024-03-24T00:18:43.894364+0000","last_became_peered":"2024-03-24T00:18:43.894364+0000","last_unstale":"2024-03-24T00:19:41.450799+0000","last_undegraded":"2024-03-24T00:19:41.450799+0000","last_fullsized":"2024-03-24T00:19:41.450799+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:52:01.538415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450781+0000","last_change":"2024-03-24T00:18:43.898453+0000","last_active":"2024-03-24T00:19:41.450781+0000","last_peered":"2024-03-24T00:19:41.450781+0000","last_clean":"2024-03-24T00:19:41.450781+0000","last_became_active":"2024-03-24T00:18:43.898118+0000","last_became_peered":"2024-03-24T00:18:43.898118+0000","last_unstale":"2024-03-24T00:19:41.450781+0000","last_undegraded":"2024-03-24T00:19:41.450781+0000","last_fullsized":"2024-03-24T00:19:41.450781+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:18:00.478010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.446702+0000","last_change":"2024-03-24T00:18:43.894133+0000","last_active":"2024-03-24T00:19:41.446702+0000","last_peered":"2024-03-24T00:19:41.446702+0000","last_clean":"2024-03-24T00:19:41.446702+0000","last_became_active":"2024-03-24T00:18:43.893777+0000","last_became_peered":"2024-03-24T00:18:43.893777+0000","last_unstale":"2024-03-24T00:19:41.446702+0000","last_undegraded":"2024-03-24T00:19:41.446702+0000","last_fullsized":"2024-03-24T00:19:41.446702+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:46:33.029934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.452898+0000","last_change":"2024-03-24T00:18:43.888024+0000","last_active":"2024-03-24T00:19:41.452898+0000","last_peered":"2024-03-24T00:19:41.452898+0000","last_clean":"2024-03-24T00:19:41.452898+0000","last_became_active":"2024-03-24T00:18:43.887430+0000","last_became_peered":"2024-03-24T00:18:43.887430+0000","last_unstale":"2024-03-24T00:19:41.452898+0000","last_undegraded":"2024-03-24T00:19:41.452898+0000","last_fullsized":"2024-03-24T00:19:41.452898+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:54:29.299788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449774+0000","last_change":"2024-03-24T00:18:43.889240+0000","last_active":"2024-03-24T00:19:41.449774+0000","last_peered":"2024-03-24T00:19:41.449774+0000","last_clean":"2024-03-24T00:19:41.449774+0000","last_became_active":"2024-03-24T00:18:43.888861+0000","last_became_peered":"2024-03-24T00:18:43.888861+0000","last_unstale":"2024-03-24T00:19:41.449774+0000","last_undegraded":"2024-03-24T00:19:41.449774+0000","last_fullsized":"2024-03-24T00:19:41.449774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:41:56.637532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.455705+0000","last_change":"2024-03-24T00:19:41.455705+0000","last_active":"2024-03-24T00:19:41.455705+0000","last_peered":"2024-03-24T00:19:41.455705+0000","last_clean":"2024-03-24T00:19:41.455705+0000","last_became_active":"2024-03-24T00:19:41.455320+0000","last_became_peered":"2024-03-24T00:19:41.455320+0000","last_unstale":"2024-03-24T00:19:41.455705+0000","last_undegraded":"2024-03-24T00:19:41.455705+0000","last_fullsized":"2024-03-24T00:19:41.455705+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:06.280616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.467662+0000","last_change":"2024-03-24T00:18:43.883717+0000","last_active":"2024-03-24T00:19:41.467662+0000","last_peered":"2024-03-24T00:19:41.467662+0000","last_clean":"2024-03-24T00:19:41.467662+0000","last_became_active":"2024-03-24T00:18:43.883329+0000","last_became_peered":"2024-03-24T00:18:43.883329+0000","last_unstale":"2024-03-24T00:19:41.467662+0000","last_undegraded":"2024-03-24T00:19:41.467662+0000","last_fullsized":"2024-03-24T00:19:41.467662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:26:15.127054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.467573+0000","last_change":"2024-03-24T00:18:43.890831+0000","last_active":"2024-03-24T00:19:41.467573+0000","last_peered":"2024-03-24T00:19:41.467573+0000","last_clean":"2024-03-24T00:19:41.467573+0000","last_became_active":"2024-03-24T00:18:43.890448+0000","last_became_peered":"2024-03-24T00:18:43.890448+0000","last_unstale":"2024-03-24T00:19:41.467573+0000","last_undegraded":"2024-03-24T00:19:41.467573+0000","last_fullsized":"2024-03-24T00:19:41.467573+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:56:13.222271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.448637+0000","last_change":"2024-03-24T00:18:43.890309+0000","last_active":"2024-03-24T00:19:41.448637+0000","last_peered":"2024-03-24T00:19:41.448637+0000","last_clean":"2024-03-24T00:19:41.448637+0000","last_became_active":"2024-03-24T00:18:43.889138+0000","last_became_peered":"2024-03-24T00:18:43.889138+0000","last_unstale":"2024-03-24T00:19:41.448637+0000","last_undegraded":"2024-03-24T00:19:41.448637+0000","last_fullsized":"2024-03-24T00:19:41.448637+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:13:52.480938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.448337+0000","last_change":"2024-03-24T00:18:43.891633+0000","last_active":"2024-03-24T00:19:41.448337+0000","last_peered":"2024-03-24T00:19:41.448337+0000","last_clean":"2024-03-24T00:19:41.448337+0000","last_became_active":"2024-03-24T00:18:43.891303+0000","last_became_peered":"2024-03-24T00:18:43.891303+0000","last_unstale":"2024-03-24T00:19:41.448337+0000","last_undegraded":"2024-03-24T00:19:41.448337+0000","last_fullsized":"2024-03-24T00:19:41.448337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:34:01.874142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.448130+0000","last_change":"2024-03-24T00:18:43.887695+0000","last_active":"2024-03-24T00:19:41.448130+0000","last_peered":"2024-03-24T00:19:41.448130+0000","last_clean":"2024-03-24T00:19:41.448130+0000","last_became_active":"2024-03-24T00:18:43.887359+0000","last_became_peered":"2024-03-24T00:18:43.887359+0000","last_unstale":"2024-03-24T00:19:41.448130+0000","last_undegraded":"2024-03-24T00:19:41.448130+0000","last_fullsized":"2024-03-24T00:19:41.448130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:00:50.951362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.445281+0000","last_change":"2024-03-24T00:18:43.895671+0000","last_active":"2024-03-24T00:19:41.445281+0000","last_peered":"2024-03-24T00:19:41.445281+0000","last_clean":"2024-03-24T00:19:41.445281+0000","last_became_active":"2024-03-24T00:18:43.895267+0000","last_became_peered":"2024-03-24T00:18:43.895267+0000","last_unstale":"2024-03-24T00:19:41.445281+0000","last_undegraded":"2024-03-24T00:19:41.445281+0000","last_fullsized":"2024-03-24T00:19:41.445281+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:24:48.014768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.444913+0000","last_change":"2024-03-24T00:18:47.896879+0000","last_active":"2024-03-24T00:19:41.444913+0000","last_peered":"2024-03-24T00:19:41.444913+0000","last_clean":"2024-03-24T00:19:41.444913+0000","last_became_active":"2024-03-24T00:18:47.896517+0000","last_became_peered":"2024-03-24T00:18:47.896517+0000","last_unstale":"2024-03-24T00:19:41.444913+0000","last_undegraded":"2024-03-24T00:19:41.444913+0000","last_fullsized":"2024-03-24T00:19:41.444913+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:28:34.331404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.461363+0000","last_change":"2024-03-24T00:19:41.461363+0000","last_active":"2024-03-24T00:19:41.461363+0000","last_peered":"2024-03-24T00:19:41.461363+0000","last_clean":"2024-03-24T00:19:41.461363+0000","last_became_active":"2024-03-24T00:19:41.454206+0000","last_became_peered":"2024-03-24T00:19:41.454206+0000","last_unstale":"2024-03-24T00:19:41.461363+0000","last_undegraded":"2024-03-24T00:19:41.461363+0000","last_fullsized":"2024-03-24T00:19:41.461363+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:56:22.294313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,4],"acting":[9,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.451138+0000","last_change":"2024-03-24T00:18:43.892643+0000","last_active":"2024-03-24T00:19:41.451138+0000","last_peered":"2024-03-24T00:19:41.451138+0000","last_clean":"2024-03-24T00:19:41.451138+0000","last_became_active":"2024-03-24T00:18:43.892317+0000","last_became_peered":"2024-03-24T00:18:43.892317+0000","last_unstale":"2024-03-24T00:19:41.451138+0000","last_undegraded":"2024-03-24T00:19:41.451138+0000","last_fullsized":"2024-03-24T00:19:41.451138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:26:28.374587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449815+0000","last_change":"2024-03-24T00:18:47.895307+0000","last_active":"2024-03-24T00:19:41.449815+0000","last_peered":"2024-03-24T00:19:41.449815+0000","last_clean":"2024-03-24T00:19:41.449815+0000","last_became_active":"2024-03-24T00:18:47.894988+0000","last_became_peered":"2024-03-24T00:18:47.894988+0000","last_unstale":"2024-03-24T00:19:41.449815+0000","last_undegraded":"2024-03-24T00:19:41.449815+0000","last_fullsized":"2024-03-24T00:19:41.449815+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:48:41.068076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.451359+0000","last_change":"2024-03-24T00:18:43.255861+0000","last_active":"2024-03-24T00:19:41.451359+0000","last_peered":"2024-03-24T00:19:41.451359+0000","last_clean":"2024-03-24T00:19:41.451359+0000","last_became_active":"2024-03-24T00:18:43.255529+0000","last_became_peered":"2024-03-24T00:18:43.255529+0000","last_unstale":"2024-03-24T00:19:41.451359+0000","last_undegraded":"2024-03-24T00:19:41.451359+0000","last_fullsized":"2024-03-24T00:19:41.451359+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:43:19.617673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.467953+0000","last_change":"2024-03-24T00:18:43.886651+0000","last_active":"2024-03-24T00:19:41.467953+0000","last_peered":"2024-03-24T00:19:41.467953+0000","last_clean":"2024-03-24T00:19:41.467953+0000","last_became_active":"2024-03-24T00:18:43.886341+0000","last_became_peered":"2024-03-24T00:18:43.886341+0000","last_unstale":"2024-03-24T00:19:41.467953+0000","last_undegraded":"2024-03-24T00:19:41.467953+0000","last_fullsized":"2024-03-24T00:19:41.467953+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:14:11.993717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449913+0000","last_change":"2024-03-24T00:18:47.897123+0000","last_active":"2024-03-24T00:19:41.449913+0000","last_peered":"2024-03-24T00:19:41.449913+0000","last_clean":"2024-03-24T00:19:41.449913+0000","last_became_active":"2024-03-24T00:18:47.896762+0000","last_became_peered":"2024-03-24T00:18:47.896762+0000","last_unstale":"2024-03-24T00:19:41.449913+0000","last_undegraded":"2024-03-24T00:19:41.449913+0000","last_fullsized":"2024-03-24T00:19:41.449913+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:45:50.639038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450830+0000","last_change":"2024-03-24T00:18:43.263895+0000","last_active":"2024-03-24T00:19:41.450830+0000","last_peered":"2024-03-24T00:19:41.450830+0000","last_clean":"2024-03-24T00:19:41.450830+0000","last_became_active":"2024-03-24T00:18:43.263557+0000","last_became_peered":"2024-03-24T00:18:43.263557+0000","last_unstale":"2024-03-24T00:19:41.450830+0000","last_undegraded":"2024-03-24T00:19:41.450830+0000","last_fullsized":"2024-03-24T00:19:41.450830+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:44:28.763293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450978+0000","last_change":"2024-03-24T00:18:43.888820+0000","last_active":"2024-03-24T00:19:41.450978+0000","last_peered":"2024-03-24T00:19:41.450978+0000","last_clean":"2024-03-24T00:19:41.450978+0000","last_became_active":"2024-03-24T00:18:43.888382+0000","last_became_peered":"2024-03-24T00:18:43.888382+0000","last_unstale":"2024-03-24T00:19:41.450978+0000","last_undegraded":"2024-03-24T00:19:41.450978+0000","last_fullsized":"2024-03-24T00:19:41.450978+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:04.097113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449456+0000","last_change":"2024-03-24T00:18:47.896443+0000","last_active":"2024-03-24T00:19:41.449456+0000","last_peered":"2024-03-24T00:19:41.449456+0000","last_clean":"2024-03-24T00:19:41.449456+0000","last_became_active":"2024-03-24T00:18:47.895792+0000","last_became_peered":"2024-03-24T00:18:47.895792+0000","last_unstale":"2024-03-24T00:19:41.449456+0000","last_undegraded":"2024-03-24T00:19:41.449456+0000","last_fullsized":"2024-03-24T00:19:41.449456+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:59:42.360902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450638+0000","last_change":"2024-03-24T00:18:43.261085+0000","last_active":"2024-03-24T00:19:41.450638+0000","last_peered":"2024-03-24T00:19:41.450638+0000","last_clean":"2024-03-24T00:19:41.450638+0000","last_became_active":"2024-03-24T00:18:43.260724+0000","last_became_peered":"2024-03-24T00:18:43.260724+0000","last_unstale":"2024-03-24T00:19:41.450638+0000","last_undegraded":"2024-03-24T00:19:41.450638+0000","last_fullsized":"2024-03-24T00:19:41.450638+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:05:20.992776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.448076+0000","last_change":"2024-03-24T00:18:43.892891+0000","last_active":"2024-03-24T00:19:41.448076+0000","last_peered":"2024-03-24T00:19:41.448076+0000","last_clean":"2024-03-24T00:19:41.448076+0000","last_became_active":"2024-03-24T00:18:43.892430+0000","last_became_peered":"2024-03-24T00:18:43.892430+0000","last_unstale":"2024-03-24T00:19:41.448076+0000","last_undegraded":"2024-03-24T00:19:41.448076+0000","last_fullsized":"2024-03-24T00:19:41.448076+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:30:24.926031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.445303+0000","last_change":"2024-03-24T00:18:47.889119+0000","last_active":"2024-03-24T00:19:41.445303+0000","last_peered":"2024-03-24T00:19:41.445303+0000","last_clean":"2024-03-24T00:19:41.445303+0000","last_became_active":"2024-03-24T00:18:47.888620+0000","last_became_peered":"2024-03-24T00:18:47.888620+0000","last_unstale":"2024-03-24T00:19:41.445303+0000","last_undegraded":"2024-03-24T00:19:41.445303+0000","last_fullsized":"2024-03-24T00:19:41.445303+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:51.483286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'6","reported_seq":104,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:20:14.947206+0000","last_change":"2024-03-24T00:18:43.251824+0000","last_active":"2024-03-24T00:20:14.947206+0000","last_peered":"2024-03-24T00:20:14.947206+0000","last_clean":"2024-03-24T00:20:14.947206+0000","last_became_active":"2024-03-24T00:18:43.251458+0000","last_became_peered":"2024-03-24T00:18:43.251458+0000","last_unstale":"2024-03-24T00:20:14.947206+0000","last_undegraded":"2024-03-24T00:20:14.947206+0000","last_fullsized":"2024-03-24T00:20:14.947206+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:13:02.303670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450437+0000","last_change":"2024-03-24T00:18:43.891289+0000","last_active":"2024-03-24T00:19:41.450437+0000","last_peered":"2024-03-24T00:19:41.450437+0000","last_clean":"2024-03-24T00:19:41.450437+0000","last_became_active":"2024-03-24T00:18:43.890725+0000","last_became_peered":"2024-03-24T00:18:43.890725+0000","last_unstale":"2024-03-24T00:19:41.450437+0000","last_undegraded":"2024-03-24T00:19:41.450437+0000","last_fullsized":"2024-03-24T00:19:41.450437+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:37:41.096520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.445139+0000","last_change":"2024-03-24T00:18:47.896175+0000","last_active":"2024-03-24T00:19:41.445139+0000","last_peered":"2024-03-24T00:19:41.445139+0000","last_clean":"2024-03-24T00:19:41.445139+0000","last_became_active":"2024-03-24T00:18:47.895723+0000","last_became_peered":"2024-03-24T00:18:47.895723+0000","last_unstale":"2024-03-24T00:19:41.445139+0000","last_undegraded":"2024-03-24T00:19:41.445139+0000","last_fullsized":"2024-03-24T00:19:41.445139+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:41:49.167532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.446324+0000","last_change":"2024-03-24T00:18:43.258986+0000","last_active":"2024-03-24T00:19:41.446324+0000","last_peered":"2024-03-24T00:19:41.446324+0000","last_clean":"2024-03-24T00:19:41.446324+0000","last_became_active":"2024-03-24T00:18:43.258654+0000","last_became_peered":"2024-03-24T00:18:43.258654+0000","last_unstale":"2024-03-24T00:19:41.446324+0000","last_undegraded":"2024-03-24T00:19:41.446324+0000","last_fullsized":"2024-03-24T00:19:41.446324+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:34:44.339991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.447947+0000","last_change":"2024-03-24T00:18:43.885852+0000","last_active":"2024-03-24T00:19:41.447947+0000","last_peered":"2024-03-24T00:19:41.447947+0000","last_clean":"2024-03-24T00:19:41.447947+0000","last_became_active":"2024-03-24T00:18:43.885538+0000","last_became_peered":"2024-03-24T00:18:43.885538+0000","last_unstale":"2024-03-24T00:19:41.447947+0000","last_undegraded":"2024-03-24T00:19:41.447947+0000","last_fullsized":"2024-03-24T00:19:41.447947+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:42:02.879892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.448233+0000","last_change":"2024-03-24T00:18:47.893230+0000","last_active":"2024-03-24T00:19:41.448233+0000","last_peered":"2024-03-24T00:19:41.448233+0000","last_clean":"2024-03-24T00:19:41.448233+0000","last_became_active":"2024-03-24T00:18:47.892522+0000","last_became_peered":"2024-03-24T00:18:47.892522+0000","last_unstale":"2024-03-24T00:19:41.448233+0000","last_undegraded":"2024-03-24T00:19:41.448233+0000","last_fullsized":"2024-03-24T00:19:41.448233+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:10:11.768154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449760+0000","last_change":"2024-03-24T00:18:43.262255+0000","last_active":"2024-03-24T00:19:41.449760+0000","last_peered":"2024-03-24T00:19:41.449760+0000","last_clean":"2024-03-24T00:19:41.449760+0000","last_became_active":"2024-03-24T00:18:43.261940+0000","last_became_peered":"2024-03-24T00:18:43.261940+0000","last_unstale":"2024-03-24T00:19:41.449760+0000","last_undegraded":"2024-03-24T00:19:41.449760+0000","last_fullsized":"2024-03-24T00:19:41.449760+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:58:48.789845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450755+0000","last_change":"2024-03-24T00:18:43.883046+0000","last_active":"2024-03-24T00:19:41.450755+0000","last_peered":"2024-03-24T00:19:41.450755+0000","last_clean":"2024-03-24T00:19:41.450755+0000","last_became_active":"2024-03-24T00:18:43.881405+0000","last_became_peered":"2024-03-24T00:18:43.881405+0000","last_unstale":"2024-03-24T00:19:41.450755+0000","last_undegraded":"2024-03-24T00:19:41.450755+0000","last_fullsized":"2024-03-24T00:19:41.450755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:21:00.047019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.467844+0000","last_change":"2024-03-24T00:18:47.897630+0000","last_active":"2024-03-24T00:19:41.467844+0000","last_peered":"2024-03-24T00:19:41.467844+0000","last_clean":"2024-03-24T00:19:41.467844+0000","last_became_active":"2024-03-24T00:18:47.897211+0000","last_became_peered":"2024-03-24T00:18:47.897211+0000","last_unstale":"2024-03-24T00:19:41.467844+0000","last_undegraded":"2024-03-24T00:19:41.467844+0000","last_fullsized":"2024-03-24T00:19:41.467844+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:22:48.762870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.447211+0000","last_change":"2024-03-24T00:18:43.262134+0000","last_active":"2024-03-24T00:19:41.447211+0000","last_peered":"2024-03-24T00:19:41.447211+0000","last_clean":"2024-03-24T00:19:41.447211+0000","last_became_active":"2024-03-24T00:18:43.261783+0000","last_became_peered":"2024-03-24T00:18:43.261783+0000","last_unstale":"2024-03-24T00:19:41.447211+0000","last_undegraded":"2024-03-24T00:19:41.447211+0000","last_fullsized":"2024-03-24T00:19:41.447211+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:16:31.060204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.447601+0000","last_change":"2024-03-24T00:18:43.880532+0000","last_active":"2024-03-24T00:19:41.447601+0000","last_peered":"2024-03-24T00:19:41.447601+0000","last_clean":"2024-03-24T00:19:41.447601+0000","last_became_active":"2024-03-24T00:18:43.880198+0000","last_became_peered":"2024-03-24T00:18:43.880198+0000","last_unstale":"2024-03-24T00:19:41.447601+0000","last_undegraded":"2024-03-24T00:19:41.447601+0000","last_fullsized":"2024-03-24T00:19:41.447601+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:33:16.710842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450898+0000","last_change":"2024-03-24T00:18:47.893607+0000","last_active":"2024-03-24T00:19:41.450898+0000","last_peered":"2024-03-24T00:19:41.450898+0000","last_clean":"2024-03-24T00:19:41.450898+0000","last_became_active":"2024-03-24T00:18:47.893026+0000","last_became_peered":"2024-03-24T00:18:47.893026+0000","last_unstale":"2024-03-24T00:19:41.450898+0000","last_undegraded":"2024-03-24T00:19:41.450898+0000","last_fullsized":"2024-03-24T00:19:41.450898+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:37:58.994146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":101,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:20:17.949877+0000","last_change":"2024-03-24T00:18:43.257520+0000","last_active":"2024-03-24T00:20:17.949877+0000","last_peered":"2024-03-24T00:20:17.949877+0000","last_clean":"2024-03-24T00:20:17.949877+0000","last_became_active":"2024-03-24T00:18:43.257175+0000","last_became_peered":"2024-03-24T00:18:43.257175+0000","last_unstale":"2024-03-24T00:20:17.949877+0000","last_undegraded":"2024-03-24T00:20:17.949877+0000","last_fullsized":"2024-03-24T00:20:17.949877+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:52:45.244983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":71,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450509+0000","last_change":"2024-03-24T00:18:43.887739+0000","last_active":"2024-03-24T00:19:41.450509+0000","last_peered":"2024-03-24T00:19:41.450509+0000","last_clean":"2024-03-24T00:19:41.450509+0000","last_became_active":"2024-03-24T00:18:43.887379+0000","last_became_peered":"2024-03-24T00:18:43.887379+0000","last_unstale":"2024-03-24T00:19:41.450509+0000","last_undegraded":"2024-03-24T00:19:41.450509+0000","last_fullsized":"2024-03-24T00:19:41.450509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:10:50.384759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.447332+0000","last_change":"2024-03-24T00:18:47.890827+0000","last_active":"2024-03-24T00:19:41.447332+0000","last_peered":"2024-03-24T00:19:41.447332+0000","last_clean":"2024-03-24T00:19:41.447332+0000","last_became_active":"2024-03-24T00:18:47.890399+0000","last_became_peered":"2024-03-24T00:18:47.890399+0000","last_unstale":"2024-03-24T00:19:41.447332+0000","last_undegraded":"2024-03-24T00:19:41.447332+0000","last_fullsized":"2024-03-24T00:19:41.447332+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:46:54.575468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.452042+0000","last_change":"2024-03-24T00:18:43.248132+0000","last_active":"2024-03-24T00:19:41.452042+0000","last_peered":"2024-03-24T00:19:41.452042+0000","last_clean":"2024-03-24T00:19:41.452042+0000","last_became_active":"2024-03-24T00:18:43.247772+0000","last_became_peered":"2024-03-24T00:18:43.247772+0000","last_unstale":"2024-03-24T00:19:41.452042+0000","last_undegraded":"2024-03-24T00:19:41.452042+0000","last_fullsized":"2024-03-24T00:19:41.452042+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:09.330155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.467974+0000","last_change":"2024-03-24T00:18:43.889920+0000","last_active":"2024-03-24T00:19:41.467974+0000","last_peered":"2024-03-24T00:19:41.467974+0000","last_clean":"2024-03-24T00:19:41.467974+0000","last_became_active":"2024-03-24T00:18:43.889593+0000","last_became_peered":"2024-03-24T00:18:43.889593+0000","last_unstale":"2024-03-24T00:19:41.467974+0000","last_undegraded":"2024-03-24T00:19:41.467974+0000","last_fullsized":"2024-03-24T00:19:41.467974+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:48:41.101019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450161+0000","last_change":"2024-03-24T00:18:47.893133+0000","last_active":"2024-03-24T00:19:41.450161+0000","last_peered":"2024-03-24T00:19:41.450161+0000","last_clean":"2024-03-24T00:19:41.450161+0000","last_became_active":"2024-03-24T00:18:47.892731+0000","last_became_peered":"2024-03-24T00:18:47.892731+0000","last_unstale":"2024-03-24T00:19:41.450161+0000","last_undegraded":"2024-03-24T00:19:41.450161+0000","last_fullsized":"2024-03-24T00:19:41.450161+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:51:09.539852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.448061+0000","last_change":"2024-03-24T00:18:43.257697+0000","last_active":"2024-03-24T00:19:41.448061+0000","last_peered":"2024-03-24T00:19:41.448061+0000","last_clean":"2024-03-24T00:19:41.448061+0000","last_became_active":"2024-03-24T00:18:43.257334+0000","last_became_peered":"2024-03-24T00:18:43.257334+0000","last_unstale":"2024-03-24T00:19:41.448061+0000","last_undegraded":"2024-03-24T00:19:41.448061+0000","last_fullsized":"2024-03-24T00:19:41.448061+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:35:22.817064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.451128+0000","last_change":"2024-03-24T00:18:43.896238+0000","last_active":"2024-03-24T00:19:41.451128+0000","last_peered":"2024-03-24T00:19:41.451128+0000","last_clean":"2024-03-24T00:19:41.451128+0000","last_became_active":"2024-03-24T00:18:43.895792+0000","last_became_peered":"2024-03-24T00:18:43.895792+0000","last_unstale":"2024-03-24T00:19:41.451128+0000","last_undegraded":"2024-03-24T00:19:41.451128+0000","last_fullsized":"2024-03-24T00:19:41.451128+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:55:11.865771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.447408+0000","last_change":"2024-03-24T00:18:47.891407+0000","last_active":"2024-03-24T00:19:41.447408+0000","last_peered":"2024-03-24T00:19:41.447408+0000","last_clean":"2024-03-24T00:19:41.447408+0000","last_became_active":"2024-03-24T00:18:47.891042+0000","last_became_peered":"2024-03-24T00:18:47.891042+0000","last_unstale":"2024-03-24T00:19:41.447408+0000","last_undegraded":"2024-03-24T00:19:41.447408+0000","last_fullsized":"2024-03-24T00:19:41.447408+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:14:02.939199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'31","reported_seq":223,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:20:18.497258+0000","last_change":"2024-03-24T00:19:41.466575+0000","last_active":"2024-03-24T00:20:18.497258+0000","last_peered":"2024-03-24T00:20:18.497258+0000","last_clean":"2024-03-24T00:20:18.497258+0000","last_became_active":"2024-03-24T00:19:41.454621+0000","last_became_peered":"2024-03-24T00:19:41.454621+0000","last_unstale":"2024-03-24T00:20:18.497258+0000","last_undegraded":"2024-03-24T00:20:18.497258+0000","last_fullsized":"2024-03-24T00:20:18.497258+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:21:54.306449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61775,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":145,"num_read_kb":133,"num_write":32,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":61775,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450502+0000","last_change":"2024-03-24T00:18:43.893740+0000","last_active":"2024-03-24T00:19:41.450502+0000","last_peered":"2024-03-24T00:19:41.450502+0000","last_clean":"2024-03-24T00:19:41.450502+0000","last_became_active":"2024-03-24T00:18:43.893277+0000","last_became_peered":"2024-03-24T00:18:43.893277+0000","last_unstale":"2024-03-24T00:19:41.450502+0000","last_undegraded":"2024-03-24T00:19:41.450502+0000","last_fullsized":"2024-03-24T00:19:41.450502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:21:47.040629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.448337+0000","last_change":"2024-03-24T00:18:47.898769+0000","last_active":"2024-03-24T00:19:41.448337+0000","last_peered":"2024-03-24T00:19:41.448337+0000","last_clean":"2024-03-24T00:19:41.448337+0000","last_became_active":"2024-03-24T00:18:47.898214+0000","last_became_peered":"2024-03-24T00:18:47.898214+0000","last_unstale":"2024-03-24T00:19:41.448337+0000","last_undegraded":"2024-03-24T00:19:41.448337+0000","last_fullsized":"2024-03-24T00:19:41.448337+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:24:42.022429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.461301+0000","last_change":"2024-03-24T00:19:41.461301+0000","last_active":"2024-03-24T00:19:41.461301+0000","last_peered":"2024-03-24T00:19:41.461301+0000","last_clean":"2024-03-24T00:19:41.461301+0000","last_became_active":"2024-03-24T00:19:41.452017+0000","last_became_peered":"2024-03-24T00:19:41.452017+0000","last_unstale":"2024-03-24T00:19:41.461301+0000","last_undegraded":"2024-03-24T00:19:41.461301+0000","last_fullsized":"2024-03-24T00:19:41.461301+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:32:25.295992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.448803+0000","last_change":"2024-03-24T00:18:43.881691+0000","last_active":"2024-03-24T00:19:41.448803+0000","last_peered":"2024-03-24T00:19:41.448803+0000","last_clean":"2024-03-24T00:19:41.448803+0000","last_became_active":"2024-03-24T00:18:43.881386+0000","last_became_peered":"2024-03-24T00:18:43.881386+0000","last_unstale":"2024-03-24T00:19:41.448803+0000","last_undegraded":"2024-03-24T00:19:41.448803+0000","last_fullsized":"2024-03-24T00:19:41.448803+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:33:24.808315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.446722+0000","last_change":"2024-03-24T00:18:47.883340+0000","last_active":"2024-03-24T00:19:41.446722+0000","last_peered":"2024-03-24T00:19:41.446722+0000","last_clean":"2024-03-24T00:19:41.446722+0000","last_became_active":"2024-03-24T00:18:47.882934+0000","last_became_peered":"2024-03-24T00:18:47.882934+0000","last_unstale":"2024-03-24T00:19:41.446722+0000","last_undegraded":"2024-03-24T00:19:41.446722+0000","last_fullsized":"2024-03-24T00:19:41.446722+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:30:56.659709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.447888+0000","last_change":"2024-03-24T00:18:43.253444+0000","last_active":"2024-03-24T00:19:41.447888+0000","last_peered":"2024-03-24T00:19:41.447888+0000","last_clean":"2024-03-24T00:19:41.447888+0000","last_became_active":"2024-03-24T00:18:43.252121+0000","last_became_peered":"2024-03-24T00:18:43.252121+0000","last_unstale":"2024-03-24T00:19:41.447888+0000","last_undegraded":"2024-03-24T00:19:41.447888+0000","last_fullsized":"2024-03-24T00:19:41.447888+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:44:48.663995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.454454+0000","last_change":"2024-03-24T00:19:41.454454+0000","last_active":"2024-03-24T00:19:41.454454+0000","last_peered":"2024-03-24T00:19:41.454454+0000","last_clean":"2024-03-24T00:19:41.454454+0000","last_became_active":"2024-03-24T00:19:41.454112+0000","last_became_peered":"2024-03-24T00:19:41.454112+0000","last_unstale":"2024-03-24T00:19:41.454454+0000","last_undegraded":"2024-03-24T00:19:41.454454+0000","last_fullsized":"2024-03-24T00:19:41.454454+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:33:12.282662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.469175+0000","last_change":"2024-03-24T00:19:41.468803+0000","last_active":"2024-03-24T00:19:41.469175+0000","last_peered":"2024-03-24T00:19:41.469175+0000","last_clean":"2024-03-24T00:19:41.469175+0000","last_became_active":"2024-03-24T00:19:41.468406+0000","last_became_peered":"2024-03-24T00:19:41.468406+0000","last_unstale":"2024-03-24T00:19:41.469175+0000","last_undegraded":"2024-03-24T00:19:41.469175+0000","last_fullsized":"2024-03-24T00:19:41.469175+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:18:09.543843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"89'3","reported_seq":101,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:20:15.953209+0000","last_change":"2024-03-24T00:18:43.251682+0000","last_active":"2024-03-24T00:20:15.953209+0000","last_peered":"2024-03-24T00:20:15.953209+0000","last_clean":"2024-03-24T00:20:15.953209+0000","last_became_active":"2024-03-24T00:18:43.250953+0000","last_became_peered":"2024-03-24T00:18:43.250953+0000","last_unstale":"2024-03-24T00:20:15.953209+0000","last_undegraded":"2024-03-24T00:20:15.953209+0000","last_fullsized":"2024-03-24T00:20:15.953209+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:22:35.218781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.451662+0000","last_change":"2024-03-24T00:18:43.876334+0000","last_active":"2024-03-24T00:19:41.451662+0000","last_peered":"2024-03-24T00:19:41.451662+0000","last_clean":"2024-03-24T00:19:41.451662+0000","last_became_active":"2024-03-24T00:18:43.875397+0000","last_became_peered":"2024-03-24T00:18:43.875397+0000","last_unstale":"2024-03-24T00:19:41.451662+0000","last_undegraded":"2024-03-24T00:19:41.451662+0000","last_fullsized":"2024-03-24T00:19:41.451662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:40:20.474055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449651+0000","last_change":"2024-03-24T00:18:47.897488+0000","last_active":"2024-03-24T00:19:41.449651+0000","last_peered":"2024-03-24T00:19:41.449651+0000","last_clean":"2024-03-24T00:19:41.449651+0000","last_became_active":"2024-03-24T00:18:47.897042+0000","last_became_peered":"2024-03-24T00:18:47.897042+0000","last_unstale":"2024-03-24T00:19:41.449651+0000","last_undegraded":"2024-03-24T00:19:41.449651+0000","last_fullsized":"2024-03-24T00:19:41.449651+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:23:13.071695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450298+0000","last_change":"2024-03-24T00:18:43.255278+0000","last_active":"2024-03-24T00:19:41.450298+0000","last_peered":"2024-03-24T00:19:41.450298+0000","last_clean":"2024-03-24T00:19:41.450298+0000","last_became_active":"2024-03-24T00:18:43.254943+0000","last_became_peered":"2024-03-24T00:18:43.254943+0000","last_unstale":"2024-03-24T00:19:41.450298+0000","last_undegraded":"2024-03-24T00:19:41.450298+0000","last_fullsized":"2024-03-24T00:19:41.450298+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:09:56.755303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.447385+0000","last_change":"2024-03-24T00:18:43.892620+0000","last_active":"2024-03-24T00:19:41.447385+0000","last_peered":"2024-03-24T00:19:41.447385+0000","last_clean":"2024-03-24T00:19:41.447385+0000","last_became_active":"2024-03-24T00:18:43.892033+0000","last_became_peered":"2024-03-24T00:18:43.892033+0000","last_unstale":"2024-03-24T00:19:41.447385+0000","last_undegraded":"2024-03-24T00:19:41.447385+0000","last_fullsized":"2024-03-24T00:19:41.447385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:30:19.472861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.447287+0000","last_change":"2024-03-24T00:18:47.895623+0000","last_active":"2024-03-24T00:19:41.447287+0000","last_peered":"2024-03-24T00:19:41.447287+0000","last_clean":"2024-03-24T00:19:41.447287+0000","last_became_active":"2024-03-24T00:18:47.895242+0000","last_became_peered":"2024-03-24T00:18:47.895242+0000","last_unstale":"2024-03-24T00:19:41.447287+0000","last_undegraded":"2024-03-24T00:19:41.447287+0000","last_fullsized":"2024-03-24T00:19:41.447287+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:50:32.775988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.451373+0000","last_change":"2024-03-24T00:18:43.265575+0000","last_active":"2024-03-24T00:19:41.451373+0000","last_peered":"2024-03-24T00:19:41.451373+0000","last_clean":"2024-03-24T00:19:41.451373+0000","last_became_active":"2024-03-24T00:18:43.265265+0000","last_became_peered":"2024-03-24T00:18:43.265265+0000","last_unstale":"2024-03-24T00:19:41.451373+0000","last_undegraded":"2024-03-24T00:19:41.451373+0000","last_fullsized":"2024-03-24T00:19:41.451373+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:15:46.566812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.448055+0000","last_change":"2024-03-24T00:18:43.898574+0000","last_active":"2024-03-24T00:19:41.448055+0000","last_peered":"2024-03-24T00:19:41.448055+0000","last_clean":"2024-03-24T00:19:41.448055+0000","last_became_active":"2024-03-24T00:18:43.898153+0000","last_became_peered":"2024-03-24T00:18:43.898153+0000","last_unstale":"2024-03-24T00:19:41.448055+0000","last_undegraded":"2024-03-24T00:19:41.448055+0000","last_fullsized":"2024-03-24T00:19:41.448055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:50:15.317257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450297+0000","last_change":"2024-03-24T00:18:47.896336+0000","last_active":"2024-03-24T00:19:41.450297+0000","last_peered":"2024-03-24T00:19:41.450297+0000","last_clean":"2024-03-24T00:19:41.450297+0000","last_became_active":"2024-03-24T00:18:47.895939+0000","last_became_peered":"2024-03-24T00:18:47.895939+0000","last_unstale":"2024-03-24T00:19:41.450297+0000","last_undegraded":"2024-03-24T00:19:41.450297+0000","last_fullsized":"2024-03-24T00:19:41.450297+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:54:03.195827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.447779+0000","last_change":"2024-03-24T00:18:43.253081+0000","last_active":"2024-03-24T00:19:41.447779+0000","last_peered":"2024-03-24T00:19:41.447779+0000","last_clean":"2024-03-24T00:19:41.447779+0000","last_became_active":"2024-03-24T00:18:43.252415+0000","last_became_peered":"2024-03-24T00:18:43.252415+0000","last_unstale":"2024-03-24T00:19:41.447779+0000","last_undegraded":"2024-03-24T00:19:41.447779+0000","last_fullsized":"2024-03-24T00:19:41.447779+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:06:45.200233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.446412+0000","last_change":"2024-03-24T00:18:43.885820+0000","last_active":"2024-03-24T00:19:41.446412+0000","last_peered":"2024-03-24T00:19:41.446412+0000","last_clean":"2024-03-24T00:19:41.446412+0000","last_became_active":"2024-03-24T00:18:43.885485+0000","last_became_peered":"2024-03-24T00:18:43.885485+0000","last_unstale":"2024-03-24T00:19:41.446412+0000","last_undegraded":"2024-03-24T00:19:41.446412+0000","last_fullsized":"2024-03-24T00:19:41.446412+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:09:12.286000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449050+0000","last_change":"2024-03-24T00:18:47.899054+0000","last_active":"2024-03-24T00:19:41.449050+0000","last_peered":"2024-03-24T00:19:41.449050+0000","last_clean":"2024-03-24T00:19:41.449050+0000","last_became_active":"2024-03-24T00:18:47.898412+0000","last_became_peered":"2024-03-24T00:18:47.898412+0000","last_unstale":"2024-03-24T00:19:41.449050+0000","last_undegraded":"2024-03-24T00:19:41.449050+0000","last_fullsized":"2024-03-24T00:19:41.449050+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:17:08.273323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.451877+0000","last_change":"2024-03-24T00:18:43.251402+0000","last_active":"2024-03-24T00:19:41.451877+0000","last_peered":"2024-03-24T00:19:41.451877+0000","last_clean":"2024-03-24T00:19:41.451877+0000","last_became_active":"2024-03-24T00:18:43.251055+0000","last_became_peered":"2024-03-24T00:18:43.251055+0000","last_unstale":"2024-03-24T00:19:41.451877+0000","last_undegraded":"2024-03-24T00:19:41.451877+0000","last_fullsized":"2024-03-24T00:19:41.451877+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:07:30.966065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.446439+0000","last_change":"2024-03-24T00:18:43.886594+0000","last_active":"2024-03-24T00:19:41.446439+0000","last_peered":"2024-03-24T00:19:41.446439+0000","last_clean":"2024-03-24T00:19:41.446439+0000","last_became_active":"2024-03-24T00:18:43.886251+0000","last_became_peered":"2024-03-24T00:18:43.886251+0000","last_unstale":"2024-03-24T00:19:41.446439+0000","last_undegraded":"2024-03-24T00:19:41.446439+0000","last_fullsized":"2024-03-24T00:19:41.446439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:55:24.662318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449658+0000","last_change":"2024-03-24T00:18:47.901352+0000","last_active":"2024-03-24T00:19:41.449658+0000","last_peered":"2024-03-24T00:19:41.449658+0000","last_clean":"2024-03-24T00:19:41.449658+0000","last_became_active":"2024-03-24T00:18:47.900943+0000","last_became_peered":"2024-03-24T00:18:47.900943+0000","last_unstale":"2024-03-24T00:19:41.449658+0000","last_undegraded":"2024-03-24T00:19:41.449658+0000","last_fullsized":"2024-03-24T00:19:41.449658+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:25:29.626277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.451990+0000","last_change":"2024-03-24T00:19:41.451990+0000","last_active":"2024-03-24T00:19:41.451990+0000","last_peered":"2024-03-24T00:19:41.451990+0000","last_clean":"2024-03-24T00:19:41.451990+0000","last_became_active":"2024-03-24T00:19:41.451609+0000","last_became_peered":"2024-03-24T00:19:41.451609+0000","last_unstale":"2024-03-24T00:19:41.451990+0000","last_undegraded":"2024-03-24T00:19:41.451990+0000","last_fullsized":"2024-03-24T00:19:41.451990+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:19:18.965909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":99,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:20:14.951625+0000","last_change":"2024-03-24T00:18:43.263213+0000","last_active":"2024-03-24T00:20:14.951625+0000","last_peered":"2024-03-24T00:20:14.951625+0000","last_clean":"2024-03-24T00:20:14.951625+0000","last_became_active":"2024-03-24T00:18:43.262668+0000","last_became_peered":"2024-03-24T00:18:43.262668+0000","last_unstale":"2024-03-24T00:20:14.951625+0000","last_undegraded":"2024-03-24T00:20:14.951625+0000","last_fullsized":"2024-03-24T00:20:14.951625+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:18:15.518187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":68,"num_read_kb":68,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.452250+0000","last_change":"2024-03-24T00:18:47.891607+0000","last_active":"2024-03-24T00:19:41.452250+0000","last_peered":"2024-03-24T00:19:41.452250+0000","last_clean":"2024-03-24T00:19:41.452250+0000","last_became_active":"2024-03-24T00:18:47.891241+0000","last_became_peered":"2024-03-24T00:18:47.891241+0000","last_unstale":"2024-03-24T00:19:41.452250+0000","last_undegraded":"2024-03-24T00:19:41.452250+0000","last_fullsized":"2024-03-24T00:19:41.452250+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:14:29.593767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.451002+0000","last_change":"2024-03-24T00:18:43.894492+0000","last_active":"2024-03-24T00:19:41.451002+0000","last_peered":"2024-03-24T00:19:41.451002+0000","last_clean":"2024-03-24T00:19:41.451002+0000","last_became_active":"2024-03-24T00:18:43.893104+0000","last_became_peered":"2024-03-24T00:18:43.893104+0000","last_unstale":"2024-03-24T00:19:41.451002+0000","last_undegraded":"2024-03-24T00:19:41.451002+0000","last_fullsized":"2024-03-24T00:19:41.451002+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:26:32.130449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450059+0000","last_change":"2024-03-24T00:18:43.251344+0000","last_active":"2024-03-24T00:19:41.450059+0000","last_peered":"2024-03-24T00:19:41.450059+0000","last_clean":"2024-03-24T00:19:41.450059+0000","last_became_active":"2024-03-24T00:18:43.250961+0000","last_became_peered":"2024-03-24T00:18:43.250961+0000","last_unstale":"2024-03-24T00:19:41.450059+0000","last_undegraded":"2024-03-24T00:19:41.450059+0000","last_fullsized":"2024-03-24T00:19:41.450059+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:13:38.423465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":361,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.451467+0000","last_change":"2024-03-24T00:17:27.864318+0000","last_active":"2024-03-24T00:19:41.451467+0000","last_peered":"2024-03-24T00:19:41.451467+0000","last_clean":"2024-03-24T00:19:41.451467+0000","last_became_active":"2024-03-24T00:17:27.863875+0000","last_became_peered":"2024-03-24T00:17:27.863875+0000","last_unstale":"2024-03-24T00:19:41.451467+0000","last_undegraded":"2024-03-24T00:19:41.451467+0000","last_fullsized":"2024-03-24T00:19:41.451467+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:14:04.731181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:14:04.731181+0000","last_clean_scrub_stamp":"2024-03-24T00:14:04.731181+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:53:25.036280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.447796+0000","last_change":"2024-03-24T00:18:47.895344+0000","last_active":"2024-03-24T00:19:41.447796+0000","last_peered":"2024-03-24T00:19:41.447796+0000","last_clean":"2024-03-24T00:19:41.447796+0000","last_became_active":"2024-03-24T00:18:47.894694+0000","last_became_peered":"2024-03-24T00:18:47.894694+0000","last_unstale":"2024-03-24T00:19:41.447796+0000","last_undegraded":"2024-03-24T00:19:41.447796+0000","last_fullsized":"2024-03-24T00:19:41.447796+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:16.969143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449647+0000","last_change":"2024-03-24T00:18:43.890236+0000","last_active":"2024-03-24T00:19:41.449647+0000","last_peered":"2024-03-24T00:19:41.449647+0000","last_clean":"2024-03-24T00:19:41.449647+0000","last_became_active":"2024-03-24T00:18:43.889896+0000","last_became_peered":"2024-03-24T00:18:43.889896+0000","last_unstale":"2024-03-24T00:19:41.449647+0000","last_undegraded":"2024-03-24T00:19:41.449647+0000","last_fullsized":"2024-03-24T00:19:41.449647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:13:19.443445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.451550+0000","last_change":"2024-03-24T00:18:43.260722+0000","last_active":"2024-03-24T00:19:41.451550+0000","last_peered":"2024-03-24T00:19:41.451550+0000","last_clean":"2024-03-24T00:19:41.451550+0000","last_became_active":"2024-03-24T00:18:43.260244+0000","last_became_peered":"2024-03-24T00:18:43.260244+0000","last_unstale":"2024-03-24T00:19:41.451550+0000","last_undegraded":"2024-03-24T00:19:41.451550+0000","last_fullsized":"2024-03-24T00:19:41.451550+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:37:37.838046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.447211+0000","last_change":"2024-03-24T00:18:47.891168+0000","last_active":"2024-03-24T00:19:41.447211+0000","last_peered":"2024-03-24T00:19:41.447211+0000","last_clean":"2024-03-24T00:19:41.447211+0000","last_became_active":"2024-03-24T00:18:47.890713+0000","last_became_peered":"2024-03-24T00:18:47.890713+0000","last_unstale":"2024-03-24T00:19:41.447211+0000","last_undegraded":"2024-03-24T00:19:41.447211+0000","last_fullsized":"2024-03-24T00:19:41.447211+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:20:30.858381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450697+0000","last_change":"2024-03-24T00:18:43.256638+0000","last_active":"2024-03-24T00:19:41.450697+0000","last_peered":"2024-03-24T00:19:41.450697+0000","last_clean":"2024-03-24T00:19:41.450697+0000","last_became_active":"2024-03-24T00:18:43.256118+0000","last_became_peered":"2024-03-24T00:18:43.256118+0000","last_unstale":"2024-03-24T00:19:41.450697+0000","last_undegraded":"2024-03-24T00:19:41.450697+0000","last_fullsized":"2024-03-24T00:19:41.450697+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:40:05.523388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449330+0000","last_change":"2024-03-24T00:18:43.885293+0000","last_active":"2024-03-24T00:19:41.449330+0000","last_peered":"2024-03-24T00:19:41.449330+0000","last_clean":"2024-03-24T00:19:41.449330+0000","last_became_active":"2024-03-24T00:18:43.884989+0000","last_became_peered":"2024-03-24T00:18:43.884989+0000","last_unstale":"2024-03-24T00:19:41.449330+0000","last_undegraded":"2024-03-24T00:19:41.449330+0000","last_fullsized":"2024-03-24T00:19:41.449330+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:32:01.519131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450657+0000","last_change":"2024-03-24T00:18:47.894253+0000","last_active":"2024-03-24T00:19:41.450657+0000","last_peered":"2024-03-24T00:19:41.450657+0000","last_clean":"2024-03-24T00:19:41.450657+0000","last_became_active":"2024-03-24T00:18:47.892654+0000","last_became_peered":"2024-03-24T00:18:47.892654+0000","last_unstale":"2024-03-24T00:19:41.450657+0000","last_undegraded":"2024-03-24T00:19:41.450657+0000","last_fullsized":"2024-03-24T00:19:41.450657+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:59:43.400810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'9","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.455998+0000","last_change":"2024-03-24T00:19:41.455998+0000","last_active":"2024-03-24T00:19:41.455998+0000","last_peered":"2024-03-24T00:19:41.455998+0000","last_clean":"2024-03-24T00:19:41.455998+0000","last_became_active":"2024-03-24T00:19:41.452128+0000","last_became_peered":"2024-03-24T00:19:41.452128+0000","last_unstale":"2024-03-24T00:19:41.455998+0000","last_undegraded":"2024-03-24T00:19:41.455998+0000","last_fullsized":"2024-03-24T00:19:41.455998+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:46.106925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.451383+0000","last_change":"2024-03-24T00:18:43.883654+0000","last_active":"2024-03-24T00:19:41.451383+0000","last_peered":"2024-03-24T00:19:41.451383+0000","last_clean":"2024-03-24T00:19:41.451383+0000","last_became_active":"2024-03-24T00:18:43.883335+0000","last_became_peered":"2024-03-24T00:18:43.883335+0000","last_unstale":"2024-03-24T00:19:41.451383+0000","last_undegraded":"2024-03-24T00:19:41.451383+0000","last_fullsized":"2024-03-24T00:19:41.451383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:42:10.856037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.444757+0000","last_change":"2024-03-24T00:18:47.888562+0000","last_active":"2024-03-24T00:19:41.444757+0000","last_peered":"2024-03-24T00:19:41.444757+0000","last_clean":"2024-03-24T00:19:41.444757+0000","last_became_active":"2024-03-24T00:18:47.888074+0000","last_became_peered":"2024-03-24T00:18:47.888074+0000","last_unstale":"2024-03-24T00:19:41.444757+0000","last_undegraded":"2024-03-24T00:19:41.444757+0000","last_fullsized":"2024-03-24T00:19:41.444757+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:01:04.617098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.451564+0000","last_change":"2024-03-24T00:18:43.261668+0000","last_active":"2024-03-24T00:19:41.451564+0000","last_peered":"2024-03-24T00:19:41.451564+0000","last_clean":"2024-03-24T00:19:41.451564+0000","last_became_active":"2024-03-24T00:18:43.261279+0000","last_became_peered":"2024-03-24T00:18:43.261279+0000","last_unstale":"2024-03-24T00:19:41.451564+0000","last_undegraded":"2024-03-24T00:19:41.451564+0000","last_fullsized":"2024-03-24T00:19:41.451564+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:36:05.390006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.467969+0000","last_change":"2024-03-24T00:18:43.889854+0000","last_active":"2024-03-24T00:19:41.467969+0000","last_peered":"2024-03-24T00:19:41.467969+0000","last_clean":"2024-03-24T00:19:41.467969+0000","last_became_active":"2024-03-24T00:18:43.889540+0000","last_became_peered":"2024-03-24T00:18:43.889540+0000","last_unstale":"2024-03-24T00:19:41.467969+0000","last_undegraded":"2024-03-24T00:19:41.467969+0000","last_fullsized":"2024-03-24T00:19:41.467969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:33:58.060564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.454081+0000","last_change":"2024-03-24T00:19:41.454081+0000","last_active":"2024-03-24T00:19:41.454081+0000","last_peered":"2024-03-24T00:19:41.454081+0000","last_clean":"2024-03-24T00:19:41.454081+0000","last_became_active":"2024-03-24T00:19:41.453621+0000","last_became_peered":"2024-03-24T00:19:41.453621+0000","last_unstale":"2024-03-24T00:19:41.454081+0000","last_undegraded":"2024-03-24T00:19:41.454081+0000","last_fullsized":"2024-03-24T00:19:41.454081+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:18:27.484085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449225+0000","last_change":"2024-03-24T00:18:43.253063+0000","last_active":"2024-03-24T00:19:41.449225+0000","last_peered":"2024-03-24T00:19:41.449225+0000","last_clean":"2024-03-24T00:19:41.449225+0000","last_became_active":"2024-03-24T00:18:43.252534+0000","last_became_peered":"2024-03-24T00:18:43.252534+0000","last_unstale":"2024-03-24T00:19:41.449225+0000","last_undegraded":"2024-03-24T00:19:41.449225+0000","last_fullsized":"2024-03-24T00:19:41.449225+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:50:59.512780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450119+0000","last_change":"2024-03-24T00:18:43.885574+0000","last_active":"2024-03-24T00:19:41.450119+0000","last_peered":"2024-03-24T00:19:41.450119+0000","last_clean":"2024-03-24T00:19:41.450119+0000","last_became_active":"2024-03-24T00:18:43.885257+0000","last_became_peered":"2024-03-24T00:18:43.885257+0000","last_unstale":"2024-03-24T00:19:41.450119+0000","last_undegraded":"2024-03-24T00:19:41.450119+0000","last_fullsized":"2024-03-24T00:19:41.450119+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:10.868436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.448145+0000","last_change":"2024-03-24T00:18:47.892739+0000","last_active":"2024-03-24T00:19:41.448145+0000","last_peered":"2024-03-24T00:19:41.448145+0000","last_clean":"2024-03-24T00:19:41.448145+0000","last_became_active":"2024-03-24T00:18:47.892291+0000","last_became_peered":"2024-03-24T00:18:47.892291+0000","last_unstale":"2024-03-24T00:19:41.448145+0000","last_undegraded":"2024-03-24T00:19:41.448145+0000","last_fullsized":"2024-03-24T00:19:41.448145+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:06:16.027968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450981+0000","last_change":"2024-03-24T00:18:43.263856+0000","last_active":"2024-03-24T00:19:41.450981+0000","last_peered":"2024-03-24T00:19:41.450981+0000","last_clean":"2024-03-24T00:19:41.450981+0000","last_became_active":"2024-03-24T00:18:43.263032+0000","last_became_peered":"2024-03-24T00:18:43.263032+0000","last_unstale":"2024-03-24T00:19:41.450981+0000","last_undegraded":"2024-03-24T00:19:41.450981+0000","last_fullsized":"2024-03-24T00:19:41.450981+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:25:03.239696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.446613+0000","last_change":"2024-03-24T00:18:43.884674+0000","last_active":"2024-03-24T00:19:41.446613+0000","last_peered":"2024-03-24T00:19:41.446613+0000","last_clean":"2024-03-24T00:19:41.446613+0000","last_became_active":"2024-03-24T00:18:43.884346+0000","last_became_peered":"2024-03-24T00:18:43.884346+0000","last_unstale":"2024-03-24T00:19:41.446613+0000","last_undegraded":"2024-03-24T00:19:41.446613+0000","last_fullsized":"2024-03-24T00:19:41.446613+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:58:31.474515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450029+0000","last_change":"2024-03-24T00:18:47.892472+0000","last_active":"2024-03-24T00:19:41.450029+0000","last_peered":"2024-03-24T00:19:41.450029+0000","last_clean":"2024-03-24T00:19:41.450029+0000","last_became_active":"2024-03-24T00:18:47.892082+0000","last_became_peered":"2024-03-24T00:18:47.892082+0000","last_unstale":"2024-03-24T00:19:41.450029+0000","last_undegraded":"2024-03-24T00:19:41.450029+0000","last_fullsized":"2024-03-24T00:19:41.450029+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:26:01.084042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.445073+0000","last_change":"2024-03-24T00:18:43.263780+0000","last_active":"2024-03-24T00:19:41.445073+0000","last_peered":"2024-03-24T00:19:41.445073+0000","last_clean":"2024-03-24T00:19:41.445073+0000","last_became_active":"2024-03-24T00:18:43.263446+0000","last_became_peered":"2024-03-24T00:18:43.263446+0000","last_unstale":"2024-03-24T00:19:41.445073+0000","last_undegraded":"2024-03-24T00:19:41.445073+0000","last_fullsized":"2024-03-24T00:19:41.445073+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:09:09.821842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450851+0000","last_change":"2024-03-24T00:18:43.888516+0000","last_active":"2024-03-24T00:19:41.450851+0000","last_peered":"2024-03-24T00:19:41.450851+0000","last_clean":"2024-03-24T00:19:41.450851+0000","last_became_active":"2024-03-24T00:18:43.888190+0000","last_became_peered":"2024-03-24T00:18:43.888190+0000","last_unstale":"2024-03-24T00:19:41.450851+0000","last_undegraded":"2024-03-24T00:19:41.450851+0000","last_fullsized":"2024-03-24T00:19:41.450851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:53:35.685128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449644+0000","last_change":"2024-03-24T00:18:47.897115+0000","last_active":"2024-03-24T00:19:41.449644+0000","last_peered":"2024-03-24T00:19:41.449644+0000","last_clean":"2024-03-24T00:19:41.449644+0000","last_became_active":"2024-03-24T00:18:47.896722+0000","last_became_peered":"2024-03-24T00:18:47.896722+0000","last_unstale":"2024-03-24T00:19:41.449644+0000","last_undegraded":"2024-03-24T00:19:41.449644+0000","last_fullsized":"2024-03-24T00:19:41.449644+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:44:40.541618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450272+0000","last_change":"2024-03-24T00:18:43.263459+0000","last_active":"2024-03-24T00:19:41.450272+0000","last_peered":"2024-03-24T00:19:41.450272+0000","last_clean":"2024-03-24T00:19:41.450272+0000","last_became_active":"2024-03-24T00:18:43.263019+0000","last_became_peered":"2024-03-24T00:18:43.263019+0000","last_unstale":"2024-03-24T00:19:41.450272+0000","last_undegraded":"2024-03-24T00:19:41.450272+0000","last_fullsized":"2024-03-24T00:19:41.450272+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:10:57.171625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.451721+0000","last_change":"2024-03-24T00:18:43.886547+0000","last_active":"2024-03-24T00:19:41.451721+0000","last_peered":"2024-03-24T00:19:41.451721+0000","last_clean":"2024-03-24T00:19:41.451721+0000","last_became_active":"2024-03-24T00:18:43.886217+0000","last_became_peered":"2024-03-24T00:18:43.886217+0000","last_unstale":"2024-03-24T00:19:41.451721+0000","last_undegraded":"2024-03-24T00:19:41.451721+0000","last_fullsized":"2024-03-24T00:19:41.451721+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:59:33.576577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449538+0000","last_change":"2024-03-24T00:18:47.894246+0000","last_active":"2024-03-24T00:19:41.449538+0000","last_peered":"2024-03-24T00:19:41.449538+0000","last_clean":"2024-03-24T00:19:41.449538+0000","last_became_active":"2024-03-24T00:18:47.893837+0000","last_became_peered":"2024-03-24T00:18:47.893837+0000","last_unstale":"2024-03-24T00:19:41.449538+0000","last_undegraded":"2024-03-24T00:19:41.449538+0000","last_fullsized":"2024-03-24T00:19:41.449538+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:33:06.044355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.445352+0000","last_change":"2024-03-24T00:18:43.256808+0000","last_active":"2024-03-24T00:19:41.445352+0000","last_peered":"2024-03-24T00:19:41.445352+0000","last_clean":"2024-03-24T00:19:41.445352+0000","last_became_active":"2024-03-24T00:18:43.256397+0000","last_became_peered":"2024-03-24T00:18:43.256397+0000","last_unstale":"2024-03-24T00:19:41.445352+0000","last_undegraded":"2024-03-24T00:19:41.445352+0000","last_fullsized":"2024-03-24T00:19:41.445352+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:36:13.505016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.452492+0000","last_change":"2024-03-24T00:18:43.887754+0000","last_active":"2024-03-24T00:19:41.452492+0000","last_peered":"2024-03-24T00:19:41.452492+0000","last_clean":"2024-03-24T00:19:41.452492+0000","last_became_active":"2024-03-24T00:18:43.886978+0000","last_became_peered":"2024-03-24T00:18:43.886978+0000","last_unstale":"2024-03-24T00:19:41.452492+0000","last_undegraded":"2024-03-24T00:19:41.452492+0000","last_fullsized":"2024-03-24T00:19:41.452492+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:43:18.186682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.467600+0000","last_change":"2024-03-24T00:18:47.887576+0000","last_active":"2024-03-24T00:19:41.467600+0000","last_peered":"2024-03-24T00:19:41.467600+0000","last_clean":"2024-03-24T00:19:41.467600+0000","last_became_active":"2024-03-24T00:18:47.887190+0000","last_became_peered":"2024-03-24T00:18:47.887190+0000","last_unstale":"2024-03-24T00:19:41.467600+0000","last_undegraded":"2024-03-24T00:19:41.467600+0000","last_fullsized":"2024-03-24T00:19:41.467600+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:10:10.396432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.448598+0000","last_change":"2024-03-24T00:18:43.259927+0000","last_active":"2024-03-24T00:19:41.448598+0000","last_peered":"2024-03-24T00:19:41.448598+0000","last_clean":"2024-03-24T00:19:41.448598+0000","last_became_active":"2024-03-24T00:18:43.259587+0000","last_became_peered":"2024-03-24T00:18:43.259587+0000","last_unstale":"2024-03-24T00:19:41.448598+0000","last_undegraded":"2024-03-24T00:19:41.448598+0000","last_fullsized":"2024-03-24T00:19:41.448598+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:26:06.380010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.452723+0000","last_change":"2024-03-24T00:18:43.897041+0000","last_active":"2024-03-24T00:19:41.452723+0000","last_peered":"2024-03-24T00:19:41.452723+0000","last_clean":"2024-03-24T00:19:41.452723+0000","last_became_active":"2024-03-24T00:18:43.896766+0000","last_became_peered":"2024-03-24T00:18:43.896766+0000","last_unstale":"2024-03-24T00:19:41.452723+0000","last_undegraded":"2024-03-24T00:19:41.452723+0000","last_fullsized":"2024-03-24T00:19:41.452723+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:27:59.577589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.447021+0000","last_change":"2024-03-24T00:18:47.893146+0000","last_active":"2024-03-24T00:19:41.447021+0000","last_peered":"2024-03-24T00:19:41.447021+0000","last_clean":"2024-03-24T00:19:41.447021+0000","last_became_active":"2024-03-24T00:18:47.892150+0000","last_became_peered":"2024-03-24T00:18:47.892150+0000","last_unstale":"2024-03-24T00:19:41.447021+0000","last_undegraded":"2024-03-24T00:19:41.447021+0000","last_fullsized":"2024-03-24T00:19:41.447021+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:28:13.887194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449742+0000","last_change":"2024-03-24T00:18:43.263972+0000","last_active":"2024-03-24T00:19:41.449742+0000","last_peered":"2024-03-24T00:19:41.449742+0000","last_clean":"2024-03-24T00:19:41.449742+0000","last_became_active":"2024-03-24T00:18:43.263577+0000","last_became_peered":"2024-03-24T00:18:43.263577+0000","last_unstale":"2024-03-24T00:19:41.449742+0000","last_undegraded":"2024-03-24T00:19:41.449742+0000","last_fullsized":"2024-03-24T00:19:41.449742+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:55:36.725420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.453461+0000","last_change":"2024-03-24T00:18:43.894722+0000","last_active":"2024-03-24T00:19:41.453461+0000","last_peered":"2024-03-24T00:19:41.453461+0000","last_clean":"2024-03-24T00:19:41.453461+0000","last_became_active":"2024-03-24T00:18:43.894359+0000","last_became_peered":"2024-03-24T00:18:43.894359+0000","last_unstale":"2024-03-24T00:19:41.453461+0000","last_undegraded":"2024-03-24T00:19:41.453461+0000","last_fullsized":"2024-03-24T00:19:41.453461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:35:23.974163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.446675+0000","last_change":"2024-03-24T00:18:43.892749+0000","last_active":"2024-03-24T00:19:41.446675+0000","last_peered":"2024-03-24T00:19:41.446675+0000","last_clean":"2024-03-24T00:19:41.446675+0000","last_became_active":"2024-03-24T00:18:43.892014+0000","last_became_peered":"2024-03-24T00:18:43.892014+0000","last_unstale":"2024-03-24T00:19:41.446675+0000","last_undegraded":"2024-03-24T00:19:41.446675+0000","last_fullsized":"2024-03-24T00:19:41.446675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:53:14.592384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.445003+0000","last_change":"2024-03-24T00:18:43.899540+0000","last_active":"2024-03-24T00:19:41.445003+0000","last_peered":"2024-03-24T00:19:41.445003+0000","last_clean":"2024-03-24T00:19:41.445003+0000","last_became_active":"2024-03-24T00:18:43.899206+0000","last_became_peered":"2024-03-24T00:18:43.899206+0000","last_unstale":"2024-03-24T00:19:41.445003+0000","last_undegraded":"2024-03-24T00:19:41.445003+0000","last_fullsized":"2024-03-24T00:19:41.445003+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:51:17.214705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.444958+0000","last_change":"2024-03-24T00:18:43.896254+0000","last_active":"2024-03-24T00:19:41.444958+0000","last_peered":"2024-03-24T00:19:41.444958+0000","last_clean":"2024-03-24T00:19:41.444958+0000","last_became_active":"2024-03-24T00:18:43.895889+0000","last_became_peered":"2024-03-24T00:18:43.895889+0000","last_unstale":"2024-03-24T00:19:41.444958+0000","last_undegraded":"2024-03-24T00:19:41.444958+0000","last_fullsized":"2024-03-24T00:19:41.444958+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:07:46.403117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.467746+0000","last_change":"2024-03-24T00:18:43.890813+0000","last_active":"2024-03-24T00:19:41.467746+0000","last_peered":"2024-03-24T00:19:41.467746+0000","last_clean":"2024-03-24T00:19:41.467746+0000","last_became_active":"2024-03-24T00:18:43.890407+0000","last_became_peered":"2024-03-24T00:18:43.890407+0000","last_unstale":"2024-03-24T00:19:41.467746+0000","last_undegraded":"2024-03-24T00:19:41.467746+0000","last_fullsized":"2024-03-24T00:19:41.467746+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:57:45.146055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449204+0000","last_change":"2024-03-24T00:18:43.890280+0000","last_active":"2024-03-24T00:19:41.449204+0000","last_peered":"2024-03-24T00:19:41.449204+0000","last_clean":"2024-03-24T00:19:41.449204+0000","last_became_active":"2024-03-24T00:18:43.889888+0000","last_became_peered":"2024-03-24T00:18:43.889888+0000","last_unstale":"2024-03-24T00:19:41.449204+0000","last_undegraded":"2024-03-24T00:19:41.449204+0000","last_fullsized":"2024-03-24T00:19:41.449204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:49:17.010306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.447973+0000","last_change":"2024-03-24T00:18:43.898453+0000","last_active":"2024-03-24T00:19:41.447973+0000","last_peered":"2024-03-24T00:19:41.447973+0000","last_clean":"2024-03-24T00:19:41.447973+0000","last_became_active":"2024-03-24T00:18:43.898081+0000","last_became_peered":"2024-03-24T00:18:43.898081+0000","last_unstale":"2024-03-24T00:19:41.447973+0000","last_undegraded":"2024-03-24T00:19:41.447973+0000","last_fullsized":"2024-03-24T00:19:41.447973+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:03:50.856755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.445528+0000","last_change":"2024-03-24T00:18:43.894667+0000","last_active":"2024-03-24T00:19:41.445528+0000","last_peered":"2024-03-24T00:19:41.445528+0000","last_clean":"2024-03-24T00:19:41.445528+0000","last_became_active":"2024-03-24T00:18:43.893869+0000","last_became_peered":"2024-03-24T00:18:43.893869+0000","last_unstale":"2024-03-24T00:19:41.445528+0000","last_undegraded":"2024-03-24T00:19:41.445528+0000","last_fullsized":"2024-03-24T00:19:41.445528+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:51:43.273867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.451424+0000","last_change":"2024-03-24T00:18:43.893201+0000","last_active":"2024-03-24T00:19:41.451424+0000","last_peered":"2024-03-24T00:19:41.451424+0000","last_clean":"2024-03-24T00:19:41.451424+0000","last_became_active":"2024-03-24T00:18:43.892866+0000","last_became_peered":"2024-03-24T00:18:43.892866+0000","last_unstale":"2024-03-24T00:19:41.451424+0000","last_undegraded":"2024-03-24T00:19:41.451424+0000","last_fullsized":"2024-03-24T00:19:41.451424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:03:16.108806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.447316+0000","last_change":"2024-03-24T00:18:43.882211+0000","last_active":"2024-03-24T00:19:41.447316+0000","last_peered":"2024-03-24T00:19:41.447316+0000","last_clean":"2024-03-24T00:19:41.447316+0000","last_became_active":"2024-03-24T00:18:43.881857+0000","last_became_peered":"2024-03-24T00:18:43.881857+0000","last_unstale":"2024-03-24T00:19:41.447316+0000","last_undegraded":"2024-03-24T00:19:41.447316+0000","last_fullsized":"2024-03-24T00:19:41.447316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:36:45.911853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.451050+0000","last_change":"2024-03-24T00:18:43.892599+0000","last_active":"2024-03-24T00:19:41.451050+0000","last_peered":"2024-03-24T00:19:41.451050+0000","last_clean":"2024-03-24T00:19:41.451050+0000","last_became_active":"2024-03-24T00:18:43.891966+0000","last_became_peered":"2024-03-24T00:18:43.891966+0000","last_unstale":"2024-03-24T00:19:41.451050+0000","last_undegraded":"2024-03-24T00:19:41.451050+0000","last_fullsized":"2024-03-24T00:19:41.451050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:22:03.486466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449819+0000","last_change":"2024-03-24T00:18:43.892410+0000","last_active":"2024-03-24T00:19:41.449819+0000","last_peered":"2024-03-24T00:19:41.449819+0000","last_clean":"2024-03-24T00:19:41.449819+0000","last_became_active":"2024-03-24T00:18:43.891140+0000","last_became_peered":"2024-03-24T00:18:43.891140+0000","last_unstale":"2024-03-24T00:19:41.449819+0000","last_undegraded":"2024-03-24T00:19:41.449819+0000","last_fullsized":"2024-03-24T00:19:41.449819+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:03:14.252863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449463+0000","last_change":"2024-03-24T00:18:43.876508+0000","last_active":"2024-03-24T00:19:41.449463+0000","last_peered":"2024-03-24T00:19:41.449463+0000","last_clean":"2024-03-24T00:19:41.449463+0000","last_became_active":"2024-03-24T00:18:43.876204+0000","last_became_peered":"2024-03-24T00:18:43.876204+0000","last_unstale":"2024-03-24T00:19:41.449463+0000","last_undegraded":"2024-03-24T00:19:41.449463+0000","last_fullsized":"2024-03-24T00:19:41.449463+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:35.358964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.446831+0000","last_change":"2024-03-24T00:18:43.887855+0000","last_active":"2024-03-24T00:19:41.446831+0000","last_peered":"2024-03-24T00:19:41.446831+0000","last_clean":"2024-03-24T00:19:41.446831+0000","last_became_active":"2024-03-24T00:18:43.887520+0000","last_became_peered":"2024-03-24T00:18:43.887520+0000","last_unstale":"2024-03-24T00:19:41.446831+0000","last_undegraded":"2024-03-24T00:19:41.446831+0000","last_fullsized":"2024-03-24T00:19:41.446831+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:17:15.930626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65776,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":440,"num_read_kb":291,"num_write":161,"num_write_kb":114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":62391,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":197328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383713,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1028096,"data_stored":500865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645350,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37016,"kb_used_data":1672,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708136,"statfs":{"total":95995035648,"available":95957131264,"internally_reserved":0,"allocated":1712128,"data_stored":1149874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808876,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1044480,"data_stored":505163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070512,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1036288,"data_stored":500405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332150,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28620,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716532,"statfs":{"total":95995035648,"available":95965728768,"internally_reserved":0,"allocated":1634304,"data_stored":1096896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495674,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1019904,"data_stored":496243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1085440,"data_stored":557928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1044480,"data_stored":497631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1036288,"data_stored":497860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182473,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1032192,"data_stored":496437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920837,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36424,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708728,"statfs":{"total":95995035648,"available":95957737472,"internally_reserved":0,"allocated":1105920,"data_stored":563250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757311,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36944,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708208,"statfs":{"total":95995035648,"available":95957204992,"internally_reserved":0,"allocated":1642496,"data_stored":1092484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":63001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:20:21.123 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T00:20:21.366 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:20:21.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:21 smithi106 ceph-mon[36505]: pgmap v341: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:20:21.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:21 smithi106 ceph-mon[36505]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:20:21.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:21 smithi078 ceph-mon[32429]: pgmap v341: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:20:21.717 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:21 smithi078 ceph-mon[32429]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:20:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:21 smithi107 ceph-mon[36485]: pgmap v341: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:20:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:21 smithi107 ceph-mon[36485]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:20:22.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.984+0000 7f44dd8da700 1 -- 172.21.15.78:0/1590803965 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44d81070d0 msgr2=0x7f44d81074b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:22.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.984+0000 7f44dd8da700 1 --2- 172.21.15.78:0/1590803965 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44d81070d0 0x7f44d81074b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f44d0009990 tx=0x7f44d0009ca0 comp rx=0 tx=0).stop 2024-03-24T00:20:22.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.984+0000 7f44dd8da700 1 -- 172.21.15.78:0/1590803965 shutdown_connections 2024-03-24T00:20:22.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.984+0000 7f44dd8da700 1 --2- 172.21.15.78:0/1590803965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44d8101a30 0x7f44d810df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:22.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.984+0000 7f44dd8da700 1 --2- 172.21.15.78:0/1590803965 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44d81070d0 0x7f44d81074b0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:22.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.984+0000 7f44dd8da700 1 --2- 172.21.15.78:0/1590803965 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44d8101090 0x7f44d81014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:22.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.984+0000 7f44dd8da700 1 -- 172.21.15.78:0/1590803965 >> 172.21.15.78:0/1590803965 conn(0x7f44d8077fc0 msgr2=0x7f44d80783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:22.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.984+0000 7f44dd8da700 1 -- 172.21.15.78:0/1590803965 shutdown_connections 2024-03-24T00:20:22.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.984+0000 7f44dd8da700 1 -- 172.21.15.78:0/1590803965 wait complete. 2024-03-24T00:20:22.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.985+0000 7f44dd8da700 1 Processor -- start 2024-03-24T00:20:22.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.985+0000 7f44dd8da700 1 -- start start 2024-03-24T00:20:22.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.985+0000 7f44dd8da700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44d8101090 0x7f44d8199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:22.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.985+0000 7f44dd8da700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44d8101a30 0x7f44d8199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:22.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.985+0000 7f44dd8da700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44d81070d0 0x7f44d81a47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:22.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.985+0000 7f44dd8da700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d8111330 con 0x7f44d8101090 2024-03-24T00:20:22.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.985+0000 7f44dd8da700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d81111b0 con 0x7f44d81070d0 2024-03-24T00:20:22.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.985+0000 7f44dd8da700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d81114b0 con 0x7f44d8101a30 2024-03-24T00:20:22.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.985+0000 7f44d6ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44d8101090 0x7f44d8199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:22.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.985+0000 7f44d6ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44d8101090 0x7f44d8199750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41504/0 (socket says 172.21.15.78:41504) 2024-03-24T00:20:22.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.985+0000 7f44d6ffd700 1 -- 172.21.15.78:0/550068048 learned_addr learned my addr 172.21.15.78:0/550068048 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:22.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.985+0000 7f44d67fc700 1 --2- 172.21.15.78:0/550068048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44d8101a30 0x7f44d8199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:22.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.985+0000 7f44d77fe700 1 --2- 172.21.15.78:0/550068048 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44d81070d0 0x7f44d81a47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:22.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.986+0000 7f44d67fc700 1 -- 172.21.15.78:0/550068048 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44d81070d0 msgr2=0x7f44d81a47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:22.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.986+0000 7f44d67fc700 1 --2- 172.21.15.78:0/550068048 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44d81070d0 0x7f44d81a47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:22.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.986+0000 7f44d67fc700 1 -- 172.21.15.78:0/550068048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44d8101090 msgr2=0x7f44d8199750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:22.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.986+0000 7f44d67fc700 1 --2- 172.21.15.78:0/550068048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44d8101090 0x7f44d8199750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:22.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.986+0000 7f44d67fc700 1 -- 172.21.15.78:0/550068048 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d81a4d10 con 0x7f44d8101a30 2024-03-24T00:20:22.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.986+0000 7f44d6ffd700 1 --2- 172.21.15.78:0/550068048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44d8101090 0x7f44d8199750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:22.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.986+0000 7f44d67fc700 1 --2- 172.21.15.78:0/550068048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44d8101a30 0x7f44d8199c90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f44cc00e9f0 tx=0x7f44cc00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:22.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.986+0000 7f44dc8d8700 1 -- 172.21.15.78:0/550068048 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44cc00c9c0 con 0x7f44d8101a30 2024-03-24T00:20:22.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.986+0000 7f44dd8da700 1 -- 172.21.15.78:0/550068048 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44d81a4f40 con 0x7f44d8101a30 2024-03-24T00:20:22.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.986+0000 7f44dc8d8700 1 -- 172.21.15.78:0/550068048 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f44cc00cb60 con 0x7f44d8101a30 2024-03-24T00:20:22.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.986+0000 7f44dc8d8700 1 -- 172.21.15.78:0/550068048 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44cc018680 con 0x7f44d8101a30 2024-03-24T00:20:22.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.986+0000 7f44dd8da700 1 -- 172.21.15.78:0/550068048 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d81a5440 con 0x7f44d8101a30 2024-03-24T00:20:22.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.987+0000 7f44dd8da700 1 -- 172.21.15.78:0/550068048 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44d8068e50 con 0x7f44d8101a30 2024-03-24T00:20:22.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.988+0000 7f44dc8d8700 1 -- 172.21.15.78:0/550068048 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f44cc047a90 con 0x7f44d8101a30 2024-03-24T00:20:22.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.989+0000 7f44dc8d8700 1 --2- 172.21.15.78:0/550068048 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f44c00a2920 0x7f44c00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:22.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.989+0000 7f44d6ffd700 1 --2- 172.21.15.78:0/550068048 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f44c00a2920 0x7f44c00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:22.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.989+0000 7f44dc8d8700 1 -- 172.21.15.78:0/550068048 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f44cc0cefc0 con 0x7f44d8101a30 2024-03-24T00:20:22.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.989+0000 7f44d6ffd700 1 --2- 172.21.15.78:0/550068048 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f44c00a2920 0x7f44c00a4de0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f44d000b2e0 tx=0x7f44d0009590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:22.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:22.992+0000 7f44dc8d8700 1 -- 172.21.15.78:0/550068048 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f44cc09bba0 con 0x7f44d8101a30 2024-03-24T00:20:23.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:23.184+0000 7f44dd8da700 1 -- 172.21.15.78:0/550068048 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f44d80625d0 con 0x7f44c00a2920 2024-03-24T00:20:23.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:23.193+0000 7f44dc8d8700 1 -- 172.21.15.78:0/550068048 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340378 (secure 0 0 0) 0x7f44d007b1b0 con 0x7f44c00a2920 2024-03-24T00:20:23.197 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:20:23.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:23.197+0000 7f44dd8da700 1 -- 172.21.15.78:0/550068048 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f44c00a2920 msgr2=0x7f44c00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:23.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:23.197+0000 7f44dd8da700 1 --2- 172.21.15.78:0/550068048 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f44c00a2920 0x7f44c00a4de0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f44d000b2e0 tx=0x7f44d0009590 comp rx=0 tx=0).stop 2024-03-24T00:20:23.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:23.197+0000 7f44dd8da700 1 -- 172.21.15.78:0/550068048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44d8101a30 msgr2=0x7f44d8199c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:23.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:23.197+0000 7f44dd8da700 1 --2- 172.21.15.78:0/550068048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44d8101a30 0x7f44d8199c90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f44cc00e9f0 tx=0x7f44cc00ed00 comp rx=0 tx=0).stop 2024-03-24T00:20:23.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:23.197+0000 7f44dd8da700 1 -- 172.21.15.78:0/550068048 shutdown_connections 2024-03-24T00:20:23.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:23.197+0000 7f44dd8da700 1 --2- 172.21.15.78:0/550068048 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f44c00a2920 0x7f44c00a4de0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:23.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:23.197+0000 7f44dd8da700 1 --2- 172.21.15.78:0/550068048 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44d81070d0 0x7f44d81a47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:23.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:23.197+0000 7f44dd8da700 1 --2- 172.21.15.78:0/550068048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44d8101090 0x7f44d8199750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:23.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:23.197+0000 7f44dd8da700 1 --2- 172.21.15.78:0/550068048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44d8101a30 0x7f44d8199c90 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:23.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:23.197+0000 7f44dd8da700 1 -- 172.21.15.78:0/550068048 >> 172.21.15.78:0/550068048 conn(0x7f44d8077fc0 msgr2=0x7f44d80ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:23.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:23.197+0000 7f44dd8da700 1 -- 172.21.15.78:0/550068048 shutdown_connections 2024-03-24T00:20:23.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:23.197+0000 7f44dd8da700 1 -- 172.21.15.78:0/550068048 wait complete. 2024-03-24T00:20:23.201 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-03-24T00:20:23.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:23 smithi078 ceph-mon[32429]: pgmap v342: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:20:23.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:23 smithi106 ceph-mon[36505]: pgmap v342: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:20:23.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:23 smithi107 ceph-mon[36485]: pgmap v342: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:20:23.949 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-03-24T00:20:22.305813+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656378,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":586,"num_read_kb":415,"num_write":344,"num_write_kb":3054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1243127,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":278,"ondisk_log_size":278,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":413248,"kb_used_data":14084,"kb_used_omap":18,"kb_used_meta":398957,"kb_avail":1124528576,"statfs":{"total":1151940427776,"available":1151517261824,"internally_reserved":0,"allocated":14422016,"data_stored":7955036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":408532284},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":35,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003404"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.448551+0000","last_change":"2024-03-24T00:18:43.890863+0000","last_active":"2024-03-24T00:19:41.448551+0000","last_peered":"2024-03-24T00:19:41.448551+0000","last_clean":"2024-03-24T00:19:41.448551+0000","last_became_active":"2024-03-24T00:18:43.890503+0000","last_became_peered":"2024-03-24T00:18:43.890503+0000","last_unstale":"2024-03-24T00:19:41.448551+0000","last_undegraded":"2024-03-24T00:19:41.448551+0000","last_fullsized":"2024-03-24T00:19:41.448551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:33:10.518835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.467919+0000","last_change":"2024-03-24T00:18:43.882039+0000","last_active":"2024-03-24T00:19:41.467919+0000","last_peered":"2024-03-24T00:19:41.467919+0000","last_clean":"2024-03-24T00:19:41.467919+0000","last_became_active":"2024-03-24T00:18:43.881704+0000","last_became_peered":"2024-03-24T00:18:43.881704+0000","last_unstale":"2024-03-24T00:19:41.467919+0000","last_undegraded":"2024-03-24T00:19:41.467919+0000","last_fullsized":"2024-03-24T00:19:41.467919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:04:46.386032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.467770+0000","last_change":"2024-03-24T00:18:43.888649+0000","last_active":"2024-03-24T00:19:41.467770+0000","last_peered":"2024-03-24T00:19:41.467770+0000","last_clean":"2024-03-24T00:19:41.467770+0000","last_became_active":"2024-03-24T00:18:43.888339+0000","last_became_peered":"2024-03-24T00:18:43.888339+0000","last_unstale":"2024-03-24T00:19:41.467770+0000","last_undegraded":"2024-03-24T00:19:41.467770+0000","last_fullsized":"2024-03-24T00:19:41.467770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:28:35.787820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.452431+0000","last_change":"2024-03-24T00:19:41.451897+0000","last_active":"2024-03-24T00:19:41.452431+0000","last_peered":"2024-03-24T00:19:41.452431+0000","last_clean":"2024-03-24T00:19:41.452431+0000","last_became_active":"2024-03-24T00:19:41.451597+0000","last_became_peered":"2024-03-24T00:19:41.451597+0000","last_unstale":"2024-03-24T00:19:41.452431+0000","last_undegraded":"2024-03-24T00:19:41.452431+0000","last_fullsized":"2024-03-24T00:19:41.452431+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:10:32.717345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450240+0000","last_change":"2024-03-24T00:18:43.890677+0000","last_active":"2024-03-24T00:19:41.450240+0000","last_peered":"2024-03-24T00:19:41.450240+0000","last_clean":"2024-03-24T00:19:41.450240+0000","last_became_active":"2024-03-24T00:18:43.887161+0000","last_became_peered":"2024-03-24T00:18:43.887161+0000","last_unstale":"2024-03-24T00:19:41.450240+0000","last_undegraded":"2024-03-24T00:19:41.450240+0000","last_fullsized":"2024-03-24T00:19:41.450240+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:09:32.160568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.446421+0000","last_change":"2024-03-24T00:18:43.887293+0000","last_active":"2024-03-24T00:19:41.446421+0000","last_peered":"2024-03-24T00:19:41.446421+0000","last_clean":"2024-03-24T00:19:41.446421+0000","last_became_active":"2024-03-24T00:18:43.886961+0000","last_became_peered":"2024-03-24T00:18:43.886961+0000","last_unstale":"2024-03-24T00:19:41.446421+0000","last_undegraded":"2024-03-24T00:19:41.446421+0000","last_fullsized":"2024-03-24T00:19:41.446421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:48:14.312376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.453058+0000","last_change":"2024-03-24T00:18:43.876014+0000","last_active":"2024-03-24T00:19:41.453058+0000","last_peered":"2024-03-24T00:19:41.453058+0000","last_clean":"2024-03-24T00:19:41.453058+0000","last_became_active":"2024-03-24T00:18:43.875619+0000","last_became_peered":"2024-03-24T00:18:43.875619+0000","last_unstale":"2024-03-24T00:19:41.453058+0000","last_undegraded":"2024-03-24T00:19:41.453058+0000","last_fullsized":"2024-03-24T00:19:41.453058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:10:26.707681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450799+0000","last_change":"2024-03-24T00:18:43.894851+0000","last_active":"2024-03-24T00:19:41.450799+0000","last_peered":"2024-03-24T00:19:41.450799+0000","last_clean":"2024-03-24T00:19:41.450799+0000","last_became_active":"2024-03-24T00:18:43.894364+0000","last_became_peered":"2024-03-24T00:18:43.894364+0000","last_unstale":"2024-03-24T00:19:41.450799+0000","last_undegraded":"2024-03-24T00:19:41.450799+0000","last_fullsized":"2024-03-24T00:19:41.450799+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:52:01.538415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450781+0000","last_change":"2024-03-24T00:18:43.898453+0000","last_active":"2024-03-24T00:19:41.450781+0000","last_peered":"2024-03-24T00:19:41.450781+0000","last_clean":"2024-03-24T00:19:41.450781+0000","last_became_active":"2024-03-24T00:18:43.898118+0000","last_became_peered":"2024-03-24T00:18:43.898118+0000","last_unstale":"2024-03-24T00:19:41.450781+0000","last_undegraded":"2024-03-24T00:19:41.450781+0000","last_fullsized":"2024-03-24T00:19:41.450781+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:18:00.478010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.446702+0000","last_change":"2024-03-24T00:18:43.894133+0000","last_active":"2024-03-24T00:19:41.446702+0000","last_peered":"2024-03-24T00:19:41.446702+0000","last_clean":"2024-03-24T00:19:41.446702+0000","last_became_active":"2024-03-24T00:18:43.893777+0000","last_became_peered":"2024-03-24T00:18:43.893777+0000","last_unstale":"2024-03-24T00:19:41.446702+0000","last_undegraded":"2024-03-24T00:19:41.446702+0000","last_fullsized":"2024-03-24T00:19:41.446702+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:46:33.029934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.452898+0000","last_change":"2024-03-24T00:18:43.888024+0000","last_active":"2024-03-24T00:19:41.452898+0000","last_peered":"2024-03-24T00:19:41.452898+0000","last_clean":"2024-03-24T00:19:41.452898+0000","last_became_active":"2024-03-24T00:18:43.887430+0000","last_became_peered":"2024-03-24T00:18:43.887430+0000","last_unstale":"2024-03-24T00:19:41.452898+0000","last_undegraded":"2024-03-24T00:19:41.452898+0000","last_fullsized":"2024-03-24T00:19:41.452898+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:54:29.299788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449774+0000","last_change":"2024-03-24T00:18:43.889240+0000","last_active":"2024-03-24T00:19:41.449774+0000","last_peered":"2024-03-24T00:19:41.449774+0000","last_clean":"2024-03-24T00:19:41.449774+0000","last_became_active":"2024-03-24T00:18:43.888861+0000","last_became_peered":"2024-03-24T00:18:43.888861+0000","last_unstale":"2024-03-24T00:19:41.449774+0000","last_undegraded":"2024-03-24T00:19:41.449774+0000","last_fullsized":"2024-03-24T00:19:41.449774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:41:56.637532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.455705+0000","last_change":"2024-03-24T00:19:41.455705+0000","last_active":"2024-03-24T00:19:41.455705+0000","last_peered":"2024-03-24T00:19:41.455705+0000","last_clean":"2024-03-24T00:19:41.455705+0000","last_became_active":"2024-03-24T00:19:41.455320+0000","last_became_peered":"2024-03-24T00:19:41.455320+0000","last_unstale":"2024-03-24T00:19:41.455705+0000","last_undegraded":"2024-03-24T00:19:41.455705+0000","last_fullsized":"2024-03-24T00:19:41.455705+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:06.280616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.467662+0000","last_change":"2024-03-24T00:18:43.883717+0000","last_active":"2024-03-24T00:19:41.467662+0000","last_peered":"2024-03-24T00:19:41.467662+0000","last_clean":"2024-03-24T00:19:41.467662+0000","last_became_active":"2024-03-24T00:18:43.883329+0000","last_became_peered":"2024-03-24T00:18:43.883329+0000","last_unstale":"2024-03-24T00:19:41.467662+0000","last_undegraded":"2024-03-24T00:19:41.467662+0000","last_fullsized":"2024-03-24T00:19:41.467662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:26:15.127054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.467573+0000","last_change":"2024-03-24T00:18:43.890831+0000","last_active":"2024-03-24T00:19:41.467573+0000","last_peered":"2024-03-24T00:19:41.467573+0000","last_clean":"2024-03-24T00:19:41.467573+0000","last_became_active":"2024-03-24T00:18:43.890448+0000","last_became_peered":"2024-03-24T00:18:43.890448+0000","last_unstale":"2024-03-24T00:19:41.467573+0000","last_undegraded":"2024-03-24T00:19:41.467573+0000","last_fullsized":"2024-03-24T00:19:41.467573+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:56:13.222271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.448637+0000","last_change":"2024-03-24T00:18:43.890309+0000","last_active":"2024-03-24T00:19:41.448637+0000","last_peered":"2024-03-24T00:19:41.448637+0000","last_clean":"2024-03-24T00:19:41.448637+0000","last_became_active":"2024-03-24T00:18:43.889138+0000","last_became_peered":"2024-03-24T00:18:43.889138+0000","last_unstale":"2024-03-24T00:19:41.448637+0000","last_undegraded":"2024-03-24T00:19:41.448637+0000","last_fullsized":"2024-03-24T00:19:41.448637+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:13:52.480938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.448337+0000","last_change":"2024-03-24T00:18:43.891633+0000","last_active":"2024-03-24T00:19:41.448337+0000","last_peered":"2024-03-24T00:19:41.448337+0000","last_clean":"2024-03-24T00:19:41.448337+0000","last_became_active":"2024-03-24T00:18:43.891303+0000","last_became_peered":"2024-03-24T00:18:43.891303+0000","last_unstale":"2024-03-24T00:19:41.448337+0000","last_undegraded":"2024-03-24T00:19:41.448337+0000","last_fullsized":"2024-03-24T00:19:41.448337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:34:01.874142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.448130+0000","last_change":"2024-03-24T00:18:43.887695+0000","last_active":"2024-03-24T00:19:41.448130+0000","last_peered":"2024-03-24T00:19:41.448130+0000","last_clean":"2024-03-24T00:19:41.448130+0000","last_became_active":"2024-03-24T00:18:43.887359+0000","last_became_peered":"2024-03-24T00:18:43.887359+0000","last_unstale":"2024-03-24T00:19:41.448130+0000","last_undegraded":"2024-03-24T00:19:41.448130+0000","last_fullsized":"2024-03-24T00:19:41.448130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:00:50.951362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.445281+0000","last_change":"2024-03-24T00:18:43.895671+0000","last_active":"2024-03-24T00:19:41.445281+0000","last_peered":"2024-03-24T00:19:41.445281+0000","last_clean":"2024-03-24T00:19:41.445281+0000","last_became_active":"2024-03-24T00:18:43.895267+0000","last_became_peered":"2024-03-24T00:18:43.895267+0000","last_unstale":"2024-03-24T00:19:41.445281+0000","last_undegraded":"2024-03-24T00:19:41.445281+0000","last_fullsized":"2024-03-24T00:19:41.445281+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:24:48.014768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.444913+0000","last_change":"2024-03-24T00:18:47.896879+0000","last_active":"2024-03-24T00:19:41.444913+0000","last_peered":"2024-03-24T00:19:41.444913+0000","last_clean":"2024-03-24T00:19:41.444913+0000","last_became_active":"2024-03-24T00:18:47.896517+0000","last_became_peered":"2024-03-24T00:18:47.896517+0000","last_unstale":"2024-03-24T00:19:41.444913+0000","last_undegraded":"2024-03-24T00:19:41.444913+0000","last_fullsized":"2024-03-24T00:19:41.444913+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:28:34.331404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.461363+0000","last_change":"2024-03-24T00:19:41.461363+0000","last_active":"2024-03-24T00:19:41.461363+0000","last_peered":"2024-03-24T00:19:41.461363+0000","last_clean":"2024-03-24T00:19:41.461363+0000","last_became_active":"2024-03-24T00:19:41.454206+0000","last_became_peered":"2024-03-24T00:19:41.454206+0000","last_unstale":"2024-03-24T00:19:41.461363+0000","last_undegraded":"2024-03-24T00:19:41.461363+0000","last_fullsized":"2024-03-24T00:19:41.461363+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:56:22.294313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,4],"acting":[9,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.451138+0000","last_change":"2024-03-24T00:18:43.892643+0000","last_active":"2024-03-24T00:19:41.451138+0000","last_peered":"2024-03-24T00:19:41.451138+0000","last_clean":"2024-03-24T00:19:41.451138+0000","last_became_active":"2024-03-24T00:18:43.892317+0000","last_became_peered":"2024-03-24T00:18:43.892317+0000","last_unstale":"2024-03-24T00:19:41.451138+0000","last_undegraded":"2024-03-24T00:19:41.451138+0000","last_fullsized":"2024-03-24T00:19:41.451138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:26:28.374587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449815+0000","last_change":"2024-03-24T00:18:47.895307+0000","last_active":"2024-03-24T00:19:41.449815+0000","last_peered":"2024-03-24T00:19:41.449815+0000","last_clean":"2024-03-24T00:19:41.449815+0000","last_became_active":"2024-03-24T00:18:47.894988+0000","last_became_peered":"2024-03-24T00:18:47.894988+0000","last_unstale":"2024-03-24T00:19:41.449815+0000","last_undegraded":"2024-03-24T00:19:41.449815+0000","last_fullsized":"2024-03-24T00:19:41.449815+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:48:41.068076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.451359+0000","last_change":"2024-03-24T00:18:43.255861+0000","last_active":"2024-03-24T00:19:41.451359+0000","last_peered":"2024-03-24T00:19:41.451359+0000","last_clean":"2024-03-24T00:19:41.451359+0000","last_became_active":"2024-03-24T00:18:43.255529+0000","last_became_peered":"2024-03-24T00:18:43.255529+0000","last_unstale":"2024-03-24T00:19:41.451359+0000","last_undegraded":"2024-03-24T00:19:41.451359+0000","last_fullsized":"2024-03-24T00:19:41.451359+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:43:19.617673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.467953+0000","last_change":"2024-03-24T00:18:43.886651+0000","last_active":"2024-03-24T00:19:41.467953+0000","last_peered":"2024-03-24T00:19:41.467953+0000","last_clean":"2024-03-24T00:19:41.467953+0000","last_became_active":"2024-03-24T00:18:43.886341+0000","last_became_peered":"2024-03-24T00:18:43.886341+0000","last_unstale":"2024-03-24T00:19:41.467953+0000","last_undegraded":"2024-03-24T00:19:41.467953+0000","last_fullsized":"2024-03-24T00:19:41.467953+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:14:11.993717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449913+0000","last_change":"2024-03-24T00:18:47.897123+0000","last_active":"2024-03-24T00:19:41.449913+0000","last_peered":"2024-03-24T00:19:41.449913+0000","last_clean":"2024-03-24T00:19:41.449913+0000","last_became_active":"2024-03-24T00:18:47.896762+0000","last_became_peered":"2024-03-24T00:18:47.896762+0000","last_unstale":"2024-03-24T00:19:41.449913+0000","last_undegraded":"2024-03-24T00:19:41.449913+0000","last_fullsized":"2024-03-24T00:19:41.449913+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:45:50.639038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450830+0000","last_change":"2024-03-24T00:18:43.263895+0000","last_active":"2024-03-24T00:19:41.450830+0000","last_peered":"2024-03-24T00:19:41.450830+0000","last_clean":"2024-03-24T00:19:41.450830+0000","last_became_active":"2024-03-24T00:18:43.263557+0000","last_became_peered":"2024-03-24T00:18:43.263557+0000","last_unstale":"2024-03-24T00:19:41.450830+0000","last_undegraded":"2024-03-24T00:19:41.450830+0000","last_fullsized":"2024-03-24T00:19:41.450830+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:44:28.763293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450978+0000","last_change":"2024-03-24T00:18:43.888820+0000","last_active":"2024-03-24T00:19:41.450978+0000","last_peered":"2024-03-24T00:19:41.450978+0000","last_clean":"2024-03-24T00:19:41.450978+0000","last_became_active":"2024-03-24T00:18:43.888382+0000","last_became_peered":"2024-03-24T00:18:43.888382+0000","last_unstale":"2024-03-24T00:19:41.450978+0000","last_undegraded":"2024-03-24T00:19:41.450978+0000","last_fullsized":"2024-03-24T00:19:41.450978+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:04.097113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449456+0000","last_change":"2024-03-24T00:18:47.896443+0000","last_active":"2024-03-24T00:19:41.449456+0000","last_peered":"2024-03-24T00:19:41.449456+0000","last_clean":"2024-03-24T00:19:41.449456+0000","last_became_active":"2024-03-24T00:18:47.895792+0000","last_became_peered":"2024-03-24T00:18:47.895792+0000","last_unstale":"2024-03-24T00:19:41.449456+0000","last_undegraded":"2024-03-24T00:19:41.449456+0000","last_fullsized":"2024-03-24T00:19:41.449456+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:59:42.360902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450638+0000","last_change":"2024-03-24T00:18:43.261085+0000","last_active":"2024-03-24T00:19:41.450638+0000","last_peered":"2024-03-24T00:19:41.450638+0000","last_clean":"2024-03-24T00:19:41.450638+0000","last_became_active":"2024-03-24T00:18:43.260724+0000","last_became_peered":"2024-03-24T00:18:43.260724+0000","last_unstale":"2024-03-24T00:19:41.450638+0000","last_undegraded":"2024-03-24T00:19:41.450638+0000","last_fullsized":"2024-03-24T00:19:41.450638+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:05:20.992776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.448076+0000","last_change":"2024-03-24T00:18:43.892891+0000","last_active":"2024-03-24T00:19:41.448076+0000","last_peered":"2024-03-24T00:19:41.448076+0000","last_clean":"2024-03-24T00:19:41.448076+0000","last_became_active":"2024-03-24T00:18:43.892430+0000","last_became_peered":"2024-03-24T00:18:43.892430+0000","last_unstale":"2024-03-24T00:19:41.448076+0000","last_undegraded":"2024-03-24T00:19:41.448076+0000","last_fullsized":"2024-03-24T00:19:41.448076+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:30:24.926031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.445303+0000","last_change":"2024-03-24T00:18:47.889119+0000","last_active":"2024-03-24T00:19:41.445303+0000","last_peered":"2024-03-24T00:19:41.445303+0000","last_clean":"2024-03-24T00:19:41.445303+0000","last_became_active":"2024-03-24T00:18:47.888620+0000","last_became_peered":"2024-03-24T00:18:47.888620+0000","last_unstale":"2024-03-24T00:19:41.445303+0000","last_undegraded":"2024-03-24T00:19:41.445303+0000","last_fullsized":"2024-03-24T00:19:41.445303+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:51.483286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'6","reported_seq":109,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:20:19.952313+0000","last_change":"2024-03-24T00:18:43.251824+0000","last_active":"2024-03-24T00:20:19.952313+0000","last_peered":"2024-03-24T00:20:19.952313+0000","last_clean":"2024-03-24T00:20:19.952313+0000","last_became_active":"2024-03-24T00:18:43.251458+0000","last_became_peered":"2024-03-24T00:18:43.251458+0000","last_unstale":"2024-03-24T00:20:19.952313+0000","last_undegraded":"2024-03-24T00:20:19.952313+0000","last_fullsized":"2024-03-24T00:20:19.952313+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:13:02.303670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450437+0000","last_change":"2024-03-24T00:18:43.891289+0000","last_active":"2024-03-24T00:19:41.450437+0000","last_peered":"2024-03-24T00:19:41.450437+0000","last_clean":"2024-03-24T00:19:41.450437+0000","last_became_active":"2024-03-24T00:18:43.890725+0000","last_became_peered":"2024-03-24T00:18:43.890725+0000","last_unstale":"2024-03-24T00:19:41.450437+0000","last_undegraded":"2024-03-24T00:19:41.450437+0000","last_fullsized":"2024-03-24T00:19:41.450437+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:37:41.096520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.445139+0000","last_change":"2024-03-24T00:18:47.896175+0000","last_active":"2024-03-24T00:19:41.445139+0000","last_peered":"2024-03-24T00:19:41.445139+0000","last_clean":"2024-03-24T00:19:41.445139+0000","last_became_active":"2024-03-24T00:18:47.895723+0000","last_became_peered":"2024-03-24T00:18:47.895723+0000","last_unstale":"2024-03-24T00:19:41.445139+0000","last_undegraded":"2024-03-24T00:19:41.445139+0000","last_fullsized":"2024-03-24T00:19:41.445139+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:41:49.167532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.446324+0000","last_change":"2024-03-24T00:18:43.258986+0000","last_active":"2024-03-24T00:19:41.446324+0000","last_peered":"2024-03-24T00:19:41.446324+0000","last_clean":"2024-03-24T00:19:41.446324+0000","last_became_active":"2024-03-24T00:18:43.258654+0000","last_became_peered":"2024-03-24T00:18:43.258654+0000","last_unstale":"2024-03-24T00:19:41.446324+0000","last_undegraded":"2024-03-24T00:19:41.446324+0000","last_fullsized":"2024-03-24T00:19:41.446324+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:34:44.339991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.447947+0000","last_change":"2024-03-24T00:18:43.885852+0000","last_active":"2024-03-24T00:19:41.447947+0000","last_peered":"2024-03-24T00:19:41.447947+0000","last_clean":"2024-03-24T00:19:41.447947+0000","last_became_active":"2024-03-24T00:18:43.885538+0000","last_became_peered":"2024-03-24T00:18:43.885538+0000","last_unstale":"2024-03-24T00:19:41.447947+0000","last_undegraded":"2024-03-24T00:19:41.447947+0000","last_fullsized":"2024-03-24T00:19:41.447947+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:42:02.879892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.448233+0000","last_change":"2024-03-24T00:18:47.893230+0000","last_active":"2024-03-24T00:19:41.448233+0000","last_peered":"2024-03-24T00:19:41.448233+0000","last_clean":"2024-03-24T00:19:41.448233+0000","last_became_active":"2024-03-24T00:18:47.892522+0000","last_became_peered":"2024-03-24T00:18:47.892522+0000","last_unstale":"2024-03-24T00:19:41.448233+0000","last_undegraded":"2024-03-24T00:19:41.448233+0000","last_fullsized":"2024-03-24T00:19:41.448233+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:10:11.768154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449760+0000","last_change":"2024-03-24T00:18:43.262255+0000","last_active":"2024-03-24T00:19:41.449760+0000","last_peered":"2024-03-24T00:19:41.449760+0000","last_clean":"2024-03-24T00:19:41.449760+0000","last_became_active":"2024-03-24T00:18:43.261940+0000","last_became_peered":"2024-03-24T00:18:43.261940+0000","last_unstale":"2024-03-24T00:19:41.449760+0000","last_undegraded":"2024-03-24T00:19:41.449760+0000","last_fullsized":"2024-03-24T00:19:41.449760+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:58:48.789845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450755+0000","last_change":"2024-03-24T00:18:43.883046+0000","last_active":"2024-03-24T00:19:41.450755+0000","last_peered":"2024-03-24T00:19:41.450755+0000","last_clean":"2024-03-24T00:19:41.450755+0000","last_became_active":"2024-03-24T00:18:43.881405+0000","last_became_peered":"2024-03-24T00:18:43.881405+0000","last_unstale":"2024-03-24T00:19:41.450755+0000","last_undegraded":"2024-03-24T00:19:41.450755+0000","last_fullsized":"2024-03-24T00:19:41.450755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:21:00.047019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.467844+0000","last_change":"2024-03-24T00:18:47.897630+0000","last_active":"2024-03-24T00:19:41.467844+0000","last_peered":"2024-03-24T00:19:41.467844+0000","last_clean":"2024-03-24T00:19:41.467844+0000","last_became_active":"2024-03-24T00:18:47.897211+0000","last_became_peered":"2024-03-24T00:18:47.897211+0000","last_unstale":"2024-03-24T00:19:41.467844+0000","last_undegraded":"2024-03-24T00:19:41.467844+0000","last_fullsized":"2024-03-24T00:19:41.467844+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:22:48.762870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.447211+0000","last_change":"2024-03-24T00:18:43.262134+0000","last_active":"2024-03-24T00:19:41.447211+0000","last_peered":"2024-03-24T00:19:41.447211+0000","last_clean":"2024-03-24T00:19:41.447211+0000","last_became_active":"2024-03-24T00:18:43.261783+0000","last_became_peered":"2024-03-24T00:18:43.261783+0000","last_unstale":"2024-03-24T00:19:41.447211+0000","last_undegraded":"2024-03-24T00:19:41.447211+0000","last_fullsized":"2024-03-24T00:19:41.447211+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:16:31.060204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.447601+0000","last_change":"2024-03-24T00:18:43.880532+0000","last_active":"2024-03-24T00:19:41.447601+0000","last_peered":"2024-03-24T00:19:41.447601+0000","last_clean":"2024-03-24T00:19:41.447601+0000","last_became_active":"2024-03-24T00:18:43.880198+0000","last_became_peered":"2024-03-24T00:18:43.880198+0000","last_unstale":"2024-03-24T00:19:41.447601+0000","last_undegraded":"2024-03-24T00:19:41.447601+0000","last_fullsized":"2024-03-24T00:19:41.447601+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:33:16.710842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450898+0000","last_change":"2024-03-24T00:18:47.893607+0000","last_active":"2024-03-24T00:19:41.450898+0000","last_peered":"2024-03-24T00:19:41.450898+0000","last_clean":"2024-03-24T00:19:41.450898+0000","last_became_active":"2024-03-24T00:18:47.893026+0000","last_became_peered":"2024-03-24T00:18:47.893026+0000","last_unstale":"2024-03-24T00:19:41.450898+0000","last_undegraded":"2024-03-24T00:19:41.450898+0000","last_fullsized":"2024-03-24T00:19:41.450898+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:37:58.994146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":101,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:20:17.949877+0000","last_change":"2024-03-24T00:18:43.257520+0000","last_active":"2024-03-24T00:20:17.949877+0000","last_peered":"2024-03-24T00:20:17.949877+0000","last_clean":"2024-03-24T00:20:17.949877+0000","last_became_active":"2024-03-24T00:18:43.257175+0000","last_became_peered":"2024-03-24T00:18:43.257175+0000","last_unstale":"2024-03-24T00:20:17.949877+0000","last_undegraded":"2024-03-24T00:20:17.949877+0000","last_fullsized":"2024-03-24T00:20:17.949877+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:52:45.244983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":71,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450509+0000","last_change":"2024-03-24T00:18:43.887739+0000","last_active":"2024-03-24T00:19:41.450509+0000","last_peered":"2024-03-24T00:19:41.450509+0000","last_clean":"2024-03-24T00:19:41.450509+0000","last_became_active":"2024-03-24T00:18:43.887379+0000","last_became_peered":"2024-03-24T00:18:43.887379+0000","last_unstale":"2024-03-24T00:19:41.450509+0000","last_undegraded":"2024-03-24T00:19:41.450509+0000","last_fullsized":"2024-03-24T00:19:41.450509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:10:50.384759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.447332+0000","last_change":"2024-03-24T00:18:47.890827+0000","last_active":"2024-03-24T00:19:41.447332+0000","last_peered":"2024-03-24T00:19:41.447332+0000","last_clean":"2024-03-24T00:19:41.447332+0000","last_became_active":"2024-03-24T00:18:47.890399+0000","last_became_peered":"2024-03-24T00:18:47.890399+0000","last_unstale":"2024-03-24T00:19:41.447332+0000","last_undegraded":"2024-03-24T00:19:41.447332+0000","last_fullsized":"2024-03-24T00:19:41.447332+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:46:54.575468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.452042+0000","last_change":"2024-03-24T00:18:43.248132+0000","last_active":"2024-03-24T00:19:41.452042+0000","last_peered":"2024-03-24T00:19:41.452042+0000","last_clean":"2024-03-24T00:19:41.452042+0000","last_became_active":"2024-03-24T00:18:43.247772+0000","last_became_peered":"2024-03-24T00:18:43.247772+0000","last_unstale":"2024-03-24T00:19:41.452042+0000","last_undegraded":"2024-03-24T00:19:41.452042+0000","last_fullsized":"2024-03-24T00:19:41.452042+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:09.330155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.467974+0000","last_change":"2024-03-24T00:18:43.889920+0000","last_active":"2024-03-24T00:19:41.467974+0000","last_peered":"2024-03-24T00:19:41.467974+0000","last_clean":"2024-03-24T00:19:41.467974+0000","last_became_active":"2024-03-24T00:18:43.889593+0000","last_became_peered":"2024-03-24T00:18:43.889593+0000","last_unstale":"2024-03-24T00:19:41.467974+0000","last_undegraded":"2024-03-24T00:19:41.467974+0000","last_fullsized":"2024-03-24T00:19:41.467974+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:48:41.101019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450161+0000","last_change":"2024-03-24T00:18:47.893133+0000","last_active":"2024-03-24T00:19:41.450161+0000","last_peered":"2024-03-24T00:19:41.450161+0000","last_clean":"2024-03-24T00:19:41.450161+0000","last_became_active":"2024-03-24T00:18:47.892731+0000","last_became_peered":"2024-03-24T00:18:47.892731+0000","last_unstale":"2024-03-24T00:19:41.450161+0000","last_undegraded":"2024-03-24T00:19:41.450161+0000","last_fullsized":"2024-03-24T00:19:41.450161+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:51:09.539852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.448061+0000","last_change":"2024-03-24T00:18:43.257697+0000","last_active":"2024-03-24T00:19:41.448061+0000","last_peered":"2024-03-24T00:19:41.448061+0000","last_clean":"2024-03-24T00:19:41.448061+0000","last_became_active":"2024-03-24T00:18:43.257334+0000","last_became_peered":"2024-03-24T00:18:43.257334+0000","last_unstale":"2024-03-24T00:19:41.448061+0000","last_undegraded":"2024-03-24T00:19:41.448061+0000","last_fullsized":"2024-03-24T00:19:41.448061+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:35:22.817064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.451128+0000","last_change":"2024-03-24T00:18:43.896238+0000","last_active":"2024-03-24T00:19:41.451128+0000","last_peered":"2024-03-24T00:19:41.451128+0000","last_clean":"2024-03-24T00:19:41.451128+0000","last_became_active":"2024-03-24T00:18:43.895792+0000","last_became_peered":"2024-03-24T00:18:43.895792+0000","last_unstale":"2024-03-24T00:19:41.451128+0000","last_undegraded":"2024-03-24T00:19:41.451128+0000","last_fullsized":"2024-03-24T00:19:41.451128+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:55:11.865771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.447408+0000","last_change":"2024-03-24T00:18:47.891407+0000","last_active":"2024-03-24T00:19:41.447408+0000","last_peered":"2024-03-24T00:19:41.447408+0000","last_clean":"2024-03-24T00:19:41.447408+0000","last_became_active":"2024-03-24T00:18:47.891042+0000","last_became_peered":"2024-03-24T00:18:47.891042+0000","last_unstale":"2024-03-24T00:19:41.447408+0000","last_undegraded":"2024-03-24T00:19:41.447408+0000","last_fullsized":"2024-03-24T00:19:41.447408+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:14:02.939199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'31","reported_seq":223,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:20:18.497258+0000","last_change":"2024-03-24T00:19:41.466575+0000","last_active":"2024-03-24T00:20:18.497258+0000","last_peered":"2024-03-24T00:20:18.497258+0000","last_clean":"2024-03-24T00:20:18.497258+0000","last_became_active":"2024-03-24T00:19:41.454621+0000","last_became_peered":"2024-03-24T00:19:41.454621+0000","last_unstale":"2024-03-24T00:20:18.497258+0000","last_undegraded":"2024-03-24T00:20:18.497258+0000","last_fullsized":"2024-03-24T00:20:18.497258+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:21:54.306449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61775,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":145,"num_read_kb":133,"num_write":32,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":61775,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450502+0000","last_change":"2024-03-24T00:18:43.893740+0000","last_active":"2024-03-24T00:19:41.450502+0000","last_peered":"2024-03-24T00:19:41.450502+0000","last_clean":"2024-03-24T00:19:41.450502+0000","last_became_active":"2024-03-24T00:18:43.893277+0000","last_became_peered":"2024-03-24T00:18:43.893277+0000","last_unstale":"2024-03-24T00:19:41.450502+0000","last_undegraded":"2024-03-24T00:19:41.450502+0000","last_fullsized":"2024-03-24T00:19:41.450502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:21:47.040629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.448337+0000","last_change":"2024-03-24T00:18:47.898769+0000","last_active":"2024-03-24T00:19:41.448337+0000","last_peered":"2024-03-24T00:19:41.448337+0000","last_clean":"2024-03-24T00:19:41.448337+0000","last_became_active":"2024-03-24T00:18:47.898214+0000","last_became_peered":"2024-03-24T00:18:47.898214+0000","last_unstale":"2024-03-24T00:19:41.448337+0000","last_undegraded":"2024-03-24T00:19:41.448337+0000","last_fullsized":"2024-03-24T00:19:41.448337+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:24:42.022429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.461301+0000","last_change":"2024-03-24T00:19:41.461301+0000","last_active":"2024-03-24T00:19:41.461301+0000","last_peered":"2024-03-24T00:19:41.461301+0000","last_clean":"2024-03-24T00:19:41.461301+0000","last_became_active":"2024-03-24T00:19:41.452017+0000","last_became_peered":"2024-03-24T00:19:41.452017+0000","last_unstale":"2024-03-24T00:19:41.461301+0000","last_undegraded":"2024-03-24T00:19:41.461301+0000","last_fullsized":"2024-03-24T00:19:41.461301+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:32:25.295992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.448803+0000","last_change":"2024-03-24T00:18:43.881691+0000","last_active":"2024-03-24T00:19:41.448803+0000","last_peered":"2024-03-24T00:19:41.448803+0000","last_clean":"2024-03-24T00:19:41.448803+0000","last_became_active":"2024-03-24T00:18:43.881386+0000","last_became_peered":"2024-03-24T00:18:43.881386+0000","last_unstale":"2024-03-24T00:19:41.448803+0000","last_undegraded":"2024-03-24T00:19:41.448803+0000","last_fullsized":"2024-03-24T00:19:41.448803+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:33:24.808315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.446722+0000","last_change":"2024-03-24T00:18:47.883340+0000","last_active":"2024-03-24T00:19:41.446722+0000","last_peered":"2024-03-24T00:19:41.446722+0000","last_clean":"2024-03-24T00:19:41.446722+0000","last_became_active":"2024-03-24T00:18:47.882934+0000","last_became_peered":"2024-03-24T00:18:47.882934+0000","last_unstale":"2024-03-24T00:19:41.446722+0000","last_undegraded":"2024-03-24T00:19:41.446722+0000","last_fullsized":"2024-03-24T00:19:41.446722+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:30:56.659709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.447888+0000","last_change":"2024-03-24T00:18:43.253444+0000","last_active":"2024-03-24T00:19:41.447888+0000","last_peered":"2024-03-24T00:19:41.447888+0000","last_clean":"2024-03-24T00:19:41.447888+0000","last_became_active":"2024-03-24T00:18:43.252121+0000","last_became_peered":"2024-03-24T00:18:43.252121+0000","last_unstale":"2024-03-24T00:19:41.447888+0000","last_undegraded":"2024-03-24T00:19:41.447888+0000","last_fullsized":"2024-03-24T00:19:41.447888+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:44:48.663995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.454454+0000","last_change":"2024-03-24T00:19:41.454454+0000","last_active":"2024-03-24T00:19:41.454454+0000","last_peered":"2024-03-24T00:19:41.454454+0000","last_clean":"2024-03-24T00:19:41.454454+0000","last_became_active":"2024-03-24T00:19:41.454112+0000","last_became_peered":"2024-03-24T00:19:41.454112+0000","last_unstale":"2024-03-24T00:19:41.454454+0000","last_undegraded":"2024-03-24T00:19:41.454454+0000","last_fullsized":"2024-03-24T00:19:41.454454+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:33:12.282662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.469175+0000","last_change":"2024-03-24T00:19:41.468803+0000","last_active":"2024-03-24T00:19:41.469175+0000","last_peered":"2024-03-24T00:19:41.469175+0000","last_clean":"2024-03-24T00:19:41.469175+0000","last_became_active":"2024-03-24T00:19:41.468406+0000","last_became_peered":"2024-03-24T00:19:41.468406+0000","last_unstale":"2024-03-24T00:19:41.469175+0000","last_undegraded":"2024-03-24T00:19:41.469175+0000","last_fullsized":"2024-03-24T00:19:41.469175+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:18:09.543843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"89'3","reported_seq":101,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:20:15.953209+0000","last_change":"2024-03-24T00:18:43.251682+0000","last_active":"2024-03-24T00:20:15.953209+0000","last_peered":"2024-03-24T00:20:15.953209+0000","last_clean":"2024-03-24T00:20:15.953209+0000","last_became_active":"2024-03-24T00:18:43.250953+0000","last_became_peered":"2024-03-24T00:18:43.250953+0000","last_unstale":"2024-03-24T00:20:15.953209+0000","last_undegraded":"2024-03-24T00:20:15.953209+0000","last_fullsized":"2024-03-24T00:20:15.953209+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:22:35.218781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.451662+0000","last_change":"2024-03-24T00:18:43.876334+0000","last_active":"2024-03-24T00:19:41.451662+0000","last_peered":"2024-03-24T00:19:41.451662+0000","last_clean":"2024-03-24T00:19:41.451662+0000","last_became_active":"2024-03-24T00:18:43.875397+0000","last_became_peered":"2024-03-24T00:18:43.875397+0000","last_unstale":"2024-03-24T00:19:41.451662+0000","last_undegraded":"2024-03-24T00:19:41.451662+0000","last_fullsized":"2024-03-24T00:19:41.451662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:40:20.474055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449651+0000","last_change":"2024-03-24T00:18:47.897488+0000","last_active":"2024-03-24T00:19:41.449651+0000","last_peered":"2024-03-24T00:19:41.449651+0000","last_clean":"2024-03-24T00:19:41.449651+0000","last_became_active":"2024-03-24T00:18:47.897042+0000","last_became_peered":"2024-03-24T00:18:47.897042+0000","last_unstale":"2024-03-24T00:19:41.449651+0000","last_undegraded":"2024-03-24T00:19:41.449651+0000","last_fullsized":"2024-03-24T00:19:41.449651+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:23:13.071695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450298+0000","last_change":"2024-03-24T00:18:43.255278+0000","last_active":"2024-03-24T00:19:41.450298+0000","last_peered":"2024-03-24T00:19:41.450298+0000","last_clean":"2024-03-24T00:19:41.450298+0000","last_became_active":"2024-03-24T00:18:43.254943+0000","last_became_peered":"2024-03-24T00:18:43.254943+0000","last_unstale":"2024-03-24T00:19:41.450298+0000","last_undegraded":"2024-03-24T00:19:41.450298+0000","last_fullsized":"2024-03-24T00:19:41.450298+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:09:56.755303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.447385+0000","last_change":"2024-03-24T00:18:43.892620+0000","last_active":"2024-03-24T00:19:41.447385+0000","last_peered":"2024-03-24T00:19:41.447385+0000","last_clean":"2024-03-24T00:19:41.447385+0000","last_became_active":"2024-03-24T00:18:43.892033+0000","last_became_peered":"2024-03-24T00:18:43.892033+0000","last_unstale":"2024-03-24T00:19:41.447385+0000","last_undegraded":"2024-03-24T00:19:41.447385+0000","last_fullsized":"2024-03-24T00:19:41.447385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:30:19.472861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.447287+0000","last_change":"2024-03-24T00:18:47.895623+0000","last_active":"2024-03-24T00:19:41.447287+0000","last_peered":"2024-03-24T00:19:41.447287+0000","last_clean":"2024-03-24T00:19:41.447287+0000","last_became_active":"2024-03-24T00:18:47.895242+0000","last_became_peered":"2024-03-24T00:18:47.895242+0000","last_unstale":"2024-03-24T00:19:41.447287+0000","last_undegraded":"2024-03-24T00:19:41.447287+0000","last_fullsized":"2024-03-24T00:19:41.447287+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:50:32.775988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.451373+0000","last_change":"2024-03-24T00:18:43.265575+0000","last_active":"2024-03-24T00:19:41.451373+0000","last_peered":"2024-03-24T00:19:41.451373+0000","last_clean":"2024-03-24T00:19:41.451373+0000","last_became_active":"2024-03-24T00:18:43.265265+0000","last_became_peered":"2024-03-24T00:18:43.265265+0000","last_unstale":"2024-03-24T00:19:41.451373+0000","last_undegraded":"2024-03-24T00:19:41.451373+0000","last_fullsized":"2024-03-24T00:19:41.451373+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:15:46.566812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.448055+0000","last_change":"2024-03-24T00:18:43.898574+0000","last_active":"2024-03-24T00:19:41.448055+0000","last_peered":"2024-03-24T00:19:41.448055+0000","last_clean":"2024-03-24T00:19:41.448055+0000","last_became_active":"2024-03-24T00:18:43.898153+0000","last_became_peered":"2024-03-24T00:18:43.898153+0000","last_unstale":"2024-03-24T00:19:41.448055+0000","last_undegraded":"2024-03-24T00:19:41.448055+0000","last_fullsized":"2024-03-24T00:19:41.448055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:50:15.317257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450297+0000","last_change":"2024-03-24T00:18:47.896336+0000","last_active":"2024-03-24T00:19:41.450297+0000","last_peered":"2024-03-24T00:19:41.450297+0000","last_clean":"2024-03-24T00:19:41.450297+0000","last_became_active":"2024-03-24T00:18:47.895939+0000","last_became_peered":"2024-03-24T00:18:47.895939+0000","last_unstale":"2024-03-24T00:19:41.450297+0000","last_undegraded":"2024-03-24T00:19:41.450297+0000","last_fullsized":"2024-03-24T00:19:41.450297+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:54:03.195827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.447779+0000","last_change":"2024-03-24T00:18:43.253081+0000","last_active":"2024-03-24T00:19:41.447779+0000","last_peered":"2024-03-24T00:19:41.447779+0000","last_clean":"2024-03-24T00:19:41.447779+0000","last_became_active":"2024-03-24T00:18:43.252415+0000","last_became_peered":"2024-03-24T00:18:43.252415+0000","last_unstale":"2024-03-24T00:19:41.447779+0000","last_undegraded":"2024-03-24T00:19:41.447779+0000","last_fullsized":"2024-03-24T00:19:41.447779+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:06:45.200233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.446412+0000","last_change":"2024-03-24T00:18:43.885820+0000","last_active":"2024-03-24T00:19:41.446412+0000","last_peered":"2024-03-24T00:19:41.446412+0000","last_clean":"2024-03-24T00:19:41.446412+0000","last_became_active":"2024-03-24T00:18:43.885485+0000","last_became_peered":"2024-03-24T00:18:43.885485+0000","last_unstale":"2024-03-24T00:19:41.446412+0000","last_undegraded":"2024-03-24T00:19:41.446412+0000","last_fullsized":"2024-03-24T00:19:41.446412+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:09:12.286000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449050+0000","last_change":"2024-03-24T00:18:47.899054+0000","last_active":"2024-03-24T00:19:41.449050+0000","last_peered":"2024-03-24T00:19:41.449050+0000","last_clean":"2024-03-24T00:19:41.449050+0000","last_became_active":"2024-03-24T00:18:47.898412+0000","last_became_peered":"2024-03-24T00:18:47.898412+0000","last_unstale":"2024-03-24T00:19:41.449050+0000","last_undegraded":"2024-03-24T00:19:41.449050+0000","last_fullsized":"2024-03-24T00:19:41.449050+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:17:08.273323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.451877+0000","last_change":"2024-03-24T00:18:43.251402+0000","last_active":"2024-03-24T00:19:41.451877+0000","last_peered":"2024-03-24T00:19:41.451877+0000","last_clean":"2024-03-24T00:19:41.451877+0000","last_became_active":"2024-03-24T00:18:43.251055+0000","last_became_peered":"2024-03-24T00:18:43.251055+0000","last_unstale":"2024-03-24T00:19:41.451877+0000","last_undegraded":"2024-03-24T00:19:41.451877+0000","last_fullsized":"2024-03-24T00:19:41.451877+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:07:30.966065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.446439+0000","last_change":"2024-03-24T00:18:43.886594+0000","last_active":"2024-03-24T00:19:41.446439+0000","last_peered":"2024-03-24T00:19:41.446439+0000","last_clean":"2024-03-24T00:19:41.446439+0000","last_became_active":"2024-03-24T00:18:43.886251+0000","last_became_peered":"2024-03-24T00:18:43.886251+0000","last_unstale":"2024-03-24T00:19:41.446439+0000","last_undegraded":"2024-03-24T00:19:41.446439+0000","last_fullsized":"2024-03-24T00:19:41.446439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:55:24.662318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449658+0000","last_change":"2024-03-24T00:18:47.901352+0000","last_active":"2024-03-24T00:19:41.449658+0000","last_peered":"2024-03-24T00:19:41.449658+0000","last_clean":"2024-03-24T00:19:41.449658+0000","last_became_active":"2024-03-24T00:18:47.900943+0000","last_became_peered":"2024-03-24T00:18:47.900943+0000","last_unstale":"2024-03-24T00:19:41.449658+0000","last_undegraded":"2024-03-24T00:19:41.449658+0000","last_fullsized":"2024-03-24T00:19:41.449658+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:25:29.626277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.451990+0000","last_change":"2024-03-24T00:19:41.451990+0000","last_active":"2024-03-24T00:19:41.451990+0000","last_peered":"2024-03-24T00:19:41.451990+0000","last_clean":"2024-03-24T00:19:41.451990+0000","last_became_active":"2024-03-24T00:19:41.451609+0000","last_became_peered":"2024-03-24T00:19:41.451609+0000","last_unstale":"2024-03-24T00:19:41.451990+0000","last_undegraded":"2024-03-24T00:19:41.451990+0000","last_fullsized":"2024-03-24T00:19:41.451990+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:19:18.965909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":104,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:20:19.957001+0000","last_change":"2024-03-24T00:18:43.263213+0000","last_active":"2024-03-24T00:20:19.957001+0000","last_peered":"2024-03-24T00:20:19.957001+0000","last_clean":"2024-03-24T00:20:19.957001+0000","last_became_active":"2024-03-24T00:18:43.262668+0000","last_became_peered":"2024-03-24T00:18:43.262668+0000","last_unstale":"2024-03-24T00:20:19.957001+0000","last_undegraded":"2024-03-24T00:20:19.957001+0000","last_fullsized":"2024-03-24T00:20:19.957001+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:18:15.518187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":73,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.452250+0000","last_change":"2024-03-24T00:18:47.891607+0000","last_active":"2024-03-24T00:19:41.452250+0000","last_peered":"2024-03-24T00:19:41.452250+0000","last_clean":"2024-03-24T00:19:41.452250+0000","last_became_active":"2024-03-24T00:18:47.891241+0000","last_became_peered":"2024-03-24T00:18:47.891241+0000","last_unstale":"2024-03-24T00:19:41.452250+0000","last_undegraded":"2024-03-24T00:19:41.452250+0000","last_fullsized":"2024-03-24T00:19:41.452250+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:14:29.593767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.451002+0000","last_change":"2024-03-24T00:18:43.894492+0000","last_active":"2024-03-24T00:19:41.451002+0000","last_peered":"2024-03-24T00:19:41.451002+0000","last_clean":"2024-03-24T00:19:41.451002+0000","last_became_active":"2024-03-24T00:18:43.893104+0000","last_became_peered":"2024-03-24T00:18:43.893104+0000","last_unstale":"2024-03-24T00:19:41.451002+0000","last_undegraded":"2024-03-24T00:19:41.451002+0000","last_fullsized":"2024-03-24T00:19:41.451002+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:26:32.130449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450059+0000","last_change":"2024-03-24T00:18:43.251344+0000","last_active":"2024-03-24T00:19:41.450059+0000","last_peered":"2024-03-24T00:19:41.450059+0000","last_clean":"2024-03-24T00:19:41.450059+0000","last_became_active":"2024-03-24T00:18:43.250961+0000","last_became_peered":"2024-03-24T00:18:43.250961+0000","last_unstale":"2024-03-24T00:19:41.450059+0000","last_undegraded":"2024-03-24T00:19:41.450059+0000","last_fullsized":"2024-03-24T00:19:41.450059+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:13:38.423465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":361,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.451467+0000","last_change":"2024-03-24T00:17:27.864318+0000","last_active":"2024-03-24T00:19:41.451467+0000","last_peered":"2024-03-24T00:19:41.451467+0000","last_clean":"2024-03-24T00:19:41.451467+0000","last_became_active":"2024-03-24T00:17:27.863875+0000","last_became_peered":"2024-03-24T00:17:27.863875+0000","last_unstale":"2024-03-24T00:19:41.451467+0000","last_undegraded":"2024-03-24T00:19:41.451467+0000","last_fullsized":"2024-03-24T00:19:41.451467+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:14:04.731181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:14:04.731181+0000","last_clean_scrub_stamp":"2024-03-24T00:14:04.731181+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:53:25.036280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.447796+0000","last_change":"2024-03-24T00:18:47.895344+0000","last_active":"2024-03-24T00:19:41.447796+0000","last_peered":"2024-03-24T00:19:41.447796+0000","last_clean":"2024-03-24T00:19:41.447796+0000","last_became_active":"2024-03-24T00:18:47.894694+0000","last_became_peered":"2024-03-24T00:18:47.894694+0000","last_unstale":"2024-03-24T00:19:41.447796+0000","last_undegraded":"2024-03-24T00:19:41.447796+0000","last_fullsized":"2024-03-24T00:19:41.447796+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:16.969143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449647+0000","last_change":"2024-03-24T00:18:43.890236+0000","last_active":"2024-03-24T00:19:41.449647+0000","last_peered":"2024-03-24T00:19:41.449647+0000","last_clean":"2024-03-24T00:19:41.449647+0000","last_became_active":"2024-03-24T00:18:43.889896+0000","last_became_peered":"2024-03-24T00:18:43.889896+0000","last_unstale":"2024-03-24T00:19:41.449647+0000","last_undegraded":"2024-03-24T00:19:41.449647+0000","last_fullsized":"2024-03-24T00:19:41.449647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:13:19.443445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.451550+0000","last_change":"2024-03-24T00:18:43.260722+0000","last_active":"2024-03-24T00:19:41.451550+0000","last_peered":"2024-03-24T00:19:41.451550+0000","last_clean":"2024-03-24T00:19:41.451550+0000","last_became_active":"2024-03-24T00:18:43.260244+0000","last_became_peered":"2024-03-24T00:18:43.260244+0000","last_unstale":"2024-03-24T00:19:41.451550+0000","last_undegraded":"2024-03-24T00:19:41.451550+0000","last_fullsized":"2024-03-24T00:19:41.451550+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:37:37.838046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.447211+0000","last_change":"2024-03-24T00:18:47.891168+0000","last_active":"2024-03-24T00:19:41.447211+0000","last_peered":"2024-03-24T00:19:41.447211+0000","last_clean":"2024-03-24T00:19:41.447211+0000","last_became_active":"2024-03-24T00:18:47.890713+0000","last_became_peered":"2024-03-24T00:18:47.890713+0000","last_unstale":"2024-03-24T00:19:41.447211+0000","last_undegraded":"2024-03-24T00:19:41.447211+0000","last_fullsized":"2024-03-24T00:19:41.447211+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:20:30.858381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450697+0000","last_change":"2024-03-24T00:18:43.256638+0000","last_active":"2024-03-24T00:19:41.450697+0000","last_peered":"2024-03-24T00:19:41.450697+0000","last_clean":"2024-03-24T00:19:41.450697+0000","last_became_active":"2024-03-24T00:18:43.256118+0000","last_became_peered":"2024-03-24T00:18:43.256118+0000","last_unstale":"2024-03-24T00:19:41.450697+0000","last_undegraded":"2024-03-24T00:19:41.450697+0000","last_fullsized":"2024-03-24T00:19:41.450697+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:40:05.523388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449330+0000","last_change":"2024-03-24T00:18:43.885293+0000","last_active":"2024-03-24T00:19:41.449330+0000","last_peered":"2024-03-24T00:19:41.449330+0000","last_clean":"2024-03-24T00:19:41.449330+0000","last_became_active":"2024-03-24T00:18:43.884989+0000","last_became_peered":"2024-03-24T00:18:43.884989+0000","last_unstale":"2024-03-24T00:19:41.449330+0000","last_undegraded":"2024-03-24T00:19:41.449330+0000","last_fullsized":"2024-03-24T00:19:41.449330+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:32:01.519131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450657+0000","last_change":"2024-03-24T00:18:47.894253+0000","last_active":"2024-03-24T00:19:41.450657+0000","last_peered":"2024-03-24T00:19:41.450657+0000","last_clean":"2024-03-24T00:19:41.450657+0000","last_became_active":"2024-03-24T00:18:47.892654+0000","last_became_peered":"2024-03-24T00:18:47.892654+0000","last_unstale":"2024-03-24T00:19:41.450657+0000","last_undegraded":"2024-03-24T00:19:41.450657+0000","last_fullsized":"2024-03-24T00:19:41.450657+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:59:43.400810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'9","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.455998+0000","last_change":"2024-03-24T00:19:41.455998+0000","last_active":"2024-03-24T00:19:41.455998+0000","last_peered":"2024-03-24T00:19:41.455998+0000","last_clean":"2024-03-24T00:19:41.455998+0000","last_became_active":"2024-03-24T00:19:41.452128+0000","last_became_peered":"2024-03-24T00:19:41.452128+0000","last_unstale":"2024-03-24T00:19:41.455998+0000","last_undegraded":"2024-03-24T00:19:41.455998+0000","last_fullsized":"2024-03-24T00:19:41.455998+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:46.106925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.451383+0000","last_change":"2024-03-24T00:18:43.883654+0000","last_active":"2024-03-24T00:19:41.451383+0000","last_peered":"2024-03-24T00:19:41.451383+0000","last_clean":"2024-03-24T00:19:41.451383+0000","last_became_active":"2024-03-24T00:18:43.883335+0000","last_became_peered":"2024-03-24T00:18:43.883335+0000","last_unstale":"2024-03-24T00:19:41.451383+0000","last_undegraded":"2024-03-24T00:19:41.451383+0000","last_fullsized":"2024-03-24T00:19:41.451383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:42:10.856037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.444757+0000","last_change":"2024-03-24T00:18:47.888562+0000","last_active":"2024-03-24T00:19:41.444757+0000","last_peered":"2024-03-24T00:19:41.444757+0000","last_clean":"2024-03-24T00:19:41.444757+0000","last_became_active":"2024-03-24T00:18:47.888074+0000","last_became_peered":"2024-03-24T00:18:47.888074+0000","last_unstale":"2024-03-24T00:19:41.444757+0000","last_undegraded":"2024-03-24T00:19:41.444757+0000","last_fullsized":"2024-03-24T00:19:41.444757+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:01:04.617098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.451564+0000","last_change":"2024-03-24T00:18:43.261668+0000","last_active":"2024-03-24T00:19:41.451564+0000","last_peered":"2024-03-24T00:19:41.451564+0000","last_clean":"2024-03-24T00:19:41.451564+0000","last_became_active":"2024-03-24T00:18:43.261279+0000","last_became_peered":"2024-03-24T00:18:43.261279+0000","last_unstale":"2024-03-24T00:19:41.451564+0000","last_undegraded":"2024-03-24T00:19:41.451564+0000","last_fullsized":"2024-03-24T00:19:41.451564+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:36:05.390006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.467969+0000","last_change":"2024-03-24T00:18:43.889854+0000","last_active":"2024-03-24T00:19:41.467969+0000","last_peered":"2024-03-24T00:19:41.467969+0000","last_clean":"2024-03-24T00:19:41.467969+0000","last_became_active":"2024-03-24T00:18:43.889540+0000","last_became_peered":"2024-03-24T00:18:43.889540+0000","last_unstale":"2024-03-24T00:19:41.467969+0000","last_undegraded":"2024-03-24T00:19:41.467969+0000","last_fullsized":"2024-03-24T00:19:41.467969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:33:58.060564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.454081+0000","last_change":"2024-03-24T00:19:41.454081+0000","last_active":"2024-03-24T00:19:41.454081+0000","last_peered":"2024-03-24T00:19:41.454081+0000","last_clean":"2024-03-24T00:19:41.454081+0000","last_became_active":"2024-03-24T00:19:41.453621+0000","last_became_peered":"2024-03-24T00:19:41.453621+0000","last_unstale":"2024-03-24T00:19:41.454081+0000","last_undegraded":"2024-03-24T00:19:41.454081+0000","last_fullsized":"2024-03-24T00:19:41.454081+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:18:27.484085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449225+0000","last_change":"2024-03-24T00:18:43.253063+0000","last_active":"2024-03-24T00:19:41.449225+0000","last_peered":"2024-03-24T00:19:41.449225+0000","last_clean":"2024-03-24T00:19:41.449225+0000","last_became_active":"2024-03-24T00:18:43.252534+0000","last_became_peered":"2024-03-24T00:18:43.252534+0000","last_unstale":"2024-03-24T00:19:41.449225+0000","last_undegraded":"2024-03-24T00:19:41.449225+0000","last_fullsized":"2024-03-24T00:19:41.449225+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:50:59.512780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450119+0000","last_change":"2024-03-24T00:18:43.885574+0000","last_active":"2024-03-24T00:19:41.450119+0000","last_peered":"2024-03-24T00:19:41.450119+0000","last_clean":"2024-03-24T00:19:41.450119+0000","last_became_active":"2024-03-24T00:18:43.885257+0000","last_became_peered":"2024-03-24T00:18:43.885257+0000","last_unstale":"2024-03-24T00:19:41.450119+0000","last_undegraded":"2024-03-24T00:19:41.450119+0000","last_fullsized":"2024-03-24T00:19:41.450119+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:10.868436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.448145+0000","last_change":"2024-03-24T00:18:47.892739+0000","last_active":"2024-03-24T00:19:41.448145+0000","last_peered":"2024-03-24T00:19:41.448145+0000","last_clean":"2024-03-24T00:19:41.448145+0000","last_became_active":"2024-03-24T00:18:47.892291+0000","last_became_peered":"2024-03-24T00:18:47.892291+0000","last_unstale":"2024-03-24T00:19:41.448145+0000","last_undegraded":"2024-03-24T00:19:41.448145+0000","last_fullsized":"2024-03-24T00:19:41.448145+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:06:16.027968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450981+0000","last_change":"2024-03-24T00:18:43.263856+0000","last_active":"2024-03-24T00:19:41.450981+0000","last_peered":"2024-03-24T00:19:41.450981+0000","last_clean":"2024-03-24T00:19:41.450981+0000","last_became_active":"2024-03-24T00:18:43.263032+0000","last_became_peered":"2024-03-24T00:18:43.263032+0000","last_unstale":"2024-03-24T00:19:41.450981+0000","last_undegraded":"2024-03-24T00:19:41.450981+0000","last_fullsized":"2024-03-24T00:19:41.450981+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:25:03.239696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.446613+0000","last_change":"2024-03-24T00:18:43.884674+0000","last_active":"2024-03-24T00:19:41.446613+0000","last_peered":"2024-03-24T00:19:41.446613+0000","last_clean":"2024-03-24T00:19:41.446613+0000","last_became_active":"2024-03-24T00:18:43.884346+0000","last_became_peered":"2024-03-24T00:18:43.884346+0000","last_unstale":"2024-03-24T00:19:41.446613+0000","last_undegraded":"2024-03-24T00:19:41.446613+0000","last_fullsized":"2024-03-24T00:19:41.446613+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:58:31.474515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450029+0000","last_change":"2024-03-24T00:18:47.892472+0000","last_active":"2024-03-24T00:19:41.450029+0000","last_peered":"2024-03-24T00:19:41.450029+0000","last_clean":"2024-03-24T00:19:41.450029+0000","last_became_active":"2024-03-24T00:18:47.892082+0000","last_became_peered":"2024-03-24T00:18:47.892082+0000","last_unstale":"2024-03-24T00:19:41.450029+0000","last_undegraded":"2024-03-24T00:19:41.450029+0000","last_fullsized":"2024-03-24T00:19:41.450029+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:26:01.084042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.445073+0000","last_change":"2024-03-24T00:18:43.263780+0000","last_active":"2024-03-24T00:19:41.445073+0000","last_peered":"2024-03-24T00:19:41.445073+0000","last_clean":"2024-03-24T00:19:41.445073+0000","last_became_active":"2024-03-24T00:18:43.263446+0000","last_became_peered":"2024-03-24T00:18:43.263446+0000","last_unstale":"2024-03-24T00:19:41.445073+0000","last_undegraded":"2024-03-24T00:19:41.445073+0000","last_fullsized":"2024-03-24T00:19:41.445073+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:09:09.821842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450851+0000","last_change":"2024-03-24T00:18:43.888516+0000","last_active":"2024-03-24T00:19:41.450851+0000","last_peered":"2024-03-24T00:19:41.450851+0000","last_clean":"2024-03-24T00:19:41.450851+0000","last_became_active":"2024-03-24T00:18:43.888190+0000","last_became_peered":"2024-03-24T00:18:43.888190+0000","last_unstale":"2024-03-24T00:19:41.450851+0000","last_undegraded":"2024-03-24T00:19:41.450851+0000","last_fullsized":"2024-03-24T00:19:41.450851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:53:35.685128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449644+0000","last_change":"2024-03-24T00:18:47.897115+0000","last_active":"2024-03-24T00:19:41.449644+0000","last_peered":"2024-03-24T00:19:41.449644+0000","last_clean":"2024-03-24T00:19:41.449644+0000","last_became_active":"2024-03-24T00:18:47.896722+0000","last_became_peered":"2024-03-24T00:18:47.896722+0000","last_unstale":"2024-03-24T00:19:41.449644+0000","last_undegraded":"2024-03-24T00:19:41.449644+0000","last_fullsized":"2024-03-24T00:19:41.449644+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:44:40.541618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.450272+0000","last_change":"2024-03-24T00:18:43.263459+0000","last_active":"2024-03-24T00:19:41.450272+0000","last_peered":"2024-03-24T00:19:41.450272+0000","last_clean":"2024-03-24T00:19:41.450272+0000","last_became_active":"2024-03-24T00:18:43.263019+0000","last_became_peered":"2024-03-24T00:18:43.263019+0000","last_unstale":"2024-03-24T00:19:41.450272+0000","last_undegraded":"2024-03-24T00:19:41.450272+0000","last_fullsized":"2024-03-24T00:19:41.450272+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:10:57.171625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.451721+0000","last_change":"2024-03-24T00:18:43.886547+0000","last_active":"2024-03-24T00:19:41.451721+0000","last_peered":"2024-03-24T00:19:41.451721+0000","last_clean":"2024-03-24T00:19:41.451721+0000","last_became_active":"2024-03-24T00:18:43.886217+0000","last_became_peered":"2024-03-24T00:18:43.886217+0000","last_unstale":"2024-03-24T00:19:41.451721+0000","last_undegraded":"2024-03-24T00:19:41.451721+0000","last_fullsized":"2024-03-24T00:19:41.451721+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:59:33.576577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449538+0000","last_change":"2024-03-24T00:18:47.894246+0000","last_active":"2024-03-24T00:19:41.449538+0000","last_peered":"2024-03-24T00:19:41.449538+0000","last_clean":"2024-03-24T00:19:41.449538+0000","last_became_active":"2024-03-24T00:18:47.893837+0000","last_became_peered":"2024-03-24T00:18:47.893837+0000","last_unstale":"2024-03-24T00:19:41.449538+0000","last_undegraded":"2024-03-24T00:19:41.449538+0000","last_fullsized":"2024-03-24T00:19:41.449538+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:33:06.044355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.445352+0000","last_change":"2024-03-24T00:18:43.256808+0000","last_active":"2024-03-24T00:19:41.445352+0000","last_peered":"2024-03-24T00:19:41.445352+0000","last_clean":"2024-03-24T00:19:41.445352+0000","last_became_active":"2024-03-24T00:18:43.256397+0000","last_became_peered":"2024-03-24T00:18:43.256397+0000","last_unstale":"2024-03-24T00:19:41.445352+0000","last_undegraded":"2024-03-24T00:19:41.445352+0000","last_fullsized":"2024-03-24T00:19:41.445352+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:36:13.505016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.452492+0000","last_change":"2024-03-24T00:18:43.887754+0000","last_active":"2024-03-24T00:19:41.452492+0000","last_peered":"2024-03-24T00:19:41.452492+0000","last_clean":"2024-03-24T00:19:41.452492+0000","last_became_active":"2024-03-24T00:18:43.886978+0000","last_became_peered":"2024-03-24T00:18:43.886978+0000","last_unstale":"2024-03-24T00:19:41.452492+0000","last_undegraded":"2024-03-24T00:19:41.452492+0000","last_fullsized":"2024-03-24T00:19:41.452492+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:43:18.186682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.467600+0000","last_change":"2024-03-24T00:18:47.887576+0000","last_active":"2024-03-24T00:19:41.467600+0000","last_peered":"2024-03-24T00:19:41.467600+0000","last_clean":"2024-03-24T00:19:41.467600+0000","last_became_active":"2024-03-24T00:18:47.887190+0000","last_became_peered":"2024-03-24T00:18:47.887190+0000","last_unstale":"2024-03-24T00:19:41.467600+0000","last_undegraded":"2024-03-24T00:19:41.467600+0000","last_fullsized":"2024-03-24T00:19:41.467600+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:10:10.396432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.448598+0000","last_change":"2024-03-24T00:18:43.259927+0000","last_active":"2024-03-24T00:19:41.448598+0000","last_peered":"2024-03-24T00:19:41.448598+0000","last_clean":"2024-03-24T00:19:41.448598+0000","last_became_active":"2024-03-24T00:18:43.259587+0000","last_became_peered":"2024-03-24T00:18:43.259587+0000","last_unstale":"2024-03-24T00:19:41.448598+0000","last_undegraded":"2024-03-24T00:19:41.448598+0000","last_fullsized":"2024-03-24T00:19:41.448598+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:26:06.380010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.452723+0000","last_change":"2024-03-24T00:18:43.897041+0000","last_active":"2024-03-24T00:19:41.452723+0000","last_peered":"2024-03-24T00:19:41.452723+0000","last_clean":"2024-03-24T00:19:41.452723+0000","last_became_active":"2024-03-24T00:18:43.896766+0000","last_became_peered":"2024-03-24T00:18:43.896766+0000","last_unstale":"2024-03-24T00:19:41.452723+0000","last_undegraded":"2024-03-24T00:19:41.452723+0000","last_fullsized":"2024-03-24T00:19:41.452723+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:27:59.577589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.447021+0000","last_change":"2024-03-24T00:18:47.893146+0000","last_active":"2024-03-24T00:19:41.447021+0000","last_peered":"2024-03-24T00:19:41.447021+0000","last_clean":"2024-03-24T00:19:41.447021+0000","last_became_active":"2024-03-24T00:18:47.892150+0000","last_became_peered":"2024-03-24T00:18:47.892150+0000","last_unstale":"2024-03-24T00:19:41.447021+0000","last_undegraded":"2024-03-24T00:19:41.447021+0000","last_fullsized":"2024-03-24T00:19:41.447021+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:46.870160+0000","last_clean_scrub_stamp":"2024-03-24T00:18:46.870160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:28:13.887194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449742+0000","last_change":"2024-03-24T00:18:43.263972+0000","last_active":"2024-03-24T00:19:41.449742+0000","last_peered":"2024-03-24T00:19:41.449742+0000","last_clean":"2024-03-24T00:19:41.449742+0000","last_became_active":"2024-03-24T00:18:43.263577+0000","last_became_peered":"2024-03-24T00:18:43.263577+0000","last_unstale":"2024-03-24T00:19:41.449742+0000","last_undegraded":"2024-03-24T00:19:41.449742+0000","last_fullsized":"2024-03-24T00:19:41.449742+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:42.228553+0000","last_clean_scrub_stamp":"2024-03-24T00:18:42.228553+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:55:36.725420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.453461+0000","last_change":"2024-03-24T00:18:43.894722+0000","last_active":"2024-03-24T00:19:41.453461+0000","last_peered":"2024-03-24T00:19:41.453461+0000","last_clean":"2024-03-24T00:19:41.453461+0000","last_became_active":"2024-03-24T00:18:43.894359+0000","last_became_peered":"2024-03-24T00:18:43.894359+0000","last_unstale":"2024-03-24T00:19:41.453461+0000","last_undegraded":"2024-03-24T00:19:41.453461+0000","last_fullsized":"2024-03-24T00:19:41.453461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:35:23.974163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.446675+0000","last_change":"2024-03-24T00:18:43.892749+0000","last_active":"2024-03-24T00:19:41.446675+0000","last_peered":"2024-03-24T00:19:41.446675+0000","last_clean":"2024-03-24T00:19:41.446675+0000","last_became_active":"2024-03-24T00:18:43.892014+0000","last_became_peered":"2024-03-24T00:18:43.892014+0000","last_unstale":"2024-03-24T00:19:41.446675+0000","last_undegraded":"2024-03-24T00:19:41.446675+0000","last_fullsized":"2024-03-24T00:19:41.446675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:53:14.592384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.445003+0000","last_change":"2024-03-24T00:18:43.899540+0000","last_active":"2024-03-24T00:19:41.445003+0000","last_peered":"2024-03-24T00:19:41.445003+0000","last_clean":"2024-03-24T00:19:41.445003+0000","last_became_active":"2024-03-24T00:18:43.899206+0000","last_became_peered":"2024-03-24T00:18:43.899206+0000","last_unstale":"2024-03-24T00:19:41.445003+0000","last_undegraded":"2024-03-24T00:19:41.445003+0000","last_fullsized":"2024-03-24T00:19:41.445003+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:51:17.214705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.444958+0000","last_change":"2024-03-24T00:18:43.896254+0000","last_active":"2024-03-24T00:19:41.444958+0000","last_peered":"2024-03-24T00:19:41.444958+0000","last_clean":"2024-03-24T00:19:41.444958+0000","last_became_active":"2024-03-24T00:18:43.895889+0000","last_became_peered":"2024-03-24T00:18:43.895889+0000","last_unstale":"2024-03-24T00:19:41.444958+0000","last_undegraded":"2024-03-24T00:19:41.444958+0000","last_fullsized":"2024-03-24T00:19:41.444958+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:07:46.403117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.467746+0000","last_change":"2024-03-24T00:18:43.890813+0000","last_active":"2024-03-24T00:19:41.467746+0000","last_peered":"2024-03-24T00:19:41.467746+0000","last_clean":"2024-03-24T00:19:41.467746+0000","last_became_active":"2024-03-24T00:18:43.890407+0000","last_became_peered":"2024-03-24T00:18:43.890407+0000","last_unstale":"2024-03-24T00:19:41.467746+0000","last_undegraded":"2024-03-24T00:19:41.467746+0000","last_fullsized":"2024-03-24T00:19:41.467746+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:57:45.146055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449204+0000","last_change":"2024-03-24T00:18:43.890280+0000","last_active":"2024-03-24T00:19:41.449204+0000","last_peered":"2024-03-24T00:19:41.449204+0000","last_clean":"2024-03-24T00:19:41.449204+0000","last_became_active":"2024-03-24T00:18:43.889888+0000","last_became_peered":"2024-03-24T00:18:43.889888+0000","last_unstale":"2024-03-24T00:19:41.449204+0000","last_undegraded":"2024-03-24T00:19:41.449204+0000","last_fullsized":"2024-03-24T00:19:41.449204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:49:17.010306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.447973+0000","last_change":"2024-03-24T00:18:43.898453+0000","last_active":"2024-03-24T00:19:41.447973+0000","last_peered":"2024-03-24T00:19:41.447973+0000","last_clean":"2024-03-24T00:19:41.447973+0000","last_became_active":"2024-03-24T00:18:43.898081+0000","last_became_peered":"2024-03-24T00:18:43.898081+0000","last_unstale":"2024-03-24T00:19:41.447973+0000","last_undegraded":"2024-03-24T00:19:41.447973+0000","last_fullsized":"2024-03-24T00:19:41.447973+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:03:50.856755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.445528+0000","last_change":"2024-03-24T00:18:43.894667+0000","last_active":"2024-03-24T00:19:41.445528+0000","last_peered":"2024-03-24T00:19:41.445528+0000","last_clean":"2024-03-24T00:19:41.445528+0000","last_became_active":"2024-03-24T00:18:43.893869+0000","last_became_peered":"2024-03-24T00:18:43.893869+0000","last_unstale":"2024-03-24T00:19:41.445528+0000","last_undegraded":"2024-03-24T00:19:41.445528+0000","last_fullsized":"2024-03-24T00:19:41.445528+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:51:43.273867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.451424+0000","last_change":"2024-03-24T00:18:43.893201+0000","last_active":"2024-03-24T00:19:41.451424+0000","last_peered":"2024-03-24T00:19:41.451424+0000","last_clean":"2024-03-24T00:19:41.451424+0000","last_became_active":"2024-03-24T00:18:43.892866+0000","last_became_peered":"2024-03-24T00:18:43.892866+0000","last_unstale":"2024-03-24T00:19:41.451424+0000","last_undegraded":"2024-03-24T00:19:41.451424+0000","last_fullsized":"2024-03-24T00:19:41.451424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:03:16.108806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.447316+0000","last_change":"2024-03-24T00:18:43.882211+0000","last_active":"2024-03-24T00:19:41.447316+0000","last_peered":"2024-03-24T00:19:41.447316+0000","last_clean":"2024-03-24T00:19:41.447316+0000","last_became_active":"2024-03-24T00:18:43.881857+0000","last_became_peered":"2024-03-24T00:18:43.881857+0000","last_unstale":"2024-03-24T00:19:41.447316+0000","last_undegraded":"2024-03-24T00:19:41.447316+0000","last_fullsized":"2024-03-24T00:19:41.447316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:36:45.911853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.451050+0000","last_change":"2024-03-24T00:18:43.892599+0000","last_active":"2024-03-24T00:19:41.451050+0000","last_peered":"2024-03-24T00:19:41.451050+0000","last_clean":"2024-03-24T00:19:41.451050+0000","last_became_active":"2024-03-24T00:18:43.891966+0000","last_became_peered":"2024-03-24T00:18:43.891966+0000","last_unstale":"2024-03-24T00:19:41.451050+0000","last_undegraded":"2024-03-24T00:19:41.451050+0000","last_fullsized":"2024-03-24T00:19:41.451050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:22:03.486466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449819+0000","last_change":"2024-03-24T00:18:43.892410+0000","last_active":"2024-03-24T00:19:41.449819+0000","last_peered":"2024-03-24T00:19:41.449819+0000","last_clean":"2024-03-24T00:19:41.449819+0000","last_became_active":"2024-03-24T00:18:43.891140+0000","last_became_peered":"2024-03-24T00:18:43.891140+0000","last_unstale":"2024-03-24T00:19:41.449819+0000","last_undegraded":"2024-03-24T00:19:41.449819+0000","last_fullsized":"2024-03-24T00:19:41.449819+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:03:14.252863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.449463+0000","last_change":"2024-03-24T00:18:43.876508+0000","last_active":"2024-03-24T00:19:41.449463+0000","last_peered":"2024-03-24T00:19:41.449463+0000","last_clean":"2024-03-24T00:19:41.449463+0000","last_became_active":"2024-03-24T00:18:43.876204+0000","last_became_peered":"2024-03-24T00:18:43.876204+0000","last_unstale":"2024-03-24T00:19:41.449463+0000","last_undegraded":"2024-03-24T00:19:41.449463+0000","last_fullsized":"2024-03-24T00:19:41.449463+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:35.358964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T00:19:41.446831+0000","last_change":"2024-03-24T00:18:43.887855+0000","last_active":"2024-03-24T00:19:41.446831+0000","last_peered":"2024-03-24T00:19:41.446831+0000","last_clean":"2024-03-24T00:19:41.446831+0000","last_became_active":"2024-03-24T00:18:43.887520+0000","last_became_peered":"2024-03-24T00:18:43.887520+0000","last_unstale":"2024-03-24T00:19:41.446831+0000","last_undegraded":"2024-03-24T00:19:41.446831+0000","last_fullsized":"2024-03-24T00:19:41.446831+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:18:43.232741+0000","last_clean_scrub_stamp":"2024-03-24T00:18:43.232741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:17:15.930626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65776,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":450,"num_read_kb":296,"num_write":161,"num_write_kb":114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":62391,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":197328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383714,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1028096,"data_stored":500865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645350,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37016,"kb_used_data":1672,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708136,"statfs":{"total":95995035648,"available":95957131264,"internally_reserved":0,"allocated":1712128,"data_stored":1149874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808876,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1044480,"data_stored":505163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070513,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1036288,"data_stored":500405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332150,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28620,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716532,"statfs":{"total":95995035648,"available":95965728768,"internally_reserved":0,"allocated":1634304,"data_stored":1096896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495675,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1019904,"data_stored":496243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1085440,"data_stored":557928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1044480,"data_stored":497631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1036288,"data_stored":497860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182473,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1032192,"data_stored":496437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920837,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36424,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708728,"statfs":{"total":95995035648,"available":95957737472,"internally_reserved":0,"allocated":1105920,"data_stored":563250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757312,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36944,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708208,"statfs":{"total":95995035648,"available":95957204992,"internally_reserved":0,"allocated":1642496,"data_stored":1092484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":63001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:20:23.955 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-24T00:20:23.955 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-24T00:20:23.955 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-24T00:20:23.955 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph health --format=json 2024-03-24T00:20:24.197 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:20:24.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:24 smithi078 ceph-mon[32429]: from='client.24892 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:20:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:24 smithi107 ceph-mon[36485]: from='client.24892 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:20:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:24 smithi106 ceph-mon[36505]: from='client.24892 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:20:25.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:25 smithi078 ceph-mon[32429]: pgmap v343: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:20:25.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:25 smithi107 ceph-mon[36485]: pgmap v343: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:20:25.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:25 smithi106 ceph-mon[36505]: pgmap v343: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:20:25.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.907+0000 7f88e4155700 1 -- 172.21.15.78:0/3916373271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88dc102ba0 msgr2=0x7f88dc10efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:25.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.907+0000 7f88e4155700 1 --2- 172.21.15.78:0/3916373271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88dc102ba0 0x7f88dc10efe0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f88d400b0d0 tx=0x7f88d400b3e0 comp rx=0 tx=0).stop 2024-03-24T00:20:25.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.907+0000 7f88e4155700 1 -- 172.21.15.78:0/3916373271 shutdown_connections 2024-03-24T00:20:25.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.907+0000 7f88e4155700 1 --2- 172.21.15.78:0/3916373271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88dc102ba0 0x7f88dc10efe0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:25.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.907+0000 7f88e4155700 1 --2- 172.21.15.78:0/3916373271 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88dc1081c0 0x7f88dc1085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:25.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.907+0000 7f88e4155700 1 --2- 172.21.15.78:0/3916373271 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88dc1021e0 0x7f88dc102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:25.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.907+0000 7f88e4155700 1 -- 172.21.15.78:0/3916373271 >> 172.21.15.78:0/3916373271 conn(0x7f88dc0792c0 msgr2=0x7f88dc0796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:25.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.907+0000 7f88e4155700 1 -- 172.21.15.78:0/3916373271 shutdown_connections 2024-03-24T00:20:25.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.907+0000 7f88e4155700 1 -- 172.21.15.78:0/3916373271 wait complete. 2024-03-24T00:20:25.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.908+0000 7f88e4155700 1 Processor -- start 2024-03-24T00:20:25.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.908+0000 7f88e4155700 1 -- start start 2024-03-24T00:20:25.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.908+0000 7f88e4155700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88dc1021e0 0x7f88dc074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:25.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.908+0000 7f88e4155700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88dc102ba0 0x7f88dc0749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:25.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.908+0000 7f88e4155700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88dc1081c0 0x7f88dc074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:25.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.908+0000 7f88e4155700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88dc112470 con 0x7f88dc1021e0 2024-03-24T00:20:25.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.908+0000 7f88e4155700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88dc1122f0 con 0x7f88dc102ba0 2024-03-24T00:20:25.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.908+0000 7f88e4155700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88dc1125f0 con 0x7f88dc1081c0 2024-03-24T00:20:25.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.908+0000 7f88e1ef1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88dc1021e0 0x7f88dc074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:25.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.908+0000 7f88e16f0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88dc102ba0 0x7f88dc0749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:25.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.908+0000 7f88e1ef1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88dc1021e0 0x7f88dc074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41938/0 (socket says 172.21.15.78:41938) 2024-03-24T00:20:25.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.908+0000 7f88e1ef1700 1 -- 172.21.15.78:0/2154259430 learned_addr learned my addr 172.21.15.78:0/2154259430 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:25.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.908+0000 7f88e26f2700 1 --2- 172.21.15.78:0/2154259430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88dc1081c0 0x7f88dc074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:25.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.909+0000 7f88e1ef1700 1 -- 172.21.15.78:0/2154259430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88dc1081c0 msgr2=0x7f88dc074f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:25.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.909+0000 7f88e1ef1700 1 --2- 172.21.15.78:0/2154259430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88dc1081c0 0x7f88dc074f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:25.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.909+0000 7f88e1ef1700 1 -- 172.21.15.78:0/2154259430 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88dc102ba0 msgr2=0x7f88dc0749d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:25.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.909+0000 7f88e1ef1700 1 --2- 172.21.15.78:0/2154259430 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88dc102ba0 0x7f88dc0749d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:25.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.909+0000 7f88e1ef1700 1 -- 172.21.15.78:0/2154259430 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88dc1a6e80 con 0x7f88dc1021e0 2024-03-24T00:20:25.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.909+0000 7f88e16f0700 1 --2- 172.21.15.78:0/2154259430 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88dc102ba0 0x7f88dc0749d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:25.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.909+0000 7f88e26f2700 1 --2- 172.21.15.78:0/2154259430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88dc1081c0 0x7f88dc074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:25.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.909+0000 7f88e1ef1700 1 --2- 172.21.15.78:0/2154259430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88dc1021e0 0x7f88dc074490 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f88d800e9a0 tx=0x7f88d800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:25.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.909+0000 7f88d2ffd700 1 -- 172.21.15.78:0/2154259430 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d800ca10 con 0x7f88dc1021e0 2024-03-24T00:20:25.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.909+0000 7f88e4155700 1 -- 172.21.15.78:0/2154259430 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88dc1a7170 con 0x7f88dc1021e0 2024-03-24T00:20:25.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.909+0000 7f88d2ffd700 1 -- 172.21.15.78:0/2154259430 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88d800cbb0 con 0x7f88dc1021e0 2024-03-24T00:20:25.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.909+0000 7f88d2ffd700 1 -- 172.21.15.78:0/2154259430 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d8018680 con 0x7f88dc1021e0 2024-03-24T00:20:25.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.909+0000 7f88e4155700 1 -- 172.21.15.78:0/2154259430 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88dc1a7700 con 0x7f88dc1021e0 2024-03-24T00:20:25.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.910+0000 7f88e4155700 1 -- 172.21.15.78:0/2154259430 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88dc068e50 con 0x7f88dc1021e0 2024-03-24T00:20:25.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.911+0000 7f88d2ffd700 1 -- 172.21.15.78:0/2154259430 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f88d8010430 con 0x7f88dc1021e0 2024-03-24T00:20:25.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.912+0000 7f88d2ffd700 1 --2- 172.21.15.78:0/2154259430 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f88c80a2920 0x7f88c80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:25.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.912+0000 7f88e16f0700 1 --2- 172.21.15.78:0/2154259430 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f88c80a2920 0x7f88c80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:25.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.912+0000 7f88d2ffd700 1 -- 172.21.15.78:0/2154259430 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f88d80cf610 con 0x7f88dc1021e0 2024-03-24T00:20:25.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.912+0000 7f88e16f0700 1 --2- 172.21.15.78:0/2154259430 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f88c80a2920 0x7f88c80a4de0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f88dc0798a0 tx=0x7f88cc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:25.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:25.916+0000 7f88d2ffd700 1 -- 172.21.15.78:0/2154259430 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f88d809c190 con 0x7f88dc1021e0 2024-03-24T00:20:26.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:26.156+0000 7f88e4155700 1 -- 172.21.15.78:0/2154259430 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f88dc075970 con 0x7f88dc1021e0 2024-03-24T00:20:26.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:26.157+0000 7f88d2ffd700 1 -- 172.21.15.78:0/2154259430 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f88d809ba90 con 0x7f88dc1021e0 2024-03-24T00:20:26.200 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:20:26.200 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-24T00:20:26.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:26.159+0000 7f88e4155700 1 -- 172.21.15.78:0/2154259430 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f88c80a2920 msgr2=0x7f88c80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:26.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:26.159+0000 7f88e4155700 1 --2- 172.21.15.78:0/2154259430 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f88c80a2920 0x7f88c80a4de0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f88dc0798a0 tx=0x7f88cc009260 comp rx=0 tx=0).stop 2024-03-24T00:20:26.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:26.159+0000 7f88e4155700 1 -- 172.21.15.78:0/2154259430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88dc1021e0 msgr2=0x7f88dc074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:26.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:26.159+0000 7f88e4155700 1 --2- 172.21.15.78:0/2154259430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88dc1021e0 0x7f88dc074490 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f88d800e9a0 tx=0x7f88d800ecb0 comp rx=0 tx=0).stop 2024-03-24T00:20:26.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:26.159+0000 7f88e4155700 1 -- 172.21.15.78:0/2154259430 shutdown_connections 2024-03-24T00:20:26.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:26.159+0000 7f88e4155700 1 --2- 172.21.15.78:0/2154259430 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f88c80a2920 0x7f88c80a4de0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:26.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:26.159+0000 7f88e4155700 1 --2- 172.21.15.78:0/2154259430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88dc1081c0 0x7f88dc074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:26.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:26.159+0000 7f88e4155700 1 --2- 172.21.15.78:0/2154259430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88dc1021e0 0x7f88dc074490 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:26.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:26.159+0000 7f88e4155700 1 --2- 172.21.15.78:0/2154259430 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88dc102ba0 0x7f88dc0749d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:26.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:26.159+0000 7f88e4155700 1 -- 172.21.15.78:0/2154259430 >> 172.21.15.78:0/2154259430 conn(0x7f88dc0792c0 msgr2=0x7f88dc100e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:26.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:26.159+0000 7f88e4155700 1 -- 172.21.15.78:0/2154259430 shutdown_connections 2024-03-24T00:20:26.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:26.159+0000 7f88e4155700 1 -- 172.21.15.78:0/2154259430 wait complete. 2024-03-24T00:20:26.478 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:26 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2154259430' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T00:20:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:26 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2154259430' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T00:20:26.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:26 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2154259430' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T00:20:26.909 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-24T00:20:26.910 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:20:27.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.026+0000 7f6b6a868640 1 -- 172.21.15.78:0/368038774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b641028a0 msgr2=0x7f6b64102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:27.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.026+0000 7f6b6a868640 1 --2- 172.21.15.78:0/368038774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b641028a0 0x7f6b64102d20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6b54009900 tx=0x7f6b5402bc30 comp rx=0 tx=0).stop 2024-03-24T00:20:27.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.026+0000 7f6b6a868640 1 -- 172.21.15.78:0/368038774 shutdown_connections 2024-03-24T00:20:27.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.026+0000 7f6b6a868640 1 --2- 172.21.15.78:0/368038774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b64103260 0x7f6b64109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.026+0000 7f6b6a868640 1 --2- 172.21.15.78:0/368038774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b641028a0 0x7f6b64102d20 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.026+0000 7f6b6a868640 1 --2- 172.21.15.78:0/368038774 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b641016b0 0x7f6b64101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.026+0000 7f6b6a868640 1 -- 172.21.15.78:0/368038774 >> 172.21.15.78:0/368038774 conn(0x7f6b640fcc80 msgr2=0x7f6b640ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:27.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.026+0000 7f6b6a868640 1 -- 172.21.15.78:0/368038774 shutdown_connections 2024-03-24T00:20:27.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.026+0000 7f6b6a868640 1 -- 172.21.15.78:0/368038774 wait complete. 2024-03-24T00:20:27.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.027+0000 7f6b6a868640 1 Processor -- start 2024-03-24T00:20:27.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.027+0000 7f6b6a868640 1 -- start start 2024-03-24T00:20:27.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.027+0000 7f6b6a868640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b641016b0 0x7f6b641979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:27.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.027+0000 7f6b6a868640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b641028a0 0x7f6b64197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:27.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.027+0000 7f6b6a868640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b64103260 0x7f6b6419ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:27.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.027+0000 7f6b6a868640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6410d310 con 0x7f6b641028a0 2024-03-24T00:20:27.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.027+0000 7f6b6a868640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6410d190 con 0x7f6b64103260 2024-03-24T00:20:27.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.027+0000 7f6b6a868640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6410d490 con 0x7f6b641016b0 2024-03-24T00:20:27.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.027+0000 7f6b637fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b641028a0 0x7f6b64197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:27.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.027+0000 7f6b63fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b641016b0 0x7f6b641979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:27.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.027+0000 7f6b637fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b641028a0 0x7f6b64197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41950/0 (socket says 172.21.15.78:41950) 2024-03-24T00:20:27.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.027+0000 7f6b63fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b641016b0 0x7f6b641979c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.78:34730/0 (socket says 172.21.15.78:34730) 2024-03-24T00:20:27.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.027+0000 7f6b637fe640 1 -- 172.21.15.78:0/1305857144 learned_addr learned my addr 172.21.15.78:0/1305857144 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:27.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.027+0000 7f6b68e05640 1 --2- 172.21.15.78:0/1305857144 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b64103260 0x7f6b6419ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:27.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.027+0000 7f6b637fe640 1 -- 172.21.15.78:0/1305857144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b641016b0 msgr2=0x7f6b641979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:27.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.027+0000 7f6b637fe640 1 --2- 172.21.15.78:0/1305857144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b641016b0 0x7f6b641979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.027+0000 7f6b637fe640 1 -- 172.21.15.78:0/1305857144 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b64103260 msgr2=0x7f6b6419ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:27.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.027+0000 7f6b637fe640 1 --2- 172.21.15.78:0/1305857144 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b64103260 0x7f6b6419ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.027+0000 7f6b637fe640 1 -- 172.21.15.78:0/1305857144 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b6419f5e0 con 0x7f6b641028a0 2024-03-24T00:20:27.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.027+0000 7f6b63fff640 1 --2- 172.21.15.78:0/1305857144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b641016b0 0x7f6b641979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:27.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.027+0000 7f6b68e05640 1 --2- 172.21.15.78:0/1305857144 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b64103260 0x7f6b6419ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:27.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.028+0000 7f6b637fe640 1 --2- 172.21.15.78:0/1305857144 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b641028a0 0x7f6b64197f00 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6b54009230 tx=0x7f6b5402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:27.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.028+0000 7f6b617fa640 1 -- 172.21.15.78:0/1305857144 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b54041070 con 0x7f6b641028a0 2024-03-24T00:20:27.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.028+0000 7f6b617fa640 1 -- 172.21.15.78:0/1305857144 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b54004450 con 0x7f6b641028a0 2024-03-24T00:20:27.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.028+0000 7f6b6a868640 1 -- 172.21.15.78:0/1305857144 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b6419f870 con 0x7f6b641028a0 2024-03-24T00:20:27.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.028+0000 7f6b617fa640 1 -- 172.21.15.78:0/1305857144 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b54003e20 con 0x7f6b641028a0 2024-03-24T00:20:27.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.028+0000 7f6b6a868640 1 -- 172.21.15.78:0/1305857144 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b6419fd60 con 0x7f6b641028a0 2024-03-24T00:20:27.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.029+0000 7f6b6a868640 1 -- 172.21.15.78:0/1305857144 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b641089d0 con 0x7f6b641028a0 2024-03-24T00:20:27.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.030+0000 7f6b617fa640 1 -- 172.21.15.78:0/1305857144 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f6b54044070 con 0x7f6b641028a0 2024-03-24T00:20:27.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.031+0000 7f6b617fa640 1 --2- 172.21.15.78:0/1305857144 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f6b4c0a2890 0x7f6b4c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:27.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.031+0000 7f6b63fff640 1 --2- 172.21.15.78:0/1305857144 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f6b4c0a2890 0x7f6b4c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:27.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.031+0000 7f6b617fa640 1 -- 172.21.15.78:0/1305857144 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f6b540f09d0 con 0x7f6b641028a0 2024-03-24T00:20:27.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.031+0000 7f6b63fff640 1 --2- 172.21.15.78:0/1305857144 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f6b4c0a2890 0x7f6b4c0a4d50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6b64102700 tx=0x7f6b58005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:27.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.035+0000 7f6b617fa640 1 -- 172.21.15.78:0/1305857144 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6b540bd070 con 0x7f6b641028a0 2024-03-24T00:20:27.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.214+0000 7f6b6a868640 1 -- 172.21.15.78:0/1305857144 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b6402d2c0 con 0x7f6b641028a0 2024-03-24T00:20:27.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.216+0000 7f6b617fa640 1 -- 172.21.15.78:0/1305857144 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24601 (secure 0 0 0) 0x7f6b540c1110 con 0x7f6b641028a0 2024-03-24T00:20:27.216 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:20:27.216 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":91,"fsid":"c7e2806a-e972-11ee-95cd-87774f69a715","created":"2024-03-24T00:10:44.134563+0000","modified":"2024-03-24T00:19:41.438250+0000","last_up_change":"2024-03-24T00:17:50.237686+0000","last_in_change":"2024-03-24T00:17:29.788699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:14:04.534391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:18:42.178418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:18:42.800424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:18:46.112800+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b950388a-c437-426b-b703-3f8ae821dd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6803","nonce":1816879397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6805","nonce":1816879397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6809","nonce":1816879397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6807","nonce":1816879397}]},"public_addr":"172.21.15.78:6803/1816879397","cluster_addr":"172.21.15.78:6805/1816879397","heartbeat_back_addr":"172.21.15.78:6809/1816879397","heartbeat_front_addr":"172.21.15.78:6807/1816879397","state":["exists","up"]},{"osd":1,"uuid":"b8d64714-dd65-400f-b3f4-673ab14e1365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6801","nonce":1647915612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6803","nonce":1647915612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6807","nonce":1647915612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6805","nonce":1647915612}]},"public_addr":"172.21.15.106:6801/1647915612","cluster_addr":"172.21.15.106:6803/1647915612","heartbeat_back_addr":"172.21.15.106:6807/1647915612","heartbeat_front_addr":"172.21.15.106:6805/1647915612","state":["exists","up"]},{"osd":2,"uuid":"fcfc4dd1-aaf8-4131-a023-5e832212158d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6801","nonce":2010770435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6803","nonce":2010770435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6807","nonce":2010770435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6805","nonce":2010770435}]},"public_addr":"172.21.15.107:6801/2010770435","cluster_addr":"172.21.15.107:6803/2010770435","heartbeat_back_addr":"172.21.15.107:6807/2010770435","heartbeat_front_addr":"172.21.15.107:6805/2010770435","state":["exists","up"]},{"osd":3,"uuid":"47c718ab-a0f4-4e52-83c5-b89386b262d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6811","nonce":1912251346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6813","nonce":1912251346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6817","nonce":1912251346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6815","nonce":1912251346}]},"public_addr":"172.21.15.78:6811/1912251346","cluster_addr":"172.21.15.78:6813/1912251346","heartbeat_back_addr":"172.21.15.78:6817/1912251346","heartbeat_front_addr":"172.21.15.78:6815/1912251346","state":["exists","up"]},{"osd":4,"uuid":"7dad64dd-677e-4c1c-9638-e4f4f2195983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6809","nonce":374641205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6811","nonce":374641205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6815","nonce":374641205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6813","nonce":374641205}]},"public_addr":"172.21.15.106:6809/374641205","cluster_addr":"172.21.15.106:6811/374641205","heartbeat_back_addr":"172.21.15.106:6815/374641205","heartbeat_front_addr":"172.21.15.106:6813/374641205","state":["exists","up"]},{"osd":5,"uuid":"bbd524e5-328d-4309-8877-1149cbe110dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6809","nonce":3272806073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6811","nonce":3272806073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6815","nonce":3272806073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6813","nonce":3272806073}]},"public_addr":"172.21.15.107:6809/3272806073","cluster_addr":"172.21.15.107:6811/3272806073","heartbeat_back_addr":"172.21.15.107:6815/3272806073","heartbeat_front_addr":"172.21.15.107:6813/3272806073","state":["exists","up"]},{"osd":6,"uuid":"5696389a-918f-4ea8-b20d-4e2a0d4cb3bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6819","nonce":3091148508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6821","nonce":3091148508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6825","nonce":3091148508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6823","nonce":3091148508}]},"public_addr":"172.21.15.78:6819/3091148508","cluster_addr":"172.21.15.78:6821/3091148508","heartbeat_back_addr":"172.21.15.78:6825/3091148508","heartbeat_front_addr":"172.21.15.78:6823/3091148508","state":["exists","up"]},{"osd":7,"uuid":"f6b764e8-ed4e-4c9a-a85b-94af6dc6b924","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6817","nonce":4262103080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6819","nonce":4262103080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6823","nonce":4262103080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6821","nonce":4262103080}]},"public_addr":"172.21.15.106:6817/4262103080","cluster_addr":"172.21.15.106:6819/4262103080","heartbeat_back_addr":"172.21.15.106:6823/4262103080","heartbeat_front_addr":"172.21.15.106:6821/4262103080","state":["exists","up"]},{"osd":8,"uuid":"ba16039f-8093-4fa0-87ad-e28b7d142b23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6817","nonce":4032799356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6819","nonce":4032799356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6823","nonce":4032799356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6821","nonce":4032799356}]},"public_addr":"172.21.15.107:6817/4032799356","cluster_addr":"172.21.15.107:6819/4032799356","heartbeat_back_addr":"172.21.15.107:6823/4032799356","heartbeat_front_addr":"172.21.15.107:6821/4032799356","state":["exists","up"]},{"osd":9,"uuid":"651357f2-0669-42e6-aa91-1d5c02d00c5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6827","nonce":214023431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6829","nonce":214023431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6833","nonce":214023431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6831","nonce":214023431}]},"public_addr":"172.21.15.78:6827/214023431","cluster_addr":"172.21.15.78:6829/214023431","heartbeat_back_addr":"172.21.15.78:6833/214023431","heartbeat_front_addr":"172.21.15.78:6831/214023431","state":["exists","up"]},{"osd":10,"uuid":"2375119d-346e-426c-8a65-b64af71ad63a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6825","nonce":400029804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6827","nonce":400029804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6831","nonce":400029804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6829","nonce":400029804}]},"public_addr":"172.21.15.106:6825/400029804","cluster_addr":"172.21.15.106:6827/400029804","heartbeat_back_addr":"172.21.15.106:6831/400029804","heartbeat_front_addr":"172.21.15.106:6829/400029804","state":["exists","up"]},{"osd":11,"uuid":"b6b609d9-dd78-4c01-ae66-ba2a662ab22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6825","nonce":4191916230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6827","nonce":4191916230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6831","nonce":4191916230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6829","nonce":4191916230}]},"public_addr":"172.21.15.107:6825/4191916230","cluster_addr":"172.21.15.107:6827/4191916230","heartbeat_back_addr":"172.21.15.107:6831/4191916230","heartbeat_front_addr":"172.21.15.107:6829/4191916230","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:13.728602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:37.320809+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:01.269452+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:25.758281+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:50.298863+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:14.345219+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:38.980489+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:04.749214+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:29.992673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:56.269202+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:22.270316+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:48.074907+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/1982202541":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/1933054339":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6801/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6800/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/465691596":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6801/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/3962523609":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6800/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/1502752625":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/2236879401":"2024-03-25T00:11:40.199569+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:20:27.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.218+0000 7f6b6a868640 1 -- 172.21.15.78:0/1305857144 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f6b4c0a2890 msgr2=0x7f6b4c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:27.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.218+0000 7f6b6a868640 1 --2- 172.21.15.78:0/1305857144 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f6b4c0a2890 0x7f6b4c0a4d50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6b64102700 tx=0x7f6b58005f90 comp rx=0 tx=0).stop 2024-03-24T00:20:27.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.218+0000 7f6b6a868640 1 -- 172.21.15.78:0/1305857144 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b641028a0 msgr2=0x7f6b64197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:27.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.218+0000 7f6b6a868640 1 --2- 172.21.15.78:0/1305857144 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b641028a0 0x7f6b64197f00 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6b54009230 tx=0x7f6b5402c650 comp rx=0 tx=0).stop 2024-03-24T00:20:27.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.218+0000 7f6b6a868640 1 -- 172.21.15.78:0/1305857144 shutdown_connections 2024-03-24T00:20:27.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.218+0000 7f6b6a868640 1 --2- 172.21.15.78:0/1305857144 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f6b4c0a2890 0x7f6b4c0a4d50 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.218+0000 7f6b6a868640 1 --2- 172.21.15.78:0/1305857144 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b64103260 0x7f6b6419ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.218+0000 7f6b6a868640 1 --2- 172.21.15.78:0/1305857144 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b641028a0 0x7f6b64197f00 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.218+0000 7f6b6a868640 1 --2- 172.21.15.78:0/1305857144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b641016b0 0x7f6b641979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.218+0000 7f6b6a868640 1 -- 172.21.15.78:0/1305857144 >> 172.21.15.78:0/1305857144 conn(0x7f6b640fcc80 msgr2=0x7f6b64105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:27.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.218+0000 7f6b6a868640 1 -- 172.21.15.78:0/1305857144 shutdown_connections 2024-03-24T00:20:27.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.218+0000 7f6b6a868640 1 -- 172.21.15.78:0/1305857144 wait complete. 2024-03-24T00:20:27.230 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:14:04.534391+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:18:42.178418+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:18:42.800424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T00:18:46.112800+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T00:20:27.230 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:20:27.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.337+0000 7fde6d7f1640 1 -- 172.21.15.78:0/624655918 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde680fffb0 msgr2=0x7fde681003d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:27.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.337+0000 7fde6d7f1640 1 --2- 172.21.15.78:0/624655918 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde680fffb0 0x7fde681003d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fde58009900 tx=0x7fde5802bcd0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.337+0000 7fde6d7f1640 1 -- 172.21.15.78:0/624655918 shutdown_connections 2024-03-24T00:20:27.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.337+0000 7fde6d7f1640 1 --2- 172.21.15.78:0/624655918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde68100910 0x7fde68100d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.337+0000 7fde6d7f1640 1 --2- 172.21.15.78:0/624655918 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde681012b0 0x7fde68101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.337+0000 7fde6d7f1640 1 --2- 172.21.15.78:0/624655918 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde680fffb0 0x7fde681003d0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.337+0000 7fde6d7f1640 1 -- 172.21.15.78:0/624655918 >> 172.21.15.78:0/624655918 conn(0x7fde680fbb30 msgr2=0x7fde680fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:27.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.337+0000 7fde6d7f1640 1 -- 172.21.15.78:0/624655918 shutdown_connections 2024-03-24T00:20:27.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.337+0000 7fde6d7f1640 1 -- 172.21.15.78:0/624655918 wait complete. 2024-03-24T00:20:27.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.338+0000 7fde6d7f1640 1 Processor -- start 2024-03-24T00:20:27.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.338+0000 7fde6d7f1640 1 -- start start 2024-03-24T00:20:27.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.338+0000 7fde6d7f1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde680fffb0 0x7fde68199ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:27.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.338+0000 7fde6d7f1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde68100910 0x7fde6819a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:27.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.338+0000 7fde6d7f1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde681012b0 0x7fde681a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:27.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.338+0000 7fde6d7f1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6810f540 con 0x7fde681012b0 2024-03-24T00:20:27.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.338+0000 7fde6d7f1640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6810f3c0 con 0x7fde680fffb0 2024-03-24T00:20:27.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.338+0000 7fde6d7f1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6810f6c0 con 0x7fde68100910 2024-03-24T00:20:27.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.338+0000 7fde677fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde681012b0 0x7fde681a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:27.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.338+0000 7fde677fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde681012b0 0x7fde681a1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41960/0 (socket says 172.21.15.78:41960) 2024-03-24T00:20:27.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.338+0000 7fde677fe640 1 -- 172.21.15.78:0/3061922857 learned_addr learned my addr 172.21.15.78:0/3061922857 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:27.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.338+0000 7fde5ffff640 1 --2- 172.21.15.78:0/3061922857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde68100910 0x7fde6819a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:27.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.338+0000 7fde66ffd640 1 --2- 172.21.15.78:0/3061922857 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde680fffb0 0x7fde68199ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:27.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.338+0000 7fde677fe640 1 -- 172.21.15.78:0/3061922857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde68100910 msgr2=0x7fde6819a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:27.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.338+0000 7fde677fe640 1 --2- 172.21.15.78:0/3061922857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde68100910 0x7fde6819a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.338+0000 7fde677fe640 1 -- 172.21.15.78:0/3061922857 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde680fffb0 msgr2=0x7fde68199ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:27.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.338+0000 7fde677fe640 1 --2- 172.21.15.78:0/3061922857 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde680fffb0 0x7fde68199ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.338+0000 7fde677fe640 1 -- 172.21.15.78:0/3061922857 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde681a18d0 con 0x7fde681012b0 2024-03-24T00:20:27.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.338+0000 7fde677fe640 1 --2- 172.21.15.78:0/3061922857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde681012b0 0x7fde681a1230 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fde6000efd0 tx=0x7fde6000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:27.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.339+0000 7fde64ff9640 1 -- 172.21.15.78:0/3061922857 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde60003b70 con 0x7fde681012b0 2024-03-24T00:20:27.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.339+0000 7fde64ff9640 1 -- 172.21.15.78:0/3061922857 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde60003d10 con 0x7fde681012b0 2024-03-24T00:20:27.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.339+0000 7fde6d7f1640 1 -- 172.21.15.78:0/3061922857 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde681a1bc0 con 0x7fde681012b0 2024-03-24T00:20:27.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.339+0000 7fde64ff9640 1 -- 172.21.15.78:0/3061922857 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde600109f0 con 0x7fde681012b0 2024-03-24T00:20:27.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.339+0000 7fde6d7f1640 1 -- 172.21.15.78:0/3061922857 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde681a2110 con 0x7fde681012b0 2024-03-24T00:20:27.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.340+0000 7fde6d7f1640 1 -- 172.21.15.78:0/3061922857 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde6810afe0 con 0x7fde681012b0 2024-03-24T00:20:27.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.341+0000 7fde64ff9640 1 -- 172.21.15.78:0/3061922857 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fde60004020 con 0x7fde681012b0 2024-03-24T00:20:27.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.341+0000 7fde64ff9640 1 --2- 172.21.15.78:0/3061922857 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fde540a2910 0x7fde540a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:27.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.342+0000 7fde66ffd640 1 --2- 172.21.15.78:0/3061922857 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fde540a2910 0x7fde540a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:27.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.342+0000 7fde64ff9640 1 -- 172.21.15.78:0/3061922857 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fde600cf2e0 con 0x7fde681012b0 2024-03-24T00:20:27.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.342+0000 7fde66ffd640 1 --2- 172.21.15.78:0/3061922857 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fde540a2910 0x7fde540a4dd0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fde680fde40 tx=0x7fde58005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:27.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.346+0000 7fde64ff9640 1 -- 172.21.15.78:0/3061922857 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fde6009bec0 con 0x7fde681012b0 2024-03-24T00:20:27.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.524+0000 7fde6d7f1640 1 -- 172.21.15.78:0/3061922857 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fde6802d2c0 con 0x7fde681012b0 2024-03-24T00:20:27.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.524+0000 7fde64ff9640 1 -- 172.21.15.78:0/3061922857 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7fde6009b7c0 con 0x7fde681012b0 2024-03-24T00:20:27.524 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-24T00:20:27.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.526+0000 7fde6d7f1640 1 -- 172.21.15.78:0/3061922857 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fde540a2910 msgr2=0x7fde540a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:27.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.526+0000 7fde6d7f1640 1 --2- 172.21.15.78:0/3061922857 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fde540a2910 0x7fde540a4dd0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fde680fde40 tx=0x7fde58005fd0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.526+0000 7fde6d7f1640 1 -- 172.21.15.78:0/3061922857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde681012b0 msgr2=0x7fde681a1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:27.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.526+0000 7fde6d7f1640 1 --2- 172.21.15.78:0/3061922857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde681012b0 0x7fde681a1230 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fde6000efd0 tx=0x7fde6000c6a0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.526+0000 7fde6d7f1640 1 -- 172.21.15.78:0/3061922857 shutdown_connections 2024-03-24T00:20:27.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.526+0000 7fde6d7f1640 1 --2- 172.21.15.78:0/3061922857 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fde540a2910 0x7fde540a4dd0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.526+0000 7fde6d7f1640 1 --2- 172.21.15.78:0/3061922857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde68100910 0x7fde6819a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.526+0000 7fde6d7f1640 1 --2- 172.21.15.78:0/3061922857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde681012b0 0x7fde681a1230 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.526+0000 7fde6d7f1640 1 --2- 172.21.15.78:0/3061922857 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde680fffb0 0x7fde68199ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.526+0000 7fde6d7f1640 1 -- 172.21.15.78:0/3061922857 >> 172.21.15.78:0/3061922857 conn(0x7fde680fbb30 msgr2=0x7fde680fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:27.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.526+0000 7fde6d7f1640 1 -- 172.21.15.78:0/3061922857 shutdown_connections 2024-03-24T00:20:27.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.526+0000 7fde6d7f1640 1 -- 172.21.15.78:0/3061922857 wait complete. 2024-03-24T00:20:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:27 smithi078 ceph-mon[32429]: pgmap v344: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:20:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:27 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1305857144' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:20:27.537 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:20:27.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.642+0000 7f34b94dc640 1 -- 172.21.15.78:0/2030628800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34b4102150 msgr2=0x7f34b41089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:27.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.642+0000 7f34b94dc640 1 --2- 172.21.15.78:0/2030628800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34b4102150 0x7f34b41089e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f34ac009900 tx=0x7f34ac02bc30 comp rx=0 tx=0).stop 2024-03-24T00:20:27.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.642+0000 7f34b94dc640 1 -- 172.21.15.78:0/2030628800 shutdown_connections 2024-03-24T00:20:27.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.642+0000 7f34b94dc640 1 --2- 172.21.15.78:0/2030628800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34b4101790 0x7f34b4101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.642+0000 7f34b94dc640 1 --2- 172.21.15.78:0/2030628800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34b4102150 0x7f34b41089e0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.642+0000 7f34b94dc640 1 --2- 172.21.15.78:0/2030628800 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34b41005a0 0x7f34b41009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.642+0000 7f34b94dc640 1 -- 172.21.15.78:0/2030628800 >> 172.21.15.78:0/2030628800 conn(0x7f34b40fbb30 msgr2=0x7f34b40fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:27.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.642+0000 7f34b94dc640 1 -- 172.21.15.78:0/2030628800 shutdown_connections 2024-03-24T00:20:27.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.642+0000 7f34b94dc640 1 -- 172.21.15.78:0/2030628800 wait complete. 2024-03-24T00:20:27.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.643+0000 7f34b94dc640 1 Processor -- start 2024-03-24T00:20:27.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.643+0000 7f34b94dc640 1 -- start start 2024-03-24T00:20:27.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.643+0000 7f34b94dc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34b41005a0 0x7f34b4197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:27.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.643+0000 7f34b94dc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34b4101790 0x7f34b4197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:27.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.643+0000 7f34b94dc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34b4102150 0x7f34b419ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:27.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.643+0000 7f34b94dc640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b410c230 con 0x7f34b4102150 2024-03-24T00:20:27.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.643+0000 7f34b94dc640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b410c0b0 con 0x7f34b4101790 2024-03-24T00:20:27.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.643+0000 7f34b94dc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b410c3b0 con 0x7f34b41005a0 2024-03-24T00:20:27.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.643+0000 7f34b37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34b4102150 0x7f34b419ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:27.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.643+0000 7f34b37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34b4102150 0x7f34b419ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41990/0 (socket says 172.21.15.78:41990) 2024-03-24T00:20:27.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.643+0000 7f34b37fe640 1 -- 172.21.15.78:0/2765561250 learned_addr learned my addr 172.21.15.78:0/2765561250 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:27.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.643+0000 7f34b27fc640 1 --2- 172.21.15.78:0/2765561250 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34b4101790 0x7f34b4197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:27.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.643+0000 7f34b2ffd640 1 --2- 172.21.15.78:0/2765561250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34b41005a0 0x7f34b4197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:27.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.643+0000 7f34b37fe640 1 -- 172.21.15.78:0/2765561250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34b41005a0 msgr2=0x7f34b4197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:27.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.643+0000 7f34b37fe640 1 --2- 172.21.15.78:0/2765561250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34b41005a0 0x7f34b4197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.643+0000 7f34b37fe640 1 -- 172.21.15.78:0/2765561250 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34b4101790 msgr2=0x7f34b4197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:27.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.643+0000 7f34b37fe640 1 --2- 172.21.15.78:0/2765561250 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34b4101790 0x7f34b4197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.644+0000 7f34b37fe640 1 -- 172.21.15.78:0/2765561250 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b419f5a0 con 0x7f34b4102150 2024-03-24T00:20:27.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.644+0000 7f34b37fe640 1 --2- 172.21.15.78:0/2765561250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34b4102150 0x7f34b419ef30 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f34ac005ab0 tx=0x7f34ac02c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:27.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.644+0000 7f349bfff640 1 -- 172.21.15.78:0/2765561250 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ac041070 con 0x7f34b4102150 2024-03-24T00:20:27.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.644+0000 7f349bfff640 1 -- 172.21.15.78:0/2765561250 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f34ac004450 con 0x7f34b4102150 2024-03-24T00:20:27.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.644+0000 7f34b94dc640 1 -- 172.21.15.78:0/2765561250 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34b419f830 con 0x7f34b4102150 2024-03-24T00:20:27.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.644+0000 7f349bfff640 1 -- 172.21.15.78:0/2765561250 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ac003e60 con 0x7f34b4102150 2024-03-24T00:20:27.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.644+0000 7f34b94dc640 1 -- 172.21.15.78:0/2765561250 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34b419fd20 con 0x7f34b4102150 2024-03-24T00:20:27.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.645+0000 7f34b94dc640 1 -- 172.21.15.78:0/2765561250 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34b419ff80 con 0x7f34b4102150 2024-03-24T00:20:27.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.646+0000 7f349bfff640 1 -- 172.21.15.78:0/2765561250 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f34ac044070 con 0x7f34b4102150 2024-03-24T00:20:27.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.646+0000 7f349bfff640 1 --2- 172.21.15.78:0/2765561250 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f349c0a2890 0x7f349c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:27.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.646+0000 7f349bfff640 1 -- 172.21.15.78:0/2765561250 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f34ac0f0e20 con 0x7f34b4102150 2024-03-24T00:20:27.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.646+0000 7f34b2ffd640 1 --2- 172.21.15.78:0/2765561250 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f349c0a2890 0x7f349c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:27.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.647+0000 7f34b2ffd640 1 --2- 172.21.15.78:0/2765561250 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f349c0a2890 0x7f349c0a4d50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f34a40099a0 tx=0x7f34a4009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:27.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.650+0000 7f349bfff640 1 -- 172.21.15.78:0/2765561250 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f34ac0bd4c0 con 0x7f34b4102150 2024-03-24T00:20:27.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.826+0000 7f34b94dc640 1 -- 172.21.15.78:0/2765561250 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f34b41a0270 con 0x7f34b4102150 2024-03-24T00:20:27.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.826+0000 7f349bfff640 1 -- 172.21.15.78:0/2765561250 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7f34ac0c1560 con 0x7f34b4102150 2024-03-24T00:20:27.826 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-24T00:20:27.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.828+0000 7f34b94dc640 1 -- 172.21.15.78:0/2765561250 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f349c0a2890 msgr2=0x7f349c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:27.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.828+0000 7f34b94dc640 1 --2- 172.21.15.78:0/2765561250 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f349c0a2890 0x7f349c0a4d50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f34a40099a0 tx=0x7f34a4009500 comp rx=0 tx=0).stop 2024-03-24T00:20:27.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.828+0000 7f34b94dc640 1 -- 172.21.15.78:0/2765561250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34b4102150 msgr2=0x7f34b419ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:27.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.828+0000 7f34b94dc640 1 --2- 172.21.15.78:0/2765561250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34b4102150 0x7f34b419ef30 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f34ac005ab0 tx=0x7f34ac02c6c0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.828+0000 7f34b94dc640 1 -- 172.21.15.78:0/2765561250 shutdown_connections 2024-03-24T00:20:27.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.828+0000 7f34b94dc640 1 --2- 172.21.15.78:0/2765561250 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f349c0a2890 0x7f349c0a4d50 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.828+0000 7f34b94dc640 1 --2- 172.21.15.78:0/2765561250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34b4102150 0x7f34b419ef30 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.828+0000 7f34b94dc640 1 --2- 172.21.15.78:0/2765561250 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34b4101790 0x7f34b4197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.828+0000 7f34b94dc640 1 --2- 172.21.15.78:0/2765561250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34b41005a0 0x7f34b4197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.828+0000 7f34b94dc640 1 -- 172.21.15.78:0/2765561250 >> 172.21.15.78:0/2765561250 conn(0x7f34b40fbb30 msgr2=0x7f34b4068800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:27.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.829+0000 7f34b94dc640 1 -- 172.21.15.78:0/2765561250 shutdown_connections 2024-03-24T00:20:27.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.829+0000 7f34b94dc640 1 -- 172.21.15.78:0/2765561250 wait complete. 2024-03-24T00:20:27.840 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:20:27.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:27 smithi107 ceph-mon[36485]: pgmap v344: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:20:27.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:27 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1305857144' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:20:27.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:27 smithi106 ceph-mon[36505]: pgmap v344: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:20:27.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:27 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1305857144' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:20:27.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.947+0000 7f03d79e6640 1 -- 172.21.15.78:0/2383172023 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d0101790 msgr2=0x7f03d0101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:27.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.947+0000 7f03d79e6640 1 --2- 172.21.15.78:0/2383172023 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d0101790 0x7f03d0101c10 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f03c4009900 tx=0x7f03c402bc30 comp rx=0 tx=0).stop 2024-03-24T00:20:27.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.948+0000 7f03d79e6640 1 -- 172.21.15.78:0/2383172023 shutdown_connections 2024-03-24T00:20:27.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.948+0000 7f03d79e6640 1 --2- 172.21.15.78:0/2383172023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03d0102150 0x7f03d01089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.948+0000 7f03d79e6640 1 --2- 172.21.15.78:0/2383172023 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d0101790 0x7f03d0101c10 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.948+0000 7f03d79e6640 1 --2- 172.21.15.78:0/2383172023 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f03d01005a0 0x7f03d01009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.948+0000 7f03d79e6640 1 -- 172.21.15.78:0/2383172023 >> 172.21.15.78:0/2383172023 conn(0x7f03d00fbb30 msgr2=0x7f03d00fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:27.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.948+0000 7f03d79e6640 1 -- 172.21.15.78:0/2383172023 shutdown_connections 2024-03-24T00:20:27.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.948+0000 7f03d79e6640 1 -- 172.21.15.78:0/2383172023 wait complete. 2024-03-24T00:20:27.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.948+0000 7f03d79e6640 1 Processor -- start 2024-03-24T00:20:27.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.948+0000 7f03d79e6640 1 -- start start 2024-03-24T00:20:27.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.949+0000 7f03d79e6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f03d01005a0 0x7f03d0196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:27.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.949+0000 7f03d79e6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d0101790 0x7f03d0196e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:27.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.949+0000 7f03d79e6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03d0102150 0x7f03d019dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:27.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.949+0000 7f03d79e6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d010c230 con 0x7f03d0101790 2024-03-24T00:20:27.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.949+0000 7f03d79e6640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d010c0b0 con 0x7f03d01005a0 2024-03-24T00:20:27.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.949+0000 7f03d79e6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d010c3b0 con 0x7f03d0102150 2024-03-24T00:20:27.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.949+0000 7f03d4f81640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d0101790 0x7f03d0196e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:27.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.949+0000 7f03d4f81640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d0101790 0x7f03d0196e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42006/0 (socket says 172.21.15.78:42006) 2024-03-24T00:20:27.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.949+0000 7f03d5782640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f03d01005a0 0x7f03d0196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:27.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.949+0000 7f03d4f81640 1 -- 172.21.15.78:0/2090458505 learned_addr learned my addr 172.21.15.78:0/2090458505 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:27.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.949+0000 7f03d5f83640 1 --2- 172.21.15.78:0/2090458505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03d0102150 0x7f03d019dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:27.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.949+0000 7f03d4f81640 1 -- 172.21.15.78:0/2090458505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03d0102150 msgr2=0x7f03d019dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:27.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.949+0000 7f03d4f81640 1 --2- 172.21.15.78:0/2090458505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03d0102150 0x7f03d019dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.949+0000 7f03d4f81640 1 -- 172.21.15.78:0/2090458505 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f03d01005a0 msgr2=0x7f03d0196910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:27.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.949+0000 7f03d4f81640 1 --2- 172.21.15.78:0/2090458505 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f03d01005a0 0x7f03d0196910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:27.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.949+0000 7f03d4f81640 1 -- 172.21.15.78:0/2090458505 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d019e530 con 0x7f03d0101790 2024-03-24T00:20:27.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.949+0000 7f03d4f81640 1 --2- 172.21.15.78:0/2090458505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d0101790 0x7f03d0196e50 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f03c4009230 tx=0x7f03c402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:27.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.949+0000 7f03ce7fc640 1 -- 172.21.15.78:0/2090458505 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c4041070 con 0x7f03d0101790 2024-03-24T00:20:27.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.949+0000 7f03ce7fc640 1 -- 172.21.15.78:0/2090458505 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f03c4004450 con 0x7f03d0101790 2024-03-24T00:20:27.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.949+0000 7f03d79e6640 1 -- 172.21.15.78:0/2090458505 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d019e7c0 con 0x7f03d0101790 2024-03-24T00:20:27.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.949+0000 7f03ce7fc640 1 -- 172.21.15.78:0/2090458505 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c4003db0 con 0x7f03d0101790 2024-03-24T00:20:27.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.949+0000 7f03d79e6640 1 -- 172.21.15.78:0/2090458505 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d019ecb0 con 0x7f03d0101790 2024-03-24T00:20:27.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.951+0000 7f03d79e6640 1 -- 172.21.15.78:0/2090458505 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d0107990 con 0x7f03d0101790 2024-03-24T00:20:27.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.951+0000 7f03ce7fc640 1 -- 172.21.15.78:0/2090458505 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f03c4044070 con 0x7f03d0101790 2024-03-24T00:20:27.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.952+0000 7f03ce7fc640 1 --2- 172.21.15.78:0/2090458505 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f03b80a2890 0x7f03b80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:27.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.952+0000 7f03d5782640 1 --2- 172.21.15.78:0/2090458505 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f03b80a2890 0x7f03b80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:27.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.952+0000 7f03ce7fc640 1 -- 172.21.15.78:0/2090458505 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f03c40f09f0 con 0x7f03d0101790 2024-03-24T00:20:27.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.953+0000 7f03d5782640 1 --2- 172.21.15.78:0/2090458505 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f03b80a2890 0x7f03b80a4d50 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f03d01015f0 tx=0x7f03c0009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:27.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:27.957+0000 7f03ce7fc640 1 -- 172.21.15.78:0/2090458505 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f03c40bd6e0 con 0x7f03d0101790 2024-03-24T00:20:28.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.137+0000 7f03d79e6640 1 -- 172.21.15.78:0/2090458505 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f03d002d070 con 0x7f03d0101790 2024-03-24T00:20:28.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.137+0000 7f03ce7fc640 1 -- 172.21.15.78:0/2090458505 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7f03c40bcfe0 con 0x7f03d0101790 2024-03-24T00:20:28.137 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-03-24T00:20:28.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.139+0000 7f03d79e6640 1 -- 172.21.15.78:0/2090458505 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f03b80a2890 msgr2=0x7f03b80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:28.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.139+0000 7f03d79e6640 1 --2- 172.21.15.78:0/2090458505 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f03b80a2890 0x7f03b80a4d50 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f03d01015f0 tx=0x7f03c0009500 comp rx=0 tx=0).stop 2024-03-24T00:20:28.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.139+0000 7f03d79e6640 1 -- 172.21.15.78:0/2090458505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d0101790 msgr2=0x7f03d0196e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:28.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.139+0000 7f03d79e6640 1 --2- 172.21.15.78:0/2090458505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d0101790 0x7f03d0196e50 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f03c4009230 tx=0x7f03c402c650 comp rx=0 tx=0).stop 2024-03-24T00:20:28.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.139+0000 7f03d79e6640 1 -- 172.21.15.78:0/2090458505 shutdown_connections 2024-03-24T00:20:28.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.139+0000 7f03d79e6640 1 --2- 172.21.15.78:0/2090458505 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f03b80a2890 0x7f03b80a4d50 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:28.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.139+0000 7f03d79e6640 1 --2- 172.21.15.78:0/2090458505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03d0102150 0x7f03d019dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:28.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.139+0000 7f03d79e6640 1 --2- 172.21.15.78:0/2090458505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d0101790 0x7f03d0196e50 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:28.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.139+0000 7f03d79e6640 1 --2- 172.21.15.78:0/2090458505 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f03d01005a0 0x7f03d0196910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:28.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.139+0000 7f03d79e6640 1 -- 172.21.15.78:0/2090458505 >> 172.21.15.78:0/2090458505 conn(0x7f03d00fbb30 msgr2=0x7f03d0104b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:28.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.140+0000 7f03d79e6640 1 -- 172.21.15.78:0/2090458505 shutdown_connections 2024-03-24T00:20:28.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.140+0000 7f03d79e6640 1 -- 172.21.15.78:0/2090458505 wait complete. 2024-03-24T00:20:28.150 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T00:20:28.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.256+0000 7f7803df4640 1 -- 172.21.15.78:0/1850590840 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77fc0ff980 msgr2=0x7f77fc0ffda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:28.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.256+0000 7f7803df4640 1 --2- 172.21.15.78:0/1850590840 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77fc0ff980 0x7f77fc0ffda0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f77f8009860 tx=0x7f77f802bb70 comp rx=0 tx=0).stop 2024-03-24T00:20:28.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.256+0000 7f7803df4640 1 -- 172.21.15.78:0/1850590840 shutdown_connections 2024-03-24T00:20:28.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.256+0000 7f7803df4640 1 --2- 172.21.15.78:0/1850590840 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77fc103310 0x7f77fc109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:28.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.256+0000 7f7803df4640 1 --2- 172.21.15.78:0/1850590840 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77fc102950 0x7f77fc102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:28.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.256+0000 7f7803df4640 1 --2- 172.21.15.78:0/1850590840 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77fc0ff980 0x7f77fc0ffda0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:28.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.256+0000 7f7803df4640 1 -- 172.21.15.78:0/1850590840 >> 172.21.15.78:0/1850590840 conn(0x7f77fc0fb2b0 msgr2=0x7f77fc0fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:28.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.256+0000 7f7803df4640 1 -- 172.21.15.78:0/1850590840 shutdown_connections 2024-03-24T00:20:28.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.256+0000 7f7803df4640 1 -- 172.21.15.78:0/1850590840 wait complete. 2024-03-24T00:20:28.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.257+0000 7f7803df4640 1 Processor -- start 2024-03-24T00:20:28.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.257+0000 7f7803df4640 1 -- start start 2024-03-24T00:20:28.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.257+0000 7f7803df4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77fc0ff980 0x7f77fc1022c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:28.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.257+0000 7f7803df4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77fc102950 0x7f77fc100900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:28.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.257+0000 7f7803df4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77fc103310 0x7f77fc100e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:28.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.257+0000 7f7803df4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77fc10d360 con 0x7f77fc0ff980 2024-03-24T00:20:28.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.257+0000 7f7803df4640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77fc10d1e0 con 0x7f77fc103310 2024-03-24T00:20:28.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.257+0000 7f7803df4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77fc10d4e0 con 0x7f77fc102950 2024-03-24T00:20:28.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.257+0000 7f7801b90640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77fc0ff980 0x7f77fc1022c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:28.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.257+0000 7f7801b90640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77fc0ff980 0x7f77fc1022c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42022/0 (socket says 172.21.15.78:42022) 2024-03-24T00:20:28.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.257+0000 7f7801b90640 1 -- 172.21.15.78:0/806906795 learned_addr learned my addr 172.21.15.78:0/806906795 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:28.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.257+0000 7f780138f640 1 --2- 172.21.15.78:0/806906795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77fc102950 0x7f77fc100900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:28.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.257+0000 7f7802391640 1 --2- 172.21.15.78:0/806906795 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77fc103310 0x7f77fc100e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:28.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.257+0000 7f7801b90640 1 -- 172.21.15.78:0/806906795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77fc102950 msgr2=0x7f77fc100900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:28.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.257+0000 7f7801b90640 1 --2- 172.21.15.78:0/806906795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77fc102950 0x7f77fc100900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:28.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.257+0000 7f7801b90640 1 -- 172.21.15.78:0/806906795 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77fc103310 msgr2=0x7f77fc100e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:28.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.257+0000 7f7801b90640 1 --2- 172.21.15.78:0/806906795 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77fc103310 0x7f77fc100e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:28.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.257+0000 7f7801b90640 1 -- 172.21.15.78:0/806906795 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77fc1016d0 con 0x7f77fc0ff980 2024-03-24T00:20:28.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.257+0000 7f7801b90640 1 --2- 172.21.15.78:0/806906795 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77fc0ff980 0x7f77fc1022c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f77f802c0b0 tx=0x7f77f8034930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:28.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.258+0000 7f77f2ffd640 1 -- 172.21.15.78:0/806906795 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f8041020 con 0x7f77fc0ff980 2024-03-24T00:20:28.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.258+0000 7f7803df4640 1 -- 172.21.15.78:0/806906795 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77fc1a1a90 con 0x7f77fc0ff980 2024-03-24T00:20:28.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.258+0000 7f77f2ffd640 1 -- 172.21.15.78:0/806906795 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f77f802c9f0 con 0x7f77fc0ff980 2024-03-24T00:20:28.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.258+0000 7f77f2ffd640 1 -- 172.21.15.78:0/806906795 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f8039040 con 0x7f77fc0ff980 2024-03-24T00:20:28.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.258+0000 7f7803df4640 1 -- 172.21.15.78:0/806906795 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77fc1a2030 con 0x7f77fc0ff980 2024-03-24T00:20:28.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.258+0000 7f7803df4640 1 -- 172.21.15.78:0/806906795 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77fc108a80 con 0x7f77fc0ff980 2024-03-24T00:20:28.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.260+0000 7f77f2ffd640 1 -- 172.21.15.78:0/806906795 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f77f8044020 con 0x7f77fc0ff980 2024-03-24T00:20:28.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.260+0000 7f77f2ffd640 1 --2- 172.21.15.78:0/806906795 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f77e80a2890 0x7f77e80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:28.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.260+0000 7f780138f640 1 --2- 172.21.15.78:0/806906795 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f77e80a2890 0x7f77e80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:28.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.260+0000 7f77f2ffd640 1 -- 172.21.15.78:0/806906795 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f77f80f06a0 con 0x7f77fc0ff980 2024-03-24T00:20:28.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.261+0000 7f780138f640 1 --2- 172.21.15.78:0/806906795 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f77e80a2890 0x7f77e80a4d50 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f77fc102030 tx=0x7f77ec009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:28.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.264+0000 7f77f2ffd640 1 -- 172.21.15.78:0/806906795 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f77f804b020 con 0x7f77fc0ff980 2024-03-24T00:20:28.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.441+0000 7f7803df4640 1 -- 172.21.15.78:0/806906795 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f77fc02d2c0 con 0x7f77fc0ff980 2024-03-24T00:20:28.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.441+0000 7f77f2ffd640 1 -- 172.21.15.78:0/806906795 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7f77f80bd390 con 0x7f77fc0ff980 2024-03-24T00:20:28.441 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-24T00:20:28.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.443+0000 7f7803df4640 1 -- 172.21.15.78:0/806906795 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f77e80a2890 msgr2=0x7f77e80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:28.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.443+0000 7f7803df4640 1 --2- 172.21.15.78:0/806906795 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f77e80a2890 0x7f77e80a4d50 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f77fc102030 tx=0x7f77ec009260 comp rx=0 tx=0).stop 2024-03-24T00:20:28.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.443+0000 7f7803df4640 1 -- 172.21.15.78:0/806906795 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77fc0ff980 msgr2=0x7f77fc1022c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:28.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.443+0000 7f7803df4640 1 --2- 172.21.15.78:0/806906795 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77fc0ff980 0x7f77fc1022c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f77f802c0b0 tx=0x7f77f8034930 comp rx=0 tx=0).stop 2024-03-24T00:20:28.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.444+0000 7f7803df4640 1 -- 172.21.15.78:0/806906795 shutdown_connections 2024-03-24T00:20:28.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.444+0000 7f7803df4640 1 --2- 172.21.15.78:0/806906795 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f77e80a2890 0x7f77e80a4d50 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:28.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.444+0000 7f7803df4640 1 --2- 172.21.15.78:0/806906795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77fc102950 0x7f77fc100900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:28.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.444+0000 7f7803df4640 1 --2- 172.21.15.78:0/806906795 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77fc103310 0x7f77fc100e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:28.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.444+0000 7f7803df4640 1 --2- 172.21.15.78:0/806906795 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77fc0ff980 0x7f77fc1022c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:28.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.444+0000 7f7803df4640 1 -- 172.21.15.78:0/806906795 >> 172.21.15.78:0/806906795 conn(0x7f77fc0fb2b0 msgr2=0x7f77fc105c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:28.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.444+0000 7f7803df4640 1 -- 172.21.15.78:0/806906795 shutdown_connections 2024-03-24T00:20:28.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.444+0000 7f7803df4640 1 -- 172.21.15.78:0/806906795 wait complete. 2024-03-24T00:20:28.454 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:20:28.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.559+0000 7f39f5036640 1 -- 172.21.15.78:0/2133513431 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39f01005c0 msgr2=0x7f39f01009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:28.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.559+0000 7f39f5036640 1 --2- 172.21.15.78:0/2133513431 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39f01005c0 0x7f39f01009e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f39e0009900 tx=0x7f39e002bc30 comp rx=0 tx=0).stop 2024-03-24T00:20:28.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.559+0000 7f39f5036640 1 -- 172.21.15.78:0/2133513431 shutdown_connections 2024-03-24T00:20:28.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.559+0000 7f39f5036640 1 --2- 172.21.15.78:0/2133513431 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39f01017b0 0x7f39f0101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:28.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.559+0000 7f39f5036640 1 --2- 172.21.15.78:0/2133513431 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39f0102170 0x7f39f0108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:28.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.559+0000 7f39f5036640 1 --2- 172.21.15.78:0/2133513431 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39f01005c0 0x7f39f01009e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:28.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.559+0000 7f39f5036640 1 -- 172.21.15.78:0/2133513431 >> 172.21.15.78:0/2133513431 conn(0x7f39f00fbb30 msgr2=0x7f39f00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:28.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.559+0000 7f39f5036640 1 -- 172.21.15.78:0/2133513431 shutdown_connections 2024-03-24T00:20:28.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.559+0000 7f39f5036640 1 -- 172.21.15.78:0/2133513431 wait complete. 2024-03-24T00:20:28.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.560+0000 7f39f5036640 1 Processor -- start 2024-03-24T00:20:28.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.560+0000 7f39f5036640 1 -- start start 2024-03-24T00:20:28.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.560+0000 7f39f5036640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39f01005c0 0x7f39f0196900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:28.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.560+0000 7f39f5036640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39f01017b0 0x7f39f0196e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:28.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.560+0000 7f39f5036640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39f0102170 0x7f39f019deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:28.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.560+0000 7f39f5036640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f010c230 con 0x7f39f0102170 2024-03-24T00:20:28.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.560+0000 7f39f5036640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f010c0b0 con 0x7f39f01017b0 2024-03-24T00:20:28.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.560+0000 7f39f5036640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f010c3b0 con 0x7f39f01005c0 2024-03-24T00:20:28.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.560+0000 7f39ef59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39f0102170 0x7f39f019deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:28.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.560+0000 7f39ef59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39f0102170 0x7f39f019deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42034/0 (socket says 172.21.15.78:42034) 2024-03-24T00:20:28.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.560+0000 7f39eed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39f01005c0 0x7f39f0196900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:28.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.560+0000 7f39ef59e640 1 -- 172.21.15.78:0/1942514625 learned_addr learned my addr 172.21.15.78:0/1942514625 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:28.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.560+0000 7f39ee59c640 1 --2- 172.21.15.78:0/1942514625 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39f01017b0 0x7f39f0196e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:28.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.560+0000 7f39ef59e640 1 -- 172.21.15.78:0/1942514625 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39f01005c0 msgr2=0x7f39f0196900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:28.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.560+0000 7f39ef59e640 1 --2- 172.21.15.78:0/1942514625 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39f01005c0 0x7f39f0196900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:28.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.560+0000 7f39ef59e640 1 -- 172.21.15.78:0/1942514625 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39f01017b0 msgr2=0x7f39f0196e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:28.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.560+0000 7f39ef59e640 1 --2- 172.21.15.78:0/1942514625 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39f01017b0 0x7f39f0196e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:28.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.560+0000 7f39ef59e640 1 -- 172.21.15.78:0/1942514625 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f019e520 con 0x7f39f0102170 2024-03-24T00:20:28.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.560+0000 7f39eed9d640 1 --2- 172.21.15.78:0/1942514625 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39f01005c0 0x7f39f0196900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:28.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.561+0000 7f39ef59e640 1 --2- 172.21.15.78:0/1942514625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39f0102170 0x7f39f019deb0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f39e800eb10 tx=0x7f39e800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:28.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.561+0000 7f39d7fff640 1 -- 172.21.15.78:0/1942514625 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e8004b20 con 0x7f39f0102170 2024-03-24T00:20:28.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.561+0000 7f39f5036640 1 -- 172.21.15.78:0/1942514625 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39f019e810 con 0x7f39f0102170 2024-03-24T00:20:28.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.561+0000 7f39d7fff640 1 -- 172.21.15.78:0/1942514625 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f39e8004cc0 con 0x7f39f0102170 2024-03-24T00:20:28.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.561+0000 7f39d7fff640 1 -- 172.21.15.78:0/1942514625 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e8010760 con 0x7f39f0102170 2024-03-24T00:20:28.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.561+0000 7f39f5036640 1 -- 172.21.15.78:0/1942514625 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39f019ed60 con 0x7f39f0102170 2024-03-24T00:20:28.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.562+0000 7f39f5036640 1 -- 172.21.15.78:0/1942514625 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39f0107960 con 0x7f39f0102170 2024-03-24T00:20:28.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.563+0000 7f39d7fff640 1 -- 172.21.15.78:0/1942514625 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f39e8005020 con 0x7f39f0102170 2024-03-24T00:20:28.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.564+0000 7f39d7fff640 1 --2- 172.21.15.78:0/1942514625 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f39d80a2890 0x7f39d80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:28.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.564+0000 7f39eed9d640 1 --2- 172.21.15.78:0/1942514625 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f39d80a2890 0x7f39d80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:28.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.564+0000 7f39d7fff640 1 -- 172.21.15.78:0/1942514625 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f39e80cf4b0 con 0x7f39f0102170 2024-03-24T00:20:28.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.564+0000 7f39eed9d640 1 --2- 172.21.15.78:0/1942514625 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f39d80a2890 0x7f39d80a4d50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f39e0000c00 tx=0x7f39e0005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:28.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.568+0000 7f39d7fff640 1 -- 172.21.15.78:0/1942514625 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f39e809c090 con 0x7f39f0102170 2024-03-24T00:20:28.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.783+0000 7f39f5036640 1 -- 172.21.15.78:0/1942514625 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39f00622a0 con 0x7f39f0102170 2024-03-24T00:20:28.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.784+0000 7f39d7fff640 1 -- 172.21.15.78:0/1942514625 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11406 (secure 0 0 0) 0x7f39e809b990 con 0x7f39f0102170 2024-03-24T00:20:28.784 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:20:28.784 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1416685123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1416685123},{"type":"v1","addr":"172.21.15.78:6835","nonce":1416685123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1071274623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1071274623},{"type":"v1","addr":"172.21.15.78:6837","nonce":1071274623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2758357551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2758357551},{"type":"v1","addr":"172.21.15.78:6839","nonce":2758357551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/3424505910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":3424505910},{"type":"v1","addr":"172.21.15.78:6841","nonce":3424505910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/3086318443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3086318443},{"type":"v1","addr":"172.21.15.107:6833","nonce":3086318443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/2367018868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2367018868},{"type":"v1","addr":"172.21.15.106:6833","nonce":2367018868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:18:43.592215+0000","modified":"2024-03-24T00:19:08.871384+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24500,"mds_1":24481,"mds_2":24464},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/2231611477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":2231611477},{"type":"v1","addr":"172.21.15.106:6835","nonce":2231611477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6835/108936439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":108936439},{"type":"v1","addr":"172.21.15.107:6835","nonce":108936439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6837/2939571540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2939571540},{"type":"v1","addr":"172.21.15.107:6837","nonce":2939571540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24464":{"gid":24464,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.106:6837/2090968633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":2090968633},{"type":"v1","addr":"172.21.15.106:6837","nonce":2090968633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.107:6839/117119462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":117119462},{"type":"v1","addr":"172.21.15.107:6839","nonce":117119462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6839/2482071304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2482071304},{"type":"v1","addr":"172.21.15.106:6839","nonce":2482071304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24481,24500,24464]},"id":1}]} 2024-03-24T00:20:28.784 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 23 2024-03-24T00:20:28.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.786+0000 7f39f5036640 1 -- 172.21.15.78:0/1942514625 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f39d80a2890 msgr2=0x7f39d80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:28.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.786+0000 7f39f5036640 1 --2- 172.21.15.78:0/1942514625 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f39d80a2890 0x7f39d80a4d50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f39e0000c00 tx=0x7f39e0005fd0 comp rx=0 tx=0).stop 2024-03-24T00:20:28.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.786+0000 7f39f5036640 1 -- 172.21.15.78:0/1942514625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39f0102170 msgr2=0x7f39f019deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:28.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.786+0000 7f39f5036640 1 --2- 172.21.15.78:0/1942514625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39f0102170 0x7f39f019deb0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f39e800eb10 tx=0x7f39e800c2d0 comp rx=0 tx=0).stop 2024-03-24T00:20:28.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.786+0000 7f39f5036640 1 -- 172.21.15.78:0/1942514625 shutdown_connections 2024-03-24T00:20:28.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.786+0000 7f39f5036640 1 --2- 172.21.15.78:0/1942514625 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f39d80a2890 0x7f39d80a4d50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:28.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.786+0000 7f39f5036640 1 --2- 172.21.15.78:0/1942514625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39f0102170 0x7f39f019deb0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:28.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.786+0000 7f39f5036640 1 --2- 172.21.15.78:0/1942514625 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39f01017b0 0x7f39f0196e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:28.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.786+0000 7f39f5036640 1 --2- 172.21.15.78:0/1942514625 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39f01005c0 0x7f39f0196900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:28.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.786+0000 7f39f5036640 1 -- 172.21.15.78:0/1942514625 >> 172.21.15.78:0/1942514625 conn(0x7f39f00fbb30 msgr2=0x7f39f0104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:28.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.786+0000 7f39f5036640 1 -- 172.21.15.78:0/1942514625 shutdown_connections 2024-03-24T00:20:28.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.786+0000 7f39f5036640 1 -- 172.21.15.78:0/1942514625 wait complete. 2024-03-24T00:20:28.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:28 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3061922857' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:20:28.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:28 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2765561250' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:20:28.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:28 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2090458505' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:20:28.797 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:20:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:28 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3061922857' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:20:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:28 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2765561250' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:20:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:28 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2090458505' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:20:28.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.900+0000 7f7833340640 1 -- 172.21.15.78:0/1963409413 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f782c103280 msgr2=0x7f782c109b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:28.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.900+0000 7f7833340640 1 --2- 172.21.15.78:0/1963409413 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f782c103280 0x7f782c109b10 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7824009ca0 tx=0x7f782402c0f0 comp rx=0 tx=0).stop 2024-03-24T00:20:28.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.900+0000 7f7833340640 1 -- 172.21.15.78:0/1963409413 shutdown_connections 2024-03-24T00:20:28.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.900+0000 7f7833340640 1 --2- 172.21.15.78:0/1963409413 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f782c103280 0x7f782c109b10 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:28.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.900+0000 7f7833340640 1 --2- 172.21.15.78:0/1963409413 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f782c1028c0 0x7f782c102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:28.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.900+0000 7f7833340640 1 --2- 172.21.15.78:0/1963409413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f782c1016d0 0x7f782c101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:28.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.900+0000 7f7833340640 1 -- 172.21.15.78:0/1963409413 >> 172.21.15.78:0/1963409413 conn(0x7f782c0fcc80 msgr2=0x7f782c0ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:28.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.900+0000 7f7833340640 1 -- 172.21.15.78:0/1963409413 shutdown_connections 2024-03-24T00:20:28.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.900+0000 7f7833340640 1 -- 172.21.15.78:0/1963409413 wait complete. 2024-03-24T00:20:28.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.901+0000 7f7833340640 1 Processor -- start 2024-03-24T00:20:28.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.901+0000 7f7833340640 1 -- start start 2024-03-24T00:20:28.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.901+0000 7f7833340640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f782c1016d0 0x7f782c075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:28.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.901+0000 7f7833340640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f782c1028c0 0x7f782c073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:28.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.901+0000 7f7833340640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f782c103280 0x7f782c073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:28.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.901+0000 7f7833340640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782c10d330 con 0x7f782c1016d0 2024-03-24T00:20:28.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.901+0000 7f7833340640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782c10d1b0 con 0x7f782c103280 2024-03-24T00:20:28.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.901+0000 7f7833340640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782c10d4b0 con 0x7f782c1028c0 2024-03-24T00:20:28.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.901+0000 7f78310dc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f782c1016d0 0x7f782c075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:28.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.901+0000 7f78310dc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f782c1016d0 0x7f782c075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42042/0 (socket says 172.21.15.78:42042) 2024-03-24T00:20:28.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.901+0000 7f78310dc640 1 -- 172.21.15.78:0/2456051606 learned_addr learned my addr 172.21.15.78:0/2456051606 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:28.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.901+0000 7f78308db640 1 --2- 172.21.15.78:0/2456051606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f782c1028c0 0x7f782c073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:28.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.901+0000 7f78318dd640 1 --2- 172.21.15.78:0/2456051606 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f782c103280 0x7f782c073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:28.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.901+0000 7f78310dc640 1 -- 172.21.15.78:0/2456051606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f782c1028c0 msgr2=0x7f782c073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:28.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.901+0000 7f78310dc640 1 --2- 172.21.15.78:0/2456051606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f782c1028c0 0x7f782c073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:28.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.901+0000 7f78310dc640 1 -- 172.21.15.78:0/2456051606 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f782c103280 msgr2=0x7f782c073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:28.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.901+0000 7f78310dc640 1 --2- 172.21.15.78:0/2456051606 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f782c103280 0x7f782c073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:28.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.901+0000 7f78310dc640 1 -- 172.21.15.78:0/2456051606 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f782c074860 con 0x7f782c1016d0 2024-03-24T00:20:28.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.901+0000 7f78310dc640 1 --2- 172.21.15.78:0/2456051606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f782c1016d0 0x7f782c075450 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f782000e9c0 tx=0x7f782000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:28.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.901+0000 7f781e7fc640 1 -- 172.21.15.78:0/2456051606 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f782000cde0 con 0x7f782c1016d0 2024-03-24T00:20:28.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.902+0000 7f7833340640 1 -- 172.21.15.78:0/2456051606 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f782c1a1a90 con 0x7f782c1016d0 2024-03-24T00:20:28.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.902+0000 7f781e7fc640 1 -- 172.21.15.78:0/2456051606 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7820010910 con 0x7f782c1016d0 2024-03-24T00:20:28.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.902+0000 7f781e7fc640 1 -- 172.21.15.78:0/2456051606 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78200189c0 con 0x7f782c1016d0 2024-03-24T00:20:28.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.902+0000 7f7833340640 1 -- 172.21.15.78:0/2456051606 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f782c1a1fe0 con 0x7f782c1016d0 2024-03-24T00:20:28.905 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:28 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3061922857' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:20:28.905 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:28 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2765561250' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:20:28.905 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:28 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2090458505' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:20:28.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.903+0000 7f7833340640 1 -- 172.21.15.78:0/2456051606 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f782c0688e0 con 0x7f782c1016d0 2024-03-24T00:20:28.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.903+0000 7f781e7fc640 1 -- 172.21.15.78:0/2456051606 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f7820010ac0 con 0x7f782c1016d0 2024-03-24T00:20:28.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.904+0000 7f781e7fc640 1 --2- 172.21.15.78:0/2456051606 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f78180a2890 0x7f78180a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:28.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.904+0000 7f78308db640 1 --2- 172.21.15.78:0/2456051606 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f78180a2890 0x7f78180a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:28.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.904+0000 7f781e7fc640 1 -- 172.21.15.78:0/2456051606 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f78200cf490 con 0x7f782c1016d0 2024-03-24T00:20:28.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.904+0000 7f78308db640 1 --2- 172.21.15.78:0/2456051606 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f78180a2890 0x7f78180a4d50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f782c0751c0 tx=0x7f7828005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:28.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:28.909+0000 7f781e7fc640 1 -- 172.21.15.78:0/2456051606 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7820053740 con 0x7f782c1016d0 2024-03-24T00:20:29.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.086+0000 7f7833340640 1 -- 172.21.15.78:0/2456051606 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f782c02d070 con 0x7f782c1016d0 2024-03-24T00:20:29.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.088+0000 7f781e7fc640 1 -- 172.21.15.78:0/2456051606 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24601 (secure 0 0 0) 0x7f782009c070 con 0x7f782c1016d0 2024-03-24T00:20:29.088 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:20:29.088 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":91,"fsid":"c7e2806a-e972-11ee-95cd-87774f69a715","created":"2024-03-24T00:10:44.134563+0000","modified":"2024-03-24T00:19:41.438250+0000","last_up_change":"2024-03-24T00:17:50.237686+0000","last_in_change":"2024-03-24T00:17:29.788699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:14:04.534391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:18:42.178418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:18:42.800424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:18:46.112800+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b950388a-c437-426b-b703-3f8ae821dd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6803","nonce":1816879397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6805","nonce":1816879397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6809","nonce":1816879397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6807","nonce":1816879397}]},"public_addr":"172.21.15.78:6803/1816879397","cluster_addr":"172.21.15.78:6805/1816879397","heartbeat_back_addr":"172.21.15.78:6809/1816879397","heartbeat_front_addr":"172.21.15.78:6807/1816879397","state":["exists","up"]},{"osd":1,"uuid":"b8d64714-dd65-400f-b3f4-673ab14e1365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6801","nonce":1647915612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6803","nonce":1647915612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6807","nonce":1647915612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6805","nonce":1647915612}]},"public_addr":"172.21.15.106:6801/1647915612","cluster_addr":"172.21.15.106:6803/1647915612","heartbeat_back_addr":"172.21.15.106:6807/1647915612","heartbeat_front_addr":"172.21.15.106:6805/1647915612","state":["exists","up"]},{"osd":2,"uuid":"fcfc4dd1-aaf8-4131-a023-5e832212158d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6801","nonce":2010770435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6803","nonce":2010770435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6807","nonce":2010770435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6805","nonce":2010770435}]},"public_addr":"172.21.15.107:6801/2010770435","cluster_addr":"172.21.15.107:6803/2010770435","heartbeat_back_addr":"172.21.15.107:6807/2010770435","heartbeat_front_addr":"172.21.15.107:6805/2010770435","state":["exists","up"]},{"osd":3,"uuid":"47c718ab-a0f4-4e52-83c5-b89386b262d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6811","nonce":1912251346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6813","nonce":1912251346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6817","nonce":1912251346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6815","nonce":1912251346}]},"public_addr":"172.21.15.78:6811/1912251346","cluster_addr":"172.21.15.78:6813/1912251346","heartbeat_back_addr":"172.21.15.78:6817/1912251346","heartbeat_front_addr":"172.21.15.78:6815/1912251346","state":["exists","up"]},{"osd":4,"uuid":"7dad64dd-677e-4c1c-9638-e4f4f2195983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6809","nonce":374641205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6811","nonce":374641205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6815","nonce":374641205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6813","nonce":374641205}]},"public_addr":"172.21.15.106:6809/374641205","cluster_addr":"172.21.15.106:6811/374641205","heartbeat_back_addr":"172.21.15.106:6815/374641205","heartbeat_front_addr":"172.21.15.106:6813/374641205","state":["exists","up"]},{"osd":5,"uuid":"bbd524e5-328d-4309-8877-1149cbe110dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6809","nonce":3272806073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6811","nonce":3272806073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6815","nonce":3272806073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6813","nonce":3272806073}]},"public_addr":"172.21.15.107:6809/3272806073","cluster_addr":"172.21.15.107:6811/3272806073","heartbeat_back_addr":"172.21.15.107:6815/3272806073","heartbeat_front_addr":"172.21.15.107:6813/3272806073","state":["exists","up"]},{"osd":6,"uuid":"5696389a-918f-4ea8-b20d-4e2a0d4cb3bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6819","nonce":3091148508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6821","nonce":3091148508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6825","nonce":3091148508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6823","nonce":3091148508}]},"public_addr":"172.21.15.78:6819/3091148508","cluster_addr":"172.21.15.78:6821/3091148508","heartbeat_back_addr":"172.21.15.78:6825/3091148508","heartbeat_front_addr":"172.21.15.78:6823/3091148508","state":["exists","up"]},{"osd":7,"uuid":"f6b764e8-ed4e-4c9a-a85b-94af6dc6b924","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6817","nonce":4262103080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6819","nonce":4262103080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6823","nonce":4262103080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6821","nonce":4262103080}]},"public_addr":"172.21.15.106:6817/4262103080","cluster_addr":"172.21.15.106:6819/4262103080","heartbeat_back_addr":"172.21.15.106:6823/4262103080","heartbeat_front_addr":"172.21.15.106:6821/4262103080","state":["exists","up"]},{"osd":8,"uuid":"ba16039f-8093-4fa0-87ad-e28b7d142b23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6817","nonce":4032799356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6819","nonce":4032799356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6823","nonce":4032799356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6821","nonce":4032799356}]},"public_addr":"172.21.15.107:6817/4032799356","cluster_addr":"172.21.15.107:6819/4032799356","heartbeat_back_addr":"172.21.15.107:6823/4032799356","heartbeat_front_addr":"172.21.15.107:6821/4032799356","state":["exists","up"]},{"osd":9,"uuid":"651357f2-0669-42e6-aa91-1d5c02d00c5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6827","nonce":214023431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6829","nonce":214023431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6833","nonce":214023431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6831","nonce":214023431}]},"public_addr":"172.21.15.78:6827/214023431","cluster_addr":"172.21.15.78:6829/214023431","heartbeat_back_addr":"172.21.15.78:6833/214023431","heartbeat_front_addr":"172.21.15.78:6831/214023431","state":["exists","up"]},{"osd":10,"uuid":"2375119d-346e-426c-8a65-b64af71ad63a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6825","nonce":400029804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6827","nonce":400029804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6831","nonce":400029804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6829","nonce":400029804}]},"public_addr":"172.21.15.106:6825/400029804","cluster_addr":"172.21.15.106:6827/400029804","heartbeat_back_addr":"172.21.15.106:6831/400029804","heartbeat_front_addr":"172.21.15.106:6829/400029804","state":["exists","up"]},{"osd":11,"uuid":"b6b609d9-dd78-4c01-ae66-ba2a662ab22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6825","nonce":4191916230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6827","nonce":4191916230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6831","nonce":4191916230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6829","nonce":4191916230}]},"public_addr":"172.21.15.107:6825/4191916230","cluster_addr":"172.21.15.107:6827/4191916230","heartbeat_back_addr":"172.21.15.107:6831/4191916230","heartbeat_front_addr":"172.21.15.107:6829/4191916230","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:13.728602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:37.320809+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:01.269452+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:25.758281+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:50.298863+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:14.345219+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:38.980489+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:04.749214+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:29.992673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:56.269202+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:22.270316+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:48.074907+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/1982202541":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/1933054339":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6801/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6800/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/465691596":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6801/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/3962523609":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6800/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/1502752625":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/2236879401":"2024-03-25T00:11:40.199569+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:20:29.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.090+0000 7f7833340640 1 -- 172.21.15.78:0/2456051606 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f78180a2890 msgr2=0x7f78180a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:29.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.090+0000 7f7833340640 1 --2- 172.21.15.78:0/2456051606 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f78180a2890 0x7f78180a4d50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f782c0751c0 tx=0x7f7828005f90 comp rx=0 tx=0).stop 2024-03-24T00:20:29.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.090+0000 7f7833340640 1 -- 172.21.15.78:0/2456051606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f782c1016d0 msgr2=0x7f782c075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:29.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.090+0000 7f7833340640 1 --2- 172.21.15.78:0/2456051606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f782c1016d0 0x7f782c075450 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f782000e9c0 tx=0x7f782000ee80 comp rx=0 tx=0).stop 2024-03-24T00:20:29.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.090+0000 7f7833340640 1 -- 172.21.15.78:0/2456051606 shutdown_connections 2024-03-24T00:20:29.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.090+0000 7f7833340640 1 --2- 172.21.15.78:0/2456051606 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f78180a2890 0x7f78180a4d50 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:29.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.090+0000 7f7833340640 1 --2- 172.21.15.78:0/2456051606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f782c1028c0 0x7f782c073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:29.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.090+0000 7f7833340640 1 --2- 172.21.15.78:0/2456051606 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f782c103280 0x7f782c073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:29.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.090+0000 7f7833340640 1 --2- 172.21.15.78:0/2456051606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f782c1016d0 0x7f782c075450 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:29.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.090+0000 7f7833340640 1 -- 172.21.15.78:0/2456051606 >> 172.21.15.78:0/2456051606 conn(0x7f782c0fcc80 msgr2=0x7f782c105bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:29.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.090+0000 7f7833340640 1 -- 172.21.15.78:0/2456051606 shutdown_connections 2024-03-24T00:20:29.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.090+0000 7f7833340640 1 -- 172.21.15.78:0/2456051606 wait complete. 2024-03-24T00:20:29.102 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:14:04.534391+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:18:42.178418+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:18:42.800424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T00:18:46.112800+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T00:20:29.103 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:20:29.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.210+0000 7efecd948640 1 -- 172.21.15.78:0/2424537408 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efec8101c40 msgr2=0x7efec8102060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:29.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.210+0000 7efecd948640 1 --2- 172.21.15.78:0/2424537408 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efec8101c40 0x7efec8102060 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7efeb8009900 tx=0x7efeb802bc80 comp rx=0 tx=0).stop 2024-03-24T00:20:29.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.210+0000 7efecd948640 1 -- 172.21.15.78:0/2424537408 shutdown_connections 2024-03-24T00:20:29.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.210+0000 7efecd948640 1 --2- 172.21.15.78:0/2424537408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efec8100800 0x7efec8100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:29.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.210+0000 7efecd948640 1 --2- 172.21.15.78:0/2424537408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efec81011c0 0x7efec8108a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:29.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.210+0000 7efecd948640 1 --2- 172.21.15.78:0/2424537408 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efec8101c40 0x7efec8102060 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:29.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.210+0000 7efecd948640 1 -- 172.21.15.78:0/2424537408 >> 172.21.15.78:0/2424537408 conn(0x7efec80fb2b0 msgr2=0x7efec80fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:29.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.210+0000 7efecd948640 1 -- 172.21.15.78:0/2424537408 shutdown_connections 2024-03-24T00:20:29.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.210+0000 7efecd948640 1 -- 172.21.15.78:0/2424537408 wait complete. 2024-03-24T00:20:29.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.211+0000 7efecd948640 1 Processor -- start 2024-03-24T00:20:29.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.211+0000 7efecd948640 1 -- start start 2024-03-24T00:20:29.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.211+0000 7efecd948640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efec8100800 0x7efec8100260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:29.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.211+0000 7efecd948640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efec81011c0 0x7efec819c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:29.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.211+0000 7efecd948640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efec8101c40 0x7efec819ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:29.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.211+0000 7efecd948640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec810c260 con 0x7efec8101c40 2024-03-24T00:20:29.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.211+0000 7efecd948640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec810c0e0 con 0x7efec81011c0 2024-03-24T00:20:29.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.211+0000 7efecd948640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec810c3e0 con 0x7efec8100800 2024-03-24T00:20:29.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.211+0000 7efec77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efec8101c40 0x7efec819ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:29.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.211+0000 7efec77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efec8101c40 0x7efec819ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42054/0 (socket says 172.21.15.78:42054) 2024-03-24T00:20:29.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.211+0000 7efec77fe640 1 -- 172.21.15.78:0/2268845424 learned_addr learned my addr 172.21.15.78:0/2268845424 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:29.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.211+0000 7efebffff640 1 --2- 172.21.15.78:0/2268845424 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efec81011c0 0x7efec819c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:29.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.211+0000 7efec6ffd640 1 --2- 172.21.15.78:0/2268845424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efec8100800 0x7efec8100260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:29.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.211+0000 7efebffff640 1 -- 172.21.15.78:0/2268845424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efec8100800 msgr2=0x7efec8100260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:29.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.211+0000 7efebffff640 1 --2- 172.21.15.78:0/2268845424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efec8100800 0x7efec8100260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:29.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.211+0000 7efebffff640 1 -- 172.21.15.78:0/2268845424 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efec8101c40 msgr2=0x7efec819ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:29.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.211+0000 7efebffff640 1 --2- 172.21.15.78:0/2268845424 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efec8101c40 0x7efec819ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:29.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.211+0000 7efebffff640 1 -- 172.21.15.78:0/2268845424 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec819f680 con 0x7efec81011c0 2024-03-24T00:20:29.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.211+0000 7efec77fe640 1 --2- 172.21.15.78:0/2268845424 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efec8101c40 0x7efec819ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:29.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.211+0000 7efebffff640 1 --2- 172.21.15.78:0/2268845424 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efec81011c0 0x7efec819c750 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7efeb000e9c0 tx=0x7efeb000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:29.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.212+0000 7efecc946640 1 -- 172.21.15.78:0/2268845424 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb000ccc0 con 0x7efec81011c0 2024-03-24T00:20:29.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.212+0000 7efecc946640 1 -- 172.21.15.78:0/2268845424 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efeb00035f0 con 0x7efec81011c0 2024-03-24T00:20:29.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.212+0000 7efecd948640 1 -- 172.21.15.78:0/2268845424 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efec819f910 con 0x7efec81011c0 2024-03-24T00:20:29.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.212+0000 7efecc946640 1 -- 172.21.15.78:0/2268845424 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb000ce60 con 0x7efec81011c0 2024-03-24T00:20:29.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.212+0000 7efecd948640 1 -- 172.21.15.78:0/2268845424 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efec819fe10 con 0x7efec81011c0 2024-03-24T00:20:29.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.213+0000 7efecd948640 1 -- 172.21.15.78:0/2268845424 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efec8107940 con 0x7efec81011c0 2024-03-24T00:20:29.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.214+0000 7efecc946640 1 -- 172.21.15.78:0/2268845424 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7efeb00037a0 con 0x7efec81011c0 2024-03-24T00:20:29.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.214+0000 7efecc946640 1 --2- 172.21.15.78:0/2268845424 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7efeb40a2890 0x7efeb40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:29.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.214+0000 7efec6ffd640 1 --2- 172.21.15.78:0/2268845424 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7efeb40a2890 0x7efeb40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:29.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.214+0000 7efecc946640 1 -- 172.21.15.78:0/2268845424 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7efeb00cf430 con 0x7efec81011c0 2024-03-24T00:20:29.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.215+0000 7efec6ffd640 1 --2- 172.21.15.78:0/2268845424 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7efeb40a2890 0x7efeb40a4d50 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7efeb8006010 tx=0x7efeb8000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:29.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.218+0000 7efecc946640 1 -- 172.21.15.78:0/2268845424 <== mon.2 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efeb009c010 con 0x7efec81011c0 2024-03-24T00:20:29.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.399+0000 7efecd948640 1 -- 172.21.15.78:0/2268845424 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efec802d2c0 con 0x7efec81011c0 2024-03-24T00:20:29.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.400+0000 7efecc946640 1 -- 172.21.15.78:0/2268845424 <== mon.2 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7efeb009b910 con 0x7efec81011c0 2024-03-24T00:20:29.399 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-24T00:20:29.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.402+0000 7efecd948640 1 -- 172.21.15.78:0/2268845424 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7efeb40a2890 msgr2=0x7efeb40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:29.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.402+0000 7efecd948640 1 --2- 172.21.15.78:0/2268845424 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7efeb40a2890 0x7efeb40a4d50 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7efeb8006010 tx=0x7efeb8000bc0 comp rx=0 tx=0).stop 2024-03-24T00:20:29.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.402+0000 7efecd948640 1 -- 172.21.15.78:0/2268845424 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efec81011c0 msgr2=0x7efec819c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:29.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.402+0000 7efecd948640 1 --2- 172.21.15.78:0/2268845424 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efec81011c0 0x7efec819c750 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7efeb000e9c0 tx=0x7efeb000ee80 comp rx=0 tx=0).stop 2024-03-24T00:20:29.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.402+0000 7efecd948640 1 -- 172.21.15.78:0/2268845424 shutdown_connections 2024-03-24T00:20:29.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.402+0000 7efecd948640 1 --2- 172.21.15.78:0/2268845424 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7efeb40a2890 0x7efeb40a4d50 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:29.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.402+0000 7efecd948640 1 --2- 172.21.15.78:0/2268845424 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efec8101c40 0x7efec819ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:29.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.402+0000 7efecd948640 1 --2- 172.21.15.78:0/2268845424 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efec81011c0 0x7efec819c750 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:29.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.402+0000 7efecd948640 1 --2- 172.21.15.78:0/2268845424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efec8100800 0x7efec8100260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:29.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.402+0000 7efecd948640 1 -- 172.21.15.78:0/2268845424 >> 172.21.15.78:0/2268845424 conn(0x7efec80fb2b0 msgr2=0x7efec8104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:29.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.402+0000 7efecd948640 1 -- 172.21.15.78:0/2268845424 shutdown_connections 2024-03-24T00:20:29.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.402+0000 7efecd948640 1 -- 172.21.15.78:0/2268845424 wait complete. 2024-03-24T00:20:29.413 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:20:29.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.523+0000 7f72d0859640 1 -- 172.21.15.78:0/53619056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72cc1005e0 msgr2=0x7f72cc100a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:29.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.523+0000 7f72d0859640 1 --2- 172.21.15.78:0/53619056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72cc1005e0 0x7f72cc100a00 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f72bc009900 tx=0x7f72bc02bc30 comp rx=0 tx=0).stop 2024-03-24T00:20:29.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.523+0000 7f72d0859640 1 -- 172.21.15.78:0/53619056 shutdown_connections 2024-03-24T00:20:29.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.523+0000 7f72d0859640 1 --2- 172.21.15.78:0/53619056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72cc102190 0x7f72cc108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:29.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.523+0000 7f72d0859640 1 --2- 172.21.15.78:0/53619056 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72cc1017d0 0x7f72cc101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:29.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.523+0000 7f72d0859640 1 --2- 172.21.15.78:0/53619056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72cc1005e0 0x7f72cc100a00 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:29.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.523+0000 7f72d0859640 1 -- 172.21.15.78:0/53619056 >> 172.21.15.78:0/53619056 conn(0x7f72cc0fbb50 msgr2=0x7f72cc0fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:29.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.523+0000 7f72d0859640 1 -- 172.21.15.78:0/53619056 shutdown_connections 2024-03-24T00:20:29.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.523+0000 7f72d0859640 1 -- 172.21.15.78:0/53619056 wait complete. 2024-03-24T00:20:29.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.524+0000 7f72d0859640 1 Processor -- start 2024-03-24T00:20:29.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.524+0000 7f72d0859640 1 -- start start 2024-03-24T00:20:29.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.524+0000 7f72d0859640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72cc1005e0 0x7f72cc06a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:29.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.524+0000 7f72d0859640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72cc1017d0 0x7f72cc19c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:29.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.524+0000 7f72d0859640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72cc102190 0x7f72cc19ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:29.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.524+0000 7f72d0859640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72cc10c250 con 0x7f72cc102190 2024-03-24T00:20:29.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.524+0000 7f72d0859640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72cc10c0d0 con 0x7f72cc1017d0 2024-03-24T00:20:29.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.524+0000 7f72d0859640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72cc10c3d0 con 0x7f72cc1005e0 2024-03-24T00:20:29.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.524+0000 7f72cad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72cc102190 0x7f72cc19ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:29.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.524+0000 7f72cad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72cc102190 0x7f72cc19ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42070/0 (socket says 172.21.15.78:42070) 2024-03-24T00:20:29.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.524+0000 7f72cad9d640 1 -- 172.21.15.78:0/1183352827 learned_addr learned my addr 172.21.15.78:0/1183352827 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:29.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.524+0000 7f72ca59c640 1 --2- 172.21.15.78:0/1183352827 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72cc1005e0 0x7f72cc06a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:29.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.524+0000 7f72c3fff640 1 --2- 172.21.15.78:0/1183352827 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72cc1017d0 0x7f72cc19c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:29.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.525+0000 7f72cad9d640 1 -- 172.21.15.78:0/1183352827 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72cc1005e0 msgr2=0x7f72cc06a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:29.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.525+0000 7f72cad9d640 1 --2- 172.21.15.78:0/1183352827 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72cc1005e0 0x7f72cc06a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:29.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.525+0000 7f72cad9d640 1 -- 172.21.15.78:0/1183352827 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72cc1017d0 msgr2=0x7f72cc19c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:29.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.525+0000 7f72cad9d640 1 --2- 172.21.15.78:0/1183352827 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72cc1017d0 0x7f72cc19c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:29.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.525+0000 7f72cad9d640 1 -- 172.21.15.78:0/1183352827 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72cc19f6a0 con 0x7f72cc102190 2024-03-24T00:20:29.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.525+0000 7f72ca59c640 1 --2- 172.21.15.78:0/1183352827 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72cc1005e0 0x7f72cc06a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:29.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.525+0000 7f72cad9d640 1 --2- 172.21.15.78:0/1183352827 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72cc102190 0x7f72cc19ede0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f72c400eb10 tx=0x7f72c400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:29.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.525+0000 7f72c37fe640 1 -- 172.21.15.78:0/1183352827 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c4004b20 con 0x7f72cc102190 2024-03-24T00:20:29.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.525+0000 7f72c37fe640 1 -- 172.21.15.78:0/1183352827 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72c4004cc0 con 0x7f72cc102190 2024-03-24T00:20:29.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.525+0000 7f72d0859640 1 -- 172.21.15.78:0/1183352827 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72cc19f990 con 0x7f72cc102190 2024-03-24T00:20:29.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.525+0000 7f72c37fe640 1 -- 172.21.15.78:0/1183352827 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c4010760 con 0x7f72cc102190 2024-03-24T00:20:29.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.525+0000 7f72d0859640 1 -- 172.21.15.78:0/1183352827 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72cc19fee0 con 0x7f72cc102190 2024-03-24T00:20:29.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.526+0000 7f72d0859640 1 -- 172.21.15.78:0/1183352827 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72cc107a20 con 0x7f72cc102190 2024-03-24T00:20:29.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.527+0000 7f72c37fe640 1 -- 172.21.15.78:0/1183352827 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f72c4005000 con 0x7f72cc102190 2024-03-24T00:20:29.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.528+0000 7f72c37fe640 1 --2- 172.21.15.78:0/1183352827 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f72b80a2890 0x7f72b80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:29.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.528+0000 7f72ca59c640 1 --2- 172.21.15.78:0/1183352827 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f72b80a2890 0x7f72b80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:29.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.528+0000 7f72c37fe640 1 -- 172.21.15.78:0/1183352827 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f72c40cf960 con 0x7f72cc102190 2024-03-24T00:20:29.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.528+0000 7f72ca59c640 1 --2- 172.21.15.78:0/1183352827 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f72b80a2890 0x7f72b80a4d50 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f72bc005ab0 tx=0x7f72bc005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:29.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.531+0000 7f72c37fe640 1 -- 172.21.15.78:0/1183352827 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f72c409c540 con 0x7f72cc102190 2024-03-24T00:20:29.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.712+0000 7f72d0859640 1 -- 172.21.15.78:0/1183352827 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f72cc02d270 con 0x7f72cc102190 2024-03-24T00:20:29.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.712+0000 7f72c37fe640 1 -- 172.21.15.78:0/1183352827 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7f72c409be40 con 0x7f72cc102190 2024-03-24T00:20:29.712 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-24T00:20:29.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.714+0000 7f72d0859640 1 -- 172.21.15.78:0/1183352827 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f72b80a2890 msgr2=0x7f72b80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:29.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.714+0000 7f72d0859640 1 --2- 172.21.15.78:0/1183352827 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f72b80a2890 0x7f72b80a4d50 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f72bc005ab0 tx=0x7f72bc005fd0 comp rx=0 tx=0).stop 2024-03-24T00:20:29.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.714+0000 7f72d0859640 1 -- 172.21.15.78:0/1183352827 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72cc102190 msgr2=0x7f72cc19ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:29.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.714+0000 7f72d0859640 1 --2- 172.21.15.78:0/1183352827 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72cc102190 0x7f72cc19ede0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f72c400eb10 tx=0x7f72c400c2d0 comp rx=0 tx=0).stop 2024-03-24T00:20:29.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.714+0000 7f72d0859640 1 -- 172.21.15.78:0/1183352827 shutdown_connections 2024-03-24T00:20:29.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.714+0000 7f72d0859640 1 --2- 172.21.15.78:0/1183352827 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f72b80a2890 0x7f72b80a4d50 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:29.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.714+0000 7f72d0859640 1 --2- 172.21.15.78:0/1183352827 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72cc102190 0x7f72cc19ede0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:29.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.714+0000 7f72d0859640 1 --2- 172.21.15.78:0/1183352827 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72cc1017d0 0x7f72cc19c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:29.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.714+0000 7f72d0859640 1 --2- 172.21.15.78:0/1183352827 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72cc1005e0 0x7f72cc06a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:29.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.714+0000 7f72d0859640 1 -- 172.21.15.78:0/1183352827 >> 172.21.15.78:0/1183352827 conn(0x7f72cc0fbb50 msgr2=0x7f72cc104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:29.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.714+0000 7f72d0859640 1 -- 172.21.15.78:0/1183352827 shutdown_connections 2024-03-24T00:20:29.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.714+0000 7f72d0859640 1 -- 172.21.15.78:0/1183352827 wait complete. 2024-03-24T00:20:29.724 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:29 smithi078 ceph-mon[32429]: pgmap v345: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:20:29.725 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:29 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/806906795' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T00:20:29.725 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:29 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1942514625' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:20:29.725 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:29 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2456051606' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:20:29.725 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:29 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2268845424' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:20:29.726 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:20:29.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.833+0000 7fd2fcc30640 1 -- 172.21.15.78:0/262515381 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2f8102150 msgr2=0x7fd2f81089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:29.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.833+0000 7fd2fcc30640 1 --2- 172.21.15.78:0/262515381 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2f8102150 0x7fd2f81089e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8009900 tx=0x7fd2e802bc30 comp rx=0 tx=0).stop 2024-03-24T00:20:29.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.833+0000 7fd2fcc30640 1 -- 172.21.15.78:0/262515381 shutdown_connections 2024-03-24T00:20:29.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.833+0000 7fd2fcc30640 1 --2- 172.21.15.78:0/262515381 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2f8102150 0x7fd2f81089e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:29.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.833+0000 7fd2fcc30640 1 --2- 172.21.15.78:0/262515381 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2f8101790 0x7fd2f8101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:29.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.833+0000 7fd2fcc30640 1 --2- 172.21.15.78:0/262515381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2f81005a0 0x7fd2f81009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:29.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.833+0000 7fd2fcc30640 1 -- 172.21.15.78:0/262515381 >> 172.21.15.78:0/262515381 conn(0x7fd2f80fbb30 msgr2=0x7fd2f80fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:29.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.833+0000 7fd2fcc30640 1 -- 172.21.15.78:0/262515381 shutdown_connections 2024-03-24T00:20:29.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.833+0000 7fd2fcc30640 1 -- 172.21.15.78:0/262515381 wait complete. 2024-03-24T00:20:29.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.833+0000 7fd2fcc30640 1 Processor -- start 2024-03-24T00:20:29.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.833+0000 7fd2fcc30640 1 -- start start 2024-03-24T00:20:29.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.834+0000 7fd2fcc30640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2f81005a0 0x7fd2f81044c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:29.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.834+0000 7fd2fcc30640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2f8101790 0x7fd2f8102b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:29.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.834+0000 7fd2fcc30640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2f8102150 0x7fd2f81030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:29.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.834+0000 7fd2fcc30640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f810c230 con 0x7fd2f81005a0 2024-03-24T00:20:29.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.834+0000 7fd2fcc30640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f810c0b0 con 0x7fd2f8101790 2024-03-24T00:20:29.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.834+0000 7fd2fcc30640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f810c3b0 con 0x7fd2f8102150 2024-03-24T00:20:29.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.834+0000 7fd2f659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2f81005a0 0x7fd2f81044c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:29.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.834+0000 7fd2f659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2f81005a0 0x7fd2f81044c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42074/0 (socket says 172.21.15.78:42074) 2024-03-24T00:20:29.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.834+0000 7fd2f659c640 1 -- 172.21.15.78:0/3286608331 learned_addr learned my addr 172.21.15.78:0/3286608331 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:29.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.834+0000 7fd2f6d9d640 1 --2- 172.21.15.78:0/3286608331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2f8102150 0x7fd2f81030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:29.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.834+0000 7fd2f5d9b640 1 --2- 172.21.15.78:0/3286608331 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2f8101790 0x7fd2f8102b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:29.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.834+0000 7fd2f659c640 1 -- 172.21.15.78:0/3286608331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2f8102150 msgr2=0x7fd2f81030b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:29.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.834+0000 7fd2f659c640 1 --2- 172.21.15.78:0/3286608331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2f8102150 0x7fd2f81030b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:29.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.834+0000 7fd2f659c640 1 -- 172.21.15.78:0/3286608331 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2f8101790 msgr2=0x7fd2f8102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:29.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.834+0000 7fd2f659c640 1 --2- 172.21.15.78:0/3286608331 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2f8101790 0x7fd2f8102b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:29.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.834+0000 7fd2f659c640 1 -- 172.21.15.78:0/3286608331 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f81038d0 con 0x7fd2f81005a0 2024-03-24T00:20:29.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.834+0000 7fd2f659c640 1 --2- 172.21.15.78:0/3286608331 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2f81005a0 0x7fd2f81044c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd2f000eb70 tx=0x7fd2f000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:29.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.834+0000 7fd2e77fe640 1 -- 172.21.15.78:0/3286608331 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f0003b70 con 0x7fd2f81005a0 2024-03-24T00:20:29.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.834+0000 7fd2fcc30640 1 -- 172.21.15.78:0/3286608331 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2f81a1b20 con 0x7fd2f81005a0 2024-03-24T00:20:29.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.834+0000 7fd2e77fe640 1 -- 172.21.15.78:0/3286608331 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd2f000ee70 con 0x7fd2f81005a0 2024-03-24T00:20:29.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.834+0000 7fd2e77fe640 1 -- 172.21.15.78:0/3286608331 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f0010830 con 0x7fd2f81005a0 2024-03-24T00:20:29.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.835+0000 7fd2fcc30640 1 -- 172.21.15.78:0/3286608331 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2f81a2070 con 0x7fd2f81005a0 2024-03-24T00:20:29.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.836+0000 7fd2fcc30640 1 -- 172.21.15.78:0/3286608331 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2f81a2240 con 0x7fd2f81005a0 2024-03-24T00:20:29.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.837+0000 7fd2e77fe640 1 -- 172.21.15.78:0/3286608331 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fd2f0004020 con 0x7fd2f81005a0 2024-03-24T00:20:29.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.837+0000 7fd2e77fe640 1 --2- 172.21.15.78:0/3286608331 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd2e00a2890 0x7fd2e00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:29.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.837+0000 7fd2f5d9b640 1 --2- 172.21.15.78:0/3286608331 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd2e00a2890 0x7fd2e00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:29.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.837+0000 7fd2e77fe640 1 -- 172.21.15.78:0/3286608331 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fd2f00cf5c0 con 0x7fd2f81005a0 2024-03-24T00:20:29.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.838+0000 7fd2f5d9b640 1 --2- 172.21.15.78:0/3286608331 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd2e00a2890 0x7fd2e00a4d50 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8104230 tx=0x7fd2ec005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:29.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:29.842+0000 7fd2e77fe640 1 -- 172.21.15.78:0/3286608331 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd2f009c1a0 con 0x7fd2f81005a0 2024-03-24T00:20:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:29 smithi107 ceph-mon[36485]: pgmap v345: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:20:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:29 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/806906795' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T00:20:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:29 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1942514625' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:20:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:29 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2456051606' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:20:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:29 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2268845424' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:20:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:29 smithi106 ceph-mon[36505]: pgmap v345: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:20:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:29 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/806906795' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T00:20:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:29 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1942514625' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:20:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:29 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2456051606' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:20:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:29 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2268845424' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:20:30.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.023+0000 7fd2fcc30640 1 -- 172.21.15.78:0/3286608331 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd2f81a2530 con 0x7fd2f81005a0 2024-03-24T00:20:30.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.023+0000 7fd2e77fe640 1 -- 172.21.15.78:0/3286608331 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd2f009baa0 con 0x7fd2f81005a0 2024-03-24T00:20:30.023 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-03-24T00:20:30.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.025+0000 7fd2fcc30640 1 -- 172.21.15.78:0/3286608331 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd2e00a2890 msgr2=0x7fd2e00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:30.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.025+0000 7fd2fcc30640 1 --2- 172.21.15.78:0/3286608331 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd2e00a2890 0x7fd2e00a4d50 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8104230 tx=0x7fd2ec005f90 comp rx=0 tx=0).stop 2024-03-24T00:20:30.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.025+0000 7fd2fcc30640 1 -- 172.21.15.78:0/3286608331 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2f81005a0 msgr2=0x7fd2f81044c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:30.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.025+0000 7fd2fcc30640 1 --2- 172.21.15.78:0/3286608331 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2f81005a0 0x7fd2f81044c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd2f000eb70 tx=0x7fd2f000c6a0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.025+0000 7fd2fcc30640 1 -- 172.21.15.78:0/3286608331 shutdown_connections 2024-03-24T00:20:30.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.025+0000 7fd2fcc30640 1 --2- 172.21.15.78:0/3286608331 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd2e00a2890 0x7fd2e00a4d50 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.025+0000 7fd2fcc30640 1 --2- 172.21.15.78:0/3286608331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2f8102150 0x7fd2f81030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.025+0000 7fd2fcc30640 1 --2- 172.21.15.78:0/3286608331 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2f8101790 0x7fd2f8102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.025+0000 7fd2fcc30640 1 --2- 172.21.15.78:0/3286608331 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2f81005a0 0x7fd2f81044c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.025+0000 7fd2fcc30640 1 -- 172.21.15.78:0/3286608331 >> 172.21.15.78:0/3286608331 conn(0x7fd2f80fbb30 msgr2=0x7fd2f80fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:30.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.025+0000 7fd2fcc30640 1 -- 172.21.15.78:0/3286608331 shutdown_connections 2024-03-24T00:20:30.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.025+0000 7fd2fcc30640 1 -- 172.21.15.78:0/3286608331 wait complete. 2024-03-24T00:20:30.036 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T00:20:30.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.141+0000 7f793e0a1640 1 -- 172.21.15.78:0/1544077976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7938104a50 msgr2=0x7f7938106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:30.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.141+0000 7f793e0a1640 1 --2- 172.21.15.78:0/1544077976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7938104a50 0x7f7938106e40 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7920009900 tx=0x7f792002bc30 comp rx=0 tx=0).stop 2024-03-24T00:20:30.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.141+0000 7f793e0a1640 1 -- 172.21.15.78:0/1544077976 shutdown_connections 2024-03-24T00:20:30.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.141+0000 7f793e0a1640 1 --2- 172.21.15.78:0/1544077976 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7938107380 0x7f79381097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.141+0000 7f793e0a1640 1 --2- 172.21.15.78:0/1544077976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7938104a50 0x7f7938106e40 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.141+0000 7f793e0a1640 1 --2- 172.21.15.78:0/1544077976 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7938102120 0x7f7938104510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.141+0000 7f793e0a1640 1 -- 172.21.15.78:0/1544077976 >> 172.21.15.78:0/1544077976 conn(0x7f79380fbb30 msgr2=0x7f79380fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:30.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.141+0000 7f793e0a1640 1 -- 172.21.15.78:0/1544077976 shutdown_connections 2024-03-24T00:20:30.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.141+0000 7f793e0a1640 1 -- 172.21.15.78:0/1544077976 wait complete. 2024-03-24T00:20:30.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.142+0000 7f793e0a1640 1 Processor -- start 2024-03-24T00:20:30.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.142+0000 7f793e0a1640 1 -- start start 2024-03-24T00:20:30.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.142+0000 7f793e0a1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7938102120 0x7f7938197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:30.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.142+0000 7f793e0a1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7938104a50 0x7f7938197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:30.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.142+0000 7f793e0a1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7938107380 0x7f793819efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:30.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.142+0000 7f793e0a1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f793810d310 con 0x7f7938102120 2024-03-24T00:20:30.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.142+0000 7f793e0a1640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f793810d190 con 0x7f7938104a50 2024-03-24T00:20:30.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.142+0000 7f793e0a1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f793810d490 con 0x7f7938107380 2024-03-24T00:20:30.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.142+0000 7f79377fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7938102120 0x7f7938197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:30.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.142+0000 7f79377fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7938102120 0x7f7938197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42088/0 (socket says 172.21.15.78:42088) 2024-03-24T00:20:30.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.142+0000 7f79377fe640 1 -- 172.21.15.78:0/4281398553 learned_addr learned my addr 172.21.15.78:0/4281398553 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:30.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.142+0000 7f792effd640 1 --2- 172.21.15.78:0/4281398553 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7938104a50 0x7f7938197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:30.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.142+0000 7f7937fff640 1 --2- 172.21.15.78:0/4281398553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7938107380 0x7f793819efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:30.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.143+0000 7f79377fe640 1 -- 172.21.15.78:0/4281398553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7938107380 msgr2=0x7f793819efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:30.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.143+0000 7f79377fe640 1 --2- 172.21.15.78:0/4281398553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7938107380 0x7f793819efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.143+0000 7f79377fe640 1 -- 172.21.15.78:0/4281398553 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7938104a50 msgr2=0x7f7938197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:30.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.143+0000 7f79377fe640 1 --2- 172.21.15.78:0/4281398553 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7938104a50 0x7f7938197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.143+0000 7f79377fe640 1 -- 172.21.15.78:0/4281398553 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f793819f630 con 0x7f7938102120 2024-03-24T00:20:30.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.143+0000 7f792effd640 1 --2- 172.21.15.78:0/4281398553 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7938104a50 0x7f7938197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:30.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.143+0000 7f79377fe640 1 --2- 172.21.15.78:0/4281398553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7938102120 0x7f7938197a10 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f792800e9f0 tx=0x7f792800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:30.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.143+0000 7f79357fa640 1 -- 172.21.15.78:0/4281398553 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f792800cd80 con 0x7f7938102120 2024-03-24T00:20:30.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.143+0000 7f79357fa640 1 -- 172.21.15.78:0/4281398553 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7928010910 con 0x7f7938102120 2024-03-24T00:20:30.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.143+0000 7f793e0a1640 1 -- 172.21.15.78:0/4281398553 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f793819f920 con 0x7f7938102120 2024-03-24T00:20:30.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.143+0000 7f79357fa640 1 -- 172.21.15.78:0/4281398553 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7928018990 con 0x7f7938102120 2024-03-24T00:20:30.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.143+0000 7f793e0a1640 1 -- 172.21.15.78:0/4281398553 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f793819fe40 con 0x7f7938102120 2024-03-24T00:20:30.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.144+0000 7f793e0a1640 1 -- 172.21.15.78:0/4281398553 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79380fd620 con 0x7f7938102120 2024-03-24T00:20:30.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.146+0000 7f79357fa640 1 -- 172.21.15.78:0/4281398553 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f7928010ac0 con 0x7f7938102120 2024-03-24T00:20:30.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.146+0000 7f79357fa640 1 --2- 172.21.15.78:0/4281398553 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f79240a2890 0x7f79240a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:30.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.147+0000 7f792effd640 1 --2- 172.21.15.78:0/4281398553 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f79240a2890 0x7f79240a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:30.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.147+0000 7f79357fa640 1 -- 172.21.15.78:0/4281398553 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f79280cf670 con 0x7f7938102120 2024-03-24T00:20:30.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.147+0000 7f792effd640 1 --2- 172.21.15.78:0/4281398553 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f79240a2890 0x7f79240a4d50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7920000c00 tx=0x7f7920005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:30.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.150+0000 7f79357fa640 1 -- 172.21.15.78:0/4281398553 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f792809bc00 con 0x7f7938102120 2024-03-24T00:20:30.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.331+0000 7f793e0a1640 1 -- 172.21.15.78:0/4281398553 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f79380622a0 con 0x7f7938102120 2024-03-24T00:20:30.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.332+0000 7f79357fa640 1 -- 172.21.15.78:0/4281398553 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7f792809fca0 con 0x7f7938102120 2024-03-24T00:20:30.332 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-24T00:20:30.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.334+0000 7f793e0a1640 1 -- 172.21.15.78:0/4281398553 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f79240a2890 msgr2=0x7f79240a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:30.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.334+0000 7f793e0a1640 1 --2- 172.21.15.78:0/4281398553 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f79240a2890 0x7f79240a4d50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7920000c00 tx=0x7f7920005fd0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.334+0000 7f793e0a1640 1 -- 172.21.15.78:0/4281398553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7938102120 msgr2=0x7f7938197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:30.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.334+0000 7f793e0a1640 1 --2- 172.21.15.78:0/4281398553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7938102120 0x7f7938197a10 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f792800e9f0 tx=0x7f792800eeb0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.334+0000 7f793e0a1640 1 -- 172.21.15.78:0/4281398553 shutdown_connections 2024-03-24T00:20:30.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.334+0000 7f793e0a1640 1 --2- 172.21.15.78:0/4281398553 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f79240a2890 0x7f79240a4d50 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.334+0000 7f793e0a1640 1 --2- 172.21.15.78:0/4281398553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7938107380 0x7f793819efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.334+0000 7f793e0a1640 1 --2- 172.21.15.78:0/4281398553 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7938104a50 0x7f7938197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.334+0000 7f793e0a1640 1 --2- 172.21.15.78:0/4281398553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7938102120 0x7f7938197a10 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.334+0000 7f793e0a1640 1 -- 172.21.15.78:0/4281398553 >> 172.21.15.78:0/4281398553 conn(0x7f79380fbb30 msgr2=0x7f7938102b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:30.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.334+0000 7f793e0a1640 1 -- 172.21.15.78:0/4281398553 shutdown_connections 2024-03-24T00:20:30.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.334+0000 7f793e0a1640 1 -- 172.21.15.78:0/4281398553 wait complete. 2024-03-24T00:20:30.345 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:20:30.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.448+0000 7f9107944640 1 -- 172.21.15.78:0/4108313129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9100103310 msgr2=0x7f9100109ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:30.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.448+0000 7f9107944640 1 --2- 172.21.15.78:0/4108313129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9100103310 0x7f9100109ba0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f90f8009cd0 tx=0x7f90f802c0f0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.448+0000 7f9107944640 1 -- 172.21.15.78:0/4108313129 shutdown_connections 2024-03-24T00:20:30.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.448+0000 7f9107944640 1 --2- 172.21.15.78:0/4108313129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9100102950 0x7f9100102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.448+0000 7f9107944640 1 --2- 172.21.15.78:0/4108313129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9100103310 0x7f9100109ba0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.448+0000 7f9107944640 1 --2- 172.21.15.78:0/4108313129 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f91000ff980 0x7f91000ffda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.448+0000 7f9107944640 1 -- 172.21.15.78:0/4108313129 >> 172.21.15.78:0/4108313129 conn(0x7f91000fb2d0 msgr2=0x7f91000fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:30.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.448+0000 7f9107944640 1 -- 172.21.15.78:0/4108313129 shutdown_connections 2024-03-24T00:20:30.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.448+0000 7f9107944640 1 -- 172.21.15.78:0/4108313129 wait complete. 2024-03-24T00:20:30.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.449+0000 7f9107944640 1 Processor -- start 2024-03-24T00:20:30.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.449+0000 7f9107944640 1 -- start start 2024-03-24T00:20:30.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.449+0000 7f9107944640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f91000ff980 0x7f91001979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:30.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.449+0000 7f9107944640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9100102950 0x7f9100197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:30.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.449+0000 7f9107944640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9100103310 0x7f910019ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:30.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.449+0000 7f9107944640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f910010d360 con 0x7f9100102950 2024-03-24T00:20:30.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.449+0000 7f9107944640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f910010d1e0 con 0x7f91000ff980 2024-03-24T00:20:30.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.449+0000 7f9107944640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f910010d4e0 con 0x7f9100103310 2024-03-24T00:20:30.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.449+0000 7f9104edf640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9100102950 0x7f9100197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:30.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.449+0000 7f91056e0640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f91000ff980 0x7f91001979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:30.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.449+0000 7f9104edf640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9100102950 0x7f9100197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42100/0 (socket says 172.21.15.78:42100) 2024-03-24T00:20:30.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.449+0000 7f9104edf640 1 -- 172.21.15.78:0/2682881687 learned_addr learned my addr 172.21.15.78:0/2682881687 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:30.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.449+0000 7f9105ee1640 1 --2- 172.21.15.78:0/2682881687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9100103310 0x7f910019ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:30.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.450+0000 7f9104edf640 1 -- 172.21.15.78:0/2682881687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9100103310 msgr2=0x7f910019ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:30.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.450+0000 7f9104edf640 1 --2- 172.21.15.78:0/2682881687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9100103310 0x7f910019ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.450+0000 7f9104edf640 1 -- 172.21.15.78:0/2682881687 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f91000ff980 msgr2=0x7f91001979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:30.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.450+0000 7f9104edf640 1 --2- 172.21.15.78:0/2682881687 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f91000ff980 0x7f91001979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.450+0000 7f9104edf640 1 -- 172.21.15.78:0/2682881687 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f910019f5d0 con 0x7f9100102950 2024-03-24T00:20:30.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.450+0000 7f91056e0640 1 --2- 172.21.15.78:0/2682881687 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f91000ff980 0x7f91001979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:30.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.450+0000 7f9104edf640 1 --2- 172.21.15.78:0/2682881687 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9100102950 0x7f9100197ef0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f90f000e9f0 tx=0x7f90f000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:30.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.450+0000 7f90f67fc640 1 -- 172.21.15.78:0/2682881687 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f000cd80 con 0x7f9100102950 2024-03-24T00:20:30.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.450+0000 7f9107944640 1 -- 172.21.15.78:0/2682881687 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f910019f8c0 con 0x7f9100102950 2024-03-24T00:20:30.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.450+0000 7f90f67fc640 1 -- 172.21.15.78:0/2682881687 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f90f0005500 con 0x7f9100102950 2024-03-24T00:20:30.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.450+0000 7f90f67fc640 1 -- 172.21.15.78:0/2682881687 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f0010690 con 0x7f9100102950 2024-03-24T00:20:30.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.450+0000 7f9107944640 1 -- 172.21.15.78:0/2682881687 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f910019fe10 con 0x7f9100102950 2024-03-24T00:20:30.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.451+0000 7f9107944640 1 -- 172.21.15.78:0/2682881687 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9100108ad0 con 0x7f9100102950 2024-03-24T00:20:30.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.452+0000 7f90f67fc640 1 -- 172.21.15.78:0/2682881687 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f90f00046c0 con 0x7f9100102950 2024-03-24T00:20:30.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.452+0000 7f90f67fc640 1 --2- 172.21.15.78:0/2682881687 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f90ec0a2890 0x7f90ec0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:30.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.452+0000 7f91056e0640 1 --2- 172.21.15.78:0/2682881687 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f90ec0a2890 0x7f90ec0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:30.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.453+0000 7f90f67fc640 1 -- 172.21.15.78:0/2682881687 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f90f00cf340 con 0x7f9100102950 2024-03-24T00:20:30.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.453+0000 7f91056e0640 1 --2- 172.21.15.78:0/2682881687 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f90ec0a2890 0x7f90ec0a4d50 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9100069b70 tx=0x7f90fc005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:30.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.457+0000 7f90f67fc640 1 -- 172.21.15.78:0/2682881687 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f90f009bf20 con 0x7f9100102950 2024-03-24T00:20:30.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.669+0000 7f9107944640 1 -- 172.21.15.78:0/2682881687 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f910002d200 con 0x7f9100102950 2024-03-24T00:20:30.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.669+0000 7f90f67fc640 1 -- 172.21.15.78:0/2682881687 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11406 (secure 0 0 0) 0x7f90f009b820 con 0x7f9100102950 2024-03-24T00:20:30.669 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:20:30.669 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1416685123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1416685123},{"type":"v1","addr":"172.21.15.78:6835","nonce":1416685123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1071274623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1071274623},{"type":"v1","addr":"172.21.15.78:6837","nonce":1071274623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2758357551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2758357551},{"type":"v1","addr":"172.21.15.78:6839","nonce":2758357551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/3424505910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":3424505910},{"type":"v1","addr":"172.21.15.78:6841","nonce":3424505910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/3086318443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3086318443},{"type":"v1","addr":"172.21.15.107:6833","nonce":3086318443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/2367018868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2367018868},{"type":"v1","addr":"172.21.15.106:6833","nonce":2367018868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:18:43.592215+0000","modified":"2024-03-24T00:19:08.871384+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24500,"mds_1":24481,"mds_2":24464},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/2231611477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":2231611477},{"type":"v1","addr":"172.21.15.106:6835","nonce":2231611477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6835/108936439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":108936439},{"type":"v1","addr":"172.21.15.107:6835","nonce":108936439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6837/2939571540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2939571540},{"type":"v1","addr":"172.21.15.107:6837","nonce":2939571540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24464":{"gid":24464,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.106:6837/2090968633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":2090968633},{"type":"v1","addr":"172.21.15.106:6837","nonce":2090968633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.107:6839/117119462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":117119462},{"type":"v1","addr":"172.21.15.107:6839","nonce":117119462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6839/2482071304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2482071304},{"type":"v1","addr":"172.21.15.106:6839","nonce":2482071304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24481,24500,24464]},"id":1}]} 2024-03-24T00:20:30.670 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 23 2024-03-24T00:20:30.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.671+0000 7f9107944640 1 -- 172.21.15.78:0/2682881687 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f90ec0a2890 msgr2=0x7f90ec0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:30.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.671+0000 7f9107944640 1 --2- 172.21.15.78:0/2682881687 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f90ec0a2890 0x7f90ec0a4d50 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9100069b70 tx=0x7f90fc005f90 comp rx=0 tx=0).stop 2024-03-24T00:20:30.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.671+0000 7f9107944640 1 -- 172.21.15.78:0/2682881687 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9100102950 msgr2=0x7f9100197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:30.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.671+0000 7f9107944640 1 --2- 172.21.15.78:0/2682881687 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9100102950 0x7f9100197ef0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f90f000e9f0 tx=0x7f90f000edf0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.672+0000 7f9107944640 1 -- 172.21.15.78:0/2682881687 shutdown_connections 2024-03-24T00:20:30.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.672+0000 7f9107944640 1 --2- 172.21.15.78:0/2682881687 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f90ec0a2890 0x7f90ec0a4d50 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.672+0000 7f9107944640 1 --2- 172.21.15.78:0/2682881687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9100103310 0x7f910019ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.672+0000 7f9107944640 1 --2- 172.21.15.78:0/2682881687 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9100102950 0x7f9100197ef0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.672+0000 7f9107944640 1 --2- 172.21.15.78:0/2682881687 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f91000ff980 0x7f91001979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.672+0000 7f9107944640 1 -- 172.21.15.78:0/2682881687 >> 172.21.15.78:0/2682881687 conn(0x7f91000fb2d0 msgr2=0x7f9100105ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:30.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.672+0000 7f9107944640 1 -- 172.21.15.78:0/2682881687 shutdown_connections 2024-03-24T00:20:30.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.672+0000 7f9107944640 1 -- 172.21.15.78:0/2682881687 wait complete. 2024-03-24T00:20:30.681 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:30 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1183352827' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:20:30.682 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:30 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3286608331' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:20:30.682 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:30 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/4281398553' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T00:20:30.683 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:20:30.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.785+0000 7f12b59e6640 1 -- 172.21.15.78:0/504431181 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12b0102170 msgr2=0x7f12b0108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:30.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.785+0000 7f12b59e6640 1 --2- 172.21.15.78:0/504431181 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12b0102170 0x7f12b0108a00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f12a8009900 tx=0x7f12a802bc30 comp rx=0 tx=0).stop 2024-03-24T00:20:30.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.785+0000 7f12b59e6640 1 -- 172.21.15.78:0/504431181 shutdown_connections 2024-03-24T00:20:30.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.785+0000 7f12b59e6640 1 --2- 172.21.15.78:0/504431181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12b01017b0 0x7f12b0101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.785+0000 7f12b59e6640 1 --2- 172.21.15.78:0/504431181 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12b0102170 0x7f12b0108a00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.785+0000 7f12b59e6640 1 --2- 172.21.15.78:0/504431181 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12b01005c0 0x7f12b01009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.785+0000 7f12b59e6640 1 -- 172.21.15.78:0/504431181 >> 172.21.15.78:0/504431181 conn(0x7f12b00fbb50 msgr2=0x7f12b00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:30.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.785+0000 7f12b59e6640 1 -- 172.21.15.78:0/504431181 shutdown_connections 2024-03-24T00:20:30.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.785+0000 7f12b59e6640 1 -- 172.21.15.78:0/504431181 wait complete. 2024-03-24T00:20:30.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.785+0000 7f12b59e6640 1 Processor -- start 2024-03-24T00:20:30.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.786+0000 7f12b59e6640 1 -- start start 2024-03-24T00:20:30.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.786+0000 7f12b59e6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12b01005c0 0x7f12b0197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:30.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.786+0000 7f12b59e6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12b01017b0 0x7f12b0197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:30.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.786+0000 7f12b59e6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12b0102170 0x7f12b019eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:30.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.786+0000 7f12b59e6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b010c220 con 0x7f12b0102170 2024-03-24T00:20:30.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.786+0000 7f12b59e6640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b010c0a0 con 0x7f12b01005c0 2024-03-24T00:20:30.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.786+0000 7f12b59e6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b010c3a0 con 0x7f12b01017b0 2024-03-24T00:20:30.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.786+0000 7f12af7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12b0102170 0x7f12b019eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:30.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.786+0000 7f12aeffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12b01005c0 0x7f12b0197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:30.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.786+0000 7f12af7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12b0102170 0x7f12b019eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42118/0 (socket says 172.21.15.78:42118) 2024-03-24T00:20:30.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.786+0000 7f12af7fe640 1 -- 172.21.15.78:0/4232739175 learned_addr learned my addr 172.21.15.78:0/4232739175 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:30.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.786+0000 7f12ae7fc640 1 --2- 172.21.15.78:0/4232739175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12b01017b0 0x7f12b0197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:30.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.786+0000 7f12af7fe640 1 -- 172.21.15.78:0/4232739175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12b01017b0 msgr2=0x7f12b0197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:30.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.786+0000 7f12af7fe640 1 --2- 172.21.15.78:0/4232739175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12b01017b0 0x7f12b0197e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.786+0000 7f12af7fe640 1 -- 172.21.15.78:0/4232739175 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12b01005c0 msgr2=0x7f12b0197940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:30.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.786+0000 7f12af7fe640 1 --2- 172.21.15.78:0/4232739175 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12b01005c0 0x7f12b0197940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.786+0000 7f12af7fe640 1 -- 172.21.15.78:0/4232739175 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b019f560 con 0x7f12b0102170 2024-03-24T00:20:30.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.786+0000 7f12af7fe640 1 --2- 172.21.15.78:0/4232739175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12b0102170 0x7f12b019eef0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f12a8005ab0 tx=0x7f12a802c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:30.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.786+0000 7f12b49e4640 1 -- 172.21.15.78:0/4232739175 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a8041070 con 0x7f12b0102170 2024-03-24T00:20:30.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.786+0000 7f12b59e6640 1 -- 172.21.15.78:0/4232739175 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b019f7f0 con 0x7f12b0102170 2024-03-24T00:20:30.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.786+0000 7f12b49e4640 1 -- 172.21.15.78:0/4232739175 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12a8004450 con 0x7f12b0102170 2024-03-24T00:20:30.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.787+0000 7f12b49e4640 1 -- 172.21.15.78:0/4232739175 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a8003e60 con 0x7f12b0102170 2024-03-24T00:20:30.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.787+0000 7f12b59e6640 1 -- 172.21.15.78:0/4232739175 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b019fd30 con 0x7f12b0102170 2024-03-24T00:20:30.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.788+0000 7f12b59e6640 1 -- 172.21.15.78:0/4232739175 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b019ff90 con 0x7f12b0102170 2024-03-24T00:20:30.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.789+0000 7f12b49e4640 1 -- 172.21.15.78:0/4232739175 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f12a8044070 con 0x7f12b0102170 2024-03-24T00:20:30.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.789+0000 7f12b49e4640 1 --2- 172.21.15.78:0/4232739175 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f12980a2890 0x7f12980a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:30.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.789+0000 7f12aeffd640 1 --2- 172.21.15.78:0/4232739175 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f12980a2890 0x7f12980a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:30.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.790+0000 7f12b49e4640 1 -- 172.21.15.78:0/4232739175 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f12a80f0a60 con 0x7f12b0102170 2024-03-24T00:20:30.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.790+0000 7f12aeffd640 1 --2- 172.21.15.78:0/4232739175 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f12980a2890 0x7f12980a4d50 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f12a00099a0 tx=0x7f12a0009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:30.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.794+0000 7f12b49e4640 1 -- 172.21.15.78:0/4232739175 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f12a80bd100 con 0x7f12b0102170 2024-03-24T00:20:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:30 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1183352827' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:20:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:30 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3286608331' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:20:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:30 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/4281398553' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T00:20:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:30 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1183352827' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:20:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:30 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3286608331' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:20:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:30 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/4281398553' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T00:20:30.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.972+0000 7f12b59e6640 1 -- 172.21.15.78:0/4232739175 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12b01a0160 con 0x7f12b0102170 2024-03-24T00:20:30.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.973+0000 7f12b49e4640 1 -- 172.21.15.78:0/4232739175 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24601 (secure 0 0 0) 0x7f12a80c11a0 con 0x7f12b0102170 2024-03-24T00:20:30.973 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:20:30.973 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":91,"fsid":"c7e2806a-e972-11ee-95cd-87774f69a715","created":"2024-03-24T00:10:44.134563+0000","modified":"2024-03-24T00:19:41.438250+0000","last_up_change":"2024-03-24T00:17:50.237686+0000","last_in_change":"2024-03-24T00:17:29.788699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:14:04.534391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:18:42.178418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:18:42.800424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:18:46.112800+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b950388a-c437-426b-b703-3f8ae821dd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6803","nonce":1816879397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6805","nonce":1816879397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6809","nonce":1816879397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6807","nonce":1816879397}]},"public_addr":"172.21.15.78:6803/1816879397","cluster_addr":"172.21.15.78:6805/1816879397","heartbeat_back_addr":"172.21.15.78:6809/1816879397","heartbeat_front_addr":"172.21.15.78:6807/1816879397","state":["exists","up"]},{"osd":1,"uuid":"b8d64714-dd65-400f-b3f4-673ab14e1365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6801","nonce":1647915612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6803","nonce":1647915612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6807","nonce":1647915612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6805","nonce":1647915612}]},"public_addr":"172.21.15.106:6801/1647915612","cluster_addr":"172.21.15.106:6803/1647915612","heartbeat_back_addr":"172.21.15.106:6807/1647915612","heartbeat_front_addr":"172.21.15.106:6805/1647915612","state":["exists","up"]},{"osd":2,"uuid":"fcfc4dd1-aaf8-4131-a023-5e832212158d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6801","nonce":2010770435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6803","nonce":2010770435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6807","nonce":2010770435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6805","nonce":2010770435}]},"public_addr":"172.21.15.107:6801/2010770435","cluster_addr":"172.21.15.107:6803/2010770435","heartbeat_back_addr":"172.21.15.107:6807/2010770435","heartbeat_front_addr":"172.21.15.107:6805/2010770435","state":["exists","up"]},{"osd":3,"uuid":"47c718ab-a0f4-4e52-83c5-b89386b262d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6811","nonce":1912251346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6813","nonce":1912251346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6817","nonce":1912251346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6815","nonce":1912251346}]},"public_addr":"172.21.15.78:6811/1912251346","cluster_addr":"172.21.15.78:6813/1912251346","heartbeat_back_addr":"172.21.15.78:6817/1912251346","heartbeat_front_addr":"172.21.15.78:6815/1912251346","state":["exists","up"]},{"osd":4,"uuid":"7dad64dd-677e-4c1c-9638-e4f4f2195983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6809","nonce":374641205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6811","nonce":374641205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6815","nonce":374641205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6813","nonce":374641205}]},"public_addr":"172.21.15.106:6809/374641205","cluster_addr":"172.21.15.106:6811/374641205","heartbeat_back_addr":"172.21.15.106:6815/374641205","heartbeat_front_addr":"172.21.15.106:6813/374641205","state":["exists","up"]},{"osd":5,"uuid":"bbd524e5-328d-4309-8877-1149cbe110dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6809","nonce":3272806073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6811","nonce":3272806073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6815","nonce":3272806073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6813","nonce":3272806073}]},"public_addr":"172.21.15.107:6809/3272806073","cluster_addr":"172.21.15.107:6811/3272806073","heartbeat_back_addr":"172.21.15.107:6815/3272806073","heartbeat_front_addr":"172.21.15.107:6813/3272806073","state":["exists","up"]},{"osd":6,"uuid":"5696389a-918f-4ea8-b20d-4e2a0d4cb3bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6819","nonce":3091148508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6821","nonce":3091148508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6825","nonce":3091148508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6823","nonce":3091148508}]},"public_addr":"172.21.15.78:6819/3091148508","cluster_addr":"172.21.15.78:6821/3091148508","heartbeat_back_addr":"172.21.15.78:6825/3091148508","heartbeat_front_addr":"172.21.15.78:6823/3091148508","state":["exists","up"]},{"osd":7,"uuid":"f6b764e8-ed4e-4c9a-a85b-94af6dc6b924","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6817","nonce":4262103080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6819","nonce":4262103080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6823","nonce":4262103080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6821","nonce":4262103080}]},"public_addr":"172.21.15.106:6817/4262103080","cluster_addr":"172.21.15.106:6819/4262103080","heartbeat_back_addr":"172.21.15.106:6823/4262103080","heartbeat_front_addr":"172.21.15.106:6821/4262103080","state":["exists","up"]},{"osd":8,"uuid":"ba16039f-8093-4fa0-87ad-e28b7d142b23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6817","nonce":4032799356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6819","nonce":4032799356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6823","nonce":4032799356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6821","nonce":4032799356}]},"public_addr":"172.21.15.107:6817/4032799356","cluster_addr":"172.21.15.107:6819/4032799356","heartbeat_back_addr":"172.21.15.107:6823/4032799356","heartbeat_front_addr":"172.21.15.107:6821/4032799356","state":["exists","up"]},{"osd":9,"uuid":"651357f2-0669-42e6-aa91-1d5c02d00c5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6827","nonce":214023431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6829","nonce":214023431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6833","nonce":214023431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6831","nonce":214023431}]},"public_addr":"172.21.15.78:6827/214023431","cluster_addr":"172.21.15.78:6829/214023431","heartbeat_back_addr":"172.21.15.78:6833/214023431","heartbeat_front_addr":"172.21.15.78:6831/214023431","state":["exists","up"]},{"osd":10,"uuid":"2375119d-346e-426c-8a65-b64af71ad63a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6825","nonce":400029804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6827","nonce":400029804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6831","nonce":400029804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6829","nonce":400029804}]},"public_addr":"172.21.15.106:6825/400029804","cluster_addr":"172.21.15.106:6827/400029804","heartbeat_back_addr":"172.21.15.106:6831/400029804","heartbeat_front_addr":"172.21.15.106:6829/400029804","state":["exists","up"]},{"osd":11,"uuid":"b6b609d9-dd78-4c01-ae66-ba2a662ab22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6825","nonce":4191916230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6827","nonce":4191916230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6831","nonce":4191916230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6829","nonce":4191916230}]},"public_addr":"172.21.15.107:6825/4191916230","cluster_addr":"172.21.15.107:6827/4191916230","heartbeat_back_addr":"172.21.15.107:6831/4191916230","heartbeat_front_addr":"172.21.15.107:6829/4191916230","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:13.728602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:37.320809+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:01.269452+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:25.758281+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:50.298863+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:14.345219+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:38.980489+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:04.749214+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:29.992673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:56.269202+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:22.270316+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:48.074907+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/1982202541":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/1933054339":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6801/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6800/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/465691596":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6801/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/3962523609":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6800/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/1502752625":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/2236879401":"2024-03-25T00:11:40.199569+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:20:30.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.975+0000 7f12b59e6640 1 -- 172.21.15.78:0/4232739175 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f12980a2890 msgr2=0x7f12980a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:30.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.975+0000 7f12b59e6640 1 --2- 172.21.15.78:0/4232739175 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f12980a2890 0x7f12980a4d50 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f12a00099a0 tx=0x7f12a0009500 comp rx=0 tx=0).stop 2024-03-24T00:20:30.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.975+0000 7f12b59e6640 1 -- 172.21.15.78:0/4232739175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12b0102170 msgr2=0x7f12b019eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:30.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.975+0000 7f12b59e6640 1 --2- 172.21.15.78:0/4232739175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12b0102170 0x7f12b019eef0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f12a8005ab0 tx=0x7f12a802c6c0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.975+0000 7f12b59e6640 1 -- 172.21.15.78:0/4232739175 shutdown_connections 2024-03-24T00:20:30.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.975+0000 7f12b59e6640 1 --2- 172.21.15.78:0/4232739175 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f12980a2890 0x7f12980a4d50 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.975+0000 7f12b59e6640 1 --2- 172.21.15.78:0/4232739175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12b01017b0 0x7f12b0197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.975+0000 7f12b59e6640 1 --2- 172.21.15.78:0/4232739175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12b0102170 0x7f12b019eef0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.975+0000 7f12b59e6640 1 --2- 172.21.15.78:0/4232739175 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12b01005c0 0x7f12b0197940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:30.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.975+0000 7f12b59e6640 1 -- 172.21.15.78:0/4232739175 >> 172.21.15.78:0/4232739175 conn(0x7f12b00fbb50 msgr2=0x7f12b0069c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:30.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.975+0000 7f12b59e6640 1 -- 172.21.15.78:0/4232739175 shutdown_connections 2024-03-24T00:20:30.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:30.975+0000 7f12b59e6640 1 -- 172.21.15.78:0/4232739175 wait complete. 2024-03-24T00:20:30.987 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:20:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.087+0000 7f535acff640 1 -- 172.21.15.78:0/3817340247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53541073a0 msgr2=0x7f53541097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.087+0000 7f535acff640 1 --2- 172.21.15.78:0/3817340247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53541073a0 0x7f53541097f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f534c009ca0 tx=0x7f534c02c0f0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.087+0000 7f535acff640 1 -- 172.21.15.78:0/3817340247 shutdown_connections 2024-03-24T00:20:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.087+0000 7f535acff640 1 --2- 172.21.15.78:0/3817340247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5354104a70 0x7f5354106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.087+0000 7f535acff640 1 --2- 172.21.15.78:0/3817340247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53541073a0 0x7f53541097f0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.087+0000 7f535acff640 1 --2- 172.21.15.78:0/3817340247 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5354102140 0x7f5354104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.087+0000 7f535acff640 1 -- 172.21.15.78:0/3817340247 >> 172.21.15.78:0/3817340247 conn(0x7f53540fbb30 msgr2=0x7f53540fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.088+0000 7f535acff640 1 -- 172.21.15.78:0/3817340247 shutdown_connections 2024-03-24T00:20:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.088+0000 7f535acff640 1 -- 172.21.15.78:0/3817340247 wait complete. 2024-03-24T00:20:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.088+0000 7f535acff640 1 Processor -- start 2024-03-24T00:20:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.088+0000 7f535acff640 1 -- start start 2024-03-24T00:20:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.088+0000 7f535acff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5354102140 0x7f53541979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.088+0000 7f535acff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5354104a70 0x7f5354197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.088+0000 7f535acff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53541073a0 0x7f535419ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.088+0000 7f535acff640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f535410d380 con 0x7f5354102140 2024-03-24T00:20:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.088+0000 7f535acff640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f535410d200 con 0x7f53541073a0 2024-03-24T00:20:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.088+0000 7f535acff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f535410d500 con 0x7f5354104a70 2024-03-24T00:20:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.088+0000 7f5358a9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5354102140 0x7f53541979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.088+0000 7f5358a9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5354102140 0x7f53541979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42128/0 (socket says 172.21.15.78:42128) 2024-03-24T00:20:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.088+0000 7f5358a9b640 1 -- 172.21.15.78:0/778969408 learned_addr learned my addr 172.21.15.78:0/778969408 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.088+0000 7f5353fff640 1 --2- 172.21.15.78:0/778969408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5354104a70 0x7f5354197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.088+0000 7f535929c640 1 --2- 172.21.15.78:0/778969408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53541073a0 0x7f535419ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.089+0000 7f5358a9b640 1 -- 172.21.15.78:0/778969408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5354104a70 msgr2=0x7f5354197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.089+0000 7f5358a9b640 1 --2- 172.21.15.78:0/778969408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5354104a70 0x7f5354197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.089+0000 7f5358a9b640 1 -- 172.21.15.78:0/778969408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53541073a0 msgr2=0x7f535419ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.089+0000 7f5358a9b640 1 --2- 172.21.15.78:0/778969408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53541073a0 0x7f535419ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.089+0000 7f5358a9b640 1 -- 172.21.15.78:0/778969408 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f535419f5c0 con 0x7f5354102140 2024-03-24T00:20:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.089+0000 7f5353fff640 1 --2- 172.21.15.78:0/778969408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5354104a70 0x7f5354197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.089+0000 7f5358a9b640 1 --2- 172.21.15.78:0/778969408 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5354102140 0x7f53541979a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f534400e9f0 tx=0x7f534400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.089+0000 7f5351ffb640 1 -- 172.21.15.78:0/778969408 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534400cd80 con 0x7f5354102140 2024-03-24T00:20:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.089+0000 7f5351ffb640 1 -- 172.21.15.78:0/778969408 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5344005500 con 0x7f5354102140 2024-03-24T00:20:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.089+0000 7f535acff640 1 -- 172.21.15.78:0/778969408 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f535419f8b0 con 0x7f5354102140 2024-03-24T00:20:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.089+0000 7f5351ffb640 1 -- 172.21.15.78:0/778969408 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5344010690 con 0x7f5354102140 2024-03-24T00:20:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.089+0000 7f535acff640 1 -- 172.21.15.78:0/778969408 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f535419fdb0 con 0x7f5354102140 2024-03-24T00:20:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.090+0000 7f535acff640 1 -- 172.21.15.78:0/778969408 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53540fd620 con 0x7f5354102140 2024-03-24T00:20:31.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.091+0000 7f5351ffb640 1 -- 172.21.15.78:0/778969408 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f53440046c0 con 0x7f5354102140 2024-03-24T00:20:31.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.091+0000 7f5351ffb640 1 --2- 172.21.15.78:0/778969408 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f533c0a2890 0x7f533c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.092+0000 7f5353fff640 1 --2- 172.21.15.78:0/778969408 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f533c0a2890 0x7f533c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.092+0000 7f5351ffb640 1 -- 172.21.15.78:0/778969408 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f53440cf340 con 0x7f5354102140 2024-03-24T00:20:31.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.092+0000 7f5353fff640 1 --2- 172.21.15.78:0/778969408 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f533c0a2890 0x7f533c0a4d50 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5354198eb0 tx=0x7f5348005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:31.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.096+0000 7f5351ffb640 1 -- 172.21.15.78:0/778969408 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f534409bf20 con 0x7f5354102140 2024-03-24T00:20:31.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.310+0000 7f535acff640 1 -- 172.21.15.78:0/778969408 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f535402d240 con 0x7f5354102140 2024-03-24T00:20:31.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.311+0000 7f5351ffb640 1 -- 172.21.15.78:0/778969408 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11406 (secure 0 0 0) 0x7f534409b820 con 0x7f5354102140 2024-03-24T00:20:31.311 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:20:31.311 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1416685123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1416685123},{"type":"v1","addr":"172.21.15.78:6835","nonce":1416685123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1071274623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1071274623},{"type":"v1","addr":"172.21.15.78:6837","nonce":1071274623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2758357551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2758357551},{"type":"v1","addr":"172.21.15.78:6839","nonce":2758357551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/3424505910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":3424505910},{"type":"v1","addr":"172.21.15.78:6841","nonce":3424505910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/3086318443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3086318443},{"type":"v1","addr":"172.21.15.107:6833","nonce":3086318443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/2367018868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2367018868},{"type":"v1","addr":"172.21.15.106:6833","nonce":2367018868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:18:43.592215+0000","modified":"2024-03-24T00:19:08.871384+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24500,"mds_1":24481,"mds_2":24464},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/2231611477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":2231611477},{"type":"v1","addr":"172.21.15.106:6835","nonce":2231611477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6835/108936439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":108936439},{"type":"v1","addr":"172.21.15.107:6835","nonce":108936439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6837/2939571540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2939571540},{"type":"v1","addr":"172.21.15.107:6837","nonce":2939571540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24464":{"gid":24464,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.106:6837/2090968633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":2090968633},{"type":"v1","addr":"172.21.15.106:6837","nonce":2090968633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.107:6839/117119462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":117119462},{"type":"v1","addr":"172.21.15.107:6839","nonce":117119462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6839/2482071304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2482071304},{"type":"v1","addr":"172.21.15.106:6839","nonce":2482071304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24481,24500,24464]},"id":1}]} 2024-03-24T00:20:31.311 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 23 2024-03-24T00:20:31.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.313+0000 7f535acff640 1 -- 172.21.15.78:0/778969408 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f533c0a2890 msgr2=0x7f533c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:31.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.313+0000 7f535acff640 1 --2- 172.21.15.78:0/778969408 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f533c0a2890 0x7f533c0a4d50 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5354198eb0 tx=0x7f5348005f90 comp rx=0 tx=0).stop 2024-03-24T00:20:31.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.313+0000 7f535acff640 1 -- 172.21.15.78:0/778969408 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5354102140 msgr2=0x7f53541979a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:31.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.313+0000 7f535acff640 1 --2- 172.21.15.78:0/778969408 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5354102140 0x7f53541979a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f534400e9f0 tx=0x7f534400edf0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.313+0000 7f535acff640 1 -- 172.21.15.78:0/778969408 shutdown_connections 2024-03-24T00:20:31.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.313+0000 7f535acff640 1 --2- 172.21.15.78:0/778969408 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f533c0a2890 0x7f533c0a4d50 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.313+0000 7f535acff640 1 --2- 172.21.15.78:0/778969408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5354104a70 0x7f5354197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.313+0000 7f535acff640 1 --2- 172.21.15.78:0/778969408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53541073a0 0x7f535419ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.313+0000 7f535acff640 1 --2- 172.21.15.78:0/778969408 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5354102140 0x7f53541979a0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.313+0000 7f535acff640 1 -- 172.21.15.78:0/778969408 >> 172.21.15.78:0/778969408 conn(0x7f53540fbb30 msgr2=0x7f5354102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:31.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.313+0000 7f535acff640 1 -- 172.21.15.78:0/778969408 shutdown_connections 2024-03-24T00:20:31.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.313+0000 7f535acff640 1 -- 172.21.15.78:0/778969408 wait complete. 2024-03-24T00:20:31.324 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-24T00:18:43.592215+0000', 'modified': '2024-03-24T00:19:08.871384+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24500, 'mds_1': 24481, 'mds_2': 24464}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24437': {'gid': 24437, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.106:6835/2231611477', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6834', 'nonce': 2231611477}, {'type': 'v1', 'addr': '172.21.15.106:6835', 'nonce': 2231611477}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24442': {'gid': 24442, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.107:6835/108936439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6834', 'nonce': 108936439}, {'type': 'v1', 'addr': '172.21.15.107:6835', 'nonce': 108936439}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24454': {'gid': 24454, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.107:6837/2939571540', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6836', 'nonce': 2939571540}, {'type': 'v1', 'addr': '172.21.15.107:6837', 'nonce': 2939571540}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24464': {'gid': 24464, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.106:6837/2090968633', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6836', 'nonce': 2090968633}, {'type': 'v1', 'addr': '172.21.15.106:6837', 'nonce': 2090968633}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24481': {'gid': 24481, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.107:6839/117119462', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6838', 'nonce': 117119462}, {'type': 'v1', 'addr': '172.21.15.107:6839', 'nonce': 117119462}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24500': {'gid': 24500, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.106:6839/2482071304', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6838', 'nonce': 2482071304}, {'type': 'v1', 'addr': '172.21.15.106:6839', 'nonce': 2482071304}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24500, 'qdb_cluster': [24481, 24500, 24464]} 2024-03-24T00:20:31.324 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-24T00:20:31.325 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-24T00:20:31.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.428+0000 7f9f901d4640 1 -- 172.21.15.78:0/2215730974 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f88101790 msgr2=0x7f9f88101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:31.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.428+0000 7f9f901d4640 1 --2- 172.21.15.78:0/2215730974 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f88101790 0x7f9f88101c10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9f78009860 tx=0x7f9f7802bb70 comp rx=0 tx=0).stop 2024-03-24T00:20:31.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.428+0000 7f9f901d4640 1 -- 172.21.15.78:0/2215730974 shutdown_connections 2024-03-24T00:20:31.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.428+0000 7f9f901d4640 1 --2- 172.21.15.78:0/2215730974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f88102150 0x7f9f881089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.428+0000 7f9f901d4640 1 --2- 172.21.15.78:0/2215730974 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f88101790 0x7f9f88101c10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.428+0000 7f9f901d4640 1 --2- 172.21.15.78:0/2215730974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f881005a0 0x7f9f881009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.428+0000 7f9f901d4640 1 -- 172.21.15.78:0/2215730974 >> 172.21.15.78:0/2215730974 conn(0x7f9f880fbb30 msgr2=0x7f9f880fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:31.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.428+0000 7f9f901d4640 1 -- 172.21.15.78:0/2215730974 shutdown_connections 2024-03-24T00:20:31.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.428+0000 7f9f901d4640 1 -- 172.21.15.78:0/2215730974 wait complete. 2024-03-24T00:20:31.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.429+0000 7f9f901d4640 1 Processor -- start 2024-03-24T00:20:31.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.429+0000 7f9f901d4640 1 -- start start 2024-03-24T00:20:31.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.429+0000 7f9f901d4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f881005a0 0x7f9f88069370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.429+0000 7f9f901d4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f88101790 0x7f9f8819c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.429+0000 7f9f901d4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f88102150 0x7f9f8819ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.429+0000 7f9f901d4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8810c230 con 0x7f9f88102150 2024-03-24T00:20:31.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.429+0000 7f9f901d4640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8810c0b0 con 0x7f9f88101790 2024-03-24T00:20:31.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.429+0000 7f9f901d4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8810c3b0 con 0x7f9f881005a0 2024-03-24T00:20:31.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.429+0000 7f9f8e771640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f88102150 0x7f9f8819ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.429+0000 7f9f8e771640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f88102150 0x7f9f8819ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42138/0 (socket says 172.21.15.78:42138) 2024-03-24T00:20:31.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.429+0000 7f9f8df70640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f881005a0 0x7f9f88069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.429+0000 7f9f8e771640 1 -- 172.21.15.78:0/2614663427 learned_addr learned my addr 172.21.15.78:0/2614663427 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:31.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.429+0000 7f9f8d76f640 1 --2- 172.21.15.78:0/2614663427 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f88101790 0x7f9f8819c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.429+0000 7f9f8e771640 1 -- 172.21.15.78:0/2614663427 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f881005a0 msgr2=0x7f9f88069370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:31.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.429+0000 7f9f8e771640 1 --2- 172.21.15.78:0/2614663427 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f881005a0 0x7f9f88069370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.429+0000 7f9f8e771640 1 -- 172.21.15.78:0/2614663427 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f88101790 msgr2=0x7f9f8819c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:31.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.429+0000 7f9f8e771640 1 --2- 172.21.15.78:0/2614663427 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f88101790 0x7f9f8819c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.429+0000 7f9f8e771640 1 -- 172.21.15.78:0/2614663427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f8819f670 con 0x7f9f88102150 2024-03-24T00:20:31.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.429+0000 7f9f8df70640 1 --2- 172.21.15.78:0/2614663427 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f881005a0 0x7f9f88069370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:31.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.429+0000 7f9f8d76f640 1 --2- 172.21.15.78:0/2614663427 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f88101790 0x7f9f8819c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:31.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.429+0000 7f9f8e771640 1 --2- 172.21.15.78:0/2614663427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f88102150 0x7f9f8819ed60 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9f8000eb10 tx=0x7f9f8000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:31.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.429+0000 7f9f7effd640 1 -- 172.21.15.78:0/2614663427 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f80004b20 con 0x7f9f88102150 2024-03-24T00:20:31.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.430+0000 7f9f901d4640 1 -- 172.21.15.78:0/2614663427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f8819f900 con 0x7f9f88102150 2024-03-24T00:20:31.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.430+0000 7f9f7effd640 1 -- 172.21.15.78:0/2614663427 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f80004cc0 con 0x7f9f88102150 2024-03-24T00:20:31.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.430+0000 7f9f7effd640 1 -- 172.21.15.78:0/2614663427 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f80010760 con 0x7f9f88102150 2024-03-24T00:20:31.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.430+0000 7f9f901d4640 1 -- 172.21.15.78:0/2614663427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f8819fe70 con 0x7f9f88102150 2024-03-24T00:20:31.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.432+0000 7f9f7effd640 1 -- 172.21.15.78:0/2614663427 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f9f80005020 con 0x7f9f88102150 2024-03-24T00:20:31.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.432+0000 7f9f92dc1740 1 Processor -- start 2024-03-24T00:20:31.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.432+0000 7f9f92dc1740 1 -- start start 2024-03-24T00:20:31.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.432+0000 7f9f92dc1740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f75b8cc740 0x55f75b8ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.432+0000 7f9f92dc1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f75b8cd130 0x55f75b8d1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.432+0000 7f9f92dc1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f75b8d22a0 0x55f75b8d4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.432+0000 7f9f92dc1740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f75b831c20 con 0x55f75b8cc740 2024-03-24T00:20:31.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.432+0000 7f9f92dc1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f75b7daa40 con 0x55f75b8d22a0 2024-03-24T00:20:31.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.432+0000 7f9f92dc1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f75b82dcc0 con 0x55f75b8cd130 2024-03-24T00:20:31.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.432+0000 7f9f8d76f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f75b8cc740 0x55f75b8ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.432+0000 7f9f8d76f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f75b8cc740 0x55f75b8ccb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42152/0 (socket says 172.21.15.78:42152) 2024-03-24T00:20:31.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.432+0000 7f9f8d76f640 1 -- 172.21.15.78:0/3895346115 learned_addr learned my addr 172.21.15.78:0/3895346115 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:31.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.432+0000 7f9f8df70640 1 --2- 172.21.15.78:0/3895346115 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f75b8cd130 0x55f75b8d1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.432+0000 7f9f8d76f640 1 --2- 172.21.15.78:0/3895346115 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f75b8d22a0 0x55f75b8d4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.432+0000 7f9f7effd640 1 --2- 172.21.15.78:0/2614663427 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f9f740a2890 0x7f9f740a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.433+0000 7f9f8d76f640 1 -- 172.21.15.78:0/3895346115 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f75b8cd130 msgr2=0x55f75b8d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:31.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.433+0000 7f9f8d76f640 1 --2- 172.21.15.78:0/3895346115 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f75b8cd130 0x55f75b8d1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.433+0000 7f9f8d76f640 1 -- 172.21.15.78:0/3895346115 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f75b8d22a0 msgr2=0x55f75b8d4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:31.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.433+0000 7f9f8d76f640 1 --2- 172.21.15.78:0/3895346115 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f75b8d22a0 0x55f75b8d4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.433+0000 7f9f8d76f640 1 -- 172.21.15.78:0/3895346115 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f75b7fcf90 con 0x55f75b8cc740 2024-03-24T00:20:31.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.433+0000 7f9f8e771640 1 --2- 172.21.15.78:0/2614663427 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f9f740a2890 0x7f9f740a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.433+0000 7f9f7effd640 1 -- 172.21.15.78:0/2614663427 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f9f800cf630 con 0x7f9f88102150 2024-03-24T00:20:31.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.433+0000 7f9f8d76f640 1 --2- 172.21.15.78:0/3895346115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f75b8cc740 0x55f75b8ccb60 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f9f78037040 tx=0x7f9f78034ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=413818f68efea440 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:31.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.433+0000 7f9f73fff640 1 -- 172.21.15.78:0/3895346115 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f78039040 con 0x55f75b8cc740 2024-03-24T00:20:31.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.433+0000 7f9f73fff640 1 -- 172.21.15.78:0/3895346115 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f7802c510 con 0x55f75b8cc740 2024-03-24T00:20:31.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.433+0000 7f9f73fff640 1 -- 172.21.15.78:0/3895346115 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f78032860 con 0x55f75b8cc740 2024-03-24T00:20:31.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.434+0000 7f9f92dc1740 1 -- 172.21.15.78:0/3895346115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f75b8cc740 msgr2=0x55f75b8ccb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:31.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.434+0000 7f9f8e771640 1 --2- 172.21.15.78:0/2614663427 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f9f740a2890 0x7f9f740a4d50 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9f800a3d80 tx=0x7f9f8000efd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:31.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.434+0000 7f9f92dc1740 1 --2- 172.21.15.78:0/3895346115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f75b8cc740 0x55f75b8ccb60 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f9f78037040 tx=0x7f9f78034ad0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.434+0000 7f9f92dc1740 1 -- 172.21.15.78:0/3895346115 shutdown_connections 2024-03-24T00:20:31.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.434+0000 7f9f92dc1740 1 --2- 172.21.15.78:0/3895346115 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f75b8cd130 0x55f75b8d1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.434+0000 7f9f92dc1740 1 --2- 172.21.15.78:0/3895346115 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f75b8d22a0 0x55f75b8d4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.434+0000 7f9f92dc1740 1 --2- 172.21.15.78:0/3895346115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f75b8cc740 0x55f75b8ccb60 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.434+0000 7f9f92dc1740 1 -- 172.21.15.78:0/3895346115 >> 172.21.15.78:0/3895346115 conn(0x55f75b894a60 msgr2=0x55f75b8c80b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:31.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.434+0000 7f9f92dc1740 1 -- 172.21.15.78:0/3895346115 shutdown_connections 2024-03-24T00:20:31.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.434+0000 7f9f92dc1740 1 -- 172.21.15.78:0/3895346115 wait complete. 2024-03-24T00:20:31.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.434+0000 7f9f92dc1740 1 -- start start 2024-03-24T00:20:31.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.434+0000 7f9f92dc1740 1 Processor -- start 2024-03-24T00:20:31.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.434+0000 7f9f92dc1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f75b8cc740 0x55f75b8d6c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.434+0000 7f9f92dc1740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f75b8cd130 0x55f75b8d71c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.434+0000 7f9f92dc1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f75b8d22a0 0x55f75b8de200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.435+0000 7f9f92dc1740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f75b8d51f0 con 0x55f75b8cd130 2024-03-24T00:20:31.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.435+0000 7f9f92dc1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f75b7ede90 con 0x55f75b8d22a0 2024-03-24T00:20:31.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.435+0000 7f9f92dc1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f75b8d5370 con 0x55f75b8cc740 2024-03-24T00:20:31.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.435+0000 7f9f92dc1740 10 client.0 _ll_register_callbacks cb 0x55f75b545ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:20:31.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.435+0000 7f9f8df70640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f75b8cd130 0x55f75b8d71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.435+0000 7f9f8d76f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f75b8cc740 0x55f75b8d6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.435+0000 7f9f8df70640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f75b8cd130 0x55f75b8d71c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42162/0 (socket says 172.21.15.78:42162) 2024-03-24T00:20:31.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.435+0000 7f9f8df70640 1 -- 172.21.15.78:0/469186723 learned_addr learned my addr 172.21.15.78:0/469186723 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:31.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.435+0000 7f9f8d76f640 1 --2- 172.21.15.78:0/469186723 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f75b8d22a0 0x55f75b8de200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.435+0000 7f9f8df70640 1 -- 172.21.15.78:0/469186723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f75b8cc740 msgr2=0x55f75b8d6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:31.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.435+0000 7f9f8df70640 1 --2- 172.21.15.78:0/469186723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f75b8cc740 0x55f75b8d6c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.435+0000 7f9f8df70640 1 -- 172.21.15.78:0/469186723 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f75b8d22a0 msgr2=0x55f75b8de200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:31.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.435+0000 7f9f8df70640 1 --2- 172.21.15.78:0/469186723 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f75b8d22a0 0x55f75b8de200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.435+0000 7f9f8df70640 1 -- 172.21.15.78:0/469186723 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f75b7fa9d0 con 0x55f75b8cd130 2024-03-24T00:20:31.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.435+0000 7f9f8df70640 1 --2- 172.21.15.78:0/469186723 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f75b8cd130 0x55f75b8d71c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f9f84005fd0 tx=0x7f9f8400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a84757f3a6a3c0b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:31.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.435+0000 7f9f717fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-24T00:20:31.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.435+0000 7f9f717fa640 1 -- 172.21.15.78:0/469186723 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f840120d0 con 0x55f75b8cd130 2024-03-24T00:20:31.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.435+0000 7f9f717fa640 1 -- 172.21.15.78:0/469186723 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f9f84010430 con 0x55f75b8cd130 2024-03-24T00:20:31.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.435+0000 7f9f92dc1740 1 -- 172.21.15.78:0/469186723 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9f7802c510 con 0x55f75b8cd130 2024-03-24T00:20:31.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.435+0000 7f9f717fa640 1 -- 172.21.15.78:0/469186723 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f84012270 con 0x55f75b8cd130 2024-03-24T00:20:31.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.435+0000 7f9f92dc1740 10 client.15237 fetch_fsmap learned FSMap version 23 2024-03-24T00:20:31.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.435+0000 7f9f92dc1740 1 -- 172.21.15.78:0/469186723 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55f75b5462d0 con 0x55f75b8cd130 2024-03-24T00:20:31.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.435+0000 7f9f717fa640 1 -- 172.21.15.78:0/469186723 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f7802c510 con 0x55f75b8cd130 2024-03-24T00:20:31.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.436+0000 7f9f717fa640 1 -- 172.21.15.78:0/469186723 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 23) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f9f8401c7b0 con 0x55f75b8cd130 2024-03-24T00:20:31.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.436+0000 7f9f92dc1740 10 client.15237 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T00:20:31.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.436+0000 7f9f92dc1740 10 client.15237 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-24T00:20:31.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.436+0000 7f9f92dc1740 20 client.15237 populate_metadata read hostname 'smithi078' 2024-03-24T00:20:31.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.436+0000 7f9f92dc1740 1 --2- 172.21.15.78:0/469186723 >> [v2:172.21.15.106:6836/2090968633,v1:172.21.15.106:6837/2090968633] conn(0x55f75b8e3560 0x55f75b8e25f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.436+0000 7f9f92dc1740 4 client.15237 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:20:31.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.436+0000 7f9f92dc1740 1 -- 172.21.15.78:0/469186723 --> [v2:172.21.15.106:6836/2090968633,v1:172.21.15.106:6837/2090968633] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f75b83f160 con 0x55f75b8e3560 2024-03-24T00:20:31.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.436+0000 7f9f8d76f640 1 --2- 172.21.15.78:0/469186723 >> [v2:172.21.15.106:6836/2090968633,v1:172.21.15.106:6837/2090968633] conn(0x55f75b8e3560 0x55f75b8e25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.436+0000 7f9f8d76f640 1 --2- 172.21.15.78:0/469186723 >> [v2:172.21.15.106:6836/2090968633,v1:172.21.15.106:6837/2090968633] conn(0x55f75b8e3560 0x55f75b8e25f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f97341dd96093489 server_cookie=1c384eba78909002 in_seq=0 out_seq=0 2024-03-24T00:20:31.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.436+0000 7f9f717fa640 10 client.15237 ms_handle_connect on v2:172.21.15.106:6836/2090968633 2024-03-24T00:20:31.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.437+0000 7f9f717fa640 1 -- 172.21.15.78:0/469186723 <== mds.2 v2:172.21.15.106:6836/2090968633 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f75b83f160 con 0x55f75b8e3560 2024-03-24T00:20:31.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.437+0000 7f9f717fa640 10 client.15237 handle_command_reply: tid=0 2024-03-24T00:20:31.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.437+0000 7f9f92dc1740 1 client.15237 shutdown 2024-03-24T00:20:31.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.437+0000 7f9f92dc1740 1 -- 172.21.15.78:0/469186723 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f75b8cd130 msgr2=0x55f75b8d71c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:31.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.437+0000 7f9f92dc1740 1 --2- 172.21.15.78:0/469186723 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f75b8cd130 0x55f75b8d71c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f9f84005fd0 tx=0x7f9f8400f790 comp rx=0 tx=0).stop 2024-03-24T00:20:31.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.437+0000 7f9f92dc1740 1 -- 172.21.15.78:0/469186723 shutdown_connections 2024-03-24T00:20:31.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.437+0000 7f9f92dc1740 1 --2- 172.21.15.78:0/469186723 >> [v2:172.21.15.106:6836/2090968633,v1:172.21.15.106:6837/2090968633] conn(0x55f75b8e3560 0x55f75b8e25f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.437+0000 7f9f92dc1740 1 --2- 172.21.15.78:0/469186723 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f75b8d22a0 0x55f75b8de200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.437+0000 7f9f92dc1740 1 --2- 172.21.15.78:0/469186723 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f75b8cd130 0x55f75b8d71c0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.437+0000 7f9f92dc1740 1 --2- 172.21.15.78:0/469186723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f75b8cc740 0x55f75b8d6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.437+0000 7f9f717fa640 0 client.15237 ms_handle_reset on v2:172.21.15.106:6836/2090968633 2024-03-24T00:20:31.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.437+0000 7f9f92dc1740 1 -- 172.21.15.78:0/469186723 >> 172.21.15.78:0/469186723 conn(0x55f75b894a60 msgr2=0x55f75b8c72f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:31.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.437+0000 7f9f92dc1740 1 -- 172.21.15.78:0/469186723 shutdown_connections 2024-03-24T00:20:31.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.437+0000 7f9f92dc1740 1 -- 172.21.15.78:0/469186723 wait complete. 2024-03-24T00:20:31.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.437+0000 7f9f92dc1740 20 client.15237 trim_cache size 0 max 16384 2024-03-24T00:20:31.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.453+0000 7f9f92dc1740 1 Processor -- start 2024-03-24T00:20:31.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.454+0000 7f9f92dc1740 1 -- start start 2024-03-24T00:20:31.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.454+0000 7f9f92dc1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f75b8cc740 0x55f75b8ce6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.454+0000 7f9f92dc1740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f75b8cd130 0x55f75b8cec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.454+0000 7f9f92dc1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f75b8d22a0 0x55f75b8cb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.454+0000 7f9f92dc1740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f75b8e2d80 con 0x55f75b8cd130 2024-03-24T00:20:31.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.454+0000 7f9f92dc1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f75b8e33e0 con 0x55f75b8d22a0 2024-03-24T00:20:31.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.454+0000 7f9f92dc1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f75b8e3a40 con 0x55f75b8cc740 2024-03-24T00:20:31.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.454+0000 7f9f8df70640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f75b8cd130 0x55f75b8cec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.454+0000 7f9f8df70640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f75b8cd130 0x55f75b8cec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42172/0 (socket says 172.21.15.78:42172) 2024-03-24T00:20:31.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.454+0000 7f9f8df70640 1 -- 172.21.15.78:0/384781014 learned_addr learned my addr 172.21.15.78:0/384781014 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:31.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.454+0000 7f9f8d76f640 1 --2- 172.21.15.78:0/384781014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f75b8cc740 0x55f75b8ce6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.454+0000 7f9f8d76f640 1 --2- 172.21.15.78:0/384781014 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f75b8d22a0 0x55f75b8cb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.454+0000 7f9f8df70640 1 -- 172.21.15.78:0/384781014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f75b8cc740 msgr2=0x55f75b8ce6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:31.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.454+0000 7f9f8df70640 1 --2- 172.21.15.78:0/384781014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f75b8cc740 0x55f75b8ce6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.454+0000 7f9f8df70640 1 -- 172.21.15.78:0/384781014 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f75b8d22a0 msgr2=0x55f75b8cb2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:31.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.454+0000 7f9f8df70640 1 --2- 172.21.15.78:0/384781014 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f75b8d22a0 0x55f75b8cb2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.454+0000 7f9f8df70640 1 -- 172.21.15.78:0/384781014 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f75b6106b0 con 0x55f75b8cd130 2024-03-24T00:20:31.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.454+0000 7f9f8df70640 1 --2- 172.21.15.78:0/384781014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f75b8cd130 0x55f75b8cec10 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f9f84012870 tx=0x7f9f84005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf4b1a8d3a4ee6d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:31.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.454+0000 7f9f737fe640 1 -- 172.21.15.78:0/384781014 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f840120d0 con 0x55f75b8cd130 2024-03-24T00:20:31.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.454+0000 7f9f737fe640 1 -- 172.21.15.78:0/384781014 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f8401c900 con 0x55f75b8cd130 2024-03-24T00:20:31.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.454+0000 7f9f737fe640 1 -- 172.21.15.78:0/384781014 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f840048a0 con 0x55f75b8cd130 2024-03-24T00:20:31.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.455+0000 7f9f92dc1740 1 -- 172.21.15.78:0/384781014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f75b8cd130 msgr2=0x55f75b8cec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:31.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.455+0000 7f9f92dc1740 1 --2- 172.21.15.78:0/384781014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f75b8cd130 0x55f75b8cec10 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f9f84012870 tx=0x7f9f84005970 comp rx=0 tx=0).stop 2024-03-24T00:20:31.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.455+0000 7f9f92dc1740 1 -- 172.21.15.78:0/384781014 shutdown_connections 2024-03-24T00:20:31.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.455+0000 7f9f92dc1740 1 --2- 172.21.15.78:0/384781014 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f75b8d22a0 0x55f75b8cb2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.455+0000 7f9f92dc1740 1 --2- 172.21.15.78:0/384781014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f75b8cd130 0x55f75b8cec10 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.455+0000 7f9f92dc1740 1 --2- 172.21.15.78:0/384781014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f75b8cc740 0x55f75b8ce6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.455+0000 7f9f92dc1740 1 -- 172.21.15.78:0/384781014 >> 172.21.15.78:0/384781014 conn(0x55f75b894a60 msgr2=0x55f75b8c7bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:31.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.455+0000 7f9f92dc1740 1 -- 172.21.15.78:0/384781014 shutdown_connections 2024-03-24T00:20:31.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.455+0000 7f9f92dc1740 1 -- 172.21.15.78:0/384781014 wait complete. 2024-03-24T00:20:31.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.455+0000 7f9f92dc1740 1 -- start start 2024-03-24T00:20:31.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.455+0000 7f9f92dc1740 1 Processor -- start 2024-03-24T00:20:31.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.456+0000 7f9f92dc1740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f75b8cc740 0x55f75b93ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.456+0000 7f9f92dc1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f75b8cd130 0x55f75b93b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.456+0000 7f9f92dc1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f75b8d22a0 0x55f75b942010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.456+0000 7f9f92dc1740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f75b8e4700 con 0x55f75b8cc740 2024-03-24T00:20:31.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.456+0000 7f9f92dc1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f75b8e40a0 con 0x55f75b8cd130 2024-03-24T00:20:31.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.456+0000 7f9f92dc1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f75b8e4d60 con 0x55f75b8d22a0 2024-03-24T00:20:31.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.456+0000 7f9f92dc1740 10 client.0 _ll_register_callbacks cb 0x7f9f84012930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:20:31.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.456+0000 7f9f8d76f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f75b8cc740 0x55f75b93ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.456+0000 7f9f8d76f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f75b8cc740 0x55f75b93ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42176/0 (socket says 172.21.15.78:42176) 2024-03-24T00:20:31.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.456+0000 7f9f8d76f640 1 -- 172.21.15.78:0/553092354 learned_addr learned my addr 172.21.15.78:0/553092354 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:31.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.456+0000 7f9f8df70640 1 --2- 172.21.15.78:0/553092354 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f75b8cd130 0x55f75b93b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.456+0000 7f9f8d76f640 1 --2- 172.21.15.78:0/553092354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f75b8d22a0 0x55f75b942010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.456+0000 7f9f8d76f640 1 -- 172.21.15.78:0/553092354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f75b8d22a0 msgr2=0x55f75b942010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:31.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.456+0000 7f9f8d76f640 1 --2- 172.21.15.78:0/553092354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f75b8d22a0 0x55f75b942010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.456+0000 7f9f8d76f640 1 -- 172.21.15.78:0/553092354 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f75b8cd130 msgr2=0x55f75b93b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:31.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.456+0000 7f9f8d76f640 1 --2- 172.21.15.78:0/553092354 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f75b8cd130 0x55f75b93b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.456+0000 7f9f8d76f640 1 -- 172.21.15.78:0/553092354 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f75b942600 con 0x55f75b8cc740 2024-03-24T00:20:31.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.456+0000 7f9f8d76f640 1 --2- 172.21.15.78:0/553092354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f75b8cc740 0x55f75b93ab00 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f9f7802c360 tx=0x7f9f840128a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62084dc11631e997 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:31.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.456+0000 7f9f717fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-24T00:20:31.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.456+0000 7f9f717fa640 1 -- 172.21.15.78:0/553092354 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f78039040 con 0x55f75b8cc740 2024-03-24T00:20:31.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.456+0000 7f9f717fa640 1 -- 172.21.15.78:0/553092354 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f9f7802c960 con 0x55f75b8cc740 2024-03-24T00:20:31.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.456+0000 7f9f92dc1740 1 -- 172.21.15.78:0/553092354 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9f8401c900 con 0x55f75b8cc740 2024-03-24T00:20:31.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.456+0000 7f9f717fa640 1 -- 172.21.15.78:0/553092354 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f78069050 con 0x55f75b8cc740 2024-03-24T00:20:31.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.456+0000 7f9f92dc1740 10 client.15243 fetch_fsmap learned FSMap version 23 2024-03-24T00:20:31.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.456+0000 7f9f92dc1740 1 -- 172.21.15.78:0/553092354 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55f75b943900 con 0x55f75b8cc740 2024-03-24T00:20:31.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.457+0000 7f9f717fa640 1 -- 172.21.15.78:0/553092354 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f8401c900 con 0x55f75b8cc740 2024-03-24T00:20:31.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.457+0000 7f9f717fa640 1 -- 172.21.15.78:0/553092354 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 23) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f9f78064030 con 0x55f75b8cc740 2024-03-24T00:20:31.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.457+0000 7f9f92dc1740 10 client.15243 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T00:20:31.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.457+0000 7f9f92dc1740 10 client.15243 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-24T00:20:31.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.457+0000 7f9f92dc1740 20 client.15243 populate_metadata read hostname 'smithi078' 2024-03-24T00:20:31.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.457+0000 7f9f92dc1740 1 --2- 172.21.15.78:0/553092354 >> [v2:172.21.15.106:6836/2090968633,v1:172.21.15.106:6837/2090968633] conn(0x55f75b947ba0 0x55f75b943ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.457+0000 7f9f92dc1740 4 client.15243 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:20:31.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.457+0000 7f9f92dc1740 1 -- 172.21.15.78:0/553092354 --> [v2:172.21.15.106:6836/2090968633,v1:172.21.15.106:6837/2090968633] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f75b8394b0 con 0x55f75b947ba0 2024-03-24T00:20:31.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.457+0000 7f9f8df70640 1 --2- 172.21.15.78:0/553092354 >> [v2:172.21.15.106:6836/2090968633,v1:172.21.15.106:6837/2090968633] conn(0x55f75b947ba0 0x55f75b943ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.457+0000 7f9f717fa640 10 client.15243 ms_handle_connect on v2:172.21.15.106:6836/2090968633 2024-03-24T00:20:31.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.457+0000 7f9f8df70640 1 --2- 172.21.15.78:0/553092354 >> [v2:172.21.15.106:6836/2090968633,v1:172.21.15.106:6837/2090968633] conn(0x55f75b947ba0 0x55f75b943ad0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1777bb038d4f333c server_cookie=7da01a27bdabd0ab in_seq=0 out_seq=0 2024-03-24T00:20:31.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.458+0000 7f9f717fa640 1 -- 172.21.15.78:0/553092354 <== mds.2 v2:172.21.15.106:6836/2090968633 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+321 (crc 0 0 0) 0x55f75b8394b0 con 0x55f75b947ba0 2024-03-24T00:20:31.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.458+0000 7f9f717fa640 10 client.15243 handle_command_reply: tid=0 2024-03-24T00:20:31.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.458+0000 7f9f92dc1740 1 client.15243 shutdown 2024-03-24T00:20:31.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.458+0000 7f9f92dc1740 1 -- 172.21.15.78:0/553092354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f75b8cc740 msgr2=0x55f75b93ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:31.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.458+0000 7f9f92dc1740 1 --2- 172.21.15.78:0/553092354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f75b8cc740 0x55f75b93ab00 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f9f7802c360 tx=0x7f9f840128a0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.458+0000 7f9f92dc1740 1 -- 172.21.15.78:0/553092354 shutdown_connections 2024-03-24T00:20:31.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.458+0000 7f9f92dc1740 1 --2- 172.21.15.78:0/553092354 >> [v2:172.21.15.106:6836/2090968633,v1:172.21.15.106:6837/2090968633] conn(0x55f75b947ba0 0x55f75b943ad0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.458+0000 7f9f92dc1740 1 --2- 172.21.15.78:0/553092354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f75b8d22a0 0x55f75b942010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.458+0000 7f9f92dc1740 1 --2- 172.21.15.78:0/553092354 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f75b8cd130 0x55f75b93b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.458+0000 7f9f92dc1740 1 --2- 172.21.15.78:0/553092354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f75b8cc740 0x55f75b93ab00 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.458+0000 7f9f717fa640 0 client.15243 ms_handle_reset on v2:172.21.15.106:6836/2090968633 2024-03-24T00:20:31.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.458+0000 7f9f92dc1740 1 -- 172.21.15.78:0/553092354 >> 172.21.15.78:0/553092354 conn(0x55f75b894a60 msgr2=0x55f75b8c7bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:31.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.458+0000 7f9f92dc1740 1 -- 172.21.15.78:0/553092354 shutdown_connections 2024-03-24T00:20:31.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.458+0000 7f9f92dc1740 1 -- 172.21.15.78:0/553092354 wait complete. 2024-03-24T00:20:31.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.458+0000 7f9f92dc1740 20 client.15243 trim_cache size 0 max 16384 2024-03-24T00:20:31.465 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-24T00:20:31.465 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "c7e2806a-e972-11ee-95cd-87774f69a715", 2024-03-24T00:20:31.465 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 2, 2024-03-24T00:20:31.465 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24464, 2024-03-24T00:20:31.465 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-03-24T00:20:31.465 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-03-24T00:20:31.465 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-03-24T00:20:31.465 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 83.078593622, 2024-03-24T00:20:31.465 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 23, 2024-03-24T00:20:31.465 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 91, 2024-03-24T00:20:31.465 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 89, 2024-03-24T00:20:31.466 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 124.923043824 2024-03-24T00:20:31.466 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-24T00:20:31.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.458+0000 7f9f901d4640 1 -- 172.21.15.78:0/2614663427 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f9f740a2890 msgr2=0x7f9f740a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:31.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.458+0000 7f9f901d4640 1 --2- 172.21.15.78:0/2614663427 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f9f740a2890 0x7f9f740a4d50 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9f800a3d80 tx=0x7f9f8000efd0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.458+0000 7f9f901d4640 1 -- 172.21.15.78:0/2614663427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f88102150 msgr2=0x7f9f8819ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:31.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.458+0000 7f9f901d4640 1 --2- 172.21.15.78:0/2614663427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f88102150 0x7f9f8819ed60 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9f8000eb10 tx=0x7f9f8000c2d0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.458+0000 7f9f901d4640 1 -- 172.21.15.78:0/2614663427 shutdown_connections 2024-03-24T00:20:31.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.458+0000 7f9f901d4640 1 --2- 172.21.15.78:0/2614663427 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f9f740a2890 0x7f9f740a4d50 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.458+0000 7f9f901d4640 1 --2- 172.21.15.78:0/2614663427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f88102150 0x7f9f8819ed60 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.458+0000 7f9f901d4640 1 --2- 172.21.15.78:0/2614663427 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f88101790 0x7f9f8819c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.458+0000 7f9f901d4640 1 --2- 172.21.15.78:0/2614663427 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f881005a0 0x7f9f88069370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.458+0000 7f9f901d4640 1 -- 172.21.15.78:0/2614663427 >> 172.21.15.78:0/2614663427 conn(0x7f9f880fbb30 msgr2=0x7f9f88104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:31.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.459+0000 7f9f901d4640 1 -- 172.21.15.78:0/2614663427 shutdown_connections 2024-03-24T00:20:31.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.459+0000 7f9f901d4640 1 -- 172.21.15.78:0/2614663427 wait complete. 2024-03-24T00:20:31.469 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-24T00:20:31.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.571+0000 7f25124d1640 1 -- 172.21.15.78:0/269943588 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f250c1017a0 msgr2=0x7f250c101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:31.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.571+0000 7f25124d1640 1 --2- 172.21.15.78:0/269943588 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f250c1017a0 0x7f250c101c20 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f2500009900 tx=0x7f250002bc30 comp rx=0 tx=0).stop 2024-03-24T00:20:31.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.571+0000 7f25124d1640 1 -- 172.21.15.78:0/269943588 shutdown_connections 2024-03-24T00:20:31.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.571+0000 7f25124d1640 1 --2- 172.21.15.78:0/269943588 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f250c102160 0x7f250c1089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.571+0000 7f25124d1640 1 --2- 172.21.15.78:0/269943588 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f250c1017a0 0x7f250c101c20 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.571+0000 7f25124d1640 1 --2- 172.21.15.78:0/269943588 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f250c1005b0 0x7f250c1009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.571+0000 7f25124d1640 1 -- 172.21.15.78:0/269943588 >> 172.21.15.78:0/269943588 conn(0x7f250c0fbb40 msgr2=0x7f250c0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:31.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.572+0000 7f25124d1640 1 -- 172.21.15.78:0/269943588 shutdown_connections 2024-03-24T00:20:31.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.572+0000 7f25124d1640 1 -- 172.21.15.78:0/269943588 wait complete. 2024-03-24T00:20:31.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.572+0000 7f25124d1640 1 Processor -- start 2024-03-24T00:20:31.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.572+0000 7f25124d1640 1 -- start start 2024-03-24T00:20:31.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.572+0000 7f25124d1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f250c1005b0 0x7f250c06a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.572+0000 7f25124d1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f250c1017a0 0x7f250c19c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.572+0000 7f25124d1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f250c102160 0x7f250c19ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.572+0000 7f25124d1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250c10c230 con 0x7f250c102160 2024-03-24T00:20:31.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.572+0000 7f25124d1640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250c10c0b0 con 0x7f250c1017a0 2024-03-24T00:20:31.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.572+0000 7f25124d1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250c10c3b0 con 0x7f250c1005b0 2024-03-24T00:20:31.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.573+0000 7f2510a6e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f250c102160 0x7f250c19ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.573+0000 7f2510a6e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f250c102160 0x7f250c19ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42206/0 (socket says 172.21.15.78:42206) 2024-03-24T00:20:31.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.573+0000 7f250b7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f250c1017a0 0x7f250c19c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.573+0000 7f2510a6e640 1 -- 172.21.15.78:0/3947926929 learned_addr learned my addr 172.21.15.78:0/3947926929 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:31.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.573+0000 7f250bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f250c1005b0 0x7f250c06a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.573+0000 7f2510a6e640 1 -- 172.21.15.78:0/3947926929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f250c1005b0 msgr2=0x7f250c06a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:31.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.573+0000 7f2510a6e640 1 --2- 172.21.15.78:0/3947926929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f250c1005b0 0x7f250c06a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.573+0000 7f2510a6e640 1 -- 172.21.15.78:0/3947926929 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f250c1017a0 msgr2=0x7f250c19c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:31.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.573+0000 7f2510a6e640 1 --2- 172.21.15.78:0/3947926929 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f250c1017a0 0x7f250c19c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.573+0000 7f2510a6e640 1 -- 172.21.15.78:0/3947926929 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f250c19f640 con 0x7f250c102160 2024-03-24T00:20:31.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.573+0000 7f250bfff640 1 --2- 172.21.15.78:0/3947926929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f250c1005b0 0x7f250c06a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:31.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.573+0000 7f2510a6e640 1 --2- 172.21.15.78:0/3947926929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f250c102160 0x7f250c19ed30 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f250400eb10 tx=0x7f250400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:31.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.573+0000 7f25097fa640 1 -- 172.21.15.78:0/3947926929 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2504004b20 con 0x7f250c102160 2024-03-24T00:20:31.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.573+0000 7f25097fa640 1 -- 172.21.15.78:0/3947926929 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2504004cc0 con 0x7f250c102160 2024-03-24T00:20:31.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.573+0000 7f25124d1640 1 -- 172.21.15.78:0/3947926929 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f250c19f8d0 con 0x7f250c102160 2024-03-24T00:20:31.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.573+0000 7f25097fa640 1 -- 172.21.15.78:0/3947926929 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2504010760 con 0x7f250c102160 2024-03-24T00:20:31.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.573+0000 7f25124d1640 1 -- 172.21.15.78:0/3947926929 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f250c19fda0 con 0x7f250c102160 2024-03-24T00:20:31.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.575+0000 7f25097fa640 1 -- 172.21.15.78:0/3947926929 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f2504005020 con 0x7f250c102160 2024-03-24T00:20:31.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.575+0000 7f2515197740 1 Processor -- start 2024-03-24T00:20:31.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.575+0000 7f2515197740 1 -- start start 2024-03-24T00:20:31.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.576+0000 7f2515197740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561fee904b50 0x561fee904f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.576+0000 7f2515197740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fee905540 0x561fee90a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.576+0000 7f2515197740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561fee90a6b0 0x561fee90cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.576+0000 7f2515197740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fee865c20 con 0x561fee905540 2024-03-24T00:20:31.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.576+0000 7f2515197740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fee80ea40 con 0x561fee90a6b0 2024-03-24T00:20:31.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.576+0000 7f2515197740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fee861cc0 con 0x561fee904b50 2024-03-24T00:20:31.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.576+0000 7f25097fa640 1 --2- 172.21.15.78:0/3947926929 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f24f40a2890 0x7f24f40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.576+0000 7f250bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fee905540 0x561fee90a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.576+0000 7f250bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fee905540 0x561fee90a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42214/0 (socket says 172.21.15.78:42214) 2024-03-24T00:20:31.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.576+0000 7f250bfff640 1 -- 172.21.15.78:0/730448793 learned_addr learned my addr 172.21.15.78:0/730448793 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:31.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.576+0000 7f250b7fe640 1 --2- 172.21.15.78:0/730448793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561fee904b50 0x561fee904f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.576+0000 7f250b7fe640 1 --2- 172.21.15.78:0/730448793 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561fee90a6b0 0x561fee90cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.576+0000 7f2510a6e640 1 --2- 172.21.15.78:0/3947926929 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f24f40a2890 0x7f24f40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.576+0000 7f250b7fe640 1 -- 172.21.15.78:0/730448793 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561fee90a6b0 msgr2=0x561fee90cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:31.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.576+0000 7f250b7fe640 1 --2- 172.21.15.78:0/730448793 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561fee90a6b0 0x561fee90cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.576+0000 7f250b7fe640 1 -- 172.21.15.78:0/730448793 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fee905540 msgr2=0x561fee90a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:31.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.576+0000 7f250b7fe640 1 --2- 172.21.15.78:0/730448793 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fee905540 0x561fee90a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.576+0000 7f250b7fe640 1 -- 172.21.15.78:0/730448793 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561fee830f90 con 0x561fee904b50 2024-03-24T00:20:31.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.576+0000 7f250bfff640 1 --2- 172.21.15.78:0/730448793 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fee905540 0x561fee90a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:31.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.576+0000 7f25097fa640 1 -- 172.21.15.78:0/3947926929 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f25040cf4c0 con 0x7f250c102160 2024-03-24T00:20:31.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.576+0000 7f250b7fe640 1 --2- 172.21.15.78:0/730448793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561fee904b50 0x561fee904f70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2500009230 tx=0x7f2500005b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=114980f71c3dffc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:31.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.576+0000 7f2510a6e640 1 --2- 172.21.15.78:0/3947926929 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f24f40a2890 0x7f24f40a4d50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2504066730 tx=0x7f25040c63f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:31.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.576+0000 7f24f27fc640 1 -- 172.21.15.78:0/730448793 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2500004030 con 0x561fee904b50 2024-03-24T00:20:31.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.577+0000 7f24f27fc640 1 -- 172.21.15.78:0/730448793 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f25000041d0 con 0x561fee904b50 2024-03-24T00:20:31.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.577+0000 7f24f27fc640 1 -- 172.21.15.78:0/730448793 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250003f440 con 0x561fee904b50 2024-03-24T00:20:31.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.577+0000 7f2515197740 1 -- 172.21.15.78:0/730448793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561fee904b50 msgr2=0x561fee904f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:31.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.577+0000 7f2515197740 1 --2- 172.21.15.78:0/730448793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561fee904b50 0x561fee904f70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2500009230 tx=0x7f2500005b50 comp rx=0 tx=0).stop 2024-03-24T00:20:31.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.577+0000 7f2515197740 1 -- 172.21.15.78:0/730448793 shutdown_connections 2024-03-24T00:20:31.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.577+0000 7f2515197740 1 --2- 172.21.15.78:0/730448793 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561fee90a6b0 0x561fee90cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.577+0000 7f2515197740 1 --2- 172.21.15.78:0/730448793 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fee905540 0x561fee90a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.577+0000 7f2515197740 1 --2- 172.21.15.78:0/730448793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561fee904b50 0x561fee904f70 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.577+0000 7f2515197740 1 -- 172.21.15.78:0/730448793 >> 172.21.15.78:0/730448793 conn(0x561fee8c8a60 msgr2=0x561fee8fc0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:31.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.577+0000 7f2515197740 1 -- 172.21.15.78:0/730448793 shutdown_connections 2024-03-24T00:20:31.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.577+0000 7f2515197740 1 -- 172.21.15.78:0/730448793 wait complete. 2024-03-24T00:20:31.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.577+0000 7f2515197740 1 -- start start 2024-03-24T00:20:31.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.578+0000 7f2515197740 1 Processor -- start 2024-03-24T00:20:31.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.578+0000 7f2515197740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fee904b50 0x561fee90f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.578+0000 7f2515197740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561fee905540 0x561fee90f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.578+0000 7f2515197740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561fee90a6b0 0x561fee9165e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.578+0000 7f2515197740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fee90d600 con 0x561fee904b50 2024-03-24T00:20:31.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.578+0000 7f2515197740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fee821e90 con 0x561fee90a6b0 2024-03-24T00:20:31.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.578+0000 7f2515197740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fee90d780 con 0x561fee905540 2024-03-24T00:20:31.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.578+0000 7f2515197740 10 client.0 _ll_register_callbacks cb 0x561fee579ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:20:31.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.578+0000 7f250b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fee904b50 0x561fee90f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.578+0000 7f250b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fee904b50 0x561fee90f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42222/0 (socket says 172.21.15.78:42222) 2024-03-24T00:20:31.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.578+0000 7f250b7fe640 1 -- 172.21.15.78:0/1774106709 learned_addr learned my addr 172.21.15.78:0/1774106709 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:31.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.578+0000 7f250bfff640 1 --2- 172.21.15.78:0/1774106709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561fee905540 0x561fee90f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.578+0000 7f250b7fe640 1 --2- 172.21.15.78:0/1774106709 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561fee90a6b0 0x561fee9165e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.578+0000 7f250b7fe640 1 -- 172.21.15.78:0/1774106709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561fee905540 msgr2=0x561fee90f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:31.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.578+0000 7f250b7fe640 1 --2- 172.21.15.78:0/1774106709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561fee905540 0x561fee90f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.578+0000 7f250b7fe640 1 -- 172.21.15.78:0/1774106709 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561fee90a6b0 msgr2=0x561fee9165e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:31.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.578+0000 7f250b7fe640 1 --2- 172.21.15.78:0/1774106709 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561fee90a6b0 0x561fee9165e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.578+0000 7f250b7fe640 1 -- 172.21.15.78:0/1774106709 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561fee82e9d0 con 0x561fee904b50 2024-03-24T00:20:31.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.579+0000 7f250b7fe640 1 --2- 172.21.15.78:0/1774106709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fee904b50 0x561fee90f060 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f250002c170 tx=0x7f2500005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca987a8b16cbed7b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:31.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.579+0000 7f24eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-24T00:20:31.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.579+0000 7f24eb7fe640 1 -- 172.21.15.78:0/1774106709 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25000057b0 con 0x561fee904b50 2024-03-24T00:20:31.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.579+0000 7f24eb7fe640 1 -- 172.21.15.78:0/1774106709 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f2500004380 con 0x561fee904b50 2024-03-24T00:20:31.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.579+0000 7f2515197740 1 -- 172.21.15.78:0/1774106709 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f25000041d0 con 0x561fee904b50 2024-03-24T00:20:31.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.579+0000 7f24eb7fe640 1 -- 172.21.15.78:0/1774106709 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f250005c050 con 0x561fee904b50 2024-03-24T00:20:31.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.579+0000 7f2515197740 10 client.15255 fetch_fsmap learned FSMap version 23 2024-03-24T00:20:31.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.579+0000 7f2515197740 1 -- 172.21.15.78:0/1774106709 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x561fee57a2d0 con 0x561fee904b50 2024-03-24T00:20:31.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.579+0000 7f24eb7fe640 1 -- 172.21.15.78:0/1774106709 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f25000041d0 con 0x561fee904b50 2024-03-24T00:20:31.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.579+0000 7f24eb7fe640 1 -- 172.21.15.78:0/1774106709 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 23) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f250003f440 con 0x561fee904b50 2024-03-24T00:20:31.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.579+0000 7f2515197740 10 client.15255 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T00:20:31.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.579+0000 7f2515197740 10 client.15255 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-24T00:20:31.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.579+0000 7f2515197740 20 client.15255 populate_metadata read hostname 'smithi078' 2024-03-24T00:20:31.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.579+0000 7f2515197740 1 --2- 172.21.15.78:0/1774106709 >> [v2:172.21.15.107:6838/117119462,v1:172.21.15.107:6839/117119462] conn(0x561fee91a740 0x561fee91f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.579+0000 7f2515197740 4 client.15255 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:20:31.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.579+0000 7f2515197740 1 -- 172.21.15.78:0/1774106709 --> [v2:172.21.15.107:6838/117119462,v1:172.21.15.107:6839/117119462] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561fee873160 con 0x561fee91a740 2024-03-24T00:20:31.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.579+0000 7f250bfff640 1 --2- 172.21.15.78:0/1774106709 >> [v2:172.21.15.107:6838/117119462,v1:172.21.15.107:6839/117119462] conn(0x561fee91a740 0x561fee91f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.580+0000 7f250bfff640 1 --2- 172.21.15.78:0/1774106709 >> [v2:172.21.15.107:6838/117119462,v1:172.21.15.107:6839/117119462] conn(0x561fee91a740 0x561fee91f7d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=237558c5cc596142 server_cookie=62fda58b6909a8f6 in_seq=0 out_seq=0 2024-03-24T00:20:31.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.580+0000 7f24eb7fe640 10 client.15255 ms_handle_connect on v2:172.21.15.107:6838/117119462 2024-03-24T00:20:31.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.580+0000 7f24eb7fe640 1 -- 172.21.15.78:0/1774106709 <== mds.1 v2:172.21.15.107:6838/117119462 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561fee873160 con 0x561fee91a740 2024-03-24T00:20:31.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.580+0000 7f24eb7fe640 10 client.15255 handle_command_reply: tid=0 2024-03-24T00:20:31.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.581+0000 7f2515197740 1 client.15255 shutdown 2024-03-24T00:20:31.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.581+0000 7f2515197740 1 -- 172.21.15.78:0/1774106709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fee904b50 msgr2=0x561fee90f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:31.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.581+0000 7f2515197740 1 --2- 172.21.15.78:0/1774106709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fee904b50 0x561fee90f060 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f250002c170 tx=0x7f2500005d50 comp rx=0 tx=0).stop 2024-03-24T00:20:31.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.581+0000 7f2515197740 1 -- 172.21.15.78:0/1774106709 shutdown_connections 2024-03-24T00:20:31.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.581+0000 7f2515197740 1 --2- 172.21.15.78:0/1774106709 >> [v2:172.21.15.107:6838/117119462,v1:172.21.15.107:6839/117119462] conn(0x561fee91a740 0x561fee91f7d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.581+0000 7f2515197740 1 --2- 172.21.15.78:0/1774106709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561fee905540 0x561fee90f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.581+0000 7f2515197740 1 --2- 172.21.15.78:0/1774106709 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561fee90a6b0 0x561fee9165e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.581+0000 7f24eb7fe640 0 client.15255 ms_handle_reset on v2:172.21.15.107:6838/117119462 2024-03-24T00:20:31.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.581+0000 7f2515197740 1 --2- 172.21.15.78:0/1774106709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fee904b50 0x561fee90f060 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.581+0000 7f2515197740 1 -- 172.21.15.78:0/1774106709 >> 172.21.15.78:0/1774106709 conn(0x561fee8c8a60 msgr2=0x561fee8fb2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:31.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.581+0000 7f2515197740 1 -- 172.21.15.78:0/1774106709 shutdown_connections 2024-03-24T00:20:31.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.581+0000 7f2515197740 1 -- 172.21.15.78:0/1774106709 wait complete. 2024-03-24T00:20:31.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.581+0000 7f2515197740 20 client.15255 trim_cache size 0 max 16384 2024-03-24T00:20:31.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:31 smithi078 ceph-mon[32429]: pgmap v346: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:20:31.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:31 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2682881687' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:20:31.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:31 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/4232739175' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:20:31.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:31 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/778969408' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:20:31.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.597+0000 7f2515197740 1 Processor -- start 2024-03-24T00:20:31.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.598+0000 7f2515197740 1 -- start start 2024-03-24T00:20:31.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.598+0000 7f2515197740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561fee904b50 0x561fee90f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.598+0000 7f2515197740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fee905540 0x561fee9165a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.598+0000 7f2515197740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561fee90a6b0 0x561fee8faa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.598+0000 7f2515197740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fee91ab90 con 0x561fee905540 2024-03-24T00:20:31.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.598+0000 7f2515197740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fee91b1d0 con 0x561fee904b50 2024-03-24T00:20:31.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.598+0000 7f2515197740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fee91b830 con 0x561fee90a6b0 2024-03-24T00:20:31.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.598+0000 7f250bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fee905540 0x561fee9165a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.598+0000 7f250bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fee905540 0x561fee9165a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42232/0 (socket says 172.21.15.78:42232) 2024-03-24T00:20:31.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.598+0000 7f250bfff640 1 -- 172.21.15.78:0/3717206673 learned_addr learned my addr 172.21.15.78:0/3717206673 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:31.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.598+0000 7f250b7fe640 1 --2- 172.21.15.78:0/3717206673 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561fee904b50 0x561fee90f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.598+0000 7f250b7fe640 1 --2- 172.21.15.78:0/3717206673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561fee90a6b0 0x561fee8faa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.598+0000 7f250bfff640 1 -- 172.21.15.78:0/3717206673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561fee90a6b0 msgr2=0x561fee8faa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:31.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.598+0000 7f250bfff640 1 --2- 172.21.15.78:0/3717206673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561fee90a6b0 0x561fee8faa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.598+0000 7f250bfff640 1 -- 172.21.15.78:0/3717206673 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561fee904b50 msgr2=0x561fee90f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:31.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.598+0000 7f250bfff640 1 --2- 172.21.15.78:0/3717206673 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561fee904b50 0x561fee90f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.598+0000 7f250bfff640 1 -- 172.21.15.78:0/3717206673 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561fee6446b0 con 0x561fee905540 2024-03-24T00:20:31.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.598+0000 7f250bfff640 1 --2- 172.21.15.78:0/3717206673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fee905540 0x561fee9165a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f24fc005d00 tx=0x7f24fc005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3b0f5937ed5433d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:31.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.598+0000 7f24f1ffb640 1 -- 172.21.15.78:0/3717206673 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24fc0202f0 con 0x561fee905540 2024-03-24T00:20:31.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.598+0000 7f24f1ffb640 1 -- 172.21.15.78:0/3717206673 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f24fc004490 con 0x561fee905540 2024-03-24T00:20:31.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.598+0000 7f24f1ffb640 1 -- 172.21.15.78:0/3717206673 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24fc020d30 con 0x561fee905540 2024-03-24T00:20:31.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.599+0000 7f2515197740 1 -- 172.21.15.78:0/3717206673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fee905540 msgr2=0x561fee9165a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:31.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.599+0000 7f2515197740 1 --2- 172.21.15.78:0/3717206673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fee905540 0x561fee9165a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f24fc005d00 tx=0x7f24fc005f90 comp rx=0 tx=0).stop 2024-03-24T00:20:31.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.599+0000 7f2515197740 1 -- 172.21.15.78:0/3717206673 shutdown_connections 2024-03-24T00:20:31.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.599+0000 7f2515197740 1 --2- 172.21.15.78:0/3717206673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561fee90a6b0 0x561fee8faa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.599+0000 7f2515197740 1 --2- 172.21.15.78:0/3717206673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fee905540 0x561fee9165a0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.599+0000 7f2515197740 1 --2- 172.21.15.78:0/3717206673 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561fee904b50 0x561fee90f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.599+0000 7f2515197740 1 -- 172.21.15.78:0/3717206673 >> 172.21.15.78:0/3717206673 conn(0x561fee8c8a60 msgr2=0x561fee90ec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:31.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.599+0000 7f2515197740 1 -- 172.21.15.78:0/3717206673 shutdown_connections 2024-03-24T00:20:31.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.599+0000 7f2515197740 1 -- 172.21.15.78:0/3717206673 wait complete. 2024-03-24T00:20:31.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.599+0000 7f2515197740 1 -- start start 2024-03-24T00:20:31.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.600+0000 7f2515197740 1 Processor -- start 2024-03-24T00:20:31.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.600+0000 7f2515197740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561fee904b50 0x561fee915f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.600+0000 7f2515197740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fee905540 0x561fee916480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.600+0000 7f2515197740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561fee90a6b0 0x561fee9129e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.600+0000 7f2515197740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fee91c4f0 con 0x561fee905540 2024-03-24T00:20:31.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.600+0000 7f2515197740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fee91be90 con 0x561fee90a6b0 2024-03-24T00:20:31.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.600+0000 7f2515197740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fee91cb50 con 0x561fee904b50 2024-03-24T00:20:31.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.600+0000 7f250bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fee905540 0x561fee916480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.600+0000 7f2515197740 10 client.0 _ll_register_callbacks cb 0x7f24fc009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:20:31.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.600+0000 7f250bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fee905540 0x561fee916480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42246/0 (socket says 172.21.15.78:42246) 2024-03-24T00:20:31.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.600+0000 7f250bfff640 1 -- 172.21.15.78:0/2709349763 learned_addr learned my addr 172.21.15.78:0/2709349763 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:31.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.600+0000 7f250b7fe640 1 --2- 172.21.15.78:0/2709349763 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561fee90a6b0 0x561fee9129e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.600+0000 7f250b7fe640 1 --2- 172.21.15.78:0/2709349763 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561fee904b50 0x561fee915f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.600+0000 7f250bfff640 1 -- 172.21.15.78:0/2709349763 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561fee904b50 msgr2=0x561fee915f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:31.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.600+0000 7f250bfff640 1 --2- 172.21.15.78:0/2709349763 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561fee904b50 0x561fee915f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.600+0000 7f250bfff640 1 -- 172.21.15.78:0/2709349763 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561fee90a6b0 msgr2=0x561fee9129e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:31.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.600+0000 7f250bfff640 1 --2- 172.21.15.78:0/2709349763 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561fee90a6b0 0x561fee9129e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.600+0000 7f250bfff640 1 -- 172.21.15.78:0/2709349763 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561fee9131e0 con 0x561fee905540 2024-03-24T00:20:31.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.600+0000 7f250b7fe640 1 --2- 172.21.15.78:0/2709349763 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561fee90a6b0 0x561fee9129e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:20:31.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.600+0000 7f250bfff640 1 --2- 172.21.15.78:0/2709349763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fee905540 0x561fee916480 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f24fc005eb0 tx=0x7f24fc004080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff820167e2507849 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:31.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.600+0000 7f24eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-24T00:20:31.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.600+0000 7f24eb7fe640 1 -- 172.21.15.78:0/2709349763 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24fc02e020 con 0x561fee905540 2024-03-24T00:20:31.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.600+0000 7f24eb7fe640 1 -- 172.21.15.78:0/2709349763 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f24fc012320 con 0x561fee905540 2024-03-24T00:20:31.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.600+0000 7f2515197740 1 -- 172.21.15.78:0/2709349763 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f24fc004490 con 0x561fee905540 2024-03-24T00:20:31.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.601+0000 7f24eb7fe640 1 -- 172.21.15.78:0/2709349763 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f24fc039050 con 0x561fee905540 2024-03-24T00:20:31.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.601+0000 7f2515197740 10 client.15261 fetch_fsmap learned FSMap version 23 2024-03-24T00:20:31.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.601+0000 7f2515197740 1 -- 172.21.15.78:0/2709349763 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x561fee977660 con 0x561fee905540 2024-03-24T00:20:31.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.601+0000 7f24eb7fe640 1 -- 172.21.15.78:0/2709349763 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f24fc004490 con 0x561fee905540 2024-03-24T00:20:31.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.601+0000 7f24eb7fe640 1 -- 172.21.15.78:0/2709349763 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 23) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f24fc005460 con 0x561fee905540 2024-03-24T00:20:31.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.601+0000 7f2515197740 10 client.15261 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T00:20:31.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.601+0000 7f2515197740 10 client.15261 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-24T00:20:31.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.601+0000 7f2515197740 20 client.15261 populate_metadata read hostname 'smithi078' 2024-03-24T00:20:31.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.601+0000 7f2515197740 1 --2- 172.21.15.78:0/2709349763 >> [v2:172.21.15.107:6838/117119462,v1:172.21.15.107:6839/117119462] conn(0x561fee97ba40 0x561fee977940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.601+0000 7f2515197740 4 client.15261 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:20:31.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.601+0000 7f2515197740 1 -- 172.21.15.78:0/2709349763 --> [v2:172.21.15.107:6838/117119462,v1:172.21.15.107:6839/117119462] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561fee86d4b0 con 0x561fee97ba40 2024-03-24T00:20:31.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.601+0000 7f250b7fe640 1 --2- 172.21.15.78:0/2709349763 >> [v2:172.21.15.107:6838/117119462,v1:172.21.15.107:6839/117119462] conn(0x561fee97ba40 0x561fee977940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.601+0000 7f250b7fe640 1 --2- 172.21.15.78:0/2709349763 >> [v2:172.21.15.107:6838/117119462,v1:172.21.15.107:6839/117119462] conn(0x561fee97ba40 0x561fee977940 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c4b7de49f3e979aa server_cookie=c72f2f6538e7beff in_seq=0 out_seq=0 2024-03-24T00:20:31.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.601+0000 7f24eb7fe640 10 client.15261 ms_handle_connect on v2:172.21.15.107:6838/117119462 2024-03-24T00:20:31.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.602+0000 7f24eb7fe640 1 -- 172.21.15.78:0/2709349763 <== mds.1 v2:172.21.15.107:6838/117119462 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x561fee86d4b0 con 0x561fee97ba40 2024-03-24T00:20:31.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.602+0000 7f24eb7fe640 10 client.15261 handle_command_reply: tid=0 2024-03-24T00:20:31.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.602+0000 7f2515197740 1 client.15261 shutdown 2024-03-24T00:20:31.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.602+0000 7f2515197740 1 -- 172.21.15.78:0/2709349763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fee905540 msgr2=0x561fee916480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:31.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.602+0000 7f2515197740 1 --2- 172.21.15.78:0/2709349763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fee905540 0x561fee916480 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f24fc005eb0 tx=0x7f24fc004080 comp rx=0 tx=0).stop 2024-03-24T00:20:31.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.602+0000 7f2515197740 1 -- 172.21.15.78:0/2709349763 shutdown_connections 2024-03-24T00:20:31.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.602+0000 7f2515197740 1 --2- 172.21.15.78:0/2709349763 >> [v2:172.21.15.107:6838/117119462,v1:172.21.15.107:6839/117119462] conn(0x561fee97ba40 0x561fee977940 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.602+0000 7f2515197740 1 --2- 172.21.15.78:0/2709349763 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561fee90a6b0 0x561fee9129e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.602+0000 7f2515197740 1 --2- 172.21.15.78:0/2709349763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fee905540 0x561fee916480 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.602+0000 7f24eb7fe640 0 client.15261 ms_handle_reset on v2:172.21.15.107:6838/117119462 2024-03-24T00:20:31.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.602+0000 7f2515197740 1 --2- 172.21.15.78:0/2709349763 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561fee904b50 0x561fee915f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.602+0000 7f2515197740 1 -- 172.21.15.78:0/2709349763 >> 172.21.15.78:0/2709349763 conn(0x561fee8c8a60 msgr2=0x561fee8ff2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:31.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.602+0000 7f2515197740 1 -- 172.21.15.78:0/2709349763 shutdown_connections 2024-03-24T00:20:31.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.602+0000 7f2515197740 1 -- 172.21.15.78:0/2709349763 wait complete. 2024-03-24T00:20:31.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.602+0000 7f2515197740 20 client.15261 trim_cache size 0 max 16384 2024-03-24T00:20:31.611 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-24T00:20:31.612 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "c7e2806a-e972-11ee-95cd-87774f69a715", 2024-03-24T00:20:31.612 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 1, 2024-03-24T00:20:31.612 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24481, 2024-03-24T00:20:31.612 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-03-24T00:20:31.612 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-03-24T00:20:31.612 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-03-24T00:20:31.612 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 84.233667156999999, 2024-03-24T00:20:31.612 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 23, 2024-03-24T00:20:31.612 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 91, 2024-03-24T00:20:31.612 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 89, 2024-03-24T00:20:31.612 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 122.24410002800001 2024-03-24T00:20:31.612 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-24T00:20:31.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.602+0000 7f25124d1640 1 -- 172.21.15.78:0/3947926929 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f24f40a2890 msgr2=0x7f24f40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:31.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.602+0000 7f25124d1640 1 --2- 172.21.15.78:0/3947926929 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f24f40a2890 0x7f24f40a4d50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2504066730 tx=0x7f25040c63f0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.602+0000 7f25124d1640 1 -- 172.21.15.78:0/3947926929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f250c102160 msgr2=0x7f250c19ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:31.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.602+0000 7f25124d1640 1 --2- 172.21.15.78:0/3947926929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f250c102160 0x7f250c19ed30 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f250400eb10 tx=0x7f250400c2d0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.603+0000 7f25124d1640 1 -- 172.21.15.78:0/3947926929 shutdown_connections 2024-03-24T00:20:31.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.603+0000 7f25124d1640 1 --2- 172.21.15.78:0/3947926929 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f24f40a2890 0x7f24f40a4d50 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.603+0000 7f25124d1640 1 --2- 172.21.15.78:0/3947926929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f250c102160 0x7f250c19ed30 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.603+0000 7f25124d1640 1 --2- 172.21.15.78:0/3947926929 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f250c1017a0 0x7f250c19c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.603+0000 7f25124d1640 1 --2- 172.21.15.78:0/3947926929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f250c1005b0 0x7f250c06a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.603+0000 7f25124d1640 1 -- 172.21.15.78:0/3947926929 >> 172.21.15.78:0/3947926929 conn(0x7f250c0fbb40 msgr2=0x7f250c104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:31.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.603+0000 7f25124d1640 1 -- 172.21.15.78:0/3947926929 shutdown_connections 2024-03-24T00:20:31.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.603+0000 7f25124d1640 1 -- 172.21.15.78:0/3947926929 wait complete. 2024-03-24T00:20:31.615 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-24T00:20:31.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.718+0000 7fa097cc6640 1 -- 172.21.15.78:0/3917409314 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa090103260 msgr2=0x7fa090109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:31.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.718+0000 7fa097cc6640 1 --2- 172.21.15.78:0/3917409314 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa090103260 0x7fa090109af0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa088009ca0 tx=0x7fa08802c0f0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.718+0000 7fa097cc6640 1 -- 172.21.15.78:0/3917409314 shutdown_connections 2024-03-24T00:20:31.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.718+0000 7fa097cc6640 1 --2- 172.21.15.78:0/3917409314 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa090103260 0x7fa090109af0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.718+0000 7fa097cc6640 1 --2- 172.21.15.78:0/3917409314 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa0901028a0 0x7fa090102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.718+0000 7fa097cc6640 1 --2- 172.21.15.78:0/3917409314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0901016b0 0x7fa090101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.718+0000 7fa097cc6640 1 -- 172.21.15.78:0/3917409314 >> 172.21.15.78:0/3917409314 conn(0x7fa0900fcc40 msgr2=0x7fa0900ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:31.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.718+0000 7fa097cc6640 1 -- 172.21.15.78:0/3917409314 shutdown_connections 2024-03-24T00:20:31.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.718+0000 7fa097cc6640 1 -- 172.21.15.78:0/3917409314 wait complete. 2024-03-24T00:20:31.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.719+0000 7fa097cc6640 1 Processor -- start 2024-03-24T00:20:31.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.719+0000 7fa097cc6640 1 -- start start 2024-03-24T00:20:31.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.719+0000 7fa097cc6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0901016b0 0x7fa090197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.719+0000 7fa097cc6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0901028a0 0x7fa090197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.719+0000 7fa097cc6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa090103260 0x7fa09019ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.719+0000 7fa097cc6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa09010d330 con 0x7fa0901016b0 2024-03-24T00:20:31.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.719+0000 7fa097cc6640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa09010d1b0 con 0x7fa090103260 2024-03-24T00:20:31.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.719+0000 7fa097cc6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa09010d4b0 con 0x7fa0901028a0 2024-03-24T00:20:31.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.719+0000 7fa095a62640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0901016b0 0x7fa090197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.719+0000 7fa095a62640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0901016b0 0x7fa090197980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42270/0 (socket says 172.21.15.78:42270) 2024-03-24T00:20:31.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.719+0000 7fa095a62640 1 -- 172.21.15.78:0/3356808022 learned_addr learned my addr 172.21.15.78:0/3356808022 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:31.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.719+0000 7fa096263640 1 --2- 172.21.15.78:0/3356808022 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa090103260 0x7fa09019ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.719+0000 7fa095261640 1 --2- 172.21.15.78:0/3356808022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0901028a0 0x7fa090197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.719+0000 7fa095a62640 1 -- 172.21.15.78:0/3356808022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0901028a0 msgr2=0x7fa090197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:31.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.719+0000 7fa095a62640 1 --2- 172.21.15.78:0/3356808022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0901028a0 0x7fa090197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.719+0000 7fa095a62640 1 -- 172.21.15.78:0/3356808022 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa090103260 msgr2=0x7fa09019ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:31.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.719+0000 7fa095a62640 1 --2- 172.21.15.78:0/3356808022 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa090103260 0x7fa09019ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.719+0000 7fa095a62640 1 -- 172.21.15.78:0/3356808022 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa09019f5a0 con 0x7fa0901016b0 2024-03-24T00:20:31.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.720+0000 7fa095a62640 1 --2- 172.21.15.78:0/3356808022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0901016b0 0x7fa090197980 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fa08c00e9c0 tx=0x7fa08c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:31.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.720+0000 7fa086ffd640 1 -- 172.21.15.78:0/3356808022 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa08c00cd80 con 0x7fa0901016b0 2024-03-24T00:20:31.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.720+0000 7fa097cc6640 1 -- 172.21.15.78:0/3356808022 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa09019f890 con 0x7fa0901016b0 2024-03-24T00:20:31.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.720+0000 7fa086ffd640 1 -- 172.21.15.78:0/3356808022 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa08c0035f0 con 0x7fa0901016b0 2024-03-24T00:20:31.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.720+0000 7fa086ffd640 1 -- 172.21.15.78:0/3356808022 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa08c010690 con 0x7fa0901016b0 2024-03-24T00:20:31.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.720+0000 7fa097cc6640 1 -- 172.21.15.78:0/3356808022 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa09019fd90 con 0x7fa0901016b0 2024-03-24T00:20:31.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.722+0000 7fa086ffd640 1 -- 172.21.15.78:0/3356808022 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fa08c0037a0 con 0x7fa0901016b0 2024-03-24T00:20:31.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.722+0000 7fa09a994740 1 Processor -- start 2024-03-24T00:20:31.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.723+0000 7fa09a994740 1 -- start start 2024-03-24T00:20:31.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.723+0000 7fa09a994740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564628502b50 0x564628502f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.723+0000 7fa09a994740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564628503540 0x564628508040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.723+0000 7fa09a994740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5646285086b0 0x56462850ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.723+0000 7fa09a994740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x564628463c20 con 0x564628502b50 2024-03-24T00:20:31.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.723+0000 7fa09a994740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x56462840ca40 con 0x5646285086b0 2024-03-24T00:20:31.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.723+0000 7fa09a994740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x56462845fcc0 con 0x564628503540 2024-03-24T00:20:31.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.723+0000 7fa086ffd640 1 --2- 172.21.15.78:0/3356808022 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa07c0a2890 0x7fa07c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.723+0000 7fa095261640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5646285086b0 0x56462850ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.723+0000 7fa095a62640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564628503540 0x564628508040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.723+0000 7fa096263640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564628502b50 0x564628502f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.723+0000 7fa095261640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5646285086b0 0x56462850ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.78:45428/0 (socket says 172.21.15.78:45428) 2024-03-24T00:20:31.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.723+0000 7fa095261640 1 -- 172.21.15.78:0/85634351 learned_addr learned my addr 172.21.15.78:0/85634351 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:31.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.723+0000 7fa096263640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564628502b50 0x564628502f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42284/0 (socket says 172.21.15.78:42284) 2024-03-24T00:20:31.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.723+0000 7fa096263640 1 --2- 172.21.15.78:0/3356808022 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa07c0a2890 0x7fa07c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.723+0000 7fa095261640 1 -- 172.21.15.78:0/85634351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564628503540 msgr2=0x564628508040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:31.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.723+0000 7fa095261640 1 --2- 172.21.15.78:0/85634351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564628503540 0x564628508040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.723+0000 7fa086ffd640 1 -- 172.21.15.78:0/3356808022 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fa08c0cf2b0 con 0x7fa0901016b0 2024-03-24T00:20:31.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.723+0000 7fa095261640 1 -- 172.21.15.78:0/85634351 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564628502b50 msgr2=0x564628502f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:31.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.723+0000 7fa095261640 1 --2- 172.21.15.78:0/85634351 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564628502b50 0x564628502f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.723+0000 7fa095261640 1 -- 172.21.15.78:0/85634351 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56462842ef90 con 0x5646285086b0 2024-03-24T00:20:31.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.723+0000 7fa096263640 1 --2- 172.21.15.78:0/3356808022 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa07c0a2890 0x7fa07c0a4d50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa088005e30 tx=0x7fa088032590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:31.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.723+0000 7fa095261640 1 --2- 172.21.15.78:0/85634351 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5646285086b0 0x56462850ab40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa080009c90 tx=0x7fa080011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34dd7ec91fe83aa0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:31.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.724+0000 7fa07bfff640 1 -- 172.21.15.78:0/85634351 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa08001e070 con 0x5646285086b0 2024-03-24T00:20:31.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.724+0000 7fa07bfff640 1 -- 172.21.15.78:0/85634351 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa080019b10 con 0x5646285086b0 2024-03-24T00:20:31.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.724+0000 7fa07bfff640 1 -- 172.21.15.78:0/85634351 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa08000ea00 con 0x5646285086b0 2024-03-24T00:20:31.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.724+0000 7fa09a994740 1 -- 172.21.15.78:0/85634351 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5646285086b0 msgr2=0x56462850ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:31.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.724+0000 7fa09a994740 1 --2- 172.21.15.78:0/85634351 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5646285086b0 0x56462850ab40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa080009c90 tx=0x7fa080011810 comp rx=0 tx=0).stop 2024-03-24T00:20:31.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.724+0000 7fa09a994740 1 -- 172.21.15.78:0/85634351 shutdown_connections 2024-03-24T00:20:31.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.724+0000 7fa09a994740 1 --2- 172.21.15.78:0/85634351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564628503540 0x564628508040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.724+0000 7fa09a994740 1 --2- 172.21.15.78:0/85634351 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5646285086b0 0x56462850ab40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.724+0000 7fa09a994740 1 --2- 172.21.15.78:0/85634351 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564628502b50 0x564628502f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.724+0000 7fa09a994740 1 -- 172.21.15.78:0/85634351 >> 172.21.15.78:0/85634351 conn(0x5646284c6a60 msgr2=0x5646284fa0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:31.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.724+0000 7fa09a994740 1 -- 172.21.15.78:0/85634351 shutdown_connections 2024-03-24T00:20:31.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.725+0000 7fa09a994740 1 -- 172.21.15.78:0/85634351 wait complete. 2024-03-24T00:20:31.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.725+0000 7fa09a994740 1 -- start start 2024-03-24T00:20:31.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.725+0000 7fa09a994740 1 Processor -- start 2024-03-24T00:20:31.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.725+0000 7fa09a994740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564628502b50 0x56462850d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.725+0000 7fa09a994740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564628503540 0x56462850d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.725+0000 7fa09a994740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5646285086b0 0x564628514580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.725+0000 7fa09a994740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56462850b660 con 0x5646285086b0 2024-03-24T00:20:31.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.725+0000 7fa09a994740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x56462841fe90 con 0x564628502b50 2024-03-24T00:20:31.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.725+0000 7fa09a994740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x56462850b7e0 con 0x564628503540 2024-03-24T00:20:31.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.725+0000 7fa09a994740 10 client.0 _ll_register_callbacks cb 0x564628177ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:20:31.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.725+0000 7fa096263640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5646285086b0 0x564628514580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.725+0000 7fa096263640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5646285086b0 0x564628514580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42294/0 (socket says 172.21.15.78:42294) 2024-03-24T00:20:31.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.725+0000 7fa095a62640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564628502b50 0x56462850d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.725+0000 7fa096263640 1 -- 172.21.15.78:0/165542634 learned_addr learned my addr 172.21.15.78:0/165542634 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:31.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.725+0000 7fa095261640 1 --2- 172.21.15.78:0/165542634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564628503540 0x56462850d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.726+0000 7fa096263640 1 -- 172.21.15.78:0/165542634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564628503540 msgr2=0x56462850d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:31.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.726+0000 7fa096263640 1 --2- 172.21.15.78:0/165542634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564628503540 0x56462850d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.726+0000 7fa096263640 1 -- 172.21.15.78:0/165542634 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564628502b50 msgr2=0x56462850d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:31.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.726+0000 7fa096263640 1 --2- 172.21.15.78:0/165542634 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564628502b50 0x56462850d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.726+0000 7fa096263640 1 -- 172.21.15.78:0/165542634 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56462842c9d0 con 0x5646285086b0 2024-03-24T00:20:31.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.726+0000 7fa095a62640 1 --2- 172.21.15.78:0/165542634 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564628502b50 0x56462850d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:31.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.726+0000 7fa096263640 1 --2- 172.21.15.78:0/165542634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5646285086b0 0x564628514580 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa088036000 tx=0x7fa0880095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ce9d663644e38c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:31.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.726+0000 7fa0797fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-24T00:20:31.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.726+0000 7fa0797fa640 1 -- 172.21.15.78:0/165542634 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa088054060 con 0x5646285086b0 2024-03-24T00:20:31.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.726+0000 7fa0797fa640 1 -- 172.21.15.78:0/165542634 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fa088004250 con 0x5646285086b0 2024-03-24T00:20:31.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.726+0000 7fa09a994740 1 -- 172.21.15.78:0/165542634 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa080019b10 con 0x5646285086b0 2024-03-24T00:20:31.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.726+0000 7fa0797fa640 1 -- 172.21.15.78:0/165542634 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa088039040 con 0x5646285086b0 2024-03-24T00:20:31.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.726+0000 7fa09a994740 10 client.15273 fetch_fsmap learned FSMap version 23 2024-03-24T00:20:31.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.726+0000 7fa09a994740 1 -- 172.21.15.78:0/165542634 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5646281782d0 con 0x5646285086b0 2024-03-24T00:20:31.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.726+0000 7fa0797fa640 1 -- 172.21.15.78:0/165542634 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa080019b10 con 0x5646285086b0 2024-03-24T00:20:31.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.726+0000 7fa0797fa640 1 -- 172.21.15.78:0/165542634 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 23) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fa08804f4e0 con 0x5646285086b0 2024-03-24T00:20:31.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.726+0000 7fa09a994740 10 client.15273 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T00:20:31.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.726+0000 7fa09a994740 10 client.15273 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:20:31.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.726+0000 7fa09a994740 20 client.15273 populate_metadata read hostname 'smithi078' 2024-03-24T00:20:31.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.726+0000 7fa09a994740 1 --2- 172.21.15.78:0/165542634 >> [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] conn(0x564628519830 0x5646285188c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.726+0000 7fa09a994740 4 client.15273 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:20:31.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.726+0000 7fa09a994740 1 -- 172.21.15.78:0/165542634 --> [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564628471160 con 0x564628519830 2024-03-24T00:20:31.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.727+0000 7fa095a62640 1 --2- 172.21.15.78:0/165542634 >> [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] conn(0x564628519830 0x5646285188c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.727+0000 7fa095a62640 1 --2- 172.21.15.78:0/165542634 >> [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] conn(0x564628519830 0x5646285188c0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c228d3ab043171a server_cookie=8004f6430b75c73a in_seq=0 out_seq=0 2024-03-24T00:20:31.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.727+0000 7fa0797fa640 10 client.15273 ms_handle_connect on v2:172.21.15.106:6838/2482071304 2024-03-24T00:20:31.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.728+0000 7fa0797fa640 1 -- 172.21.15.78:0/165542634 <== mds.0 v2:172.21.15.106:6838/2482071304 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564628471160 con 0x564628519830 2024-03-24T00:20:31.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.728+0000 7fa0797fa640 10 client.15273 handle_command_reply: tid=0 2024-03-24T00:20:31.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.728+0000 7fa09a994740 1 client.15273 shutdown 2024-03-24T00:20:31.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.728+0000 7fa09a994740 1 -- 172.21.15.78:0/165542634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5646285086b0 msgr2=0x564628514580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:31.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.728+0000 7fa09a994740 1 --2- 172.21.15.78:0/165542634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5646285086b0 0x564628514580 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa088036000 tx=0x7fa0880095c0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.728+0000 7fa09a994740 1 -- 172.21.15.78:0/165542634 shutdown_connections 2024-03-24T00:20:31.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.728+0000 7fa09a994740 1 --2- 172.21.15.78:0/165542634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564628503540 0x56462850d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.728+0000 7fa09a994740 1 --2- 172.21.15.78:0/165542634 >> [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] conn(0x564628519830 0x5646285188c0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.728+0000 7fa09a994740 1 --2- 172.21.15.78:0/165542634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5646285086b0 0x564628514580 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.728+0000 7fa09a994740 1 --2- 172.21.15.78:0/165542634 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564628502b50 0x56462850d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.728+0000 7fa0797fa640 0 client.15273 ms_handle_reset on v2:172.21.15.106:6838/2482071304 2024-03-24T00:20:31.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.728+0000 7fa09a994740 1 -- 172.21.15.78:0/165542634 >> 172.21.15.78:0/165542634 conn(0x5646284c6a60 msgr2=0x5646284f92f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:31.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.728+0000 7fa09a994740 1 -- 172.21.15.78:0/165542634 shutdown_connections 2024-03-24T00:20:31.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.728+0000 7fa09a994740 1 -- 172.21.15.78:0/165542634 wait complete. 2024-03-24T00:20:31.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.728+0000 7fa09a994740 20 client.15273 trim_cache size 0 max 16384 2024-03-24T00:20:31.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.745+0000 7fa09a994740 1 Processor -- start 2024-03-24T00:20:31.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.745+0000 7fa09a994740 1 -- start start 2024-03-24T00:20:31.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.745+0000 7fa09a994740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564628502b50 0x5646284f8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.745+0000 7fa09a994740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564628503540 0x564628514540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.745+0000 7fa09a994740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646285086b0 0x5646284f8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.745+0000 7fa09a994740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x564628519050 con 0x564628503540 2024-03-24T00:20:31.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.745+0000 7fa09a994740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646285196b0 con 0x564628502b50 2024-03-24T00:20:31.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.745+0000 7fa09a994740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x564628519d10 con 0x5646285086b0 2024-03-24T00:20:31.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.745+0000 7fa095261640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564628503540 0x564628514540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.745+0000 7fa095261640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564628503540 0x564628514540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42306/0 (socket says 172.21.15.78:42306) 2024-03-24T00:20:31.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.745+0000 7fa095261640 1 -- 172.21.15.78:0/3794501664 learned_addr learned my addr 172.21.15.78:0/3794501664 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:31.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.745+0000 7fa095a62640 1 --2- 172.21.15.78:0/3794501664 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564628502b50 0x5646284f8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.745+0000 7fa096263640 1 --2- 172.21.15.78:0/3794501664 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646285086b0 0x5646284f8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.745+0000 7fa095261640 1 -- 172.21.15.78:0/3794501664 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646285086b0 msgr2=0x5646284f8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:31.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.745+0000 7fa095261640 1 --2- 172.21.15.78:0/3794501664 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646285086b0 0x5646284f8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.745+0000 7fa095261640 1 -- 172.21.15.78:0/3794501664 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564628502b50 msgr2=0x5646284f8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:31.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.745+0000 7fa095261640 1 --2- 172.21.15.78:0/3794501664 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564628502b50 0x5646284f8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.745+0000 7fa095261640 1 -- 172.21.15.78:0/3794501664 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5646282426b0 con 0x564628503540 2024-03-24T00:20:31.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.745+0000 7fa096263640 1 --2- 172.21.15.78:0/3794501664 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646285086b0 0x5646284f8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:20:31.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.745+0000 7fa095261640 1 --2- 172.21.15.78:0/3794501664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564628503540 0x564628514540 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa08001b000 tx=0x7fa080009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca6d578a776a8daa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:31.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.745+0000 7fa07b7fe640 1 -- 172.21.15.78:0/3794501664 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa080028020 con 0x564628503540 2024-03-24T00:20:31.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.745+0000 7fa07b7fe640 1 -- 172.21.15.78:0/3794501664 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa08001f040 con 0x564628503540 2024-03-24T00:20:31.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.745+0000 7fa07b7fe640 1 -- 172.21.15.78:0/3794501664 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa08001e070 con 0x564628503540 2024-03-24T00:20:31.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.746+0000 7fa09a994740 1 -- 172.21.15.78:0/3794501664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564628503540 msgr2=0x564628514540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:31.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.746+0000 7fa09a994740 1 --2- 172.21.15.78:0/3794501664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564628503540 0x564628514540 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa08001b000 tx=0x7fa080009fd0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.746+0000 7fa09a994740 1 -- 172.21.15.78:0/3794501664 shutdown_connections 2024-03-24T00:20:31.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.746+0000 7fa09a994740 1 --2- 172.21.15.78:0/3794501664 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646285086b0 0x5646284f8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.746+0000 7fa09a994740 1 --2- 172.21.15.78:0/3794501664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564628503540 0x564628514540 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.746+0000 7fa09a994740 1 --2- 172.21.15.78:0/3794501664 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564628502b50 0x5646284f8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.746+0000 7fa09a994740 1 -- 172.21.15.78:0/3794501664 >> 172.21.15.78:0/3794501664 conn(0x5646284c6a60 msgr2=0x5646285049e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:31.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.746+0000 7fa09a994740 1 -- 172.21.15.78:0/3794501664 shutdown_connections 2024-03-24T00:20:31.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.746+0000 7fa09a994740 1 -- 172.21.15.78:0/3794501664 wait complete. 2024-03-24T00:20:31.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.746+0000 7fa09a994740 1 -- start start 2024-03-24T00:20:31.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.746+0000 7fa09a994740 1 Processor -- start 2024-03-24T00:20:31.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.747+0000 7fa09a994740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564628502b50 0x564628513fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.747+0000 7fa09a994740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564628503540 0x564628514510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.747+0000 7fa09a994740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5646285086b0 0x564628510b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.747+0000 7fa09a994740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56462851a9d0 con 0x564628502b50 2024-03-24T00:20:31.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.747+0000 7fa09a994740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x56462851a370 con 0x5646285086b0 2024-03-24T00:20:31.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.747+0000 7fa09a994740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x56462851b030 con 0x564628503540 2024-03-24T00:20:31.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.747+0000 7fa09a994740 10 client.0 _ll_register_callbacks cb 0x7fa08c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:20:31.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.747+0000 7fa095a62640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564628502b50 0x564628513fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.747+0000 7fa095a62640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564628502b50 0x564628513fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42310/0 (socket says 172.21.15.78:42310) 2024-03-24T00:20:31.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.747+0000 7fa095a62640 1 -- 172.21.15.78:0/278085718 learned_addr learned my addr 172.21.15.78:0/278085718 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:31.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.747+0000 7fa095261640 1 --2- 172.21.15.78:0/278085718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564628503540 0x564628514510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.747+0000 7fa096263640 1 --2- 172.21.15.78:0/278085718 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5646285086b0 0x564628510b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.747+0000 7fa095a62640 1 -- 172.21.15.78:0/278085718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564628503540 msgr2=0x564628514510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:31.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.747+0000 7fa095a62640 1 --2- 172.21.15.78:0/278085718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564628503540 0x564628514510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.747+0000 7fa095a62640 1 -- 172.21.15.78:0/278085718 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5646285086b0 msgr2=0x564628510b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:31.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.747+0000 7fa095a62640 1 --2- 172.21.15.78:0/278085718 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5646285086b0 0x564628510b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.747+0000 7fa095a62640 1 -- 172.21.15.78:0/278085718 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564628511370 con 0x564628502b50 2024-03-24T00:20:31.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.747+0000 7fa096263640 1 --2- 172.21.15.78:0/278085718 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5646285086b0 0x564628510b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:20:31.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.747+0000 7fa095a62640 1 --2- 172.21.15.78:0/278085718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564628502b50 0x564628513fd0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa08c003ec0 tx=0x7fa08c06a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c1e6eea33907af8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:31.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.747+0000 7fa0797fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-24T00:20:31.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.747+0000 7fa0797fa640 1 -- 172.21.15.78:0/278085718 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa08c09b790 con 0x564628502b50 2024-03-24T00:20:31.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.747+0000 7fa0797fa640 1 -- 172.21.15.78:0/278085718 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fa08c003b10 con 0x564628502b50 2024-03-24T00:20:31.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.747+0000 7fa09a994740 1 -- 172.21.15.78:0/278085718 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa08001f040 con 0x564628502b50 2024-03-24T00:20:31.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.748+0000 7fa0797fa640 1 -- 172.21.15.78:0/278085718 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa08c09f8e0 con 0x564628502b50 2024-03-24T00:20:31.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.748+0000 7fa09a994740 10 client.15279 fetch_fsmap learned FSMap version 23 2024-03-24T00:20:31.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.748+0000 7fa09a994740 1 -- 172.21.15.78:0/278085718 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5646285757b0 con 0x564628502b50 2024-03-24T00:20:31.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.748+0000 7fa0797fa640 1 -- 172.21.15.78:0/278085718 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa08001f040 con 0x564628502b50 2024-03-24T00:20:31.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.748+0000 7fa0797fa640 1 -- 172.21.15.78:0/278085718 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 23) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fa08c0041e0 con 0x564628502b50 2024-03-24T00:20:31.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.748+0000 7fa09a994740 10 client.15279 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T00:20:31.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.748+0000 7fa09a994740 10 client.15279 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:20:31.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.748+0000 7fa09a994740 20 client.15279 populate_metadata read hostname 'smithi078' 2024-03-24T00:20:31.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.748+0000 7fa09a994740 1 --2- 172.21.15.78:0/278085718 >> [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] conn(0x564628579da0 0x564628575aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:31.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.748+0000 7fa09a994740 4 client.15279 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:20:31.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.748+0000 7fa09a994740 1 -- 172.21.15.78:0/278085718 --> [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56462846b4b0 con 0x564628579da0 2024-03-24T00:20:31.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.748+0000 7fa095261640 1 --2- 172.21.15.78:0/278085718 >> [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] conn(0x564628579da0 0x564628575aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:31.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.748+0000 7fa0797fa640 10 client.15279 ms_handle_connect on v2:172.21.15.106:6838/2482071304 2024-03-24T00:20:31.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.748+0000 7fa095261640 1 --2- 172.21.15.78:0/278085718 >> [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] conn(0x564628579da0 0x564628575aa0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2dac5d60e79c5330 server_cookie=7044518ee97d7bbd in_seq=0 out_seq=0 2024-03-24T00:20:31.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.749+0000 7fa0797fa640 1 -- 172.21.15.78:0/278085718 <== mds.0 v2:172.21.15.106:6838/2482071304 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x56462846b4b0 con 0x564628579da0 2024-03-24T00:20:31.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.749+0000 7fa0797fa640 10 client.15279 handle_command_reply: tid=0 2024-03-24T00:20:31.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.749+0000 7fa09a994740 1 client.15279 shutdown 2024-03-24T00:20:31.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.749+0000 7fa09a994740 1 -- 172.21.15.78:0/278085718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564628502b50 msgr2=0x564628513fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:31.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.749+0000 7fa09a994740 1 --2- 172.21.15.78:0/278085718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564628502b50 0x564628513fd0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa08c003ec0 tx=0x7fa08c06a5f0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.749+0000 7fa09a994740 1 -- 172.21.15.78:0/278085718 shutdown_connections 2024-03-24T00:20:31.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.749+0000 7fa09a994740 1 --2- 172.21.15.78:0/278085718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564628503540 0x564628514510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.749+0000 7fa09a994740 1 --2- 172.21.15.78:0/278085718 >> [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] conn(0x564628579da0 0x564628575aa0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.749+0000 7fa09a994740 1 --2- 172.21.15.78:0/278085718 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5646285086b0 0x564628510b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.749+0000 7fa09a994740 1 --2- 172.21.15.78:0/278085718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564628502b50 0x564628513fd0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.749+0000 7fa0797fa640 0 client.15279 ms_handle_reset on v2:172.21.15.106:6838/2482071304 2024-03-24T00:20:31.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.749+0000 7fa09a994740 1 -- 172.21.15.78:0/278085718 >> 172.21.15.78:0/278085718 conn(0x5646284c6a60 msgr2=0x5646284fd2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:31.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.749+0000 7fa09a994740 1 -- 172.21.15.78:0/278085718 shutdown_connections 2024-03-24T00:20:31.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.749+0000 7fa09a994740 1 -- 172.21.15.78:0/278085718 wait complete. 2024-03-24T00:20:31.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.749+0000 7fa09a994740 20 client.15279 trim_cache size 0 max 16384 2024-03-24T00:20:31.761 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-24T00:20:31.761 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "c7e2806a-e972-11ee-95cd-87774f69a715", 2024-03-24T00:20:31.761 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-03-24T00:20:31.761 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24500, 2024-03-24T00:20:31.761 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-03-24T00:20:31.761 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-03-24T00:20:31.761 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-03-24T00:20:31.761 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 107.507280483, 2024-03-24T00:20:31.761 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 23, 2024-03-24T00:20:31.761 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 91, 2024-03-24T00:20:31.761 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 89, 2024-03-24T00:20:31.761 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 113.88268436200001 2024-03-24T00:20:31.762 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-24T00:20:31.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.749+0000 7fa097cc6640 1 -- 172.21.15.78:0/3356808022 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa07c0a2890 msgr2=0x7fa07c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:31.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.749+0000 7fa097cc6640 1 --2- 172.21.15.78:0/3356808022 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa07c0a2890 0x7fa07c0a4d50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa088005e30 tx=0x7fa088032590 comp rx=0 tx=0).stop 2024-03-24T00:20:31.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.749+0000 7fa097cc6640 1 -- 172.21.15.78:0/3356808022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0901016b0 msgr2=0x7fa090197980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:31.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.749+0000 7fa097cc6640 1 --2- 172.21.15.78:0/3356808022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0901016b0 0x7fa090197980 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fa08c00e9c0 tx=0x7fa08c00ee80 comp rx=0 tx=0).stop 2024-03-24T00:20:31.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.749+0000 7fa097cc6640 1 -- 172.21.15.78:0/3356808022 shutdown_connections 2024-03-24T00:20:31.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.750+0000 7fa097cc6640 1 --2- 172.21.15.78:0/3356808022 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fa07c0a2890 0x7fa07c0a4d50 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.750+0000 7fa097cc6640 1 --2- 172.21.15.78:0/3356808022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0901028a0 0x7fa090197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.750+0000 7fa097cc6640 1 --2- 172.21.15.78:0/3356808022 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa090103260 0x7fa09019ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.750+0000 7fa097cc6640 1 --2- 172.21.15.78:0/3356808022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0901016b0 0x7fa090197980 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:31.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.750+0000 7fa097cc6640 1 -- 172.21.15.78:0/3356808022 >> 172.21.15.78:0/3356808022 conn(0x7fa0900fcc40 msgr2=0x7fa090105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:31.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.750+0000 7fa097cc6640 1 -- 172.21.15.78:0/3356808022 shutdown_connections 2024-03-24T00:20:31.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:31.750+0000 7fa097cc6640 1 -- 172.21.15.78:0/3356808022 wait complete. 2024-03-24T00:20:31.764 INFO:tasks.cephadm:Setup complete, yielding 2024-03-24T00:20:31.765 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T00:20:31.774 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2024-03-24T00:20:31.775 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- bash -c 'ceph orch status' 2024-03-24T00:20:31.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:31 smithi107 ceph-mon[36485]: pgmap v346: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:20:31.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:31 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2682881687' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:20:31.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:31 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/4232739175' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:20:31.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:31 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/778969408' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:20:31.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:31 smithi106 ceph-mon[36505]: pgmap v346: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:20:31.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:31 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2682881687' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:20:31.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:31 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/4232739175' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:20:31.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:31 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/778969408' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:20:32.017 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:20:33.707 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:33 smithi078 ceph-mon[32429]: pgmap v347: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:20:33.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.807+0000 7f8f45c1a700 1 -- 172.21.15.78:0/186691546 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f401022c0 msgr2=0x7f8f40102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:33.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.807+0000 7f8f45c1a700 1 --2- 172.21.15.78:0/186691546 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f401022c0 0x7f8f40102720 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8f340099c0 tx=0x7f8f34009cd0 comp rx=0 tx=0).stop 2024-03-24T00:20:33.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.807+0000 7f8f45c1a700 1 -- 172.21.15.78:0/186691546 shutdown_connections 2024-03-24T00:20:33.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.807+0000 7f8f45c1a700 1 --2- 172.21.15.78:0/186691546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f40102c60 0x7f8f40109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:33.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.807+0000 7f8f45c1a700 1 --2- 172.21.15.78:0/186691546 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f401010c0 0x7f8f401014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:33.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.807+0000 7f8f45c1a700 1 --2- 172.21.15.78:0/186691546 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f401022c0 0x7f8f40102720 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:33.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.807+0000 7f8f45c1a700 1 -- 172.21.15.78:0/186691546 >> 172.21.15.78:0/186691546 conn(0x7f8f400fc680 msgr2=0x7f8f400feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:33.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.808+0000 7f8f45c1a700 1 -- 172.21.15.78:0/186691546 shutdown_connections 2024-03-24T00:20:33.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.808+0000 7f8f45c1a700 1 -- 172.21.15.78:0/186691546 wait complete. 2024-03-24T00:20:33.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.808+0000 7f8f45c1a700 1 Processor -- start 2024-03-24T00:20:33.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.808+0000 7f8f45c1a700 1 -- start start 2024-03-24T00:20:33.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.808+0000 7f8f45c1a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f401010c0 0x7f8f401972a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:33.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.808+0000 7f8f45c1a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f401022c0 0x7f8f401977e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:33.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.808+0000 7f8f45c1a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f40102c60 0x7f8f4019e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:33.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.809+0000 7f8f45c1a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4010cc80 con 0x7f8f401010c0 2024-03-24T00:20:33.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.809+0000 7f8f45c1a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4010cb00 con 0x7f8f401022c0 2024-03-24T00:20:33.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.809+0000 7f8f45c1a700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4010ce00 con 0x7f8f40102c60 2024-03-24T00:20:33.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.809+0000 7f8f3effd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f401022c0 0x7f8f401977e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:33.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.809+0000 7f8f3f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f401010c0 0x7f8f401972a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:33.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.809+0000 7f8f3effd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f401022c0 0x7f8f401977e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.78:45478/0 (socket says 172.21.15.78:45478) 2024-03-24T00:20:33.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.809+0000 7f8f3effd700 1 -- 172.21.15.78:0/2599240445 learned_addr learned my addr 172.21.15.78:0/2599240445 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:33.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.809+0000 7f8f3ffff700 1 --2- 172.21.15.78:0/2599240445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f40102c60 0x7f8f4019e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:33.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.809+0000 7f8f3effd700 1 -- 172.21.15.78:0/2599240445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f40102c60 msgr2=0x7f8f4019e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:33.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.809+0000 7f8f3effd700 1 --2- 172.21.15.78:0/2599240445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f40102c60 0x7f8f4019e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:33.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.809+0000 7f8f3effd700 1 -- 172.21.15.78:0/2599240445 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f401010c0 msgr2=0x7f8f401972a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:33.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.809+0000 7f8f3effd700 1 --2- 172.21.15.78:0/2599240445 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f401010c0 0x7f8f401972a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:33.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.809+0000 7f8f3effd700 1 -- 172.21.15.78:0/2599240445 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f4019ef00 con 0x7f8f401022c0 2024-03-24T00:20:33.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.809+0000 7f8f3f7fe700 1 --2- 172.21.15.78:0/2599240445 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f401010c0 0x7f8f401972a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:33.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.809+0000 7f8f3effd700 1 --2- 172.21.15.78:0/2599240445 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f401022c0 0x7f8f401977e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8f34003790 tx=0x7f8f3401ca40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:33.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.809+0000 7f8f3cff9700 1 -- 172.21.15.78:0/2599240445 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f34009e60 con 0x7f8f401022c0 2024-03-24T00:20:33.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.809+0000 7f8f45c1a700 1 -- 172.21.15.78:0/2599240445 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f4006a8b0 con 0x7f8f401022c0 2024-03-24T00:20:33.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.809+0000 7f8f3cff9700 1 -- 172.21.15.78:0/2599240445 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f34003e40 con 0x7f8f401022c0 2024-03-24T00:20:33.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.809+0000 7f8f3cff9700 1 -- 172.21.15.78:0/2599240445 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f340286f0 con 0x7f8f401022c0 2024-03-24T00:20:33.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.809+0000 7f8f45c1a700 1 -- 172.21.15.78:0/2599240445 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f4006ade0 con 0x7f8f401022c0 2024-03-24T00:20:33.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.810+0000 7f8f45c1a700 1 -- 172.21.15.78:0/2599240445 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f40068e50 con 0x7f8f401022c0 2024-03-24T00:20:33.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.811+0000 7f8f3cff9700 1 -- 172.21.15.78:0/2599240445 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f8f340050d0 con 0x7f8f401022c0 2024-03-24T00:20:33.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.812+0000 7f8f3cff9700 1 --2- 172.21.15.78:0/2599240445 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f8f280a2920 0x7f8f280a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:33.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.812+0000 7f8f3cff9700 1 -- 172.21.15.78:0/2599240445 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f8f340d44e0 con 0x7f8f401022c0 2024-03-24T00:20:33.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.812+0000 7f8f3f7fe700 1 --2- 172.21.15.78:0/2599240445 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f8f280a2920 0x7f8f280a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:33.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.813+0000 7f8f3f7fe700 1 --2- 172.21.15.78:0/2599240445 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f8f280a2920 0x7f8f280a4de0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8f40102120 tx=0x7f8f30010040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:33.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:33.816+0000 7f8f3cff9700 1 -- 172.21.15.78:0/2599240445 <== mon.2 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8f34013070 con 0x7f8f401022c0 2024-03-24T00:20:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:33 smithi107 ceph-mon[36485]: pgmap v347: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:20:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:33 smithi106 ceph-mon[36505]: pgmap v347: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:20:34.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:34.032+0000 7f8f45c1a700 1 -- 172.21.15.78:0/2599240445 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8f4006b040 con 0x7f8f280a2920 2024-03-24T00:20:34.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:34.033+0000 7f8f3cff9700 1 -- 172.21.15.78:0/2599240445 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f8f4006b040 con 0x7f8f280a2920 2024-03-24T00:20:34.033 INFO:teuthology.orchestra.run.smithi078.stdout:Backend: cephadm 2024-03-24T00:20:34.033 INFO:teuthology.orchestra.run.smithi078.stdout:Available: Yes 2024-03-24T00:20:34.033 INFO:teuthology.orchestra.run.smithi078.stdout:Paused: No 2024-03-24T00:20:34.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:34.035+0000 7f8f45c1a700 1 -- 172.21.15.78:0/2599240445 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f8f280a2920 msgr2=0x7f8f280a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:34.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:34.035+0000 7f8f45c1a700 1 --2- 172.21.15.78:0/2599240445 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f8f280a2920 0x7f8f280a4de0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8f40102120 tx=0x7f8f30010040 comp rx=0 tx=0).stop 2024-03-24T00:20:34.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:34.035+0000 7f8f45c1a700 1 -- 172.21.15.78:0/2599240445 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f401022c0 msgr2=0x7f8f401977e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:34.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:34.035+0000 7f8f45c1a700 1 --2- 172.21.15.78:0/2599240445 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f401022c0 0x7f8f401977e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8f34003790 tx=0x7f8f3401ca40 comp rx=0 tx=0).stop 2024-03-24T00:20:34.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:34.035+0000 7f8f45c1a700 1 -- 172.21.15.78:0/2599240445 shutdown_connections 2024-03-24T00:20:34.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:34.035+0000 7f8f45c1a700 1 --2- 172.21.15.78:0/2599240445 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f8f280a2920 0x7f8f280a4de0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:34.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:34.035+0000 7f8f45c1a700 1 --2- 172.21.15.78:0/2599240445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f40102c60 0x7f8f4019e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:34.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:34.035+0000 7f8f45c1a700 1 --2- 172.21.15.78:0/2599240445 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f401010c0 0x7f8f401972a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:34.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:34.035+0000 7f8f45c1a700 1 --2- 172.21.15.78:0/2599240445 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f401022c0 0x7f8f401977e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:34.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:34.035+0000 7f8f45c1a700 1 -- 172.21.15.78:0/2599240445 >> 172.21.15.78:0/2599240445 conn(0x7f8f400fc680 msgr2=0x7f8f401055c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:34.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:34.035+0000 7f8f45c1a700 1 -- 172.21.15.78:0/2599240445 shutdown_connections 2024-03-24T00:20:34.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:34.035+0000 7f8f45c1a700 1 -- 172.21.15.78:0/2599240445 wait complete. 2024-03-24T00:20:34.747 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-24T00:20:34.987 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:20:35.826 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:35 smithi078 ceph-mon[32429]: from='client.25103 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:20:35.826 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:35 smithi078 ceph-mon[32429]: pgmap v348: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:20:35.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:35 smithi107 ceph-mon[36485]: from='client.25103 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:20:35.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:35 smithi107 ceph-mon[36485]: pgmap v348: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:20:35.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:35 smithi106 ceph-mon[36505]: from='client.25103 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:20:35.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:35 smithi106 ceph-mon[36505]: pgmap v348: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:20:36.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.790+0000 7f7733e3c700 1 -- 172.21.15.78:0/2694479682 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f772c102ca0 msgr2=0x7f772c109550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:36.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.790+0000 7f7733e3c700 1 --2- 172.21.15.78:0/2694479682 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f772c102ca0 0x7f772c109550 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f772400b0d0 tx=0x7f772400b3e0 comp rx=0 tx=0).stop 2024-03-24T00:20:36.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.790+0000 7f7733e3c700 1 -- 172.21.15.78:0/2694479682 shutdown_connections 2024-03-24T00:20:36.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.790+0000 7f7733e3c700 1 --2- 172.21.15.78:0/2694479682 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f772c102ca0 0x7f772c109550 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:36.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.790+0000 7f7733e3c700 1 --2- 172.21.15.78:0/2694479682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f772c101100 0x7f772c101520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:36.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.790+0000 7f7733e3c700 1 --2- 172.21.15.78:0/2694479682 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f772c102300 0x7f772c102760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:36.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.790+0000 7f7733e3c700 1 -- 172.21.15.78:0/2694479682 >> 172.21.15.78:0/2694479682 conn(0x7f772c0fc680 msgr2=0x7f772c0feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:36.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.791+0000 7f7733e3c700 1 -- 172.21.15.78:0/2694479682 shutdown_connections 2024-03-24T00:20:36.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.791+0000 7f7733e3c700 1 -- 172.21.15.78:0/2694479682 wait complete. 2024-03-24T00:20:36.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.791+0000 7f7733e3c700 1 Processor -- start 2024-03-24T00:20:36.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.791+0000 7f7733e3c700 1 -- start start 2024-03-24T00:20:36.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.792+0000 7f7733e3c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f772c101100 0x7f772c06b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:36.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.792+0000 7f7733e3c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f772c102300 0x7f772c19d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:36.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.792+0000 7f7733e3c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f772c102ca0 0x7f772c19f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:36.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.792+0000 7f7733e3c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772c10ccf0 con 0x7f772c101100 2024-03-24T00:20:36.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.792+0000 7f7733e3c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772c10cb70 con 0x7f772c102ca0 2024-03-24T00:20:36.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.792+0000 7f7733e3c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772c10ce70 con 0x7f772c102300 2024-03-24T00:20:36.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.792+0000 7f7731bd8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f772c101100 0x7f772c06b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:36.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.792+0000 7f7731bd8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f772c101100 0x7f772c06b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53886/0 (socket says 172.21.15.78:53886) 2024-03-24T00:20:36.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.792+0000 7f7731bd8700 1 -- 172.21.15.78:0/522180800 learned_addr learned my addr 172.21.15.78:0/522180800 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:36.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.792+0000 7f77313d7700 1 --2- 172.21.15.78:0/522180800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f772c102300 0x7f772c19d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:36.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.792+0000 7f77323d9700 1 --2- 172.21.15.78:0/522180800 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f772c102ca0 0x7f772c19f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:36.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.792+0000 7f7731bd8700 1 -- 172.21.15.78:0/522180800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f772c102300 msgr2=0x7f772c19d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:36.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.792+0000 7f7731bd8700 1 --2- 172.21.15.78:0/522180800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f772c102300 0x7f772c19d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:36.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.792+0000 7f7731bd8700 1 -- 172.21.15.78:0/522180800 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f772c102ca0 msgr2=0x7f772c19f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:36.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.792+0000 7f7731bd8700 1 --2- 172.21.15.78:0/522180800 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f772c102ca0 0x7f772c19f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:36.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.792+0000 7f7731bd8700 1 -- 172.21.15.78:0/522180800 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f772c19ffa0 con 0x7f772c101100 2024-03-24T00:20:36.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.792+0000 7f7731bd8700 1 --2- 172.21.15.78:0/522180800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f772c101100 0x7f772c06b430 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f772800e950 tx=0x7f772800ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:36.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.792+0000 7f7722ffd700 1 -- 172.21.15.78:0/522180800 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7728004230 con 0x7f772c101100 2024-03-24T00:20:36.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.793+0000 7f7722ffd700 1 -- 172.21.15.78:0/522180800 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f77280043d0 con 0x7f772c101100 2024-03-24T00:20:36.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.793+0000 7f7733e3c700 1 -- 172.21.15.78:0/522180800 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f772c1a0290 con 0x7f772c101100 2024-03-24T00:20:36.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.793+0000 7f7722ffd700 1 -- 172.21.15.78:0/522180800 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7728003dd0 con 0x7f772c101100 2024-03-24T00:20:36.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.793+0000 7f7733e3c700 1 -- 172.21.15.78:0/522180800 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f772c1a07d0 con 0x7f772c101100 2024-03-24T00:20:36.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.793+0000 7f7733e3c700 1 -- 172.21.15.78:0/522180800 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f772c068e50 con 0x7f772c101100 2024-03-24T00:20:36.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.795+0000 7f7722ffd700 1 -- 172.21.15.78:0/522180800 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f772800c7e0 con 0x7f772c101100 2024-03-24T00:20:36.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.795+0000 7f7722ffd700 1 --2- 172.21.15.78:0/522180800 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f77180a29a0 0x7f77180a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:36.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.795+0000 7f77313d7700 1 --2- 172.21.15.78:0/522180800 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f77180a29a0 0x7f77180a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:36.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.795+0000 7f7722ffd700 1 -- 172.21.15.78:0/522180800 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f77280cf280 con 0x7f772c101100 2024-03-24T00:20:36.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.796+0000 7f77313d7700 1 --2- 172.21.15.78:0/522180800 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f77180a29a0 0x7f77180a4e60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f772c06b990 tx=0x7f771c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:36.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:36.799+0000 7f7722ffd700 1 -- 172.21.15.78:0/522180800 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f772809be60 con 0x7f772c101100 2024-03-24T00:20:37.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:37.002+0000 7f7733e3c700 1 -- 172.21.15.78:0/522180800 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f772c0625d0 con 0x7f77180a29a0 2024-03-24T00:20:37.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:37.016+0000 7f7722ffd700 1 -- 172.21.15.78:0/522180800 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4185 (secure 0 0 0) 0x7f772c0625d0 con 0x7f77180a29a0 2024-03-24T00:20:37.016 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T00:20:37.016 INFO:teuthology.orchestra.run.smithi078.stdout:mds.a smithi078 running (2m) 19s ago 2m 16.1M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 ec37728ad1d3 2024-03-24T00:20:37.016 INFO:teuthology.orchestra.run.smithi078.stdout:mds.b smithi106 running (119s) 52s ago 119s 26.0M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 1f15b94cc52e 2024-03-24T00:20:37.016 INFO:teuthology.orchestra.run.smithi078.stdout:mds.c smithi107 running (2m) 52s ago 2m 18.4M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 cedd2902e47d 2024-03-24T00:20:37.016 INFO:teuthology.orchestra.run.smithi078.stdout:mds.d smithi078 running (2m) 19s ago 2m 16.4M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 e84c2439bea2 2024-03-24T00:20:37.016 INFO:teuthology.orchestra.run.smithi078.stdout:mds.e smithi106 running (2m) 52s ago 2m 16.1M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 d421b8ece3df 2024-03-24T00:20:37.016 INFO:teuthology.orchestra.run.smithi078.stdout:mds.f smithi107 running (2m) 52s ago 2m 18.3M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 bf5d224a6f1e 2024-03-24T00:20:37.016 INFO:teuthology.orchestra.run.smithi078.stdout:mds.g smithi078 running (2m) 19s ago 2m 16.1M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 fc59ea233761 2024-03-24T00:20:37.016 INFO:teuthology.orchestra.run.smithi078.stdout:mds.h smithi106 running (2m) 52s ago 2m 18.4M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 cb8a0c8e7dbc 2024-03-24T00:20:37.016 INFO:teuthology.orchestra.run.smithi078.stdout:mds.i smithi107 running (2m) 52s ago 2m 19.3M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 06ef8cdf5fcc 2024-03-24T00:20:37.016 INFO:teuthology.orchestra.run.smithi078.stdout:mds.j smithi078 running (2m) 19s ago 2m 14.0M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 2bae411f7011 2024-03-24T00:20:37.016 INFO:teuthology.orchestra.run.smithi078.stdout:mds.k smithi106 running (2m) 52s ago 2m 13.0M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 98bb5f2ad84d 2024-03-24T00:20:37.016 INFO:teuthology.orchestra.run.smithi078.stdout:mds.l smithi107 running (2m) 52s ago 2m 15.5M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 b4c419602016 2024-03-24T00:20:37.017 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi078 *:9283,8765 running (9m) 19s ago 9m 515M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 f41546bd5b7f 2024-03-24T00:20:37.017 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi106 *:8443,8765 running (7m) 52s ago 7m 454M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 1d774065926d 2024-03-24T00:20:37.017 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.z smithi107 *:8443,8765 running (7m) 52s ago 7m 452M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 2ac0a0181356 2024-03-24T00:20:37.017 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (9m) 19s ago 9m 52.5M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 c766f205ef0e 2024-03-24T00:20:37.017 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi106 running (8m) 52s ago 8m 38.4M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 e957d719501d 2024-03-24T00:20:37.017 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi107 running (8m) 52s ago 8m 39.8M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 37d0db3ece2c 2024-03-24T00:20:37.017 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7m) 19s ago 7m 53.5M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 b307c2120341 2024-03-24T00:20:37.017 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi106 running (7m) 52s ago 7m 54.1M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 08dd91ac5525 2024-03-24T00:20:37.017 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi107 running (6m) 52s ago 6m 54.9M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 c7ecb3310794 2024-03-24T00:20:37.017 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6m) 19s ago 6m 53.6M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 a88b01b13dd0 2024-03-24T00:20:37.017 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi106 running (5m) 52s ago 5m 52.5M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 561904edeeb3 2024-03-24T00:20:37.017 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi107 running (5m) 52s ago 5m 52.5M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 d7016c8c7fe0 2024-03-24T00:20:37.018 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi078 running (5m) 19s ago 5m 41.7M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 fbf7e964a41f 2024-03-24T00:20:37.018 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi106 running (4m) 52s ago 4m 43.5M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 77a85d20b30e 2024-03-24T00:20:37.018 INFO:teuthology.orchestra.run.smithi078.stdout:osd.8 smithi107 running (4m) 52s ago 4m 51.0M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 5518906f7345 2024-03-24T00:20:37.018 INFO:teuthology.orchestra.run.smithi078.stdout:osd.9 smithi078 running (3m) 19s ago 3m 51.7M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 e4a6cd44d703 2024-03-24T00:20:37.018 INFO:teuthology.orchestra.run.smithi078.stdout:osd.10 smithi106 running (3m) 52s ago 3m 51.8M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 7e8f139366db 2024-03-24T00:20:37.018 INFO:teuthology.orchestra.run.smithi078.stdout:osd.11 smithi107 running (2m) 52s ago 2m 49.0M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 f3d648a121e2 2024-03-24T00:20:37.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:37.018+0000 7f7733e3c700 1 -- 172.21.15.78:0/522180800 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f77180a29a0 msgr2=0x7f77180a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:37.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:37.018+0000 7f7733e3c700 1 --2- 172.21.15.78:0/522180800 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f77180a29a0 0x7f77180a4e60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f772c06b990 tx=0x7f771c009260 comp rx=0 tx=0).stop 2024-03-24T00:20:37.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:37.018+0000 7f7733e3c700 1 -- 172.21.15.78:0/522180800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f772c101100 msgr2=0x7f772c06b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:37.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:37.018+0000 7f7733e3c700 1 --2- 172.21.15.78:0/522180800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f772c101100 0x7f772c06b430 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f772800e950 tx=0x7f772800ec60 comp rx=0 tx=0).stop 2024-03-24T00:20:37.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:37.018+0000 7f7733e3c700 1 -- 172.21.15.78:0/522180800 shutdown_connections 2024-03-24T00:20:37.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:37.018+0000 7f7733e3c700 1 --2- 172.21.15.78:0/522180800 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f77180a29a0 0x7f77180a4e60 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:37.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:37.018+0000 7f7733e3c700 1 --2- 172.21.15.78:0/522180800 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f772c102ca0 0x7f772c19f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:37.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:37.018+0000 7f7733e3c700 1 --2- 172.21.15.78:0/522180800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f772c101100 0x7f772c06b430 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:37.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:37.018+0000 7f7733e3c700 1 --2- 172.21.15.78:0/522180800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f772c102300 0x7f772c19d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:37.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:37.018+0000 7f7733e3c700 1 -- 172.21.15.78:0/522180800 >> 172.21.15.78:0/522180800 conn(0x7f772c0fc680 msgr2=0x7f772c1055a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:37.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:37.018+0000 7f7733e3c700 1 -- 172.21.15.78:0/522180800 shutdown_connections 2024-03-24T00:20:37.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:37.018+0000 7f7733e3c700 1 -- 172.21.15.78:0/522180800 wait complete. 2024-03-24T00:20:37.759 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:37 smithi078 ceph-mon[32429]: pgmap v349: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:20:37.787 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-24T00:20:37.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:37 smithi107 ceph-mon[36485]: pgmap v349: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:20:37.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:37 smithi106 ceph-mon[36505]: pgmap v349: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:20:38.026 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:20:38.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:38 smithi078 ceph-mon[32429]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:20:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:38 smithi107 ceph-mon[36485]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:20:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:38 smithi106 ceph-mon[36505]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:20:39.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.661+0000 7f7ef94b0700 1 -- 172.21.15.78:0/3642513914 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ef4102d20 msgr2=0x7f7ef4105110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:39.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.661+0000 7f7ef94b0700 1 --2- 172.21.15.78:0/3642513914 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ef4102d20 0x7f7ef4105110 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4009990 tx=0x7f7ee4009ca0 comp rx=0 tx=0).stop 2024-03-24T00:20:39.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.661+0000 7f7ef94b0700 1 -- 172.21.15.78:0/3642513914 shutdown_connections 2024-03-24T00:20:39.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.661+0000 7f7ef94b0700 1 --2- 172.21.15.78:0/3642513914 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ef4107f80 0x7f7ef410a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:39.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.661+0000 7f7ef94b0700 1 --2- 172.21.15.78:0/3642513914 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ef4102d20 0x7f7ef4105110 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:39.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.661+0000 7f7ef94b0700 1 --2- 172.21.15.78:0/3642513914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ef4105650 0x7f7ef4107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:39.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.661+0000 7f7ef94b0700 1 -- 172.21.15.78:0/3642513914 >> 172.21.15.78:0/3642513914 conn(0x7f7ef40fc690 msgr2=0x7f7ef40fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:39.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.661+0000 7f7ef94b0700 1 -- 172.21.15.78:0/3642513914 shutdown_connections 2024-03-24T00:20:39.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.662+0000 7f7ef94b0700 1 -- 172.21.15.78:0/3642513914 wait complete. 2024-03-24T00:20:39.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.662+0000 7f7ef94b0700 1 Processor -- start 2024-03-24T00:20:39.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.662+0000 7f7ef94b0700 1 -- start start 2024-03-24T00:20:39.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.663+0000 7f7ef94b0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ef4102d20 0x7f7ef4102680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:39.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.663+0000 7f7ef94b0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ef4105650 0x7f7ef4100cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:39.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.663+0000 7f7ef94b0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ef4107f80 0x7f7ef4101210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:39.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.663+0000 7f7ef94b0700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef410dda0 con 0x7f7ef4105650 2024-03-24T00:20:39.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.663+0000 7f7ef94b0700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef410dc20 con 0x7f7ef4107f80 2024-03-24T00:20:39.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.663+0000 7f7ef94b0700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef410df20 con 0x7f7ef4102d20 2024-03-24T00:20:39.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.663+0000 7f7ef27fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ef4105650 0x7f7ef4100cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:39.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.663+0000 7f7ef27fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ef4105650 0x7f7ef4100cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53900/0 (socket says 172.21.15.78:53900) 2024-03-24T00:20:39.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.663+0000 7f7ef27fc700 1 -- 172.21.15.78:0/1214273732 learned_addr learned my addr 172.21.15.78:0/1214273732 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:39.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.663+0000 7f7ef2ffd700 1 --2- 172.21.15.78:0/1214273732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ef4102d20 0x7f7ef4102680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:39.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.663+0000 7f7ef37fe700 1 --2- 172.21.15.78:0/1214273732 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ef4107f80 0x7f7ef4101210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:39.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.663+0000 7f7ef27fc700 1 -- 172.21.15.78:0/1214273732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ef4102d20 msgr2=0x7f7ef4102680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:39.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.663+0000 7f7ef27fc700 1 --2- 172.21.15.78:0/1214273732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ef4102d20 0x7f7ef4102680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:39.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.663+0000 7f7ef27fc700 1 -- 172.21.15.78:0/1214273732 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ef4107f80 msgr2=0x7f7ef4101210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:39.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.663+0000 7f7ef27fc700 1 --2- 172.21.15.78:0/1214273732 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ef4107f80 0x7f7ef4101210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:39.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.663+0000 7f7ef27fc700 1 -- 172.21.15.78:0/1214273732 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef4101ac0 con 0x7f7ef4105650 2024-03-24T00:20:39.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.663+0000 7f7ef2ffd700 1 --2- 172.21.15.78:0/1214273732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ef4102d20 0x7f7ef4102680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:39.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.663+0000 7f7ef27fc700 1 --2- 172.21.15.78:0/1214273732 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ef4105650 0x7f7ef4100cd0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7ee800e9a0 tx=0x7f7ee800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:39.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.663+0000 7f7edbfff700 1 -- 172.21.15.78:0/1214273732 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee800ca10 con 0x7f7ef4105650 2024-03-24T00:20:39.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.663+0000 7f7edbfff700 1 -- 172.21.15.78:0/1214273732 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ee800cbb0 con 0x7f7ef4105650 2024-03-24T00:20:39.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.663+0000 7f7edbfff700 1 -- 172.21.15.78:0/1214273732 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee8018680 con 0x7f7ef4105650 2024-03-24T00:20:39.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.663+0000 7f7ef94b0700 1 -- 172.21.15.78:0/1214273732 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ef41a2540 con 0x7f7ef4105650 2024-03-24T00:20:39.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.664+0000 7f7ef94b0700 1 -- 172.21.15.78:0/1214273732 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ef41a2a50 con 0x7f7ef4105650 2024-03-24T00:20:39.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.664+0000 7f7ef94b0700 1 -- 172.21.15.78:0/1214273732 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ef4068e50 con 0x7f7ef4105650 2024-03-24T00:20:39.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.666+0000 7f7edbfff700 1 -- 172.21.15.78:0/1214273732 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f7ee8010430 con 0x7f7ef4105650 2024-03-24T00:20:39.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.666+0000 7f7edbfff700 1 --2- 172.21.15.78:0/1214273732 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f7edc0a2920 0x7f7edc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:39.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.667+0000 7f7ef2ffd700 1 --2- 172.21.15.78:0/1214273732 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f7edc0a2920 0x7f7edc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:39.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.667+0000 7f7edbfff700 1 -- 172.21.15.78:0/1214273732 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f7ee80cf660 con 0x7f7ef4105650 2024-03-24T00:20:39.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.667+0000 7f7ef2ffd700 1 --2- 172.21.15.78:0/1214273732 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f7edc0a2920 0x7f7edc0a4de0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7ee400b2e0 tx=0x7f7ee4005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:39.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.670+0000 7f7edbfff700 1 -- 172.21.15.78:0/1214273732 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7ee809c1e0 con 0x7f7ef4105650 2024-03-24T00:20:39.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:39 smithi078 ceph-mon[32429]: pgmap v350: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:20:39.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:39 smithi107 ceph-mon[36485]: pgmap v350: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:20:39.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.879+0000 7f7ef94b0700 1 -- 172.21.15.78:0/1214273732 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ef402de80 con 0x7f7edc0a2920 2024-03-24T00:20:39.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.885+0000 7f7edbfff700 1 -- 172.21.15.78:0/1214273732 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7f7ef402de80 con 0x7f7edc0a2920 2024-03-24T00:20:39.884 INFO:teuthology.orchestra.run.smithi078.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-24T00:20:39.884 INFO:teuthology.orchestra.run.smithi078.stdout:mds.all 12/12 55s ago 2m smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi106=b;smithi106=e;smithi106=h;smithi106=k;smithi107=c;smithi107=f;smithi107=i;smithi107=l;count:12 2024-03-24T00:20:39.885 INFO:teuthology.orchestra.run.smithi078.stdout:mgr 3/3 55s ago 7m smithi078=x;smithi106=y;smithi107=z;count:3 2024-03-24T00:20:39.885 INFO:teuthology.orchestra.run.smithi078.stdout:mon 3/3 55s ago 8m smithi078:172.21.15.78=a;smithi106:172.21.15.106=b;smithi107:172.21.15.107=c;count:3 2024-03-24T00:20:39.885 INFO:teuthology.orchestra.run.smithi078.stdout:osd 12 55s ago - 2024-03-24T00:20:39.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.887+0000 7f7ef94b0700 1 -- 172.21.15.78:0/1214273732 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f7edc0a2920 msgr2=0x7f7edc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:39.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.887+0000 7f7ef94b0700 1 --2- 172.21.15.78:0/1214273732 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f7edc0a2920 0x7f7edc0a4de0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7ee400b2e0 tx=0x7f7ee4005fd0 comp rx=0 tx=0).stop 2024-03-24T00:20:39.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.887+0000 7f7ef94b0700 1 -- 172.21.15.78:0/1214273732 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ef4105650 msgr2=0x7f7ef4100cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:39.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.887+0000 7f7ef94b0700 1 --2- 172.21.15.78:0/1214273732 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ef4105650 0x7f7ef4100cd0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7ee800e9a0 tx=0x7f7ee800ecb0 comp rx=0 tx=0).stop 2024-03-24T00:20:39.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.887+0000 7f7ef94b0700 1 -- 172.21.15.78:0/1214273732 shutdown_connections 2024-03-24T00:20:39.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.887+0000 7f7ef94b0700 1 --2- 172.21.15.78:0/1214273732 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f7edc0a2920 0x7f7edc0a4de0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:39.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.887+0000 7f7ef94b0700 1 --2- 172.21.15.78:0/1214273732 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ef4107f80 0x7f7ef4101210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:39.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.887+0000 7f7ef94b0700 1 --2- 172.21.15.78:0/1214273732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ef4102d20 0x7f7ef4102680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:39.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.887+0000 7f7ef94b0700 1 --2- 172.21.15.78:0/1214273732 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ef4105650 0x7f7ef4100cd0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:39.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.887+0000 7f7ef94b0700 1 -- 172.21.15.78:0/1214273732 >> 172.21.15.78:0/1214273732 conn(0x7f7ef40fc690 msgr2=0x7f7ef4103740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:39.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.887+0000 7f7ef94b0700 1 -- 172.21.15.78:0/1214273732 shutdown_connections 2024-03-24T00:20:39.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:39.887+0000 7f7ef94b0700 1 -- 172.21.15.78:0/1214273732 wait complete. 2024-03-24T00:20:39.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:39 smithi106 ceph-mon[36505]: pgmap v350: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:20:40.625 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-24T00:20:40.864 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:20:41.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:41 smithi078 ceph-mon[32429]: from='client.15297 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:20:41.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:41 smithi078 ceph-mon[32429]: pgmap v351: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:20:41.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:41 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]: dispatch 2024-03-24T00:20:41.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:41 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2024-03-24T00:20:41.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:41 smithi107 ceph-mon[36485]: from='client.15297 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:20:41.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:41 smithi107 ceph-mon[36485]: pgmap v351: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:20:41.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:41 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]: dispatch 2024-03-24T00:20:41.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:41 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2024-03-24T00:20:41.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:41 smithi106 ceph-mon[36505]: from='client.15297 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:20:41.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:41 smithi106 ceph-mon[36505]: pgmap v351: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:20:41.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:41 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]: dispatch 2024-03-24T00:20:41.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:41 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2024-03-24T00:20:42.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.548+0000 7f4291750700 1 -- 172.21.15.78:0/1357094989 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f428c1010c0 msgr2=0x7f428c1014e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:42.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.548+0000 7f4291750700 1 --2- 172.21.15.78:0/1357094989 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f428c1010c0 0x7f428c1014e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f427c0099c0 tx=0x7f427c009cd0 comp rx=0 tx=0).stop 2024-03-24T00:20:42.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.548+0000 7f4291750700 1 -- 172.21.15.78:0/1357094989 shutdown_connections 2024-03-24T00:20:42.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.548+0000 7f4291750700 1 --2- 172.21.15.78:0/1357094989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f428c102c60 0x7f428c109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:42.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.548+0000 7f4291750700 1 --2- 172.21.15.78:0/1357094989 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f428c1010c0 0x7f428c1014e0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:42.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.548+0000 7f4291750700 1 --2- 172.21.15.78:0/1357094989 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f428c1022c0 0x7f428c102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:42.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.548+0000 7f4291750700 1 -- 172.21.15.78:0/1357094989 >> 172.21.15.78:0/1357094989 conn(0x7f428c0fc680 msgr2=0x7f428c0feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:42.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.548+0000 7f4291750700 1 -- 172.21.15.78:0/1357094989 shutdown_connections 2024-03-24T00:20:42.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.548+0000 7f4291750700 1 -- 172.21.15.78:0/1357094989 wait complete. 2024-03-24T00:20:42.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.548+0000 7f4291750700 1 Processor -- start 2024-03-24T00:20:42.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.549+0000 7f4291750700 1 -- start start 2024-03-24T00:20:42.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.549+0000 7f4291750700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f428c1010c0 0x7f428c06b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:42.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.549+0000 7f4291750700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f428c1022c0 0x7f428c19d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:42.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.549+0000 7f4291750700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f428c102c60 0x7f428c19f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:42.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.549+0000 7f4291750700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428c10ccb0 con 0x7f428c1010c0 2024-03-24T00:20:42.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.549+0000 7f4291750700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428c10cb30 con 0x7f428c1022c0 2024-03-24T00:20:42.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.549+0000 7f4291750700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428c10ce30 con 0x7f428c102c60 2024-03-24T00:20:42.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.549+0000 7f428affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f428c1010c0 0x7f428c06b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:42.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.549+0000 7f428affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f428c1010c0 0x7f428c06b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53912/0 (socket says 172.21.15.78:53912) 2024-03-24T00:20:42.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.549+0000 7f428affd700 1 -- 172.21.15.78:0/1887591185 learned_addr learned my addr 172.21.15.78:0/1887591185 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:42.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.549+0000 7f428b7fe700 1 --2- 172.21.15.78:0/1887591185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f428c102c60 0x7f428c19f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:42.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.549+0000 7f428a7fc700 1 --2- 172.21.15.78:0/1887591185 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f428c1022c0 0x7f428c19d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:42.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.549+0000 7f428affd700 1 -- 172.21.15.78:0/1887591185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f428c102c60 msgr2=0x7f428c19f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:42.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.549+0000 7f428affd700 1 --2- 172.21.15.78:0/1887591185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f428c102c60 0x7f428c19f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:42.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.549+0000 7f428affd700 1 -- 172.21.15.78:0/1887591185 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f428c1022c0 msgr2=0x7f428c19d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:42.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.549+0000 7f428affd700 1 --2- 172.21.15.78:0/1887591185 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f428c1022c0 0x7f428c19d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:42.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.549+0000 7f428affd700 1 -- 172.21.15.78:0/1887591185 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f428c1a0050 con 0x7f428c1010c0 2024-03-24T00:20:42.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.550+0000 7f428affd700 1 --2- 172.21.15.78:0/1887591185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f428c1010c0 0x7f428c06b430 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f427c003790 tx=0x7f427c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:42.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.550+0000 7f4273fff700 1 -- 172.21.15.78:0/1887591185 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427c009e60 con 0x7f428c1010c0 2024-03-24T00:20:42.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.550+0000 7f4291750700 1 -- 172.21.15.78:0/1887591185 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f428c1a02e0 con 0x7f428c1010c0 2024-03-24T00:20:42.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.550+0000 7f4273fff700 1 -- 172.21.15.78:0/1887591185 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f427c003960 con 0x7f428c1010c0 2024-03-24T00:20:42.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.550+0000 7f4273fff700 1 -- 172.21.15.78:0/1887591185 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427c0055e0 con 0x7f428c1010c0 2024-03-24T00:20:42.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.550+0000 7f4291750700 1 -- 172.21.15.78:0/1887591185 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f428c1a07c0 con 0x7f428c1010c0 2024-03-24T00:20:42.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.551+0000 7f4291750700 1 -- 172.21.15.78:0/1887591185 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f428c068e50 con 0x7f428c1010c0 2024-03-24T00:20:42.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.552+0000 7f4273fff700 1 -- 172.21.15.78:0/1887591185 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f427c003b10 con 0x7f428c1010c0 2024-03-24T00:20:42.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:42 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]': finished 2024-03-24T00:20:42.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:42 smithi078 ceph-mon[32429]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2024-03-24T00:20:42.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:42 smithi078 ceph-mon[32429]: osdmap e92: 12 total, 12 up, 12 in 2024-03-24T00:20:42.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:42 smithi078 ceph-mon[32429]: mgrmap e18: x(active, since 9m), standbys: y, z 2024-03-24T00:20:42.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.552+0000 7f4273fff700 1 --2- 172.21.15.78:0/1887591185 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f42740a2890 0x7f42740a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:42.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.553+0000 7f428a7fc700 1 --2- 172.21.15.78:0/1887591185 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f42740a2890 0x7f42740a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:42.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.553+0000 7f4273fff700 1 -- 172.21.15.78:0/1887591185 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f427c0cfd10 con 0x7f428c1010c0 2024-03-24T00:20:42.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.553+0000 7f428a7fc700 1 --2- 172.21.15.78:0/1887591185 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f42740a2890 0x7f42740a4d50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f428c06b990 tx=0x7f4280009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:42.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.556+0000 7f4273fff700 1 -- 172.21.15.78:0/1887591185 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f427c013070 con 0x7f428c1010c0 2024-03-24T00:20:42.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.758+0000 7f4291750700 1 -- 172.21.15.78:0/1887591185 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f428c0625d0 con 0x7f42740a2890 2024-03-24T00:20:42.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.760+0000 7f4273fff700 1 -- 172.21.15.78:0/1887591185 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7f428c0625d0 con 0x7f42740a2890 2024-03-24T00:20:42.760 INFO:teuthology.orchestra.run.smithi078.stdout:HOST ADDR LABELS STATUS 2024-03-24T00:20:42.760 INFO:teuthology.orchestra.run.smithi078.stdout:smithi078 172.21.15.78 2024-03-24T00:20:42.760 INFO:teuthology.orchestra.run.smithi078.stdout:smithi106 172.21.15.106 2024-03-24T00:20:42.760 INFO:teuthology.orchestra.run.smithi078.stdout:smithi107 172.21.15.107 2024-03-24T00:20:42.760 INFO:teuthology.orchestra.run.smithi078.stdout:3 hosts in cluster 2024-03-24T00:20:42.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.762+0000 7f4291750700 1 -- 172.21.15.78:0/1887591185 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f42740a2890 msgr2=0x7f42740a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:42.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.762+0000 7f4291750700 1 --2- 172.21.15.78:0/1887591185 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f42740a2890 0x7f42740a4d50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f428c06b990 tx=0x7f4280009260 comp rx=0 tx=0).stop 2024-03-24T00:20:42.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.762+0000 7f4291750700 1 -- 172.21.15.78:0/1887591185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f428c1010c0 msgr2=0x7f428c06b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:42.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.762+0000 7f4291750700 1 --2- 172.21.15.78:0/1887591185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f428c1010c0 0x7f428c06b430 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f427c003790 tx=0x7f427c009580 comp rx=0 tx=0).stop 2024-03-24T00:20:42.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.762+0000 7f4291750700 1 -- 172.21.15.78:0/1887591185 shutdown_connections 2024-03-24T00:20:42.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.762+0000 7f4291750700 1 --2- 172.21.15.78:0/1887591185 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f42740a2890 0x7f42740a4d50 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:42.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.762+0000 7f4291750700 1 --2- 172.21.15.78:0/1887591185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f428c102c60 0x7f428c19f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:42.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.762+0000 7f4291750700 1 --2- 172.21.15.78:0/1887591185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f428c1010c0 0x7f428c06b430 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:42.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.762+0000 7f4291750700 1 --2- 172.21.15.78:0/1887591185 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f428c1022c0 0x7f428c19d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:42.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.762+0000 7f4291750700 1 -- 172.21.15.78:0/1887591185 >> 172.21.15.78:0/1887591185 conn(0x7f428c0fc680 msgr2=0x7f428c105640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:42.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.762+0000 7f4291750700 1 -- 172.21.15.78:0/1887591185 shutdown_connections 2024-03-24T00:20:42.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:42.762+0000 7f4291750700 1 -- 172.21.15.78:0/1887591185 wait complete. 2024-03-24T00:20:42.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:42 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]': finished 2024-03-24T00:20:42.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:42 smithi107 ceph-mon[36485]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2024-03-24T00:20:42.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:42 smithi107 ceph-mon[36485]: osdmap e92: 12 total, 12 up, 12 in 2024-03-24T00:20:42.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:42 smithi107 ceph-mon[36485]: mgrmap e18: x(active, since 9m), standbys: y, z 2024-03-24T00:20:42.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:42 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]': finished 2024-03-24T00:20:42.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:42 smithi106 ceph-mon[36505]: from='mgr.14150 172.21.15.78:0/1355257121' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2024-03-24T00:20:42.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:42 smithi106 ceph-mon[36505]: osdmap e92: 12 total, 12 up, 12 in 2024-03-24T00:20:42.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:42 smithi106 ceph-mon[36505]: mgrmap e18: x(active, since 9m), standbys: y, z 2024-03-24T00:20:43.424 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-24T00:20:43.666 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:20:43.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:43 smithi078 ceph-mon[32429]: pgmap v353: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-24T00:20:43.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:43 smithi078 ceph-mon[32429]: osdmap e93: 12 total, 12 up, 12 in 2024-03-24T00:20:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:43 smithi107 ceph-mon[36485]: pgmap v353: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-24T00:20:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:43 smithi107 ceph-mon[36485]: osdmap e93: 12 total, 12 up, 12 in 2024-03-24T00:20:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:43 smithi106 ceph-mon[36505]: pgmap v353: 129 pgs: 129 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-24T00:20:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:43 smithi106 ceph-mon[36505]: osdmap e93: 12 total, 12 up, 12 in 2024-03-24T00:20:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:44 smithi078 ceph-mon[32429]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:20:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:44 smithi107 ceph-mon[36485]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:20:44.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:44 smithi106 ceph-mon[36505]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:20:45.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.343+0000 7effdc8de700 1 -- 172.21.15.78:0/21844005 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effd81033c0 msgr2=0x7effd8103820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:45.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.343+0000 7effdc8de700 1 --2- 172.21.15.78:0/21844005 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effd81033c0 0x7effd8103820 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7effc80099c0 tx=0x7effc8009cd0 comp rx=0 tx=0).stop 2024-03-24T00:20:45.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.344+0000 7effdc8de700 1 -- 172.21.15.78:0/21844005 shutdown_connections 2024-03-24T00:20:45.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.344+0000 7effdc8de700 1 --2- 172.21.15.78:0/21844005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effd8103d60 0x7effd810a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:45.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.344+0000 7effdc8de700 1 --2- 172.21.15.78:0/21844005 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effd81021c0 0x7effd81025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:45.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.344+0000 7effdc8de700 1 --2- 172.21.15.78:0/21844005 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effd81033c0 0x7effd8103820 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:45.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.344+0000 7effdc8de700 1 -- 172.21.15.78:0/21844005 >> 172.21.15.78:0/21844005 conn(0x7effd80fd7a0 msgr2=0x7effd80ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:45.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.344+0000 7effdc8de700 1 -- 172.21.15.78:0/21844005 shutdown_connections 2024-03-24T00:20:45.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.344+0000 7effdc8de700 1 -- 172.21.15.78:0/21844005 wait complete. 2024-03-24T00:20:45.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.344+0000 7effdc8de700 1 Processor -- start 2024-03-24T00:20:45.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.344+0000 7effdc8de700 1 -- start start 2024-03-24T00:20:45.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.344+0000 7effdc8de700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effd81021c0 0x7effd81983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:45.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.345+0000 7effdc8de700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effd81033c0 0x7effd8198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:45.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.345+0000 7effdc8de700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effd8103d60 0x7effd819f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:45.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.345+0000 7effdc8de700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd810dd80 con 0x7effd8103d60 2024-03-24T00:20:45.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.345+0000 7effdc8de700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd810dc00 con 0x7effd81021c0 2024-03-24T00:20:45.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.345+0000 7effdc8de700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd810df00 con 0x7effd81033c0 2024-03-24T00:20:45.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.345+0000 7effd659c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effd81021c0 0x7effd81983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:45.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.345+0000 7effd659c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effd81021c0 0x7effd81983c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.78:44006/0 (socket says 172.21.15.78:44006) 2024-03-24T00:20:45.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.345+0000 7effd659c700 1 -- 172.21.15.78:0/3989123548 learned_addr learned my addr 172.21.15.78:0/3989123548 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:45.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.345+0000 7effd6d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effd8103d60 0x7effd819f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:45.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.345+0000 7effd5d9b700 1 --2- 172.21.15.78:0/3989123548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effd81033c0 0x7effd8198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:45.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.345+0000 7effd659c700 1 -- 172.21.15.78:0/3989123548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effd81033c0 msgr2=0x7effd8198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:45.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.345+0000 7effd659c700 1 --2- 172.21.15.78:0/3989123548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effd81033c0 0x7effd8198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:45.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.345+0000 7effd659c700 1 -- 172.21.15.78:0/3989123548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effd8103d60 msgr2=0x7effd819f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:45.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.345+0000 7effd659c700 1 --2- 172.21.15.78:0/3989123548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effd8103d60 0x7effd819f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:45.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.345+0000 7effd659c700 1 -- 172.21.15.78:0/3989123548 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd81a0020 con 0x7effd81021c0 2024-03-24T00:20:45.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.345+0000 7effd659c700 1 --2- 172.21.15.78:0/3989123548 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effd81021c0 0x7effd81983c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7effcc00e970 tx=0x7effcc00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:45.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.345+0000 7effc77fe700 1 -- 172.21.15.78:0/3989123548 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effcc004220 con 0x7effd81021c0 2024-03-24T00:20:45.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.345+0000 7effc77fe700 1 -- 172.21.15.78:0/3989123548 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7effcc0043c0 con 0x7effd81021c0 2024-03-24T00:20:45.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.345+0000 7effc77fe700 1 -- 172.21.15.78:0/3989123548 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effcc003df0 con 0x7effd81021c0 2024-03-24T00:20:45.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.345+0000 7effdc8de700 1 -- 172.21.15.78:0/3989123548 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effd81a0310 con 0x7effd81021c0 2024-03-24T00:20:45.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.345+0000 7effdc8de700 1 -- 172.21.15.78:0/3989123548 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effd81a08a0 con 0x7effd81021c0 2024-03-24T00:20:45.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.346+0000 7effdc8de700 1 -- 172.21.15.78:0/3989123548 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effd8068e50 con 0x7effd81021c0 2024-03-24T00:20:45.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.347+0000 7effc77fe700 1 -- 172.21.15.78:0/3989123548 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7effcc00c7e0 con 0x7effd81021c0 2024-03-24T00:20:45.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.348+0000 7effc77fe700 1 --2- 172.21.15.78:0/3989123548 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7effc00a2910 0x7effc00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:45.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.348+0000 7effd5d9b700 1 --2- 172.21.15.78:0/3989123548 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7effc00a2910 0x7effc00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:45.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.348+0000 7effc77fe700 1 -- 172.21.15.78:0/3989123548 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7effcc0cef20 con 0x7effd81021c0 2024-03-24T00:20:45.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.348+0000 7effd5d9b700 1 --2- 172.21.15.78:0/3989123548 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7effc00a2910 0x7effc00a4dd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7effc8000c00 tx=0x7effc8005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:45.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.351+0000 7effc77fe700 1 -- 172.21.15.78:0/3989123548 <== mon.2 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7effcc09bc30 con 0x7effd81021c0 2024-03-24T00:20:45.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.558+0000 7effdc8de700 1 -- 172.21.15.78:0/3989123548 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7effd80625d0 con 0x7effc00a2910 2024-03-24T00:20:45.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.560+0000 7effc77fe700 1 -- 172.21.15.78:0/3989123548 <== mgr.14150 v2:172.21.15.78:6800/1983101026 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+532 (secure 0 0 0) 0x7effd80625d0 con 0x7effc00a2910 2024-03-24T00:20:45.560 INFO:teuthology.orchestra.run.smithi078.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-24T00:20:45.560 INFO:teuthology.orchestra.run.smithi078.stdout:smithi078 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400QR400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-24T00:20:45.560 INFO:teuthology.orchestra.run.smithi078.stdout:smithi106 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204013R400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-24T00:20:45.560 INFO:teuthology.orchestra.run.smithi078.stdout:smithi107 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400NK400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-24T00:20:45.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.562+0000 7effdc8de700 1 -- 172.21.15.78:0/3989123548 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7effc00a2910 msgr2=0x7effc00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:45.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.562+0000 7effdc8de700 1 --2- 172.21.15.78:0/3989123548 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7effc00a2910 0x7effc00a4dd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7effc8000c00 tx=0x7effc8005fb0 comp rx=0 tx=0).stop 2024-03-24T00:20:45.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.562+0000 7effdc8de700 1 -- 172.21.15.78:0/3989123548 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effd81021c0 msgr2=0x7effd81983c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:45.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.562+0000 7effdc8de700 1 --2- 172.21.15.78:0/3989123548 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effd81021c0 0x7effd81983c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7effcc00e970 tx=0x7effcc00ec80 comp rx=0 tx=0).stop 2024-03-24T00:20:45.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.562+0000 7effdc8de700 1 -- 172.21.15.78:0/3989123548 shutdown_connections 2024-03-24T00:20:45.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.562+0000 7effdc8de700 1 --2- 172.21.15.78:0/3989123548 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7effc00a2910 0x7effc00a4dd0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:45.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.562+0000 7effdc8de700 1 --2- 172.21.15.78:0/3989123548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effd8103d60 0x7effd819f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:45.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.562+0000 7effdc8de700 1 --2- 172.21.15.78:0/3989123548 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effd81021c0 0x7effd81983c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:45.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.562+0000 7effdc8de700 1 --2- 172.21.15.78:0/3989123548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effd81033c0 0x7effd8198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:45.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.562+0000 7effdc8de700 1 -- 172.21.15.78:0/3989123548 >> 172.21.15.78:0/3989123548 conn(0x7effd80fd7a0 msgr2=0x7effd8106740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:45.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.562+0000 7effdc8de700 1 -- 172.21.15.78:0/3989123548 shutdown_connections 2024-03-24T00:20:45.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:45.562+0000 7effdc8de700 1 -- 172.21.15.78:0/3989123548 wait complete. 2024-03-24T00:20:45.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:45 smithi078 ceph-mon[32429]: pgmap v355: 129 pgs: 1 activating+degraded, 128 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 639 B/s wr, 8 op/s; 1/194 objects degraded (0.515%) 2024-03-24T00:20:45.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:45 smithi078 ceph-mon[32429]: Health check failed: Degraded data redundancy: 1/194 objects degraded (0.515%), 1 pg degraded (PG_DEGRADED) 2024-03-24T00:20:45.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:45 smithi107 ceph-mon[36485]: pgmap v355: 129 pgs: 1 activating+degraded, 128 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 639 B/s wr, 8 op/s; 1/194 objects degraded (0.515%) 2024-03-24T00:20:45.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:45 smithi107 ceph-mon[36485]: Health check failed: Degraded data redundancy: 1/194 objects degraded (0.515%), 1 pg degraded (PG_DEGRADED) 2024-03-24T00:20:45.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:45 smithi106 ceph-mon[36505]: pgmap v355: 129 pgs: 1 activating+degraded, 128 active+clean; 641 KiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 639 B/s wr, 8 op/s; 1/194 objects degraded (0.515%) 2024-03-24T00:20:45.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:45 smithi106 ceph-mon[36505]: Health check failed: Degraded data redundancy: 1/194 objects degraded (0.515%), 1 pg degraded (PG_DEGRADED) 2024-03-24T00:20:46.212 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T00:20:46.221 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2024-03-24T00:20:46.221 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-24T00:20:46.467 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:20:47.826 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:47 smithi078 ceph-mon[32429]: from='client.25127 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:20:47.826 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:47 smithi078 ceph-mon[32429]: pgmap v356: 129 pgs: 1 activating+degraded, 128 active+clean; 641 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 639 B/s wr, 6 op/s; 1/194 objects degraded (0.515%) 2024-03-24T00:20:47.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:47 smithi107 ceph-mon[36485]: from='client.25127 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:20:47.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:47 smithi107 ceph-mon[36485]: pgmap v356: 129 pgs: 1 activating+degraded, 128 active+clean; 641 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 639 B/s wr, 6 op/s; 1/194 objects degraded (0.515%) 2024-03-24T00:20:47.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:47 smithi106 ceph-mon[36505]: from='client.25127 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:20:47.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:47 smithi106 ceph-mon[36505]: pgmap v356: 129 pgs: 1 activating+degraded, 128 active+clean; 641 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 639 B/s wr, 6 op/s; 1/194 objects degraded (0.515%) 2024-03-24T00:20:48.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.209+0000 7f438b5a2700 1 -- 172.21.15.78:0/1234336548 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4384101cd0 msgr2=0x7f4384109570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:48.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.209+0000 7f438b5a2700 1 --2- 172.21.15.78:0/1234336548 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4384101cd0 0x7f4384109570 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f437c009f90 tx=0x7f437c00b330 comp rx=0 tx=0).stop 2024-03-24T00:20:48.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.209+0000 7f438b5a2700 1 -- 172.21.15.78:0/1234336548 shutdown_connections 2024-03-24T00:20:48.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.209+0000 7f438b5a2700 1 --2- 172.21.15.78:0/1234336548 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4384101cd0 0x7f4384109570 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:48.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.209+0000 7f438b5a2700 1 --2- 172.21.15.78:0/1234336548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4384102740 0x7f4384102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:48.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.209+0000 7f438b5a2700 1 --2- 172.21.15.78:0/1234336548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4384101310 0x7f4384101790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:48.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.209+0000 7f438b5a2700 1 -- 172.21.15.78:0/1234336548 >> 172.21.15.78:0/1234336548 conn(0x7f43840fbe00 msgr2=0x7f43840fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:48.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.209+0000 7f438b5a2700 1 -- 172.21.15.78:0/1234336548 shutdown_connections 2024-03-24T00:20:48.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.209+0000 7f438b5a2700 1 -- 172.21.15.78:0/1234336548 wait complete. 2024-03-24T00:20:48.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.210+0000 7f438b5a2700 1 Processor -- start 2024-03-24T00:20:48.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.210+0000 7f438b5a2700 1 -- start start 2024-03-24T00:20:48.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.210+0000 7f438b5a2700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4384101310 0x7f4384197230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:48.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.210+0000 7f438b5a2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4384101cd0 0x7f4384197770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:48.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.210+0000 7f438b5a2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4384102740 0x7f438419e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:48.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.210+0000 7f438b5a2700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438410cce0 con 0x7f4384102740 2024-03-24T00:20:48.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.210+0000 7f438b5a2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438410cb60 con 0x7f4384101cd0 2024-03-24T00:20:48.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.210+0000 7f438b5a2700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438410ce60 con 0x7f4384101310 2024-03-24T00:20:48.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.210+0000 7f4389b3f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4384102740 0x7f438419e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:48.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.210+0000 7f438933e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4384101310 0x7f4384197230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:48.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.210+0000 7f4389b3f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4384102740 0x7f438419e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37860/0 (socket says 172.21.15.78:37860) 2024-03-24T00:20:48.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.210+0000 7f4388b3d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4384101cd0 0x7f4384197770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:48.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.210+0000 7f438933e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4384101310 0x7f4384197230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.78:55776/0 (socket says 172.21.15.78:55776) 2024-03-24T00:20:48.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.210+0000 7f4389b3f700 1 -- 172.21.15.78:0/3205316654 learned_addr learned my addr 172.21.15.78:0/3205316654 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:48.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.211+0000 7f438933e700 1 -- 172.21.15.78:0/3205316654 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4384101cd0 msgr2=0x7f4384197770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:48.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.211+0000 7f438933e700 1 --2- 172.21.15.78:0/3205316654 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4384101cd0 0x7f4384197770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:48.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.211+0000 7f438933e700 1 -- 172.21.15.78:0/3205316654 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4384102740 msgr2=0x7f438419e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:48.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.211+0000 7f438933e700 1 --2- 172.21.15.78:0/3205316654 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4384102740 0x7f438419e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:48.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.211+0000 7f438933e700 1 -- 172.21.15.78:0/3205316654 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f438419ee90 con 0x7f4384101310 2024-03-24T00:20:48.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.211+0000 7f4389b3f700 1 --2- 172.21.15.78:0/3205316654 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4384102740 0x7f438419e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:48.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.211+0000 7f438933e700 1 --2- 172.21.15.78:0/3205316654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4384101310 0x7f4384197230 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f437400ea20 tx=0x7f437400ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:48.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.211+0000 7f437a7fc700 1 -- 172.21.15.78:0/3205316654 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437400cbd0 con 0x7f4384101310 2024-03-24T00:20:48.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.211+0000 7f437a7fc700 1 -- 172.21.15.78:0/3205316654 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4374004500 con 0x7f4384101310 2024-03-24T00:20:48.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.211+0000 7f437a7fc700 1 -- 172.21.15.78:0/3205316654 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4374005020 con 0x7f4384101310 2024-03-24T00:20:48.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.211+0000 7f438b5a2700 1 -- 172.21.15.78:0/3205316654 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f438419f180 con 0x7f4384101310 2024-03-24T00:20:48.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.211+0000 7f438b5a2700 1 -- 172.21.15.78:0/3205316654 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f438419f6c0 con 0x7f4384101310 2024-03-24T00:20:48.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.212+0000 7f438b5a2700 1 -- 172.21.15.78:0/3205316654 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4384068e50 con 0x7f4384101310 2024-03-24T00:20:48.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.213+0000 7f437a7fc700 1 -- 172.21.15.78:0/3205316654 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4374005200 con 0x7f4384101310 2024-03-24T00:20:48.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.214+0000 7f437a7fc700 1 --2- 172.21.15.78:0/3205316654 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f43700a2910 0x7f43700a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:48.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.214+0000 7f4388b3d700 1 --2- 172.21.15.78:0/3205316654 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f43700a2910 0x7f43700a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:48.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.214+0000 7f437a7fc700 1 -- 172.21.15.78:0/3205316654 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f43740cec10 con 0x7f4384101310 2024-03-24T00:20:48.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.214+0000 7f4388b3d700 1 --2- 172.21.15.78:0/3205316654 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f43700a2910 0x7f43700a4dd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4380009750 tx=0x7f4380009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:48.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.218+0000 7f437a7fc700 1 -- 172.21.15.78:0/3205316654 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f437409b920 con 0x7f4384101310 2024-03-24T00:20:48.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.497+0000 7f438b5a2700 1 -- 172.21.15.78:0/3205316654 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f438402de50 con 0x7f4384101310 2024-03-24T00:20:48.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.497+0000 7f437a7fc700 1 -- 172.21.15.78:0/3205316654 <== mon.1 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 23 v23) v1 ==== 76+0+3002 (secure 0 0 0) 0x7f437400ee60 con 0x7f4384101310 2024-03-24T00:20:48.498 INFO:teuthology.orchestra.run.smithi078.stdout:e23 2024-03-24T00:20:48.498 INFO:teuthology.orchestra.run.smithi078.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T00:20:48.498 INFO:teuthology.orchestra.run.smithi078.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-24T00:20:48.498 INFO:teuthology.orchestra.run.smithi078.stdout:legacy client fscid: 1 2024-03-24T00:20:48.498 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:20:48.498 INFO:teuthology.orchestra.run.smithi078.stdout:Filesystem 'cephfs' (1) 2024-03-24T00:20:48.498 INFO:teuthology.orchestra.run.smithi078.stdout:fs_name cephfs 2024-03-24T00:20:48.498 INFO:teuthology.orchestra.run.smithi078.stdout:epoch 23 2024-03-24T00:20:48.498 INFO:teuthology.orchestra.run.smithi078.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-24T00:20:48.498 INFO:teuthology.orchestra.run.smithi078.stdout:created 2024-03-24T00:18:43.592215+0000 2024-03-24T00:20:48.498 INFO:teuthology.orchestra.run.smithi078.stdout:modified 2024-03-24T00:19:08.871384+0000 2024-03-24T00:20:48.498 INFO:teuthology.orchestra.run.smithi078.stdout:tableserver 0 2024-03-24T00:20:48.499 INFO:teuthology.orchestra.run.smithi078.stdout:root 0 2024-03-24T00:20:48.499 INFO:teuthology.orchestra.run.smithi078.stdout:session_timeout 300 2024-03-24T00:20:48.499 INFO:teuthology.orchestra.run.smithi078.stdout:session_autoclose 300 2024-03-24T00:20:48.499 INFO:teuthology.orchestra.run.smithi078.stdout:max_file_size 1099511627776 2024-03-24T00:20:48.499 INFO:teuthology.orchestra.run.smithi078.stdout:max_xattr_size 65536 2024-03-24T00:20:48.499 INFO:teuthology.orchestra.run.smithi078.stdout:required_client_features {} 2024-03-24T00:20:48.499 INFO:teuthology.orchestra.run.smithi078.stdout:last_failure 0 2024-03-24T00:20:48.499 INFO:teuthology.orchestra.run.smithi078.stdout:last_failure_osd_epoch 0 2024-03-24T00:20:48.499 INFO:teuthology.orchestra.run.smithi078.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-24T00:20:48.499 INFO:teuthology.orchestra.run.smithi078.stdout:max_mds 3 2024-03-24T00:20:48.499 INFO:teuthology.orchestra.run.smithi078.stdout:in 0,1,2 2024-03-24T00:20:48.499 INFO:teuthology.orchestra.run.smithi078.stdout:up {0=24500,1=24481,2=24464} 2024-03-24T00:20:48.499 INFO:teuthology.orchestra.run.smithi078.stdout:failed 2024-03-24T00:20:48.500 INFO:teuthology.orchestra.run.smithi078.stdout:damaged 2024-03-24T00:20:48.500 INFO:teuthology.orchestra.run.smithi078.stdout:stopped 2024-03-24T00:20:48.500 INFO:teuthology.orchestra.run.smithi078.stdout:data_pools [3,4] 2024-03-24T00:20:48.500 INFO:teuthology.orchestra.run.smithi078.stdout:metadata_pool 2 2024-03-24T00:20:48.500 INFO:teuthology.orchestra.run.smithi078.stdout:inline_data disabled 2024-03-24T00:20:48.500 INFO:teuthology.orchestra.run.smithi078.stdout:balancer 2024-03-24T00:20:48.500 INFO:teuthology.orchestra.run.smithi078.stdout:bal_rank_mask -1 2024-03-24T00:20:48.500 INFO:teuthology.orchestra.run.smithi078.stdout:standby_count_wanted 0 2024-03-24T00:20:48.500 INFO:teuthology.orchestra.run.smithi078.stdout:qdb_cluster leader: 24500 members: 24481,24500,24464 2024-03-24T00:20:48.500 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.b{0:24500} state up:active seq 3 addr [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:20:48.500 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.c{0:24454} state up:standby-replay seq 1 addr [v2:172.21.15.107:6836/2939571540,v1:172.21.15.107:6837/2939571540] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:20:48.500 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.i{1:24481} state up:active seq 11 addr [v2:172.21.15.107:6838/117119462,v1:172.21.15.107:6839/117119462] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:20:48.500 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.f{1:24442} state up:standby-replay seq 1 addr [v2:172.21.15.107:6834/108936439,v1:172.21.15.107:6835/108936439] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:20:48.500 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.h{2:24464} state up:active seq 12 addr [v2:172.21.15.106:6836/2090968633,v1:172.21.15.106:6837/2090968633] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:20:48.501 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.e{2:24437} state up:standby-replay seq 1 addr [v2:172.21.15.106:6834/2231611477,v1:172.21.15.106:6835/2231611477] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:20:48.501 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:20:48.501 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:20:48.501 INFO:teuthology.orchestra.run.smithi078.stdout:Standby daemons: 2024-03-24T00:20:48.501 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:20:48.501 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.j{-1:14559} state up:standby seq 1 addr [v2:172.21.15.78:6834/1416685123,v1:172.21.15.78:6835/1416685123] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:20:48.501 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.d{-1:14571} state up:standby seq 1 addr [v2:172.21.15.78:6836/1071274623,v1:172.21.15.78:6837/1071274623] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:20:48.501 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.g{-1:14634} state up:standby seq 1 addr [v2:172.21.15.78:6838/2758357551,v1:172.21.15.78:6839/2758357551] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:20:48.501 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.a{-1:14646} state up:standby seq 1 addr [v2:172.21.15.78:6840/3424505910,v1:172.21.15.78:6841/3424505910] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:20:48.501 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.l{-1:24397} state up:standby seq 1 addr [v2:172.21.15.107:6832/3086318443,v1:172.21.15.107:6833/3086318443] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:20:48.501 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.k{-1:24407} state up:standby seq 1 addr [v2:172.21.15.106:6832/2367018868,v1:172.21.15.106:6833/2367018868] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:20:48.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.500+0000 7f438b5a2700 1 -- 172.21.15.78:0/3205316654 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f43700a2910 msgr2=0x7f43700a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:48.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.500+0000 7f438b5a2700 1 --2- 172.21.15.78:0/3205316654 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f43700a2910 0x7f43700a4dd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4380009750 tx=0x7f4380009310 comp rx=0 tx=0).stop 2024-03-24T00:20:48.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.500+0000 7f438b5a2700 1 -- 172.21.15.78:0/3205316654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4384101310 msgr2=0x7f4384197230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:48.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.500+0000 7f438b5a2700 1 --2- 172.21.15.78:0/3205316654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4384101310 0x7f4384197230 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f437400ea20 tx=0x7f437400ed30 comp rx=0 tx=0).stop 2024-03-24T00:20:48.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.500+0000 7f438b5a2700 1 -- 172.21.15.78:0/3205316654 shutdown_connections 2024-03-24T00:20:48.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.500+0000 7f438b5a2700 1 --2- 172.21.15.78:0/3205316654 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f43700a2910 0x7f43700a4dd0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:48.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.500+0000 7f438b5a2700 1 --2- 172.21.15.78:0/3205316654 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4384102740 0x7f438419e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:48.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.500+0000 7f438b5a2700 1 --2- 172.21.15.78:0/3205316654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4384101310 0x7f4384197230 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:48.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.500+0000 7f438b5a2700 1 --2- 172.21.15.78:0/3205316654 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4384101cd0 0x7f4384197770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:48.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.500+0000 7f438b5a2700 1 -- 172.21.15.78:0/3205316654 >> 172.21.15.78:0/3205316654 conn(0x7f43840fbe00 msgr2=0x7f4384105620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:48.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.500+0000 7f438b5a2700 1 -- 172.21.15.78:0/3205316654 shutdown_connections 2024-03-24T00:20:48.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:48.500+0000 7f438b5a2700 1 -- 172.21.15.78:0/3205316654 wait complete. 2024-03-24T00:20:48.503 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 23 2024-03-24T00:20:48.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:48 smithi078 ceph-mon[32429]: pgmap v357: 129 pgs: 129 active+clean; 641 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 639 B/s wr, 6 op/s; 14 B/s, 0 objects/s recovering 2024-03-24T00:20:48.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:48 smithi078 ceph-mon[32429]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/194 objects degraded (0.515%), 1 pg degraded) 2024-03-24T00:20:48.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:48 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:20:48.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:48 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3205316654' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-24T00:20:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:48 smithi107 ceph-mon[36485]: pgmap v357: 129 pgs: 129 active+clean; 641 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 639 B/s wr, 6 op/s; 14 B/s, 0 objects/s recovering 2024-03-24T00:20:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:48 smithi107 ceph-mon[36485]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/194 objects degraded (0.515%), 1 pg degraded) 2024-03-24T00:20:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:48 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:20:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:48 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3205316654' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-24T00:20:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:48 smithi106 ceph-mon[36505]: pgmap v357: 129 pgs: 129 active+clean; 641 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 639 B/s wr, 6 op/s; 14 B/s, 0 objects/s recovering 2024-03-24T00:20:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:48 smithi106 ceph-mon[36505]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/194 objects degraded (0.515%), 1 pg degraded) 2024-03-24T00:20:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:48 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:20:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:48 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3205316654' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-24T00:20:49.295 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-24T00:20:49.539 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:20:51.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.202+0000 7fe841fee700 1 -- 172.21.15.78:0/1816845887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe83c102c90 msgr2=0x7fe83c109540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:51.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.202+0000 7fe841fee700 1 --2- 172.21.15.78:0/1816845887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe83c102c90 0x7fe83c109540 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe83400b0d0 tx=0x7fe83400b3e0 comp rx=0 tx=0).stop 2024-03-24T00:20:51.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.202+0000 7fe841fee700 1 -- 172.21.15.78:0/1816845887 shutdown_connections 2024-03-24T00:20:51.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.202+0000 7fe841fee700 1 --2- 172.21.15.78:0/1816845887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe83c102c90 0x7fe83c109540 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:51.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.202+0000 7fe841fee700 1 --2- 172.21.15.78:0/1816845887 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe83c1010f0 0x7fe83c101510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:51.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.202+0000 7fe841fee700 1 --2- 172.21.15.78:0/1816845887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe83c1022f0 0x7fe83c102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:51.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.202+0000 7fe841fee700 1 -- 172.21.15.78:0/1816845887 >> 172.21.15.78:0/1816845887 conn(0x7fe83c0fc6b0 msgr2=0x7fe83c0fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:51.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.202+0000 7fe841fee700 1 -- 172.21.15.78:0/1816845887 shutdown_connections 2024-03-24T00:20:51.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.202+0000 7fe841fee700 1 -- 172.21.15.78:0/1816845887 wait complete. 2024-03-24T00:20:51.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.203+0000 7fe841fee700 1 Processor -- start 2024-03-24T00:20:51.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.203+0000 7fe841fee700 1 -- start start 2024-03-24T00:20:51.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.203+0000 7fe841fee700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe83c1010f0 0x7fe83c06b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:51.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.203+0000 7fe841fee700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe83c1022f0 0x7fe83c19d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:51.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.203+0000 7fe841fee700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe83c102c90 0x7fe83c19f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:51.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.203+0000 7fe841fee700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe83c10cce0 con 0x7fe83c1022f0 2024-03-24T00:20:51.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.203+0000 7fe841fee700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe83c10cb60 con 0x7fe83c102c90 2024-03-24T00:20:51.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.203+0000 7fe841fee700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe83c10ce60 con 0x7fe83c1010f0 2024-03-24T00:20:51.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.203+0000 7fe832ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe83c1022f0 0x7fe83c19d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:51.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.203+0000 7fe832ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe83c1022f0 0x7fe83c19d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37878/0 (socket says 172.21.15.78:37878) 2024-03-24T00:20:51.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.203+0000 7fe832ffd700 1 -- 172.21.15.78:0/2677657470 learned_addr learned my addr 172.21.15.78:0/2677657470 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:51.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.203+0000 7fe83b7fe700 1 --2- 172.21.15.78:0/2677657470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe83c1010f0 0x7fe83c06b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:51.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.203+0000 7fe83bfff700 1 --2- 172.21.15.78:0/2677657470 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe83c102c90 0x7fe83c19f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:51.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.203+0000 7fe832ffd700 1 -- 172.21.15.78:0/2677657470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe83c1010f0 msgr2=0x7fe83c06b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:51.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.203+0000 7fe832ffd700 1 --2- 172.21.15.78:0/2677657470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe83c1010f0 0x7fe83c06b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:51.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.203+0000 7fe832ffd700 1 -- 172.21.15.78:0/2677657470 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe83c102c90 msgr2=0x7fe83c19f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:51.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.203+0000 7fe832ffd700 1 --2- 172.21.15.78:0/2677657470 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe83c102c90 0x7fe83c19f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:51.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.203+0000 7fe832ffd700 1 -- 172.21.15.78:0/2677657470 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe83c19ffd0 con 0x7fe83c1022f0 2024-03-24T00:20:51.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.203+0000 7fe83bfff700 1 --2- 172.21.15.78:0/2677657470 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe83c102c90 0x7fe83c19f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:51.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.204+0000 7fe832ffd700 1 --2- 172.21.15.78:0/2677657470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe83c1022f0 0x7fe83c19d0c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe82400e9c0 tx=0x7fe82400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:51.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.204+0000 7fe8397fa700 1 -- 172.21.15.78:0/2677657470 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82400cbd0 con 0x7fe83c1022f0 2024-03-24T00:20:51.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.204+0000 7fe841fee700 1 -- 172.21.15.78:0/2677657470 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe83c1a02c0 con 0x7fe83c1022f0 2024-03-24T00:20:51.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.204+0000 7fe8397fa700 1 -- 172.21.15.78:0/2677657470 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe82400ee60 con 0x7fe83c1022f0 2024-03-24T00:20:51.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.204+0000 7fe8397fa700 1 -- 172.21.15.78:0/2677657470 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe824005630 con 0x7fe83c1022f0 2024-03-24T00:20:51.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.204+0000 7fe841fee700 1 -- 172.21.15.78:0/2677657470 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe83c1a0800 con 0x7fe83c1022f0 2024-03-24T00:20:51.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.205+0000 7fe841fee700 1 -- 172.21.15.78:0/2677657470 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe83c068e50 con 0x7fe83c1022f0 2024-03-24T00:20:51.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.206+0000 7fe8397fa700 1 -- 172.21.15.78:0/2677657470 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe824004750 con 0x7fe83c1022f0 2024-03-24T00:20:51.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.207+0000 7fe8397fa700 1 --2- 172.21.15.78:0/2677657470 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe8280a2910 0x7fe8280a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:51.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.207+0000 7fe83b7fe700 1 --2- 172.21.15.78:0/2677657470 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe8280a2910 0x7fe8280a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:51.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.207+0000 7fe8397fa700 1 -- 172.21.15.78:0/2677657470 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7fe8240cefd0 con 0x7fe83c1022f0 2024-03-24T00:20:51.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.207+0000 7fe83b7fe700 1 --2- 172.21.15.78:0/2677657470 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe8280a2910 0x7fe8280a4dd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe83c102150 tx=0x7fe82c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:51.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.212+0000 7fe8397fa700 1 -- 172.21.15.78:0/2677657470 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe82409bce0 con 0x7fe83c1022f0 2024-03-24T00:20:51.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.411+0000 7fe841fee700 1 -- 172.21.15.78:0/2677657470 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7fe83c062da0 con 0x7fe83c1022f0 2024-03-24T00:20:51.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.412+0000 7fe8397fa700 1 -- 172.21.15.78:0/2677657470 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v93) v1 ==== 56+0+5656 (secure 0 0 0) 0x7fe82409b5e0 con 0x7fe83c1022f0 2024-03-24T00:20:51.413 INFO:teuthology.orchestra.run.smithi078.stdout:epoch 93 2024-03-24T00:20:51.413 INFO:teuthology.orchestra.run.smithi078.stdout:fsid c7e2806a-e972-11ee-95cd-87774f69a715 2024-03-24T00:20:51.414 INFO:teuthology.orchestra.run.smithi078.stdout:created 2024-03-24T00:10:44.134563+0000 2024-03-24T00:20:51.414 INFO:teuthology.orchestra.run.smithi078.stdout:modified 2024-03-24T00:20:42.446867+0000 2024-03-24T00:20:51.414 INFO:teuthology.orchestra.run.smithi078.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-24T00:20:51.414 INFO:teuthology.orchestra.run.smithi078.stdout:crush_version 27 2024-03-24T00:20:51.414 INFO:teuthology.orchestra.run.smithi078.stdout:full_ratio 0.95 2024-03-24T00:20:51.414 INFO:teuthology.orchestra.run.smithi078.stdout:backfillfull_ratio 0.9 2024-03-24T00:20:51.414 INFO:teuthology.orchestra.run.smithi078.stdout:nearfull_ratio 0.85 2024-03-24T00:20:51.414 INFO:teuthology.orchestra.run.smithi078.stdout:require_min_compat_client luminous 2024-03-24T00:20:51.414 INFO:teuthology.orchestra.run.smithi078.stdout:min_compat_client luminous 2024-03-24T00:20:51.415 INFO:teuthology.orchestra.run.smithi078.stdout:require_osd_release squid 2024-03-24T00:20:51.415 INFO:teuthology.orchestra.run.smithi078.stdout:stretch_mode_enabled false 2024-03-24T00:20:51.415 INFO:teuthology.orchestra.run.smithi078.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-24T00:20:51.415 INFO:teuthology.orchestra.run.smithi078.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2024-03-24T00:20:51.415 INFO:teuthology.orchestra.run.smithi078.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-03-24T00:20:51.415 INFO:teuthology.orchestra.run.smithi078.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-03-24T00:20:51.415 INFO:teuthology.orchestra.run.smithi078.stdout:max_osd 12 2024-03-24T00:20:51.415 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6802/1816879397,v1:172.21.15.78:6803/1816879397] [v2:172.21.15.78:6804/1816879397,v1:172.21.15.78:6805/1816879397] exists,up b950388a-c437-426b-b703-3f8ae821dd0d 2024-03-24T00:20:51.415 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 up in weight 1 up_from 13 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.106:6800/1647915612,v1:172.21.15.106:6801/1647915612] [v2:172.21.15.106:6802/1647915612,v1:172.21.15.106:6803/1647915612] exists,up b8d64714-dd65-400f-b3f4-673ab14e1365 2024-03-24T00:20:51.416 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 up in weight 1 up_from 18 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6800/2010770435,v1:172.21.15.107:6801/2010770435] [v2:172.21.15.107:6802/2010770435,v1:172.21.15.107:6803/2010770435] exists,up fcfc4dd1-aaf8-4131-a023-5e832212158d 2024-03-24T00:20:51.416 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6810/1912251346,v1:172.21.15.78:6811/1912251346] [v2:172.21.15.78:6812/1912251346,v1:172.21.15.78:6813/1912251346] exists,up 47c718ab-a0f4-4e52-83c5-b89386b262d0 2024-03-24T00:20:51.416 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 up in weight 1 up_from 33 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.106:6808/374641205,v1:172.21.15.106:6809/374641205] [v2:172.21.15.106:6810/374641205,v1:172.21.15.106:6811/374641205] exists,up 7dad64dd-677e-4c1c-9638-e4f4f2195983 2024-03-24T00:20:51.416 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 up in weight 1 up_from 38 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6808/3272806073,v1:172.21.15.107:6809/3272806073] [v2:172.21.15.107:6810/3272806073,v1:172.21.15.107:6811/3272806073] exists,up bbd524e5-328d-4309-8877-1149cbe110dd 2024-03-24T00:20:51.416 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 up in weight 1 up_from 46 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6818/3091148508,v1:172.21.15.78:6819/3091148508] [v2:172.21.15.78:6820/3091148508,v1:172.21.15.78:6821/3091148508] exists,up 5696389a-918f-4ea8-b20d-4e2a0d4cb3bf 2024-03-24T00:20:51.416 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 up in weight 1 up_from 51 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.106:6816/4262103080,v1:172.21.15.106:6817/4262103080] [v2:172.21.15.106:6818/4262103080,v1:172.21.15.106:6819/4262103080] exists,up f6b764e8-ed4e-4c9a-a85b-94af6dc6b924 2024-03-24T00:20:51.416 INFO:teuthology.orchestra.run.smithi078.stdout:osd.8 up in weight 1 up_from 59 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6816/4032799356,v1:172.21.15.107:6817/4032799356] [v2:172.21.15.107:6818/4032799356,v1:172.21.15.107:6819/4032799356] exists,up ba16039f-8093-4fa0-87ad-e28b7d142b23 2024-03-24T00:20:51.416 INFO:teuthology.orchestra.run.smithi078.stdout:osd.9 up in weight 1 up_from 67 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6826/214023431,v1:172.21.15.78:6827/214023431] [v2:172.21.15.78:6828/214023431,v1:172.21.15.78:6829/214023431] exists,up 651357f2-0669-42e6-aa91-1d5c02d00c5c 2024-03-24T00:20:51.416 INFO:teuthology.orchestra.run.smithi078.stdout:osd.10 up in weight 1 up_from 72 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.106:6824/400029804,v1:172.21.15.106:6825/400029804] [v2:172.21.15.106:6826/400029804,v1:172.21.15.106:6827/400029804] exists,up 2375119d-346e-426c-8a65-b64af71ad63a 2024-03-24T00:20:51.416 INFO:teuthology.orchestra.run.smithi078.stdout:osd.11 up in weight 1 up_from 80 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6824/4191916230,v1:172.21.15.107:6825/4191916230] [v2:172.21.15.107:6826/4191916230,v1:172.21.15.107:6827/4191916230] exists,up b6b609d9-dd78-4c01-ae66-ba2a662ab22f 2024-03-24T00:20:51.416 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 2.6 [8,11] 2024-03-24T00:20:51.417 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 2.13 [2,11] 2024-03-24T00:20:51.417 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 2.14 [2,4] 2024-03-24T00:20:51.417 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 2.1f [8,4] 2024-03-24T00:20:51.417 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 3.0 [4,8] 2024-03-24T00:20:51.417 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 3.13 [4,9] 2024-03-24T00:20:51.417 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 3.33 [5,9] 2024-03-24T00:20:51.417 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 3.3c [4,9] 2024-03-24T00:20:51.417 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 4.5 [5,10] 2024-03-24T00:20:51.417 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 4.e [9,11] 2024-03-24T00:20:51.417 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 4.17 [9,8] 2024-03-24T00:20:51.417 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 4.1c [9,8] 2024-03-24T00:20:51.417 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/1982202541 expires 2024-03-25T00:11:40.199569+0000 2024-03-24T00:20:51.418 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/1933054339 expires 2024-03-25T00:11:40.199569+0000 2024-03-24T00:20:51.418 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:6801/2748920178 expires 2024-03-25T00:11:40.199569+0000 2024-03-24T00:20:51.418 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:6800/2748920178 expires 2024-03-25T00:11:40.199569+0000 2024-03-24T00:20:51.418 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/465691596 expires 2024-03-25T00:11:14.637881+0000 2024-03-24T00:20:51.418 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:6801/2913847518 expires 2024-03-25T00:11:14.637881+0000 2024-03-24T00:20:51.418 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/3962523609 expires 2024-03-25T00:11:14.637881+0000 2024-03-24T00:20:51.418 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:6800/2913847518 expires 2024-03-25T00:11:14.637881+0000 2024-03-24T00:20:51.418 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/1502752625 expires 2024-03-25T00:11:14.637881+0000 2024-03-24T00:20:51.418 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/2236879401 expires 2024-03-25T00:11:40.199569+0000 2024-03-24T00:20:51.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.414+0000 7fe841fee700 1 -- 172.21.15.78:0/2677657470 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe8280a2910 msgr2=0x7fe8280a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:51.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.414+0000 7fe841fee700 1 --2- 172.21.15.78:0/2677657470 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe8280a2910 0x7fe8280a4dd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe83c102150 tx=0x7fe82c009260 comp rx=0 tx=0).stop 2024-03-24T00:20:51.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.414+0000 7fe841fee700 1 -- 172.21.15.78:0/2677657470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe83c1022f0 msgr2=0x7fe83c19d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:51.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.414+0000 7fe841fee700 1 --2- 172.21.15.78:0/2677657470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe83c1022f0 0x7fe83c19d0c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe82400e9c0 tx=0x7fe82400ecd0 comp rx=0 tx=0).stop 2024-03-24T00:20:51.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.415+0000 7fe841fee700 1 -- 172.21.15.78:0/2677657470 shutdown_connections 2024-03-24T00:20:51.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.415+0000 7fe841fee700 1 --2- 172.21.15.78:0/2677657470 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe8280a2910 0x7fe8280a4dd0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:51.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.415+0000 7fe841fee700 1 --2- 172.21.15.78:0/2677657470 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe83c102c90 0x7fe83c19f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:51.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.415+0000 7fe841fee700 1 --2- 172.21.15.78:0/2677657470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe83c1010f0 0x7fe83c06b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:51.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.415+0000 7fe841fee700 1 --2- 172.21.15.78:0/2677657470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe83c1022f0 0x7fe83c19d0c0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:51.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.415+0000 7fe841fee700 1 -- 172.21.15.78:0/2677657470 >> 172.21.15.78:0/2677657470 conn(0x7fe83c0fc6b0 msgr2=0x7fe83c105590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:51.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.415+0000 7fe841fee700 1 -- 172.21.15.78:0/2677657470 shutdown_connections 2024-03-24T00:20:51.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:51.415+0000 7fe841fee700 1 -- 172.21.15.78:0/2677657470 wait complete. 2024-03-24T00:20:51.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:51 smithi078 ceph-mon[32429]: pgmap v358: 129 pgs: 129 active+clean; 641 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 461 B/s wr, 6 op/s; 13 B/s, 0 objects/s recovering 2024-03-24T00:20:51.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:51 smithi106 ceph-mon[36505]: pgmap v358: 129 pgs: 129 active+clean; 641 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 461 B/s wr, 6 op/s; 13 B/s, 0 objects/s recovering 2024-03-24T00:20:51.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:51 smithi107 ceph-mon[36485]: pgmap v358: 129 pgs: 129 active+clean; 641 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 461 B/s wr, 6 op/s; 13 B/s, 0 objects/s recovering 2024-03-24T00:20:52.127 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-24T00:20:52.144 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:20:52.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.252+0000 7fdfd5627640 1 -- 172.21.15.78:0/2465769822 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfd01005c0 msgr2=0x7fdfd01009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:52.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.252+0000 7fdfd5627640 1 --2- 172.21.15.78:0/2465769822 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfd01005c0 0x7fdfd01009e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fdfc0009ca0 tx=0x7fdfc002c0f0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.253+0000 7fdfd5627640 1 -- 172.21.15.78:0/2465769822 shutdown_connections 2024-03-24T00:20:52.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.253+0000 7fdfd5627640 1 --2- 172.21.15.78:0/2465769822 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfd0102170 0x7fdfd0108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.253+0000 7fdfd5627640 1 --2- 172.21.15.78:0/2465769822 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdfd01017b0 0x7fdfd0101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.253+0000 7fdfd5627640 1 --2- 172.21.15.78:0/2465769822 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfd01005c0 0x7fdfd01009e0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.253+0000 7fdfd5627640 1 -- 172.21.15.78:0/2465769822 >> 172.21.15.78:0/2465769822 conn(0x7fdfd00fbb30 msgr2=0x7fdfd00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:52.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.253+0000 7fdfd5627640 1 -- 172.21.15.78:0/2465769822 shutdown_connections 2024-03-24T00:20:52.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.253+0000 7fdfd5627640 1 -- 172.21.15.78:0/2465769822 wait complete. 2024-03-24T00:20:52.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.253+0000 7fdfd5627640 1 Processor -- start 2024-03-24T00:20:52.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.253+0000 7fdfd5627640 1 -- start start 2024-03-24T00:20:52.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.253+0000 7fdfd5627640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfd01005c0 0x7fdfd01968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:52.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.253+0000 7fdfd5627640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfd01017b0 0x7fdfd0196de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:52.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.253+0000 7fdfd5627640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdfd0102170 0x7fdfd019de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:52.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.253+0000 7fdfd5627640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd010c230 con 0x7fdfd01017b0 2024-03-24T00:20:52.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.253+0000 7fdfd5627640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd010c0b0 con 0x7fdfd0102170 2024-03-24T00:20:52.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.253+0000 7fdfd5627640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd010c3b0 con 0x7fdfd01005c0 2024-03-24T00:20:52.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.254+0000 7fdfce7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfd01017b0 0x7fdfd0196de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:52.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.254+0000 7fdfce7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfd01017b0 0x7fdfd0196de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37898/0 (socket says 172.21.15.78:37898) 2024-03-24T00:20:52.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.254+0000 7fdfcf7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdfd0102170 0x7fdfd019de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:52.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.254+0000 7fdfce7fc640 1 -- 172.21.15.78:0/3316324138 learned_addr learned my addr 172.21.15.78:0/3316324138 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:52.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.254+0000 7fdfceffd640 1 --2- 172.21.15.78:0/3316324138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfd01005c0 0x7fdfd01968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:52.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.254+0000 7fdfce7fc640 1 -- 172.21.15.78:0/3316324138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfd01005c0 msgr2=0x7fdfd01968a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:52.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.254+0000 7fdfce7fc640 1 --2- 172.21.15.78:0/3316324138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfd01005c0 0x7fdfd01968a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.254+0000 7fdfce7fc640 1 -- 172.21.15.78:0/3316324138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdfd0102170 msgr2=0x7fdfd019de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:52.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.254+0000 7fdfce7fc640 1 --2- 172.21.15.78:0/3316324138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdfd0102170 0x7fdfd019de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.254+0000 7fdfce7fc640 1 -- 172.21.15.78:0/3316324138 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd019e4c0 con 0x7fdfd01017b0 2024-03-24T00:20:52.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.254+0000 7fdfcf7fe640 1 --2- 172.21.15.78:0/3316324138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdfd0102170 0x7fdfd019de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:52.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.254+0000 7fdfce7fc640 1 --2- 172.21.15.78:0/3316324138 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfd01017b0 0x7fdfd0196de0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fdfc400e9f0 tx=0x7fdfc400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:52.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.254+0000 7fdfb7fff640 1 -- 172.21.15.78:0/3316324138 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc400cd80 con 0x7fdfd01017b0 2024-03-24T00:20:52.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.254+0000 7fdfb7fff640 1 -- 172.21.15.78:0/3316324138 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdfc4010910 con 0x7fdfd01017b0 2024-03-24T00:20:52.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.254+0000 7fdfd5627640 1 -- 172.21.15.78:0/3316324138 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfd019e7b0 con 0x7fdfd01017b0 2024-03-24T00:20:52.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.254+0000 7fdfb7fff640 1 -- 172.21.15.78:0/3316324138 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc4018990 con 0x7fdfd01017b0 2024-03-24T00:20:52.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.254+0000 7fdfd5627640 1 -- 172.21.15.78:0/3316324138 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfd019ed00 con 0x7fdfd01017b0 2024-03-24T00:20:52.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.255+0000 7fdfd5627640 1 -- 172.21.15.78:0/3316324138 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfd01078e0 con 0x7fdfd01017b0 2024-03-24T00:20:52.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.256+0000 7fdfb7fff640 1 -- 172.21.15.78:0/3316324138 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fdfc4010ac0 con 0x7fdfd01017b0 2024-03-24T00:20:52.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.257+0000 7fdfb7fff640 1 --2- 172.21.15.78:0/3316324138 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fdfb80a2800 0x7fdfb80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:52.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.257+0000 7fdfceffd640 1 --2- 172.21.15.78:0/3316324138 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fdfb80a2800 0x7fdfb80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:52.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.257+0000 7fdfb7fff640 1 -- 172.21.15.78:0/3316324138 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7fdfc40cf140 con 0x7fdfd01017b0 2024-03-24T00:20:52.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.258+0000 7fdfceffd640 1 --2- 172.21.15.78:0/3316324138 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fdfb80a2800 0x7fdfb80a4cc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fdfc002c700 tx=0x7fdfc0032590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:52.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.262+0000 7fdfb7fff640 1 -- 172.21.15.78:0/3316324138 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdfc409b800 con 0x7fdfd01017b0 2024-03-24T00:20:52.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.439+0000 7fdfd5627640 1 -- 172.21.15.78:0/3316324138 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdfd00622a0 con 0x7fdfd01017b0 2024-03-24T00:20:52.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.441+0000 7fdfb7fff640 1 -- 172.21.15.78:0/3316324138 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24695 (secure 0 0 0) 0x7fdfc409f8a0 con 0x7fdfd01017b0 2024-03-24T00:20:52.442 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:20:52.442 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":93,"fsid":"c7e2806a-e972-11ee-95cd-87774f69a715","created":"2024-03-24T00:10:44.134563+0000","modified":"2024-03-24T00:20:42.446867+0000","last_up_change":"2024-03-24T00:17:50.237686+0000","last_in_change":"2024-03-24T00:17:29.788699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:14:04.534391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:18:42.178418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:18:42.800424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:18:46.112800+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b950388a-c437-426b-b703-3f8ae821dd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6803","nonce":1816879397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6805","nonce":1816879397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6809","nonce":1816879397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6807","nonce":1816879397}]},"public_addr":"172.21.15.78:6803/1816879397","cluster_addr":"172.21.15.78:6805/1816879397","heartbeat_back_addr":"172.21.15.78:6809/1816879397","heartbeat_front_addr":"172.21.15.78:6807/1816879397","state":["exists","up"]},{"osd":1,"uuid":"b8d64714-dd65-400f-b3f4-673ab14e1365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6801","nonce":1647915612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6803","nonce":1647915612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6807","nonce":1647915612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6805","nonce":1647915612}]},"public_addr":"172.21.15.106:6801/1647915612","cluster_addr":"172.21.15.106:6803/1647915612","heartbeat_back_addr":"172.21.15.106:6807/1647915612","heartbeat_front_addr":"172.21.15.106:6805/1647915612","state":["exists","up"]},{"osd":2,"uuid":"fcfc4dd1-aaf8-4131-a023-5e832212158d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6801","nonce":2010770435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6803","nonce":2010770435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6807","nonce":2010770435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6805","nonce":2010770435}]},"public_addr":"172.21.15.107:6801/2010770435","cluster_addr":"172.21.15.107:6803/2010770435","heartbeat_back_addr":"172.21.15.107:6807/2010770435","heartbeat_front_addr":"172.21.15.107:6805/2010770435","state":["exists","up"]},{"osd":3,"uuid":"47c718ab-a0f4-4e52-83c5-b89386b262d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6811","nonce":1912251346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6813","nonce":1912251346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6817","nonce":1912251346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6815","nonce":1912251346}]},"public_addr":"172.21.15.78:6811/1912251346","cluster_addr":"172.21.15.78:6813/1912251346","heartbeat_back_addr":"172.21.15.78:6817/1912251346","heartbeat_front_addr":"172.21.15.78:6815/1912251346","state":["exists","up"]},{"osd":4,"uuid":"7dad64dd-677e-4c1c-9638-e4f4f2195983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6809","nonce":374641205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6811","nonce":374641205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6815","nonce":374641205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6813","nonce":374641205}]},"public_addr":"172.21.15.106:6809/374641205","cluster_addr":"172.21.15.106:6811/374641205","heartbeat_back_addr":"172.21.15.106:6815/374641205","heartbeat_front_addr":"172.21.15.106:6813/374641205","state":["exists","up"]},{"osd":5,"uuid":"bbd524e5-328d-4309-8877-1149cbe110dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6809","nonce":3272806073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6811","nonce":3272806073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6815","nonce":3272806073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6813","nonce":3272806073}]},"public_addr":"172.21.15.107:6809/3272806073","cluster_addr":"172.21.15.107:6811/3272806073","heartbeat_back_addr":"172.21.15.107:6815/3272806073","heartbeat_front_addr":"172.21.15.107:6813/3272806073","state":["exists","up"]},{"osd":6,"uuid":"5696389a-918f-4ea8-b20d-4e2a0d4cb3bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6819","nonce":3091148508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6821","nonce":3091148508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6825","nonce":3091148508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6823","nonce":3091148508}]},"public_addr":"172.21.15.78:6819/3091148508","cluster_addr":"172.21.15.78:6821/3091148508","heartbeat_back_addr":"172.21.15.78:6825/3091148508","heartbeat_front_addr":"172.21.15.78:6823/3091148508","state":["exists","up"]},{"osd":7,"uuid":"f6b764e8-ed4e-4c9a-a85b-94af6dc6b924","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6817","nonce":4262103080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6819","nonce":4262103080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6823","nonce":4262103080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6821","nonce":4262103080}]},"public_addr":"172.21.15.106:6817/4262103080","cluster_addr":"172.21.15.106:6819/4262103080","heartbeat_back_addr":"172.21.15.106:6823/4262103080","heartbeat_front_addr":"172.21.15.106:6821/4262103080","state":["exists","up"]},{"osd":8,"uuid":"ba16039f-8093-4fa0-87ad-e28b7d142b23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6817","nonce":4032799356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6819","nonce":4032799356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6823","nonce":4032799356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6821","nonce":4032799356}]},"public_addr":"172.21.15.107:6817/4032799356","cluster_addr":"172.21.15.107:6819/4032799356","heartbeat_back_addr":"172.21.15.107:6823/4032799356","heartbeat_front_addr":"172.21.15.107:6821/4032799356","state":["exists","up"]},{"osd":9,"uuid":"651357f2-0669-42e6-aa91-1d5c02d00c5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6827","nonce":214023431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6829","nonce":214023431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6833","nonce":214023431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6831","nonce":214023431}]},"public_addr":"172.21.15.78:6827/214023431","cluster_addr":"172.21.15.78:6829/214023431","heartbeat_back_addr":"172.21.15.78:6833/214023431","heartbeat_front_addr":"172.21.15.78:6831/214023431","state":["exists","up"]},{"osd":10,"uuid":"2375119d-346e-426c-8a65-b64af71ad63a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6825","nonce":400029804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6827","nonce":400029804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6831","nonce":400029804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6829","nonce":400029804}]},"public_addr":"172.21.15.106:6825/400029804","cluster_addr":"172.21.15.106:6827/400029804","heartbeat_back_addr":"172.21.15.106:6831/400029804","heartbeat_front_addr":"172.21.15.106:6829/400029804","state":["exists","up"]},{"osd":11,"uuid":"b6b609d9-dd78-4c01-ae66-ba2a662ab22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6825","nonce":4191916230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6827","nonce":4191916230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6831","nonce":4191916230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6829","nonce":4191916230}]},"public_addr":"172.21.15.107:6825/4191916230","cluster_addr":"172.21.15.107:6827/4191916230","heartbeat_back_addr":"172.21.15.107:6831/4191916230","heartbeat_front_addr":"172.21.15.107:6829/4191916230","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:13.728602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:37.320809+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:01.269452+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:25.758281+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:50.298863+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:14.345219+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:38.980489+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:04.749214+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:29.992673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:56.269202+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:22.270316+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:48.074907+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/1982202541":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/1933054339":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6801/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6800/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/465691596":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6801/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/3962523609":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6800/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/1502752625":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/2236879401":"2024-03-25T00:11:40.199569+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:20:52.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.443+0000 7fdfd5627640 1 -- 172.21.15.78:0/3316324138 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fdfb80a2800 msgr2=0x7fdfb80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:52.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.443+0000 7fdfd5627640 1 --2- 172.21.15.78:0/3316324138 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fdfb80a2800 0x7fdfb80a4cc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fdfc002c700 tx=0x7fdfc0032590 comp rx=0 tx=0).stop 2024-03-24T00:20:52.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.443+0000 7fdfd5627640 1 -- 172.21.15.78:0/3316324138 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfd01017b0 msgr2=0x7fdfd0196de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:52.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.443+0000 7fdfd5627640 1 --2- 172.21.15.78:0/3316324138 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfd01017b0 0x7fdfd0196de0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fdfc400e9f0 tx=0x7fdfc400eeb0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.443+0000 7fdfd5627640 1 -- 172.21.15.78:0/3316324138 shutdown_connections 2024-03-24T00:20:52.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.443+0000 7fdfd5627640 1 --2- 172.21.15.78:0/3316324138 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fdfb80a2800 0x7fdfb80a4cc0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.443+0000 7fdfd5627640 1 --2- 172.21.15.78:0/3316324138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdfd0102170 0x7fdfd019de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.443+0000 7fdfd5627640 1 --2- 172.21.15.78:0/3316324138 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfd01017b0 0x7fdfd0196de0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.443+0000 7fdfd5627640 1 --2- 172.21.15.78:0/3316324138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfd01005c0 0x7fdfd01968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.443+0000 7fdfd5627640 1 -- 172.21.15.78:0/3316324138 >> 172.21.15.78:0/3316324138 conn(0x7fdfd00fbb30 msgr2=0x7fdfd0104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:52.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.443+0000 7fdfd5627640 1 -- 172.21.15.78:0/3316324138 shutdown_connections 2024-03-24T00:20:52.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.443+0000 7fdfd5627640 1 -- 172.21.15.78:0/3316324138 wait complete. 2024-03-24T00:20:52.456 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:14:04.534391+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:18:42.178418+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:18:42.800424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T00:18:46.112800+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T00:20:52.456 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:20:52.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.560+0000 7f246d019640 1 -- 172.21.15.78:0/2034397052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2468102150 msgr2=0x7f24681089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:52.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.560+0000 7f246d019640 1 --2- 172.21.15.78:0/2034397052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2468102150 0x7f24681089e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f2460009900 tx=0x7f246002bc30 comp rx=0 tx=0).stop 2024-03-24T00:20:52.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.560+0000 7f246d019640 1 -- 172.21.15.78:0/2034397052 shutdown_connections 2024-03-24T00:20:52.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.560+0000 7f246d019640 1 --2- 172.21.15.78:0/2034397052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2468102150 0x7f24681089e0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.560+0000 7f246d019640 1 --2- 172.21.15.78:0/2034397052 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2468101790 0x7f2468101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.560+0000 7f246d019640 1 --2- 172.21.15.78:0/2034397052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24681005a0 0x7f24681009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.560+0000 7f246d019640 1 -- 172.21.15.78:0/2034397052 >> 172.21.15.78:0/2034397052 conn(0x7f24680fbb30 msgr2=0x7f24680fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:52.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.560+0000 7f246d019640 1 -- 172.21.15.78:0/2034397052 shutdown_connections 2024-03-24T00:20:52.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.560+0000 7f246d019640 1 -- 172.21.15.78:0/2034397052 wait complete. 2024-03-24T00:20:52.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.561+0000 7f246d019640 1 Processor -- start 2024-03-24T00:20:52.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.561+0000 7f246d019640 1 -- start start 2024-03-24T00:20:52.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.561+0000 7f246d019640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f24681005a0 0x7f2468069370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:52.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.561+0000 7f246d019640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2468101790 0x7f246819c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:52.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.561+0000 7f246d019640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2468102150 0x7f246819ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:52.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.561+0000 7f246d019640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246810c230 con 0x7f2468102150 2024-03-24T00:20:52.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.561+0000 7f246d019640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246810c0b0 con 0x7f24681005a0 2024-03-24T00:20:52.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.561+0000 7f246d019640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246810c3b0 con 0x7f2468101790 2024-03-24T00:20:52.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.561+0000 7f246759e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2468102150 0x7f246819ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:52.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.561+0000 7f246759e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2468102150 0x7f246819ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37908/0 (socket says 172.21.15.78:37908) 2024-03-24T00:20:52.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.561+0000 7f246759e640 1 -- 172.21.15.78:0/1278181116 learned_addr learned my addr 172.21.15.78:0/1278181116 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:52.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.561+0000 7f2466d9d640 1 --2- 172.21.15.78:0/1278181116 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f24681005a0 0x7f2468069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:52.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.561+0000 7f246659c640 1 --2- 172.21.15.78:0/1278181116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2468101790 0x7f246819c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:52.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.562+0000 7f246759e640 1 -- 172.21.15.78:0/1278181116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2468101790 msgr2=0x7f246819c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:52.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.562+0000 7f246759e640 1 --2- 172.21.15.78:0/1278181116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2468101790 0x7f246819c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.562+0000 7f246759e640 1 -- 172.21.15.78:0/1278181116 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f24681005a0 msgr2=0x7f2468069370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:52.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.562+0000 7f246759e640 1 --2- 172.21.15.78:0/1278181116 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f24681005a0 0x7f2468069370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.562+0000 7f246759e640 1 -- 172.21.15.78:0/1278181116 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f246819f5e0 con 0x7f2468102150 2024-03-24T00:20:52.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.562+0000 7f246759e640 1 --2- 172.21.15.78:0/1278181116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2468102150 0x7f246819ecd0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2460005ab0 tx=0x7f246002c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:52.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.562+0000 7f244ffff640 1 -- 172.21.15.78:0/1278181116 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2460041070 con 0x7f2468102150 2024-03-24T00:20:52.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.562+0000 7f246d019640 1 -- 172.21.15.78:0/1278181116 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f246819f810 con 0x7f2468102150 2024-03-24T00:20:52.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.562+0000 7f244ffff640 1 -- 172.21.15.78:0/1278181116 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2460004450 con 0x7f2468102150 2024-03-24T00:20:52.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.562+0000 7f244ffff640 1 -- 172.21.15.78:0/1278181116 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2460003e60 con 0x7f2468102150 2024-03-24T00:20:52.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.562+0000 7f246d019640 1 -- 172.21.15.78:0/1278181116 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f246819fd50 con 0x7f2468102150 2024-03-24T00:20:52.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.563+0000 7f246d019640 1 -- 172.21.15.78:0/1278181116 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24681078c0 con 0x7f2468102150 2024-03-24T00:20:52.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.564+0000 7f244ffff640 1 -- 172.21.15.78:0/1278181116 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2460044070 con 0x7f2468102150 2024-03-24T00:20:52.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.565+0000 7f244ffff640 1 --2- 172.21.15.78:0/1278181116 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f24500a2800 0x7f24500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:52.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.565+0000 7f244ffff640 1 -- 172.21.15.78:0/1278181116 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f24600f0a10 con 0x7f2468102150 2024-03-24T00:20:52.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.565+0000 7f2466d9d640 1 --2- 172.21.15.78:0/1278181116 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f24500a2800 0x7f24500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:52.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.565+0000 7f2466d9d640 1 --2- 172.21.15.78:0/1278181116 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f24500a2800 0x7f24500a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f24681015f0 tx=0x7f2458009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:52.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.569+0000 7f244ffff640 1 -- 172.21.15.78:0/1278181116 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f24600bd0d0 con 0x7f2468102150 2024-03-24T00:20:52.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:52 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2677657470' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-24T00:20:52.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.742+0000 7f246d019640 1 -- 172.21.15.78:0/1278181116 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f24680622a0 con 0x7f2468102150 2024-03-24T00:20:52.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.742+0000 7f244ffff640 1 -- 172.21.15.78:0/1278181116 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7f24600c1170 con 0x7f2468102150 2024-03-24T00:20:52.743 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-24T00:20:52.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.744+0000 7f246d019640 1 -- 172.21.15.78:0/1278181116 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f24500a2800 msgr2=0x7f24500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:52.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.744+0000 7f246d019640 1 --2- 172.21.15.78:0/1278181116 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f24500a2800 0x7f24500a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f24681015f0 tx=0x7f2458009500 comp rx=0 tx=0).stop 2024-03-24T00:20:52.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.744+0000 7f246d019640 1 -- 172.21.15.78:0/1278181116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2468102150 msgr2=0x7f246819ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:52.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.744+0000 7f246d019640 1 --2- 172.21.15.78:0/1278181116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2468102150 0x7f246819ecd0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2460005ab0 tx=0x7f246002c6c0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.744+0000 7f246d019640 1 -- 172.21.15.78:0/1278181116 shutdown_connections 2024-03-24T00:20:52.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.744+0000 7f246d019640 1 --2- 172.21.15.78:0/1278181116 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f24500a2800 0x7f24500a4cc0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.744+0000 7f246d019640 1 --2- 172.21.15.78:0/1278181116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2468101790 0x7f246819c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.744+0000 7f246d019640 1 --2- 172.21.15.78:0/1278181116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2468102150 0x7f246819ecd0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.744+0000 7f246d019640 1 --2- 172.21.15.78:0/1278181116 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f24681005a0 0x7f2468069370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.744+0000 7f246d019640 1 -- 172.21.15.78:0/1278181116 >> 172.21.15.78:0/1278181116 conn(0x7f24680fbb30 msgr2=0x7f2468104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:52.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.744+0000 7f246d019640 1 -- 172.21.15.78:0/1278181116 shutdown_connections 2024-03-24T00:20:52.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.744+0000 7f246d019640 1 -- 172.21.15.78:0/1278181116 wait complete. 2024-03-24T00:20:52.755 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:52 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2677657470' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-24T00:20:52.756 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:20:52.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:52 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2677657470' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-24T00:20:52.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.860+0000 7f87bb13b640 1 -- 172.21.15.78:0/1192710041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87b4103280 msgr2=0x7f87b4109b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:52.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.860+0000 7f87bb13b640 1 --2- 172.21.15.78:0/1192710041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87b4103280 0x7f87b4109b10 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f87ac009ca0 tx=0x7f87ac02c0f0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.860+0000 7f87bb13b640 1 -- 172.21.15.78:0/1192710041 shutdown_connections 2024-03-24T00:20:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.860+0000 7f87bb13b640 1 --2- 172.21.15.78:0/1192710041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87b4103280 0x7f87b4109b10 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.860+0000 7f87bb13b640 1 --2- 172.21.15.78:0/1192710041 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f87b41028c0 0x7f87b4102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.860+0000 7f87bb13b640 1 --2- 172.21.15.78:0/1192710041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87b41016d0 0x7f87b4101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.860+0000 7f87bb13b640 1 -- 172.21.15.78:0/1192710041 >> 172.21.15.78:0/1192710041 conn(0x7f87b40fcc40 msgr2=0x7f87b40ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.860+0000 7f87bb13b640 1 -- 172.21.15.78:0/1192710041 shutdown_connections 2024-03-24T00:20:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.860+0000 7f87bb13b640 1 -- 172.21.15.78:0/1192710041 wait complete. 2024-03-24T00:20:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.860+0000 7f87bb13b640 1 Processor -- start 2024-03-24T00:20:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.860+0000 7f87bb13b640 1 -- start start 2024-03-24T00:20:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.860+0000 7f87bb13b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87b41016d0 0x7f87b4197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.860+0000 7f87bb13b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f87b41028c0 0x7f87b4197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.860+0000 7f87bb13b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87b4103280 0x7f87b419ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.860+0000 7f87bb13b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b410d350 con 0x7f87b41016d0 2024-03-24T00:20:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.860+0000 7f87bb13b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b410d1d0 con 0x7f87b41028c0 2024-03-24T00:20:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.860+0000 7f87bb13b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b410d4d0 con 0x7f87b4103280 2024-03-24T00:20:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.860+0000 7f87b8ed7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87b41016d0 0x7f87b4197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.860+0000 7f87b8ed7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87b41016d0 0x7f87b4197970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37930/0 (socket says 172.21.15.78:37930) 2024-03-24T00:20:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.860+0000 7f87b8ed7640 1 -- 172.21.15.78:0/556204859 learned_addr learned my addr 172.21.15.78:0/556204859 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.861+0000 7f87b96d8640 1 --2- 172.21.15.78:0/556204859 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87b4103280 0x7f87b419ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.861+0000 7f87b3fff640 1 --2- 172.21.15.78:0/556204859 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f87b41028c0 0x7f87b4197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.861+0000 7f87b8ed7640 1 -- 172.21.15.78:0/556204859 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87b4103280 msgr2=0x7f87b419ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.861+0000 7f87b8ed7640 1 --2- 172.21.15.78:0/556204859 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87b4103280 0x7f87b419ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.861+0000 7f87b8ed7640 1 -- 172.21.15.78:0/556204859 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f87b41028c0 msgr2=0x7f87b4197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.861+0000 7f87b8ed7640 1 --2- 172.21.15.78:0/556204859 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f87b41028c0 0x7f87b4197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.861+0000 7f87b8ed7640 1 -- 172.21.15.78:0/556204859 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b419f590 con 0x7f87b41016d0 2024-03-24T00:20:52.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.861+0000 7f87b8ed7640 1 --2- 172.21.15.78:0/556204859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87b41016d0 0x7f87b4197970 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f87a800e9c0 tx=0x7f87a800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:52.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.861+0000 7f87b1ffb640 1 -- 172.21.15.78:0/556204859 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a800cd80 con 0x7f87b41016d0 2024-03-24T00:20:52.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.861+0000 7f87b1ffb640 1 -- 172.21.15.78:0/556204859 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f87a80035f0 con 0x7f87b41016d0 2024-03-24T00:20:52.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.861+0000 7f87bb13b640 1 -- 172.21.15.78:0/556204859 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b419f880 con 0x7f87b41016d0 2024-03-24T00:20:52.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.861+0000 7f87b1ffb640 1 -- 172.21.15.78:0/556204859 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a8010690 con 0x7f87b41016d0 2024-03-24T00:20:52.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.861+0000 7f87bb13b640 1 -- 172.21.15.78:0/556204859 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b419fd80 con 0x7f87b41016d0 2024-03-24T00:20:52.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.862+0000 7f87bb13b640 1 -- 172.21.15.78:0/556204859 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87b41089f0 con 0x7f87b41016d0 2024-03-24T00:20:52.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.863+0000 7f87b1ffb640 1 -- 172.21.15.78:0/556204859 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f87a80037a0 con 0x7f87b41016d0 2024-03-24T00:20:52.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.863+0000 7f87b1ffb640 1 --2- 172.21.15.78:0/556204859 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f87a40a2800 0x7f87a40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:52.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.864+0000 7f87b3fff640 1 --2- 172.21.15.78:0/556204859 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f87a40a2800 0x7f87a40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:52.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.864+0000 7f87b1ffb640 1 -- 172.21.15.78:0/556204859 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f87a80cf150 con 0x7f87b41016d0 2024-03-24T00:20:52.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.864+0000 7f87b3fff640 1 --2- 172.21.15.78:0/556204859 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f87a40a2800 0x7f87a40a4cc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f87a00095f0 tx=0x7f87a0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:52.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:52.868+0000 7f87b1ffb640 1 -- 172.21.15.78:0/556204859 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f87a809be60 con 0x7f87b41016d0 2024-03-24T00:20:53.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.042+0000 7f87bb13b640 1 -- 172.21.15.78:0/556204859 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f87b402d070 con 0x7f87b41016d0 2024-03-24T00:20:53.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.042+0000 7f87b1ffb640 1 -- 172.21.15.78:0/556204859 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7f87a809b760 con 0x7f87b41016d0 2024-03-24T00:20:53.043 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-24T00:20:53.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.044+0000 7f87bb13b640 1 -- 172.21.15.78:0/556204859 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f87a40a2800 msgr2=0x7f87a40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:53.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.044+0000 7f87bb13b640 1 --2- 172.21.15.78:0/556204859 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f87a40a2800 0x7f87a40a4cc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f87a00095f0 tx=0x7f87a0005f90 comp rx=0 tx=0).stop 2024-03-24T00:20:53.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.044+0000 7f87bb13b640 1 -- 172.21.15.78:0/556204859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87b41016d0 msgr2=0x7f87b4197970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:53.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.044+0000 7f87bb13b640 1 --2- 172.21.15.78:0/556204859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87b41016d0 0x7f87b4197970 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f87a800e9c0 tx=0x7f87a800ee80 comp rx=0 tx=0).stop 2024-03-24T00:20:53.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.044+0000 7f87bb13b640 1 -- 172.21.15.78:0/556204859 shutdown_connections 2024-03-24T00:20:53.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.044+0000 7f87bb13b640 1 --2- 172.21.15.78:0/556204859 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f87a40a2800 0x7f87a40a4cc0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:53.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.044+0000 7f87bb13b640 1 --2- 172.21.15.78:0/556204859 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87b4103280 0x7f87b419ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:53.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.044+0000 7f87bb13b640 1 --2- 172.21.15.78:0/556204859 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f87b41028c0 0x7f87b4197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:53.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.044+0000 7f87bb13b640 1 --2- 172.21.15.78:0/556204859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87b41016d0 0x7f87b4197970 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:53.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.044+0000 7f87bb13b640 1 -- 172.21.15.78:0/556204859 >> 172.21.15.78:0/556204859 conn(0x7f87b40fcc40 msgr2=0x7f87b4105bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:53.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.044+0000 7f87bb13b640 1 -- 172.21.15.78:0/556204859 shutdown_connections 2024-03-24T00:20:53.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.044+0000 7f87bb13b640 1 -- 172.21.15.78:0/556204859 wait complete. 2024-03-24T00:20:53.056 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:20:53.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.160+0000 7f5b54c0b640 1 -- 172.21.15.78:0/224790384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b50104a50 msgr2=0x7f5b50106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:53.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.160+0000 7f5b54c0b640 1 --2- 172.21.15.78:0/224790384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b50104a50 0x7f5b50106e40 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f5b44009900 tx=0x7f5b4402bc30 comp rx=0 tx=0).stop 2024-03-24T00:20:53.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.160+0000 7f5b54c0b640 1 -- 172.21.15.78:0/224790384 shutdown_connections 2024-03-24T00:20:53.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.160+0000 7f5b54c0b640 1 --2- 172.21.15.78:0/224790384 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5b50107380 0x7f5b501097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:53.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.160+0000 7f5b54c0b640 1 --2- 172.21.15.78:0/224790384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b50104a50 0x7f5b50106e40 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:53.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.160+0000 7f5b54c0b640 1 --2- 172.21.15.78:0/224790384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b50102120 0x7f5b50104510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:53.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.160+0000 7f5b54c0b640 1 -- 172.21.15.78:0/224790384 >> 172.21.15.78:0/224790384 conn(0x7f5b500fbb30 msgr2=0x7f5b500fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:53.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.160+0000 7f5b54c0b640 1 -- 172.21.15.78:0/224790384 shutdown_connections 2024-03-24T00:20:53.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.160+0000 7f5b54c0b640 1 -- 172.21.15.78:0/224790384 wait complete. 2024-03-24T00:20:53.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.161+0000 7f5b54c0b640 1 Processor -- start 2024-03-24T00:20:53.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.161+0000 7f5b54c0b640 1 -- start start 2024-03-24T00:20:53.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.161+0000 7f5b54c0b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b50102120 0x7f5b50197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:53.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.161+0000 7f5b54c0b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5b50104a50 0x7f5b50197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:53.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.161+0000 7f5b54c0b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b50107380 0x7f5b5019efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:53.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.161+0000 7f5b54c0b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5010d300 con 0x7f5b50107380 2024-03-24T00:20:53.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.161+0000 7f5b54c0b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5010d180 con 0x7f5b50104a50 2024-03-24T00:20:53.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.161+0000 7f5b54c0b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5010d480 con 0x7f5b50102120 2024-03-24T00:20:53.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.161+0000 7f5b4ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b50107380 0x7f5b5019efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:53.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.161+0000 7f5b4ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b50107380 0x7f5b5019efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37950/0 (socket says 172.21.15.78:37950) 2024-03-24T00:20:53.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.161+0000 7f5b4ed9d640 1 -- 172.21.15.78:0/3663169133 learned_addr learned my addr 172.21.15.78:0/3663169133 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:53.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.161+0000 7f5b4e59c640 1 --2- 172.21.15.78:0/3663169133 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b50102120 0x7f5b50197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:53.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.161+0000 7f5b4dd9b640 1 --2- 172.21.15.78:0/3663169133 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5b50104a50 0x7f5b50197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:53.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.162+0000 7f5b4ed9d640 1 -- 172.21.15.78:0/3663169133 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b50102120 msgr2=0x7f5b50197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:53.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.162+0000 7f5b4ed9d640 1 --2- 172.21.15.78:0/3663169133 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b50102120 0x7f5b50197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:53.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.162+0000 7f5b4ed9d640 1 -- 172.21.15.78:0/3663169133 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5b50104a50 msgr2=0x7f5b50197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:53.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.162+0000 7f5b4ed9d640 1 --2- 172.21.15.78:0/3663169133 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5b50104a50 0x7f5b50197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:53.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.162+0000 7f5b4ed9d640 1 -- 172.21.15.78:0/3663169133 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b5019f650 con 0x7f5b50107380 2024-03-24T00:20:53.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.162+0000 7f5b4ed9d640 1 --2- 172.21.15.78:0/3663169133 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b50107380 0x7f5b5019efe0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f5b4800e9b0 tx=0x7f5b4800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:53.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.162+0000 7f5b3f7fe640 1 -- 172.21.15.78:0/3663169133 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b4800cd80 con 0x7f5b50107380 2024-03-24T00:20:53.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.162+0000 7f5b3f7fe640 1 -- 172.21.15.78:0/3663169133 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b480035f0 con 0x7f5b50107380 2024-03-24T00:20:53.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.162+0000 7f5b54c0b640 1 -- 172.21.15.78:0/3663169133 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b5019f940 con 0x7f5b50107380 2024-03-24T00:20:53.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.162+0000 7f5b3f7fe640 1 -- 172.21.15.78:0/3663169133 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b48010670 con 0x7f5b50107380 2024-03-24T00:20:53.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.162+0000 7f5b54c0b640 1 -- 172.21.15.78:0/3663169133 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b5019fe60 con 0x7f5b50107380 2024-03-24T00:20:53.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.163+0000 7f5b54c0b640 1 -- 172.21.15.78:0/3663169133 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b500fd620 con 0x7f5b50107380 2024-03-24T00:20:53.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.164+0000 7f5b3f7fe640 1 -- 172.21.15.78:0/3663169133 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5b480037a0 con 0x7f5b50107380 2024-03-24T00:20:53.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.164+0000 7f5b3f7fe640 1 --2- 172.21.15.78:0/3663169133 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f5b380a2880 0x7f5b380a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:53.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.165+0000 7f5b3f7fe640 1 -- 172.21.15.78:0/3663169133 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f5b480cf070 con 0x7f5b50107380 2024-03-24T00:20:53.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.165+0000 7f5b4e59c640 1 --2- 172.21.15.78:0/3663169133 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f5b380a2880 0x7f5b380a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:53.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.165+0000 7f5b4e59c640 1 --2- 172.21.15.78:0/3663169133 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f5b380a2880 0x7f5b380a4d40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5b40005fd0 tx=0x7f5b40009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:53.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.169+0000 7f5b3f7fe640 1 -- 172.21.15.78:0/3663169133 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5b4809bd80 con 0x7f5b50107380 2024-03-24T00:20:53.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.343+0000 7f5b54c0b640 1 -- 172.21.15.78:0/3663169133 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5b500622a0 con 0x7f5b50107380 2024-03-24T00:20:53.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.343+0000 7f5b3f7fe640 1 -- 172.21.15.78:0/3663169133 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5b4809b680 con 0x7f5b50107380 2024-03-24T00:20:53.344 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-03-24T00:20:53.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.345+0000 7f5b54c0b640 1 -- 172.21.15.78:0/3663169133 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f5b380a2880 msgr2=0x7f5b380a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:53.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.345+0000 7f5b54c0b640 1 --2- 172.21.15.78:0/3663169133 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f5b380a2880 0x7f5b380a4d40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5b40005fd0 tx=0x7f5b40009500 comp rx=0 tx=0).stop 2024-03-24T00:20:53.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.345+0000 7f5b54c0b640 1 -- 172.21.15.78:0/3663169133 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b50107380 msgr2=0x7f5b5019efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:53.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.345+0000 7f5b54c0b640 1 --2- 172.21.15.78:0/3663169133 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b50107380 0x7f5b5019efe0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f5b4800e9b0 tx=0x7f5b4800ee70 comp rx=0 tx=0).stop 2024-03-24T00:20:53.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.345+0000 7f5b54c0b640 1 -- 172.21.15.78:0/3663169133 shutdown_connections 2024-03-24T00:20:53.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.345+0000 7f5b54c0b640 1 --2- 172.21.15.78:0/3663169133 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f5b380a2880 0x7f5b380a4d40 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:53.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.345+0000 7f5b54c0b640 1 --2- 172.21.15.78:0/3663169133 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b50107380 0x7f5b5019efe0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:53.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.345+0000 7f5b54c0b640 1 --2- 172.21.15.78:0/3663169133 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5b50104a50 0x7f5b50197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:53.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.345+0000 7f5b54c0b640 1 --2- 172.21.15.78:0/3663169133 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b50102120 0x7f5b50197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:53.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.345+0000 7f5b54c0b640 1 -- 172.21.15.78:0/3663169133 >> 172.21.15.78:0/3663169133 conn(0x7f5b500fbb30 msgr2=0x7f5b50102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:53.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.345+0000 7f5b54c0b640 1 -- 172.21.15.78:0/3663169133 shutdown_connections 2024-03-24T00:20:53.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.345+0000 7f5b54c0b640 1 -- 172.21.15.78:0/3663169133 wait complete. 2024-03-24T00:20:53.357 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T00:20:53.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.463+0000 7f71a5fd4640 1 -- 172.21.15.78:0/343122351 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71a01017a0 msgr2=0x7f71a0101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.463+0000 7f71a5fd4640 1 --2- 172.21.15.78:0/343122351 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71a01017a0 0x7f71a0101c20 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f7194009900 tx=0x7f719402bc30 comp rx=0 tx=0).stop 2024-03-24T00:20:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.463+0000 7f71a5fd4640 1 -- 172.21.15.78:0/343122351 shutdown_connections 2024-03-24T00:20:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.463+0000 7f71a5fd4640 1 --2- 172.21.15.78:0/343122351 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71a0102160 0x7f71a01089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.463+0000 7f71a5fd4640 1 --2- 172.21.15.78:0/343122351 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71a01017a0 0x7f71a0101c20 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.464+0000 7f71a5fd4640 1 --2- 172.21.15.78:0/343122351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71a01005b0 0x7f71a01009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.464+0000 7f71a5fd4640 1 -- 172.21.15.78:0/343122351 >> 172.21.15.78:0/343122351 conn(0x7f71a00fbb40 msgr2=0x7f71a00fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.464+0000 7f71a5fd4640 1 -- 172.21.15.78:0/343122351 shutdown_connections 2024-03-24T00:20:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.464+0000 7f71a5fd4640 1 -- 172.21.15.78:0/343122351 wait complete. 2024-03-24T00:20:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.464+0000 7f71a5fd4640 1 Processor -- start 2024-03-24T00:20:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.464+0000 7f71a5fd4640 1 -- start start 2024-03-24T00:20:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.464+0000 7f71a5fd4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71a01005b0 0x7f71a01968e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:53.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.464+0000 7f71a5fd4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71a01017a0 0x7f71a0196e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:53.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.464+0000 7f71a5fd4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71a0102160 0x7f71a019de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:53.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.464+0000 7f71a5fd4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a010c230 con 0x7f71a01017a0 2024-03-24T00:20:53.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.464+0000 7f71a5fd4640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a010c0b0 con 0x7f71a0102160 2024-03-24T00:20:53.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.464+0000 7f71a5fd4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a010c3b0 con 0x7f71a01005b0 2024-03-24T00:20:53.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.465+0000 7f719effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71a01017a0 0x7f71a0196e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:53.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.465+0000 7f719f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71a01005b0 0x7f71a01968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:53.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.465+0000 7f719effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71a01017a0 0x7f71a0196e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37970/0 (socket says 172.21.15.78:37970) 2024-03-24T00:20:53.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.465+0000 7f719f7fe640 1 --2- 172.21.15.78:0/800587219 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71a01005b0 0x7f71a01968e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.78:55874/0 (socket says 172.21.15.78:55874) 2024-03-24T00:20:53.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.465+0000 7f719effd640 1 -- 172.21.15.78:0/800587219 learned_addr learned my addr 172.21.15.78:0/800587219 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:53.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.465+0000 7f719ffff640 1 --2- 172.21.15.78:0/800587219 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71a0102160 0x7f71a019de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:53.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.465+0000 7f719effd640 1 -- 172.21.15.78:0/800587219 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71a01005b0 msgr2=0x7f71a01968e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:53.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.465+0000 7f719effd640 1 --2- 172.21.15.78:0/800587219 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71a01005b0 0x7f71a01968e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.465+0000 7f719effd640 1 -- 172.21.15.78:0/800587219 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71a0102160 msgr2=0x7f71a019de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.465+0000 7f719effd640 1 --2- 172.21.15.78:0/800587219 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71a0102160 0x7f71a019de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.465+0000 7f719effd640 1 -- 172.21.15.78:0/800587219 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a019e500 con 0x7f71a01017a0 2024-03-24T00:20:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.465+0000 7f719f7fe640 1 --2- 172.21.15.78:0/800587219 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71a01005b0 0x7f71a01968e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.465+0000 7f719ffff640 1 --2- 172.21.15.78:0/800587219 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71a0102160 0x7f71a019de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.465+0000 7f719effd640 1 --2- 172.21.15.78:0/800587219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71a01017a0 0x7f71a0196e20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7194009230 tx=0x7f719402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.465+0000 7f719cff9640 1 -- 172.21.15.78:0/800587219 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7194041070 con 0x7f71a01017a0 2024-03-24T00:20:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.465+0000 7f719cff9640 1 -- 172.21.15.78:0/800587219 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7194004450 con 0x7f71a01017a0 2024-03-24T00:20:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.465+0000 7f71a5fd4640 1 -- 172.21.15.78:0/800587219 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71a0069c70 con 0x7f71a01017a0 2024-03-24T00:20:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.465+0000 7f719cff9640 1 -- 172.21.15.78:0/800587219 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7194003db0 con 0x7f71a01017a0 2024-03-24T00:20:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.465+0000 7f71a5fd4640 1 -- 172.21.15.78:0/800587219 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71a006a1b0 con 0x7f71a01017a0 2024-03-24T00:20:53.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.466+0000 7f71a5fd4640 1 -- 172.21.15.78:0/800587219 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71a00688e0 con 0x7f71a01017a0 2024-03-24T00:20:53.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.467+0000 7f719cff9640 1 -- 172.21.15.78:0/800587219 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7194044070 con 0x7f71a01017a0 2024-03-24T00:20:53.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.468+0000 7f719cff9640 1 --2- 172.21.15.78:0/800587219 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f71880a2880 0x7f71880a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:53.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.468+0000 7f719cff9640 1 -- 172.21.15.78:0/800587219 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f71940f0910 con 0x7f71a01017a0 2024-03-24T00:20:53.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.468+0000 7f719f7fe640 1 --2- 172.21.15.78:0/800587219 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f71880a2880 0x7f71880a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:53.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.468+0000 7f719f7fe640 1 --2- 172.21.15.78:0/800587219 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f71880a2880 0x7f71880a4d40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f71a0101600 tx=0x7f7190010040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:53.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.472+0000 7f719cff9640 1 -- 172.21.15.78:0/800587219 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f71940bd620 con 0x7f71a01017a0 2024-03-24T00:20:53.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.645+0000 7f71a5fd4640 1 -- 172.21.15.78:0/800587219 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f71a019e790 con 0x7f71a01017a0 2024-03-24T00:20:53.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.646+0000 7f719cff9640 1 -- 172.21.15.78:0/800587219 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7f71940bcf20 con 0x7f71a01017a0 2024-03-24T00:20:53.647 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-24T00:20:53.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.648+0000 7f71a5fd4640 1 -- 172.21.15.78:0/800587219 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f71880a2880 msgr2=0x7f71880a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:53.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.648+0000 7f71a5fd4640 1 --2- 172.21.15.78:0/800587219 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f71880a2880 0x7f71880a4d40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f71a0101600 tx=0x7f7190010040 comp rx=0 tx=0).stop 2024-03-24T00:20:53.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.648+0000 7f71a5fd4640 1 -- 172.21.15.78:0/800587219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71a01017a0 msgr2=0x7f71a0196e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:53.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.648+0000 7f71a5fd4640 1 --2- 172.21.15.78:0/800587219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71a01017a0 0x7f71a0196e20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7194009230 tx=0x7f719402c650 comp rx=0 tx=0).stop 2024-03-24T00:20:53.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.648+0000 7f71a5fd4640 1 -- 172.21.15.78:0/800587219 shutdown_connections 2024-03-24T00:20:53.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.648+0000 7f71a5fd4640 1 --2- 172.21.15.78:0/800587219 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f71880a2880 0x7f71880a4d40 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:53.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.648+0000 7f71a5fd4640 1 --2- 172.21.15.78:0/800587219 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71a0102160 0x7f71a019de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:53.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.648+0000 7f71a5fd4640 1 --2- 172.21.15.78:0/800587219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71a01017a0 0x7f71a0196e20 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:53.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.648+0000 7f71a5fd4640 1 --2- 172.21.15.78:0/800587219 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71a01005b0 0x7f71a01968e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:53.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.648+0000 7f71a5fd4640 1 -- 172.21.15.78:0/800587219 >> 172.21.15.78:0/800587219 conn(0x7f71a00fbb40 msgr2=0x7f71a0104bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:53.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.648+0000 7f71a5fd4640 1 -- 172.21.15.78:0/800587219 shutdown_connections 2024-03-24T00:20:53.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.648+0000 7f71a5fd4640 1 -- 172.21.15.78:0/800587219 wait complete. 2024-03-24T00:20:53.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:53 smithi106 ceph-mon[36505]: pgmap v359: 129 pgs: 129 active+clean; 641 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s; 11 B/s, 0 objects/s recovering 2024-03-24T00:20:53.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:53 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3316324138' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:20:53.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:53 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1278181116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:20:53.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:53 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/556204859' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:20:53.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:53 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3663169133' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:20:53.662 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:53 smithi078 ceph-mon[32429]: pgmap v359: 129 pgs: 129 active+clean; 641 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s; 11 B/s, 0 objects/s recovering 2024-03-24T00:20:53.662 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:53 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3316324138' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:20:53.662 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:53 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1278181116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:20:53.662 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:53 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/556204859' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:20:53.662 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:53 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3663169133' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:20:53.663 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:20:53.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.763+0000 7efe5373b640 1 -- 172.21.15.78:0/1311077616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe4c1021e0 msgr2=0x7efe4c1045d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:53.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.763+0000 7efe5373b640 1 --2- 172.21.15.78:0/1311077616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe4c1021e0 0x7efe4c1045d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7efe48009900 tx=0x7efe4802bc30 comp rx=0 tx=0).stop 2024-03-24T00:20:53.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.763+0000 7efe5373b640 1 -- 172.21.15.78:0/1311077616 shutdown_connections 2024-03-24T00:20:53.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.763+0000 7efe5373b640 1 --2- 172.21.15.78:0/1311077616 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe4c107440 0x7efe4c109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:53.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.763+0000 7efe5373b640 1 --2- 172.21.15.78:0/1311077616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe4c104b10 0x7efe4c106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:53.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.763+0000 7efe5373b640 1 --2- 172.21.15.78:0/1311077616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe4c1021e0 0x7efe4c1045d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:53.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.763+0000 7efe5373b640 1 -- 172.21.15.78:0/1311077616 >> 172.21.15.78:0/1311077616 conn(0x7efe4c0fbb50 msgr2=0x7efe4c0fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:53.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.763+0000 7efe5373b640 1 -- 172.21.15.78:0/1311077616 shutdown_connections 2024-03-24T00:20:53.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.763+0000 7efe5373b640 1 -- 172.21.15.78:0/1311077616 wait complete. 2024-03-24T00:20:53.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.764+0000 7efe5373b640 1 Processor -- start 2024-03-24T00:20:53.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.764+0000 7efe5373b640 1 -- start start 2024-03-24T00:20:53.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.764+0000 7efe5373b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe4c1021e0 0x7efe4c197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:53.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.764+0000 7efe5373b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe4c104b10 0x7efe4c197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:53.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.764+0000 7efe5373b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe4c107440 0x7efe4c19efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:53.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.764+0000 7efe5373b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe4c10d380 con 0x7efe4c1021e0 2024-03-24T00:20:53.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.764+0000 7efe5373b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe4c10d200 con 0x7efe4c104b10 2024-03-24T00:20:53.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.764+0000 7efe5373b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe4c10d500 con 0x7efe4c107440 2024-03-24T00:20:53.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.764+0000 7efe514d7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe4c1021e0 0x7efe4c197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:53.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.764+0000 7efe514d7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe4c1021e0 0x7efe4c197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37992/0 (socket says 172.21.15.78:37992) 2024-03-24T00:20:53.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.764+0000 7efe514d7640 1 -- 172.21.15.78:0/2418702720 learned_addr learned my addr 172.21.15.78:0/2418702720 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:53.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.764+0000 7efe50cd6640 1 --2- 172.21.15.78:0/2418702720 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe4c104b10 0x7efe4c197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:53.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.764+0000 7efe51cd8640 1 --2- 172.21.15.78:0/2418702720 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe4c107440 0x7efe4c19efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:53.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.764+0000 7efe514d7640 1 -- 172.21.15.78:0/2418702720 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe4c107440 msgr2=0x7efe4c19efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:53.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.764+0000 7efe514d7640 1 --2- 172.21.15.78:0/2418702720 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe4c107440 0x7efe4c19efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:53.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.764+0000 7efe514d7640 1 -- 172.21.15.78:0/2418702720 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe4c104b10 msgr2=0x7efe4c197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:53.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.764+0000 7efe514d7640 1 --2- 172.21.15.78:0/2418702720 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe4c104b10 0x7efe4c197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:53.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.764+0000 7efe514d7640 1 -- 172.21.15.78:0/2418702720 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe4c19f610 con 0x7efe4c1021e0 2024-03-24T00:20:53.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.764+0000 7efe514d7640 1 --2- 172.21.15.78:0/2418702720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe4c1021e0 0x7efe4c197a90 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7efe48009230 tx=0x7efe4802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:53.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.765+0000 7efe427fc640 1 -- 172.21.15.78:0/2418702720 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe48041070 con 0x7efe4c1021e0 2024-03-24T00:20:53.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.765+0000 7efe5373b640 1 -- 172.21.15.78:0/2418702720 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe4c19f8a0 con 0x7efe4c1021e0 2024-03-24T00:20:53.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.765+0000 7efe427fc640 1 -- 172.21.15.78:0/2418702720 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efe48004450 con 0x7efe4c1021e0 2024-03-24T00:20:53.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.765+0000 7efe427fc640 1 -- 172.21.15.78:0/2418702720 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe48003e20 con 0x7efe4c1021e0 2024-03-24T00:20:53.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.765+0000 7efe5373b640 1 -- 172.21.15.78:0/2418702720 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe4c19fd90 con 0x7efe4c1021e0 2024-03-24T00:20:53.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.766+0000 7efe5373b640 1 -- 172.21.15.78:0/2418702720 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe4c0fd640 con 0x7efe4c1021e0 2024-03-24T00:20:53.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.767+0000 7efe427fc640 1 -- 172.21.15.78:0/2418702720 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7efe48044070 con 0x7efe4c1021e0 2024-03-24T00:20:53.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.767+0000 7efe427fc640 1 --2- 172.21.15.78:0/2418702720 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7efe380a2800 0x7efe380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:53.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.767+0000 7efe50cd6640 1 --2- 172.21.15.78:0/2418702720 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7efe380a2800 0x7efe380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:53.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.767+0000 7efe427fc640 1 -- 172.21.15.78:0/2418702720 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7efe480f0980 con 0x7efe4c1021e0 2024-03-24T00:20:53.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.768+0000 7efe50cd6640 1 --2- 172.21.15.78:0/2418702720 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7efe380a2800 0x7efe380a4cc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7efe3c0095f0 tx=0x7efe3c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:53.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.772+0000 7efe427fc640 1 -- 172.21.15.78:0/2418702720 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efe480bd040 con 0x7efe4c1021e0 2024-03-24T00:20:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:53 smithi107 ceph-mon[36485]: pgmap v359: 129 pgs: 129 active+clean; 641 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s; 11 B/s, 0 objects/s recovering 2024-03-24T00:20:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:53 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3316324138' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:20:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:53 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1278181116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:20:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:53 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/556204859' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:20:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:53 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3663169133' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:20:53.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.991+0000 7efe5373b640 1 -- 172.21.15.78:0/2418702720 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe4c02d290 con 0x7efe4c1021e0 2024-03-24T00:20:53.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.991+0000 7efe427fc640 1 -- 172.21.15.78:0/2418702720 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11406 (secure 0 0 0) 0x7efe480c10e0 con 0x7efe4c1021e0 2024-03-24T00:20:53.992 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:20:53.992 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1416685123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1416685123},{"type":"v1","addr":"172.21.15.78:6835","nonce":1416685123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1071274623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1071274623},{"type":"v1","addr":"172.21.15.78:6837","nonce":1071274623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2758357551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2758357551},{"type":"v1","addr":"172.21.15.78:6839","nonce":2758357551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/3424505910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":3424505910},{"type":"v1","addr":"172.21.15.78:6841","nonce":3424505910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/3086318443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3086318443},{"type":"v1","addr":"172.21.15.107:6833","nonce":3086318443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/2367018868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2367018868},{"type":"v1","addr":"172.21.15.106:6833","nonce":2367018868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:18:43.592215+0000","modified":"2024-03-24T00:19:08.871384+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24500,"mds_1":24481,"mds_2":24464},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/2231611477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":2231611477},{"type":"v1","addr":"172.21.15.106:6835","nonce":2231611477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6835/108936439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":108936439},{"type":"v1","addr":"172.21.15.107:6835","nonce":108936439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6837/2939571540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2939571540},{"type":"v1","addr":"172.21.15.107:6837","nonce":2939571540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24464":{"gid":24464,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.106:6837/2090968633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":2090968633},{"type":"v1","addr":"172.21.15.106:6837","nonce":2090968633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.107:6839/117119462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":117119462},{"type":"v1","addr":"172.21.15.107:6839","nonce":117119462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6839/2482071304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2482071304},{"type":"v1","addr":"172.21.15.106:6839","nonce":2482071304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24481,24500,24464]},"id":1}]} 2024-03-24T00:20:53.993 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 23 2024-03-24T00:20:53.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.993+0000 7efe5373b640 1 -- 172.21.15.78:0/2418702720 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7efe380a2800 msgr2=0x7efe380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:53.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.993+0000 7efe5373b640 1 --2- 172.21.15.78:0/2418702720 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7efe380a2800 0x7efe380a4cc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7efe3c0095f0 tx=0x7efe3c005f90 comp rx=0 tx=0).stop 2024-03-24T00:20:53.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.993+0000 7efe5373b640 1 -- 172.21.15.78:0/2418702720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe4c1021e0 msgr2=0x7efe4c197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:53.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.993+0000 7efe5373b640 1 --2- 172.21.15.78:0/2418702720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe4c1021e0 0x7efe4c197a90 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7efe48009230 tx=0x7efe4802c650 comp rx=0 tx=0).stop 2024-03-24T00:20:53.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.994+0000 7efe5373b640 1 -- 172.21.15.78:0/2418702720 shutdown_connections 2024-03-24T00:20:53.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.994+0000 7efe5373b640 1 --2- 172.21.15.78:0/2418702720 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7efe380a2800 0x7efe380a4cc0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:53.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.994+0000 7efe5373b640 1 --2- 172.21.15.78:0/2418702720 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe4c107440 0x7efe4c19efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:53.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.994+0000 7efe5373b640 1 --2- 172.21.15.78:0/2418702720 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe4c104b10 0x7efe4c197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:53.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.994+0000 7efe5373b640 1 --2- 172.21.15.78:0/2418702720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe4c1021e0 0x7efe4c197a90 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:53.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.994+0000 7efe5373b640 1 -- 172.21.15.78:0/2418702720 >> 172.21.15.78:0/2418702720 conn(0x7efe4c0fbb50 msgr2=0x7efe4c102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:53.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.994+0000 7efe5373b640 1 -- 172.21.15.78:0/2418702720 shutdown_connections 2024-03-24T00:20:53.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:53.994+0000 7efe5373b640 1 -- 172.21.15.78:0/2418702720 wait complete. 2024-03-24T00:20:54.006 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:20:54.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.107+0000 7fdb9d51b640 1 -- 172.21.15.78:0/250512293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb98102170 msgr2=0x7fdb98108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:54.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.107+0000 7fdb9d51b640 1 --2- 172.21.15.78:0/250512293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb98102170 0x7fdb98108a00 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fdb90009ca0 tx=0x7fdb9002c0f0 comp rx=0 tx=0).stop 2024-03-24T00:20:54.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.107+0000 7fdb9d51b640 1 -- 172.21.15.78:0/250512293 shutdown_connections 2024-03-24T00:20:54.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.107+0000 7fdb9d51b640 1 --2- 172.21.15.78:0/250512293 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb981017b0 0x7fdb98101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:54.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.107+0000 7fdb9d51b640 1 --2- 172.21.15.78:0/250512293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb98102170 0x7fdb98108a00 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:54.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.107+0000 7fdb9d51b640 1 --2- 172.21.15.78:0/250512293 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb981005c0 0x7fdb981009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:54.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.107+0000 7fdb9d51b640 1 -- 172.21.15.78:0/250512293 >> 172.21.15.78:0/250512293 conn(0x7fdb980fbb30 msgr2=0x7fdb980fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:54.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.107+0000 7fdb9d51b640 1 -- 172.21.15.78:0/250512293 shutdown_connections 2024-03-24T00:20:54.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.107+0000 7fdb9d51b640 1 -- 172.21.15.78:0/250512293 wait complete. 2024-03-24T00:20:54.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.107+0000 7fdb9d51b640 1 Processor -- start 2024-03-24T00:20:54.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.107+0000 7fdb9d51b640 1 -- start start 2024-03-24T00:20:54.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.108+0000 7fdb9d51b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb981005c0 0x7fdb98196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:54.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.108+0000 7fdb9d51b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb981017b0 0x7fdb98196da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:54.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.108+0000 7fdb9d51b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb98102170 0x7fdb9819de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:54.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.108+0000 7fdb9d51b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9810c240 con 0x7fdb98102170 2024-03-24T00:20:54.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.108+0000 7fdb9d51b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9810c0c0 con 0x7fdb981017b0 2024-03-24T00:20:54.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.108+0000 7fdb9d51b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9810c3c0 con 0x7fdb981005c0 2024-03-24T00:20:54.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.108+0000 7fdb977fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb98102170 0x7fdb9819de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:54.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.108+0000 7fdb8e7fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb981017b0 0x7fdb98196da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:54.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.108+0000 7fdb977fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb98102170 0x7fdb9819de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37998/0 (socket says 172.21.15.78:37998) 2024-03-24T00:20:54.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.108+0000 7fdb977fe640 1 -- 172.21.15.78:0/237245930 learned_addr learned my addr 172.21.15.78:0/237245930 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:54.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.108+0000 7fdb96ffd640 1 --2- 172.21.15.78:0/237245930 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb981005c0 0x7fdb98196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:54.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.108+0000 7fdb8e7fc640 1 -- 172.21.15.78:0/237245930 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb981005c0 msgr2=0x7fdb98196860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:54.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.108+0000 7fdb8e7fc640 1 --2- 172.21.15.78:0/237245930 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb981005c0 0x7fdb98196860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:54.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.108+0000 7fdb8e7fc640 1 -- 172.21.15.78:0/237245930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb98102170 msgr2=0x7fdb9819de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:54.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.108+0000 7fdb8e7fc640 1 --2- 172.21.15.78:0/237245930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb98102170 0x7fdb9819de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:54.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.108+0000 7fdb8e7fc640 1 -- 172.21.15.78:0/237245930 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb9819e480 con 0x7fdb981017b0 2024-03-24T00:20:54.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.108+0000 7fdb977fe640 1 --2- 172.21.15.78:0/237245930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb98102170 0x7fdb9819de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:54.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.108+0000 7fdb8e7fc640 1 --2- 172.21.15.78:0/237245930 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb981017b0 0x7fdb98196da0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdb8000e9f0 tx=0x7fdb8000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:54.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.108+0000 7fdb94ff9640 1 -- 172.21.15.78:0/237245930 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8000ccc0 con 0x7fdb981017b0 2024-03-24T00:20:54.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.108+0000 7fdb9d51b640 1 -- 172.21.15.78:0/237245930 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb9819e770 con 0x7fdb981017b0 2024-03-24T00:20:54.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.108+0000 7fdb9d51b640 1 -- 172.21.15.78:0/237245930 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb9819ecc0 con 0x7fdb981017b0 2024-03-24T00:20:54.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.108+0000 7fdb94ff9640 1 -- 172.21.15.78:0/237245930 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdb80005020 con 0x7fdb981017b0 2024-03-24T00:20:54.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.108+0000 7fdb94ff9640 1 -- 172.21.15.78:0/237245930 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8001c070 con 0x7fdb981017b0 2024-03-24T00:20:54.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.109+0000 7fdb9d51b640 1 -- 172.21.15.78:0/237245930 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb98107880 con 0x7fdb981017b0 2024-03-24T00:20:54.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.110+0000 7fdb94ff9640 1 -- 172.21.15.78:0/237245930 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fdb80005540 con 0x7fdb981017b0 2024-03-24T00:20:54.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.111+0000 7fdb94ff9640 1 --2- 172.21.15.78:0/237245930 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fdb780a2800 0x7fdb780a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:54.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.111+0000 7fdb96ffd640 1 --2- 172.21.15.78:0/237245930 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fdb780a2800 0x7fdb780a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:54.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.111+0000 7fdb94ff9640 1 -- 172.21.15.78:0/237245930 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7fdb800cf0a0 con 0x7fdb981017b0 2024-03-24T00:20:54.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.111+0000 7fdb96ffd640 1 --2- 172.21.15.78:0/237245930 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fdb780a2800 0x7fdb780a4cc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdb880095f0 tx=0x7fdb88005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:54.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.117+0000 7fdb94ff9640 1 -- 172.21.15.78:0/237245930 <== mon.2 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdb8009bde0 con 0x7fdb981017b0 2024-03-24T00:20:54.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.292+0000 7fdb9d51b640 1 -- 172.21.15.78:0/237245930 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb9802d270 con 0x7fdb981017b0 2024-03-24T00:20:54.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.295+0000 7fdb94ff9640 1 -- 172.21.15.78:0/237245930 <== mon.2 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24695 (secure 0 0 0) 0x7fdb8009b6e0 con 0x7fdb981017b0 2024-03-24T00:20:54.295 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:20:54.295 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":93,"fsid":"c7e2806a-e972-11ee-95cd-87774f69a715","created":"2024-03-24T00:10:44.134563+0000","modified":"2024-03-24T00:20:42.446867+0000","last_up_change":"2024-03-24T00:17:50.237686+0000","last_in_change":"2024-03-24T00:17:29.788699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:14:04.534391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:18:42.178418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:18:42.800424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:18:46.112800+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b950388a-c437-426b-b703-3f8ae821dd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6803","nonce":1816879397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6805","nonce":1816879397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6809","nonce":1816879397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6807","nonce":1816879397}]},"public_addr":"172.21.15.78:6803/1816879397","cluster_addr":"172.21.15.78:6805/1816879397","heartbeat_back_addr":"172.21.15.78:6809/1816879397","heartbeat_front_addr":"172.21.15.78:6807/1816879397","state":["exists","up"]},{"osd":1,"uuid":"b8d64714-dd65-400f-b3f4-673ab14e1365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6801","nonce":1647915612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6803","nonce":1647915612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6807","nonce":1647915612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6805","nonce":1647915612}]},"public_addr":"172.21.15.106:6801/1647915612","cluster_addr":"172.21.15.106:6803/1647915612","heartbeat_back_addr":"172.21.15.106:6807/1647915612","heartbeat_front_addr":"172.21.15.106:6805/1647915612","state":["exists","up"]},{"osd":2,"uuid":"fcfc4dd1-aaf8-4131-a023-5e832212158d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6801","nonce":2010770435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6803","nonce":2010770435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6807","nonce":2010770435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6805","nonce":2010770435}]},"public_addr":"172.21.15.107:6801/2010770435","cluster_addr":"172.21.15.107:6803/2010770435","heartbeat_back_addr":"172.21.15.107:6807/2010770435","heartbeat_front_addr":"172.21.15.107:6805/2010770435","state":["exists","up"]},{"osd":3,"uuid":"47c718ab-a0f4-4e52-83c5-b89386b262d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6811","nonce":1912251346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6813","nonce":1912251346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6817","nonce":1912251346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6815","nonce":1912251346}]},"public_addr":"172.21.15.78:6811/1912251346","cluster_addr":"172.21.15.78:6813/1912251346","heartbeat_back_addr":"172.21.15.78:6817/1912251346","heartbeat_front_addr":"172.21.15.78:6815/1912251346","state":["exists","up"]},{"osd":4,"uuid":"7dad64dd-677e-4c1c-9638-e4f4f2195983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6809","nonce":374641205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6811","nonce":374641205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6815","nonce":374641205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6813","nonce":374641205}]},"public_addr":"172.21.15.106:6809/374641205","cluster_addr":"172.21.15.106:6811/374641205","heartbeat_back_addr":"172.21.15.106:6815/374641205","heartbeat_front_addr":"172.21.15.106:6813/374641205","state":["exists","up"]},{"osd":5,"uuid":"bbd524e5-328d-4309-8877-1149cbe110dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6809","nonce":3272806073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6811","nonce":3272806073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6815","nonce":3272806073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6813","nonce":3272806073}]},"public_addr":"172.21.15.107:6809/3272806073","cluster_addr":"172.21.15.107:6811/3272806073","heartbeat_back_addr":"172.21.15.107:6815/3272806073","heartbeat_front_addr":"172.21.15.107:6813/3272806073","state":["exists","up"]},{"osd":6,"uuid":"5696389a-918f-4ea8-b20d-4e2a0d4cb3bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6819","nonce":3091148508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6821","nonce":3091148508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6825","nonce":3091148508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6823","nonce":3091148508}]},"public_addr":"172.21.15.78:6819/3091148508","cluster_addr":"172.21.15.78:6821/3091148508","heartbeat_back_addr":"172.21.15.78:6825/3091148508","heartbeat_front_addr":"172.21.15.78:6823/3091148508","state":["exists","up"]},{"osd":7,"uuid":"f6b764e8-ed4e-4c9a-a85b-94af6dc6b924","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6817","nonce":4262103080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6819","nonce":4262103080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6823","nonce":4262103080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6821","nonce":4262103080}]},"public_addr":"172.21.15.106:6817/4262103080","cluster_addr":"172.21.15.106:6819/4262103080","heartbeat_back_addr":"172.21.15.106:6823/4262103080","heartbeat_front_addr":"172.21.15.106:6821/4262103080","state":["exists","up"]},{"osd":8,"uuid":"ba16039f-8093-4fa0-87ad-e28b7d142b23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6817","nonce":4032799356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6819","nonce":4032799356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6823","nonce":4032799356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6821","nonce":4032799356}]},"public_addr":"172.21.15.107:6817/4032799356","cluster_addr":"172.21.15.107:6819/4032799356","heartbeat_back_addr":"172.21.15.107:6823/4032799356","heartbeat_front_addr":"172.21.15.107:6821/4032799356","state":["exists","up"]},{"osd":9,"uuid":"651357f2-0669-42e6-aa91-1d5c02d00c5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6827","nonce":214023431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6829","nonce":214023431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6833","nonce":214023431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6831","nonce":214023431}]},"public_addr":"172.21.15.78:6827/214023431","cluster_addr":"172.21.15.78:6829/214023431","heartbeat_back_addr":"172.21.15.78:6833/214023431","heartbeat_front_addr":"172.21.15.78:6831/214023431","state":["exists","up"]},{"osd":10,"uuid":"2375119d-346e-426c-8a65-b64af71ad63a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6825","nonce":400029804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6827","nonce":400029804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6831","nonce":400029804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6829","nonce":400029804}]},"public_addr":"172.21.15.106:6825/400029804","cluster_addr":"172.21.15.106:6827/400029804","heartbeat_back_addr":"172.21.15.106:6831/400029804","heartbeat_front_addr":"172.21.15.106:6829/400029804","state":["exists","up"]},{"osd":11,"uuid":"b6b609d9-dd78-4c01-ae66-ba2a662ab22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6825","nonce":4191916230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6827","nonce":4191916230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6831","nonce":4191916230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6829","nonce":4191916230}]},"public_addr":"172.21.15.107:6825/4191916230","cluster_addr":"172.21.15.107:6827/4191916230","heartbeat_back_addr":"172.21.15.107:6831/4191916230","heartbeat_front_addr":"172.21.15.107:6829/4191916230","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:13.728602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:37.320809+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:01.269452+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:25.758281+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:50.298863+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:14.345219+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:38.980489+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:04.749214+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:29.992673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:56.269202+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:22.270316+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:48.074907+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/1982202541":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/1933054339":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6801/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6800/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/465691596":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6801/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/3962523609":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6800/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/1502752625":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/2236879401":"2024-03-25T00:11:40.199569+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:20:54.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.296+0000 7fdb9d51b640 1 -- 172.21.15.78:0/237245930 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fdb780a2800 msgr2=0x7fdb780a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:54.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.296+0000 7fdb9d51b640 1 --2- 172.21.15.78:0/237245930 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fdb780a2800 0x7fdb780a4cc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdb880095f0 tx=0x7fdb88005f90 comp rx=0 tx=0).stop 2024-03-24T00:20:54.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.296+0000 7fdb9d51b640 1 -- 172.21.15.78:0/237245930 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb981017b0 msgr2=0x7fdb98196da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:54.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.296+0000 7fdb9d51b640 1 --2- 172.21.15.78:0/237245930 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb981017b0 0x7fdb98196da0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdb8000e9f0 tx=0x7fdb8000eeb0 comp rx=0 tx=0).stop 2024-03-24T00:20:54.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.296+0000 7fdb9d51b640 1 -- 172.21.15.78:0/237245930 shutdown_connections 2024-03-24T00:20:54.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.296+0000 7fdb9d51b640 1 --2- 172.21.15.78:0/237245930 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fdb780a2800 0x7fdb780a4cc0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:54.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.296+0000 7fdb9d51b640 1 --2- 172.21.15.78:0/237245930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb98102170 0x7fdb9819de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:54.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.296+0000 7fdb9d51b640 1 --2- 172.21.15.78:0/237245930 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb981017b0 0x7fdb98196da0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:54.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.296+0000 7fdb9d51b640 1 --2- 172.21.15.78:0/237245930 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb981005c0 0x7fdb98196860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:54.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.296+0000 7fdb9d51b640 1 -- 172.21.15.78:0/237245930 >> 172.21.15.78:0/237245930 conn(0x7fdb980fbb30 msgr2=0x7fdb98104a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:54.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.296+0000 7fdb9d51b640 1 -- 172.21.15.78:0/237245930 shutdown_connections 2024-03-24T00:20:54.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.296+0000 7fdb9d51b640 1 -- 172.21.15.78:0/237245930 wait complete. 2024-03-24T00:20:54.309 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:14:04.534391+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:18:42.178418+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:18:42.800424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T00:18:46.112800+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T00:20:54.309 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:20:54.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.414+0000 7fe3c3377640 1 -- 172.21.15.78:0/3616906625 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3bc0fff20 msgr2=0x7fe3bc102350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:54.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.414+0000 7fe3c3377640 1 --2- 172.21.15.78:0/3616906625 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3bc0fff20 0x7fe3bc102350 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8009900 tx=0x7fe3b802bc30 comp rx=0 tx=0).stop 2024-03-24T00:20:54.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.414+0000 7fe3c3377640 1 -- 172.21.15.78:0/3616906625 shutdown_connections 2024-03-24T00:20:54.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.414+0000 7fe3c3377640 1 --2- 172.21.15.78:0/3616906625 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3bc105240 0x7fe3bc1056c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:54.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.414+0000 7fe3c3377640 1 --2- 172.21.15.78:0/3616906625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3bc102890 0x7fe3bc104d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:54.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.414+0000 7fe3c3377640 1 --2- 172.21.15.78:0/3616906625 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3bc0fff20 0x7fe3bc102350 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:54.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.414+0000 7fe3c3377640 1 -- 172.21.15.78:0/3616906625 >> 172.21.15.78:0/3616906625 conn(0x7fe3bc0fbb40 msgr2=0x7fe3bc0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:54.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.415+0000 7fe3c3377640 1 -- 172.21.15.78:0/3616906625 shutdown_connections 2024-03-24T00:20:54.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.415+0000 7fe3c3377640 1 -- 172.21.15.78:0/3616906625 wait complete. 2024-03-24T00:20:54.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.415+0000 7fe3c3377640 1 Processor -- start 2024-03-24T00:20:54.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.415+0000 7fe3c3377640 1 -- start start 2024-03-24T00:20:54.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.415+0000 7fe3c3377640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3bc0fff20 0x7fe3bc19e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:54.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.415+0000 7fe3c3377640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3bc102890 0x7fe3bc19e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:54.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.415+0000 7fe3c3377640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3bc105240 0x7fe3bc1a5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:54.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.415+0000 7fe3c3377640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc113920 con 0x7fe3bc102890 2024-03-24T00:20:54.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.415+0000 7fe3c3377640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc1137a0 con 0x7fe3bc105240 2024-03-24T00:20:54.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.415+0000 7fe3c3377640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc113aa0 con 0x7fe3bc0fff20 2024-03-24T00:20:54.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.415+0000 7fe3c0912640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3bc102890 0x7fe3bc19e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:54.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.415+0000 7fe3c0912640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3bc102890 0x7fe3bc19e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38014/0 (socket says 172.21.15.78:38014) 2024-03-24T00:20:54.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.415+0000 7fe3c0912640 1 -- 172.21.15.78:0/1259093562 learned_addr learned my addr 172.21.15.78:0/1259093562 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:54.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.415+0000 7fe3c1113640 1 --2- 172.21.15.78:0/1259093562 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3bc0fff20 0x7fe3bc19e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:54.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.415+0000 7fe3c1914640 1 --2- 172.21.15.78:0/1259093562 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3bc105240 0x7fe3bc1a5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:54.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.416+0000 7fe3c0912640 1 -- 172.21.15.78:0/1259093562 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3bc0fff20 msgr2=0x7fe3bc19e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:54.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.416+0000 7fe3c0912640 1 --2- 172.21.15.78:0/1259093562 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3bc0fff20 0x7fe3bc19e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:54.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.416+0000 7fe3c0912640 1 -- 172.21.15.78:0/1259093562 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3bc105240 msgr2=0x7fe3bc1a5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:54.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.416+0000 7fe3c0912640 1 --2- 172.21.15.78:0/1259093562 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3bc105240 0x7fe3bc1a5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:54.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.416+0000 7fe3c0912640 1 -- 172.21.15.78:0/1259093562 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3bc1a5c00 con 0x7fe3bc102890 2024-03-24T00:20:54.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.416+0000 7fe3c0912640 1 --2- 172.21.15.78:0/1259093562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3bc102890 0x7fe3bc19e570 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac00ea60 tx=0x7fe3ac00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:54.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.416+0000 7fe3b27fc640 1 -- 172.21.15.78:0/1259093562 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ac00cde0 con 0x7fe3bc102890 2024-03-24T00:20:54.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.416+0000 7fe3b27fc640 1 -- 172.21.15.78:0/1259093562 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe3ac0035f0 con 0x7fe3bc102890 2024-03-24T00:20:54.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.416+0000 7fe3c3377640 1 -- 172.21.15.78:0/1259093562 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3bc1a5ef0 con 0x7fe3bc102890 2024-03-24T00:20:54.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.416+0000 7fe3b27fc640 1 -- 172.21.15.78:0/1259093562 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ac010670 con 0x7fe3bc102890 2024-03-24T00:20:54.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.416+0000 7fe3c3377640 1 -- 172.21.15.78:0/1259093562 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3bc1a63b0 con 0x7fe3bc102890 2024-03-24T00:20:54.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.417+0000 7fe3c3377640 1 -- 172.21.15.78:0/1259093562 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3bc0fd630 con 0x7fe3bc102890 2024-03-24T00:20:54.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.418+0000 7fe3b27fc640 1 -- 172.21.15.78:0/1259093562 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe3ac0037a0 con 0x7fe3bc102890 2024-03-24T00:20:54.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.418+0000 7fe3b27fc640 1 --2- 172.21.15.78:0/1259093562 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe3a80a2800 0x7fe3a80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:54.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.419+0000 7fe3c1113640 1 --2- 172.21.15.78:0/1259093562 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe3a80a2800 0x7fe3a80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:54.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.419+0000 7fe3b27fc640 1 -- 172.21.15.78:0/1259093562 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7fe3ac0cf0d0 con 0x7fe3bc102890 2024-03-24T00:20:54.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.419+0000 7fe3c1113640 1 --2- 172.21.15.78:0/1259093562 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe3a80a2800 0x7fe3a80a4cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe3b802c7b0 tx=0x7fe3b8005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:54.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.423+0000 7fe3b27fc640 1 -- 172.21.15.78:0/1259093562 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe3ac09bde0 con 0x7fe3bc102890 2024-03-24T00:20:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:54 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/800587219' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T00:20:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:54 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2418702720' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:20:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:54 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/237245930' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:20:54.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.597+0000 7fe3c3377640 1 -- 172.21.15.78:0/1259093562 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe3bc062340 con 0x7fe3bc102890 2024-03-24T00:20:54.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.598+0000 7fe3b27fc640 1 -- 172.21.15.78:0/1259093562 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe3ac0860f0 con 0x7fe3bc102890 2024-03-24T00:20:54.601 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-24T00:20:54.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.599+0000 7fe3c3377640 1 -- 172.21.15.78:0/1259093562 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe3a80a2800 msgr2=0x7fe3a80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:54.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.600+0000 7fe3c3377640 1 --2- 172.21.15.78:0/1259093562 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe3a80a2800 0x7fe3a80a4cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe3b802c7b0 tx=0x7fe3b8005fd0 comp rx=0 tx=0).stop 2024-03-24T00:20:54.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.600+0000 7fe3c3377640 1 -- 172.21.15.78:0/1259093562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3bc102890 msgr2=0x7fe3bc19e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:54.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.600+0000 7fe3c3377640 1 --2- 172.21.15.78:0/1259093562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3bc102890 0x7fe3bc19e570 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac00ea60 tx=0x7fe3ac00ef20 comp rx=0 tx=0).stop 2024-03-24T00:20:54.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.600+0000 7fe3c3377640 1 -- 172.21.15.78:0/1259093562 shutdown_connections 2024-03-24T00:20:54.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.600+0000 7fe3c3377640 1 --2- 172.21.15.78:0/1259093562 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe3a80a2800 0x7fe3a80a4cc0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:54.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.600+0000 7fe3c3377640 1 --2- 172.21.15.78:0/1259093562 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3bc105240 0x7fe3bc1a5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:54.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.600+0000 7fe3c3377640 1 --2- 172.21.15.78:0/1259093562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3bc102890 0x7fe3bc19e570 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:54.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.600+0000 7fe3c3377640 1 --2- 172.21.15.78:0/1259093562 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3bc0fff20 0x7fe3bc19e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:54.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.600+0000 7fe3c3377640 1 -- 172.21.15.78:0/1259093562 >> 172.21.15.78:0/1259093562 conn(0x7fe3bc0fbb40 msgr2=0x7fe3bc103240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:54.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.600+0000 7fe3c3377640 1 -- 172.21.15.78:0/1259093562 shutdown_connections 2024-03-24T00:20:54.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.600+0000 7fe3c3377640 1 -- 172.21.15.78:0/1259093562 wait complete. 2024-03-24T00:20:54.612 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:20:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:54 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/800587219' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T00:20:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:54 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2418702720' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:20:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:54 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/237245930' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:20:54.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.715+0000 7f9ff4153640 1 -- 172.21.15.78:0/1507449362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fec102cb0 msgr2=0x7f9fec1030d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:54.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.715+0000 7f9ff4153640 1 --2- 172.21.15.78:0/1507449362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fec102cb0 0x7f9fec1030d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc009900 tx=0x7f9fdc02bc30 comp rx=0 tx=0).stop 2024-03-24T00:20:54.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.715+0000 7f9ff4153640 1 -- 172.21.15.78:0/1507449362 shutdown_connections 2024-03-24T00:20:54.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.715+0000 7f9ff4153640 1 --2- 172.21.15.78:0/1507449362 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9fec1000e0 0x7f9fec109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:54.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.715+0000 7f9ff4153640 1 --2- 172.21.15.78:0/1507449362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fec0ff720 0x7f9fec0ffba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:54.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.715+0000 7f9ff4153640 1 --2- 172.21.15.78:0/1507449362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fec102cb0 0x7f9fec1030d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:54.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.715+0000 7f9ff4153640 1 -- 172.21.15.78:0/1507449362 >> 172.21.15.78:0/1507449362 conn(0x7f9fec0fb2a0 msgr2=0x7f9fec0fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:54.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.715+0000 7f9ff4153640 1 -- 172.21.15.78:0/1507449362 shutdown_connections 2024-03-24T00:20:54.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.715+0000 7f9ff4153640 1 -- 172.21.15.78:0/1507449362 wait complete. 2024-03-24T00:20:54.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.716+0000 7f9ff4153640 1 Processor -- start 2024-03-24T00:20:54.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.716+0000 7f9ff4153640 1 -- start start 2024-03-24T00:20:54.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.716+0000 7f9ff4153640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fec0ff720 0x7f9fec197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:54.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.716+0000 7f9ff4153640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fec1000e0 0x7f9fec197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:54.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.716+0000 7f9ff4153640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9fec102cb0 0x7f9fec19efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:54.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.716+0000 7f9ff4153640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fec10d350 con 0x7f9fec0ff720 2024-03-24T00:20:54.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.716+0000 7f9ff4153640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fec10d1d0 con 0x7f9fec102cb0 2024-03-24T00:20:54.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.716+0000 7f9ff4153640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fec10d4d0 con 0x7f9fec1000e0 2024-03-24T00:20:54.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.716+0000 7f9ff1eef640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fec0ff720 0x7f9fec197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:54.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.716+0000 7f9ff1eef640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fec0ff720 0x7f9fec197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55320/0 (socket says 172.21.15.78:55320) 2024-03-24T00:20:54.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.716+0000 7f9ff1eef640 1 -- 172.21.15.78:0/2814608158 learned_addr learned my addr 172.21.15.78:0/2814608158 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:54.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.716+0000 7f9ff16ee640 1 --2- 172.21.15.78:0/2814608158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fec1000e0 0x7f9fec197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:54.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.716+0000 7f9ff26f0640 1 --2- 172.21.15.78:0/2814608158 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9fec102cb0 0x7f9fec19efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:54.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.716+0000 7f9ff1eef640 1 -- 172.21.15.78:0/2814608158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fec1000e0 msgr2=0x7f9fec197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:54.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.716+0000 7f9ff1eef640 1 --2- 172.21.15.78:0/2814608158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fec1000e0 0x7f9fec197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:54.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.716+0000 7f9ff1eef640 1 -- 172.21.15.78:0/2814608158 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9fec102cb0 msgr2=0x7f9fec19efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:54.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.716+0000 7f9ff1eef640 1 --2- 172.21.15.78:0/2814608158 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9fec102cb0 0x7f9fec19efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:54.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.716+0000 7f9ff1eef640 1 -- 172.21.15.78:0/2814608158 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fec19f620 con 0x7f9fec0ff720 2024-03-24T00:20:54.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.716+0000 7f9ff1eef640 1 --2- 172.21.15.78:0/2814608158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fec0ff720 0x7f9fec197a00 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc000c00 tx=0x7f9fdc02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:54.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.716+0000 7f9fe2ffd640 1 -- 172.21.15.78:0/2814608158 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc041070 con 0x7f9fec0ff720 2024-03-24T00:20:54.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.716+0000 7f9ff4153640 1 -- 172.21.15.78:0/2814608158 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fec19f8b0 con 0x7f9fec0ff720 2024-03-24T00:20:54.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.716+0000 7f9fe2ffd640 1 -- 172.21.15.78:0/2814608158 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9fdc004450 con 0x7f9fec0ff720 2024-03-24T00:20:54.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.716+0000 7f9fe2ffd640 1 -- 172.21.15.78:0/2814608158 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc034440 con 0x7f9fec0ff720 2024-03-24T00:20:54.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.717+0000 7f9ff4153640 1 -- 172.21.15.78:0/2814608158 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fec19fdf0 con 0x7f9fec0ff720 2024-03-24T00:20:54.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.718+0000 7f9ff4153640 1 -- 172.21.15.78:0/2814608158 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fec108b90 con 0x7f9fec0ff720 2024-03-24T00:20:54.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.718+0000 7f9fe2ffd640 1 -- 172.21.15.78:0/2814608158 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9fdc044070 con 0x7f9fec0ff720 2024-03-24T00:20:54.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.719+0000 7f9fe2ffd640 1 --2- 172.21.15.78:0/2814608158 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f9fd80a2800 0x7f9fd80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:54.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.719+0000 7f9ff16ee640 1 --2- 172.21.15.78:0/2814608158 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f9fd80a2800 0x7f9fd80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:54.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.719+0000 7f9fe2ffd640 1 -- 172.21.15.78:0/2814608158 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f9fdc0f08b0 con 0x7f9fec0ff720 2024-03-24T00:20:54.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.719+0000 7f9ff16ee640 1 --2- 172.21.15.78:0/2814608158 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f9fd80a2800 0x7f9fd80a4cc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9fe80095f0 tx=0x7f9fe8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:54.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.723+0000 7f9fe2ffd640 1 -- 172.21.15.78:0/2814608158 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9fdc0bcf70 con 0x7f9fec0ff720 2024-03-24T00:20:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:54 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/800587219' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T00:20:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:54 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2418702720' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:20:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:54 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/237245930' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:20:54.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.896+0000 7f9ff4153640 1 -- 172.21.15.78:0/2814608158 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9fec0622a0 con 0x7f9fec0ff720 2024-03-24T00:20:54.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.897+0000 7f9fe2ffd640 1 -- 172.21.15.78:0/2814608158 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9fdc0c1010 con 0x7f9fec0ff720 2024-03-24T00:20:54.897 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-24T00:20:54.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.899+0000 7f9ff4153640 1 -- 172.21.15.78:0/2814608158 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f9fd80a2800 msgr2=0x7f9fd80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:54.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.899+0000 7f9ff4153640 1 --2- 172.21.15.78:0/2814608158 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f9fd80a2800 0x7f9fd80a4cc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9fe80095f0 tx=0x7f9fe8005f90 comp rx=0 tx=0).stop 2024-03-24T00:20:54.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.899+0000 7f9ff4153640 1 -- 172.21.15.78:0/2814608158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fec0ff720 msgr2=0x7f9fec197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:54.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.899+0000 7f9ff4153640 1 --2- 172.21.15.78:0/2814608158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fec0ff720 0x7f9fec197a00 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc000c00 tx=0x7f9fdc02c650 comp rx=0 tx=0).stop 2024-03-24T00:20:54.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.899+0000 7f9ff4153640 1 -- 172.21.15.78:0/2814608158 shutdown_connections 2024-03-24T00:20:54.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.899+0000 7f9ff4153640 1 --2- 172.21.15.78:0/2814608158 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f9fd80a2800 0x7f9fd80a4cc0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:54.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.899+0000 7f9ff4153640 1 --2- 172.21.15.78:0/2814608158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fec1000e0 0x7f9fec197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:54.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.899+0000 7f9ff4153640 1 --2- 172.21.15.78:0/2814608158 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9fec102cb0 0x7f9fec19efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:54.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.899+0000 7f9ff4153640 1 --2- 172.21.15.78:0/2814608158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fec0ff720 0x7f9fec197a00 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:54.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.899+0000 7f9ff4153640 1 -- 172.21.15.78:0/2814608158 >> 172.21.15.78:0/2814608158 conn(0x7f9fec0fb2a0 msgr2=0x7f9fec105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:54.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.899+0000 7f9ff4153640 1 -- 172.21.15.78:0/2814608158 shutdown_connections 2024-03-24T00:20:54.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:54.899+0000 7f9ff4153640 1 -- 172.21.15.78:0/2814608158 wait complete. 2024-03-24T00:20:54.911 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:20:55.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.016+0000 7f4731aea640 1 -- 172.21.15.78:0/142872153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f472c102150 msgr2=0x7f472c1089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:55.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.016+0000 7f4731aea640 1 --2- 172.21.15.78:0/142872153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f472c102150 0x7f472c1089e0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f4724009ca0 tx=0x7f472402c0f0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.016+0000 7f4731aea640 1 -- 172.21.15.78:0/142872153 shutdown_connections 2024-03-24T00:20:55.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.016+0000 7f4731aea640 1 --2- 172.21.15.78:0/142872153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f472c102150 0x7f472c1089e0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.016+0000 7f4731aea640 1 --2- 172.21.15.78:0/142872153 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f472c101790 0x7f472c101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.016+0000 7f4731aea640 1 --2- 172.21.15.78:0/142872153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f472c1005a0 0x7f472c1009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.016+0000 7f4731aea640 1 -- 172.21.15.78:0/142872153 >> 172.21.15.78:0/142872153 conn(0x7f472c0fbb30 msgr2=0x7f472c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:55.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.016+0000 7f4731aea640 1 -- 172.21.15.78:0/142872153 shutdown_connections 2024-03-24T00:20:55.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.016+0000 7f4731aea640 1 -- 172.21.15.78:0/142872153 wait complete. 2024-03-24T00:20:55.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.016+0000 7f4731aea640 1 Processor -- start 2024-03-24T00:20:55.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.016+0000 7f4731aea640 1 -- start start 2024-03-24T00:20:55.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.016+0000 7f4731aea640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f472c1005a0 0x7f472c197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:55.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.016+0000 7f4731aea640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f472c101790 0x7f472c197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:55.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.016+0000 7f4731aea640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f472c102150 0x7f472c19ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:55.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.016+0000 7f4731aea640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472c10c230 con 0x7f472c102150 2024-03-24T00:20:55.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.016+0000 7f4731aea640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472c10c0b0 con 0x7f472c101790 2024-03-24T00:20:55.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.016+0000 7f4731aea640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472c10c3b0 con 0x7f472c1005a0 2024-03-24T00:20:55.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.017+0000 7f472bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f472c102150 0x7f472c19ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:55.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.017+0000 7f472bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f472c102150 0x7f472c19ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55330/0 (socket says 172.21.15.78:55330) 2024-03-24T00:20:55.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.017+0000 7f472bfff640 1 -- 172.21.15.78:0/906745921 learned_addr learned my addr 172.21.15.78:0/906745921 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:55.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.017+0000 7f4723fff640 1 --2- 172.21.15.78:0/906745921 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f472c101790 0x7f472c197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:55.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.017+0000 7f472b7fe640 1 --2- 172.21.15.78:0/906745921 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f472c1005a0 0x7f472c197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:55.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.017+0000 7f472bfff640 1 -- 172.21.15.78:0/906745921 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f472c1005a0 msgr2=0x7f472c197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:55.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.017+0000 7f472bfff640 1 --2- 172.21.15.78:0/906745921 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f472c1005a0 0x7f472c197990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.017+0000 7f472bfff640 1 -- 172.21.15.78:0/906745921 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f472c101790 msgr2=0x7f472c197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:55.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.017+0000 7f472bfff640 1 --2- 172.21.15.78:0/906745921 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f472c101790 0x7f472c197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.017+0000 7f472bfff640 1 -- 172.21.15.78:0/906745921 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f472c19f5b0 con 0x7f472c102150 2024-03-24T00:20:55.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.017+0000 7f472b7fe640 1 --2- 172.21.15.78:0/906745921 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f472c1005a0 0x7f472c197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:20:55.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.017+0000 7f472bfff640 1 --2- 172.21.15.78:0/906745921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f472c102150 0x7f472c19ef40 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f472402c630 tx=0x7f4724032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:55.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.017+0000 7f47297fa640 1 -- 172.21.15.78:0/906745921 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4724041020 con 0x7f472c102150 2024-03-24T00:20:55.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.017+0000 7f47297fa640 1 -- 172.21.15.78:0/906745921 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4724034cd0 con 0x7f472c102150 2024-03-24T00:20:55.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.017+0000 7f4731aea640 1 -- 172.21.15.78:0/906745921 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f472c19f840 con 0x7f472c102150 2024-03-24T00:20:55.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.017+0000 7f47297fa640 1 -- 172.21.15.78:0/906745921 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4724039040 con 0x7f472c102150 2024-03-24T00:20:55.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.017+0000 7f4731aea640 1 -- 172.21.15.78:0/906745921 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f472c19fd80 con 0x7f472c102150 2024-03-24T00:20:55.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.018+0000 7f4731aea640 1 -- 172.21.15.78:0/906745921 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f472c19ffe0 con 0x7f472c102150 2024-03-24T00:20:55.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.019+0000 7f47297fa640 1 -- 172.21.15.78:0/906745921 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4724044020 con 0x7f472c102150 2024-03-24T00:20:55.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.020+0000 7f47297fa640 1 --2- 172.21.15.78:0/906745921 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f47180a2880 0x7f47180a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:55.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.020+0000 7f472b7fe640 1 --2- 172.21.15.78:0/906745921 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f47180a2880 0x7f47180a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:55.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.020+0000 7f47297fa640 1 -- 172.21.15.78:0/906745921 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f47240f1050 con 0x7f472c102150 2024-03-24T00:20:55.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.020+0000 7f472b7fe640 1 --2- 172.21.15.78:0/906745921 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f47180a2880 0x7f47180a4d40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f472c1015f0 tx=0x7f471c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:55.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.023+0000 7f47297fa640 1 -- 172.21.15.78:0/906745921 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f472404b020 con 0x7f472c102150 2024-03-24T00:20:55.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.196+0000 7f4731aea640 1 -- 172.21.15.78:0/906745921 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f472c1a01b0 con 0x7f472c102150 2024-03-24T00:20:55.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.197+0000 7f47297fa640 1 -- 172.21.15.78:0/906745921 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7f47240bddf0 con 0x7f472c102150 2024-03-24T00:20:55.198 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-03-24T00:20:55.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.199+0000 7f4731aea640 1 -- 172.21.15.78:0/906745921 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f47180a2880 msgr2=0x7f47180a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:55.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.199+0000 7f4731aea640 1 --2- 172.21.15.78:0/906745921 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f47180a2880 0x7f47180a4d40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f472c1015f0 tx=0x7f471c005f90 comp rx=0 tx=0).stop 2024-03-24T00:20:55.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.199+0000 7f4731aea640 1 -- 172.21.15.78:0/906745921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f472c102150 msgr2=0x7f472c19ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:55.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.199+0000 7f4731aea640 1 --2- 172.21.15.78:0/906745921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f472c102150 0x7f472c19ef40 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f472402c630 tx=0x7f4724032ea0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.199+0000 7f4731aea640 1 -- 172.21.15.78:0/906745921 shutdown_connections 2024-03-24T00:20:55.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.199+0000 7f4731aea640 1 --2- 172.21.15.78:0/906745921 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f47180a2880 0x7f47180a4d40 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.199+0000 7f4731aea640 1 --2- 172.21.15.78:0/906745921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f472c102150 0x7f472c19ef40 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.199+0000 7f4731aea640 1 --2- 172.21.15.78:0/906745921 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f472c101790 0x7f472c197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.199+0000 7f4731aea640 1 --2- 172.21.15.78:0/906745921 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f472c1005a0 0x7f472c197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.199+0000 7f4731aea640 1 -- 172.21.15.78:0/906745921 >> 172.21.15.78:0/906745921 conn(0x7f472c0fbb30 msgr2=0x7f472c068800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:55.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.199+0000 7f4731aea640 1 -- 172.21.15.78:0/906745921 shutdown_connections 2024-03-24T00:20:55.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.199+0000 7f4731aea640 1 -- 172.21.15.78:0/906745921 wait complete. 2024-03-24T00:20:55.211 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T00:20:55.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.315+0000 7f73119e8640 1 -- 172.21.15.78:0/2137838271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f730c1028a0 msgr2=0x7f730c102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:55.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.315+0000 7f73119e8640 1 --2- 172.21.15.78:0/2137838271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f730c1028a0 0x7f730c102d20 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f7300009900 tx=0x7f730002bc30 comp rx=0 tx=0).stop 2024-03-24T00:20:55.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.315+0000 7f73119e8640 1 -- 172.21.15.78:0/2137838271 shutdown_connections 2024-03-24T00:20:55.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.315+0000 7f73119e8640 1 --2- 172.21.15.78:0/2137838271 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f730c103260 0x7f730c109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.315+0000 7f73119e8640 1 --2- 172.21.15.78:0/2137838271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f730c1028a0 0x7f730c102d20 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.315+0000 7f73119e8640 1 --2- 172.21.15.78:0/2137838271 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f730c1016b0 0x7f730c101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.315+0000 7f73119e8640 1 -- 172.21.15.78:0/2137838271 >> 172.21.15.78:0/2137838271 conn(0x7f730c0fcc40 msgr2=0x7f730c0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:55.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.315+0000 7f73119e8640 1 -- 172.21.15.78:0/2137838271 shutdown_connections 2024-03-24T00:20:55.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.315+0000 7f73119e8640 1 -- 172.21.15.78:0/2137838271 wait complete. 2024-03-24T00:20:55.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.316+0000 7f73119e8640 1 Processor -- start 2024-03-24T00:20:55.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.316+0000 7f73119e8640 1 -- start start 2024-03-24T00:20:55.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.316+0000 7f73119e8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f730c1016b0 0x7f730c197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:55.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.316+0000 7f73119e8640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f730c1028a0 0x7f730c197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:55.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.316+0000 7f73119e8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f730c103260 0x7f730c19efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:55.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.316+0000 7f73119e8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730c10d310 con 0x7f730c1016b0 2024-03-24T00:20:55.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.316+0000 7f73119e8640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730c10d190 con 0x7f730c1028a0 2024-03-24T00:20:55.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.316+0000 7f73119e8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730c10d490 con 0x7f730c103260 2024-03-24T00:20:55.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.316+0000 7f730affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f730c1016b0 0x7f730c197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:55.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.316+0000 7f730affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f730c1016b0 0x7f730c197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55362/0 (socket says 172.21.15.78:55362) 2024-03-24T00:20:55.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.316+0000 7f730affd640 1 -- 172.21.15.78:0/104404465 learned_addr learned my addr 172.21.15.78:0/104404465 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:55.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.316+0000 7f730a7fc640 1 --2- 172.21.15.78:0/104404465 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f730c1028a0 0x7f730c197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:55.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.317+0000 7f730b7fe640 1 --2- 172.21.15.78:0/104404465 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f730c103260 0x7f730c19efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:55.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.317+0000 7f730affd640 1 -- 172.21.15.78:0/104404465 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f730c103260 msgr2=0x7f730c19efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:55.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.317+0000 7f730affd640 1 --2- 172.21.15.78:0/104404465 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f730c103260 0x7f730c19efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.317+0000 7f730affd640 1 -- 172.21.15.78:0/104404465 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f730c1028a0 msgr2=0x7f730c197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:55.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.317+0000 7f730affd640 1 --2- 172.21.15.78:0/104404465 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f730c1028a0 0x7f730c197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.317+0000 7f730affd640 1 -- 172.21.15.78:0/104404465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730c19f630 con 0x7f730c1016b0 2024-03-24T00:20:55.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.317+0000 7f730affd640 1 --2- 172.21.15.78:0/104404465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f730c1016b0 0x7f730c197a10 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f72fc00e9c0 tx=0x7f72fc00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:55.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.317+0000 7f73109e6640 1 -- 172.21.15.78:0/104404465 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc00ccc0 con 0x7f730c1016b0 2024-03-24T00:20:55.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.317+0000 7f73109e6640 1 -- 172.21.15.78:0/104404465 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72fc0035f0 con 0x7f730c1016b0 2024-03-24T00:20:55.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.317+0000 7f73109e6640 1 -- 172.21.15.78:0/104404465 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc00ce60 con 0x7f730c1016b0 2024-03-24T00:20:55.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.317+0000 7f73119e8640 1 -- 172.21.15.78:0/104404465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f730c19f920 con 0x7f730c1016b0 2024-03-24T00:20:55.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.317+0000 7f73119e8640 1 -- 172.21.15.78:0/104404465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f730c19fe40 con 0x7f730c1016b0 2024-03-24T00:20:55.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.318+0000 7f73119e8640 1 -- 172.21.15.78:0/104404465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f730c108a50 con 0x7f730c1016b0 2024-03-24T00:20:55.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.319+0000 7f73109e6640 1 -- 172.21.15.78:0/104404465 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f72fc0037a0 con 0x7f730c1016b0 2024-03-24T00:20:55.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.320+0000 7f73109e6640 1 --2- 172.21.15.78:0/104404465 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f72f40a2800 0x7f72f40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:55.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.320+0000 7f730a7fc640 1 --2- 172.21.15.78:0/104404465 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f72f40a2800 0x7f72f40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:55.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.320+0000 7f73109e6640 1 -- 172.21.15.78:0/104404465 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f72fc0cf150 con 0x7f730c1016b0 2024-03-24T00:20:55.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.320+0000 7f730a7fc640 1 --2- 172.21.15.78:0/104404465 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f72f40a2800 0x7f72f40a4cc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f730002c7b0 tx=0x7f7300005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:55.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.324+0000 7f73109e6640 1 -- 172.21.15.78:0/104404465 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f72fc09be60 con 0x7f730c1016b0 2024-03-24T00:20:55.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.501+0000 7f73119e8640 1 -- 172.21.15.78:0/104404465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f730c062360 con 0x7f730c1016b0 2024-03-24T00:20:55.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.502+0000 7f73109e6640 1 -- 172.21.15.78:0/104404465 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7f72fc09b760 con 0x7f730c1016b0 2024-03-24T00:20:55.503 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-24T00:20:55.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.504+0000 7f73119e8640 1 -- 172.21.15.78:0/104404465 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f72f40a2800 msgr2=0x7f72f40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:55.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.504+0000 7f73119e8640 1 --2- 172.21.15.78:0/104404465 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f72f40a2800 0x7f72f40a4cc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f730002c7b0 tx=0x7f7300005fd0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.504+0000 7f73119e8640 1 -- 172.21.15.78:0/104404465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f730c1016b0 msgr2=0x7f730c197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:55.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.504+0000 7f73119e8640 1 --2- 172.21.15.78:0/104404465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f730c1016b0 0x7f730c197a10 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f72fc00e9c0 tx=0x7f72fc00edc0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.504+0000 7f73119e8640 1 -- 172.21.15.78:0/104404465 shutdown_connections 2024-03-24T00:20:55.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.504+0000 7f73119e8640 1 --2- 172.21.15.78:0/104404465 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f72f40a2800 0x7f72f40a4cc0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.504+0000 7f73119e8640 1 --2- 172.21.15.78:0/104404465 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f730c103260 0x7f730c19efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.504+0000 7f73119e8640 1 --2- 172.21.15.78:0/104404465 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f730c1028a0 0x7f730c197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.504+0000 7f73119e8640 1 --2- 172.21.15.78:0/104404465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f730c1016b0 0x7f730c197a10 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.504+0000 7f73119e8640 1 -- 172.21.15.78:0/104404465 >> 172.21.15.78:0/104404465 conn(0x7f730c0fcc40 msgr2=0x7f730c105c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:55.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.504+0000 7f73119e8640 1 -- 172.21.15.78:0/104404465 shutdown_connections 2024-03-24T00:20:55.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.504+0000 7f73119e8640 1 -- 172.21.15.78:0/104404465 wait complete. 2024-03-24T00:20:55.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:55 smithi078 ceph-mon[32429]: pgmap v360: 129 pgs: 129 active+clean; 641 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 345 B/s wr, 7 op/s; 9 B/s, 0 objects/s recovering 2024-03-24T00:20:55.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:55 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1259093562' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:20:55.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:55 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2814608158' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:20:55.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:55 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/906745921' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:20:55.517 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:20:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.618+0000 7f4cdeeac640 1 -- 172.21.15.78:0/2505433555 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4cd8101210 msgr2=0x7f4cd8108ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.618+0000 7f4cdeeac640 1 --2- 172.21.15.78:0/2505433555 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4cd8101210 0x7f4cd8108ab0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0009cd0 tx=0x7f4cd002c0f0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.618+0000 7f4cdeeac640 1 -- 172.21.15.78:0/2505433555 shutdown_connections 2024-03-24T00:20:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.618+0000 7f4cdeeac640 1 --2- 172.21.15.78:0/2505433555 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4cd8101210 0x7f4cd8108ab0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.618+0000 7f4cdeeac640 1 --2- 172.21.15.78:0/2505433555 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cd8100850 0x7f4cd8100cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.618+0000 7f4cdeeac640 1 --2- 172.21.15.78:0/2505433555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cd8101bb0 0x7f4cd8101fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.618+0000 7f4cdeeac640 1 -- 172.21.15.78:0/2505433555 >> 172.21.15.78:0/2505433555 conn(0x7f4cd80fb2c0 msgr2=0x7f4cd80fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.618+0000 7f4cdeeac640 1 -- 172.21.15.78:0/2505433555 shutdown_connections 2024-03-24T00:20:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.618+0000 7f4cdeeac640 1 -- 172.21.15.78:0/2505433555 wait complete. 2024-03-24T00:20:55.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.618+0000 7f4cdeeac640 1 Processor -- start 2024-03-24T00:20:55.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.618+0000 7f4cdeeac640 1 -- start start 2024-03-24T00:20:55.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.619+0000 7f4cdeeac640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cd8100850 0x7f4cd8197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:55.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.619+0000 7f4cdeeac640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cd8101210 0x7f4cd8197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:55.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.619+0000 7f4cdeeac640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4cd8101bb0 0x7f4cd819ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:55.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.619+0000 7f4cdeeac640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd810c270 con 0x7f4cd8101bb0 2024-03-24T00:20:55.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.619+0000 7f4cdeeac640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd810c0f0 con 0x7f4cd8101210 2024-03-24T00:20:55.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.619+0000 7f4cdeeac640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd810c3f0 con 0x7f4cd8100850 2024-03-24T00:20:55.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.619+0000 7f4cdd449640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4cd8101bb0 0x7f4cd819ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:55.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.619+0000 7f4cdd449640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4cd8101bb0 0x7f4cd819ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55370/0 (socket says 172.21.15.78:55370) 2024-03-24T00:20:55.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.619+0000 7f4cdd449640 1 -- 172.21.15.78:0/541947473 learned_addr learned my addr 172.21.15.78:0/541947473 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:55.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.619+0000 7f4cdcc48640 1 --2- 172.21.15.78:0/541947473 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cd8100850 0x7f4cd8197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:55.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.619+0000 7f4cd7fff640 1 --2- 172.21.15.78:0/541947473 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cd8101210 0x7f4cd8197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:55.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.619+0000 7f4cdd449640 1 -- 172.21.15.78:0/541947473 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cd8100850 msgr2=0x7f4cd8197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:55.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.619+0000 7f4cdd449640 1 --2- 172.21.15.78:0/541947473 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cd8100850 0x7f4cd8197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.619+0000 7f4cdd449640 1 -- 172.21.15.78:0/541947473 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cd8101210 msgr2=0x7f4cd8197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:55.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.619+0000 7f4cdd449640 1 --2- 172.21.15.78:0/541947473 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cd8101210 0x7f4cd8197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.619+0000 7f4cdd449640 1 -- 172.21.15.78:0/541947473 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd819f590 con 0x7f4cd8101bb0 2024-03-24T00:20:55.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.619+0000 7f4cdd449640 1 --2- 172.21.15.78:0/541947473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4cd8101bb0 0x7f4cd819ef20 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0009ca0 tx=0x7f4cd00346d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:55.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.619+0000 7f4cd5ffb640 1 -- 172.21.15.78:0/541947473 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0041020 con 0x7f4cd8101bb0 2024-03-24T00:20:55.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.619+0000 7f4cd5ffb640 1 -- 172.21.15.78:0/541947473 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4cd0032890 con 0x7f4cd8101bb0 2024-03-24T00:20:55.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.619+0000 7f4cdeeac640 1 -- 172.21.15.78:0/541947473 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cd819f820 con 0x7f4cd8101bb0 2024-03-24T00:20:55.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.619+0000 7f4cd5ffb640 1 -- 172.21.15.78:0/541947473 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0039040 con 0x7f4cd8101bb0 2024-03-24T00:20:55.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.620+0000 7f4cdeeac640 1 -- 172.21.15.78:0/541947473 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cd819fd10 con 0x7f4cd8101bb0 2024-03-24T00:20:55.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.620+0000 7f4cdeeac640 1 -- 172.21.15.78:0/541947473 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cd819ff70 con 0x7f4cd8101bb0 2024-03-24T00:20:55.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.621+0000 7f4cd5ffb640 1 -- 172.21.15.78:0/541947473 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4cd0044020 con 0x7f4cd8101bb0 2024-03-24T00:20:55.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.622+0000 7f4cd5ffb640 1 --2- 172.21.15.78:0/541947473 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4cc80a2800 0x7f4cc80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:55.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.622+0000 7f4cdcc48640 1 --2- 172.21.15.78:0/541947473 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4cc80a2800 0x7f4cc80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:55.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.622+0000 7f4cd5ffb640 1 -- 172.21.15.78:0/541947473 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f4cd00f0b90 con 0x7f4cd8101bb0 2024-03-24T00:20:55.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.622+0000 7f4cdcc48640 1 --2- 172.21.15.78:0/541947473 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4cc80a2800 0x7f4cc80a4cc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8069b10 tx=0x7f4ccc005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:55.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.626+0000 7f4cd5ffb640 1 -- 172.21.15.78:0/541947473 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4cd004b020 con 0x7f4cd8101bb0 2024-03-24T00:20:55.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:55 smithi106 ceph-mon[36505]: pgmap v360: 129 pgs: 129 active+clean; 641 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 345 B/s wr, 7 op/s; 9 B/s, 0 objects/s recovering 2024-03-24T00:20:55.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:55 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1259093562' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:20:55.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:55 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2814608158' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:20:55.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:55 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/906745921' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:20:55.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.839+0000 7f4cdeeac640 1 -- 172.21.15.78:0/541947473 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4cd81a0300 con 0x7f4cd8101bb0 2024-03-24T00:20:55.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.840+0000 7f4cd5ffb640 1 -- 172.21.15.78:0/541947473 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11406 (secure 0 0 0) 0x7f4cd00bd8a0 con 0x7f4cd8101bb0 2024-03-24T00:20:55.841 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:20:55.841 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1416685123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1416685123},{"type":"v1","addr":"172.21.15.78:6835","nonce":1416685123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1071274623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1071274623},{"type":"v1","addr":"172.21.15.78:6837","nonce":1071274623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2758357551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2758357551},{"type":"v1","addr":"172.21.15.78:6839","nonce":2758357551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/3424505910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":3424505910},{"type":"v1","addr":"172.21.15.78:6841","nonce":3424505910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/3086318443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3086318443},{"type":"v1","addr":"172.21.15.107:6833","nonce":3086318443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/2367018868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2367018868},{"type":"v1","addr":"172.21.15.106:6833","nonce":2367018868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:18:43.592215+0000","modified":"2024-03-24T00:19:08.871384+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24500,"mds_1":24481,"mds_2":24464},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/2231611477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":2231611477},{"type":"v1","addr":"172.21.15.106:6835","nonce":2231611477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6835/108936439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":108936439},{"type":"v1","addr":"172.21.15.107:6835","nonce":108936439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6837/2939571540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2939571540},{"type":"v1","addr":"172.21.15.107:6837","nonce":2939571540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24464":{"gid":24464,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.106:6837/2090968633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":2090968633},{"type":"v1","addr":"172.21.15.106:6837","nonce":2090968633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.107:6839/117119462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":117119462},{"type":"v1","addr":"172.21.15.107:6839","nonce":117119462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6839/2482071304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2482071304},{"type":"v1","addr":"172.21.15.106:6839","nonce":2482071304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24481,24500,24464]},"id":1}]} 2024-03-24T00:20:55.841 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 23 2024-03-24T00:20:55.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.842+0000 7f4cdeeac640 1 -- 172.21.15.78:0/541947473 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4cc80a2800 msgr2=0x7f4cc80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:55.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.842+0000 7f4cdeeac640 1 --2- 172.21.15.78:0/541947473 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4cc80a2800 0x7f4cc80a4cc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8069b10 tx=0x7f4ccc005f90 comp rx=0 tx=0).stop 2024-03-24T00:20:55.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.842+0000 7f4cdeeac640 1 -- 172.21.15.78:0/541947473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4cd8101bb0 msgr2=0x7f4cd819ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:55.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.842+0000 7f4cdeeac640 1 --2- 172.21.15.78:0/541947473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4cd8101bb0 0x7f4cd819ef20 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0009ca0 tx=0x7f4cd00346d0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.842+0000 7f4cdeeac640 1 -- 172.21.15.78:0/541947473 shutdown_connections 2024-03-24T00:20:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.842+0000 7f4cdeeac640 1 --2- 172.21.15.78:0/541947473 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4cc80a2800 0x7f4cc80a4cc0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.842+0000 7f4cdeeac640 1 --2- 172.21.15.78:0/541947473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4cd8101bb0 0x7f4cd819ef20 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.842+0000 7f4cdeeac640 1 --2- 172.21.15.78:0/541947473 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cd8101210 0x7f4cd8197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.842+0000 7f4cdeeac640 1 --2- 172.21.15.78:0/541947473 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cd8100850 0x7f4cd8197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.842+0000 7f4cdeeac640 1 -- 172.21.15.78:0/541947473 >> 172.21.15.78:0/541947473 conn(0x7f4cd80fb2c0 msgr2=0x7f4cd80fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.842+0000 7f4cdeeac640 1 -- 172.21.15.78:0/541947473 shutdown_connections 2024-03-24T00:20:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.842+0000 7f4cdeeac640 1 -- 172.21.15.78:0/541947473 wait complete. 2024-03-24T00:20:55.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:55 smithi107 ceph-mon[36485]: pgmap v360: 129 pgs: 129 active+clean; 641 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 345 B/s wr, 7 op/s; 9 B/s, 0 objects/s recovering 2024-03-24T00:20:55.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:55 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1259093562' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:20:55.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:55 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2814608158' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:20:55.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:55 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/906745921' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:20:55.855 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:20:55.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.958+0000 7f4cd2259640 1 -- 172.21.15.78:0/2303500032 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ccc102170 msgr2=0x7f4ccc108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:55.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.958+0000 7f4cd2259640 1 --2- 172.21.15.78:0/2303500032 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ccc102170 0x7f4ccc108a00 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f4cc4009900 tx=0x7f4cc402bc30 comp rx=0 tx=0).stop 2024-03-24T00:20:55.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.958+0000 7f4cd2259640 1 -- 172.21.15.78:0/2303500032 shutdown_connections 2024-03-24T00:20:55.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.958+0000 7f4cd2259640 1 --2- 172.21.15.78:0/2303500032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ccc1017b0 0x7f4ccc101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.958+0000 7f4cd2259640 1 --2- 172.21.15.78:0/2303500032 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ccc102170 0x7f4ccc108a00 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.958+0000 7f4cd2259640 1 --2- 172.21.15.78:0/2303500032 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ccc1005c0 0x7f4ccc1009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.958+0000 7f4cd2259640 1 -- 172.21.15.78:0/2303500032 >> 172.21.15.78:0/2303500032 conn(0x7f4ccc0fbb30 msgr2=0x7f4ccc0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:55.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.959+0000 7f4cd2259640 1 -- 172.21.15.78:0/2303500032 shutdown_connections 2024-03-24T00:20:55.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.959+0000 7f4cd2259640 1 -- 172.21.15.78:0/2303500032 wait complete. 2024-03-24T00:20:55.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.959+0000 7f4cd2259640 1 Processor -- start 2024-03-24T00:20:55.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.959+0000 7f4cd2259640 1 -- start start 2024-03-24T00:20:55.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.959+0000 7f4cd2259640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ccc1005c0 0x7f4ccc1979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:55.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.959+0000 7f4cd2259640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ccc1017b0 0x7f4ccc197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:55.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.959+0000 7f4cd2259640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ccc102170 0x7f4ccc19ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:55.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.959+0000 7f4cd2259640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ccc10c240 con 0x7f4ccc102170 2024-03-24T00:20:55.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.959+0000 7f4cd2259640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ccc10c0c0 con 0x7f4ccc1005c0 2024-03-24T00:20:55.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.959+0000 7f4cd2259640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ccc10c3c0 con 0x7f4ccc1017b0 2024-03-24T00:20:55.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.959+0000 7f4ccbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ccc102170 0x7f4ccc19ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:55.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.960+0000 7f4ccbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ccc102170 0x7f4ccc19ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55388/0 (socket says 172.21.15.78:55388) 2024-03-24T00:20:55.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.960+0000 7f4ccb7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ccc1005c0 0x7f4ccc1979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:55.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.960+0000 7f4ccbfff640 1 -- 172.21.15.78:0/2887117193 learned_addr learned my addr 172.21.15.78:0/2887117193 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:55.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.960+0000 7f4ccaffd640 1 --2- 172.21.15.78:0/2887117193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ccc1017b0 0x7f4ccc197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:55.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.960+0000 7f4ccaffd640 1 -- 172.21.15.78:0/2887117193 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ccc1005c0 msgr2=0x7f4ccc1979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:55.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.960+0000 7f4ccaffd640 1 --2- 172.21.15.78:0/2887117193 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ccc1005c0 0x7f4ccc1979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.960+0000 7f4ccaffd640 1 -- 172.21.15.78:0/2887117193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ccc102170 msgr2=0x7f4ccc19ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:55.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.960+0000 7f4ccaffd640 1 --2- 172.21.15.78:0/2887117193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ccc102170 0x7f4ccc19ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:55.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.960+0000 7f4ccaffd640 1 -- 172.21.15.78:0/2887117193 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ccc19f5c0 con 0x7f4ccc1017b0 2024-03-24T00:20:55.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.960+0000 7f4ccbfff640 1 --2- 172.21.15.78:0/2887117193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ccc102170 0x7f4ccc19ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:55.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.960+0000 7f4ccaffd640 1 --2- 172.21.15.78:0/2887117193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ccc1017b0 0x7f4ccc197ee0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4cb400e9f0 tx=0x7f4cb400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:55.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.960+0000 7f4cc8ff9640 1 -- 172.21.15.78:0/2887117193 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb400ccc0 con 0x7f4ccc1017b0 2024-03-24T00:20:55.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.960+0000 7f4cc8ff9640 1 -- 172.21.15.78:0/2887117193 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4cb4005020 con 0x7f4ccc1017b0 2024-03-24T00:20:55.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.960+0000 7f4cd2259640 1 -- 172.21.15.78:0/2887117193 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ccc19f8b0 con 0x7f4ccc1017b0 2024-03-24T00:20:55.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.960+0000 7f4cc8ff9640 1 -- 172.21.15.78:0/2887117193 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb40049e0 con 0x7f4ccc1017b0 2024-03-24T00:20:55.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.960+0000 7f4cd2259640 1 -- 172.21.15.78:0/2887117193 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ccc19fdb0 con 0x7f4ccc1017b0 2024-03-24T00:20:55.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.961+0000 7f4cd2259640 1 -- 172.21.15.78:0/2887117193 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ccc1a0070 con 0x7f4ccc1017b0 2024-03-24T00:20:55.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.962+0000 7f4cc8ff9640 1 -- 172.21.15.78:0/2887117193 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4cb4005560 con 0x7f4ccc1017b0 2024-03-24T00:20:55.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.963+0000 7f4cc8ff9640 1 --2- 172.21.15.78:0/2887117193 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4cb80a2800 0x7f4cb80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:55.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.963+0000 7f4ccb7fe640 1 --2- 172.21.15.78:0/2887117193 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4cb80a2800 0x7f4cb80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:55.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.963+0000 7f4cc8ff9640 1 -- 172.21.15.78:0/2887117193 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f4cb40cedb0 con 0x7f4ccc1017b0 2024-03-24T00:20:55.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.963+0000 7f4ccb7fe640 1 --2- 172.21.15.78:0/2887117193 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4cb80a2800 0x7f4cb80a4cc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc101610 tx=0x7f4cbc009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:55.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:55.966+0000 7f4cc8ff9640 1 -- 172.21.15.78:0/2887117193 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4cb40051d0 con 0x7f4ccc1017b0 2024-03-24T00:20:56.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.143+0000 7f4cd2259640 1 -- 172.21.15.78:0/2887117193 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ccc1a0240 con 0x7f4ccc1017b0 2024-03-24T00:20:56.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.145+0000 7f4cc8ff9640 1 -- 172.21.15.78:0/2887117193 <== mon.1 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24695 (secure 0 0 0) 0x7f4cb409ba60 con 0x7f4ccc1017b0 2024-03-24T00:20:56.146 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T00:20:56.146 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":93,"fsid":"c7e2806a-e972-11ee-95cd-87774f69a715","created":"2024-03-24T00:10:44.134563+0000","modified":"2024-03-24T00:20:42.446867+0000","last_up_change":"2024-03-24T00:17:50.237686+0000","last_in_change":"2024-03-24T00:17:29.788699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:14:04.534391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:18:42.178418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:18:42.800424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:18:46.112800+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b950388a-c437-426b-b703-3f8ae821dd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6803","nonce":1816879397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6805","nonce":1816879397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6809","nonce":1816879397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6807","nonce":1816879397}]},"public_addr":"172.21.15.78:6803/1816879397","cluster_addr":"172.21.15.78:6805/1816879397","heartbeat_back_addr":"172.21.15.78:6809/1816879397","heartbeat_front_addr":"172.21.15.78:6807/1816879397","state":["exists","up"]},{"osd":1,"uuid":"b8d64714-dd65-400f-b3f4-673ab14e1365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6801","nonce":1647915612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6803","nonce":1647915612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6807","nonce":1647915612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6805","nonce":1647915612}]},"public_addr":"172.21.15.106:6801/1647915612","cluster_addr":"172.21.15.106:6803/1647915612","heartbeat_back_addr":"172.21.15.106:6807/1647915612","heartbeat_front_addr":"172.21.15.106:6805/1647915612","state":["exists","up"]},{"osd":2,"uuid":"fcfc4dd1-aaf8-4131-a023-5e832212158d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6801","nonce":2010770435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6803","nonce":2010770435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6807","nonce":2010770435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6805","nonce":2010770435}]},"public_addr":"172.21.15.107:6801/2010770435","cluster_addr":"172.21.15.107:6803/2010770435","heartbeat_back_addr":"172.21.15.107:6807/2010770435","heartbeat_front_addr":"172.21.15.107:6805/2010770435","state":["exists","up"]},{"osd":3,"uuid":"47c718ab-a0f4-4e52-83c5-b89386b262d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6811","nonce":1912251346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6813","nonce":1912251346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6817","nonce":1912251346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6815","nonce":1912251346}]},"public_addr":"172.21.15.78:6811/1912251346","cluster_addr":"172.21.15.78:6813/1912251346","heartbeat_back_addr":"172.21.15.78:6817/1912251346","heartbeat_front_addr":"172.21.15.78:6815/1912251346","state":["exists","up"]},{"osd":4,"uuid":"7dad64dd-677e-4c1c-9638-e4f4f2195983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6809","nonce":374641205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6811","nonce":374641205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6815","nonce":374641205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6813","nonce":374641205}]},"public_addr":"172.21.15.106:6809/374641205","cluster_addr":"172.21.15.106:6811/374641205","heartbeat_back_addr":"172.21.15.106:6815/374641205","heartbeat_front_addr":"172.21.15.106:6813/374641205","state":["exists","up"]},{"osd":5,"uuid":"bbd524e5-328d-4309-8877-1149cbe110dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6809","nonce":3272806073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6811","nonce":3272806073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6815","nonce":3272806073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6813","nonce":3272806073}]},"public_addr":"172.21.15.107:6809/3272806073","cluster_addr":"172.21.15.107:6811/3272806073","heartbeat_back_addr":"172.21.15.107:6815/3272806073","heartbeat_front_addr":"172.21.15.107:6813/3272806073","state":["exists","up"]},{"osd":6,"uuid":"5696389a-918f-4ea8-b20d-4e2a0d4cb3bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6819","nonce":3091148508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6821","nonce":3091148508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6825","nonce":3091148508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6823","nonce":3091148508}]},"public_addr":"172.21.15.78:6819/3091148508","cluster_addr":"172.21.15.78:6821/3091148508","heartbeat_back_addr":"172.21.15.78:6825/3091148508","heartbeat_front_addr":"172.21.15.78:6823/3091148508","state":["exists","up"]},{"osd":7,"uuid":"f6b764e8-ed4e-4c9a-a85b-94af6dc6b924","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6817","nonce":4262103080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6819","nonce":4262103080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6823","nonce":4262103080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6821","nonce":4262103080}]},"public_addr":"172.21.15.106:6817/4262103080","cluster_addr":"172.21.15.106:6819/4262103080","heartbeat_back_addr":"172.21.15.106:6823/4262103080","heartbeat_front_addr":"172.21.15.106:6821/4262103080","state":["exists","up"]},{"osd":8,"uuid":"ba16039f-8093-4fa0-87ad-e28b7d142b23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6817","nonce":4032799356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6819","nonce":4032799356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6823","nonce":4032799356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6821","nonce":4032799356}]},"public_addr":"172.21.15.107:6817/4032799356","cluster_addr":"172.21.15.107:6819/4032799356","heartbeat_back_addr":"172.21.15.107:6823/4032799356","heartbeat_front_addr":"172.21.15.107:6821/4032799356","state":["exists","up"]},{"osd":9,"uuid":"651357f2-0669-42e6-aa91-1d5c02d00c5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6827","nonce":214023431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6829","nonce":214023431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6833","nonce":214023431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6831","nonce":214023431}]},"public_addr":"172.21.15.78:6827/214023431","cluster_addr":"172.21.15.78:6829/214023431","heartbeat_back_addr":"172.21.15.78:6833/214023431","heartbeat_front_addr":"172.21.15.78:6831/214023431","state":["exists","up"]},{"osd":10,"uuid":"2375119d-346e-426c-8a65-b64af71ad63a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6825","nonce":400029804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6827","nonce":400029804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6831","nonce":400029804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6829","nonce":400029804}]},"public_addr":"172.21.15.106:6825/400029804","cluster_addr":"172.21.15.106:6827/400029804","heartbeat_back_addr":"172.21.15.106:6831/400029804","heartbeat_front_addr":"172.21.15.106:6829/400029804","state":["exists","up"]},{"osd":11,"uuid":"b6b609d9-dd78-4c01-ae66-ba2a662ab22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6825","nonce":4191916230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6827","nonce":4191916230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6831","nonce":4191916230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6829","nonce":4191916230}]},"public_addr":"172.21.15.107:6825/4191916230","cluster_addr":"172.21.15.107:6827/4191916230","heartbeat_back_addr":"172.21.15.107:6831/4191916230","heartbeat_front_addr":"172.21.15.107:6829/4191916230","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:13.728602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:37.320809+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:01.269452+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:25.758281+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:50.298863+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:14.345219+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:38.980489+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:04.749214+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:29.992673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:56.269202+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:22.270316+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:48.074907+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/1982202541":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/1933054339":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6801/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6800/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:0/465691596":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6801/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/3962523609":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6800/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/1502752625":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/2236879401":"2024-03-25T00:11:40.199569+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:20:56.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.147+0000 7f4cd2259640 1 -- 172.21.15.78:0/2887117193 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4cb80a2800 msgr2=0x7f4cb80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:56.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.147+0000 7f4cd2259640 1 --2- 172.21.15.78:0/2887117193 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4cb80a2800 0x7f4cb80a4cc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc101610 tx=0x7f4cbc009500 comp rx=0 tx=0).stop 2024-03-24T00:20:56.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.147+0000 7f4cd2259640 1 -- 172.21.15.78:0/2887117193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ccc1017b0 msgr2=0x7f4ccc197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:56.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.147+0000 7f4cd2259640 1 --2- 172.21.15.78:0/2887117193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ccc1017b0 0x7f4ccc197ee0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4cb400e9f0 tx=0x7f4cb400eeb0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.147+0000 7f4cd2259640 1 -- 172.21.15.78:0/2887117193 shutdown_connections 2024-03-24T00:20:56.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.147+0000 7f4cd2259640 1 --2- 172.21.15.78:0/2887117193 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f4cb80a2800 0x7f4cb80a4cc0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.147+0000 7f4cd2259640 1 --2- 172.21.15.78:0/2887117193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ccc1017b0 0x7f4ccc197ee0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.147+0000 7f4cd2259640 1 --2- 172.21.15.78:0/2887117193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ccc102170 0x7f4ccc19ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.147+0000 7f4cd2259640 1 --2- 172.21.15.78:0/2887117193 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ccc1005c0 0x7f4ccc1979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.147+0000 7f4cd2259640 1 -- 172.21.15.78:0/2887117193 >> 172.21.15.78:0/2887117193 conn(0x7f4ccc0fbb30 msgr2=0x7f4ccc0fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:56.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.147+0000 7f4cd2259640 1 -- 172.21.15.78:0/2887117193 shutdown_connections 2024-03-24T00:20:56.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.147+0000 7f4cd2259640 1 -- 172.21.15.78:0/2887117193 wait complete. 2024-03-24T00:20:56.160 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-24T00:18:43.592215+0000', 'modified': '2024-03-24T00:19:08.871384+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24500, 'mds_1': 24481, 'mds_2': 24464}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24437': {'gid': 24437, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.106:6835/2231611477', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6834', 'nonce': 2231611477}, {'type': 'v1', 'addr': '172.21.15.106:6835', 'nonce': 2231611477}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24442': {'gid': 24442, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.107:6835/108936439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6834', 'nonce': 108936439}, {'type': 'v1', 'addr': '172.21.15.107:6835', 'nonce': 108936439}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24454': {'gid': 24454, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.107:6837/2939571540', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6836', 'nonce': 2939571540}, {'type': 'v1', 'addr': '172.21.15.107:6837', 'nonce': 2939571540}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24464': {'gid': 24464, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.106:6837/2090968633', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6836', 'nonce': 2090968633}, {'type': 'v1', 'addr': '172.21.15.106:6837', 'nonce': 2090968633}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24481': {'gid': 24481, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.107:6839/117119462', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6838', 'nonce': 117119462}, {'type': 'v1', 'addr': '172.21.15.107:6839', 'nonce': 117119462}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24500': {'gid': 24500, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.106:6839/2482071304', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6838', 'nonce': 2482071304}, {'type': 'v1', 'addr': '172.21.15.106:6839', 'nonce': 2482071304}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24500, 'qdb_cluster': [24481, 24500, 24464]} 2024-03-24T00:20:56.160 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-24T00:20:56.160 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-24T00:20:56.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.262+0000 7faf5ec52640 1 -- 172.21.15.78:0/422225973 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf58103310 msgr2=0x7faf58109ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:56.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.262+0000 7faf5ec52640 1 --2- 172.21.15.78:0/422225973 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf58103310 0x7faf58109ba0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7faf50009cd0 tx=0x7faf5002c0f0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.262+0000 7faf5ec52640 1 -- 172.21.15.78:0/422225973 shutdown_connections 2024-03-24T00:20:56.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.262+0000 7faf5ec52640 1 --2- 172.21.15.78:0/422225973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf58102950 0x7faf58102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.262+0000 7faf5ec52640 1 --2- 172.21.15.78:0/422225973 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf58103310 0x7faf58109ba0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.262+0000 7faf5ec52640 1 --2- 172.21.15.78:0/422225973 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf580ff980 0x7faf580ffda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.262+0000 7faf5ec52640 1 -- 172.21.15.78:0/422225973 >> 172.21.15.78:0/422225973 conn(0x7faf580fb290 msgr2=0x7faf580fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:56.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.262+0000 7faf5ec52640 1 -- 172.21.15.78:0/422225973 shutdown_connections 2024-03-24T00:20:56.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.262+0000 7faf5ec52640 1 -- 172.21.15.78:0/422225973 wait complete. 2024-03-24T00:20:56.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.263+0000 7faf5ec52640 1 Processor -- start 2024-03-24T00:20:56.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.263+0000 7faf5ec52640 1 -- start start 2024-03-24T00:20:56.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.263+0000 7faf5ec52640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf580ff980 0x7faf58199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.263+0000 7faf5ec52640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf58102950 0x7faf5819a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.263+0000 7faf5ec52640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf58103310 0x7faf581a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.263+0000 7faf5ec52640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf5810d360 con 0x7faf58103310 2024-03-24T00:20:56.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.263+0000 7faf5ec52640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf5810d1e0 con 0x7faf58102950 2024-03-24T00:20:56.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.263+0000 7faf5ec52640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf5810d4e0 con 0x7faf580ff980 2024-03-24T00:20:56.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.263+0000 7faf5d1ef640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf58103310 0x7faf581a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.263+0000 7faf5d1ef640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf58103310 0x7faf581a1140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55398/0 (socket says 172.21.15.78:55398) 2024-03-24T00:20:56.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.263+0000 7faf5d1ef640 1 -- 172.21.15.78:0/3890645906 learned_addr learned my addr 172.21.15.78:0/3890645906 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:56.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.263+0000 7faf57fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf58102950 0x7faf5819a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.263+0000 7faf5c9ee640 1 --2- 172.21.15.78:0/3890645906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf580ff980 0x7faf58199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.263+0000 7faf5d1ef640 1 -- 172.21.15.78:0/3890645906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf580ff980 msgr2=0x7faf58199b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:56.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.263+0000 7faf5d1ef640 1 --2- 172.21.15.78:0/3890645906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf580ff980 0x7faf58199b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.263+0000 7faf5d1ef640 1 -- 172.21.15.78:0/3890645906 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf58102950 msgr2=0x7faf5819a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:56.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.263+0000 7faf5d1ef640 1 --2- 172.21.15.78:0/3890645906 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf58102950 0x7faf5819a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.263+0000 7faf5d1ef640 1 -- 172.21.15.78:0/3890645906 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf581a17b0 con 0x7faf58103310 2024-03-24T00:20:56.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.263+0000 7faf5c9ee640 1 --2- 172.21.15.78:0/3890645906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf580ff980 0x7faf58199b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:20:56.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.264+0000 7faf5d1ef640 1 --2- 172.21.15.78:0/3890645906 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf58103310 0x7faf581a1140 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7faf50009ca0 tx=0x7faf500346d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:56.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.264+0000 7faf55ffb640 1 -- 172.21.15.78:0/3890645906 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf50041020 con 0x7faf58103310 2024-03-24T00:20:56.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.264+0000 7faf55ffb640 1 -- 172.21.15.78:0/3890645906 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faf50032890 con 0x7faf58103310 2024-03-24T00:20:56.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.264+0000 7faf5ec52640 1 -- 172.21.15.78:0/3890645906 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf581a1a40 con 0x7faf58103310 2024-03-24T00:20:56.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.264+0000 7faf55ffb640 1 -- 172.21.15.78:0/3890645906 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf50039040 con 0x7faf58103310 2024-03-24T00:20:56.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.264+0000 7faf5ec52640 1 -- 172.21.15.78:0/3890645906 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf581a1f30 con 0x7faf58103310 2024-03-24T00:20:56.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.266+0000 7faf55ffb640 1 -- 172.21.15.78:0/3890645906 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7faf50044020 con 0x7faf58103310 2024-03-24T00:20:56.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.266+0000 7faf6183e740 1 Processor -- start 2024-03-24T00:20:56.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.267+0000 7faf6183e740 1 -- start start 2024-03-24T00:20:56.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.267+0000 7faf6183e740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b0202e740 0x557b0202eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.267+0000 7faf6183e740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b0202f130 0x557b02033c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.267+0000 7faf6183e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b020342a0 0x557b02036730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.267+0000 7faf6183e740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b01f93c20 con 0x557b0202f130 2024-03-24T00:20:56.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.267+0000 7faf6183e740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b01f3ca40 con 0x557b0202e740 2024-03-24T00:20:56.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.267+0000 7faf6183e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b01f8fcc0 con 0x557b020342a0 2024-03-24T00:20:56.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.267+0000 7faf55ffb640 1 --2- 172.21.15.78:0/3890645906 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7faf480a2800 0x7faf480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.267+0000 7faf57fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b0202e740 0x557b0202eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.267+0000 7faf5c9ee640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b0202f130 0x557b02033c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.267+0000 7faf57fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b0202e740 0x557b0202eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.78:36596/0 (socket says 172.21.15.78:36596) 2024-03-24T00:20:56.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.267+0000 7faf57fff640 1 -- 172.21.15.78:0/3751477899 learned_addr learned my addr 172.21.15.78:0/3751477899 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:56.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.267+0000 7faf57fff640 1 --2- 172.21.15.78:0/3751477899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b020342a0 0x557b02036730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.267+0000 7faf5d1ef640 1 --2- 172.21.15.78:0/3890645906 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7faf480a2800 0x7faf480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.267+0000 7faf55ffb640 1 -- 172.21.15.78:0/3890645906 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7faf500f0c40 con 0x7faf58103310 2024-03-24T00:20:56.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.267+0000 7faf57fff640 1 -- 172.21.15.78:0/3751477899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b020342a0 msgr2=0x557b02036730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:56.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.267+0000 7faf57fff640 1 --2- 172.21.15.78:0/3751477899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b020342a0 0x557b02036730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.267+0000 7faf57fff640 1 -- 172.21.15.78:0/3751477899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b0202f130 msgr2=0x557b02033c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:56.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.267+0000 7faf57fff640 1 --2- 172.21.15.78:0/3751477899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b0202f130 0x557b02033c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.267+0000 7faf57fff640 1 -- 172.21.15.78:0/3751477899 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557b01f5ef90 con 0x557b0202e740 2024-03-24T00:20:56.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.267+0000 7faf5c9ee640 1 --2- 172.21.15.78:0/3751477899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b0202f130 0x557b02033c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:56.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.267+0000 7faf57fff640 1 --2- 172.21.15.78:0/3751477899 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b0202e740 0x557b0202eb60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7faf44011810 tx=0x7faf440097d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc5c86976ba1d068 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:56.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.267+0000 7faf5d1ef640 1 --2- 172.21.15.78:0/3890645906 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7faf480a2800 0x7faf480a4cc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7faf500b96b0 tx=0x7faf50087f60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:56.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.267+0000 7faf42ffd640 1 -- 172.21.15.78:0/3751477899 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf44004030 con 0x557b0202e740 2024-03-24T00:20:56.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.268+0000 7faf42ffd640 1 -- 172.21.15.78:0/3751477899 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faf44011e50 con 0x557b0202e740 2024-03-24T00:20:56.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.268+0000 7faf42ffd640 1 -- 172.21.15.78:0/3751477899 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf44003850 con 0x557b0202e740 2024-03-24T00:20:56.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.268+0000 7faf6183e740 1 -- 172.21.15.78:0/3751477899 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b0202e740 msgr2=0x557b0202eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:56.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.268+0000 7faf6183e740 1 --2- 172.21.15.78:0/3751477899 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b0202e740 0x557b0202eb60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7faf44011810 tx=0x7faf440097d0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.268+0000 7faf6183e740 1 -- 172.21.15.78:0/3751477899 shutdown_connections 2024-03-24T00:20:56.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.268+0000 7faf6183e740 1 --2- 172.21.15.78:0/3751477899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b020342a0 0x557b02036730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.268+0000 7faf6183e740 1 --2- 172.21.15.78:0/3751477899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b0202f130 0x557b02033c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.268+0000 7faf6183e740 1 --2- 172.21.15.78:0/3751477899 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b0202e740 0x557b0202eb60 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.268+0000 7faf6183e740 1 -- 172.21.15.78:0/3751477899 >> 172.21.15.78:0/3751477899 conn(0x557b01ff6a60 msgr2=0x557b0202a0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:56.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.268+0000 7faf6183e740 1 -- 172.21.15.78:0/3751477899 shutdown_connections 2024-03-24T00:20:56.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.268+0000 7faf6183e740 1 -- 172.21.15.78:0/3751477899 wait complete. 2024-03-24T00:20:56.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.268+0000 7faf6183e740 1 -- start start 2024-03-24T00:20:56.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.269+0000 7faf6183e740 1 Processor -- start 2024-03-24T00:20:56.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.269+0000 7faf6183e740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b0202e740 0x557b02038c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.269+0000 7faf6183e740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b0202f130 0x557b020391c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.269+0000 7faf6183e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b020342a0 0x557b02040200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.269+0000 7faf6183e740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b02037250 con 0x557b0202f130 2024-03-24T00:20:56.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.269+0000 7faf6183e740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b01f4fe90 con 0x557b0202e740 2024-03-24T00:20:56.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.269+0000 7faf6183e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b020373d0 con 0x557b020342a0 2024-03-24T00:20:56.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.269+0000 7faf6183e740 10 client.0 _ll_register_callbacks cb 0x557b01ca7ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:20:56.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.269+0000 7faf5c9ee640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b0202f130 0x557b020391c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.269+0000 7faf57fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b0202e740 0x557b02038c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.269+0000 7faf5c9ee640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b0202f130 0x557b020391c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55424/0 (socket says 172.21.15.78:55424) 2024-03-24T00:20:56.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.269+0000 7faf5c9ee640 1 -- 172.21.15.78:0/3441555525 learned_addr learned my addr 172.21.15.78:0/3441555525 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:56.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.269+0000 7faf57fff640 1 --2- 172.21.15.78:0/3441555525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b020342a0 0x557b02040200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.269+0000 7faf5c9ee640 1 -- 172.21.15.78:0/3441555525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b020342a0 msgr2=0x557b02040200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:56.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.269+0000 7faf5c9ee640 1 --2- 172.21.15.78:0/3441555525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b020342a0 0x557b02040200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.269+0000 7faf5c9ee640 1 -- 172.21.15.78:0/3441555525 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b0202e740 msgr2=0x557b02038c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:56.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.269+0000 7faf5c9ee640 1 --2- 172.21.15.78:0/3441555525 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b0202e740 0x557b02038c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.269+0000 7faf5c9ee640 1 -- 172.21.15.78:0/3441555525 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557b01f5c9d0 con 0x557b0202f130 2024-03-24T00:20:56.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.269+0000 7faf57fff640 1 --2- 172.21.15.78:0/3441555525 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b0202e740 0x557b02038c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:56.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.270+0000 7faf5c9ee640 1 --2- 172.21.15.78:0/3441555525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b0202f130 0x557b020391c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7faf4c009d20 tx=0x7faf4c0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85a1f82d82d35a45 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:56.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.270+0000 7faf37fff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-24T00:20:56.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.270+0000 7faf37fff640 1 -- 172.21.15.78:0/3441555525 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4c012860 con 0x557b0202f130 2024-03-24T00:20:56.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.270+0000 7faf6183e740 1 -- 172.21.15.78:0/3441555525 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faf44011e50 con 0x557b0202f130 2024-03-24T00:20:56.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.270+0000 7faf37fff640 1 -- 172.21.15.78:0/3441555525 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7faf4c012a00 con 0x557b0202f130 2024-03-24T00:20:56.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.270+0000 7faf37fff640 1 -- 172.21.15.78:0/3441555525 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7faf4c012c00 con 0x557b0202f130 2024-03-24T00:20:56.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.270+0000 7faf6183e740 10 client.15411 fetch_fsmap learned FSMap version 23 2024-03-24T00:20:56.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.270+0000 7faf6183e740 1 -- 172.21.15.78:0/3441555525 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x557b01ca82d0 con 0x557b0202f130 2024-03-24T00:20:56.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.270+0000 7faf37fff640 1 -- 172.21.15.78:0/3441555525 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faf44011e50 con 0x557b0202f130 2024-03-24T00:20:56.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.270+0000 7faf37fff640 1 -- 172.21.15.78:0/3441555525 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 23) v1 ==== 7926+0+0 (secure 0 0 0) 0x7faf4c01a030 con 0x557b0202f130 2024-03-24T00:20:56.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.270+0000 7faf6183e740 10 client.15411 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T00:20:56.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.270+0000 7faf6183e740 10 client.15411 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-24T00:20:56.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.270+0000 7faf6183e740 20 client.15411 populate_metadata read hostname 'smithi078' 2024-03-24T00:20:56.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.270+0000 7faf6183e740 1 --2- 172.21.15.78:0/3441555525 >> [v2:172.21.15.106:6836/2090968633,v1:172.21.15.106:6837/2090968633] conn(0x557b02045580 0x557b020445d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.270+0000 7faf6183e740 4 client.15411 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:20:56.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.270+0000 7faf6183e740 1 -- 172.21.15.78:0/3441555525 --> [v2:172.21.15.106:6836/2090968633,v1:172.21.15.106:6837/2090968633] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557b01fa1160 con 0x557b02045580 2024-03-24T00:20:56.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.270+0000 7faf57fff640 1 --2- 172.21.15.78:0/3441555525 >> [v2:172.21.15.106:6836/2090968633,v1:172.21.15.106:6837/2090968633] conn(0x557b02045580 0x557b020445d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.271+0000 7faf57fff640 1 --2- 172.21.15.78:0/3441555525 >> [v2:172.21.15.106:6836/2090968633,v1:172.21.15.106:6837/2090968633] conn(0x557b02045580 0x557b020445d0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1d4de40354321339 server_cookie=9198942c29efb99b in_seq=0 out_seq=0 2024-03-24T00:20:56.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.271+0000 7faf37fff640 10 client.15411 ms_handle_connect on v2:172.21.15.106:6836/2090968633 2024-03-24T00:20:56.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.271+0000 7faf37fff640 1 -- 172.21.15.78:0/3441555525 <== mds.2 v2:172.21.15.106:6836/2090968633 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557b01fa1160 con 0x557b02045580 2024-03-24T00:20:56.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.271+0000 7faf37fff640 10 client.15411 handle_command_reply: tid=0 2024-03-24T00:20:56.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.271+0000 7faf6183e740 1 client.15411 shutdown 2024-03-24T00:20:56.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.272+0000 7faf6183e740 1 -- 172.21.15.78:0/3441555525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b0202f130 msgr2=0x557b020391c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:56.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.272+0000 7faf6183e740 1 --2- 172.21.15.78:0/3441555525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b0202f130 0x557b020391c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7faf4c009d20 tx=0x7faf4c0039c0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.272+0000 7faf6183e740 1 -- 172.21.15.78:0/3441555525 shutdown_connections 2024-03-24T00:20:56.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.272+0000 7faf6183e740 1 --2- 172.21.15.78:0/3441555525 >> [v2:172.21.15.106:6836/2090968633,v1:172.21.15.106:6837/2090968633] conn(0x557b02045580 0x557b020445d0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.272+0000 7faf6183e740 1 --2- 172.21.15.78:0/3441555525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b020342a0 0x557b02040200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.272+0000 7faf6183e740 1 --2- 172.21.15.78:0/3441555525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b0202f130 0x557b020391c0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.272+0000 7faf6183e740 1 --2- 172.21.15.78:0/3441555525 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b0202e740 0x557b02038c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.272+0000 7faf37fff640 0 client.15411 ms_handle_reset on v2:172.21.15.106:6836/2090968633 2024-03-24T00:20:56.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.272+0000 7faf6183e740 1 -- 172.21.15.78:0/3441555525 >> 172.21.15.78:0/3441555525 conn(0x557b01ff6a60 msgr2=0x557b020292f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:56.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.272+0000 7faf6183e740 1 -- 172.21.15.78:0/3441555525 shutdown_connections 2024-03-24T00:20:56.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.272+0000 7faf6183e740 1 -- 172.21.15.78:0/3441555525 wait complete. 2024-03-24T00:20:56.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.272+0000 7faf6183e740 20 client.15411 trim_cache size 0 max 16384 2024-03-24T00:20:56.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.288+0000 7faf6183e740 1 Processor -- start 2024-03-24T00:20:56.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.288+0000 7faf6183e740 1 -- start start 2024-03-24T00:20:56.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.288+0000 7faf6183e740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b0202e740 0x557b02030630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.288+0000 7faf6183e740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b0202f130 0x557b02030b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.288+0000 7faf6183e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b020342a0 0x557b020401c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.288+0000 7faf6183e740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b02044da0 con 0x557b0202f130 2024-03-24T00:20:56.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.288+0000 7faf6183e740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b02045400 con 0x557b0202e740 2024-03-24T00:20:56.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.288+0000 7faf6183e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b02045a60 con 0x557b020342a0 2024-03-24T00:20:56.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.288+0000 7faf5c9ee640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b0202f130 0x557b02030b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.288+0000 7faf5c9ee640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b0202f130 0x557b02030b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55430/0 (socket says 172.21.15.78:55430) 2024-03-24T00:20:56.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.288+0000 7faf5c9ee640 1 -- 172.21.15.78:0/413819648 learned_addr learned my addr 172.21.15.78:0/413819648 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:56.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.288+0000 7faf57fff640 1 --2- 172.21.15.78:0/413819648 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b0202e740 0x557b02030630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.288+0000 7faf57fff640 1 --2- 172.21.15.78:0/413819648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b020342a0 0x557b020401c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.289+0000 7faf5c9ee640 1 -- 172.21.15.78:0/413819648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b020342a0 msgr2=0x557b020401c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:56.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.289+0000 7faf5c9ee640 1 --2- 172.21.15.78:0/413819648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b020342a0 0x557b020401c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.289+0000 7faf5c9ee640 1 -- 172.21.15.78:0/413819648 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b0202e740 msgr2=0x557b02030630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:56.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.289+0000 7faf5c9ee640 1 --2- 172.21.15.78:0/413819648 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b0202e740 0x557b02030630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.289+0000 7faf5c9ee640 1 -- 172.21.15.78:0/413819648 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557b01d726b0 con 0x557b0202f130 2024-03-24T00:20:56.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.289+0000 7faf57fff640 1 --2- 172.21.15.78:0/413819648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b020342a0 0x557b020401c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:20:56.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.289+0000 7faf5c9ee640 1 --2- 172.21.15.78:0/413819648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b0202f130 0x557b02030b70 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7faf4c01b3f0 tx=0x7faf4c009910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de4c368c41f51ea0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:56.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.289+0000 7faf427fc640 1 -- 172.21.15.78:0/413819648 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4c02c020 con 0x557b0202f130 2024-03-24T00:20:56.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.289+0000 7faf427fc640 1 -- 172.21.15.78:0/413819648 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faf4c012660 con 0x557b0202f130 2024-03-24T00:20:56.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.289+0000 7faf427fc640 1 -- 172.21.15.78:0/413819648 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4c005540 con 0x557b0202f130 2024-03-24T00:20:56.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.289+0000 7faf6183e740 1 -- 172.21.15.78:0/413819648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b0202f130 msgr2=0x557b02030b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:56.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.289+0000 7faf6183e740 1 --2- 172.21.15.78:0/413819648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b0202f130 0x557b02030b70 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7faf4c01b3f0 tx=0x7faf4c009910 comp rx=0 tx=0).stop 2024-03-24T00:20:56.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.289+0000 7faf6183e740 1 -- 172.21.15.78:0/413819648 shutdown_connections 2024-03-24T00:20:56.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.289+0000 7faf6183e740 1 --2- 172.21.15.78:0/413819648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b020342a0 0x557b020401c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.289+0000 7faf6183e740 1 --2- 172.21.15.78:0/413819648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b0202f130 0x557b02030b70 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.289+0000 7faf6183e740 1 --2- 172.21.15.78:0/413819648 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b0202e740 0x557b02030630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.289+0000 7faf6183e740 1 -- 172.21.15.78:0/413819648 >> 172.21.15.78:0/413819648 conn(0x557b01ff6a60 msgr2=0x557b02029bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:56.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.290+0000 7faf6183e740 1 -- 172.21.15.78:0/413819648 shutdown_connections 2024-03-24T00:20:56.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.290+0000 7faf6183e740 1 -- 172.21.15.78:0/413819648 wait complete. 2024-03-24T00:20:56.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.290+0000 7faf6183e740 1 -- start start 2024-03-24T00:20:56.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.290+0000 7faf6183e740 1 Processor -- start 2024-03-24T00:20:56.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.290+0000 7faf6183e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b0202e740 0x557b0203c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.290+0000 7faf6183e740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b0202f130 0x557b0203c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.290+0000 7faf6183e740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b020342a0 0x557b0203f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.290+0000 7faf6183e740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b02046720 con 0x557b020342a0 2024-03-24T00:20:56.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.290+0000 7faf6183e740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b020460c0 con 0x557b0202f130 2024-03-24T00:20:56.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.290+0000 7faf6183e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b02046d80 con 0x557b0202e740 2024-03-24T00:20:56.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.290+0000 7faf6183e740 10 client.0 _ll_register_callbacks cb 0x7faf4c004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:20:56.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.290+0000 7faf57fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b020342a0 0x557b0203f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.290+0000 7faf5c9ee640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b0202f130 0x557b0203c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.290+0000 7faf5c9ee640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b0202f130 0x557b0203c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.78:36624/0 (socket says 172.21.15.78:36624) 2024-03-24T00:20:56.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.290+0000 7faf57fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b020342a0 0x557b0203f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55444/0 (socket says 172.21.15.78:55444) 2024-03-24T00:20:56.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.290+0000 7faf5c9ee640 1 -- 172.21.15.78:0/1363235611 learned_addr learned my addr 172.21.15.78:0/1363235611 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:56.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.290+0000 7faf57fff640 1 --2- 172.21.15.78:0/1363235611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b0202e740 0x557b0203c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.291+0000 7faf5c9ee640 1 -- 172.21.15.78:0/1363235611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b0202e740 msgr2=0x557b0203c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:56.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.291+0000 7faf5c9ee640 1 --2- 172.21.15.78:0/1363235611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b0202e740 0x557b0203c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.291+0000 7faf5c9ee640 1 -- 172.21.15.78:0/1363235611 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b020342a0 msgr2=0x557b0203f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:56.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.291+0000 7faf5c9ee640 1 --2- 172.21.15.78:0/1363235611 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b020342a0 0x557b0203f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.291+0000 7faf5c9ee640 1 -- 172.21.15.78:0/1363235611 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557b0203fe20 con 0x557b0202f130 2024-03-24T00:20:56.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.291+0000 7faf57fff640 1 --2- 172.21.15.78:0/1363235611 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b020342a0 0x557b0203f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:56.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.291+0000 7faf5c9ee640 1 --2- 172.21.15.78:0/1363235611 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b0202f130 0x557b0203c810 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7faf4c000c00 tx=0x7faf4c005570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4181c1c655e8241 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:56.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.291+0000 7faf37fff640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-24T00:20:56.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.291+0000 7faf37fff640 1 -- 172.21.15.78:0/1363235611 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4c02c030 con 0x557b0202f130 2024-03-24T00:20:56.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.291+0000 7faf6183e740 1 -- 172.21.15.78:0/1363235611 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faf4c012660 con 0x557b0202f130 2024-03-24T00:20:56.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.291+0000 7faf37fff640 1 -- 172.21.15.78:0/1363235611 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7faf4c01b920 con 0x557b0202f130 2024-03-24T00:20:56.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.291+0000 7faf37fff640 1 -- 172.21.15.78:0/1363235611 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7faf4c01bb20 con 0x557b0202f130 2024-03-24T00:20:56.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.291+0000 7faf6183e740 10 client.25235 fetch_fsmap learned FSMap version 23 2024-03-24T00:20:56.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.291+0000 7faf6183e740 1 -- 172.21.15.78:0/1363235611 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x557b020a0fd0 con 0x557b0202f130 2024-03-24T00:20:56.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.291+0000 7faf37fff640 1 -- 172.21.15.78:0/1363235611 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faf4c012660 con 0x557b0202f130 2024-03-24T00:20:56.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.291+0000 7faf37fff640 1 -- 172.21.15.78:0/1363235611 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 23) v1 ==== 7926+0+0 (secure 0 0 0) 0x7faf4c01a040 con 0x557b0202f130 2024-03-24T00:20:56.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.291+0000 7faf6183e740 10 client.25235 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T00:20:56.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.291+0000 7faf6183e740 10 client.25235 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-24T00:20:56.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.292+0000 7faf6183e740 20 client.25235 populate_metadata read hostname 'smithi078' 2024-03-24T00:20:56.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.292+0000 7faf6183e740 1 --2- 172.21.15.78:0/1363235611 >> [v2:172.21.15.106:6836/2090968633,v1:172.21.15.106:6837/2090968633] conn(0x557b020a54d0 0x557b020a1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.292+0000 7faf6183e740 4 client.25235 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:20:56.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.292+0000 7faf6183e740 1 -- 172.21.15.78:0/1363235611 --> [v2:172.21.15.106:6836/2090968633,v1:172.21.15.106:6837/2090968633] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557b01f9b4b0 con 0x557b020a54d0 2024-03-24T00:20:56.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.292+0000 7faf57fff640 1 --2- 172.21.15.78:0/1363235611 >> [v2:172.21.15.106:6836/2090968633,v1:172.21.15.106:6837/2090968633] conn(0x557b020a54d0 0x557b020a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.292+0000 7faf37fff640 10 client.25235 ms_handle_connect on v2:172.21.15.106:6836/2090968633 2024-03-24T00:20:56.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.292+0000 7faf57fff640 1 --2- 172.21.15.78:0/1363235611 >> [v2:172.21.15.106:6836/2090968633,v1:172.21.15.106:6837/2090968633] conn(0x557b020a54d0 0x557b020a1260 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4d36bff95611806c server_cookie=59dd6224c338b079 in_seq=0 out_seq=0 2024-03-24T00:20:56.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.292+0000 7faf37fff640 1 -- 172.21.15.78:0/1363235611 <== mds.2 v2:172.21.15.106:6836/2090968633 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+321 (crc 0 0 0) 0x557b01f9b4b0 con 0x557b020a54d0 2024-03-24T00:20:56.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.292+0000 7faf37fff640 10 client.25235 handle_command_reply: tid=0 2024-03-24T00:20:56.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.292+0000 7faf6183e740 1 client.25235 shutdown 2024-03-24T00:20:56.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.292+0000 7faf6183e740 1 -- 172.21.15.78:0/1363235611 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b0202f130 msgr2=0x557b0203c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:56.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.292+0000 7faf6183e740 1 --2- 172.21.15.78:0/1363235611 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b0202f130 0x557b0203c810 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7faf4c000c00 tx=0x7faf4c005570 comp rx=0 tx=0).stop 2024-03-24T00:20:56.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.292+0000 7faf6183e740 1 -- 172.21.15.78:0/1363235611 shutdown_connections 2024-03-24T00:20:56.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.292+0000 7faf6183e740 1 --2- 172.21.15.78:0/1363235611 >> [v2:172.21.15.106:6836/2090968633,v1:172.21.15.106:6837/2090968633] conn(0x557b020a54d0 0x557b020a1260 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.292+0000 7faf6183e740 1 --2- 172.21.15.78:0/1363235611 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557b020342a0 0x557b0203f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.292+0000 7faf6183e740 1 --2- 172.21.15.78:0/1363235611 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b0202f130 0x557b0203c810 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.292+0000 7faf6183e740 1 --2- 172.21.15.78:0/1363235611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b0202e740 0x557b0203c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.292+0000 7faf37fff640 0 client.25235 ms_handle_reset on v2:172.21.15.106:6836/2090968633 2024-03-24T00:20:56.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.292+0000 7faf6183e740 1 -- 172.21.15.78:0/1363235611 >> 172.21.15.78:0/1363235611 conn(0x557b01ff6a60 msgr2=0x557b02030630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:56.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.292+0000 7faf6183e740 1 -- 172.21.15.78:0/1363235611 shutdown_connections 2024-03-24T00:20:56.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.292+0000 7faf6183e740 1 -- 172.21.15.78:0/1363235611 wait complete. 2024-03-24T00:20:56.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.293+0000 7faf6183e740 20 client.25235 trim_cache size 0 max 16384 2024-03-24T00:20:56.301 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-24T00:20:56.301 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "c7e2806a-e972-11ee-95cd-87774f69a715", 2024-03-24T00:20:56.301 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 2, 2024-03-24T00:20:56.301 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24464, 2024-03-24T00:20:56.301 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-03-24T00:20:56.301 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-03-24T00:20:56.301 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-03-24T00:20:56.301 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 107.914200092, 2024-03-24T00:20:56.301 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 23, 2024-03-24T00:20:56.301 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 93, 2024-03-24T00:20:56.302 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 89, 2024-03-24T00:20:56.302 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 149.75865256 2024-03-24T00:20:56.302 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-24T00:20:56.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.293+0000 7faf5ec52640 1 -- 172.21.15.78:0/3890645906 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7faf480a2800 msgr2=0x7faf480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:56.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.293+0000 7faf5ec52640 1 --2- 172.21.15.78:0/3890645906 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7faf480a2800 0x7faf480a4cc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7faf500b96b0 tx=0x7faf50087f60 comp rx=0 tx=0).stop 2024-03-24T00:20:56.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.293+0000 7faf5ec52640 1 -- 172.21.15.78:0/3890645906 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf58103310 msgr2=0x7faf581a1140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:56.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.293+0000 7faf5ec52640 1 --2- 172.21.15.78:0/3890645906 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf58103310 0x7faf581a1140 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7faf50009ca0 tx=0x7faf500346d0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.293+0000 7faf5ec52640 1 -- 172.21.15.78:0/3890645906 shutdown_connections 2024-03-24T00:20:56.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.293+0000 7faf5ec52640 1 --2- 172.21.15.78:0/3890645906 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7faf480a2800 0x7faf480a4cc0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.293+0000 7faf5ec52640 1 --2- 172.21.15.78:0/3890645906 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf58103310 0x7faf581a1140 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.293+0000 7faf5ec52640 1 --2- 172.21.15.78:0/3890645906 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf58102950 0x7faf5819a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.293+0000 7faf5ec52640 1 --2- 172.21.15.78:0/3890645906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf580ff980 0x7faf58199b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.293+0000 7faf5ec52640 1 -- 172.21.15.78:0/3890645906 >> 172.21.15.78:0/3890645906 conn(0x7faf580fb290 msgr2=0x7faf58100740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:56.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.293+0000 7faf5ec52640 1 -- 172.21.15.78:0/3890645906 shutdown_connections 2024-03-24T00:20:56.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.293+0000 7faf5ec52640 1 -- 172.21.15.78:0/3890645906 wait complete. 2024-03-24T00:20:56.305 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-24T00:20:56.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.409+0000 7f91512dc640 1 -- 172.21.15.78:0/3136540240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f914c1016b0 msgr2=0x7f914c101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:56.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.409+0000 7f91512dc640 1 --2- 172.21.15.78:0/3136540240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f914c1016b0 0x7f914c101ad0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f913c009900 tx=0x7f913c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:20:56.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.409+0000 7f91512dc640 1 -- 172.21.15.78:0/3136540240 shutdown_connections 2024-03-24T00:20:56.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.409+0000 7f91512dc640 1 --2- 172.21.15.78:0/3136540240 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f914c1028a0 0x7f914c102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.409+0000 7f91512dc640 1 --2- 172.21.15.78:0/3136540240 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f914c103260 0x7f914c109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.409+0000 7f91512dc640 1 --2- 172.21.15.78:0/3136540240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f914c1016b0 0x7f914c101ad0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.409+0000 7f91512dc640 1 -- 172.21.15.78:0/3136540240 >> 172.21.15.78:0/3136540240 conn(0x7f914c0fcc40 msgr2=0x7f914c0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:56.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.409+0000 7f91512dc640 1 -- 172.21.15.78:0/3136540240 shutdown_connections 2024-03-24T00:20:56.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.409+0000 7f91512dc640 1 -- 172.21.15.78:0/3136540240 wait complete. 2024-03-24T00:20:56.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.410+0000 7f91512dc640 1 Processor -- start 2024-03-24T00:20:56.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.410+0000 7f91512dc640 1 -- start start 2024-03-24T00:20:56.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.410+0000 7f91512dc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f914c1016b0 0x7f914c197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.410+0000 7f91512dc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f914c1028a0 0x7f914c197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.410+0000 7f91512dc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f914c103260 0x7f914c19efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.410+0000 7f91512dc640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914c10d300 con 0x7f914c1016b0 2024-03-24T00:20:56.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.410+0000 7f91512dc640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914c10d180 con 0x7f914c103260 2024-03-24T00:20:56.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.410+0000 7f91512dc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914c10d480 con 0x7f914c1028a0 2024-03-24T00:20:56.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.410+0000 7f914affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f914c1016b0 0x7f914c197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.410+0000 7f914affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f914c1016b0 0x7f914c197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55462/0 (socket says 172.21.15.78:55462) 2024-03-24T00:20:56.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.410+0000 7f914affd640 1 -- 172.21.15.78:0/3388824270 learned_addr learned my addr 172.21.15.78:0/3388824270 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:56.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.410+0000 7f91427fc640 1 --2- 172.21.15.78:0/3388824270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f914c1028a0 0x7f914c197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.410+0000 7f914b7fe640 1 --2- 172.21.15.78:0/3388824270 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f914c103260 0x7f914c19efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.410+0000 7f914affd640 1 -- 172.21.15.78:0/3388824270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f914c1028a0 msgr2=0x7f914c197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:56.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.410+0000 7f914affd640 1 --2- 172.21.15.78:0/3388824270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f914c1028a0 0x7f914c197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.410+0000 7f914affd640 1 -- 172.21.15.78:0/3388824270 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f914c103260 msgr2=0x7f914c19efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:56.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.410+0000 7f914affd640 1 --2- 172.21.15.78:0/3388824270 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f914c103260 0x7f914c19efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.410+0000 7f914affd640 1 -- 172.21.15.78:0/3388824270 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f914c19f620 con 0x7f914c1016b0 2024-03-24T00:20:56.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.410+0000 7f914affd640 1 --2- 172.21.15.78:0/3388824270 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f914c1016b0 0x7f914c197a00 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f913c000c00 tx=0x7f913c02c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:56.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.411+0000 7f9148ff9640 1 -- 172.21.15.78:0/3388824270 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913c041070 con 0x7f914c1016b0 2024-03-24T00:20:56.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.411+0000 7f91512dc640 1 -- 172.21.15.78:0/3388824270 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f914c19f8b0 con 0x7f914c1016b0 2024-03-24T00:20:56.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.411+0000 7f9148ff9640 1 -- 172.21.15.78:0/3388824270 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f913c004450 con 0x7f914c1016b0 2024-03-24T00:20:56.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.411+0000 7f9148ff9640 1 -- 172.21.15.78:0/3388824270 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913c034440 con 0x7f914c1016b0 2024-03-24T00:20:56.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.411+0000 7f91512dc640 1 -- 172.21.15.78:0/3388824270 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f914c19fdf0 con 0x7f914c1016b0 2024-03-24T00:20:56.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.413+0000 7f9148ff9640 1 -- 172.21.15.78:0/3388824270 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f913c044070 con 0x7f914c1016b0 2024-03-24T00:20:56.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.413+0000 7f9153fa4740 1 Processor -- start 2024-03-24T00:20:56.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.414+0000 7f9153fa4740 1 -- start start 2024-03-24T00:20:56.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.414+0000 7f9153fa4740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560eb13b9b50 0x560eb13b9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.414+0000 7f9153fa4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eb13ba540 0x560eb13bf040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.414+0000 7f9153fa4740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eb13bf6b0 0x560eb13c1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.414+0000 7f9153fa4740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560eb131ac20 con 0x560eb13b9b50 2024-03-24T00:20:56.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.414+0000 7f9153fa4740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x560eb12c3a40 con 0x560eb13bf6b0 2024-03-24T00:20:56.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.414+0000 7f9153fa4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x560eb1316cc0 con 0x560eb13ba540 2024-03-24T00:20:56.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.414+0000 7f914b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560eb13b9b50 0x560eb13b9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.414+0000 7f914affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eb13ba540 0x560eb13bf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.414+0000 7f91427fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eb13bf6b0 0x560eb13c1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.414+0000 7f914affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eb13ba540 0x560eb13bf040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.78:44318/0 (socket says 172.21.15.78:44318) 2024-03-24T00:20:56.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.414+0000 7f91427fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eb13bf6b0 0x560eb13c1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.78:36646/0 (socket says 172.21.15.78:36646) 2024-03-24T00:20:56.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.414+0000 7f914affd640 1 -- 172.21.15.78:0/3976831707 learned_addr learned my addr 172.21.15.78:0/3976831707 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:56.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.414+0000 7f914b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560eb13b9b50 0x560eb13b9f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55478/0 (socket says 172.21.15.78:55478) 2024-03-24T00:20:56.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.414+0000 7f9148ff9640 1 --2- 172.21.15.78:0/3388824270 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f91380a2800 0x7f91380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.414+0000 7f914b7fe640 1 --2- 172.21.15.78:0/3388824270 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f91380a2800 0x7f91380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.414+0000 7f91427fc640 1 -- 172.21.15.78:0/3976831707 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eb13ba540 msgr2=0x560eb13bf040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:56.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.414+0000 7f91427fc640 1 --2- 172.21.15.78:0/3976831707 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eb13ba540 0x560eb13bf040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.414+0000 7f91427fc640 1 -- 172.21.15.78:0/3976831707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560eb13b9b50 msgr2=0x560eb13b9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:56.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.414+0000 7f91427fc640 1 --2- 172.21.15.78:0/3976831707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560eb13b9b50 0x560eb13b9f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.414+0000 7f91427fc640 1 -- 172.21.15.78:0/3976831707 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560eb12e5f90 con 0x560eb13bf6b0 2024-03-24T00:20:56.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.414+0000 7f9148ff9640 1 -- 172.21.15.78:0/3388824270 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f913c0f08b0 con 0x7f914c1016b0 2024-03-24T00:20:56.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.414+0000 7f914affd640 1 --2- 172.21.15.78:0/3976831707 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eb13ba540 0x560eb13bf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:56.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.415+0000 7f91427fc640 1 --2- 172.21.15.78:0/3976831707 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eb13bf6b0 0x560eb13c1b40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9134009bf0 tx=0x7f9134011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d896be3e3ef06839 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:56.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.415+0000 7f914b7fe640 1 --2- 172.21.15.78:0/3388824270 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f91380a2800 0x7f91380a4cc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x560eb13bada0 tx=0x7f9144004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:56.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.415+0000 7f91417fa640 1 -- 172.21.15.78:0/3976831707 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913401e070 con 0x560eb13bf6b0 2024-03-24T00:20:56.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.415+0000 7f91417fa640 1 -- 172.21.15.78:0/3976831707 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9134009da0 con 0x560eb13bf6b0 2024-03-24T00:20:56.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.415+0000 7f91417fa640 1 -- 172.21.15.78:0/3976831707 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91340187a0 con 0x560eb13bf6b0 2024-03-24T00:20:56.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.415+0000 7f9153fa4740 1 -- 172.21.15.78:0/3976831707 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eb13bf6b0 msgr2=0x560eb13c1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:56.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.415+0000 7f9153fa4740 1 --2- 172.21.15.78:0/3976831707 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eb13bf6b0 0x560eb13c1b40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9134009bf0 tx=0x7f9134011810 comp rx=0 tx=0).stop 2024-03-24T00:20:56.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.416+0000 7f9153fa4740 1 -- 172.21.15.78:0/3976831707 shutdown_connections 2024-03-24T00:20:56.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.416+0000 7f9153fa4740 1 --2- 172.21.15.78:0/3976831707 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eb13ba540 0x560eb13bf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.416+0000 7f9153fa4740 1 --2- 172.21.15.78:0/3976831707 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eb13bf6b0 0x560eb13c1b40 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.416+0000 7f9153fa4740 1 --2- 172.21.15.78:0/3976831707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560eb13b9b50 0x560eb13b9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.416+0000 7f9153fa4740 1 -- 172.21.15.78:0/3976831707 >> 172.21.15.78:0/3976831707 conn(0x560eb137da60 msgr2=0x560eb13b10b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:56.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.416+0000 7f9153fa4740 1 -- 172.21.15.78:0/3976831707 shutdown_connections 2024-03-24T00:20:56.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.416+0000 7f9153fa4740 1 -- 172.21.15.78:0/3976831707 wait complete. 2024-03-24T00:20:56.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.416+0000 7f9153fa4740 1 -- start start 2024-03-24T00:20:56.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.416+0000 7f9153fa4740 1 Processor -- start 2024-03-24T00:20:56.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.416+0000 7f9153fa4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eb13b9b50 0x560eb13c40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.416+0000 7f9153fa4740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eb13ba540 0x560eb13c4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.416+0000 7f9153fa4740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560eb13bf6b0 0x560eb13cb670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.416+0000 7f9153fa4740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560eb13c2660 con 0x560eb13bf6b0 2024-03-24T00:20:56.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.416+0000 7f9153fa4740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x560eb12d6e90 con 0x560eb13ba540 2024-03-24T00:20:56.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.416+0000 7f9153fa4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x560eb13c27e0 con 0x560eb13b9b50 2024-03-24T00:20:56.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.416+0000 7f9153fa4740 10 client.0 _ll_register_callbacks cb 0x560eb102eac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:20:56.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.416+0000 7f914b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560eb13bf6b0 0x560eb13cb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.416+0000 7f914b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560eb13bf6b0 0x560eb13cb670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55492/0 (socket says 172.21.15.78:55492) 2024-03-24T00:20:56.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.416+0000 7f914b7fe640 1 -- 172.21.15.78:0/559929933 learned_addr learned my addr 172.21.15.78:0/559929933 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:56.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.416+0000 7f91427fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eb13ba540 0x560eb13c4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.416+0000 7f914affd640 1 --2- 172.21.15.78:0/559929933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eb13b9b50 0x560eb13c40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.416+0000 7f914b7fe640 1 -- 172.21.15.78:0/559929933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eb13b9b50 msgr2=0x560eb13c40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:56.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.416+0000 7f914b7fe640 1 --2- 172.21.15.78:0/559929933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eb13b9b50 0x560eb13c40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.416+0000 7f914b7fe640 1 -- 172.21.15.78:0/559929933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eb13ba540 msgr2=0x560eb13c4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.416+0000 7f914b7fe640 1 --2- 172.21.15.78:0/559929933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eb13ba540 0x560eb13c4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.417+0000 7f914b7fe640 1 -- 172.21.15.78:0/559929933 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560eb12e39d0 con 0x560eb13bf6b0 2024-03-24T00:20:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.417+0000 7f914b7fe640 1 --2- 172.21.15.78:0/559929933 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560eb13bf6b0 0x560eb13cb670 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f9144003dc0 tx=0x7f9144004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5cc377fed714242 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.417+0000 7f91327fc640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-24T00:20:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.417+0000 7f91327fc640 1 -- 172.21.15.78:0/559929933 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9144025030 con 0x560eb13bf6b0 2024-03-24T00:20:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.417+0000 7f91327fc640 1 -- 172.21.15.78:0/559929933 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f9144004550 con 0x560eb13bf6b0 2024-03-24T00:20:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.417+0000 7f9153fa4740 1 -- 172.21.15.78:0/559929933 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9134009da0 con 0x560eb13bf6b0 2024-03-24T00:20:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.417+0000 7f91327fc640 1 -- 172.21.15.78:0/559929933 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9144031050 con 0x560eb13bf6b0 2024-03-24T00:20:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.417+0000 7f9153fa4740 10 client.15429 fetch_fsmap learned FSMap version 23 2024-03-24T00:20:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.417+0000 7f9153fa4740 1 -- 172.21.15.78:0/559929933 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x560eb102f2d0 con 0x560eb13bf6b0 2024-03-24T00:20:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.417+0000 7f91327fc640 1 -- 172.21.15.78:0/559929933 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9134009da0 con 0x560eb13bf6b0 2024-03-24T00:20:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.417+0000 7f91327fc640 1 -- 172.21.15.78:0/559929933 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 23) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f914401f040 con 0x560eb13bf6b0 2024-03-24T00:20:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.417+0000 7f9153fa4740 10 client.15429 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T00:20:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.417+0000 7f9153fa4740 10 client.15429 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-24T00:20:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.417+0000 7f9153fa4740 20 client.15429 populate_metadata read hostname 'smithi078' 2024-03-24T00:20:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.417+0000 7f9153fa4740 1 --2- 172.21.15.78:0/559929933 >> [v2:172.21.15.107:6838/117119462,v1:172.21.15.107:6839/117119462] conn(0x560eb13d0940 0x560eb13cf960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.417+0000 7f9153fa4740 4 client.15429 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:20:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.417+0000 7f9153fa4740 1 -- 172.21.15.78:0/559929933 --> [v2:172.21.15.107:6838/117119462,v1:172.21.15.107:6839/117119462] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560eb1328160 con 0x560eb13d0940 2024-03-24T00:20:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.418+0000 7f914affd640 1 --2- 172.21.15.78:0/559929933 >> [v2:172.21.15.107:6838/117119462,v1:172.21.15.107:6839/117119462] conn(0x560eb13d0940 0x560eb13cf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.418+0000 7f914affd640 1 --2- 172.21.15.78:0/559929933 >> [v2:172.21.15.107:6838/117119462,v1:172.21.15.107:6839/117119462] conn(0x560eb13d0940 0x560eb13cf960 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b43275e8e8ef1661 server_cookie=a8a2c264888858b8 in_seq=0 out_seq=0 2024-03-24T00:20:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.418+0000 7f91327fc640 10 client.15429 ms_handle_connect on v2:172.21.15.107:6838/117119462 2024-03-24T00:20:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.419+0000 7f91327fc640 1 -- 172.21.15.78:0/559929933 <== mds.1 v2:172.21.15.107:6838/117119462 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560eb1328160 con 0x560eb13d0940 2024-03-24T00:20:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.419+0000 7f91327fc640 10 client.15429 handle_command_reply: tid=0 2024-03-24T00:20:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.419+0000 7f9153fa4740 1 client.15429 shutdown 2024-03-24T00:20:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.419+0000 7f9153fa4740 1 -- 172.21.15.78:0/559929933 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560eb13bf6b0 msgr2=0x560eb13cb670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.419+0000 7f9153fa4740 1 --2- 172.21.15.78:0/559929933 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560eb13bf6b0 0x560eb13cb670 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f9144003dc0 tx=0x7f9144004a40 comp rx=0 tx=0).stop 2024-03-24T00:20:56.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.419+0000 7f9153fa4740 1 -- 172.21.15.78:0/559929933 shutdown_connections 2024-03-24T00:20:56.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.419+0000 7f9153fa4740 1 --2- 172.21.15.78:0/559929933 >> [v2:172.21.15.107:6838/117119462,v1:172.21.15.107:6839/117119462] conn(0x560eb13d0940 0x560eb13cf960 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.419+0000 7f9153fa4740 1 --2- 172.21.15.78:0/559929933 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560eb13bf6b0 0x560eb13cb670 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.419+0000 7f9153fa4740 1 --2- 172.21.15.78:0/559929933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eb13ba540 0x560eb13c4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.419+0000 7f91327fc640 0 client.15429 ms_handle_reset on v2:172.21.15.107:6838/117119462 2024-03-24T00:20:56.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.419+0000 7f9153fa4740 1 --2- 172.21.15.78:0/559929933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eb13b9b50 0x560eb13c40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.419+0000 7f9153fa4740 1 -- 172.21.15.78:0/559929933 >> 172.21.15.78:0/559929933 conn(0x560eb137da60 msgr2=0x560eb13b02f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:56.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.419+0000 7f9153fa4740 1 -- 172.21.15.78:0/559929933 shutdown_connections 2024-03-24T00:20:56.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.419+0000 7f9153fa4740 1 -- 172.21.15.78:0/559929933 wait complete. 2024-03-24T00:20:56.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.419+0000 7f9153fa4740 20 client.15429 trim_cache size 0 max 16384 2024-03-24T00:20:56.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.435+0000 7f9153fa4740 1 Processor -- start 2024-03-24T00:20:56.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.436+0000 7f9153fa4740 1 -- start start 2024-03-24T00:20:56.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.436+0000 7f9153fa4740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560eb13b9b50 0x560eb13bba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.436+0000 7f9153fa4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eb13ba540 0x560eb13bbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.436+0000 7f9153fa4740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eb13bf6b0 0x560eb13cb590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.436+0000 7f9153fa4740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560eb13d0160 con 0x560eb13b9b50 2024-03-24T00:20:56.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.436+0000 7f9153fa4740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x560eb13d07c0 con 0x560eb13bf6b0 2024-03-24T00:20:56.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.436+0000 7f9153fa4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x560eb13d0e20 con 0x560eb13ba540 2024-03-24T00:20:56.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.436+0000 7f914affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560eb13b9b50 0x560eb13bba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.436+0000 7f914affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560eb13b9b50 0x560eb13bba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55494/0 (socket says 172.21.15.78:55494) 2024-03-24T00:20:56.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.436+0000 7f914affd640 1 -- 172.21.15.78:0/3978500924 learned_addr learned my addr 172.21.15.78:0/3978500924 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:56.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.436+0000 7f91427fc640 1 --2- 172.21.15.78:0/3978500924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eb13ba540 0x560eb13bbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.436+0000 7f914b7fe640 1 --2- 172.21.15.78:0/3978500924 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eb13bf6b0 0x560eb13cb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.436+0000 7f914affd640 1 -- 172.21.15.78:0/3978500924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eb13ba540 msgr2=0x560eb13bbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:56.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.436+0000 7f914affd640 1 --2- 172.21.15.78:0/3978500924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eb13ba540 0x560eb13bbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.436+0000 7f914affd640 1 -- 172.21.15.78:0/3978500924 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eb13bf6b0 msgr2=0x560eb13cb590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:56.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.436+0000 7f914affd640 1 --2- 172.21.15.78:0/3978500924 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eb13bf6b0 0x560eb13cb590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.436+0000 7f914affd640 1 -- 172.21.15.78:0/3978500924 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560eb10f96b0 con 0x560eb13b9b50 2024-03-24T00:20:56.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.436+0000 7f914affd640 1 --2- 172.21.15.78:0/3978500924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560eb13b9b50 0x560eb13bba40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f913c0b99d0 tx=0x7f913c0b93d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e58badb354468c55 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:56.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.437+0000 7f9140ff9640 1 -- 172.21.15.78:0/3978500924 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913c04b040 con 0x560eb13b9b50 2024-03-24T00:20:56.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.437+0000 7f9140ff9640 1 -- 172.21.15.78:0/3978500924 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f913c004c80 con 0x560eb13b9b50 2024-03-24T00:20:56.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.437+0000 7f9140ff9640 1 -- 172.21.15.78:0/3978500924 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913c0c10a0 con 0x560eb13b9b50 2024-03-24T00:20:56.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.437+0000 7f9153fa4740 1 -- 172.21.15.78:0/3978500924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560eb13b9b50 msgr2=0x560eb13bba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:56.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.437+0000 7f9153fa4740 1 --2- 172.21.15.78:0/3978500924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560eb13b9b50 0x560eb13bba40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f913c0b99d0 tx=0x7f913c0b93d0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.437+0000 7f9153fa4740 1 -- 172.21.15.78:0/3978500924 shutdown_connections 2024-03-24T00:20:56.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.437+0000 7f9153fa4740 1 --2- 172.21.15.78:0/3978500924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eb13ba540 0x560eb13bbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.437+0000 7f9153fa4740 1 --2- 172.21.15.78:0/3978500924 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eb13bf6b0 0x560eb13cb590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.437+0000 7f9153fa4740 1 --2- 172.21.15.78:0/3978500924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560eb13b9b50 0x560eb13bba40 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.437+0000 7f9153fa4740 1 -- 172.21.15.78:0/3978500924 >> 172.21.15.78:0/3978500924 conn(0x560eb137da60 msgr2=0x560eb13c2be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:56.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.437+0000 7f9153fa4740 1 -- 172.21.15.78:0/3978500924 shutdown_connections 2024-03-24T00:20:56.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.437+0000 7f9153fa4740 1 -- 172.21.15.78:0/3978500924 wait complete. 2024-03-24T00:20:56.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.437+0000 7f9153fa4740 1 -- start start 2024-03-24T00:20:56.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.438+0000 7f9153fa4740 1 Processor -- start 2024-03-24T00:20:56.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.438+0000 7f9153fa4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eb13b9b50 0x560eb13cb330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.438+0000 7f9153fa4740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eb13ba540 0x560eb13c7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.438+0000 7f9153fa4740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560eb13bf6b0 0x560eb13c8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.438+0000 7f9153fa4740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560eb13d1ae0 con 0x560eb13bf6b0 2024-03-24T00:20:56.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.438+0000 7f9153fa4740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x560eb13d1480 con 0x560eb13ba540 2024-03-24T00:20:56.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.438+0000 7f9153fa4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x560eb13d2140 con 0x560eb13b9b50 2024-03-24T00:20:56.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.438+0000 7f9153fa4740 10 client.0 _ll_register_callbacks cb 0x7f9144003a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:20:56.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.438+0000 7f914b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560eb13bf6b0 0x560eb13c8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.438+0000 7f914b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560eb13bf6b0 0x560eb13c8310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55496/0 (socket says 172.21.15.78:55496) 2024-03-24T00:20:56.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.438+0000 7f914b7fe640 1 -- 172.21.15.78:0/3146049670 learned_addr learned my addr 172.21.15.78:0/3146049670 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:56.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.438+0000 7f91427fc640 1 --2- 172.21.15.78:0/3146049670 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eb13ba540 0x560eb13c7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.438+0000 7f914affd640 1 --2- 172.21.15.78:0/3146049670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eb13b9b50 0x560eb13cb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.438+0000 7f914b7fe640 1 -- 172.21.15.78:0/3146049670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eb13b9b50 msgr2=0x560eb13cb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:56.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.438+0000 7f914b7fe640 1 --2- 172.21.15.78:0/3146049670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eb13b9b50 0x560eb13cb330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.438+0000 7f914b7fe640 1 -- 172.21.15.78:0/3146049670 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eb13ba540 msgr2=0x560eb13c7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:56.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.438+0000 7f914b7fe640 1 --2- 172.21.15.78:0/3146049670 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eb13ba540 0x560eb13c7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.438+0000 7f914b7fe640 1 -- 172.21.15.78:0/3146049670 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560eb13c8850 con 0x560eb13bf6b0 2024-03-24T00:20:56.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.438+0000 7f914b7fe640 1 --2- 172.21.15.78:0/3146049670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560eb13bf6b0 0x560eb13c8310 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f9144003d90 tx=0x7f9144005950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b24c85e6b2c0d60 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:56.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.438+0000 7f91327fc640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-24T00:20:56.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.438+0000 7f91327fc640 1 -- 172.21.15.78:0/3146049670 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9144025040 con 0x560eb13bf6b0 2024-03-24T00:20:56.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.438+0000 7f91327fc640 1 -- 172.21.15.78:0/3146049670 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f9144004830 con 0x560eb13bf6b0 2024-03-24T00:20:56.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.438+0000 7f9153fa4740 1 -- 172.21.15.78:0/3146049670 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f913c004c80 con 0x560eb13bf6b0 2024-03-24T00:20:56.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.439+0000 7f91327fc640 1 -- 172.21.15.78:0/3146049670 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f914400fe90 con 0x560eb13bf6b0 2024-03-24T00:20:56.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.439+0000 7f9153fa4740 10 client.15435 fetch_fsmap learned FSMap version 23 2024-03-24T00:20:56.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.439+0000 7f9153fa4740 1 -- 172.21.15.78:0/3146049670 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x560eb142c9e0 con 0x560eb13bf6b0 2024-03-24T00:20:56.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.439+0000 7f91327fc640 1 -- 172.21.15.78:0/3146049670 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f913c004c80 con 0x560eb13bf6b0 2024-03-24T00:20:56.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.439+0000 7f91327fc640 1 -- 172.21.15.78:0/3146049670 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 23) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f91440217b0 con 0x560eb13bf6b0 2024-03-24T00:20:56.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.439+0000 7f9153fa4740 10 client.15435 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T00:20:56.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.439+0000 7f9153fa4740 10 client.15435 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-24T00:20:56.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.439+0000 7f9153fa4740 20 client.15435 populate_metadata read hostname 'smithi078' 2024-03-24T00:20:56.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.439+0000 7f9153fa4740 1 --2- 172.21.15.78:0/3146049670 >> [v2:172.21.15.107:6838/117119462,v1:172.21.15.107:6839/117119462] conn(0x560eb1431120 0x560eb142ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.439+0000 7f9153fa4740 4 client.15435 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:20:56.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.439+0000 7f9153fa4740 1 -- 172.21.15.78:0/3146049670 --> [v2:172.21.15.107:6838/117119462,v1:172.21.15.107:6839/117119462] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560eb13c3ee0 con 0x560eb1431120 2024-03-24T00:20:56.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.439+0000 7f914affd640 1 --2- 172.21.15.78:0/3146049670 >> [v2:172.21.15.107:6838/117119462,v1:172.21.15.107:6839/117119462] conn(0x560eb1431120 0x560eb142ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.440+0000 7f91327fc640 10 client.15435 ms_handle_connect on v2:172.21.15.107:6838/117119462 2024-03-24T00:20:56.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.440+0000 7f914affd640 1 --2- 172.21.15.78:0/3146049670 >> [v2:172.21.15.107:6838/117119462,v1:172.21.15.107:6839/117119462] conn(0x560eb1431120 0x560eb142ccb0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f9a4ae01eea924c9 server_cookie=34b48ecd2144e79e in_seq=0 out_seq=0 2024-03-24T00:20:56.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.440+0000 7f91327fc640 1 -- 172.21.15.78:0/3146049670 <== mds.1 v2:172.21.15.107:6838/117119462 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x560eb13c3ee0 con 0x560eb1431120 2024-03-24T00:20:56.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.440+0000 7f91327fc640 10 client.15435 handle_command_reply: tid=0 2024-03-24T00:20:56.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.440+0000 7f9153fa4740 1 client.15435 shutdown 2024-03-24T00:20:56.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.440+0000 7f9153fa4740 1 -- 172.21.15.78:0/3146049670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560eb13bf6b0 msgr2=0x560eb13c8310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:56.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.440+0000 7f9153fa4740 1 --2- 172.21.15.78:0/3146049670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560eb13bf6b0 0x560eb13c8310 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f9144003d90 tx=0x7f9144005950 comp rx=0 tx=0).stop 2024-03-24T00:20:56.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.440+0000 7f9153fa4740 1 -- 172.21.15.78:0/3146049670 shutdown_connections 2024-03-24T00:20:56.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.440+0000 7f9153fa4740 1 --2- 172.21.15.78:0/3146049670 >> [v2:172.21.15.107:6838/117119462,v1:172.21.15.107:6839/117119462] conn(0x560eb1431120 0x560eb142ccb0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.440+0000 7f9153fa4740 1 --2- 172.21.15.78:0/3146049670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560eb13bf6b0 0x560eb13c8310 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.440+0000 7f9153fa4740 1 --2- 172.21.15.78:0/3146049670 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eb13ba540 0x560eb13c7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.440+0000 7f9153fa4740 1 --2- 172.21.15.78:0/3146049670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eb13b9b50 0x560eb13cb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.440+0000 7f91327fc640 0 client.15435 ms_handle_reset on v2:172.21.15.107:6838/117119462 2024-03-24T00:20:56.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.440+0000 7f9153fa4740 1 -- 172.21.15.78:0/3146049670 >> 172.21.15.78:0/3146049670 conn(0x560eb137da60 msgr2=0x560eb13c2be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:56.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.440+0000 7f9153fa4740 1 -- 172.21.15.78:0/3146049670 shutdown_connections 2024-03-24T00:20:56.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.440+0000 7f9153fa4740 1 -- 172.21.15.78:0/3146049670 wait complete. 2024-03-24T00:20:56.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.440+0000 7f9153fa4740 20 client.15435 trim_cache size 0 max 16384 2024-03-24T00:20:56.451 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-24T00:20:56.451 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "c7e2806a-e972-11ee-95cd-87774f69a715", 2024-03-24T00:20:56.451 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 1, 2024-03-24T00:20:56.451 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24481, 2024-03-24T00:20:56.451 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-03-24T00:20:56.451 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-03-24T00:20:56.451 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-03-24T00:20:56.451 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 109.07280809, 2024-03-24T00:20:56.451 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 23, 2024-03-24T00:20:56.451 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 93, 2024-03-24T00:20:56.451 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 89, 2024-03-24T00:20:56.452 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 147.08324022299999 2024-03-24T00:20:56.452 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-24T00:20:56.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.440+0000 7f91512dc640 1 -- 172.21.15.78:0/3388824270 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f91380a2800 msgr2=0x7f91380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:56.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.440+0000 7f91512dc640 1 --2- 172.21.15.78:0/3388824270 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f91380a2800 0x7f91380a4cc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x560eb13bada0 tx=0x7f9144004ab0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.440+0000 7f91512dc640 1 -- 172.21.15.78:0/3388824270 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f914c1016b0 msgr2=0x7f914c197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:56.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.440+0000 7f91512dc640 1 --2- 172.21.15.78:0/3388824270 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f914c1016b0 0x7f914c197a00 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f913c000c00 tx=0x7f913c02c640 comp rx=0 tx=0).stop 2024-03-24T00:20:56.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.440+0000 7f91512dc640 1 -- 172.21.15.78:0/3388824270 shutdown_connections 2024-03-24T00:20:56.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.440+0000 7f91512dc640 1 --2- 172.21.15.78:0/3388824270 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f91380a2800 0x7f91380a4cc0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.440+0000 7f91512dc640 1 --2- 172.21.15.78:0/3388824270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f914c1028a0 0x7f914c197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.440+0000 7f91512dc640 1 --2- 172.21.15.78:0/3388824270 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f914c103260 0x7f914c19efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.440+0000 7f91512dc640 1 --2- 172.21.15.78:0/3388824270 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f914c1016b0 0x7f914c197a00 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.440+0000 7f91512dc640 1 -- 172.21.15.78:0/3388824270 >> 172.21.15.78:0/3388824270 conn(0x7f914c0fcc40 msgr2=0x7f914c105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:56.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.441+0000 7f91512dc640 1 -- 172.21.15.78:0/3388824270 shutdown_connections 2024-03-24T00:20:56.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.441+0000 7f91512dc640 1 -- 172.21.15.78:0/3388824270 wait complete. 2024-03-24T00:20:56.454 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:56 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/104404465' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T00:20:56.454 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:56 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/541947473' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:20:56.454 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:56 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2887117193' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:20:56.455 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-24T00:20:56.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.556+0000 7f6b82790640 1 -- 172.21.15.78:0/2330237738 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b7c102150 msgr2=0x7f6b7c1089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:56.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.556+0000 7f6b82790640 1 --2- 172.21.15.78:0/2330237738 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b7c102150 0x7f6b7c1089e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6b74009ca0 tx=0x7f6b7402c0f0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.556+0000 7f6b82790640 1 -- 172.21.15.78:0/2330237738 shutdown_connections 2024-03-24T00:20:56.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.556+0000 7f6b82790640 1 --2- 172.21.15.78:0/2330237738 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b7c101790 0x7f6b7c101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.556+0000 7f6b82790640 1 --2- 172.21.15.78:0/2330237738 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b7c102150 0x7f6b7c1089e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.556+0000 7f6b82790640 1 --2- 172.21.15.78:0/2330237738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b7c1005a0 0x7f6b7c1009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.556+0000 7f6b82790640 1 -- 172.21.15.78:0/2330237738 >> 172.21.15.78:0/2330237738 conn(0x7f6b7c0fbb30 msgr2=0x7f6b7c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:56.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.556+0000 7f6b82790640 1 -- 172.21.15.78:0/2330237738 shutdown_connections 2024-03-24T00:20:56.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.556+0000 7f6b82790640 1 -- 172.21.15.78:0/2330237738 wait complete. 2024-03-24T00:20:56.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.557+0000 7f6b82790640 1 Processor -- start 2024-03-24T00:20:56.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.557+0000 7f6b82790640 1 -- start start 2024-03-24T00:20:56.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.557+0000 7f6b82790640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b7c1005a0 0x7f6b7c069390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.557+0000 7f6b82790640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b7c101790 0x7f6b7c19c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.557+0000 7f6b82790640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b7c102150 0x7f6b7c19eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.557+0000 7f6b82790640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7c10c230 con 0x7f6b7c101790 2024-03-24T00:20:56.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.557+0000 7f6b82790640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7c10c0b0 con 0x7f6b7c1005a0 2024-03-24T00:20:56.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.557+0000 7f6b82790640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7c10c3b0 con 0x7f6b7c102150 2024-03-24T00:20:56.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.557+0000 7f6b7b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b7c101790 0x7f6b7c19c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.557+0000 7f6b7b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b7c101790 0x7f6b7c19c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55510/0 (socket says 172.21.15.78:55510) 2024-03-24T00:20:56.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.557+0000 7f6b7b7fe640 1 -- 172.21.15.78:0/1075581965 learned_addr learned my addr 172.21.15.78:0/1075581965 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:56.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.557+0000 7f6b7bfff640 1 --2- 172.21.15.78:0/1075581965 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b7c1005a0 0x7f6b7c069390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.558+0000 7f6b80d2d640 1 --2- 172.21.15.78:0/1075581965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b7c102150 0x7f6b7c19eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.558+0000 7f6b7b7fe640 1 -- 172.21.15.78:0/1075581965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b7c102150 msgr2=0x7f6b7c19eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:56.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.558+0000 7f6b7b7fe640 1 --2- 172.21.15.78:0/1075581965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b7c102150 0x7f6b7c19eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.558+0000 7f6b7b7fe640 1 -- 172.21.15.78:0/1075581965 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b7c1005a0 msgr2=0x7f6b7c069390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:56.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.558+0000 7f6b7b7fe640 1 --2- 172.21.15.78:0/1075581965 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b7c1005a0 0x7f6b7c069390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.558+0000 7f6b7b7fe640 1 -- 172.21.15.78:0/1075581965 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b7c19f560 con 0x7f6b7c101790 2024-03-24T00:20:56.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.558+0000 7f6b7b7fe640 1 --2- 172.21.15.78:0/1075581965 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b7c101790 0x7f6b7c19c660 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f6b7000ea60 tx=0x7f6b7000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:56.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.558+0000 7f6b797fa640 1 -- 172.21.15.78:0/1075581965 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7000cde0 con 0x7f6b7c101790 2024-03-24T00:20:56.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.558+0000 7f6b82790640 1 -- 172.21.15.78:0/1075581965 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b7c19f850 con 0x7f6b7c101790 2024-03-24T00:20:56.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.558+0000 7f6b797fa640 1 -- 172.21.15.78:0/1075581965 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b700035f0 con 0x7f6b7c101790 2024-03-24T00:20:56.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.558+0000 7f6b797fa640 1 -- 172.21.15.78:0/1075581965 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b70010670 con 0x7f6b7c101790 2024-03-24T00:20:56.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.558+0000 7f6b82790640 1 -- 172.21.15.78:0/1075581965 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b7c19fda0 con 0x7f6b7c101790 2024-03-24T00:20:56.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.561+0000 7f6b8537d740 1 Processor -- start 2024-03-24T00:20:56.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.561+0000 7f6b797fa640 1 -- 172.21.15.78:0/1075581965 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6b700037a0 con 0x7f6b7c101790 2024-03-24T00:20:56.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.561+0000 7f6b8537d740 1 -- start start 2024-03-24T00:20:56.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.561+0000 7f6b8537d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557dc5774740 0x557dc5774b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.561+0000 7f6b8537d740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557dc5775130 0x557dc5779c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.561+0000 7f6b8537d740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557dc577a2a0 0x557dc577c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.561+0000 7f6b8537d740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557dc56d9c20 con 0x557dc577a2a0 2024-03-24T00:20:56.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.561+0000 7f6b8537d740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x557dc5682a40 con 0x557dc5775130 2024-03-24T00:20:56.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.561+0000 7f6b8537d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x557dc56d5cc0 con 0x557dc5774740 2024-03-24T00:20:56.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.561+0000 7f6b7b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557dc577a2a0 0x557dc577c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.561+0000 7f6b7bfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557dc5775130 0x557dc5779c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.561+0000 7f6b80d2d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557dc5774740 0x557dc5774b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.561+0000 7f6b7b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557dc577a2a0 0x557dc577c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55512/0 (socket says 172.21.15.78:55512) 2024-03-24T00:20:56.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.561+0000 7f6b7b7fe640 1 -- 172.21.15.78:0/4057303449 learned_addr learned my addr 172.21.15.78:0/4057303449 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:56.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.561+0000 7f6b7bfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557dc5775130 0x557dc5779c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.78:36732/0 (socket says 172.21.15.78:36732) 2024-03-24T00:20:56.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.562+0000 7f6b797fa640 1 --2- 172.21.15.78:0/1075581965 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f6b640a2800 0x7f6b640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.562+0000 7f6b7b7fe640 1 -- 172.21.15.78:0/4057303449 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557dc5774740 msgr2=0x557dc5774b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:56.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.562+0000 7f6b7b7fe640 1 --2- 172.21.15.78:0/4057303449 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557dc5774740 0x557dc5774b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.562+0000 7f6b7b7fe640 1 -- 172.21.15.78:0/4057303449 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557dc5775130 msgr2=0x557dc5779c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:56.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.562+0000 7f6b7b7fe640 1 --2- 172.21.15.78:0/4057303449 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557dc5775130 0x557dc5779c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.562+0000 7f6b7b7fe640 1 -- 172.21.15.78:0/4057303449 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557dc56a4f90 con 0x557dc577a2a0 2024-03-24T00:20:56.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.562+0000 7f6b80d2d640 1 --2- 172.21.15.78:0/1075581965 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f6b640a2800 0x7f6b640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.562+0000 7f6b797fa640 1 -- 172.21.15.78:0/1075581965 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f6b700cf140 con 0x7f6b7c101790 2024-03-24T00:20:56.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.562+0000 7f6b7b7fe640 1 --2- 172.21.15.78:0/4057303449 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557dc577a2a0 0x557dc577c730 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6b700acda0 tx=0x7f6b700af710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51261f3776ffc661 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:56.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.562+0000 7f6b80d2d640 1 --2- 172.21.15.78:0/1075581965 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f6b640a2800 0x7f6b640a4cc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6b74005e30 tx=0x7f6b7402e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:56.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.562+0000 7f6b627fc640 1 -- 172.21.15.78:0/4057303449 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7009b230 con 0x557dc577a2a0 2024-03-24T00:20:56.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.562+0000 7f6b627fc640 1 -- 172.21.15.78:0/4057303449 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b70086110 con 0x557dc577a2a0 2024-03-24T00:20:56.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.562+0000 7f6b627fc640 1 -- 172.21.15.78:0/4057303449 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b70099cb0 con 0x557dc577a2a0 2024-03-24T00:20:56.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.563+0000 7f6b8537d740 1 -- 172.21.15.78:0/4057303449 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557dc577a2a0 msgr2=0x557dc577c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:56.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.563+0000 7f6b8537d740 1 --2- 172.21.15.78:0/4057303449 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557dc577a2a0 0x557dc577c730 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6b700acda0 tx=0x7f6b700af710 comp rx=0 tx=0).stop 2024-03-24T00:20:56.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.563+0000 7f6b8537d740 1 -- 172.21.15.78:0/4057303449 shutdown_connections 2024-03-24T00:20:56.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.563+0000 7f6b8537d740 1 --2- 172.21.15.78:0/4057303449 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557dc577a2a0 0x557dc577c730 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.563+0000 7f6b8537d740 1 --2- 172.21.15.78:0/4057303449 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557dc5775130 0x557dc5779c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.563+0000 7f6b8537d740 1 --2- 172.21.15.78:0/4057303449 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557dc5774740 0x557dc5774b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.563+0000 7f6b8537d740 1 -- 172.21.15.78:0/4057303449 >> 172.21.15.78:0/4057303449 conn(0x557dc573ca60 msgr2=0x557dc57700b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:56.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.563+0000 7f6b8537d740 1 -- 172.21.15.78:0/4057303449 shutdown_connections 2024-03-24T00:20:56.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.563+0000 7f6b8537d740 1 -- 172.21.15.78:0/4057303449 wait complete. 2024-03-24T00:20:56.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.563+0000 7f6b8537d740 1 -- start start 2024-03-24T00:20:56.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.563+0000 7f6b8537d740 1 Processor -- start 2024-03-24T00:20:56.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.564+0000 7f6b8537d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557dc5774740 0x557dc577ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.564+0000 7f6b8537d740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557dc5775130 0x557dc577f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.564+0000 7f6b8537d740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557dc577a2a0 0x557dc57861a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.564+0000 7f6b8537d740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557dc577d250 con 0x557dc5775130 2024-03-24T00:20:56.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.564+0000 7f6b8537d740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x557dc5695e90 con 0x557dc577a2a0 2024-03-24T00:20:56.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.564+0000 7f6b8537d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x557dc577d3d0 con 0x557dc5774740 2024-03-24T00:20:56.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.564+0000 7f6b8537d740 10 client.0 _ll_register_callbacks cb 0x557dc53edac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:20:56.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.564+0000 7f6b7b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557dc5775130 0x557dc577f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.564+0000 7f6b7b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557dc5775130 0x557dc577f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55518/0 (socket says 172.21.15.78:55518) 2024-03-24T00:20:56.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.564+0000 7f6b7bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557dc5774740 0x557dc577ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.564+0000 7f6b7b7fe640 1 -- 172.21.15.78:0/2820620827 learned_addr learned my addr 172.21.15.78:0/2820620827 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:56.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.564+0000 7f6b80d2d640 1 --2- 172.21.15.78:0/2820620827 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557dc577a2a0 0x557dc57861a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.564+0000 7f6b7bfff640 1 -- 172.21.15.78:0/2820620827 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557dc577a2a0 msgr2=0x557dc57861a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:56.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.564+0000 7f6b7bfff640 1 --2- 172.21.15.78:0/2820620827 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557dc577a2a0 0x557dc57861a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.564+0000 7f6b7bfff640 1 -- 172.21.15.78:0/2820620827 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557dc5775130 msgr2=0x557dc577f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:56.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.564+0000 7f6b7bfff640 1 --2- 172.21.15.78:0/2820620827 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557dc5775130 0x557dc577f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.564+0000 7f6b7bfff640 1 -- 172.21.15.78:0/2820620827 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557dc56a29d0 con 0x557dc5774740 2024-03-24T00:20:56.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.564+0000 7f6b7bfff640 1 --2- 172.21.15.78:0/2820620827 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557dc5774740 0x557dc577ec20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x557dc5775530 tx=0x7f6b6c012b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9987c8515cc40516 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:56.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.564+0000 7f6b5b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-24T00:20:56.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.564+0000 7f6b5b7fe640 1 -- 172.21.15.78:0/2820620827 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c00f890 con 0x557dc5774740 2024-03-24T00:20:56.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.564+0000 7f6b5b7fe640 1 -- 172.21.15.78:0/2820620827 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f6b6c00fa30 con 0x557dc5774740 2024-03-24T00:20:56.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.564+0000 7f6b8537d740 1 -- 172.21.15.78:0/2820620827 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6b70086110 con 0x557dc5774740 2024-03-24T00:20:56.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.565+0000 7f6b5b7fe640 1 -- 172.21.15.78:0/2820620827 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b6c00fc30 con 0x557dc5774740 2024-03-24T00:20:56.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.565+0000 7f6b8537d740 10 client.25204 fetch_fsmap learned FSMap version 23 2024-03-24T00:20:56.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.565+0000 7f6b8537d740 1 -- 172.21.15.78:0/2820620827 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x557dc53ee2d0 con 0x557dc5774740 2024-03-24T00:20:56.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.565+0000 7f6b5b7fe640 1 -- 172.21.15.78:0/2820620827 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b70086110 con 0x557dc5774740 2024-03-24T00:20:56.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.565+0000 7f6b5b7fe640 1 -- 172.21.15.78:0/2820620827 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 23) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f6b6c01c420 con 0x557dc5774740 2024-03-24T00:20:56.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.565+0000 7f6b8537d740 10 client.25204 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T00:20:56.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.565+0000 7f6b8537d740 10 client.25204 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:20:56.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.565+0000 7f6b8537d740 20 client.25204 populate_metadata read hostname 'smithi078' 2024-03-24T00:20:56.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.565+0000 7f6b8537d740 1 --2- 172.21.15.78:0/2820620827 >> [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] conn(0x557dc578b4c0 0x557dc578a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.565+0000 7f6b8537d740 4 client.25204 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:20:56.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.565+0000 7f6b8537d740 1 -- 172.21.15.78:0/2820620827 --> [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557dc56e7160 con 0x557dc578b4c0 2024-03-24T00:20:56.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.565+0000 7f6b7b7fe640 1 --2- 172.21.15.78:0/2820620827 >> [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] conn(0x557dc578b4c0 0x557dc578a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.565+0000 7f6b5b7fe640 10 client.25204 ms_handle_connect on v2:172.21.15.106:6838/2482071304 2024-03-24T00:20:56.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.565+0000 7f6b7b7fe640 1 --2- 172.21.15.78:0/2820620827 >> [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] conn(0x557dc578b4c0 0x557dc578a540 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b37d0b385ee5ed22 server_cookie=5c1e171b0686e3e4 in_seq=0 out_seq=0 2024-03-24T00:20:56.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.566+0000 7f6b5b7fe640 1 -- 172.21.15.78:0/2820620827 <== mds.0 v2:172.21.15.106:6838/2482071304 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557dc56e7160 con 0x557dc578b4c0 2024-03-24T00:20:56.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.566+0000 7f6b5b7fe640 10 client.25204 handle_command_reply: tid=0 2024-03-24T00:20:56.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.566+0000 7f6b8537d740 1 client.25204 shutdown 2024-03-24T00:20:56.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.566+0000 7f6b8537d740 1 -- 172.21.15.78:0/2820620827 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557dc5774740 msgr2=0x557dc577ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:56.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.566+0000 7f6b8537d740 1 --2- 172.21.15.78:0/2820620827 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557dc5774740 0x557dc577ec20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x557dc5775530 tx=0x7f6b6c012b60 comp rx=0 tx=0).stop 2024-03-24T00:20:56.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.566+0000 7f6b8537d740 1 -- 172.21.15.78:0/2820620827 shutdown_connections 2024-03-24T00:20:56.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.566+0000 7f6b8537d740 1 --2- 172.21.15.78:0/2820620827 >> [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] conn(0x557dc578b4c0 0x557dc578a540 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.566+0000 7f6b8537d740 1 --2- 172.21.15.78:0/2820620827 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557dc577a2a0 0x557dc57861a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.566+0000 7f6b8537d740 1 --2- 172.21.15.78:0/2820620827 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557dc5775130 0x557dc577f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.566+0000 7f6b8537d740 1 --2- 172.21.15.78:0/2820620827 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557dc5774740 0x557dc577ec20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.566+0000 7f6b5b7fe640 0 client.25204 ms_handle_reset on v2:172.21.15.106:6838/2482071304 2024-03-24T00:20:56.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.566+0000 7f6b8537d740 1 -- 172.21.15.78:0/2820620827 >> 172.21.15.78:0/2820620827 conn(0x557dc573ca60 msgr2=0x557dc576f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:56.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.566+0000 7f6b8537d740 1 -- 172.21.15.78:0/2820620827 shutdown_connections 2024-03-24T00:20:56.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.566+0000 7f6b8537d740 1 -- 172.21.15.78:0/2820620827 wait complete. 2024-03-24T00:20:56.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.566+0000 7f6b8537d740 20 client.25204 trim_cache size 0 max 16384 2024-03-24T00:20:56.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.583+0000 7f6b8537d740 1 Processor -- start 2024-03-24T00:20:56.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.583+0000 7f6b8537d740 1 -- start start 2024-03-24T00:20:56.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.583+0000 7f6b8537d740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557dc5774740 0x557dc576f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.583+0000 7f6b8537d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557dc5775130 0x557dc57860c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.583+0000 7f6b8537d740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557dc577a2a0 0x557dc5793960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.583+0000 7f6b8537d740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557dc578ace0 con 0x557dc5774740 2024-03-24T00:20:56.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.583+0000 7f6b8537d740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x557dc578b340 con 0x557dc577a2a0 2024-03-24T00:20:56.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.583+0000 7f6b8537d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x557dc578b9a0 con 0x557dc5775130 2024-03-24T00:20:56.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.583+0000 7f6b7bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557dc5774740 0x557dc576f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.583+0000 7f6b7bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557dc5774740 0x557dc576f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55522/0 (socket says 172.21.15.78:55522) 2024-03-24T00:20:56.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.583+0000 7f6b7bfff640 1 -- 172.21.15.78:0/947633518 learned_addr learned my addr 172.21.15.78:0/947633518 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:56.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.583+0000 7f6b7b7fe640 1 --2- 172.21.15.78:0/947633518 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557dc5775130 0x557dc57860c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.583+0000 7f6b80d2d640 1 --2- 172.21.15.78:0/947633518 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557dc577a2a0 0x557dc5793960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.583+0000 7f6b7bfff640 1 -- 172.21.15.78:0/947633518 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557dc5775130 msgr2=0x557dc57860c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:56.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.583+0000 7f6b7bfff640 1 --2- 172.21.15.78:0/947633518 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557dc5775130 0x557dc57860c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.583+0000 7f6b7bfff640 1 -- 172.21.15.78:0/947633518 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557dc577a2a0 msgr2=0x557dc5793960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:56.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.583+0000 7f6b7bfff640 1 --2- 172.21.15.78:0/947633518 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557dc577a2a0 0x557dc5793960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.583+0000 7f6b7bfff640 1 -- 172.21.15.78:0/947633518 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557dc54b86b0 con 0x557dc5774740 2024-03-24T00:20:56.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.583+0000 7f6b7bfff640 1 --2- 172.21.15.78:0/947633518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557dc5774740 0x557dc576f170 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c01c930 tx=0x7f6b6c009ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3248d101cc012775 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:56.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.584+0000 7f6b61ffb640 1 -- 172.21.15.78:0/947633518 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c00fdd0 con 0x557dc5774740 2024-03-24T00:20:56.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.584+0000 7f6b61ffb640 1 -- 172.21.15.78:0/947633518 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b6c01c420 con 0x557dc5774740 2024-03-24T00:20:56.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.584+0000 7f6b61ffb640 1 -- 172.21.15.78:0/947633518 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c005420 con 0x557dc5774740 2024-03-24T00:20:56.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.584+0000 7f6b8537d740 1 -- 172.21.15.78:0/947633518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557dc5774740 msgr2=0x557dc576f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:56.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.584+0000 7f6b8537d740 1 --2- 172.21.15.78:0/947633518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557dc5774740 0x557dc576f170 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c01c930 tx=0x7f6b6c009ea0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.584+0000 7f6b8537d740 1 -- 172.21.15.78:0/947633518 shutdown_connections 2024-03-24T00:20:56.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.584+0000 7f6b8537d740 1 --2- 172.21.15.78:0/947633518 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557dc5775130 0x557dc57860c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.584+0000 7f6b8537d740 1 --2- 172.21.15.78:0/947633518 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557dc577a2a0 0x557dc5793960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.584+0000 7f6b8537d740 1 --2- 172.21.15.78:0/947633518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557dc5774740 0x557dc576f170 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.584+0000 7f6b8537d740 1 -- 172.21.15.78:0/947633518 >> 172.21.15.78:0/947633518 conn(0x557dc573ca60 msgr2=0x557dc576ea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:56.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.584+0000 7f6b8537d740 1 -- 172.21.15.78:0/947633518 shutdown_connections 2024-03-24T00:20:56.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.584+0000 7f6b8537d740 1 -- 172.21.15.78:0/947633518 wait complete. 2024-03-24T00:20:56.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.584+0000 7f6b8537d740 1 -- start start 2024-03-24T00:20:56.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.585+0000 7f6b8537d740 1 Processor -- start 2024-03-24T00:20:56.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.585+0000 7f6b8537d740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557dc5774740 0x557dc5785e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.585+0000 7f6b8537d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557dc5775130 0x557dc57828e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.585+0000 7f6b8537d740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557dc577a2a0 0x557dc5782e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.585+0000 7f6b8537d740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557dc578c660 con 0x557dc577a2a0 2024-03-24T00:20:56.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.585+0000 7f6b8537d740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x557dc578c000 con 0x557dc5774740 2024-03-24T00:20:56.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.585+0000 7f6b8537d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x557dc578ccc0 con 0x557dc5775130 2024-03-24T00:20:56.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.585+0000 7f6b8537d740 10 client.0 _ll_register_callbacks cb 0x7f6b6c009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:20:56.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.585+0000 7f6b7b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557dc5775130 0x557dc57828e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.585+0000 7f6b7bfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557dc5774740 0x557dc5785e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.585+0000 7f6b80d2d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557dc577a2a0 0x557dc5782e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.585+0000 7f6b7b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557dc5775130 0x557dc57828e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.78:44414/0 (socket says 172.21.15.78:44414) 2024-03-24T00:20:56.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.585+0000 7f6b7bfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557dc5774740 0x557dc5785e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.78:36750/0 (socket says 172.21.15.78:36750) 2024-03-24T00:20:56.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.585+0000 7f6b7b7fe640 1 -- 172.21.15.78:0/256084669 learned_addr learned my addr 172.21.15.78:0/256084669 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:56.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.585+0000 7f6b80d2d640 1 -- 172.21.15.78:0/256084669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557dc5775130 msgr2=0x557dc57828e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:56.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.585+0000 7f6b80d2d640 1 --2- 172.21.15.78:0/256084669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557dc5775130 0x557dc57828e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.585+0000 7f6b80d2d640 1 -- 172.21.15.78:0/256084669 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557dc5774740 msgr2=0x557dc5785e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:56.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.585+0000 7f6b80d2d640 1 --2- 172.21.15.78:0/256084669 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557dc5774740 0x557dc5785e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.585+0000 7f6b80d2d640 1 -- 172.21.15.78:0/256084669 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557dc57834c0 con 0x557dc577a2a0 2024-03-24T00:20:56.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.585+0000 7f6b7b7fe640 1 --2- 172.21.15.78:0/256084669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557dc5775130 0x557dc57828e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:56.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.586+0000 7f6b80d2d640 1 --2- 172.21.15.78:0/256084669 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557dc577a2a0 0x557dc5782e20 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f6b7402c630 tx=0x7f6b74032590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb76b634cf914c56 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:56.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.586+0000 7f6b5b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-24T00:20:56.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.586+0000 7f6b5b7fe640 1 -- 172.21.15.78:0/256084669 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b74039040 con 0x557dc577a2a0 2024-03-24T00:20:56.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.586+0000 7f6b5b7fe640 1 -- 172.21.15.78:0/256084669 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f6b74041b00 con 0x557dc577a2a0 2024-03-24T00:20:56.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.586+0000 7f6b8537d740 1 -- 172.21.15.78:0/256084669 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6b6c01c420 con 0x557dc577a2a0 2024-03-24T00:20:56.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.586+0000 7f6b5b7fe640 1 -- 172.21.15.78:0/256084669 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b74034440 con 0x557dc577a2a0 2024-03-24T00:20:56.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.586+0000 7f6b8537d740 10 client.15453 fetch_fsmap learned FSMap version 23 2024-03-24T00:20:56.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.586+0000 7f6b8537d740 1 -- 172.21.15.78:0/256084669 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x557dc57e76c0 con 0x557dc577a2a0 2024-03-24T00:20:56.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.586+0000 7f6b5b7fe640 1 -- 172.21.15.78:0/256084669 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b6c01c420 con 0x557dc577a2a0 2024-03-24T00:20:56.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.586+0000 7f6b5b7fe640 1 -- 172.21.15.78:0/256084669 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 23) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f6b7403f040 con 0x557dc577a2a0 2024-03-24T00:20:56.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.586+0000 7f6b8537d740 10 client.15453 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T00:20:56.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.586+0000 7f6b8537d740 10 client.15453 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:20:56.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.586+0000 7f6b8537d740 20 client.15453 populate_metadata read hostname 'smithi078' 2024-03-24T00:20:56.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.586+0000 7f6b8537d740 1 --2- 172.21.15.78:0/256084669 >> [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] conn(0x557dc57ebda0 0x557dc57e7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:56.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.586+0000 7f6b8537d740 4 client.15453 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:20:56.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.586+0000 7f6b8537d740 1 -- 172.21.15.78:0/256084669 --> [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557dc56e14b0 con 0x557dc57ebda0 2024-03-24T00:20:56.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.586+0000 7f6b7bfff640 1 --2- 172.21.15.78:0/256084669 >> [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] conn(0x557dc57ebda0 0x557dc57e7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:56.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.587+0000 7f6b5b7fe640 10 client.15453 ms_handle_connect on v2:172.21.15.106:6838/2482071304 2024-03-24T00:20:56.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.587+0000 7f6b7bfff640 1 --2- 172.21.15.78:0/256084669 >> [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] conn(0x557dc57ebda0 0x557dc57e7a00 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d170b30f36517b8 server_cookie=9b6e3b92565b9ad3 in_seq=0 out_seq=0 2024-03-24T00:20:56.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.587+0000 7f6b5b7fe640 1 -- 172.21.15.78:0/256084669 <== mds.0 v2:172.21.15.106:6838/2482071304 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x557dc56e14b0 con 0x557dc57ebda0 2024-03-24T00:20:56.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.587+0000 7f6b5b7fe640 10 client.15453 handle_command_reply: tid=0 2024-03-24T00:20:56.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.587+0000 7f6b8537d740 1 client.15453 shutdown 2024-03-24T00:20:56.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.587+0000 7f6b8537d740 1 -- 172.21.15.78:0/256084669 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557dc577a2a0 msgr2=0x557dc5782e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:56.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.587+0000 7f6b8537d740 1 --2- 172.21.15.78:0/256084669 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557dc577a2a0 0x557dc5782e20 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f6b7402c630 tx=0x7f6b74032590 comp rx=0 tx=0).stop 2024-03-24T00:20:56.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.587+0000 7f6b8537d740 1 -- 172.21.15.78:0/256084669 shutdown_connections 2024-03-24T00:20:56.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.587+0000 7f6b8537d740 1 --2- 172.21.15.78:0/256084669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557dc5775130 0x557dc57828e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.587+0000 7f6b8537d740 1 --2- 172.21.15.78:0/256084669 >> [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] conn(0x557dc57ebda0 0x557dc57e7a00 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.587+0000 7f6b8537d740 1 --2- 172.21.15.78:0/256084669 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557dc577a2a0 0x557dc5782e20 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.587+0000 7f6b5b7fe640 0 client.15453 ms_handle_reset on v2:172.21.15.106:6838/2482071304 2024-03-24T00:20:56.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.587+0000 7f6b8537d740 1 --2- 172.21.15.78:0/256084669 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557dc5774740 0x557dc5785e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.587+0000 7f6b8537d740 1 -- 172.21.15.78:0/256084669 >> 172.21.15.78:0/256084669 conn(0x557dc573ca60 msgr2=0x557dc576f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:56.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.587+0000 7f6b8537d740 1 -- 172.21.15.78:0/256084669 shutdown_connections 2024-03-24T00:20:56.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.587+0000 7f6b8537d740 1 -- 172.21.15.78:0/256084669 wait complete. 2024-03-24T00:20:56.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.587+0000 7f6b8537d740 20 client.15453 trim_cache size 0 max 16384 2024-03-24T00:20:56.596 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-24T00:20:56.596 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "c7e2806a-e972-11ee-95cd-87774f69a715", 2024-03-24T00:20:56.597 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-03-24T00:20:56.597 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24500, 2024-03-24T00:20:56.597 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-03-24T00:20:56.597 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-03-24T00:20:56.597 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-03-24T00:20:56.597 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 132.346542841, 2024-03-24T00:20:56.597 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 23, 2024-03-24T00:20:56.597 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 93, 2024-03-24T00:20:56.597 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 89, 2024-03-24T00:20:56.597 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 138.721950765 2024-03-24T00:20:56.597 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-24T00:20:56.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.588+0000 7f6b82790640 1 -- 172.21.15.78:0/1075581965 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f6b640a2800 msgr2=0x7f6b640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:56.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.588+0000 7f6b82790640 1 --2- 172.21.15.78:0/1075581965 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f6b640a2800 0x7f6b640a4cc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6b74005e30 tx=0x7f6b7402e040 comp rx=0 tx=0).stop 2024-03-24T00:20:56.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.588+0000 7f6b82790640 1 -- 172.21.15.78:0/1075581965 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b7c101790 msgr2=0x7f6b7c19c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:56.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.588+0000 7f6b82790640 1 --2- 172.21.15.78:0/1075581965 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b7c101790 0x7f6b7c19c660 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f6b7000ea60 tx=0x7f6b7000ef20 comp rx=0 tx=0).stop 2024-03-24T00:20:56.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.588+0000 7f6b82790640 1 -- 172.21.15.78:0/1075581965 shutdown_connections 2024-03-24T00:20:56.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.588+0000 7f6b82790640 1 --2- 172.21.15.78:0/1075581965 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f6b640a2800 0x7f6b640a4cc0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.588+0000 7f6b82790640 1 --2- 172.21.15.78:0/1075581965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b7c102150 0x7f6b7c19eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.588+0000 7f6b82790640 1 --2- 172.21.15.78:0/1075581965 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b7c101790 0x7f6b7c19c660 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.588+0000 7f6b82790640 1 --2- 172.21.15.78:0/1075581965 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b7c1005a0 0x7f6b7c069390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:56.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.588+0000 7f6b82790640 1 -- 172.21.15.78:0/1075581965 >> 172.21.15.78:0/1075581965 conn(0x7f6b7c0fbb30 msgr2=0x7f6b7c104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:56.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.588+0000 7f6b82790640 1 -- 172.21.15.78:0/1075581965 shutdown_connections 2024-03-24T00:20:56.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:56.588+0000 7f6b82790640 1 -- 172.21.15.78:0/1075581965 wait complete. 2024-03-24T00:20:56.600 INFO:teuthology.run_tasks:Running task kclient... 2024-03-24T00:20:56.618 INFO:tasks.kclient:Mounting kernel clients... 2024-03-24T00:20:56.618 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-24T00:20:56.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:20:56.619 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2024-03-24T00:20:56.634 INFO:teuthology.orchestra.run.smithi078.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-24T00:20:56.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:20:56.635 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-24T00:20:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:56 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/104404465' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T00:20:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:56 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/541947473' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:20:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:56 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2887117193' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:20:56.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:20:56.703 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link delete ceph-brx 2024-03-24T00:20:56.797 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'wsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-03-24T00:20:56.797 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-24T00:20:56.797 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T00:20:56.798 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph fs ls 2024-03-24T00:20:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:56 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/104404465' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T00:20:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:56 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/541947473' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:20:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:56 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2887117193' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:20:57.037 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:20:57.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:57 smithi078 ceph-mon[32429]: pgmap v361: 129 pgs: 129 active+clean; 641 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 9 B/s, 0 objects/s recovering 2024-03-24T00:20:57.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:57 smithi106 ceph-mon[36505]: pgmap v361: 129 pgs: 129 active+clean; 641 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 9 B/s, 0 objects/s recovering 2024-03-24T00:20:57.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:57 smithi107 ceph-mon[36485]: pgmap v361: 129 pgs: 129 active+clean; 641 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 9 B/s, 0 objects/s recovering 2024-03-24T00:20:58.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.591+0000 7fd9b4a56700 1 -- 172.21.15.78:0/2153995234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9b0101a30 msgr2=0x7fd9b010ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:58.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.591+0000 7fd9b4a56700 1 --2- 172.21.15.78:0/2153995234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9b0101a30 0x7fd9b010ded0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd9a800b0a0 tx=0x7fd9a800b3b0 comp rx=0 tx=0).stop 2024-03-24T00:20:58.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.591+0000 7fd9b4a56700 1 -- 172.21.15.78:0/2153995234 shutdown_connections 2024-03-24T00:20:58.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.591+0000 7fd9b4a56700 1 --2- 172.21.15.78:0/2153995234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9b0101a30 0x7fd9b010ded0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:58.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.591+0000 7fd9b4a56700 1 --2- 172.21.15.78:0/2153995234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9b0107120 0x7fd9b0107500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:58.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.591+0000 7fd9b4a56700 1 --2- 172.21.15.78:0/2153995234 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9b0101070 0x7fd9b01014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:58.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.591+0000 7fd9b4a56700 1 -- 172.21.15.78:0/2153995234 >> 172.21.15.78:0/2153995234 conn(0x7fd9b00780a0 msgr2=0x7fd9b00784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:58.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.591+0000 7fd9b4a56700 1 -- 172.21.15.78:0/2153995234 shutdown_connections 2024-03-24T00:20:58.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.591+0000 7fd9b4a56700 1 -- 172.21.15.78:0/2153995234 wait complete. 2024-03-24T00:20:58.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.592+0000 7fd9b4a56700 1 Processor -- start 2024-03-24T00:20:58.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.592+0000 7fd9b4a56700 1 -- start start 2024-03-24T00:20:58.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.592+0000 7fd9b4a56700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9b0101070 0x7fd9b019d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:58.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.592+0000 7fd9b4a56700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9b0101a30 0x7fd9b019d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:58.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.592+0000 7fd9b4a56700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9b0107120 0x7fd9b0197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:58.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.592+0000 7fd9b4a56700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b0111350 con 0x7fd9b0101070 2024-03-24T00:20:58.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.592+0000 7fd9b4a56700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b01111d0 con 0x7fd9b0101a30 2024-03-24T00:20:58.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.592+0000 7fd9b4a56700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b01114d0 con 0x7fd9b0107120 2024-03-24T00:20:58.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.592+0000 7fd9ae59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9b0101070 0x7fd9b019d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:58.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.592+0000 7fd9ae59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9b0101070 0x7fd9b019d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55538/0 (socket says 172.21.15.78:55538) 2024-03-24T00:20:58.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.592+0000 7fd9ae59c700 1 -- 172.21.15.78:0/766049049 learned_addr learned my addr 172.21.15.78:0/766049049 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:20:58.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.592+0000 7fd9aed9d700 1 --2- 172.21.15.78:0/766049049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9b0107120 0x7fd9b0197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:58.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.592+0000 7fd9add9b700 1 --2- 172.21.15.78:0/766049049 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9b0101a30 0x7fd9b019d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:58.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.593+0000 7fd9aed9d700 1 -- 172.21.15.78:0/766049049 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9b0101a30 msgr2=0x7fd9b019d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:58.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.593+0000 7fd9aed9d700 1 --2- 172.21.15.78:0/766049049 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9b0101a30 0x7fd9b019d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:58.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.593+0000 7fd9aed9d700 1 -- 172.21.15.78:0/766049049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9b0101070 msgr2=0x7fd9b019d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:58.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.593+0000 7fd9aed9d700 1 --2- 172.21.15.78:0/766049049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9b0101070 0x7fd9b019d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:58.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.593+0000 7fd9aed9d700 1 -- 172.21.15.78:0/766049049 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b0197d70 con 0x7fd9b0107120 2024-03-24T00:20:58.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.593+0000 7fd9ae59c700 1 --2- 172.21.15.78:0/766049049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9b0101070 0x7fd9b019d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:58.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.593+0000 7fd9aed9d700 1 --2- 172.21.15.78:0/766049049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9b0107120 0x7fd9b0197610 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd9a800b0a0 tx=0x7fd9a8004d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:58.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.593+0000 7fd99f7fe700 1 -- 172.21.15.78:0/766049049 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a8021070 con 0x7fd9b0107120 2024-03-24T00:20:58.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.593+0000 7fd99f7fe700 1 -- 172.21.15.78:0/766049049 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9a8007b80 con 0x7fd9b0107120 2024-03-24T00:20:58.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.593+0000 7fd9b4a56700 1 -- 172.21.15.78:0/766049049 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9b0198000 con 0x7fd9b0107120 2024-03-24T00:20:58.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.593+0000 7fd99f7fe700 1 -- 172.21.15.78:0/766049049 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a8005040 con 0x7fd9b0107120 2024-03-24T00:20:58.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.593+0000 7fd9b4a56700 1 -- 172.21.15.78:0/766049049 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9b01a4320 con 0x7fd9b0107120 2024-03-24T00:20:58.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.594+0000 7fd9b4a56700 1 -- 172.21.15.78:0/766049049 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9b0109470 con 0x7fd9b0107120 2024-03-24T00:20:58.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.595+0000 7fd99f7fe700 1 -- 172.21.15.78:0/766049049 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd9a8024070 con 0x7fd9b0107120 2024-03-24T00:20:58.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.595+0000 7fd99f7fe700 1 --2- 172.21.15.78:0/766049049 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd9980a2890 0x7fd9980a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:58.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.596+0000 7fd9ae59c700 1 --2- 172.21.15.78:0/766049049 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd9980a2890 0x7fd9980a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:58.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.596+0000 7fd99f7fe700 1 -- 172.21.15.78:0/766049049 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7fd9a800f020 con 0x7fd9b0107120 2024-03-24T00:20:58.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.596+0000 7fd9ae59c700 1 --2- 172.21.15.78:0/766049049 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd9980a2890 0x7fd9980a4d50 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd9a0011550 tx=0x7fd9a0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:58.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.600+0000 7fd99f7fe700 1 -- 172.21.15.78:0/766049049 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd9a802b020 con 0x7fd9b0107120 2024-03-24T00:20:58.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.847+0000 7fd9b4a56700 1 -- 172.21.15.78:0/766049049 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd9b0198b40 con 0x7fd9b0107120 2024-03-24T00:20:58.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.847+0000 7fd99f7fe700 1 -- 172.21.15.78:0/766049049 <== mon.1 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v23) v1 ==== 53+0+88 (secure 0 0 0) 0x7fd9a8007d30 con 0x7fd9b0107120 2024-03-24T00:20:58.849 INFO:teuthology.orchestra.run.smithi078.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-24T00:20:58.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.850+0000 7fd9b4a56700 1 -- 172.21.15.78:0/766049049 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd9980a2890 msgr2=0x7fd9980a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:58.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.850+0000 7fd9b4a56700 1 --2- 172.21.15.78:0/766049049 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd9980a2890 0x7fd9980a4d50 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd9a0011550 tx=0x7fd9a0009260 comp rx=0 tx=0).stop 2024-03-24T00:20:58.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.850+0000 7fd9b4a56700 1 -- 172.21.15.78:0/766049049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9b0107120 msgr2=0x7fd9b0197610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:58.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.850+0000 7fd9b4a56700 1 --2- 172.21.15.78:0/766049049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9b0107120 0x7fd9b0197610 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd9a800b0a0 tx=0x7fd9a8004d00 comp rx=0 tx=0).stop 2024-03-24T00:20:58.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.850+0000 7fd9b4a56700 1 -- 172.21.15.78:0/766049049 shutdown_connections 2024-03-24T00:20:58.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.850+0000 7fd9b4a56700 1 --2- 172.21.15.78:0/766049049 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fd9980a2890 0x7fd9980a4d50 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:58.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.850+0000 7fd9b4a56700 1 --2- 172.21.15.78:0/766049049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9b0107120 0x7fd9b0197610 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:58.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.850+0000 7fd9b4a56700 1 --2- 172.21.15.78:0/766049049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9b0101070 0x7fd9b019d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:58.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.850+0000 7fd9b4a56700 1 --2- 172.21.15.78:0/766049049 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9b0101a30 0x7fd9b019d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:58.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.850+0000 7fd9b4a56700 1 -- 172.21.15.78:0/766049049 >> 172.21.15.78:0/766049049 conn(0x7fd9b00780a0 msgr2=0x7fd9b00ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:58.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.850+0000 7fd9b4a56700 1 -- 172.21.15.78:0/766049049 shutdown_connections 2024-03-24T00:20:58.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:20:58.850+0000 7fd9b4a56700 1 -- 172.21.15.78:0/766049049 wait complete. 2024-03-24T00:20:59.539 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T00:20:59.539 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:20:59.539 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi078.front.sepia.ceph.com 2024-03-24T00:20:59.539 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T00:20:59.539 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T00:20:59.540 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T00:20:59.540 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T00:20:59.540 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:20:59.540 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:20:59.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:20:59.540 DEBUG:teuthology.orchestra.run.smithi078:> ip addr 2024-03-24T00:20:59.556 INFO:teuthology.orchestra.run.smithi078.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:20:59.556 INFO:teuthology.orchestra.run.smithi078.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:20:59.556 INFO:teuthology.orchestra.run.smithi078.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:20:59.556 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:20:59.556 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 ::1/128 scope host 2024-03-24T00:20:59.557 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:20:59.557 INFO:teuthology.orchestra.run.smithi078.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:20:59.557 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:22 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:20:59.557 INFO:teuthology.orchestra.run.smithi078.stdout: altname enp7s0f0 2024-03-24T00:20:59.557 INFO:teuthology.orchestra.run.smithi078.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:20:59.557 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:23 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:20:59.557 INFO:teuthology.orchestra.run.smithi078.stdout: altname enp7s0f1 2024-03-24T00:20:59.557 INFO:teuthology.orchestra.run.smithi078.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:20:59.557 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dc brd ff:ff:ff:ff:ff:ff 2024-03-24T00:20:59.557 INFO:teuthology.orchestra.run.smithi078.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:20:59.557 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dd brd ff:ff:ff:ff:ff:ff 2024-03-24T00:20:59.557 INFO:teuthology.orchestra.run.smithi078.stdout: inet 172.21.15.78/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T00:20:59.557 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft 42094sec preferred_lft 42094sec 2024-03-24T00:20:59.558 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 fe80::ec4:7aff:fe8f:d4dd/64 scope link 2024-03-24T00:20:59.558 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:20:59.558 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-24T00:20:59.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:20:59.558 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2024-03-24T00:20:59.558 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-03-24T00:20:59.558 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link add name ceph-brx type bridge 2024-03-24T00:20:59.558 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip addr flush dev ceph-brx 2024-03-24T00:20:59.558 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set ceph-brx up 2024-03-24T00:20:59.559 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-24T00:20:59.559 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-03-24T00:20:59.638 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:59 smithi078 ceph-mon[32429]: pgmap v362: 129 pgs: 129 active+clean; 641 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s; 9 B/s, 0 objects/s recovering 2024-03-24T00:20:59.638 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:20:59 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/766049049' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T00:20:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:59 smithi106 ceph-mon[36505]: pgmap v362: 129 pgs: 129 active+clean; 641 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s; 9 B/s, 0 objects/s recovering 2024-03-24T00:20:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:20:59 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/766049049' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T00:20:59.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:20:59.668 DEBUG:teuthology.orchestra.run.smithi078:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:20:59.736 INFO:teuthology.orchestra.run.smithi078.stdout:1 2024-03-24T00:20:59.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:20:59.738 DEBUG:teuthology.orchestra.run.smithi078:> route 2024-03-24T00:20:59.794 INFO:teuthology.orchestra.run.smithi078.stdout:Kernel IP routing table 2024-03-24T00:20:59.794 INFO:teuthology.orchestra.run.smithi078.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:20:59.794 INFO:teuthology.orchestra.run.smithi078.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T00:20:59.795 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T00:20:59.795 INFO:teuthology.orchestra.run.smithi078.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:20:59.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:20:59.795 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2024-03-24T00:20:59.795 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-03-24T00:20:59.795 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T00:20:59.795 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T00:20:59.795 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T00:20:59.795 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-03-24T00:20:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:59 smithi107 ceph-mon[36485]: pgmap v362: 129 pgs: 129 active+clean; 641 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s; 9 B/s, 0 objects/s recovering 2024-03-24T00:20:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:20:59 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/766049049' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T00:20:59.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:20:59.892 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2024-03-24T00:20:59.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:20:59.906 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list-id 2024-03-24T00:20:59.961 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:20:59.961 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2024-03-24T00:20:59.961 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-03-24T00:20:59.961 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:20:59.962 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-24T00:20:59.962 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-03-24T00:21:00.045 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-24T00:21:00.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:21:00.045 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2024-03-24T00:21:00.045 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-03-24T00:21:00.045 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-24T00:21:00.045 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-24T00:21:00.045 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-24T00:21:00.045 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-24T00:21:00.045 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-24T00:21:00.046 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-03-24T00:21:00.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:21:00.173 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2024-03-24T00:21:00.174 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-03-24T00:21:00.174 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set brx.0 up 2024-03-24T00:21:00.174 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set dev brx.0 master ceph-brx 2024-03-24T00:21:00.174 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-03-24T00:21:00.216 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:21:00.216 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T00:21:00.271 INFO:teuthology.orchestra.run.smithi078.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:21:00.272 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:21:00.272 DEBUG:teuthology.orchestra.run.smithi078:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:21:00.326 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-24T00:21:00.327 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-24T00:21:00.327 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/9775ff22-04a1-4999-bfcb-98d63ec721c1 2024-03-24T00:21:00.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:21:00.327 DEBUG:teuthology.orchestra.run.smithi078:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/9775ff22-04a1-4999-bfcb-98d63ec721c1 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,wsync,nofallback 2024-03-24T00:21:00.412 INFO:teuthology.orchestra.run.smithi078.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,wsync,nofallback 2024-03-24T00:21:00.412 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,wsync". 2024-03-24T00:21:00.540 INFO:teuthology.orchestra.run.smithi078.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,wsync,nofallback 2024-03-24T00:21:00.540 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,wsync". 2024-03-24T00:21:00.540 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: resolved to: "172.21.15.78:6789,172.21.15.106:6789,172.21.15.107:6789" 2024-03-24T00:21:00.540 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: trying mount with new device syntax: 0@c7e2806a-e972-11ee-95cd-87774f69a715.cephfs=/volumes/qa/sv_1/9775ff22-04a1-4999-bfcb-98d63ec721c1 2024-03-24T00:21:00.540 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,wsync,name=0,key=0,mon_addr=172.21.15.78:6789/172.21.15.106:6789/172.21.15.107:6789" will pass to kernel 2024-03-24T00:21:00.541 INFO:teuthology.orchestra.run.smithi078.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-24T00:21:00.541 INFO:teuthology.orchestra.run.smithi078.stdout: You just mounted a file system that supports labels which does not 2024-03-24T00:21:00.541 INFO:teuthology.orchestra.run.smithi078.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-24T00:21:00.541 INFO:teuthology.orchestra.run.smithi078.stdout: applications will generate AVC messages and not be allowed access to 2024-03-24T00:21:00.541 INFO:teuthology.orchestra.run.smithi078.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-24T00:21:00.543 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-24T00:21:00.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:21:00.543 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:21:00.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:21:00.574 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:21:00.631 INFO:teuthology.orchestra.run.smithi078.stdout:client15462 2024-03-24T00:21:00.631 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-24T00:21:00.631 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:21:00.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:21:00.688 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:21:00.744 INFO:teuthology.orchestra.run.smithi078.stdout:client15462 2024-03-24T00:21:00.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:21:00.744 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:21:00.800 INFO:teuthology.orchestra.run.smithi078.stdout:client15462 2024-03-24T00:21:00.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:21:00.801 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/c7e2806a-e972-11ee-95cd-87774f69a715.client15462/status') 2024-03-24T00:21:00.866 INFO:teuthology.orchestra.run.smithi078.stdout:instance: client.15462 (3)192.168.0.1:0/3344586265 2024-03-24T00:21:00.866 INFO:teuthology.orchestra.run.smithi078.stdout:blocklisted: false 2024-03-24T00:21:00.867 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records in 2024-03-24T00:21:00.867 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records out 2024-03-24T00:21:00.867 INFO:teuthology.orchestra.run.smithi078.stderr:70 bytes copied, 2.2819e-05 s, 3.1 MB/s 2024-03-24T00:21:00.868 INFO:teuthology.run_tasks:Running task exec... 2024-03-24T00:21:00.880 INFO:teuthology.task.exec:Executing custom commands... 2024-03-24T00:21:00.880 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2024-03-24T00:21:00.880 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-03-24T00:21:01.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.012+0000 7f3f94075640 1 -- 172.21.15.78:0/4123840203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f8c107420 msgr2=0x7f3f8c109810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:01.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.012+0000 7f3f94075640 1 --2- 172.21.15.78:0/4123840203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f8c107420 0x7f3f8c109810 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f3f84009ca0 tx=0x7f3f8402c0f0 comp rx=0 tx=0).stop 2024-03-24T00:21:01.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.012+0000 7f3f94075640 1 -- 172.21.15.78:0/4123840203 shutdown_connections 2024-03-24T00:21:01.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.012+0000 7f3f94075640 1 --2- 172.21.15.78:0/4123840203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f8c107420 0x7f3f8c109810 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:01.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.012+0000 7f3f94075640 1 --2- 172.21.15.78:0/4123840203 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f8c104af0 0x7f3f8c106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:01.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.012+0000 7f3f94075640 1 --2- 172.21.15.78:0/4123840203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f8c1021c0 0x7f3f8c1045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:01.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.012+0000 7f3f94075640 1 -- 172.21.15.78:0/4123840203 >> 172.21.15.78:0/4123840203 conn(0x7f3f8c0fbb30 msgr2=0x7f3f8c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:01.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.013+0000 7f3f94075640 1 -- 172.21.15.78:0/4123840203 shutdown_connections 2024-03-24T00:21:01.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.013+0000 7f3f94075640 1 -- 172.21.15.78:0/4123840203 wait complete. 2024-03-24T00:21:01.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.013+0000 7f3f94075640 1 Processor -- start 2024-03-24T00:21:01.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.013+0000 7f3f94075640 1 -- start start 2024-03-24T00:21:01.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.013+0000 7f3f94075640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f8c1021c0 0x7f3f8c197910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:01.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.013+0000 7f3f94075640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f8c104af0 0x7f3f8c197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:01.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.013+0000 7f3f94075640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f8c107420 0x7f3f8c19eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:01.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.013+0000 7f3f94075640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8c10d360 con 0x7f3f8c104af0 2024-03-24T00:21:01.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.013+0000 7f3f94075640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8c10d1e0 con 0x7f3f8c1021c0 2024-03-24T00:21:01.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.013+0000 7f3f94075640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8c10d4e0 con 0x7f3f8c107420 2024-03-24T00:21:01.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.013+0000 7f3f91610640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f8c104af0 0x7f3f8c197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:01.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.013+0000 7f3f91610640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f8c104af0 0x7f3f8c197e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55544/0 (socket says 172.21.15.78:55544) 2024-03-24T00:21:01.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.013+0000 7f3f91610640 1 -- 172.21.15.78:0/2151414121 learned_addr learned my addr 172.21.15.78:0/2151414121 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:21:01.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.013+0000 7f3f91e11640 1 --2- 172.21.15.78:0/2151414121 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f8c1021c0 0x7f3f8c197910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:01.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.013+0000 7f3f92612640 1 --2- 172.21.15.78:0/2151414121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f8c107420 0x7f3f8c19eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:01.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.014+0000 7f3f91610640 1 -- 172.21.15.78:0/2151414121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f8c107420 msgr2=0x7f3f8c19eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:01.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.014+0000 7f3f91610640 1 --2- 172.21.15.78:0/2151414121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f8c107420 0x7f3f8c19eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:01.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.014+0000 7f3f91610640 1 -- 172.21.15.78:0/2151414121 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f8c1021c0 msgr2=0x7f3f8c197910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:01.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.014+0000 7f3f91610640 1 --2- 172.21.15.78:0/2151414121 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f8c1021c0 0x7f3f8c197910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:01.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.014+0000 7f3f91610640 1 -- 172.21.15.78:0/2151414121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f8c19f530 con 0x7f3f8c104af0 2024-03-24T00:21:01.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.014+0000 7f3f91610640 1 --2- 172.21.15.78:0/2151414121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f8c104af0 0x7f3f8c197e50 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c005fd0 tx=0x7f3f7c00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:01.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.014+0000 7f3f82ffd640 1 -- 172.21.15.78:0/2151414121 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c017070 con 0x7f3f8c104af0 2024-03-24T00:21:01.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.014+0000 7f3f94075640 1 -- 172.21.15.78:0/2151414121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f8c19f820 con 0x7f3f8c104af0 2024-03-24T00:21:01.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.014+0000 7f3f82ffd640 1 -- 172.21.15.78:0/2151414121 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3f7c009c30 con 0x7f3f8c104af0 2024-03-24T00:21:01.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.014+0000 7f3f82ffd640 1 -- 172.21.15.78:0/2151414121 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c010460 con 0x7f3f8c104af0 2024-03-24T00:21:01.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.014+0000 7f3f94075640 1 -- 172.21.15.78:0/2151414121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f8c19fdc0 con 0x7f3f8c104af0 2024-03-24T00:21:01.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.015+0000 7f3f94075640 1 -- 172.21.15.78:0/2151414121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f8c0fd620 con 0x7f3f8c104af0 2024-03-24T00:21:01.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.016+0000 7f3f82ffd640 1 -- 172.21.15.78:0/2151414121 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3f7c003e70 con 0x7f3f8c104af0 2024-03-24T00:21:01.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.016+0000 7f3f82ffd640 1 --2- 172.21.15.78:0/2151414121 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3f780a28a0 0x7f3f780a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:01.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.017+0000 7f3f91e11640 1 --2- 172.21.15.78:0/2151414121 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3f780a28a0 0x7f3f780a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:01.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.017+0000 7f3f82ffd640 1 -- 172.21.15.78:0/2151414121 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f3f7c0cea00 con 0x7f3f8c104af0 2024-03-24T00:21:01.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.017+0000 7f3f91e11640 1 --2- 172.21.15.78:0/2151414121 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3f780a28a0 0x7f3f780a4d60 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3f88009bd0 tx=0x7f3f88005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:01.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.021+0000 7f3f82ffd640 1 -- 172.21.15.78:0/2151414121 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3f7c020030 con 0x7f3f8c104af0 2024-03-24T00:21:01.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.244+0000 7f3f94075640 1 -- 172.21.15.78:0/2151414121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) v1 -- 0x7f3f8c0622a0 con 0x7f3f8c104af0 2024-03-24T00:21:01.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.390+0000 7f3f82ffd640 1 -- 172.21.15.78:0/2151414121 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v24) v1 ==== 117+0+0 (secure 0 0 0) 0x7f3f7c09b710 con 0x7f3f8c104af0 2024-03-24T00:21:01.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.392+0000 7f3f94075640 1 -- 172.21.15.78:0/2151414121 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3f780a28a0 msgr2=0x7f3f780a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:01.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.392+0000 7f3f94075640 1 --2- 172.21.15.78:0/2151414121 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3f780a28a0 0x7f3f780a4d60 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3f88009bd0 tx=0x7f3f88005f90 comp rx=0 tx=0).stop 2024-03-24T00:21:01.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.392+0000 7f3f94075640 1 -- 172.21.15.78:0/2151414121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f8c104af0 msgr2=0x7f3f8c197e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:01.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.392+0000 7f3f94075640 1 --2- 172.21.15.78:0/2151414121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f8c104af0 0x7f3f8c197e50 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c005fd0 tx=0x7f3f7c00dde0 comp rx=0 tx=0).stop 2024-03-24T00:21:01.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.392+0000 7f3f94075640 1 -- 172.21.15.78:0/2151414121 shutdown_connections 2024-03-24T00:21:01.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.392+0000 7f3f94075640 1 --2- 172.21.15.78:0/2151414121 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3f780a28a0 0x7f3f780a4d60 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:01.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.392+0000 7f3f94075640 1 --2- 172.21.15.78:0/2151414121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f8c107420 0x7f3f8c19eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:01.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.392+0000 7f3f94075640 1 --2- 172.21.15.78:0/2151414121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f8c104af0 0x7f3f8c197e50 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:01.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.392+0000 7f3f94075640 1 --2- 172.21.15.78:0/2151414121 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f8c1021c0 0x7f3f8c197910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:01.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.392+0000 7f3f94075640 1 -- 172.21.15.78:0/2151414121 >> 172.21.15.78:0/2151414121 conn(0x7f3f8c0fbb30 msgr2=0x7f3f8c102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:01.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.392+0000 7f3f94075640 1 -- 172.21.15.78:0/2151414121 shutdown_connections 2024-03-24T00:21:01.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.392+0000 7f3f94075640 1 -- 172.21.15.78:0/2151414121 wait complete. 2024-03-24T00:21:01.403 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-24T00:21:01.414 INFO:teuthology.run_tasks:Running task exec... 2024-03-24T00:21:01.425 INFO:teuthology.task.exec:Executing custom commands... 2024-03-24T00:21:01.425 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2024-03-24T00:21:01.425 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-03-24T00:21:01.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.528+0000 7ffaf075b640 1 -- 172.21.15.78:0/2607069485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffae81005e0 msgr2=0x7ffae8100a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:01.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.528+0000 7ffaf075b640 1 --2- 172.21.15.78:0/2607069485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffae81005e0 0x7ffae8100a00 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ffae4009900 tx=0x7ffae402bc30 comp rx=0 tx=0).stop 2024-03-24T00:21:01.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.528+0000 7ffaf075b640 1 -- 172.21.15.78:0/2607069485 shutdown_connections 2024-03-24T00:21:01.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.528+0000 7ffaf075b640 1 --2- 172.21.15.78:0/2607069485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffae81017d0 0x7ffae8101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:01.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.528+0000 7ffaf075b640 1 --2- 172.21.15.78:0/2607069485 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffae8102190 0x7ffae8108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:01.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.528+0000 7ffaf075b640 1 --2- 172.21.15.78:0/2607069485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffae81005e0 0x7ffae8100a00 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:01.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.528+0000 7ffaf075b640 1 -- 172.21.15.78:0/2607069485 >> 172.21.15.78:0/2607069485 conn(0x7ffae80fbb50 msgr2=0x7ffae80fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:01.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.528+0000 7ffaf075b640 1 -- 172.21.15.78:0/2607069485 shutdown_connections 2024-03-24T00:21:01.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.528+0000 7ffaf075b640 1 -- 172.21.15.78:0/2607069485 wait complete. 2024-03-24T00:21:01.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.529+0000 7ffaf075b640 1 Processor -- start 2024-03-24T00:21:01.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.529+0000 7ffaf075b640 1 -- start start 2024-03-24T00:21:01.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.529+0000 7ffaf075b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffae81005e0 0x7ffae8199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:01.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.529+0000 7ffaf075b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffae81017d0 0x7ffae819a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:01.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.529+0000 7ffaf075b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffae8102190 0x7ffae81a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:01.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.529+0000 7ffaf075b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae810d380 con 0x7ffae81017d0 2024-03-24T00:21:01.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.529+0000 7ffaf075b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae810d200 con 0x7ffae81005e0 2024-03-24T00:21:01.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.529+0000 7ffaf075b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae810d500 con 0x7ffae8102190 2024-03-24T00:21:01.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.529+0000 7ffaedcf6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffae81017d0 0x7ffae819a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:01.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.529+0000 7ffaedcf6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffae81017d0 0x7ffae819a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55556/0 (socket says 172.21.15.78:55556) 2024-03-24T00:21:01.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.529+0000 7ffaedcf6640 1 -- 172.21.15.78:0/1475436602 learned_addr learned my addr 172.21.15.78:0/1475436602 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:21:01.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.529+0000 7ffaee4f7640 1 --2- 172.21.15.78:0/1475436602 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffae81005e0 0x7ffae8199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:01.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.529+0000 7ffaeecf8640 1 --2- 172.21.15.78:0/1475436602 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffae8102190 0x7ffae81a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:01.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.529+0000 7ffaedcf6640 1 -- 172.21.15.78:0/1475436602 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffae8102190 msgr2=0x7ffae81a10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:01.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.529+0000 7ffaedcf6640 1 --2- 172.21.15.78:0/1475436602 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffae8102190 0x7ffae81a10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:01.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.529+0000 7ffaedcf6640 1 -- 172.21.15.78:0/1475436602 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffae81005e0 msgr2=0x7ffae8199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:01.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.529+0000 7ffaedcf6640 1 --2- 172.21.15.78:0/1475436602 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffae81005e0 0x7ffae8199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:01.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.529+0000 7ffaedcf6640 1 -- 172.21.15.78:0/1475436602 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae81a1750 con 0x7ffae81017d0 2024-03-24T00:21:01.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.529+0000 7ffaedcf6640 1 --2- 172.21.15.78:0/1475436602 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffae81017d0 0x7ffae819a070 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ffad800e9f0 tx=0x7ffad800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:01.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.529+0000 7ffadf7fe640 1 -- 172.21.15.78:0/1475436602 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad800cd80 con 0x7ffae81017d0 2024-03-24T00:21:01.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.529+0000 7ffadf7fe640 1 -- 172.21.15.78:0/1475436602 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffad8005500 con 0x7ffae81017d0 2024-03-24T00:21:01.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.529+0000 7ffaf075b640 1 -- 172.21.15.78:0/1475436602 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffae81a1a40 con 0x7ffae81017d0 2024-03-24T00:21:01.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.529+0000 7ffadf7fe640 1 -- 172.21.15.78:0/1475436602 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad8010690 con 0x7ffae81017d0 2024-03-24T00:21:01.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.529+0000 7ffaf075b640 1 -- 172.21.15.78:0/1475436602 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffae81a1f40 con 0x7ffae81017d0 2024-03-24T00:21:01.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.530+0000 7ffaf075b640 1 -- 172.21.15.78:0/1475436602 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffae81a21d0 con 0x7ffae81017d0 2024-03-24T00:21:01.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.531+0000 7ffadf7fe640 1 -- 172.21.15.78:0/1475436602 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ffad80046c0 con 0x7ffae81017d0 2024-03-24T00:21:01.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.532+0000 7ffadf7fe640 1 --2- 172.21.15.78:0/1475436602 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ffad40a28a0 0x7ffad40a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:01.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.532+0000 7ffaee4f7640 1 --2- 172.21.15.78:0/1475436602 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ffad40a28a0 0x7ffad40a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:01.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.532+0000 7ffadf7fe640 1 -- 172.21.15.78:0/1475436602 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7ffad80cf1e0 con 0x7ffae81017d0 2024-03-24T00:21:01.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.532+0000 7ffaee4f7640 1 --2- 172.21.15.78:0/1475436602 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ffad40a28a0 0x7ffad40a4d60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ffae4000c00 tx=0x7ffae4005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:01.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.536+0000 7ffadf7fe640 1 -- 172.21.15.78:0/1475436602 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ffad809bef0 con 0x7ffae81017d0 2024-03-24T00:21:01.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:01 smithi078 ceph-mon[32429]: pgmap v363: 129 pgs: 129 active+clean; 641 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:21:01.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:01 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2151414121' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-24T00:21:01.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:01 smithi106 ceph-mon[36505]: pgmap v363: 129 pgs: 129 active+clean; 641 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:21:01.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:01 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2151414121' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-24T00:21:01.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:01.763+0000 7ffaf075b640 1 -- 172.21.15.78:0/1475436602 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7ffae81a23a0 con 0x7ffae81017d0 2024-03-24T00:21:01.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:01 smithi107 ceph-mon[36485]: pgmap v363: 129 pgs: 129 active+clean; 641 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:21:01.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:01 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2151414121' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-24T00:21:02.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.399+0000 7ffadf7fe640 1 -- 172.21.15.78:0/1475436602 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) v1 ==== 92+0+0 (secure 0 0 0) 0x7ffad809b7f0 con 0x7ffae81017d0 2024-03-24T00:21:02.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.402+0000 7ffaf075b640 1 -- 172.21.15.78:0/1475436602 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ffad40a28a0 msgr2=0x7ffad40a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:02.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.402+0000 7ffaf075b640 1 --2- 172.21.15.78:0/1475436602 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ffad40a28a0 0x7ffad40a4d60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ffae4000c00 tx=0x7ffae4005fd0 comp rx=0 tx=0).stop 2024-03-24T00:21:02.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.402+0000 7ffaf075b640 1 -- 172.21.15.78:0/1475436602 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffae81017d0 msgr2=0x7ffae819a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:02.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.402+0000 7ffaf075b640 1 --2- 172.21.15.78:0/1475436602 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffae81017d0 0x7ffae819a070 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ffad800e9f0 tx=0x7ffad800edf0 comp rx=0 tx=0).stop 2024-03-24T00:21:02.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.402+0000 7ffaf075b640 1 -- 172.21.15.78:0/1475436602 shutdown_connections 2024-03-24T00:21:02.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.402+0000 7ffaf075b640 1 --2- 172.21.15.78:0/1475436602 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7ffad40a28a0 0x7ffad40a4d60 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:02.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.402+0000 7ffaf075b640 1 --2- 172.21.15.78:0/1475436602 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffae8102190 0x7ffae81a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:02.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.402+0000 7ffaf075b640 1 --2- 172.21.15.78:0/1475436602 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffae81017d0 0x7ffae819a070 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:02.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.402+0000 7ffaf075b640 1 --2- 172.21.15.78:0/1475436602 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffae81005e0 0x7ffae8199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:02.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.402+0000 7ffaf075b640 1 -- 172.21.15.78:0/1475436602 >> 172.21.15.78:0/1475436602 conn(0x7ffae80fbb50 msgr2=0x7ffae81093e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:02.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.402+0000 7ffaf075b640 1 -- 172.21.15.78:0/1475436602 shutdown_connections 2024-03-24T00:21:02.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.404+0000 7ffaf075b640 1 -- 172.21.15.78:0/1475436602 wait complete. 2024-03-24T00:21:02.419 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-03-24T00:21:02.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.529+0000 7f5da5959640 1 -- 172.21.15.78:0/1196513134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5da01017f0 msgr2=0x7f5da0101c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:02.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.529+0000 7f5da5959640 1 --2- 172.21.15.78:0/1196513134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5da01017f0 0x7f5da0101c70 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f5d94009900 tx=0x7f5d9402bc30 comp rx=0 tx=0).stop 2024-03-24T00:21:02.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.529+0000 7f5da5959640 1 -- 172.21.15.78:0/1196513134 shutdown_connections 2024-03-24T00:21:02.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.529+0000 7f5da5959640 1 --2- 172.21.15.78:0/1196513134 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5da01021b0 0x7f5da0108a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:02.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.529+0000 7f5da5959640 1 --2- 172.21.15.78:0/1196513134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5da01017f0 0x7f5da0101c70 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:02.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.529+0000 7f5da5959640 1 --2- 172.21.15.78:0/1196513134 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5da0100600 0x7f5da0100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:02.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.529+0000 7f5da5959640 1 -- 172.21.15.78:0/1196513134 >> 172.21.15.78:0/1196513134 conn(0x7f5da00fbb70 msgr2=0x7f5da00fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:02.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.529+0000 7f5da5959640 1 -- 172.21.15.78:0/1196513134 shutdown_connections 2024-03-24T00:21:02.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.529+0000 7f5da5959640 1 -- 172.21.15.78:0/1196513134 wait complete. 2024-03-24T00:21:02.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.530+0000 7f5da5959640 1 Processor -- start 2024-03-24T00:21:02.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.530+0000 7f5da5959640 1 -- start start 2024-03-24T00:21:02.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.530+0000 7f5da5959640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5da0100600 0x7f5da006a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:02.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.530+0000 7f5da5959640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5da01017f0 0x7f5da019c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:02.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.530+0000 7f5da5959640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5da01021b0 0x7f5da019ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:02.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.530+0000 7f5da5959640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da010c280 con 0x7f5da0100600 2024-03-24T00:21:02.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.530+0000 7f5da5959640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da010c100 con 0x7f5da01021b0 2024-03-24T00:21:02.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.530+0000 7f5da5959640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da010c400 con 0x7f5da01017f0 2024-03-24T00:21:02.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.530+0000 7f5d9effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5da0100600 0x7f5da006a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:02.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.530+0000 7f5d9effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5da0100600 0x7f5da006a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55588/0 (socket says 172.21.15.78:55588) 2024-03-24T00:21:02.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.530+0000 7f5d9effd640 1 -- 172.21.15.78:0/3372850153 learned_addr learned my addr 172.21.15.78:0/3372850153 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:21:02.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.530+0000 7f5d9e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5da01017f0 0x7f5da019c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:02.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.530+0000 7f5d9f7fe640 1 --2- 172.21.15.78:0/3372850153 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5da01021b0 0x7f5da019ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:02.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.531+0000 7f5d9effd640 1 -- 172.21.15.78:0/3372850153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5da01017f0 msgr2=0x7f5da019c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:02.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.531+0000 7f5d9effd640 1 --2- 172.21.15.78:0/3372850153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5da01017f0 0x7f5da019c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:02.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.531+0000 7f5d9effd640 1 -- 172.21.15.78:0/3372850153 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5da01021b0 msgr2=0x7f5da019ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:02.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.531+0000 7f5d9effd640 1 --2- 172.21.15.78:0/3372850153 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5da01021b0 0x7f5da019ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:02.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.531+0000 7f5d9effd640 1 -- 172.21.15.78:0/3372850153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da019f590 con 0x7f5da0100600 2024-03-24T00:21:02.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.531+0000 7f5d9e7fc640 1 --2- 172.21.15.78:0/3372850153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5da01017f0 0x7f5da019c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:21:02.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.531+0000 7f5d9effd640 1 --2- 172.21.15.78:0/3372850153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5da0100600 0x7f5da006a460 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f5d9800eb70 tx=0x7f5d9800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:02.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.531+0000 7f5da4957640 1 -- 172.21.15.78:0/3372850153 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d98003b70 con 0x7f5da0100600 2024-03-24T00:21:02.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.531+0000 7f5da5959640 1 -- 172.21.15.78:0/3372850153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5da019f880 con 0x7f5da0100600 2024-03-24T00:21:02.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.531+0000 7f5da4957640 1 -- 172.21.15.78:0/3372850153 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5d9800ee70 con 0x7f5da0100600 2024-03-24T00:21:02.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.531+0000 7f5da4957640 1 -- 172.21.15.78:0/3372850153 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d98010830 con 0x7f5da0100600 2024-03-24T00:21:02.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.531+0000 7f5da5959640 1 -- 172.21.15.78:0/3372850153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5da019fda0 con 0x7f5da0100600 2024-03-24T00:21:02.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.532+0000 7f5da5959640 1 -- 172.21.15.78:0/3372850153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5da0107850 con 0x7f5da0100600 2024-03-24T00:21:02.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.533+0000 7f5da4957640 1 -- 172.21.15.78:0/3372850153 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 137307+0+0 (secure 0 0 0) 0x7f5d98004020 con 0x7f5da0100600 2024-03-24T00:21:02.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.534+0000 7f5da4957640 1 --2- 172.21.15.78:0/3372850153 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f5d880a28f0 0x7f5d880a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:02.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.534+0000 7f5d9e7fc640 1 -- 172.21.15.78:0/3372850153 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f5d880a28f0 msgr2=0x7f5d880a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1983101026 2024-03-24T00:21:02.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.534+0000 7f5da4957640 1 -- 172.21.15.78:0/3372850153 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f5d980cf4c0 con 0x7f5da0100600 2024-03-24T00:21:02.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.534+0000 7f5d9e7fc640 1 --2- 172.21.15.78:0/3372850153 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f5d880a28f0 0x7f5d880a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:21:02.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.538+0000 7f5da4957640 1 -- 172.21.15.78:0/3372850153 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5d9809c1d0 con 0x7f5da0100600 2024-03-24T00:21:02.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:02 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2151414121' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-24T00:21:02.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:02 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:21:02.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:02 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1475436602' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-24T00:21:02.652 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:21:02 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: ignoring --setuser ceph since I am not root 2024-03-24T00:21:02.653 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:21:02 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: ignoring --setgroup ceph since I am not root 2024-03-24T00:21:02.653 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:21:02 smithi106 ceph-mgr[37896]: -- 172.21.15.106:0/2878218152 <== mon.2 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563675c465a0 con 0x563675c1b000 2024-03-24T00:21:02.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.719+0000 7f5da5959640 1 -- 172.21.15.78:0/3372850153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f5da00622a0 con 0x7f5da0100600 2024-03-24T00:21:02.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.723+0000 7f5da4957640 1 -- 172.21.15.78:0/3372850153 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v33)=0 v33) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5d9809bad0 con 0x7f5da0100600 2024-03-24T00:21:02.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.725+0000 7f5da5959640 1 -- 172.21.15.78:0/3372850153 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f5d880a28f0 msgr2=0x7f5d880a4db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T00:21:02.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.725+0000 7f5da5959640 1 --2- 172.21.15.78:0/3372850153 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f5d880a28f0 0x7f5d880a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:02.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.725+0000 7f5da5959640 1 -- 172.21.15.78:0/3372850153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5da0100600 msgr2=0x7f5da006a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:02.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.725+0000 7f5da5959640 1 --2- 172.21.15.78:0/3372850153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5da0100600 0x7f5da006a460 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f5d9800eb70 tx=0x7f5d9800c6a0 comp rx=0 tx=0).stop 2024-03-24T00:21:02.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.725+0000 7f5da5959640 1 -- 172.21.15.78:0/3372850153 shutdown_connections 2024-03-24T00:21:02.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.725+0000 7f5da5959640 1 --2- 172.21.15.78:0/3372850153 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f5d880a28f0 0x7f5d880a4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:02.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.725+0000 7f5da5959640 1 --2- 172.21.15.78:0/3372850153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5da01017f0 0x7f5da019c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:02.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.725+0000 7f5da5959640 1 --2- 172.21.15.78:0/3372850153 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5da01021b0 0x7f5da019ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:02.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.725+0000 7f5da5959640 1 --2- 172.21.15.78:0/3372850153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5da0100600 0x7f5da006a460 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:02.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.725+0000 7f5da5959640 1 -- 172.21.15.78:0/3372850153 >> 172.21.15.78:0/3372850153 conn(0x7f5da00fbb70 msgr2=0x7f5da0104a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:02.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.725+0000 7f5da5959640 1 -- 172.21.15.78:0/3372850153 shutdown_connections 2024-03-24T00:21:02.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.725+0000 7f5da5959640 1 -- 172.21.15.78:0/3372850153 wait complete. 2024-03-24T00:21:02.735 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:02 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2151414121' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-24T00:21:02.735 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:02 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:21:02.735 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:02 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1475436602' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-24T00:21:02.736 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:02 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: ignoring --setuser ceph since I am not root 2024-03-24T00:21:02.736 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:02 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: ignoring --setgroup ceph since I am not root 2024-03-24T00:21:02.737 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-03-24T00:21:02.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.841+0000 7fe8c0c3d640 1 -- 172.21.15.78:0/616536739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8bc102140 msgr2=0x7fe8bc104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:02.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.841+0000 7fe8c0c3d640 1 --2- 172.21.15.78:0/616536739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8bc102140 0x7fe8bc104530 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac009900 tx=0x7fe8ac02bc30 comp rx=0 tx=0).stop 2024-03-24T00:21:02.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.842+0000 7fe8c0c3d640 1 -- 172.21.15.78:0/616536739 shutdown_connections 2024-03-24T00:21:02.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.842+0000 7fe8c0c3d640 1 --2- 172.21.15.78:0/616536739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8bc1073a0 0x7fe8bc1097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:02.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.842+0000 7fe8c0c3d640 1 --2- 172.21.15.78:0/616536739 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8bc104a70 0x7fe8bc106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:02.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.842+0000 7fe8c0c3d640 1 --2- 172.21.15.78:0/616536739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8bc102140 0x7fe8bc104530 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:02.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.842+0000 7fe8c0c3d640 1 -- 172.21.15.78:0/616536739 >> 172.21.15.78:0/616536739 conn(0x7fe8bc0fbb30 msgr2=0x7fe8bc0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:02.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.842+0000 7fe8c0c3d640 1 -- 172.21.15.78:0/616536739 shutdown_connections 2024-03-24T00:21:02.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.842+0000 7fe8c0c3d640 1 -- 172.21.15.78:0/616536739 wait complete. 2024-03-24T00:21:02.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.842+0000 7fe8c0c3d640 1 Processor -- start 2024-03-24T00:21:02.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.842+0000 7fe8c0c3d640 1 -- start start 2024-03-24T00:21:02.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.842+0000 7fe8c0c3d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8bc102140 0x7fe8bc101ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:02.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.842+0000 7fe8c0c3d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8bc104a70 0x7fe8bc1000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:02.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.842+0000 7fe8c0c3d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8bc1073a0 0x7fe8bc100630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:02.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.842+0000 7fe8c0c3d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8bc10d350 con 0x7fe8bc102140 2024-03-24T00:21:02.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.842+0000 7fe8c0c3d640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8bc10d1d0 con 0x7fe8bc1073a0 2024-03-24T00:21:02.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.842+0000 7fe8c0c3d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8bc10d4d0 con 0x7fe8bc104a70 2024-03-24T00:21:02.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.843+0000 7fe8ba59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8bc102140 0x7fe8bc101ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:02.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.843+0000 7fe8ba59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8bc102140 0x7fe8bc101ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55596/0 (socket says 172.21.15.78:55596) 2024-03-24T00:21:02.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.843+0000 7fe8ba59c640 1 -- 172.21.15.78:0/1586336724 learned_addr learned my addr 172.21.15.78:0/1586336724 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:21:02.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.843+0000 7fe8bad9d640 1 --2- 172.21.15.78:0/1586336724 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8bc1073a0 0x7fe8bc100630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:02.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.843+0000 7fe8b3fff640 1 --2- 172.21.15.78:0/1586336724 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8bc104a70 0x7fe8bc1000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:02.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.843+0000 7fe8ba59c640 1 -- 172.21.15.78:0/1586336724 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8bc104a70 msgr2=0x7fe8bc1000f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:02.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.843+0000 7fe8ba59c640 1 --2- 172.21.15.78:0/1586336724 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8bc104a70 0x7fe8bc1000f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:02.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.843+0000 7fe8ba59c640 1 -- 172.21.15.78:0/1586336724 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8bc1073a0 msgr2=0x7fe8bc100630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:02.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.843+0000 7fe8ba59c640 1 --2- 172.21.15.78:0/1586336724 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8bc1073a0 0x7fe8bc100630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:02.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.843+0000 7fe8ba59c640 1 -- 172.21.15.78:0/1586336724 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8bc100ec0 con 0x7fe8bc102140 2024-03-24T00:21:02.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.843+0000 7fe8ba59c640 1 --2- 172.21.15.78:0/1586336724 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8bc102140 0x7fe8bc101ab0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac009230 tx=0x7fe8ac005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:02.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.843+0000 7fe8b37fe640 1 -- 172.21.15.78:0/1586336724 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac041070 con 0x7fe8bc102140 2024-03-24T00:21:02.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.843+0000 7fe8b37fe640 1 -- 172.21.15.78:0/1586336724 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe8ac0054e0 con 0x7fe8bc102140 2024-03-24T00:21:02.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.843+0000 7fe8b37fe640 1 -- 172.21.15.78:0/1586336724 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac034440 con 0x7fe8bc102140 2024-03-24T00:21:02.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.843+0000 7fe8c0c3d640 1 -- 172.21.15.78:0/1586336724 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8bc1a1a50 con 0x7fe8bc102140 2024-03-24T00:21:02.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.843+0000 7fe8c0c3d640 1 -- 172.21.15.78:0/1586336724 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8bc1a1ff0 con 0x7fe8bc102140 2024-03-24T00:21:02.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:02 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2151414121' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-24T00:21:02.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:02 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:21:02.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:02 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1475436602' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-24T00:21:02.847 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:21:02 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: ignoring --setuser ceph since I am not root 2024-03-24T00:21:02.847 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:21:02 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: ignoring --setgroup ceph since I am not root 2024-03-24T00:21:02.847 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:21:02 smithi107 ceph-mgr[38355]: -- 172.21.15.107:0/6012226 <== mon.2 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55948a3205a0 con 0x55948a2f4800 2024-03-24T00:21:02.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.845+0000 7fe8c0c3d640 1 -- 172.21.15.78:0/1586336724 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8bc0fd620 con 0x7fe8bc102140 2024-03-24T00:21:02.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.846+0000 7fe8b37fe640 1 -- 172.21.15.78:0/1586336724 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 137307+0+0 (secure 0 0 0) 0x7fe8ac044070 con 0x7fe8bc102140 2024-03-24T00:21:02.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.847+0000 7fe8b37fe640 1 --2- 172.21.15.78:0/1586336724 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe8a80a2970 0x7fe8a80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:02.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.847+0000 7fe8b3fff640 1 -- 172.21.15.78:0/1586336724 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe8a80a2970 msgr2=0x7fe8a80a4e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1983101026 2024-03-24T00:21:02.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.847+0000 7fe8b37fe640 1 -- 172.21.15.78:0/1586336724 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7fe8ac0f0a30 con 0x7fe8bc102140 2024-03-24T00:21:02.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.847+0000 7fe8b3fff640 1 --2- 172.21.15.78:0/1586336724 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe8a80a2970 0x7fe8a80a4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:21:02.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:02.851+0000 7fe8b37fe640 1 -- 172.21.15.78:0/1586336724 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe8ac0bd0f0 con 0x7fe8bc102140 2024-03-24T00:21:03.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.034+0000 7fe8c0c3d640 1 -- 172.21.15.78:0/1586336724 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fe8bc02d070 con 0x7fe8bc102140 2024-03-24T00:21:03.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.038+0000 7fe8b37fe640 1 -- 172.21.15.78:0/1586336724 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v34)=0 v34) v1 ==== 133+0+0 (secure 0 0 0) 0x7fe8ac0c1190 con 0x7fe8bc102140 2024-03-24T00:21:03.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.041+0000 7fe8c0c3d640 1 -- 172.21.15.78:0/1586336724 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe8a80a2970 msgr2=0x7fe8a80a4e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T00:21:03.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.041+0000 7fe8c0c3d640 1 --2- 172.21.15.78:0/1586336724 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe8a80a2970 0x7fe8a80a4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:03.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.041+0000 7fe8c0c3d640 1 -- 172.21.15.78:0/1586336724 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8bc102140 msgr2=0x7fe8bc101ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:03.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.041+0000 7fe8c0c3d640 1 --2- 172.21.15.78:0/1586336724 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8bc102140 0x7fe8bc101ab0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac009230 tx=0x7fe8ac005dd0 comp rx=0 tx=0).stop 2024-03-24T00:21:03.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.041+0000 7fe8c0c3d640 1 -- 172.21.15.78:0/1586336724 shutdown_connections 2024-03-24T00:21:03.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.041+0000 7fe8c0c3d640 1 --2- 172.21.15.78:0/1586336724 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7fe8a80a2970 0x7fe8a80a4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:03.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.041+0000 7fe8c0c3d640 1 --2- 172.21.15.78:0/1586336724 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8bc104a70 0x7fe8bc1000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:03.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.041+0000 7fe8c0c3d640 1 --2- 172.21.15.78:0/1586336724 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8bc1073a0 0x7fe8bc100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:03.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.041+0000 7fe8c0c3d640 1 --2- 172.21.15.78:0/1586336724 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8bc102140 0x7fe8bc101ab0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:03.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.041+0000 7fe8c0c3d640 1 -- 172.21.15.78:0/1586336724 >> 172.21.15.78:0/1586336724 conn(0x7fe8bc0fbb30 msgr2=0x7fe8bc102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:03.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.041+0000 7fe8c0c3d640 1 -- 172.21.15.78:0/1586336724 shutdown_connections 2024-03-24T00:21:03.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.042+0000 7fe8c0c3d640 1 -- 172.21.15.78:0/1586336724 wait complete. 2024-03-24T00:21:03.056 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-03-24T00:21:03.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.161+0000 7f3ef84de640 1 -- 172.21.15.78:0/1782078177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ef01005c0 msgr2=0x7f3ef01009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:03.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.161+0000 7f3ef84de640 1 --2- 172.21.15.78:0/1782078177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ef01005c0 0x7f3ef01009e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0009900 tx=0x7f3ee002bc30 comp rx=0 tx=0).stop 2024-03-24T00:21:03.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.161+0000 7f3ef84de640 1 -- 172.21.15.78:0/1782078177 shutdown_connections 2024-03-24T00:21:03.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.161+0000 7f3ef84de640 1 --2- 172.21.15.78:0/1782078177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ef01017b0 0x7f3ef0101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:03.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.161+0000 7f3ef84de640 1 --2- 172.21.15.78:0/1782078177 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ef0102170 0x7f3ef0108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:03.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.161+0000 7f3ef84de640 1 --2- 172.21.15.78:0/1782078177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ef01005c0 0x7f3ef01009e0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:03.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.161+0000 7f3ef84de640 1 -- 172.21.15.78:0/1782078177 >> 172.21.15.78:0/1782078177 conn(0x7f3ef00fbb30 msgr2=0x7f3ef00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:03.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.161+0000 7f3ef84de640 1 -- 172.21.15.78:0/1782078177 shutdown_connections 2024-03-24T00:21:03.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.161+0000 7f3ef84de640 1 -- 172.21.15.78:0/1782078177 wait complete. 2024-03-24T00:21:03.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.162+0000 7f3ef84de640 1 Processor -- start 2024-03-24T00:21:03.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.162+0000 7f3ef84de640 1 -- start start 2024-03-24T00:21:03.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.162+0000 7f3ef84de640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ef01005c0 0x7f3ef0196850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:03.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.162+0000 7f3ef84de640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ef01017b0 0x7f3ef0196d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:03.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.162+0000 7f3ef84de640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ef0102170 0x7f3ef019de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:03.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.162+0000 7f3ef84de640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef010c210 con 0x7f3ef0102170 2024-03-24T00:21:03.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.162+0000 7f3ef84de640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef010c090 con 0x7f3ef01005c0 2024-03-24T00:21:03.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.162+0000 7f3ef84de640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef010c390 con 0x7f3ef01017b0 2024-03-24T00:21:03.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.162+0000 7f3ef627a640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ef01005c0 0x7f3ef0196850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:03.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.162+0000 7f3ef5a79640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ef01017b0 0x7f3ef0196d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:03.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.162+0000 7f3ef6a7b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ef0102170 0x7f3ef019de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:03.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.162+0000 7f3ef627a640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ef01005c0 0x7f3ef0196850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.78:36884/0 (socket says 172.21.15.78:36884) 2024-03-24T00:21:03.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.162+0000 7f3ef627a640 1 -- 172.21.15.78:0/2761563606 learned_addr learned my addr 172.21.15.78:0/2761563606 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:21:03.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.163+0000 7f3ef627a640 1 -- 172.21.15.78:0/2761563606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ef01017b0 msgr2=0x7f3ef0196d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:03.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.163+0000 7f3ef627a640 1 --2- 172.21.15.78:0/2761563606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ef01017b0 0x7f3ef0196d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:03.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.163+0000 7f3ef627a640 1 -- 172.21.15.78:0/2761563606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ef0102170 msgr2=0x7f3ef019de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:03.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.163+0000 7f3ef627a640 1 --2- 172.21.15.78:0/2761563606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ef0102170 0x7f3ef019de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:03.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.163+0000 7f3ef627a640 1 -- 172.21.15.78:0/2761563606 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ef019e470 con 0x7f3ef01005c0 2024-03-24T00:21:03.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.163+0000 7f3ef6a7b640 1 --2- 172.21.15.78:0/2761563606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ef0102170 0x7f3ef019de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:21:03.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.163+0000 7f3ef627a640 1 --2- 172.21.15.78:0/2761563606 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ef01005c0 0x7f3ef0196850 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0005ab0 tx=0x7f3ee0005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:03.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.163+0000 7f3ee77fe640 1 -- 172.21.15.78:0/2761563606 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee0005000 con 0x7f3ef01005c0 2024-03-24T00:21:03.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.163+0000 7f3ef84de640 1 -- 172.21.15.78:0/2761563606 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ef0068d20 con 0x7f3ef01005c0 2024-03-24T00:21:03.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.163+0000 7f3ee77fe640 1 -- 172.21.15.78:0/2761563606 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3ee00035f0 con 0x7f3ef01005c0 2024-03-24T00:21:03.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.163+0000 7f3ee77fe640 1 -- 172.21.15.78:0/2761563606 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee003b6a0 con 0x7f3ef01005c0 2024-03-24T00:21:03.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.163+0000 7f3ef84de640 1 -- 172.21.15.78:0/2761563606 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ef0069210 con 0x7f3ef01005c0 2024-03-24T00:21:03.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.164+0000 7f3ef84de640 1 -- 172.21.15.78:0/2761563606 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ef004f400 con 0x7f3ef01005c0 2024-03-24T00:21:03.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.166+0000 7f3ee77fe640 1 -- 172.21.15.78:0/2761563606 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 19) v1 ==== 137307+0+0 (secure 0 0 0) 0x7f3ee0003b30 con 0x7f3ef01005c0 2024-03-24T00:21:03.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.166+0000 7f3ee77fe640 1 --2- 172.21.15.78:0/2761563606 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3edc0a28f0 0x7f3edc0a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:03.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.166+0000 7f3ef5a79640 1 -- 172.21.15.78:0/2761563606 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3edc0a28f0 msgr2=0x7f3edc0a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1983101026 2024-03-24T00:21:03.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.166+0000 7f3ee77fe640 1 -- 172.21.15.78:0/2761563606 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f3ee00f2d10 con 0x7f3ef01005c0 2024-03-24T00:21:03.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.166+0000 7f3ef5a79640 1 --2- 172.21.15.78:0/2761563606 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3edc0a28f0 0x7f3edc0a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:21:03.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.169+0000 7f3ee77fe640 1 -- 172.21.15.78:0/2761563606 <== mon.2 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3ee00bf9c0 con 0x7f3ef01005c0 2024-03-24T00:21:03.340 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:02 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:02.990+0000 7fb04b642200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T00:21:03.340 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:03.094+0000 7fb04b642200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T00:21:03.343 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:21:02 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:21:02.987+0000 7f5d62ac8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T00:21:03.343 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:21:03 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:21:03.084+0000 7f5d62ac8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T00:21:03.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.367+0000 7f3ef5a79640 1 -- 172.21.15.78:0/2761563606 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3edc0a28f0 msgr2=0x7f3edc0a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1983101026 2024-03-24T00:21:03.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.367+0000 7f3ef5a79640 1 --2- 172.21.15.78:0/2761563606 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3edc0a28f0 0x7f3edc0a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:21:03.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.369+0000 7f3ef84de640 1 -- 172.21.15.78:0/2761563606 --> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ef0069470 con 0x7f3edc0a28f0 2024-03-24T00:21:03.402 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:21:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:21:02.998+0000 7f3077cd0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T00:21:03.402 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:21:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:21:03.098+0000 7f3077cd0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T00:21:03.734 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:03 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1475436602' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-24T00:21:03.734 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:03 smithi078 ceph-mon[32429]: mgrmap e19: x(active, since 9m), standbys: y, z 2024-03-24T00:21:03.734 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:03 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3372850153' entity='client.admin' 2024-03-24T00:21:03.734 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:03 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1586336724' entity='client.admin' 2024-03-24T00:21:03.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.768+0000 7f3ef5a79640 1 -- 172.21.15.78:0/2761563606 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3edc0a28f0 msgr2=0x7f3edc0a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1983101026 2024-03-24T00:21:03.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:03.768+0000 7f3ef5a79640 1 --2- 172.21.15.78:0/2761563606 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3edc0a28f0 0x7f3edc0a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:21:03.771 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:03 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1475436602' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-24T00:21:03.771 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:03 smithi106 ceph-mon[36505]: mgrmap e19: x(active, since 9m), standbys: y, z 2024-03-24T00:21:03.771 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:03 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3372850153' entity='client.admin' 2024-03-24T00:21:03.771 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:03 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1586336724' entity='client.admin' 2024-03-24T00:21:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:03 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1475436602' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-24T00:21:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:03 smithi107 ceph-mon[36485]: mgrmap e19: x(active, since 9m), standbys: y, z 2024-03-24T00:21:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:03 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3372850153' entity='client.admin' 2024-03-24T00:21:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:03 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1586336724' entity='client.admin' 2024-03-24T00:21:04.039 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:21:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:21:03.770+0000 7f3077cd0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T00:21:04.039 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:21:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:21:03.879+0000 7f3077cd0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T00:21:04.039 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:21:04 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:21:04.038+0000 7f3077cd0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T00:21:04.064 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:03.734+0000 7fb04b642200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T00:21:04.064 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:03.827+0000 7fb04b642200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T00:21:04.064 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:03.970+0000 7fb04b642200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T00:21:04.064 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:04.064+0000 7fb04b642200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T00:21:04.211 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:21:03 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:21:03.877+0000 7f5d62ac8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T00:21:04.211 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:21:03 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:21:03.974+0000 7f5d62ac8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T00:21:04.211 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:21:04 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:21:04.116+0000 7f5d62ac8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T00:21:04.211 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:21:04 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:21:04.211+0000 7f5d62ac8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T00:21:04.340 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:04.146+0000 7fb04b642200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T00:21:04.402 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:21:04 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:21:04.173+0000 7f3077cd0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T00:21:04.402 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:21:04 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:21:04.273+0000 7f3077cd0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T00:21:04.558 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:21:04 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:21:04.295+0000 7f5d62ac8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T00:21:04.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:04.569+0000 7f3ef5a79640 1 -- 172.21.15.78:0/2761563606 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3edc0a28f0 msgr2=0x7f3edc0a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1983101026 2024-03-24T00:21:04.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:04.569+0000 7f3ef5a79640 1 --2- 172.21.15.78:0/2761563606 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3edc0a28f0 0x7f3edc0a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:21:04.694 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:04.408+0000 7fb04b642200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T00:21:04.694 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:04.516+0000 7fb04b642200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T00:21:04.694 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:04.600+0000 7fb04b642200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T00:21:04.694 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:04.694+0000 7fb04b642200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T00:21:04.843 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:21:04 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:21:04.558+0000 7f5d62ac8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T00:21:04.843 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:21:04 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:21:04.668+0000 7f5d62ac8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T00:21:04.843 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:21:04 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:21:04.753+0000 7f5d62ac8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T00:21:04.872 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:21:04 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:21:04.571+0000 7f3077cd0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T00:21:04.872 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:21:04 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:21:04.689+0000 7f3077cd0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T00:21:04.872 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:21:04 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:21:04.775+0000 7f3077cd0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T00:21:05.090 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:04.867+0000 7fb04b642200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T00:21:05.152 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:21:04 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:21:04.871+0000 7f3077cd0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T00:21:05.152 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:21:05 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:21:05.048+0000 7f3077cd0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T00:21:05.343 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:21:04 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:21:04.848+0000 7f5d62ac8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T00:21:05.343 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:21:05 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:21:05.025+0000 7f5d62ac8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T00:21:06.090 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:05 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:05.813+0000 7fb04b642200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T00:21:06.090 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:05 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:05.941+0000 7fb04b642200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T00:21:06.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:06.170+0000 7f3ef5a79640 1 -- 172.21.15.78:0/2761563606 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3edc0a28f0 msgr2=0x7f3edc0a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1983101026 2024-03-24T00:21:06.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:06.170+0000 7f3ef5a79640 1 --2- 172.21.15.78:0/2761563606 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3edc0a28f0 0x7f3edc0a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T00:21:06.343 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:21:06 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:21:06.001+0000 7f5d62ac8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T00:21:06.343 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:21:06 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:21:06.133+0000 7f5d62ac8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T00:21:06.402 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:21:06 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:21:06.037+0000 7f3077cd0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T00:21:06.402 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:21:06 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:21:06.220+0000 7f3077cd0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T00:21:06.556 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:06.271+0000 7fb04b642200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T00:21:06.556 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:06.361+0000 7fb04b642200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T00:21:06.758 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:21:06 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:21:06.470+0000 7f5d62ac8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T00:21:06.758 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:21:06 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:21:06.560+0000 7f5d62ac8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T00:21:06.758 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:21:06 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:21:06.758+0000 7f5d62ac8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:21:06.840 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:06.556+0000 7fb04b642200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:21:06.840 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:06.740+0000 7fb04b642200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T00:21:06.899 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:21:06 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:21:06.594+0000 7f3077cd0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T00:21:06.899 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:21:06 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:21:06.699+0000 7f3077cd0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T00:21:07.093 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:21:06 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:21:06.945+0000 7f5d62ac8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T00:21:07.152 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:21:06 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:21:06.898+0000 7f3077cd0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:21:07.152 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:21:07 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:21:07.084+0000 7f3077cd0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T00:21:07.283 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:07 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:07.027+0000 7fb04b642200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:21:07.496 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:21:07 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:21:07.232+0000 7f5d62ac8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:21:07.590 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:07 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:07.283+0000 7fb04b642200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T00:21:07.590 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:07 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:07.519+0000 7fb04b642200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T00:21:07.691 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:21:07 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:21:07.403+0000 7f3077cd0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:21:07.760 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:21:07 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:21:07.496+0000 7f5d62ac8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T00:21:08.093 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:21:07 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:21:07.760+0000 7f5d62ac8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T00:21:08.152 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:21:07 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:21:07.690+0000 7f3077cd0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T00:21:08.152 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:21:07 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:21:07.936+0000 7f3077cd0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T00:21:08.590 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:08.322+0000 7fb04b642200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T00:21:09.093 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:21:08 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:21:08.647+0000 7f5d62ac8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T00:21:09.152 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:21:08 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:21:08.829+0000 7f3077cd0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T00:21:09.205 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:08.887+0000 7fb04b642200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T00:21:09.205 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:08.987+0000 7fb04b642200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T00:21:09.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:09.232+0000 7f3ee77fe640 1 -- 172.21.15.78:0/2761563606 <== mon.2 v2:172.21.15.106:3300/0 7 ==== mgrmap(e 20) v1 ==== 136975+0+0 (secure 0 0 0) 0x7f3ee00bb930 con 0x7f3ef01005c0 2024-03-24T00:21:09.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:09.233+0000 7f3ee77fe640 1 -- 172.21.15.78:0/2761563606 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3edc0a28f0 msgr2=0x7f3edc0a4db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T00:21:09.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:09.233+0000 7f3ee77fe640 1 --2- 172.21.15.78:0/2761563606 >> [v2:172.21.15.78:6800/1983101026,v1:172.21.15.78:6801/1983101026] conn(0x7f3edc0a28f0 0x7f3edc0a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:09.459 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:09.205+0000 7fb04b642200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T00:21:09.460 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: Active manager daemon x restarted 2024-03-24T00:21:09.460 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: Activating manager daemon x 2024-03-24T00:21:09.460 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: osdmap e94: 12 total, 12 up, 12 in 2024-03-24T00:21:09.460 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: mgrmap e20: x(active, starting, since 0.0110272s), standbys: y, z 2024-03-24T00:21:09.460 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:21:09.460 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:21:09.460 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:21:09.460 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:21:09.460 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:21:09.460 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:21:09.460 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T00:21:09.461 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:21:09.461 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T00:21:09.461 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T00:21:09.461 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:21:09.461 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T00:21:09.461 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T00:21:09.461 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T00:21:09.461 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T00:21:09.461 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T00:21:09.461 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T00:21:09.461 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T00:21:09.461 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:21:09.461 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:21:09.461 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:21:09.461 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:21:09.462 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:21:09.462 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:21:09.462 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:21:09.462 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:21:09.462 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:21:09.462 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:21:09.462 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:21:09.462 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:21:09.462 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T00:21:09.462 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T00:21:09.462 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T00:21:09.462 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-mon[32429]: Manager daemon x is now available 2024-03-24T00:21:09.570 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:21:09.224+0000 7f5d62ac8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T00:21:09.570 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:21:09.328+0000 7f5d62ac8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T00:21:09.571 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: Active manager daemon x restarted 2024-03-24T00:21:09.571 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: Activating manager daemon x 2024-03-24T00:21:09.571 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: osdmap e94: 12 total, 12 up, 12 in 2024-03-24T00:21:09.571 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: mgrmap e20: x(active, starting, since 0.0110272s), standbys: y, z 2024-03-24T00:21:09.571 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:21:09.572 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:21:09.572 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:21:09.572 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:21:09.572 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:21:09.572 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:21:09.572 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T00:21:09.572 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:21:09.572 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T00:21:09.572 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T00:21:09.572 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:21:09.572 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T00:21:09.572 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T00:21:09.572 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T00:21:09.572 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T00:21:09.573 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T00:21:09.573 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T00:21:09.573 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T00:21:09.573 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:21:09.573 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:21:09.573 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:21:09.573 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:21:09.573 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:21:09.573 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:21:09.573 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:21:09.573 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:21:09.573 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:21:09.574 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:21:09.574 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:21:09.574 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:21:09.574 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T00:21:09.574 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T00:21:09.574 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T00:21:09.574 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-mon[36485]: Manager daemon x is now available 2024-03-24T00:21:09.575 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: Active manager daemon x restarted 2024-03-24T00:21:09.575 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: Activating manager daemon x 2024-03-24T00:21:09.575 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: osdmap e94: 12 total, 12 up, 12 in 2024-03-24T00:21:09.575 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: mgrmap e20: x(active, starting, since 0.0110272s), standbys: y, z 2024-03-24T00:21:09.575 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:21:09.575 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:21:09.575 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:21:09.575 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:21:09.575 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:21:09.575 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:21:09.576 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T00:21:09.576 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:21:09.576 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T00:21:09.576 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T00:21:09.576 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:21:09.576 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T00:21:09.576 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T00:21:09.576 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T00:21:09.576 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T00:21:09.576 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T00:21:09.576 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T00:21:09.576 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T00:21:09.576 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:21:09.576 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:21:09.577 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:21:09.577 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:21:09.577 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:21:09.577 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:21:09.577 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:21:09.577 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:21:09.577 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:21:09.577 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:21:09.577 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:21:09.577 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:21:09.577 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T00:21:09.577 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T00:21:09.577 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T00:21:09.577 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-mon[36505]: Manager daemon x is now available 2024-03-24T00:21:09.841 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:09.471+0000 7fafae097700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:21:09.841 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:09.471+0000 7fafae097700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:21:09.841 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:09.471+0000 7fafae097700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:21:09.841 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:09.471+0000 7fafae097700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:21:09.841 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:09.471+0000 7fafae097700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:21:09.841 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:09.471+0000 7fafab091700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:21:09.841 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:09.471+0000 7fafab091700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:21:09.841 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:09.471+0000 7fafab091700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:21:09.841 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:09.471+0000 7fafab091700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:21:09.841 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:21:09 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:21:09.471+0000 7fafab091700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:21:09.843 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 00:21:09 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z[38351]: 2024-03-24T00:21:09.570+0000 7f5d62ac8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T00:21:09.902 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:21:09.572+0000 7f3077cd0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T00:21:09.902 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:21:09.688+0000 7f3077cd0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T00:21:10.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.239+0000 7f3ee77fe640 1 -- 172.21.15.78:0/2761563606 <== mon.2 v2:172.21.15.106:3300/0 8 ==== mgrmap(e 21) v1 ==== 137102+0+0 (secure 0 0 0) 0x7f3ee00f04a0 con 0x7f3ef01005c0 2024-03-24T00:21:10.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.239+0000 7f3ee77fe640 1 --2- 172.21.15.78:0/2761563606 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f3edc0511d0 0x7f3edc0515b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:10.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.239+0000 7f3ee77fe640 1 -- 172.21.15.78:0/2761563606 --> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ee00c3200 con 0x7f3edc0511d0 2024-03-24T00:21:10.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.240+0000 7f3ef5a79640 1 --2- 172.21.15.78:0/2761563606 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f3edc0511d0 0x7f3edc0515b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:10.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.242+0000 7f3ef5a79640 1 --2- 172.21.15.78:0/2761563606 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f3edc0511d0 0x7f3edc0515b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3ef0197d60 tx=0x7f3eec00b390 comp rx=0 tx=0).ready entity=mgr.25243 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:10.274 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 00:21:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y[37892]: 2024-03-24T00:21:09.933+0000 7f3077cd0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T00:21:10.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.303+0000 7f3ee77fe640 1 -- 172.21.15.78:0/2761563606 <== mgr.25243 v2:172.21.15.78:6800/3337965055 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f3ee00c3200 con 0x7f3edc0511d0 2024-03-24T00:21:10.303 INFO:teuthology.orchestra.run.smithi078.stdout:Schedule set for path / 2024-03-24T00:21:10.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.305+0000 7f3ef84de640 1 -- 172.21.15.78:0/2761563606 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f3edc0511d0 msgr2=0x7f3edc0515b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:10.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.305+0000 7f3ef84de640 1 --2- 172.21.15.78:0/2761563606 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f3edc0511d0 0x7f3edc0515b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3ef0197d60 tx=0x7f3eec00b390 comp rx=0 tx=0).stop 2024-03-24T00:21:10.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.305+0000 7f3ef84de640 1 -- 172.21.15.78:0/2761563606 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ef01005c0 msgr2=0x7f3ef0196850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:10.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.305+0000 7f3ef84de640 1 --2- 172.21.15.78:0/2761563606 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ef01005c0 0x7f3ef0196850 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0005ab0 tx=0x7f3ee0005dc0 comp rx=0 tx=0).stop 2024-03-24T00:21:10.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.305+0000 7f3ef6a7b640 1 -- 172.21.15.78:0/2761563606 reap_dead start 2024-03-24T00:21:10.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.305+0000 7f3ef84de640 1 -- 172.21.15.78:0/2761563606 shutdown_connections 2024-03-24T00:21:10.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.305+0000 7f3ef84de640 1 -- 172.21.15.78:0/2761563606 >> 172.21.15.78:0/2761563606 conn(0x7f3ef00fbb30 msgr2=0x7f3ef0104ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:10.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.305+0000 7f3ef84de640 1 -- 172.21.15.78:0/2761563606 shutdown_connections 2024-03-24T00:21:10.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.305+0000 7f3ef84de640 1 -- 172.21.15.78:0/2761563606 wait complete. 2024-03-24T00:21:10.316 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-03-24T00:21:10.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.423+0000 7fa738e26640 1 -- 172.21.15.78:0/919228377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa734102170 msgr2=0x7fa734108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:10.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.423+0000 7fa738e26640 1 --2- 172.21.15.78:0/919228377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa734102170 0x7fa734108a00 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fa72c009ca0 tx=0x7fa72c02c0f0 comp rx=0 tx=0).stop 2024-03-24T00:21:10.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.423+0000 7fa738e26640 1 -- 172.21.15.78:0/919228377 shutdown_connections 2024-03-24T00:21:10.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.423+0000 7fa738e26640 1 --2- 172.21.15.78:0/919228377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa734102170 0x7fa734108a00 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:10.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.423+0000 7fa738e26640 1 --2- 172.21.15.78:0/919228377 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7341017b0 0x7fa734101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:10.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.423+0000 7fa738e26640 1 --2- 172.21.15.78:0/919228377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7341005c0 0x7fa7341009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:10.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.423+0000 7fa738e26640 1 -- 172.21.15.78:0/919228377 >> 172.21.15.78:0/919228377 conn(0x7fa7340fbb30 msgr2=0x7fa7340fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:10.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.423+0000 7fa738e26640 1 -- 172.21.15.78:0/919228377 shutdown_connections 2024-03-24T00:21:10.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.423+0000 7fa738e26640 1 -- 172.21.15.78:0/919228377 wait complete. 2024-03-24T00:21:10.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.424+0000 7fa738e26640 1 Processor -- start 2024-03-24T00:21:10.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.424+0000 7fa738e26640 1 -- start start 2024-03-24T00:21:10.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.424+0000 7fa738e26640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7341005c0 0x7fa734196800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:10.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.424+0000 7fa738e26640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7341017b0 0x7fa734196d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:10.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.424+0000 7fa738e26640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa734102170 0x7fa73419ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:10.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.424+0000 7fa738e26640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa73410c240 con 0x7fa7341017b0 2024-03-24T00:21:10.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.424+0000 7fa738e26640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa73410c0c0 con 0x7fa734102170 2024-03-24T00:21:10.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.424+0000 7fa738e26640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa73410c3c0 con 0x7fa7341005c0 2024-03-24T00:21:10.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.424+0000 7fa72bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7341017b0 0x7fa734196d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:10.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.424+0000 7fa72bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7341017b0 0x7fa734196d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48712/0 (socket says 172.21.15.78:48712) 2024-03-24T00:21:10.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.424+0000 7fa72bfff640 1 -- 172.21.15.78:0/897324436 learned_addr learned my addr 172.21.15.78:0/897324436 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:21:10.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.424+0000 7fa73259c640 1 --2- 172.21.15.78:0/897324436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7341005c0 0x7fa734196800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:10.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.424+0000 7fa732d9d640 1 --2- 172.21.15.78:0/897324436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa734102170 0x7fa73419ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:10.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.425+0000 7fa72bfff640 1 -- 172.21.15.78:0/897324436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7341005c0 msgr2=0x7fa734196800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:10.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.425+0000 7fa72bfff640 1 --2- 172.21.15.78:0/897324436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7341005c0 0x7fa734196800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:10.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.425+0000 7fa72bfff640 1 -- 172.21.15.78:0/897324436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa734102170 msgr2=0x7fa73419ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:10.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.425+0000 7fa72bfff640 1 --2- 172.21.15.78:0/897324436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa734102170 0x7fa73419ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:10.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.425+0000 7fa72bfff640 1 -- 172.21.15.78:0/897324436 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa73419e420 con 0x7fa7341017b0 2024-03-24T00:21:10.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.425+0000 7fa72bfff640 1 --2- 172.21.15.78:0/897324436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7341017b0 0x7fa734196d40 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fa71c00e9f0 tx=0x7fa71c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:10.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.425+0000 7fa72b7fe640 1 -- 172.21.15.78:0/897324436 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa71c00cd80 con 0x7fa7341017b0 2024-03-24T00:21:10.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.425+0000 7fa72b7fe640 1 -- 172.21.15.78:0/897324436 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa71c010910 con 0x7fa7341017b0 2024-03-24T00:21:10.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.425+0000 7fa738e26640 1 -- 172.21.15.78:0/897324436 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa73419e710 con 0x7fa7341017b0 2024-03-24T00:21:10.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.425+0000 7fa72b7fe640 1 -- 172.21.15.78:0/897324436 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa71c018990 con 0x7fa7341017b0 2024-03-24T00:21:10.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.425+0000 7fa738e26640 1 -- 172.21.15.78:0/897324436 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa73419ec60 con 0x7fa7341017b0 2024-03-24T00:21:10.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.426+0000 7fa738e26640 1 -- 172.21.15.78:0/897324436 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7341078e0 con 0x7fa7341017b0 2024-03-24T00:21:10.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.427+0000 7fa72b7fe640 1 -- 172.21.15.78:0/897324436 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 21) v1 ==== 137102+0+0 (secure 0 0 0) 0x7fa71c010ac0 con 0x7fa7341017b0 2024-03-24T00:21:10.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.428+0000 7fa72b7fe640 1 --2- 172.21.15.78:0/897324436 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fa7200a26b0 0x7fa7200a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:10.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.428+0000 7fa73259c640 1 --2- 172.21.15.78:0/897324436 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fa7200a26b0 0x7fa7200a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:10.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.428+0000 7fa72b7fe640 1 -- 172.21.15.78:0/897324436 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fa71c0cee40 con 0x7fa7341017b0 2024-03-24T00:21:10.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.428+0000 7fa73259c640 1 --2- 172.21.15.78:0/897324436 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fa7200a26b0 0x7fa7200a4b70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa734101610 tx=0x7fa724005f90 comp rx=0 tx=0).ready entity=mgr.25243 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:10.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.432+0000 7fa72b7fe640 1 -- 172.21.15.78:0/897324436 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa71c09b6f0 con 0x7fa7341017b0 2024-03-24T00:21:10.567 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:21:10.568 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:21:10.568 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T00:21:10.568 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T00:21:10.568 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:10 smithi106 ceph-mon[36505]: Standby manager daemon z restarted 2024-03-24T00:21:10.568 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:10 smithi106 ceph-mon[36505]: Standby manager daemon z started 2024-03-24T00:21:10.568 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:10 smithi106 ceph-mon[36505]: from='mgr.? 172.21.15.107:0/350880485' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T00:21:10.568 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:10 smithi106 ceph-mon[36505]: from='mgr.? 172.21.15.107:0/350880485' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T00:21:10.568 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:10 smithi106 ceph-mon[36505]: from='mgr.? 172.21.15.107:0/350880485' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T00:21:10.568 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:10 smithi106 ceph-mon[36505]: from='mgr.? 172.21.15.107:0/350880485' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T00:21:10.568 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:10 smithi106 ceph-mon[36505]: Standby manager daemon y restarted 2024-03-24T00:21:10.568 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:10 smithi106 ceph-mon[36505]: Standby manager daemon y started 2024-03-24T00:21:10.568 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:10 smithi106 ceph-mon[36505]: from='mgr.? 172.21.15.106:0/1114738150' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T00:21:10.568 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:10 smithi106 ceph-mon[36505]: from='mgr.? 172.21.15.106:0/1114738150' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T00:21:10.569 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:10 smithi106 ceph-mon[36505]: from='mgr.? 172.21.15.106:0/1114738150' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T00:21:10.569 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:10 smithi106 ceph-mon[36505]: from='mgr.? 172.21.15.106:0/1114738150' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T00:21:10.569 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:10 smithi106 ceph-mon[36505]: mgrmap e21: x(active, since 1.01801s), standbys: z, y 2024-03-24T00:21:10.570 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:21:10.570 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:21:10.570 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T00:21:10.570 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T00:21:10.570 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:10 smithi078 ceph-mon[32429]: Standby manager daemon z restarted 2024-03-24T00:21:10.570 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:10 smithi078 ceph-mon[32429]: Standby manager daemon z started 2024-03-24T00:21:10.570 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:10 smithi078 ceph-mon[32429]: from='mgr.? 172.21.15.107:0/350880485' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T00:21:10.570 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:10 smithi078 ceph-mon[32429]: from='mgr.? 172.21.15.107:0/350880485' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T00:21:10.570 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:10 smithi078 ceph-mon[32429]: from='mgr.? 172.21.15.107:0/350880485' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T00:21:10.570 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:10 smithi078 ceph-mon[32429]: from='mgr.? 172.21.15.107:0/350880485' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T00:21:10.570 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:10 smithi078 ceph-mon[32429]: Standby manager daemon y restarted 2024-03-24T00:21:10.570 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:10 smithi078 ceph-mon[32429]: Standby manager daemon y started 2024-03-24T00:21:10.571 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:10 smithi078 ceph-mon[32429]: from='mgr.? 172.21.15.106:0/1114738150' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T00:21:10.571 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:10 smithi078 ceph-mon[32429]: from='mgr.? 172.21.15.106:0/1114738150' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T00:21:10.571 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:10 smithi078 ceph-mon[32429]: from='mgr.? 172.21.15.106:0/1114738150' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T00:21:10.571 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:10 smithi078 ceph-mon[32429]: from='mgr.? 172.21.15.106:0/1114738150' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T00:21:10.571 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:10 smithi078 ceph-mon[32429]: mgrmap e21: x(active, since 1.01801s), standbys: z, y 2024-03-24T00:21:10.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:21:10.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:21:10.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T00:21:10.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T00:21:10.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:10 smithi107 ceph-mon[36485]: Standby manager daemon z restarted 2024-03-24T00:21:10.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:10 smithi107 ceph-mon[36485]: Standby manager daemon z started 2024-03-24T00:21:10.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:10 smithi107 ceph-mon[36485]: from='mgr.? 172.21.15.107:0/350880485' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T00:21:10.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:10 smithi107 ceph-mon[36485]: from='mgr.? 172.21.15.107:0/350880485' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T00:21:10.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:10 smithi107 ceph-mon[36485]: from='mgr.? 172.21.15.107:0/350880485' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T00:21:10.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:10 smithi107 ceph-mon[36485]: from='mgr.? 172.21.15.107:0/350880485' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T00:21:10.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:10 smithi107 ceph-mon[36485]: Standby manager daemon y restarted 2024-03-24T00:21:10.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:10 smithi107 ceph-mon[36485]: Standby manager daemon y started 2024-03-24T00:21:10.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:10 smithi107 ceph-mon[36485]: from='mgr.? 172.21.15.106:0/1114738150' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T00:21:10.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:10 smithi107 ceph-mon[36485]: from='mgr.? 172.21.15.106:0/1114738150' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T00:21:10.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:10 smithi107 ceph-mon[36485]: from='mgr.? 172.21.15.106:0/1114738150' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T00:21:10.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:10 smithi107 ceph-mon[36485]: from='mgr.? 172.21.15.106:0/1114738150' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T00:21:10.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:10 smithi107 ceph-mon[36485]: mgrmap e21: x(active, since 1.01801s), standbys: z, y 2024-03-24T00:21:10.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.659+0000 7fa738e26640 1 -- 172.21.15.78:0/897324436 --> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) v1 -- 0x7fa734061c40 con 0x7fa7200a26b0 2024-03-24T00:21:10.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.711+0000 7fa72b7fe640 1 -- 172.21.15.78:0/897324436 <== mgr.25243 v2:172.21.15.78:6800/3337965055 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fa734061c40 con 0x7fa7200a26b0 2024-03-24T00:21:10.711 INFO:teuthology.orchestra.run.smithi078.stdout:Retention added to path / 2024-03-24T00:21:10.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.713+0000 7fa738e26640 1 -- 172.21.15.78:0/897324436 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fa7200a26b0 msgr2=0x7fa7200a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.713+0000 7fa738e26640 1 --2- 172.21.15.78:0/897324436 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fa7200a26b0 0x7fa7200a4b70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa734101610 tx=0x7fa724005f90 comp rx=0 tx=0).stop 2024-03-24T00:21:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.713+0000 7fa738e26640 1 -- 172.21.15.78:0/897324436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7341017b0 msgr2=0x7fa734196d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.713+0000 7fa738e26640 1 --2- 172.21.15.78:0/897324436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7341017b0 0x7fa734196d40 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fa71c00e9f0 tx=0x7fa71c00eeb0 comp rx=0 tx=0).stop 2024-03-24T00:21:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.713+0000 7fa738e26640 1 -- 172.21.15.78:0/897324436 shutdown_connections 2024-03-24T00:21:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.713+0000 7fa738e26640 1 --2- 172.21.15.78:0/897324436 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fa7200a26b0 0x7fa7200a4b70 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.713+0000 7fa738e26640 1 --2- 172.21.15.78:0/897324436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa734102170 0x7fa73419ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.713+0000 7fa738e26640 1 --2- 172.21.15.78:0/897324436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7341017b0 0x7fa734196d40 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.713+0000 7fa738e26640 1 --2- 172.21.15.78:0/897324436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7341005c0 0x7fa734196800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.713+0000 7fa738e26640 1 -- 172.21.15.78:0/897324436 >> 172.21.15.78:0/897324436 conn(0x7fa7340fbb30 msgr2=0x7fa734104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.713+0000 7fa738e26640 1 -- 172.21.15.78:0/897324436 shutdown_connections 2024-03-24T00:21:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.714+0000 7fa738e26640 1 -- 172.21.15.78:0/897324436 wait complete. 2024-03-24T00:21:10.725 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-24T00:21:10.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.831+0000 7f0629d1b640 1 -- 172.21.15.78:0/1523634529 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0624100600 msgr2=0x7f0624100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:10.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.831+0000 7f0629d1b640 1 --2- 172.21.15.78:0/1523634529 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0624100600 0x7f0624100a20 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f0614009900 tx=0x7f061402bc30 comp rx=0 tx=0).stop 2024-03-24T00:21:10.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.831+0000 7f0629d1b640 1 -- 172.21.15.78:0/1523634529 shutdown_connections 2024-03-24T00:21:10.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.831+0000 7f0629d1b640 1 --2- 172.21.15.78:0/1523634529 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06241021b0 0x7f0624108a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:10.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.831+0000 7f0629d1b640 1 --2- 172.21.15.78:0/1523634529 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f06241017f0 0x7f0624101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:10.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.831+0000 7f0629d1b640 1 --2- 172.21.15.78:0/1523634529 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0624100600 0x7f0624100a20 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:10.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.831+0000 7f0629d1b640 1 -- 172.21.15.78:0/1523634529 >> 172.21.15.78:0/1523634529 conn(0x7f06240fbb70 msgr2=0x7f06240fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:10.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.831+0000 7f0629d1b640 1 -- 172.21.15.78:0/1523634529 shutdown_connections 2024-03-24T00:21:10.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.831+0000 7f0629d1b640 1 -- 172.21.15.78:0/1523634529 wait complete. 2024-03-24T00:21:10.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.831+0000 7f0629d1b640 1 Processor -- start 2024-03-24T00:21:10.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.832+0000 7f0629d1b640 1 -- start start 2024-03-24T00:21:10.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.832+0000 7f0629d1b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0624100600 0x7f062406aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:10.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.832+0000 7f0629d1b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06241017f0 0x7f062419c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:10.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.832+0000 7f0629d1b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06241021b0 0x7f062419ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:10.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.832+0000 7f0629d1b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f062410c270 con 0x7f06241017f0 2024-03-24T00:21:10.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.832+0000 7f0629d1b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f062410c0f0 con 0x7f0624100600 2024-03-24T00:21:10.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.832+0000 7f0629d1b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f062410c3f0 con 0x7f06241021b0 2024-03-24T00:21:10.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.832+0000 7f0622ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06241017f0 0x7f062419c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:10.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.832+0000 7f0622ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06241017f0 0x7f062419c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48742/0 (socket says 172.21.15.78:48742) 2024-03-24T00:21:10.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.832+0000 7f0622ffd640 1 -- 172.21.15.78:0/1891641379 learned_addr learned my addr 172.21.15.78:0/1891641379 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:21:10.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.832+0000 7f0623fff640 1 --2- 172.21.15.78:0/1891641379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06241021b0 0x7f062419ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:10.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.832+0000 7f06237fe640 1 --2- 172.21.15.78:0/1891641379 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0624100600 0x7f062406aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:10.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.832+0000 7f0622ffd640 1 -- 172.21.15.78:0/1891641379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06241021b0 msgr2=0x7f062419ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:10.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.832+0000 7f0622ffd640 1 --2- 172.21.15.78:0/1891641379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06241021b0 0x7f062419ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:10.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.832+0000 7f0622ffd640 1 -- 172.21.15.78:0/1891641379 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0624100600 msgr2=0x7f062406aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:10.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.832+0000 7f0622ffd640 1 --2- 172.21.15.78:0/1891641379 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0624100600 0x7f062406aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:10.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.832+0000 7f0622ffd640 1 -- 172.21.15.78:0/1891641379 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f062419f590 con 0x7f06241017f0 2024-03-24T00:21:10.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.832+0000 7f0622ffd640 1 --2- 172.21.15.78:0/1891641379 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06241017f0 0x7f062419c690 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f061800e9f0 tx=0x7f061800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:10.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.833+0000 7f0620ff9640 1 -- 172.21.15.78:0/1891641379 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061800cd80 con 0x7f06241017f0 2024-03-24T00:21:10.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.833+0000 7f0620ff9640 1 -- 172.21.15.78:0/1891641379 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0618010910 con 0x7f06241017f0 2024-03-24T00:21:10.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.833+0000 7f0629d1b640 1 -- 172.21.15.78:0/1891641379 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f062419f880 con 0x7f06241017f0 2024-03-24T00:21:10.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.833+0000 7f0620ff9640 1 -- 172.21.15.78:0/1891641379 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0618018990 con 0x7f06241017f0 2024-03-24T00:21:10.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.833+0000 7f0629d1b640 1 -- 172.21.15.78:0/1891641379 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f062419fdd0 con 0x7f06241017f0 2024-03-24T00:21:10.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.834+0000 7f0629d1b640 1 -- 172.21.15.78:0/1891641379 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0624107920 con 0x7f06241017f0 2024-03-24T00:21:10.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.835+0000 7f0620ff9640 1 -- 172.21.15.78:0/1891641379 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 21) v1 ==== 137102+0+0 (secure 0 0 0) 0x7f0618010ac0 con 0x7f06241017f0 2024-03-24T00:21:10.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.835+0000 7f0620ff9640 1 --2- 172.21.15.78:0/1891641379 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f060c0a26b0 0x7f060c0a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:10.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.835+0000 7f06237fe640 1 --2- 172.21.15.78:0/1891641379 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f060c0a26b0 0x7f060c0a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:10.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.835+0000 7f0620ff9640 1 -- 172.21.15.78:0/1891641379 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f06180cee50 con 0x7f06241017f0 2024-03-24T00:21:10.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.836+0000 7f06237fe640 1 --2- 172.21.15.78:0/1891641379 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f060c0a26b0 0x7f060c0a4b70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0614000c00 tx=0x7f0614005fd0 comp rx=0 tx=0).ready entity=mgr.25243 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:10.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:10.839+0000 7f0620ff9640 1 -- 172.21.15.78:0/1891641379 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f061809b700 con 0x7f06241017f0 2024-03-24T00:21:11.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.064+0000 7f0629d1b640 1 -- 172.21.15.78:0/1891641379 --> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f0624061ae0 con 0x7f060c0a26b0 2024-03-24T00:21:11.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.072+0000 7f0620ff9640 1 -- 172.21.15.78:0/1891641379 <== mgr.25243 v2:172.21.15.78:6800/3337965055 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+298 (secure 0 0 0) 0x7f0624061ae0 con 0x7f060c0a26b0 2024-03-24T00:21:11.072 INFO:teuthology.orchestra.run.smithi078.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-24T00:00:00", "created": "2024-03-24T00:21:10", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-03-24T00:21:11.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.074+0000 7f0629d1b640 1 -- 172.21.15.78:0/1891641379 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f060c0a26b0 msgr2=0x7f060c0a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:11.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.074+0000 7f0629d1b640 1 --2- 172.21.15.78:0/1891641379 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f060c0a26b0 0x7f060c0a4b70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0614000c00 tx=0x7f0614005fd0 comp rx=0 tx=0).stop 2024-03-24T00:21:11.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.074+0000 7f0629d1b640 1 -- 172.21.15.78:0/1891641379 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06241017f0 msgr2=0x7f062419c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:11.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.074+0000 7f0629d1b640 1 --2- 172.21.15.78:0/1891641379 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06241017f0 0x7f062419c690 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f061800e9f0 tx=0x7f061800eeb0 comp rx=0 tx=0).stop 2024-03-24T00:21:11.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.074+0000 7f0629d1b640 1 -- 172.21.15.78:0/1891641379 shutdown_connections 2024-03-24T00:21:11.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.074+0000 7f0629d1b640 1 --2- 172.21.15.78:0/1891641379 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f060c0a26b0 0x7f060c0a4b70 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:11.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.074+0000 7f0629d1b640 1 --2- 172.21.15.78:0/1891641379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06241021b0 0x7f062419ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:11.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.074+0000 7f0629d1b640 1 --2- 172.21.15.78:0/1891641379 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06241017f0 0x7f062419c690 secure :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f061800e9f0 tx=0x7f061800eeb0 comp rx=0 tx=0).stop 2024-03-24T00:21:11.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.074+0000 7f0629d1b640 1 --2- 172.21.15.78:0/1891641379 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0624100600 0x7f062406aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:11.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.074+0000 7f0629d1b640 1 -- 172.21.15.78:0/1891641379 >> 172.21.15.78:0/1891641379 conn(0x7f06240fbb70 msgr2=0x7f0624104af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:11.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.074+0000 7f0629d1b640 1 -- 172.21.15.78:0/1891641379 shutdown_connections 2024-03-24T00:21:11.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.074+0000 7f0629d1b640 1 -- 172.21.15.78:0/1891641379 wait complete. 2024-03-24T00:21:11.086 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-24T00:21:11.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.197+0000 7fcb3aa27640 1 -- 172.21.15.78:0/1134178795 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb34102170 msgr2=0x7fcb34108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:11.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.197+0000 7fcb3aa27640 1 --2- 172.21.15.78:0/1134178795 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb34102170 0x7fcb34108a00 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c009ca0 tx=0x7fcb2c02c0f0 comp rx=0 tx=0).stop 2024-03-24T00:21:11.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.197+0000 7fcb3aa27640 1 -- 172.21.15.78:0/1134178795 shutdown_connections 2024-03-24T00:21:11.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.197+0000 7fcb3aa27640 1 --2- 172.21.15.78:0/1134178795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb341017b0 0x7fcb34101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:11.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.197+0000 7fcb3aa27640 1 --2- 172.21.15.78:0/1134178795 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb34102170 0x7fcb34108a00 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:11.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.197+0000 7fcb3aa27640 1 --2- 172.21.15.78:0/1134178795 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb341005c0 0x7fcb341009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:11.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.197+0000 7fcb3aa27640 1 -- 172.21.15.78:0/1134178795 >> 172.21.15.78:0/1134178795 conn(0x7fcb340fbb30 msgr2=0x7fcb340fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:11.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.197+0000 7fcb3aa27640 1 -- 172.21.15.78:0/1134178795 shutdown_connections 2024-03-24T00:21:11.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.197+0000 7fcb3aa27640 1 -- 172.21.15.78:0/1134178795 wait complete. 2024-03-24T00:21:11.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.198+0000 7fcb3aa27640 1 Processor -- start 2024-03-24T00:21:11.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.198+0000 7fcb3aa27640 1 -- start start 2024-03-24T00:21:11.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.198+0000 7fcb3aa27640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb341005c0 0x7fcb34197910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:11.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.198+0000 7fcb3aa27640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb341017b0 0x7fcb34197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:11.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.198+0000 7fcb3aa27640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb34102170 0x7fcb3419eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:11.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.198+0000 7fcb3aa27640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3410c240 con 0x7fcb341005c0 2024-03-24T00:21:11.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.198+0000 7fcb3aa27640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3410c0c0 con 0x7fcb34102170 2024-03-24T00:21:11.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.198+0000 7fcb3aa27640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3410c3c0 con 0x7fcb341017b0 2024-03-24T00:21:11.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.198+0000 7fcb33fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb341005c0 0x7fcb34197910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:11.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.198+0000 7fcb33fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb341005c0 0x7fcb34197910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48762/0 (socket says 172.21.15.78:48762) 2024-03-24T00:21:11.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.198+0000 7fcb33fff640 1 -- 172.21.15.78:0/4047731124 learned_addr learned my addr 172.21.15.78:0/4047731124 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T00:21:11.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.198+0000 7fcb337fe640 1 --2- 172.21.15.78:0/4047731124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb341017b0 0x7fcb34197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:11.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.198+0000 7fcb38fc4640 1 --2- 172.21.15.78:0/4047731124 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb34102170 0x7fcb3419eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:11.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.198+0000 7fcb33fff640 1 -- 172.21.15.78:0/4047731124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb341017b0 msgr2=0x7fcb34197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:11.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.198+0000 7fcb33fff640 1 --2- 172.21.15.78:0/4047731124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb341017b0 0x7fcb34197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:11.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.198+0000 7fcb33fff640 1 -- 172.21.15.78:0/4047731124 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb34102170 msgr2=0x7fcb3419eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:11.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.198+0000 7fcb33fff640 1 --2- 172.21.15.78:0/4047731124 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb34102170 0x7fcb3419eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:11.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.198+0000 7fcb33fff640 1 -- 172.21.15.78:0/4047731124 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb3419f530 con 0x7fcb341005c0 2024-03-24T00:21:11.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.199+0000 7fcb33fff640 1 --2- 172.21.15.78:0/4047731124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb341005c0 0x7fcb34197910 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fcb2400e9c0 tx=0x7fcb2400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:11.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.199+0000 7fcb317fa640 1 -- 172.21.15.78:0/4047731124 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2400cd80 con 0x7fcb341005c0 2024-03-24T00:21:11.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.199+0000 7fcb317fa640 1 -- 172.21.15.78:0/4047731124 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcb240035f0 con 0x7fcb341005c0 2024-03-24T00:21:11.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.199+0000 7fcb317fa640 1 -- 172.21.15.78:0/4047731124 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb24010690 con 0x7fcb341005c0 2024-03-24T00:21:11.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.199+0000 7fcb3aa27640 1 -- 172.21.15.78:0/4047731124 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb3419f820 con 0x7fcb341005c0 2024-03-24T00:21:11.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.199+0000 7fcb3aa27640 1 -- 172.21.15.78:0/4047731124 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb3419fd20 con 0x7fcb341005c0 2024-03-24T00:21:11.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.201+0000 7fcb3aa27640 1 -- 172.21.15.78:0/4047731124 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb3419ffe0 con 0x7fcb341005c0 2024-03-24T00:21:11.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.202+0000 7fcb317fa640 1 -- 172.21.15.78:0/4047731124 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 21) v1 ==== 137102+0+0 (secure 0 0 0) 0x7fcb240037a0 con 0x7fcb341005c0 2024-03-24T00:21:11.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.202+0000 7fcb317fa640 1 --2- 172.21.15.78:0/4047731124 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fcb1c0a2800 0x7fcb1c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:11.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.202+0000 7fcb317fa640 1 -- 172.21.15.78:0/4047731124 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fcb240ceda0 con 0x7fcb341005c0 2024-03-24T00:21:11.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.202+0000 7fcb337fe640 1 --2- 172.21.15.78:0/4047731124 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fcb1c0a2800 0x7fcb1c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:11.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.203+0000 7fcb337fe640 1 --2- 172.21.15.78:0/4047731124 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fcb1c0a2800 0x7fcb1c0a4cc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcb34198e20 tx=0x7fcb28005f90 comp rx=0 tx=0).ready entity=mgr.25243 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:11.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.208+0000 7fcb317fa640 1 -- 172.21.15.78:0/4047731124 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcb2409bca0 con 0x7fcb341005c0 2024-03-24T00:21:11.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:11 smithi078 ceph-mon[32429]: [24/Mar/2024:00:21:10] ENGINE Bus STARTING 2024-03-24T00:21:11.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:11 smithi078 ceph-mon[32429]: [24/Mar/2024:00:21:10] ENGINE Serving on http://172.21.15.78:8765 2024-03-24T00:21:11.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:11 smithi078 ceph-mon[32429]: [24/Mar/2024:00:21:10] ENGINE Serving on https://172.21.15.78:7150 2024-03-24T00:21:11.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:11 smithi078 ceph-mon[32429]: [24/Mar/2024:00:21:10] ENGINE Bus STARTED 2024-03-24T00:21:11.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:11 smithi078 ceph-mon[32429]: from='client.25322 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:21:11.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:11 smithi078 ceph-mon[32429]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:21:11.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.494+0000 7fcb3aa27640 1 -- 172.21.15.78:0/4047731124 --> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fcb340fdc50 con 0x7fcb1c0a2800 2024-03-24T00:21:11.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.503+0000 7fcb317fa640 1 -- 172.21.15.78:0/4047731124 <== mgr.25243 v2:172.21.15.78:6800/3337965055 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7fcb340fdc50 con 0x7fcb1c0a2800 2024-03-24T00:21:11.503 INFO:teuthology.orchestra.run.smithi078.stdout:/ 1m 6m3h 2024-03-24T00:21:11.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.505+0000 7fcb3aa27640 1 -- 172.21.15.78:0/4047731124 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fcb1c0a2800 msgr2=0x7fcb1c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:11.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.505+0000 7fcb3aa27640 1 --2- 172.21.15.78:0/4047731124 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fcb1c0a2800 0x7fcb1c0a4cc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcb34198e20 tx=0x7fcb28005f90 comp rx=0 tx=0).stop 2024-03-24T00:21:11.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.505+0000 7fcb3aa27640 1 -- 172.21.15.78:0/4047731124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb341005c0 msgr2=0x7fcb34197910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:11.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.505+0000 7fcb3aa27640 1 --2- 172.21.15.78:0/4047731124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb341005c0 0x7fcb34197910 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fcb2400e9c0 tx=0x7fcb2400ee80 comp rx=0 tx=0).stop 2024-03-24T00:21:11.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.506+0000 7fcb3aa27640 1 -- 172.21.15.78:0/4047731124 shutdown_connections 2024-03-24T00:21:11.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.506+0000 7fcb3aa27640 1 --2- 172.21.15.78:0/4047731124 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fcb1c0a2800 0x7fcb1c0a4cc0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:11.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.506+0000 7fcb3aa27640 1 --2- 172.21.15.78:0/4047731124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb341017b0 0x7fcb34197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:11.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.506+0000 7fcb3aa27640 1 --2- 172.21.15.78:0/4047731124 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb34102170 0x7fcb3419eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:11.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.506+0000 7fcb3aa27640 1 --2- 172.21.15.78:0/4047731124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb341005c0 0x7fcb34197910 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:11.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.506+0000 7fcb3aa27640 1 -- 172.21.15.78:0/4047731124 >> 172.21.15.78:0/4047731124 conn(0x7fcb340fbb30 msgr2=0x7fcb340fd540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:11.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.506+0000 7fcb3aa27640 1 -- 172.21.15.78:0/4047731124 shutdown_connections 2024-03-24T00:21:11.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T00:21:11.506+0000 7fcb3aa27640 1 -- 172.21.15.78:0/4047731124 wait complete. 2024-03-24T00:21:11.518 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-03-24T00:21:11.547 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-03-24T00:21:11.563 INFO:teuthology.run_tasks:Running task workunit... 2024-03-24T00:21:11.582 INFO:tasks.workunit:Pulling workunits from ref 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:21:11.582 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-24T00:21:11.583 INFO:tasks.workunit:timeout=3h 2024-03-24T00:21:11.583 INFO:tasks.workunit:cleanup=True 2024-03-24T00:21:11.583 DEBUG:teuthology.orchestra.run.smithi078:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-24T00:21:11.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:11 smithi107 ceph-mon[36485]: [24/Mar/2024:00:21:10] ENGINE Bus STARTING 2024-03-24T00:21:11.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:11 smithi107 ceph-mon[36485]: [24/Mar/2024:00:21:10] ENGINE Serving on http://172.21.15.78:8765 2024-03-24T00:21:11.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:11 smithi107 ceph-mon[36485]: [24/Mar/2024:00:21:10] ENGINE Serving on https://172.21.15.78:7150 2024-03-24T00:21:11.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:11 smithi107 ceph-mon[36485]: [24/Mar/2024:00:21:10] ENGINE Bus STARTED 2024-03-24T00:21:11.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:11 smithi107 ceph-mon[36485]: from='client.25322 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:21:11.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:11 smithi107 ceph-mon[36485]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:21:11.602 INFO:teuthology.orchestra.run.smithi078.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-24T00:21:11.603 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-24T00:21:11.603 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-03-24T00:21:11.603 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:21:11.603 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-24T00:21:11.603 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-03-24 00:19:10.252351137 +0000 2024-03-24T00:21:11.603 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-03-24 00:19:10.252351137 +0000 2024-03-24T00:21:11.603 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-03-24 00:21:00.568792782 +0000 2024-03-24T00:21:11.603 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-03-24 00:19:10.252351137 +0000 2024-03-24T00:21:11.603 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-24T00:21:11.603 DEBUG:teuthology.orchestra.run.smithi078:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-24T00:21:11.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:11 smithi106 ceph-mon[36505]: [24/Mar/2024:00:21:10] ENGINE Bus STARTING 2024-03-24T00:21:11.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:11 smithi106 ceph-mon[36505]: [24/Mar/2024:00:21:10] ENGINE Serving on http://172.21.15.78:8765 2024-03-24T00:21:11.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:11 smithi106 ceph-mon[36505]: [24/Mar/2024:00:21:10] ENGINE Serving on https://172.21.15.78:7150 2024-03-24T00:21:11.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:11 smithi106 ceph-mon[36505]: [24/Mar/2024:00:21:10] ENGINE Bus STARTED 2024-03-24T00:21:11.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:11 smithi106 ceph-mon[36505]: from='client.25322 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:21:11.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:11 smithi106 ceph-mon[36505]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:21:11.677 DEBUG:teuthology.orchestra.run.smithi078:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:21:11.732 INFO:tasks.workunit.client.0.smithi078.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-24T00:21:12.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:12 smithi106 ceph-mon[36505]: from='client.15552 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:21:12.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:12 smithi106 ceph-mon[36505]: from='client.15558 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:21:12.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:12 smithi106 ceph-mon[36505]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:21:12.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:12 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:12.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:12 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:12.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:12 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:12.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:12 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:12.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:12 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:12.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:12 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:12.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:12 smithi106 ceph-mon[36505]: mgrmap e22: x(active, since 3s), standbys: z, y 2024-03-24T00:21:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:12 smithi078 ceph-mon[32429]: from='client.15552 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:21:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:12 smithi078 ceph-mon[32429]: from='client.15558 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:21:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:12 smithi078 ceph-mon[32429]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:21:12.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:12 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:12.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:12 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:12.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:12 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:12.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:12 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:12.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:12 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:12.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:12 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:12.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:12 smithi078 ceph-mon[32429]: mgrmap e22: x(active, since 3s), standbys: z, y 2024-03-24T00:21:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:12 smithi107 ceph-mon[36485]: from='client.15552 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:21:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:12 smithi107 ceph-mon[36485]: from='client.15558 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:21:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:12 smithi107 ceph-mon[36485]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:21:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:12 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:12 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:12 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:12 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:12 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:12 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:12 smithi107 ceph-mon[36485]: mgrmap e22: x(active, since 3s), standbys: z, y 2024-03-24T00:21:13.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:13 smithi106 ceph-mon[36505]: from='client.15564 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:21:13.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:13 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:13.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:13 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:13.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:13 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:13 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:13 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:13 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:13 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:13.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:13 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:13.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:13 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:13 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:13 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:13 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.655 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:13 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:13.655 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:13 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:13.655 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:13 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.655 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:13 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.655 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:13 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.655 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:13 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.656 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:13 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:21:13.656 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:13 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:21:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:13 smithi078 ceph-mon[32429]: from='client.15564 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:21:13.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:13 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:13.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:13 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:13.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:13 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:13 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:13 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:13 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:13 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:13.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:13 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:13.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:13 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:13 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:13 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:13 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:13 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:13.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:13 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:13.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:13 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:13 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:13 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.844 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:13 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.844 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:13 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:21:13.844 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:13 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:21:13.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:13 smithi107 ceph-mon[36485]: from='client.15564 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:21:13.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:13 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:13.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:13 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:13.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:13 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:13 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:13 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:13 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:13 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:13.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:13 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:13.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:13 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:13 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:13 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:13 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:13 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:13.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:13 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:13.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:13 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.848 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:13 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.848 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:13 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.848 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:13 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:13.848 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:13 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:21:13.848 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:13 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:21:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:14 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T00:21:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:14 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T00:21:14.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:14 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T00:21:14.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:14 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T00:21:14.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:14 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T00:21:14.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:14 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T00:21:14.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:14 smithi106 ceph-mon[36505]: Updating smithi078:/etc/ceph/ceph.conf 2024-03-24T00:21:14.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:14 smithi106 ceph-mon[36505]: Updating smithi106:/etc/ceph/ceph.conf 2024-03-24T00:21:14.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:14 smithi106 ceph-mon[36505]: Updating smithi107:/etc/ceph/ceph.conf 2024-03-24T00:21:14.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:14 smithi106 ceph-mon[36505]: Updating smithi078:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:21:14.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:14 smithi106 ceph-mon[36505]: Updating smithi107:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:21:14.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:14 smithi106 ceph-mon[36505]: Updating smithi106:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:21:14.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:14 smithi106 ceph-mon[36505]: Updating smithi078:/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:21:14.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:14 smithi106 ceph-mon[36505]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:21:14.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:14 smithi106 ceph-mon[36505]: Updating smithi106:/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:21:14.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:14 smithi106 ceph-mon[36505]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:21:14.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:14 smithi106 ceph-mon[36505]: mgrmap e23: x(active, since 4s), standbys: z, y 2024-03-24T00:21:14.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:14 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:14.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:14 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:14.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:14 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:14.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:14 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:14.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:14 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:14.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:14 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:14.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:14 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:14.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:14 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:21:14.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:14 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:21:14.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:14 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:21:14.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:14 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:14 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T00:21:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:14 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T00:21:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:14 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T00:21:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:14 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T00:21:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:14 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T00:21:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:14 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T00:21:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:14 smithi078 ceph-mon[32429]: Updating smithi078:/etc/ceph/ceph.conf 2024-03-24T00:21:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:14 smithi078 ceph-mon[32429]: Updating smithi106:/etc/ceph/ceph.conf 2024-03-24T00:21:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:14 smithi078 ceph-mon[32429]: Updating smithi107:/etc/ceph/ceph.conf 2024-03-24T00:21:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:14 smithi078 ceph-mon[32429]: Updating smithi078:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:21:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:14 smithi078 ceph-mon[32429]: Updating smithi107:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:21:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:14 smithi078 ceph-mon[32429]: Updating smithi106:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:21:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:14 smithi078 ceph-mon[32429]: Updating smithi078:/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:21:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:14 smithi078 ceph-mon[32429]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:21:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:14 smithi078 ceph-mon[32429]: Updating smithi106:/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:21:14.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:14 smithi078 ceph-mon[32429]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:21:14.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:14 smithi078 ceph-mon[32429]: mgrmap e23: x(active, since 4s), standbys: z, y 2024-03-24T00:21:14.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:14 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:14.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:14 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:14.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:14 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:14.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:14 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:14.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:14 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:14.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:14 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:14.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:14 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:14.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:14 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:21:14.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:14 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:21:14.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:14 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:21:14.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:14 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:14 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T00:21:14.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:14 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T00:21:14.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:14 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T00:21:14.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:14 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T00:21:14.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:14 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T00:21:14.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:14 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T00:21:14.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:14 smithi107 ceph-mon[36485]: Updating smithi078:/etc/ceph/ceph.conf 2024-03-24T00:21:14.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:14 smithi107 ceph-mon[36485]: Updating smithi106:/etc/ceph/ceph.conf 2024-03-24T00:21:14.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:14 smithi107 ceph-mon[36485]: Updating smithi107:/etc/ceph/ceph.conf 2024-03-24T00:21:14.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:14 smithi107 ceph-mon[36485]: Updating smithi078:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:21:14.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:14 smithi107 ceph-mon[36485]: Updating smithi107:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:21:14.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:14 smithi107 ceph-mon[36485]: Updating smithi106:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:21:14.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:14 smithi107 ceph-mon[36485]: Updating smithi078:/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:21:14.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:14 smithi107 ceph-mon[36485]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:21:14.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:14 smithi107 ceph-mon[36485]: Updating smithi106:/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:21:14.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:14 smithi107 ceph-mon[36485]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:21:14.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:14 smithi107 ceph-mon[36485]: mgrmap e23: x(active, since 4s), standbys: z, y 2024-03-24T00:21:14.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:14 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:14.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:14 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:14.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:14 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:14.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:14 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:14.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:14 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:14.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:14 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:14.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:14 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:14.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:14 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:21:14.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:14 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:21:14.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:14 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:21:14.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:14 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:21:15.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:15 smithi106 ceph-mon[36505]: Updating smithi078:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T00:21:15.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:15 smithi106 ceph-mon[36505]: Updating smithi107:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T00:21:15.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:15 smithi106 ceph-mon[36505]: Updating smithi106:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T00:21:15.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:15 smithi078 ceph-mon[32429]: Updating smithi078:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T00:21:15.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:15 smithi078 ceph-mon[32429]: Updating smithi107:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T00:21:15.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:15 smithi078 ceph-mon[32429]: Updating smithi106:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T00:21:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:15 smithi107 ceph-mon[36485]: Updating smithi078:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T00:21:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:15 smithi107 ceph-mon[36485]: Updating smithi107:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T00:21:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:15 smithi107 ceph-mon[36485]: Updating smithi106:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T00:21:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:16 smithi106 ceph-mon[36505]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:21:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:16 smithi078 ceph-mon[32429]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:21:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:16 smithi107 ceph-mon[36485]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:21:18.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:18 smithi106 ceph-mon[36505]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 132 KiB/s wr, 77 op/s 2024-03-24T00:21:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:18 smithi078 ceph-mon[32429]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 132 KiB/s wr, 77 op/s 2024-03-24T00:21:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:18 smithi107 ceph-mon[36485]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 132 KiB/s wr, 77 op/s 2024-03-24T00:21:20.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:20 smithi106 ceph-mon[36505]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 103 KiB/s wr, 60 op/s 2024-03-24T00:21:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:20 smithi078 ceph-mon[32429]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 103 KiB/s wr, 60 op/s 2024-03-24T00:21:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:20 smithi107 ceph-mon[36485]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 103 KiB/s wr, 60 op/s 2024-03-24T00:21:22.645 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:22 smithi107 ceph-mon[36485]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-03-24T00:21:22.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:22 smithi106 ceph-mon[36505]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-03-24T00:21:22.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:22 smithi078 ceph-mon[32429]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-03-24T00:21:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:24 smithi106 ceph-mon[36505]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-24T00:21:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:24 smithi078 ceph-mon[32429]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-24T00:21:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:24 smithi107 ceph-mon[36485]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-24T00:21:26.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:26 smithi106 ceph-mon[36505]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 49 op/s 2024-03-24T00:21:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:26 smithi078 ceph-mon[32429]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 49 op/s 2024-03-24T00:21:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:26 smithi107 ceph-mon[36485]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 49 op/s 2024-03-24T00:21:28.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:28 smithi106 ceph-mon[36505]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 49 op/s 2024-03-24T00:21:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:28 smithi078 ceph-mon[32429]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 49 op/s 2024-03-24T00:21:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:28 smithi107 ceph-mon[36485]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 49 op/s 2024-03-24T00:21:30.642 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:30 smithi078 ceph-mon[32429]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:21:30.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:30 smithi106 ceph-mon[36505]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:21:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:30 smithi107 ceph-mon[36485]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:21:32.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:32 smithi106 ceph-mon[36505]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:21:32.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:32 smithi078 ceph-mon[32429]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:21:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:32 smithi107 ceph-mon[36485]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:21:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:34 smithi106 ceph-mon[36505]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:21:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:34 smithi078 ceph-mon[32429]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:21:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:34 smithi107 ceph-mon[36485]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:21:36.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:36 smithi106 ceph-mon[36505]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:21:36.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:36 smithi078 ceph-mon[32429]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:21:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:36 smithi107 ceph-mon[36485]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:21:38.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:38 smithi106 ceph-mon[36505]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:21:38.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:38 smithi078 ceph-mon[32429]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:21:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:38 smithi107 ceph-mon[36485]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:21:40.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:40 smithi106 ceph-mon[36505]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:21:40.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:40 smithi078 ceph-mon[32429]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:21:40.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:40 smithi107 ceph-mon[36485]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:21:42.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:42 smithi106 ceph-mon[36505]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:21:42.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:42 smithi078 ceph-mon[32429]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:21:42.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:42 smithi107 ceph-mon[36485]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:21:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:44 smithi106 ceph-mon[36505]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:21:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:44 smithi078 ceph-mon[32429]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:21:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:44 smithi107 ceph-mon[36485]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:21:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:46 smithi106 ceph-mon[36505]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-24T00:21:46.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:46 smithi078 ceph-mon[32429]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-24T00:21:46.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:46 smithi107 ceph-mon[36485]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-24T00:21:48.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:48 smithi078 ceph-mon[32429]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 597 B/s wr, 5 op/s 2024-03-24T00:21:48.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:48 smithi107 ceph-mon[36485]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 597 B/s wr, 5 op/s 2024-03-24T00:21:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:48 smithi106 ceph-mon[36505]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 597 B/s wr, 5 op/s 2024-03-24T00:21:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:50 smithi078 ceph-mon[32429]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 597 B/s wr, 5 op/s 2024-03-24T00:21:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:50 smithi107 ceph-mon[36485]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 597 B/s wr, 5 op/s 2024-03-24T00:21:50.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:50 smithi106 ceph-mon[36505]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 597 B/s wr, 5 op/s 2024-03-24T00:21:52.703 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:52 smithi106 ceph-mon[36505]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-24T00:21:52.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:52 smithi078 ceph-mon[32429]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-24T00:21:52.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:52 smithi107 ceph-mon[36485]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-24T00:21:54.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:54 smithi078 ceph-mon[32429]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-24T00:21:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:54 smithi107 ceph-mon[36485]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-24T00:21:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:54 smithi106 ceph-mon[36505]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-24T00:21:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:56 smithi078 ceph-mon[32429]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-24T00:21:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:56 smithi107 ceph-mon[36485]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-24T00:21:56.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:56 smithi106 ceph-mon[36505]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-24T00:21:58.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:21:58 smithi078 ceph-mon[32429]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-24T00:21:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:21:58 smithi107 ceph-mon[36485]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-24T00:21:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:21:58 smithi106 ceph-mon[36505]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-24T00:22:00.722 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:00 smithi078 ceph-mon[32429]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-24T00:22:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:00 smithi107 ceph-mon[36485]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-24T00:22:00.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:00 smithi106 ceph-mon[36505]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-24T00:22:01.090 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:22:00 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:22:00.721+0000 7faf8bbd3700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:22:01.090 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:22:00 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:22:00.721+0000 7faf8bbd3700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:22:01.090 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:22:00 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:22:00.721+0000 7faf8bbd3700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:22:01.091 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:22:00 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:22:00.721+0000 7faf8bbd3700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:22:01.091 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 00:22:00 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x[32658]: 2024-03-24T00:22:00.721+0000 7faf8bbd3700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:22:02.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:02 smithi078 ceph-mon[32429]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-24T00:22:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:02 smithi107 ceph-mon[36485]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-24T00:22:02.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:02 smithi106 ceph-mon[36505]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-24T00:22:03.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:03 smithi078 ceph-mon[32429]: mgrmap e24: x(active, since 53s), standbys: z, y 2024-03-24T00:22:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:03 smithi107 ceph-mon[36485]: mgrmap e24: x(active, since 53s), standbys: z, y 2024-03-24T00:22:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:03 smithi106 ceph-mon[36505]: mgrmap e24: x(active, since 53s), standbys: z, y 2024-03-24T00:22:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:04 smithi078 ceph-mon[32429]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:22:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:04 smithi107 ceph-mon[36485]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:22:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:04 smithi106 ceph-mon[36505]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:22:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:06 smithi078 ceph-mon[32429]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-24T00:22:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:06 smithi107 ceph-mon[36485]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-24T00:22:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:06 smithi106 ceph-mon[36505]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-24T00:22:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:08 smithi078 ceph-mon[32429]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-24T00:22:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:08 smithi107 ceph-mon[36485]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-24T00:22:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:08 smithi106 ceph-mon[36505]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-24T00:22:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:10 smithi078 ceph-mon[32429]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-24T00:22:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:10 smithi107 ceph-mon[36485]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-24T00:22:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:10 smithi106 ceph-mon[36505]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-24T00:22:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:12 smithi078 ceph-mon[32429]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T00:22:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:12 smithi107 ceph-mon[36485]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T00:22:12.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:12 smithi106 ceph-mon[36505]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T00:22:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:14 smithi078 ceph-mon[32429]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-24T00:22:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:14 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:22:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:14 smithi107 ceph-mon[36485]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-24T00:22:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:14 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:22:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:14 smithi106 ceph-mon[36505]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-24T00:22:14.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:14 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:22:15.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:22:15.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:22:15.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:22:15.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:22:15.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:22:15.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:22:15.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:15 smithi078 ceph-mon[32429]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T00:22:15.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:22:15.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:22:15.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:22:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:22:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:22:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:22:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:22:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:22:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:22:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:15 smithi107 ceph-mon[36485]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T00:22:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:22:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:22:15.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:22:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:22:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:22:15.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:22:15.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:22:15.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:22:15.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:22:15.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:15 smithi106 ceph-mon[36505]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T00:22:15.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:22:15.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:22:15.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:22:18.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:18 smithi078 ceph-mon[32429]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-24T00:22:18.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:18 smithi107 ceph-mon[36485]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-24T00:22:18.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:18 smithi106 ceph-mon[36505]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-24T00:22:20.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:20 smithi078 ceph-mon[32429]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-24T00:22:20.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:20 smithi107 ceph-mon[36485]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-24T00:22:20.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:20 smithi106 ceph-mon[36505]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-24T00:22:22.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:22 smithi078 ceph-mon[32429]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-24T00:22:22.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:22 smithi107 ceph-mon[36485]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-24T00:22:22.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:22 smithi106 ceph-mon[36505]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-24T00:22:24.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:24 smithi078 ceph-mon[32429]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:22:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:24 smithi107 ceph-mon[36485]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:22:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:24 smithi106 ceph-mon[36505]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:22:26.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:26 smithi078 ceph-mon[32429]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:22:26.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:26 smithi107 ceph-mon[36485]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:22:26.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:26 smithi106 ceph-mon[36505]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:22:28.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:28 smithi078 ceph-mon[32429]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:22:28.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:28 smithi107 ceph-mon[36485]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:22:28.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:28 smithi106 ceph-mon[36505]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:22:30.453 INFO:tasks.workunit.client.0.smithi078.stderr:Note: switching to '3be9a5c9bc793e11e2800a8c0c696e8b46742033'. 2024-03-24T00:22:30.453 INFO:tasks.workunit.client.0.smithi078.stderr: 2024-03-24T00:22:30.453 INFO:tasks.workunit.client.0.smithi078.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-24T00:22:30.453 INFO:tasks.workunit.client.0.smithi078.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-24T00:22:30.453 INFO:tasks.workunit.client.0.smithi078.stderr:state without impacting any branches by switching back to a branch. 2024-03-24T00:22:30.454 INFO:tasks.workunit.client.0.smithi078.stderr: 2024-03-24T00:22:30.454 INFO:tasks.workunit.client.0.smithi078.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-24T00:22:30.454 INFO:tasks.workunit.client.0.smithi078.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-24T00:22:30.454 INFO:tasks.workunit.client.0.smithi078.stderr: 2024-03-24T00:22:30.454 INFO:tasks.workunit.client.0.smithi078.stderr: git switch -c 2024-03-24T00:22:30.454 INFO:tasks.workunit.client.0.smithi078.stderr: 2024-03-24T00:22:30.454 INFO:tasks.workunit.client.0.smithi078.stderr:Or undo this operation with: 2024-03-24T00:22:30.454 INFO:tasks.workunit.client.0.smithi078.stderr: 2024-03-24T00:22:30.454 INFO:tasks.workunit.client.0.smithi078.stderr: git switch - 2024-03-24T00:22:30.454 INFO:tasks.workunit.client.0.smithi078.stderr: 2024-03-24T00:22:30.454 INFO:tasks.workunit.client.0.smithi078.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-24T00:22:30.454 INFO:tasks.workunit.client.0.smithi078.stderr: 2024-03-24T00:22:30.454 INFO:tasks.workunit.client.0.smithi078.stderr:HEAD is now at 3be9a5c9bc7 dummy commit #1 2024-03-24T00:22:30.458 DEBUG:teuthology.orchestra.run.smithi078:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-24T00:22:30.519 INFO:tasks.workunit.client.0.smithi078.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-24T00:22:30.521 INFO:tasks.workunit.client.0.smithi078.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-24T00:22:30.521 INFO:tasks.workunit.client.0.smithi078.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-24T00:22:30.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:30 smithi078 ceph-mon[32429]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:22:30.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:30 smithi107 ceph-mon[36485]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:22:30.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:30 smithi106 ceph-mon[36505]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:22:31.256 INFO:tasks.workunit.client.0.smithi078.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-24T00:22:31.334 INFO:tasks.workunit.client.0.smithi078.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-24T00:22:31.373 INFO:tasks.workunit.client.0.smithi078.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-24T00:22:31.375 INFO:tasks.workunit.client.0.smithi078.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-24T00:22:31.375 INFO:tasks.workunit.client.0.smithi078.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-24T00:22:31.414 INFO:tasks.workunit.client.0.smithi078.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-24T00:22:31.417 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-24T00:22:31.418 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-24T00:22:31.473 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-03-24T00:22:31.474 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-03-24T00:22:31.475 DEBUG:teuthology.orchestra.run.smithi078:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-03-24T00:22:31.536 INFO:tasks.workunit.client.0.smithi078.stderr:+ dbench 1 2024-03-24T00:22:31.546 INFO:tasks.workunit.client.0.smithi078.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-03-24T00:22:31.546 INFO:tasks.workunit.client.0.smithi078.stdout: 2024-03-24T00:22:31.546 INFO:tasks.workunit.client.0.smithi078.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-03-24T00:22:31.550 INFO:tasks.workunit.client.0.smithi078.stdout:0 of 1 processes prepared for launch 0 sec 2024-03-24T00:22:31.650 INFO:tasks.workunit.client.0.smithi078.stdout:1 of 1 processes prepared for launch 0 sec 2024-03-24T00:22:31.651 INFO:tasks.workunit.client.0.smithi078.stdout:releasing clients 2024-03-24T00:22:32.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:32 smithi078 ceph-mon[32429]: pgmap v44: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:22:32.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:32 smithi107 ceph-mon[36485]: pgmap v44: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:22:32.650 INFO:tasks.workunit.client.0.smithi078.stdout: 1 2917 33.47 MB/sec warmup 1 sec latency 6.443 ms 2024-03-24T00:22:32.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:32 smithi106 ceph-mon[36505]: pgmap v44: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:22:33.650 INFO:tasks.workunit.client.0.smithi078.stdout: 1 4663 21.03 MB/sec warmup 2 sec latency 6.615 ms 2024-03-24T00:22:34.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:34 smithi078 ceph-mon[32429]: pgmap v45: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:22:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:34 smithi107 ceph-mon[36485]: pgmap v45: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:22:34.650 INFO:tasks.workunit.client.0.smithi078.stdout: 1 7472 19.73 MB/sec warmup 3 sec latency 7.954 ms 2024-03-24T00:22:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:34 smithi106 ceph-mon[36505]: pgmap v45: 129 pgs: 129 active+clean; 1.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:22:35.650 INFO:tasks.workunit.client.0.smithi078.stdout: 1 9581 16.74 MB/sec warmup 4 sec latency 17.738 ms 2024-03-24T00:22:36.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:36 smithi078 ceph-mon[32429]: pgmap v46: 129 pgs: 129 active+clean; 5.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 452 KiB/s wr, 73 op/s 2024-03-24T00:22:36.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:36 smithi107 ceph-mon[36485]: pgmap v46: 129 pgs: 129 active+clean; 5.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 452 KiB/s wr, 73 op/s 2024-03-24T00:22:36.650 INFO:tasks.workunit.client.0.smithi078.stdout: 1 11451 15.76 MB/sec warmup 5 sec latency 8.890 ms 2024-03-24T00:22:36.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:36 smithi106 ceph-mon[36505]: pgmap v46: 129 pgs: 129 active+clean; 5.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 452 KiB/s wr, 73 op/s 2024-03-24T00:22:37.651 INFO:tasks.workunit.client.0.smithi078.stdout: 1 14407 16.19 MB/sec warmup 6 sec latency 9.310 ms 2024-03-24T00:22:38.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:38 smithi078 ceph-mon[32429]: pgmap v47: 129 pgs: 129 active+clean; 5.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 452 KiB/s wr, 71 op/s 2024-03-24T00:22:38.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-mon[36485]: pgmap v47: 129 pgs: 129 active+clean; 5.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 452 KiB/s wr, 71 op/s 2024-03-24T00:22:38.593 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2f52136700 time 2024-03-24T00:22:38.416623+0000 2024-03-24T00:22:38.593 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:22:38.594 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:38.594 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2f5c7604a1] 2024-03-24T00:22:38.594 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2f5c760667] 2024-03-24T00:22:38.594 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560d30934223] 2024-03-24T00:22:38.594 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560d307ae5d6] 2024-03-24T00:22:38.594 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560d307aeda1] 2024-03-24T00:22:38.594 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560d307b12d5] 2024-03-24T00:22:38.594 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560d30801233] 2024-03-24T00:22:38.594 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560d30803197] 2024-03-24T00:22:38.594 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560d3070211b] 2024-03-24T00:22:38.594 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560d307026bc] 2024-03-24T00:22:38.594 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560d306ec1ff] 2024-03-24T00:22:38.594 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2f5c9f2488] 2024-03-24T00:22:38.595 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 13: (DispatchQueue::entry()+0x51f) [0x7f2f5c9ef60f] 2024-03-24T00:22:38.595 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2f5cac2e81] 2024-03-24T00:22:38.595 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2f5b4dd1ca] 2024-03-24T00:22:38.595 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 16: clone() 2024-03-24T00:22:38.596 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 2024-03-24T00:22:38.417+0000 7f2f52136700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2f52136700 time 2024-03-24T00:22:38.416623+0000 2024-03-24T00:22:38.596 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:22:38.596 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 2024-03-24T00:22:38.596 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:38.596 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2f5c7604a1] 2024-03-24T00:22:38.596 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2f5c760667] 2024-03-24T00:22:38.596 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560d30934223] 2024-03-24T00:22:38.596 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560d307ae5d6] 2024-03-24T00:22:38.596 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560d307aeda1] 2024-03-24T00:22:38.596 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560d307b12d5] 2024-03-24T00:22:38.596 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560d30801233] 2024-03-24T00:22:38.596 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560d30803197] 2024-03-24T00:22:38.597 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560d3070211b] 2024-03-24T00:22:38.597 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560d307026bc] 2024-03-24T00:22:38.597 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560d306ec1ff] 2024-03-24T00:22:38.597 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2f5c9f2488] 2024-03-24T00:22:38.597 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 13: (DispatchQueue::entry()+0x51f) [0x7f2f5c9ef60f] 2024-03-24T00:22:38.597 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2f5cac2e81] 2024-03-24T00:22:38.597 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2f5b4dd1ca] 2024-03-24T00:22:38.597 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 16: clone() 2024-03-24T00:22:38.597 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 2024-03-24T00:22:38.597 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: *** Caught signal (Aborted) ** 2024-03-24T00:22:38.597 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: in thread 7f2f52136700 thread_name:ms_dispatch 2024-03-24T00:22:38.597 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:38.597 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2f5b4e7d20] 2024-03-24T00:22:38.598 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 2: gsignal() 2024-03-24T00:22:38.598 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 3: abort() 2024-03-24T00:22:38.598 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2f5c7604fb] 2024-03-24T00:22:38.598 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2f5c760667] 2024-03-24T00:22:38.598 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560d30934223] 2024-03-24T00:22:38.598 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560d307ae5d6] 2024-03-24T00:22:38.598 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560d307aeda1] 2024-03-24T00:22:38.598 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560d307b12d5] 2024-03-24T00:22:38.598 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560d30801233] 2024-03-24T00:22:38.598 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560d30803197] 2024-03-24T00:22:38.598 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560d3070211b] 2024-03-24T00:22:38.598 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560d307026bc] 2024-03-24T00:22:38.598 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560d306ec1ff] 2024-03-24T00:22:38.598 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2f5c9f2488] 2024-03-24T00:22:38.598 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 16: (DispatchQueue::entry()+0x51f) [0x7f2f5c9ef60f] 2024-03-24T00:22:38.599 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2f5cac2e81] 2024-03-24T00:22:38.599 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2f5b4dd1ca] 2024-03-24T00:22:38.599 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 19: clone() 2024-03-24T00:22:38.599 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 2024-03-24T00:22:38.420+0000 7f2f52136700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:22:38.599 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: in thread 7f2f52136700 thread_name:ms_dispatch 2024-03-24T00:22:38.599 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 2024-03-24T00:22:38.599 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:38.599 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2f5b4e7d20] 2024-03-24T00:22:38.599 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 2: gsignal() 2024-03-24T00:22:38.599 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 3: abort() 2024-03-24T00:22:38.599 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2f5c7604fb] 2024-03-24T00:22:38.599 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2f5c760667] 2024-03-24T00:22:38.599 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560d30934223] 2024-03-24T00:22:38.599 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560d307ae5d6] 2024-03-24T00:22:38.600 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560d307aeda1] 2024-03-24T00:22:38.600 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560d307b12d5] 2024-03-24T00:22:38.600 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560d30801233] 2024-03-24T00:22:38.600 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560d30803197] 2024-03-24T00:22:38.600 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560d3070211b] 2024-03-24T00:22:38.600 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560d307026bc] 2024-03-24T00:22:38.600 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560d306ec1ff] 2024-03-24T00:22:38.600 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2f5c9f2488] 2024-03-24T00:22:38.600 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 16: (DispatchQueue::entry()+0x51f) [0x7f2f5c9ef60f] 2024-03-24T00:22:38.600 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2f5cac2e81] 2024-03-24T00:22:38.600 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2f5b4dd1ca] 2024-03-24T00:22:38.600 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 19: clone() 2024-03-24T00:22:38.600 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:22:38.600 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 2024-03-24T00:22:38.601 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: -1> 2024-03-24T00:22:38.417+0000 7f2f52136700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2f52136700 time 2024-03-24T00:22:38.416623+0000 2024-03-24T00:22:38.601 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:22:38.601 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 2024-03-24T00:22:38.601 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:38.601 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2f5c7604a1] 2024-03-24T00:22:38.601 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2f5c760667] 2024-03-24T00:22:38.601 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560d30934223] 2024-03-24T00:22:38.601 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560d307ae5d6] 2024-03-24T00:22:38.601 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560d307aeda1] 2024-03-24T00:22:38.601 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560d307b12d5] 2024-03-24T00:22:38.601 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560d30801233] 2024-03-24T00:22:38.601 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560d30803197] 2024-03-24T00:22:38.601 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560d3070211b] 2024-03-24T00:22:38.601 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560d307026bc] 2024-03-24T00:22:38.601 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560d306ec1ff] 2024-03-24T00:22:38.602 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2f5c9f2488] 2024-03-24T00:22:38.602 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 13: (DispatchQueue::entry()+0x51f) [0x7f2f5c9ef60f] 2024-03-24T00:22:38.602 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2f5cac2e81] 2024-03-24T00:22:38.602 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2f5b4dd1ca] 2024-03-24T00:22:38.602 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 16: clone() 2024-03-24T00:22:38.602 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 2024-03-24T00:22:38.602 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 0> 2024-03-24T00:22:38.420+0000 7f2f52136700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:22:38.602 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: in thread 7f2f52136700 thread_name:ms_dispatch 2024-03-24T00:22:38.602 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 2024-03-24T00:22:38.602 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:38.602 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2f5b4e7d20] 2024-03-24T00:22:38.602 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 2: gsignal() 2024-03-24T00:22:38.602 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 3: abort() 2024-03-24T00:22:38.602 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2f5c7604fb] 2024-03-24T00:22:38.602 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2f5c760667] 2024-03-24T00:22:38.603 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560d30934223] 2024-03-24T00:22:38.603 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560d307ae5d6] 2024-03-24T00:22:38.603 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560d307aeda1] 2024-03-24T00:22:38.603 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560d307b12d5] 2024-03-24T00:22:38.603 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560d30801233] 2024-03-24T00:22:38.603 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560d30803197] 2024-03-24T00:22:38.603 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560d3070211b] 2024-03-24T00:22:38.603 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560d307026bc] 2024-03-24T00:22:38.603 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560d306ec1ff] 2024-03-24T00:22:38.603 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2f5c9f2488] 2024-03-24T00:22:38.603 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 16: (DispatchQueue::entry()+0x51f) [0x7f2f5c9ef60f] 2024-03-24T00:22:38.603 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2f5cac2e81] 2024-03-24T00:22:38.603 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2f5b4dd1ca] 2024-03-24T00:22:38.603 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 19: clone() 2024-03-24T00:22:38.603 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:22:38.604 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 2024-03-24T00:22:38.604 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: -1> 2024-03-24T00:22:38.417+0000 7f2f52136700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2f52136700 time 2024-03-24T00:22:38.416623+0000 2024-03-24T00:22:38.604 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:22:38.604 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 2024-03-24T00:22:38.604 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:38.604 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2f5c7604a1] 2024-03-24T00:22:38.604 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2f5c760667] 2024-03-24T00:22:38.604 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560d30934223] 2024-03-24T00:22:38.604 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560d307ae5d6] 2024-03-24T00:22:38.605 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560d307aeda1] 2024-03-24T00:22:38.605 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560d307b12d5] 2024-03-24T00:22:38.605 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560d30801233] 2024-03-24T00:22:38.605 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560d30803197] 2024-03-24T00:22:38.605 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560d3070211b] 2024-03-24T00:22:38.605 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560d307026bc] 2024-03-24T00:22:38.605 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560d306ec1ff] 2024-03-24T00:22:38.605 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2f5c9f2488] 2024-03-24T00:22:38.605 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 13: (DispatchQueue::entry()+0x51f) [0x7f2f5c9ef60f] 2024-03-24T00:22:38.605 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2f5cac2e81] 2024-03-24T00:22:38.605 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2f5b4dd1ca] 2024-03-24T00:22:38.605 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 16: clone() 2024-03-24T00:22:38.605 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 2024-03-24T00:22:38.606 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 0> 2024-03-24T00:22:38.420+0000 7f2f52136700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:22:38.606 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: in thread 7f2f52136700 thread_name:ms_dispatch 2024-03-24T00:22:38.606 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 2024-03-24T00:22:38.606 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:38.606 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2f5b4e7d20] 2024-03-24T00:22:38.606 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 2: gsignal() 2024-03-24T00:22:38.606 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 3: abort() 2024-03-24T00:22:38.606 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2f5c7604fb] 2024-03-24T00:22:38.606 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2f5c760667] 2024-03-24T00:22:38.606 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560d30934223] 2024-03-24T00:22:38.606 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560d307ae5d6] 2024-03-24T00:22:38.606 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560d307aeda1] 2024-03-24T00:22:38.606 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560d307b12d5] 2024-03-24T00:22:38.607 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560d30801233] 2024-03-24T00:22:38.607 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560d30803197] 2024-03-24T00:22:38.607 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560d3070211b] 2024-03-24T00:22:38.607 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560d307026bc] 2024-03-24T00:22:38.607 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560d306ec1ff] 2024-03-24T00:22:38.607 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2f5c9f2488] 2024-03-24T00:22:38.607 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 16: (DispatchQueue::entry()+0x51f) [0x7f2f5c9ef60f] 2024-03-24T00:22:38.607 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2f5cac2e81] 2024-03-24T00:22:38.607 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2f5b4dd1ca] 2024-03-24T00:22:38.607 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 19: clone() 2024-03-24T00:22:38.607 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:22:38.607 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[67511]: 2024-03-24T00:22:38.607 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:38 smithi107 podman[70700]: 2024-03-24 00:22:38.479513478 +0000 UTC m=+0.019746271 container died 06ef8cdf5fccc67cd8cc7604888e4363b69f84405e2e625e9e31cbf4960161fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-24T00:22:38.651 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 14.06 MB/sec warmup 7 sec latency 241.997 ms 2024-03-24T00:22:38.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:38 smithi106 ceph-mon[36505]: pgmap v47: 129 pgs: 129 active+clean; 5.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 452 KiB/s wr, 71 op/s 2024-03-24T00:22:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:39 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:22:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:39 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:22:39.593 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:39 smithi107 podman[70700]: 2024-03-24 00:22:39.281375829 +0000 UTC m=+0.821608624 container remove 06ef8cdf5fccc67cd8cc7604888e4363b69f84405e2e625e9e31cbf4960161fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:22:39.593 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:39 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:22:39.593 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:39 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.i.service: Failed with result 'exit-code'. 2024-03-24T00:22:39.593 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:39 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.i.service: Consumed 1min 2.601s CPU time. 2024-03-24T00:22:39.651 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 12.30 MB/sec warmup 8 sec latency 1242.076 ms 2024-03-24T00:22:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:39 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:22:40.651 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 10.93 MB/sec warmup 9 sec latency 2242.135 ms 2024-03-24T00:22:40.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:40 smithi106 ceph-mon[36505]: pgmap v48: 129 pgs: 129 active+clean; 5.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 452 KiB/s wr, 71 op/s 2024-03-24T00:22:40.764 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:40 smithi078 ceph-mon[32429]: pgmap v48: 129 pgs: 129 active+clean; 5.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 452 KiB/s wr, 71 op/s 2024-03-24T00:22:40.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:40 smithi107 ceph-mon[36485]: pgmap v48: 129 pgs: 129 active+clean; 5.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 452 KiB/s wr, 71 op/s 2024-03-24T00:22:41.651 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 9.84 MB/sec warmup 10 sec latency 3242.233 ms 2024-03-24T00:22:42.651 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 8.94 MB/sec warmup 11 sec latency 4242.340 ms 2024-03-24T00:22:42.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:42 smithi106 ceph-mon[36505]: pgmap v49: 129 pgs: 129 active+clean; 8.9 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 831 KiB/s wr, 136 op/s 2024-03-24T00:22:42.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:42 smithi078 ceph-mon[32429]: pgmap v49: 129 pgs: 129 active+clean; 8.9 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 831 KiB/s wr, 136 op/s 2024-03-24T00:22:42.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:42 smithi107 ceph-mon[36485]: pgmap v49: 129 pgs: 129 active+clean; 8.9 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 831 KiB/s wr, 136 op/s 2024-03-24T00:22:43.651 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 8.20 MB/sec warmup 12 sec latency 5242.445 ms 2024-03-24T00:22:44.651 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 7.57 MB/sec warmup 13 sec latency 6242.531 ms 2024-03-24T00:22:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:44 smithi106 ceph-mon[36505]: pgmap v50: 129 pgs: 129 active+clean; 8.9 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 831 KiB/s wr, 134 op/s 2024-03-24T00:22:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:44 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:22:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:44 smithi078 ceph-mon[32429]: pgmap v50: 129 pgs: 129 active+clean; 8.9 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 831 KiB/s wr, 134 op/s 2024-03-24T00:22:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:44 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:22:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:44 smithi107 ceph-mon[36485]: pgmap v50: 129 pgs: 129 active+clean; 8.9 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 831 KiB/s wr, 134 op/s 2024-03-24T00:22:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:44 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:22:45.651 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 7.03 MB/sec warmup 14 sec latency 7242.613 ms 2024-03-24T00:22:46.651 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 6.56 MB/sec warmup 15 sec latency 8242.665 ms 2024-03-24T00:22:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:46 smithi106 ceph-mon[36505]: pgmap v51: 129 pgs: 129 active+clean; 9.0 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 842 KiB/s wr, 138 op/s 2024-03-24T00:22:46.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:46 smithi078 ceph-mon[32429]: pgmap v51: 129 pgs: 129 active+clean; 9.0 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 842 KiB/s wr, 138 op/s 2024-03-24T00:22:46.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:46 smithi107 ceph-mon[36485]: pgmap v51: 129 pgs: 129 active+clean; 9.0 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 842 KiB/s wr, 138 op/s 2024-03-24T00:22:47.652 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 6.15 MB/sec warmup 16 sec latency 9242.741 ms 2024-03-24T00:22:48.652 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 5.79 MB/sec warmup 17 sec latency 10242.838 ms 2024-03-24T00:22:48.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:48 smithi106 ceph-mon[36505]: pgmap v52: 129 pgs: 129 active+clean; 9.0 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 390 KiB/s wr, 70 op/s 2024-03-24T00:22:48.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:48 smithi078 ceph-mon[32429]: pgmap v52: 129 pgs: 129 active+clean; 9.0 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 390 KiB/s wr, 70 op/s 2024-03-24T00:22:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:48 smithi107 ceph-mon[36485]: pgmap v52: 129 pgs: 129 active+clean; 9.0 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 390 KiB/s wr, 70 op/s 2024-03-24T00:22:49.651 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 5.47 MB/sec warmup 18 sec latency 11242.883 ms 2024-03-24T00:22:49.864 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:49 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.i.service: Scheduled restart job, restart counter is at 1. 2024-03-24T00:22:49.864 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:49 smithi107 systemd[1]: Stopped Ceph mds.i for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:22:49.864 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:49 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.i.service: Consumed 1min 2.601s CPU time. 2024-03-24T00:22:49.865 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:49 smithi107 systemd[1]: Starting Ceph mds.i for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:22:49.865 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:49 smithi107 podman[70807]: 2024-03-24 00:22:49.738717153 +0000 UTC m=+0.018581906 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:22:49.865 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:49 smithi107 podman[70807]: 2024-03-24 00:22:49.865019853 +0000 UTC m=+0.144884623 container create 47cae41d7349a719589e600bd47905167025109acddc5922a46804ea6bcda99e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-24T00:22:50.125 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:49 smithi107 ceph-mon[36485]: pgmap v53: 129 pgs: 129 active+clean; 9.0 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 390 KiB/s wr, 70 op/s 2024-03-24T00:22:50.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:49 smithi078 ceph-mon[32429]: pgmap v53: 129 pgs: 129 active+clean; 9.0 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 390 KiB/s wr, 70 op/s 2024-03-24T00:22:50.391 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:50 smithi107 podman[70807]: 2024-03-24 00:22:50.126396531 +0000 UTC m=+0.406261293 container init 47cae41d7349a719589e600bd47905167025109acddc5922a46804ea6bcda99e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-24T00:22:50.391 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:50 smithi107 podman[70807]: 2024-03-24 00:22:50.129311934 +0000 UTC m=+0.409176695 container start 47cae41d7349a719589e600bd47905167025109acddc5922a46804ea6bcda99e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True) 2024-03-24T00:22:50.392 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:50 smithi107 ceph-mds[70826]: -- 172.21.15.107:0/753303391 <== mon.1 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5595608d1a40 con 0x5595608a4800 2024-03-24T00:22:50.392 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:50 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: starting mds.i at 2024-03-24T00:22:50.392 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:50 smithi107 bash[70807]: 47cae41d7349a719589e600bd47905167025109acddc5922a46804ea6bcda99e 2024-03-24T00:22:50.392 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:22:50 smithi107 systemd[1]: Started Ceph mds.i for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:22:50.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:49 smithi106 ceph-mon[36505]: pgmap v53: 129 pgs: 129 active+clean; 9.0 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 390 KiB/s wr, 70 op/s 2024-03-24T00:22:50.652 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 5.18 MB/sec warmup 19 sec latency 12243.002 ms 2024-03-24T00:22:51.326 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:50 smithi107 ceph-mon[36485]: daemon mds.i restarted 2024-03-24T00:22:51.326 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:50 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:22:51.326 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:50 smithi107 ceph-mon[36485]: osdmap e95: 12 total, 12 up, 12 in 2024-03-24T00:22:51.326 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:50 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6838/3531100722,v1:172.21.15.107:6839/3531100722] up:boot 2024-03-24T00:22:51.326 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:50 smithi107 ceph-mon[36485]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-24T00:22:51.326 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:50 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-24T00:22:51.326 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:50 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T00:22:51.326 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:50 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:replay,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:22:51.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:50 smithi078 ceph-mon[32429]: daemon mds.i restarted 2024-03-24T00:22:51.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:50 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:22:51.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:50 smithi078 ceph-mon[32429]: osdmap e95: 12 total, 12 up, 12 in 2024-03-24T00:22:51.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:50 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6838/3531100722,v1:172.21.15.107:6839/3531100722] up:boot 2024-03-24T00:22:51.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:50 smithi078 ceph-mon[32429]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-24T00:22:51.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:50 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-24T00:22:51.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:50 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T00:22:51.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:50 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:replay,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:22:51.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:50 smithi106 ceph-mon[36505]: daemon mds.i restarted 2024-03-24T00:22:51.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:50 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:22:51.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:50 smithi106 ceph-mon[36505]: osdmap e95: 12 total, 12 up, 12 in 2024-03-24T00:22:51.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:50 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6838/3531100722,v1:172.21.15.107:6839/3531100722] up:boot 2024-03-24T00:22:51.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:50 smithi106 ceph-mon[36505]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-24T00:22:51.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:50 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-24T00:22:51.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:50 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T00:22:51.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:50 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:replay,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:22:51.652 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 4.92 MB/sec warmup 20 sec latency 13243.048 ms 2024-03-24T00:22:52.303 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:51 smithi106 ceph-mon[36505]: pgmap v55: 129 pgs: 129 active+clean; 27 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 3.1 MiB/s wr, 57 op/s 2024-03-24T00:22:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:51 smithi078 ceph-mon[32429]: pgmap v55: 129 pgs: 129 active+clean; 27 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 3.1 MiB/s wr, 57 op/s 2024-03-24T00:22:52.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:51 smithi107 ceph-mon[36485]: pgmap v55: 129 pgs: 129 active+clean; 27 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 3.1 MiB/s wr, 57 op/s 2024-03-24T00:22:52.652 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 4.69 MB/sec warmup 21 sec latency 14243.157 ms 2024-03-24T00:22:53.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:52 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6834/108936439,v1:172.21.15.107:6835/108936439] up:resolve 2024-03-24T00:22:53.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:52 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:resolve,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:22:53.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:52 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6834/108936439,v1:172.21.15.107:6835/108936439] up:resolve 2024-03-24T00:22:53.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:52 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:resolve,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:22:53.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:52 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6834/108936439,v1:172.21.15.107:6835/108936439] up:resolve 2024-03-24T00:22:53.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:52 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:resolve,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:22:53.652 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 4.47 MB/sec warmup 22 sec latency 15243.206 ms 2024-03-24T00:22:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:53 smithi078 ceph-mon[32429]: reconnect by client.25352 172.21.15.78:0/3677336756 after 0 2024-03-24T00:22:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:53 smithi078 ceph-mon[32429]: reconnect by client.25297 172.21.15.78:0/2282579457 after 0 2024-03-24T00:22:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:53 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999983 2024-03-24T00:22:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:53 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6834/108936439,v1:172.21.15.107:6835/108936439] up:reconnect 2024-03-24T00:22:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:53 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:reconnect,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:22:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:53 smithi078 ceph-mon[32429]: pgmap v56: 129 pgs: 129 active+clean; 27 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 3.1 MiB/s wr, 57 op/s 2024-03-24T00:22:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:53 smithi107 ceph-mon[36485]: reconnect by client.25352 172.21.15.78:0/3677336756 after 0 2024-03-24T00:22:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:53 smithi107 ceph-mon[36485]: reconnect by client.25297 172.21.15.78:0/2282579457 after 0 2024-03-24T00:22:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:53 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999983 2024-03-24T00:22:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:53 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6834/108936439,v1:172.21.15.107:6835/108936439] up:reconnect 2024-03-24T00:22:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:53 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:reconnect,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:22:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:53 smithi107 ceph-mon[36485]: pgmap v56: 129 pgs: 129 active+clean; 27 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 3.1 MiB/s wr, 57 op/s 2024-03-24T00:22:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:53 smithi106 ceph-mon[36505]: reconnect by client.25352 172.21.15.78:0/3677336756 after 0 2024-03-24T00:22:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:53 smithi106 ceph-mon[36505]: reconnect by client.25297 172.21.15.78:0/2282579457 after 0 2024-03-24T00:22:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:53 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999983 2024-03-24T00:22:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:53 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6834/108936439,v1:172.21.15.107:6835/108936439] up:reconnect 2024-03-24T00:22:54.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:53 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:reconnect,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:22:54.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:53 smithi106 ceph-mon[36505]: pgmap v56: 129 pgs: 129 active+clean; 27 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 3.1 MiB/s wr, 57 op/s 2024-03-24T00:22:54.652 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 4.28 MB/sec warmup 23 sec latency 16243.286 ms 2024-03-24T00:22:55.249 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:54 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6834/108936439,v1:172.21.15.107:6835/108936439] up:rejoin 2024-03-24T00:22:55.249 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:54 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:rejoin,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:22:55.249 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:54 smithi106 ceph-mon[36505]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-24T00:22:55.249 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:54 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:22:55.249 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:54 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:22:55.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:54 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6834/108936439,v1:172.21.15.107:6835/108936439] up:rejoin 2024-03-24T00:22:55.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:54 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:rejoin,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:22:55.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:54 smithi078 ceph-mon[32429]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-24T00:22:55.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:54 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:22:55.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:54 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:22:55.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6834/108936439,v1:172.21.15.107:6835/108936439] up:rejoin 2024-03-24T00:22:55.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:rejoin,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:22:55.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-mon[36485]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-24T00:22:55.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:22:55.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:22:55.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f83a8cfb700 time 2024-03-24T00:22:54.988636+0000 2024-03-24T00:22:55.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:22:55.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:55.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f83b33254a1] 2024-03-24T00:22:55.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f83b3325667] 2024-03-24T00:22:55.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556780ce7223] 2024-03-24T00:22:55.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556780b615d6] 2024-03-24T00:22:55.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556780b61da1] 2024-03-24T00:22:55.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556780b642d5] 2024-03-24T00:22:55.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556780bb4233] 2024-03-24T00:22:55.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556780bb6197] 2024-03-24T00:22:55.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556780ab511b] 2024-03-24T00:22:55.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556780ab56bc] 2024-03-24T00:22:55.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556780a9f1ff] 2024-03-24T00:22:55.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f83b35b7488] 2024-03-24T00:22:55.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 13: (DispatchQueue::entry()+0x51f) [0x7f83b35b460f] 2024-03-24T00:22:55.346 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f83b3687e81] 2024-03-24T00:22:55.346 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f83b20a21ca] 2024-03-24T00:22:55.346 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 16: clone() 2024-03-24T00:22:55.346 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 2024-03-24T00:22:54.989+0000 7f83a8cfb700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f83a8cfb700 time 2024-03-24T00:22:54.988636+0000 2024-03-24T00:22:55.346 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:22:55.346 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 2024-03-24T00:22:55.346 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:55.346 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f83b33254a1] 2024-03-24T00:22:55.346 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f83b3325667] 2024-03-24T00:22:55.346 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556780ce7223] 2024-03-24T00:22:55.346 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556780b615d6] 2024-03-24T00:22:55.346 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556780b61da1] 2024-03-24T00:22:55.346 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556780b642d5] 2024-03-24T00:22:55.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556780bb4233] 2024-03-24T00:22:55.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556780bb6197] 2024-03-24T00:22:55.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556780ab511b] 2024-03-24T00:22:55.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556780ab56bc] 2024-03-24T00:22:55.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556780a9f1ff] 2024-03-24T00:22:55.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f83b35b7488] 2024-03-24T00:22:55.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 13: (DispatchQueue::entry()+0x51f) [0x7f83b35b460f] 2024-03-24T00:22:55.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f83b3687e81] 2024-03-24T00:22:55.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f83b20a21ca] 2024-03-24T00:22:55.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 16: clone() 2024-03-24T00:22:55.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 2024-03-24T00:22:55.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: *** Caught signal (Aborted) ** 2024-03-24T00:22:55.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: in thread 7f83a8cfb700 thread_name:ms_dispatch 2024-03-24T00:22:55.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:55.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f83b20acd20] 2024-03-24T00:22:55.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 2: gsignal() 2024-03-24T00:22:55.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 3: abort() 2024-03-24T00:22:55.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f83b33254fb] 2024-03-24T00:22:55.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f83b3325667] 2024-03-24T00:22:55.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556780ce7223] 2024-03-24T00:22:55.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556780b615d6] 2024-03-24T00:22:55.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556780b61da1] 2024-03-24T00:22:55.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556780b642d5] 2024-03-24T00:22:55.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556780bb4233] 2024-03-24T00:22:55.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556780bb6197] 2024-03-24T00:22:55.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556780ab511b] 2024-03-24T00:22:55.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556780ab56bc] 2024-03-24T00:22:55.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556780a9f1ff] 2024-03-24T00:22:55.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f83b35b7488] 2024-03-24T00:22:55.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 16: (DispatchQueue::entry()+0x51f) [0x7f83b35b460f] 2024-03-24T00:22:55.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f83b3687e81] 2024-03-24T00:22:55.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f83b20a21ca] 2024-03-24T00:22:55.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 19: clone() 2024-03-24T00:22:55.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 2024-03-24T00:22:54.992+0000 7f83a8cfb700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:22:55.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: in thread 7f83a8cfb700 thread_name:ms_dispatch 2024-03-24T00:22:55.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 2024-03-24T00:22:55.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:55.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f83b20acd20] 2024-03-24T00:22:55.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 2: gsignal() 2024-03-24T00:22:55.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 3: abort() 2024-03-24T00:22:55.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f83b33254fb] 2024-03-24T00:22:55.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f83b3325667] 2024-03-24T00:22:55.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556780ce7223] 2024-03-24T00:22:55.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556780b615d6] 2024-03-24T00:22:55.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556780b61da1] 2024-03-24T00:22:55.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556780b642d5] 2024-03-24T00:22:55.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556780bb4233] 2024-03-24T00:22:55.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556780bb6197] 2024-03-24T00:22:55.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556780ab511b] 2024-03-24T00:22:55.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556780ab56bc] 2024-03-24T00:22:55.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556780a9f1ff] 2024-03-24T00:22:55.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f83b35b7488] 2024-03-24T00:22:55.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 16: (DispatchQueue::entry()+0x51f) [0x7f83b35b460f] 2024-03-24T00:22:55.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f83b3687e81] 2024-03-24T00:22:55.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f83b20a21ca] 2024-03-24T00:22:55.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 19: clone() 2024-03-24T00:22:55.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:22:55.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 2024-03-24T00:22:55.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: -1> 2024-03-24T00:22:54.989+0000 7f83a8cfb700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f83a8cfb700 time 2024-03-24T00:22:54.988636+0000 2024-03-24T00:22:55.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:22:55.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 2024-03-24T00:22:55.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:55.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f83b33254a1] 2024-03-24T00:22:55.352 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f83b3325667] 2024-03-24T00:22:55.352 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556780ce7223] 2024-03-24T00:22:55.352 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556780b615d6] 2024-03-24T00:22:55.352 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556780b61da1] 2024-03-24T00:22:55.352 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556780b642d5] 2024-03-24T00:22:55.352 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556780bb4233] 2024-03-24T00:22:55.352 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556780bb6197] 2024-03-24T00:22:55.352 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556780ab511b] 2024-03-24T00:22:55.352 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556780ab56bc] 2024-03-24T00:22:55.352 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556780a9f1ff] 2024-03-24T00:22:55.352 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f83b35b7488] 2024-03-24T00:22:55.352 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 13: (DispatchQueue::entry()+0x51f) [0x7f83b35b460f] 2024-03-24T00:22:55.352 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f83b3687e81] 2024-03-24T00:22:55.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f83b20a21ca] 2024-03-24T00:22:55.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 16: clone() 2024-03-24T00:22:55.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 2024-03-24T00:22:55.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 0> 2024-03-24T00:22:54.992+0000 7f83a8cfb700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:22:55.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: in thread 7f83a8cfb700 thread_name:ms_dispatch 2024-03-24T00:22:55.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 2024-03-24T00:22:55.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:55.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f83b20acd20] 2024-03-24T00:22:55.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 2: gsignal() 2024-03-24T00:22:55.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 3: abort() 2024-03-24T00:22:55.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f83b33254fb] 2024-03-24T00:22:55.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f83b3325667] 2024-03-24T00:22:55.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556780ce7223] 2024-03-24T00:22:55.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556780b615d6] 2024-03-24T00:22:55.354 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556780b61da1] 2024-03-24T00:22:55.354 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556780b642d5] 2024-03-24T00:22:55.354 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556780bb4233] 2024-03-24T00:22:55.354 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556780bb6197] 2024-03-24T00:22:55.354 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556780ab511b] 2024-03-24T00:22:55.354 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556780ab56bc] 2024-03-24T00:22:55.354 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556780a9f1ff] 2024-03-24T00:22:55.354 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f83b35b7488] 2024-03-24T00:22:55.354 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 16: (DispatchQueue::entry()+0x51f) [0x7f83b35b460f] 2024-03-24T00:22:55.354 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f83b3687e81] 2024-03-24T00:22:55.354 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f83b20a21ca] 2024-03-24T00:22:55.354 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 19: clone() 2024-03-24T00:22:55.354 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:22:55.355 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 2024-03-24T00:22:55.355 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: -1> 2024-03-24T00:22:54.989+0000 7f83a8cfb700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f83a8cfb700 time 2024-03-24T00:22:54.988636+0000 2024-03-24T00:22:55.355 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:22:55.355 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 2024-03-24T00:22:55.355 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:55.355 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f83b33254a1] 2024-03-24T00:22:55.355 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f83b3325667] 2024-03-24T00:22:55.355 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556780ce7223] 2024-03-24T00:22:55.355 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556780b615d6] 2024-03-24T00:22:55.355 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556780b61da1] 2024-03-24T00:22:55.355 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556780b642d5] 2024-03-24T00:22:55.355 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556780bb4233] 2024-03-24T00:22:55.355 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556780bb6197] 2024-03-24T00:22:55.355 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556780ab511b] 2024-03-24T00:22:55.355 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556780ab56bc] 2024-03-24T00:22:55.356 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556780a9f1ff] 2024-03-24T00:22:55.356 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f83b35b7488] 2024-03-24T00:22:55.356 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 13: (DispatchQueue::entry()+0x51f) [0x7f83b35b460f] 2024-03-24T00:22:55.356 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f83b3687e81] 2024-03-24T00:22:55.356 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f83b20a21ca] 2024-03-24T00:22:55.356 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 16: clone() 2024-03-24T00:22:55.356 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 2024-03-24T00:22:55.356 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 0> 2024-03-24T00:22:54.992+0000 7f83a8cfb700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:22:55.356 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: in thread 7f83a8cfb700 thread_name:ms_dispatch 2024-03-24T00:22:55.356 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 2024-03-24T00:22:55.356 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:55.356 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f83b20acd20] 2024-03-24T00:22:55.356 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 2: gsignal() 2024-03-24T00:22:55.356 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 3: abort() 2024-03-24T00:22:55.356 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f83b33254fb] 2024-03-24T00:22:55.357 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f83b3325667] 2024-03-24T00:22:55.357 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556780ce7223] 2024-03-24T00:22:55.357 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556780b615d6] 2024-03-24T00:22:55.357 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556780b61da1] 2024-03-24T00:22:55.357 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556780b642d5] 2024-03-24T00:22:55.357 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556780bb4233] 2024-03-24T00:22:55.357 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556780bb6197] 2024-03-24T00:22:55.357 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556780ab511b] 2024-03-24T00:22:55.357 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556780ab56bc] 2024-03-24T00:22:55.357 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556780a9f1ff] 2024-03-24T00:22:55.357 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f83b35b7488] 2024-03-24T00:22:55.357 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 16: (DispatchQueue::entry()+0x51f) [0x7f83b35b460f] 2024-03-24T00:22:55.357 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f83b3687e81] 2024-03-24T00:22:55.357 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f83b20a21ca] 2024-03-24T00:22:55.358 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 19: clone() 2024-03-24T00:22:55.358 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:22:55.358 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[66884]: 2024-03-24T00:22:55.358 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 podman[70874]: 2024-03-24 00:22:55.07482626 +0000 UTC m=+0.019127772 container died bf5d224a6f1e013d539c03a8d089cddc3e20d940dabd9119c540a98a499b2847 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-24T00:22:55.652 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 4.10 MB/sec warmup 24 sec latency 17243.387 ms 2024-03-24T00:22:55.653 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 24 00:22:55 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-b[66985]: 2024-03-24T00:22:55.248+0000 7f978f4ee700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-24T00:22:56.060 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6834/108936439,v1:172.21.15.107:6835/108936439] up:active 2024-03-24T00:22:56.060 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:22:56.060 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:22:56.060 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:55 smithi107 ceph-mon[36485]: pgmap v57: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 3.2 MiB/s wr, 62 op/s 2024-03-24T00:22:56.061 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 podman[70874]: 2024-03-24 00:22:55.809004433 +0000 UTC m=+0.753305961 container remove bf5d224a6f1e013d539c03a8d089cddc3e20d940dabd9119c540a98a499b2847 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-24T00:22:56.061 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:55 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:22:56.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:55 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6834/108936439,v1:172.21.15.107:6835/108936439] up:active 2024-03-24T00:22:56.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:55 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:22:56.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:55 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:22:56.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:55 smithi078 ceph-mon[32429]: pgmap v57: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 3.2 MiB/s wr, 62 op/s 2024-03-24T00:22:56.343 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:22:56 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.f.service: Failed with result 'exit-code'. 2024-03-24T00:22:56.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:55 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6834/108936439,v1:172.21.15.107:6835/108936439] up:active 2024-03-24T00:22:56.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:55 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:22:56.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:55 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:22:56.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:55 smithi106 ceph-mon[36505]: pgmap v57: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 3.2 MiB/s wr, 62 op/s 2024-03-24T00:22:56.652 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 3.94 MB/sec warmup 25 sec latency 18243.494 ms 2024-03-24T00:22:57.652 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 3.78 MB/sec warmup 26 sec latency 19243.556 ms 2024-03-24T00:22:58.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:22:58 smithi078 ceph-mon[32429]: pgmap v58: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 3.2 MiB/s wr, 62 op/s 2024-03-24T00:22:58.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:22:58 smithi107 ceph-mon[36485]: pgmap v58: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 3.2 MiB/s wr, 62 op/s 2024-03-24T00:22:58.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:22:58 smithi106 ceph-mon[36505]: pgmap v58: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 3.2 MiB/s wr, 62 op/s 2024-03-24T00:22:58.652 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 3.64 MB/sec warmup 27 sec latency 20243.667 ms 2024-03-24T00:22:59.653 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 3.51 MB/sec warmup 28 sec latency 21243.814 ms 2024-03-24T00:23:00.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:00 smithi078 ceph-mon[32429]: pgmap v59: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 3.2 MiB/s wr, 62 op/s 2024-03-24T00:23:00.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:00 smithi107 ceph-mon[36485]: pgmap v59: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 3.2 MiB/s wr, 62 op/s 2024-03-24T00:23:00.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:00 smithi106 ceph-mon[36505]: pgmap v59: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 3.2 MiB/s wr, 62 op/s 2024-03-24T00:23:00.653 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 3.39 MB/sec warmup 29 sec latency 22243.915 ms 2024-03-24T00:23:01.653 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 3.28 MB/sec warmup 30 sec latency 23244.013 ms 2024-03-24T00:23:02.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:02 smithi078 ceph-mon[32429]: pgmap v60: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 2.8 MiB/s wr, 60 op/s 2024-03-24T00:23:02.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:02 smithi107 ceph-mon[36485]: pgmap v60: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 2.8 MiB/s wr, 60 op/s 2024-03-24T00:23:02.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:02 smithi106 ceph-mon[36505]: pgmap v60: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 2.8 MiB/s wr, 60 op/s 2024-03-24T00:23:02.652 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 3.17 MB/sec warmup 31 sec latency 24244.073 ms 2024-03-24T00:23:03.653 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 3.07 MB/sec warmup 32 sec latency 25244.175 ms 2024-03-24T00:23:04.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:04 smithi078 ceph-mon[32429]: pgmap v61: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-24T00:23:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:04 smithi107 ceph-mon[36485]: pgmap v61: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-24T00:23:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:04 smithi106 ceph-mon[36505]: pgmap v61: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-24T00:23:04.653 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 2.98 MB/sec warmup 33 sec latency 26244.228 ms 2024-03-24T00:23:05.653 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 2.89 MB/sec warmup 34 sec latency 27244.320 ms 2024-03-24T00:23:06.349 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:06 smithi107 ceph-mon[36485]: pgmap v62: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-24T00:23:06.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:23:06 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.f.service: Scheduled restart job, restart counter is at 1. 2024-03-24T00:23:06.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:23:06 smithi107 systemd[1]: Stopped Ceph mds.f for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:23:06.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:23:06 smithi107 systemd[1]: Starting Ceph mds.f for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:23:06.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:23:06 smithi107 podman[70972]: 2024-03-24 00:23:06.228080948 +0000 UTC m=+0.018060789 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:23:06.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:23:06 smithi107 podman[70972]: 2024-03-24 00:23:06.350490288 +0000 UTC m=+0.140470147 container create a463eeb32c327238756d28b593be94ad10653c5e3cfe427e6a0be58d3dd83dec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-24T00:23:06.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:06 smithi078 ceph-mon[32429]: pgmap v62: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-24T00:23:06.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:06 smithi106 ceph-mon[36505]: pgmap v62: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-24T00:23:06.653 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 2.81 MB/sec warmup 35 sec latency 28244.358 ms 2024-03-24T00:23:07.093 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:23:06 smithi107 podman[70972]: 2024-03-24 00:23:06.603286867 +0000 UTC m=+0.393266705 container init a463eeb32c327238756d28b593be94ad10653c5e3cfe427e6a0be58d3dd83dec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-24T00:23:07.093 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:23:06 smithi107 podman[70972]: 2024-03-24 00:23:06.606009747 +0000 UTC m=+0.395989567 container start a463eeb32c327238756d28b593be94ad10653c5e3cfe427e6a0be58d3dd83dec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:23:07.093 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:23:06 smithi107 ceph-mds[70990]: -- 172.21.15.107:0/3366734072 <== mon.1 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ffa9c7ba40 con 0x55ffa9c4ec00 2024-03-24T00:23:07.094 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:23:06 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: starting mds.f at 2024-03-24T00:23:07.094 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:23:06 smithi107 bash[70972]: a463eeb32c327238756d28b593be94ad10653c5e3cfe427e6a0be58d3dd83dec 2024-03-24T00:23:07.094 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:23:06 smithi107 systemd[1]: Started Ceph mds.f for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:23:07.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:07 smithi078 ceph-mon[32429]: daemon mds.f restarted 2024-03-24T00:23:07.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:07 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:23:07.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:07 smithi078 ceph-mon[32429]: osdmap e96: 12 total, 12 up, 12 in 2024-03-24T00:23:07.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:07 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6834/4046882186,v1:172.21.15.107:6835/4046882186] up:boot 2024-03-24T00:23:07.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:07 smithi078 ceph-mon[32429]: Standby daemon mds.i assigned to filesystem cephfs as rank 1 2024-03-24T00:23:07.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:07 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-24T00:23:07.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:07 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:23:07.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:07 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:replay,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:23:07.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:07 smithi107 ceph-mon[36485]: daemon mds.f restarted 2024-03-24T00:23:07.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:07 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:23:07.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:07 smithi107 ceph-mon[36485]: osdmap e96: 12 total, 12 up, 12 in 2024-03-24T00:23:07.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:07 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6834/4046882186,v1:172.21.15.107:6835/4046882186] up:boot 2024-03-24T00:23:07.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:07 smithi107 ceph-mon[36485]: Standby daemon mds.i assigned to filesystem cephfs as rank 1 2024-03-24T00:23:07.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:07 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-24T00:23:07.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:07 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:23:07.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:07 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:replay,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:23:07.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:07 smithi106 ceph-mon[36505]: daemon mds.f restarted 2024-03-24T00:23:07.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:07 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:23:07.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:07 smithi106 ceph-mon[36505]: osdmap e96: 12 total, 12 up, 12 in 2024-03-24T00:23:07.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:07 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6834/4046882186,v1:172.21.15.107:6835/4046882186] up:boot 2024-03-24T00:23:07.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:07 smithi106 ceph-mon[36505]: Standby daemon mds.i assigned to filesystem cephfs as rank 1 2024-03-24T00:23:07.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:07 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-24T00:23:07.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:07 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:23:07.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:07 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:replay,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:23:07.653 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 2.73 MB/sec warmup 36 sec latency 29244.464 ms 2024-03-24T00:23:08.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:08 smithi078 ceph-mon[32429]: pgmap v64: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 2.5 KiB/s wr, 8 op/s 2024-03-24T00:23:08.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:08 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6838/3531100722,v1:172.21.15.107:6839/3531100722] up:resolve 2024-03-24T00:23:08.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:08 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:resolve,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:23:08.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:08 smithi107 ceph-mon[36485]: pgmap v64: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 2.5 KiB/s wr, 8 op/s 2024-03-24T00:23:08.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:08 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6838/3531100722,v1:172.21.15.107:6839/3531100722] up:resolve 2024-03-24T00:23:08.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:08 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:resolve,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:23:08.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:08 smithi106 ceph-mon[36505]: pgmap v64: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 2.5 KiB/s wr, 8 op/s 2024-03-24T00:23:08.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:08 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6838/3531100722,v1:172.21.15.107:6839/3531100722] up:resolve 2024-03-24T00:23:08.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:08 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:resolve,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:23:08.653 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 2.66 MB/sec warmup 37 sec latency 30244.580 ms 2024-03-24T00:23:09.653 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 2.59 MB/sec warmup 38 sec latency 31244.689 ms 2024-03-24T00:23:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:09 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:23:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:09 smithi078 ceph-mon[32429]: reconnect by client.25352 172.21.15.78:0/3677336756 after 0.000999983 2024-03-24T00:23:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:09 smithi078 ceph-mon[32429]: reconnect by client.25297 172.21.15.78:0/2282579457 after 0.000999983 2024-03-24T00:23:10.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:09 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6838/3531100722,v1:172.21.15.107:6839/3531100722] up:reconnect 2024-03-24T00:23:10.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:09 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:reconnect,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:23:10.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:09 smithi078 ceph-mon[32429]: pgmap v65: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 2.5 KiB/s wr, 8 op/s 2024-03-24T00:23:10.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:09 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:23:10.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:09 smithi107 ceph-mon[36485]: reconnect by client.25352 172.21.15.78:0/3677336756 after 0.000999983 2024-03-24T00:23:10.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:09 smithi107 ceph-mon[36485]: reconnect by client.25297 172.21.15.78:0/2282579457 after 0.000999983 2024-03-24T00:23:10.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:09 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6838/3531100722,v1:172.21.15.107:6839/3531100722] up:reconnect 2024-03-24T00:23:10.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:09 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:reconnect,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:23:10.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:09 smithi107 ceph-mon[36485]: pgmap v65: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 2.5 KiB/s wr, 8 op/s 2024-03-24T00:23:10.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:09 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:23:10.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:09 smithi106 ceph-mon[36505]: reconnect by client.25352 172.21.15.78:0/3677336756 after 0.000999983 2024-03-24T00:23:10.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:09 smithi106 ceph-mon[36505]: reconnect by client.25297 172.21.15.78:0/2282579457 after 0.000999983 2024-03-24T00:23:10.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:09 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6838/3531100722,v1:172.21.15.107:6839/3531100722] up:reconnect 2024-03-24T00:23:10.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:09 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:reconnect,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:23:10.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:09 smithi106 ceph-mon[36505]: pgmap v65: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 2.5 KiB/s wr, 8 op/s 2024-03-24T00:23:10.653 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 2.52 MB/sec warmup 39 sec latency 32244.797 ms 2024-03-24T00:23:11.064 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6838/3531100722,v1:172.21.15.107:6839/3531100722] up:rejoin 2024-03-24T00:23:11.065 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:rejoin,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:23:11.065 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-mon[36485]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-24T00:23:11.065 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f167b65e700 time 2024-03-24T00:23:10.701102+0000 2024-03-24T00:23:11.065 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:23:11.065 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:11.065 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1685c884a1] 2024-03-24T00:23:11.065 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1685c88667] 2024-03-24T00:23:11.065 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55955e379223] 2024-03-24T00:23:11.065 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55955e1f35d6] 2024-03-24T00:23:11.066 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55955e1f3da1] 2024-03-24T00:23:11.066 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55955e1f62d5] 2024-03-24T00:23:11.066 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55955e246233] 2024-03-24T00:23:11.066 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55955e248197] 2024-03-24T00:23:11.066 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55955e14711b] 2024-03-24T00:23:11.066 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55955e1476bc] 2024-03-24T00:23:11.066 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55955e1311ff] 2024-03-24T00:23:11.066 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1685f1a488] 2024-03-24T00:23:11.066 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 13: (DispatchQueue::entry()+0x51f) [0x7f1685f1760f] 2024-03-24T00:23:11.066 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1685feae81] 2024-03-24T00:23:11.066 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1684a051ca] 2024-03-24T00:23:11.066 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 16: clone() 2024-03-24T00:23:11.066 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: *** Caught signal (Aborted) ** 2024-03-24T00:23:11.066 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: in thread 7f167b65e700 thread_name:ms_dispatch 2024-03-24T00:23:11.067 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 2024-03-24T00:23:10.702+0000 7f167b65e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f167b65e700 time 2024-03-24T00:23:10.701102+0000 2024-03-24T00:23:11.067 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:23:11.068 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 2024-03-24T00:23:11.068 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:11.068 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1685c884a1] 2024-03-24T00:23:11.068 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1685c88667] 2024-03-24T00:23:11.068 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55955e379223] 2024-03-24T00:23:11.068 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55955e1f35d6] 2024-03-24T00:23:11.068 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55955e1f3da1] 2024-03-24T00:23:11.068 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55955e1f62d5] 2024-03-24T00:23:11.068 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55955e246233] 2024-03-24T00:23:11.068 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55955e248197] 2024-03-24T00:23:11.068 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55955e14711b] 2024-03-24T00:23:11.068 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55955e1476bc] 2024-03-24T00:23:11.068 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55955e1311ff] 2024-03-24T00:23:11.068 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1685f1a488] 2024-03-24T00:23:11.069 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 13: (DispatchQueue::entry()+0x51f) [0x7f1685f1760f] 2024-03-24T00:23:11.069 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1685feae81] 2024-03-24T00:23:11.069 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1684a051ca] 2024-03-24T00:23:11.069 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 16: clone() 2024-03-24T00:23:11.069 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 2024-03-24T00:23:11.069 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:11.069 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1684a0fd20] 2024-03-24T00:23:11.069 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 2: gsignal() 2024-03-24T00:23:11.069 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 3: abort() 2024-03-24T00:23:11.069 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1685c884fb] 2024-03-24T00:23:11.069 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1685c88667] 2024-03-24T00:23:11.069 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55955e379223] 2024-03-24T00:23:11.069 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55955e1f35d6] 2024-03-24T00:23:11.070 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55955e1f3da1] 2024-03-24T00:23:11.070 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55955e1f62d5] 2024-03-24T00:23:11.070 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55955e246233] 2024-03-24T00:23:11.070 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55955e248197] 2024-03-24T00:23:11.070 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55955e14711b] 2024-03-24T00:23:11.070 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55955e1476bc] 2024-03-24T00:23:11.070 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55955e1311ff] 2024-03-24T00:23:11.070 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1685f1a488] 2024-03-24T00:23:11.070 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 16: (DispatchQueue::entry()+0x51f) [0x7f1685f1760f] 2024-03-24T00:23:11.070 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1685feae81] 2024-03-24T00:23:11.070 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1684a051ca] 2024-03-24T00:23:11.070 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 19: clone() 2024-03-24T00:23:11.070 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 2024-03-24T00:23:10.704+0000 7f167b65e700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:23:11.070 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: in thread 7f167b65e700 thread_name:ms_dispatch 2024-03-24T00:23:11.070 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 2024-03-24T00:23:11.071 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:11.071 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1684a0fd20] 2024-03-24T00:23:11.071 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 2: gsignal() 2024-03-24T00:23:11.071 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 3: abort() 2024-03-24T00:23:11.071 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1685c884fb] 2024-03-24T00:23:11.071 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1685c88667] 2024-03-24T00:23:11.071 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55955e379223] 2024-03-24T00:23:11.071 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55955e1f35d6] 2024-03-24T00:23:11.071 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55955e1f3da1] 2024-03-24T00:23:11.071 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55955e1f62d5] 2024-03-24T00:23:11.071 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55955e246233] 2024-03-24T00:23:11.071 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55955e248197] 2024-03-24T00:23:11.071 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55955e14711b] 2024-03-24T00:23:11.071 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55955e1476bc] 2024-03-24T00:23:11.071 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55955e1311ff] 2024-03-24T00:23:11.072 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1685f1a488] 2024-03-24T00:23:11.072 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 16: (DispatchQueue::entry()+0x51f) [0x7f1685f1760f] 2024-03-24T00:23:11.072 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1685feae81] 2024-03-24T00:23:11.072 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1684a051ca] 2024-03-24T00:23:11.072 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 19: clone() 2024-03-24T00:23:11.072 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:23:11.072 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 2024-03-24T00:23:11.072 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: -1> 2024-03-24T00:23:10.702+0000 7f167b65e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f167b65e700 time 2024-03-24T00:23:10.701102+0000 2024-03-24T00:23:11.072 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:23:11.073 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 2024-03-24T00:23:11.073 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:11.073 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1685c884a1] 2024-03-24T00:23:11.073 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1685c88667] 2024-03-24T00:23:11.073 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55955e379223] 2024-03-24T00:23:11.073 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55955e1f35d6] 2024-03-24T00:23:11.073 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55955e1f3da1] 2024-03-24T00:23:11.073 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55955e1f62d5] 2024-03-24T00:23:11.073 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55955e246233] 2024-03-24T00:23:11.073 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55955e248197] 2024-03-24T00:23:11.073 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55955e14711b] 2024-03-24T00:23:11.073 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55955e1476bc] 2024-03-24T00:23:11.073 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55955e1311ff] 2024-03-24T00:23:11.074 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1685f1a488] 2024-03-24T00:23:11.074 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 13: (DispatchQueue::entry()+0x51f) [0x7f1685f1760f] 2024-03-24T00:23:11.074 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1685feae81] 2024-03-24T00:23:11.074 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1684a051ca] 2024-03-24T00:23:11.074 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 16: clone() 2024-03-24T00:23:11.074 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 2024-03-24T00:23:11.074 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 0> 2024-03-24T00:23:10.704+0000 7f167b65e700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:23:11.074 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: in thread 7f167b65e700 thread_name:ms_dispatch 2024-03-24T00:23:11.074 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 2024-03-24T00:23:11.074 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:11.074 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1684a0fd20] 2024-03-24T00:23:11.074 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 2: gsignal() 2024-03-24T00:23:11.074 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 3: abort() 2024-03-24T00:23:11.074 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1685c884fb] 2024-03-24T00:23:11.075 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1685c88667] 2024-03-24T00:23:11.075 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55955e379223] 2024-03-24T00:23:11.075 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55955e1f35d6] 2024-03-24T00:23:11.075 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55955e1f3da1] 2024-03-24T00:23:11.075 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55955e1f62d5] 2024-03-24T00:23:11.075 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55955e246233] 2024-03-24T00:23:11.075 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55955e248197] 2024-03-24T00:23:11.075 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55955e14711b] 2024-03-24T00:23:11.075 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55955e1476bc] 2024-03-24T00:23:11.075 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55955e1311ff] 2024-03-24T00:23:11.075 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1685f1a488] 2024-03-24T00:23:11.075 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 16: (DispatchQueue::entry()+0x51f) [0x7f1685f1760f] 2024-03-24T00:23:11.075 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1685feae81] 2024-03-24T00:23:11.075 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1684a051ca] 2024-03-24T00:23:11.076 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 19: clone() 2024-03-24T00:23:11.076 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:23:11.076 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 2024-03-24T00:23:11.076 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: -1> 2024-03-24T00:23:10.702+0000 7f167b65e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f167b65e700 time 2024-03-24T00:23:10.701102+0000 2024-03-24T00:23:11.076 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:23:11.076 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 2024-03-24T00:23:11.076 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:11.076 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1685c884a1] 2024-03-24T00:23:11.076 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1685c88667] 2024-03-24T00:23:11.076 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55955e379223] 2024-03-24T00:23:11.076 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55955e1f35d6] 2024-03-24T00:23:11.077 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55955e1f3da1] 2024-03-24T00:23:11.077 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55955e1f62d5] 2024-03-24T00:23:11.077 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55955e246233] 2024-03-24T00:23:11.077 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55955e248197] 2024-03-24T00:23:11.077 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55955e14711b] 2024-03-24T00:23:11.077 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55955e1476bc] 2024-03-24T00:23:11.077 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55955e1311ff] 2024-03-24T00:23:11.077 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1685f1a488] 2024-03-24T00:23:11.077 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 13: (DispatchQueue::entry()+0x51f) [0x7f1685f1760f] 2024-03-24T00:23:11.077 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1685feae81] 2024-03-24T00:23:11.077 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1684a051ca] 2024-03-24T00:23:11.077 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 16: clone() 2024-03-24T00:23:11.077 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 2024-03-24T00:23:11.077 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 0> 2024-03-24T00:23:10.704+0000 7f167b65e700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:23:11.078 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: in thread 7f167b65e700 thread_name:ms_dispatch 2024-03-24T00:23:11.078 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 2024-03-24T00:23:11.078 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:11.078 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1684a0fd20] 2024-03-24T00:23:11.078 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 2: gsignal() 2024-03-24T00:23:11.078 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 3: abort() 2024-03-24T00:23:11.078 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1685c884fb] 2024-03-24T00:23:11.078 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1685c88667] 2024-03-24T00:23:11.078 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55955e379223] 2024-03-24T00:23:11.078 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55955e1f35d6] 2024-03-24T00:23:11.078 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55955e1f3da1] 2024-03-24T00:23:11.078 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55955e1f62d5] 2024-03-24T00:23:11.078 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55955e246233] 2024-03-24T00:23:11.079 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55955e248197] 2024-03-24T00:23:11.079 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55955e14711b] 2024-03-24T00:23:11.079 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55955e1476bc] 2024-03-24T00:23:11.079 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55955e1311ff] 2024-03-24T00:23:11.079 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1685f1a488] 2024-03-24T00:23:11.079 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 16: (DispatchQueue::entry()+0x51f) [0x7f1685f1760f] 2024-03-24T00:23:11.079 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1685feae81] 2024-03-24T00:23:11.079 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1684a051ca] 2024-03-24T00:23:11.079 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 19: clone() 2024-03-24T00:23:11.079 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:23:11.079 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[70822]: 2024-03-24T00:23:11.079 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:10 smithi107 podman[71025]: 2024-03-24 00:23:10.751447854 +0000 UTC m=+0.020376864 container died 47cae41d7349a719589e600bd47905167025109acddc5922a46804ea6bcda99e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:23:11.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:10 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6838/3531100722,v1:172.21.15.107:6839/3531100722] up:rejoin 2024-03-24T00:23:11.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:10 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:rejoin,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:23:11.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:10 smithi078 ceph-mon[32429]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-24T00:23:11.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:10 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6838/3531100722,v1:172.21.15.107:6839/3531100722] up:rejoin 2024-03-24T00:23:11.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:10 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:rejoin,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:23:11.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:10 smithi106 ceph-mon[36505]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-24T00:23:11.654 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 2.46 MB/sec warmup 40 sec latency 33244.911 ms 2024-03-24T00:23:11.843 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:11 smithi107 podman[71025]: 2024-03-24 00:23:11.483352438 +0000 UTC m=+0.752281422 container remove 47cae41d7349a719589e600bd47905167025109acddc5922a46804ea6bcda99e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:23:11.843 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:11 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:23:11.843 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:11 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.i.service: Failed with result 'exit-code'. 2024-03-24T00:23:11.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:11 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:23:11.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:11 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:23:11.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:11 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6838/3531100722,v1:172.21.15.107:6839/3531100722] up:active 2024-03-24T00:23:11.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:11 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:23:11.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:11 smithi107 ceph-mon[36485]: mgrmap e25: x(active, since 2m), standbys: z, y 2024-03-24T00:23:11.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:11 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:23:11.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:11 smithi107 ceph-mon[36485]: pgmap v66: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.3 KiB/s wr, 8 op/s 2024-03-24T00:23:12.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:11 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:23:12.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:11 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:23:12.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:11 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6838/3531100722,v1:172.21.15.107:6839/3531100722] up:active 2024-03-24T00:23:12.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:11 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:23:12.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:11 smithi078 ceph-mon[32429]: mgrmap e25: x(active, since 2m), standbys: z, y 2024-03-24T00:23:12.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:11 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:23:12.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:11 smithi078 ceph-mon[32429]: pgmap v66: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.3 KiB/s wr, 8 op/s 2024-03-24T00:23:12.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:11 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:23:12.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:11 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:23:12.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:11 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6838/3531100722,v1:172.21.15.107:6839/3531100722] up:active 2024-03-24T00:23:12.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:11 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:23:12.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:11 smithi106 ceph-mon[36505]: mgrmap e25: x(active, since 2m), standbys: z, y 2024-03-24T00:23:12.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:11 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:23:12.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:11 smithi106 ceph-mon[36505]: pgmap v66: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.3 KiB/s wr, 8 op/s 2024-03-24T00:23:12.654 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 2.40 MB/sec warmup 41 sec latency 34244.966 ms 2024-03-24T00:23:13.654 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 2.34 MB/sec warmup 42 sec latency 35245.077 ms 2024-03-24T00:23:14.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:14 smithi078 ceph-mon[32429]: pgmap v67: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.3 KiB/s wr, 8 op/s 2024-03-24T00:23:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:14 smithi107 ceph-mon[36485]: pgmap v67: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.3 KiB/s wr, 8 op/s 2024-03-24T00:23:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:14 smithi106 ceph-mon[36505]: pgmap v67: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.3 KiB/s wr, 8 op/s 2024-03-24T00:23:14.654 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 2.29 MB/sec warmup 43 sec latency 36245.149 ms 2024-03-24T00:23:15.652 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 24 00:23:15 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-b[66985]: 2024-03-24T00:23:15.248+0000 7f978f4ee700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-24T00:23:15.654 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 2.24 MB/sec warmup 44 sec latency 37245.206 ms 2024-03-24T00:23:16.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:16 smithi078 ceph-mon[32429]: pgmap v68: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 28 op/s 2024-03-24T00:23:16.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:23:16.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:23:16.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:23:16.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:23:16.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:16 smithi107 ceph-mon[36485]: pgmap v68: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 28 op/s 2024-03-24T00:23:16.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:23:16.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:23:16.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:23:16.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:23:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:16 smithi106 ceph-mon[36505]: pgmap v68: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 28 op/s 2024-03-24T00:23:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:23:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:23:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:23:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:23:16.654 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 2.19 MB/sec warmup 45 sec latency 38245.286 ms 2024-03-24T00:23:17.654 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 2.14 MB/sec warmup 46 sec latency 39245.385 ms 2024-03-24T00:23:18.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:18 smithi078 ceph-mon[32429]: pgmap v69: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 26 op/s 2024-03-24T00:23:18.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:18 smithi107 ceph-mon[36485]: pgmap v69: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 26 op/s 2024-03-24T00:23:18.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:18 smithi106 ceph-mon[36505]: pgmap v69: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 26 op/s 2024-03-24T00:23:18.654 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 2.09 MB/sec warmup 47 sec latency 40245.444 ms 2024-03-24T00:23:19.654 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 2.05 MB/sec warmup 48 sec latency 41245.560 ms 2024-03-24T00:23:20.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:20 smithi078 ceph-mon[32429]: pgmap v70: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-03-24T00:23:20.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:20 smithi107 ceph-mon[36485]: pgmap v70: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-03-24T00:23:20.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:20 smithi106 ceph-mon[36505]: pgmap v70: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-03-24T00:23:20.654 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 2.01 MB/sec warmup 49 sec latency 42245.661 ms 2024-03-24T00:23:21.654 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.97 MB/sec warmup 50 sec latency 43245.744 ms 2024-03-24T00:23:22.099 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:21 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.i.service: Scheduled restart job, restart counter is at 2. 2024-03-24T00:23:22.099 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:21 smithi107 systemd[1]: Stopped Ceph mds.i for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:23:22.099 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:21 smithi107 systemd[1]: Starting Ceph mds.i for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:23:22.099 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:22 smithi107 podman[71254]: 2024-03-24 00:23:21.978541718 +0000 UTC m=+0.018329458 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:23:22.351 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:22 smithi107 ceph-mon[36485]: pgmap v71: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 24 KiB/s wr, 26 op/s 2024-03-24T00:23:22.352 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:22 smithi107 podman[71254]: 2024-03-24 00:23:22.100131242 +0000 UTC m=+0.139918966 container create 8737b638b692bcfb2a756c65aa074df2f21d588e0116c20af637f2336579087a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5) 2024-03-24T00:23:22.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:22 smithi078 ceph-mon[32429]: pgmap v71: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 24 KiB/s wr, 26 op/s 2024-03-24T00:23:22.617 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:22 smithi107 podman[71254]: 2024-03-24 00:23:22.352599058 +0000 UTC m=+0.392386780 container init 8737b638b692bcfb2a756c65aa074df2f21d588e0116c20af637f2336579087a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-24T00:23:22.617 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:22 smithi107 podman[71254]: 2024-03-24 00:23:22.355468821 +0000 UTC m=+0.395256555 container start 8737b638b692bcfb2a756c65aa074df2f21d588e0116c20af637f2336579087a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-24T00:23:22.617 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:22 smithi107 ceph-mds[71273]: -- 172.21.15.107:0/3895117843 <== mon.1 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55726f5c7a40 con 0x55726f59ac00 2024-03-24T00:23:22.617 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:22 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: starting mds.i at 2024-03-24T00:23:22.617 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:22 smithi107 bash[71254]: 8737b638b692bcfb2a756c65aa074df2f21d588e0116c20af637f2336579087a 2024-03-24T00:23:22.617 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:23:22 smithi107 systemd[1]: Started Ceph mds.i for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:23:22.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:22 smithi106 ceph-mon[36505]: pgmap v71: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 24 KiB/s wr, 26 op/s 2024-03-24T00:23:22.654 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.93 MB/sec warmup 51 sec latency 44245.831 ms 2024-03-24T00:23:23.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:23 smithi107 ceph-mon[36485]: daemon mds.i restarted 2024-03-24T00:23:23.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:23 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:23:23.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:23 smithi107 ceph-mon[36485]: osdmap e97: 12 total, 12 up, 12 in 2024-03-24T00:23:23.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:23 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6838/3131891440,v1:172.21.15.107:6839/3131891440] up:boot 2024-03-24T00:23:23.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:23 smithi107 ceph-mon[36485]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-24T00:23:23.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:23 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-24T00:23:23.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:23 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T00:23:23.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:23 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:replay,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:23:23.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:23 smithi106 ceph-mon[36505]: daemon mds.i restarted 2024-03-24T00:23:23.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:23 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:23:23.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:23 smithi106 ceph-mon[36505]: osdmap e97: 12 total, 12 up, 12 in 2024-03-24T00:23:23.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:23 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6838/3131891440,v1:172.21.15.107:6839/3131891440] up:boot 2024-03-24T00:23:23.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:23 smithi106 ceph-mon[36505]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-24T00:23:23.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:23 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-24T00:23:23.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:23 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T00:23:23.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:23 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:replay,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:23:23.654 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.89 MB/sec warmup 52 sec latency 45245.923 ms 2024-03-24T00:23:23.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:23 smithi078 ceph-mon[32429]: daemon mds.i restarted 2024-03-24T00:23:23.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:23 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:23:23.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:23 smithi078 ceph-mon[32429]: osdmap e97: 12 total, 12 up, 12 in 2024-03-24T00:23:23.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:23 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6838/3131891440,v1:172.21.15.107:6839/3131891440] up:boot 2024-03-24T00:23:23.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:23 smithi078 ceph-mon[32429]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-24T00:23:23.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:23 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-24T00:23:23.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:23 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T00:23:23.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:23 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:replay,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:23:24.655 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.86 MB/sec warmup 53 sec latency 46246.014 ms 2024-03-24T00:23:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:24 smithi078 ceph-mon[32429]: pgmap v73: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:23:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:24 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6834/4046882186,v1:172.21.15.107:6835/4046882186] up:resolve 2024-03-24T00:23:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:24 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:resolve,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:23:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:24 smithi107 ceph-mon[36485]: pgmap v73: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:23:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:24 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6834/4046882186,v1:172.21.15.107:6835/4046882186] up:resolve 2024-03-24T00:23:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:24 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:resolve,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:23:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:24 smithi106 ceph-mon[36505]: pgmap v73: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:23:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:24 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6834/4046882186,v1:172.21.15.107:6835/4046882186] up:resolve 2024-03-24T00:23:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:24 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:resolve,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:23:25.655 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.82 MB/sec warmup 54 sec latency 47246.164 ms 2024-03-24T00:23:25.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:25 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:23:25.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:25 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6834/4046882186,v1:172.21.15.107:6835/4046882186] up:reconnect 2024-03-24T00:23:25.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:25 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:reconnect,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:23:25.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:25 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:23:25.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:25 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6834/4046882186,v1:172.21.15.107:6835/4046882186] up:reconnect 2024-03-24T00:23:25.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:25 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:reconnect,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:23:25.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:25 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:23:25.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:25 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6834/4046882186,v1:172.21.15.107:6835/4046882186] up:reconnect 2024-03-24T00:23:25.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:25 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:reconnect,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:23:26.657 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.79 MB/sec warmup 55 sec latency 48246.256 ms 2024-03-24T00:23:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:26 smithi078 ceph-mon[32429]: pgmap v74: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-24T00:23:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:26 smithi107 ceph-mon[36485]: pgmap v74: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-24T00:23:26.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:26 smithi106 ceph-mon[36505]: pgmap v74: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-24T00:23:27.655 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.76 MB/sec warmup 56 sec latency 49246.354 ms 2024-03-24T00:23:28.655 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.73 MB/sec warmup 57 sec latency 50246.475 ms 2024-03-24T00:23:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:28 smithi078 ceph-mon[32429]: pgmap v75: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-24T00:23:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:28 smithi107 ceph-mon[36485]: pgmap v75: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-24T00:23:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:28 smithi106 ceph-mon[36505]: pgmap v75: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-24T00:23:29.655 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.70 MB/sec warmup 58 sec latency 51246.591 ms 2024-03-24T00:23:30.655 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.67 MB/sec warmup 59 sec latency 52246.691 ms 2024-03-24T00:23:30.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:30 smithi078 ceph-mon[32429]: pgmap v76: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-24T00:23:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:30 smithi107 ceph-mon[36485]: pgmap v76: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-24T00:23:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:30 smithi106 ceph-mon[36505]: pgmap v76: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-24T00:23:31.655 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.64 MB/sec warmup 60 sec latency 53246.747 ms 2024-03-24T00:23:32.655 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.61 MB/sec warmup 61 sec latency 54246.803 ms 2024-03-24T00:23:32.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:32 smithi078 ceph-mon[32429]: pgmap v77: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T00:23:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:32 smithi107 ceph-mon[36485]: pgmap v77: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T00:23:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:32 smithi106 ceph-mon[36505]: pgmap v77: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T00:23:33.656 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.59 MB/sec warmup 62 sec latency 55246.876 ms 2024-03-24T00:23:34.655 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.56 MB/sec warmup 63 sec latency 56246.913 ms 2024-03-24T00:23:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:34 smithi078 ceph-mon[32429]: pgmap v78: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T00:23:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:34 smithi107 ceph-mon[36485]: pgmap v78: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T00:23:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:34 smithi106 ceph-mon[36505]: pgmap v78: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T00:23:35.656 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.54 MB/sec warmup 64 sec latency 57247.032 ms 2024-03-24T00:23:36.656 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.51 MB/sec warmup 65 sec latency 58247.166 ms 2024-03-24T00:23:36.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:36 smithi078 ceph-mon[32429]: pgmap v79: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:23:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:36 smithi107 ceph-mon[36485]: pgmap v79: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:23:36.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:36 smithi106 ceph-mon[36505]: pgmap v79: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:23:37.656 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.49 MB/sec warmup 66 sec latency 59247.280 ms 2024-03-24T00:23:38.656 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.47 MB/sec warmup 67 sec latency 60247.379 ms 2024-03-24T00:23:38.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:38 smithi078 ceph-mon[32429]: pgmap v80: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:23:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:38 smithi107 ceph-mon[36485]: pgmap v80: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:23:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:38 smithi106 ceph-mon[36505]: pgmap v80: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:23:39.656 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.45 MB/sec warmup 68 sec latency 61247.493 ms 2024-03-24T00:23:40.656 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.43 MB/sec warmup 69 sec latency 62247.592 ms 2024-03-24T00:23:40.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:40 smithi078 ceph-mon[32429]: pgmap v81: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:23:40.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:40 smithi107 ceph-mon[36485]: pgmap v81: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:23:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:40 smithi106 ceph-mon[36505]: pgmap v81: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:23:41.656 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.41 MB/sec warmup 70 sec latency 63247.698 ms 2024-03-24T00:23:42.657 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.39 MB/sec warmup 71 sec latency 64247.787 ms 2024-03-24T00:23:42.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:42 smithi078 ceph-mon[32429]: pgmap v82: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:23:42.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:42 smithi107 ceph-mon[36485]: pgmap v82: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:23:42.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:42 smithi106 ceph-mon[36505]: pgmap v82: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:23:43.657 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.37 MB/sec warmup 72 sec latency 65247.916 ms 2024-03-24T00:23:44.657 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.35 MB/sec warmup 73 sec latency 66248.020 ms 2024-03-24T00:23:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:44 smithi078 ceph-mon[32429]: pgmap v83: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:23:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:44 smithi107 ceph-mon[36485]: pgmap v83: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:23:44.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:44 smithi106 ceph-mon[36505]: pgmap v83: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:23:45.657 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.33 MB/sec warmup 74 sec latency 67248.108 ms 2024-03-24T00:23:46.657 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.31 MB/sec warmup 75 sec latency 68248.215 ms 2024-03-24T00:23:46.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:46 smithi078 ceph-mon[32429]: pgmap v84: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-24T00:23:46.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:46 smithi107 ceph-mon[36485]: pgmap v84: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-24T00:23:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:46 smithi106 ceph-mon[36505]: pgmap v84: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-24T00:23:47.657 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.29 MB/sec warmup 76 sec latency 69248.335 ms 2024-03-24T00:23:48.657 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.28 MB/sec warmup 77 sec latency 70248.453 ms 2024-03-24T00:23:48.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:48 smithi078 ceph-mon[32429]: pgmap v85: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T00:23:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:48 smithi107 ceph-mon[36485]: pgmap v85: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T00:23:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:48 smithi106 ceph-mon[36505]: pgmap v85: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T00:23:49.657 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.26 MB/sec warmup 78 sec latency 71248.503 ms 2024-03-24T00:23:50.658 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.25 MB/sec warmup 79 sec latency 72248.642 ms 2024-03-24T00:23:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:50 smithi078 ceph-mon[32429]: pgmap v86: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T00:23:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:50 smithi107 ceph-mon[36485]: pgmap v86: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T00:23:50.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:50 smithi106 ceph-mon[36505]: pgmap v86: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T00:23:51.657 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.23 MB/sec warmup 80 sec latency 73248.679 ms 2024-03-24T00:23:52.657 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.21 MB/sec warmup 81 sec latency 74248.719 ms 2024-03-24T00:23:52.704 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:52 smithi106 ceph-mon[36505]: pgmap v87: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-24T00:23:52.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:52 smithi078 ceph-mon[32429]: pgmap v87: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-24T00:23:52.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:52 smithi107 ceph-mon[36485]: pgmap v87: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-24T00:23:53.657 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.20 MB/sec warmup 82 sec latency 75248.803 ms 2024-03-24T00:23:54.658 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.19 MB/sec warmup 83 sec latency 76248.877 ms 2024-03-24T00:23:54.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:54 smithi078 ceph-mon[32429]: pgmap v88: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T00:23:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:54 smithi107 ceph-mon[36485]: pgmap v88: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T00:23:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:54 smithi106 ceph-mon[36505]: pgmap v88: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T00:23:55.658 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.17 MB/sec warmup 84 sec latency 77248.938 ms 2024-03-24T00:23:56.658 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.16 MB/sec warmup 85 sec latency 78248.983 ms 2024-03-24T00:23:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:56 smithi078 ceph-mon[32429]: pgmap v89: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-24T00:23:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:56 smithi107 ceph-mon[36485]: pgmap v89: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-24T00:23:56.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:56 smithi106 ceph-mon[36505]: pgmap v89: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-24T00:23:57.658 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.14 MB/sec warmup 86 sec latency 79249.097 ms 2024-03-24T00:23:58.658 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.13 MB/sec warmup 87 sec latency 80249.186 ms 2024-03-24T00:23:58.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:23:58 smithi078 ceph-mon[32429]: pgmap v90: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:23:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:23:58 smithi107 ceph-mon[36485]: pgmap v90: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:23:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:23:58 smithi106 ceph-mon[36505]: pgmap v90: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:23:59.658 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.12 MB/sec warmup 88 sec latency 81249.300 ms 2024-03-24T00:24:00.658 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.11 MB/sec warmup 89 sec latency 82249.419 ms 2024-03-24T00:24:00.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:00 smithi078 ceph-mon[32429]: pgmap v91: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:24:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:00 smithi107 ceph-mon[36485]: pgmap v91: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:24:00.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:00 smithi106 ceph-mon[36505]: pgmap v91: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:24:01.658 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.09 MB/sec warmup 90 sec latency 83249.498 ms 2024-03-24T00:24:02.658 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.08 MB/sec warmup 91 sec latency 84249.592 ms 2024-03-24T00:24:02.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:02 smithi078 ceph-mon[32429]: pgmap v92: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:24:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:02 smithi107 ceph-mon[36485]: pgmap v92: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:24:02.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:02 smithi106 ceph-mon[36505]: pgmap v92: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:24:03.658 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.07 MB/sec warmup 92 sec latency 85249.697 ms 2024-03-24T00:24:04.658 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.06 MB/sec warmup 93 sec latency 86249.735 ms 2024-03-24T00:24:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:04 smithi078 ceph-mon[32429]: pgmap v93: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:24:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:04 smithi107 ceph-mon[36485]: pgmap v93: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:24:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:04 smithi106 ceph-mon[36505]: pgmap v93: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:24:05.658 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.05 MB/sec warmup 94 sec latency 87249.768 ms 2024-03-24T00:24:06.658 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.04 MB/sec warmup 95 sec latency 88249.808 ms 2024-03-24T00:24:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:06 smithi078 ceph-mon[32429]: pgmap v94: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:24:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:06 smithi107 ceph-mon[36485]: pgmap v94: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:24:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:06 smithi106 ceph-mon[36505]: pgmap v94: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:24:07.659 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.02 MB/sec warmup 96 sec latency 89249.892 ms 2024-03-24T00:24:08.659 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.01 MB/sec warmup 97 sec latency 90249.956 ms 2024-03-24T00:24:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:08 smithi078 ceph-mon[32429]: pgmap v95: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-24T00:24:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:08 smithi107 ceph-mon[36485]: pgmap v95: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-24T00:24:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:08 smithi106 ceph-mon[36505]: pgmap v95: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-24T00:24:09.659 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 1.00 MB/sec warmup 98 sec latency 91250.034 ms 2024-03-24T00:24:10.659 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.99 MB/sec warmup 99 sec latency 92250.091 ms 2024-03-24T00:24:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:10 smithi078 ceph-mon[32429]: pgmap v96: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-24T00:24:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:10 smithi107 ceph-mon[36485]: pgmap v96: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-24T00:24:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:10 smithi106 ceph-mon[36505]: pgmap v96: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-24T00:24:11.659 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.98 MB/sec warmup 100 sec latency 93250.147 ms 2024-03-24T00:24:12.659 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.97 MB/sec warmup 101 sec latency 94250.212 ms 2024-03-24T00:24:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:12 smithi078 ceph-mon[32429]: pgmap v97: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:24:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:12 smithi078 ceph-mon[32429]: evicting unresponsive client smithi078: (25297), after waiting 47.2332 seconds during MDS startup 2024-03-24T00:24:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:12 smithi078 ceph-mon[32429]: Evicting (and blocklisting) client session 25297 (172.21.15.78:0/2282579457) 2024-03-24T00:24:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:12 smithi078 ceph-mon[32429]: evicting unresponsive client smithi078: (25352), after waiting 47.2332 seconds during MDS startup 2024-03-24T00:24:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:12 smithi078 ceph-mon[32429]: Evicting (and blocklisting) client session 25352 (172.21.15.78:0/3677336756) 2024-03-24T00:24:12.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:12 smithi078 ceph-mon[32429]: from='mds.? [v2:172.21.15.107:6834/4046882186,v1:172.21.15.107:6835/4046882186]' entity='mds.f' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.78:0/2282579457"}]: dispatch 2024-03-24T00:24:12.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:12 smithi078 ceph-mon[32429]: from='mds.? [v2:172.21.15.107:6834/4046882186,v1:172.21.15.107:6835/4046882186]' entity='mds.f' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.78:0/3677336756"}]: dispatch 2024-03-24T00:24:12.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:12 smithi078 ceph-mon[32429]: from='mds.? ' entity='mds.f' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.78:0/2282579457"}]: dispatch 2024-03-24T00:24:12.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:12 smithi078 ceph-mon[32429]: from='mds.? ' entity='mds.f' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.78:0/3677336756"}]: dispatch 2024-03-24T00:24:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:12 smithi107 ceph-mon[36485]: pgmap v97: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:24:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:12 smithi107 ceph-mon[36485]: evicting unresponsive client smithi078: (25297), after waiting 47.2332 seconds during MDS startup 2024-03-24T00:24:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:12 smithi107 ceph-mon[36485]: Evicting (and blocklisting) client session 25297 (172.21.15.78:0/2282579457) 2024-03-24T00:24:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:12 smithi107 ceph-mon[36485]: evicting unresponsive client smithi078: (25352), after waiting 47.2332 seconds during MDS startup 2024-03-24T00:24:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:12 smithi107 ceph-mon[36485]: Evicting (and blocklisting) client session 25352 (172.21.15.78:0/3677336756) 2024-03-24T00:24:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:12 smithi107 ceph-mon[36485]: from='mds.? [v2:172.21.15.107:6834/4046882186,v1:172.21.15.107:6835/4046882186]' entity='mds.f' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.78:0/2282579457"}]: dispatch 2024-03-24T00:24:12.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:12 smithi107 ceph-mon[36485]: from='mds.? [v2:172.21.15.107:6834/4046882186,v1:172.21.15.107:6835/4046882186]' entity='mds.f' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.78:0/3677336756"}]: dispatch 2024-03-24T00:24:12.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:12 smithi107 ceph-mon[36485]: from='mds.? ' entity='mds.f' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.78:0/2282579457"}]: dispatch 2024-03-24T00:24:12.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:12 smithi107 ceph-mon[36485]: from='mds.? ' entity='mds.f' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.78:0/3677336756"}]: dispatch 2024-03-24T00:24:12.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:12 smithi106 ceph-mon[36505]: pgmap v97: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:24:12.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:12 smithi106 ceph-mon[36505]: evicting unresponsive client smithi078: (25297), after waiting 47.2332 seconds during MDS startup 2024-03-24T00:24:12.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:12 smithi106 ceph-mon[36505]: Evicting (and blocklisting) client session 25297 (172.21.15.78:0/2282579457) 2024-03-24T00:24:12.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:12 smithi106 ceph-mon[36505]: evicting unresponsive client smithi078: (25352), after waiting 47.2332 seconds during MDS startup 2024-03-24T00:24:12.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:12 smithi106 ceph-mon[36505]: Evicting (and blocklisting) client session 25352 (172.21.15.78:0/3677336756) 2024-03-24T00:24:12.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:12 smithi106 ceph-mon[36505]: from='mds.? [v2:172.21.15.107:6834/4046882186,v1:172.21.15.107:6835/4046882186]' entity='mds.f' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.78:0/2282579457"}]: dispatch 2024-03-24T00:24:12.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:12 smithi106 ceph-mon[36505]: from='mds.? [v2:172.21.15.107:6834/4046882186,v1:172.21.15.107:6835/4046882186]' entity='mds.f' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.78:0/3677336756"}]: dispatch 2024-03-24T00:24:12.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:12 smithi106 ceph-mon[36505]: from='mds.? ' entity='mds.f' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.78:0/2282579457"}]: dispatch 2024-03-24T00:24:12.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:12 smithi106 ceph-mon[36505]: from='mds.? ' entity='mds.f' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.78:0/3677336756"}]: dispatch 2024-03-24T00:24:13.659 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.96 MB/sec warmup 102 sec latency 95250.276 ms 2024-03-24T00:24:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:13 smithi078 ceph-mon[32429]: from='mds.? ' entity='mds.f' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.78:0/2282579457"}]': finished 2024-03-24T00:24:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:13 smithi078 ceph-mon[32429]: from='mds.? ' entity='mds.f' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.78:0/3677336756"}]': finished 2024-03-24T00:24:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:13 smithi078 ceph-mon[32429]: osdmap e98: 12 total, 12 up, 12 in 2024-03-24T00:24:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:13 smithi078 ceph-mon[32429]: pgmap v99: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-24T00:24:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-mon[36485]: from='mds.? ' entity='mds.f' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.78:0/2282579457"}]': finished 2024-03-24T00:24:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-mon[36485]: from='mds.? ' entity='mds.f' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.78:0/3677336756"}]': finished 2024-03-24T00:24:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-mon[36485]: osdmap e98: 12 total, 12 up, 12 in 2024-03-24T00:24:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-mon[36485]: pgmap v99: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-24T00:24:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:13 smithi106 ceph-mon[36505]: from='mds.? ' entity='mds.f' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.78:0/2282579457"}]': finished 2024-03-24T00:24:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:13 smithi106 ceph-mon[36505]: from='mds.? ' entity='mds.f' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.78:0/3677336756"}]': finished 2024-03-24T00:24:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:13 smithi106 ceph-mon[36505]: osdmap e98: 12 total, 12 up, 12 in 2024-03-24T00:24:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:13 smithi106 ceph-mon[36505]: pgmap v99: 129 pgs: 129 active+clean; 27 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-24T00:24:14.325 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f62c88c1700 time 2024-03-24T00:24:13.968606+0000 2024-03-24T00:24:14.326 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:14.326 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:14.326 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f62d2eeb4a1] 2024-03-24T00:24:14.326 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f62d2eeb667] 2024-03-24T00:24:14.326 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ffa8b23223] 2024-03-24T00:24:14.326 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ffa899d5d6] 2024-03-24T00:24:14.326 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ffa899dda1] 2024-03-24T00:24:14.326 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ffa89a02d5] 2024-03-24T00:24:14.326 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ffa89f0233] 2024-03-24T00:24:14.327 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ffa89f2197] 2024-03-24T00:24:14.327 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ffa88f111b] 2024-03-24T00:24:14.327 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ffa88f16bc] 2024-03-24T00:24:14.327 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ffa88db1ff] 2024-03-24T00:24:14.327 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f62d317d488] 2024-03-24T00:24:14.327 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 13: (DispatchQueue::entry()+0x51f) [0x7f62d317a60f] 2024-03-24T00:24:14.327 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f62d324de81] 2024-03-24T00:24:14.327 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f62d1c681ca] 2024-03-24T00:24:14.327 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 16: clone() 2024-03-24T00:24:14.329 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 2024-03-24T00:24:13.969+0000 7f62c88c1700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f62c88c1700 time 2024-03-24T00:24:13.968606+0000 2024-03-24T00:24:14.329 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:14.329 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 2024-03-24T00:24:14.329 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:14.329 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f62d2eeb4a1] 2024-03-24T00:24:14.329 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f62d2eeb667] 2024-03-24T00:24:14.329 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ffa8b23223] 2024-03-24T00:24:14.329 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ffa899d5d6] 2024-03-24T00:24:14.329 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ffa899dda1] 2024-03-24T00:24:14.329 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ffa89a02d5] 2024-03-24T00:24:14.329 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ffa89f0233] 2024-03-24T00:24:14.330 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ffa89f2197] 2024-03-24T00:24:14.330 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ffa88f111b] 2024-03-24T00:24:14.330 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ffa88f16bc] 2024-03-24T00:24:14.330 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ffa88db1ff] 2024-03-24T00:24:14.330 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f62d317d488] 2024-03-24T00:24:14.330 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 13: (DispatchQueue::entry()+0x51f) [0x7f62d317a60f] 2024-03-24T00:24:14.330 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f62d324de81] 2024-03-24T00:24:14.330 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f62d1c681ca] 2024-03-24T00:24:14.330 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 16: clone() 2024-03-24T00:24:14.330 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 2024-03-24T00:24:14.330 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: *** Caught signal (Aborted) ** 2024-03-24T00:24:14.330 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: in thread 7f62c88c1700 thread_name:ms_dispatch 2024-03-24T00:24:14.331 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:14.331 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f62d1c72d20] 2024-03-24T00:24:14.331 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 2: gsignal() 2024-03-24T00:24:14.331 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 3: abort() 2024-03-24T00:24:14.331 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f62d2eeb4fb] 2024-03-24T00:24:14.331 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f62d2eeb667] 2024-03-24T00:24:14.331 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ffa8b23223] 2024-03-24T00:24:14.331 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ffa899d5d6] 2024-03-24T00:24:14.331 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ffa899dda1] 2024-03-24T00:24:14.331 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ffa89a02d5] 2024-03-24T00:24:14.331 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ffa89f0233] 2024-03-24T00:24:14.331 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ffa89f2197] 2024-03-24T00:24:14.331 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ffa88f111b] 2024-03-24T00:24:14.331 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ffa88f16bc] 2024-03-24T00:24:14.331 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ffa88db1ff] 2024-03-24T00:24:14.332 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f62d317d488] 2024-03-24T00:24:14.332 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 16: (DispatchQueue::entry()+0x51f) [0x7f62d317a60f] 2024-03-24T00:24:14.332 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f62d324de81] 2024-03-24T00:24:14.332 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f62d1c681ca] 2024-03-24T00:24:14.332 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 19: clone() 2024-03-24T00:24:14.332 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 2024-03-24T00:24:13.971+0000 7f62c88c1700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:24:14.332 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: in thread 7f62c88c1700 thread_name:ms_dispatch 2024-03-24T00:24:14.332 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 2024-03-24T00:24:14.332 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:14.332 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f62d1c72d20] 2024-03-24T00:24:14.332 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 2: gsignal() 2024-03-24T00:24:14.332 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 3: abort() 2024-03-24T00:24:14.332 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f62d2eeb4fb] 2024-03-24T00:24:14.332 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f62d2eeb667] 2024-03-24T00:24:14.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ffa8b23223] 2024-03-24T00:24:14.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ffa899d5d6] 2024-03-24T00:24:14.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ffa899dda1] 2024-03-24T00:24:14.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ffa89a02d5] 2024-03-24T00:24:14.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ffa89f0233] 2024-03-24T00:24:14.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ffa89f2197] 2024-03-24T00:24:14.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ffa88f111b] 2024-03-24T00:24:14.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ffa88f16bc] 2024-03-24T00:24:14.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ffa88db1ff] 2024-03-24T00:24:14.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f62d317d488] 2024-03-24T00:24:14.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 16: (DispatchQueue::entry()+0x51f) [0x7f62d317a60f] 2024-03-24T00:24:14.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f62d324de81] 2024-03-24T00:24:14.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f62d1c681ca] 2024-03-24T00:24:14.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 19: clone() 2024-03-24T00:24:14.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:24:14.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 2024-03-24T00:24:14.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: -1> 2024-03-24T00:24:13.969+0000 7f62c88c1700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f62c88c1700 time 2024-03-24T00:24:13.968606+0000 2024-03-24T00:24:14.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:14.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 2024-03-24T00:24:14.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:14.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f62d2eeb4a1] 2024-03-24T00:24:14.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f62d2eeb667] 2024-03-24T00:24:14.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ffa8b23223] 2024-03-24T00:24:14.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ffa899d5d6] 2024-03-24T00:24:14.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ffa899dda1] 2024-03-24T00:24:14.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ffa89a02d5] 2024-03-24T00:24:14.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ffa89f0233] 2024-03-24T00:24:14.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ffa89f2197] 2024-03-24T00:24:14.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ffa88f111b] 2024-03-24T00:24:14.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ffa88f16bc] 2024-03-24T00:24:14.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ffa88db1ff] 2024-03-24T00:24:14.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f62d317d488] 2024-03-24T00:24:14.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 13: (DispatchQueue::entry()+0x51f) [0x7f62d317a60f] 2024-03-24T00:24:14.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f62d324de81] 2024-03-24T00:24:14.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f62d1c681ca] 2024-03-24T00:24:14.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 16: clone() 2024-03-24T00:24:14.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 2024-03-24T00:24:14.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 0> 2024-03-24T00:24:13.971+0000 7f62c88c1700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:24:14.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: in thread 7f62c88c1700 thread_name:ms_dispatch 2024-03-24T00:24:14.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 2024-03-24T00:24:14.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:14.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f62d1c72d20] 2024-03-24T00:24:14.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 2: gsignal() 2024-03-24T00:24:14.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 3: abort() 2024-03-24T00:24:14.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f62d2eeb4fb] 2024-03-24T00:24:14.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f62d2eeb667] 2024-03-24T00:24:14.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ffa8b23223] 2024-03-24T00:24:14.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ffa899d5d6] 2024-03-24T00:24:14.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ffa899dda1] 2024-03-24T00:24:14.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ffa89a02d5] 2024-03-24T00:24:14.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ffa89f0233] 2024-03-24T00:24:14.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ffa89f2197] 2024-03-24T00:24:14.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ffa88f111b] 2024-03-24T00:24:14.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ffa88f16bc] 2024-03-24T00:24:14.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ffa88db1ff] 2024-03-24T00:24:14.337 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f62d317d488] 2024-03-24T00:24:14.337 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 16: (DispatchQueue::entry()+0x51f) [0x7f62d317a60f] 2024-03-24T00:24:14.337 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f62d324de81] 2024-03-24T00:24:14.337 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f62d1c681ca] 2024-03-24T00:24:14.337 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 19: clone() 2024-03-24T00:24:14.337 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:24:14.337 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 2024-03-24T00:24:14.337 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: -1> 2024-03-24T00:24:13.969+0000 7f62c88c1700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f62c88c1700 time 2024-03-24T00:24:13.968606+0000 2024-03-24T00:24:14.337 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:14.337 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 2024-03-24T00:24:14.337 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:14.337 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f62d2eeb4a1] 2024-03-24T00:24:14.337 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f62d2eeb667] 2024-03-24T00:24:14.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ffa8b23223] 2024-03-24T00:24:14.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ffa899d5d6] 2024-03-24T00:24:14.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ffa899dda1] 2024-03-24T00:24:14.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ffa89a02d5] 2024-03-24T00:24:14.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ffa89f0233] 2024-03-24T00:24:14.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ffa89f2197] 2024-03-24T00:24:14.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ffa88f111b] 2024-03-24T00:24:14.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ffa88f16bc] 2024-03-24T00:24:14.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ffa88db1ff] 2024-03-24T00:24:14.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f62d317d488] 2024-03-24T00:24:14.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 13: (DispatchQueue::entry()+0x51f) [0x7f62d317a60f] 2024-03-24T00:24:14.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f62d324de81] 2024-03-24T00:24:14.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f62d1c681ca] 2024-03-24T00:24:14.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 16: clone() 2024-03-24T00:24:14.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 2024-03-24T00:24:14.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 0> 2024-03-24T00:24:13.971+0000 7f62c88c1700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:24:14.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: in thread 7f62c88c1700 thread_name:ms_dispatch 2024-03-24T00:24:14.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 2024-03-24T00:24:14.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:14.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f62d1c72d20] 2024-03-24T00:24:14.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 2: gsignal() 2024-03-24T00:24:14.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 3: abort() 2024-03-24T00:24:14.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f62d2eeb4fb] 2024-03-24T00:24:14.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f62d2eeb667] 2024-03-24T00:24:14.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ffa8b23223] 2024-03-24T00:24:14.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ffa899d5d6] 2024-03-24T00:24:14.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ffa899dda1] 2024-03-24T00:24:14.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ffa89a02d5] 2024-03-24T00:24:14.340 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ffa89f0233] 2024-03-24T00:24:14.340 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ffa89f2197] 2024-03-24T00:24:14.340 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ffa88f111b] 2024-03-24T00:24:14.340 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ffa88f16bc] 2024-03-24T00:24:14.340 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ffa88db1ff] 2024-03-24T00:24:14.340 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f62d317d488] 2024-03-24T00:24:14.340 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 16: (DispatchQueue::entry()+0x51f) [0x7f62d317a60f] 2024-03-24T00:24:14.340 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f62d324de81] 2024-03-24T00:24:14.340 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f62d1c681ca] 2024-03-24T00:24:14.340 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 19: clone() 2024-03-24T00:24:14.340 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:24:14.340 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[70986]: 2024-03-24T00:24:14.340 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:14 smithi107 podman[71320]: 2024-03-24 00:24:14.022387973 +0000 UTC m=+0.020592980 container died a463eeb32c327238756d28b593be94ad10653c5e3cfe427e6a0be58d3dd83dec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-24T00:24:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:14 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6834/4046882186,v1:172.21.15.107:6835/4046882186] up:rejoin 2024-03-24T00:24:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:14 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:rejoin,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:14 smithi107 ceph-mon[36485]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-24T00:24:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:14 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:24:14.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:14 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:24:14.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:14 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6834/4046882186,v1:172.21.15.107:6835/4046882186] up:active 2024-03-24T00:24:14.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:14 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:24:14.659 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.96 MB/sec warmup 103 sec latency 96250.352 ms 2024-03-24T00:24:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:14 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6834/4046882186,v1:172.21.15.107:6835/4046882186] up:rejoin 2024-03-24T00:24:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:14 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:rejoin,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:14 smithi078 ceph-mon[32429]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-24T00:24:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:14 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:24:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:14 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:24:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:14 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6834/4046882186,v1:172.21.15.107:6835/4046882186] up:active 2024-03-24T00:24:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:14 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:24:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:14 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6834/4046882186,v1:172.21.15.107:6835/4046882186] up:rejoin 2024-03-24T00:24:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:14 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:rejoin,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:14 smithi106 ceph-mon[36505]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-24T00:24:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:14 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:24:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:14 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:24:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:14 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6834/4046882186,v1:172.21.15.107:6835/4046882186] up:active 2024-03-24T00:24:14.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:14 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:24:15.093 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:14 smithi107 podman[71320]: 2024-03-24 00:24:14.761443816 +0000 UTC m=+0.759648884 container remove a463eeb32c327238756d28b593be94ad10653c5e3cfe427e6a0be58d3dd83dec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-24T00:24:15.093 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:14 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:24:15.093 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:15 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.f.service: Failed with result 'exit-code'. 2024-03-24T00:24:15.507 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 24 00:24:15 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-b[66985]: 2024-03-24T00:24:15.252+0000 7f978f4ee700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-24T00:24:15.659 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.95 MB/sec warmup 104 sec latency 97250.461 ms 2024-03-24T00:24:15.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:15 smithi078 ceph-mon[32429]: pgmap v100: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 KiB/s wr, 26 op/s 2024-03-24T00:24:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:15 smithi107 ceph-mon[36485]: pgmap v100: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 KiB/s wr, 26 op/s 2024-03-24T00:24:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:15 smithi106 ceph-mon[36505]: pgmap v100: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 KiB/s wr, 26 op/s 2024-03-24T00:24:16.659 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.94 MB/sec warmup 105 sec latency 98250.515 ms 2024-03-24T00:24:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:24:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:24:16.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:24:17.659 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.93 MB/sec warmup 106 sec latency 99250.642 ms 2024-03-24T00:24:17.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:17 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:24:17.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:17 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:24:17.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:17 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:24:17.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:17 smithi078 ceph-mon[32429]: pgmap v101: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 KiB/s wr, 26 op/s 2024-03-24T00:24:17.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:17 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:24:17.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:17 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:24:17.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:17 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:24:17.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:17 smithi107 ceph-mon[36485]: pgmap v101: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 KiB/s wr, 26 op/s 2024-03-24T00:24:17.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:17 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:24:17.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:17 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:24:17.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:17 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:24:17.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:17 smithi106 ceph-mon[36505]: pgmap v101: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 KiB/s wr, 26 op/s 2024-03-24T00:24:18.659 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.92 MB/sec warmup 107 sec latency 100250.731 ms 2024-03-24T00:24:19.660 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.91 MB/sec warmup 108 sec latency 101250.830 ms 2024-03-24T00:24:20.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:20 smithi106 ceph-mon[36505]: pgmap v102: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 KiB/s wr, 26 op/s 2024-03-24T00:24:20.659 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.90 MB/sec warmup 109 sec latency 102250.880 ms 2024-03-24T00:24:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:20 smithi078 ceph-mon[32429]: pgmap v102: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 KiB/s wr, 26 op/s 2024-03-24T00:24:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:20 smithi107 ceph-mon[36485]: pgmap v102: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 KiB/s wr, 26 op/s 2024-03-24T00:24:21.660 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.89 MB/sec warmup 110 sec latency 103250.995 ms 2024-03-24T00:24:22.645 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:22 smithi107 ceph-mon[36485]: pgmap v103: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 27 op/s 2024-03-24T00:24:22.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:22 smithi106 ceph-mon[36505]: pgmap v103: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 27 op/s 2024-03-24T00:24:22.660 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.89 MB/sec warmup 111 sec latency 104251.085 ms 2024-03-24T00:24:22.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:22 smithi078 ceph-mon[32429]: pgmap v103: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 27 op/s 2024-03-24T00:24:23.660 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.88 MB/sec warmup 112 sec latency 105251.190 ms 2024-03-24T00:24:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:24 smithi106 ceph-mon[36505]: pgmap v104: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 25 op/s 2024-03-24T00:24:24.660 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.87 MB/sec warmup 113 sec latency 106251.297 ms 2024-03-24T00:24:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:24 smithi078 ceph-mon[32429]: pgmap v104: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 25 op/s 2024-03-24T00:24:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:24 smithi107 ceph-mon[36485]: pgmap v104: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 25 op/s 2024-03-24T00:24:25.360 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:25 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.f.service: Scheduled restart job, restart counter is at 2. 2024-03-24T00:24:25.361 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:25 smithi107 systemd[1]: Stopped Ceph mds.f for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:24:25.361 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:25 smithi107 systemd[1]: Starting Ceph mds.f for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:24:25.361 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:25 smithi107 podman[71546]: 2024-03-24 00:24:25.230922225 +0000 UTC m=+0.018050934 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:24:25.361 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:25 smithi107 podman[71546]: 2024-03-24 00:24:25.361484336 +0000 UTC m=+0.148613034 container create ef5cc80c6b13b90d304f79d58ecbcf4d413c97122058e6981a1d8ab2b1d4fcc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=) 2024-03-24T00:24:25.630 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:25 smithi107 podman[71546]: 2024-03-24 00:24:25.606246027 +0000 UTC m=+0.393374739 container init ef5cc80c6b13b90d304f79d58ecbcf4d413c97122058e6981a1d8ab2b1d4fcc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-24T00:24:25.630 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:25 smithi107 podman[71546]: 2024-03-24 00:24:25.608817846 +0000 UTC m=+0.395946553 container start ef5cc80c6b13b90d304f79d58ecbcf4d413c97122058e6981a1d8ab2b1d4fcc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-24T00:24:25.660 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.86 MB/sec warmup 114 sec latency 107251.345 ms 2024-03-24T00:24:26.093 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:25 smithi107 ceph-mds[71566]: -- 172.21.15.107:0/2158821679 <== mon.1 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55acfeb55a40 con 0x55acfeb29000 2024-03-24T00:24:26.093 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:25 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: starting mds.f at 2024-03-24T00:24:26.093 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:25 smithi107 bash[71546]: ef5cc80c6b13b90d304f79d58ecbcf4d413c97122058e6981a1d8ab2b1d4fcc9 2024-03-24T00:24:26.093 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:25 smithi107 systemd[1]: Started Ceph mds.f for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:24:26.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:26 smithi106 ceph-mon[36505]: pgmap v105: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-03-24T00:24:26.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:26 smithi106 ceph-mon[36505]: daemon mds.f restarted 2024-03-24T00:24:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:26 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:24:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:26 smithi106 ceph-mon[36505]: osdmap e99: 12 total, 12 up, 12 in 2024-03-24T00:24:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:26 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6834/23104891,v1:172.21.15.107:6835/23104891] up:boot 2024-03-24T00:24:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:26 smithi106 ceph-mon[36505]: Standby daemon mds.i assigned to filesystem cephfs as rank 1 2024-03-24T00:24:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:26 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-24T00:24:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:26 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:24:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:26 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:replay,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:26.660 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.86 MB/sec warmup 115 sec latency 108251.413 ms 2024-03-24T00:24:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:26 smithi078 ceph-mon[32429]: pgmap v105: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-03-24T00:24:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:26 smithi078 ceph-mon[32429]: daemon mds.f restarted 2024-03-24T00:24:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:26 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:24:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:26 smithi078 ceph-mon[32429]: osdmap e99: 12 total, 12 up, 12 in 2024-03-24T00:24:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:26 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6834/23104891,v1:172.21.15.107:6835/23104891] up:boot 2024-03-24T00:24:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:26 smithi078 ceph-mon[32429]: Standby daemon mds.i assigned to filesystem cephfs as rank 1 2024-03-24T00:24:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:26 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-24T00:24:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:26 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:24:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:26 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:replay,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:26 smithi107 ceph-mon[36485]: pgmap v105: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-03-24T00:24:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:26 smithi107 ceph-mon[36485]: daemon mds.f restarted 2024-03-24T00:24:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:26 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:24:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:26 smithi107 ceph-mon[36485]: osdmap e99: 12 total, 12 up, 12 in 2024-03-24T00:24:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:26 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6834/23104891,v1:172.21.15.107:6835/23104891] up:boot 2024-03-24T00:24:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:26 smithi107 ceph-mon[36485]: Standby daemon mds.i assigned to filesystem cephfs as rank 1 2024-03-24T00:24:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:26 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-24T00:24:26.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:26 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:24:26.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:26 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:replay,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:27.660 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.85 MB/sec warmup 116 sec latency 109251.521 ms 2024-03-24T00:24:28.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:27 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6838/3131891440,v1:172.21.15.107:6839/3131891440] up:resolve 2024-03-24T00:24:28.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:27 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:resolve,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:28.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:27 smithi078 ceph-mon[32429]: pgmap v107: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:24:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:27 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6838/3131891440,v1:172.21.15.107:6839/3131891440] up:resolve 2024-03-24T00:24:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:27 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:resolve,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:27 smithi107 ceph-mon[36485]: pgmap v107: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:24:28.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:27 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6838/3131891440,v1:172.21.15.107:6839/3131891440] up:resolve 2024-03-24T00:24:28.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:27 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:resolve,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:28.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:27 smithi106 ceph-mon[36505]: pgmap v107: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:24:28.660 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.84 MB/sec warmup 117 sec latency 110251.619 ms 2024-03-24T00:24:28.954 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:28 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:24:28.955 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:28 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6838/3131891440,v1:172.21.15.107:6839/3131891440] up:reconnect 2024-03-24T00:24:28.955 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:28 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:reconnect,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:28.967 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:24:28.967 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6838/3131891440,v1:172.21.15.107:6839/3131891440] up:reconnect 2024-03-24T00:24:28.967 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:reconnect,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:28 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:24:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:28 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6838/3131891440,v1:172.21.15.107:6839/3131891440] up:reconnect 2024-03-24T00:24:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:28 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:reconnect,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:29.259 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f9135100700 time 2024-03-24T00:24:28.968525+0000 2024-03-24T00:24:29.259 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:29.259 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:29.259 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f913f72a4a1] 2024-03-24T00:24:29.259 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f913f72a667] 2024-03-24T00:24:29.259 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55726cc8c223] 2024-03-24T00:24:29.259 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55726cb065d6] 2024-03-24T00:24:29.260 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55726cb06da1] 2024-03-24T00:24:29.260 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55726cb092d5] 2024-03-24T00:24:29.260 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55726cb59233] 2024-03-24T00:24:29.260 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55726cb5b197] 2024-03-24T00:24:29.260 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55726ca5a11b] 2024-03-24T00:24:29.260 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55726ca5a6bc] 2024-03-24T00:24:29.260 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55726ca441ff] 2024-03-24T00:24:29.260 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f913f9bc488] 2024-03-24T00:24:29.260 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 13: (DispatchQueue::entry()+0x51f) [0x7f913f9b960f] 2024-03-24T00:24:29.260 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f913fa8ce81] 2024-03-24T00:24:29.260 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f913e4a71ca] 2024-03-24T00:24:29.260 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 16: clone() 2024-03-24T00:24:29.262 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 2024-03-24T00:24:28.969+0000 7f9135100700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f9135100700 time 2024-03-24T00:24:28.968525+0000 2024-03-24T00:24:29.262 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:29.262 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 2024-03-24T00:24:29.262 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:29.262 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f913f72a4a1] 2024-03-24T00:24:29.262 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f913f72a667] 2024-03-24T00:24:29.262 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55726cc8c223] 2024-03-24T00:24:29.262 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55726cb065d6] 2024-03-24T00:24:29.263 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55726cb06da1] 2024-03-24T00:24:29.263 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55726cb092d5] 2024-03-24T00:24:29.263 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55726cb59233] 2024-03-24T00:24:29.263 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55726cb5b197] 2024-03-24T00:24:29.263 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55726ca5a11b] 2024-03-24T00:24:29.263 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55726ca5a6bc] 2024-03-24T00:24:29.263 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55726ca441ff] 2024-03-24T00:24:29.263 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f913f9bc488] 2024-03-24T00:24:29.263 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 13: (DispatchQueue::entry()+0x51f) [0x7f913f9b960f] 2024-03-24T00:24:29.263 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f913fa8ce81] 2024-03-24T00:24:29.263 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f913e4a71ca] 2024-03-24T00:24:29.263 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 16: clone() 2024-03-24T00:24:29.263 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 2024-03-24T00:24:29.264 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: *** Caught signal (Aborted) ** 2024-03-24T00:24:29.264 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: in thread 7f9135100700 thread_name:ms_dispatch 2024-03-24T00:24:29.264 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:29.264 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f913e4b1d20] 2024-03-24T00:24:29.264 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 2: gsignal() 2024-03-24T00:24:29.264 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 3: abort() 2024-03-24T00:24:29.264 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f913f72a4fb] 2024-03-24T00:24:29.264 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f913f72a667] 2024-03-24T00:24:29.264 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55726cc8c223] 2024-03-24T00:24:29.264 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55726cb065d6] 2024-03-24T00:24:29.264 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55726cb06da1] 2024-03-24T00:24:29.264 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55726cb092d5] 2024-03-24T00:24:29.264 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55726cb59233] 2024-03-24T00:24:29.264 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55726cb5b197] 2024-03-24T00:24:29.265 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55726ca5a11b] 2024-03-24T00:24:29.265 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55726ca5a6bc] 2024-03-24T00:24:29.265 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55726ca441ff] 2024-03-24T00:24:29.265 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f913f9bc488] 2024-03-24T00:24:29.265 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 16: (DispatchQueue::entry()+0x51f) [0x7f913f9b960f] 2024-03-24T00:24:29.265 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f913fa8ce81] 2024-03-24T00:24:29.265 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f913e4a71ca] 2024-03-24T00:24:29.265 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 19: clone() 2024-03-24T00:24:29.265 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 2024-03-24T00:24:28.971+0000 7f9135100700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:24:29.265 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: in thread 7f9135100700 thread_name:ms_dispatch 2024-03-24T00:24:29.265 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 2024-03-24T00:24:29.265 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:29.265 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f913e4b1d20] 2024-03-24T00:24:29.265 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 2: gsignal() 2024-03-24T00:24:29.266 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 3: abort() 2024-03-24T00:24:29.266 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f913f72a4fb] 2024-03-24T00:24:29.266 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f913f72a667] 2024-03-24T00:24:29.266 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55726cc8c223] 2024-03-24T00:24:29.266 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55726cb065d6] 2024-03-24T00:24:29.266 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55726cb06da1] 2024-03-24T00:24:29.266 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55726cb092d5] 2024-03-24T00:24:29.266 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55726cb59233] 2024-03-24T00:24:29.266 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55726cb5b197] 2024-03-24T00:24:29.266 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55726ca5a11b] 2024-03-24T00:24:29.266 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55726ca5a6bc] 2024-03-24T00:24:29.266 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55726ca441ff] 2024-03-24T00:24:29.266 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f913f9bc488] 2024-03-24T00:24:29.267 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 16: (DispatchQueue::entry()+0x51f) [0x7f913f9b960f] 2024-03-24T00:24:29.267 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f913fa8ce81] 2024-03-24T00:24:29.267 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f913e4a71ca] 2024-03-24T00:24:29.267 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 19: clone() 2024-03-24T00:24:29.267 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:24:29.267 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 2024-03-24T00:24:29.267 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: -1> 2024-03-24T00:24:28.969+0000 7f9135100700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f9135100700 time 2024-03-24T00:24:28.968525+0000 2024-03-24T00:24:29.267 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:29.267 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 2024-03-24T00:24:29.267 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:29.267 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f913f72a4a1] 2024-03-24T00:24:29.267 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f913f72a667] 2024-03-24T00:24:29.267 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55726cc8c223] 2024-03-24T00:24:29.267 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55726cb065d6] 2024-03-24T00:24:29.268 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55726cb06da1] 2024-03-24T00:24:29.268 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55726cb092d5] 2024-03-24T00:24:29.268 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55726cb59233] 2024-03-24T00:24:29.268 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55726cb5b197] 2024-03-24T00:24:29.268 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55726ca5a11b] 2024-03-24T00:24:29.268 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55726ca5a6bc] 2024-03-24T00:24:29.268 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55726ca441ff] 2024-03-24T00:24:29.268 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f913f9bc488] 2024-03-24T00:24:29.268 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 13: (DispatchQueue::entry()+0x51f) [0x7f913f9b960f] 2024-03-24T00:24:29.268 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f913fa8ce81] 2024-03-24T00:24:29.268 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f913e4a71ca] 2024-03-24T00:24:29.268 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 16: clone() 2024-03-24T00:24:29.268 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 2024-03-24T00:24:29.268 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 0> 2024-03-24T00:24:28.971+0000 7f9135100700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:24:29.269 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: in thread 7f9135100700 thread_name:ms_dispatch 2024-03-24T00:24:29.269 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 2024-03-24T00:24:29.269 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:29.269 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f913e4b1d20] 2024-03-24T00:24:29.269 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 2: gsignal() 2024-03-24T00:24:29.269 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 3: abort() 2024-03-24T00:24:29.269 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f913f72a4fb] 2024-03-24T00:24:29.269 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f913f72a667] 2024-03-24T00:24:29.269 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55726cc8c223] 2024-03-24T00:24:29.269 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55726cb065d6] 2024-03-24T00:24:29.269 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55726cb06da1] 2024-03-24T00:24:29.269 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55726cb092d5] 2024-03-24T00:24:29.269 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55726cb59233] 2024-03-24T00:24:29.269 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55726cb5b197] 2024-03-24T00:24:29.270 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55726ca5a11b] 2024-03-24T00:24:29.270 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55726ca5a6bc] 2024-03-24T00:24:29.270 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55726ca441ff] 2024-03-24T00:24:29.270 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f913f9bc488] 2024-03-24T00:24:29.270 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 16: (DispatchQueue::entry()+0x51f) [0x7f913f9b960f] 2024-03-24T00:24:29.270 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f913fa8ce81] 2024-03-24T00:24:29.270 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f913e4a71ca] 2024-03-24T00:24:29.270 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 19: clone() 2024-03-24T00:24:29.270 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:24:29.270 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 2024-03-24T00:24:29.270 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: -1> 2024-03-24T00:24:28.969+0000 7f9135100700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f9135100700 time 2024-03-24T00:24:28.968525+0000 2024-03-24T00:24:29.270 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:29.270 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 2024-03-24T00:24:29.271 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:29.271 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f913f72a4a1] 2024-03-24T00:24:29.271 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f913f72a667] 2024-03-24T00:24:29.271 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55726cc8c223] 2024-03-24T00:24:29.271 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55726cb065d6] 2024-03-24T00:24:29.271 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55726cb06da1] 2024-03-24T00:24:29.271 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55726cb092d5] 2024-03-24T00:24:29.271 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55726cb59233] 2024-03-24T00:24:29.271 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55726cb5b197] 2024-03-24T00:24:29.271 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55726ca5a11b] 2024-03-24T00:24:29.271 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55726ca5a6bc] 2024-03-24T00:24:29.271 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55726ca441ff] 2024-03-24T00:24:29.271 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f913f9bc488] 2024-03-24T00:24:29.271 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 13: (DispatchQueue::entry()+0x51f) [0x7f913f9b960f] 2024-03-24T00:24:29.272 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f913fa8ce81] 2024-03-24T00:24:29.272 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f913e4a71ca] 2024-03-24T00:24:29.272 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 16: clone() 2024-03-24T00:24:29.272 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 2024-03-24T00:24:29.272 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 0> 2024-03-24T00:24:28.971+0000 7f9135100700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:24:29.272 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: in thread 7f9135100700 thread_name:ms_dispatch 2024-03-24T00:24:29.272 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 2024-03-24T00:24:29.272 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:29.272 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f913e4b1d20] 2024-03-24T00:24:29.272 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 2: gsignal() 2024-03-24T00:24:29.272 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 3: abort() 2024-03-24T00:24:29.272 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f913f72a4fb] 2024-03-24T00:24:29.272 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f913f72a667] 2024-03-24T00:24:29.272 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55726cc8c223] 2024-03-24T00:24:29.272 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55726cb065d6] 2024-03-24T00:24:29.273 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55726cb06da1] 2024-03-24T00:24:29.273 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55726cb092d5] 2024-03-24T00:24:29.273 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55726cb59233] 2024-03-24T00:24:29.273 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55726cb5b197] 2024-03-24T00:24:29.273 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55726ca5a11b] 2024-03-24T00:24:29.273 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55726ca5a6bc] 2024-03-24T00:24:29.273 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55726ca441ff] 2024-03-24T00:24:29.273 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f913f9bc488] 2024-03-24T00:24:29.273 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 16: (DispatchQueue::entry()+0x51f) [0x7f913f9b960f] 2024-03-24T00:24:29.273 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f913fa8ce81] 2024-03-24T00:24:29.273 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f913e4a71ca] 2024-03-24T00:24:29.273 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 19: clone() 2024-03-24T00:24:29.273 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:24:29.273 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71269]: 2024-03-24T00:24:29.273 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:29 smithi107 podman[71600]: 2024-03-24 00:24:29.024874584 +0000 UTC m=+0.020942575 container died 8737b638b692bcfb2a756c65aa074df2f21d588e0116c20af637f2336579087a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-24T00:24:29.660 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.83 MB/sec warmup 118 sec latency 111251.709 ms 2024-03-24T00:24:29.843 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:29 smithi107 podman[71600]: 2024-03-24 00:24:29.586097794 +0000 UTC m=+0.582165795 container remove 8737b638b692bcfb2a756c65aa074df2f21d588e0116c20af637f2336579087a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True) 2024-03-24T00:24:29.843 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:29 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:24:29.843 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:29 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.i.service: Failed with result 'exit-code'. 2024-03-24T00:24:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:29 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6838/3131891440,v1:172.21.15.107:6839/3131891440] up:rejoin 2024-03-24T00:24:29.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:29 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:rejoin,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:29.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:29 smithi107 ceph-mon[36485]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-24T00:24:29.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:29 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:24:29.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:29 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:24:29.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:29 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6838/3131891440,v1:172.21.15.107:6839/3131891440] up:active 2024-03-24T00:24:29.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:29 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:24:29.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:29 smithi107 ceph-mon[36485]: pgmap v108: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:24:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:29 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6838/3131891440,v1:172.21.15.107:6839/3131891440] up:rejoin 2024-03-24T00:24:30.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:29 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:rejoin,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:30.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:29 smithi078 ceph-mon[32429]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-24T00:24:30.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:29 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:24:30.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:29 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:24:30.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:29 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6838/3131891440,v1:172.21.15.107:6839/3131891440] up:active 2024-03-24T00:24:30.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:29 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:24:30.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:29 smithi078 ceph-mon[32429]: pgmap v108: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:24:30.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:29 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6838/3131891440,v1:172.21.15.107:6839/3131891440] up:rejoin 2024-03-24T00:24:30.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:29 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:rejoin,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:30.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:29 smithi106 ceph-mon[36505]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-24T00:24:30.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:29 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:24:30.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:29 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:24:30.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:29 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6838/3131891440,v1:172.21.15.107:6839/3131891440] up:active 2024-03-24T00:24:30.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:29 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:24:30.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:29 smithi106 ceph-mon[36505]: pgmap v108: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:24:30.652 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 24 00:24:30 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-b[66985]: 2024-03-24T00:24:30.253+0000 7f978f4ee700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-24T00:24:30.660 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.83 MB/sec warmup 119 sec latency 112251.814 ms 2024-03-24T00:24:32.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:32 smithi106 ceph-mon[36505]: pgmap v109: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 2.4 KiB/s wr, 11 op/s 2024-03-24T00:24:32.660 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 1 sec latency 114251.886 ms 2024-03-24T00:24:32.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:32 smithi078 ceph-mon[32429]: pgmap v109: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 2.4 KiB/s wr, 11 op/s 2024-03-24T00:24:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:32 smithi107 ceph-mon[36485]: pgmap v109: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 2.4 KiB/s wr, 11 op/s 2024-03-24T00:24:33.661 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 2 sec latency 115252.017 ms 2024-03-24T00:24:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:34 smithi106 ceph-mon[36505]: pgmap v110: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 2.4 KiB/s wr, 11 op/s 2024-03-24T00:24:34.661 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 3 sec latency 116252.124 ms 2024-03-24T00:24:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:34 smithi078 ceph-mon[32429]: pgmap v110: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 2.4 KiB/s wr, 11 op/s 2024-03-24T00:24:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:34 smithi107 ceph-mon[36485]: pgmap v110: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 2.4 KiB/s wr, 11 op/s 2024-03-24T00:24:35.661 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 4 sec latency 117252.244 ms 2024-03-24T00:24:36.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:36 smithi106 ceph-mon[36505]: pgmap v111: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 2.3 KiB/s wr, 11 op/s 2024-03-24T00:24:36.661 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 5 sec latency 118252.344 ms 2024-03-24T00:24:36.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:36 smithi078 ceph-mon[32429]: pgmap v111: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 2.3 KiB/s wr, 11 op/s 2024-03-24T00:24:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:36 smithi107 ceph-mon[36485]: pgmap v111: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 2.3 KiB/s wr, 11 op/s 2024-03-24T00:24:37.661 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 6 sec latency 119252.434 ms 2024-03-24T00:24:38.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:38 smithi106 ceph-mon[36505]: pgmap v112: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-24T00:24:38.661 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 7 sec latency 120252.561 ms 2024-03-24T00:24:38.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:38 smithi078 ceph-mon[32429]: pgmap v112: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-24T00:24:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:38 smithi107 ceph-mon[36485]: pgmap v112: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-24T00:24:39.661 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 8 sec latency 121252.653 ms 2024-03-24T00:24:40.214 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:39 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.i.service: Scheduled restart job, restart counter is at 3. 2024-03-24T00:24:40.214 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:39 smithi107 systemd[1]: Stopped Ceph mds.i for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:24:40.214 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:39 smithi107 systemd[1]: Starting Ceph mds.i for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:24:40.214 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:40 smithi107 podman[71695]: 2024-03-24 00:24:39.983269777 +0000 UTC m=+0.017960034 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:24:40.214 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:40 smithi107 podman[71695]: 2024-03-24 00:24:40.085865997 +0000 UTC m=+0.120556271 container create 3f1b7b4571971b3d9c936633bee1d64781fcefb1a2ac296cd4b145880ae7f578 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True) 2024-03-24T00:24:40.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:40 smithi107 ceph-mon[36485]: pgmap v113: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 1.9 KiB/s wr, 9 op/s 2024-03-24T00:24:40.593 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:40 smithi107 podman[71695]: 2024-03-24 00:24:40.330202172 +0000 UTC m=+0.364892425 container init 3f1b7b4571971b3d9c936633bee1d64781fcefb1a2ac296cd4b145880ae7f578 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-24T00:24:40.593 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:40 smithi107 podman[71695]: 2024-03-24 00:24:40.332965728 +0000 UTC m=+0.367655977 container start 3f1b7b4571971b3d9c936633bee1d64781fcefb1a2ac296cd4b145880ae7f578 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:24:40.593 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:40 smithi107 ceph-mds[71714]: -- 172.21.15.107:0/3027179479 <== mon.1 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e51140da40 con 0x55e5113e0800 2024-03-24T00:24:40.593 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:40 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: starting mds.i at 2024-03-24T00:24:40.594 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:40 smithi107 bash[71695]: 3f1b7b4571971b3d9c936633bee1d64781fcefb1a2ac296cd4b145880ae7f578 2024-03-24T00:24:40.594 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:40 smithi107 systemd[1]: Started Ceph mds.i for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:24:40.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:40 smithi106 ceph-mon[36505]: pgmap v113: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 1.9 KiB/s wr, 9 op/s 2024-03-24T00:24:40.661 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 9 sec latency 122252.704 ms 2024-03-24T00:24:40.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:40 smithi078 ceph-mon[32429]: pgmap v113: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 1.9 KiB/s wr, 9 op/s 2024-03-24T00:24:41.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:41 smithi106 ceph-mon[36505]: daemon mds.i restarted 2024-03-24T00:24:41.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:41 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:24:41.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:41 smithi106 ceph-mon[36505]: osdmap e100: 12 total, 12 up, 12 in 2024-03-24T00:24:41.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:41 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6838/607202225,v1:172.21.15.107:6839/607202225] up:boot 2024-03-24T00:24:41.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:41 smithi106 ceph-mon[36505]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-24T00:24:41.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:41 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-24T00:24:41.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:41 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T00:24:41.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:41 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:replay,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:41.661 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 10 sec latency 123252.754 ms 2024-03-24T00:24:41.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:41 smithi078 ceph-mon[32429]: daemon mds.i restarted 2024-03-24T00:24:41.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:41 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:24:41.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:41 smithi078 ceph-mon[32429]: osdmap e100: 12 total, 12 up, 12 in 2024-03-24T00:24:41.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:41 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6838/607202225,v1:172.21.15.107:6839/607202225] up:boot 2024-03-24T00:24:41.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:41 smithi078 ceph-mon[32429]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-24T00:24:41.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:41 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-24T00:24:41.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:41 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T00:24:41.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:41 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:replay,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:41.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:41 smithi107 ceph-mon[36485]: daemon mds.i restarted 2024-03-24T00:24:41.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:41 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:24:41.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:41 smithi107 ceph-mon[36485]: osdmap e100: 12 total, 12 up, 12 in 2024-03-24T00:24:41.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:41 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6838/607202225,v1:172.21.15.107:6839/607202225] up:boot 2024-03-24T00:24:41.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:41 smithi107 ceph-mon[36485]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-24T00:24:41.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:41 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-24T00:24:41.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:41 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T00:24:41.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:41 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:replay,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:42.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:42 smithi106 ceph-mon[36505]: pgmap v115: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:24:42.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:42 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6834/23104891,v1:172.21.15.107:6835/23104891] up:resolve 2024-03-24T00:24:42.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:42 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:resolve,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:42.661 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 11 sec latency 124252.814 ms 2024-03-24T00:24:42.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:42 smithi078 ceph-mon[32429]: pgmap v115: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:24:42.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:42 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6834/23104891,v1:172.21.15.107:6835/23104891] up:resolve 2024-03-24T00:24:42.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:42 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:resolve,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:42.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:42 smithi107 ceph-mon[36485]: pgmap v115: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:24:42.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:42 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6834/23104891,v1:172.21.15.107:6835/23104891] up:resolve 2024-03-24T00:24:42.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:42 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:resolve,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:43.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:43 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999983 2024-03-24T00:24:43.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:43 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6834/23104891,v1:172.21.15.107:6835/23104891] up:reconnect 2024-03-24T00:24:43.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:43 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:reconnect,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:43.661 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 12 sec latency 125252.927 ms 2024-03-24T00:24:43.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:43 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999983 2024-03-24T00:24:43.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:43 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6834/23104891,v1:172.21.15.107:6835/23104891] up:reconnect 2024-03-24T00:24:43.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:43 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:reconnect,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999983 2024-03-24T00:24:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6834/23104891,v1:172.21.15.107:6835/23104891] up:reconnect 2024-03-24T00:24:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:reconnect,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:44.325 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2303224700 time 2024-03-24T00:24:43.972887+0000 2024-03-24T00:24:44.325 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:44.325 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:44.325 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f230d84e4a1] 2024-03-24T00:24:44.325 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f230d84e667] 2024-03-24T00:24:44.326 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55acfcc46223] 2024-03-24T00:24:44.326 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55acfcac05d6] 2024-03-24T00:24:44.326 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55acfcac0da1] 2024-03-24T00:24:44.326 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55acfcac32d5] 2024-03-24T00:24:44.326 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55acfcb13233] 2024-03-24T00:24:44.326 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55acfcb15197] 2024-03-24T00:24:44.326 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55acfca1411b] 2024-03-24T00:24:44.326 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55acfca146bc] 2024-03-24T00:24:44.326 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55acfc9fe1ff] 2024-03-24T00:24:44.326 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f230dae0488] 2024-03-24T00:24:44.326 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 13: (DispatchQueue::entry()+0x51f) [0x7f230dadd60f] 2024-03-24T00:24:44.326 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f230dbb0e81] 2024-03-24T00:24:44.326 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f230c5cb1ca] 2024-03-24T00:24:44.327 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 16: clone() 2024-03-24T00:24:44.327 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 2024-03-24T00:24:43.973+0000 7f2303224700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2303224700 time 2024-03-24T00:24:43.972887+0000 2024-03-24T00:24:44.328 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:44.328 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 2024-03-24T00:24:44.328 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:44.328 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f230d84e4a1] 2024-03-24T00:24:44.328 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f230d84e667] 2024-03-24T00:24:44.328 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55acfcc46223] 2024-03-24T00:24:44.328 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55acfcac05d6] 2024-03-24T00:24:44.328 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55acfcac0da1] 2024-03-24T00:24:44.328 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55acfcac32d5] 2024-03-24T00:24:44.328 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55acfcb13233] 2024-03-24T00:24:44.328 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55acfcb15197] 2024-03-24T00:24:44.328 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55acfca1411b] 2024-03-24T00:24:44.329 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55acfca146bc] 2024-03-24T00:24:44.329 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55acfc9fe1ff] 2024-03-24T00:24:44.329 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f230dae0488] 2024-03-24T00:24:44.329 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 13: (DispatchQueue::entry()+0x51f) [0x7f230dadd60f] 2024-03-24T00:24:44.329 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f230dbb0e81] 2024-03-24T00:24:44.329 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f230c5cb1ca] 2024-03-24T00:24:44.329 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 16: clone() 2024-03-24T00:24:44.329 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 2024-03-24T00:24:44.329 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: *** Caught signal (Aborted) ** 2024-03-24T00:24:44.329 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: in thread 7f2303224700 thread_name:ms_dispatch 2024-03-24T00:24:44.329 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:44.329 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f230c5d5d20] 2024-03-24T00:24:44.329 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 2: gsignal() 2024-03-24T00:24:44.329 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 3: abort() 2024-03-24T00:24:44.330 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f230d84e4fb] 2024-03-24T00:24:44.330 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f230d84e667] 2024-03-24T00:24:44.330 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55acfcc46223] 2024-03-24T00:24:44.330 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55acfcac05d6] 2024-03-24T00:24:44.330 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55acfcac0da1] 2024-03-24T00:24:44.330 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55acfcac32d5] 2024-03-24T00:24:44.330 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55acfcb13233] 2024-03-24T00:24:44.330 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55acfcb15197] 2024-03-24T00:24:44.330 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55acfca1411b] 2024-03-24T00:24:44.330 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55acfca146bc] 2024-03-24T00:24:44.330 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55acfc9fe1ff] 2024-03-24T00:24:44.331 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f230dae0488] 2024-03-24T00:24:44.331 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 16: (DispatchQueue::entry()+0x51f) [0x7f230dadd60f] 2024-03-24T00:24:44.331 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f230dbb0e81] 2024-03-24T00:24:44.331 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f230c5cb1ca] 2024-03-24T00:24:44.331 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 19: clone() 2024-03-24T00:24:44.331 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 2024-03-24T00:24:43.975+0000 7f2303224700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:24:44.331 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: in thread 7f2303224700 thread_name:ms_dispatch 2024-03-24T00:24:44.331 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 2024-03-24T00:24:44.331 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:44.331 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f230c5d5d20] 2024-03-24T00:24:44.331 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 2: gsignal() 2024-03-24T00:24:44.331 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 3: abort() 2024-03-24T00:24:44.331 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f230d84e4fb] 2024-03-24T00:24:44.331 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f230d84e667] 2024-03-24T00:24:44.332 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55acfcc46223] 2024-03-24T00:24:44.332 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55acfcac05d6] 2024-03-24T00:24:44.332 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55acfcac0da1] 2024-03-24T00:24:44.332 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55acfcac32d5] 2024-03-24T00:24:44.332 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55acfcb13233] 2024-03-24T00:24:44.332 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55acfcb15197] 2024-03-24T00:24:44.332 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55acfca1411b] 2024-03-24T00:24:44.332 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55acfca146bc] 2024-03-24T00:24:44.332 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55acfc9fe1ff] 2024-03-24T00:24:44.332 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f230dae0488] 2024-03-24T00:24:44.332 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 16: (DispatchQueue::entry()+0x51f) [0x7f230dadd60f] 2024-03-24T00:24:44.332 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f230dbb0e81] 2024-03-24T00:24:44.332 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f230c5cb1ca] 2024-03-24T00:24:44.332 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 19: clone() 2024-03-24T00:24:44.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:24:44.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 2024-03-24T00:24:44.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: -1> 2024-03-24T00:24:43.973+0000 7f2303224700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2303224700 time 2024-03-24T00:24:43.972887+0000 2024-03-24T00:24:44.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:44.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 2024-03-24T00:24:44.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:44.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f230d84e4a1] 2024-03-24T00:24:44.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f230d84e667] 2024-03-24T00:24:44.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55acfcc46223] 2024-03-24T00:24:44.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55acfcac05d6] 2024-03-24T00:24:44.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55acfcac0da1] 2024-03-24T00:24:44.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55acfcac32d5] 2024-03-24T00:24:44.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55acfcb13233] 2024-03-24T00:24:44.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55acfcb15197] 2024-03-24T00:24:44.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55acfca1411b] 2024-03-24T00:24:44.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55acfca146bc] 2024-03-24T00:24:44.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55acfc9fe1ff] 2024-03-24T00:24:44.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f230dae0488] 2024-03-24T00:24:44.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 13: (DispatchQueue::entry()+0x51f) [0x7f230dadd60f] 2024-03-24T00:24:44.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f230dbb0e81] 2024-03-24T00:24:44.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f230c5cb1ca] 2024-03-24T00:24:44.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 16: clone() 2024-03-24T00:24:44.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 2024-03-24T00:24:44.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 0> 2024-03-24T00:24:43.975+0000 7f2303224700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:24:44.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: in thread 7f2303224700 thread_name:ms_dispatch 2024-03-24T00:24:44.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 2024-03-24T00:24:44.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:44.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f230c5d5d20] 2024-03-24T00:24:44.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 2: gsignal() 2024-03-24T00:24:44.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 3: abort() 2024-03-24T00:24:44.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f230d84e4fb] 2024-03-24T00:24:44.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f230d84e667] 2024-03-24T00:24:44.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55acfcc46223] 2024-03-24T00:24:44.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55acfcac05d6] 2024-03-24T00:24:44.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55acfcac0da1] 2024-03-24T00:24:44.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55acfcac32d5] 2024-03-24T00:24:44.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55acfcb13233] 2024-03-24T00:24:44.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55acfcb15197] 2024-03-24T00:24:44.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55acfca1411b] 2024-03-24T00:24:44.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55acfca146bc] 2024-03-24T00:24:44.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55acfc9fe1ff] 2024-03-24T00:24:44.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f230dae0488] 2024-03-24T00:24:44.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 16: (DispatchQueue::entry()+0x51f) [0x7f230dadd60f] 2024-03-24T00:24:44.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f230dbb0e81] 2024-03-24T00:24:44.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f230c5cb1ca] 2024-03-24T00:24:44.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 19: clone() 2024-03-24T00:24:44.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:24:44.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 2024-03-24T00:24:44.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: -1> 2024-03-24T00:24:43.973+0000 7f2303224700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2303224700 time 2024-03-24T00:24:43.972887+0000 2024-03-24T00:24:44.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:44.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 2024-03-24T00:24:44.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:44.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f230d84e4a1] 2024-03-24T00:24:44.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f230d84e667] 2024-03-24T00:24:44.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55acfcc46223] 2024-03-24T00:24:44.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55acfcac05d6] 2024-03-24T00:24:44.337 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55acfcac0da1] 2024-03-24T00:24:44.337 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55acfcac32d5] 2024-03-24T00:24:44.337 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55acfcb13233] 2024-03-24T00:24:44.337 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55acfcb15197] 2024-03-24T00:24:44.337 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55acfca1411b] 2024-03-24T00:24:44.337 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55acfca146bc] 2024-03-24T00:24:44.337 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55acfc9fe1ff] 2024-03-24T00:24:44.337 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f230dae0488] 2024-03-24T00:24:44.337 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 13: (DispatchQueue::entry()+0x51f) [0x7f230dadd60f] 2024-03-24T00:24:44.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f230dbb0e81] 2024-03-24T00:24:44.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f230c5cb1ca] 2024-03-24T00:24:44.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 16: clone() 2024-03-24T00:24:44.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 2024-03-24T00:24:44.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 0> 2024-03-24T00:24:43.975+0000 7f2303224700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:24:44.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: in thread 7f2303224700 thread_name:ms_dispatch 2024-03-24T00:24:44.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 2024-03-24T00:24:44.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:44.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f230c5d5d20] 2024-03-24T00:24:44.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 2: gsignal() 2024-03-24T00:24:44.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 3: abort() 2024-03-24T00:24:44.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f230d84e4fb] 2024-03-24T00:24:44.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f230d84e667] 2024-03-24T00:24:44.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55acfcc46223] 2024-03-24T00:24:44.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55acfcac05d6] 2024-03-24T00:24:44.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55acfcac0da1] 2024-03-24T00:24:44.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55acfcac32d5] 2024-03-24T00:24:44.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55acfcb13233] 2024-03-24T00:24:44.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55acfcb15197] 2024-03-24T00:24:44.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55acfca1411b] 2024-03-24T00:24:44.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55acfca146bc] 2024-03-24T00:24:44.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55acfc9fe1ff] 2024-03-24T00:24:44.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f230dae0488] 2024-03-24T00:24:44.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 16: (DispatchQueue::entry()+0x51f) [0x7f230dadd60f] 2024-03-24T00:24:44.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f230dbb0e81] 2024-03-24T00:24:44.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f230c5cb1ca] 2024-03-24T00:24:44.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 19: clone() 2024-03-24T00:24:44.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:24:44.340 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71562]: 2024-03-24T00:24:44.340 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:44 smithi107 podman[71748]: 2024-03-24 00:24:44.021546578 +0000 UTC m=+0.020223862 container died ef5cc80c6b13b90d304f79d58ecbcf4d413c97122058e6981a1d8ab2b1d4fcc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-24T00:24:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:44 smithi107 ceph-mon[36485]: pgmap v116: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:24:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:44 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6834/23104891,v1:172.21.15.107:6835/23104891] up:rejoin 2024-03-24T00:24:44.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:44 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:rejoin,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:44.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:44 smithi107 ceph-mon[36485]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-24T00:24:44.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:44 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:24:44.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:44 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:24:44.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:44 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6834/23104891,v1:172.21.15.107:6835/23104891] up:active 2024-03-24T00:24:44.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:44 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:24:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:44 smithi106 ceph-mon[36505]: pgmap v116: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:24:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:44 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6834/23104891,v1:172.21.15.107:6835/23104891] up:rejoin 2024-03-24T00:24:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:44 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:rejoin,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:44 smithi106 ceph-mon[36505]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-24T00:24:44.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:44 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:24:44.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:44 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:24:44.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:44 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6834/23104891,v1:172.21.15.107:6835/23104891] up:active 2024-03-24T00:24:44.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:44 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:24:44.662 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 13 sec latency 126253.032 ms 2024-03-24T00:24:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:44 smithi078 ceph-mon[32429]: pgmap v116: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:24:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:44 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6834/23104891,v1:172.21.15.107:6835/23104891] up:rejoin 2024-03-24T00:24:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:44 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:rejoin,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:44.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:44 smithi078 ceph-mon[32429]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-24T00:24:44.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:44 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:24:44.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:44 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:24:44.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:44 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6834/23104891,v1:172.21.15.107:6835/23104891] up:active 2024-03-24T00:24:44.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:44 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:24:45.093 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:44 smithi107 podman[71748]: 2024-03-24 00:24:44.710154947 +0000 UTC m=+0.708832222 container remove ef5cc80c6b13b90d304f79d58ecbcf4d413c97122058e6981a1d8ab2b1d4fcc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-24T00:24:45.093 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:44 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:24:45.093 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:44 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.f.service: Failed with result 'exit-code'. 2024-03-24T00:24:45.652 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 24 00:24:45 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-b[66985]: 2024-03-24T00:24:45.252+0000 7f978f4ee700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-24T00:24:45.662 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 14 sec latency 127253.080 ms 2024-03-24T00:24:46.662 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 15 sec latency 128253.149 ms 2024-03-24T00:24:46.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:46 smithi078 ceph-mon[32429]: pgmap v117: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-03-24T00:24:46.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:46 smithi107 ceph-mon[36485]: pgmap v117: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-03-24T00:24:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:46 smithi106 ceph-mon[36505]: pgmap v117: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-03-24T00:24:47.662 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 16 sec latency 129253.264 ms 2024-03-24T00:24:48.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:48 smithi078 ceph-mon[32429]: pgmap v118: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-03-24T00:24:48.662 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 17 sec latency 130253.321 ms 2024-03-24T00:24:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:48 smithi107 ceph-mon[36485]: pgmap v118: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-03-24T00:24:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:48 smithi106 ceph-mon[36505]: pgmap v118: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-03-24T00:24:49.662 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 18 sec latency 131253.452 ms 2024-03-24T00:24:50.662 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 19 sec latency 132253.543 ms 2024-03-24T00:24:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:50 smithi078 ceph-mon[32429]: pgmap v119: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-03-24T00:24:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:50 smithi107 ceph-mon[36485]: pgmap v119: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-03-24T00:24:50.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:50 smithi106 ceph-mon[36505]: pgmap v119: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-03-24T00:24:51.662 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 20 sec latency 133253.591 ms 2024-03-24T00:24:52.662 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 21 sec latency 134253.703 ms 2024-03-24T00:24:52.704 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:52 smithi106 ceph-mon[36505]: pgmap v120: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-03-24T00:24:52.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:52 smithi078 ceph-mon[32429]: pgmap v120: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-03-24T00:24:52.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:52 smithi107 ceph-mon[36485]: pgmap v120: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-03-24T00:24:53.662 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 22 sec latency 135253.759 ms 2024-03-24T00:24:54.663 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 23 sec latency 136253.860 ms 2024-03-24T00:24:54.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:54 smithi078 ceph-mon[32429]: pgmap v121: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-24T00:24:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:54 smithi107 ceph-mon[36485]: pgmap v121: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-24T00:24:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:54 smithi106 ceph-mon[36505]: pgmap v121: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-24T00:24:55.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:55 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.f.service: Scheduled restart job, restart counter is at 3. 2024-03-24T00:24:55.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:55 smithi107 systemd[1]: Stopped Ceph mds.f for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:24:55.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:55 smithi107 systemd[1]: Starting Ceph mds.f for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:24:55.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:55 smithi107 podman[71847]: 2024-03-24 00:24:55.232609672 +0000 UTC m=+0.017750366 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:24:55.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:55 smithi107 podman[71847]: 2024-03-24 00:24:55.35188628 +0000 UTC m=+0.137026959 container create abeffabf64f52ce13a2c9f9ab61ea7e8db058b592598dd0f6817ed60598dcfe7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:24:55.663 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 24 sec latency 137253.954 ms 2024-03-24T00:24:56.094 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:55 smithi107 podman[71847]: 2024-03-24 00:24:55.604295388 +0000 UTC m=+0.389436074 container init abeffabf64f52ce13a2c9f9ab61ea7e8db058b592598dd0f6817ed60598dcfe7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-24T00:24:56.095 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:55 smithi107 podman[71847]: 2024-03-24 00:24:55.606835568 +0000 UTC m=+0.391976257 container start abeffabf64f52ce13a2c9f9ab61ea7e8db058b592598dd0f6817ed60598dcfe7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:24:56.095 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:55 smithi107 ceph-mds[71865]: -- 172.21.15.107:0/1624017334 <== mon.1 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5644e12bfa40 con 0x5644e1293000 2024-03-24T00:24:56.095 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: starting mds.f at 2024-03-24T00:24:56.095 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:55 smithi107 bash[71847]: abeffabf64f52ce13a2c9f9ab61ea7e8db058b592598dd0f6817ed60598dcfe7 2024-03-24T00:24:56.095 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:24:55 smithi107 systemd[1]: Started Ceph mds.f for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:24:56.663 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 25 sec latency 138254.005 ms 2024-03-24T00:24:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:56 smithi078 ceph-mon[32429]: pgmap v122: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-24T00:24:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:56 smithi078 ceph-mon[32429]: daemon mds.f restarted 2024-03-24T00:24:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:56 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:24:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:56 smithi078 ceph-mon[32429]: osdmap e101: 12 total, 12 up, 12 in 2024-03-24T00:24:56.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:56 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6834/2950120609,v1:172.21.15.107:6835/2950120609] up:boot 2024-03-24T00:24:56.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:56 smithi078 ceph-mon[32429]: Standby daemon mds.i assigned to filesystem cephfs as rank 1 2024-03-24T00:24:56.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:56 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-24T00:24:56.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:24:56.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:56 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:replay,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:56 smithi107 ceph-mon[36485]: pgmap v122: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-24T00:24:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:56 smithi107 ceph-mon[36485]: daemon mds.f restarted 2024-03-24T00:24:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:56 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:24:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:56 smithi107 ceph-mon[36485]: osdmap e101: 12 total, 12 up, 12 in 2024-03-24T00:24:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:56 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6834/2950120609,v1:172.21.15.107:6835/2950120609] up:boot 2024-03-24T00:24:56.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:56 smithi107 ceph-mon[36485]: Standby daemon mds.i assigned to filesystem cephfs as rank 1 2024-03-24T00:24:56.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:56 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-24T00:24:56.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:24:56.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:56 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:replay,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:56.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:56 smithi106 ceph-mon[36505]: pgmap v122: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-24T00:24:56.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:56 smithi106 ceph-mon[36505]: daemon mds.f restarted 2024-03-24T00:24:56.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:56 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:24:56.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:56 smithi106 ceph-mon[36505]: osdmap e101: 12 total, 12 up, 12 in 2024-03-24T00:24:56.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:56 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6834/2950120609,v1:172.21.15.107:6835/2950120609] up:boot 2024-03-24T00:24:56.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:56 smithi106 ceph-mon[36505]: Standby daemon mds.i assigned to filesystem cephfs as rank 1 2024-03-24T00:24:56.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:56 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-24T00:24:56.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:24:56.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:56 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:replay,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:57.666 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 26 sec latency 139257.397 ms 2024-03-24T00:24:58.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:57 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6838/607202225,v1:172.21.15.107:6839/607202225] up:resolve 2024-03-24T00:24:58.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:57 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:resolve,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:58.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:57 smithi078 ceph-mon[32429]: pgmap v124: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:24:58.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:57 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6838/607202225,v1:172.21.15.107:6839/607202225] up:resolve 2024-03-24T00:24:58.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:57 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:resolve,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:58.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:57 smithi107 ceph-mon[36485]: pgmap v124: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:24:58.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:57 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6838/607202225,v1:172.21.15.107:6839/607202225] up:resolve 2024-03-24T00:24:58.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:57 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:resolve,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:58.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:57 smithi106 ceph-mon[36505]: pgmap v124: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:24:58.666 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 27 sec latency 140257.437 ms 2024-03-24T00:24:58.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:58 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:24:58.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:58 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6838/607202225,v1:172.21.15.107:6839/607202225] up:reconnect 2024-03-24T00:24:58.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:58 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:reconnect,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:58.977 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:24:58.977 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6838/607202225,v1:172.21.15.107:6839/607202225] up:reconnect 2024-03-24T00:24:58.977 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:reconnect,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:58 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:24:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:58 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6838/607202225,v1:172.21.15.107:6839/607202225] up:reconnect 2024-03-24T00:24:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:58 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:reconnect,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:59.345 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa38f774700 time 2024-03-24T00:24:58.978226+0000 2024-03-24T00:24:59.345 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:59.345 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:59.346 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa399d9e4a1] 2024-03-24T00:24:59.346 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa399d9e667] 2024-03-24T00:24:59.346 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e50ef66223] 2024-03-24T00:24:59.346 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e50ede05d6] 2024-03-24T00:24:59.346 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e50ede0da1] 2024-03-24T00:24:59.346 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e50ede32d5] 2024-03-24T00:24:59.346 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e50ee33233] 2024-03-24T00:24:59.346 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e50ee35197] 2024-03-24T00:24:59.346 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e50ed3411b] 2024-03-24T00:24:59.346 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e50ed346bc] 2024-03-24T00:24:59.347 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e50ed1e1ff] 2024-03-24T00:24:59.347 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa39a030488] 2024-03-24T00:24:59.347 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 13: (DispatchQueue::entry()+0x51f) [0x7fa39a02d60f] 2024-03-24T00:24:59.347 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa39a100e81] 2024-03-24T00:24:59.347 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa398b1b1ca] 2024-03-24T00:24:59.347 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 16: clone() 2024-03-24T00:24:59.347 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 2024-03-24T00:24:58.979+0000 7fa38f774700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa38f774700 time 2024-03-24T00:24:58.978226+0000 2024-03-24T00:24:59.347 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:59.347 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 2024-03-24T00:24:59.347 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:59.347 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa399d9e4a1] 2024-03-24T00:24:59.348 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa399d9e667] 2024-03-24T00:24:59.348 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e50ef66223] 2024-03-24T00:24:59.348 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e50ede05d6] 2024-03-24T00:24:59.348 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e50ede0da1] 2024-03-24T00:24:59.348 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e50ede32d5] 2024-03-24T00:24:59.348 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e50ee33233] 2024-03-24T00:24:59.348 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e50ee35197] 2024-03-24T00:24:59.348 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e50ed3411b] 2024-03-24T00:24:59.348 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e50ed346bc] 2024-03-24T00:24:59.348 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e50ed1e1ff] 2024-03-24T00:24:59.348 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa39a030488] 2024-03-24T00:24:59.348 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 13: (DispatchQueue::entry()+0x51f) [0x7fa39a02d60f] 2024-03-24T00:24:59.348 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa39a100e81] 2024-03-24T00:24:59.349 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa398b1b1ca] 2024-03-24T00:24:59.349 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 16: clone() 2024-03-24T00:24:59.349 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 2024-03-24T00:24:59.349 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: *** Caught signal (Aborted) ** 2024-03-24T00:24:59.349 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: in thread 7fa38f774700 thread_name:ms_dispatch 2024-03-24T00:24:59.349 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:59.349 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa398b25d20] 2024-03-24T00:24:59.349 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 2: gsignal() 2024-03-24T00:24:59.349 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 3: abort() 2024-03-24T00:24:59.349 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa399d9e4fb] 2024-03-24T00:24:59.349 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa399d9e667] 2024-03-24T00:24:59.349 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e50ef66223] 2024-03-24T00:24:59.350 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e50ede05d6] 2024-03-24T00:24:59.350 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e50ede0da1] 2024-03-24T00:24:59.350 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e50ede32d5] 2024-03-24T00:24:59.350 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e50ee33233] 2024-03-24T00:24:59.350 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e50ee35197] 2024-03-24T00:24:59.350 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e50ed3411b] 2024-03-24T00:24:59.350 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e50ed346bc] 2024-03-24T00:24:59.350 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e50ed1e1ff] 2024-03-24T00:24:59.350 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa39a030488] 2024-03-24T00:24:59.350 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 16: (DispatchQueue::entry()+0x51f) [0x7fa39a02d60f] 2024-03-24T00:24:59.350 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa39a100e81] 2024-03-24T00:24:59.350 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa398b1b1ca] 2024-03-24T00:24:59.351 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 19: clone() 2024-03-24T00:24:59.351 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 2024-03-24T00:24:58.980+0000 7fa38f774700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:24:59.351 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: in thread 7fa38f774700 thread_name:ms_dispatch 2024-03-24T00:24:59.351 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 2024-03-24T00:24:59.351 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:59.351 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa398b25d20] 2024-03-24T00:24:59.351 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 2: gsignal() 2024-03-24T00:24:59.351 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 3: abort() 2024-03-24T00:24:59.351 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa399d9e4fb] 2024-03-24T00:24:59.351 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa399d9e667] 2024-03-24T00:24:59.352 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e50ef66223] 2024-03-24T00:24:59.352 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e50ede05d6] 2024-03-24T00:24:59.353 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e50ede0da1] 2024-03-24T00:24:59.353 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e50ede32d5] 2024-03-24T00:24:59.353 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e50ee33233] 2024-03-24T00:24:59.353 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e50ee35197] 2024-03-24T00:24:59.353 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e50ed3411b] 2024-03-24T00:24:59.353 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e50ed346bc] 2024-03-24T00:24:59.353 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e50ed1e1ff] 2024-03-24T00:24:59.353 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa39a030488] 2024-03-24T00:24:59.353 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 16: (DispatchQueue::entry()+0x51f) [0x7fa39a02d60f] 2024-03-24T00:24:59.353 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa39a100e81] 2024-03-24T00:24:59.353 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa398b1b1ca] 2024-03-24T00:24:59.354 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 19: clone() 2024-03-24T00:24:59.354 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:24:59.354 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 2024-03-24T00:24:59.354 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: -1> 2024-03-24T00:24:58.979+0000 7fa38f774700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa38f774700 time 2024-03-24T00:24:58.978226+0000 2024-03-24T00:24:59.354 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:59.354 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 2024-03-24T00:24:59.354 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:59.354 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa399d9e4a1] 2024-03-24T00:24:59.354 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa399d9e667] 2024-03-24T00:24:59.354 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e50ef66223] 2024-03-24T00:24:59.354 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e50ede05d6] 2024-03-24T00:24:59.355 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e50ede0da1] 2024-03-24T00:24:59.355 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e50ede32d5] 2024-03-24T00:24:59.355 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e50ee33233] 2024-03-24T00:24:59.355 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e50ee35197] 2024-03-24T00:24:59.355 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e50ed3411b] 2024-03-24T00:24:59.355 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e50ed346bc] 2024-03-24T00:24:59.355 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e50ed1e1ff] 2024-03-24T00:24:59.355 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa39a030488] 2024-03-24T00:24:59.355 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 13: (DispatchQueue::entry()+0x51f) [0x7fa39a02d60f] 2024-03-24T00:24:59.355 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa39a100e81] 2024-03-24T00:24:59.355 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa398b1b1ca] 2024-03-24T00:24:59.355 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 16: clone() 2024-03-24T00:24:59.356 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 2024-03-24T00:24:59.356 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 0> 2024-03-24T00:24:58.980+0000 7fa38f774700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:24:59.356 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: in thread 7fa38f774700 thread_name:ms_dispatch 2024-03-24T00:24:59.356 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 2024-03-24T00:24:59.356 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:59.356 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa398b25d20] 2024-03-24T00:24:59.356 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 2: gsignal() 2024-03-24T00:24:59.356 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 3: abort() 2024-03-24T00:24:59.356 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa399d9e4fb] 2024-03-24T00:24:59.356 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa399d9e667] 2024-03-24T00:24:59.356 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e50ef66223] 2024-03-24T00:24:59.356 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e50ede05d6] 2024-03-24T00:24:59.357 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e50ede0da1] 2024-03-24T00:24:59.357 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e50ede32d5] 2024-03-24T00:24:59.357 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e50ee33233] 2024-03-24T00:24:59.357 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e50ee35197] 2024-03-24T00:24:59.357 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e50ed3411b] 2024-03-24T00:24:59.357 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e50ed346bc] 2024-03-24T00:24:59.357 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e50ed1e1ff] 2024-03-24T00:24:59.357 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa39a030488] 2024-03-24T00:24:59.357 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 16: (DispatchQueue::entry()+0x51f) [0x7fa39a02d60f] 2024-03-24T00:24:59.357 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa39a100e81] 2024-03-24T00:24:59.357 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa398b1b1ca] 2024-03-24T00:24:59.358 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 19: clone() 2024-03-24T00:24:59.358 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:24:59.358 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 2024-03-24T00:24:59.358 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: -1> 2024-03-24T00:24:58.979+0000 7fa38f774700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa38f774700 time 2024-03-24T00:24:58.978226+0000 2024-03-24T00:24:59.358 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:59.358 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 2024-03-24T00:24:59.358 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:59.358 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa399d9e4a1] 2024-03-24T00:24:59.358 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa399d9e667] 2024-03-24T00:24:59.358 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e50ef66223] 2024-03-24T00:24:59.358 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e50ede05d6] 2024-03-24T00:24:59.359 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e50ede0da1] 2024-03-24T00:24:59.359 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e50ede32d5] 2024-03-24T00:24:59.359 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e50ee33233] 2024-03-24T00:24:59.359 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e50ee35197] 2024-03-24T00:24:59.359 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e50ed3411b] 2024-03-24T00:24:59.359 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e50ed346bc] 2024-03-24T00:24:59.359 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e50ed1e1ff] 2024-03-24T00:24:59.359 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa39a030488] 2024-03-24T00:24:59.359 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 13: (DispatchQueue::entry()+0x51f) [0x7fa39a02d60f] 2024-03-24T00:24:59.359 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa39a100e81] 2024-03-24T00:24:59.359 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa398b1b1ca] 2024-03-24T00:24:59.360 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 16: clone() 2024-03-24T00:24:59.360 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 2024-03-24T00:24:59.360 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 0> 2024-03-24T00:24:58.980+0000 7fa38f774700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:24:59.360 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: in thread 7fa38f774700 thread_name:ms_dispatch 2024-03-24T00:24:59.360 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 2024-03-24T00:24:59.360 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:59.360 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa398b25d20] 2024-03-24T00:24:59.360 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 2: gsignal() 2024-03-24T00:24:59.360 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 3: abort() 2024-03-24T00:24:59.360 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa399d9e4fb] 2024-03-24T00:24:59.360 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa399d9e667] 2024-03-24T00:24:59.360 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e50ef66223] 2024-03-24T00:24:59.361 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e50ede05d6] 2024-03-24T00:24:59.361 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e50ede0da1] 2024-03-24T00:24:59.361 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e50ede32d5] 2024-03-24T00:24:59.361 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e50ee33233] 2024-03-24T00:24:59.361 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e50ee35197] 2024-03-24T00:24:59.361 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e50ed3411b] 2024-03-24T00:24:59.361 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e50ed346bc] 2024-03-24T00:24:59.361 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e50ed1e1ff] 2024-03-24T00:24:59.361 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa39a030488] 2024-03-24T00:24:59.361 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 16: (DispatchQueue::entry()+0x51f) [0x7fa39a02d60f] 2024-03-24T00:24:59.361 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa39a100e81] 2024-03-24T00:24:59.362 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa398b1b1ca] 2024-03-24T00:24:59.362 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 19: clone() 2024-03-24T00:24:59.362 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:24:59.362 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[71710]: 2024-03-24T00:24:59.362 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:59 smithi107 podman[71899]: 2024-03-24 00:24:59.025527302 +0000 UTC m=+0.020132025 container died 3f1b7b4571971b3d9c936633bee1d64781fcefb1a2ac296cd4b145880ae7f578 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:24:59.666 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 28 sec latency 141257.517 ms 2024-03-24T00:24:59.734 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:59 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6838/607202225,v1:172.21.15.107:6839/607202225] up:rejoin 2024-03-24T00:24:59.734 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:59 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:rejoin,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:24:59.734 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:59 smithi107 ceph-mon[36485]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-24T00:24:59.734 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:59 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:24:59.734 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:59 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:24:59.734 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:59 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6838/607202225,v1:172.21.15.107:6839/607202225] up:active 2024-03-24T00:24:59.735 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:59 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:24:59.735 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:24:59 smithi107 ceph-mon[36485]: pgmap v125: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:24:59.987 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:59 smithi107 podman[71899]: 2024-03-24 00:24:59.734881739 +0000 UTC m=+0.729486504 container remove 3f1b7b4571971b3d9c936633bee1d64781fcefb1a2ac296cd4b145880ae7f578 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-24T00:24:59.987 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:59 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:25:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:59 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6838/607202225,v1:172.21.15.107:6839/607202225] up:rejoin 2024-03-24T00:25:00.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:59 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:rejoin,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:25:00.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:59 smithi078 ceph-mon[32429]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-24T00:25:00.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:59 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:25:00.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:59 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:25:00.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:59 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6838/607202225,v1:172.21.15.107:6839/607202225] up:active 2024-03-24T00:25:00.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:59 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:25:00.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:24:59 smithi078 ceph-mon[32429]: pgmap v125: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:25:00.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:59 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6838/607202225,v1:172.21.15.107:6839/607202225] up:rejoin 2024-03-24T00:25:00.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:59 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:rejoin,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:25:00.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:59 smithi106 ceph-mon[36505]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-24T00:25:00.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:59 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:25:00.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:59 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:25:00.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:59 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6838/607202225,v1:172.21.15.107:6839/607202225] up:active 2024-03-24T00:25:00.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:59 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:25:00.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:24:59 smithi106 ceph-mon[36505]: pgmap v125: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:25:00.343 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:24:59 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.i.service: Failed with result 'exit-code'. 2024-03-24T00:25:00.652 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 24 00:25:00 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-b[66985]: 2024-03-24T00:25:00.254+0000 7f978f4ee700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-24T00:25:00.666 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 29 sec latency 142257.581 ms 2024-03-24T00:25:01.666 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 30 sec latency 143257.702 ms 2024-03-24T00:25:02.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:02 smithi106 ceph-mon[36505]: pgmap v126: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 204 B/s wr, 12 op/s 2024-03-24T00:25:02.666 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 31 sec latency 144257.784 ms 2024-03-24T00:25:02.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:02 smithi078 ceph-mon[32429]: pgmap v126: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 204 B/s wr, 12 op/s 2024-03-24T00:25:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:02 smithi107 ceph-mon[36485]: pgmap v126: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 204 B/s wr, 12 op/s 2024-03-24T00:25:03.666 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 32 sec latency 145257.863 ms 2024-03-24T00:25:04.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:04 smithi078 ceph-mon[32429]: pgmap v127: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 204 B/s wr, 12 op/s 2024-03-24T00:25:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:04 smithi106 ceph-mon[36505]: pgmap v127: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 204 B/s wr, 12 op/s 2024-03-24T00:25:04.666 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 33 sec latency 146257.922 ms 2024-03-24T00:25:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:04 smithi107 ceph-mon[36485]: pgmap v127: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 204 B/s wr, 12 op/s 2024-03-24T00:25:05.667 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 34 sec latency 147258.047 ms 2024-03-24T00:25:06.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:06 smithi106 ceph-mon[36505]: pgmap v128: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 204 B/s wr, 12 op/s 2024-03-24T00:25:06.667 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 35 sec latency 148258.148 ms 2024-03-24T00:25:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:06 smithi078 ceph-mon[32429]: pgmap v128: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 204 B/s wr, 12 op/s 2024-03-24T00:25:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:06 smithi107 ceph-mon[36485]: pgmap v128: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 204 B/s wr, 12 op/s 2024-03-24T00:25:07.667 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 36 sec latency 149258.263 ms 2024-03-24T00:25:08.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:08 smithi106 ceph-mon[36505]: pgmap v129: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 175 B/s wr, 10 op/s 2024-03-24T00:25:08.667 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 37 sec latency 150258.308 ms 2024-03-24T00:25:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:08 smithi078 ceph-mon[32429]: pgmap v129: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 175 B/s wr, 10 op/s 2024-03-24T00:25:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:08 smithi107 ceph-mon[36485]: pgmap v129: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 175 B/s wr, 10 op/s 2024-03-24T00:25:09.667 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 38 sec latency 151258.427 ms 2024-03-24T00:25:10.350 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:10 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.i.service: Scheduled restart job, restart counter is at 4. 2024-03-24T00:25:10.350 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:10 smithi107 systemd[1]: Stopped Ceph mds.i for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:25:10.350 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:10 smithi107 systemd[1]: Starting Ceph mds.i for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:25:10.350 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:10 smithi107 podman[71995]: 2024-03-24 00:25:10.227692771 +0000 UTC m=+0.018240851 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:25:10.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:10 smithi107 ceph-mon[36485]: pgmap v130: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-24T00:25:10.611 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:10 smithi107 podman[71995]: 2024-03-24 00:25:10.351183549 +0000 UTC m=+0.141731623 container create c64578d01d47a7bc8e8718f340492e07eaf99a6fbf81527e49ae03e23a061f31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True) 2024-03-24T00:25:10.611 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:10 smithi107 podman[71995]: 2024-03-24 00:25:10.586928436 +0000 UTC m=+0.377476529 container init c64578d01d47a7bc8e8718f340492e07eaf99a6fbf81527e49ae03e23a061f31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-24T00:25:10.611 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:10 smithi107 podman[71995]: 2024-03-24 00:25:10.5899688 +0000 UTC m=+0.380516874 container start c64578d01d47a7bc8e8718f340492e07eaf99a6fbf81527e49ae03e23a061f31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-24T00:25:10.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:10 smithi106 ceph-mon[36505]: pgmap v130: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-24T00:25:10.667 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 39 sec latency 152258.534 ms 2024-03-24T00:25:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:10 smithi078 ceph-mon[32429]: pgmap v130: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-24T00:25:11.093 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:10 smithi107 ceph-mds[72014]: -- 172.21.15.107:0/2854711891 <== mon.1 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563530bdda40 con 0x563530bb1000 2024-03-24T00:25:11.093 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: starting mds.i at 2024-03-24T00:25:11.093 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:10 smithi107 bash[71995]: c64578d01d47a7bc8e8718f340492e07eaf99a6fbf81527e49ae03e23a061f31 2024-03-24T00:25:11.093 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:10 smithi107 systemd[1]: Started Ceph mds.i for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:25:11.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:11 smithi106 ceph-mon[36505]: daemon mds.i restarted 2024-03-24T00:25:11.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:11 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:25:11.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:11 smithi106 ceph-mon[36505]: osdmap e102: 12 total, 12 up, 12 in 2024-03-24T00:25:11.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:11 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6838/3949153161,v1:172.21.15.107:6839/3949153161] up:boot 2024-03-24T00:25:11.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:11 smithi106 ceph-mon[36505]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-24T00:25:11.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:11 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-24T00:25:11.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:11 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T00:25:11.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:11 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:replay,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:25:11.667 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 40 sec latency 153258.638 ms 2024-03-24T00:25:11.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:11 smithi078 ceph-mon[32429]: daemon mds.i restarted 2024-03-24T00:25:11.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:11 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:25:11.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:11 smithi078 ceph-mon[32429]: osdmap e102: 12 total, 12 up, 12 in 2024-03-24T00:25:11.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:11 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6838/3949153161,v1:172.21.15.107:6839/3949153161] up:boot 2024-03-24T00:25:11.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:11 smithi078 ceph-mon[32429]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-24T00:25:11.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:11 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-24T00:25:11.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:11 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T00:25:11.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:11 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:replay,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:25:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:11 smithi107 ceph-mon[36485]: daemon mds.i restarted 2024-03-24T00:25:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:11 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:25:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:11 smithi107 ceph-mon[36485]: osdmap e102: 12 total, 12 up, 12 in 2024-03-24T00:25:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:11 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6838/3949153161,v1:172.21.15.107:6839/3949153161] up:boot 2024-03-24T00:25:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:11 smithi107 ceph-mon[36485]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-24T00:25:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:11 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-24T00:25:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:11 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T00:25:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:11 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:replay,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:25:12.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:12 smithi078 ceph-mon[32429]: pgmap v132: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:25:12.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:12 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6834/2950120609,v1:172.21.15.107:6835/2950120609] up:resolve 2024-03-24T00:25:12.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:12 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:resolve,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:25:12.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:12 smithi106 ceph-mon[36505]: pgmap v132: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:25:12.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:12 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6834/2950120609,v1:172.21.15.107:6835/2950120609] up:resolve 2024-03-24T00:25:12.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:12 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:resolve,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:25:12.667 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 41 sec latency 154258.734 ms 2024-03-24T00:25:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:12 smithi107 ceph-mon[36485]: pgmap v132: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:25:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:12 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6834/2950120609,v1:172.21.15.107:6835/2950120609] up:resolve 2024-03-24T00:25:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:12 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:resolve,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:25:13.667 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 42 sec latency 155258.856 ms 2024-03-24T00:25:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:13 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6834/2950120609,v1:172.21.15.107:6835/2950120609] up:reconnect 2024-03-24T00:25:13.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:13 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:reconnect,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:25:13.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:13 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.00199997 2024-03-24T00:25:13.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:13 smithi106 ceph-mon[36505]: pgmap v133: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:25:13.966 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:13 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6834/2950120609,v1:172.21.15.107:6835/2950120609] up:reconnect 2024-03-24T00:25:13.966 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:13 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:reconnect,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:25:13.967 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:13 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.00199997 2024-03-24T00:25:13.967 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:13 smithi078 ceph-mon[32429]: pgmap v133: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:25:13.981 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6834/2950120609,v1:172.21.15.107:6835/2950120609] up:reconnect 2024-03-24T00:25:13.982 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:reconnect,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:25:13.982 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.00199997 2024-03-24T00:25:13.982 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-mon[36485]: pgmap v133: 129 pgs: 129 active+clean; 27 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:25:14.343 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fc2623ce700 time 2024-03-24T00:25:13.982544+0000 2024-03-24T00:25:14.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:25:14.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:14.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fc26c9f84a1] 2024-03-24T00:25:14.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc26c9f8667] 2024-03-24T00:25:14.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5644de781223] 2024-03-24T00:25:14.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5644de5fb5d6] 2024-03-24T00:25:14.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5644de5fbda1] 2024-03-24T00:25:14.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5644de5fe2d5] 2024-03-24T00:25:14.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5644de64e233] 2024-03-24T00:25:14.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5644de650197] 2024-03-24T00:25:14.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5644de54f11b] 2024-03-24T00:25:14.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5644de54f6bc] 2024-03-24T00:25:14.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5644de5391ff] 2024-03-24T00:25:14.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc26cc8a488] 2024-03-24T00:25:14.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 13: (DispatchQueue::entry()+0x51f) [0x7fc26cc8760f] 2024-03-24T00:25:14.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc26cd5ae81] 2024-03-24T00:25:14.346 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc26b7751ca] 2024-03-24T00:25:14.346 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 16: clone() 2024-03-24T00:25:14.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 2024-03-24T00:25:13.983+0000 7fc2623ce700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fc2623ce700 time 2024-03-24T00:25:13.982544+0000 2024-03-24T00:25:14.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:25:14.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 2024-03-24T00:25:14.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:14.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fc26c9f84a1] 2024-03-24T00:25:14.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc26c9f8667] 2024-03-24T00:25:14.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5644de781223] 2024-03-24T00:25:14.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5644de5fb5d6] 2024-03-24T00:25:14.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5644de5fbda1] 2024-03-24T00:25:14.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5644de5fe2d5] 2024-03-24T00:25:14.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5644de64e233] 2024-03-24T00:25:14.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5644de650197] 2024-03-24T00:25:14.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5644de54f11b] 2024-03-24T00:25:14.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5644de54f6bc] 2024-03-24T00:25:14.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5644de5391ff] 2024-03-24T00:25:14.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc26cc8a488] 2024-03-24T00:25:14.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 13: (DispatchQueue::entry()+0x51f) [0x7fc26cc8760f] 2024-03-24T00:25:14.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc26cd5ae81] 2024-03-24T00:25:14.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc26b7751ca] 2024-03-24T00:25:14.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 16: clone() 2024-03-24T00:25:14.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 2024-03-24T00:25:14.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: *** Caught signal (Aborted) ** 2024-03-24T00:25:14.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: in thread 7fc2623ce700 thread_name:ms_dispatch 2024-03-24T00:25:14.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:14.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc26b77fd20] 2024-03-24T00:25:14.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 2: gsignal() 2024-03-24T00:25:14.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 3: abort() 2024-03-24T00:25:14.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fc26c9f84fb] 2024-03-24T00:25:14.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc26c9f8667] 2024-03-24T00:25:14.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5644de781223] 2024-03-24T00:25:14.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5644de5fb5d6] 2024-03-24T00:25:14.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5644de5fbda1] 2024-03-24T00:25:14.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5644de5fe2d5] 2024-03-24T00:25:14.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5644de64e233] 2024-03-24T00:25:14.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5644de650197] 2024-03-24T00:25:14.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5644de54f11b] 2024-03-24T00:25:14.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5644de54f6bc] 2024-03-24T00:25:14.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5644de5391ff] 2024-03-24T00:25:14.352 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc26cc8a488] 2024-03-24T00:25:14.352 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 16: (DispatchQueue::entry()+0x51f) [0x7fc26cc8760f] 2024-03-24T00:25:14.352 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc26cd5ae81] 2024-03-24T00:25:14.352 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc26b7751ca] 2024-03-24T00:25:14.352 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 19: clone() 2024-03-24T00:25:14.352 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 2024-03-24T00:25:13.985+0000 7fc2623ce700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:25:14.352 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: in thread 7fc2623ce700 thread_name:ms_dispatch 2024-03-24T00:25:14.352 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 2024-03-24T00:25:14.352 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:14.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc26b77fd20] 2024-03-24T00:25:14.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 2: gsignal() 2024-03-24T00:25:14.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 3: abort() 2024-03-24T00:25:14.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fc26c9f84fb] 2024-03-24T00:25:14.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc26c9f8667] 2024-03-24T00:25:14.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5644de781223] 2024-03-24T00:25:14.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5644de5fb5d6] 2024-03-24T00:25:14.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5644de5fbda1] 2024-03-24T00:25:14.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5644de5fe2d5] 2024-03-24T00:25:14.354 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5644de64e233] 2024-03-24T00:25:14.354 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5644de650197] 2024-03-24T00:25:14.354 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5644de54f11b] 2024-03-24T00:25:14.354 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5644de54f6bc] 2024-03-24T00:25:14.354 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5644de5391ff] 2024-03-24T00:25:14.354 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc26cc8a488] 2024-03-24T00:25:14.354 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 16: (DispatchQueue::entry()+0x51f) [0x7fc26cc8760f] 2024-03-24T00:25:14.355 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc26cd5ae81] 2024-03-24T00:25:14.355 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc26b7751ca] 2024-03-24T00:25:14.355 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 19: clone() 2024-03-24T00:25:14.355 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:25:14.355 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 2024-03-24T00:25:14.355 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: -1> 2024-03-24T00:25:13.983+0000 7fc2623ce700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fc2623ce700 time 2024-03-24T00:25:13.982544+0000 2024-03-24T00:25:14.355 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:25:14.355 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 2024-03-24T00:25:14.356 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:14.356 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fc26c9f84a1] 2024-03-24T00:25:14.356 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc26c9f8667] 2024-03-24T00:25:14.356 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5644de781223] 2024-03-24T00:25:14.356 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5644de5fb5d6] 2024-03-24T00:25:14.356 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5644de5fbda1] 2024-03-24T00:25:14.356 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5644de5fe2d5] 2024-03-24T00:25:14.356 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5644de64e233] 2024-03-24T00:25:14.356 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5644de650197] 2024-03-24T00:25:14.357 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5644de54f11b] 2024-03-24T00:25:14.357 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5644de54f6bc] 2024-03-24T00:25:14.357 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5644de5391ff] 2024-03-24T00:25:14.357 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc26cc8a488] 2024-03-24T00:25:14.357 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 13: (DispatchQueue::entry()+0x51f) [0x7fc26cc8760f] 2024-03-24T00:25:14.357 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc26cd5ae81] 2024-03-24T00:25:14.357 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc26b7751ca] 2024-03-24T00:25:14.357 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 16: clone() 2024-03-24T00:25:14.357 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 2024-03-24T00:25:14.358 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 0> 2024-03-24T00:25:13.985+0000 7fc2623ce700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:25:14.358 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: in thread 7fc2623ce700 thread_name:ms_dispatch 2024-03-24T00:25:14.358 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 2024-03-24T00:25:14.358 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:14.358 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc26b77fd20] 2024-03-24T00:25:14.358 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 2: gsignal() 2024-03-24T00:25:14.358 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 3: abort() 2024-03-24T00:25:14.358 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fc26c9f84fb] 2024-03-24T00:25:14.359 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc26c9f8667] 2024-03-24T00:25:14.359 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5644de781223] 2024-03-24T00:25:14.359 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5644de5fb5d6] 2024-03-24T00:25:14.359 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5644de5fbda1] 2024-03-24T00:25:14.359 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5644de5fe2d5] 2024-03-24T00:25:14.359 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5644de64e233] 2024-03-24T00:25:14.359 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5644de650197] 2024-03-24T00:25:14.359 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5644de54f11b] 2024-03-24T00:25:14.360 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5644de54f6bc] 2024-03-24T00:25:14.360 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5644de5391ff] 2024-03-24T00:25:14.360 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc26cc8a488] 2024-03-24T00:25:14.360 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 16: (DispatchQueue::entry()+0x51f) [0x7fc26cc8760f] 2024-03-24T00:25:14.360 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc26cd5ae81] 2024-03-24T00:25:14.360 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc26b7751ca] 2024-03-24T00:25:14.360 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 19: clone() 2024-03-24T00:25:14.360 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:25:14.360 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 2024-03-24T00:25:14.361 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: -1> 2024-03-24T00:25:13.983+0000 7fc2623ce700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fc2623ce700 time 2024-03-24T00:25:13.982544+0000 2024-03-24T00:25:14.361 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:25:14.361 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 2024-03-24T00:25:14.361 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:14.361 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fc26c9f84a1] 2024-03-24T00:25:14.361 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc26c9f8667] 2024-03-24T00:25:14.361 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5644de781223] 2024-03-24T00:25:14.361 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5644de5fb5d6] 2024-03-24T00:25:14.362 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5644de5fbda1] 2024-03-24T00:25:14.362 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5644de5fe2d5] 2024-03-24T00:25:14.362 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5644de64e233] 2024-03-24T00:25:14.362 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5644de650197] 2024-03-24T00:25:14.362 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5644de54f11b] 2024-03-24T00:25:14.362 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5644de54f6bc] 2024-03-24T00:25:14.362 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5644de5391ff] 2024-03-24T00:25:14.362 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc26cc8a488] 2024-03-24T00:25:14.363 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 13: (DispatchQueue::entry()+0x51f) [0x7fc26cc8760f] 2024-03-24T00:25:14.363 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc26cd5ae81] 2024-03-24T00:25:14.363 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc26b7751ca] 2024-03-24T00:25:14.363 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 16: clone() 2024-03-24T00:25:14.363 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 2024-03-24T00:25:14.363 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 0> 2024-03-24T00:25:13.985+0000 7fc2623ce700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:25:14.363 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: in thread 7fc2623ce700 thread_name:ms_dispatch 2024-03-24T00:25:14.363 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 2024-03-24T00:25:14.363 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:14.364 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc26b77fd20] 2024-03-24T00:25:14.364 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 2: gsignal() 2024-03-24T00:25:14.364 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 3: abort() 2024-03-24T00:25:14.364 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fc26c9f84fb] 2024-03-24T00:25:14.364 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc26c9f8667] 2024-03-24T00:25:14.364 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5644de781223] 2024-03-24T00:25:14.364 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5644de5fb5d6] 2024-03-24T00:25:14.364 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5644de5fbda1] 2024-03-24T00:25:14.365 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5644de5fe2d5] 2024-03-24T00:25:14.365 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5644de64e233] 2024-03-24T00:25:14.365 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5644de650197] 2024-03-24T00:25:14.365 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5644de54f11b] 2024-03-24T00:25:14.365 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5644de54f6bc] 2024-03-24T00:25:14.365 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5644de5391ff] 2024-03-24T00:25:14.365 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc26cc8a488] 2024-03-24T00:25:14.365 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 16: (DispatchQueue::entry()+0x51f) [0x7fc26cc8760f] 2024-03-24T00:25:14.365 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc26cd5ae81] 2024-03-24T00:25:14.366 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc26b7751ca] 2024-03-24T00:25:14.366 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 19: clone() 2024-03-24T00:25:14.366 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:25:14.366 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[71861]: 2024-03-24T00:25:14.366 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 podman[72048]: 2024-03-24 00:25:14.031854591 +0000 UTC m=+0.020652593 container died abeffabf64f52ce13a2c9f9ab61ea7e8db058b592598dd0f6817ed60598dcfe7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-24T00:25:14.668 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 43 sec latency 156258.894 ms 2024-03-24T00:25:14.750 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6834/2950120609,v1:172.21.15.107:6835/2950120609] up:rejoin 2024-03-24T00:25:14.750 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:rejoin,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:25:14.750 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-mon[36485]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-24T00:25:14.750 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:25:14.750 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:25:14.750 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6834/2950120609,v1:172.21.15.107:6835/2950120609] up:active 2024-03-24T00:25:14.750 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:14 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:25:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:14 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6834/2950120609,v1:172.21.15.107:6835/2950120609] up:rejoin 2024-03-24T00:25:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:14 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:rejoin,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:25:14.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:14 smithi106 ceph-mon[36505]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-24T00:25:14.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:14 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:25:14.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:14 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:25:14.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:14 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6834/2950120609,v1:172.21.15.107:6835/2950120609] up:active 2024-03-24T00:25:14.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:14 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:25:15.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:14 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6834/2950120609,v1:172.21.15.107:6835/2950120609] up:rejoin 2024-03-24T00:25:15.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:14 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:rejoin,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:25:15.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:14 smithi078 ceph-mon[32429]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-24T00:25:15.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:14 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:25:15.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:14 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:25:15.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:14 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6834/2950120609,v1:172.21.15.107:6835/2950120609] up:active 2024-03-24T00:25:15.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:14 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:25:15.093 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 podman[72048]: 2024-03-24 00:25:14.750523897 +0000 UTC m=+0.739321929 container remove abeffabf64f52ce13a2c9f9ab61ea7e8db058b592598dd0f6817ed60598dcfe7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-24T00:25:15.093 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:25:15.093 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:14 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.f.service: Failed with result 'exit-code'. 2024-03-24T00:25:15.652 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 24 00:25:15 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-b[66985]: 2024-03-24T00:25:15.255+0000 7f978f4ee700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-24T00:25:15.667 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 44 sec latency 157258.987 ms 2024-03-24T00:25:16.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:15 smithi078 ceph-mon[32429]: pgmap v134: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:25:16.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:15 smithi107 ceph-mon[36485]: pgmap v134: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:25:16.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:15 smithi106 ceph-mon[36505]: pgmap v134: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:25:16.668 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 45 sec latency 158259.079 ms 2024-03-24T00:25:17.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:17 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:25:17.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:17 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:25:17.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:17 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:25:17.668 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 46 sec latency 159259.136 ms 2024-03-24T00:25:18.460 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:18 smithi078 ceph-mon[32429]: pgmap v135: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:25:18.460 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:18 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:25:18.460 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:18 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:25:18.460 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:18 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:25:18.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:18 smithi107 ceph-mon[36485]: pgmap v135: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:25:18.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:18 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:25:18.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:18 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:25:18.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:18 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:25:18.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:18 smithi106 ceph-mon[36505]: pgmap v135: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:25:18.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:18 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:25:18.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:18 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:25:18.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:18 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:25:18.668 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 47 sec latency 160259.249 ms 2024-03-24T00:25:19.668 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 48 sec latency 161259.357 ms 2024-03-24T00:25:20.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:20 smithi106 ceph-mon[36505]: pgmap v136: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:25:20.668 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 49 sec latency 162259.478 ms 2024-03-24T00:25:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:20 smithi078 ceph-mon[32429]: pgmap v136: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:25:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:20 smithi107 ceph-mon[36485]: pgmap v136: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:25:21.668 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 50 sec latency 163259.535 ms 2024-03-24T00:25:22.645 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:22 smithi107 ceph-mon[36485]: pgmap v137: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 26 op/s 2024-03-24T00:25:22.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:22 smithi106 ceph-mon[36505]: pgmap v137: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 26 op/s 2024-03-24T00:25:22.668 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 51 sec latency 164259.668 ms 2024-03-24T00:25:22.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:22 smithi078 ceph-mon[32429]: pgmap v137: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 26 op/s 2024-03-24T00:25:23.668 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 52 sec latency 165259.787 ms 2024-03-24T00:25:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:24 smithi106 ceph-mon[36505]: pgmap v138: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-24T00:25:24.668 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 53 sec latency 166259.901 ms 2024-03-24T00:25:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:24 smithi078 ceph-mon[32429]: pgmap v138: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-24T00:25:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:24 smithi107 ceph-mon[36485]: pgmap v138: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-24T00:25:25.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:25 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.f.service: Scheduled restart job, restart counter is at 4. 2024-03-24T00:25:25.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:25 smithi107 systemd[1]: Stopped Ceph mds.f for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:25:25.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:25 smithi107 systemd[1]: Starting Ceph mds.f for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:25:25.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:25 smithi107 podman[72292]: 2024-03-24 00:25:25.227652208 +0000 UTC m=+0.018524591 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:25:25.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:25 smithi107 podman[72292]: 2024-03-24 00:25:25.350463679 +0000 UTC m=+0.141336069 container create 34e1f540b2943919ead28c0956ca4ce48239dd63aff0bc744d3ab6d720d87fb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-24T00:25:25.611 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:25 smithi107 podman[72292]: 2024-03-24 00:25:25.586521606 +0000 UTC m=+0.377393990 container init 34e1f540b2943919ead28c0956ca4ce48239dd63aff0bc744d3ab6d720d87fb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True) 2024-03-24T00:25:25.611 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:25 smithi107 podman[72292]: 2024-03-24 00:25:25.589006235 +0000 UTC m=+0.379878622 container start 34e1f540b2943919ead28c0956ca4ce48239dd63aff0bc744d3ab6d720d87fb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:25:25.668 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 54 sec latency 167259.937 ms 2024-03-24T00:25:26.094 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:25 smithi107 ceph-mds[72311]: -- 172.21.15.107:0/3796300753 <== mon.1 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ff46f19a40 con 0x55ff46eecc00 2024-03-24T00:25:26.094 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:25 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: starting mds.f at 2024-03-24T00:25:26.094 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:25 smithi107 bash[72292]: 34e1f540b2943919ead28c0956ca4ce48239dd63aff0bc744d3ab6d720d87fb0 2024-03-24T00:25:26.094 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:25 smithi107 systemd[1]: Started Ceph mds.f for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:25:26.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:26 smithi106 ceph-mon[36505]: pgmap v139: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-03-24T00:25:26.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:26 smithi106 ceph-mon[36505]: daemon mds.f restarted 2024-03-24T00:25:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:26 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:25:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:26 smithi106 ceph-mon[36505]: osdmap e103: 12 total, 12 up, 12 in 2024-03-24T00:25:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:26 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6834/758053763,v1:172.21.15.107:6835/758053763] up:boot 2024-03-24T00:25:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:26 smithi106 ceph-mon[36505]: Standby daemon mds.i assigned to filesystem cephfs as rank 1 2024-03-24T00:25:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:26 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-24T00:25:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:26 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:25:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:26 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:replay,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:25:26.669 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 55 sec latency 168260.025 ms 2024-03-24T00:25:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:26 smithi078 ceph-mon[32429]: pgmap v139: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-03-24T00:25:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:26 smithi078 ceph-mon[32429]: daemon mds.f restarted 2024-03-24T00:25:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:26 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:25:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:26 smithi078 ceph-mon[32429]: osdmap e103: 12 total, 12 up, 12 in 2024-03-24T00:25:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:26 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6834/758053763,v1:172.21.15.107:6835/758053763] up:boot 2024-03-24T00:25:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:26 smithi078 ceph-mon[32429]: Standby daemon mds.i assigned to filesystem cephfs as rank 1 2024-03-24T00:25:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:26 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-24T00:25:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:26 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:25:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:26 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:replay,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:25:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:26 smithi107 ceph-mon[36485]: pgmap v139: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-03-24T00:25:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:26 smithi107 ceph-mon[36485]: daemon mds.f restarted 2024-03-24T00:25:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:26 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:25:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:26 smithi107 ceph-mon[36485]: osdmap e103: 12 total, 12 up, 12 in 2024-03-24T00:25:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:26 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6834/758053763,v1:172.21.15.107:6835/758053763] up:boot 2024-03-24T00:25:26.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:26 smithi107 ceph-mon[36485]: Standby daemon mds.i assigned to filesystem cephfs as rank 1 2024-03-24T00:25:26.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:26 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-03-24T00:25:26.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:26 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:25:26.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:26 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:replay,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:25:27.669 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 56 sec latency 169260.118 ms 2024-03-24T00:25:27.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:27 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6838/3949153161,v1:172.21.15.107:6839/3949153161] up:resolve 2024-03-24T00:25:27.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:27 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:resolve,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:25:27.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:27 smithi106 ceph-mon[36505]: pgmap v141: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:25:28.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:27 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6838/3949153161,v1:172.21.15.107:6839/3949153161] up:resolve 2024-03-24T00:25:28.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:27 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:resolve,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:25:28.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:27 smithi078 ceph-mon[32429]: pgmap v141: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:25:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:27 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6838/3949153161,v1:172.21.15.107:6839/3949153161] up:resolve 2024-03-24T00:25:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:27 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:resolve,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:25:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:27 smithi107 ceph-mon[36485]: pgmap v141: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:25:28.669 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 57 sec latency 170260.226 ms 2024-03-24T00:25:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:28 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999983 2024-03-24T00:25:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:28 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6838/3949153161,v1:172.21.15.107:6839/3949153161] up:reconnect 2024-03-24T00:25:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:28 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:reconnect,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:25:28.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:28 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999983 2024-03-24T00:25:28.970 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:28 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6838/3949153161,v1:172.21.15.107:6839/3949153161] up:reconnect 2024-03-24T00:25:28.970 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:28 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:reconnect,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:25:28.982 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999983 2024-03-24T00:25:28.982 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6838/3949153161,v1:172.21.15.107:6839/3949153161] up:reconnect 2024-03-24T00:25:28.982 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:reconnect,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:25:29.344 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f5f7b1b3700 time 2024-03-24T00:25:28.983135+0000 2024-03-24T00:25:29.344 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:25:29.344 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:29.345 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f5f857dd4a1] 2024-03-24T00:25:29.345 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f5f857dd667] 2024-03-24T00:25:29.345 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56352f1e2223] 2024-03-24T00:25:29.345 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56352f05c5d6] 2024-03-24T00:25:29.345 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56352f05cda1] 2024-03-24T00:25:29.345 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56352f05f2d5] 2024-03-24T00:25:29.345 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56352f0af233] 2024-03-24T00:25:29.345 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56352f0b1197] 2024-03-24T00:25:29.345 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56352efb011b] 2024-03-24T00:25:29.345 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56352efb06bc] 2024-03-24T00:25:29.346 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56352ef9a1ff] 2024-03-24T00:25:29.346 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5f85a6f488] 2024-03-24T00:25:29.346 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 13: (DispatchQueue::entry()+0x51f) [0x7f5f85a6c60f] 2024-03-24T00:25:29.346 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5f85b3fe81] 2024-03-24T00:25:29.346 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5f8455a1ca] 2024-03-24T00:25:29.346 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 16: clone() 2024-03-24T00:25:29.346 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: *** Caught signal (Aborted) ** 2024-03-24T00:25:29.346 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: in thread 7f5f7b1b3700 thread_name:ms_dispatch 2024-03-24T00:25:29.346 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 2024-03-24T00:25:28.983+0000 7f5f7b1b3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f5f7b1b3700 time 2024-03-24T00:25:28.983135+0000 2024-03-24T00:25:29.346 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:25:29.346 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 2024-03-24T00:25:29.347 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:29.347 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f5f857dd4a1] 2024-03-24T00:25:29.347 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f5f857dd667] 2024-03-24T00:25:29.347 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56352f1e2223] 2024-03-24T00:25:29.347 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56352f05c5d6] 2024-03-24T00:25:29.347 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56352f05cda1] 2024-03-24T00:25:29.347 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56352f05f2d5] 2024-03-24T00:25:29.347 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56352f0af233] 2024-03-24T00:25:29.347 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56352f0b1197] 2024-03-24T00:25:29.347 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56352efb011b] 2024-03-24T00:25:29.347 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56352efb06bc] 2024-03-24T00:25:29.347 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56352ef9a1ff] 2024-03-24T00:25:29.348 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5f85a6f488] 2024-03-24T00:25:29.348 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 13: (DispatchQueue::entry()+0x51f) [0x7f5f85a6c60f] 2024-03-24T00:25:29.348 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5f85b3fe81] 2024-03-24T00:25:29.348 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5f8455a1ca] 2024-03-24T00:25:29.348 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 16: clone() 2024-03-24T00:25:29.348 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 2024-03-24T00:25:29.348 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:29.348 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5f84564d20] 2024-03-24T00:25:29.349 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 2: gsignal() 2024-03-24T00:25:29.349 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 3: abort() 2024-03-24T00:25:29.349 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f5f857dd4fb] 2024-03-24T00:25:29.349 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f5f857dd667] 2024-03-24T00:25:29.349 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56352f1e2223] 2024-03-24T00:25:29.349 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56352f05c5d6] 2024-03-24T00:25:29.349 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56352f05cda1] 2024-03-24T00:25:29.350 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56352f05f2d5] 2024-03-24T00:25:29.350 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56352f0af233] 2024-03-24T00:25:29.350 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56352f0b1197] 2024-03-24T00:25:29.350 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56352efb011b] 2024-03-24T00:25:29.350 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56352efb06bc] 2024-03-24T00:25:29.350 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56352ef9a1ff] 2024-03-24T00:25:29.350 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5f85a6f488] 2024-03-24T00:25:29.350 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 16: (DispatchQueue::entry()+0x51f) [0x7f5f85a6c60f] 2024-03-24T00:25:29.350 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5f85b3fe81] 2024-03-24T00:25:29.350 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5f8455a1ca] 2024-03-24T00:25:29.350 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 19: clone() 2024-03-24T00:25:29.350 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 2024-03-24T00:25:28.985+0000 7f5f7b1b3700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:25:29.350 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: in thread 7f5f7b1b3700 thread_name:ms_dispatch 2024-03-24T00:25:29.351 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 2024-03-24T00:25:29.351 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:29.351 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5f84564d20] 2024-03-24T00:25:29.351 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 2: gsignal() 2024-03-24T00:25:29.351 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 3: abort() 2024-03-24T00:25:29.351 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f5f857dd4fb] 2024-03-24T00:25:29.351 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f5f857dd667] 2024-03-24T00:25:29.351 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56352f1e2223] 2024-03-24T00:25:29.351 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56352f05c5d6] 2024-03-24T00:25:29.351 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56352f05cda1] 2024-03-24T00:25:29.351 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56352f05f2d5] 2024-03-24T00:25:29.351 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56352f0af233] 2024-03-24T00:25:29.351 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56352f0b1197] 2024-03-24T00:25:29.351 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56352efb011b] 2024-03-24T00:25:29.352 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56352efb06bc] 2024-03-24T00:25:29.352 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56352ef9a1ff] 2024-03-24T00:25:29.352 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5f85a6f488] 2024-03-24T00:25:29.352 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 16: (DispatchQueue::entry()+0x51f) [0x7f5f85a6c60f] 2024-03-24T00:25:29.352 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5f85b3fe81] 2024-03-24T00:25:29.352 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5f8455a1ca] 2024-03-24T00:25:29.352 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 19: clone() 2024-03-24T00:25:29.352 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:25:29.352 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 2024-03-24T00:25:29.352 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: -1> 2024-03-24T00:25:28.983+0000 7f5f7b1b3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f5f7b1b3700 time 2024-03-24T00:25:28.983135+0000 2024-03-24T00:25:29.352 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:25:29.352 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 2024-03-24T00:25:29.352 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:29.352 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f5f857dd4a1] 2024-03-24T00:25:29.353 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f5f857dd667] 2024-03-24T00:25:29.353 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56352f1e2223] 2024-03-24T00:25:29.353 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56352f05c5d6] 2024-03-24T00:25:29.353 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56352f05cda1] 2024-03-24T00:25:29.353 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56352f05f2d5] 2024-03-24T00:25:29.353 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56352f0af233] 2024-03-24T00:25:29.353 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56352f0b1197] 2024-03-24T00:25:29.353 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56352efb011b] 2024-03-24T00:25:29.353 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56352efb06bc] 2024-03-24T00:25:29.353 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56352ef9a1ff] 2024-03-24T00:25:29.353 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5f85a6f488] 2024-03-24T00:25:29.353 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 13: (DispatchQueue::entry()+0x51f) [0x7f5f85a6c60f] 2024-03-24T00:25:29.353 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5f85b3fe81] 2024-03-24T00:25:29.354 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5f8455a1ca] 2024-03-24T00:25:29.354 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 16: clone() 2024-03-24T00:25:29.354 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 2024-03-24T00:25:29.354 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 0> 2024-03-24T00:25:28.985+0000 7f5f7b1b3700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:25:29.354 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: in thread 7f5f7b1b3700 thread_name:ms_dispatch 2024-03-24T00:25:29.354 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 2024-03-24T00:25:29.354 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:29.354 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5f84564d20] 2024-03-24T00:25:29.354 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 2: gsignal() 2024-03-24T00:25:29.354 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 3: abort() 2024-03-24T00:25:29.354 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f5f857dd4fb] 2024-03-24T00:25:29.354 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f5f857dd667] 2024-03-24T00:25:29.355 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56352f1e2223] 2024-03-24T00:25:29.355 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56352f05c5d6] 2024-03-24T00:25:29.355 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56352f05cda1] 2024-03-24T00:25:29.355 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56352f05f2d5] 2024-03-24T00:25:29.355 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56352f0af233] 2024-03-24T00:25:29.355 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56352f0b1197] 2024-03-24T00:25:29.355 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56352efb011b] 2024-03-24T00:25:29.355 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56352efb06bc] 2024-03-24T00:25:29.355 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56352ef9a1ff] 2024-03-24T00:25:29.355 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5f85a6f488] 2024-03-24T00:25:29.355 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 16: (DispatchQueue::entry()+0x51f) [0x7f5f85a6c60f] 2024-03-24T00:25:29.355 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5f85b3fe81] 2024-03-24T00:25:29.355 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5f8455a1ca] 2024-03-24T00:25:29.355 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 19: clone() 2024-03-24T00:25:29.356 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:25:29.356 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 2024-03-24T00:25:29.356 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: -1> 2024-03-24T00:25:28.983+0000 7f5f7b1b3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f5f7b1b3700 time 2024-03-24T00:25:28.983135+0000 2024-03-24T00:25:29.356 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:25:29.356 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 2024-03-24T00:25:29.356 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:29.356 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f5f857dd4a1] 2024-03-24T00:25:29.356 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f5f857dd667] 2024-03-24T00:25:29.356 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56352f1e2223] 2024-03-24T00:25:29.356 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56352f05c5d6] 2024-03-24T00:25:29.356 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56352f05cda1] 2024-03-24T00:25:29.356 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56352f05f2d5] 2024-03-24T00:25:29.356 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56352f0af233] 2024-03-24T00:25:29.356 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56352f0b1197] 2024-03-24T00:25:29.357 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56352efb011b] 2024-03-24T00:25:29.357 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56352efb06bc] 2024-03-24T00:25:29.357 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56352ef9a1ff] 2024-03-24T00:25:29.357 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5f85a6f488] 2024-03-24T00:25:29.357 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 13: (DispatchQueue::entry()+0x51f) [0x7f5f85a6c60f] 2024-03-24T00:25:29.357 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5f85b3fe81] 2024-03-24T00:25:29.357 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5f8455a1ca] 2024-03-24T00:25:29.357 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 16: clone() 2024-03-24T00:25:29.357 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 2024-03-24T00:25:29.357 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 0> 2024-03-24T00:25:28.985+0000 7f5f7b1b3700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:25:29.357 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: in thread 7f5f7b1b3700 thread_name:ms_dispatch 2024-03-24T00:25:29.357 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 2024-03-24T00:25:29.357 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:29.357 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5f84564d20] 2024-03-24T00:25:29.358 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 2: gsignal() 2024-03-24T00:25:29.358 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 3: abort() 2024-03-24T00:25:29.358 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f5f857dd4fb] 2024-03-24T00:25:29.358 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f5f857dd667] 2024-03-24T00:25:29.358 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56352f1e2223] 2024-03-24T00:25:29.358 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56352f05c5d6] 2024-03-24T00:25:29.358 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56352f05cda1] 2024-03-24T00:25:29.358 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56352f05f2d5] 2024-03-24T00:25:29.358 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56352f0af233] 2024-03-24T00:25:29.358 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56352f0b1197] 2024-03-24T00:25:29.358 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56352efb011b] 2024-03-24T00:25:29.358 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56352efb06bc] 2024-03-24T00:25:29.358 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56352ef9a1ff] 2024-03-24T00:25:29.359 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5f85a6f488] 2024-03-24T00:25:29.359 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 16: (DispatchQueue::entry()+0x51f) [0x7f5f85a6c60f] 2024-03-24T00:25:29.359 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5f85b3fe81] 2024-03-24T00:25:29.359 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5f8455a1ca] 2024-03-24T00:25:29.359 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 19: clone() 2024-03-24T00:25:29.359 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:25:29.359 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i[72010]: 2024-03-24T00:25:29.359 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 podman[72345]: 2024-03-24 00:25:29.029604823 +0000 UTC m=+0.019864957 container died c64578d01d47a7bc8e8718f340492e07eaf99a6fbf81527e49ae03e23a061f31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-24T00:25:29.669 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 58 sec latency 171260.316 ms 2024-03-24T00:25:29.749 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6838/3949153161,v1:172.21.15.107:6839/3949153161] up:rejoin 2024-03-24T00:25:29.749 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:rejoin,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:25:29.749 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-mon[36485]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-24T00:25:29.749 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:25:29.749 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:25:29.750 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6838/3949153161,v1:172.21.15.107:6839/3949153161] up:active 2024-03-24T00:25:29.750 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:25:29.750 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:29 smithi107 ceph-mon[36485]: pgmap v142: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:25:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:29 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6838/3949153161,v1:172.21.15.107:6839/3949153161] up:rejoin 2024-03-24T00:25:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:29 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:rejoin,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:25:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:29 smithi078 ceph-mon[32429]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-24T00:25:30.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:29 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:25:30.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:29 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:25:30.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:29 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6838/3949153161,v1:172.21.15.107:6839/3949153161] up:active 2024-03-24T00:25:30.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:29 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:25:30.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:29 smithi078 ceph-mon[32429]: pgmap v142: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:25:30.093 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 podman[72345]: 2024-03-24 00:25:29.75011728 +0000 UTC m=+0.740377395 container remove c64578d01d47a7bc8e8718f340492e07eaf99a6fbf81527e49ae03e23a061f31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-i, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-24T00:25:30.093 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:25:30.093 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:29 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.i.service: Failed with result 'exit-code'. 2024-03-24T00:25:30.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:29 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6838/3949153161,v1:172.21.15.107:6839/3949153161] up:rejoin 2024-03-24T00:25:30.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:29 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:rejoin,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:25:30.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:29 smithi106 ceph-mon[36505]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-24T00:25:30.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:29 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:25:30.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:29 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:25:30.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:29 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6838/3949153161,v1:172.21.15.107:6839/3949153161] up:active 2024-03-24T00:25:30.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:29 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T00:25:30.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:29 smithi106 ceph-mon[36505]: pgmap v142: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:25:30.652 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 24 00:25:30 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-b[66985]: 2024-03-24T00:25:30.255+0000 7f978f4ee700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-24T00:25:30.669 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 59 sec latency 172260.422 ms 2024-03-24T00:25:31.669 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 60 sec latency 173260.524 ms 2024-03-24T00:25:32.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:32 smithi078 ceph-mon[32429]: pgmap v143: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-24T00:25:32.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:32 smithi106 ceph-mon[36505]: pgmap v143: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-24T00:25:32.669 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 61 sec latency 174260.627 ms 2024-03-24T00:25:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:32 smithi107 ceph-mon[36485]: pgmap v143: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-24T00:25:33.669 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 62 sec latency 175260.708 ms 2024-03-24T00:25:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:34 smithi106 ceph-mon[36505]: pgmap v144: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-24T00:25:34.669 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 63 sec latency 176260.792 ms 2024-03-24T00:25:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:34 smithi078 ceph-mon[32429]: pgmap v144: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-24T00:25:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:34 smithi107 ceph-mon[36485]: pgmap v144: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-24T00:25:35.670 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 64 sec latency 177260.895 ms 2024-03-24T00:25:36.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:36 smithi106 ceph-mon[36505]: pgmap v145: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:25:36.670 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 65 sec latency 178260.976 ms 2024-03-24T00:25:36.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:36 smithi078 ceph-mon[32429]: pgmap v145: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:25:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:36 smithi107 ceph-mon[36485]: pgmap v145: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:25:37.670 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 66 sec latency 179261.081 ms 2024-03-24T00:25:38.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:38 smithi106 ceph-mon[36505]: pgmap v146: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 175 B/s wr, 9 op/s 2024-03-24T00:25:38.670 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 67 sec latency 180261.185 ms 2024-03-24T00:25:38.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:38 smithi078 ceph-mon[32429]: pgmap v146: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 175 B/s wr, 9 op/s 2024-03-24T00:25:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:38 smithi107 ceph-mon[36485]: pgmap v146: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 175 B/s wr, 9 op/s 2024-03-24T00:25:39.670 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 68 sec latency 181261.231 ms 2024-03-24T00:25:40.381 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:40 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.i.service: Scheduled restart job, restart counter is at 5. 2024-03-24T00:25:40.381 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:40 smithi107 systemd[1]: Stopped Ceph mds.i for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:25:40.381 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:40 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.i.service: Start request repeated too quickly. 2024-03-24T00:25:40.381 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:40 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.i.service: Failed with result 'exit-code'. 2024-03-24T00:25:40.381 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 24 00:25:40 smithi107 systemd[1]: Failed to start Ceph mds.i for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:25:40.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:40 smithi106 ceph-mon[36505]: pgmap v147: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-24T00:25:40.670 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 69 sec latency 182261.342 ms 2024-03-24T00:25:40.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:40 smithi078 ceph-mon[32429]: pgmap v147: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-24T00:25:40.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:40 smithi107 ceph-mon[36485]: pgmap v147: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-24T00:25:41.670 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 70 sec latency 183261.433 ms 2024-03-24T00:25:42.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:42 smithi106 ceph-mon[36505]: pgmap v148: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-24T00:25:42.670 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 71 sec latency 184261.534 ms 2024-03-24T00:25:42.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:42 smithi078 ceph-mon[32429]: pgmap v148: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-24T00:25:42.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:42 smithi107 ceph-mon[36485]: pgmap v148: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-24T00:25:43.670 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 72 sec latency 185261.632 ms 2024-03-24T00:25:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:44 smithi106 ceph-mon[36505]: pgmap v149: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:25:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:44 smithi106 ceph-mon[36505]: Replacing daemon mds.i as rank 1 with standby daemon mds.f 2024-03-24T00:25:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:44 smithi106 ceph-mon[36505]: MDS daemon mds.i is removed because it is dead or otherwise unavailable. 2024-03-24T00:25:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:44 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:25:44.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:44 smithi106 ceph-mon[36505]: osdmap e104: 12 total, 12 up, 12 in 2024-03-24T00:25:44.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:44 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:replay,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:25:44.670 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 73 sec latency 186261.683 ms 2024-03-24T00:25:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:44 smithi078 ceph-mon[32429]: pgmap v149: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:25:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:44 smithi078 ceph-mon[32429]: Replacing daemon mds.i as rank 1 with standby daemon mds.f 2024-03-24T00:25:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:44 smithi078 ceph-mon[32429]: MDS daemon mds.i is removed because it is dead or otherwise unavailable. 2024-03-24T00:25:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:44 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:25:44.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:44 smithi078 ceph-mon[32429]: osdmap e104: 12 total, 12 up, 12 in 2024-03-24T00:25:44.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:44 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:replay,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:25:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:44 smithi107 ceph-mon[36485]: pgmap v149: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:25:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:44 smithi107 ceph-mon[36485]: Replacing daemon mds.i as rank 1 with standby daemon mds.f 2024-03-24T00:25:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:44 smithi107 ceph-mon[36485]: MDS daemon mds.i is removed because it is dead or otherwise unavailable. 2024-03-24T00:25:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:44 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:25:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:44 smithi107 ceph-mon[36485]: osdmap e104: 12 total, 12 up, 12 in 2024-03-24T00:25:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:44 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:replay,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:25:45.670 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 74 sec latency 187261.768 ms 2024-03-24T00:25:46.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:45 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6834/758053763,v1:172.21.15.107:6835/758053763] up:resolve 2024-03-24T00:25:46.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:45 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:resolve,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:25:46.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:45 smithi078 ceph-mon[32429]: pgmap v151: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:25:46.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:45 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6834/758053763,v1:172.21.15.107:6835/758053763] up:resolve 2024-03-24T00:25:46.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:45 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:resolve,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:25:46.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:45 smithi107 ceph-mon[36485]: pgmap v151: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:25:46.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:45 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6834/758053763,v1:172.21.15.107:6835/758053763] up:resolve 2024-03-24T00:25:46.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:45 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:resolve,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:25:46.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:45 smithi106 ceph-mon[36505]: pgmap v151: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:25:46.670 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 75 sec latency 188261.867 ms 2024-03-24T00:25:47.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:46 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6834/758053763,v1:172.21.15.107:6835/758053763] up:reconnect 2024-03-24T00:25:47.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:46 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:reconnect,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:25:47.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:46 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.00399993 2024-03-24T00:25:47.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:46 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6834/758053763,v1:172.21.15.107:6835/758053763] up:reconnect 2024-03-24T00:25:47.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:46 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:reconnect,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:25:47.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:46 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.00399993 2024-03-24T00:25:47.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:46 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6834/758053763,v1:172.21.15.107:6835/758053763] up:reconnect 2024-03-24T00:25:47.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:46 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:reconnect,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:25:47.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:46 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.00399993 2024-03-24T00:25:47.671 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 76 sec latency 189261.944 ms 2024-03-24T00:25:48.014 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:47 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6834/758053763,v1:172.21.15.107:6835/758053763] up:rejoin 2024-03-24T00:25:48.014 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:47 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:rejoin,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:25:48.014 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:47 smithi107 ceph-mon[36485]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-24T00:25:48.014 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:47 smithi107 ceph-mon[36485]: pgmap v152: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:25:48.015 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fb93c5fb700 time 2024-03-24T00:25:48.009375+0000 2024-03-24T00:25:48.015 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:25:48.015 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:48.015 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fb946c254a1] 2024-03-24T00:25:48.015 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb946c25667] 2024-03-24T00:25:48.015 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ff4467f223] 2024-03-24T00:25:48.015 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ff444f95d6] 2024-03-24T00:25:48.015 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ff444f9da1] 2024-03-24T00:25:48.015 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ff444fc2d5] 2024-03-24T00:25:48.016 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ff4454c233] 2024-03-24T00:25:48.016 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ff4454e197] 2024-03-24T00:25:48.016 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ff4444d11b] 2024-03-24T00:25:48.016 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ff4444d6bc] 2024-03-24T00:25:48.016 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ff444371ff] 2024-03-24T00:25:48.016 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb946eb7488] 2024-03-24T00:25:48.016 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 13: (DispatchQueue::entry()+0x51f) [0x7fb946eb460f] 2024-03-24T00:25:48.016 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb946f87e81] 2024-03-24T00:25:48.016 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb9459a21ca] 2024-03-24T00:25:48.016 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 16: clone() 2024-03-24T00:25:48.016 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 2024-03-24T00:25:48.009+0000 7fb93c5fb700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fb93c5fb700 time 2024-03-24T00:25:48.009375+0000 2024-03-24T00:25:48.016 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:25:48.016 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 2024-03-24T00:25:48.016 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:48.017 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fb946c254a1] 2024-03-24T00:25:48.017 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb946c25667] 2024-03-24T00:25:48.017 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ff4467f223] 2024-03-24T00:25:48.017 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ff444f95d6] 2024-03-24T00:25:48.017 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ff444f9da1] 2024-03-24T00:25:48.017 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ff444fc2d5] 2024-03-24T00:25:48.017 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ff4454c233] 2024-03-24T00:25:48.017 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ff4454e197] 2024-03-24T00:25:48.017 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ff4444d11b] 2024-03-24T00:25:48.017 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ff4444d6bc] 2024-03-24T00:25:48.017 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ff444371ff] 2024-03-24T00:25:48.017 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb946eb7488] 2024-03-24T00:25:48.017 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 13: (DispatchQueue::entry()+0x51f) [0x7fb946eb460f] 2024-03-24T00:25:48.017 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb946f87e81] 2024-03-24T00:25:48.018 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb9459a21ca] 2024-03-24T00:25:48.018 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 16: clone() 2024-03-24T00:25:48.018 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 2024-03-24T00:25:48.018 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: *** Caught signal (Aborted) ** 2024-03-24T00:25:48.018 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: in thread 7fb93c5fb700 thread_name:ms_dispatch 2024-03-24T00:25:48.018 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:48.018 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb9459acd20] 2024-03-24T00:25:48.018 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 2: gsignal() 2024-03-24T00:25:48.018 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 3: abort() 2024-03-24T00:25:48.018 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb946c254fb] 2024-03-24T00:25:48.019 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb946c25667] 2024-03-24T00:25:48.019 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ff4467f223] 2024-03-24T00:25:48.019 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ff444f95d6] 2024-03-24T00:25:48.019 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ff444f9da1] 2024-03-24T00:25:48.019 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ff444fc2d5] 2024-03-24T00:25:48.019 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ff4454c233] 2024-03-24T00:25:48.019 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ff4454e197] 2024-03-24T00:25:48.019 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ff4444d11b] 2024-03-24T00:25:48.019 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ff4444d6bc] 2024-03-24T00:25:48.019 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ff444371ff] 2024-03-24T00:25:48.019 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb946eb7488] 2024-03-24T00:25:48.019 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 16: (DispatchQueue::entry()+0x51f) [0x7fb946eb460f] 2024-03-24T00:25:48.019 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb946f87e81] 2024-03-24T00:25:48.019 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb9459a21ca] 2024-03-24T00:25:48.019 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 19: clone() 2024-03-24T00:25:48.020 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 2024-03-24T00:25:48.012+0000 7fb93c5fb700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:25:48.020 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: in thread 7fb93c5fb700 thread_name:ms_dispatch 2024-03-24T00:25:48.020 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 2024-03-24T00:25:48.020 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:48.020 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb9459acd20] 2024-03-24T00:25:48.020 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 2: gsignal() 2024-03-24T00:25:48.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:47 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6834/758053763,v1:172.21.15.107:6835/758053763] up:rejoin 2024-03-24T00:25:48.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:47 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:rejoin,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:25:48.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:47 smithi078 ceph-mon[32429]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-24T00:25:48.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:47 smithi078 ceph-mon[32429]: pgmap v152: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:25:48.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 3: abort() 2024-03-24T00:25:48.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb946c254fb] 2024-03-24T00:25:48.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb946c25667] 2024-03-24T00:25:48.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ff4467f223] 2024-03-24T00:25:48.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ff444f95d6] 2024-03-24T00:25:48.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ff444f9da1] 2024-03-24T00:25:48.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ff444fc2d5] 2024-03-24T00:25:48.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ff4454c233] 2024-03-24T00:25:48.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ff4454e197] 2024-03-24T00:25:48.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ff4444d11b] 2024-03-24T00:25:48.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ff4444d6bc] 2024-03-24T00:25:48.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ff444371ff] 2024-03-24T00:25:48.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb946eb7488] 2024-03-24T00:25:48.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 16: (DispatchQueue::entry()+0x51f) [0x7fb946eb460f] 2024-03-24T00:25:48.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb946f87e81] 2024-03-24T00:25:48.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb9459a21ca] 2024-03-24T00:25:48.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 19: clone() 2024-03-24T00:25:48.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:25:48.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 2024-03-24T00:25:48.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: -1> 2024-03-24T00:25:48.009+0000 7fb93c5fb700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fb93c5fb700 time 2024-03-24T00:25:48.009375+0000 2024-03-24T00:25:48.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:25:48.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 2024-03-24T00:25:48.346 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:48.346 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fb946c254a1] 2024-03-24T00:25:48.346 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb946c25667] 2024-03-24T00:25:48.346 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ff4467f223] 2024-03-24T00:25:48.346 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ff444f95d6] 2024-03-24T00:25:48.346 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ff444f9da1] 2024-03-24T00:25:48.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ff444fc2d5] 2024-03-24T00:25:48.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ff4454c233] 2024-03-24T00:25:48.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ff4454e197] 2024-03-24T00:25:48.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ff4444d11b] 2024-03-24T00:25:48.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ff4444d6bc] 2024-03-24T00:25:48.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ff444371ff] 2024-03-24T00:25:48.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb946eb7488] 2024-03-24T00:25:48.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 13: (DispatchQueue::entry()+0x51f) [0x7fb946eb460f] 2024-03-24T00:25:48.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb946f87e81] 2024-03-24T00:25:48.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb9459a21ca] 2024-03-24T00:25:48.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 16: clone() 2024-03-24T00:25:48.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 2024-03-24T00:25:48.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 0> 2024-03-24T00:25:48.012+0000 7fb93c5fb700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:25:48.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: in thread 7fb93c5fb700 thread_name:ms_dispatch 2024-03-24T00:25:48.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 2024-03-24T00:25:48.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:48.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb9459acd20] 2024-03-24T00:25:48.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 2: gsignal() 2024-03-24T00:25:48.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 3: abort() 2024-03-24T00:25:48.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb946c254fb] 2024-03-24T00:25:48.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb946c25667] 2024-03-24T00:25:48.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ff4467f223] 2024-03-24T00:25:48.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ff444f95d6] 2024-03-24T00:25:48.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ff444f9da1] 2024-03-24T00:25:48.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ff444fc2d5] 2024-03-24T00:25:48.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ff4454c233] 2024-03-24T00:25:48.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ff4454e197] 2024-03-24T00:25:48.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ff4444d11b] 2024-03-24T00:25:48.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ff4444d6bc] 2024-03-24T00:25:48.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ff444371ff] 2024-03-24T00:25:48.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb946eb7488] 2024-03-24T00:25:48.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 16: (DispatchQueue::entry()+0x51f) [0x7fb946eb460f] 2024-03-24T00:25:48.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb946f87e81] 2024-03-24T00:25:48.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb9459a21ca] 2024-03-24T00:25:48.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 19: clone() 2024-03-24T00:25:48.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:25:48.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 2024-03-24T00:25:48.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: -1> 2024-03-24T00:25:48.009+0000 7fb93c5fb700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fb93c5fb700 time 2024-03-24T00:25:48.009375+0000 2024-03-24T00:25:48.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:25:48.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 2024-03-24T00:25:48.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:48.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fb946c254a1] 2024-03-24T00:25:48.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb946c25667] 2024-03-24T00:25:48.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ff4467f223] 2024-03-24T00:25:48.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ff444f95d6] 2024-03-24T00:25:48.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ff444f9da1] 2024-03-24T00:25:48.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ff444fc2d5] 2024-03-24T00:25:48.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ff4454c233] 2024-03-24T00:25:48.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ff4454e197] 2024-03-24T00:25:48.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ff4444d11b] 2024-03-24T00:25:48.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ff4444d6bc] 2024-03-24T00:25:48.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ff444371ff] 2024-03-24T00:25:48.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb946eb7488] 2024-03-24T00:25:48.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 13: (DispatchQueue::entry()+0x51f) [0x7fb946eb460f] 2024-03-24T00:25:48.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb946f87e81] 2024-03-24T00:25:48.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb9459a21ca] 2024-03-24T00:25:48.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 16: clone() 2024-03-24T00:25:48.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 2024-03-24T00:25:48.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 0> 2024-03-24T00:25:48.012+0000 7fb93c5fb700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:25:48.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: in thread 7fb93c5fb700 thread_name:ms_dispatch 2024-03-24T00:25:48.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 2024-03-24T00:25:48.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:48.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb9459acd20] 2024-03-24T00:25:48.351 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 2: gsignal() 2024-03-24T00:25:48.352 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 3: abort() 2024-03-24T00:25:48.352 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb946c254fb] 2024-03-24T00:25:48.352 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb946c25667] 2024-03-24T00:25:48.352 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ff4467f223] 2024-03-24T00:25:48.352 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ff444f95d6] 2024-03-24T00:25:48.352 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ff444f9da1] 2024-03-24T00:25:48.352 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55ff444fc2d5] 2024-03-24T00:25:48.352 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ff4454c233] 2024-03-24T00:25:48.352 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ff4454e197] 2024-03-24T00:25:48.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ff4444d11b] 2024-03-24T00:25:48.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ff4444d6bc] 2024-03-24T00:25:48.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ff444371ff] 2024-03-24T00:25:48.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb946eb7488] 2024-03-24T00:25:48.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 16: (DispatchQueue::entry()+0x51f) [0x7fb946eb460f] 2024-03-24T00:25:48.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb946f87e81] 2024-03-24T00:25:48.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb9459a21ca] 2024-03-24T00:25:48.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 19: clone() 2024-03-24T00:25:48.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:25:48.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72307]: 2024-03-24T00:25:48.353 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 podman[72391]: 2024-03-24 00:25:48.06465547 +0000 UTC m=+0.020031722 container died 34e1f540b2943919ead28c0956ca4ce48239dd63aff0bc744d3ab6d720d87fb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True) 2024-03-24T00:25:48.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:47 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6834/758053763,v1:172.21.15.107:6835/758053763] up:rejoin 2024-03-24T00:25:48.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:47 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:rejoin,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:25:48.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:47 smithi106 ceph-mon[36505]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-24T00:25:48.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:47 smithi106 ceph-mon[36505]: pgmap v152: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:25:48.674 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 77 sec latency 190262.029 ms 2024-03-24T00:25:48.997 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 podman[72391]: 2024-03-24 00:25:48.685843462 +0000 UTC m=+0.641219714 container remove 34e1f540b2943919ead28c0956ca4ce48239dd63aff0bc744d3ab6d720d87fb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-24T00:25:48.997 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:25:48.997 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:48 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.f.service: Failed with result 'exit-code'. 2024-03-24T00:25:49.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:48 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:25:49.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:48 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:25:49.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:48 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6834/758053763,v1:172.21.15.107:6835/758053763] up:active 2024-03-24T00:25:49.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:48 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:25:49.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:48 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=h=up:active} 3 up:standby-replay 5 up:standby 2024-03-24T00:25:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:25:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:25:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6834/758053763,v1:172.21.15.107:6835/758053763] up:active 2024-03-24T00:25:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:25:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:48 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=h=up:active} 3 up:standby-replay 5 up:standby 2024-03-24T00:25:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:48 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:25:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:48 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:25:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:48 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6834/758053763,v1:172.21.15.107:6835/758053763] up:active 2024-03-24T00:25:49.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:48 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:25:49.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:48 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=h=up:active} 3 up:standby-replay 5 up:standby 2024-03-24T00:25:49.671 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 78 sec latency 191262.102 ms 2024-03-24T00:25:50.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:50 smithi078 ceph-mon[32429]: pgmap v153: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:25:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:50 smithi107 ceph-mon[36485]: pgmap v153: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:25:50.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:50 smithi106 ceph-mon[36505]: pgmap v153: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:25:50.671 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 79 sec latency 192262.182 ms 2024-03-24T00:25:51.671 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 80 sec latency 193262.331 ms 2024-03-24T00:25:52.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:52 smithi106 ceph-mon[36505]: pgmap v154: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:25:52.671 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 81 sec latency 194262.439 ms 2024-03-24T00:25:52.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:52 smithi078 ceph-mon[32429]: pgmap v154: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:25:52.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:52 smithi107 ceph-mon[36485]: pgmap v154: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:25:53.671 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 82 sec latency 195262.542 ms 2024-03-24T00:25:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:54 smithi078 ceph-mon[32429]: pgmap v155: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:25:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:54 smithi106 ceph-mon[36505]: pgmap v155: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:25:54.671 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 83 sec latency 196262.644 ms 2024-03-24T00:25:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:54 smithi107 ceph-mon[36485]: pgmap v155: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:25:55.671 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 84 sec latency 197262.732 ms 2024-03-24T00:25:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:56 smithi106 ceph-mon[36505]: pgmap v156: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 180 B/s wr, 12 op/s 2024-03-24T00:25:56.671 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 85 sec latency 198262.821 ms 2024-03-24T00:25:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:56 smithi078 ceph-mon[32429]: pgmap v156: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 180 B/s wr, 12 op/s 2024-03-24T00:25:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:56 smithi107 ceph-mon[36485]: pgmap v156: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 180 B/s wr, 12 op/s 2024-03-24T00:25:57.672 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 86 sec latency 199262.914 ms 2024-03-24T00:25:58.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:25:58 smithi106 ceph-mon[36505]: pgmap v157: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T00:25:58.672 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 87 sec latency 200262.971 ms 2024-03-24T00:25:58.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:25:58 smithi078 ceph-mon[32429]: pgmap v157: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T00:25:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:25:58 smithi107 ceph-mon[36485]: pgmap v157: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T00:25:59.456 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:59 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.f.service: Scheduled restart job, restart counter is at 5. 2024-03-24T00:25:59.456 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:59 smithi107 systemd[1]: Stopped Ceph mds.f for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:25:59.456 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:59 smithi107 systemd[1]: Starting Ceph mds.f for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:25:59.456 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:59 smithi107 podman[72488]: 2024-03-24 00:25:59.224647841 +0000 UTC m=+0.018141437 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:25:59.457 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:59 smithi107 podman[72488]: 2024-03-24 00:25:59.343947683 +0000 UTC m=+0.137441262 container create 310b5224fa343558e07f539a4221887285698889a5e87d49bf6716f76193d25a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:25:59.672 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 88 sec latency 201263.061 ms 2024-03-24T00:25:59.843 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:59 smithi107 podman[72488]: 2024-03-24 00:25:59.580272055 +0000 UTC m=+0.373765629 container init 310b5224fa343558e07f539a4221887285698889a5e87d49bf6716f76193d25a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:25:59.843 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:59 smithi107 podman[72488]: 2024-03-24 00:25:59.583012101 +0000 UTC m=+0.376505678 container start 310b5224fa343558e07f539a4221887285698889a5e87d49bf6716f76193d25a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-24T00:25:59.844 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:59 smithi107 ceph-mds[72507]: -- 172.21.15.107:0/812557342 <== mon.1 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e57896da40 con 0x55e578940800 2024-03-24T00:25:59.844 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:59 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: starting mds.f at 2024-03-24T00:25:59.844 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:59 smithi107 bash[72488]: 310b5224fa343558e07f539a4221887285698889a5e87d49bf6716f76193d25a 2024-03-24T00:25:59.844 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:25:59 smithi107 systemd[1]: Started Ceph mds.f for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:26:00.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:00 smithi078 ceph-mon[32429]: pgmap v158: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T00:26:00.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:00 smithi078 ceph-mon[32429]: daemon mds.f restarted 2024-03-24T00:26:00.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:00 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:26:00.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:00 smithi078 ceph-mon[32429]: osdmap e105: 12 total, 12 up, 12 in 2024-03-24T00:26:00.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:00 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6834/2447512361,v1:172.21.15.107:6835/2447512361] up:boot 2024-03-24T00:26:00.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:00 smithi078 ceph-mon[32429]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-03-24T00:26:00.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:00 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby, 1 failed 2024-03-24T00:26:00.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:00 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:26:00.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:00 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:replay,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:26:00.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:00 smithi106 ceph-mon[36505]: pgmap v158: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T00:26:00.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:00 smithi106 ceph-mon[36505]: daemon mds.f restarted 2024-03-24T00:26:00.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:00 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:26:00.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:00 smithi106 ceph-mon[36505]: osdmap e105: 12 total, 12 up, 12 in 2024-03-24T00:26:00.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:00 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6834/2447512361,v1:172.21.15.107:6835/2447512361] up:boot 2024-03-24T00:26:00.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:00 smithi106 ceph-mon[36505]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-03-24T00:26:00.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:00 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby, 1 failed 2024-03-24T00:26:00.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:00 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:26:00.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:00 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:replay,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:26:00.672 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 89 sec latency 202263.146 ms 2024-03-24T00:26:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:00 smithi107 ceph-mon[36485]: pgmap v158: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T00:26:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:00 smithi107 ceph-mon[36485]: daemon mds.f restarted 2024-03-24T00:26:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:00 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:26:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:00 smithi107 ceph-mon[36485]: osdmap e105: 12 total, 12 up, 12 in 2024-03-24T00:26:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:00 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6834/2447512361,v1:172.21.15.107:6835/2447512361] up:boot 2024-03-24T00:26:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:00 smithi107 ceph-mon[36485]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-03-24T00:26:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:00 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby, 1 failed 2024-03-24T00:26:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:00 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:26:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:00 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:replay,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:26:01.672 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 90 sec latency 203263.233 ms 2024-03-24T00:26:01.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:01 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6832/2367018868,v1:172.21.15.106:6833/2367018868] up:resolve 2024-03-24T00:26:01.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:01 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:resolve,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:26:01.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:01 smithi106 ceph-mon[36505]: pgmap v160: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:26:02.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:01 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6832/2367018868,v1:172.21.15.106:6833/2367018868] up:resolve 2024-03-24T00:26:02.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:01 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:resolve,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:26:02.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:01 smithi078 ceph-mon[32429]: pgmap v160: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:26:02.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:01 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6832/2367018868,v1:172.21.15.106:6833/2367018868] up:resolve 2024-03-24T00:26:02.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:01 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:resolve,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:26:02.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:01 smithi107 ceph-mon[36485]: pgmap v160: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:26:02.672 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 91 sec latency 204263.340 ms 2024-03-24T00:26:02.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:02 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:26:02.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:02 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6832/2367018868,v1:172.21.15.106:6833/2367018868] up:reconnect 2024-03-24T00:26:02.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:02 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:reconnect,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:26:03.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:02 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:26:03.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:02 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6832/2367018868,v1:172.21.15.106:6833/2367018868] up:reconnect 2024-03-24T00:26:03.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:02 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:reconnect,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:26:03.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:02 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:26:03.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:02 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6832/2367018868,v1:172.21.15.106:6833/2367018868] up:reconnect 2024-03-24T00:26:03.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:02 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:reconnect,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:26:03.672 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 92 sec latency 205263.387 ms 2024-03-24T00:26:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6832/2367018868,v1:172.21.15.106:6833/2367018868] up:rejoin 2024-03-24T00:26:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:rejoin,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:26:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-mon[36505]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-24T00:26:03.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-mon[36505]: pgmap v161: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:26:03.903 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3a887c9700 time 2024-03-24T00:26:03.665768+0000 2024-03-24T00:26:03.903 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:26:03.903 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:03.903 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3a92df34a1] 2024-03-24T00:26:03.904 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3a92df3667] 2024-03-24T00:26:03.904 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55dc1bf76223] 2024-03-24T00:26:03.904 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55dc1bdf05d6] 2024-03-24T00:26:03.904 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55dc1bdf0da1] 2024-03-24T00:26:03.904 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55dc1bdf32d5] 2024-03-24T00:26:03.904 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55dc1be43233] 2024-03-24T00:26:03.904 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55dc1be45197] 2024-03-24T00:26:03.904 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55dc1bd4411b] 2024-03-24T00:26:03.904 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55dc1bd446bc] 2024-03-24T00:26:03.904 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55dc1bd2e1ff] 2024-03-24T00:26:03.904 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3a93085488] 2024-03-24T00:26:03.904 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 13: (DispatchQueue::entry()+0x51f) [0x7f3a9308260f] 2024-03-24T00:26:03.904 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3a93155e81] 2024-03-24T00:26:03.905 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3a91b701ca] 2024-03-24T00:26:03.905 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 16: clone() 2024-03-24T00:26:03.905 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 2024-03-24T00:26:03.667+0000 7f3a887c9700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3a887c9700 time 2024-03-24T00:26:03.665768+0000 2024-03-24T00:26:03.905 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:26:03.905 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 2024-03-24T00:26:03.905 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:03.905 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3a92df34a1] 2024-03-24T00:26:03.905 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3a92df3667] 2024-03-24T00:26:03.905 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55dc1bf76223] 2024-03-24T00:26:03.905 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55dc1bdf05d6] 2024-03-24T00:26:03.905 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55dc1bdf0da1] 2024-03-24T00:26:03.905 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55dc1bdf32d5] 2024-03-24T00:26:03.905 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55dc1be43233] 2024-03-24T00:26:03.906 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55dc1be45197] 2024-03-24T00:26:03.906 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55dc1bd4411b] 2024-03-24T00:26:03.906 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55dc1bd446bc] 2024-03-24T00:26:03.906 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55dc1bd2e1ff] 2024-03-24T00:26:03.906 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3a93085488] 2024-03-24T00:26:03.906 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 13: (DispatchQueue::entry()+0x51f) [0x7f3a9308260f] 2024-03-24T00:26:03.906 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3a93155e81] 2024-03-24T00:26:03.906 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3a91b701ca] 2024-03-24T00:26:03.906 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 16: clone() 2024-03-24T00:26:03.906 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 2024-03-24T00:26:03.906 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: *** Caught signal (Aborted) ** 2024-03-24T00:26:03.906 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: in thread 7f3a887c9700 thread_name:ms_dispatch 2024-03-24T00:26:03.906 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:03.906 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3a91b7ad20] 2024-03-24T00:26:03.907 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 2: gsignal() 2024-03-24T00:26:03.908 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 3: abort() 2024-03-24T00:26:03.908 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3a92df34fb] 2024-03-24T00:26:03.908 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3a92df3667] 2024-03-24T00:26:03.908 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55dc1bf76223] 2024-03-24T00:26:03.908 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55dc1bdf05d6] 2024-03-24T00:26:03.908 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55dc1bdf0da1] 2024-03-24T00:26:03.908 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55dc1bdf32d5] 2024-03-24T00:26:03.908 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55dc1be43233] 2024-03-24T00:26:03.908 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55dc1be45197] 2024-03-24T00:26:03.908 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55dc1bd4411b] 2024-03-24T00:26:03.908 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55dc1bd446bc] 2024-03-24T00:26:03.908 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55dc1bd2e1ff] 2024-03-24T00:26:03.908 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3a93085488] 2024-03-24T00:26:03.908 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 16: (DispatchQueue::entry()+0x51f) [0x7f3a9308260f] 2024-03-24T00:26:03.908 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3a93155e81] 2024-03-24T00:26:03.909 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3a91b701ca] 2024-03-24T00:26:03.909 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 19: clone() 2024-03-24T00:26:03.909 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 2024-03-24T00:26:03.668+0000 7f3a887c9700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:26:03.909 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: in thread 7f3a887c9700 thread_name:ms_dispatch 2024-03-24T00:26:03.909 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 2024-03-24T00:26:03.909 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:03.909 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3a91b7ad20] 2024-03-24T00:26:03.909 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 2: gsignal() 2024-03-24T00:26:03.909 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 3: abort() 2024-03-24T00:26:03.909 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3a92df34fb] 2024-03-24T00:26:03.909 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3a92df3667] 2024-03-24T00:26:03.909 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55dc1bf76223] 2024-03-24T00:26:03.909 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55dc1bdf05d6] 2024-03-24T00:26:03.909 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55dc1bdf0da1] 2024-03-24T00:26:03.909 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55dc1bdf32d5] 2024-03-24T00:26:03.909 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55dc1be43233] 2024-03-24T00:26:03.910 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55dc1be45197] 2024-03-24T00:26:03.910 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55dc1bd4411b] 2024-03-24T00:26:03.910 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55dc1bd446bc] 2024-03-24T00:26:03.910 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55dc1bd2e1ff] 2024-03-24T00:26:03.910 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3a93085488] 2024-03-24T00:26:03.910 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 16: (DispatchQueue::entry()+0x51f) [0x7f3a9308260f] 2024-03-24T00:26:03.910 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3a93155e81] 2024-03-24T00:26:03.910 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3a91b701ca] 2024-03-24T00:26:03.910 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 19: clone() 2024-03-24T00:26:03.910 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:26:03.910 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 2024-03-24T00:26:03.910 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: -1> 2024-03-24T00:26:03.667+0000 7f3a887c9700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3a887c9700 time 2024-03-24T00:26:03.665768+0000 2024-03-24T00:26:03.910 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:26:03.910 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 2024-03-24T00:26:03.910 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:03.911 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3a92df34a1] 2024-03-24T00:26:03.911 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3a92df3667] 2024-03-24T00:26:03.911 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55dc1bf76223] 2024-03-24T00:26:03.911 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55dc1bdf05d6] 2024-03-24T00:26:03.911 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55dc1bdf0da1] 2024-03-24T00:26:03.911 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55dc1bdf32d5] 2024-03-24T00:26:03.911 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55dc1be43233] 2024-03-24T00:26:03.911 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55dc1be45197] 2024-03-24T00:26:03.911 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55dc1bd4411b] 2024-03-24T00:26:03.911 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55dc1bd446bc] 2024-03-24T00:26:03.911 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55dc1bd2e1ff] 2024-03-24T00:26:03.911 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3a93085488] 2024-03-24T00:26:03.911 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 13: (DispatchQueue::entry()+0x51f) [0x7f3a9308260f] 2024-03-24T00:26:03.911 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3a93155e81] 2024-03-24T00:26:03.911 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3a91b701ca] 2024-03-24T00:26:03.912 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 16: clone() 2024-03-24T00:26:03.912 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 2024-03-24T00:26:03.912 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 0> 2024-03-24T00:26:03.668+0000 7f3a887c9700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:26:03.912 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: in thread 7f3a887c9700 thread_name:ms_dispatch 2024-03-24T00:26:03.912 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 2024-03-24T00:26:03.912 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:03.912 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3a91b7ad20] 2024-03-24T00:26:03.912 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 2: gsignal() 2024-03-24T00:26:03.912 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 3: abort() 2024-03-24T00:26:03.912 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3a92df34fb] 2024-03-24T00:26:03.912 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3a92df3667] 2024-03-24T00:26:03.912 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55dc1bf76223] 2024-03-24T00:26:03.912 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55dc1bdf05d6] 2024-03-24T00:26:03.912 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55dc1bdf0da1] 2024-03-24T00:26:03.912 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55dc1bdf32d5] 2024-03-24T00:26:03.913 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55dc1be43233] 2024-03-24T00:26:03.913 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55dc1be45197] 2024-03-24T00:26:03.913 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55dc1bd4411b] 2024-03-24T00:26:03.913 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55dc1bd446bc] 2024-03-24T00:26:03.913 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55dc1bd2e1ff] 2024-03-24T00:26:03.913 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3a93085488] 2024-03-24T00:26:03.913 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 16: (DispatchQueue::entry()+0x51f) [0x7f3a9308260f] 2024-03-24T00:26:03.913 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3a93155e81] 2024-03-24T00:26:03.913 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3a91b701ca] 2024-03-24T00:26:03.913 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 19: clone() 2024-03-24T00:26:03.913 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:26:03.913 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 2024-03-24T00:26:03.913 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: -1> 2024-03-24T00:26:03.667+0000 7f3a887c9700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3a887c9700 time 2024-03-24T00:26:03.665768+0000 2024-03-24T00:26:03.913 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:26:03.913 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 2024-03-24T00:26:03.913 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:03.914 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3a92df34a1] 2024-03-24T00:26:03.914 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3a92df3667] 2024-03-24T00:26:03.914 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55dc1bf76223] 2024-03-24T00:26:03.914 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55dc1bdf05d6] 2024-03-24T00:26:03.914 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55dc1bdf0da1] 2024-03-24T00:26:03.914 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55dc1bdf32d5] 2024-03-24T00:26:03.914 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55dc1be43233] 2024-03-24T00:26:03.914 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55dc1be45197] 2024-03-24T00:26:03.914 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55dc1bd4411b] 2024-03-24T00:26:03.914 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55dc1bd446bc] 2024-03-24T00:26:03.914 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55dc1bd2e1ff] 2024-03-24T00:26:03.914 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3a93085488] 2024-03-24T00:26:03.914 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 13: (DispatchQueue::entry()+0x51f) [0x7f3a9308260f] 2024-03-24T00:26:03.914 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3a93155e81] 2024-03-24T00:26:03.914 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3a91b701ca] 2024-03-24T00:26:03.915 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 16: clone() 2024-03-24T00:26:03.915 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 2024-03-24T00:26:03.915 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 0> 2024-03-24T00:26:03.668+0000 7f3a887c9700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:26:03.915 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: in thread 7f3a887c9700 thread_name:ms_dispatch 2024-03-24T00:26:03.915 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 2024-03-24T00:26:03.915 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:03.915 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3a91b7ad20] 2024-03-24T00:26:03.915 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 2: gsignal() 2024-03-24T00:26:03.915 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 3: abort() 2024-03-24T00:26:03.915 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3a92df34fb] 2024-03-24T00:26:03.915 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3a92df3667] 2024-03-24T00:26:03.915 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55dc1bf76223] 2024-03-24T00:26:03.915 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55dc1bdf05d6] 2024-03-24T00:26:03.915 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55dc1bdf0da1] 2024-03-24T00:26:03.915 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55dc1bdf32d5] 2024-03-24T00:26:03.915 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55dc1be43233] 2024-03-24T00:26:03.916 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55dc1be45197] 2024-03-24T00:26:03.916 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55dc1bd4411b] 2024-03-24T00:26:03.916 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55dc1bd446bc] 2024-03-24T00:26:03.916 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55dc1bd2e1ff] 2024-03-24T00:26:03.916 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3a93085488] 2024-03-24T00:26:03.916 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 16: (DispatchQueue::entry()+0x51f) [0x7f3a9308260f] 2024-03-24T00:26:03.916 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3a93155e81] 2024-03-24T00:26:03.916 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3a91b701ca] 2024-03-24T00:26:03.916 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 19: clone() 2024-03-24T00:26:03.916 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:26:03.916 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[66042]: 2024-03-24T00:26:03.916 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:03 smithi106 podman[70490]: 2024-03-24 00:26:03.720506318 +0000 UTC m=+0.022785169 container died 98bb5f2ad84ddb896d35c650fbdfcc6b9b3f5bb989d008f61c9c311c18922d17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:26:04.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:03 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6832/2367018868,v1:172.21.15.106:6833/2367018868] up:rejoin 2024-03-24T00:26:04.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:03 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:rejoin,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:26:04.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:03 smithi078 ceph-mon[32429]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-24T00:26:04.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:03 smithi078 ceph-mon[32429]: pgmap v161: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:26:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:03 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6832/2367018868,v1:172.21.15.106:6833/2367018868] up:rejoin 2024-03-24T00:26:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:03 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:rejoin,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:26:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:03 smithi107 ceph-mon[36485]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-24T00:26:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:03 smithi107 ceph-mon[36485]: pgmap v161: 129 pgs: 129 active+clean; 27 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:26:04.672 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 93 sec latency 206263.492 ms 2024-03-24T00:26:04.827 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:04 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:26:04.827 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:04 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:26:04.827 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:04 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6832/2367018868,v1:172.21.15.106:6833/2367018868] up:active 2024-03-24T00:26:04.828 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:04 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:26:04.828 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:04 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=h=up:active} 3 up:standby-replay 5 up:standby 2024-03-24T00:26:04.828 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:04 smithi106 podman[70490]: 2024-03-24 00:26:04.565099339 +0000 UTC m=+0.867378169 container remove 98bb5f2ad84ddb896d35c650fbdfcc6b9b3f5bb989d008f61c9c311c18922d17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-24T00:26:04.828 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:04 smithi106 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:26:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:04 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:26:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:04 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:26:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:04 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6832/2367018868,v1:172.21.15.106:6833/2367018868] up:active 2024-03-24T00:26:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:04 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:26:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:04 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=h=up:active} 3 up:standby-replay 5 up:standby 2024-03-24T00:26:05.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:04 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:26:05.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:04 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:26:05.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:04 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6832/2367018868,v1:172.21.15.106:6833/2367018868] up:active 2024-03-24T00:26:05.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:04 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:26:05.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:04 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=h=up:active} 3 up:standby-replay 5 up:standby 2024-03-24T00:26:05.152 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:04 smithi106 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.k.service: Failed with result 'exit-code'. 2024-03-24T00:26:05.672 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 94 sec latency 207263.624 ms 2024-03-24T00:26:05.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:05 smithi078 ceph-mon[32429]: pgmap v162: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-24T00:26:05.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:05 smithi106 ceph-mon[36505]: pgmap v162: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-24T00:26:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:05 smithi107 ceph-mon[36485]: pgmap v162: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-24T00:26:06.672 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 95 sec latency 208263.717 ms 2024-03-24T00:26:07.673 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 96 sec latency 209263.763 ms 2024-03-24T00:26:08.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:08 smithi106 ceph-mon[36505]: pgmap v163: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-24T00:26:08.672 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 97 sec latency 210263.871 ms 2024-03-24T00:26:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:08 smithi078 ceph-mon[32429]: pgmap v163: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-24T00:26:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:08 smithi107 ceph-mon[36485]: pgmap v163: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-24T00:26:09.673 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 98 sec latency 211264.018 ms 2024-03-24T00:26:10.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:10 smithi106 ceph-mon[36505]: pgmap v164: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-24T00:26:10.673 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 99 sec latency 212264.066 ms 2024-03-24T00:26:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:10 smithi078 ceph-mon[32429]: pgmap v164: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-24T00:26:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:10 smithi107 ceph-mon[36485]: pgmap v164: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-24T00:26:11.673 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 100 sec latency 213264.171 ms 2024-03-24T00:26:12.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:12 smithi106 ceph-mon[36505]: pgmap v165: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-24T00:26:12.674 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 101 sec latency 214264.267 ms 2024-03-24T00:26:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:12 smithi078 ceph-mon[32429]: pgmap v165: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-24T00:26:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:12 smithi107 ceph-mon[36485]: pgmap v165: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-24T00:26:13.673 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 102 sec latency 215264.382 ms 2024-03-24T00:26:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:14 smithi106 ceph-mon[36505]: pgmap v166: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-24T00:26:14.673 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 103 sec latency 216264.500 ms 2024-03-24T00:26:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:14 smithi078 ceph-mon[32429]: pgmap v166: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-24T00:26:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:14 smithi107 ceph-mon[36485]: pgmap v166: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-24T00:26:15.270 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:14 smithi106 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.k.service: Scheduled restart job, restart counter is at 1. 2024-03-24T00:26:15.270 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:14 smithi106 systemd[1]: Stopped Ceph mds.k for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:26:15.270 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:14 smithi106 systemd[1]: Starting Ceph mds.k for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:26:15.270 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:15 smithi106 podman[70583]: 2024-03-24 00:26:15.031845042 +0000 UTC m=+0.018451223 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:26:15.270 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:15 smithi106 podman[70583]: 2024-03-24 00:26:15.14963775 +0000 UTC m=+0.136243916 container create 8cb69d332574a1f9c6e077e8cc6b05ee14708764207e3adacac9a21f26cbaaa2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-24T00:26:15.652 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:15 smithi106 podman[70583]: 2024-03-24 00:26:15.402564817 +0000 UTC m=+0.389171006 container init 8cb69d332574a1f9c6e077e8cc6b05ee14708764207e3adacac9a21f26cbaaa2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-24T00:26:15.652 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:15 smithi106 podman[70583]: 2024-03-24 00:26:15.405229362 +0000 UTC m=+0.391835543 container start 8cb69d332574a1f9c6e077e8cc6b05ee14708764207e3adacac9a21f26cbaaa2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-24T00:26:15.653 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:15 smithi106 ceph-mds[70602]: -- 172.21.15.106:0/1300467658 <== mon.2 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556fbe783a40 con 0x556fbe757000 2024-03-24T00:26:15.653 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:15 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: starting mds.k at 2024-03-24T00:26:15.653 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:15 smithi106 bash[70583]: 8cb69d332574a1f9c6e077e8cc6b05ee14708764207e3adacac9a21f26cbaaa2 2024-03-24T00:26:15.653 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:15 smithi106 systemd[1]: Started Ceph mds.k for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:26:15.673 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 104 sec latency 217264.630 ms 2024-03-24T00:26:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:16 smithi106 ceph-mon[36505]: pgmap v167: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-03-24T00:26:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:16 smithi106 ceph-mon[36505]: daemon mds.k restarted 2024-03-24T00:26:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:16 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:26:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:16 smithi106 ceph-mon[36505]: osdmap e106: 12 total, 12 up, 12 in 2024-03-24T00:26:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:16 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6832/2611554095,v1:172.21.15.106:6833/2611554095] up:boot 2024-03-24T00:26:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:16 smithi106 ceph-mon[36505]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-24T00:26:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:16 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby, 1 failed 2024-03-24T00:26:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T00:26:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:16 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:replay,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:26:16.673 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 105 sec latency 218264.706 ms 2024-03-24T00:26:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:16 smithi078 ceph-mon[32429]: pgmap v167: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-03-24T00:26:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:16 smithi078 ceph-mon[32429]: daemon mds.k restarted 2024-03-24T00:26:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:16 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:26:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:16 smithi078 ceph-mon[32429]: osdmap e106: 12 total, 12 up, 12 in 2024-03-24T00:26:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:16 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6832/2611554095,v1:172.21.15.106:6833/2611554095] up:boot 2024-03-24T00:26:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:16 smithi078 ceph-mon[32429]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-24T00:26:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:16 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby, 1 failed 2024-03-24T00:26:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T00:26:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:16 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:replay,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:26:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:16 smithi107 ceph-mon[36485]: pgmap v167: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-03-24T00:26:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:16 smithi107 ceph-mon[36485]: daemon mds.k restarted 2024-03-24T00:26:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:16 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:26:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:16 smithi107 ceph-mon[36485]: osdmap e106: 12 total, 12 up, 12 in 2024-03-24T00:26:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:16 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6832/2611554095,v1:172.21.15.106:6833/2611554095] up:boot 2024-03-24T00:26:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:16 smithi107 ceph-mon[36485]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-24T00:26:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:16 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby, 1 failed 2024-03-24T00:26:16.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T00:26:16.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:16 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:replay,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:26:17.673 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 106 sec latency 219264.790 ms 2024-03-24T00:26:17.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:17 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6834/2447512361,v1:172.21.15.107:6835/2447512361] up:resolve 2024-03-24T00:26:17.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:17 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:resolve,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:26:17.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:17 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6834/2447512361,v1:172.21.15.107:6835/2447512361] up:resolve 2024-03-24T00:26:17.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:17 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:resolve,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:26:17.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:17 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6834/2447512361,v1:172.21.15.107:6835/2447512361] up:resolve 2024-03-24T00:26:17.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:17 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:resolve,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:26:18.673 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 107 sec latency 220264.868 ms 2024-03-24T00:26:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:18 smithi078 ceph-mon[32429]: pgmap v169: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:26:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:18 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:26:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:18 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6834/2447512361,v1:172.21.15.107:6835/2447512361] up:reconnect 2024-03-24T00:26:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:18 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:reconnect,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:26:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:18 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:26:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-mon[36485]: pgmap v169: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:26:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:26:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6834/2447512361,v1:172.21.15.107:6835/2447512361] up:reconnect 2024-03-24T00:26:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:reconnect,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:26:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:26:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:18 smithi106 ceph-mon[36505]: pgmap v169: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:26:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:18 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:26:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:18 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6834/2447512361,v1:172.21.15.107:6835/2447512361] up:reconnect 2024-03-24T00:26:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:18 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:reconnect,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:26:18.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:18 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:26:19.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f4505016700 time 2024-03-24T00:26:18.997198+0000 2024-03-24T00:26:19.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:26:19.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:19.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f450f6404a1] 2024-03-24T00:26:19.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f450f640667] 2024-03-24T00:26:19.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e577ade223] 2024-03-24T00:26:19.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e5779585d6] 2024-03-24T00:26:19.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e577958da1] 2024-03-24T00:26:19.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e57795b2d5] 2024-03-24T00:26:19.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e5779ab233] 2024-03-24T00:26:19.333 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e5779ad197] 2024-03-24T00:26:19.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e5778ac11b] 2024-03-24T00:26:19.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e5778ac6bc] 2024-03-24T00:26:19.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e5778961ff] 2024-03-24T00:26:19.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f450f8d2488] 2024-03-24T00:26:19.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 13: (DispatchQueue::entry()+0x51f) [0x7f450f8cf60f] 2024-03-24T00:26:19.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f450f9a2e81] 2024-03-24T00:26:19.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f450e3bd1ca] 2024-03-24T00:26:19.334 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 16: clone() 2024-03-24T00:26:19.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 2024-03-24T00:26:18.997+0000 7f4505016700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f4505016700 time 2024-03-24T00:26:18.997198+0000 2024-03-24T00:26:19.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:26:19.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 2024-03-24T00:26:19.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:19.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f450f6404a1] 2024-03-24T00:26:19.335 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f450f640667] 2024-03-24T00:26:19.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e577ade223] 2024-03-24T00:26:19.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e5779585d6] 2024-03-24T00:26:19.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e577958da1] 2024-03-24T00:26:19.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e57795b2d5] 2024-03-24T00:26:19.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e5779ab233] 2024-03-24T00:26:19.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e5779ad197] 2024-03-24T00:26:19.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e5778ac11b] 2024-03-24T00:26:19.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e5778ac6bc] 2024-03-24T00:26:19.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e5778961ff] 2024-03-24T00:26:19.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f450f8d2488] 2024-03-24T00:26:19.336 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 13: (DispatchQueue::entry()+0x51f) [0x7f450f8cf60f] 2024-03-24T00:26:19.337 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f450f9a2e81] 2024-03-24T00:26:19.337 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f450e3bd1ca] 2024-03-24T00:26:19.337 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 16: clone() 2024-03-24T00:26:19.337 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 2024-03-24T00:26:19.337 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: *** Caught signal (Aborted) ** 2024-03-24T00:26:19.337 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: in thread 7f4505016700 thread_name:ms_dispatch 2024-03-24T00:26:19.337 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:19.337 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f450e3c7d20] 2024-03-24T00:26:19.337 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 2: gsignal() 2024-03-24T00:26:19.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 3: abort() 2024-03-24T00:26:19.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f450f6404fb] 2024-03-24T00:26:19.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f450f640667] 2024-03-24T00:26:19.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e577ade223] 2024-03-24T00:26:19.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e5779585d6] 2024-03-24T00:26:19.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e577958da1] 2024-03-24T00:26:19.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e57795b2d5] 2024-03-24T00:26:19.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e5779ab233] 2024-03-24T00:26:19.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e5779ad197] 2024-03-24T00:26:19.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e5778ac11b] 2024-03-24T00:26:19.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e5778ac6bc] 2024-03-24T00:26:19.338 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e5778961ff] 2024-03-24T00:26:19.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f450f8d2488] 2024-03-24T00:26:19.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 16: (DispatchQueue::entry()+0x51f) [0x7f450f8cf60f] 2024-03-24T00:26:19.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f450f9a2e81] 2024-03-24T00:26:19.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f450e3bd1ca] 2024-03-24T00:26:19.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 19: clone() 2024-03-24T00:26:19.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 2024-03-24T00:26:18.999+0000 7f4505016700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:26:19.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: in thread 7f4505016700 thread_name:ms_dispatch 2024-03-24T00:26:19.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 2024-03-24T00:26:19.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:19.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f450e3c7d20] 2024-03-24T00:26:19.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 2: gsignal() 2024-03-24T00:26:19.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 3: abort() 2024-03-24T00:26:19.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f450f6404fb] 2024-03-24T00:26:19.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f450f640667] 2024-03-24T00:26:19.339 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e577ade223] 2024-03-24T00:26:19.340 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e5779585d6] 2024-03-24T00:26:19.340 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e577958da1] 2024-03-24T00:26:19.340 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e57795b2d5] 2024-03-24T00:26:19.340 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e5779ab233] 2024-03-24T00:26:19.340 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e5779ad197] 2024-03-24T00:26:19.340 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e5778ac11b] 2024-03-24T00:26:19.340 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e5778ac6bc] 2024-03-24T00:26:19.340 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e5778961ff] 2024-03-24T00:26:19.340 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f450f8d2488] 2024-03-24T00:26:19.340 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 16: (DispatchQueue::entry()+0x51f) [0x7f450f8cf60f] 2024-03-24T00:26:19.340 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f450f9a2e81] 2024-03-24T00:26:19.340 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f450e3bd1ca] 2024-03-24T00:26:19.340 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 19: clone() 2024-03-24T00:26:19.341 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:26:19.341 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 2024-03-24T00:26:19.341 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: -1> 2024-03-24T00:26:18.997+0000 7f4505016700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f4505016700 time 2024-03-24T00:26:18.997198+0000 2024-03-24T00:26:19.341 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:26:19.341 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 2024-03-24T00:26:19.341 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:19.341 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f450f6404a1] 2024-03-24T00:26:19.341 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f450f640667] 2024-03-24T00:26:19.341 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e577ade223] 2024-03-24T00:26:19.341 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e5779585d6] 2024-03-24T00:26:19.341 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e577958da1] 2024-03-24T00:26:19.341 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e57795b2d5] 2024-03-24T00:26:19.342 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e5779ab233] 2024-03-24T00:26:19.342 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e5779ad197] 2024-03-24T00:26:19.342 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e5778ac11b] 2024-03-24T00:26:19.342 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e5778ac6bc] 2024-03-24T00:26:19.342 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e5778961ff] 2024-03-24T00:26:19.342 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f450f8d2488] 2024-03-24T00:26:19.342 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 13: (DispatchQueue::entry()+0x51f) [0x7f450f8cf60f] 2024-03-24T00:26:19.342 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f450f9a2e81] 2024-03-24T00:26:19.342 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f450e3bd1ca] 2024-03-24T00:26:19.343 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 16: clone() 2024-03-24T00:26:19.343 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 2024-03-24T00:26:19.343 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 0> 2024-03-24T00:26:18.999+0000 7f4505016700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:26:19.343 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: in thread 7f4505016700 thread_name:ms_dispatch 2024-03-24T00:26:19.343 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 2024-03-24T00:26:19.343 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:19.343 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f450e3c7d20] 2024-03-24T00:26:19.343 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 2: gsignal() 2024-03-24T00:26:19.343 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 3: abort() 2024-03-24T00:26:19.343 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f450f6404fb] 2024-03-24T00:26:19.343 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f450f640667] 2024-03-24T00:26:19.343 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e577ade223] 2024-03-24T00:26:19.343 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e5779585d6] 2024-03-24T00:26:19.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e577958da1] 2024-03-24T00:26:19.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e57795b2d5] 2024-03-24T00:26:19.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e5779ab233] 2024-03-24T00:26:19.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e5779ad197] 2024-03-24T00:26:19.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e5778ac11b] 2024-03-24T00:26:19.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e5778ac6bc] 2024-03-24T00:26:19.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e5778961ff] 2024-03-24T00:26:19.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f450f8d2488] 2024-03-24T00:26:19.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 16: (DispatchQueue::entry()+0x51f) [0x7f450f8cf60f] 2024-03-24T00:26:19.344 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f450f9a2e81] 2024-03-24T00:26:19.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f450e3bd1ca] 2024-03-24T00:26:19.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 19: clone() 2024-03-24T00:26:19.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:26:19.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 2024-03-24T00:26:19.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: -1> 2024-03-24T00:26:18.997+0000 7f4505016700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f4505016700 time 2024-03-24T00:26:18.997198+0000 2024-03-24T00:26:19.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:26:19.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 2024-03-24T00:26:19.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:19.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f450f6404a1] 2024-03-24T00:26:19.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f450f640667] 2024-03-24T00:26:19.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e577ade223] 2024-03-24T00:26:19.345 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e5779585d6] 2024-03-24T00:26:19.346 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e577958da1] 2024-03-24T00:26:19.346 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e57795b2d5] 2024-03-24T00:26:19.346 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e5779ab233] 2024-03-24T00:26:19.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e5779ad197] 2024-03-24T00:26:19.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e5778ac11b] 2024-03-24T00:26:19.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e5778ac6bc] 2024-03-24T00:26:19.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e5778961ff] 2024-03-24T00:26:19.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f450f8d2488] 2024-03-24T00:26:19.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 13: (DispatchQueue::entry()+0x51f) [0x7f450f8cf60f] 2024-03-24T00:26:19.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f450f9a2e81] 2024-03-24T00:26:19.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f450e3bd1ca] 2024-03-24T00:26:19.347 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 16: clone() 2024-03-24T00:26:19.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 2024-03-24T00:26:19.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 0> 2024-03-24T00:26:18.999+0000 7f4505016700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:26:19.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: in thread 7f4505016700 thread_name:ms_dispatch 2024-03-24T00:26:19.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 2024-03-24T00:26:19.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:19.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f450e3c7d20] 2024-03-24T00:26:19.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 2: gsignal() 2024-03-24T00:26:19.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 3: abort() 2024-03-24T00:26:19.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f450f6404fb] 2024-03-24T00:26:19.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f450f640667] 2024-03-24T00:26:19.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e577ade223] 2024-03-24T00:26:19.348 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e5779585d6] 2024-03-24T00:26:19.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e577958da1] 2024-03-24T00:26:19.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e57795b2d5] 2024-03-24T00:26:19.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e5779ab233] 2024-03-24T00:26:19.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e5779ad197] 2024-03-24T00:26:19.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e5778ac11b] 2024-03-24T00:26:19.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e5778ac6bc] 2024-03-24T00:26:19.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e5778961ff] 2024-03-24T00:26:19.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f450f8d2488] 2024-03-24T00:26:19.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 16: (DispatchQueue::entry()+0x51f) [0x7f450f8cf60f] 2024-03-24T00:26:19.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f450f9a2e81] 2024-03-24T00:26:19.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f450e3bd1ca] 2024-03-24T00:26:19.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 19: clone() 2024-03-24T00:26:19.349 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:26:19.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f[72503]: 2024-03-24T00:26:19.350 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 podman[72673]: 2024-03-24 00:26:19.042218716 +0000 UTC m=+0.018987177 container died 310b5224fa343558e07f539a4221887285698889a5e87d49bf6716f76193d25a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-24T00:26:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6834/2447512361,v1:172.21.15.107:6835/2447512361] up:rejoin 2024-03-24T00:26:19.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:rejoin,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:26:19.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-mon[36485]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-24T00:26:19.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:26:19.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:26:19.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:26:19.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:26:19.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:26:19.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6834/2447512361,v1:172.21.15.107:6835/2447512361] up:active 2024-03-24T00:26:19.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:19 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=h=up:active} 3 up:standby-replay 5 up:standby 2024-03-24T00:26:19.674 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 108 sec latency 221264.969 ms 2024-03-24T00:26:19.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:19 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6834/2447512361,v1:172.21.15.107:6835/2447512361] up:rejoin 2024-03-24T00:26:19.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:19 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:rejoin,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:26:19.844 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:19 smithi078 ceph-mon[32429]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-24T00:26:19.844 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:19 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:26:19.844 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:19 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:26:19.844 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:19 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:26:19.844 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:19 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:26:19.844 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:19 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:26:19.844 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:19 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6834/2447512361,v1:172.21.15.107:6835/2447512361] up:active 2024-03-24T00:26:19.844 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:19 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=h=up:active} 3 up:standby-replay 5 up:standby 2024-03-24T00:26:19.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:19 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6834/2447512361,v1:172.21.15.107:6835/2447512361] up:rejoin 2024-03-24T00:26:19.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:19 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:rejoin,2=h=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:26:19.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:19 smithi106 ceph-mon[36505]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-24T00:26:19.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:19 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:26:19.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:19 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:26:19.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:19 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:26:19.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:19 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:26:19.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:19 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:26:19.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:19 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6834/2447512361,v1:172.21.15.107:6835/2447512361] up:active 2024-03-24T00:26:19.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:19 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=h=up:active} 3 up:standby-replay 5 up:standby 2024-03-24T00:26:20.093 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 podman[72673]: 2024-03-24 00:26:19.703018318 +0000 UTC m=+0.679786798 container remove 310b5224fa343558e07f539a4221887285698889a5e87d49bf6716f76193d25a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-f, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-24T00:26:20.093 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:26:20.094 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:19 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.f.service: Failed with result 'exit-code'. 2024-03-24T00:26:20.674 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 109 sec latency 222265.020 ms 2024-03-24T00:26:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:20 smithi078 ceph-mon[32429]: pgmap v170: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:26:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:20 smithi107 ceph-mon[36485]: pgmap v170: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:26:20.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:20 smithi106 ceph-mon[36505]: pgmap v170: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:26:21.674 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 110 sec latency 223265.060 ms 2024-03-24T00:26:22.674 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 111 sec latency 224265.165 ms 2024-03-24T00:26:22.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:22 smithi078 ceph-mon[32429]: pgmap v171: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-24T00:26:22.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:22 smithi107 ceph-mon[36485]: pgmap v171: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-24T00:26:22.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:22 smithi106 ceph-mon[36505]: pgmap v171: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-24T00:26:23.674 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 112 sec latency 225265.256 ms 2024-03-24T00:26:24.674 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 113 sec latency 226265.348 ms 2024-03-24T00:26:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:24 smithi078 ceph-mon[32429]: pgmap v172: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-24T00:26:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:24 smithi107 ceph-mon[36485]: pgmap v172: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-24T00:26:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:24 smithi106 ceph-mon[36505]: pgmap v172: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-24T00:26:25.674 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 114 sec latency 227265.424 ms 2024-03-24T00:26:26.674 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 115 sec latency 228265.510 ms 2024-03-24T00:26:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:26 smithi078 ceph-mon[32429]: pgmap v173: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:26:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:26 smithi107 ceph-mon[36485]: pgmap v173: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:26:26.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:26 smithi106 ceph-mon[36505]: pgmap v173: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:26:27.674 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 116 sec latency 229265.616 ms 2024-03-24T00:26:28.674 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 117 sec latency 230265.667 ms 2024-03-24T00:26:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:28 smithi078 ceph-mon[32429]: pgmap v174: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 172 B/s wr, 9 op/s 2024-03-24T00:26:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:28 smithi107 ceph-mon[36485]: pgmap v174: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 172 B/s wr, 9 op/s 2024-03-24T00:26:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:28 smithi106 ceph-mon[36505]: pgmap v174: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 172 B/s wr, 9 op/s 2024-03-24T00:26:29.674 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 118 sec latency 231265.736 ms 2024-03-24T00:26:30.487 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:30 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.f.service: Scheduled restart job, restart counter is at 6. 2024-03-24T00:26:30.487 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:30 smithi107 systemd[1]: Stopped Ceph mds.f for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:26:30.488 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:30 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.f.service: Start request repeated too quickly. 2024-03-24T00:26:30.488 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:30 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.f.service: Failed with result 'exit-code'. 2024-03-24T00:26:30.488 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 24 00:26:30 smithi107 systemd[1]: Failed to start Ceph mds.f for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:26:30.675 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 119 sec latency 232265.822 ms 2024-03-24T00:26:30.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:30 smithi078 ceph-mon[32429]: pgmap v175: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-24T00:26:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:30 smithi107 ceph-mon[36485]: pgmap v175: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-24T00:26:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:30 smithi106 ceph-mon[36505]: pgmap v175: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-24T00:26:31.675 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 120 sec latency 233265.909 ms 2024-03-24T00:26:32.675 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 121 sec latency 234265.997 ms 2024-03-24T00:26:32.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:32 smithi078 ceph-mon[32429]: pgmap v176: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-24T00:26:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:32 smithi107 ceph-mon[36485]: pgmap v176: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-24T00:26:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:32 smithi106 ceph-mon[36505]: pgmap v176: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-24T00:26:33.675 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 122 sec latency 235266.044 ms 2024-03-24T00:26:34.675 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 123 sec latency 236266.135 ms 2024-03-24T00:26:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:34 smithi078 ceph-mon[32429]: pgmap v177: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:26:34.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:34 smithi078 ceph-mon[32429]: Replacing daemon mds.f as rank 1 with standby daemon mds.k 2024-03-24T00:26:34.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:34 smithi078 ceph-mon[32429]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-03-24T00:26:34.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:34 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:26:34.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:34 smithi078 ceph-mon[32429]: osdmap e107: 12 total, 12 up, 12 in 2024-03-24T00:26:34.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:34 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:replay,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:26:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:34 smithi107 ceph-mon[36485]: pgmap v177: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:26:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:34 smithi107 ceph-mon[36485]: Replacing daemon mds.f as rank 1 with standby daemon mds.k 2024-03-24T00:26:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:34 smithi107 ceph-mon[36485]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-03-24T00:26:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:34 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:26:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:34 smithi107 ceph-mon[36485]: osdmap e107: 12 total, 12 up, 12 in 2024-03-24T00:26:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:34 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:replay,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:26:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:34 smithi106 ceph-mon[36505]: pgmap v177: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:26:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:34 smithi106 ceph-mon[36505]: Replacing daemon mds.f as rank 1 with standby daemon mds.k 2024-03-24T00:26:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:34 smithi106 ceph-mon[36505]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-03-24T00:26:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:34 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:26:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:34 smithi106 ceph-mon[36505]: osdmap e107: 12 total, 12 up, 12 in 2024-03-24T00:26:34.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:34 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:replay,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:26:35.675 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 124 sec latency 237266.253 ms 2024-03-24T00:26:36.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:35 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6832/2611554095,v1:172.21.15.106:6833/2611554095] up:resolve 2024-03-24T00:26:36.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:35 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:resolve,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:26:36.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:35 smithi078 ceph-mon[32429]: pgmap v179: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:26:36.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:35 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6832/2611554095,v1:172.21.15.106:6833/2611554095] up:resolve 2024-03-24T00:26:36.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:35 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:resolve,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:26:36.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:35 smithi107 ceph-mon[36485]: pgmap v179: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:26:36.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:35 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6832/2611554095,v1:172.21.15.106:6833/2611554095] up:resolve 2024-03-24T00:26:36.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:35 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:resolve,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:26:36.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:35 smithi106 ceph-mon[36505]: pgmap v179: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:26:36.675 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 125 sec latency 238266.354 ms 2024-03-24T00:26:37.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:36 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999978 2024-03-24T00:26:37.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:36 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6832/2611554095,v1:172.21.15.106:6833/2611554095] up:reconnect 2024-03-24T00:26:37.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:36 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:reconnect,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:26:37.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:36 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6832/2611554095,v1:172.21.15.106:6833/2611554095] up:rejoin 2024-03-24T00:26:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:36 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999978 2024-03-24T00:26:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:36 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6832/2611554095,v1:172.21.15.106:6833/2611554095] up:reconnect 2024-03-24T00:26:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:36 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:reconnect,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:26:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:36 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6832/2611554095,v1:172.21.15.106:6833/2611554095] up:rejoin 2024-03-24T00:26:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:36 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999978 2024-03-24T00:26:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:36 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6832/2611554095,v1:172.21.15.106:6833/2611554095] up:reconnect 2024-03-24T00:26:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:36 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:reconnect,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:26:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:36 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6832/2611554095,v1:172.21.15.106:6833/2611554095] up:rejoin 2024-03-24T00:26:37.675 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 126 sec latency 239266.465 ms 2024-03-24T00:26:38.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:38 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:rejoin,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:26:38.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:38 smithi078 ceph-mon[32429]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-24T00:26:38.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:38 smithi078 ceph-mon[32429]: pgmap v180: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:26:38.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:38 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:rejoin,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:26:38.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:38 smithi107 ceph-mon[36485]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-24T00:26:38.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:38 smithi107 ceph-mon[36485]: pgmap v180: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:26:38.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:rejoin,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:26:38.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-mon[36505]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-24T00:26:38.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-mon[36505]: pgmap v180: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:26:38.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa08bc90700 time 2024-03-24T00:26:38.019583+0000 2024-03-24T00:26:38.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:26:38.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:38.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa0962ba4a1] 2024-03-24T00:26:38.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa0962ba667] 2024-03-24T00:26:38.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556fbce72223] 2024-03-24T00:26:38.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556fbccec5d6] 2024-03-24T00:26:38.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556fbccecda1] 2024-03-24T00:26:38.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556fbccef2d5] 2024-03-24T00:26:38.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556fbcd3f233] 2024-03-24T00:26:38.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556fbcd41197] 2024-03-24T00:26:38.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556fbcc4011b] 2024-03-24T00:26:38.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556fbcc406bc] 2024-03-24T00:26:38.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556fbcc2a1ff] 2024-03-24T00:26:38.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa09654c488] 2024-03-24T00:26:38.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 13: (DispatchQueue::entry()+0x51f) [0x7fa09654960f] 2024-03-24T00:26:38.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa09661ce81] 2024-03-24T00:26:38.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa0950371ca] 2024-03-24T00:26:38.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 16: clone() 2024-03-24T00:26:38.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 2024-03-24T00:26:38.020+0000 7fa08bc90700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa08bc90700 time 2024-03-24T00:26:38.019583+0000 2024-03-24T00:26:38.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:26:38.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 2024-03-24T00:26:38.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:38.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa0962ba4a1] 2024-03-24T00:26:38.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa0962ba667] 2024-03-24T00:26:38.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556fbce72223] 2024-03-24T00:26:38.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556fbccec5d6] 2024-03-24T00:26:38.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556fbccecda1] 2024-03-24T00:26:38.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556fbccef2d5] 2024-03-24T00:26:38.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556fbcd3f233] 2024-03-24T00:26:38.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556fbcd41197] 2024-03-24T00:26:38.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556fbcc4011b] 2024-03-24T00:26:38.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556fbcc406bc] 2024-03-24T00:26:38.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556fbcc2a1ff] 2024-03-24T00:26:38.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa09654c488] 2024-03-24T00:26:38.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 13: (DispatchQueue::entry()+0x51f) [0x7fa09654960f] 2024-03-24T00:26:38.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa09661ce81] 2024-03-24T00:26:38.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa0950371ca] 2024-03-24T00:26:38.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 16: clone() 2024-03-24T00:26:38.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 2024-03-24T00:26:38.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: *** Caught signal (Aborted) ** 2024-03-24T00:26:38.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: in thread 7fa08bc90700 thread_name:ms_dispatch 2024-03-24T00:26:38.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:38.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa095041d20] 2024-03-24T00:26:38.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 2: gsignal() 2024-03-24T00:26:38.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 3: abort() 2024-03-24T00:26:38.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa0962ba4fb] 2024-03-24T00:26:38.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa0962ba667] 2024-03-24T00:26:38.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556fbce72223] 2024-03-24T00:26:38.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556fbccec5d6] 2024-03-24T00:26:38.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556fbccecda1] 2024-03-24T00:26:38.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556fbccef2d5] 2024-03-24T00:26:38.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556fbcd3f233] 2024-03-24T00:26:38.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556fbcd41197] 2024-03-24T00:26:38.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556fbcc4011b] 2024-03-24T00:26:38.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556fbcc406bc] 2024-03-24T00:26:38.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556fbcc2a1ff] 2024-03-24T00:26:38.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa09654c488] 2024-03-24T00:26:38.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 16: (DispatchQueue::entry()+0x51f) [0x7fa09654960f] 2024-03-24T00:26:38.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa09661ce81] 2024-03-24T00:26:38.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa0950371ca] 2024-03-24T00:26:38.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 19: clone() 2024-03-24T00:26:38.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 2024-03-24T00:26:38.022+0000 7fa08bc90700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:26:38.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: in thread 7fa08bc90700 thread_name:ms_dispatch 2024-03-24T00:26:38.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 2024-03-24T00:26:38.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:38.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa095041d20] 2024-03-24T00:26:38.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 2: gsignal() 2024-03-24T00:26:38.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 3: abort() 2024-03-24T00:26:38.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa0962ba4fb] 2024-03-24T00:26:38.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa0962ba667] 2024-03-24T00:26:38.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556fbce72223] 2024-03-24T00:26:38.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556fbccec5d6] 2024-03-24T00:26:38.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556fbccecda1] 2024-03-24T00:26:38.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556fbccef2d5] 2024-03-24T00:26:38.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556fbcd3f233] 2024-03-24T00:26:38.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556fbcd41197] 2024-03-24T00:26:38.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556fbcc4011b] 2024-03-24T00:26:38.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556fbcc406bc] 2024-03-24T00:26:38.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556fbcc2a1ff] 2024-03-24T00:26:38.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa09654c488] 2024-03-24T00:26:38.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 16: (DispatchQueue::entry()+0x51f) [0x7fa09654960f] 2024-03-24T00:26:38.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa09661ce81] 2024-03-24T00:26:38.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa0950371ca] 2024-03-24T00:26:38.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 19: clone() 2024-03-24T00:26:38.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:26:38.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 2024-03-24T00:26:38.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: -1> 2024-03-24T00:26:38.020+0000 7fa08bc90700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa08bc90700 time 2024-03-24T00:26:38.019583+0000 2024-03-24T00:26:38.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:26:38.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 2024-03-24T00:26:38.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:38.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa0962ba4a1] 2024-03-24T00:26:38.412 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa0962ba667] 2024-03-24T00:26:38.412 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556fbce72223] 2024-03-24T00:26:38.412 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556fbccec5d6] 2024-03-24T00:26:38.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556fbccecda1] 2024-03-24T00:26:38.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556fbccef2d5] 2024-03-24T00:26:38.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556fbcd3f233] 2024-03-24T00:26:38.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556fbcd41197] 2024-03-24T00:26:38.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556fbcc4011b] 2024-03-24T00:26:38.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556fbcc406bc] 2024-03-24T00:26:38.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556fbcc2a1ff] 2024-03-24T00:26:38.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa09654c488] 2024-03-24T00:26:38.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 13: (DispatchQueue::entry()+0x51f) [0x7fa09654960f] 2024-03-24T00:26:38.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa09661ce81] 2024-03-24T00:26:38.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa0950371ca] 2024-03-24T00:26:38.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 16: clone() 2024-03-24T00:26:38.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 2024-03-24T00:26:38.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 0> 2024-03-24T00:26:38.022+0000 7fa08bc90700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:26:38.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: in thread 7fa08bc90700 thread_name:ms_dispatch 2024-03-24T00:26:38.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 2024-03-24T00:26:38.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:38.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa095041d20] 2024-03-24T00:26:38.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 2: gsignal() 2024-03-24T00:26:38.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 3: abort() 2024-03-24T00:26:38.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa0962ba4fb] 2024-03-24T00:26:38.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa0962ba667] 2024-03-24T00:26:38.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556fbce72223] 2024-03-24T00:26:38.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556fbccec5d6] 2024-03-24T00:26:38.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556fbccecda1] 2024-03-24T00:26:38.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556fbccef2d5] 2024-03-24T00:26:38.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556fbcd3f233] 2024-03-24T00:26:38.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556fbcd41197] 2024-03-24T00:26:38.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556fbcc4011b] 2024-03-24T00:26:38.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556fbcc406bc] 2024-03-24T00:26:38.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556fbcc2a1ff] 2024-03-24T00:26:38.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa09654c488] 2024-03-24T00:26:38.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 16: (DispatchQueue::entry()+0x51f) [0x7fa09654960f] 2024-03-24T00:26:38.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa09661ce81] 2024-03-24T00:26:38.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa0950371ca] 2024-03-24T00:26:38.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 19: clone() 2024-03-24T00:26:38.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:26:38.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 2024-03-24T00:26:38.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: -1> 2024-03-24T00:26:38.020+0000 7fa08bc90700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa08bc90700 time 2024-03-24T00:26:38.019583+0000 2024-03-24T00:26:38.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:26:38.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 2024-03-24T00:26:38.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:38.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa0962ba4a1] 2024-03-24T00:26:38.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa0962ba667] 2024-03-24T00:26:38.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556fbce72223] 2024-03-24T00:26:38.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556fbccec5d6] 2024-03-24T00:26:38.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556fbccecda1] 2024-03-24T00:26:38.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556fbccef2d5] 2024-03-24T00:26:38.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556fbcd3f233] 2024-03-24T00:26:38.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556fbcd41197] 2024-03-24T00:26:38.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556fbcc4011b] 2024-03-24T00:26:38.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556fbcc406bc] 2024-03-24T00:26:38.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556fbcc2a1ff] 2024-03-24T00:26:38.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa09654c488] 2024-03-24T00:26:38.417 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 13: (DispatchQueue::entry()+0x51f) [0x7fa09654960f] 2024-03-24T00:26:38.417 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa09661ce81] 2024-03-24T00:26:38.417 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa0950371ca] 2024-03-24T00:26:38.417 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 16: clone() 2024-03-24T00:26:38.417 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 2024-03-24T00:26:38.417 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 0> 2024-03-24T00:26:38.022+0000 7fa08bc90700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:26:38.417 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: in thread 7fa08bc90700 thread_name:ms_dispatch 2024-03-24T00:26:38.417 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 2024-03-24T00:26:38.417 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:38.417 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa095041d20] 2024-03-24T00:26:38.417 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 2: gsignal() 2024-03-24T00:26:38.417 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 3: abort() 2024-03-24T00:26:38.417 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa0962ba4fb] 2024-03-24T00:26:38.417 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa0962ba667] 2024-03-24T00:26:38.417 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556fbce72223] 2024-03-24T00:26:38.418 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556fbccec5d6] 2024-03-24T00:26:38.418 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556fbccecda1] 2024-03-24T00:26:38.418 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556fbccef2d5] 2024-03-24T00:26:38.418 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556fbcd3f233] 2024-03-24T00:26:38.418 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556fbcd41197] 2024-03-24T00:26:38.418 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556fbcc4011b] 2024-03-24T00:26:38.418 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556fbcc406bc] 2024-03-24T00:26:38.418 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556fbcc2a1ff] 2024-03-24T00:26:38.418 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa09654c488] 2024-03-24T00:26:38.418 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 16: (DispatchQueue::entry()+0x51f) [0x7fa09654960f] 2024-03-24T00:26:38.418 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa09661ce81] 2024-03-24T00:26:38.418 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa0950371ca] 2024-03-24T00:26:38.418 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 19: clone() 2024-03-24T00:26:38.419 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:26:38.419 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70598]: 2024-03-24T00:26:38.419 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 podman[70767]: 2024-03-24 00:26:38.075672402 +0000 UTC m=+0.025250559 container died 8cb69d332574a1f9c6e077e8cc6b05ee14708764207e3adacac9a21f26cbaaa2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-24T00:26:38.675 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 127 sec latency 240266.623 ms 2024-03-24T00:26:39.002 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 podman[70767]: 2024-03-24 00:26:38.72481675 +0000 UTC m=+0.674394881 container remove 8cb69d332574a1f9c6e077e8cc6b05ee14708764207e3adacac9a21f26cbaaa2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-24T00:26:39.003 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:38 smithi106 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:26:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:39 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:26:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:39 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:26:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:39 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6832/2611554095,v1:172.21.15.106:6833/2611554095] up:active 2024-03-24T00:26:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:39 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:26:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:39 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=h=up:active} 3 up:standby-replay 4 up:standby 2024-03-24T00:26:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:39 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:26:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:39 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:26:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:39 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6832/2611554095,v1:172.21.15.106:6833/2611554095] up:active 2024-03-24T00:26:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:39 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:26:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:39 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=h=up:active} 3 up:standby-replay 4 up:standby 2024-03-24T00:26:39.402 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:39 smithi106 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.k.service: Failed with result 'exit-code'. 2024-03-24T00:26:39.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:39 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:26:39.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:39 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:26:39.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:39 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6832/2611554095,v1:172.21.15.106:6833/2611554095] up:active 2024-03-24T00:26:39.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:39 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:26:39.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:39 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=h=up:active} 3 up:standby-replay 4 up:standby 2024-03-24T00:26:39.675 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 128 sec latency 241266.732 ms 2024-03-24T00:26:40.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:40 smithi078 ceph-mon[32429]: pgmap v181: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:26:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:40 smithi107 ceph-mon[36485]: pgmap v181: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:26:40.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:40 smithi106 ceph-mon[36505]: pgmap v181: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:26:40.675 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 129 sec latency 242266.834 ms 2024-03-24T00:26:41.676 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 130 sec latency 243266.893 ms 2024-03-24T00:26:42.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:42 smithi106 ceph-mon[36505]: pgmap v182: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:26:42.676 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 131 sec latency 244266.993 ms 2024-03-24T00:26:42.684 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:42 smithi078 ceph-mon[32429]: pgmap v182: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:26:42.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:42 smithi107 ceph-mon[36485]: pgmap v182: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:26:43.676 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 132 sec latency 245267.111 ms 2024-03-24T00:26:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:44 smithi106 ceph-mon[36505]: pgmap v183: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:26:44.676 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 133 sec latency 246267.209 ms 2024-03-24T00:26:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:44 smithi078 ceph-mon[32429]: pgmap v183: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:26:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:44 smithi107 ceph-mon[36485]: pgmap v183: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:26:45.676 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 134 sec latency 247267.257 ms 2024-03-24T00:26:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:46 smithi106 ceph-mon[36505]: pgmap v184: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 180 B/s wr, 12 op/s 2024-03-24T00:26:46.676 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 135 sec latency 248267.344 ms 2024-03-24T00:26:46.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:46 smithi078 ceph-mon[32429]: pgmap v184: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 180 B/s wr, 12 op/s 2024-03-24T00:26:46.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:46 smithi107 ceph-mon[36485]: pgmap v184: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 180 B/s wr, 12 op/s 2024-03-24T00:26:47.676 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 136 sec latency 249267.464 ms 2024-03-24T00:26:48.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:48 smithi106 ceph-mon[36505]: pgmap v185: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T00:26:48.676 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 137 sec latency 250267.558 ms 2024-03-24T00:26:48.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:48 smithi078 ceph-mon[32429]: pgmap v185: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T00:26:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:48 smithi107 ceph-mon[36485]: pgmap v185: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T00:26:49.509 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:49 smithi106 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.k.service: Scheduled restart job, restart counter is at 2. 2024-03-24T00:26:49.509 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:49 smithi106 systemd[1]: Stopped Ceph mds.k for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:26:49.509 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:49 smithi106 systemd[1]: Starting Ceph mds.k for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:26:49.509 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:49 smithi106 podman[70865]: 2024-03-24 00:26:49.281973968 +0000 UTC m=+0.018004530 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:26:49.509 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:49 smithi106 podman[70865]: 2024-03-24 00:26:49.509636774 +0000 UTC m=+0.245667315 container create 953b3f1b70f094eb1338b43ee4aa7e08b853675e9eee8fa3481b1e1508997103 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD) 2024-03-24T00:26:49.678 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 138 sec latency 251267.676 ms 2024-03-24T00:26:49.779 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:49 smithi106 podman[70865]: 2024-03-24 00:26:49.745493357 +0000 UTC m=+0.481523907 container init 953b3f1b70f094eb1338b43ee4aa7e08b853675e9eee8fa3481b1e1508997103 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-24T00:26:49.779 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:49 smithi106 podman[70865]: 2024-03-24 00:26:49.748717374 +0000 UTC m=+0.484747921 container start 953b3f1b70f094eb1338b43ee4aa7e08b853675e9eee8fa3481b1e1508997103 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-24T00:26:50.126 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:50 smithi106 ceph-mon[36505]: pgmap v186: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T00:26:50.126 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:50 smithi106 ceph-mon[36505]: daemon mds.k restarted 2024-03-24T00:26:50.126 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:50 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:26:50.126 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:50 smithi106 ceph-mon[36505]: osdmap e108: 12 total, 12 up, 12 in 2024-03-24T00:26:50.126 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:50 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6832/2299054161,v1:172.21.15.106:6833/2299054161] up:boot 2024-03-24T00:26:50.126 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:50 smithi106 ceph-mon[36505]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-24T00:26:50.126 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:50 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-24T00:26:50.126 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:50 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T00:26:50.126 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:50 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:replay,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:26:50.127 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:49 smithi106 ceph-mds[70883]: -- 172.21.15.106:0/2240664321 <== mon.2 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558f3e905a40 con 0x558f3e8d9000 2024-03-24T00:26:50.127 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:49 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: starting mds.k at 2024-03-24T00:26:50.127 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:49 smithi106 bash[70865]: 953b3f1b70f094eb1338b43ee4aa7e08b853675e9eee8fa3481b1e1508997103 2024-03-24T00:26:50.127 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:26:49 smithi106 systemd[1]: Started Ceph mds.k for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:26:50.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:49 smithi078 ceph-mon[32429]: pgmap v186: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T00:26:50.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:49 smithi078 ceph-mon[32429]: daemon mds.k restarted 2024-03-24T00:26:50.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:49 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:26:50.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:49 smithi078 ceph-mon[32429]: osdmap e108: 12 total, 12 up, 12 in 2024-03-24T00:26:50.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:49 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6832/2299054161,v1:172.21.15.106:6833/2299054161] up:boot 2024-03-24T00:26:50.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:49 smithi078 ceph-mon[32429]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-24T00:26:50.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:49 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-24T00:26:50.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:49 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T00:26:50.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:49 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:replay,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:26:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:49 smithi107 ceph-mon[36485]: pgmap v186: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T00:26:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:49 smithi107 ceph-mon[36485]: daemon mds.k restarted 2024-03-24T00:26:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:49 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:26:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:49 smithi107 ceph-mon[36485]: osdmap e108: 12 total, 12 up, 12 in 2024-03-24T00:26:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:49 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6832/2299054161,v1:172.21.15.106:6833/2299054161] up:boot 2024-03-24T00:26:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:49 smithi107 ceph-mon[36485]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-24T00:26:50.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:49 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-24T00:26:50.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:49 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T00:26:50.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:49 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:replay,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:26:50.677 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 139 sec latency 252267.796 ms 2024-03-24T00:26:51.677 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 140 sec latency 253267.889 ms 2024-03-24T00:26:52.303 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:52 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6832/3086318443,v1:172.21.15.107:6833/3086318443] up:resolve 2024-03-24T00:26:52.303 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:52 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:resolve,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:26:52.303 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:52 smithi106 ceph-mon[36505]: pgmap v188: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:26:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:52 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6832/3086318443,v1:172.21.15.107:6833/3086318443] up:resolve 2024-03-24T00:26:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:52 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:resolve,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:26:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:52 smithi078 ceph-mon[32429]: pgmap v188: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:26:52.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:52 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6832/3086318443,v1:172.21.15.107:6833/3086318443] up:resolve 2024-03-24T00:26:52.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:52 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:resolve,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:26:52.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:52 smithi107 ceph-mon[36485]: pgmap v188: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:26:52.677 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 141 sec latency 254267.974 ms 2024-03-24T00:26:53.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:53 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999984 2024-03-24T00:26:53.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:53 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6832/3086318443,v1:172.21.15.107:6833/3086318443] up:reconnect 2024-03-24T00:26:53.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:53 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:reconnect,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:26:53.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:53 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999984 2024-03-24T00:26:53.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:53 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6832/3086318443,v1:172.21.15.107:6833/3086318443] up:reconnect 2024-03-24T00:26:53.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:53 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:reconnect,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:26:53.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:53 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999984 2024-03-24T00:26:53.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:53 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6832/3086318443,v1:172.21.15.107:6833/3086318443] up:reconnect 2024-03-24T00:26:53.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:53 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:reconnect,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:26:53.677 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 142 sec latency 255268.073 ms 2024-03-24T00:26:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:54 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6832/3086318443,v1:172.21.15.107:6833/3086318443] up:rejoin 2024-03-24T00:26:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:54 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:rejoin,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:26:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:54 smithi078 ceph-mon[32429]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-24T00:26:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:54 smithi078 ceph-mon[32429]: pgmap v189: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:26:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:54 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:26:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:54 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:26:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:54 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6832/3086318443,v1:172.21.15.107:6833/3086318443] up:active 2024-03-24T00:26:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:54 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=h=up:active} 3 up:standby-replay 4 up:standby 2024-03-24T00:26:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6832/3086318443,v1:172.21.15.107:6833/3086318443] up:rejoin 2024-03-24T00:26:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:rejoin,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:26:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-mon[36485]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-24T00:26:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-mon[36485]: pgmap v189: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:26:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:26:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:26:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6832/3086318443,v1:172.21.15.107:6833/3086318443] up:active 2024-03-24T00:26:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=h=up:active} 3 up:standby-replay 4 up:standby 2024-03-24T00:26:54.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f91ece0d700 time 2024-03-24T00:26:54.007152+0000 2024-03-24T00:26:54.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:26:54.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:54.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f91f74374a1] 2024-03-24T00:26:54.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f91f7437667] 2024-03-24T00:26:54.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c63c2f1223] 2024-03-24T00:26:54.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c63c16b5d6] 2024-03-24T00:26:54.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c63c16bda1] 2024-03-24T00:26:54.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c63c16e2d5] 2024-03-24T00:26:54.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c63c1be233] 2024-03-24T00:26:54.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c63c1c0197] 2024-03-24T00:26:54.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c63c0bf11b] 2024-03-24T00:26:54.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c63c0bf6bc] 2024-03-24T00:26:54.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c63c0a91ff] 2024-03-24T00:26:54.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f91f76c9488] 2024-03-24T00:26:54.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 13: (DispatchQueue::entry()+0x51f) [0x7f91f76c660f] 2024-03-24T00:26:54.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f91f7799e81] 2024-03-24T00:26:54.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f91f61b41ca] 2024-03-24T00:26:54.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 16: clone() 2024-03-24T00:26:54.346 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 2024-03-24T00:26:54.007+0000 7f91ece0d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f91ece0d700 time 2024-03-24T00:26:54.007152+0000 2024-03-24T00:26:54.346 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:26:54.346 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 2024-03-24T00:26:54.346 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:54.346 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f91f74374a1] 2024-03-24T00:26:54.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f91f7437667] 2024-03-24T00:26:54.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c63c2f1223] 2024-03-24T00:26:54.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c63c16b5d6] 2024-03-24T00:26:54.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c63c16bda1] 2024-03-24T00:26:54.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c63c16e2d5] 2024-03-24T00:26:54.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c63c1be233] 2024-03-24T00:26:54.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c63c1c0197] 2024-03-24T00:26:54.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c63c0bf11b] 2024-03-24T00:26:54.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c63c0bf6bc] 2024-03-24T00:26:54.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c63c0a91ff] 2024-03-24T00:26:54.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f91f76c9488] 2024-03-24T00:26:54.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 13: (DispatchQueue::entry()+0x51f) [0x7f91f76c660f] 2024-03-24T00:26:54.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f91f7799e81] 2024-03-24T00:26:54.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f91f61b41ca] 2024-03-24T00:26:54.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 16: clone() 2024-03-24T00:26:54.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 2024-03-24T00:26:54.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: *** Caught signal (Aborted) ** 2024-03-24T00:26:54.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: in thread 7f91ece0d700 thread_name:ms_dispatch 2024-03-24T00:26:54.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:54.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f91f61bed20] 2024-03-24T00:26:54.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 2: gsignal() 2024-03-24T00:26:54.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 3: abort() 2024-03-24T00:26:54.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f91f74374fb] 2024-03-24T00:26:54.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f91f7437667] 2024-03-24T00:26:54.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c63c2f1223] 2024-03-24T00:26:54.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c63c16b5d6] 2024-03-24T00:26:54.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c63c16bda1] 2024-03-24T00:26:54.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c63c16e2d5] 2024-03-24T00:26:54.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c63c1be233] 2024-03-24T00:26:54.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c63c1c0197] 2024-03-24T00:26:54.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c63c0bf11b] 2024-03-24T00:26:54.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c63c0bf6bc] 2024-03-24T00:26:54.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c63c0a91ff] 2024-03-24T00:26:54.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f91f76c9488] 2024-03-24T00:26:54.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 16: (DispatchQueue::entry()+0x51f) [0x7f91f76c660f] 2024-03-24T00:26:54.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f91f7799e81] 2024-03-24T00:26:54.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f91f61b41ca] 2024-03-24T00:26:54.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 19: clone() 2024-03-24T00:26:54.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 2024-03-24T00:26:54.009+0000 7f91ece0d700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:26:54.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: in thread 7f91ece0d700 thread_name:ms_dispatch 2024-03-24T00:26:54.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 2024-03-24T00:26:54.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:54.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f91f61bed20] 2024-03-24T00:26:54.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 2: gsignal() 2024-03-24T00:26:54.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 3: abort() 2024-03-24T00:26:54.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f91f74374fb] 2024-03-24T00:26:54.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f91f7437667] 2024-03-24T00:26:54.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c63c2f1223] 2024-03-24T00:26:54.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c63c16b5d6] 2024-03-24T00:26:54.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c63c16bda1] 2024-03-24T00:26:54.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c63c16e2d5] 2024-03-24T00:26:54.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c63c1be233] 2024-03-24T00:26:54.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c63c1c0197] 2024-03-24T00:26:54.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c63c0bf11b] 2024-03-24T00:26:54.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c63c0bf6bc] 2024-03-24T00:26:54.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c63c0a91ff] 2024-03-24T00:26:54.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f91f76c9488] 2024-03-24T00:26:54.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 16: (DispatchQueue::entry()+0x51f) [0x7f91f76c660f] 2024-03-24T00:26:54.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f91f7799e81] 2024-03-24T00:26:54.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f91f61b41ca] 2024-03-24T00:26:54.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 19: clone() 2024-03-24T00:26:54.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:26:54.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 2024-03-24T00:26:54.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: -1> 2024-03-24T00:26:54.007+0000 7f91ece0d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f91ece0d700 time 2024-03-24T00:26:54.007152+0000 2024-03-24T00:26:54.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:26:54.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 2024-03-24T00:26:54.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:54.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f91f74374a1] 2024-03-24T00:26:54.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f91f7437667] 2024-03-24T00:26:54.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c63c2f1223] 2024-03-24T00:26:54.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c63c16b5d6] 2024-03-24T00:26:54.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c63c16bda1] 2024-03-24T00:26:54.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c63c16e2d5] 2024-03-24T00:26:54.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c63c1be233] 2024-03-24T00:26:54.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c63c1c0197] 2024-03-24T00:26:54.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c63c0bf11b] 2024-03-24T00:26:54.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c63c0bf6bc] 2024-03-24T00:26:54.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c63c0a91ff] 2024-03-24T00:26:54.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f91f76c9488] 2024-03-24T00:26:54.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 13: (DispatchQueue::entry()+0x51f) [0x7f91f76c660f] 2024-03-24T00:26:54.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f91f7799e81] 2024-03-24T00:26:54.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f91f61b41ca] 2024-03-24T00:26:54.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 16: clone() 2024-03-24T00:26:54.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 2024-03-24T00:26:54.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 0> 2024-03-24T00:26:54.009+0000 7f91ece0d700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:26:54.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: in thread 7f91ece0d700 thread_name:ms_dispatch 2024-03-24T00:26:54.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 2024-03-24T00:26:54.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:54.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f91f61bed20] 2024-03-24T00:26:54.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 2: gsignal() 2024-03-24T00:26:54.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 3: abort() 2024-03-24T00:26:54.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f91f74374fb] 2024-03-24T00:26:54.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f91f7437667] 2024-03-24T00:26:54.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c63c2f1223] 2024-03-24T00:26:54.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c63c16b5d6] 2024-03-24T00:26:54.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c63c16bda1] 2024-03-24T00:26:54.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c63c16e2d5] 2024-03-24T00:26:54.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c63c1be233] 2024-03-24T00:26:54.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c63c1c0197] 2024-03-24T00:26:54.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c63c0bf11b] 2024-03-24T00:26:54.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c63c0bf6bc] 2024-03-24T00:26:54.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c63c0a91ff] 2024-03-24T00:26:54.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f91f76c9488] 2024-03-24T00:26:54.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 16: (DispatchQueue::entry()+0x51f) [0x7f91f76c660f] 2024-03-24T00:26:54.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f91f7799e81] 2024-03-24T00:26:54.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f91f61b41ca] 2024-03-24T00:26:54.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 19: clone() 2024-03-24T00:26:54.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:26:54.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 2024-03-24T00:26:54.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: -1> 2024-03-24T00:26:54.007+0000 7f91ece0d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f91ece0d700 time 2024-03-24T00:26:54.007152+0000 2024-03-24T00:26:54.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:26:54.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 2024-03-24T00:26:54.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:54.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f91f74374a1] 2024-03-24T00:26:54.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f91f7437667] 2024-03-24T00:26:54.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c63c2f1223] 2024-03-24T00:26:54.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c63c16b5d6] 2024-03-24T00:26:54.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c63c16bda1] 2024-03-24T00:26:54.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c63c16e2d5] 2024-03-24T00:26:54.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c63c1be233] 2024-03-24T00:26:54.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c63c1c0197] 2024-03-24T00:26:54.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c63c0bf11b] 2024-03-24T00:26:54.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c63c0bf6bc] 2024-03-24T00:26:54.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c63c0a91ff] 2024-03-24T00:26:54.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f91f76c9488] 2024-03-24T00:26:54.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 13: (DispatchQueue::entry()+0x51f) [0x7f91f76c660f] 2024-03-24T00:26:54.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f91f7799e81] 2024-03-24T00:26:54.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f91f61b41ca] 2024-03-24T00:26:54.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 16: clone() 2024-03-24T00:26:54.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 2024-03-24T00:26:54.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 0> 2024-03-24T00:26:54.009+0000 7f91ece0d700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:26:54.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: in thread 7f91ece0d700 thread_name:ms_dispatch 2024-03-24T00:26:54.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 2024-03-24T00:26:54.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:26:54.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f91f61bed20] 2024-03-24T00:26:54.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 2: gsignal() 2024-03-24T00:26:54.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 3: abort() 2024-03-24T00:26:54.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f91f74374fb] 2024-03-24T00:26:54.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f91f7437667] 2024-03-24T00:26:54.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c63c2f1223] 2024-03-24T00:26:54.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c63c16b5d6] 2024-03-24T00:26:54.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c63c16bda1] 2024-03-24T00:26:54.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c63c16e2d5] 2024-03-24T00:26:54.359 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c63c1be233] 2024-03-24T00:26:54.359 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c63c1c0197] 2024-03-24T00:26:54.359 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c63c0bf11b] 2024-03-24T00:26:54.359 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c63c0bf6bc] 2024-03-24T00:26:54.359 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c63c0a91ff] 2024-03-24T00:26:54.359 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f91f76c9488] 2024-03-24T00:26:54.359 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 16: (DispatchQueue::entry()+0x51f) [0x7f91f76c660f] 2024-03-24T00:26:54.359 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f91f7799e81] 2024-03-24T00:26:54.359 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f91f61b41ca] 2024-03-24T00:26:54.359 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 19: clone() 2024-03-24T00:26:54.359 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:26:54.360 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[66571]: 2024-03-24T00:26:54.360 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 podman[72741]: 2024-03-24 00:26:54.062733403 +0000 UTC m=+0.021504614 container died b4c419602016bf59b2129f162e4842086c872fbbb83a4b36ea7d5bc990682587 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-24T00:26:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:54 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6832/3086318443,v1:172.21.15.107:6833/3086318443] up:rejoin 2024-03-24T00:26:54.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:54 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:rejoin,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:26:54.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:54 smithi106 ceph-mon[36505]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-24T00:26:54.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:54 smithi106 ceph-mon[36505]: pgmap v189: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:26:54.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:54 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:26:54.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:54 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:26:54.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:54 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6832/3086318443,v1:172.21.15.107:6833/3086318443] up:active 2024-03-24T00:26:54.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:54 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=h=up:active} 3 up:standby-replay 4 up:standby 2024-03-24T00:26:54.677 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 143 sec latency 256268.207 ms 2024-03-24T00:26:55.343 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 podman[72741]: 2024-03-24 00:26:54.86186547 +0000 UTC m=+0.820636682 container remove b4c419602016bf59b2129f162e4842086c872fbbb83a4b36ea7d5bc990682587 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:26:55.343 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:54 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:26:55.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:26:55 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.l.service: Failed with result 'exit-code'. 2024-03-24T00:26:55.677 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 144 sec latency 257268.311 ms 2024-03-24T00:26:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:56 smithi106 ceph-mon[36505]: pgmap v190: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:26:56.677 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 145 sec latency 258268.438 ms 2024-03-24T00:26:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:56 smithi078 ceph-mon[32429]: pgmap v190: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:26:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:56 smithi107 ceph-mon[36485]: pgmap v190: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:26:57.677 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 146 sec latency 259268.520 ms 2024-03-24T00:26:58.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:26:58 smithi106 ceph-mon[36505]: pgmap v191: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:26:58.677 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 147 sec latency 260268.627 ms 2024-03-24T00:26:58.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:26:58 smithi078 ceph-mon[32429]: pgmap v191: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:26:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:26:58 smithi107 ceph-mon[36485]: pgmap v191: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:26:59.677 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 148 sec latency 261268.725 ms 2024-03-24T00:27:00.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:00 smithi106 ceph-mon[36505]: pgmap v192: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:27:00.677 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 149 sec latency 262268.830 ms 2024-03-24T00:27:00.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:00 smithi078 ceph-mon[32429]: pgmap v192: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:27:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:00 smithi107 ceph-mon[36485]: pgmap v192: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:27:01.678 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 150 sec latency 263268.920 ms 2024-03-24T00:27:02.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:02 smithi106 ceph-mon[36505]: pgmap v193: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 177 B/s wr, 12 op/s 2024-03-24T00:27:02.678 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 151 sec latency 264269.061 ms 2024-03-24T00:27:02.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:02 smithi078 ceph-mon[32429]: pgmap v193: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 177 B/s wr, 12 op/s 2024-03-24T00:27:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:02 smithi107 ceph-mon[36485]: pgmap v193: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 177 B/s wr, 12 op/s 2024-03-24T00:27:03.678 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 152 sec latency 265269.147 ms 2024-03-24T00:27:04.678 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 153 sec latency 266269.288 ms 2024-03-24T00:27:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:04 smithi078 ceph-mon[32429]: pgmap v194: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T00:27:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:04 smithi107 ceph-mon[36485]: pgmap v194: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T00:27:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:04 smithi106 ceph-mon[36505]: pgmap v194: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T00:27:05.611 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:05 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.l.service: Scheduled restart job, restart counter is at 1. 2024-03-24T00:27:05.611 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:05 smithi107 systemd[1]: Stopped Ceph mds.l for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:27:05.611 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:05 smithi107 systemd[1]: Starting Ceph mds.l for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:27:05.611 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:05 smithi107 podman[72838]: 2024-03-24 00:27:05.479018841 +0000 UTC m=+0.017647323 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:27:05.611 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:05 smithi107 podman[72838]: 2024-03-24 00:27:05.611589833 +0000 UTC m=+0.150218313 container create 4bf4122f4b784737ee0d1400942fcda02a38ebd4f987861e2d45de54f0599df4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:27:05.678 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 154 sec latency 267269.339 ms 2024-03-24T00:27:05.948 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:05 smithi107 podman[72838]: 2024-03-24 00:27:05.82309956 +0000 UTC m=+0.361728042 container init 4bf4122f4b784737ee0d1400942fcda02a38ebd4f987861e2d45de54f0599df4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-24T00:27:05.948 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:05 smithi107 podman[72838]: 2024-03-24 00:27:05.82590579 +0000 UTC m=+0.364534276 container start 4bf4122f4b784737ee0d1400942fcda02a38ebd4f987861e2d45de54f0599df4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5) 2024-03-24T00:27:05.948 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:05 smithi107 ceph-mds[72857]: -- 172.21.15.107:0/2316687461 <== mon.1 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c5a7229a40 con 0x55c5a71fd000 2024-03-24T00:27:05.948 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:05 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: starting mds.l at 2024-03-24T00:27:06.343 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:05 smithi107 bash[72838]: 4bf4122f4b784737ee0d1400942fcda02a38ebd4f987861e2d45de54f0599df4 2024-03-24T00:27:06.343 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:05 smithi107 systemd[1]: Started Ceph mds.l for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:27:06.678 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 155 sec latency 268269.454 ms 2024-03-24T00:27:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:06 smithi078 ceph-mon[32429]: pgmap v195: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-24T00:27:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:06 smithi078 ceph-mon[32429]: daemon mds.l restarted 2024-03-24T00:27:06.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:06 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:27:06.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:06 smithi078 ceph-mon[32429]: osdmap e109: 12 total, 12 up, 12 in 2024-03-24T00:27:06.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:06 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6832/1751539133,v1:172.21.15.107:6833/1751539133] up:boot 2024-03-24T00:27:06.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:06 smithi078 ceph-mon[32429]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-03-24T00:27:06.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:06 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-24T00:27:06.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:06 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:27:06.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:06 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:replay,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:06 smithi107 ceph-mon[36485]: pgmap v195: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-24T00:27:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:06 smithi107 ceph-mon[36485]: daemon mds.l restarted 2024-03-24T00:27:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:06 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:27:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:06 smithi107 ceph-mon[36485]: osdmap e109: 12 total, 12 up, 12 in 2024-03-24T00:27:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:06 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6832/1751539133,v1:172.21.15.107:6833/1751539133] up:boot 2024-03-24T00:27:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:06 smithi107 ceph-mon[36485]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-03-24T00:27:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:06 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-24T00:27:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:06 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:27:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:06 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:replay,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:06 smithi106 ceph-mon[36505]: pgmap v195: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-24T00:27:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:06 smithi106 ceph-mon[36505]: daemon mds.l restarted 2024-03-24T00:27:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:06 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:27:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:06 smithi106 ceph-mon[36505]: osdmap e109: 12 total, 12 up, 12 in 2024-03-24T00:27:06.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:06 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6832/1751539133,v1:172.21.15.107:6833/1751539133] up:boot 2024-03-24T00:27:06.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:06 smithi106 ceph-mon[36505]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-03-24T00:27:06.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:06 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-24T00:27:06.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:06 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:27:06.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:06 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:replay,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:07.678 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 156 sec latency 269269.577 ms 2024-03-24T00:27:08.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:07 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6832/2299054161,v1:172.21.15.106:6833/2299054161] up:resolve 2024-03-24T00:27:08.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:07 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:resolve,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:08.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:07 smithi106 ceph-mon[36505]: pgmap v197: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:27:08.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:07 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6832/2299054161,v1:172.21.15.106:6833/2299054161] up:resolve 2024-03-24T00:27:08.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:07 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:resolve,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:08.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:07 smithi078 ceph-mon[32429]: pgmap v197: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:27:08.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:07 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6832/2299054161,v1:172.21.15.106:6833/2299054161] up:resolve 2024-03-24T00:27:08.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:07 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:resolve,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:08.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:07 smithi107 ceph-mon[36485]: pgmap v197: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:27:08.678 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 157 sec latency 270269.686 ms 2024-03-24T00:27:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:08 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999978 2024-03-24T00:27:09.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:08 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6832/2299054161,v1:172.21.15.106:6833/2299054161] up:reconnect 2024-03-24T00:27:09.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:08 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:reconnect,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:09.153 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f34c2d1a700 time 2024-03-24T00:27:09.011408+0000 2024-03-24T00:27:09.153 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:27:09.153 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:27:09.153 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f34cd3444a1] 2024-03-24T00:27:09.153 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f34cd344667] 2024-03-24T00:27:09.154 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558f3d749223] 2024-03-24T00:27:09.154 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558f3d5c35d6] 2024-03-24T00:27:09.154 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558f3d5c3da1] 2024-03-24T00:27:09.154 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558f3d5c62d5] 2024-03-24T00:27:09.154 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558f3d616233] 2024-03-24T00:27:09.154 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558f3d618197] 2024-03-24T00:27:09.154 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558f3d51711b] 2024-03-24T00:27:09.154 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558f3d5176bc] 2024-03-24T00:27:09.154 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558f3d5011ff] 2024-03-24T00:27:09.154 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f34cd5d6488] 2024-03-24T00:27:09.154 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 13: (DispatchQueue::entry()+0x51f) [0x7f34cd5d360f] 2024-03-24T00:27:09.155 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f34cd6a6e81] 2024-03-24T00:27:09.155 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f34cc0c11ca] 2024-03-24T00:27:09.155 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 16: clone() 2024-03-24T00:27:09.155 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 2024-03-24T00:27:09.011+0000 7f34c2d1a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f34c2d1a700 time 2024-03-24T00:27:09.011408+0000 2024-03-24T00:27:09.155 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:27:09.155 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 2024-03-24T00:27:09.155 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:27:09.155 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f34cd3444a1] 2024-03-24T00:27:09.155 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f34cd344667] 2024-03-24T00:27:09.155 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558f3d749223] 2024-03-24T00:27:09.155 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558f3d5c35d6] 2024-03-24T00:27:09.155 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558f3d5c3da1] 2024-03-24T00:27:09.155 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558f3d5c62d5] 2024-03-24T00:27:09.155 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558f3d616233] 2024-03-24T00:27:09.156 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558f3d618197] 2024-03-24T00:27:09.156 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558f3d51711b] 2024-03-24T00:27:09.156 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558f3d5176bc] 2024-03-24T00:27:09.156 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558f3d5011ff] 2024-03-24T00:27:09.156 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f34cd5d6488] 2024-03-24T00:27:09.156 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 13: (DispatchQueue::entry()+0x51f) [0x7f34cd5d360f] 2024-03-24T00:27:09.156 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f34cd6a6e81] 2024-03-24T00:27:09.156 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f34cc0c11ca] 2024-03-24T00:27:09.156 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 16: clone() 2024-03-24T00:27:09.156 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 2024-03-24T00:27:09.156 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: *** Caught signal (Aborted) ** 2024-03-24T00:27:09.156 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: in thread 7f34c2d1a700 thread_name:ms_dispatch 2024-03-24T00:27:09.156 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:27:09.156 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f34cc0cbd20] 2024-03-24T00:27:09.157 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 2: gsignal() 2024-03-24T00:27:09.158 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 3: abort() 2024-03-24T00:27:09.158 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f34cd3444fb] 2024-03-24T00:27:09.158 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f34cd344667] 2024-03-24T00:27:09.158 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558f3d749223] 2024-03-24T00:27:09.158 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558f3d5c35d6] 2024-03-24T00:27:09.158 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558f3d5c3da1] 2024-03-24T00:27:09.158 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558f3d5c62d5] 2024-03-24T00:27:09.158 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558f3d616233] 2024-03-24T00:27:09.158 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558f3d618197] 2024-03-24T00:27:09.158 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558f3d51711b] 2024-03-24T00:27:09.158 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558f3d5176bc] 2024-03-24T00:27:09.158 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558f3d5011ff] 2024-03-24T00:27:09.158 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f34cd5d6488] 2024-03-24T00:27:09.159 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 16: (DispatchQueue::entry()+0x51f) [0x7f34cd5d360f] 2024-03-24T00:27:09.159 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f34cd6a6e81] 2024-03-24T00:27:09.159 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f34cc0c11ca] 2024-03-24T00:27:09.159 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 19: clone() 2024-03-24T00:27:09.159 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 2024-03-24T00:27:09.013+0000 7f34c2d1a700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:27:09.159 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: in thread 7f34c2d1a700 thread_name:ms_dispatch 2024-03-24T00:27:09.159 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 2024-03-24T00:27:09.159 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:27:09.159 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f34cc0cbd20] 2024-03-24T00:27:09.159 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 2: gsignal() 2024-03-24T00:27:09.159 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 3: abort() 2024-03-24T00:27:09.159 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f34cd3444fb] 2024-03-24T00:27:09.159 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f34cd344667] 2024-03-24T00:27:09.159 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558f3d749223] 2024-03-24T00:27:09.160 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558f3d5c35d6] 2024-03-24T00:27:09.160 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558f3d5c3da1] 2024-03-24T00:27:09.160 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558f3d5c62d5] 2024-03-24T00:27:09.160 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558f3d616233] 2024-03-24T00:27:09.160 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558f3d618197] 2024-03-24T00:27:09.160 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558f3d51711b] 2024-03-24T00:27:09.160 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558f3d5176bc] 2024-03-24T00:27:09.160 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558f3d5011ff] 2024-03-24T00:27:09.160 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f34cd5d6488] 2024-03-24T00:27:09.160 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 16: (DispatchQueue::entry()+0x51f) [0x7f34cd5d360f] 2024-03-24T00:27:09.160 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f34cd6a6e81] 2024-03-24T00:27:09.160 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f34cc0c11ca] 2024-03-24T00:27:09.160 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 19: clone() 2024-03-24T00:27:09.160 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:27:09.160 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 2024-03-24T00:27:09.161 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: -1> 2024-03-24T00:27:09.011+0000 7f34c2d1a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f34c2d1a700 time 2024-03-24T00:27:09.011408+0000 2024-03-24T00:27:09.161 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:27:09.161 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 2024-03-24T00:27:09.161 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:27:09.161 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f34cd3444a1] 2024-03-24T00:27:09.161 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f34cd344667] 2024-03-24T00:27:09.161 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558f3d749223] 2024-03-24T00:27:09.161 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558f3d5c35d6] 2024-03-24T00:27:09.161 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558f3d5c3da1] 2024-03-24T00:27:09.161 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558f3d5c62d5] 2024-03-24T00:27:09.161 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558f3d616233] 2024-03-24T00:27:09.161 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558f3d618197] 2024-03-24T00:27:09.161 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558f3d51711b] 2024-03-24T00:27:09.161 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558f3d5176bc] 2024-03-24T00:27:09.162 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558f3d5011ff] 2024-03-24T00:27:09.162 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f34cd5d6488] 2024-03-24T00:27:09.162 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 13: (DispatchQueue::entry()+0x51f) [0x7f34cd5d360f] 2024-03-24T00:27:09.162 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f34cd6a6e81] 2024-03-24T00:27:09.162 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f34cc0c11ca] 2024-03-24T00:27:09.162 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 16: clone() 2024-03-24T00:27:09.162 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 2024-03-24T00:27:09.162 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 0> 2024-03-24T00:27:09.013+0000 7f34c2d1a700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:27:09.162 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: in thread 7f34c2d1a700 thread_name:ms_dispatch 2024-03-24T00:27:09.162 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 2024-03-24T00:27:09.162 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:27:09.162 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f34cc0cbd20] 2024-03-24T00:27:09.162 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 2: gsignal() 2024-03-24T00:27:09.163 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 3: abort() 2024-03-24T00:27:09.163 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f34cd3444fb] 2024-03-24T00:27:09.163 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f34cd344667] 2024-03-24T00:27:09.163 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558f3d749223] 2024-03-24T00:27:09.163 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558f3d5c35d6] 2024-03-24T00:27:09.163 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558f3d5c3da1] 2024-03-24T00:27:09.163 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558f3d5c62d5] 2024-03-24T00:27:09.163 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558f3d616233] 2024-03-24T00:27:09.163 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558f3d618197] 2024-03-24T00:27:09.163 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558f3d51711b] 2024-03-24T00:27:09.163 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558f3d5176bc] 2024-03-24T00:27:09.163 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558f3d5011ff] 2024-03-24T00:27:09.163 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f34cd5d6488] 2024-03-24T00:27:09.164 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 16: (DispatchQueue::entry()+0x51f) [0x7f34cd5d360f] 2024-03-24T00:27:09.164 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f34cd6a6e81] 2024-03-24T00:27:09.164 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f34cc0c11ca] 2024-03-24T00:27:09.164 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 19: clone() 2024-03-24T00:27:09.164 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:27:09.164 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 2024-03-24T00:27:09.164 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: -1> 2024-03-24T00:27:09.011+0000 7f34c2d1a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f34c2d1a700 time 2024-03-24T00:27:09.011408+0000 2024-03-24T00:27:09.164 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:27:09.164 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 2024-03-24T00:27:09.164 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:27:09.164 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f34cd3444a1] 2024-03-24T00:27:09.164 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f34cd344667] 2024-03-24T00:27:09.164 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558f3d749223] 2024-03-24T00:27:09.164 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558f3d5c35d6] 2024-03-24T00:27:09.165 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558f3d5c3da1] 2024-03-24T00:27:09.165 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558f3d5c62d5] 2024-03-24T00:27:09.165 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558f3d616233] 2024-03-24T00:27:09.165 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558f3d618197] 2024-03-24T00:27:09.165 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558f3d51711b] 2024-03-24T00:27:09.165 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558f3d5176bc] 2024-03-24T00:27:09.165 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558f3d5011ff] 2024-03-24T00:27:09.165 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f34cd5d6488] 2024-03-24T00:27:09.165 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 13: (DispatchQueue::entry()+0x51f) [0x7f34cd5d360f] 2024-03-24T00:27:09.165 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f34cd6a6e81] 2024-03-24T00:27:09.165 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f34cc0c11ca] 2024-03-24T00:27:09.165 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 16: clone() 2024-03-24T00:27:09.165 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 2024-03-24T00:27:09.165 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 0> 2024-03-24T00:27:09.013+0000 7f34c2d1a700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:27:09.165 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: in thread 7f34c2d1a700 thread_name:ms_dispatch 2024-03-24T00:27:09.166 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 2024-03-24T00:27:09.166 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:27:09.166 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f34cc0cbd20] 2024-03-24T00:27:09.166 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 2: gsignal() 2024-03-24T00:27:09.166 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 3: abort() 2024-03-24T00:27:09.166 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f34cd3444fb] 2024-03-24T00:27:09.166 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f34cd344667] 2024-03-24T00:27:09.166 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558f3d749223] 2024-03-24T00:27:09.166 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558f3d5c35d6] 2024-03-24T00:27:09.166 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558f3d5c3da1] 2024-03-24T00:27:09.166 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558f3d5c62d5] 2024-03-24T00:27:09.166 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558f3d616233] 2024-03-24T00:27:09.166 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558f3d618197] 2024-03-24T00:27:09.166 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558f3d51711b] 2024-03-24T00:27:09.167 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558f3d5176bc] 2024-03-24T00:27:09.167 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558f3d5011ff] 2024-03-24T00:27:09.167 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f34cd5d6488] 2024-03-24T00:27:09.167 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 16: (DispatchQueue::entry()+0x51f) [0x7f34cd5d360f] 2024-03-24T00:27:09.167 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f34cd6a6e81] 2024-03-24T00:27:09.167 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f34cc0c11ca] 2024-03-24T00:27:09.167 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 19: clone() 2024-03-24T00:27:09.167 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:27:09.167 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[70879]: 2024-03-24T00:27:09.167 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 podman[70932]: 2024-03-24 00:27:09.075363527 +0000 UTC m=+0.022747962 container died 953b3f1b70f094eb1338b43ee4aa7e08b853675e9eee8fa3481b1e1508997103 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-24T00:27:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:08 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999978 2024-03-24T00:27:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:08 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6832/2299054161,v1:172.21.15.106:6833/2299054161] up:reconnect 2024-03-24T00:27:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:08 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:reconnect,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:08 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999978 2024-03-24T00:27:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:08 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6832/2299054161,v1:172.21.15.106:6833/2299054161] up:reconnect 2024-03-24T00:27:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:08 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:reconnect,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:09.678 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 158 sec latency 271269.781 ms 2024-03-24T00:27:09.918 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 podman[70932]: 2024-03-24 00:27:09.660978917 +0000 UTC m=+0.608363364 container remove 953b3f1b70f094eb1338b43ee4aa7e08b853675e9eee8fa3481b1e1508997103 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:27:09.919 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:27:09.919 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6832/2299054161,v1:172.21.15.106:6833/2299054161] up:rejoin 2024-03-24T00:27:09.919 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:rejoin,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:09.919 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-mon[36505]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-24T00:27:09.919 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:27:09.919 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:27:09.919 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6832/2299054161,v1:172.21.15.106:6833/2299054161] up:active 2024-03-24T00:27:09.919 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=h=up:active} 3 up:standby-replay 4 up:standby 2024-03-24T00:27:09.919 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:09 smithi106 ceph-mon[36505]: pgmap v198: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:27:10.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:09 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6832/2299054161,v1:172.21.15.106:6833/2299054161] up:rejoin 2024-03-24T00:27:10.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:09 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:rejoin,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:10.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:09 smithi078 ceph-mon[32429]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-24T00:27:10.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:09 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:27:10.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:09 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:27:10.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:09 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6832/2299054161,v1:172.21.15.106:6833/2299054161] up:active 2024-03-24T00:27:10.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:09 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=h=up:active} 3 up:standby-replay 4 up:standby 2024-03-24T00:27:10.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:09 smithi078 ceph-mon[32429]: pgmap v198: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:27:10.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:09 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6832/2299054161,v1:172.21.15.106:6833/2299054161] up:rejoin 2024-03-24T00:27:10.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:09 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:rejoin,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:10.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:09 smithi107 ceph-mon[36485]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-24T00:27:10.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:09 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:27:10.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:09 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:27:10.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:09 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6832/2299054161,v1:172.21.15.106:6833/2299054161] up:active 2024-03-24T00:27:10.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:09 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=h=up:active} 3 up:standby-replay 4 up:standby 2024-03-24T00:27:10.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:09 smithi107 ceph-mon[36485]: pgmap v198: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:27:10.402 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:09 smithi106 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.k.service: Failed with result 'exit-code'. 2024-03-24T00:27:10.679 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 159 sec latency 272269.835 ms 2024-03-24T00:27:11.679 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 160 sec latency 273269.933 ms 2024-03-24T00:27:12.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:12 smithi106 ceph-mon[36505]: pgmap v199: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-24T00:27:12.679 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 161 sec latency 274269.984 ms 2024-03-24T00:27:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:12 smithi078 ceph-mon[32429]: pgmap v199: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-24T00:27:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:12 smithi107 ceph-mon[36485]: pgmap v199: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-24T00:27:13.679 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 162 sec latency 275270.095 ms 2024-03-24T00:27:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:14 smithi106 ceph-mon[36505]: pgmap v200: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-24T00:27:14.679 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 163 sec latency 276270.206 ms 2024-03-24T00:27:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:14 smithi078 ceph-mon[32429]: pgmap v200: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-24T00:27:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:14 smithi107 ceph-mon[36485]: pgmap v200: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-24T00:27:15.679 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 164 sec latency 277270.307 ms 2024-03-24T00:27:16.679 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 165 sec latency 278270.429 ms 2024-03-24T00:27:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:16 smithi078 ceph-mon[32429]: pgmap v201: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:27:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:16 smithi107 ceph-mon[36485]: pgmap v201: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:27:16.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:16 smithi106 ceph-mon[36505]: pgmap v201: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:27:17.679 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 166 sec latency 279270.514 ms 2024-03-24T00:27:18.679 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 167 sec latency 280270.622 ms 2024-03-24T00:27:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:18 smithi078 ceph-mon[32429]: pgmap v202: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-24T00:27:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:18 smithi107 ceph-mon[36485]: pgmap v202: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-24T00:27:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:18 smithi106 ceph-mon[36505]: pgmap v202: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-24T00:27:19.680 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 168 sec latency 281270.714 ms 2024-03-24T00:27:19.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:19 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:27:19.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:19 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:27:19.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:19 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:27:19.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:19 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:27:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:19 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:27:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:19 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:27:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:19 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:27:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:19 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:27:19.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:19 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:27:19.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:19 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:27:19.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:19 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:27:19.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:19 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:27:20.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:20 smithi106 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.k.service: Scheduled restart job, restart counter is at 3. 2024-03-24T00:27:20.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:20 smithi106 systemd[1]: Stopped Ceph mds.k for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:27:20.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:20 smithi106 systemd[1]: Starting Ceph mds.k for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:27:20.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:20 smithi106 podman[71155]: 2024-03-24 00:27:20.288229102 +0000 UTC m=+0.018632121 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:27:20.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:20 smithi106 podman[71155]: 2024-03-24 00:27:20.403775893 +0000 UTC m=+0.134178898 container create 50659a837aba2803221dc98028fad5df7d51a0b4141bf20ec5625fd54ded0cab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-24T00:27:20.680 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 169 sec latency 282270.813 ms 2024-03-24T00:27:20.722 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:20 smithi106 ceph-mon[36505]: pgmap v203: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-24T00:27:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:20 smithi078 ceph-mon[32429]: pgmap v203: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-24T00:27:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:20 smithi107 ceph-mon[36485]: pgmap v203: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-24T00:27:21.087 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:20 smithi106 podman[71155]: 2024-03-24 00:27:20.723415689 +0000 UTC m=+0.453818709 container init 50659a837aba2803221dc98028fad5df7d51a0b4141bf20ec5625fd54ded0cab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD) 2024-03-24T00:27:21.087 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:20 smithi106 podman[71155]: 2024-03-24 00:27:20.727220572 +0000 UTC m=+0.457623569 container start 50659a837aba2803221dc98028fad5df7d51a0b4141bf20ec5625fd54ded0cab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:27:21.088 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:20 smithi106 ceph-mds[71174]: -- 172.21.15.106:0/3833751443 <== mon.2 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558faf30ba40 con 0x558faf2dec00 2024-03-24T00:27:21.088 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:20 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: starting mds.k at 2024-03-24T00:27:21.088 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:20 smithi106 bash[71155]: 50659a837aba2803221dc98028fad5df7d51a0b4141bf20ec5625fd54ded0cab 2024-03-24T00:27:21.088 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:20 smithi106 systemd[1]: Started Ceph mds.k for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:27:21.680 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 170 sec latency 283270.925 ms 2024-03-24T00:27:21.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:21 smithi078 ceph-mon[32429]: daemon mds.k restarted 2024-03-24T00:27:21.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:21 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:27:21.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:21 smithi078 ceph-mon[32429]: osdmap e110: 12 total, 12 up, 12 in 2024-03-24T00:27:21.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:21 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6832/459568830,v1:172.21.15.106:6833/459568830] up:boot 2024-03-24T00:27:21.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:21 smithi078 ceph-mon[32429]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-24T00:27:21.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:21 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-24T00:27:21.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:21 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T00:27:21.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:21 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:replay,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:21 smithi107 ceph-mon[36485]: daemon mds.k restarted 2024-03-24T00:27:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:21 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:27:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:21 smithi107 ceph-mon[36485]: osdmap e110: 12 total, 12 up, 12 in 2024-03-24T00:27:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:21 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6832/459568830,v1:172.21.15.106:6833/459568830] up:boot 2024-03-24T00:27:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:21 smithi107 ceph-mon[36485]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-24T00:27:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:21 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-24T00:27:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:21 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T00:27:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:21 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:replay,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:21.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:21 smithi106 ceph-mon[36505]: daemon mds.k restarted 2024-03-24T00:27:21.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:21 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:27:21.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:21 smithi106 ceph-mon[36505]: osdmap e110: 12 total, 12 up, 12 in 2024-03-24T00:27:21.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:21 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6832/459568830,v1:172.21.15.106:6833/459568830] up:boot 2024-03-24T00:27:21.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:21 smithi106 ceph-mon[36505]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-24T00:27:21.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:21 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-24T00:27:21.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:21 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T00:27:21.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:21 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:replay,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:22.680 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 171 sec latency 284271.002 ms 2024-03-24T00:27:22.744 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:22 smithi107 ceph-mon[36485]: pgmap v205: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:27:22.744 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:22 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6832/1751539133,v1:172.21.15.107:6833/1751539133] up:resolve 2024-03-24T00:27:22.744 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:22 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:resolve,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:22.777 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:22 smithi078 ceph-mon[32429]: pgmap v205: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:27:22.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:22 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6832/1751539133,v1:172.21.15.107:6833/1751539133] up:resolve 2024-03-24T00:27:22.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:22 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:resolve,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:22.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:22 smithi106 ceph-mon[36505]: pgmap v205: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:27:22.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:22 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6832/1751539133,v1:172.21.15.107:6833/1751539133] up:resolve 2024-03-24T00:27:22.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:22 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:resolve,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:23.680 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 172 sec latency 285271.097 ms 2024-03-24T00:27:24.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:23 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999984 2024-03-24T00:27:24.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:23 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6832/1751539133,v1:172.21.15.107:6833/1751539133] up:reconnect 2024-03-24T00:27:24.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:23 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:reconnect,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:24.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:23 smithi078 ceph-mon[32429]: pgmap v206: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:27:24.065 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:23 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999984 2024-03-24T00:27:24.065 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:23 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6832/1751539133,v1:172.21.15.107:6833/1751539133] up:reconnect 2024-03-24T00:27:24.065 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:23 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:reconnect,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:24.066 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:23 smithi107 ceph-mon[36485]: pgmap v206: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:27:24.066 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f6bcc856700 time 2024-03-24T00:27:24.015667+0000 2024-03-24T00:27:24.066 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:27:24.066 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:27:24.066 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6bd6e804a1] 2024-03-24T00:27:24.067 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f6bd6e80667] 2024-03-24T00:27:24.067 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c5a46ba223] 2024-03-24T00:27:24.067 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c5a45345d6] 2024-03-24T00:27:24.067 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c5a4534da1] 2024-03-24T00:27:24.067 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c5a45372d5] 2024-03-24T00:27:24.067 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c5a4587233] 2024-03-24T00:27:24.067 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c5a4589197] 2024-03-24T00:27:24.067 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c5a448811b] 2024-03-24T00:27:24.067 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c5a44886bc] 2024-03-24T00:27:24.067 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c5a44721ff] 2024-03-24T00:27:24.067 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6bd7112488] 2024-03-24T00:27:24.067 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 13: (DispatchQueue::entry()+0x51f) [0x7f6bd710f60f] 2024-03-24T00:27:24.067 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6bd71e2e81] 2024-03-24T00:27:24.068 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6bd5bfd1ca] 2024-03-24T00:27:24.068 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 16: clone() 2024-03-24T00:27:24.068 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: *** Caught signal (Aborted) ** 2024-03-24T00:27:24.068 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: in thread 7f6bcc856700 thread_name:ms_dispatch 2024-03-24T00:27:24.068 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 2024-03-24T00:27:24.016+0000 7f6bcc856700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f6bcc856700 time 2024-03-24T00:27:24.015667+0000 2024-03-24T00:27:24.068 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:27:24.068 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 2024-03-24T00:27:24.068 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:27:24.068 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6bd6e804a1] 2024-03-24T00:27:24.068 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f6bd6e80667] 2024-03-24T00:27:24.068 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c5a46ba223] 2024-03-24T00:27:24.068 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c5a45345d6] 2024-03-24T00:27:24.068 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c5a4534da1] 2024-03-24T00:27:24.068 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c5a45372d5] 2024-03-24T00:27:24.068 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c5a4587233] 2024-03-24T00:27:24.069 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c5a4589197] 2024-03-24T00:27:24.069 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c5a448811b] 2024-03-24T00:27:24.069 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c5a44886bc] 2024-03-24T00:27:24.069 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c5a44721ff] 2024-03-24T00:27:24.069 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6bd7112488] 2024-03-24T00:27:24.069 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 13: (DispatchQueue::entry()+0x51f) [0x7f6bd710f60f] 2024-03-24T00:27:24.069 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6bd71e2e81] 2024-03-24T00:27:24.069 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6bd5bfd1ca] 2024-03-24T00:27:24.069 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 16: clone() 2024-03-24T00:27:24.069 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 2024-03-24T00:27:24.069 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:27:24.069 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6bd5c07d20] 2024-03-24T00:27:24.069 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 2: gsignal() 2024-03-24T00:27:24.070 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 3: abort() 2024-03-24T00:27:24.070 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6bd6e804fb] 2024-03-24T00:27:24.070 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f6bd6e80667] 2024-03-24T00:27:24.070 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c5a46ba223] 2024-03-24T00:27:24.070 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c5a45345d6] 2024-03-24T00:27:24.070 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c5a4534da1] 2024-03-24T00:27:24.070 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c5a45372d5] 2024-03-24T00:27:24.070 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c5a4587233] 2024-03-24T00:27:24.070 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c5a4589197] 2024-03-24T00:27:24.070 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c5a448811b] 2024-03-24T00:27:24.070 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c5a44886bc] 2024-03-24T00:27:24.070 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c5a44721ff] 2024-03-24T00:27:24.070 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6bd7112488] 2024-03-24T00:27:24.070 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 16: (DispatchQueue::entry()+0x51f) [0x7f6bd710f60f] 2024-03-24T00:27:24.071 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6bd71e2e81] 2024-03-24T00:27:24.071 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6bd5bfd1ca] 2024-03-24T00:27:24.071 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 19: clone() 2024-03-24T00:27:24.071 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 2024-03-24T00:27:24.018+0000 7f6bcc856700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:27:24.071 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: in thread 7f6bcc856700 thread_name:ms_dispatch 2024-03-24T00:27:24.071 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 2024-03-24T00:27:24.071 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:27:24.071 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6bd5c07d20] 2024-03-24T00:27:24.071 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 2: gsignal() 2024-03-24T00:27:24.071 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 3: abort() 2024-03-24T00:27:24.071 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6bd6e804fb] 2024-03-24T00:27:24.071 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f6bd6e80667] 2024-03-24T00:27:24.071 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c5a46ba223] 2024-03-24T00:27:24.071 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c5a45345d6] 2024-03-24T00:27:24.072 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c5a4534da1] 2024-03-24T00:27:24.072 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c5a45372d5] 2024-03-24T00:27:24.072 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c5a4587233] 2024-03-24T00:27:24.072 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c5a4589197] 2024-03-24T00:27:24.072 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c5a448811b] 2024-03-24T00:27:24.072 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c5a44886bc] 2024-03-24T00:27:24.072 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c5a44721ff] 2024-03-24T00:27:24.072 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6bd7112488] 2024-03-24T00:27:24.072 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 16: (DispatchQueue::entry()+0x51f) [0x7f6bd710f60f] 2024-03-24T00:27:24.072 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6bd71e2e81] 2024-03-24T00:27:24.072 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6bd5bfd1ca] 2024-03-24T00:27:24.072 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 19: clone() 2024-03-24T00:27:24.072 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:27:24.073 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 2024-03-24T00:27:24.073 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: -1> 2024-03-24T00:27:24.016+0000 7f6bcc856700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f6bcc856700 time 2024-03-24T00:27:24.015667+0000 2024-03-24T00:27:24.073 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:27:24.073 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 2024-03-24T00:27:24.073 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:27:24.073 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6bd6e804a1] 2024-03-24T00:27:24.073 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f6bd6e80667] 2024-03-24T00:27:24.074 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c5a46ba223] 2024-03-24T00:27:24.074 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c5a45345d6] 2024-03-24T00:27:24.074 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c5a4534da1] 2024-03-24T00:27:24.074 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c5a45372d5] 2024-03-24T00:27:24.074 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c5a4587233] 2024-03-24T00:27:24.074 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c5a4589197] 2024-03-24T00:27:24.074 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c5a448811b] 2024-03-24T00:27:24.074 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c5a44886bc] 2024-03-24T00:27:24.074 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c5a44721ff] 2024-03-24T00:27:24.074 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6bd7112488] 2024-03-24T00:27:24.075 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 13: (DispatchQueue::entry()+0x51f) [0x7f6bd710f60f] 2024-03-24T00:27:24.075 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6bd71e2e81] 2024-03-24T00:27:24.075 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6bd5bfd1ca] 2024-03-24T00:27:24.075 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 16: clone() 2024-03-24T00:27:24.075 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 2024-03-24T00:27:24.075 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 0> 2024-03-24T00:27:24.018+0000 7f6bcc856700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:27:24.075 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: in thread 7f6bcc856700 thread_name:ms_dispatch 2024-03-24T00:27:24.075 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 2024-03-24T00:27:24.075 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:27:24.075 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6bd5c07d20] 2024-03-24T00:27:24.075 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 2: gsignal() 2024-03-24T00:27:24.075 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 3: abort() 2024-03-24T00:27:24.075 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6bd6e804fb] 2024-03-24T00:27:24.075 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f6bd6e80667] 2024-03-24T00:27:24.076 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c5a46ba223] 2024-03-24T00:27:24.076 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c5a45345d6] 2024-03-24T00:27:24.076 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c5a4534da1] 2024-03-24T00:27:24.076 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c5a45372d5] 2024-03-24T00:27:24.076 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c5a4587233] 2024-03-24T00:27:24.076 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c5a4589197] 2024-03-24T00:27:24.076 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c5a448811b] 2024-03-24T00:27:24.076 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c5a44886bc] 2024-03-24T00:27:24.076 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c5a44721ff] 2024-03-24T00:27:24.076 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6bd7112488] 2024-03-24T00:27:24.076 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 16: (DispatchQueue::entry()+0x51f) [0x7f6bd710f60f] 2024-03-24T00:27:24.076 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6bd71e2e81] 2024-03-24T00:27:24.076 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6bd5bfd1ca] 2024-03-24T00:27:24.076 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 19: clone() 2024-03-24T00:27:24.076 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:27:24.076 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 2024-03-24T00:27:24.077 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: -1> 2024-03-24T00:27:24.016+0000 7f6bcc856700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f6bcc856700 time 2024-03-24T00:27:24.015667+0000 2024-03-24T00:27:24.077 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:27:24.077 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 2024-03-24T00:27:24.077 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:27:24.077 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6bd6e804a1] 2024-03-24T00:27:24.077 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f6bd6e80667] 2024-03-24T00:27:24.077 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c5a46ba223] 2024-03-24T00:27:24.077 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c5a45345d6] 2024-03-24T00:27:24.077 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c5a4534da1] 2024-03-24T00:27:24.077 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c5a45372d5] 2024-03-24T00:27:24.077 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c5a4587233] 2024-03-24T00:27:24.077 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c5a4589197] 2024-03-24T00:27:24.077 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c5a448811b] 2024-03-24T00:27:24.077 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c5a44886bc] 2024-03-24T00:27:24.077 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c5a44721ff] 2024-03-24T00:27:24.078 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6bd7112488] 2024-03-24T00:27:24.078 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 13: (DispatchQueue::entry()+0x51f) [0x7f6bd710f60f] 2024-03-24T00:27:24.078 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6bd71e2e81] 2024-03-24T00:27:24.078 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6bd5bfd1ca] 2024-03-24T00:27:24.078 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 16: clone() 2024-03-24T00:27:24.078 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 2024-03-24T00:27:24.078 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 0> 2024-03-24T00:27:24.018+0000 7f6bcc856700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:27:24.078 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: in thread 7f6bcc856700 thread_name:ms_dispatch 2024-03-24T00:27:24.078 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 2024-03-24T00:27:24.078 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:27:24.078 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6bd5c07d20] 2024-03-24T00:27:24.078 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 2: gsignal() 2024-03-24T00:27:24.078 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 3: abort() 2024-03-24T00:27:24.078 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6bd6e804fb] 2024-03-24T00:27:24.078 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f6bd6e80667] 2024-03-24T00:27:24.079 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c5a46ba223] 2024-03-24T00:27:24.079 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c5a45345d6] 2024-03-24T00:27:24.079 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c5a4534da1] 2024-03-24T00:27:24.079 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c5a45372d5] 2024-03-24T00:27:24.079 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c5a4587233] 2024-03-24T00:27:24.079 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c5a4589197] 2024-03-24T00:27:24.079 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c5a448811b] 2024-03-24T00:27:24.079 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c5a44886bc] 2024-03-24T00:27:24.079 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c5a44721ff] 2024-03-24T00:27:24.079 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6bd7112488] 2024-03-24T00:27:24.079 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 16: (DispatchQueue::entry()+0x51f) [0x7f6bd710f60f] 2024-03-24T00:27:24.079 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6bd71e2e81] 2024-03-24T00:27:24.079 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6bd5bfd1ca] 2024-03-24T00:27:24.079 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 19: clone() 2024-03-24T00:27:24.079 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:27:24.079 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[72853]: 2024-03-24T00:27:24.080 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 podman[73025]: 2024-03-24 00:27:24.065216393 +0000 UTC m=+0.020678348 container died 4bf4122f4b784737ee0d1400942fcda02a38ebd4f987861e2d45de54f0599df4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-24T00:27:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:23 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999984 2024-03-24T00:27:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:23 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6832/1751539133,v1:172.21.15.107:6833/1751539133] up:reconnect 2024-03-24T00:27:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:23 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:reconnect,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:23 smithi106 ceph-mon[36505]: pgmap v206: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:27:24.680 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 173 sec latency 286271.189 ms 2024-03-24T00:27:25.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:24 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6832/1751539133,v1:172.21.15.107:6833/1751539133] up:rejoin 2024-03-24T00:27:25.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:24 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:rejoin,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:25.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:24 smithi078 ceph-mon[32429]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-24T00:27:25.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:24 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:27:25.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:24 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:27:25.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:24 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6832/1751539133,v1:172.21.15.107:6833/1751539133] up:active 2024-03-24T00:27:25.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:24 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=h=up:active} 3 up:standby-replay 4 up:standby 2024-03-24T00:27:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6832/1751539133,v1:172.21.15.107:6833/1751539133] up:rejoin 2024-03-24T00:27:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:rejoin,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-mon[36485]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-24T00:27:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:27:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:27:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6832/1751539133,v1:172.21.15.107:6833/1751539133] up:active 2024-03-24T00:27:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:24 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=h=up:active} 3 up:standby-replay 4 up:standby 2024-03-24T00:27:25.094 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 podman[73025]: 2024-03-24 00:27:24.67682845 +0000 UTC m=+0.632290366 container remove 4bf4122f4b784737ee0d1400942fcda02a38ebd4f987861e2d45de54f0599df4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-24T00:27:25.094 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:27:25.094 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:24 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.l.service: Failed with result 'exit-code'. 2024-03-24T00:27:25.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:24 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6832/1751539133,v1:172.21.15.107:6833/1751539133] up:rejoin 2024-03-24T00:27:25.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:24 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:rejoin,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:25.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:24 smithi106 ceph-mon[36505]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-24T00:27:25.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:24 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:27:25.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:24 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:27:25.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:24 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6832/1751539133,v1:172.21.15.107:6833/1751539133] up:active 2024-03-24T00:27:25.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:24 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=h=up:active} 3 up:standby-replay 4 up:standby 2024-03-24T00:27:25.680 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 174 sec latency 287271.281 ms 2024-03-24T00:27:26.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:25 smithi078 ceph-mon[32429]: pgmap v207: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-24T00:27:26.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:25 smithi107 ceph-mon[36485]: pgmap v207: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-24T00:27:26.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:25 smithi106 ceph-mon[36505]: pgmap v207: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-24T00:27:26.680 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 175 sec latency 288271.358 ms 2024-03-24T00:27:27.680 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 176 sec latency 289271.441 ms 2024-03-24T00:27:28.680 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 177 sec latency 290271.532 ms 2024-03-24T00:27:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:28 smithi078 ceph-mon[32429]: pgmap v208: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-24T00:27:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:28 smithi107 ceph-mon[36485]: pgmap v208: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-24T00:27:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:28 smithi106 ceph-mon[36505]: pgmap v208: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-24T00:27:29.681 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 178 sec latency 291271.637 ms 2024-03-24T00:27:30.680 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 179 sec latency 292271.728 ms 2024-03-24T00:27:30.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:30 smithi078 ceph-mon[32429]: pgmap v209: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-24T00:27:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:30 smithi107 ceph-mon[36485]: pgmap v209: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-24T00:27:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:30 smithi106 ceph-mon[36505]: pgmap v209: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-24T00:27:31.681 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 180 sec latency 293271.777 ms 2024-03-24T00:27:32.681 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 181 sec latency 294271.842 ms 2024-03-24T00:27:32.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:32 smithi078 ceph-mon[32429]: pgmap v210: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 193 B/s wr, 10 op/s 2024-03-24T00:27:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:32 smithi107 ceph-mon[36485]: pgmap v210: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 193 B/s wr, 10 op/s 2024-03-24T00:27:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:32 smithi106 ceph-mon[36505]: pgmap v210: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 193 B/s wr, 10 op/s 2024-03-24T00:27:33.681 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 182 sec latency 295271.937 ms 2024-03-24T00:27:34.681 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 183 sec latency 296272.052 ms 2024-03-24T00:27:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:34 smithi078 ceph-mon[32429]: pgmap v211: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-24T00:27:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:34 smithi107 ceph-mon[36485]: pgmap v211: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-24T00:27:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:34 smithi106 ceph-mon[36505]: pgmap v211: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-24T00:27:35.360 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:35 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.l.service: Scheduled restart job, restart counter is at 2. 2024-03-24T00:27:35.360 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:35 smithi107 systemd[1]: Stopped Ceph mds.l for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:27:35.360 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:35 smithi107 systemd[1]: Starting Ceph mds.l for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:27:35.360 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:35 smithi107 podman[73119]: 2024-03-24 00:27:35.231773603 +0000 UTC m=+0.018058004 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:27:35.361 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:35 smithi107 podman[73119]: 2024-03-24 00:27:35.361093319 +0000 UTC m=+0.147377706 container create 7be889f7000cc5168ca30d4a1a0cd6c8806bfd41ed078a2084c7c07cdcf516ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True) 2024-03-24T00:27:35.681 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 184 sec latency 297272.116 ms 2024-03-24T00:27:35.879 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:35 smithi107 podman[73119]: 2024-03-24 00:27:35.613598867 +0000 UTC m=+0.399883253 container init 7be889f7000cc5168ca30d4a1a0cd6c8806bfd41ed078a2084c7c07cdcf516ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True) 2024-03-24T00:27:35.879 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:35 smithi107 podman[73119]: 2024-03-24 00:27:35.616759802 +0000 UTC m=+0.403044185 container start 7be889f7000cc5168ca30d4a1a0cd6c8806bfd41ed078a2084c7c07cdcf516ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:27:35.879 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:35 smithi107 ceph-mds[73138]: -- 172.21.15.107:0/4018289159 <== mon.1 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560b58effa40 con 0x560b58ed3000 2024-03-24T00:27:35.879 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:35 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: starting mds.l at 2024-03-24T00:27:35.879 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:35 smithi107 bash[73119]: 7be889f7000cc5168ca30d4a1a0cd6c8806bfd41ed078a2084c7c07cdcf516ba 2024-03-24T00:27:35.879 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:35 smithi107 systemd[1]: Started Ceph mds.l for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:27:36.681 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 185 sec latency 298272.216 ms 2024-03-24T00:27:36.800 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:36 smithi107 ceph-mon[36485]: pgmap v212: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-24T00:27:36.800 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:36 smithi107 ceph-mon[36485]: daemon mds.l restarted 2024-03-24T00:27:36.800 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:36 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:27:36.800 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:36 smithi107 ceph-mon[36485]: osdmap e111: 12 total, 12 up, 12 in 2024-03-24T00:27:36.800 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:36 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6832/794068267,v1:172.21.15.107:6833/794068267] up:boot 2024-03-24T00:27:36.800 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:36 smithi107 ceph-mon[36485]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-03-24T00:27:36.801 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:36 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-24T00:27:36.801 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:27:36.801 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:36 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:replay,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:36.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:36 smithi078 ceph-mon[32429]: pgmap v212: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-24T00:27:36.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:36 smithi078 ceph-mon[32429]: daemon mds.l restarted 2024-03-24T00:27:36.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:36 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:27:36.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:36 smithi078 ceph-mon[32429]: osdmap e111: 12 total, 12 up, 12 in 2024-03-24T00:27:36.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:36 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6832/794068267,v1:172.21.15.107:6833/794068267] up:boot 2024-03-24T00:27:36.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:36 smithi078 ceph-mon[32429]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-03-24T00:27:36.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:36 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-24T00:27:36.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:27:36.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:36 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:replay,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:36.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:36 smithi106 ceph-mon[36505]: pgmap v212: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-24T00:27:36.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:36 smithi106 ceph-mon[36505]: daemon mds.l restarted 2024-03-24T00:27:36.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:36 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:27:36.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:36 smithi106 ceph-mon[36505]: osdmap e111: 12 total, 12 up, 12 in 2024-03-24T00:27:36.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:36 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6832/794068267,v1:172.21.15.107:6833/794068267] up:boot 2024-03-24T00:27:36.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:36 smithi106 ceph-mon[36505]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-03-24T00:27:36.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:36 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-24T00:27:36.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:27:36.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:36 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:replay,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:37.681 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 186 sec latency 299272.330 ms 2024-03-24T00:27:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:37 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6832/459568830,v1:172.21.15.106:6833/459568830] up:resolve 2024-03-24T00:27:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:37 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:resolve,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:37 smithi078 ceph-mon[32429]: pgmap v214: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:27:38.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:37 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6832/459568830,v1:172.21.15.106:6833/459568830] up:resolve 2024-03-24T00:27:38.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:37 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:resolve,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:38.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:37 smithi107 ceph-mon[36485]: pgmap v214: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:27:38.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:37 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6832/459568830,v1:172.21.15.106:6833/459568830] up:resolve 2024-03-24T00:27:38.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:37 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:resolve,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:38.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:37 smithi106 ceph-mon[36505]: pgmap v214: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:27:38.681 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 187 sec latency 300272.379 ms 2024-03-24T00:27:39.006 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:38 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999978 2024-03-24T00:27:39.006 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:38 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6832/459568830,v1:172.21.15.106:6833/459568830] up:reconnect 2024-03-24T00:27:39.006 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:38 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:reconnect,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:39.018 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:38 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999978 2024-03-24T00:27:39.018 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:38 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6832/459568830,v1:172.21.15.106:6833/459568830] up:reconnect 2024-03-24T00:27:39.018 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:38 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:reconnect,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:38 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999978 2024-03-24T00:27:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:38 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6832/459568830,v1:172.21.15.106:6833/459568830] up:reconnect 2024-03-24T00:27:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:38 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:reconnect,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:39.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fcb35713700 time 2024-03-24T00:27:39.018922+0000 2024-03-24T00:27:39.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:27:39.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:27:39.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fcb3fd3d4a1] 2024-03-24T00:27:39.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcb3fd3d667] 2024-03-24T00:27:39.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558fac7f6223] 2024-03-24T00:27:39.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558fac6705d6] 2024-03-24T00:27:39.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558fac670da1] 2024-03-24T00:27:39.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558fac6732d5] 2024-03-24T00:27:39.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558fac6c3233] 2024-03-24T00:27:39.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558fac6c5197] 2024-03-24T00:27:39.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558fac5c411b] 2024-03-24T00:27:39.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558fac5c46bc] 2024-03-24T00:27:39.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558fac5ae1ff] 2024-03-24T00:27:39.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcb3ffcf488] 2024-03-24T00:27:39.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 13: (DispatchQueue::entry()+0x51f) [0x7fcb3ffcc60f] 2024-03-24T00:27:39.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcb4009fe81] 2024-03-24T00:27:39.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcb3eaba1ca] 2024-03-24T00:27:39.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 16: clone() 2024-03-24T00:27:39.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 2024-03-24T00:27:39.020+0000 7fcb35713700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fcb35713700 time 2024-03-24T00:27:39.018922+0000 2024-03-24T00:27:39.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:27:39.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 2024-03-24T00:27:39.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:27:39.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fcb3fd3d4a1] 2024-03-24T00:27:39.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcb3fd3d667] 2024-03-24T00:27:39.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558fac7f6223] 2024-03-24T00:27:39.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558fac6705d6] 2024-03-24T00:27:39.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558fac670da1] 2024-03-24T00:27:39.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558fac6732d5] 2024-03-24T00:27:39.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558fac6c3233] 2024-03-24T00:27:39.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558fac6c5197] 2024-03-24T00:27:39.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558fac5c411b] 2024-03-24T00:27:39.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558fac5c46bc] 2024-03-24T00:27:39.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558fac5ae1ff] 2024-03-24T00:27:39.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcb3ffcf488] 2024-03-24T00:27:39.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 13: (DispatchQueue::entry()+0x51f) [0x7fcb3ffcc60f] 2024-03-24T00:27:39.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcb4009fe81] 2024-03-24T00:27:39.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcb3eaba1ca] 2024-03-24T00:27:39.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 16: clone() 2024-03-24T00:27:39.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 2024-03-24T00:27:39.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: *** Caught signal (Aborted) ** 2024-03-24T00:27:39.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: in thread 7fcb35713700 thread_name:ms_dispatch 2024-03-24T00:27:39.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:27:39.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcb3eac4d20] 2024-03-24T00:27:39.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 2: gsignal() 2024-03-24T00:27:39.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 3: abort() 2024-03-24T00:27:39.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fcb3fd3d4fb] 2024-03-24T00:27:39.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcb3fd3d667] 2024-03-24T00:27:39.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558fac7f6223] 2024-03-24T00:27:39.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558fac6705d6] 2024-03-24T00:27:39.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558fac670da1] 2024-03-24T00:27:39.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558fac6732d5] 2024-03-24T00:27:39.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558fac6c3233] 2024-03-24T00:27:39.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558fac6c5197] 2024-03-24T00:27:39.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558fac5c411b] 2024-03-24T00:27:39.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558fac5c46bc] 2024-03-24T00:27:39.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558fac5ae1ff] 2024-03-24T00:27:39.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcb3ffcf488] 2024-03-24T00:27:39.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 16: (DispatchQueue::entry()+0x51f) [0x7fcb3ffcc60f] 2024-03-24T00:27:39.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcb4009fe81] 2024-03-24T00:27:39.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcb3eaba1ca] 2024-03-24T00:27:39.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 19: clone() 2024-03-24T00:27:39.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 2024-03-24T00:27:39.022+0000 7fcb35713700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:27:39.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: in thread 7fcb35713700 thread_name:ms_dispatch 2024-03-24T00:27:39.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 2024-03-24T00:27:39.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:27:39.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcb3eac4d20] 2024-03-24T00:27:39.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 2: gsignal() 2024-03-24T00:27:39.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 3: abort() 2024-03-24T00:27:39.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fcb3fd3d4fb] 2024-03-24T00:27:39.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcb3fd3d667] 2024-03-24T00:27:39.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558fac7f6223] 2024-03-24T00:27:39.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558fac6705d6] 2024-03-24T00:27:39.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558fac670da1] 2024-03-24T00:27:39.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558fac6732d5] 2024-03-24T00:27:39.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558fac6c3233] 2024-03-24T00:27:39.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558fac6c5197] 2024-03-24T00:27:39.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558fac5c411b] 2024-03-24T00:27:39.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558fac5c46bc] 2024-03-24T00:27:39.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558fac5ae1ff] 2024-03-24T00:27:39.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcb3ffcf488] 2024-03-24T00:27:39.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 16: (DispatchQueue::entry()+0x51f) [0x7fcb3ffcc60f] 2024-03-24T00:27:39.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcb4009fe81] 2024-03-24T00:27:39.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcb3eaba1ca] 2024-03-24T00:27:39.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 19: clone() 2024-03-24T00:27:39.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:27:39.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 2024-03-24T00:27:39.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: -1> 2024-03-24T00:27:39.020+0000 7fcb35713700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fcb35713700 time 2024-03-24T00:27:39.018922+0000 2024-03-24T00:27:39.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:27:39.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 2024-03-24T00:27:39.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:27:39.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fcb3fd3d4a1] 2024-03-24T00:27:39.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcb3fd3d667] 2024-03-24T00:27:39.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558fac7f6223] 2024-03-24T00:27:39.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558fac6705d6] 2024-03-24T00:27:39.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558fac670da1] 2024-03-24T00:27:39.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558fac6732d5] 2024-03-24T00:27:39.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558fac6c3233] 2024-03-24T00:27:39.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558fac6c5197] 2024-03-24T00:27:39.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558fac5c411b] 2024-03-24T00:27:39.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558fac5c46bc] 2024-03-24T00:27:39.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558fac5ae1ff] 2024-03-24T00:27:39.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcb3ffcf488] 2024-03-24T00:27:39.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 13: (DispatchQueue::entry()+0x51f) [0x7fcb3ffcc60f] 2024-03-24T00:27:39.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcb4009fe81] 2024-03-24T00:27:39.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcb3eaba1ca] 2024-03-24T00:27:39.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 16: clone() 2024-03-24T00:27:39.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 2024-03-24T00:27:39.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 0> 2024-03-24T00:27:39.022+0000 7fcb35713700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:27:39.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: in thread 7fcb35713700 thread_name:ms_dispatch 2024-03-24T00:27:39.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 2024-03-24T00:27:39.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:27:39.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcb3eac4d20] 2024-03-24T00:27:39.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 2: gsignal() 2024-03-24T00:27:39.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 3: abort() 2024-03-24T00:27:39.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fcb3fd3d4fb] 2024-03-24T00:27:39.412 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcb3fd3d667] 2024-03-24T00:27:39.412 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558fac7f6223] 2024-03-24T00:27:39.412 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558fac6705d6] 2024-03-24T00:27:39.412 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558fac670da1] 2024-03-24T00:27:39.412 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558fac6732d5] 2024-03-24T00:27:39.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558fac6c3233] 2024-03-24T00:27:39.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558fac6c5197] 2024-03-24T00:27:39.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558fac5c411b] 2024-03-24T00:27:39.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558fac5c46bc] 2024-03-24T00:27:39.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558fac5ae1ff] 2024-03-24T00:27:39.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcb3ffcf488] 2024-03-24T00:27:39.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 16: (DispatchQueue::entry()+0x51f) [0x7fcb3ffcc60f] 2024-03-24T00:27:39.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcb4009fe81] 2024-03-24T00:27:39.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcb3eaba1ca] 2024-03-24T00:27:39.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 19: clone() 2024-03-24T00:27:39.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:27:39.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 2024-03-24T00:27:39.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: -1> 2024-03-24T00:27:39.020+0000 7fcb35713700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fcb35713700 time 2024-03-24T00:27:39.018922+0000 2024-03-24T00:27:39.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:27:39.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 2024-03-24T00:27:39.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:27:39.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fcb3fd3d4a1] 2024-03-24T00:27:39.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcb3fd3d667] 2024-03-24T00:27:39.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558fac7f6223] 2024-03-24T00:27:39.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558fac6705d6] 2024-03-24T00:27:39.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558fac670da1] 2024-03-24T00:27:39.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558fac6732d5] 2024-03-24T00:27:39.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558fac6c3233] 2024-03-24T00:27:39.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558fac6c5197] 2024-03-24T00:27:39.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558fac5c411b] 2024-03-24T00:27:39.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558fac5c46bc] 2024-03-24T00:27:39.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558fac5ae1ff] 2024-03-24T00:27:39.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcb3ffcf488] 2024-03-24T00:27:39.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 13: (DispatchQueue::entry()+0x51f) [0x7fcb3ffcc60f] 2024-03-24T00:27:39.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcb4009fe81] 2024-03-24T00:27:39.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcb3eaba1ca] 2024-03-24T00:27:39.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 16: clone() 2024-03-24T00:27:39.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 2024-03-24T00:27:39.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 0> 2024-03-24T00:27:39.022+0000 7fcb35713700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:27:39.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: in thread 7fcb35713700 thread_name:ms_dispatch 2024-03-24T00:27:39.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 2024-03-24T00:27:39.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:27:39.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcb3eac4d20] 2024-03-24T00:27:39.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 2: gsignal() 2024-03-24T00:27:39.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 3: abort() 2024-03-24T00:27:39.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fcb3fd3d4fb] 2024-03-24T00:27:39.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcb3fd3d667] 2024-03-24T00:27:39.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558fac7f6223] 2024-03-24T00:27:39.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558fac6705d6] 2024-03-24T00:27:39.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558fac670da1] 2024-03-24T00:27:39.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x558fac6732d5] 2024-03-24T00:27:39.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558fac6c3233] 2024-03-24T00:27:39.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558fac6c5197] 2024-03-24T00:27:39.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558fac5c411b] 2024-03-24T00:27:39.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558fac5c46bc] 2024-03-24T00:27:39.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558fac5ae1ff] 2024-03-24T00:27:39.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcb3ffcf488] 2024-03-24T00:27:39.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 16: (DispatchQueue::entry()+0x51f) [0x7fcb3ffcc60f] 2024-03-24T00:27:39.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcb4009fe81] 2024-03-24T00:27:39.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcb3eaba1ca] 2024-03-24T00:27:39.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 19: clone() 2024-03-24T00:27:39.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:27:39.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71170]: 2024-03-24T00:27:39.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 podman[71220]: 2024-03-24 00:27:39.06615182 +0000 UTC m=+0.020352882 container died 50659a837aba2803221dc98028fad5df7d51a0b4141bf20ec5625fd54ded0cab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=) 2024-03-24T00:27:39.681 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 188 sec latency 301272.414 ms 2024-03-24T00:27:39.845 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6832/459568830,v1:172.21.15.106:6833/459568830] up:rejoin 2024-03-24T00:27:39.845 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:rejoin,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:39.845 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-mon[36505]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-24T00:27:39.846 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:27:39.846 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:27:39.846 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6832/459568830,v1:172.21.15.106:6833/459568830] up:active 2024-03-24T00:27:39.846 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=h=up:active} 3 up:standby-replay 4 up:standby 2024-03-24T00:27:39.846 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:39 smithi106 ceph-mon[36505]: pgmap v215: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:27:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:39 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6832/459568830,v1:172.21.15.106:6833/459568830] up:rejoin 2024-03-24T00:27:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:39 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:rejoin,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:40.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:39 smithi078 ceph-mon[32429]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-24T00:27:40.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:39 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:27:40.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:39 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:27:40.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:39 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6832/459568830,v1:172.21.15.106:6833/459568830] up:active 2024-03-24T00:27:40.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:39 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=h=up:active} 3 up:standby-replay 4 up:standby 2024-03-24T00:27:40.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:39 smithi078 ceph-mon[32429]: pgmap v215: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:27:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:39 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6832/459568830,v1:172.21.15.106:6833/459568830] up:rejoin 2024-03-24T00:27:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:39 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:rejoin,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:39 smithi107 ceph-mon[36485]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-24T00:27:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:39 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:27:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:39 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:27:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:39 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6832/459568830,v1:172.21.15.106:6833/459568830] up:active 2024-03-24T00:27:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:39 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=h=up:active} 3 up:standby-replay 4 up:standby 2024-03-24T00:27:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:39 smithi107 ceph-mon[36485]: pgmap v215: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:27:40.123 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 podman[71220]: 2024-03-24 00:27:39.846022557 +0000 UTC m=+0.800223617 container remove 50659a837aba2803221dc98028fad5df7d51a0b4141bf20ec5625fd54ded0cab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:27:40.123 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:39 smithi106 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:27:40.402 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:40 smithi106 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.k.service: Failed with result 'exit-code'. 2024-03-24T00:27:40.681 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 189 sec latency 302272.483 ms 2024-03-24T00:27:41.681 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 190 sec latency 303272.569 ms 2024-03-24T00:27:42.681 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 191 sec latency 304272.684 ms 2024-03-24T00:27:42.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:42 smithi078 ceph-mon[32429]: pgmap v216: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:27:42.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:42 smithi107 ceph-mon[36485]: pgmap v216: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:27:42.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:42 smithi106 ceph-mon[36505]: pgmap v216: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:27:43.682 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 192 sec latency 305272.823 ms 2024-03-24T00:27:44.682 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 193 sec latency 306272.928 ms 2024-03-24T00:27:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:44 smithi078 ceph-mon[32429]: pgmap v217: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:27:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:44 smithi107 ceph-mon[36485]: pgmap v217: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:27:44.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:44 smithi106 ceph-mon[36505]: pgmap v217: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:27:45.682 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 194 sec latency 307273.001 ms 2024-03-24T00:27:46.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:46 smithi107 ceph-mon[36485]: pgmap v218: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:27:46.682 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 195 sec latency 308273.116 ms 2024-03-24T00:27:46.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:46 smithi078 ceph-mon[32429]: pgmap v218: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:27:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:46 smithi106 ceph-mon[36505]: pgmap v218: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:27:47.682 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 196 sec latency 309273.212 ms 2024-03-24T00:27:48.682 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 197 sec latency 310273.311 ms 2024-03-24T00:27:48.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:48 smithi078 ceph-mon[32429]: pgmap v219: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 175 B/s wr, 9 op/s 2024-03-24T00:27:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:48 smithi107 ceph-mon[36485]: pgmap v219: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 175 B/s wr, 9 op/s 2024-03-24T00:27:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:48 smithi106 ceph-mon[36505]: pgmap v219: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 175 B/s wr, 9 op/s 2024-03-24T00:27:49.682 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 198 sec latency 311273.418 ms 2024-03-24T00:27:50.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:50 smithi106 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.k.service: Scheduled restart job, restart counter is at 4. 2024-03-24T00:27:50.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:50 smithi106 systemd[1]: Stopped Ceph mds.k for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:27:50.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:50 smithi106 systemd[1]: Starting Ceph mds.k for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:27:50.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:50 smithi106 podman[71315]: 2024-03-24 00:27:50.288725765 +0000 UTC m=+0.018899936 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:27:50.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:50 smithi106 podman[71315]: 2024-03-24 00:27:50.405779611 +0000 UTC m=+0.135953767 container create f7c5a3114702168b0961ad77c0a481671c9a32238c16300a76b68d150dbdef0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-24T00:27:50.674 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:50 smithi106 ceph-mon[36505]: pgmap v220: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-24T00:27:50.683 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 199 sec latency 312273.500 ms 2024-03-24T00:27:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:50 smithi078 ceph-mon[32429]: pgmap v220: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-24T00:27:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:50 smithi107 ceph-mon[36485]: pgmap v220: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-24T00:27:51.152 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:50 smithi106 podman[71315]: 2024-03-24 00:27:50.67524565 +0000 UTC m=+0.405419806 container init f7c5a3114702168b0961ad77c0a481671c9a32238c16300a76b68d150dbdef0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-24T00:27:51.152 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:50 smithi106 podman[71315]: 2024-03-24 00:27:50.679061237 +0000 UTC m=+0.409235399 container start f7c5a3114702168b0961ad77c0a481671c9a32238c16300a76b68d150dbdef0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-03-24T00:27:51.153 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:50 smithi106 ceph-mds[71333]: -- 172.21.15.106:0/659122031 <== mon.2 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c6a2789a40 con 0x55c6a275d000 2024-03-24T00:27:51.153 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:50 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: starting mds.k at 2024-03-24T00:27:51.153 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:50 smithi106 bash[71315]: f7c5a3114702168b0961ad77c0a481671c9a32238c16300a76b68d150dbdef0a 2024-03-24T00:27:51.153 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:27:50 smithi106 systemd[1]: Started Ceph mds.k for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:27:51.682 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 200 sec latency 313273.563 ms 2024-03-24T00:27:51.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:51 smithi078 ceph-mon[32429]: daemon mds.k restarted 2024-03-24T00:27:51.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:51 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:27:51.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:51 smithi078 ceph-mon[32429]: osdmap e112: 12 total, 12 up, 12 in 2024-03-24T00:27:51.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:51 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6832/93515314,v1:172.21.15.106:6833/93515314] up:boot 2024-03-24T00:27:51.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:51 smithi078 ceph-mon[32429]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-24T00:27:51.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:51 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-24T00:27:51.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:51 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T00:27:51.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:51 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:replay,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:51.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:51 smithi107 ceph-mon[36485]: daemon mds.k restarted 2024-03-24T00:27:51.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:51 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:27:51.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:51 smithi107 ceph-mon[36485]: osdmap e112: 12 total, 12 up, 12 in 2024-03-24T00:27:51.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:51 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6832/93515314,v1:172.21.15.106:6833/93515314] up:boot 2024-03-24T00:27:51.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:51 smithi107 ceph-mon[36485]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-24T00:27:51.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:51 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-24T00:27:51.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:51 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T00:27:51.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:51 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:replay,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:51 smithi106 ceph-mon[36505]: daemon mds.k restarted 2024-03-24T00:27:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:51 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:27:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:51 smithi106 ceph-mon[36505]: osdmap e112: 12 total, 12 up, 12 in 2024-03-24T00:27:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:51 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6832/93515314,v1:172.21.15.106:6833/93515314] up:boot 2024-03-24T00:27:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:51 smithi106 ceph-mon[36505]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-24T00:27:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:51 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-24T00:27:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:51 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T00:27:51.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:51 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:replay,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:52.682 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 201 sec latency 314273.625 ms 2024-03-24T00:27:52.734 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:52 smithi078 ceph-mon[32429]: pgmap v222: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:27:52.735 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:52 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6832/794068267,v1:172.21.15.107:6833/794068267] up:resolve 2024-03-24T00:27:52.735 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:52 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:resolve,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:52.744 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:52 smithi107 ceph-mon[36485]: pgmap v222: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:27:52.744 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:52 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6832/794068267,v1:172.21.15.107:6833/794068267] up:resolve 2024-03-24T00:27:52.744 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:52 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:resolve,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:52.803 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:52 smithi106 ceph-mon[36505]: pgmap v222: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:27:52.804 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:52 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6832/794068267,v1:172.21.15.107:6833/794068267] up:resolve 2024-03-24T00:27:52.804 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:52 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:resolve,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:53.682 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 202 sec latency 315273.677 ms 2024-03-24T00:27:54.010 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:53 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:27:54.010 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:53 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6832/794068267,v1:172.21.15.107:6833/794068267] up:reconnect 2024-03-24T00:27:54.010 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:53 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:reconnect,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:54.010 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:53 smithi078 ceph-mon[32429]: pgmap v223: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:27:54.026 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:53 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:27:54.026 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:53 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6832/794068267,v1:172.21.15.107:6833/794068267] up:reconnect 2024-03-24T00:27:54.026 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:53 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:reconnect,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:54.027 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:53 smithi107 ceph-mon[36485]: pgmap v223: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:27:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:53 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:27:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:53 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6832/794068267,v1:172.21.15.107:6833/794068267] up:reconnect 2024-03-24T00:27:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:53 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:reconnect,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:54.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:53 smithi106 ceph-mon[36505]: pgmap v223: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:27:54.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f4c08925700 time 2024-03-24T00:27:54.026572+0000 2024-03-24T00:27:54.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:27:54.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:27:54.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f4c12f4f4a1] 2024-03-24T00:27:54.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f4c12f4f667] 2024-03-24T00:27:54.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560b57738223] 2024-03-24T00:27:54.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560b575b25d6] 2024-03-24T00:27:54.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560b575b2da1] 2024-03-24T00:27:54.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560b575b52d5] 2024-03-24T00:27:54.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560b57605233] 2024-03-24T00:27:54.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560b57607197] 2024-03-24T00:27:54.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560b5750611b] 2024-03-24T00:27:54.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560b575066bc] 2024-03-24T00:27:54.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560b574f01ff] 2024-03-24T00:27:54.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f4c131e1488] 2024-03-24T00:27:54.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 13: (DispatchQueue::entry()+0x51f) [0x7f4c131de60f] 2024-03-24T00:27:54.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4c132b1e81] 2024-03-24T00:27:54.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4c11ccc1ca] 2024-03-24T00:27:54.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 16: clone() 2024-03-24T00:27:54.346 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 2024-03-24T00:27:54.027+0000 7f4c08925700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f4c08925700 time 2024-03-24T00:27:54.026572+0000 2024-03-24T00:27:54.346 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:27:54.346 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 2024-03-24T00:27:54.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:27:54.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f4c12f4f4a1] 2024-03-24T00:27:54.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f4c12f4f667] 2024-03-24T00:27:54.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560b57738223] 2024-03-24T00:27:54.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560b575b25d6] 2024-03-24T00:27:54.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560b575b2da1] 2024-03-24T00:27:54.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560b575b52d5] 2024-03-24T00:27:54.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560b57605233] 2024-03-24T00:27:54.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560b57607197] 2024-03-24T00:27:54.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560b5750611b] 2024-03-24T00:27:54.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560b575066bc] 2024-03-24T00:27:54.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560b574f01ff] 2024-03-24T00:27:54.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f4c131e1488] 2024-03-24T00:27:54.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 13: (DispatchQueue::entry()+0x51f) [0x7f4c131de60f] 2024-03-24T00:27:54.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4c132b1e81] 2024-03-24T00:27:54.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4c11ccc1ca] 2024-03-24T00:27:54.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 16: clone() 2024-03-24T00:27:54.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 2024-03-24T00:27:54.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: *** Caught signal (Aborted) ** 2024-03-24T00:27:54.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: in thread 7f4c08925700 thread_name:ms_dispatch 2024-03-24T00:27:54.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:27:54.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4c11cd6d20] 2024-03-24T00:27:54.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 2: gsignal() 2024-03-24T00:27:54.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 3: abort() 2024-03-24T00:27:54.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f4c12f4f4fb] 2024-03-24T00:27:54.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f4c12f4f667] 2024-03-24T00:27:54.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560b57738223] 2024-03-24T00:27:54.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560b575b25d6] 2024-03-24T00:27:54.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560b575b2da1] 2024-03-24T00:27:54.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560b575b52d5] 2024-03-24T00:27:54.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560b57605233] 2024-03-24T00:27:54.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560b57607197] 2024-03-24T00:27:54.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560b5750611b] 2024-03-24T00:27:54.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560b575066bc] 2024-03-24T00:27:54.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560b574f01ff] 2024-03-24T00:27:54.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f4c131e1488] 2024-03-24T00:27:54.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 16: (DispatchQueue::entry()+0x51f) [0x7f4c131de60f] 2024-03-24T00:27:54.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4c132b1e81] 2024-03-24T00:27:54.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4c11ccc1ca] 2024-03-24T00:27:54.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 19: clone() 2024-03-24T00:27:54.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 2024-03-24T00:27:54.028+0000 7f4c08925700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:27:54.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: in thread 7f4c08925700 thread_name:ms_dispatch 2024-03-24T00:27:54.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 2024-03-24T00:27:54.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:27:54.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4c11cd6d20] 2024-03-24T00:27:54.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 2: gsignal() 2024-03-24T00:27:54.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 3: abort() 2024-03-24T00:27:54.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f4c12f4f4fb] 2024-03-24T00:27:54.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f4c12f4f667] 2024-03-24T00:27:54.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560b57738223] 2024-03-24T00:27:54.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560b575b25d6] 2024-03-24T00:27:54.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560b575b2da1] 2024-03-24T00:27:54.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560b575b52d5] 2024-03-24T00:27:54.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560b57605233] 2024-03-24T00:27:54.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560b57607197] 2024-03-24T00:27:54.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560b5750611b] 2024-03-24T00:27:54.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560b575066bc] 2024-03-24T00:27:54.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560b574f01ff] 2024-03-24T00:27:54.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f4c131e1488] 2024-03-24T00:27:54.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 16: (DispatchQueue::entry()+0x51f) [0x7f4c131de60f] 2024-03-24T00:27:54.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4c132b1e81] 2024-03-24T00:27:54.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4c11ccc1ca] 2024-03-24T00:27:54.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 19: clone() 2024-03-24T00:27:54.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:27:54.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 2024-03-24T00:27:54.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: -1> 2024-03-24T00:27:54.027+0000 7f4c08925700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f4c08925700 time 2024-03-24T00:27:54.026572+0000 2024-03-24T00:27:54.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:27:54.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 2024-03-24T00:27:54.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:27:54.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f4c12f4f4a1] 2024-03-24T00:27:54.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f4c12f4f667] 2024-03-24T00:27:54.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560b57738223] 2024-03-24T00:27:54.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560b575b25d6] 2024-03-24T00:27:54.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560b575b2da1] 2024-03-24T00:27:54.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560b575b52d5] 2024-03-24T00:27:54.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560b57605233] 2024-03-24T00:27:54.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560b57607197] 2024-03-24T00:27:54.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560b5750611b] 2024-03-24T00:27:54.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560b575066bc] 2024-03-24T00:27:54.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560b574f01ff] 2024-03-24T00:27:54.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f4c131e1488] 2024-03-24T00:27:54.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 13: (DispatchQueue::entry()+0x51f) [0x7f4c131de60f] 2024-03-24T00:27:54.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4c132b1e81] 2024-03-24T00:27:54.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4c11ccc1ca] 2024-03-24T00:27:54.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 16: clone() 2024-03-24T00:27:54.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 2024-03-24T00:27:54.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 0> 2024-03-24T00:27:54.028+0000 7f4c08925700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:27:54.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: in thread 7f4c08925700 thread_name:ms_dispatch 2024-03-24T00:27:54.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 2024-03-24T00:27:54.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:27:54.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4c11cd6d20] 2024-03-24T00:27:54.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 2: gsignal() 2024-03-24T00:27:54.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 3: abort() 2024-03-24T00:27:54.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f4c12f4f4fb] 2024-03-24T00:27:54.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f4c12f4f667] 2024-03-24T00:27:54.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560b57738223] 2024-03-24T00:27:54.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560b575b25d6] 2024-03-24T00:27:54.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560b575b2da1] 2024-03-24T00:27:54.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560b575b52d5] 2024-03-24T00:27:54.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560b57605233] 2024-03-24T00:27:54.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560b57607197] 2024-03-24T00:27:54.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560b5750611b] 2024-03-24T00:27:54.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560b575066bc] 2024-03-24T00:27:54.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560b574f01ff] 2024-03-24T00:27:54.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f4c131e1488] 2024-03-24T00:27:54.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 16: (DispatchQueue::entry()+0x51f) [0x7f4c131de60f] 2024-03-24T00:27:54.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4c132b1e81] 2024-03-24T00:27:54.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4c11ccc1ca] 2024-03-24T00:27:54.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 19: clone() 2024-03-24T00:27:54.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:27:54.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 2024-03-24T00:27:54.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: -1> 2024-03-24T00:27:54.027+0000 7f4c08925700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f4c08925700 time 2024-03-24T00:27:54.026572+0000 2024-03-24T00:27:54.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:27:54.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 2024-03-24T00:27:54.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:27:54.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f4c12f4f4a1] 2024-03-24T00:27:54.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f4c12f4f667] 2024-03-24T00:27:54.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560b57738223] 2024-03-24T00:27:54.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560b575b25d6] 2024-03-24T00:27:54.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560b575b2da1] 2024-03-24T00:27:54.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560b575b52d5] 2024-03-24T00:27:54.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560b57605233] 2024-03-24T00:27:54.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560b57607197] 2024-03-24T00:27:54.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560b5750611b] 2024-03-24T00:27:54.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560b575066bc] 2024-03-24T00:27:54.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560b574f01ff] 2024-03-24T00:27:54.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f4c131e1488] 2024-03-24T00:27:54.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 13: (DispatchQueue::entry()+0x51f) [0x7f4c131de60f] 2024-03-24T00:27:54.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4c132b1e81] 2024-03-24T00:27:54.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4c11ccc1ca] 2024-03-24T00:27:54.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 16: clone() 2024-03-24T00:27:54.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 2024-03-24T00:27:54.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 0> 2024-03-24T00:27:54.028+0000 7f4c08925700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:27:54.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: in thread 7f4c08925700 thread_name:ms_dispatch 2024-03-24T00:27:54.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 2024-03-24T00:27:54.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:27:54.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4c11cd6d20] 2024-03-24T00:27:54.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 2: gsignal() 2024-03-24T00:27:54.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 3: abort() 2024-03-24T00:27:54.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f4c12f4f4fb] 2024-03-24T00:27:54.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f4c12f4f667] 2024-03-24T00:27:54.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560b57738223] 2024-03-24T00:27:54.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560b575b25d6] 2024-03-24T00:27:54.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560b575b2da1] 2024-03-24T00:27:54.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560b575b52d5] 2024-03-24T00:27:54.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560b57605233] 2024-03-24T00:27:54.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560b57607197] 2024-03-24T00:27:54.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560b5750611b] 2024-03-24T00:27:54.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560b575066bc] 2024-03-24T00:27:54.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560b574f01ff] 2024-03-24T00:27:54.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f4c131e1488] 2024-03-24T00:27:54.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 16: (DispatchQueue::entry()+0x51f) [0x7f4c131de60f] 2024-03-24T00:27:54.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4c132b1e81] 2024-03-24T00:27:54.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4c11ccc1ca] 2024-03-24T00:27:54.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 19: clone() 2024-03-24T00:27:54.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:27:54.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73134]: 2024-03-24T00:27:54.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 podman[73184]: 2024-03-24 00:27:54.079469679 +0000 UTC m=+0.020925563 container died 7be889f7000cc5168ca30d4a1a0cd6c8806bfd41ed078a2084c7c07cdcf516ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-24T00:27:54.683 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 203 sec latency 316273.769 ms 2024-03-24T00:27:55.020 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6832/794068267,v1:172.21.15.107:6833/794068267] up:rejoin 2024-03-24T00:27:55.020 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:rejoin,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:55.020 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-mon[36485]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-24T00:27:55.020 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:27:55.020 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:27:55.020 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6832/794068267,v1:172.21.15.107:6833/794068267] up:active 2024-03-24T00:27:55.021 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:54 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=h=up:active} 3 up:standby-replay 4 up:standby 2024-03-24T00:27:55.021 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 podman[73184]: 2024-03-24 00:27:54.792740249 +0000 UTC m=+0.734196187 container remove 7be889f7000cc5168ca30d4a1a0cd6c8806bfd41ed078a2084c7c07cdcf516ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:27:55.021 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:54 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:27:55.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:54 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6832/794068267,v1:172.21.15.107:6833/794068267] up:rejoin 2024-03-24T00:27:55.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:54 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:rejoin,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:55.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:54 smithi078 ceph-mon[32429]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-24T00:27:55.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:54 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:27:55.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:54 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:27:55.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:54 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6832/794068267,v1:172.21.15.107:6833/794068267] up:active 2024-03-24T00:27:55.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:54 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=h=up:active} 3 up:standby-replay 4 up:standby 2024-03-24T00:27:55.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:54 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6832/794068267,v1:172.21.15.107:6833/794068267] up:rejoin 2024-03-24T00:27:55.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:54 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:rejoin,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:27:55.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:54 smithi106 ceph-mon[36505]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-24T00:27:55.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:54 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:27:55.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:54 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:27:55.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:54 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6832/794068267,v1:172.21.15.107:6833/794068267] up:active 2024-03-24T00:27:55.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:54 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=h=up:active} 3 up:standby-replay 4 up:standby 2024-03-24T00:27:55.343 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:27:55 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.l.service: Failed with result 'exit-code'. 2024-03-24T00:27:55.683 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 204 sec latency 317273.853 ms 2024-03-24T00:27:56.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:55 smithi078 ceph-mon[32429]: pgmap v224: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:27:56.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:55 smithi107 ceph-mon[36485]: pgmap v224: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:27:56.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:55 smithi106 ceph-mon[36505]: pgmap v224: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:27:56.683 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 205 sec latency 318273.901 ms 2024-03-24T00:27:57.683 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 206 sec latency 319273.951 ms 2024-03-24T00:27:58.683 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 207 sec latency 320274.011 ms 2024-03-24T00:27:58.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:27:58 smithi078 ceph-mon[32429]: pgmap v225: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:27:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:27:58 smithi107 ceph-mon[36485]: pgmap v225: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:27:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:27:58 smithi106 ceph-mon[36505]: pgmap v225: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:27:59.683 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 208 sec latency 321274.121 ms 2024-03-24T00:28:00.683 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 209 sec latency 322274.217 ms 2024-03-24T00:28:00.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:00 smithi078 ceph-mon[32429]: pgmap v226: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:28:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:00 smithi107 ceph-mon[36485]: pgmap v226: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:28:00.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:00 smithi106 ceph-mon[36505]: pgmap v226: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:28:01.683 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 210 sec latency 323274.329 ms 2024-03-24T00:28:02.683 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 211 sec latency 324274.430 ms 2024-03-24T00:28:02.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:02 smithi078 ceph-mon[32429]: pgmap v227: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 192 B/s wr, 11 op/s 2024-03-24T00:28:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:02 smithi107 ceph-mon[36485]: pgmap v227: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 192 B/s wr, 11 op/s 2024-03-24T00:28:02.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:02 smithi106 ceph-mon[36505]: pgmap v227: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 192 B/s wr, 11 op/s 2024-03-24T00:28:03.683 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 212 sec latency 325274.535 ms 2024-03-24T00:28:04.683 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 213 sec latency 326274.631 ms 2024-03-24T00:28:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:04 smithi078 ceph-mon[32429]: pgmap v228: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-24T00:28:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:04 smithi107 ceph-mon[36485]: pgmap v228: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-24T00:28:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:04 smithi106 ceph-mon[36505]: pgmap v228: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-24T00:28:05.448 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:05 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.l.service: Scheduled restart job, restart counter is at 3. 2024-03-24T00:28:05.448 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:05 smithi107 systemd[1]: Stopped Ceph mds.l for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:28:05.448 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:05 smithi107 systemd[1]: Starting Ceph mds.l for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:28:05.448 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:05 smithi107 podman[73276]: 2024-03-24 00:28:05.228925814 +0000 UTC m=+0.018199511 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:28:05.449 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:05 smithi107 podman[73276]: 2024-03-24 00:28:05.334088087 +0000 UTC m=+0.123361813 container create 028ebe198cbdcfa7f04ae4a85494544f94b60111d06138fb734d03d913fa7d00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:28:05.683 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 214 sec latency 327274.752 ms 2024-03-24T00:28:05.844 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:05 smithi107 podman[73276]: 2024-03-24 00:28:05.569994377 +0000 UTC m=+0.359268061 container init 028ebe198cbdcfa7f04ae4a85494544f94b60111d06138fb734d03d913fa7d00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-24T00:28:05.844 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:05 smithi107 podman[73276]: 2024-03-24 00:28:05.572728315 +0000 UTC m=+0.362001994 container start 028ebe198cbdcfa7f04ae4a85494544f94b60111d06138fb734d03d913fa7d00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-24T00:28:05.844 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:05 smithi107 ceph-mds[73296]: -- 172.21.15.107:0/1235088130 <== mon.1 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563403f55a40 con 0x563403f29000 2024-03-24T00:28:05.844 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:05 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: starting mds.l at 2024-03-24T00:28:05.844 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:05 smithi107 bash[73276]: 028ebe198cbdcfa7f04ae4a85494544f94b60111d06138fb734d03d913fa7d00 2024-03-24T00:28:05.844 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:05 smithi107 systemd[1]: Started Ceph mds.l for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:28:06.684 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 215 sec latency 328274.871 ms 2024-03-24T00:28:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:06 smithi078 ceph-mon[32429]: pgmap v229: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-24T00:28:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:06 smithi078 ceph-mon[32429]: daemon mds.l restarted 2024-03-24T00:28:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:06 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:28:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:06 smithi078 ceph-mon[32429]: osdmap e113: 12 total, 12 up, 12 in 2024-03-24T00:28:06.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:06 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6832/3928685127,v1:172.21.15.107:6833/3928685127] up:boot 2024-03-24T00:28:06.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:06 smithi078 ceph-mon[32429]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-03-24T00:28:06.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:06 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-24T00:28:06.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:06 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:28:06.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:06 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:replay,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:06 smithi107 ceph-mon[36485]: pgmap v229: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-24T00:28:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:06 smithi107 ceph-mon[36485]: daemon mds.l restarted 2024-03-24T00:28:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:06 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:28:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:06 smithi107 ceph-mon[36485]: osdmap e113: 12 total, 12 up, 12 in 2024-03-24T00:28:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:06 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6832/3928685127,v1:172.21.15.107:6833/3928685127] up:boot 2024-03-24T00:28:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:06 smithi107 ceph-mon[36485]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-03-24T00:28:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:06 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-24T00:28:06.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:06 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:28:06.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:06 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:replay,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:06 smithi106 ceph-mon[36505]: pgmap v229: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-24T00:28:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:06 smithi106 ceph-mon[36505]: daemon mds.l restarted 2024-03-24T00:28:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:06 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:28:06.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:06 smithi106 ceph-mon[36505]: osdmap e113: 12 total, 12 up, 12 in 2024-03-24T00:28:06.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:06 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6832/3928685127,v1:172.21.15.107:6833/3928685127] up:boot 2024-03-24T00:28:06.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:06 smithi106 ceph-mon[36505]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-03-24T00:28:06.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:06 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-24T00:28:06.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:06 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:28:06.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:06 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:replay,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:07.684 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 216 sec latency 329274.994 ms 2024-03-24T00:28:07.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:07 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6832/93515314,v1:172.21.15.106:6833/93515314] up:resolve 2024-03-24T00:28:07.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:07 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:resolve,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:07.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:07 smithi106 ceph-mon[36505]: pgmap v231: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:28:08.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:07 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6832/93515314,v1:172.21.15.106:6833/93515314] up:resolve 2024-03-24T00:28:08.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:07 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:resolve,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:08.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:07 smithi078 ceph-mon[32429]: pgmap v231: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:28:08.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:07 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6832/93515314,v1:172.21.15.106:6833/93515314] up:resolve 2024-03-24T00:28:08.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:07 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:resolve,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:08.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:07 smithi107 ceph-mon[36485]: pgmap v231: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:28:08.684 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 217 sec latency 330275.075 ms 2024-03-24T00:28:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:08 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:28:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:08 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6832/93515314,v1:172.21.15.106:6833/93515314] up:reconnect 2024-03-24T00:28:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:08 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:reconnect,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:09.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:08 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:28:09.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:08 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6832/93515314,v1:172.21.15.106:6833/93515314] up:reconnect 2024-03-24T00:28:09.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:08 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:reconnect,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:08 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:28:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:08 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6832/93515314,v1:172.21.15.106:6833/93515314] up:reconnect 2024-03-24T00:28:09.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:08 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:reconnect,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:09.396 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f64c92f3700 time 2024-03-24T00:28:09.028151+0000 2024-03-24T00:28:09.396 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:28:09.396 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:28:09.396 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f64d391d4a1] 2024-03-24T00:28:09.396 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f64d391d667] 2024-03-24T00:28:09.396 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c6a055a223] 2024-03-24T00:28:09.396 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c6a03d45d6] 2024-03-24T00:28:09.396 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c6a03d4da1] 2024-03-24T00:28:09.396 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c6a03d72d5] 2024-03-24T00:28:09.397 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c6a0427233] 2024-03-24T00:28:09.397 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c6a0429197] 2024-03-24T00:28:09.397 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c6a032811b] 2024-03-24T00:28:09.397 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c6a03286bc] 2024-03-24T00:28:09.397 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c6a03121ff] 2024-03-24T00:28:09.397 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f64d3baf488] 2024-03-24T00:28:09.397 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 13: (DispatchQueue::entry()+0x51f) [0x7f64d3bac60f] 2024-03-24T00:28:09.397 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f64d3c7fe81] 2024-03-24T00:28:09.397 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f64d269a1ca] 2024-03-24T00:28:09.397 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 16: clone() 2024-03-24T00:28:09.398 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 2024-03-24T00:28:09.029+0000 7f64c92f3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f64c92f3700 time 2024-03-24T00:28:09.028151+0000 2024-03-24T00:28:09.398 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:28:09.398 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 2024-03-24T00:28:09.399 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:28:09.399 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f64d391d4a1] 2024-03-24T00:28:09.399 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f64d391d667] 2024-03-24T00:28:09.399 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c6a055a223] 2024-03-24T00:28:09.399 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c6a03d45d6] 2024-03-24T00:28:09.399 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c6a03d4da1] 2024-03-24T00:28:09.399 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c6a03d72d5] 2024-03-24T00:28:09.399 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c6a0427233] 2024-03-24T00:28:09.399 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c6a0429197] 2024-03-24T00:28:09.399 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c6a032811b] 2024-03-24T00:28:09.399 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c6a03286bc] 2024-03-24T00:28:09.399 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c6a03121ff] 2024-03-24T00:28:09.399 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f64d3baf488] 2024-03-24T00:28:09.399 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 13: (DispatchQueue::entry()+0x51f) [0x7f64d3bac60f] 2024-03-24T00:28:09.400 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f64d3c7fe81] 2024-03-24T00:28:09.400 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f64d269a1ca] 2024-03-24T00:28:09.400 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 16: clone() 2024-03-24T00:28:09.400 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 2024-03-24T00:28:09.400 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: *** Caught signal (Aborted) ** 2024-03-24T00:28:09.400 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: in thread 7f64c92f3700 thread_name:ms_dispatch 2024-03-24T00:28:09.400 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:28:09.400 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f64d26a4d20] 2024-03-24T00:28:09.400 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 2: gsignal() 2024-03-24T00:28:09.400 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 3: abort() 2024-03-24T00:28:09.400 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f64d391d4fb] 2024-03-24T00:28:09.400 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f64d391d667] 2024-03-24T00:28:09.400 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c6a055a223] 2024-03-24T00:28:09.400 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c6a03d45d6] 2024-03-24T00:28:09.401 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c6a03d4da1] 2024-03-24T00:28:09.401 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c6a03d72d5] 2024-03-24T00:28:09.401 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c6a0427233] 2024-03-24T00:28:09.401 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c6a0429197] 2024-03-24T00:28:09.401 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c6a032811b] 2024-03-24T00:28:09.401 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c6a03286bc] 2024-03-24T00:28:09.401 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c6a03121ff] 2024-03-24T00:28:09.401 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f64d3baf488] 2024-03-24T00:28:09.401 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 16: (DispatchQueue::entry()+0x51f) [0x7f64d3bac60f] 2024-03-24T00:28:09.401 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f64d3c7fe81] 2024-03-24T00:28:09.401 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f64d269a1ca] 2024-03-24T00:28:09.401 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 19: clone() 2024-03-24T00:28:09.401 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 2024-03-24T00:28:09.031+0000 7f64c92f3700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:28:09.402 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: in thread 7f64c92f3700 thread_name:ms_dispatch 2024-03-24T00:28:09.402 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 2024-03-24T00:28:09.402 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:28:09.402 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f64d26a4d20] 2024-03-24T00:28:09.402 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 2: gsignal() 2024-03-24T00:28:09.402 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 3: abort() 2024-03-24T00:28:09.402 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f64d391d4fb] 2024-03-24T00:28:09.402 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f64d391d667] 2024-03-24T00:28:09.402 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c6a055a223] 2024-03-24T00:28:09.402 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c6a03d45d6] 2024-03-24T00:28:09.402 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c6a03d4da1] 2024-03-24T00:28:09.402 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c6a03d72d5] 2024-03-24T00:28:09.402 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c6a0427233] 2024-03-24T00:28:09.402 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c6a0429197] 2024-03-24T00:28:09.402 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c6a032811b] 2024-03-24T00:28:09.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c6a03286bc] 2024-03-24T00:28:09.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c6a03121ff] 2024-03-24T00:28:09.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f64d3baf488] 2024-03-24T00:28:09.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 16: (DispatchQueue::entry()+0x51f) [0x7f64d3bac60f] 2024-03-24T00:28:09.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f64d3c7fe81] 2024-03-24T00:28:09.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f64d269a1ca] 2024-03-24T00:28:09.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 19: clone() 2024-03-24T00:28:09.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:28:09.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 2024-03-24T00:28:09.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: -1> 2024-03-24T00:28:09.029+0000 7f64c92f3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f64c92f3700 time 2024-03-24T00:28:09.028151+0000 2024-03-24T00:28:09.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:28:09.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 2024-03-24T00:28:09.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:28:09.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f64d391d4a1] 2024-03-24T00:28:09.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f64d391d667] 2024-03-24T00:28:09.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c6a055a223] 2024-03-24T00:28:09.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c6a03d45d6] 2024-03-24T00:28:09.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c6a03d4da1] 2024-03-24T00:28:09.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c6a03d72d5] 2024-03-24T00:28:09.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c6a0427233] 2024-03-24T00:28:09.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c6a0429197] 2024-03-24T00:28:09.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c6a032811b] 2024-03-24T00:28:09.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c6a03286bc] 2024-03-24T00:28:09.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c6a03121ff] 2024-03-24T00:28:09.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f64d3baf488] 2024-03-24T00:28:09.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 13: (DispatchQueue::entry()+0x51f) [0x7f64d3bac60f] 2024-03-24T00:28:09.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f64d3c7fe81] 2024-03-24T00:28:09.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f64d269a1ca] 2024-03-24T00:28:09.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 16: clone() 2024-03-24T00:28:09.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 2024-03-24T00:28:09.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 0> 2024-03-24T00:28:09.031+0000 7f64c92f3700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:28:09.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: in thread 7f64c92f3700 thread_name:ms_dispatch 2024-03-24T00:28:09.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 2024-03-24T00:28:09.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:28:09.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f64d26a4d20] 2024-03-24T00:28:09.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 2: gsignal() 2024-03-24T00:28:09.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 3: abort() 2024-03-24T00:28:09.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f64d391d4fb] 2024-03-24T00:28:09.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f64d391d667] 2024-03-24T00:28:09.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c6a055a223] 2024-03-24T00:28:09.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c6a03d45d6] 2024-03-24T00:28:09.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c6a03d4da1] 2024-03-24T00:28:09.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c6a03d72d5] 2024-03-24T00:28:09.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c6a0427233] 2024-03-24T00:28:09.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c6a0429197] 2024-03-24T00:28:09.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c6a032811b] 2024-03-24T00:28:09.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c6a03286bc] 2024-03-24T00:28:09.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c6a03121ff] 2024-03-24T00:28:09.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f64d3baf488] 2024-03-24T00:28:09.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 16: (DispatchQueue::entry()+0x51f) [0x7f64d3bac60f] 2024-03-24T00:28:09.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f64d3c7fe81] 2024-03-24T00:28:09.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f64d269a1ca] 2024-03-24T00:28:09.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 19: clone() 2024-03-24T00:28:09.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:28:09.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 2024-03-24T00:28:09.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: -1> 2024-03-24T00:28:09.029+0000 7f64c92f3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f64c92f3700 time 2024-03-24T00:28:09.028151+0000 2024-03-24T00:28:09.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:28:09.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 2024-03-24T00:28:09.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:28:09.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f64d391d4a1] 2024-03-24T00:28:09.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f64d391d667] 2024-03-24T00:28:09.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c6a055a223] 2024-03-24T00:28:09.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c6a03d45d6] 2024-03-24T00:28:09.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c6a03d4da1] 2024-03-24T00:28:09.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c6a03d72d5] 2024-03-24T00:28:09.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c6a0427233] 2024-03-24T00:28:09.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c6a0429197] 2024-03-24T00:28:09.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c6a032811b] 2024-03-24T00:28:09.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c6a03286bc] 2024-03-24T00:28:09.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c6a03121ff] 2024-03-24T00:28:09.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f64d3baf488] 2024-03-24T00:28:09.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 13: (DispatchQueue::entry()+0x51f) [0x7f64d3bac60f] 2024-03-24T00:28:09.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f64d3c7fe81] 2024-03-24T00:28:09.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f64d269a1ca] 2024-03-24T00:28:09.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 16: clone() 2024-03-24T00:28:09.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 2024-03-24T00:28:09.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 0> 2024-03-24T00:28:09.031+0000 7f64c92f3700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:28:09.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: in thread 7f64c92f3700 thread_name:ms_dispatch 2024-03-24T00:28:09.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 2024-03-24T00:28:09.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:28:09.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f64d26a4d20] 2024-03-24T00:28:09.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 2: gsignal() 2024-03-24T00:28:09.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 3: abort() 2024-03-24T00:28:09.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f64d391d4fb] 2024-03-24T00:28:09.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f64d391d667] 2024-03-24T00:28:09.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c6a055a223] 2024-03-24T00:28:09.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c6a03d45d6] 2024-03-24T00:28:09.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c6a03d4da1] 2024-03-24T00:28:09.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55c6a03d72d5] 2024-03-24T00:28:09.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c6a0427233] 2024-03-24T00:28:09.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c6a0429197] 2024-03-24T00:28:09.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c6a032811b] 2024-03-24T00:28:09.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c6a03286bc] 2024-03-24T00:28:09.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c6a03121ff] 2024-03-24T00:28:09.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f64d3baf488] 2024-03-24T00:28:09.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 16: (DispatchQueue::entry()+0x51f) [0x7f64d3bac60f] 2024-03-24T00:28:09.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f64d3c7fe81] 2024-03-24T00:28:09.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f64d269a1ca] 2024-03-24T00:28:09.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 19: clone() 2024-03-24T00:28:09.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:28:09.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71329]: 2024-03-24T00:28:09.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 podman[71367]: 2024-03-24 00:28:09.076568317 +0000 UTC m=+0.022936053 container died f7c5a3114702168b0961ad77c0a481671c9a32238c16300a76b68d150dbdef0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-24T00:28:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6832/93515314,v1:172.21.15.106:6833/93515314] up:rejoin 2024-03-24T00:28:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:rejoin,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-mon[36505]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-24T00:28:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:28:09.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:28:09.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6832/93515314,v1:172.21.15.106:6833/93515314] up:active 2024-03-24T00:28:09.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=h=up:active} 3 up:standby-replay 4 up:standby 2024-03-24T00:28:09.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:09 smithi106 ceph-mon[36505]: pgmap v232: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:28:09.684 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 218 sec latency 331275.156 ms 2024-03-24T00:28:09.989 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 podman[71367]: 2024-03-24 00:28:09.730857617 +0000 UTC m=+0.677225356 container remove f7c5a3114702168b0961ad77c0a481671c9a32238c16300a76b68d150dbdef0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-24T00:28:09.989 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:09 smithi106 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:28:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:09 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6832/93515314,v1:172.21.15.106:6833/93515314] up:rejoin 2024-03-24T00:28:10.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:09 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:rejoin,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:10.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:09 smithi078 ceph-mon[32429]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-24T00:28:10.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:09 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:28:10.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:09 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:28:10.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:09 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6832/93515314,v1:172.21.15.106:6833/93515314] up:active 2024-03-24T00:28:10.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:09 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=h=up:active} 3 up:standby-replay 4 up:standby 2024-03-24T00:28:10.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:09 smithi078 ceph-mon[32429]: pgmap v232: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:28:10.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:09 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6832/93515314,v1:172.21.15.106:6833/93515314] up:rejoin 2024-03-24T00:28:10.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:09 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:rejoin,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:10.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:09 smithi107 ceph-mon[36485]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-24T00:28:10.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:09 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:28:10.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:09 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:28:10.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:09 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6832/93515314,v1:172.21.15.106:6833/93515314] up:active 2024-03-24T00:28:10.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:09 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=h=up:active} 3 up:standby-replay 4 up:standby 2024-03-24T00:28:10.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:09 smithi107 ceph-mon[36485]: pgmap v232: 129 pgs: 129 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:28:10.402 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:10 smithi106 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.k.service: Failed with result 'exit-code'. 2024-03-24T00:28:10.684 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 219 sec latency 332275.242 ms 2024-03-24T00:28:11.684 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 220 sec latency 333275.363 ms 2024-03-24T00:28:12.684 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 221 sec latency 334275.466 ms 2024-03-24T00:28:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:12 smithi078 ceph-mon[32429]: pgmap v233: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:28:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:12 smithi107 ceph-mon[36485]: pgmap v233: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:28:12.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:12 smithi106 ceph-mon[36505]: pgmap v233: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:28:13.684 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 222 sec latency 335275.564 ms 2024-03-24T00:28:14.684 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 223 sec latency 336275.689 ms 2024-03-24T00:28:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:14 smithi078 ceph-mon[32429]: pgmap v234: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:28:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:14 smithi107 ceph-mon[36485]: pgmap v234: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:28:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:14 smithi106 ceph-mon[36505]: pgmap v234: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:28:15.684 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 224 sec latency 337275.764 ms 2024-03-24T00:28:16.685 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 225 sec latency 338275.861 ms 2024-03-24T00:28:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:16 smithi078 ceph-mon[32429]: pgmap v235: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:28:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:16 smithi107 ceph-mon[36485]: pgmap v235: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:28:16.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:16 smithi106 ceph-mon[36505]: pgmap v235: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:28:17.685 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 226 sec latency 339275.973 ms 2024-03-24T00:28:18.685 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 227 sec latency 340276.024 ms 2024-03-24T00:28:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:18 smithi078 ceph-mon[32429]: pgmap v236: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-24T00:28:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:18 smithi107 ceph-mon[36485]: pgmap v236: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-24T00:28:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:18 smithi106 ceph-mon[36505]: pgmap v236: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-24T00:28:19.685 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 228 sec latency 341276.074 ms 2024-03-24T00:28:20.436 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:20 smithi106 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.k.service: Scheduled restart job, restart counter is at 5. 2024-03-24T00:28:20.436 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:20 smithi106 systemd[1]: Stopped Ceph mds.k for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:28:20.436 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:20 smithi106 systemd[1]: Starting Ceph mds.k for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:28:20.437 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:20 smithi106 podman[71593]: 2024-03-24 00:28:20.287104549 +0000 UTC m=+0.020691702 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:28:20.437 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:20 smithi106 podman[71593]: 2024-03-24 00:28:20.391182049 +0000 UTC m=+0.124769199 container create 65573d87b095b93d761a5843ac4fec00033163ecc39678639baf1ef7f6063e25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-24T00:28:20.685 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 229 sec latency 342276.184 ms 2024-03-24T00:28:20.696 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:20 smithi106 podman[71593]: 2024-03-24 00:28:20.669508087 +0000 UTC m=+0.403095255 container init 65573d87b095b93d761a5843ac4fec00033163ecc39678639baf1ef7f6063e25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-24T00:28:20.696 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:20 smithi106 podman[71593]: 2024-03-24 00:28:20.672689626 +0000 UTC m=+0.406276779 container start 65573d87b095b93d761a5843ac4fec00033163ecc39678639baf1ef7f6063e25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k, GIT_CLEAN=True, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-24T00:28:20.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:20 smithi106 ceph-mon[36505]: pgmap v237: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T00:28:20.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:20 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:28:20.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:20 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:20.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:20 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:28:20.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:20 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:28:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:20 smithi078 ceph-mon[32429]: pgmap v237: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T00:28:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:20 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:28:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:20 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:20.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:20 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:28:20.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:20 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:28:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:20 smithi107 ceph-mon[36485]: pgmap v237: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T00:28:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:20 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:28:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:20 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:20 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:28:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:20 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:28:21.035 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:20 smithi106 ceph-mds[71611]: -- 172.21.15.106:0/3336747134 <== mon.2 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556d3983ba40 con 0x556d3980ec00 2024-03-24T00:28:21.035 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:20 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: starting mds.k at 2024-03-24T00:28:21.035 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:20 smithi106 bash[71593]: 65573d87b095b93d761a5843ac4fec00033163ecc39678639baf1ef7f6063e25 2024-03-24T00:28:21.035 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:20 smithi106 systemd[1]: Started Ceph mds.k for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:28:21.685 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 230 sec latency 343276.286 ms 2024-03-24T00:28:21.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:21 smithi078 ceph-mon[32429]: daemon mds.k restarted 2024-03-24T00:28:21.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:21 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:28:21.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:21 smithi078 ceph-mon[32429]: osdmap e114: 12 total, 12 up, 12 in 2024-03-24T00:28:21.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:21 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6832/2104934388,v1:172.21.15.106:6833/2104934388] up:boot 2024-03-24T00:28:21.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:21 smithi078 ceph-mon[32429]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-24T00:28:21.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:21 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-24T00:28:21.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:21 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T00:28:21.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:21 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:replay,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:21 smithi107 ceph-mon[36485]: daemon mds.k restarted 2024-03-24T00:28:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:21 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:28:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:21 smithi107 ceph-mon[36485]: osdmap e114: 12 total, 12 up, 12 in 2024-03-24T00:28:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:21 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6832/2104934388,v1:172.21.15.106:6833/2104934388] up:boot 2024-03-24T00:28:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:21 smithi107 ceph-mon[36485]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-24T00:28:21.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:21 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-24T00:28:21.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:21 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T00:28:21.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:21 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:replay,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:21.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:21 smithi106 ceph-mon[36505]: daemon mds.k restarted 2024-03-24T00:28:21.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:21 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:28:21.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:21 smithi106 ceph-mon[36505]: osdmap e114: 12 total, 12 up, 12 in 2024-03-24T00:28:21.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:21 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6832/2104934388,v1:172.21.15.106:6833/2104934388] up:boot 2024-03-24T00:28:21.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:21 smithi106 ceph-mon[36505]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-24T00:28:21.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:21 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-24T00:28:21.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:21 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T00:28:21.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:21 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:replay,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:22.685 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 231 sec latency 344276.403 ms 2024-03-24T00:28:22.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:22 smithi078 ceph-mon[32429]: pgmap v239: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:28:22.730 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:22 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6832/3928685127,v1:172.21.15.107:6833/3928685127] up:resolve 2024-03-24T00:28:22.730 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:22 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:resolve,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:22.745 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:22 smithi107 ceph-mon[36485]: pgmap v239: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:28:22.745 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:22 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6832/3928685127,v1:172.21.15.107:6833/3928685127] up:resolve 2024-03-24T00:28:22.745 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:22 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:resolve,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:22.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:22 smithi106 ceph-mon[36505]: pgmap v239: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:28:22.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:22 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6832/3928685127,v1:172.21.15.107:6833/3928685127] up:resolve 2024-03-24T00:28:22.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:22 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:resolve,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:23.685 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 232 sec latency 345276.517 ms 2024-03-24T00:28:24.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:23 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6832/3928685127,v1:172.21.15.107:6833/3928685127] up:reconnect 2024-03-24T00:28:24.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:23 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:reconnect,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:24.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:23 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.00399993 2024-03-24T00:28:24.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:23 smithi078 ceph-mon[32429]: pgmap v240: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:28:24.034 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:23 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6832/3928685127,v1:172.21.15.107:6833/3928685127] up:reconnect 2024-03-24T00:28:24.034 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:23 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:reconnect,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:24.034 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:23 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.00399993 2024-03-24T00:28:24.034 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:23 smithi107 ceph-mon[36485]: pgmap v240: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:28:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:23 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6832/3928685127,v1:172.21.15.107:6833/3928685127] up:reconnect 2024-03-24T00:28:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:23 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:reconnect,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:24.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:23 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.00399993 2024-03-24T00:28:24.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:23 smithi106 ceph-mon[36505]: pgmap v240: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:28:24.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f755c83c700 time 2024-03-24T00:28:24.033553+0000 2024-03-24T00:28:24.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:28:24.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:28:24.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7566e664a1] 2024-03-24T00:28:24.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7566e66667] 2024-03-24T00:28:24.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563401dfb223] 2024-03-24T00:28:24.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563401c755d6] 2024-03-24T00:28:24.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563401c75da1] 2024-03-24T00:28:24.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x563401c782d5] 2024-03-24T00:28:24.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563401cc8233] 2024-03-24T00:28:24.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563401cca197] 2024-03-24T00:28:24.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563401bc911b] 2024-03-24T00:28:24.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563401bc96bc] 2024-03-24T00:28:24.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563401bb31ff] 2024-03-24T00:28:24.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f75670f8488] 2024-03-24T00:28:24.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 13: (DispatchQueue::entry()+0x51f) [0x7f75670f560f] 2024-03-24T00:28:24.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f75671c8e81] 2024-03-24T00:28:24.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7565be31ca] 2024-03-24T00:28:24.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 16: clone() 2024-03-24T00:28:24.346 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 2024-03-24T00:28:24.034+0000 7f755c83c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f755c83c700 time 2024-03-24T00:28:24.033553+0000 2024-03-24T00:28:24.346 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:28:24.346 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 2024-03-24T00:28:24.346 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:28:24.346 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7566e664a1] 2024-03-24T00:28:24.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7566e66667] 2024-03-24T00:28:24.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563401dfb223] 2024-03-24T00:28:24.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563401c755d6] 2024-03-24T00:28:24.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563401c75da1] 2024-03-24T00:28:24.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x563401c782d5] 2024-03-24T00:28:24.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563401cc8233] 2024-03-24T00:28:24.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563401cca197] 2024-03-24T00:28:24.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563401bc911b] 2024-03-24T00:28:24.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563401bc96bc] 2024-03-24T00:28:24.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563401bb31ff] 2024-03-24T00:28:24.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f75670f8488] 2024-03-24T00:28:24.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 13: (DispatchQueue::entry()+0x51f) [0x7f75670f560f] 2024-03-24T00:28:24.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f75671c8e81] 2024-03-24T00:28:24.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7565be31ca] 2024-03-24T00:28:24.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 16: clone() 2024-03-24T00:28:24.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 2024-03-24T00:28:24.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: *** Caught signal (Aborted) ** 2024-03-24T00:28:24.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: in thread 7f755c83c700 thread_name:ms_dispatch 2024-03-24T00:28:24.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:28:24.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7565bedd20] 2024-03-24T00:28:24.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 2: gsignal() 2024-03-24T00:28:24.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 3: abort() 2024-03-24T00:28:24.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7566e664fb] 2024-03-24T00:28:24.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7566e66667] 2024-03-24T00:28:24.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563401dfb223] 2024-03-24T00:28:24.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563401c755d6] 2024-03-24T00:28:24.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563401c75da1] 2024-03-24T00:28:24.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x563401c782d5] 2024-03-24T00:28:24.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563401cc8233] 2024-03-24T00:28:24.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563401cca197] 2024-03-24T00:28:24.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563401bc911b] 2024-03-24T00:28:24.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563401bc96bc] 2024-03-24T00:28:24.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563401bb31ff] 2024-03-24T00:28:24.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f75670f8488] 2024-03-24T00:28:24.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 16: (DispatchQueue::entry()+0x51f) [0x7f75670f560f] 2024-03-24T00:28:24.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f75671c8e81] 2024-03-24T00:28:24.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7565be31ca] 2024-03-24T00:28:24.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 19: clone() 2024-03-24T00:28:24.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 2024-03-24T00:28:24.036+0000 7f755c83c700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:28:24.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: in thread 7f755c83c700 thread_name:ms_dispatch 2024-03-24T00:28:24.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 2024-03-24T00:28:24.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:28:24.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7565bedd20] 2024-03-24T00:28:24.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 2: gsignal() 2024-03-24T00:28:24.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 3: abort() 2024-03-24T00:28:24.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7566e664fb] 2024-03-24T00:28:24.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7566e66667] 2024-03-24T00:28:24.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563401dfb223] 2024-03-24T00:28:24.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563401c755d6] 2024-03-24T00:28:24.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563401c75da1] 2024-03-24T00:28:24.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x563401c782d5] 2024-03-24T00:28:24.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563401cc8233] 2024-03-24T00:28:24.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563401cca197] 2024-03-24T00:28:24.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563401bc911b] 2024-03-24T00:28:24.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563401bc96bc] 2024-03-24T00:28:24.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563401bb31ff] 2024-03-24T00:28:24.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f75670f8488] 2024-03-24T00:28:24.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 16: (DispatchQueue::entry()+0x51f) [0x7f75670f560f] 2024-03-24T00:28:24.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f75671c8e81] 2024-03-24T00:28:24.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7565be31ca] 2024-03-24T00:28:24.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 19: clone() 2024-03-24T00:28:24.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:28:24.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 2024-03-24T00:28:24.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: -1> 2024-03-24T00:28:24.034+0000 7f755c83c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f755c83c700 time 2024-03-24T00:28:24.033553+0000 2024-03-24T00:28:24.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:28:24.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 2024-03-24T00:28:24.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:28:24.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7566e664a1] 2024-03-24T00:28:24.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7566e66667] 2024-03-24T00:28:24.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563401dfb223] 2024-03-24T00:28:24.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563401c755d6] 2024-03-24T00:28:24.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563401c75da1] 2024-03-24T00:28:24.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x563401c782d5] 2024-03-24T00:28:24.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563401cc8233] 2024-03-24T00:28:24.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563401cca197] 2024-03-24T00:28:24.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563401bc911b] 2024-03-24T00:28:24.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563401bc96bc] 2024-03-24T00:28:24.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563401bb31ff] 2024-03-24T00:28:24.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f75670f8488] 2024-03-24T00:28:24.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 13: (DispatchQueue::entry()+0x51f) [0x7f75670f560f] 2024-03-24T00:28:24.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f75671c8e81] 2024-03-24T00:28:24.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7565be31ca] 2024-03-24T00:28:24.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 16: clone() 2024-03-24T00:28:24.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 2024-03-24T00:28:24.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 0> 2024-03-24T00:28:24.036+0000 7f755c83c700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:28:24.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: in thread 7f755c83c700 thread_name:ms_dispatch 2024-03-24T00:28:24.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 2024-03-24T00:28:24.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:28:24.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7565bedd20] 2024-03-24T00:28:24.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 2: gsignal() 2024-03-24T00:28:24.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 3: abort() 2024-03-24T00:28:24.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7566e664fb] 2024-03-24T00:28:24.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7566e66667] 2024-03-24T00:28:24.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563401dfb223] 2024-03-24T00:28:24.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563401c755d6] 2024-03-24T00:28:24.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563401c75da1] 2024-03-24T00:28:24.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x563401c782d5] 2024-03-24T00:28:24.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563401cc8233] 2024-03-24T00:28:24.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563401cca197] 2024-03-24T00:28:24.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563401bc911b] 2024-03-24T00:28:24.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563401bc96bc] 2024-03-24T00:28:24.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563401bb31ff] 2024-03-24T00:28:24.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f75670f8488] 2024-03-24T00:28:24.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 16: (DispatchQueue::entry()+0x51f) [0x7f75670f560f] 2024-03-24T00:28:24.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f75671c8e81] 2024-03-24T00:28:24.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7565be31ca] 2024-03-24T00:28:24.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 19: clone() 2024-03-24T00:28:24.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:28:24.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 2024-03-24T00:28:24.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: -1> 2024-03-24T00:28:24.034+0000 7f755c83c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f755c83c700 time 2024-03-24T00:28:24.033553+0000 2024-03-24T00:28:24.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:28:24.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 2024-03-24T00:28:24.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:28:24.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7566e664a1] 2024-03-24T00:28:24.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7566e66667] 2024-03-24T00:28:24.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563401dfb223] 2024-03-24T00:28:24.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563401c755d6] 2024-03-24T00:28:24.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563401c75da1] 2024-03-24T00:28:24.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x563401c782d5] 2024-03-24T00:28:24.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563401cc8233] 2024-03-24T00:28:24.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563401cca197] 2024-03-24T00:28:24.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563401bc911b] 2024-03-24T00:28:24.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563401bc96bc] 2024-03-24T00:28:24.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563401bb31ff] 2024-03-24T00:28:24.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f75670f8488] 2024-03-24T00:28:24.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 13: (DispatchQueue::entry()+0x51f) [0x7f75670f560f] 2024-03-24T00:28:24.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f75671c8e81] 2024-03-24T00:28:24.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7565be31ca] 2024-03-24T00:28:24.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 16: clone() 2024-03-24T00:28:24.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 2024-03-24T00:28:24.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 0> 2024-03-24T00:28:24.036+0000 7f755c83c700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:28:24.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: in thread 7f755c83c700 thread_name:ms_dispatch 2024-03-24T00:28:24.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 2024-03-24T00:28:24.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:28:24.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7565bedd20] 2024-03-24T00:28:24.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 2: gsignal() 2024-03-24T00:28:24.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 3: abort() 2024-03-24T00:28:24.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7566e664fb] 2024-03-24T00:28:24.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7566e66667] 2024-03-24T00:28:24.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563401dfb223] 2024-03-24T00:28:24.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563401c755d6] 2024-03-24T00:28:24.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563401c75da1] 2024-03-24T00:28:24.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x563401c782d5] 2024-03-24T00:28:24.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563401cc8233] 2024-03-24T00:28:24.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563401cca197] 2024-03-24T00:28:24.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563401bc911b] 2024-03-24T00:28:24.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563401bc96bc] 2024-03-24T00:28:24.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563401bb31ff] 2024-03-24T00:28:24.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f75670f8488] 2024-03-24T00:28:24.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 16: (DispatchQueue::entry()+0x51f) [0x7f75670f560f] 2024-03-24T00:28:24.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f75671c8e81] 2024-03-24T00:28:24.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7565be31ca] 2024-03-24T00:28:24.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 19: clone() 2024-03-24T00:28:24.359 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:28:24.359 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73291]: 2024-03-24T00:28:24.359 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 podman[73464]: 2024-03-24 00:28:24.087783767 +0000 UTC m=+0.020673598 container died 028ebe198cbdcfa7f04ae4a85494544f94b60111d06138fb734d03d913fa7d00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-24T00:28:24.685 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 233 sec latency 346276.638 ms 2024-03-24T00:28:25.087 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 podman[73464]: 2024-03-24 00:28:24.864914114 +0000 UTC m=+0.797803926 container remove 028ebe198cbdcfa7f04ae4a85494544f94b60111d06138fb734d03d913fa7d00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2) 2024-03-24T00:28:25.087 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:24 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:28:25.088 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6832/3928685127,v1:172.21.15.107:6833/3928685127] up:rejoin 2024-03-24T00:28:25.088 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:rejoin,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:25.088 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-mon[36485]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-24T00:28:25.088 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:28:25.088 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:28:25.088 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6832/3928685127,v1:172.21.15.107:6833/3928685127] up:active 2024-03-24T00:28:25.088 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:24 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=h=up:active} 3 up:standby-replay 4 up:standby 2024-03-24T00:28:25.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:24 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6832/3928685127,v1:172.21.15.107:6833/3928685127] up:rejoin 2024-03-24T00:28:25.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:24 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:rejoin,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:25.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:24 smithi078 ceph-mon[32429]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-24T00:28:25.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:24 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:28:25.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:24 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:28:25.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:24 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6832/3928685127,v1:172.21.15.107:6833/3928685127] up:active 2024-03-24T00:28:25.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:24 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=h=up:active} 3 up:standby-replay 4 up:standby 2024-03-24T00:28:25.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:24 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6832/3928685127,v1:172.21.15.107:6833/3928685127] up:rejoin 2024-03-24T00:28:25.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:24 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:rejoin,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:25.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:24 smithi106 ceph-mon[36505]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-24T00:28:25.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:24 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:28:25.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:24 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:28:25.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:24 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6832/3928685127,v1:172.21.15.107:6833/3928685127] up:active 2024-03-24T00:28:25.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:24 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=h=up:active} 3 up:standby-replay 4 up:standby 2024-03-24T00:28:25.593 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:25 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.l.service: Failed with result 'exit-code'. 2024-03-24T00:28:25.686 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 234 sec latency 347276.698 ms 2024-03-24T00:28:26.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:25 smithi078 ceph-mon[32429]: pgmap v241: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:28:26.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:25 smithi107 ceph-mon[36485]: pgmap v241: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:28:26.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:25 smithi106 ceph-mon[36505]: pgmap v241: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:28:26.686 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 235 sec latency 348276.787 ms 2024-03-24T00:28:27.686 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 236 sec latency 349276.875 ms 2024-03-24T00:28:28.686 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 237 sec latency 350276.968 ms 2024-03-24T00:28:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:28 smithi078 ceph-mon[32429]: pgmap v242: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:28:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:28 smithi107 ceph-mon[36485]: pgmap v242: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:28:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:28 smithi106 ceph-mon[36505]: pgmap v242: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:28:29.686 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 238 sec latency 351277.006 ms 2024-03-24T00:28:30.686 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 239 sec latency 352277.086 ms 2024-03-24T00:28:30.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:30 smithi078 ceph-mon[32429]: pgmap v243: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:28:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:30 smithi107 ceph-mon[36485]: pgmap v243: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:28:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:30 smithi106 ceph-mon[36505]: pgmap v243: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:28:31.686 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 240 sec latency 353277.143 ms 2024-03-24T00:28:32.686 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 241 sec latency 354277.244 ms 2024-03-24T00:28:32.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:32 smithi078 ceph-mon[32429]: pgmap v244: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 25 KiB/s wr, 13 op/s 2024-03-24T00:28:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:32 smithi107 ceph-mon[36485]: pgmap v244: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 25 KiB/s wr, 13 op/s 2024-03-24T00:28:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:32 smithi106 ceph-mon[36505]: pgmap v244: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 25 KiB/s wr, 13 op/s 2024-03-24T00:28:33.686 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 242 sec latency 355277.387 ms 2024-03-24T00:28:34.686 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 243 sec latency 356277.476 ms 2024-03-24T00:28:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:34 smithi078 ceph-mon[32429]: pgmap v245: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 11 op/s 2024-03-24T00:28:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:34 smithi078 ceph-mon[32429]: osdmap e115: 12 total, 12 up, 12 in 2024-03-24T00:28:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:34 smithi107 ceph-mon[36485]: pgmap v245: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 11 op/s 2024-03-24T00:28:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:34 smithi107 ceph-mon[36485]: osdmap e115: 12 total, 12 up, 12 in 2024-03-24T00:28:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:34 smithi106 ceph-mon[36505]: pgmap v245: 129 pgs: 129 active+clean; 27 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 11 op/s 2024-03-24T00:28:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:34 smithi106 ceph-mon[36505]: osdmap e115: 12 total, 12 up, 12 in 2024-03-24T00:28:35.597 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:35 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.l.service: Scheduled restart job, restart counter is at 4. 2024-03-24T00:28:35.598 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:35 smithi107 systemd[1]: Stopped Ceph mds.l for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:28:35.598 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:35 smithi107 systemd[1]: Starting Ceph mds.l for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:28:35.598 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:35 smithi107 podman[73566]: 2024-03-24 00:28:35.480804534 +0000 UTC m=+0.017883135 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:28:35.598 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:35 smithi107 podman[73566]: 2024-03-24 00:28:35.598183003 +0000 UTC m=+0.135261583 container create 714954b2afc0f79bb23e2cbad0c1785e783f9065c0549d4efca6bfab2b961ab8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-24T00:28:35.686 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 244 sec latency 357277.575 ms 2024-03-24T00:28:36.211 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:35 smithi107 podman[73566]: 2024-03-24 00:28:35.850687183 +0000 UTC m=+0.387765776 container init 714954b2afc0f79bb23e2cbad0c1785e783f9065c0549d4efca6bfab2b961ab8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:28:36.211 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:35 smithi107 podman[73566]: 2024-03-24 00:28:35.853360989 +0000 UTC m=+0.390439587 container start 714954b2afc0f79bb23e2cbad0c1785e783f9065c0549d4efca6bfab2b961ab8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-24T00:28:36.211 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:35 smithi107 ceph-mds[73585]: -- 172.21.15.107:0/2876210211 <== mon.1 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56116d333a40 con 0x56116d307000 2024-03-24T00:28:36.211 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:35 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: starting mds.l at 2024-03-24T00:28:36.211 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:35 smithi107 bash[73566]: 714954b2afc0f79bb23e2cbad0c1785e783f9065c0549d4efca6bfab2b961ab8 2024-03-24T00:28:36.211 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:35 smithi107 systemd[1]: Started Ceph mds.l for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:28:36.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:36 smithi107 ceph-mon[36485]: pgmap v247: 129 pgs: 129 active+clean; 27 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:28:36.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:36 smithi107 ceph-mon[36485]: daemon mds.l restarted 2024-03-24T00:28:36.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:36 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:28:36.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:36 smithi107 ceph-mon[36485]: osdmap e116: 12 total, 12 up, 12 in 2024-03-24T00:28:36.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:36 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6832/4067467714,v1:172.21.15.107:6833/4067467714] up:boot 2024-03-24T00:28:36.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:36 smithi107 ceph-mon[36485]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-03-24T00:28:36.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:36 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-24T00:28:36.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:28:36.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:36 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:replay,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:36.687 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 245 sec latency 358277.715 ms 2024-03-24T00:28:36.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:36 smithi078 ceph-mon[32429]: pgmap v247: 129 pgs: 129 active+clean; 27 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:28:36.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:36 smithi078 ceph-mon[32429]: daemon mds.l restarted 2024-03-24T00:28:36.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:36 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:28:36.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:36 smithi078 ceph-mon[32429]: osdmap e116: 12 total, 12 up, 12 in 2024-03-24T00:28:36.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:36 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6832/4067467714,v1:172.21.15.107:6833/4067467714] up:boot 2024-03-24T00:28:36.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:36 smithi078 ceph-mon[32429]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-03-24T00:28:36.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:36 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-24T00:28:36.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:28:36.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:36 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:replay,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:36.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:36 smithi106 ceph-mon[36505]: pgmap v247: 129 pgs: 129 active+clean; 27 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:28:36.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:36 smithi106 ceph-mon[36505]: daemon mds.l restarted 2024-03-24T00:28:36.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:36 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:28:36.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:36 smithi106 ceph-mon[36505]: osdmap e116: 12 total, 12 up, 12 in 2024-03-24T00:28:36.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:36 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6832/4067467714,v1:172.21.15.107:6833/4067467714] up:boot 2024-03-24T00:28:36.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:36 smithi106 ceph-mon[36505]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-03-24T00:28:36.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:36 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-03-24T00:28:36.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:28:36.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:36 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:replay,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:37.686 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 246 sec latency 359277.808 ms 2024-03-24T00:28:38.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:37 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6832/2104934388,v1:172.21.15.106:6833/2104934388] up:resolve 2024-03-24T00:28:38.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:37 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:resolve,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:38.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:37 smithi078 ceph-mon[32429]: pgmap v249: 129 pgs: 129 active+clean; 27 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-24T00:28:38.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:37 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6832/2104934388,v1:172.21.15.106:6833/2104934388] up:resolve 2024-03-24T00:28:38.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:37 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:resolve,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:38.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:37 smithi107 ceph-mon[36485]: pgmap v249: 129 pgs: 129 active+clean; 27 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-24T00:28:38.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:37 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6832/2104934388,v1:172.21.15.106:6833/2104934388] up:resolve 2024-03-24T00:28:38.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:37 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:resolve,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:38.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:37 smithi106 ceph-mon[36505]: pgmap v249: 129 pgs: 129 active+clean; 27 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-24T00:28:38.687 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 247 sec latency 360277.915 ms 2024-03-24T00:28:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:38 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999978 2024-03-24T00:28:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:38 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6832/2104934388,v1:172.21.15.106:6833/2104934388] up:reconnect 2024-03-24T00:28:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:38 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:reconnect,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:38 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999978 2024-03-24T00:28:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:38 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6832/2104934388,v1:172.21.15.106:6833/2104934388] up:reconnect 2024-03-24T00:28:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:38 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:reconnect,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:38 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999978 2024-03-24T00:28:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:38 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6832/2104934388,v1:172.21.15.106:6833/2104934388] up:reconnect 2024-03-24T00:28:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:38 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:reconnect,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:39.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f134dba5700 time 2024-03-24T00:28:39.052079+0000 2024-03-24T00:28:39.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:28:39.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:28:39.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f13581cf4a1] 2024-03-24T00:28:39.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f13581cf667] 2024-03-24T00:28:39.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556d37c30223] 2024-03-24T00:28:39.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556d37aaa5d6] 2024-03-24T00:28:39.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556d37aaada1] 2024-03-24T00:28:39.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556d37aad2d5] 2024-03-24T00:28:39.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556d37afd233] 2024-03-24T00:28:39.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556d37aff197] 2024-03-24T00:28:39.403 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556d379fe11b] 2024-03-24T00:28:39.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556d379fe6bc] 2024-03-24T00:28:39.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556d379e81ff] 2024-03-24T00:28:39.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1358461488] 2024-03-24T00:28:39.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 13: (DispatchQueue::entry()+0x51f) [0x7f135845e60f] 2024-03-24T00:28:39.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1358531e81] 2024-03-24T00:28:39.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1356f4c1ca] 2024-03-24T00:28:39.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 16: clone() 2024-03-24T00:28:39.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 2024-03-24T00:28:39.052+0000 7f134dba5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f134dba5700 time 2024-03-24T00:28:39.052079+0000 2024-03-24T00:28:39.404 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:28:39.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 2024-03-24T00:28:39.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:28:39.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f13581cf4a1] 2024-03-24T00:28:39.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f13581cf667] 2024-03-24T00:28:39.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556d37c30223] 2024-03-24T00:28:39.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556d37aaa5d6] 2024-03-24T00:28:39.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556d37aaada1] 2024-03-24T00:28:39.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556d37aad2d5] 2024-03-24T00:28:39.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556d37afd233] 2024-03-24T00:28:39.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556d37aff197] 2024-03-24T00:28:39.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556d379fe11b] 2024-03-24T00:28:39.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556d379fe6bc] 2024-03-24T00:28:39.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556d379e81ff] 2024-03-24T00:28:39.405 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1358461488] 2024-03-24T00:28:39.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 13: (DispatchQueue::entry()+0x51f) [0x7f135845e60f] 2024-03-24T00:28:39.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1358531e81] 2024-03-24T00:28:39.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1356f4c1ca] 2024-03-24T00:28:39.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 16: clone() 2024-03-24T00:28:39.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 2024-03-24T00:28:39.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: *** Caught signal (Aborted) ** 2024-03-24T00:28:39.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: in thread 7f134dba5700 thread_name:ms_dispatch 2024-03-24T00:28:39.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:28:39.406 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1356f56d20] 2024-03-24T00:28:39.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 2: gsignal() 2024-03-24T00:28:39.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 3: abort() 2024-03-24T00:28:39.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f13581cf4fb] 2024-03-24T00:28:39.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f13581cf667] 2024-03-24T00:28:39.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556d37c30223] 2024-03-24T00:28:39.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556d37aaa5d6] 2024-03-24T00:28:39.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556d37aaada1] 2024-03-24T00:28:39.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556d37aad2d5] 2024-03-24T00:28:39.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556d37afd233] 2024-03-24T00:28:39.407 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556d37aff197] 2024-03-24T00:28:39.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556d379fe11b] 2024-03-24T00:28:39.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556d379fe6bc] 2024-03-24T00:28:39.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556d379e81ff] 2024-03-24T00:28:39.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1358461488] 2024-03-24T00:28:39.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 16: (DispatchQueue::entry()+0x51f) [0x7f135845e60f] 2024-03-24T00:28:39.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1358531e81] 2024-03-24T00:28:39.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1356f4c1ca] 2024-03-24T00:28:39.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 19: clone() 2024-03-24T00:28:39.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 2024-03-24T00:28:39.054+0000 7f134dba5700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:28:39.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: in thread 7f134dba5700 thread_name:ms_dispatch 2024-03-24T00:28:39.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 2024-03-24T00:28:39.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:28:39.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1356f56d20] 2024-03-24T00:28:39.408 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 2: gsignal() 2024-03-24T00:28:39.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 3: abort() 2024-03-24T00:28:39.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f13581cf4fb] 2024-03-24T00:28:39.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f13581cf667] 2024-03-24T00:28:39.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556d37c30223] 2024-03-24T00:28:39.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556d37aaa5d6] 2024-03-24T00:28:39.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556d37aaada1] 2024-03-24T00:28:39.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556d37aad2d5] 2024-03-24T00:28:39.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556d37afd233] 2024-03-24T00:28:39.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556d37aff197] 2024-03-24T00:28:39.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556d379fe11b] 2024-03-24T00:28:39.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556d379fe6bc] 2024-03-24T00:28:39.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556d379e81ff] 2024-03-24T00:28:39.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1358461488] 2024-03-24T00:28:39.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 16: (DispatchQueue::entry()+0x51f) [0x7f135845e60f] 2024-03-24T00:28:39.409 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1358531e81] 2024-03-24T00:28:39.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1356f4c1ca] 2024-03-24T00:28:39.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 19: clone() 2024-03-24T00:28:39.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:28:39.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 2024-03-24T00:28:39.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: -1> 2024-03-24T00:28:39.052+0000 7f134dba5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f134dba5700 time 2024-03-24T00:28:39.052079+0000 2024-03-24T00:28:39.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:28:39.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 2024-03-24T00:28:39.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:28:39.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f13581cf4a1] 2024-03-24T00:28:39.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f13581cf667] 2024-03-24T00:28:39.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556d37c30223] 2024-03-24T00:28:39.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556d37aaa5d6] 2024-03-24T00:28:39.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556d37aaada1] 2024-03-24T00:28:39.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556d37aad2d5] 2024-03-24T00:28:39.410 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556d37afd233] 2024-03-24T00:28:39.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556d37aff197] 2024-03-24T00:28:39.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556d379fe11b] 2024-03-24T00:28:39.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556d379fe6bc] 2024-03-24T00:28:39.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556d379e81ff] 2024-03-24T00:28:39.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1358461488] 2024-03-24T00:28:39.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 13: (DispatchQueue::entry()+0x51f) [0x7f135845e60f] 2024-03-24T00:28:39.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1358531e81] 2024-03-24T00:28:39.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1356f4c1ca] 2024-03-24T00:28:39.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 16: clone() 2024-03-24T00:28:39.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 2024-03-24T00:28:39.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 0> 2024-03-24T00:28:39.054+0000 7f134dba5700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:28:39.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: in thread 7f134dba5700 thread_name:ms_dispatch 2024-03-24T00:28:39.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 2024-03-24T00:28:39.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:28:39.411 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1356f56d20] 2024-03-24T00:28:39.412 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 2: gsignal() 2024-03-24T00:28:39.412 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 3: abort() 2024-03-24T00:28:39.412 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f13581cf4fb] 2024-03-24T00:28:39.412 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f13581cf667] 2024-03-24T00:28:39.412 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556d37c30223] 2024-03-24T00:28:39.412 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556d37aaa5d6] 2024-03-24T00:28:39.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556d37aaada1] 2024-03-24T00:28:39.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556d37aad2d5] 2024-03-24T00:28:39.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556d37afd233] 2024-03-24T00:28:39.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556d37aff197] 2024-03-24T00:28:39.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556d379fe11b] 2024-03-24T00:28:39.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556d379fe6bc] 2024-03-24T00:28:39.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556d379e81ff] 2024-03-24T00:28:39.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1358461488] 2024-03-24T00:28:39.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 16: (DispatchQueue::entry()+0x51f) [0x7f135845e60f] 2024-03-24T00:28:39.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1358531e81] 2024-03-24T00:28:39.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1356f4c1ca] 2024-03-24T00:28:39.413 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 19: clone() 2024-03-24T00:28:39.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:28:39.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 2024-03-24T00:28:39.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: -1> 2024-03-24T00:28:39.052+0000 7f134dba5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f134dba5700 time 2024-03-24T00:28:39.052079+0000 2024-03-24T00:28:39.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:28:39.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 2024-03-24T00:28:39.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:28:39.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f13581cf4a1] 2024-03-24T00:28:39.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f13581cf667] 2024-03-24T00:28:39.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556d37c30223] 2024-03-24T00:28:39.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556d37aaa5d6] 2024-03-24T00:28:39.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556d37aaada1] 2024-03-24T00:28:39.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556d37aad2d5] 2024-03-24T00:28:39.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556d37afd233] 2024-03-24T00:28:39.414 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556d37aff197] 2024-03-24T00:28:39.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556d379fe11b] 2024-03-24T00:28:39.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556d379fe6bc] 2024-03-24T00:28:39.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556d379e81ff] 2024-03-24T00:28:39.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1358461488] 2024-03-24T00:28:39.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 13: (DispatchQueue::entry()+0x51f) [0x7f135845e60f] 2024-03-24T00:28:39.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1358531e81] 2024-03-24T00:28:39.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1356f4c1ca] 2024-03-24T00:28:39.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 16: clone() 2024-03-24T00:28:39.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 2024-03-24T00:28:39.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 0> 2024-03-24T00:28:39.054+0000 7f134dba5700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:28:39.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: in thread 7f134dba5700 thread_name:ms_dispatch 2024-03-24T00:28:39.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 2024-03-24T00:28:39.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:28:39.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1356f56d20] 2024-03-24T00:28:39.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 2: gsignal() 2024-03-24T00:28:39.415 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 3: abort() 2024-03-24T00:28:39.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f13581cf4fb] 2024-03-24T00:28:39.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f13581cf667] 2024-03-24T00:28:39.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556d37c30223] 2024-03-24T00:28:39.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556d37aaa5d6] 2024-03-24T00:28:39.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556d37aaada1] 2024-03-24T00:28:39.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556d37aad2d5] 2024-03-24T00:28:39.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556d37afd233] 2024-03-24T00:28:39.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556d37aff197] 2024-03-24T00:28:39.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556d379fe11b] 2024-03-24T00:28:39.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556d379fe6bc] 2024-03-24T00:28:39.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556d379e81ff] 2024-03-24T00:28:39.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1358461488] 2024-03-24T00:28:39.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 16: (DispatchQueue::entry()+0x51f) [0x7f135845e60f] 2024-03-24T00:28:39.416 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1358531e81] 2024-03-24T00:28:39.417 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1356f4c1ca] 2024-03-24T00:28:39.417 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 19: clone() 2024-03-24T00:28:39.417 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:28:39.417 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k[71607]: 2024-03-24T00:28:39.417 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 podman[71657]: 2024-03-24 00:28:39.110698138 +0000 UTC m=+0.020021124 container died 65573d87b095b93d761a5843ac4fec00033163ecc39678639baf1ef7f6063e25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:28:39.687 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 248 sec latency 361278.030 ms 2024-03-24T00:28:40.000 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 podman[71657]: 2024-03-24 00:28:39.749950477 +0000 UTC m=+0.659273462 container remove 65573d87b095b93d761a5843ac4fec00033163ecc39678639baf1ef7f6063e25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-k, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:28:40.000 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:39 smithi106 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:28:40.000 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6832/2104934388,v1:172.21.15.106:6833/2104934388] up:rejoin 2024-03-24T00:28:40.001 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:rejoin,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:40.001 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-mon[36505]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-24T00:28:40.001 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:28:40.001 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:28:40.001 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6832/2104934388,v1:172.21.15.106:6833/2104934388] up:active 2024-03-24T00:28:40.001 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=h=up:active} 3 up:standby-replay 4 up:standby 2024-03-24T00:28:40.001 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-mon[36505]: osdmap e117: 12 total, 12 up, 12 in 2024-03-24T00:28:40.001 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:39 smithi106 ceph-mon[36505]: pgmap v251: 129 pgs: 129 active+clean; 27 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-24T00:28:40.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:39 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6832/2104934388,v1:172.21.15.106:6833/2104934388] up:rejoin 2024-03-24T00:28:40.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:39 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:rejoin,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:40.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:39 smithi078 ceph-mon[32429]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-24T00:28:40.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:39 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:28:40.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:39 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:28:40.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:39 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6832/2104934388,v1:172.21.15.106:6833/2104934388] up:active 2024-03-24T00:28:40.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:39 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=h=up:active} 3 up:standby-replay 4 up:standby 2024-03-24T00:28:40.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:39 smithi078 ceph-mon[32429]: osdmap e117: 12 total, 12 up, 12 in 2024-03-24T00:28:40.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:39 smithi078 ceph-mon[32429]: pgmap v251: 129 pgs: 129 active+clean; 27 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-24T00:28:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:39 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6832/2104934388,v1:172.21.15.106:6833/2104934388] up:rejoin 2024-03-24T00:28:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:39 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=k=up:rejoin,2=h=up:active} 2 up:standby-replay 5 up:standby 2024-03-24T00:28:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:39 smithi107 ceph-mon[36485]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-03-24T00:28:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:39 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:28:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:39 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:28:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:39 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6832/2104934388,v1:172.21.15.106:6833/2104934388] up:active 2024-03-24T00:28:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:39 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=k=up:active,2=h=up:active} 3 up:standby-replay 4 up:standby 2024-03-24T00:28:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:39 smithi107 ceph-mon[36485]: osdmap e117: 12 total, 12 up, 12 in 2024-03-24T00:28:40.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:39 smithi107 ceph-mon[36485]: pgmap v251: 129 pgs: 129 active+clean; 27 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-24T00:28:40.402 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:40 smithi106 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.k.service: Failed with result 'exit-code'. 2024-03-24T00:28:40.687 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 249 sec latency 362278.146 ms 2024-03-24T00:28:41.687 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 250 sec latency 363278.237 ms 2024-03-24T00:28:42.689 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 251 sec latency 364278.329 ms 2024-03-24T00:28:42.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:42 smithi078 ceph-mon[32429]: pgmap v252: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 516 B/s wr, 13 op/s 2024-03-24T00:28:42.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:42 smithi107 ceph-mon[36485]: pgmap v252: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 516 B/s wr, 13 op/s 2024-03-24T00:28:42.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:42 smithi106 ceph-mon[36505]: pgmap v252: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 516 B/s wr, 13 op/s 2024-03-24T00:28:43.687 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 252 sec latency 365278.416 ms 2024-03-24T00:28:44.687 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 253 sec latency 366278.520 ms 2024-03-24T00:28:44.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:44 smithi078 ceph-mon[32429]: pgmap v253: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 383 B/s wr, 9 op/s 2024-03-24T00:28:44.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:44 smithi107 ceph-mon[36485]: pgmap v253: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 383 B/s wr, 9 op/s 2024-03-24T00:28:44.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:44 smithi106 ceph-mon[36505]: pgmap v253: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 383 B/s wr, 9 op/s 2024-03-24T00:28:45.687 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 254 sec latency 367278.652 ms 2024-03-24T00:28:46.687 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 255 sec latency 368278.724 ms 2024-03-24T00:28:46.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:46 smithi078 ceph-mon[32429]: pgmap v254: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 324 B/s wr, 11 op/s 2024-03-24T00:28:46.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:46 smithi107 ceph-mon[36485]: pgmap v254: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 324 B/s wr, 11 op/s 2024-03-24T00:28:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:46 smithi106 ceph-mon[36505]: pgmap v254: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 324 B/s wr, 11 op/s 2024-03-24T00:28:47.688 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 256 sec latency 369278.799 ms 2024-03-24T00:28:48.688 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 257 sec latency 370278.899 ms 2024-03-24T00:28:48.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:48 smithi078 ceph-mon[32429]: pgmap v255: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-24T00:28:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:48 smithi107 ceph-mon[36485]: pgmap v255: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-24T00:28:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:48 smithi106 ceph-mon[36505]: pgmap v255: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-24T00:28:49.688 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 258 sec latency 371278.945 ms 2024-03-24T00:28:50.435 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:50 smithi106 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.k.service: Scheduled restart job, restart counter is at 6. 2024-03-24T00:28:50.435 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:50 smithi106 systemd[1]: Stopped Ceph mds.k for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:28:50.435 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:50 smithi106 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.k.service: Start request repeated too quickly. 2024-03-24T00:28:50.435 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:50 smithi106 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.k.service: Failed with result 'exit-code'. 2024-03-24T00:28:50.435 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 24 00:28:50 smithi106 systemd[1]: Failed to start Ceph mds.k for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:28:50.688 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 259 sec latency 372279.042 ms 2024-03-24T00:28:50.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:50 smithi078 ceph-mon[32429]: pgmap v256: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 297 B/s wr, 10 op/s 2024-03-24T00:28:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:50 smithi107 ceph-mon[36485]: pgmap v256: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 297 B/s wr, 10 op/s 2024-03-24T00:28:50.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:50 smithi106 ceph-mon[36505]: pgmap v256: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 297 B/s wr, 10 op/s 2024-03-24T00:28:51.688 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 260 sec latency 373279.102 ms 2024-03-24T00:28:52.688 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 261 sec latency 374279.217 ms 2024-03-24T00:28:52.706 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:52 smithi106 ceph-mon[36505]: pgmap v257: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T00:28:52.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:52 smithi078 ceph-mon[32429]: pgmap v257: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T00:28:52.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:52 smithi107 ceph-mon[36485]: pgmap v257: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T00:28:53.688 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 262 sec latency 375279.305 ms 2024-03-24T00:28:54.688 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 263 sec latency 376279.392 ms 2024-03-24T00:28:54.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:54 smithi078 ceph-mon[32429]: pgmap v258: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:28:54.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:54 smithi078 ceph-mon[32429]: Replacing daemon mds.k as rank 1 with standby daemon mds.l 2024-03-24T00:28:54.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:54 smithi078 ceph-mon[32429]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-03-24T00:28:54.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:54 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:28:54.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:54 smithi078 ceph-mon[32429]: osdmap e118: 12 total, 12 up, 12 in 2024-03-24T00:28:54.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:54 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:replay,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:28:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:54 smithi107 ceph-mon[36485]: pgmap v258: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:28:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:54 smithi107 ceph-mon[36485]: Replacing daemon mds.k as rank 1 with standby daemon mds.l 2024-03-24T00:28:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:54 smithi107 ceph-mon[36485]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-03-24T00:28:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:54 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:28:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:54 smithi107 ceph-mon[36485]: osdmap e118: 12 total, 12 up, 12 in 2024-03-24T00:28:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:54 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:replay,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:28:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:54 smithi106 ceph-mon[36505]: pgmap v258: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:28:54.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:54 smithi106 ceph-mon[36505]: Replacing daemon mds.k as rank 1 with standby daemon mds.l 2024-03-24T00:28:54.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:54 smithi106 ceph-mon[36505]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-03-24T00:28:54.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:54 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:28:54.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:54 smithi106 ceph-mon[36505]: osdmap e118: 12 total, 12 up, 12 in 2024-03-24T00:28:54.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:54 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:replay,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:28:55.688 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 264 sec latency 377279.491 ms 2024-03-24T00:28:56.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:56 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6832/4067467714,v1:172.21.15.107:6833/4067467714] up:resolve 2024-03-24T00:28:56.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:56 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:resolve,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:28:56.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:56 smithi078 ceph-mon[32429]: pgmap v260: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:28:56.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:56 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6832/4067467714,v1:172.21.15.107:6833/4067467714] up:resolve 2024-03-24T00:28:56.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:56 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:resolve,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:28:56.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:56 smithi107 ceph-mon[36485]: pgmap v260: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:28:56.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:56 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6832/4067467714,v1:172.21.15.107:6833/4067467714] up:resolve 2024-03-24T00:28:56.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:56 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:resolve,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:28:56.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:56 smithi106 ceph-mon[36505]: pgmap v260: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:28:56.688 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 265 sec latency 378279.580 ms 2024-03-24T00:28:57.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:57 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999984 2024-03-24T00:28:57.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:57 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6832/4067467714,v1:172.21.15.107:6833/4067467714] up:reconnect 2024-03-24T00:28:57.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:57 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:reconnect,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:28:57.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:57 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999984 2024-03-24T00:28:57.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:57 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6832/4067467714,v1:172.21.15.107:6833/4067467714] up:reconnect 2024-03-24T00:28:57.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:57 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:reconnect,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:28:57.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:57 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999984 2024-03-24T00:28:57.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:57 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6832/4067467714,v1:172.21.15.107:6833/4067467714] up:reconnect 2024-03-24T00:28:57.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:57 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:reconnect,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:28:57.688 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 266 sec latency 379279.671 ms 2024-03-24T00:28:58.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:58 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6832/4067467714,v1:172.21.15.107:6833/4067467714] up:rejoin 2024-03-24T00:28:58.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:58 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:rejoin,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:28:58.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:58 smithi078 ceph-mon[32429]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-24T00:28:58.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:58 smithi078 ceph-mon[32429]: pgmap v261: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:28:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6832/4067467714,v1:172.21.15.107:6833/4067467714] up:rejoin 2024-03-24T00:28:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:rejoin,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:28:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-mon[36485]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-24T00:28:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-mon[36485]: pgmap v261: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:28:58.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f8753a33700 time 2024-03-24T00:28:58.057078+0000 2024-03-24T00:28:58.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:28:58.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:28:58.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f875e05d4a1] 2024-03-24T00:28:58.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f875e05d667] 2024-03-24T00:28:58.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56116b360223] 2024-03-24T00:28:58.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56116b1da5d6] 2024-03-24T00:28:58.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56116b1dada1] 2024-03-24T00:28:58.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56116b1dd2d5] 2024-03-24T00:28:58.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56116b22d233] 2024-03-24T00:28:58.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56116b22f197] 2024-03-24T00:28:58.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56116b12e11b] 2024-03-24T00:28:58.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56116b12e6bc] 2024-03-24T00:28:58.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56116b1181ff] 2024-03-24T00:28:58.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f875e2ef488] 2024-03-24T00:28:58.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 13: (DispatchQueue::entry()+0x51f) [0x7f875e2ec60f] 2024-03-24T00:28:58.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f875e3bfe81] 2024-03-24T00:28:58.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f875cdda1ca] 2024-03-24T00:28:58.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 16: clone() 2024-03-24T00:28:58.346 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 2024-03-24T00:28:58.057+0000 7f8753a33700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f8753a33700 time 2024-03-24T00:28:58.057078+0000 2024-03-24T00:28:58.346 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:28:58.346 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 2024-03-24T00:28:58.346 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:28:58.346 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f875e05d4a1] 2024-03-24T00:28:58.346 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f875e05d667] 2024-03-24T00:28:58.346 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56116b360223] 2024-03-24T00:28:58.346 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56116b1da5d6] 2024-03-24T00:28:58.346 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56116b1dada1] 2024-03-24T00:28:58.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56116b1dd2d5] 2024-03-24T00:28:58.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56116b22d233] 2024-03-24T00:28:58.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56116b22f197] 2024-03-24T00:28:58.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56116b12e11b] 2024-03-24T00:28:58.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56116b12e6bc] 2024-03-24T00:28:58.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56116b1181ff] 2024-03-24T00:28:58.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f875e2ef488] 2024-03-24T00:28:58.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 13: (DispatchQueue::entry()+0x51f) [0x7f875e2ec60f] 2024-03-24T00:28:58.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f875e3bfe81] 2024-03-24T00:28:58.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f875cdda1ca] 2024-03-24T00:28:58.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 16: clone() 2024-03-24T00:28:58.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 2024-03-24T00:28:58.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: *** Caught signal (Aborted) ** 2024-03-24T00:28:58.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: in thread 7f8753a33700 thread_name:ms_dispatch 2024-03-24T00:28:58.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:28:58.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f875cde4d20] 2024-03-24T00:28:58.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 2: gsignal() 2024-03-24T00:28:58.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 3: abort() 2024-03-24T00:28:58.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f875e05d4fb] 2024-03-24T00:28:58.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f875e05d667] 2024-03-24T00:28:58.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56116b360223] 2024-03-24T00:28:58.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56116b1da5d6] 2024-03-24T00:28:58.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56116b1dada1] 2024-03-24T00:28:58.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56116b1dd2d5] 2024-03-24T00:28:58.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56116b22d233] 2024-03-24T00:28:58.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56116b22f197] 2024-03-24T00:28:58.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56116b12e11b] 2024-03-24T00:28:58.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56116b12e6bc] 2024-03-24T00:28:58.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56116b1181ff] 2024-03-24T00:28:58.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f875e2ef488] 2024-03-24T00:28:58.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 16: (DispatchQueue::entry()+0x51f) [0x7f875e2ec60f] 2024-03-24T00:28:58.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f875e3bfe81] 2024-03-24T00:28:58.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f875cdda1ca] 2024-03-24T00:28:58.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 19: clone() 2024-03-24T00:28:58.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 2024-03-24T00:28:58.059+0000 7f8753a33700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:28:58.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: in thread 7f8753a33700 thread_name:ms_dispatch 2024-03-24T00:28:58.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 2024-03-24T00:28:58.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:28:58.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f875cde4d20] 2024-03-24T00:28:58.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 2: gsignal() 2024-03-24T00:28:58.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 3: abort() 2024-03-24T00:28:58.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f875e05d4fb] 2024-03-24T00:28:58.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f875e05d667] 2024-03-24T00:28:58.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56116b360223] 2024-03-24T00:28:58.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56116b1da5d6] 2024-03-24T00:28:58.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56116b1dada1] 2024-03-24T00:28:58.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56116b1dd2d5] 2024-03-24T00:28:58.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56116b22d233] 2024-03-24T00:28:58.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56116b22f197] 2024-03-24T00:28:58.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56116b12e11b] 2024-03-24T00:28:58.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56116b12e6bc] 2024-03-24T00:28:58.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56116b1181ff] 2024-03-24T00:28:58.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f875e2ef488] 2024-03-24T00:28:58.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 16: (DispatchQueue::entry()+0x51f) [0x7f875e2ec60f] 2024-03-24T00:28:58.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f875e3bfe81] 2024-03-24T00:28:58.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f875cdda1ca] 2024-03-24T00:28:58.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 19: clone() 2024-03-24T00:28:58.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:28:58.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 2024-03-24T00:28:58.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: -1> 2024-03-24T00:28:58.057+0000 7f8753a33700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f8753a33700 time 2024-03-24T00:28:58.057078+0000 2024-03-24T00:28:58.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:28:58.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 2024-03-24T00:28:58.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:28:58.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f875e05d4a1] 2024-03-24T00:28:58.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f875e05d667] 2024-03-24T00:28:58.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56116b360223] 2024-03-24T00:28:58.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56116b1da5d6] 2024-03-24T00:28:58.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56116b1dada1] 2024-03-24T00:28:58.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56116b1dd2d5] 2024-03-24T00:28:58.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56116b22d233] 2024-03-24T00:28:58.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56116b22f197] 2024-03-24T00:28:58.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56116b12e11b] 2024-03-24T00:28:58.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56116b12e6bc] 2024-03-24T00:28:58.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56116b1181ff] 2024-03-24T00:28:58.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f875e2ef488] 2024-03-24T00:28:58.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 13: (DispatchQueue::entry()+0x51f) [0x7f875e2ec60f] 2024-03-24T00:28:58.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f875e3bfe81] 2024-03-24T00:28:58.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f875cdda1ca] 2024-03-24T00:28:58.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 16: clone() 2024-03-24T00:28:58.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 2024-03-24T00:28:58.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 0> 2024-03-24T00:28:58.059+0000 7f8753a33700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:28:58.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: in thread 7f8753a33700 thread_name:ms_dispatch 2024-03-24T00:28:58.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 2024-03-24T00:28:58.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:28:58.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f875cde4d20] 2024-03-24T00:28:58.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 2: gsignal() 2024-03-24T00:28:58.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 3: abort() 2024-03-24T00:28:58.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f875e05d4fb] 2024-03-24T00:28:58.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f875e05d667] 2024-03-24T00:28:58.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56116b360223] 2024-03-24T00:28:58.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56116b1da5d6] 2024-03-24T00:28:58.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56116b1dada1] 2024-03-24T00:28:58.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56116b1dd2d5] 2024-03-24T00:28:58.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56116b22d233] 2024-03-24T00:28:58.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56116b22f197] 2024-03-24T00:28:58.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56116b12e11b] 2024-03-24T00:28:58.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56116b12e6bc] 2024-03-24T00:28:58.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56116b1181ff] 2024-03-24T00:28:58.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f875e2ef488] 2024-03-24T00:28:58.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 16: (DispatchQueue::entry()+0x51f) [0x7f875e2ec60f] 2024-03-24T00:28:58.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f875e3bfe81] 2024-03-24T00:28:58.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f875cdda1ca] 2024-03-24T00:28:58.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 19: clone() 2024-03-24T00:28:58.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:28:58.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 2024-03-24T00:28:58.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: -1> 2024-03-24T00:28:58.057+0000 7f8753a33700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f8753a33700 time 2024-03-24T00:28:58.057078+0000 2024-03-24T00:28:58.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:28:58.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 2024-03-24T00:28:58.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:28:58.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f875e05d4a1] 2024-03-24T00:28:58.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f875e05d667] 2024-03-24T00:28:58.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56116b360223] 2024-03-24T00:28:58.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56116b1da5d6] 2024-03-24T00:28:58.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56116b1dada1] 2024-03-24T00:28:58.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56116b1dd2d5] 2024-03-24T00:28:58.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56116b22d233] 2024-03-24T00:28:58.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56116b22f197] 2024-03-24T00:28:58.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56116b12e11b] 2024-03-24T00:28:58.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56116b12e6bc] 2024-03-24T00:28:58.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56116b1181ff] 2024-03-24T00:28:58.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f875e2ef488] 2024-03-24T00:28:58.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 13: (DispatchQueue::entry()+0x51f) [0x7f875e2ec60f] 2024-03-24T00:28:58.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f875e3bfe81] 2024-03-24T00:28:58.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f875cdda1ca] 2024-03-24T00:28:58.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 16: clone() 2024-03-24T00:28:58.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 2024-03-24T00:28:58.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 0> 2024-03-24T00:28:58.059+0000 7f8753a33700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:28:58.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: in thread 7f8753a33700 thread_name:ms_dispatch 2024-03-24T00:28:58.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 2024-03-24T00:28:58.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:28:58.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f875cde4d20] 2024-03-24T00:28:58.356 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 2: gsignal() 2024-03-24T00:28:58.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 3: abort() 2024-03-24T00:28:58.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f875e05d4fb] 2024-03-24T00:28:58.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f875e05d667] 2024-03-24T00:28:58.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56116b360223] 2024-03-24T00:28:58.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56116b1da5d6] 2024-03-24T00:28:58.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56116b1dada1] 2024-03-24T00:28:58.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56116b1dd2d5] 2024-03-24T00:28:58.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56116b22d233] 2024-03-24T00:28:58.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56116b22f197] 2024-03-24T00:28:58.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56116b12e11b] 2024-03-24T00:28:58.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56116b12e6bc] 2024-03-24T00:28:58.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56116b1181ff] 2024-03-24T00:28:58.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f875e2ef488] 2024-03-24T00:28:58.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 16: (DispatchQueue::entry()+0x51f) [0x7f875e2ec60f] 2024-03-24T00:28:58.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f875e3bfe81] 2024-03-24T00:28:58.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f875cdda1ca] 2024-03-24T00:28:58.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 19: clone() 2024-03-24T00:28:58.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:28:58.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73581]: 2024-03-24T00:28:58.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 podman[73632]: 2024-03-24 00:28:58.106844848 +0000 UTC m=+0.020976353 container died 714954b2afc0f79bb23e2cbad0c1785e783f9065c0549d4efca6bfab2b961ab8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-24T00:28:58.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:58 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6832/4067467714,v1:172.21.15.107:6833/4067467714] up:rejoin 2024-03-24T00:28:58.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:58 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:rejoin,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:28:58.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:58 smithi106 ceph-mon[36505]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-24T00:28:58.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:58 smithi106 ceph-mon[36505]: pgmap v261: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:28:58.688 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 267 sec latency 380279.745 ms 2024-03-24T00:28:59.050 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 podman[73632]: 2024-03-24 00:28:58.753350606 +0000 UTC m=+0.667482131 container remove 714954b2afc0f79bb23e2cbad0c1785e783f9065c0549d4efca6bfab2b961ab8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, CEPH_POINT_RELEASE=) 2024-03-24T00:28:59.050 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:28:59.050 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:28:58 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.l.service: Failed with result 'exit-code'. 2024-03-24T00:28:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:59 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:28:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:59 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:28:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:59 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6832/4067467714,v1:172.21.15.107:6833/4067467714] up:active 2024-03-24T00:28:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:59 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:28:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:28:59 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=h=up:active} 3 up:standby-replay 3 up:standby 2024-03-24T00:28:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:59 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:28:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:59 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:28:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:59 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6832/4067467714,v1:172.21.15.107:6833/4067467714] up:active 2024-03-24T00:28:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:59 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:28:59.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:28:59 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=h=up:active} 3 up:standby-replay 3 up:standby 2024-03-24T00:28:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:59 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:28:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:59 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:28:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:59 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6832/4067467714,v1:172.21.15.107:6833/4067467714] up:active 2024-03-24T00:28:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:59 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:28:59.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:28:59 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=h=up:active} 3 up:standby-replay 3 up:standby 2024-03-24T00:28:59.689 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 268 sec latency 381279.837 ms 2024-03-24T00:29:00.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:00 smithi078 ceph-mon[32429]: pgmap v262: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:29:00.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:00 smithi107 ceph-mon[36485]: pgmap v262: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:29:00.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:00 smithi106 ceph-mon[36505]: pgmap v262: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:29:00.689 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 269 sec latency 382279.926 ms 2024-03-24T00:29:01.689 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 270 sec latency 383279.990 ms 2024-03-24T00:29:02.689 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 271 sec latency 384280.076 ms 2024-03-24T00:29:02.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:02 smithi078 ceph-mon[32429]: pgmap v263: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 204 B/s wr, 12 op/s 2024-03-24T00:29:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:02 smithi107 ceph-mon[36485]: pgmap v263: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 204 B/s wr, 12 op/s 2024-03-24T00:29:02.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:02 smithi106 ceph-mon[36505]: pgmap v263: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 204 B/s wr, 12 op/s 2024-03-24T00:29:03.689 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 272 sec latency 385280.189 ms 2024-03-24T00:29:04.689 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 273 sec latency 386280.246 ms 2024-03-24T00:29:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:04 smithi078 ceph-mon[32429]: pgmap v264: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 204 B/s wr, 12 op/s 2024-03-24T00:29:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:04 smithi107 ceph-mon[36485]: pgmap v264: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 204 B/s wr, 12 op/s 2024-03-24T00:29:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:04 smithi106 ceph-mon[36505]: pgmap v264: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 204 B/s wr, 12 op/s 2024-03-24T00:29:05.689 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 274 sec latency 387280.363 ms 2024-03-24T00:29:06.690 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 275 sec latency 388280.481 ms 2024-03-24T00:29:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:06 smithi078 ceph-mon[32429]: pgmap v265: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 180 B/s wr, 13 op/s 2024-03-24T00:29:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:06 smithi107 ceph-mon[36485]: pgmap v265: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 180 B/s wr, 13 op/s 2024-03-24T00:29:06.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:06 smithi106 ceph-mon[36505]: pgmap v265: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 180 B/s wr, 13 op/s 2024-03-24T00:29:07.689 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 276 sec latency 389280.586 ms 2024-03-24T00:29:08.689 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 277 sec latency 390280.707 ms 2024-03-24T00:29:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:08 smithi078 ceph-mon[32429]: pgmap v266: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-24T00:29:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:08 smithi107 ceph-mon[36485]: pgmap v266: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-24T00:29:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:08 smithi106 ceph-mon[36505]: pgmap v266: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-24T00:29:09.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:09 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.l.service: Scheduled restart job, restart counter is at 5. 2024-03-24T00:29:09.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:09 smithi107 systemd[1]: Stopped Ceph mds.l for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:29:09.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:09 smithi107 systemd[1]: Starting Ceph mds.l for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:29:09.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:09 smithi107 podman[73726]: 2024-03-24 00:29:09.22801819 +0000 UTC m=+0.017930565 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:29:09.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:09 smithi107 podman[73726]: 2024-03-24 00:29:09.344907762 +0000 UTC m=+0.134820114 container create b9f62c1f18c5e94818845446625f26351b87aa7f2a3f357e45e3ab87a137bd58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-24T00:29:09.690 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 278 sec latency 391280.754 ms 2024-03-24T00:29:10.093 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:09 smithi107 podman[73726]: 2024-03-24 00:29:09.597488168 +0000 UTC m=+0.387400525 container init b9f62c1f18c5e94818845446625f26351b87aa7f2a3f357e45e3ab87a137bd58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-24T00:29:10.093 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:09 smithi107 podman[73726]: 2024-03-24 00:29:09.600198373 +0000 UTC m=+0.390110735 container start b9f62c1f18c5e94818845446625f26351b87aa7f2a3f357e45e3ab87a137bd58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:29:10.094 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:09 smithi107 ceph-mds[73745]: -- 172.21.15.107:0/149042328 <== mon.1 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d9358efa40 con 0x55d9358c2800 2024-03-24T00:29:10.094 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:09 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: starting mds.l at 2024-03-24T00:29:10.094 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:09 smithi107 bash[73726]: b9f62c1f18c5e94818845446625f26351b87aa7f2a3f357e45e3ab87a137bd58 2024-03-24T00:29:10.094 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:09 smithi107 systemd[1]: Started Ceph mds.l for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:29:10.690 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 279 sec latency 392280.868 ms 2024-03-24T00:29:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:10 smithi078 ceph-mon[32429]: pgmap v267: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-24T00:29:10.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:10 smithi078 ceph-mon[32429]: daemon mds.l restarted 2024-03-24T00:29:10.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:10 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:29:10.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:10 smithi078 ceph-mon[32429]: osdmap e119: 12 total, 12 up, 12 in 2024-03-24T00:29:10.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:10 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6832/1006503957,v1:172.21.15.107:6833/1006503957] up:boot 2024-03-24T00:29:10.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:10 smithi078 ceph-mon[32429]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-03-24T00:29:10.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:10 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-03-24T00:29:10.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:29:10.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:10 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:replay,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:29:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:10 smithi107 ceph-mon[36485]: pgmap v267: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-24T00:29:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:10 smithi107 ceph-mon[36485]: daemon mds.l restarted 2024-03-24T00:29:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:10 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:29:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:10 smithi107 ceph-mon[36485]: osdmap e119: 12 total, 12 up, 12 in 2024-03-24T00:29:10.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:10 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6832/1006503957,v1:172.21.15.107:6833/1006503957] up:boot 2024-03-24T00:29:10.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:10 smithi107 ceph-mon[36485]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-03-24T00:29:10.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:10 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-03-24T00:29:10.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:29:10.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:10 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:replay,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:29:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:10 smithi106 ceph-mon[36505]: pgmap v267: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-24T00:29:10.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:10 smithi106 ceph-mon[36505]: daemon mds.l restarted 2024-03-24T00:29:10.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:10 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:29:10.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:10 smithi106 ceph-mon[36505]: osdmap e119: 12 total, 12 up, 12 in 2024-03-24T00:29:10.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:10 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6832/1006503957,v1:172.21.15.107:6833/1006503957] up:boot 2024-03-24T00:29:10.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:10 smithi106 ceph-mon[36505]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-03-24T00:29:10.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:10 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-03-24T00:29:10.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:29:10.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:10 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:replay,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:29:11.690 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 280 sec latency 393280.956 ms 2024-03-24T00:29:12.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:11 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6840/3424505910,v1:172.21.15.78:6841/3424505910] up:resolve 2024-03-24T00:29:12.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:11 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:resolve,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:29:12.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:11 smithi078 ceph-mon[32429]: pgmap v269: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:29:12.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:11 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6840/3424505910,v1:172.21.15.78:6841/3424505910] up:resolve 2024-03-24T00:29:12.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:11 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:resolve,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:29:12.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:11 smithi107 ceph-mon[36485]: pgmap v269: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:29:12.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:11 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6840/3424505910,v1:172.21.15.78:6841/3424505910] up:resolve 2024-03-24T00:29:12.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:11 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:resolve,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:29:12.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:11 smithi106 ceph-mon[36505]: pgmap v269: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:29:12.690 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 281 sec latency 394281.031 ms 2024-03-24T00:29:13.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:12 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999982 2024-03-24T00:29:13.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:12 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6840/3424505910,v1:172.21.15.78:6841/3424505910] up:reconnect 2024-03-24T00:29:13.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:12 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:reconnect,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:29:13.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:12 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999982 2024-03-24T00:29:13.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:12 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6840/3424505910,v1:172.21.15.78:6841/3424505910] up:reconnect 2024-03-24T00:29:13.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:12 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:reconnect,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:29:13.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:12 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999982 2024-03-24T00:29:13.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:12 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6840/3424505910,v1:172.21.15.78:6841/3424505910] up:reconnect 2024-03-24T00:29:13.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:12 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:reconnect,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:29:13.690 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 282 sec latency 395281.066 ms 2024-03-24T00:29:14.082 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6840/3424505910,v1:172.21.15.78:6841/3424505910] up:rejoin 2024-03-24T00:29:14.082 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:rejoin,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:29:14.082 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-mon[32429]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-03-24T00:29:14.082 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-mon[32429]: pgmap v270: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:29:14.083 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f45021e4700 time 2024-03-24T00:29:13.680381+0000 2024-03-24T00:29:14.083 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:29:14.083 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:29:14.083 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f450c80e4a1] 2024-03-24T00:29:14.083 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f450c80e667] 2024-03-24T00:29:14.083 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5622dda9d223] 2024-03-24T00:29:14.084 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5622dd9175d6] 2024-03-24T00:29:14.084 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5622dd917da1] 2024-03-24T00:29:14.084 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5622dd91a2d5] 2024-03-24T00:29:14.084 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5622dd96a233] 2024-03-24T00:29:14.084 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5622dd96c197] 2024-03-24T00:29:14.084 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5622dd86b11b] 2024-03-24T00:29:14.084 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5622dd86b6bc] 2024-03-24T00:29:14.084 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5622dd8551ff] 2024-03-24T00:29:14.084 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f450caa0488] 2024-03-24T00:29:14.084 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 13: (DispatchQueue::entry()+0x51f) [0x7f450ca9d60f] 2024-03-24T00:29:14.084 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f450cb70e81] 2024-03-24T00:29:14.085 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f450b58b1ca] 2024-03-24T00:29:14.085 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 16: clone() 2024-03-24T00:29:14.085 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 2024-03-24T00:29:13.680+0000 7f45021e4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f45021e4700 time 2024-03-24T00:29:13.680381+0000 2024-03-24T00:29:14.085 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:29:14.085 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 2024-03-24T00:29:14.085 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:29:14.085 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f450c80e4a1] 2024-03-24T00:29:14.085 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f450c80e667] 2024-03-24T00:29:14.085 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5622dda9d223] 2024-03-24T00:29:14.085 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5622dd9175d6] 2024-03-24T00:29:14.085 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5622dd917da1] 2024-03-24T00:29:14.085 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5622dd91a2d5] 2024-03-24T00:29:14.085 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5622dd96a233] 2024-03-24T00:29:14.085 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5622dd96c197] 2024-03-24T00:29:14.086 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5622dd86b11b] 2024-03-24T00:29:14.086 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5622dd86b6bc] 2024-03-24T00:29:14.086 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5622dd8551ff] 2024-03-24T00:29:14.086 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f450caa0488] 2024-03-24T00:29:14.086 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 13: (DispatchQueue::entry()+0x51f) [0x7f450ca9d60f] 2024-03-24T00:29:14.086 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f450cb70e81] 2024-03-24T00:29:14.086 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f450b58b1ca] 2024-03-24T00:29:14.086 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 16: clone() 2024-03-24T00:29:14.086 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 2024-03-24T00:29:14.086 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: *** Caught signal (Aborted) ** 2024-03-24T00:29:14.086 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: in thread 7f45021e4700 thread_name:ms_dispatch 2024-03-24T00:29:14.086 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:29:14.086 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f450b595d20] 2024-03-24T00:29:14.087 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 2: gsignal() 2024-03-24T00:29:14.088 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 3: abort() 2024-03-24T00:29:14.088 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f450c80e4fb] 2024-03-24T00:29:14.088 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f450c80e667] 2024-03-24T00:29:14.088 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5622dda9d223] 2024-03-24T00:29:14.088 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5622dd9175d6] 2024-03-24T00:29:14.088 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5622dd917da1] 2024-03-24T00:29:14.088 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5622dd91a2d5] 2024-03-24T00:29:14.088 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5622dd96a233] 2024-03-24T00:29:14.088 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5622dd96c197] 2024-03-24T00:29:14.088 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5622dd86b11b] 2024-03-24T00:29:14.088 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5622dd86b6bc] 2024-03-24T00:29:14.088 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5622dd8551ff] 2024-03-24T00:29:14.088 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f450caa0488] 2024-03-24T00:29:14.088 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 16: (DispatchQueue::entry()+0x51f) [0x7f450ca9d60f] 2024-03-24T00:29:14.089 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f450cb70e81] 2024-03-24T00:29:14.089 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f450b58b1ca] 2024-03-24T00:29:14.089 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 19: clone() 2024-03-24T00:29:14.089 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 2024-03-24T00:29:13.682+0000 7f45021e4700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:29:14.089 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: in thread 7f45021e4700 thread_name:ms_dispatch 2024-03-24T00:29:14.089 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 2024-03-24T00:29:14.089 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:29:14.089 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f450b595d20] 2024-03-24T00:29:14.089 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 2: gsignal() 2024-03-24T00:29:14.089 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 3: abort() 2024-03-24T00:29:14.089 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f450c80e4fb] 2024-03-24T00:29:14.089 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f450c80e667] 2024-03-24T00:29:14.090 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5622dda9d223] 2024-03-24T00:29:14.090 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5622dd9175d6] 2024-03-24T00:29:14.090 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5622dd917da1] 2024-03-24T00:29:14.090 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5622dd91a2d5] 2024-03-24T00:29:14.090 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5622dd96a233] 2024-03-24T00:29:14.090 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5622dd96c197] 2024-03-24T00:29:14.090 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5622dd86b11b] 2024-03-24T00:29:14.090 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5622dd86b6bc] 2024-03-24T00:29:14.090 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5622dd8551ff] 2024-03-24T00:29:14.090 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f450caa0488] 2024-03-24T00:29:14.090 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 16: (DispatchQueue::entry()+0x51f) [0x7f450ca9d60f] 2024-03-24T00:29:14.090 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f450cb70e81] 2024-03-24T00:29:14.090 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f450b58b1ca] 2024-03-24T00:29:14.090 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 19: clone() 2024-03-24T00:29:14.090 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:29:14.091 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 2024-03-24T00:29:14.091 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: -1> 2024-03-24T00:29:13.680+0000 7f45021e4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f45021e4700 time 2024-03-24T00:29:13.680381+0000 2024-03-24T00:29:14.091 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:29:14.091 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 2024-03-24T00:29:14.091 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:29:14.091 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f450c80e4a1] 2024-03-24T00:29:14.091 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f450c80e667] 2024-03-24T00:29:14.091 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5622dda9d223] 2024-03-24T00:29:14.091 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5622dd9175d6] 2024-03-24T00:29:14.091 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5622dd917da1] 2024-03-24T00:29:14.091 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5622dd91a2d5] 2024-03-24T00:29:14.091 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5622dd96a233] 2024-03-24T00:29:14.091 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5622dd96c197] 2024-03-24T00:29:14.091 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5622dd86b11b] 2024-03-24T00:29:14.092 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5622dd86b6bc] 2024-03-24T00:29:14.092 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5622dd8551ff] 2024-03-24T00:29:14.092 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f450caa0488] 2024-03-24T00:29:14.092 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 13: (DispatchQueue::entry()+0x51f) [0x7f450ca9d60f] 2024-03-24T00:29:14.092 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f450cb70e81] 2024-03-24T00:29:14.092 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f450b58b1ca] 2024-03-24T00:29:14.092 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 16: clone() 2024-03-24T00:29:14.092 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 2024-03-24T00:29:14.092 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 0> 2024-03-24T00:29:13.682+0000 7f45021e4700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:29:14.092 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: in thread 7f45021e4700 thread_name:ms_dispatch 2024-03-24T00:29:14.092 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 2024-03-24T00:29:14.092 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:29:14.092 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f450b595d20] 2024-03-24T00:29:14.092 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 2: gsignal() 2024-03-24T00:29:14.093 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 3: abort() 2024-03-24T00:29:14.093 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f450c80e4fb] 2024-03-24T00:29:14.093 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f450c80e667] 2024-03-24T00:29:14.093 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5622dda9d223] 2024-03-24T00:29:14.093 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5622dd9175d6] 2024-03-24T00:29:14.093 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5622dd917da1] 2024-03-24T00:29:14.093 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5622dd91a2d5] 2024-03-24T00:29:14.093 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5622dd96a233] 2024-03-24T00:29:14.093 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5622dd96c197] 2024-03-24T00:29:14.093 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5622dd86b11b] 2024-03-24T00:29:14.093 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5622dd86b6bc] 2024-03-24T00:29:14.093 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5622dd8551ff] 2024-03-24T00:29:14.093 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f450caa0488] 2024-03-24T00:29:14.093 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 16: (DispatchQueue::entry()+0x51f) [0x7f450ca9d60f] 2024-03-24T00:29:14.094 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f450cb70e81] 2024-03-24T00:29:14.094 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f450b58b1ca] 2024-03-24T00:29:14.094 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 19: clone() 2024-03-24T00:29:14.094 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:29:14.094 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 2024-03-24T00:29:14.094 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: -1> 2024-03-24T00:29:13.680+0000 7f45021e4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f45021e4700 time 2024-03-24T00:29:13.680381+0000 2024-03-24T00:29:14.094 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:29:14.094 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 2024-03-24T00:29:14.094 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:29:14.094 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f450c80e4a1] 2024-03-24T00:29:14.094 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f450c80e667] 2024-03-24T00:29:14.094 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5622dda9d223] 2024-03-24T00:29:14.095 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5622dd9175d6] 2024-03-24T00:29:14.095 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5622dd917da1] 2024-03-24T00:29:14.095 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5622dd91a2d5] 2024-03-24T00:29:14.095 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5622dd96a233] 2024-03-24T00:29:14.095 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5622dd96c197] 2024-03-24T00:29:14.095 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5622dd86b11b] 2024-03-24T00:29:14.095 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5622dd86b6bc] 2024-03-24T00:29:14.095 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5622dd8551ff] 2024-03-24T00:29:14.095 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f450caa0488] 2024-03-24T00:29:14.095 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 13: (DispatchQueue::entry()+0x51f) [0x7f450ca9d60f] 2024-03-24T00:29:14.095 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f450cb70e81] 2024-03-24T00:29:14.095 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f450b58b1ca] 2024-03-24T00:29:14.095 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 16: clone() 2024-03-24T00:29:14.095 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 2024-03-24T00:29:14.096 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 0> 2024-03-24T00:29:13.682+0000 7f45021e4700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:29:14.096 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: in thread 7f45021e4700 thread_name:ms_dispatch 2024-03-24T00:29:14.096 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 2024-03-24T00:29:14.096 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:29:14.096 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f450b595d20] 2024-03-24T00:29:14.096 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 2: gsignal() 2024-03-24T00:29:14.096 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 3: abort() 2024-03-24T00:29:14.096 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f450c80e4fb] 2024-03-24T00:29:14.096 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f450c80e667] 2024-03-24T00:29:14.096 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5622dda9d223] 2024-03-24T00:29:14.096 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5622dd9175d6] 2024-03-24T00:29:14.096 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5622dd917da1] 2024-03-24T00:29:14.096 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5622dd91a2d5] 2024-03-24T00:29:14.096 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5622dd96a233] 2024-03-24T00:29:14.096 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5622dd96c197] 2024-03-24T00:29:14.097 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5622dd86b11b] 2024-03-24T00:29:14.097 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5622dd86b6bc] 2024-03-24T00:29:14.097 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5622dd8551ff] 2024-03-24T00:29:14.097 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f450caa0488] 2024-03-24T00:29:14.097 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 16: (DispatchQueue::entry()+0x51f) [0x7f450ca9d60f] 2024-03-24T00:29:14.097 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f450cb70e81] 2024-03-24T00:29:14.097 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f450b58b1ca] 2024-03-24T00:29:14.097 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 19: clone() 2024-03-24T00:29:14.097 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:29:14.097 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[71733]: 2024-03-24T00:29:14.097 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:13 smithi078 podman[87154]: 2024-03-24 00:29:13.749272835 +0000 UTC m=+0.022873451 container died ec37728ad1d30bbfa14fab540501ba6b5cb3b114cbdb3deb2591414783474e65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-24T00:29:14.098 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:13 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6840/3424505910,v1:172.21.15.78:6841/3424505910] up:rejoin 2024-03-24T00:29:14.098 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:13 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:rejoin,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:29:14.098 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:13 smithi107 ceph-mon[36485]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-03-24T00:29:14.098 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:13 smithi107 ceph-mon[36485]: pgmap v270: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:29:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:13 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6840/3424505910,v1:172.21.15.78:6841/3424505910] up:rejoin 2024-03-24T00:29:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:13 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:rejoin,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:29:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:13 smithi106 ceph-mon[36505]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-03-24T00:29:14.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:13 smithi106 ceph-mon[36505]: pgmap v270: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:29:14.690 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 283 sec latency 396281.102 ms 2024-03-24T00:29:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:14 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:29:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:14 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:29:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:14 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6840/3424505910,v1:172.21.15.78:6841/3424505910] up:active 2024-03-24T00:29:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:14 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:29:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:14 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:active} 3 up:standby-replay 3 up:standby 2024-03-24T00:29:14.841 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:14 smithi078 podman[87154]: 2024-03-24 00:29:14.499828865 +0000 UTC m=+0.773429482 container remove ec37728ad1d30bbfa14fab540501ba6b5cb3b114cbdb3deb2591414783474e65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-24T00:29:14.841 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:14 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:29:14.842 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:14 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.a.service: Failed with result 'exit-code'. 2024-03-24T00:29:15.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:14 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:29:15.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:14 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:29:15.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:14 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6840/3424505910,v1:172.21.15.78:6841/3424505910] up:active 2024-03-24T00:29:15.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:14 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:29:15.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:14 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:active} 3 up:standby-replay 3 up:standby 2024-03-24T00:29:15.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:14 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:29:15.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:14 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:29:15.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:14 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6840/3424505910,v1:172.21.15.78:6841/3424505910] up:active 2024-03-24T00:29:15.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:14 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:29:15.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:14 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:active} 3 up:standby-replay 3 up:standby 2024-03-24T00:29:15.690 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 284 sec latency 397281.134 ms 2024-03-24T00:29:16.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:15 smithi078 ceph-mon[32429]: pgmap v271: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:29:16.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:15 smithi107 ceph-mon[36485]: pgmap v271: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:29:16.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:15 smithi106 ceph-mon[36505]: pgmap v271: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:29:16.690 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 285 sec latency 398281.240 ms 2024-03-24T00:29:17.690 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 286 sec latency 399281.372 ms 2024-03-24T00:29:18.690 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 287 sec latency 400281.475 ms 2024-03-24T00:29:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:18 smithi078 ceph-mon[32429]: pgmap v272: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:29:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:18 smithi107 ceph-mon[36485]: pgmap v272: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:29:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:18 smithi106 ceph-mon[36505]: pgmap v272: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:29:19.690 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 288 sec latency 401281.621 ms 2024-03-24T00:29:20.691 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 289 sec latency 402281.671 ms 2024-03-24T00:29:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:20 smithi078 ceph-mon[32429]: pgmap v273: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:29:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:20 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:29:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:20 smithi107 ceph-mon[36485]: pgmap v273: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:29:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:20 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:29:20.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:20 smithi106 ceph-mon[36505]: pgmap v273: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:29:20.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:20 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:29:21.691 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 290 sec latency 403281.783 ms 2024-03-24T00:29:21.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:21 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:21.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:21 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:29:21.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:21 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:29:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:21 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:21 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:29:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:21 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:29:21.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:21 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:21.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:21 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:29:21.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:21 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:29:22.691 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 291 sec latency 404281.875 ms 2024-03-24T00:29:22.745 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:22 smithi107 ceph-mon[36485]: pgmap v274: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-03-24T00:29:22.803 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:22 smithi106 ceph-mon[36505]: pgmap v274: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-03-24T00:29:22.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:22 smithi078 ceph-mon[32429]: pgmap v274: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-03-24T00:29:23.691 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 292 sec latency 405281.971 ms 2024-03-24T00:29:24.691 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 293 sec latency 406282.072 ms 2024-03-24T00:29:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:24 smithi078 ceph-mon[32429]: pgmap v275: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-24T00:29:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:24 smithi107 ceph-mon[36485]: pgmap v275: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-24T00:29:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:24 smithi106 ceph-mon[36505]: pgmap v275: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-24T00:29:25.221 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:24 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.a.service: Scheduled restart job, restart counter is at 1. 2024-03-24T00:29:25.221 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:24 smithi078 systemd[1]: Stopped Ceph mds.a for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:29:25.221 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:24 smithi078 systemd[1]: Starting Ceph mds.a for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:29:25.221 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:25 smithi078 podman[87384]: 2024-03-24 00:29:24.97769549 +0000 UTC m=+0.018005919 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:29:25.221 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:25 smithi078 podman[87384]: 2024-03-24 00:29:25.092312664 +0000 UTC m=+0.132623070 container create aaf9c182a08aa8c9e46885f8e8c86111439353a5343390549e86935089514bf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-24T00:29:25.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:25 smithi078 ceph-mon[32429]: daemon mds.a restarted 2024-03-24T00:29:25.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:25 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:29:25.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:25 smithi078 ceph-mon[32429]: osdmap e120: 12 total, 12 up, 12 in 2024-03-24T00:29:25.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:25 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6840/1196331852,v1:172.21.15.78:6841/1196331852] up:boot 2024-03-24T00:29:25.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:25 smithi078 ceph-mon[32429]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-24T00:29:25.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:25 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-03-24T00:29:25.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T00:29:25.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:25 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:replay,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:29:25.591 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:25 smithi078 podman[87384]: 2024-03-24 00:29:25.328652772 +0000 UTC m=+0.368963185 container init aaf9c182a08aa8c9e46885f8e8c86111439353a5343390549e86935089514bf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:29:25.591 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:25 smithi078 podman[87384]: 2024-03-24 00:29:25.331231125 +0000 UTC m=+0.371541542 container start aaf9c182a08aa8c9e46885f8e8c86111439353a5343390549e86935089514bf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True) 2024-03-24T00:29:25.591 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:25 smithi078 ceph-mds[87403]: -- 172.21.15.78:0/1777176271 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56072543da40 con 0x560725410800 2024-03-24T00:29:25.591 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:25 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: starting mds.a at 2024-03-24T00:29:25.591 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:25 smithi078 bash[87384]: aaf9c182a08aa8c9e46885f8e8c86111439353a5343390549e86935089514bf9 2024-03-24T00:29:25.592 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:25 smithi078 systemd[1]: Started Ceph mds.a for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:29:25.691 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 294 sec latency 407282.104 ms 2024-03-24T00:29:25.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:25 smithi107 ceph-mon[36485]: daemon mds.a restarted 2024-03-24T00:29:25.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:25 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:29:25.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:25 smithi107 ceph-mon[36485]: osdmap e120: 12 total, 12 up, 12 in 2024-03-24T00:29:25.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:25 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6840/1196331852,v1:172.21.15.78:6841/1196331852] up:boot 2024-03-24T00:29:25.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:25 smithi107 ceph-mon[36485]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-24T00:29:25.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:25 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-03-24T00:29:25.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T00:29:25.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:25 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:replay,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:29:25.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:25 smithi106 ceph-mon[36505]: daemon mds.a restarted 2024-03-24T00:29:25.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:25 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:29:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:25 smithi106 ceph-mon[36505]: osdmap e120: 12 total, 12 up, 12 in 2024-03-24T00:29:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:25 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6840/1196331852,v1:172.21.15.78:6841/1196331852] up:boot 2024-03-24T00:29:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:25 smithi106 ceph-mon[36505]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-24T00:29:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:25 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 4 up:standby, 1 failed 2024-03-24T00:29:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T00:29:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:25 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:replay,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:29:26.691 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 295 sec latency 408282.202 ms 2024-03-24T00:29:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:26 smithi078 ceph-mon[32429]: pgmap v277: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:29:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:26 smithi107 ceph-mon[36485]: pgmap v277: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:29:26.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:26 smithi106 ceph-mon[36505]: pgmap v277: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:29:27.691 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 296 sec latency 409282.313 ms 2024-03-24T00:29:27.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:27 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6832/1006503957,v1:172.21.15.107:6833/1006503957] up:resolve 2024-03-24T00:29:27.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:27 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:resolve,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:29:27.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:27 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6832/1006503957,v1:172.21.15.107:6833/1006503957] up:resolve 2024-03-24T00:29:27.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:27 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:resolve,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:29:27.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:27 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6832/1006503957,v1:172.21.15.107:6833/1006503957] up:resolve 2024-03-24T00:29:27.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:27 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:resolve,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:29:28.691 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 297 sec latency 410282.416 ms 2024-03-24T00:29:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:28 smithi078 ceph-mon[32429]: pgmap v278: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:29:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:28 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6832/1006503957,v1:172.21.15.107:6833/1006503957] up:reconnect 2024-03-24T00:29:28.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:28 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:reconnect,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:29:28.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:28 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.00299995 2024-03-24T00:29:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:28 smithi107 ceph-mon[36485]: pgmap v278: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:29:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:28 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6832/1006503957,v1:172.21.15.107:6833/1006503957] up:reconnect 2024-03-24T00:29:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:28 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:reconnect,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:29:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:28 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.00299995 2024-03-24T00:29:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:28 smithi106 ceph-mon[36505]: pgmap v278: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:29:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:28 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6832/1006503957,v1:172.21.15.107:6833/1006503957] up:reconnect 2024-03-24T00:29:28.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:28 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:reconnect,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:29:28.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:28 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.00299995 2024-03-24T00:29:29.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f894195a700 time 2024-03-24T00:29:29.052390+0000 2024-03-24T00:29:29.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:29:29.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:29:29.345 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f894bf844a1] 2024-03-24T00:29:29.346 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f894bf84667] 2024-03-24T00:29:29.346 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55d933535223] 2024-03-24T00:29:29.346 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55d9333af5d6] 2024-03-24T00:29:29.346 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55d9333afda1] 2024-03-24T00:29:29.346 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55d9333b22d5] 2024-03-24T00:29:29.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55d933402233] 2024-03-24T00:29:29.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d933404197] 2024-03-24T00:29:29.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d93330311b] 2024-03-24T00:29:29.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d9333036bc] 2024-03-24T00:29:29.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55d9332ed1ff] 2024-03-24T00:29:29.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f894c216488] 2024-03-24T00:29:29.347 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 13: (DispatchQueue::entry()+0x51f) [0x7f894c21360f] 2024-03-24T00:29:29.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f894c2e6e81] 2024-03-24T00:29:29.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f894ad011ca] 2024-03-24T00:29:29.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 16: clone() 2024-03-24T00:29:29.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 2024-03-24T00:29:29.053+0000 7f894195a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f894195a700 time 2024-03-24T00:29:29.052390+0000 2024-03-24T00:29:29.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:29:29.348 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 2024-03-24T00:29:29.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:29:29.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f894bf844a1] 2024-03-24T00:29:29.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f894bf84667] 2024-03-24T00:29:29.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55d933535223] 2024-03-24T00:29:29.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55d9333af5d6] 2024-03-24T00:29:29.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55d9333afda1] 2024-03-24T00:29:29.349 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55d9333b22d5] 2024-03-24T00:29:29.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55d933402233] 2024-03-24T00:29:29.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d933404197] 2024-03-24T00:29:29.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d93330311b] 2024-03-24T00:29:29.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d9333036bc] 2024-03-24T00:29:29.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55d9332ed1ff] 2024-03-24T00:29:29.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f894c216488] 2024-03-24T00:29:29.350 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 13: (DispatchQueue::entry()+0x51f) [0x7f894c21360f] 2024-03-24T00:29:29.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f894c2e6e81] 2024-03-24T00:29:29.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f894ad011ca] 2024-03-24T00:29:29.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 16: clone() 2024-03-24T00:29:29.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 2024-03-24T00:29:29.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: *** Caught signal (Aborted) ** 2024-03-24T00:29:29.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: in thread 7f894195a700 thread_name:ms_dispatch 2024-03-24T00:29:29.351 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:29:29.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f894ad0bd20] 2024-03-24T00:29:29.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 2: gsignal() 2024-03-24T00:29:29.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 3: abort() 2024-03-24T00:29:29.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f894bf844fb] 2024-03-24T00:29:29.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f894bf84667] 2024-03-24T00:29:29.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55d933535223] 2024-03-24T00:29:29.352 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55d9333af5d6] 2024-03-24T00:29:29.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55d9333afda1] 2024-03-24T00:29:29.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55d9333b22d5] 2024-03-24T00:29:29.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55d933402233] 2024-03-24T00:29:29.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d933404197] 2024-03-24T00:29:29.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d93330311b] 2024-03-24T00:29:29.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d9333036bc] 2024-03-24T00:29:29.353 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55d9332ed1ff] 2024-03-24T00:29:29.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f894c216488] 2024-03-24T00:29:29.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 16: (DispatchQueue::entry()+0x51f) [0x7f894c21360f] 2024-03-24T00:29:29.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f894c2e6e81] 2024-03-24T00:29:29.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f894ad011ca] 2024-03-24T00:29:29.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 19: clone() 2024-03-24T00:29:29.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 2024-03-24T00:29:29.055+0000 7f894195a700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:29:29.354 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: in thread 7f894195a700 thread_name:ms_dispatch 2024-03-24T00:29:29.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 2024-03-24T00:29:29.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:29:29.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f894ad0bd20] 2024-03-24T00:29:29.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 2: gsignal() 2024-03-24T00:29:29.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 3: abort() 2024-03-24T00:29:29.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f894bf844fb] 2024-03-24T00:29:29.355 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f894bf84667] 2024-03-24T00:29:29.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55d933535223] 2024-03-24T00:29:29.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55d9333af5d6] 2024-03-24T00:29:29.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55d9333afda1] 2024-03-24T00:29:29.357 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55d9333b22d5] 2024-03-24T00:29:29.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55d933402233] 2024-03-24T00:29:29.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d933404197] 2024-03-24T00:29:29.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d93330311b] 2024-03-24T00:29:29.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d9333036bc] 2024-03-24T00:29:29.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55d9332ed1ff] 2024-03-24T00:29:29.358 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f894c216488] 2024-03-24T00:29:29.359 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 16: (DispatchQueue::entry()+0x51f) [0x7f894c21360f] 2024-03-24T00:29:29.359 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f894c2e6e81] 2024-03-24T00:29:29.359 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f894ad011ca] 2024-03-24T00:29:29.359 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 19: clone() 2024-03-24T00:29:29.359 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:29:29.359 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 2024-03-24T00:29:29.359 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: -1> 2024-03-24T00:29:29.053+0000 7f894195a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f894195a700 time 2024-03-24T00:29:29.052390+0000 2024-03-24T00:29:29.360 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:29:29.360 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 2024-03-24T00:29:29.360 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:29:29.360 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f894bf844a1] 2024-03-24T00:29:29.360 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f894bf84667] 2024-03-24T00:29:29.360 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55d933535223] 2024-03-24T00:29:29.361 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55d9333af5d6] 2024-03-24T00:29:29.361 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55d9333afda1] 2024-03-24T00:29:29.361 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55d9333b22d5] 2024-03-24T00:29:29.361 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55d933402233] 2024-03-24T00:29:29.361 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d933404197] 2024-03-24T00:29:29.361 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d93330311b] 2024-03-24T00:29:29.361 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d9333036bc] 2024-03-24T00:29:29.362 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55d9332ed1ff] 2024-03-24T00:29:29.362 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f894c216488] 2024-03-24T00:29:29.362 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 13: (DispatchQueue::entry()+0x51f) [0x7f894c21360f] 2024-03-24T00:29:29.362 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f894c2e6e81] 2024-03-24T00:29:29.362 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f894ad011ca] 2024-03-24T00:29:29.362 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 16: clone() 2024-03-24T00:29:29.362 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 2024-03-24T00:29:29.363 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 0> 2024-03-24T00:29:29.055+0000 7f894195a700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:29:29.363 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: in thread 7f894195a700 thread_name:ms_dispatch 2024-03-24T00:29:29.363 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 2024-03-24T00:29:29.363 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:29:29.363 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f894ad0bd20] 2024-03-24T00:29:29.363 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 2: gsignal() 2024-03-24T00:29:29.363 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 3: abort() 2024-03-24T00:29:29.364 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f894bf844fb] 2024-03-24T00:29:29.364 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f894bf84667] 2024-03-24T00:29:29.364 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55d933535223] 2024-03-24T00:29:29.364 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55d9333af5d6] 2024-03-24T00:29:29.364 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55d9333afda1] 2024-03-24T00:29:29.365 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55d9333b22d5] 2024-03-24T00:29:29.365 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55d933402233] 2024-03-24T00:29:29.365 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d933404197] 2024-03-24T00:29:29.365 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d93330311b] 2024-03-24T00:29:29.365 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d9333036bc] 2024-03-24T00:29:29.365 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55d9332ed1ff] 2024-03-24T00:29:29.366 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f894c216488] 2024-03-24T00:29:29.366 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 16: (DispatchQueue::entry()+0x51f) [0x7f894c21360f] 2024-03-24T00:29:29.366 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f894c2e6e81] 2024-03-24T00:29:29.366 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f894ad011ca] 2024-03-24T00:29:29.366 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 19: clone() 2024-03-24T00:29:29.366 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:29:29.367 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 2024-03-24T00:29:29.367 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: -1> 2024-03-24T00:29:29.053+0000 7f894195a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f894195a700 time 2024-03-24T00:29:29.052390+0000 2024-03-24T00:29:29.367 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:29:29.367 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 2024-03-24T00:29:29.367 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:29:29.367 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f894bf844a1] 2024-03-24T00:29:29.367 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f894bf84667] 2024-03-24T00:29:29.368 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55d933535223] 2024-03-24T00:29:29.368 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55d9333af5d6] 2024-03-24T00:29:29.368 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55d9333afda1] 2024-03-24T00:29:29.368 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55d9333b22d5] 2024-03-24T00:29:29.368 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55d933402233] 2024-03-24T00:29:29.368 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d933404197] 2024-03-24T00:29:29.369 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d93330311b] 2024-03-24T00:29:29.369 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d9333036bc] 2024-03-24T00:29:29.369 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55d9332ed1ff] 2024-03-24T00:29:29.369 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f894c216488] 2024-03-24T00:29:29.369 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 13: (DispatchQueue::entry()+0x51f) [0x7f894c21360f] 2024-03-24T00:29:29.369 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f894c2e6e81] 2024-03-24T00:29:29.369 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f894ad011ca] 2024-03-24T00:29:29.370 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 16: clone() 2024-03-24T00:29:29.370 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 2024-03-24T00:29:29.370 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 0> 2024-03-24T00:29:29.055+0000 7f894195a700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:29:29.370 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: in thread 7f894195a700 thread_name:ms_dispatch 2024-03-24T00:29:29.370 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 2024-03-24T00:29:29.370 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:29:29.370 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f894ad0bd20] 2024-03-24T00:29:29.371 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 2: gsignal() 2024-03-24T00:29:29.371 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 3: abort() 2024-03-24T00:29:29.371 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f894bf844fb] 2024-03-24T00:29:29.371 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f894bf84667] 2024-03-24T00:29:29.371 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55d933535223] 2024-03-24T00:29:29.371 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55d9333af5d6] 2024-03-24T00:29:29.371 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55d9333afda1] 2024-03-24T00:29:29.372 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55d9333b22d5] 2024-03-24T00:29:29.372 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55d933402233] 2024-03-24T00:29:29.372 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d933404197] 2024-03-24T00:29:29.372 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d93330311b] 2024-03-24T00:29:29.372 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d9333036bc] 2024-03-24T00:29:29.372 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55d9332ed1ff] 2024-03-24T00:29:29.372 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f894c216488] 2024-03-24T00:29:29.373 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 16: (DispatchQueue::entry()+0x51f) [0x7f894c21360f] 2024-03-24T00:29:29.373 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f894c2e6e81] 2024-03-24T00:29:29.373 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f894ad011ca] 2024-03-24T00:29:29.373 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 19: clone() 2024-03-24T00:29:29.373 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:29:29.373 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l[73741]: 2024-03-24T00:29:29.373 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 podman[73925]: 2024-03-24 00:29:29.102734473 +0000 UTC m=+0.020603098 container died b9f62c1f18c5e94818845446625f26351b87aa7f2a3f357e45e3ab87a137bd58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-24T00:29:29.691 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 298 sec latency 411282.510 ms 2024-03-24T00:29:29.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:29 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6832/1006503957,v1:172.21.15.107:6833/1006503957] up:rejoin 2024-03-24T00:29:29.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:29 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:rejoin,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:29:29.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:29 smithi078 ceph-mon[32429]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-24T00:29:29.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:29 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:29:29.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:29 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:29:29.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:29 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6832/1006503957,v1:172.21.15.107:6833/1006503957] up:active 2024-03-24T00:29:29.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:29 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=h=up:active} 3 up:standby-replay 3 up:standby 2024-03-24T00:29:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6832/1006503957,v1:172.21.15.107:6833/1006503957] up:rejoin 2024-03-24T00:29:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:rejoin,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:29:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-mon[36485]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-24T00:29:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:29:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:29:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6832/1006503957,v1:172.21.15.107:6833/1006503957] up:active 2024-03-24T00:29:29.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:29 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=h=up:active} 3 up:standby-replay 3 up:standby 2024-03-24T00:29:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:29 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6832/1006503957,v1:172.21.15.107:6833/1006503957] up:rejoin 2024-03-24T00:29:29.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:29 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=l=up:rejoin,2=h=up:active} 2 up:standby-replay 4 up:standby 2024-03-24T00:29:29.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:29 smithi106 ceph-mon[36505]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-24T00:29:29.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:29 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:29:29.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:29 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:29:29.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:29 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6832/1006503957,v1:172.21.15.107:6833/1006503957] up:active 2024-03-24T00:29:29.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:29 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=l=up:active,2=h=up:active} 3 up:standby-replay 3 up:standby 2024-03-24T00:29:30.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 podman[73925]: 2024-03-24 00:29:29.968298622 +0000 UTC m=+0.886167292 container remove b9f62c1f18c5e94818845446625f26351b87aa7f2a3f357e45e3ab87a137bd58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-l, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-24T00:29:30.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:29 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:29:30.344 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:30 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.l.service: Failed with result 'exit-code'. 2024-03-24T00:29:30.691 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 299 sec latency 412282.638 ms 2024-03-24T00:29:30.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:30 smithi078 ceph-mon[32429]: pgmap v279: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:29:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:30 smithi107 ceph-mon[36485]: pgmap v279: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:29:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:30 smithi106 ceph-mon[36505]: pgmap v279: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:29:31.692 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 300 sec latency 413282.749 ms 2024-03-24T00:29:32.691 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 301 sec latency 414282.819 ms 2024-03-24T00:29:32.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:32 smithi078 ceph-mon[32429]: pgmap v280: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:29:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:32 smithi107 ceph-mon[36485]: pgmap v280: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:29:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:32 smithi106 ceph-mon[36505]: pgmap v280: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:29:33.691 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 302 sec latency 415282.849 ms 2024-03-24T00:29:34.692 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 303 sec latency 416282.938 ms 2024-03-24T00:29:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:34 smithi078 ceph-mon[32429]: pgmap v281: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:29:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:34 smithi107 ceph-mon[36485]: pgmap v281: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:29:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:34 smithi106 ceph-mon[36505]: pgmap v281: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:29:35.692 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 304 sec latency 417283.016 ms 2024-03-24T00:29:36.692 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 305 sec latency 418283.125 ms 2024-03-24T00:29:36.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:36 smithi078 ceph-mon[32429]: pgmap v282: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:29:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:36 smithi107 ceph-mon[36485]: pgmap v282: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:29:36.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:36 smithi106 ceph-mon[36505]: pgmap v282: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:29:37.692 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 306 sec latency 419283.186 ms 2024-03-24T00:29:38.692 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 307 sec latency 420283.279 ms 2024-03-24T00:29:38.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:38 smithi078 ceph-mon[32429]: pgmap v283: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 22 KiB/s wr, 11 op/s 2024-03-24T00:29:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:38 smithi107 ceph-mon[36485]: pgmap v283: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 22 KiB/s wr, 11 op/s 2024-03-24T00:29:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:38 smithi106 ceph-mon[36505]: pgmap v283: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 22 KiB/s wr, 11 op/s 2024-03-24T00:29:39.692 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 308 sec latency 421283.359 ms 2024-03-24T00:29:39.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:39 smithi078 ceph-mon[32429]: osdmap e121: 12 total, 12 up, 12 in 2024-03-24T00:29:39.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:39 smithi107 ceph-mon[36485]: osdmap e121: 12 total, 12 up, 12 in 2024-03-24T00:29:39.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:39 smithi106 ceph-mon[36505]: osdmap e121: 12 total, 12 up, 12 in 2024-03-24T00:29:40.692 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 309 sec latency 422283.461 ms 2024-03-24T00:29:40.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:40 smithi078 ceph-mon[32429]: pgmap v285: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:29:40.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:40 smithi107 ceph-mon[36485]: pgmap v285: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:29:40.843 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:40 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.l.service: Scheduled restart job, restart counter is at 6. 2024-03-24T00:29:40.843 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:40 smithi107 systemd[1]: Stopped Ceph mds.l for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:29:40.844 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:40 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.l.service: Start request repeated too quickly. 2024-03-24T00:29:40.844 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:40 smithi107 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.l.service: Failed with result 'exit-code'. 2024-03-24T00:29:40.844 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 24 00:29:40 smithi107 systemd[1]: Failed to start Ceph mds.l for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:29:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:40 smithi106 ceph-mon[36505]: pgmap v285: 129 pgs: 129 active+clean; 27 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:29:41.692 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 310 sec latency 423283.577 ms 2024-03-24T00:29:42.692 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 311 sec latency 424283.673 ms 2024-03-24T00:29:42.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:42 smithi078 ceph-mon[32429]: pgmap v286: 129 pgs: 129 active+clean; 27 MiB data, 517 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:29:42.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:42 smithi107 ceph-mon[36485]: pgmap v286: 129 pgs: 129 active+clean; 27 MiB data, 517 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:29:42.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:42 smithi106 ceph-mon[36505]: pgmap v286: 129 pgs: 129 active+clean; 27 MiB data, 517 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:29:43.693 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 312 sec latency 425283.707 ms 2024-03-24T00:29:44.693 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 313 sec latency 426283.802 ms 2024-03-24T00:29:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:44 smithi078 ceph-mon[32429]: pgmap v287: 129 pgs: 129 active+clean; 27 MiB data, 517 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:29:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:44 smithi078 ceph-mon[32429]: Replacing daemon mds.l as rank 1 with standby daemon mds.a 2024-03-24T00:29:44.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:44 smithi078 ceph-mon[32429]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-03-24T00:29:44.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:44 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:29:44.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:44 smithi078 ceph-mon[32429]: osdmap e122: 12 total, 12 up, 12 in 2024-03-24T00:29:44.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:44 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:replay,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:29:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:44 smithi107 ceph-mon[36485]: pgmap v287: 129 pgs: 129 active+clean; 27 MiB data, 517 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:29:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:44 smithi107 ceph-mon[36485]: Replacing daemon mds.l as rank 1 with standby daemon mds.a 2024-03-24T00:29:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:44 smithi107 ceph-mon[36485]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-03-24T00:29:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:44 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:29:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:44 smithi107 ceph-mon[36485]: osdmap e122: 12 total, 12 up, 12 in 2024-03-24T00:29:44.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:44 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:replay,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:29:44.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:44 smithi106 ceph-mon[36505]: pgmap v287: 129 pgs: 129 active+clean; 27 MiB data, 517 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:29:44.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:44 smithi106 ceph-mon[36505]: Replacing daemon mds.l as rank 1 with standby daemon mds.a 2024-03-24T00:29:44.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:44 smithi106 ceph-mon[36505]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-03-24T00:29:44.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:44 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:29:44.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:44 smithi106 ceph-mon[36505]: osdmap e122: 12 total, 12 up, 12 in 2024-03-24T00:29:44.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:44 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:replay,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:29:45.693 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 314 sec latency 427283.877 ms 2024-03-24T00:29:46.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:46 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6840/1196331852,v1:172.21.15.78:6841/1196331852] up:resolve 2024-03-24T00:29:46.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:46 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:resolve,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:29:46.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:46 smithi078 ceph-mon[32429]: pgmap v289: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-24T00:29:46.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:46 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6840/1196331852,v1:172.21.15.78:6841/1196331852] up:resolve 2024-03-24T00:29:46.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:46 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:resolve,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:29:46.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:46 smithi107 ceph-mon[36485]: pgmap v289: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-24T00:29:46.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:46 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6840/1196331852,v1:172.21.15.78:6841/1196331852] up:resolve 2024-03-24T00:29:46.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:46 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:resolve,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:29:46.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:46 smithi106 ceph-mon[36505]: pgmap v289: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-24T00:29:46.693 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 315 sec latency 428283.999 ms 2024-03-24T00:29:47.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:47 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999982 2024-03-24T00:29:47.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:47 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6840/1196331852,v1:172.21.15.78:6841/1196331852] up:reconnect 2024-03-24T00:29:47.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:47 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:reconnect,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:29:47.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:47 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999982 2024-03-24T00:29:47.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:47 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6840/1196331852,v1:172.21.15.78:6841/1196331852] up:reconnect 2024-03-24T00:29:47.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:47 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:reconnect,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:29:47.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:47 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999982 2024-03-24T00:29:47.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:47 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6840/1196331852,v1:172.21.15.78:6841/1196331852] up:reconnect 2024-03-24T00:29:47.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:47 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:reconnect,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:29:47.693 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 316 sec latency 429284.088 ms 2024-03-24T00:29:48.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6840/1196331852,v1:172.21.15.78:6841/1196331852] up:rejoin 2024-03-24T00:29:48.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:rejoin,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:29:48.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-mon[32429]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-03-24T00:29:48.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-mon[32429]: pgmap v290: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 230 B/s wr, 7 op/s 2024-03-24T00:29:48.341 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f0f5a626700 time 2024-03-24T00:29:48.078979+0000 2024-03-24T00:29:48.342 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:29:48.342 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:29:48.342 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f0f64c504a1] 2024-03-24T00:29:48.342 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0f64c50667] 2024-03-24T00:29:48.342 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560723dd0223] 2024-03-24T00:29:48.342 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560723c4a5d6] 2024-03-24T00:29:48.342 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560723c4ada1] 2024-03-24T00:29:48.342 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560723c4d2d5] 2024-03-24T00:29:48.342 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560723c9d233] 2024-03-24T00:29:48.342 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560723c9f197] 2024-03-24T00:29:48.343 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560723b9e11b] 2024-03-24T00:29:48.343 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560723b9e6bc] 2024-03-24T00:29:48.343 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560723b881ff] 2024-03-24T00:29:48.343 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0f64ee2488] 2024-03-24T00:29:48.343 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 13: (DispatchQueue::entry()+0x51f) [0x7f0f64edf60f] 2024-03-24T00:29:48.343 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0f64fb2e81] 2024-03-24T00:29:48.343 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0f639cd1ca] 2024-03-24T00:29:48.343 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 16: clone() 2024-03-24T00:29:48.343 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 2024-03-24T00:29:48.080+0000 7f0f5a626700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f0f5a626700 time 2024-03-24T00:29:48.078979+0000 2024-03-24T00:29:48.344 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:29:48.344 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 2024-03-24T00:29:48.344 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:29:48.344 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f0f64c504a1] 2024-03-24T00:29:48.344 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0f64c50667] 2024-03-24T00:29:48.344 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560723dd0223] 2024-03-24T00:29:48.344 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560723c4a5d6] 2024-03-24T00:29:48.344 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560723c4ada1] 2024-03-24T00:29:48.344 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560723c4d2d5] 2024-03-24T00:29:48.344 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560723c9d233] 2024-03-24T00:29:48.344 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560723c9f197] 2024-03-24T00:29:48.344 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560723b9e11b] 2024-03-24T00:29:48.344 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560723b9e6bc] 2024-03-24T00:29:48.345 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560723b881ff] 2024-03-24T00:29:48.345 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0f64ee2488] 2024-03-24T00:29:48.345 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 13: (DispatchQueue::entry()+0x51f) [0x7f0f64edf60f] 2024-03-24T00:29:48.345 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0f64fb2e81] 2024-03-24T00:29:48.345 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0f639cd1ca] 2024-03-24T00:29:48.345 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 16: clone() 2024-03-24T00:29:48.345 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 2024-03-24T00:29:48.345 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: *** Caught signal (Aborted) ** 2024-03-24T00:29:48.345 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: in thread 7f0f5a626700 thread_name:ms_dispatch 2024-03-24T00:29:48.345 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:29:48.345 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0f639d7d20] 2024-03-24T00:29:48.346 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:48 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6840/1196331852,v1:172.21.15.78:6841/1196331852] up:rejoin 2024-03-24T00:29:48.346 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:48 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:rejoin,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:29:48.346 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:48 smithi107 ceph-mon[36485]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-03-24T00:29:48.346 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:48 smithi107 ceph-mon[36485]: pgmap v290: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 230 B/s wr, 7 op/s 2024-03-24T00:29:48.347 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 2: gsignal() 2024-03-24T00:29:48.347 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 3: abort() 2024-03-24T00:29:48.347 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0f64c504fb] 2024-03-24T00:29:48.347 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0f64c50667] 2024-03-24T00:29:48.348 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560723dd0223] 2024-03-24T00:29:48.348 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560723c4a5d6] 2024-03-24T00:29:48.348 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560723c4ada1] 2024-03-24T00:29:48.348 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560723c4d2d5] 2024-03-24T00:29:48.348 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560723c9d233] 2024-03-24T00:29:48.348 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560723c9f197] 2024-03-24T00:29:48.348 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560723b9e11b] 2024-03-24T00:29:48.348 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560723b9e6bc] 2024-03-24T00:29:48.348 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560723b881ff] 2024-03-24T00:29:48.348 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0f64ee2488] 2024-03-24T00:29:48.348 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 16: (DispatchQueue::entry()+0x51f) [0x7f0f64edf60f] 2024-03-24T00:29:48.348 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0f64fb2e81] 2024-03-24T00:29:48.349 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0f639cd1ca] 2024-03-24T00:29:48.349 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 19: clone() 2024-03-24T00:29:48.349 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 2024-03-24T00:29:48.082+0000 7f0f5a626700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:29:48.349 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: in thread 7f0f5a626700 thread_name:ms_dispatch 2024-03-24T00:29:48.349 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 2024-03-24T00:29:48.349 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:29:48.349 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0f639d7d20] 2024-03-24T00:29:48.349 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 2: gsignal() 2024-03-24T00:29:48.349 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 3: abort() 2024-03-24T00:29:48.349 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0f64c504fb] 2024-03-24T00:29:48.349 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0f64c50667] 2024-03-24T00:29:48.349 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560723dd0223] 2024-03-24T00:29:48.349 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560723c4a5d6] 2024-03-24T00:29:48.350 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560723c4ada1] 2024-03-24T00:29:48.350 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560723c4d2d5] 2024-03-24T00:29:48.350 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560723c9d233] 2024-03-24T00:29:48.350 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560723c9f197] 2024-03-24T00:29:48.350 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560723b9e11b] 2024-03-24T00:29:48.350 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560723b9e6bc] 2024-03-24T00:29:48.351 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560723b881ff] 2024-03-24T00:29:48.351 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0f64ee2488] 2024-03-24T00:29:48.351 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 16: (DispatchQueue::entry()+0x51f) [0x7f0f64edf60f] 2024-03-24T00:29:48.351 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0f64fb2e81] 2024-03-24T00:29:48.351 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0f639cd1ca] 2024-03-24T00:29:48.351 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 19: clone() 2024-03-24T00:29:48.351 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:29:48.351 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 2024-03-24T00:29:48.351 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: -1> 2024-03-24T00:29:48.080+0000 7f0f5a626700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f0f5a626700 time 2024-03-24T00:29:48.078979+0000 2024-03-24T00:29:48.351 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:29:48.351 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 2024-03-24T00:29:48.352 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:29:48.352 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f0f64c504a1] 2024-03-24T00:29:48.352 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0f64c50667] 2024-03-24T00:29:48.352 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560723dd0223] 2024-03-24T00:29:48.352 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560723c4a5d6] 2024-03-24T00:29:48.352 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560723c4ada1] 2024-03-24T00:29:48.352 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560723c4d2d5] 2024-03-24T00:29:48.352 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560723c9d233] 2024-03-24T00:29:48.352 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560723c9f197] 2024-03-24T00:29:48.352 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560723b9e11b] 2024-03-24T00:29:48.352 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560723b9e6bc] 2024-03-24T00:29:48.352 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560723b881ff] 2024-03-24T00:29:48.352 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0f64ee2488] 2024-03-24T00:29:48.353 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 13: (DispatchQueue::entry()+0x51f) [0x7f0f64edf60f] 2024-03-24T00:29:48.353 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0f64fb2e81] 2024-03-24T00:29:48.353 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0f639cd1ca] 2024-03-24T00:29:48.353 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 16: clone() 2024-03-24T00:29:48.353 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 2024-03-24T00:29:48.353 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 0> 2024-03-24T00:29:48.082+0000 7f0f5a626700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:29:48.353 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: in thread 7f0f5a626700 thread_name:ms_dispatch 2024-03-24T00:29:48.353 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 2024-03-24T00:29:48.353 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:29:48.353 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0f639d7d20] 2024-03-24T00:29:48.353 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 2: gsignal() 2024-03-24T00:29:48.353 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 3: abort() 2024-03-24T00:29:48.353 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0f64c504fb] 2024-03-24T00:29:48.354 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0f64c50667] 2024-03-24T00:29:48.354 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560723dd0223] 2024-03-24T00:29:48.354 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560723c4a5d6] 2024-03-24T00:29:48.354 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560723c4ada1] 2024-03-24T00:29:48.354 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560723c4d2d5] 2024-03-24T00:29:48.354 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560723c9d233] 2024-03-24T00:29:48.354 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560723c9f197] 2024-03-24T00:29:48.354 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560723b9e11b] 2024-03-24T00:29:48.354 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560723b9e6bc] 2024-03-24T00:29:48.354 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560723b881ff] 2024-03-24T00:29:48.354 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0f64ee2488] 2024-03-24T00:29:48.354 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 16: (DispatchQueue::entry()+0x51f) [0x7f0f64edf60f] 2024-03-24T00:29:48.355 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0f64fb2e81] 2024-03-24T00:29:48.355 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0f639cd1ca] 2024-03-24T00:29:48.355 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 19: clone() 2024-03-24T00:29:48.355 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:29:48.355 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 2024-03-24T00:29:48.355 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: -1> 2024-03-24T00:29:48.080+0000 7f0f5a626700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f0f5a626700 time 2024-03-24T00:29:48.078979+0000 2024-03-24T00:29:48.355 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:29:48.355 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 2024-03-24T00:29:48.355 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:29:48.355 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f0f64c504a1] 2024-03-24T00:29:48.355 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0f64c50667] 2024-03-24T00:29:48.355 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560723dd0223] 2024-03-24T00:29:48.355 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560723c4a5d6] 2024-03-24T00:29:48.356 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560723c4ada1] 2024-03-24T00:29:48.356 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560723c4d2d5] 2024-03-24T00:29:48.356 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560723c9d233] 2024-03-24T00:29:48.356 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560723c9f197] 2024-03-24T00:29:48.356 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560723b9e11b] 2024-03-24T00:29:48.356 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560723b9e6bc] 2024-03-24T00:29:48.356 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560723b881ff] 2024-03-24T00:29:48.356 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0f64ee2488] 2024-03-24T00:29:48.356 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 13: (DispatchQueue::entry()+0x51f) [0x7f0f64edf60f] 2024-03-24T00:29:48.356 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0f64fb2e81] 2024-03-24T00:29:48.357 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0f639cd1ca] 2024-03-24T00:29:48.357 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 16: clone() 2024-03-24T00:29:48.357 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 2024-03-24T00:29:48.357 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 0> 2024-03-24T00:29:48.082+0000 7f0f5a626700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:29:48.357 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: in thread 7f0f5a626700 thread_name:ms_dispatch 2024-03-24T00:29:48.357 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 2024-03-24T00:29:48.358 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:29:48.358 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0f639d7d20] 2024-03-24T00:29:48.358 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 2: gsignal() 2024-03-24T00:29:48.358 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 3: abort() 2024-03-24T00:29:48.358 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0f64c504fb] 2024-03-24T00:29:48.359 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0f64c50667] 2024-03-24T00:29:48.359 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560723dd0223] 2024-03-24T00:29:48.359 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560723c4a5d6] 2024-03-24T00:29:48.359 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560723c4ada1] 2024-03-24T00:29:48.359 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560723c4d2d5] 2024-03-24T00:29:48.359 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560723c9d233] 2024-03-24T00:29:48.359 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560723c9f197] 2024-03-24T00:29:48.360 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560723b9e11b] 2024-03-24T00:29:48.360 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560723b9e6bc] 2024-03-24T00:29:48.360 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560723b881ff] 2024-03-24T00:29:48.360 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0f64ee2488] 2024-03-24T00:29:48.360 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 16: (DispatchQueue::entry()+0x51f) [0x7f0f64edf60f] 2024-03-24T00:29:48.360 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0f64fb2e81] 2024-03-24T00:29:48.361 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0f639cd1ca] 2024-03-24T00:29:48.361 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 19: clone() 2024-03-24T00:29:48.361 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:29:48.361 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87399]: 2024-03-24T00:29:48.362 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 podman[87460]: 2024-03-24 00:29:48.141080061 +0000 UTC m=+0.020502164 container died aaf9c182a08aa8c9e46885f8e8c86111439353a5343390549e86935089514bf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:29:48.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:48 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6840/1196331852,v1:172.21.15.78:6841/1196331852] up:rejoin 2024-03-24T00:29:48.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:48 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:rejoin,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:29:48.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:48 smithi106 ceph-mon[36505]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-03-24T00:29:48.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:48 smithi106 ceph-mon[36505]: pgmap v290: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 230 B/s wr, 7 op/s 2024-03-24T00:29:48.693 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 317 sec latency 430284.147 ms 2024-03-24T00:29:49.075 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:49 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:29:49.075 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:49 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:29:49.075 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:49 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6840/1196331852,v1:172.21.15.78:6841/1196331852] up:active 2024-03-24T00:29:49.075 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:49 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:29:49.075 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:49 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:active} 3 up:standby-replay 2 up:standby 2024-03-24T00:29:49.075 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:49 smithi078 ceph-mon[32429]: osdmap e123: 12 total, 12 up, 12 in 2024-03-24T00:29:49.076 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 podman[87460]: 2024-03-24 00:29:48.825230264 +0000 UTC m=+0.704652368 container remove aaf9c182a08aa8c9e46885f8e8c86111439353a5343390549e86935089514bf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-24T00:29:49.076 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:48 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:29:49.340 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:49 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.a.service: Failed with result 'exit-code'. 2024-03-24T00:29:49.347 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:49 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:29:49.347 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:49 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:29:49.347 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:49 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6840/1196331852,v1:172.21.15.78:6841/1196331852] up:active 2024-03-24T00:29:49.347 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:49 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:29:49.347 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:49 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:active} 3 up:standby-replay 2 up:standby 2024-03-24T00:29:49.347 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:49 smithi107 ceph-mon[36485]: osdmap e123: 12 total, 12 up, 12 in 2024-03-24T00:29:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:49 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:29:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:49 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:29:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:49 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6840/1196331852,v1:172.21.15.78:6841/1196331852] up:active 2024-03-24T00:29:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:49 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:29:49.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:49 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:active} 3 up:standby-replay 2 up:standby 2024-03-24T00:29:49.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:49 smithi106 ceph-mon[36505]: osdmap e123: 12 total, 12 up, 12 in 2024-03-24T00:29:49.693 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 318 sec latency 431284.191 ms 2024-03-24T00:29:50.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:50 smithi078 ceph-mon[32429]: pgmap v292: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 127 B/s wr, 4 op/s 2024-03-24T00:29:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:50 smithi107 ceph-mon[36485]: pgmap v292: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 127 B/s wr, 4 op/s 2024-03-24T00:29:50.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:50 smithi106 ceph-mon[36505]: pgmap v292: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 127 B/s wr, 4 op/s 2024-03-24T00:29:50.693 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 319 sec latency 432284.238 ms 2024-03-24T00:29:51.693 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 320 sec latency 433284.285 ms 2024-03-24T00:29:52.693 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 321 sec latency 434284.346 ms 2024-03-24T00:29:52.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:52 smithi078 ceph-mon[32429]: pgmap v293: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 383 B/s wr, 13 op/s 2024-03-24T00:29:52.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:52 smithi107 ceph-mon[36485]: pgmap v293: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 383 B/s wr, 13 op/s 2024-03-24T00:29:52.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:52 smithi106 ceph-mon[36505]: pgmap v293: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 383 B/s wr, 13 op/s 2024-03-24T00:29:53.693 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 322 sec latency 435284.380 ms 2024-03-24T00:29:54.693 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 323 sec latency 436284.471 ms 2024-03-24T00:29:54.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:54 smithi078 ceph-mon[32429]: pgmap v294: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 328 B/s wr, 11 op/s 2024-03-24T00:29:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:54 smithi107 ceph-mon[36485]: pgmap v294: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 328 B/s wr, 11 op/s 2024-03-24T00:29:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:54 smithi106 ceph-mon[36505]: pgmap v294: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 328 B/s wr, 11 op/s 2024-03-24T00:29:55.693 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 324 sec latency 437284.549 ms 2024-03-24T00:29:56.693 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 325 sec latency 438284.636 ms 2024-03-24T00:29:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:56 smithi078 ceph-mon[32429]: pgmap v295: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:29:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:56 smithi107 ceph-mon[36485]: pgmap v295: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:29:56.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:56 smithi106 ceph-mon[36505]: pgmap v295: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:29:57.694 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 326 sec latency 439284.737 ms 2024-03-24T00:29:58.694 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 327 sec latency 440284.831 ms 2024-03-24T00:29:58.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:29:58 smithi078 ceph-mon[32429]: pgmap v296: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:29:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:29:58 smithi107 ceph-mon[36485]: pgmap v296: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:29:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:29:58 smithi106 ceph-mon[36505]: pgmap v296: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:29:59.442 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:59 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.a.service: Scheduled restart job, restart counter is at 2. 2024-03-24T00:29:59.442 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:59 smithi078 systemd[1]: Stopped Ceph mds.a for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:29:59.442 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:59 smithi078 systemd[1]: Starting Ceph mds.a for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:29:59.442 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:59 smithi078 podman[87555]: 2024-03-24 00:29:59.227934296 +0000 UTC m=+0.018171104 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:29:59.442 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:59 smithi078 podman[87555]: 2024-03-24 00:29:59.442719141 +0000 UTC m=+0.232955938 container create 688d8b9fc55a051e98bce217caedab4be492a822002ca6f752ceb31ac209487c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-24T00:29:59.694 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 328 sec latency 441284.921 ms 2024-03-24T00:30:00.091 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:59 smithi078 podman[87555]: 2024-03-24 00:29:59.712084323 +0000 UTC m=+0.502321130 container init 688d8b9fc55a051e98bce217caedab4be492a822002ca6f752ceb31ac209487c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:30:00.091 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:59 smithi078 podman[87555]: 2024-03-24 00:29:59.714830147 +0000 UTC m=+0.505066946 container start 688d8b9fc55a051e98bce217caedab4be492a822002ca6f752ceb31ac209487c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-24T00:30:00.091 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:59 smithi078 ceph-mds[87573]: -- 172.21.15.78:0/2322441948 <== mon.2 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561aeb69ba40 con 0x561aeb66e800 2024-03-24T00:30:00.091 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: starting mds.a at 2024-03-24T00:30:00.091 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:59 smithi078 bash[87555]: 688d8b9fc55a051e98bce217caedab4be492a822002ca6f752ceb31ac209487c 2024-03-24T00:30:00.091 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:29:59 smithi078 systemd[1]: Started Ceph mds.a for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:30:00.694 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 329 sec latency 442284.972 ms 2024-03-24T00:30:00.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:00 smithi078 ceph-mon[32429]: pgmap v297: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 197 B/s wr, 10 op/s 2024-03-24T00:30:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:00 smithi078 ceph-mon[32429]: daemon mds.a restarted 2024-03-24T00:30:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:00 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:30:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:00 smithi078 ceph-mon[32429]: osdmap e124: 12 total, 12 up, 12 in 2024-03-24T00:30:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:00 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6840/1759489401,v1:172.21.15.78:6841/1759489401] up:boot 2024-03-24T00:30:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:00 smithi078 ceph-mon[32429]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-03-24T00:30:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:00 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:30:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:00 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T00:30:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:00 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:00 smithi078 ceph-mon[32429]: Health detail: HEALTH_WARN 1 filesystem is degraded 2024-03-24T00:30:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:00 smithi078 ceph-mon[32429]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-24T00:30:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:00 smithi078 ceph-mon[32429]: fs cephfs is degraded 2024-03-24T00:30:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:00 smithi107 ceph-mon[36485]: pgmap v297: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 197 B/s wr, 10 op/s 2024-03-24T00:30:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:00 smithi107 ceph-mon[36485]: daemon mds.a restarted 2024-03-24T00:30:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:00 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:30:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:00 smithi107 ceph-mon[36485]: osdmap e124: 12 total, 12 up, 12 in 2024-03-24T00:30:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:00 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6840/1759489401,v1:172.21.15.78:6841/1759489401] up:boot 2024-03-24T00:30:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:00 smithi107 ceph-mon[36485]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-03-24T00:30:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:00 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:30:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:00 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T00:30:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:00 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:00 smithi107 ceph-mon[36485]: Health detail: HEALTH_WARN 1 filesystem is degraded 2024-03-24T00:30:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:00 smithi107 ceph-mon[36485]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-24T00:30:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:00 smithi107 ceph-mon[36485]: fs cephfs is degraded 2024-03-24T00:30:00.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:00 smithi106 ceph-mon[36505]: pgmap v297: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 197 B/s wr, 10 op/s 2024-03-24T00:30:00.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:00 smithi106 ceph-mon[36505]: daemon mds.a restarted 2024-03-24T00:30:00.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:00 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:30:00.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:00 smithi106 ceph-mon[36505]: osdmap e124: 12 total, 12 up, 12 in 2024-03-24T00:30:00.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:00 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6840/1759489401,v1:172.21.15.78:6841/1759489401] up:boot 2024-03-24T00:30:00.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:00 smithi106 ceph-mon[36505]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-03-24T00:30:00.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:00 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:30:00.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:00 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T00:30:00.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:00 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:00.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:00 smithi106 ceph-mon[36505]: Health detail: HEALTH_WARN 1 filesystem is degraded 2024-03-24T00:30:00.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:00 smithi106 ceph-mon[36505]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-24T00:30:00.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:00 smithi106 ceph-mon[36505]: fs cephfs is degraded 2024-03-24T00:30:01.694 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 330 sec latency 443285.068 ms 2024-03-24T00:30:02.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:01 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6838/2758357551,v1:172.21.15.78:6839/2758357551] up:resolve 2024-03-24T00:30:02.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:01 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:02.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:01 smithi078 ceph-mon[32429]: pgmap v299: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:30:02.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:01 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6838/2758357551,v1:172.21.15.78:6839/2758357551] up:resolve 2024-03-24T00:30:02.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:01 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:02.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:01 smithi107 ceph-mon[36485]: pgmap v299: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:30:02.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:01 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6838/2758357551,v1:172.21.15.78:6839/2758357551] up:resolve 2024-03-24T00:30:02.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:01 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:02.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:01 smithi106 ceph-mon[36505]: pgmap v299: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:30:02.694 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 331 sec latency 444285.143 ms 2024-03-24T00:30:03.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:02 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:30:03.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:02 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6838/2758357551,v1:172.21.15.78:6839/2758357551] up:reconnect 2024-03-24T00:30:03.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:02 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:03.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:02 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:30:03.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:02 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6838/2758357551,v1:172.21.15.78:6839/2758357551] up:reconnect 2024-03-24T00:30:03.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:02 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:03.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:02 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:30:03.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:02 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6838/2758357551,v1:172.21.15.78:6839/2758357551] up:reconnect 2024-03-24T00:30:03.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:02 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:03.694 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 332 sec latency 445285.193 ms 2024-03-24T00:30:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6838/2758357551,v1:172.21.15.78:6839/2758357551] up:rejoin 2024-03-24T00:30:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:04.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-mon[32429]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-24T00:30:04.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-mon[32429]: pgmap v300: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:30:04.091 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fcde3848700 time 2024-03-24T00:30:03.798344+0000 2024-03-24T00:30:04.091 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:30:04.091 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:30:04.091 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fcdede724a1] 2024-03-24T00:30:04.091 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcdede72667] 2024-03-24T00:30:04.092 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5572877d0223] 2024-03-24T00:30:04.092 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55728764a5d6] 2024-03-24T00:30:04.092 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55728764ada1] 2024-03-24T00:30:04.092 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55728764d2d5] 2024-03-24T00:30:04.092 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55728769d233] 2024-03-24T00:30:04.092 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55728769f197] 2024-03-24T00:30:04.092 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55728759e11b] 2024-03-24T00:30:04.092 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55728759e6bc] 2024-03-24T00:30:04.092 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5572875881ff] 2024-03-24T00:30:04.092 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcdee104488] 2024-03-24T00:30:04.092 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 13: (DispatchQueue::entry()+0x51f) [0x7fcdee10160f] 2024-03-24T00:30:04.092 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcdee1d4e81] 2024-03-24T00:30:04.093 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcdecbef1ca] 2024-03-24T00:30:04.093 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 16: clone() 2024-03-24T00:30:04.093 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 2024-03-24T00:30:03.798+0000 7fcde3848700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fcde3848700 time 2024-03-24T00:30:03.798344+0000 2024-03-24T00:30:04.093 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:30:04.093 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 2024-03-24T00:30:04.093 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:30:04.093 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fcdede724a1] 2024-03-24T00:30:04.093 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcdede72667] 2024-03-24T00:30:04.093 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5572877d0223] 2024-03-24T00:30:04.093 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55728764a5d6] 2024-03-24T00:30:04.093 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55728764ada1] 2024-03-24T00:30:04.093 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55728764d2d5] 2024-03-24T00:30:04.093 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55728769d233] 2024-03-24T00:30:04.094 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55728769f197] 2024-03-24T00:30:04.094 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55728759e11b] 2024-03-24T00:30:04.094 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55728759e6bc] 2024-03-24T00:30:04.094 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5572875881ff] 2024-03-24T00:30:04.094 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcdee104488] 2024-03-24T00:30:04.094 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 13: (DispatchQueue::entry()+0x51f) [0x7fcdee10160f] 2024-03-24T00:30:04.094 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcdee1d4e81] 2024-03-24T00:30:04.094 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcdecbef1ca] 2024-03-24T00:30:04.094 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 16: clone() 2024-03-24T00:30:04.094 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 2024-03-24T00:30:04.094 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: *** Caught signal (Aborted) ** 2024-03-24T00:30:04.094 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: in thread 7fcde3848700 thread_name:ms_dispatch 2024-03-24T00:30:04.094 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:30:04.094 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcdecbf9d20] 2024-03-24T00:30:04.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:03 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6838/2758357551,v1:172.21.15.78:6839/2758357551] up:rejoin 2024-03-24T00:30:04.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:03 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:04.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:03 smithi107 ceph-mon[36485]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-24T00:30:04.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:03 smithi107 ceph-mon[36485]: pgmap v300: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:30:04.096 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 2: gsignal() 2024-03-24T00:30:04.096 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 3: abort() 2024-03-24T00:30:04.096 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fcdede724fb] 2024-03-24T00:30:04.096 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcdede72667] 2024-03-24T00:30:04.096 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5572877d0223] 2024-03-24T00:30:04.096 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55728764a5d6] 2024-03-24T00:30:04.097 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55728764ada1] 2024-03-24T00:30:04.097 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55728764d2d5] 2024-03-24T00:30:04.097 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55728769d233] 2024-03-24T00:30:04.097 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55728769f197] 2024-03-24T00:30:04.097 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55728759e11b] 2024-03-24T00:30:04.097 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55728759e6bc] 2024-03-24T00:30:04.097 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5572875881ff] 2024-03-24T00:30:04.097 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcdee104488] 2024-03-24T00:30:04.097 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 16: (DispatchQueue::entry()+0x51f) [0x7fcdee10160f] 2024-03-24T00:30:04.097 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcdee1d4e81] 2024-03-24T00:30:04.097 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcdecbef1ca] 2024-03-24T00:30:04.097 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 19: clone() 2024-03-24T00:30:04.097 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 2024-03-24T00:30:03.800+0000 7fcde3848700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:30:04.097 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: in thread 7fcde3848700 thread_name:ms_dispatch 2024-03-24T00:30:04.098 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 2024-03-24T00:30:04.098 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:30:04.098 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcdecbf9d20] 2024-03-24T00:30:04.098 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 2: gsignal() 2024-03-24T00:30:04.098 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 3: abort() 2024-03-24T00:30:04.098 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fcdede724fb] 2024-03-24T00:30:04.098 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcdede72667] 2024-03-24T00:30:04.098 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5572877d0223] 2024-03-24T00:30:04.098 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55728764a5d6] 2024-03-24T00:30:04.098 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55728764ada1] 2024-03-24T00:30:04.098 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55728764d2d5] 2024-03-24T00:30:04.098 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55728769d233] 2024-03-24T00:30:04.098 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55728769f197] 2024-03-24T00:30:04.098 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55728759e11b] 2024-03-24T00:30:04.099 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55728759e6bc] 2024-03-24T00:30:04.099 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5572875881ff] 2024-03-24T00:30:04.099 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcdee104488] 2024-03-24T00:30:04.099 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 16: (DispatchQueue::entry()+0x51f) [0x7fcdee10160f] 2024-03-24T00:30:04.099 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcdee1d4e81] 2024-03-24T00:30:04.099 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcdecbef1ca] 2024-03-24T00:30:04.099 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 19: clone() 2024-03-24T00:30:04.099 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:30:04.099 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 2024-03-24T00:30:04.099 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: -1> 2024-03-24T00:30:03.798+0000 7fcde3848700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fcde3848700 time 2024-03-24T00:30:03.798344+0000 2024-03-24T00:30:04.099 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:30:04.099 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 2024-03-24T00:30:04.099 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:30:04.099 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fcdede724a1] 2024-03-24T00:30:04.100 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcdede72667] 2024-03-24T00:30:04.100 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5572877d0223] 2024-03-24T00:30:04.100 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55728764a5d6] 2024-03-24T00:30:04.100 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55728764ada1] 2024-03-24T00:30:04.100 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55728764d2d5] 2024-03-24T00:30:04.100 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55728769d233] 2024-03-24T00:30:04.100 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55728769f197] 2024-03-24T00:30:04.100 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55728759e11b] 2024-03-24T00:30:04.100 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55728759e6bc] 2024-03-24T00:30:04.100 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5572875881ff] 2024-03-24T00:30:04.100 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcdee104488] 2024-03-24T00:30:04.100 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 13: (DispatchQueue::entry()+0x51f) [0x7fcdee10160f] 2024-03-24T00:30:04.100 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcdee1d4e81] 2024-03-24T00:30:04.100 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcdecbef1ca] 2024-03-24T00:30:04.101 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 16: clone() 2024-03-24T00:30:04.101 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 2024-03-24T00:30:04.101 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 0> 2024-03-24T00:30:03.800+0000 7fcde3848700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:30:04.101 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: in thread 7fcde3848700 thread_name:ms_dispatch 2024-03-24T00:30:04.101 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 2024-03-24T00:30:04.101 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:30:04.101 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcdecbf9d20] 2024-03-24T00:30:04.101 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 2: gsignal() 2024-03-24T00:30:04.101 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 3: abort() 2024-03-24T00:30:04.101 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fcdede724fb] 2024-03-24T00:30:04.101 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcdede72667] 2024-03-24T00:30:04.101 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5572877d0223] 2024-03-24T00:30:04.101 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55728764a5d6] 2024-03-24T00:30:04.101 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55728764ada1] 2024-03-24T00:30:04.101 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55728764d2d5] 2024-03-24T00:30:04.102 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55728769d233] 2024-03-24T00:30:04.102 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55728769f197] 2024-03-24T00:30:04.102 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55728759e11b] 2024-03-24T00:30:04.102 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55728759e6bc] 2024-03-24T00:30:04.102 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5572875881ff] 2024-03-24T00:30:04.102 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcdee104488] 2024-03-24T00:30:04.102 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 16: (DispatchQueue::entry()+0x51f) [0x7fcdee10160f] 2024-03-24T00:30:04.102 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcdee1d4e81] 2024-03-24T00:30:04.102 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcdecbef1ca] 2024-03-24T00:30:04.102 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 19: clone() 2024-03-24T00:30:04.102 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:30:04.102 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 2024-03-24T00:30:04.102 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: -1> 2024-03-24T00:30:03.798+0000 7fcde3848700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fcde3848700 time 2024-03-24T00:30:03.798344+0000 2024-03-24T00:30:04.102 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:30:04.102 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 2024-03-24T00:30:04.103 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:30:04.103 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fcdede724a1] 2024-03-24T00:30:04.103 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcdede72667] 2024-03-24T00:30:04.103 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5572877d0223] 2024-03-24T00:30:04.103 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55728764a5d6] 2024-03-24T00:30:04.103 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55728764ada1] 2024-03-24T00:30:04.103 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55728764d2d5] 2024-03-24T00:30:04.103 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55728769d233] 2024-03-24T00:30:04.103 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55728769f197] 2024-03-24T00:30:04.103 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55728759e11b] 2024-03-24T00:30:04.103 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55728759e6bc] 2024-03-24T00:30:04.103 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5572875881ff] 2024-03-24T00:30:04.103 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcdee104488] 2024-03-24T00:30:04.103 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 13: (DispatchQueue::entry()+0x51f) [0x7fcdee10160f] 2024-03-24T00:30:04.104 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcdee1d4e81] 2024-03-24T00:30:04.104 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcdecbef1ca] 2024-03-24T00:30:04.104 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 16: clone() 2024-03-24T00:30:04.104 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 2024-03-24T00:30:04.104 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 0> 2024-03-24T00:30:03.800+0000 7fcde3848700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:30:04.104 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: in thread 7fcde3848700 thread_name:ms_dispatch 2024-03-24T00:30:04.104 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 2024-03-24T00:30:04.104 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:30:04.104 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcdecbf9d20] 2024-03-24T00:30:04.104 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 2: gsignal() 2024-03-24T00:30:04.104 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 3: abort() 2024-03-24T00:30:04.104 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fcdede724fb] 2024-03-24T00:30:04.104 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcdede72667] 2024-03-24T00:30:04.104 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5572877d0223] 2024-03-24T00:30:04.104 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55728764a5d6] 2024-03-24T00:30:04.105 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55728764ada1] 2024-03-24T00:30:04.105 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55728764d2d5] 2024-03-24T00:30:04.105 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55728769d233] 2024-03-24T00:30:04.105 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55728769f197] 2024-03-24T00:30:04.105 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55728759e11b] 2024-03-24T00:30:04.105 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55728759e6bc] 2024-03-24T00:30:04.105 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5572875881ff] 2024-03-24T00:30:04.105 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcdee104488] 2024-03-24T00:30:04.105 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 16: (DispatchQueue::entry()+0x51f) [0x7fcdee10160f] 2024-03-24T00:30:04.105 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcdee1d4e81] 2024-03-24T00:30:04.105 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcdecbef1ca] 2024-03-24T00:30:04.105 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 19: clone() 2024-03-24T00:30:04.105 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:30:04.105 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[71382]: 2024-03-24T00:30:04.105 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:03 smithi078 podman[87607]: 2024-03-24 00:30:03.867057417 +0000 UTC m=+0.022340193 container died fc59ea233761ff068cce1fb220b5da030e9e3fa27580dd222c2a12846ce01223 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS) 2024-03-24T00:30:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:03 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6838/2758357551,v1:172.21.15.78:6839/2758357551] up:rejoin 2024-03-24T00:30:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:03 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:03 smithi106 ceph-mon[36505]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-24T00:30:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:03 smithi106 ceph-mon[36505]: pgmap v300: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:30:04.694 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 333 sec latency 446285.242 ms 2024-03-24T00:30:05.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:04 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:30:05.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:04 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:30:05.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:04 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6838/2758357551,v1:172.21.15.78:6839/2758357551] up:active 2024-03-24T00:30:05.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:04 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:05.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:04 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 2 up:standby 2024-03-24T00:30:05.091 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:04 smithi078 podman[87607]: 2024-03-24 00:30:04.659262785 +0000 UTC m=+0.814545561 container remove fc59ea233761ff068cce1fb220b5da030e9e3fa27580dd222c2a12846ce01223 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:30:05.091 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:04 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:30:05.091 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:04 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.g.service: Failed with result 'exit-code'. 2024-03-24T00:30:05.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:04 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:30:05.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:04 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:30:05.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:04 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6838/2758357551,v1:172.21.15.78:6839/2758357551] up:active 2024-03-24T00:30:05.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:04 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:05.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:04 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 2 up:standby 2024-03-24T00:30:05.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:04 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:30:05.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:04 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:30:05.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:04 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6838/2758357551,v1:172.21.15.78:6839/2758357551] up:active 2024-03-24T00:30:05.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:04 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:05.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:04 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 2 up:standby 2024-03-24T00:30:05.694 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 334 sec latency 447285.313 ms 2024-03-24T00:30:06.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:05 smithi078 ceph-mon[32429]: pgmap v301: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-24T00:30:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:05 smithi107 ceph-mon[36485]: pgmap v301: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-24T00:30:06.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:05 smithi106 ceph-mon[36505]: pgmap v301: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-24T00:30:06.694 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 335 sec latency 448285.392 ms 2024-03-24T00:30:07.694 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 336 sec latency 449285.493 ms 2024-03-24T00:30:08.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:08 smithi078 ceph-mon[32429]: pgmap v302: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-24T00:30:08.694 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 337 sec latency 450285.618 ms 2024-03-24T00:30:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:08 smithi107 ceph-mon[36485]: pgmap v302: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-24T00:30:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:08 smithi106 ceph-mon[36505]: pgmap v302: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-24T00:30:09.695 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 338 sec latency 451285.708 ms 2024-03-24T00:30:10.695 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 339 sec latency 452285.814 ms 2024-03-24T00:30:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:10 smithi078 ceph-mon[32429]: pgmap v303: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-24T00:30:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:10 smithi107 ceph-mon[36485]: pgmap v303: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-24T00:30:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:10 smithi106 ceph-mon[36505]: pgmap v303: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-24T00:30:11.695 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 340 sec latency 453285.939 ms 2024-03-24T00:30:12.695 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 341 sec latency 454286.038 ms 2024-03-24T00:30:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:12 smithi078 ceph-mon[32429]: pgmap v304: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-24T00:30:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:12 smithi107 ceph-mon[36485]: pgmap v304: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-24T00:30:12.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:12 smithi106 ceph-mon[36505]: pgmap v304: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-24T00:30:13.695 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 342 sec latency 455286.121 ms 2024-03-24T00:30:14.695 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 343 sec latency 456286.177 ms 2024-03-24T00:30:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:14 smithi078 ceph-mon[32429]: pgmap v305: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T00:30:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:14 smithi107 ceph-mon[36485]: pgmap v305: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T00:30:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:14 smithi106 ceph-mon[36505]: pgmap v305: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T00:30:15.441 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:15 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.g.service: Scheduled restart job, restart counter is at 1. 2024-03-24T00:30:15.441 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:15 smithi078 systemd[1]: Stopped Ceph mds.g for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:30:15.441 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:15 smithi078 systemd[1]: Starting Ceph mds.g for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:30:15.441 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:15 smithi078 podman[87704]: 2024-03-24 00:30:15.227346084 +0000 UTC m=+0.018651957 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:30:15.441 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:15 smithi078 podman[87704]: 2024-03-24 00:30:15.335069871 +0000 UTC m=+0.126375744 container create 9dc8d561f02c4445c55f0e855cd97a386b83cd77686b9d317c0b8848a6d802b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS) 2024-03-24T00:30:15.695 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 344 sec latency 457286.296 ms 2024-03-24T00:30:15.840 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:15 smithi078 podman[87704]: 2024-03-24 00:30:15.562882303 +0000 UTC m=+0.354188176 container init 9dc8d561f02c4445c55f0e855cd97a386b83cd77686b9d317c0b8848a6d802b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-24T00:30:15.840 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:15 smithi078 podman[87704]: 2024-03-24 00:30:15.56551341 +0000 UTC m=+0.356819283 container start 9dc8d561f02c4445c55f0e855cd97a386b83cd77686b9d317c0b8848a6d802b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-24T00:30:15.840 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:15 smithi078 ceph-mds[87723]: -- 172.21.15.78:0/199214874 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560202abfa40 con 0x560202a93000 2024-03-24T00:30:15.841 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:15 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: starting mds.g at 2024-03-24T00:30:15.841 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:15 smithi078 bash[87704]: 9dc8d561f02c4445c55f0e855cd97a386b83cd77686b9d317c0b8848a6d802b7 2024-03-24T00:30:15.841 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:15 smithi078 systemd[1]: Started Ceph mds.g for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:30:16.695 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 345 sec latency 458286.395 ms 2024-03-24T00:30:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:16 smithi078 ceph-mon[32429]: pgmap v306: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-03-24T00:30:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:16 smithi078 ceph-mon[32429]: daemon mds.g restarted 2024-03-24T00:30:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:16 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:30:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:16 smithi078 ceph-mon[32429]: osdmap e125: 12 total, 12 up, 12 in 2024-03-24T00:30:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:16 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6838/2645579087,v1:172.21.15.78:6839/2645579087] up:boot 2024-03-24T00:30:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:16 smithi078 ceph-mon[32429]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-03-24T00:30:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:16 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:30:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:30:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:16 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:replay,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:16 smithi107 ceph-mon[36485]: pgmap v306: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-03-24T00:30:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:16 smithi107 ceph-mon[36485]: daemon mds.g restarted 2024-03-24T00:30:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:16 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:30:16.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:16 smithi107 ceph-mon[36485]: osdmap e125: 12 total, 12 up, 12 in 2024-03-24T00:30:16.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:16 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6838/2645579087,v1:172.21.15.78:6839/2645579087] up:boot 2024-03-24T00:30:16.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:16 smithi107 ceph-mon[36485]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-03-24T00:30:16.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:16 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:30:16.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:30:16.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:16 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:replay,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:16.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:16 smithi106 ceph-mon[36505]: pgmap v306: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-03-24T00:30:16.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:16 smithi106 ceph-mon[36505]: daemon mds.g restarted 2024-03-24T00:30:16.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:16 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:30:16.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:16 smithi106 ceph-mon[36505]: osdmap e125: 12 total, 12 up, 12 in 2024-03-24T00:30:16.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:16 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6838/2645579087,v1:172.21.15.78:6839/2645579087] up:boot 2024-03-24T00:30:16.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:16 smithi106 ceph-mon[36505]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-03-24T00:30:16.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:16 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:30:16.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:30:16.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:16 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:replay,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:17.695 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 346 sec latency 459286.493 ms 2024-03-24T00:30:17.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:17 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6840/1759489401,v1:172.21.15.78:6841/1759489401] up:resolve 2024-03-24T00:30:17.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:17 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:resolve,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:17.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:17 smithi106 ceph-mon[36505]: pgmap v308: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:30:18.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:17 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6840/1759489401,v1:172.21.15.78:6841/1759489401] up:resolve 2024-03-24T00:30:18.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:17 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:resolve,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:18.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:17 smithi078 ceph-mon[32429]: pgmap v308: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:30:18.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:17 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6840/1759489401,v1:172.21.15.78:6841/1759489401] up:resolve 2024-03-24T00:30:18.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:17 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:resolve,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:18.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:17 smithi107 ceph-mon[36485]: pgmap v308: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:30:18.695 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 347 sec latency 460286.632 ms 2024-03-24T00:30:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:18 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:30:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:18 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6840/1759489401,v1:172.21.15.78:6841/1759489401] up:reconnect 2024-03-24T00:30:18.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:18 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:reconnect,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:19.055 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:18 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:30:19.055 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:18 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6840/1759489401,v1:172.21.15.78:6841/1759489401] up:reconnect 2024-03-24T00:30:19.055 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:18 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:reconnect,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:18 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:30:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:18 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6840/1759489401,v1:172.21.15.78:6841/1759489401] up:reconnect 2024-03-24T00:30:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:18 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:reconnect,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:19.341 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff313946700 time 2024-03-24T00:30:19.070704+0000 2024-03-24T00:30:19.341 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:30:19.341 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:30:19.341 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff31df704a1] 2024-03-24T00:30:19.341 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff31df70667] 2024-03-24T00:30:19.341 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561aea807223] 2024-03-24T00:30:19.342 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561aea6815d6] 2024-03-24T00:30:19.342 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561aea681da1] 2024-03-24T00:30:19.342 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x561aea6842d5] 2024-03-24T00:30:19.342 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561aea6d4233] 2024-03-24T00:30:19.342 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561aea6d6197] 2024-03-24T00:30:19.342 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561aea5d511b] 2024-03-24T00:30:19.342 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561aea5d56bc] 2024-03-24T00:30:19.342 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561aea5bf1ff] 2024-03-24T00:30:19.342 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff31e202488] 2024-03-24T00:30:19.342 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 13: (DispatchQueue::entry()+0x51f) [0x7ff31e1ff60f] 2024-03-24T00:30:19.342 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff31e2d2e81] 2024-03-24T00:30:19.342 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff31cced1ca] 2024-03-24T00:30:19.343 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 16: clone() 2024-03-24T00:30:19.343 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 2024-03-24T00:30:19.071+0000 7ff313946700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff313946700 time 2024-03-24T00:30:19.070704+0000 2024-03-24T00:30:19.343 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:30:19.343 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 2024-03-24T00:30:19.343 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:30:19.343 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff31df704a1] 2024-03-24T00:30:19.343 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff31df70667] 2024-03-24T00:30:19.343 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561aea807223] 2024-03-24T00:30:19.343 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561aea6815d6] 2024-03-24T00:30:19.343 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561aea681da1] 2024-03-24T00:30:19.343 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x561aea6842d5] 2024-03-24T00:30:19.343 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561aea6d4233] 2024-03-24T00:30:19.343 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561aea6d6197] 2024-03-24T00:30:19.343 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561aea5d511b] 2024-03-24T00:30:19.344 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561aea5d56bc] 2024-03-24T00:30:19.344 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561aea5bf1ff] 2024-03-24T00:30:19.344 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff31e202488] 2024-03-24T00:30:19.344 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 13: (DispatchQueue::entry()+0x51f) [0x7ff31e1ff60f] 2024-03-24T00:30:19.344 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff31e2d2e81] 2024-03-24T00:30:19.344 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff31cced1ca] 2024-03-24T00:30:19.344 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 16: clone() 2024-03-24T00:30:19.344 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 2024-03-24T00:30:19.344 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: *** Caught signal (Aborted) ** 2024-03-24T00:30:19.344 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: in thread 7ff313946700 thread_name:ms_dispatch 2024-03-24T00:30:19.344 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:30:19.344 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff31ccf7d20] 2024-03-24T00:30:19.345 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 2: gsignal() 2024-03-24T00:30:19.345 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 3: abort() 2024-03-24T00:30:19.345 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff31df704fb] 2024-03-24T00:30:19.345 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff31df70667] 2024-03-24T00:30:19.346 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561aea807223] 2024-03-24T00:30:19.346 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561aea6815d6] 2024-03-24T00:30:19.346 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561aea681da1] 2024-03-24T00:30:19.346 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x561aea6842d5] 2024-03-24T00:30:19.346 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561aea6d4233] 2024-03-24T00:30:19.346 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561aea6d6197] 2024-03-24T00:30:19.346 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561aea5d511b] 2024-03-24T00:30:19.346 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561aea5d56bc] 2024-03-24T00:30:19.346 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561aea5bf1ff] 2024-03-24T00:30:19.346 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff31e202488] 2024-03-24T00:30:19.346 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 16: (DispatchQueue::entry()+0x51f) [0x7ff31e1ff60f] 2024-03-24T00:30:19.346 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff31e2d2e81] 2024-03-24T00:30:19.346 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff31cced1ca] 2024-03-24T00:30:19.347 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 19: clone() 2024-03-24T00:30:19.347 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 2024-03-24T00:30:19.074+0000 7ff313946700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:30:19.347 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: in thread 7ff313946700 thread_name:ms_dispatch 2024-03-24T00:30:19.347 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 2024-03-24T00:30:19.347 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:30:19.347 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff31ccf7d20] 2024-03-24T00:30:19.347 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 2: gsignal() 2024-03-24T00:30:19.347 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 3: abort() 2024-03-24T00:30:19.347 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff31df704fb] 2024-03-24T00:30:19.347 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff31df70667] 2024-03-24T00:30:19.347 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561aea807223] 2024-03-24T00:30:19.347 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561aea6815d6] 2024-03-24T00:30:19.347 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561aea681da1] 2024-03-24T00:30:19.347 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x561aea6842d5] 2024-03-24T00:30:19.348 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561aea6d4233] 2024-03-24T00:30:19.348 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561aea6d6197] 2024-03-24T00:30:19.348 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561aea5d511b] 2024-03-24T00:30:19.348 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561aea5d56bc] 2024-03-24T00:30:19.348 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561aea5bf1ff] 2024-03-24T00:30:19.348 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff31e202488] 2024-03-24T00:30:19.348 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 16: (DispatchQueue::entry()+0x51f) [0x7ff31e1ff60f] 2024-03-24T00:30:19.348 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff31e2d2e81] 2024-03-24T00:30:19.348 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff31cced1ca] 2024-03-24T00:30:19.348 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 19: clone() 2024-03-24T00:30:19.348 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:30:19.348 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 2024-03-24T00:30:19.348 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: -1> 2024-03-24T00:30:19.071+0000 7ff313946700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff313946700 time 2024-03-24T00:30:19.070704+0000 2024-03-24T00:30:19.348 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:30:19.348 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 2024-03-24T00:30:19.348 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:30:19.349 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff31df704a1] 2024-03-24T00:30:19.349 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff31df70667] 2024-03-24T00:30:19.349 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561aea807223] 2024-03-24T00:30:19.349 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561aea6815d6] 2024-03-24T00:30:19.349 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561aea681da1] 2024-03-24T00:30:19.349 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x561aea6842d5] 2024-03-24T00:30:19.349 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561aea6d4233] 2024-03-24T00:30:19.349 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561aea6d6197] 2024-03-24T00:30:19.349 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561aea5d511b] 2024-03-24T00:30:19.349 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561aea5d56bc] 2024-03-24T00:30:19.349 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561aea5bf1ff] 2024-03-24T00:30:19.349 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff31e202488] 2024-03-24T00:30:19.349 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 13: (DispatchQueue::entry()+0x51f) [0x7ff31e1ff60f] 2024-03-24T00:30:19.349 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff31e2d2e81] 2024-03-24T00:30:19.350 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff31cced1ca] 2024-03-24T00:30:19.350 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 16: clone() 2024-03-24T00:30:19.350 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 2024-03-24T00:30:19.350 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 0> 2024-03-24T00:30:19.074+0000 7ff313946700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:30:19.350 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: in thread 7ff313946700 thread_name:ms_dispatch 2024-03-24T00:30:19.350 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 2024-03-24T00:30:19.350 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:30:19.350 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff31ccf7d20] 2024-03-24T00:30:19.350 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 2: gsignal() 2024-03-24T00:30:19.350 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 3: abort() 2024-03-24T00:30:19.350 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff31df704fb] 2024-03-24T00:30:19.350 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff31df70667] 2024-03-24T00:30:19.350 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561aea807223] 2024-03-24T00:30:19.351 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561aea6815d6] 2024-03-24T00:30:19.351 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561aea681da1] 2024-03-24T00:30:19.351 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x561aea6842d5] 2024-03-24T00:30:19.351 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561aea6d4233] 2024-03-24T00:30:19.351 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561aea6d6197] 2024-03-24T00:30:19.351 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561aea5d511b] 2024-03-24T00:30:19.351 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561aea5d56bc] 2024-03-24T00:30:19.351 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561aea5bf1ff] 2024-03-24T00:30:19.351 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff31e202488] 2024-03-24T00:30:19.351 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 16: (DispatchQueue::entry()+0x51f) [0x7ff31e1ff60f] 2024-03-24T00:30:19.351 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff31e2d2e81] 2024-03-24T00:30:19.351 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff31cced1ca] 2024-03-24T00:30:19.351 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 19: clone() 2024-03-24T00:30:19.351 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:30:19.352 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 2024-03-24T00:30:19.352 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: -1> 2024-03-24T00:30:19.071+0000 7ff313946700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff313946700 time 2024-03-24T00:30:19.070704+0000 2024-03-24T00:30:19.352 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:30:19.352 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 2024-03-24T00:30:19.352 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:30:19.352 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff31df704a1] 2024-03-24T00:30:19.352 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff31df70667] 2024-03-24T00:30:19.352 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561aea807223] 2024-03-24T00:30:19.352 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561aea6815d6] 2024-03-24T00:30:19.352 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561aea681da1] 2024-03-24T00:30:19.352 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x561aea6842d5] 2024-03-24T00:30:19.352 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561aea6d4233] 2024-03-24T00:30:19.352 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561aea6d6197] 2024-03-24T00:30:19.352 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561aea5d511b] 2024-03-24T00:30:19.352 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561aea5d56bc] 2024-03-24T00:30:19.353 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561aea5bf1ff] 2024-03-24T00:30:19.353 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff31e202488] 2024-03-24T00:30:19.353 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 13: (DispatchQueue::entry()+0x51f) [0x7ff31e1ff60f] 2024-03-24T00:30:19.353 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff31e2d2e81] 2024-03-24T00:30:19.353 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff31cced1ca] 2024-03-24T00:30:19.353 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 16: clone() 2024-03-24T00:30:19.353 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 2024-03-24T00:30:19.353 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 0> 2024-03-24T00:30:19.074+0000 7ff313946700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:30:19.353 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: in thread 7ff313946700 thread_name:ms_dispatch 2024-03-24T00:30:19.353 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 2024-03-24T00:30:19.353 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:30:19.353 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff31ccf7d20] 2024-03-24T00:30:19.353 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 2: gsignal() 2024-03-24T00:30:19.353 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 3: abort() 2024-03-24T00:30:19.354 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff31df704fb] 2024-03-24T00:30:19.354 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff31df70667] 2024-03-24T00:30:19.354 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561aea807223] 2024-03-24T00:30:19.354 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561aea6815d6] 2024-03-24T00:30:19.354 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561aea681da1] 2024-03-24T00:30:19.354 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x561aea6842d5] 2024-03-24T00:30:19.354 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561aea6d4233] 2024-03-24T00:30:19.354 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561aea6d6197] 2024-03-24T00:30:19.354 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561aea5d511b] 2024-03-24T00:30:19.354 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561aea5d56bc] 2024-03-24T00:30:19.354 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561aea5bf1ff] 2024-03-24T00:30:19.354 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff31e202488] 2024-03-24T00:30:19.354 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 16: (DispatchQueue::entry()+0x51f) [0x7ff31e1ff60f] 2024-03-24T00:30:19.354 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff31e2d2e81] 2024-03-24T00:30:19.355 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff31cced1ca] 2024-03-24T00:30:19.355 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 19: clone() 2024-03-24T00:30:19.355 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:30:19.355 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[87569]: 2024-03-24T00:30:19.355 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 podman[87757]: 2024-03-24 00:30:19.120513566 +0000 UTC m=+0.020248918 container died 688d8b9fc55a051e98bce217caedab4be492a822002ca6f752ceb31ac209487c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-03-24T00:30:19.695 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 348 sec latency 461286.732 ms 2024-03-24T00:30:19.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:19 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6840/1759489401,v1:172.21.15.78:6841/1759489401] up:rejoin 2024-03-24T00:30:19.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:19 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:rejoin,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:19.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:19 smithi106 ceph-mon[36505]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-03-24T00:30:19.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:19 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:30:19.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:19 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:30:19.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:19 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6840/1759489401,v1:172.21.15.78:6841/1759489401] up:active 2024-03-24T00:30:19.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:19 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:active} 3 up:standby-replay 2 up:standby 2024-03-24T00:30:19.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:19 smithi106 ceph-mon[36505]: pgmap v309: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:30:19.972 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6840/1759489401,v1:172.21.15.78:6841/1759489401] up:rejoin 2024-03-24T00:30:19.972 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:rejoin,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:19.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-mon[32429]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-03-24T00:30:19.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:30:19.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:30:19.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6840/1759489401,v1:172.21.15.78:6841/1759489401] up:active 2024-03-24T00:30:19.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:active} 3 up:standby-replay 2 up:standby 2024-03-24T00:30:19.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:19 smithi078 ceph-mon[32429]: pgmap v309: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:30:19.973 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 podman[87757]: 2024-03-24 00:30:19.74334769 +0000 UTC m=+0.643083088 container remove 688d8b9fc55a051e98bce217caedab4be492a822002ca6f752ceb31ac209487c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-24T00:30:19.973 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:30:20.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:19 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6840/1759489401,v1:172.21.15.78:6841/1759489401] up:rejoin 2024-03-24T00:30:20.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:19 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:rejoin,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:20.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:19 smithi107 ceph-mon[36485]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-03-24T00:30:20.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:19 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:30:20.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:19 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:30:20.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:19 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6840/1759489401,v1:172.21.15.78:6841/1759489401] up:active 2024-03-24T00:30:20.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:19 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:active} 3 up:standby-replay 2 up:standby 2024-03-24T00:30:20.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:19 smithi107 ceph-mon[36485]: pgmap v309: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:30:20.340 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:19 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.a.service: Failed with result 'exit-code'. 2024-03-24T00:30:20.696 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 349 sec latency 462286.834 ms 2024-03-24T00:30:21.406 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:21 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:30:21.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:21 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:30:21.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:21 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:30:21.696 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 350 sec latency 463287.014 ms 2024-03-24T00:30:22.494 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:22 smithi107 ceph-mon[36485]: pgmap v310: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:30:22.495 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:22 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:30:22.495 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:22 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:30:22.495 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:22 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:30:22.577 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:22 smithi106 ceph-mon[36505]: pgmap v310: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:30:22.577 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:22 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:30:22.577 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:22 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:30:22.577 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:22 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:30:22.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:22 smithi078 ceph-mon[32429]: pgmap v310: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:30:22.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:22 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:30:22.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:22 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:30:22.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:22 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:30:22.696 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 351 sec latency 464287.106 ms 2024-03-24T00:30:23.696 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 352 sec latency 465287.187 ms 2024-03-24T00:30:24.696 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 353 sec latency 466287.256 ms 2024-03-24T00:30:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:24 smithi078 ceph-mon[32429]: pgmap v311: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:30:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:24 smithi107 ceph-mon[36485]: pgmap v311: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:30:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:24 smithi106 ceph-mon[36505]: pgmap v311: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:30:25.696 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 354 sec latency 467287.333 ms 2024-03-24T00:30:26.696 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 355 sec latency 468287.436 ms 2024-03-24T00:30:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:26 smithi078 ceph-mon[32429]: pgmap v312: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:30:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:26 smithi107 ceph-mon[36485]: pgmap v312: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:30:26.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:26 smithi106 ceph-mon[36505]: pgmap v312: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:30:27.696 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 356 sec latency 469287.536 ms 2024-03-24T00:30:28.697 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 357 sec latency 470287.666 ms 2024-03-24T00:30:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:28 smithi078 ceph-mon[32429]: pgmap v313: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 23 KiB/s wr, 12 op/s 2024-03-24T00:30:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:28 smithi107 ceph-mon[36485]: pgmap v313: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 23 KiB/s wr, 12 op/s 2024-03-24T00:30:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:28 smithi106 ceph-mon[36505]: pgmap v313: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 23 KiB/s wr, 12 op/s 2024-03-24T00:30:29.697 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 358 sec latency 471287.716 ms 2024-03-24T00:30:29.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:29 smithi078 ceph-mon[32429]: osdmap e126: 12 total, 12 up, 12 in 2024-03-24T00:30:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:29 smithi107 ceph-mon[36485]: osdmap e126: 12 total, 12 up, 12 in 2024-03-24T00:30:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:29 smithi106 ceph-mon[36505]: osdmap e126: 12 total, 12 up, 12 in 2024-03-24T00:30:30.436 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:30 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.a.service: Scheduled restart job, restart counter is at 3. 2024-03-24T00:30:30.436 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:30 smithi078 systemd[1]: Stopped Ceph mds.a for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:30:30.436 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:30 smithi078 systemd[1]: Starting Ceph mds.a for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:30:30.436 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:30 smithi078 podman[87995]: 2024-03-24 00:30:30.229410588 +0000 UTC m=+0.018170681 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:30:30.697 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 359 sec latency 472287.777 ms 2024-03-24T00:30:30.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:30 smithi078 ceph-mon[32429]: pgmap v315: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:30:30.706 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:30 smithi078 podman[87995]: 2024-03-24 00:30:30.436519546 +0000 UTC m=+0.225279633 container create 6b1db3598129b1e3debde48544c73da864efde62d4bfd87710f9b97117246cd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:30:30.706 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:30 smithi078 podman[87995]: 2024-03-24 00:30:30.681039791 +0000 UTC m=+0.469799880 container init 6b1db3598129b1e3debde48544c73da864efde62d4bfd87710f9b97117246cd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-24T00:30:30.706 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:30 smithi078 podman[87995]: 2024-03-24 00:30:30.683642608 +0000 UTC m=+0.472402707 container start 6b1db3598129b1e3debde48544c73da864efde62d4bfd87710f9b97117246cd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-24T00:30:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:30 smithi107 ceph-mon[36485]: pgmap v315: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:30:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:30 smithi106 ceph-mon[36505]: pgmap v315: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:30:31.090 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:30 smithi078 ceph-mds[88014]: -- 172.21.15.78:0/910263986 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556e57a6da40 con 0x556e57a40800 2024-03-24T00:30:31.091 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:30 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: starting mds.a at 2024-03-24T00:30:31.091 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:30 smithi078 bash[87995]: 6b1db3598129b1e3debde48544c73da864efde62d4bfd87710f9b97117246cd7 2024-03-24T00:30:31.091 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:30 smithi078 systemd[1]: Started Ceph mds.a for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:30:31.697 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 360 sec latency 473287.863 ms 2024-03-24T00:30:31.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:31 smithi078 ceph-mon[32429]: daemon mds.a restarted 2024-03-24T00:30:31.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:31 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:30:31.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:31 smithi078 ceph-mon[32429]: osdmap e127: 12 total, 12 up, 12 in 2024-03-24T00:30:31.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:31 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6840/1960340276,v1:172.21.15.78:6841/1960340276] up:boot 2024-03-24T00:30:31.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:31 smithi078 ceph-mon[32429]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-03-24T00:30:31.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:31 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:30:31.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:31 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T00:30:31.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:31 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:31.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:31 smithi107 ceph-mon[36485]: daemon mds.a restarted 2024-03-24T00:30:31.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:31 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:30:31.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:31 smithi107 ceph-mon[36485]: osdmap e127: 12 total, 12 up, 12 in 2024-03-24T00:30:31.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:31 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6840/1960340276,v1:172.21.15.78:6841/1960340276] up:boot 2024-03-24T00:30:31.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:31 smithi107 ceph-mon[36485]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-03-24T00:30:31.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:31 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:30:31.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:31 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T00:30:31.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:31 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:31.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:31 smithi106 ceph-mon[36505]: daemon mds.a restarted 2024-03-24T00:30:31.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:31 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:30:31.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:31 smithi106 ceph-mon[36505]: osdmap e127: 12 total, 12 up, 12 in 2024-03-24T00:30:31.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:31 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6840/1960340276,v1:172.21.15.78:6841/1960340276] up:boot 2024-03-24T00:30:31.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:31 smithi106 ceph-mon[36505]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-03-24T00:30:31.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:31 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:30:31.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:31 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T00:30:31.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:31 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:32.697 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 361 sec latency 474287.964 ms 2024-03-24T00:30:32.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:32 smithi078 ceph-mon[32429]: pgmap v317: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-24T00:30:32.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:32 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6838/2645579087,v1:172.21.15.78:6839/2645579087] up:resolve 2024-03-24T00:30:32.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:32 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:32 smithi107 ceph-mon[36485]: pgmap v317: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-24T00:30:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:32 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6838/2645579087,v1:172.21.15.78:6839/2645579087] up:resolve 2024-03-24T00:30:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:32 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:32 smithi106 ceph-mon[36505]: pgmap v317: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-24T00:30:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:32 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6838/2645579087,v1:172.21.15.78:6839/2645579087] up:resolve 2024-03-24T00:30:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:32 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:33.697 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 362 sec latency 475288.074 ms 2024-03-24T00:30:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:33 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999982 2024-03-24T00:30:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:33 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6838/2645579087,v1:172.21.15.78:6839/2645579087] up:reconnect 2024-03-24T00:30:33.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:33 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:33.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:33 smithi106 ceph-mon[36505]: pgmap v318: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-24T00:30:34.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:33 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999982 2024-03-24T00:30:34.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:33 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6838/2645579087,v1:172.21.15.78:6839/2645579087] up:reconnect 2024-03-24T00:30:34.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:33 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:34.060 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:33 smithi078 ceph-mon[32429]: pgmap v318: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-24T00:30:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:33 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999982 2024-03-24T00:30:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:33 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6838/2645579087,v1:172.21.15.78:6839/2645579087] up:reconnect 2024-03-24T00:30:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:33 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:33 smithi107 ceph-mon[36485]: pgmap v318: 129 pgs: 129 active+clean; 27 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-24T00:30:34.591 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fb6a51c9700 time 2024-03-24T00:30:34.093033+0000 2024-03-24T00:30:34.591 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:30:34.591 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:30:34.591 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fb6af7f34a1] 2024-03-24T00:30:34.592 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb6af7f3667] 2024-03-24T00:30:34.592 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56020000b223] 2024-03-24T00:30:34.592 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5601ffe855d6] 2024-03-24T00:30:34.592 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5601ffe85da1] 2024-03-24T00:30:34.592 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5601ffe882d5] 2024-03-24T00:30:34.592 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5601ffed8233] 2024-03-24T00:30:34.592 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5601ffeda197] 2024-03-24T00:30:34.592 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5601ffdd911b] 2024-03-24T00:30:34.592 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5601ffdd96bc] 2024-03-24T00:30:34.592 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5601ffdc31ff] 2024-03-24T00:30:34.592 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb6afa85488] 2024-03-24T00:30:34.592 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 13: (DispatchQueue::entry()+0x51f) [0x7fb6afa8260f] 2024-03-24T00:30:34.593 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb6afb55e81] 2024-03-24T00:30:34.593 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb6ae5701ca] 2024-03-24T00:30:34.593 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 16: clone() 2024-03-24T00:30:34.593 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 2024-03-24T00:30:34.093+0000 7fb6a51c9700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fb6a51c9700 time 2024-03-24T00:30:34.093033+0000 2024-03-24T00:30:34.593 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:30:34.593 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 2024-03-24T00:30:34.593 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:30:34.593 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fb6af7f34a1] 2024-03-24T00:30:34.593 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb6af7f3667] 2024-03-24T00:30:34.593 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56020000b223] 2024-03-24T00:30:34.593 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5601ffe855d6] 2024-03-24T00:30:34.593 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5601ffe85da1] 2024-03-24T00:30:34.593 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5601ffe882d5] 2024-03-24T00:30:34.594 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5601ffed8233] 2024-03-24T00:30:34.594 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5601ffeda197] 2024-03-24T00:30:34.594 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5601ffdd911b] 2024-03-24T00:30:34.594 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5601ffdd96bc] 2024-03-24T00:30:34.594 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5601ffdc31ff] 2024-03-24T00:30:34.594 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb6afa85488] 2024-03-24T00:30:34.594 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 13: (DispatchQueue::entry()+0x51f) [0x7fb6afa8260f] 2024-03-24T00:30:34.594 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb6afb55e81] 2024-03-24T00:30:34.594 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb6ae5701ca] 2024-03-24T00:30:34.594 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 16: clone() 2024-03-24T00:30:34.594 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 2024-03-24T00:30:34.594 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: *** Caught signal (Aborted) ** 2024-03-24T00:30:34.594 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: in thread 7fb6a51c9700 thread_name:ms_dispatch 2024-03-24T00:30:34.595 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:30:34.595 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb6ae57ad20] 2024-03-24T00:30:34.596 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 2: gsignal() 2024-03-24T00:30:34.596 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 3: abort() 2024-03-24T00:30:34.596 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb6af7f34fb] 2024-03-24T00:30:34.596 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb6af7f3667] 2024-03-24T00:30:34.596 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56020000b223] 2024-03-24T00:30:34.596 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5601ffe855d6] 2024-03-24T00:30:34.596 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5601ffe85da1] 2024-03-24T00:30:34.596 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5601ffe882d5] 2024-03-24T00:30:34.596 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5601ffed8233] 2024-03-24T00:30:34.596 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5601ffeda197] 2024-03-24T00:30:34.596 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5601ffdd911b] 2024-03-24T00:30:34.596 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5601ffdd96bc] 2024-03-24T00:30:34.597 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5601ffdc31ff] 2024-03-24T00:30:34.597 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb6afa85488] 2024-03-24T00:30:34.597 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 16: (DispatchQueue::entry()+0x51f) [0x7fb6afa8260f] 2024-03-24T00:30:34.597 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb6afb55e81] 2024-03-24T00:30:34.597 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb6ae5701ca] 2024-03-24T00:30:34.597 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 19: clone() 2024-03-24T00:30:34.597 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 2024-03-24T00:30:34.095+0000 7fb6a51c9700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:30:34.597 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: in thread 7fb6a51c9700 thread_name:ms_dispatch 2024-03-24T00:30:34.597 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 2024-03-24T00:30:34.597 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:30:34.597 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb6ae57ad20] 2024-03-24T00:30:34.597 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 2: gsignal() 2024-03-24T00:30:34.597 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 3: abort() 2024-03-24T00:30:34.597 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb6af7f34fb] 2024-03-24T00:30:34.598 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb6af7f3667] 2024-03-24T00:30:34.598 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56020000b223] 2024-03-24T00:30:34.598 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5601ffe855d6] 2024-03-24T00:30:34.598 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5601ffe85da1] 2024-03-24T00:30:34.598 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5601ffe882d5] 2024-03-24T00:30:34.598 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5601ffed8233] 2024-03-24T00:30:34.598 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5601ffeda197] 2024-03-24T00:30:34.598 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5601ffdd911b] 2024-03-24T00:30:34.598 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5601ffdd96bc] 2024-03-24T00:30:34.598 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5601ffdc31ff] 2024-03-24T00:30:34.598 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb6afa85488] 2024-03-24T00:30:34.598 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 16: (DispatchQueue::entry()+0x51f) [0x7fb6afa8260f] 2024-03-24T00:30:34.598 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb6afb55e81] 2024-03-24T00:30:34.599 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb6ae5701ca] 2024-03-24T00:30:34.599 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 19: clone() 2024-03-24T00:30:34.599 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:30:34.599 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 2024-03-24T00:30:34.599 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: -1> 2024-03-24T00:30:34.093+0000 7fb6a51c9700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fb6a51c9700 time 2024-03-24T00:30:34.093033+0000 2024-03-24T00:30:34.599 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:30:34.599 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 2024-03-24T00:30:34.599 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:30:34.599 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fb6af7f34a1] 2024-03-24T00:30:34.599 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb6af7f3667] 2024-03-24T00:30:34.599 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56020000b223] 2024-03-24T00:30:34.599 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5601ffe855d6] 2024-03-24T00:30:34.599 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5601ffe85da1] 2024-03-24T00:30:34.599 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5601ffe882d5] 2024-03-24T00:30:34.600 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5601ffed8233] 2024-03-24T00:30:34.600 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5601ffeda197] 2024-03-24T00:30:34.600 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5601ffdd911b] 2024-03-24T00:30:34.600 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5601ffdd96bc] 2024-03-24T00:30:34.600 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5601ffdc31ff] 2024-03-24T00:30:34.600 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb6afa85488] 2024-03-24T00:30:34.600 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 13: (DispatchQueue::entry()+0x51f) [0x7fb6afa8260f] 2024-03-24T00:30:34.600 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb6afb55e81] 2024-03-24T00:30:34.600 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb6ae5701ca] 2024-03-24T00:30:34.600 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 16: clone() 2024-03-24T00:30:34.600 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 2024-03-24T00:30:34.600 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 0> 2024-03-24T00:30:34.095+0000 7fb6a51c9700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:30:34.600 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: in thread 7fb6a51c9700 thread_name:ms_dispatch 2024-03-24T00:30:34.600 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 2024-03-24T00:30:34.601 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:30:34.601 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb6ae57ad20] 2024-03-24T00:30:34.601 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 2: gsignal() 2024-03-24T00:30:34.601 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 3: abort() 2024-03-24T00:30:34.601 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb6af7f34fb] 2024-03-24T00:30:34.601 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb6af7f3667] 2024-03-24T00:30:34.601 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56020000b223] 2024-03-24T00:30:34.601 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5601ffe855d6] 2024-03-24T00:30:34.601 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5601ffe85da1] 2024-03-24T00:30:34.601 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5601ffe882d5] 2024-03-24T00:30:34.601 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5601ffed8233] 2024-03-24T00:30:34.601 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5601ffeda197] 2024-03-24T00:30:34.601 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5601ffdd911b] 2024-03-24T00:30:34.601 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5601ffdd96bc] 2024-03-24T00:30:34.601 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5601ffdc31ff] 2024-03-24T00:30:34.602 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb6afa85488] 2024-03-24T00:30:34.602 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 16: (DispatchQueue::entry()+0x51f) [0x7fb6afa8260f] 2024-03-24T00:30:34.602 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb6afb55e81] 2024-03-24T00:30:34.602 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb6ae5701ca] 2024-03-24T00:30:34.602 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 19: clone() 2024-03-24T00:30:34.602 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:30:34.602 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 2024-03-24T00:30:34.602 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: -1> 2024-03-24T00:30:34.093+0000 7fb6a51c9700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fb6a51c9700 time 2024-03-24T00:30:34.093033+0000 2024-03-24T00:30:34.602 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:30:34.602 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 2024-03-24T00:30:34.602 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:30:34.602 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fb6af7f34a1] 2024-03-24T00:30:34.602 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb6af7f3667] 2024-03-24T00:30:34.602 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56020000b223] 2024-03-24T00:30:34.603 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5601ffe855d6] 2024-03-24T00:30:34.603 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5601ffe85da1] 2024-03-24T00:30:34.603 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5601ffe882d5] 2024-03-24T00:30:34.603 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5601ffed8233] 2024-03-24T00:30:34.603 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5601ffeda197] 2024-03-24T00:30:34.603 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5601ffdd911b] 2024-03-24T00:30:34.603 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5601ffdd96bc] 2024-03-24T00:30:34.603 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5601ffdc31ff] 2024-03-24T00:30:34.603 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb6afa85488] 2024-03-24T00:30:34.603 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 13: (DispatchQueue::entry()+0x51f) [0x7fb6afa8260f] 2024-03-24T00:30:34.603 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb6afb55e81] 2024-03-24T00:30:34.603 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb6ae5701ca] 2024-03-24T00:30:34.603 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 16: clone() 2024-03-24T00:30:34.603 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 2024-03-24T00:30:34.603 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 0> 2024-03-24T00:30:34.095+0000 7fb6a51c9700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:30:34.604 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: in thread 7fb6a51c9700 thread_name:ms_dispatch 2024-03-24T00:30:34.604 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 2024-03-24T00:30:34.604 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:30:34.604 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb6ae57ad20] 2024-03-24T00:30:34.604 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 2: gsignal() 2024-03-24T00:30:34.604 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 3: abort() 2024-03-24T00:30:34.604 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb6af7f34fb] 2024-03-24T00:30:34.604 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fb6af7f3667] 2024-03-24T00:30:34.604 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56020000b223] 2024-03-24T00:30:34.604 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5601ffe855d6] 2024-03-24T00:30:34.604 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5601ffe85da1] 2024-03-24T00:30:34.604 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5601ffe882d5] 2024-03-24T00:30:34.604 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5601ffed8233] 2024-03-24T00:30:34.604 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5601ffeda197] 2024-03-24T00:30:34.604 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5601ffdd911b] 2024-03-24T00:30:34.605 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5601ffdd96bc] 2024-03-24T00:30:34.605 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5601ffdc31ff] 2024-03-24T00:30:34.605 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fb6afa85488] 2024-03-24T00:30:34.605 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 16: (DispatchQueue::entry()+0x51f) [0x7fb6afa8260f] 2024-03-24T00:30:34.605 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb6afb55e81] 2024-03-24T00:30:34.605 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb6ae5701ca] 2024-03-24T00:30:34.605 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 19: clone() 2024-03-24T00:30:34.605 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:30:34.605 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[87719]: 2024-03-24T00:30:34.605 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:34 smithi078 podman[88048]: 2024-03-24 00:30:34.150850255 +0000 UTC m=+0.020397996 container died 9dc8d561f02c4445c55f0e855cd97a386b83cd77686b9d317c0b8848a6d802b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:30:34.697 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 363 sec latency 476288.108 ms 2024-03-24T00:30:34.876 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6838/2645579087,v1:172.21.15.78:6839/2645579087] up:rejoin 2024-03-24T00:30:34.876 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:34.876 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-mon[32429]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-24T00:30:34.876 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:30:34.876 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:30:34.876 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6838/2645579087,v1:172.21.15.78:6839/2645579087] up:active 2024-03-24T00:30:34.876 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 2 up:standby 2024-03-24T00:30:34.876 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:34 smithi078 ceph-mon[32429]: osdmap e128: 12 total, 12 up, 12 in 2024-03-24T00:30:35.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:34 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6838/2645579087,v1:172.21.15.78:6839/2645579087] up:rejoin 2024-03-24T00:30:35.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:34 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:35.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:34 smithi107 ceph-mon[36485]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-24T00:30:35.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:34 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:30:35.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:34 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:30:35.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:34 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6838/2645579087,v1:172.21.15.78:6839/2645579087] up:active 2024-03-24T00:30:35.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:34 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 2 up:standby 2024-03-24T00:30:35.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:34 smithi107 ceph-mon[36485]: osdmap e128: 12 total, 12 up, 12 in 2024-03-24T00:30:35.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:34 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6838/2645579087,v1:172.21.15.78:6839/2645579087] up:rejoin 2024-03-24T00:30:35.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:34 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:35.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:34 smithi106 ceph-mon[36505]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-24T00:30:35.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:34 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:30:35.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:34 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:30:35.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:34 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6838/2645579087,v1:172.21.15.78:6839/2645579087] up:active 2024-03-24T00:30:35.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:34 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 2 up:standby 2024-03-24T00:30:35.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:34 smithi106 ceph-mon[36505]: osdmap e128: 12 total, 12 up, 12 in 2024-03-24T00:30:35.387 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:35 smithi078 podman[88048]: 2024-03-24 00:30:35.136579633 +0000 UTC m=+1.006127358 container remove 9dc8d561f02c4445c55f0e855cd97a386b83cd77686b9d317c0b8848a6d802b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:30:35.387 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:35 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:30:35.697 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 364 sec latency 477288.206 ms 2024-03-24T00:30:35.742 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:35 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.g.service: Failed with result 'exit-code'. 2024-03-24T00:30:36.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:35 smithi078 ceph-mon[32429]: pgmap v320: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 590 B/s wr, 16 op/s 2024-03-24T00:30:36.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:35 smithi107 ceph-mon[36485]: pgmap v320: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 590 B/s wr, 16 op/s 2024-03-24T00:30:36.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:35 smithi106 ceph-mon[36505]: pgmap v320: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 590 B/s wr, 16 op/s 2024-03-24T00:30:36.697 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 365 sec latency 478288.272 ms 2024-03-24T00:30:37.697 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 366 sec latency 479288.370 ms 2024-03-24T00:30:38.697 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 367 sec latency 480288.441 ms 2024-03-24T00:30:38.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:38 smithi078 ceph-mon[32429]: pgmap v321: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 511 B/s wr, 14 op/s 2024-03-24T00:30:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:38 smithi107 ceph-mon[36485]: pgmap v321: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 511 B/s wr, 14 op/s 2024-03-24T00:30:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:38 smithi106 ceph-mon[36505]: pgmap v321: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 511 B/s wr, 14 op/s 2024-03-24T00:30:39.697 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 368 sec latency 481288.546 ms 2024-03-24T00:30:40.697 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 369 sec latency 482288.699 ms 2024-03-24T00:30:40.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:40 smithi078 ceph-mon[32429]: pgmap v322: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 353 B/s wr, 9 op/s 2024-03-24T00:30:40.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:40 smithi107 ceph-mon[36485]: pgmap v322: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 353 B/s wr, 9 op/s 2024-03-24T00:30:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:40 smithi106 ceph-mon[36505]: pgmap v322: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 353 B/s wr, 9 op/s 2024-03-24T00:30:41.698 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 370 sec latency 483288.816 ms 2024-03-24T00:30:42.698 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 371 sec latency 484288.933 ms 2024-03-24T00:30:42.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:42 smithi078 ceph-mon[32429]: pgmap v323: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-24T00:30:42.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:42 smithi107 ceph-mon[36485]: pgmap v323: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-24T00:30:42.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:42 smithi106 ceph-mon[36505]: pgmap v323: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-24T00:30:43.698 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 372 sec latency 485288.990 ms 2024-03-24T00:30:44.698 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 373 sec latency 486289.083 ms 2024-03-24T00:30:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:44 smithi078 ceph-mon[32429]: pgmap v324: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-24T00:30:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:44 smithi107 ceph-mon[36485]: pgmap v324: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-24T00:30:44.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:44 smithi106 ceph-mon[36505]: pgmap v324: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 307 B/s wr, 11 op/s 2024-03-24T00:30:45.698 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 374 sec latency 487289.131 ms 2024-03-24T00:30:45.954 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:45 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.g.service: Scheduled restart job, restart counter is at 2. 2024-03-24T00:30:45.954 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:45 smithi078 systemd[1]: Stopped Ceph mds.g for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:30:45.954 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:45 smithi078 systemd[1]: Starting Ceph mds.g for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:30:45.954 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:45 smithi078 podman[88136]: 2024-03-24 00:30:45.728591633 +0000 UTC m=+0.017938031 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:30:45.954 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:45 smithi078 podman[88136]: 2024-03-24 00:30:45.837165213 +0000 UTC m=+0.126511633 container create 5b249f096534ce03f8bcb8fec8fd073bd83d92e0ee40e616222d4af9a1fc1c14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:30:46.340 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:46 smithi078 podman[88136]: 2024-03-24 00:30:46.065299427 +0000 UTC m=+0.354645818 container init 5b249f096534ce03f8bcb8fec8fd073bd83d92e0ee40e616222d4af9a1fc1c14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:30:46.341 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:46 smithi078 podman[88136]: 2024-03-24 00:30:46.067985019 +0000 UTC m=+0.357331394 container start 5b249f096534ce03f8bcb8fec8fd073bd83d92e0ee40e616222d4af9a1fc1c14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-24T00:30:46.341 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:46 smithi078 ceph-mds[88154]: -- 172.21.15.78:0/958897721 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560fcf977a40 con 0x560fcf94b000 2024-03-24T00:30:46.341 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:46 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: starting mds.g at 2024-03-24T00:30:46.341 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:46 smithi078 bash[88136]: 5b249f096534ce03f8bcb8fec8fd073bd83d92e0ee40e616222d4af9a1fc1c14 2024-03-24T00:30:46.341 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:30:46 smithi078 systemd[1]: Started Ceph mds.g for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:30:46.698 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 375 sec latency 488289.227 ms 2024-03-24T00:30:46.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:46 smithi078 ceph-mon[32429]: pgmap v325: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 270 B/s wr, 12 op/s 2024-03-24T00:30:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:46 smithi078 ceph-mon[32429]: daemon mds.g restarted 2024-03-24T00:30:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:46 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:30:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:46 smithi078 ceph-mon[32429]: osdmap e129: 12 total, 12 up, 12 in 2024-03-24T00:30:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:46 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6838/4117918862,v1:172.21.15.78:6839/4117918862] up:boot 2024-03-24T00:30:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:46 smithi078 ceph-mon[32429]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-03-24T00:30:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:46 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:30:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:46 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:30:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:46 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:replay,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:46.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:46 smithi107 ceph-mon[36485]: pgmap v325: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 270 B/s wr, 12 op/s 2024-03-24T00:30:46.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:46 smithi107 ceph-mon[36485]: daemon mds.g restarted 2024-03-24T00:30:46.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:46 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:30:46.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:46 smithi107 ceph-mon[36485]: osdmap e129: 12 total, 12 up, 12 in 2024-03-24T00:30:46.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:46 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6838/4117918862,v1:172.21.15.78:6839/4117918862] up:boot 2024-03-24T00:30:46.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:46 smithi107 ceph-mon[36485]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-03-24T00:30:46.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:46 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:30:46.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:46 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:30:46.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:46 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:replay,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:46 smithi106 ceph-mon[36505]: pgmap v325: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 270 B/s wr, 12 op/s 2024-03-24T00:30:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:46 smithi106 ceph-mon[36505]: daemon mds.g restarted 2024-03-24T00:30:46.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:46 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:30:46.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:46 smithi106 ceph-mon[36505]: osdmap e129: 12 total, 12 up, 12 in 2024-03-24T00:30:46.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:46 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6838/4117918862,v1:172.21.15.78:6839/4117918862] up:boot 2024-03-24T00:30:46.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:46 smithi106 ceph-mon[36505]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-03-24T00:30:46.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:46 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:30:46.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:46 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:30:46.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:46 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:replay,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:47.698 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 376 sec latency 489289.580 ms 2024-03-24T00:30:48.698 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 377 sec latency 490289.657 ms 2024-03-24T00:30:48.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:48 smithi078 ceph-mon[32429]: pgmap v327: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:30:48.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:48 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6840/1960340276,v1:172.21.15.78:6841/1960340276] up:resolve 2024-03-24T00:30:48.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:48 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:resolve,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:48 smithi107 ceph-mon[36485]: pgmap v327: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:30:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:48 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6840/1960340276,v1:172.21.15.78:6841/1960340276] up:resolve 2024-03-24T00:30:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:48 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:resolve,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:48 smithi106 ceph-mon[36505]: pgmap v327: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:30:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:48 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6840/1960340276,v1:172.21.15.78:6841/1960340276] up:resolve 2024-03-24T00:30:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:48 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:resolve,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:49.699 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 378 sec latency 491289.757 ms 2024-03-24T00:30:49.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:49 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999982 2024-03-24T00:30:49.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:49 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6840/1960340276,v1:172.21.15.78:6841/1960340276] up:reconnect 2024-03-24T00:30:49.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:49 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:reconnect,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:49.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:49 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999982 2024-03-24T00:30:49.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:49 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6840/1960340276,v1:172.21.15.78:6841/1960340276] up:reconnect 2024-03-24T00:30:49.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:49 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:reconnect,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:49.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:49 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999982 2024-03-24T00:30:49.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:49 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6840/1960340276,v1:172.21.15.78:6841/1960340276] up:reconnect 2024-03-24T00:30:49.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:49 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:reconnect,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:50.699 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 379 sec latency 492289.802 ms 2024-03-24T00:30:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-mon[32429]: pgmap v328: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:30:50.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6840/1960340276,v1:172.21.15.78:6841/1960340276] up:rejoin 2024-03-24T00:30:50.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:rejoin,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:50.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-mon[32429]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-03-24T00:30:50.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:30:50.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:30:50.841 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f8e3318f700 time 2024-03-24T00:30:50.501838+0000 2024-03-24T00:30:50.842 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:30:50.842 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:30:50.842 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f8e3d7b94a1] 2024-03-24T00:30:50.842 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f8e3d7b9667] 2024-03-24T00:30:50.842 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556e569fe223] 2024-03-24T00:30:50.842 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556e568785d6] 2024-03-24T00:30:50.842 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556e56878da1] 2024-03-24T00:30:50.842 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556e5687b2d5] 2024-03-24T00:30:50.842 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556e568cb233] 2024-03-24T00:30:50.842 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556e568cd197] 2024-03-24T00:30:50.842 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556e567cc11b] 2024-03-24T00:30:50.843 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556e567cc6bc] 2024-03-24T00:30:50.843 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556e567b61ff] 2024-03-24T00:30:50.843 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f8e3da4b488] 2024-03-24T00:30:50.843 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 13: (DispatchQueue::entry()+0x51f) [0x7f8e3da4860f] 2024-03-24T00:30:50.843 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8e3db1be81] 2024-03-24T00:30:50.843 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f8e3c5361ca] 2024-03-24T00:30:50.843 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 16: clone() 2024-03-24T00:30:50.843 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: *** Caught signal (Aborted) ** 2024-03-24T00:30:50.843 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: in thread 7f8e3318f700 thread_name:ms_dispatch 2024-03-24T00:30:50.843 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 2024-03-24T00:30:50.502+0000 7f8e3318f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f8e3318f700 time 2024-03-24T00:30:50.501838+0000 2024-03-24T00:30:50.843 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:30:50.843 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 2024-03-24T00:30:50.843 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:30:50.843 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f8e3d7b94a1] 2024-03-24T00:30:50.844 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f8e3d7b9667] 2024-03-24T00:30:50.844 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556e569fe223] 2024-03-24T00:30:50.844 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556e568785d6] 2024-03-24T00:30:50.844 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556e56878da1] 2024-03-24T00:30:50.844 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556e5687b2d5] 2024-03-24T00:30:50.844 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556e568cb233] 2024-03-24T00:30:50.844 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556e568cd197] 2024-03-24T00:30:50.844 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556e567cc11b] 2024-03-24T00:30:50.844 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556e567cc6bc] 2024-03-24T00:30:50.844 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556e567b61ff] 2024-03-24T00:30:50.844 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f8e3da4b488] 2024-03-24T00:30:50.844 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 13: (DispatchQueue::entry()+0x51f) [0x7f8e3da4860f] 2024-03-24T00:30:50.844 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8e3db1be81] 2024-03-24T00:30:50.844 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f8e3c5361ca] 2024-03-24T00:30:50.845 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 16: clone() 2024-03-24T00:30:50.845 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 2024-03-24T00:30:50.845 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:30:50.845 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8e3c540d20] 2024-03-24T00:30:50.845 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 2: gsignal() 2024-03-24T00:30:50.845 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 3: abort() 2024-03-24T00:30:50.845 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f8e3d7b94fb] 2024-03-24T00:30:50.845 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f8e3d7b9667] 2024-03-24T00:30:50.845 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556e569fe223] 2024-03-24T00:30:50.845 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556e568785d6] 2024-03-24T00:30:50.845 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556e56878da1] 2024-03-24T00:30:50.845 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556e5687b2d5] 2024-03-24T00:30:50.845 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556e568cb233] 2024-03-24T00:30:50.845 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556e568cd197] 2024-03-24T00:30:50.845 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556e567cc11b] 2024-03-24T00:30:50.846 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556e567cc6bc] 2024-03-24T00:30:50.846 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556e567b61ff] 2024-03-24T00:30:50.846 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f8e3da4b488] 2024-03-24T00:30:50.846 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 16: (DispatchQueue::entry()+0x51f) [0x7f8e3da4860f] 2024-03-24T00:30:50.846 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8e3db1be81] 2024-03-24T00:30:50.846 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f8e3c5361ca] 2024-03-24T00:30:50.846 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 19: clone() 2024-03-24T00:30:50.846 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 2024-03-24T00:30:50.505+0000 7f8e3318f700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:30:50.846 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: in thread 7f8e3318f700 thread_name:ms_dispatch 2024-03-24T00:30:50.846 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 2024-03-24T00:30:50.846 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:30:50.846 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8e3c540d20] 2024-03-24T00:30:50.846 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 2: gsignal() 2024-03-24T00:30:50.846 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 3: abort() 2024-03-24T00:30:50.851 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f8e3d7b94fb] 2024-03-24T00:30:50.851 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f8e3d7b9667] 2024-03-24T00:30:50.852 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:50 smithi107 ceph-mon[36485]: pgmap v328: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:30:50.852 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:50 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6840/1960340276,v1:172.21.15.78:6841/1960340276] up:rejoin 2024-03-24T00:30:50.852 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:50 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:rejoin,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:50.852 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:50 smithi107 ceph-mon[36485]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-03-24T00:30:50.852 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:50 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:30:50.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:50 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:30:50.854 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556e569fe223] 2024-03-24T00:30:50.854 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556e568785d6] 2024-03-24T00:30:50.855 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556e56878da1] 2024-03-24T00:30:50.855 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556e5687b2d5] 2024-03-24T00:30:50.855 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556e568cb233] 2024-03-24T00:30:50.855 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556e568cd197] 2024-03-24T00:30:50.855 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556e567cc11b] 2024-03-24T00:30:50.855 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556e567cc6bc] 2024-03-24T00:30:50.856 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556e567b61ff] 2024-03-24T00:30:50.856 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f8e3da4b488] 2024-03-24T00:30:50.856 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 16: (DispatchQueue::entry()+0x51f) [0x7f8e3da4860f] 2024-03-24T00:30:50.856 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8e3db1be81] 2024-03-24T00:30:50.856 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f8e3c5361ca] 2024-03-24T00:30:50.856 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 19: clone() 2024-03-24T00:30:50.856 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:30:50.856 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 2024-03-24T00:30:50.856 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: -1> 2024-03-24T00:30:50.502+0000 7f8e3318f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f8e3318f700 time 2024-03-24T00:30:50.501838+0000 2024-03-24T00:30:50.857 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:30:50.857 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 2024-03-24T00:30:50.857 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:30:50.857 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f8e3d7b94a1] 2024-03-24T00:30:50.857 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f8e3d7b9667] 2024-03-24T00:30:50.857 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556e569fe223] 2024-03-24T00:30:50.857 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556e568785d6] 2024-03-24T00:30:50.857 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556e56878da1] 2024-03-24T00:30:50.857 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556e5687b2d5] 2024-03-24T00:30:50.857 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556e568cb233] 2024-03-24T00:30:50.857 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556e568cd197] 2024-03-24T00:30:50.857 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556e567cc11b] 2024-03-24T00:30:50.857 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556e567cc6bc] 2024-03-24T00:30:50.857 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556e567b61ff] 2024-03-24T00:30:50.857 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f8e3da4b488] 2024-03-24T00:30:50.858 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 13: (DispatchQueue::entry()+0x51f) [0x7f8e3da4860f] 2024-03-24T00:30:50.858 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8e3db1be81] 2024-03-24T00:30:50.858 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f8e3c5361ca] 2024-03-24T00:30:50.858 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 16: clone() 2024-03-24T00:30:50.858 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 2024-03-24T00:30:50.858 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 0> 2024-03-24T00:30:50.505+0000 7f8e3318f700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:30:50.858 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: in thread 7f8e3318f700 thread_name:ms_dispatch 2024-03-24T00:30:50.858 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 2024-03-24T00:30:50.858 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:30:50.858 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8e3c540d20] 2024-03-24T00:30:50.858 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 2: gsignal() 2024-03-24T00:30:50.858 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 3: abort() 2024-03-24T00:30:50.858 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f8e3d7b94fb] 2024-03-24T00:30:50.859 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f8e3d7b9667] 2024-03-24T00:30:50.859 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556e569fe223] 2024-03-24T00:30:50.859 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556e568785d6] 2024-03-24T00:30:50.859 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556e56878da1] 2024-03-24T00:30:50.859 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556e5687b2d5] 2024-03-24T00:30:50.859 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556e568cb233] 2024-03-24T00:30:50.859 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556e568cd197] 2024-03-24T00:30:50.859 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556e567cc11b] 2024-03-24T00:30:50.859 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556e567cc6bc] 2024-03-24T00:30:50.859 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556e567b61ff] 2024-03-24T00:30:50.859 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f8e3da4b488] 2024-03-24T00:30:50.859 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 16: (DispatchQueue::entry()+0x51f) [0x7f8e3da4860f] 2024-03-24T00:30:50.859 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8e3db1be81] 2024-03-24T00:30:50.860 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f8e3c5361ca] 2024-03-24T00:30:50.860 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 19: clone() 2024-03-24T00:30:50.860 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:30:50.860 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 2024-03-24T00:30:50.860 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: -1> 2024-03-24T00:30:50.502+0000 7f8e3318f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f8e3318f700 time 2024-03-24T00:30:50.501838+0000 2024-03-24T00:30:50.860 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:30:50.860 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 2024-03-24T00:30:50.860 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:30:50.860 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f8e3d7b94a1] 2024-03-24T00:30:50.860 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f8e3d7b9667] 2024-03-24T00:30:50.860 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556e569fe223] 2024-03-24T00:30:50.860 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556e568785d6] 2024-03-24T00:30:50.860 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556e56878da1] 2024-03-24T00:30:50.860 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556e5687b2d5] 2024-03-24T00:30:50.861 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556e568cb233] 2024-03-24T00:30:50.861 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556e568cd197] 2024-03-24T00:30:50.861 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556e567cc11b] 2024-03-24T00:30:50.861 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556e567cc6bc] 2024-03-24T00:30:50.861 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556e567b61ff] 2024-03-24T00:30:50.861 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f8e3da4b488] 2024-03-24T00:30:50.861 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 13: (DispatchQueue::entry()+0x51f) [0x7f8e3da4860f] 2024-03-24T00:30:50.861 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8e3db1be81] 2024-03-24T00:30:50.861 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f8e3c5361ca] 2024-03-24T00:30:50.861 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 16: clone() 2024-03-24T00:30:50.861 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 2024-03-24T00:30:50.861 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 0> 2024-03-24T00:30:50.505+0000 7f8e3318f700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:30:50.861 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: in thread 7f8e3318f700 thread_name:ms_dispatch 2024-03-24T00:30:50.862 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 2024-03-24T00:30:50.862 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:30:50.862 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8e3c540d20] 2024-03-24T00:30:50.862 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 2: gsignal() 2024-03-24T00:30:50.862 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 3: abort() 2024-03-24T00:30:50.862 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f8e3d7b94fb] 2024-03-24T00:30:50.862 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f8e3d7b9667] 2024-03-24T00:30:50.862 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556e569fe223] 2024-03-24T00:30:50.862 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556e568785d6] 2024-03-24T00:30:50.862 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556e56878da1] 2024-03-24T00:30:50.862 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556e5687b2d5] 2024-03-24T00:30:50.862 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556e568cb233] 2024-03-24T00:30:50.862 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556e568cd197] 2024-03-24T00:30:50.863 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556e567cc11b] 2024-03-24T00:30:50.863 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556e567cc6bc] 2024-03-24T00:30:50.863 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556e567b61ff] 2024-03-24T00:30:50.863 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f8e3da4b488] 2024-03-24T00:30:50.863 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 16: (DispatchQueue::entry()+0x51f) [0x7f8e3da4860f] 2024-03-24T00:30:50.863 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8e3db1be81] 2024-03-24T00:30:50.863 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f8e3c5361ca] 2024-03-24T00:30:50.863 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 19: clone() 2024-03-24T00:30:50.863 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:30:50.863 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88010]: 2024-03-24T00:30:50.863 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:50 smithi078 podman[88188]: 2024-03-24 00:30:50.560881982 +0000 UTC m=+0.021077263 container died 6b1db3598129b1e3debde48544c73da864efde62d4bfd87710f9b97117246cd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-24T00:30:50.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:50 smithi106 ceph-mon[36505]: pgmap v328: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:30:50.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:50 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6840/1960340276,v1:172.21.15.78:6841/1960340276] up:rejoin 2024-03-24T00:30:50.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:50 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:rejoin,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:50.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:50 smithi106 ceph-mon[36505]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-03-24T00:30:50.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:50 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:30:50.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:50 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:30:51.501 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:51 smithi078 podman[88188]: 2024-03-24 00:30:51.21192144 +0000 UTC m=+0.672116769 container remove 6b1db3598129b1e3debde48544c73da864efde62d4bfd87710f9b97117246cd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-24T00:30:51.501 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:51 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:30:51.501 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:30:51 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.a.service: Failed with result 'exit-code'. 2024-03-24T00:30:51.699 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 380 sec latency 493289.834 ms 2024-03-24T00:30:51.772 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:51 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6840/1960340276,v1:172.21.15.78:6841/1960340276] up:active 2024-03-24T00:30:51.772 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:51 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:51.772 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:51 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:active} 3 up:standby-replay 2 up:standby 2024-03-24T00:30:51.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:51 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6840/1960340276,v1:172.21.15.78:6841/1960340276] up:active 2024-03-24T00:30:51.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:51 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:51.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:51 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:active} 3 up:standby-replay 2 up:standby 2024-03-24T00:30:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:51 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6840/1960340276,v1:172.21.15.78:6841/1960340276] up:active 2024-03-24T00:30:51.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:51 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:30:51.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:51 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:active} 3 up:standby-replay 2 up:standby 2024-03-24T00:30:52.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:52 smithi078 ceph-mon[32429]: pgmap v329: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:30:52.699 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 381 sec latency 494289.910 ms 2024-03-24T00:30:52.803 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:52 smithi106 ceph-mon[36505]: pgmap v329: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:30:52.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:52 smithi107 ceph-mon[36485]: pgmap v329: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:30:53.699 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 382 sec latency 495290.011 ms 2024-03-24T00:30:54.699 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 383 sec latency 496290.102 ms 2024-03-24T00:30:54.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:54 smithi078 ceph-mon[32429]: pgmap v330: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:30:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:54 smithi107 ceph-mon[36485]: pgmap v330: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:30:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:54 smithi106 ceph-mon[36505]: pgmap v330: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:30:55.699 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 384 sec latency 497290.148 ms 2024-03-24T00:30:56.699 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 385 sec latency 498290.280 ms 2024-03-24T00:30:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:56 smithi078 ceph-mon[32429]: pgmap v331: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:30:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:56 smithi107 ceph-mon[36485]: pgmap v331: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:30:56.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:56 smithi106 ceph-mon[36505]: pgmap v331: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:30:57.699 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 386 sec latency 499290.386 ms 2024-03-24T00:30:58.699 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 387 sec latency 500290.480 ms 2024-03-24T00:30:58.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:30:58 smithi078 ceph-mon[32429]: pgmap v332: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 181 B/s wr, 9 op/s 2024-03-24T00:30:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:30:58 smithi107 ceph-mon[36485]: pgmap v332: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 181 B/s wr, 9 op/s 2024-03-24T00:30:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:30:58 smithi106 ceph-mon[36505]: pgmap v332: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 181 B/s wr, 9 op/s 2024-03-24T00:30:59.699 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 388 sec latency 501290.587 ms 2024-03-24T00:31:00.700 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 389 sec latency 502290.711 ms 2024-03-24T00:31:00.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:00 smithi078 ceph-mon[32429]: pgmap v333: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-24T00:31:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:00 smithi107 ceph-mon[36485]: pgmap v333: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-24T00:31:00.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:00 smithi106 ceph-mon[36505]: pgmap v333: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-24T00:31:01.700 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 390 sec latency 503290.756 ms 2024-03-24T00:31:01.862 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:01 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.a.service: Scheduled restart job, restart counter is at 4. 2024-03-24T00:31:01.862 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:01 smithi078 systemd[1]: Stopped Ceph mds.a for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:31:01.862 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:01 smithi078 systemd[1]: Starting Ceph mds.a for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:31:01.862 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:01 smithi078 podman[88286]: 2024-03-24 00:31:01.723570333 +0000 UTC m=+0.017801578 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:31:01.862 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:01 smithi078 podman[88286]: 2024-03-24 00:31:01.862763264 +0000 UTC m=+0.156994493 container create d87ce699142dba3025be8c2d89a445967e029557c6d62dde4fac14a613ad4649 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:31:02.173 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:02 smithi078 podman[88286]: 2024-03-24 00:31:02.082008846 +0000 UTC m=+0.376240086 container init d87ce699142dba3025be8c2d89a445967e029557c6d62dde4fac14a613ad4649 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-24T00:31:02.173 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:02 smithi078 podman[88286]: 2024-03-24 00:31:02.084477105 +0000 UTC m=+0.378708344 container start d87ce699142dba3025be8c2d89a445967e029557c6d62dde4fac14a613ad4649 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-24T00:31:02.173 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:02 smithi078 ceph-mds[88306]: -- 172.21.15.78:0/1150436337 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e37c3aba40 con 0x55e37c37f000 2024-03-24T00:31:02.173 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:02 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: starting mds.a at 2024-03-24T00:31:02.517 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:02 smithi078 bash[88286]: d87ce699142dba3025be8c2d89a445967e029557c6d62dde4fac14a613ad4649 2024-03-24T00:31:02.517 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:02 smithi078 systemd[1]: Started Ceph mds.a for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:31:02.700 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 391 sec latency 504290.800 ms 2024-03-24T00:31:02.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:02 smithi078 ceph-mon[32429]: pgmap v334: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-24T00:31:02.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:02 smithi078 ceph-mon[32429]: daemon mds.a restarted 2024-03-24T00:31:02.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:02 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:31:02.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:02 smithi078 ceph-mon[32429]: osdmap e130: 12 total, 12 up, 12 in 2024-03-24T00:31:02.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:02 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6840/1658072731,v1:172.21.15.78:6841/1658072731] up:boot 2024-03-24T00:31:02.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:02 smithi078 ceph-mon[32429]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-03-24T00:31:02.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:02 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:31:02.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:02 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T00:31:02.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:02 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:31:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:02 smithi107 ceph-mon[36485]: pgmap v334: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-24T00:31:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:02 smithi107 ceph-mon[36485]: daemon mds.a restarted 2024-03-24T00:31:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:02 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:31:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:02 smithi107 ceph-mon[36485]: osdmap e130: 12 total, 12 up, 12 in 2024-03-24T00:31:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:02 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6840/1658072731,v1:172.21.15.78:6841/1658072731] up:boot 2024-03-24T00:31:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:02 smithi107 ceph-mon[36485]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-03-24T00:31:02.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:02 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:31:02.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:02 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T00:31:02.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:02 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:31:02.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:02 smithi106 ceph-mon[36505]: pgmap v334: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-24T00:31:02.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:02 smithi106 ceph-mon[36505]: daemon mds.a restarted 2024-03-24T00:31:02.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:02 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:31:02.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:02 smithi106 ceph-mon[36505]: osdmap e130: 12 total, 12 up, 12 in 2024-03-24T00:31:02.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:02 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6840/1658072731,v1:172.21.15.78:6841/1658072731] up:boot 2024-03-24T00:31:02.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:02 smithi106 ceph-mon[36505]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-03-24T00:31:02.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:02 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:31:02.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:02 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T00:31:02.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:02 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:31:03.700 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 392 sec latency 505290.915 ms 2024-03-24T00:31:04.700 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 393 sec latency 506290.982 ms 2024-03-24T00:31:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:04 smithi078 ceph-mon[32429]: pgmap v336: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 10 op/s 2024-03-24T00:31:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:04 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6838/4117918862,v1:172.21.15.78:6839/4117918862] up:resolve 2024-03-24T00:31:04.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:04 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:31:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:04 smithi107 ceph-mon[36485]: pgmap v336: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 10 op/s 2024-03-24T00:31:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:04 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6838/4117918862,v1:172.21.15.78:6839/4117918862] up:resolve 2024-03-24T00:31:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:04 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:31:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:04 smithi106 ceph-mon[36505]: pgmap v336: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 10 op/s 2024-03-24T00:31:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:04 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6838/4117918862,v1:172.21.15.78:6839/4117918862] up:resolve 2024-03-24T00:31:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:04 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:31:05.700 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 394 sec latency 507291.079 ms 2024-03-24T00:31:05.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:05 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999982 2024-03-24T00:31:05.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:05 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6838/4117918862,v1:172.21.15.78:6839/4117918862] up:reconnect 2024-03-24T00:31:05.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:05 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:31:05.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:05 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999982 2024-03-24T00:31:05.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:05 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6838/4117918862,v1:172.21.15.78:6839/4117918862] up:reconnect 2024-03-24T00:31:05.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:05 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:31:05.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:05 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999982 2024-03-24T00:31:05.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:05 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6838/4117918862,v1:172.21.15.78:6839/4117918862] up:reconnect 2024-03-24T00:31:05.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:05 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:31:06.700 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 395 sec latency 508291.180 ms 2024-03-24T00:31:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-mon[32429]: pgmap v337: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:31:06.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6838/4117918862,v1:172.21.15.78:6839/4117918862] up:rejoin 2024-03-24T00:31:06.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:31:06.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-mon[32429]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-24T00:31:06.841 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f41a5b01700 time 2024-03-24T00:31:06.555417+0000 2024-03-24T00:31:06.841 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:31:06.841 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:31:06.841 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f41b012b4a1] 2024-03-24T00:31:06.842 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f41b012b667] 2024-03-24T00:31:06.842 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560fcd3aa223] 2024-03-24T00:31:06.842 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560fcd2245d6] 2024-03-24T00:31:06.842 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560fcd224da1] 2024-03-24T00:31:06.842 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560fcd2272d5] 2024-03-24T00:31:06.842 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560fcd277233] 2024-03-24T00:31:06.842 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560fcd279197] 2024-03-24T00:31:06.842 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560fcd17811b] 2024-03-24T00:31:06.842 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560fcd1786bc] 2024-03-24T00:31:06.842 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560fcd1621ff] 2024-03-24T00:31:06.842 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f41b03bd488] 2024-03-24T00:31:06.842 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 13: (DispatchQueue::entry()+0x51f) [0x7f41b03ba60f] 2024-03-24T00:31:06.843 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f41b048de81] 2024-03-24T00:31:06.843 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f41aeea81ca] 2024-03-24T00:31:06.843 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 16: clone() 2024-03-24T00:31:06.843 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 2024-03-24T00:31:06.555+0000 7f41a5b01700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f41a5b01700 time 2024-03-24T00:31:06.555417+0000 2024-03-24T00:31:06.843 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:31:06.843 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 2024-03-24T00:31:06.843 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:31:06.843 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f41b012b4a1] 2024-03-24T00:31:06.843 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f41b012b667] 2024-03-24T00:31:06.843 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560fcd3aa223] 2024-03-24T00:31:06.843 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560fcd2245d6] 2024-03-24T00:31:06.843 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560fcd224da1] 2024-03-24T00:31:06.843 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560fcd2272d5] 2024-03-24T00:31:06.844 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560fcd277233] 2024-03-24T00:31:06.844 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560fcd279197] 2024-03-24T00:31:06.844 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560fcd17811b] 2024-03-24T00:31:06.844 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560fcd1786bc] 2024-03-24T00:31:06.844 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560fcd1621ff] 2024-03-24T00:31:06.844 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f41b03bd488] 2024-03-24T00:31:06.844 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 13: (DispatchQueue::entry()+0x51f) [0x7f41b03ba60f] 2024-03-24T00:31:06.844 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f41b048de81] 2024-03-24T00:31:06.844 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f41aeea81ca] 2024-03-24T00:31:06.844 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 16: clone() 2024-03-24T00:31:06.844 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 2024-03-24T00:31:06.844 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: *** Caught signal (Aborted) ** 2024-03-24T00:31:06.844 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: in thread 7f41a5b01700 thread_name:ms_dispatch 2024-03-24T00:31:06.845 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:31:06.845 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f41aeeb2d20] 2024-03-24T00:31:06.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:06 smithi107 ceph-mon[36485]: pgmap v337: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:31:06.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:06 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6838/4117918862,v1:172.21.15.78:6839/4117918862] up:rejoin 2024-03-24T00:31:06.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:06 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:31:06.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:06 smithi107 ceph-mon[36485]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-24T00:31:06.846 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 2: gsignal() 2024-03-24T00:31:06.847 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 3: abort() 2024-03-24T00:31:06.847 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f41b012b4fb] 2024-03-24T00:31:06.847 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f41b012b667] 2024-03-24T00:31:06.847 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560fcd3aa223] 2024-03-24T00:31:06.847 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560fcd2245d6] 2024-03-24T00:31:06.847 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560fcd224da1] 2024-03-24T00:31:06.847 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560fcd2272d5] 2024-03-24T00:31:06.847 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560fcd277233] 2024-03-24T00:31:06.847 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560fcd279197] 2024-03-24T00:31:06.847 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560fcd17811b] 2024-03-24T00:31:06.847 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560fcd1786bc] 2024-03-24T00:31:06.847 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560fcd1621ff] 2024-03-24T00:31:06.848 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f41b03bd488] 2024-03-24T00:31:06.848 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 16: (DispatchQueue::entry()+0x51f) [0x7f41b03ba60f] 2024-03-24T00:31:06.848 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f41b048de81] 2024-03-24T00:31:06.848 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f41aeea81ca] 2024-03-24T00:31:06.848 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 19: clone() 2024-03-24T00:31:06.848 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 2024-03-24T00:31:06.557+0000 7f41a5b01700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:31:06.848 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: in thread 7f41a5b01700 thread_name:ms_dispatch 2024-03-24T00:31:06.848 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 2024-03-24T00:31:06.848 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:31:06.848 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f41aeeb2d20] 2024-03-24T00:31:06.848 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 2: gsignal() 2024-03-24T00:31:06.848 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 3: abort() 2024-03-24T00:31:06.848 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f41b012b4fb] 2024-03-24T00:31:06.848 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f41b012b667] 2024-03-24T00:31:06.849 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560fcd3aa223] 2024-03-24T00:31:06.849 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560fcd2245d6] 2024-03-24T00:31:06.849 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560fcd224da1] 2024-03-24T00:31:06.849 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560fcd2272d5] 2024-03-24T00:31:06.849 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560fcd277233] 2024-03-24T00:31:06.849 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560fcd279197] 2024-03-24T00:31:06.849 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560fcd17811b] 2024-03-24T00:31:06.849 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560fcd1786bc] 2024-03-24T00:31:06.849 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560fcd1621ff] 2024-03-24T00:31:06.849 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f41b03bd488] 2024-03-24T00:31:06.849 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 16: (DispatchQueue::entry()+0x51f) [0x7f41b03ba60f] 2024-03-24T00:31:06.849 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f41b048de81] 2024-03-24T00:31:06.849 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f41aeea81ca] 2024-03-24T00:31:06.849 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 19: clone() 2024-03-24T00:31:06.849 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:31:06.850 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 2024-03-24T00:31:06.850 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: -1> 2024-03-24T00:31:06.555+0000 7f41a5b01700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f41a5b01700 time 2024-03-24T00:31:06.555417+0000 2024-03-24T00:31:06.850 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:31:06.850 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 2024-03-24T00:31:06.850 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:31:06.850 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f41b012b4a1] 2024-03-24T00:31:06.850 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f41b012b667] 2024-03-24T00:31:06.850 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560fcd3aa223] 2024-03-24T00:31:06.850 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560fcd2245d6] 2024-03-24T00:31:06.850 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560fcd224da1] 2024-03-24T00:31:06.850 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560fcd2272d5] 2024-03-24T00:31:06.850 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560fcd277233] 2024-03-24T00:31:06.850 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560fcd279197] 2024-03-24T00:31:06.851 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560fcd17811b] 2024-03-24T00:31:06.851 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560fcd1786bc] 2024-03-24T00:31:06.851 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560fcd1621ff] 2024-03-24T00:31:06.851 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f41b03bd488] 2024-03-24T00:31:06.851 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 13: (DispatchQueue::entry()+0x51f) [0x7f41b03ba60f] 2024-03-24T00:31:06.851 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f41b048de81] 2024-03-24T00:31:06.851 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f41aeea81ca] 2024-03-24T00:31:06.851 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 16: clone() 2024-03-24T00:31:06.851 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 2024-03-24T00:31:06.851 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 0> 2024-03-24T00:31:06.557+0000 7f41a5b01700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:31:06.851 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: in thread 7f41a5b01700 thread_name:ms_dispatch 2024-03-24T00:31:06.851 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 2024-03-24T00:31:06.851 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:31:06.851 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f41aeeb2d20] 2024-03-24T00:31:06.852 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 2: gsignal() 2024-03-24T00:31:06.852 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 3: abort() 2024-03-24T00:31:06.852 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f41b012b4fb] 2024-03-24T00:31:06.852 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f41b012b667] 2024-03-24T00:31:06.852 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560fcd3aa223] 2024-03-24T00:31:06.852 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560fcd2245d6] 2024-03-24T00:31:06.852 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560fcd224da1] 2024-03-24T00:31:06.852 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560fcd2272d5] 2024-03-24T00:31:06.852 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560fcd277233] 2024-03-24T00:31:06.852 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560fcd279197] 2024-03-24T00:31:06.852 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560fcd17811b] 2024-03-24T00:31:06.852 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560fcd1786bc] 2024-03-24T00:31:06.852 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560fcd1621ff] 2024-03-24T00:31:06.852 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f41b03bd488] 2024-03-24T00:31:06.852 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 16: (DispatchQueue::entry()+0x51f) [0x7f41b03ba60f] 2024-03-24T00:31:06.853 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f41b048de81] 2024-03-24T00:31:06.853 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f41aeea81ca] 2024-03-24T00:31:06.853 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 19: clone() 2024-03-24T00:31:06.853 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:31:06.853 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 2024-03-24T00:31:06.853 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: -1> 2024-03-24T00:31:06.555+0000 7f41a5b01700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f41a5b01700 time 2024-03-24T00:31:06.555417+0000 2024-03-24T00:31:06.853 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:31:06.853 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 2024-03-24T00:31:06.853 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:31:06.853 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f41b012b4a1] 2024-03-24T00:31:06.853 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f41b012b667] 2024-03-24T00:31:06.853 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560fcd3aa223] 2024-03-24T00:31:06.853 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560fcd2245d6] 2024-03-24T00:31:06.853 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560fcd224da1] 2024-03-24T00:31:06.854 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560fcd2272d5] 2024-03-24T00:31:06.854 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560fcd277233] 2024-03-24T00:31:06.854 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560fcd279197] 2024-03-24T00:31:06.854 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560fcd17811b] 2024-03-24T00:31:06.854 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560fcd1786bc] 2024-03-24T00:31:06.854 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560fcd1621ff] 2024-03-24T00:31:06.854 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f41b03bd488] 2024-03-24T00:31:06.854 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 13: (DispatchQueue::entry()+0x51f) [0x7f41b03ba60f] 2024-03-24T00:31:06.854 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f41b048de81] 2024-03-24T00:31:06.854 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f41aeea81ca] 2024-03-24T00:31:06.854 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 16: clone() 2024-03-24T00:31:06.854 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 2024-03-24T00:31:06.854 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 0> 2024-03-24T00:31:06.557+0000 7f41a5b01700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:31:06.854 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: in thread 7f41a5b01700 thread_name:ms_dispatch 2024-03-24T00:31:06.855 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 2024-03-24T00:31:06.855 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:31:06.855 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f41aeeb2d20] 2024-03-24T00:31:06.855 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 2: gsignal() 2024-03-24T00:31:06.855 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 3: abort() 2024-03-24T00:31:06.855 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f41b012b4fb] 2024-03-24T00:31:06.855 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f41b012b667] 2024-03-24T00:31:06.855 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560fcd3aa223] 2024-03-24T00:31:06.855 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x560fcd2245d6] 2024-03-24T00:31:06.855 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x560fcd224da1] 2024-03-24T00:31:06.855 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x560fcd2272d5] 2024-03-24T00:31:06.855 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560fcd277233] 2024-03-24T00:31:06.856 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560fcd279197] 2024-03-24T00:31:06.856 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x560fcd17811b] 2024-03-24T00:31:06.856 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560fcd1786bc] 2024-03-24T00:31:06.856 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x560fcd1621ff] 2024-03-24T00:31:06.856 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f41b03bd488] 2024-03-24T00:31:06.856 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 16: (DispatchQueue::entry()+0x51f) [0x7f41b03ba60f] 2024-03-24T00:31:06.856 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f41b048de81] 2024-03-24T00:31:06.856 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f41aeea81ca] 2024-03-24T00:31:06.856 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 19: clone() 2024-03-24T00:31:06.856 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:31:06.856 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88150]: 2024-03-24T00:31:06.856 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:06 smithi078 podman[88340]: 2024-03-24 00:31:06.614387717 +0000 UTC m=+0.021874312 container died 5b249f096534ce03f8bcb8fec8fd073bd83d92e0ee40e616222d4af9a1fc1c14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True) 2024-03-24T00:31:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:06 smithi106 ceph-mon[36505]: pgmap v337: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:31:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:06 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6838/4117918862,v1:172.21.15.78:6839/4117918862] up:rejoin 2024-03-24T00:31:06.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:06 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:31:06.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:06 smithi106 ceph-mon[36505]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-24T00:31:07.549 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:07 smithi078 podman[88340]: 2024-03-24 00:31:07.245507974 +0000 UTC m=+0.652994557 container remove 5b249f096534ce03f8bcb8fec8fd073bd83d92e0ee40e616222d4af9a1fc1c14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-24T00:31:07.549 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:07 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:31:07.549 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:07 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.g.service: Failed with result 'exit-code'. 2024-03-24T00:31:07.700 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 396 sec latency 509291.304 ms 2024-03-24T00:31:07.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:07 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:31:07.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:07 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:31:07.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:07 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6838/4117918862,v1:172.21.15.78:6839/4117918862] up:active 2024-03-24T00:31:07.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:07 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:31:07.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:07 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 2 up:standby 2024-03-24T00:31:07.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:07 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:31:07.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:07 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:31:07.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:07 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6838/4117918862,v1:172.21.15.78:6839/4117918862] up:active 2024-03-24T00:31:07.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:07 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:31:07.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:07 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 2 up:standby 2024-03-24T00:31:07.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:07 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:31:07.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:07 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:31:07.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:07 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6838/4117918862,v1:172.21.15.78:6839/4117918862] up:active 2024-03-24T00:31:07.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:07 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:31:07.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:07 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 2 up:standby 2024-03-24T00:31:08.701 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 397 sec latency 510291.411 ms 2024-03-24T00:31:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:08 smithi078 ceph-mon[32429]: pgmap v338: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:31:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:08 smithi107 ceph-mon[36485]: pgmap v338: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:31:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:08 smithi106 ceph-mon[36505]: pgmap v338: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:31:09.700 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 398 sec latency 511291.478 ms 2024-03-24T00:31:10.700 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 399 sec latency 512291.616 ms 2024-03-24T00:31:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:10 smithi078 ceph-mon[32429]: pgmap v339: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:31:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:10 smithi107 ceph-mon[36485]: pgmap v339: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:31:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:10 smithi106 ceph-mon[36505]: pgmap v339: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:31:11.701 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 400 sec latency 513291.723 ms 2024-03-24T00:31:12.701 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 401 sec latency 514291.870 ms 2024-03-24T00:31:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:12 smithi078 ceph-mon[32429]: pgmap v340: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 29 op/s 2024-03-24T00:31:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:12 smithi107 ceph-mon[36485]: pgmap v340: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 29 op/s 2024-03-24T00:31:12.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:12 smithi106 ceph-mon[36505]: pgmap v340: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 29 op/s 2024-03-24T00:31:13.701 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 402 sec latency 515291.965 ms 2024-03-24T00:31:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:13 smithi078 ceph-mon[32429]: pgmap v341: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-03-24T00:31:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:13 smithi107 ceph-mon[36485]: pgmap v341: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-03-24T00:31:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:13 smithi106 ceph-mon[36505]: pgmap v341: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-03-24T00:31:14.701 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 403 sec latency 516292.022 ms 2024-03-24T00:31:15.701 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 404 sec latency 517292.129 ms 2024-03-24T00:31:16.701 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 405 sec latency 518292.252 ms 2024-03-24T00:31:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:16 smithi078 ceph-mon[32429]: pgmap v342: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-03-24T00:31:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:16 smithi107 ceph-mon[36485]: pgmap v342: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-03-24T00:31:16.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:16 smithi106 ceph-mon[36505]: pgmap v342: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-03-24T00:31:17.701 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 406 sec latency 519292.299 ms 2024-03-24T00:31:17.955 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:17 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.g.service: Scheduled restart job, restart counter is at 3. 2024-03-24T00:31:17.955 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:17 smithi078 systemd[1]: Stopped Ceph mds.g for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:31:17.955 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:17 smithi078 systemd[1]: Starting Ceph mds.g for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:31:17.955 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:17 smithi078 podman[88436]: 2024-03-24 00:31:17.728991406 +0000 UTC m=+0.019802288 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:31:17.955 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:17 smithi078 podman[88436]: 2024-03-24 00:31:17.837922045 +0000 UTC m=+0.128732897 container create 17254032ff2c518d22d66fd6d5c0d9419c033d8a838699743dc5c62aca52ad81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:31:18.475 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:18 smithi078 podman[88436]: 2024-03-24 00:31:18.099076803 +0000 UTC m=+0.389887676 container init 17254032ff2c518d22d66fd6d5c0d9419c033d8a838699743dc5c62aca52ad81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-24T00:31:18.475 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:18 smithi078 podman[88436]: 2024-03-24 00:31:18.101885282 +0000 UTC m=+0.392696136 container start 17254032ff2c518d22d66fd6d5c0d9419c033d8a838699743dc5c62aca52ad81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-24T00:31:18.475 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:18 smithi078 ceph-mds[88455]: -- 172.21.15.78:0/935959218 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f25f38fa40 con 0x55f25f363000 2024-03-24T00:31:18.475 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: starting mds.g at 2024-03-24T00:31:18.475 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:18 smithi078 bash[88436]: 17254032ff2c518d22d66fd6d5c0d9419c033d8a838699743dc5c62aca52ad81 2024-03-24T00:31:18.475 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:18 smithi078 systemd[1]: Started Ceph mds.g for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:31:18.701 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 407 sec latency 520292.417 ms 2024-03-24T00:31:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:18 smithi078 ceph-mon[32429]: pgmap v343: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T00:31:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:18 smithi078 ceph-mon[32429]: daemon mds.g restarted 2024-03-24T00:31:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:18 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:31:18.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:18 smithi078 ceph-mon[32429]: osdmap e131: 12 total, 12 up, 12 in 2024-03-24T00:31:18.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:18 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6838/4176145760,v1:172.21.15.78:6839/4176145760] up:boot 2024-03-24T00:31:18.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:18 smithi078 ceph-mon[32429]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-03-24T00:31:18.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:18 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:31:18.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:18 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:31:18.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:18 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:replay,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:31:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:18 smithi107 ceph-mon[36485]: pgmap v343: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T00:31:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:18 smithi107 ceph-mon[36485]: daemon mds.g restarted 2024-03-24T00:31:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:18 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:31:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:18 smithi107 ceph-mon[36485]: osdmap e131: 12 total, 12 up, 12 in 2024-03-24T00:31:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:18 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6838/4176145760,v1:172.21.15.78:6839/4176145760] up:boot 2024-03-24T00:31:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:18 smithi107 ceph-mon[36485]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-03-24T00:31:18.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:18 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:31:18.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:18 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:31:18.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:18 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:replay,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:31:18.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:18 smithi106 ceph-mon[36505]: pgmap v343: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T00:31:18.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:18 smithi106 ceph-mon[36505]: daemon mds.g restarted 2024-03-24T00:31:18.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:18 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:31:18.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:18 smithi106 ceph-mon[36505]: osdmap e131: 12 total, 12 up, 12 in 2024-03-24T00:31:18.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:18 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6838/4176145760,v1:172.21.15.78:6839/4176145760] up:boot 2024-03-24T00:31:18.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:18 smithi106 ceph-mon[36505]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-03-24T00:31:18.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:18 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:31:18.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:18 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:31:18.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:18 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:replay,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:31:19.701 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 408 sec latency 521292.479 ms 2024-03-24T00:31:20.701 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 409 sec latency 522292.565 ms 2024-03-24T00:31:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:20 smithi078 ceph-mon[32429]: pgmap v345: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:31:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:20 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6840/1658072731,v1:172.21.15.78:6841/1658072731] up:resolve 2024-03-24T00:31:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:20 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:resolve,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:31:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:20 smithi107 ceph-mon[36485]: pgmap v345: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:31:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:20 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6840/1658072731,v1:172.21.15.78:6841/1658072731] up:resolve 2024-03-24T00:31:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:20 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:resolve,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:31:20.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:20 smithi106 ceph-mon[36505]: pgmap v345: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:31:20.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:20 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6840/1658072731,v1:172.21.15.78:6841/1658072731] up:resolve 2024-03-24T00:31:20.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:20 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:resolve,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:31:21.702 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 410 sec latency 523292.696 ms 2024-03-24T00:31:21.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:21 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:31:21.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:21 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6840/1658072731,v1:172.21.15.78:6841/1658072731] up:reconnect 2024-03-24T00:31:21.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:21 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:reconnect,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:31:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:21 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:31:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:21 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6840/1658072731,v1:172.21.15.78:6841/1658072731] up:reconnect 2024-03-24T00:31:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:21 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:reconnect,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:31:21.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:21 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:31:21.977 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:21 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6840/1658072731,v1:172.21.15.78:6841/1658072731] up:reconnect 2024-03-24T00:31:21.977 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:21 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:reconnect,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:31:22.702 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 411 sec latency 524292.801 ms 2024-03-24T00:31:22.796 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:22 smithi107 ceph-mon[36485]: pgmap v346: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-24T00:31:22.796 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:22 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6840/1658072731,v1:172.21.15.78:6841/1658072731] up:rejoin 2024-03-24T00:31:22.796 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:22 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:rejoin,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:31:22.796 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:22 smithi107 ceph-mon[36485]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-03-24T00:31:22.796 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:22 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:31:22.803 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:22 smithi106 ceph-mon[36505]: pgmap v346: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-24T00:31:22.803 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:22 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6840/1658072731,v1:172.21.15.78:6841/1658072731] up:rejoin 2024-03-24T00:31:22.803 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:22 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:rejoin,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:31:22.803 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:22 smithi106 ceph-mon[36505]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-03-24T00:31:22.803 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:22 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:31:22.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-mon[32429]: pgmap v346: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-24T00:31:22.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6840/1658072731,v1:172.21.15.78:6841/1658072731] up:rejoin 2024-03-24T00:31:22.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=a=up:rejoin,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:31:22.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-mon[32429]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-03-24T00:31:22.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:31:22.841 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f923b2a1700 time 2024-03-24T00:31:22.522504+0000 2024-03-24T00:31:22.841 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:31:22.842 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:31:22.842 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f92458cb4a1] 2024-03-24T00:31:22.842 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f92458cb667] 2024-03-24T00:31:22.842 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e379ffd223] 2024-03-24T00:31:22.842 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e379e775d6] 2024-03-24T00:31:22.842 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e379e77da1] 2024-03-24T00:31:22.842 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e379e7a2d5] 2024-03-24T00:31:22.842 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e379eca233] 2024-03-24T00:31:22.842 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e379ecc197] 2024-03-24T00:31:22.842 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e379dcb11b] 2024-03-24T00:31:22.842 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e379dcb6bc] 2024-03-24T00:31:22.842 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e379db51ff] 2024-03-24T00:31:22.843 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f9245b5d488] 2024-03-24T00:31:22.843 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 13: (DispatchQueue::entry()+0x51f) [0x7f9245b5a60f] 2024-03-24T00:31:22.843 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9245c2de81] 2024-03-24T00:31:22.843 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f92446481ca] 2024-03-24T00:31:22.843 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 16: clone() 2024-03-24T00:31:22.843 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: *** Caught signal (Aborted) ** 2024-03-24T00:31:22.843 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: in thread 7f923b2a1700 thread_name:ms_dispatch 2024-03-24T00:31:22.843 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 2024-03-24T00:31:22.523+0000 7f923b2a1700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f923b2a1700 time 2024-03-24T00:31:22.522504+0000 2024-03-24T00:31:22.843 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:31:22.843 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 2024-03-24T00:31:22.843 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:31:22.843 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f92458cb4a1] 2024-03-24T00:31:22.843 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f92458cb667] 2024-03-24T00:31:22.844 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e379ffd223] 2024-03-24T00:31:22.844 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e379e775d6] 2024-03-24T00:31:22.844 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e379e77da1] 2024-03-24T00:31:22.844 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e379e7a2d5] 2024-03-24T00:31:22.844 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e379eca233] 2024-03-24T00:31:22.844 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e379ecc197] 2024-03-24T00:31:22.844 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e379dcb11b] 2024-03-24T00:31:22.844 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e379dcb6bc] 2024-03-24T00:31:22.844 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e379db51ff] 2024-03-24T00:31:22.844 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f9245b5d488] 2024-03-24T00:31:22.844 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 13: (DispatchQueue::entry()+0x51f) [0x7f9245b5a60f] 2024-03-24T00:31:22.844 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9245c2de81] 2024-03-24T00:31:22.844 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f92446481ca] 2024-03-24T00:31:22.844 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 16: clone() 2024-03-24T00:31:22.844 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 2024-03-24T00:31:22.845 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:31:22.845 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9244652d20] 2024-03-24T00:31:22.845 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 2: gsignal() 2024-03-24T00:31:22.845 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 3: abort() 2024-03-24T00:31:22.845 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f92458cb4fb] 2024-03-24T00:31:22.845 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f92458cb667] 2024-03-24T00:31:22.845 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e379ffd223] 2024-03-24T00:31:22.845 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e379e775d6] 2024-03-24T00:31:22.845 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e379e77da1] 2024-03-24T00:31:22.845 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e379e7a2d5] 2024-03-24T00:31:22.845 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e379eca233] 2024-03-24T00:31:22.845 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e379ecc197] 2024-03-24T00:31:22.845 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e379dcb11b] 2024-03-24T00:31:22.845 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e379dcb6bc] 2024-03-24T00:31:22.846 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e379db51ff] 2024-03-24T00:31:22.846 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f9245b5d488] 2024-03-24T00:31:22.846 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 16: (DispatchQueue::entry()+0x51f) [0x7f9245b5a60f] 2024-03-24T00:31:22.846 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9245c2de81] 2024-03-24T00:31:22.846 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f92446481ca] 2024-03-24T00:31:22.846 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 19: clone() 2024-03-24T00:31:22.846 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 2024-03-24T00:31:22.529+0000 7f923b2a1700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:31:22.846 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: in thread 7f923b2a1700 thread_name:ms_dispatch 2024-03-24T00:31:22.846 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 2024-03-24T00:31:22.846 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:31:22.846 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9244652d20] 2024-03-24T00:31:22.846 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 2: gsignal() 2024-03-24T00:31:22.846 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 3: abort() 2024-03-24T00:31:22.846 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f92458cb4fb] 2024-03-24T00:31:22.847 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f92458cb667] 2024-03-24T00:31:22.847 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e379ffd223] 2024-03-24T00:31:22.848 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e379e775d6] 2024-03-24T00:31:22.848 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e379e77da1] 2024-03-24T00:31:22.848 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e379e7a2d5] 2024-03-24T00:31:22.848 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e379eca233] 2024-03-24T00:31:22.848 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e379ecc197] 2024-03-24T00:31:22.848 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e379dcb11b] 2024-03-24T00:31:22.848 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e379dcb6bc] 2024-03-24T00:31:22.848 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e379db51ff] 2024-03-24T00:31:22.848 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f9245b5d488] 2024-03-24T00:31:22.848 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 16: (DispatchQueue::entry()+0x51f) [0x7f9245b5a60f] 2024-03-24T00:31:22.848 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9245c2de81] 2024-03-24T00:31:22.848 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f92446481ca] 2024-03-24T00:31:22.849 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 19: clone() 2024-03-24T00:31:22.849 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:31:22.849 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 2024-03-24T00:31:22.849 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: -1> 2024-03-24T00:31:22.523+0000 7f923b2a1700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f923b2a1700 time 2024-03-24T00:31:22.522504+0000 2024-03-24T00:31:22.849 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:31:22.849 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 2024-03-24T00:31:22.849 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:31:22.849 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f92458cb4a1] 2024-03-24T00:31:22.849 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f92458cb667] 2024-03-24T00:31:22.849 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e379ffd223] 2024-03-24T00:31:22.849 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e379e775d6] 2024-03-24T00:31:22.849 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e379e77da1] 2024-03-24T00:31:22.849 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e379e7a2d5] 2024-03-24T00:31:22.849 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e379eca233] 2024-03-24T00:31:22.850 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e379ecc197] 2024-03-24T00:31:22.850 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e379dcb11b] 2024-03-24T00:31:22.850 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e379dcb6bc] 2024-03-24T00:31:22.850 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e379db51ff] 2024-03-24T00:31:22.850 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f9245b5d488] 2024-03-24T00:31:22.850 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 13: (DispatchQueue::entry()+0x51f) [0x7f9245b5a60f] 2024-03-24T00:31:22.850 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9245c2de81] 2024-03-24T00:31:22.850 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f92446481ca] 2024-03-24T00:31:22.850 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 16: clone() 2024-03-24T00:31:22.850 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 2024-03-24T00:31:22.850 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 0> 2024-03-24T00:31:22.529+0000 7f923b2a1700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:31:22.850 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: in thread 7f923b2a1700 thread_name:ms_dispatch 2024-03-24T00:31:22.850 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 2024-03-24T00:31:22.850 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:31:22.850 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9244652d20] 2024-03-24T00:31:22.851 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 2: gsignal() 2024-03-24T00:31:22.851 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 3: abort() 2024-03-24T00:31:22.851 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f92458cb4fb] 2024-03-24T00:31:22.851 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f92458cb667] 2024-03-24T00:31:22.851 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e379ffd223] 2024-03-24T00:31:22.851 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e379e775d6] 2024-03-24T00:31:22.851 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e379e77da1] 2024-03-24T00:31:22.851 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e379e7a2d5] 2024-03-24T00:31:22.851 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e379eca233] 2024-03-24T00:31:22.851 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e379ecc197] 2024-03-24T00:31:22.851 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e379dcb11b] 2024-03-24T00:31:22.851 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e379dcb6bc] 2024-03-24T00:31:22.851 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e379db51ff] 2024-03-24T00:31:22.852 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f9245b5d488] 2024-03-24T00:31:22.852 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 16: (DispatchQueue::entry()+0x51f) [0x7f9245b5a60f] 2024-03-24T00:31:22.852 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9245c2de81] 2024-03-24T00:31:22.852 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f92446481ca] 2024-03-24T00:31:22.852 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 19: clone() 2024-03-24T00:31:22.852 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:31:22.852 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 2024-03-24T00:31:22.852 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: -1> 2024-03-24T00:31:22.523+0000 7f923b2a1700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f923b2a1700 time 2024-03-24T00:31:22.522504+0000 2024-03-24T00:31:22.852 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:31:22.852 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 2024-03-24T00:31:22.852 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:31:22.852 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f92458cb4a1] 2024-03-24T00:31:22.852 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f92458cb667] 2024-03-24T00:31:22.852 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e379ffd223] 2024-03-24T00:31:22.853 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e379e775d6] 2024-03-24T00:31:22.853 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e379e77da1] 2024-03-24T00:31:22.853 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e379e7a2d5] 2024-03-24T00:31:22.853 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e379eca233] 2024-03-24T00:31:22.853 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e379ecc197] 2024-03-24T00:31:22.853 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e379dcb11b] 2024-03-24T00:31:22.853 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e379dcb6bc] 2024-03-24T00:31:22.853 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e379db51ff] 2024-03-24T00:31:22.853 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f9245b5d488] 2024-03-24T00:31:22.853 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 13: (DispatchQueue::entry()+0x51f) [0x7f9245b5a60f] 2024-03-24T00:31:22.853 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9245c2de81] 2024-03-24T00:31:22.853 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f92446481ca] 2024-03-24T00:31:22.853 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 16: clone() 2024-03-24T00:31:22.853 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 2024-03-24T00:31:22.854 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 0> 2024-03-24T00:31:22.529+0000 7f923b2a1700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:31:22.854 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: in thread 7f923b2a1700 thread_name:ms_dispatch 2024-03-24T00:31:22.854 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 2024-03-24T00:31:22.854 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:31:22.854 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9244652d20] 2024-03-24T00:31:22.854 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 2: gsignal() 2024-03-24T00:31:22.854 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 3: abort() 2024-03-24T00:31:22.854 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f92458cb4fb] 2024-03-24T00:31:22.854 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f92458cb667] 2024-03-24T00:31:22.854 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e379ffd223] 2024-03-24T00:31:22.854 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e379e775d6] 2024-03-24T00:31:22.854 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e379e77da1] 2024-03-24T00:31:22.854 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55e379e7a2d5] 2024-03-24T00:31:22.854 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e379eca233] 2024-03-24T00:31:22.855 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e379ecc197] 2024-03-24T00:31:22.855 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e379dcb11b] 2024-03-24T00:31:22.855 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e379dcb6bc] 2024-03-24T00:31:22.855 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e379db51ff] 2024-03-24T00:31:22.855 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f9245b5d488] 2024-03-24T00:31:22.855 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 16: (DispatchQueue::entry()+0x51f) [0x7f9245b5a60f] 2024-03-24T00:31:22.855 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9245c2de81] 2024-03-24T00:31:22.855 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f92446481ca] 2024-03-24T00:31:22.855 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 19: clone() 2024-03-24T00:31:22.855 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:31:22.855 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a[88302]: 2024-03-24T00:31:22.855 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:22 smithi078 podman[88620]: 2024-03-24 00:31:22.588473243 +0000 UTC m=+0.021931790 container died d87ce699142dba3025be8c2d89a445967e029557c6d62dde4fac14a613ad4649 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-24T00:31:23.172 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:23 smithi078 podman[88620]: 2024-03-24 00:31:23.153817915 +0000 UTC m=+0.587276468 container remove d87ce699142dba3025be8c2d89a445967e029557c6d62dde4fac14a613ad4649 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-a, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-24T00:31:23.172 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:23 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:31:23.702 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 412 sec latency 525292.827 ms 2024-03-24T00:31:23.772 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:23 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:31:23.772 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:23 smithi078 ceph-mon[32429]: Cluster is now healthy 2024-03-24T00:31:23.772 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:23 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6840/1658072731,v1:172.21.15.78:6841/1658072731] up:active 2024-03-24T00:31:23.772 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:23 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:31:23.772 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:23 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:active} 3 up:standby-replay 2 up:standby 2024-03-24T00:31:23.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:23 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:31:23.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:23 smithi107 ceph-mon[36485]: Cluster is now healthy 2024-03-24T00:31:23.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:23 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6840/1658072731,v1:172.21.15.78:6841/1658072731] up:active 2024-03-24T00:31:23.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:23 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:31:23.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:23 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:active} 3 up:standby-replay 2 up:standby 2024-03-24T00:31:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:23 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:31:23.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:23 smithi106 ceph-mon[36505]: Cluster is now healthy 2024-03-24T00:31:23.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:23 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6840/1658072731,v1:172.21.15.78:6841/1658072731] up:active 2024-03-24T00:31:23.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:23 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:active} 2 up:standby-replay 3 up:standby 2024-03-24T00:31:23.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:23 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:active} 3 up:standby-replay 2 up:standby 2024-03-24T00:31:24.340 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:24 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.a.service: Failed with result 'exit-code'. 2024-03-24T00:31:24.702 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 413 sec latency 526292.853 ms 2024-03-24T00:31:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:24 smithi078 ceph-mon[32429]: pgmap v347: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-24T00:31:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:24 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:31:24.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:24 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:31:24.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:24 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:31:24.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:24 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:31:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:24 smithi107 ceph-mon[36485]: pgmap v347: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-24T00:31:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:24 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:31:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:24 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:31:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:24 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:31:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:24 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:31:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:24 smithi106 ceph-mon[36505]: pgmap v347: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-24T00:31:24.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:24 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:31:24.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:24 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:31:24.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:24 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:31:24.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:24 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:31:25.702 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 414 sec latency 527292.933 ms 2024-03-24T00:31:25.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:31:25.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:31:25.743 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:25.743 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:25.743 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:25.743 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:25.743 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:25 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T00:31:25.743 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:25 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T00:31:25.743 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:25.743 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:25.743 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:25.743 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:25.743 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:25 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T00:31:25.744 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:25 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T00:31:25.744 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:25.744 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:25.744 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:25.744 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:25.744 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:25 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T00:31:25.744 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:25 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T00:31:25.744 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:25.744 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:31:25.744 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:25 smithi078 ceph-mon[32429]: pgmap v348: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T00:31:25.744 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:31:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:31:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:31:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:25 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T00:31:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:25 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T00:31:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:25.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:25 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T00:31:25.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:25 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T00:31:25.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:25.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:25.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:25.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:25.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:25 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T00:31:25.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:25 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T00:31:25.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:25.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:31:25.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:25 smithi106 ceph-mon[36505]: pgmap v348: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T00:31:25.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:31:26.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:31:26.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:31:26.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:26.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:26.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:26.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:26.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:25 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T00:31:26.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:25 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T00:31:26.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:26.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:26.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:26.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:26.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:25 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T00:31:26.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:25 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T00:31:26.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:26.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:26.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:26.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:31:26.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:25 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T00:31:26.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:25 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T00:31:26.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:26.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:31:26.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:25 smithi107 ceph-mon[36485]: pgmap v348: 129 pgs: 129 active+clean; 27 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T00:31:26.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:31:26.702 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 415 sec latency 528292.966 ms 2024-03-24T00:31:26.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:26 smithi106 ceph-mon[36505]: Health check failed: 5 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-24T00:31:26.926 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:26 smithi078 ceph-mon[32429]: Health check failed: 5 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-24T00:31:27.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:26 smithi107 ceph-mon[36485]: Health check failed: 5 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-24T00:31:27.702 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 416 sec latency 529293.420 ms 2024-03-24T00:31:27.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:27 smithi078 ceph-mon[32429]: pgmap v349: 129 pgs: 129 active+clean; 27 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:31:27.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:27 smithi106 ceph-mon[36505]: pgmap v349: 129 pgs: 129 active+clean; 27 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:31:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:27 smithi107 ceph-mon[36485]: pgmap v349: 129 pgs: 129 active+clean; 27 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:31:28.702 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 417 sec latency 530293.518 ms 2024-03-24T00:31:29.702 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 418 sec latency 531293.637 ms 2024-03-24T00:31:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:29 smithi106 ceph-mon[36505]: osdmap e132: 12 total, 12 up, 12 in 2024-03-24T00:31:29.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:29 smithi106 ceph-mon[36505]: pgmap v351: 129 pgs: 129 active+clean; 27 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:31:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:29 smithi078 ceph-mon[32429]: osdmap e132: 12 total, 12 up, 12 in 2024-03-24T00:31:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:29 smithi078 ceph-mon[32429]: pgmap v351: 129 pgs: 129 active+clean; 27 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:31:30.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:29 smithi107 ceph-mon[36485]: osdmap e132: 12 total, 12 up, 12 in 2024-03-24T00:31:30.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:29 smithi107 ceph-mon[36485]: pgmap v351: 129 pgs: 129 active+clean; 27 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:31:30.703 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 419 sec latency 532293.737 ms 2024-03-24T00:31:31.703 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 420 sec latency 533293.844 ms 2024-03-24T00:31:32.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:32 smithi078 ceph-mon[32429]: pgmap v352: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:31:32.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:32 smithi107 ceph-mon[36485]: pgmap v352: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:31:32.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:32 smithi106 ceph-mon[36505]: pgmap v352: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:31:32.703 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 421 sec latency 534293.933 ms 2024-03-24T00:31:33.703 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 422 sec latency 535294.023 ms 2024-03-24T00:31:34.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:34 smithi078 ceph-mon[32429]: pgmap v353: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:31:34.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:34 smithi078 ceph-mon[32429]: osdmap e133: 12 total, 12 up, 12 in 2024-03-24T00:31:34.591 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:34 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.a.service: Scheduled restart job, restart counter is at 5. 2024-03-24T00:31:34.591 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:34 smithi078 systemd[1]: Stopped Ceph mds.a for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:31:34.591 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:34 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.a.service: Start request repeated too quickly. 2024-03-24T00:31:34.591 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:34 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.a.service: Failed with result 'exit-code'. 2024-03-24T00:31:34.591 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 24 00:31:34 smithi078 systemd[1]: Failed to start Ceph mds.a for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:31:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:34 smithi107 ceph-mon[36485]: pgmap v353: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:31:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:34 smithi107 ceph-mon[36485]: osdmap e133: 12 total, 12 up, 12 in 2024-03-24T00:31:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:34 smithi106 ceph-mon[36505]: pgmap v353: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:31:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:34 smithi106 ceph-mon[36505]: osdmap e133: 12 total, 12 up, 12 in 2024-03-24T00:31:34.703 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 423 sec latency 536294.115 ms 2024-03-24T00:31:35.703 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 424 sec latency 537294.148 ms 2024-03-24T00:31:36.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:36 smithi078 ceph-mon[32429]: pgmap v355: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-24T00:31:36.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:36 smithi107 ceph-mon[36485]: pgmap v355: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-24T00:31:36.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:36 smithi106 ceph-mon[36505]: pgmap v355: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-24T00:31:36.703 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 425 sec latency 538294.221 ms 2024-03-24T00:31:37.703 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 426 sec latency 539294.309 ms 2024-03-24T00:31:38.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:38 smithi078 ceph-mon[32429]: pgmap v356: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 237 B/s wr, 7 op/s 2024-03-24T00:31:38.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:38 smithi107 ceph-mon[36485]: pgmap v356: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 237 B/s wr, 7 op/s 2024-03-24T00:31:38.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:38 smithi106 ceph-mon[36505]: pgmap v356: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 237 B/s wr, 7 op/s 2024-03-24T00:31:38.703 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 427 sec latency 540294.414 ms 2024-03-24T00:31:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:39 smithi078 ceph-mon[32429]: Replacing daemon mds.a as rank 1 with standby daemon mds.g 2024-03-24T00:31:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:39 smithi078 ceph-mon[32429]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-03-24T00:31:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:39 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:31:39.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:39 smithi078 ceph-mon[32429]: osdmap e134: 12 total, 12 up, 12 in 2024-03-24T00:31:39.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:39 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:31:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:39 smithi107 ceph-mon[36485]: Replacing daemon mds.a as rank 1 with standby daemon mds.g 2024-03-24T00:31:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:39 smithi107 ceph-mon[36485]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-03-24T00:31:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:39 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:31:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:39 smithi107 ceph-mon[36485]: osdmap e134: 12 total, 12 up, 12 in 2024-03-24T00:31:39.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:39 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:31:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:39 smithi106 ceph-mon[36505]: Replacing daemon mds.a as rank 1 with standby daemon mds.g 2024-03-24T00:31:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:39 smithi106 ceph-mon[36505]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-03-24T00:31:39.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:39 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:31:39.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:39 smithi106 ceph-mon[36505]: osdmap e134: 12 total, 12 up, 12 in 2024-03-24T00:31:39.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:39 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:31:39.703 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 428 sec latency 541294.502 ms 2024-03-24T00:31:40.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:40 smithi078 ceph-mon[32429]: pgmap v358: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-24T00:31:40.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:40 smithi107 ceph-mon[36485]: pgmap v358: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-24T00:31:40.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:40 smithi106 ceph-mon[36505]: pgmap v358: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-24T00:31:40.704 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 429 sec latency 542294.623 ms 2024-03-24T00:31:41.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:41 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6838/4176145760,v1:172.21.15.78:6839/4176145760] up:resolve 2024-03-24T00:31:41.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:41 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:31:41.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:41 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6838/4176145760,v1:172.21.15.78:6839/4176145760] up:resolve 2024-03-24T00:31:41.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:41 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:31:41.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:41 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6838/4176145760,v1:172.21.15.78:6839/4176145760] up:resolve 2024-03-24T00:31:41.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:41 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:31:41.703 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 430 sec latency 543294.718 ms 2024-03-24T00:31:42.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:42 smithi078 ceph-mon[32429]: pgmap v359: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 127 B/s wr, 8 op/s 2024-03-24T00:31:42.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:42 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999982 2024-03-24T00:31:42.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:42 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6838/4176145760,v1:172.21.15.78:6839/4176145760] up:reconnect 2024-03-24T00:31:42.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:42 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:31:42.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:42 smithi107 ceph-mon[36485]: pgmap v359: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 127 B/s wr, 8 op/s 2024-03-24T00:31:42.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:42 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999982 2024-03-24T00:31:42.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:42 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6838/4176145760,v1:172.21.15.78:6839/4176145760] up:reconnect 2024-03-24T00:31:42.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:42 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:31:42.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:42 smithi106 ceph-mon[36505]: pgmap v359: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 127 B/s wr, 8 op/s 2024-03-24T00:31:42.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:42 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999982 2024-03-24T00:31:42.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:42 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6838/4176145760,v1:172.21.15.78:6839/4176145760] up:reconnect 2024-03-24T00:31:42.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:42 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:31:42.704 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 431 sec latency 544294.802 ms 2024-03-24T00:31:43.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:43 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6838/4176145760,v1:172.21.15.78:6839/4176145760] up:rejoin 2024-03-24T00:31:43.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:43 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:31:43.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:43 smithi106 ceph-mon[36505]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-24T00:31:43.704 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 432 sec latency 545294.852 ms 2024-03-24T00:31:43.828 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6838/4176145760,v1:172.21.15.78:6839/4176145760] up:rejoin 2024-03-24T00:31:43.829 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:31:43.829 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-mon[32429]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-24T00:31:43.830 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f7fe17e6700 time 2024-03-24T00:31:43.363023+0000 2024-03-24T00:31:43.830 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:31:43.830 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:31:43.830 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7febe104a1] 2024-03-24T00:31:43.830 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7febe10667] 2024-03-24T00:31:43.830 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f25cc08223] 2024-03-24T00:31:43.830 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f25ca825d6] 2024-03-24T00:31:43.830 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f25ca82da1] 2024-03-24T00:31:43.830 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f25ca852d5] 2024-03-24T00:31:43.831 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f25cad5233] 2024-03-24T00:31:43.831 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f25cad7197] 2024-03-24T00:31:43.831 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f25c9d611b] 2024-03-24T00:31:43.831 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f25c9d66bc] 2024-03-24T00:31:43.831 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f25c9c01ff] 2024-03-24T00:31:43.831 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7fec0a2488] 2024-03-24T00:31:43.831 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 13: (DispatchQueue::entry()+0x51f) [0x7f7fec09f60f] 2024-03-24T00:31:43.831 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7fec172e81] 2024-03-24T00:31:43.831 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7feab8d1ca] 2024-03-24T00:31:43.831 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 16: clone() 2024-03-24T00:31:43.831 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 2024-03-24T00:31:43.364+0000 7f7fe17e6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f7fe17e6700 time 2024-03-24T00:31:43.363023+0000 2024-03-24T00:31:43.831 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:31:43.832 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 2024-03-24T00:31:43.832 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:31:43.832 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7febe104a1] 2024-03-24T00:31:43.832 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7febe10667] 2024-03-24T00:31:43.832 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f25cc08223] 2024-03-24T00:31:43.832 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f25ca825d6] 2024-03-24T00:31:43.832 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f25ca82da1] 2024-03-24T00:31:43.832 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f25ca852d5] 2024-03-24T00:31:43.832 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f25cad5233] 2024-03-24T00:31:43.832 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f25cad7197] 2024-03-24T00:31:43.832 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f25c9d611b] 2024-03-24T00:31:43.832 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f25c9d66bc] 2024-03-24T00:31:43.832 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f25c9c01ff] 2024-03-24T00:31:43.833 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7fec0a2488] 2024-03-24T00:31:43.833 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 13: (DispatchQueue::entry()+0x51f) [0x7f7fec09f60f] 2024-03-24T00:31:43.833 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7fec172e81] 2024-03-24T00:31:43.833 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7feab8d1ca] 2024-03-24T00:31:43.833 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 16: clone() 2024-03-24T00:31:43.833 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 2024-03-24T00:31:43.833 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: *** Caught signal (Aborted) ** 2024-03-24T00:31:43.833 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: in thread 7f7fe17e6700 thread_name:ms_dispatch 2024-03-24T00:31:43.833 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:31:43.833 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7feab97d20] 2024-03-24T00:31:43.833 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 2: gsignal() 2024-03-24T00:31:43.833 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 3: abort() 2024-03-24T00:31:43.833 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7febe104fb] 2024-03-24T00:31:43.834 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7febe10667] 2024-03-24T00:31:43.834 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f25cc08223] 2024-03-24T00:31:43.834 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f25ca825d6] 2024-03-24T00:31:43.834 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f25ca82da1] 2024-03-24T00:31:43.834 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f25ca852d5] 2024-03-24T00:31:43.834 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f25cad5233] 2024-03-24T00:31:43.834 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f25cad7197] 2024-03-24T00:31:43.834 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f25c9d611b] 2024-03-24T00:31:43.834 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f25c9d66bc] 2024-03-24T00:31:43.834 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f25c9c01ff] 2024-03-24T00:31:43.835 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7fec0a2488] 2024-03-24T00:31:43.835 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 16: (DispatchQueue::entry()+0x51f) [0x7f7fec09f60f] 2024-03-24T00:31:43.835 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7fec172e81] 2024-03-24T00:31:43.835 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7feab8d1ca] 2024-03-24T00:31:43.835 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 19: clone() 2024-03-24T00:31:43.835 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 2024-03-24T00:31:43.367+0000 7f7fe17e6700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:31:43.835 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: in thread 7f7fe17e6700 thread_name:ms_dispatch 2024-03-24T00:31:43.835 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 2024-03-24T00:31:43.835 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:31:43.836 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7feab97d20] 2024-03-24T00:31:43.836 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 2: gsignal() 2024-03-24T00:31:43.836 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 3: abort() 2024-03-24T00:31:43.836 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7febe104fb] 2024-03-24T00:31:43.836 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7febe10667] 2024-03-24T00:31:43.837 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f25cc08223] 2024-03-24T00:31:43.837 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f25ca825d6] 2024-03-24T00:31:43.837 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f25ca82da1] 2024-03-24T00:31:43.837 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f25ca852d5] 2024-03-24T00:31:43.837 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f25cad5233] 2024-03-24T00:31:43.838 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f25cad7197] 2024-03-24T00:31:43.838 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f25c9d611b] 2024-03-24T00:31:43.838 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f25c9d66bc] 2024-03-24T00:31:43.838 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f25c9c01ff] 2024-03-24T00:31:43.838 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7fec0a2488] 2024-03-24T00:31:43.838 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 16: (DispatchQueue::entry()+0x51f) [0x7f7fec09f60f] 2024-03-24T00:31:43.838 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7fec172e81] 2024-03-24T00:31:43.838 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7feab8d1ca] 2024-03-24T00:31:43.838 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 19: clone() 2024-03-24T00:31:43.838 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:31:43.838 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 2024-03-24T00:31:43.838 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: -1> 2024-03-24T00:31:43.364+0000 7f7fe17e6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f7fe17e6700 time 2024-03-24T00:31:43.363023+0000 2024-03-24T00:31:43.839 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:31:43.839 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 2024-03-24T00:31:43.839 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:31:43.839 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7febe104a1] 2024-03-24T00:31:43.839 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7febe10667] 2024-03-24T00:31:43.839 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f25cc08223] 2024-03-24T00:31:43.839 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f25ca825d6] 2024-03-24T00:31:43.839 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f25ca82da1] 2024-03-24T00:31:43.839 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f25ca852d5] 2024-03-24T00:31:43.839 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f25cad5233] 2024-03-24T00:31:43.839 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f25cad7197] 2024-03-24T00:31:43.839 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f25c9d611b] 2024-03-24T00:31:43.839 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f25c9d66bc] 2024-03-24T00:31:43.839 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f25c9c01ff] 2024-03-24T00:31:43.839 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7fec0a2488] 2024-03-24T00:31:43.840 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 13: (DispatchQueue::entry()+0x51f) [0x7f7fec09f60f] 2024-03-24T00:31:43.840 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7fec172e81] 2024-03-24T00:31:43.840 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7feab8d1ca] 2024-03-24T00:31:43.840 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 16: clone() 2024-03-24T00:31:43.840 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 2024-03-24T00:31:43.840 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 0> 2024-03-24T00:31:43.367+0000 7f7fe17e6700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:31:43.840 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: in thread 7f7fe17e6700 thread_name:ms_dispatch 2024-03-24T00:31:43.840 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 2024-03-24T00:31:43.840 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:31:43.840 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7feab97d20] 2024-03-24T00:31:43.840 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 2: gsignal() 2024-03-24T00:31:43.840 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 3: abort() 2024-03-24T00:31:43.840 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7febe104fb] 2024-03-24T00:31:43.840 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7febe10667] 2024-03-24T00:31:43.840 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f25cc08223] 2024-03-24T00:31:43.841 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f25ca825d6] 2024-03-24T00:31:43.841 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f25ca82da1] 2024-03-24T00:31:43.841 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f25ca852d5] 2024-03-24T00:31:43.841 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f25cad5233] 2024-03-24T00:31:43.841 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f25cad7197] 2024-03-24T00:31:43.841 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f25c9d611b] 2024-03-24T00:31:43.841 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f25c9d66bc] 2024-03-24T00:31:43.841 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f25c9c01ff] 2024-03-24T00:31:43.841 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7fec0a2488] 2024-03-24T00:31:43.841 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 16: (DispatchQueue::entry()+0x51f) [0x7f7fec09f60f] 2024-03-24T00:31:43.841 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7fec172e81] 2024-03-24T00:31:43.841 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7feab8d1ca] 2024-03-24T00:31:43.842 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 19: clone() 2024-03-24T00:31:43.842 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:31:43.842 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 2024-03-24T00:31:43.842 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: -1> 2024-03-24T00:31:43.364+0000 7f7fe17e6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f7fe17e6700 time 2024-03-24T00:31:43.363023+0000 2024-03-24T00:31:43.842 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:31:43.842 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 2024-03-24T00:31:43.842 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:31:43.842 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7febe104a1] 2024-03-24T00:31:43.842 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7febe10667] 2024-03-24T00:31:43.842 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f25cc08223] 2024-03-24T00:31:43.842 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f25ca825d6] 2024-03-24T00:31:43.843 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f25ca82da1] 2024-03-24T00:31:43.843 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f25ca852d5] 2024-03-24T00:31:43.843 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f25cad5233] 2024-03-24T00:31:43.843 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f25cad7197] 2024-03-24T00:31:43.843 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f25c9d611b] 2024-03-24T00:31:43.843 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f25c9d66bc] 2024-03-24T00:31:43.843 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f25c9c01ff] 2024-03-24T00:31:43.843 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7fec0a2488] 2024-03-24T00:31:43.843 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 13: (DispatchQueue::entry()+0x51f) [0x7f7fec09f60f] 2024-03-24T00:31:43.843 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7fec172e81] 2024-03-24T00:31:43.843 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7feab8d1ca] 2024-03-24T00:31:43.843 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 16: clone() 2024-03-24T00:31:43.843 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 2024-03-24T00:31:43.844 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 0> 2024-03-24T00:31:43.367+0000 7f7fe17e6700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:31:43.844 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: in thread 7f7fe17e6700 thread_name:ms_dispatch 2024-03-24T00:31:43.844 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 2024-03-24T00:31:43.844 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:31:43.844 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7feab97d20] 2024-03-24T00:31:43.844 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 2: gsignal() 2024-03-24T00:31:43.844 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 3: abort() 2024-03-24T00:31:43.844 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7febe104fb] 2024-03-24T00:31:43.844 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7febe10667] 2024-03-24T00:31:43.844 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f25cc08223] 2024-03-24T00:31:43.844 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f25ca825d6] 2024-03-24T00:31:43.844 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f25ca82da1] 2024-03-24T00:31:43.844 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f25ca852d5] 2024-03-24T00:31:43.844 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f25cad5233] 2024-03-24T00:31:43.845 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f25cad7197] 2024-03-24T00:31:43.845 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f25c9d611b] 2024-03-24T00:31:43.845 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f25c9d66bc] 2024-03-24T00:31:43.845 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f25c9c01ff] 2024-03-24T00:31:43.845 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7fec0a2488] 2024-03-24T00:31:43.845 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 16: (DispatchQueue::entry()+0x51f) [0x7f7fec09f60f] 2024-03-24T00:31:43.845 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7fec172e81] 2024-03-24T00:31:43.845 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7feab8d1ca] 2024-03-24T00:31:43.845 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 19: clone() 2024-03-24T00:31:43.845 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:31:43.845 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[88451]: 2024-03-24T00:31:43.845 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:43 smithi078 podman[89032]: 2024-03-24 00:31:43.428183262 +0000 UTC m=+0.021708525 container died 17254032ff2c518d22d66fd6d5c0d9419c033d8a838699743dc5c62aca52ad81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-24T00:31:43.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:43 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6838/4176145760,v1:172.21.15.78:6839/4176145760] up:rejoin 2024-03-24T00:31:43.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:43 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:31:43.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:43 smithi107 ceph-mon[36485]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-24T00:31:44.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:44 smithi078 ceph-mon[32429]: pgmap v360: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 111 B/s wr, 7 op/s 2024-03-24T00:31:44.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:44 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:31:44.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:44 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6838/4176145760,v1:172.21.15.78:6839/4176145760] up:active 2024-03-24T00:31:44.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:44 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:31:44.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:44 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 1 up:standby 2024-03-24T00:31:44.591 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:44 smithi078 podman[89032]: 2024-03-24 00:31:44.14663143 +0000 UTC m=+0.740156694 container remove 17254032ff2c518d22d66fd6d5c0d9419c033d8a838699743dc5c62aca52ad81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:31:44.591 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:44 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:31:44.591 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:44 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.g.service: Failed with result 'exit-code'. 2024-03-24T00:31:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:44 smithi106 ceph-mon[36505]: pgmap v360: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 111 B/s wr, 7 op/s 2024-03-24T00:31:44.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:44 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:31:44.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:44 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6838/4176145760,v1:172.21.15.78:6839/4176145760] up:active 2024-03-24T00:31:44.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:44 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:31:44.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:44 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 1 up:standby 2024-03-24T00:31:44.704 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 433 sec latency 546294.902 ms 2024-03-24T00:31:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:44 smithi107 ceph-mon[36485]: pgmap v360: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 111 B/s wr, 7 op/s 2024-03-24T00:31:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:44 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:31:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:44 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6838/4176145760,v1:172.21.15.78:6839/4176145760] up:active 2024-03-24T00:31:44.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:44 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:31:44.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:44 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 1 up:standby 2024-03-24T00:31:45.704 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 434 sec latency 547295.001 ms 2024-03-24T00:31:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:46 smithi106 ceph-mon[36505]: pgmap v361: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-24T00:31:46.704 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 435 sec latency 548295.121 ms 2024-03-24T00:31:46.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:46 smithi078 ceph-mon[32429]: pgmap v361: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-24T00:31:46.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:46 smithi107 ceph-mon[36485]: pgmap v361: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-24T00:31:47.704 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 436 sec latency 549295.215 ms 2024-03-24T00:31:48.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:48 smithi106 ceph-mon[36505]: pgmap v362: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:31:48.704 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 437 sec latency 550295.250 ms 2024-03-24T00:31:48.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:48 smithi078 ceph-mon[32429]: pgmap v362: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:31:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:48 smithi107 ceph-mon[36485]: pgmap v362: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:31:49.704 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 438 sec latency 551295.361 ms 2024-03-24T00:31:50.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:50 smithi106 ceph-mon[36505]: pgmap v363: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 201 B/s wr, 10 op/s 2024-03-24T00:31:50.704 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 439 sec latency 552295.446 ms 2024-03-24T00:31:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:50 smithi078 ceph-mon[32429]: pgmap v363: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 201 B/s wr, 10 op/s 2024-03-24T00:31:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:50 smithi107 ceph-mon[36485]: pgmap v363: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 201 B/s wr, 10 op/s 2024-03-24T00:31:51.704 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 440 sec latency 553295.538 ms 2024-03-24T00:31:52.645 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:52 smithi107 ceph-mon[36485]: pgmap v364: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-24T00:31:52.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:52 smithi106 ceph-mon[36505]: pgmap v364: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-24T00:31:52.704 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 441 sec latency 554295.639 ms 2024-03-24T00:31:52.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:52 smithi078 ceph-mon[32429]: pgmap v364: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-24T00:31:53.705 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 442 sec latency 555295.730 ms 2024-03-24T00:31:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:54 smithi106 ceph-mon[36505]: pgmap v365: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T00:31:54.704 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 443 sec latency 556295.787 ms 2024-03-24T00:31:54.829 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:54 smithi078 ceph-mon[32429]: pgmap v365: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T00:31:54.830 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:54 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.g.service: Scheduled restart job, restart counter is at 4. 2024-03-24T00:31:54.830 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:54 smithi078 systemd[1]: Stopped Ceph mds.g for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:31:54.830 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:54 smithi078 systemd[1]: Starting Ceph mds.g for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:31:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:54 smithi107 ceph-mon[36485]: pgmap v365: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T00:31:55.090 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:54 smithi078 podman[89128]: 2024-03-24 00:31:54.733131141 +0000 UTC m=+0.017937468 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:31:55.091 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:54 smithi078 podman[89128]: 2024-03-24 00:31:54.85543239 +0000 UTC m=+0.140238698 container create 2d537e771551d8fae8aa9fc275867472001f13de1efcb4fcd1e2846161440880 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True) 2024-03-24T00:31:55.364 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:55 smithi078 podman[89128]: 2024-03-24 00:31:55.116301655 +0000 UTC m=+0.401107968 container init 2d537e771551d8fae8aa9fc275867472001f13de1efcb4fcd1e2846161440880 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-24T00:31:55.364 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:55 smithi078 podman[89128]: 2024-03-24 00:31:55.119122203 +0000 UTC m=+0.403928507 container start 2d537e771551d8fae8aa9fc275867472001f13de1efcb4fcd1e2846161440880 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-24T00:31:55.364 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:55 smithi078 ceph-mds[89147]: -- 172.21.15.78:0/2494561225 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f115ee5a40 con 0x55f115eb9000 2024-03-24T00:31:55.364 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:55 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: starting mds.g at 2024-03-24T00:31:55.364 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:55 smithi078 bash[89128]: 2d537e771551d8fae8aa9fc275867472001f13de1efcb4fcd1e2846161440880 2024-03-24T00:31:55.365 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:31:55 smithi078 systemd[1]: Started Ceph mds.g for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:31:55.650 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:55 smithi078 ceph-mon[32429]: daemon mds.g restarted 2024-03-24T00:31:55.650 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:55 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:31:55.650 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:55 smithi078 ceph-mon[32429]: osdmap e135: 12 total, 12 up, 12 in 2024-03-24T00:31:55.650 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:55 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6838/3377749691,v1:172.21.15.78:6839/3377749691] up:boot 2024-03-24T00:31:55.650 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:55 smithi078 ceph-mon[32429]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-24T00:31:55.650 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:55 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-03-24T00:31:55.651 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:31:55.651 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:55 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:31:55.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:55 smithi106 ceph-mon[36505]: daemon mds.g restarted 2024-03-24T00:31:55.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:55 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:31:55.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:55 smithi106 ceph-mon[36505]: osdmap e135: 12 total, 12 up, 12 in 2024-03-24T00:31:55.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:55 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6838/3377749691,v1:172.21.15.78:6839/3377749691] up:boot 2024-03-24T00:31:55.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:55 smithi106 ceph-mon[36505]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-24T00:31:55.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:55 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-03-24T00:31:55.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:31:55.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:55 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:31:55.705 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 444 sec latency 557295.826 ms 2024-03-24T00:31:55.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:55 smithi107 ceph-mon[36485]: daemon mds.g restarted 2024-03-24T00:31:55.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:55 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:31:55.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:55 smithi107 ceph-mon[36485]: osdmap e135: 12 total, 12 up, 12 in 2024-03-24T00:31:55.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:55 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6838/3377749691,v1:172.21.15.78:6839/3377749691] up:boot 2024-03-24T00:31:55.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:55 smithi107 ceph-mon[36485]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-24T00:31:55.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:55 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-03-24T00:31:55.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:31:55.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:55 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:31:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:56 smithi106 ceph-mon[36505]: pgmap v367: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:31:56.705 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 445 sec latency 558295.900 ms 2024-03-24T00:31:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:56 smithi078 ceph-mon[32429]: pgmap v367: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:31:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:56 smithi107 ceph-mon[36485]: pgmap v367: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T00:31:57.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:57 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6836/1071274623,v1:172.21.15.78:6837/1071274623] up:resolve 2024-03-24T00:31:57.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:57 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:31:57.705 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 446 sec latency 559296.003 ms 2024-03-24T00:31:57.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:57 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6836/1071274623,v1:172.21.15.78:6837/1071274623] up:resolve 2024-03-24T00:31:57.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:57 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:31:57.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:57 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6836/1071274623,v1:172.21.15.78:6837/1071274623] up:resolve 2024-03-24T00:31:57.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:57 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:31:58.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:58 smithi106 ceph-mon[36505]: pgmap v368: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:31:58.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:58 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999982 2024-03-24T00:31:58.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:58 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6836/1071274623,v1:172.21.15.78:6837/1071274623] up:reconnect 2024-03-24T00:31:58.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:58 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:31:58.705 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 447 sec latency 560296.138 ms 2024-03-24T00:31:58.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:58 smithi078 ceph-mon[32429]: pgmap v368: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:31:58.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:58 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999982 2024-03-24T00:31:58.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:58 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6836/1071274623,v1:172.21.15.78:6837/1071274623] up:reconnect 2024-03-24T00:31:58.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:58 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:31:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:58 smithi107 ceph-mon[36485]: pgmap v368: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:31:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:58 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999982 2024-03-24T00:31:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:58 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6836/1071274623,v1:172.21.15.78:6837/1071274623] up:reconnect 2024-03-24T00:31:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:58 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:31:59.341 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f0547d54700 time 2024-03-24T00:31:59.097837+0000 2024-03-24T00:31:59.341 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:31:59.341 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:31:59.342 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f055237e4a1] 2024-03-24T00:31:59.342 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f055237e667] 2024-03-24T00:31:59.342 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559f26112223] 2024-03-24T00:31:59.342 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559f25f8c5d6] 2024-03-24T00:31:59.342 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559f25f8cda1] 2024-03-24T00:31:59.342 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559f25f8f2d5] 2024-03-24T00:31:59.342 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559f25fdf233] 2024-03-24T00:31:59.342 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559f25fe1197] 2024-03-24T00:31:59.342 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559f25ee011b] 2024-03-24T00:31:59.342 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559f25ee06bc] 2024-03-24T00:31:59.342 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559f25eca1ff] 2024-03-24T00:31:59.343 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0552610488] 2024-03-24T00:31:59.343 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 13: (DispatchQueue::entry()+0x51f) [0x7f055260d60f] 2024-03-24T00:31:59.343 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f05526e0e81] 2024-03-24T00:31:59.343 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f05510fb1ca] 2024-03-24T00:31:59.343 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 16: clone() 2024-03-24T00:31:59.343 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: *** Caught signal (Aborted) ** 2024-03-24T00:31:59.343 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: in thread 7f0547d54700 thread_name:ms_dispatch 2024-03-24T00:31:59.343 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 2024-03-24T00:31:59.097+0000 7f0547d54700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f0547d54700 time 2024-03-24T00:31:59.097837+0000 2024-03-24T00:31:59.343 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:31:59.343 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 2024-03-24T00:31:59.343 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:31:59.344 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f055237e4a1] 2024-03-24T00:31:59.344 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f055237e667] 2024-03-24T00:31:59.344 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559f26112223] 2024-03-24T00:31:59.344 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559f25f8c5d6] 2024-03-24T00:31:59.344 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559f25f8cda1] 2024-03-24T00:31:59.344 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559f25f8f2d5] 2024-03-24T00:31:59.344 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559f25fdf233] 2024-03-24T00:31:59.344 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559f25fe1197] 2024-03-24T00:31:59.344 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559f25ee011b] 2024-03-24T00:31:59.344 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559f25ee06bc] 2024-03-24T00:31:59.344 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559f25eca1ff] 2024-03-24T00:31:59.344 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0552610488] 2024-03-24T00:31:59.344 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 13: (DispatchQueue::entry()+0x51f) [0x7f055260d60f] 2024-03-24T00:31:59.344 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f05526e0e81] 2024-03-24T00:31:59.345 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f05510fb1ca] 2024-03-24T00:31:59.345 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 16: clone() 2024-03-24T00:31:59.345 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 2024-03-24T00:31:59.345 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:31:59.345 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0551105d20] 2024-03-24T00:31:59.346 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 2: gsignal() 2024-03-24T00:31:59.346 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 3: abort() 2024-03-24T00:31:59.346 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f055237e4fb] 2024-03-24T00:31:59.346 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f055237e667] 2024-03-24T00:31:59.346 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559f26112223] 2024-03-24T00:31:59.346 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559f25f8c5d6] 2024-03-24T00:31:59.346 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559f25f8cda1] 2024-03-24T00:31:59.347 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559f25f8f2d5] 2024-03-24T00:31:59.347 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559f25fdf233] 2024-03-24T00:31:59.347 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559f25fe1197] 2024-03-24T00:31:59.347 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559f25ee011b] 2024-03-24T00:31:59.347 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559f25ee06bc] 2024-03-24T00:31:59.347 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559f25eca1ff] 2024-03-24T00:31:59.347 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0552610488] 2024-03-24T00:31:59.347 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 16: (DispatchQueue::entry()+0x51f) [0x7f055260d60f] 2024-03-24T00:31:59.347 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f05526e0e81] 2024-03-24T00:31:59.347 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f05510fb1ca] 2024-03-24T00:31:59.347 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 19: clone() 2024-03-24T00:31:59.347 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 2024-03-24T00:31:59.100+0000 7f0547d54700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:31:59.347 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: in thread 7f0547d54700 thread_name:ms_dispatch 2024-03-24T00:31:59.348 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 2024-03-24T00:31:59.348 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:31:59.348 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0551105d20] 2024-03-24T00:31:59.348 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 2: gsignal() 2024-03-24T00:31:59.348 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 3: abort() 2024-03-24T00:31:59.348 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f055237e4fb] 2024-03-24T00:31:59.348 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f055237e667] 2024-03-24T00:31:59.348 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559f26112223] 2024-03-24T00:31:59.348 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559f25f8c5d6] 2024-03-24T00:31:59.348 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559f25f8cda1] 2024-03-24T00:31:59.348 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559f25f8f2d5] 2024-03-24T00:31:59.348 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559f25fdf233] 2024-03-24T00:31:59.348 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559f25fe1197] 2024-03-24T00:31:59.348 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559f25ee011b] 2024-03-24T00:31:59.349 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559f25ee06bc] 2024-03-24T00:31:59.349 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559f25eca1ff] 2024-03-24T00:31:59.349 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0552610488] 2024-03-24T00:31:59.349 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 16: (DispatchQueue::entry()+0x51f) [0x7f055260d60f] 2024-03-24T00:31:59.349 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f05526e0e81] 2024-03-24T00:31:59.349 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f05510fb1ca] 2024-03-24T00:31:59.349 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 19: clone() 2024-03-24T00:31:59.349 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:31:59.349 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 2024-03-24T00:31:59.349 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: -1> 2024-03-24T00:31:59.097+0000 7f0547d54700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f0547d54700 time 2024-03-24T00:31:59.097837+0000 2024-03-24T00:31:59.349 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:31:59.349 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 2024-03-24T00:31:59.349 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:31:59.349 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f055237e4a1] 2024-03-24T00:31:59.350 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f055237e667] 2024-03-24T00:31:59.350 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559f26112223] 2024-03-24T00:31:59.350 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559f25f8c5d6] 2024-03-24T00:31:59.350 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559f25f8cda1] 2024-03-24T00:31:59.350 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559f25f8f2d5] 2024-03-24T00:31:59.350 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559f25fdf233] 2024-03-24T00:31:59.350 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559f25fe1197] 2024-03-24T00:31:59.350 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559f25ee011b] 2024-03-24T00:31:59.350 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559f25ee06bc] 2024-03-24T00:31:59.350 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559f25eca1ff] 2024-03-24T00:31:59.350 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0552610488] 2024-03-24T00:31:59.350 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 13: (DispatchQueue::entry()+0x51f) [0x7f055260d60f] 2024-03-24T00:31:59.350 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f05526e0e81] 2024-03-24T00:31:59.350 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f05510fb1ca] 2024-03-24T00:31:59.351 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 16: clone() 2024-03-24T00:31:59.351 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 2024-03-24T00:31:59.351 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 0> 2024-03-24T00:31:59.100+0000 7f0547d54700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:31:59.351 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: in thread 7f0547d54700 thread_name:ms_dispatch 2024-03-24T00:31:59.351 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 2024-03-24T00:31:59.351 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:31:59.351 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0551105d20] 2024-03-24T00:31:59.351 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 2: gsignal() 2024-03-24T00:31:59.351 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 3: abort() 2024-03-24T00:31:59.351 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f055237e4fb] 2024-03-24T00:31:59.351 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f055237e667] 2024-03-24T00:31:59.351 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559f26112223] 2024-03-24T00:31:59.351 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559f25f8c5d6] 2024-03-24T00:31:59.351 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559f25f8cda1] 2024-03-24T00:31:59.352 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559f25f8f2d5] 2024-03-24T00:31:59.352 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559f25fdf233] 2024-03-24T00:31:59.352 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559f25fe1197] 2024-03-24T00:31:59.352 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559f25ee011b] 2024-03-24T00:31:59.352 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559f25ee06bc] 2024-03-24T00:31:59.352 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559f25eca1ff] 2024-03-24T00:31:59.352 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0552610488] 2024-03-24T00:31:59.352 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 16: (DispatchQueue::entry()+0x51f) [0x7f055260d60f] 2024-03-24T00:31:59.352 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f05526e0e81] 2024-03-24T00:31:59.352 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f05510fb1ca] 2024-03-24T00:31:59.352 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 19: clone() 2024-03-24T00:31:59.352 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:31:59.352 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 2024-03-24T00:31:59.352 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: -1> 2024-03-24T00:31:59.097+0000 7f0547d54700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f0547d54700 time 2024-03-24T00:31:59.097837+0000 2024-03-24T00:31:59.353 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:31:59.353 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 2024-03-24T00:31:59.353 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:31:59.353 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f055237e4a1] 2024-03-24T00:31:59.353 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f055237e667] 2024-03-24T00:31:59.353 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559f26112223] 2024-03-24T00:31:59.353 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559f25f8c5d6] 2024-03-24T00:31:59.353 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559f25f8cda1] 2024-03-24T00:31:59.353 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559f25f8f2d5] 2024-03-24T00:31:59.353 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559f25fdf233] 2024-03-24T00:31:59.353 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559f25fe1197] 2024-03-24T00:31:59.353 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559f25ee011b] 2024-03-24T00:31:59.353 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559f25ee06bc] 2024-03-24T00:31:59.353 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559f25eca1ff] 2024-03-24T00:31:59.353 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0552610488] 2024-03-24T00:31:59.354 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 13: (DispatchQueue::entry()+0x51f) [0x7f055260d60f] 2024-03-24T00:31:59.354 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f05526e0e81] 2024-03-24T00:31:59.354 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f05510fb1ca] 2024-03-24T00:31:59.354 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 16: clone() 2024-03-24T00:31:59.354 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 2024-03-24T00:31:59.354 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 0> 2024-03-24T00:31:59.100+0000 7f0547d54700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:31:59.354 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: in thread 7f0547d54700 thread_name:ms_dispatch 2024-03-24T00:31:59.354 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 2024-03-24T00:31:59.354 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:31:59.354 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0551105d20] 2024-03-24T00:31:59.354 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 2: gsignal() 2024-03-24T00:31:59.354 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 3: abort() 2024-03-24T00:31:59.355 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f055237e4fb] 2024-03-24T00:31:59.355 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f055237e667] 2024-03-24T00:31:59.355 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559f26112223] 2024-03-24T00:31:59.355 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559f25f8c5d6] 2024-03-24T00:31:59.355 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559f25f8cda1] 2024-03-24T00:31:59.355 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x559f25f8f2d5] 2024-03-24T00:31:59.355 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559f25fdf233] 2024-03-24T00:31:59.355 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559f25fe1197] 2024-03-24T00:31:59.355 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559f25ee011b] 2024-03-24T00:31:59.355 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559f25ee06bc] 2024-03-24T00:31:59.355 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559f25eca1ff] 2024-03-24T00:31:59.355 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0552610488] 2024-03-24T00:31:59.355 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 16: (DispatchQueue::entry()+0x51f) [0x7f055260d60f] 2024-03-24T00:31:59.355 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f05526e0e81] 2024-03-24T00:31:59.356 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f05510fb1ca] 2024-03-24T00:31:59.356 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 19: clone() 2024-03-24T00:31:59.356 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:31:59.356 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[70775]: 2024-03-24T00:31:59.356 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 podman[89193]: 2024-03-24 00:31:59.162984272 +0000 UTC m=+0.019952829 container died e84c2439bea2c279fea20cb137e076397e1f0f62f137b2eeddaf2a8b6d394d5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-24T00:31:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:59 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6836/1071274623,v1:172.21.15.78:6837/1071274623] up:rejoin 2024-03-24T00:31:59.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:59 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:31:59.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:59 smithi106 ceph-mon[36505]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-24T00:31:59.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:59 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:31:59.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:59 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6836/1071274623,v1:172.21.15.78:6837/1071274623] up:active 2024-03-24T00:31:59.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:31:59 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=h=up:active} 3 up:standby-replay 1 up:standby 2024-03-24T00:31:59.705 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 448 sec latency 561296.237 ms 2024-03-24T00:31:59.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6836/1071274623,v1:172.21.15.78:6837/1071274623] up:rejoin 2024-03-24T00:31:59.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:31:59.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-mon[32429]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-24T00:31:59.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:31:59.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6836/1071274623,v1:172.21.15.78:6837/1071274623] up:active 2024-03-24T00:31:59.813 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:31:59 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=h=up:active} 3 up:standby-replay 1 up:standby 2024-03-24T00:31:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:59 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6836/1071274623,v1:172.21.15.78:6837/1071274623] up:rejoin 2024-03-24T00:31:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:59 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:31:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:59 smithi107 ceph-mon[36485]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-24T00:31:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:59 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:31:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:59 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6836/1071274623,v1:172.21.15.78:6837/1071274623] up:active 2024-03-24T00:31:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:31:59 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=h=up:active} 3 up:standby-replay 1 up:standby 2024-03-24T00:32:00.063 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 podman[89193]: 2024-03-24 00:31:59.813123309 +0000 UTC m=+0.670091855 container remove e84c2439bea2c279fea20cb137e076397e1f0f62f137b2eeddaf2a8b6d394d5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-24T00:32:00.063 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:31:59 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:32:00.340 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:00 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.d.service: Failed with result 'exit-code'. 2024-03-24T00:32:00.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:00 smithi106 ceph-mon[36505]: pgmap v369: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:32:00.705 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 449 sec latency 562296.307 ms 2024-03-24T00:32:00.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:00 smithi078 ceph-mon[32429]: pgmap v369: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:32:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:00 smithi107 ceph-mon[36485]: pgmap v369: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:32:01.705 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 450 sec latency 563296.437 ms 2024-03-24T00:32:02.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:02 smithi106 ceph-mon[36505]: pgmap v370: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:32:02.684 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:02 smithi078 ceph-mon[32429]: pgmap v370: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:32:02.705 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 451 sec latency 564296.536 ms 2024-03-24T00:32:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:02 smithi107 ceph-mon[36485]: pgmap v370: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:32:03.705 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 452 sec latency 565296.631 ms 2024-03-24T00:32:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:04 smithi106 ceph-mon[36505]: pgmap v371: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:32:04.706 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 453 sec latency 566296.737 ms 2024-03-24T00:32:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:04 smithi078 ceph-mon[32429]: pgmap v371: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:32:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:04 smithi107 ceph-mon[36485]: pgmap v371: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:32:05.706 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 454 sec latency 567296.802 ms 2024-03-24T00:32:06.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:06 smithi106 ceph-mon[36505]: pgmap v372: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 202 B/s wr, 11 op/s 2024-03-24T00:32:06.706 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 455 sec latency 568296.836 ms 2024-03-24T00:32:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:06 smithi078 ceph-mon[32429]: pgmap v372: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 202 B/s wr, 11 op/s 2024-03-24T00:32:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:06 smithi107 ceph-mon[36485]: pgmap v372: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 202 B/s wr, 11 op/s 2024-03-24T00:32:07.706 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 456 sec latency 569296.880 ms 2024-03-24T00:32:08.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:08 smithi106 ceph-mon[36505]: pgmap v373: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-24T00:32:08.706 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 457 sec latency 570296.977 ms 2024-03-24T00:32:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:08 smithi078 ceph-mon[32429]: pgmap v373: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-24T00:32:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:08 smithi107 ceph-mon[36485]: pgmap v373: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 170 B/s wr, 13 op/s 2024-03-24T00:32:09.706 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 458 sec latency 571297.061 ms 2024-03-24T00:32:10.406 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:10 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.d.service: Scheduled restart job, restart counter is at 1. 2024-03-24T00:32:10.406 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:10 smithi078 systemd[1]: Stopped Ceph mds.d for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:32:10.407 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:10 smithi078 systemd[1]: Starting Ceph mds.d for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:32:10.407 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:10 smithi078 podman[89289]: 2024-03-24 00:32:10.228569604 +0000 UTC m=+0.018408099 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:32:10.407 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:10 smithi078 podman[89289]: 2024-03-24 00:32:10.339133797 +0000 UTC m=+0.128972335 container create 5ce9b31bbc3d3b64eb7034a89ac6a41336a0f63c3282d078511d9235866d9ef0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:32:10.691 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:10 smithi078 ceph-mon[32429]: pgmap v374: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-24T00:32:10.691 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:10 smithi078 podman[89289]: 2024-03-24 00:32:10.591597526 +0000 UTC m=+0.381436040 container init 5ce9b31bbc3d3b64eb7034a89ac6a41336a0f63c3282d078511d9235866d9ef0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319) 2024-03-24T00:32:10.692 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:10 smithi078 podman[89289]: 2024-03-24 00:32:10.59434953 +0000 UTC m=+0.384188033 container start 5ce9b31bbc3d3b64eb7034a89ac6a41336a0f63c3282d078511d9235866d9ef0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-24T00:32:10.692 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:10 smithi078 ceph-mds[89308]: -- 172.21.15.78:0/4038107454 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56220f7d1a40 con 0x56220f7a4800 2024-03-24T00:32:10.692 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:10 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: starting mds.d at 2024-03-24T00:32:10.706 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 459 sec latency 572297.104 ms 2024-03-24T00:32:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:10 smithi107 ceph-mon[36485]: pgmap v374: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-24T00:32:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:10 smithi106 ceph-mon[36505]: pgmap v374: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-24T00:32:11.091 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:10 smithi078 bash[89289]: 5ce9b31bbc3d3b64eb7034a89ac6a41336a0f63c3282d078511d9235866d9ef0 2024-03-24T00:32:11.091 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:10 smithi078 systemd[1]: Started Ceph mds.d for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:32:11.706 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 460 sec latency 573297.185 ms 2024-03-24T00:32:11.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:11 smithi078 ceph-mon[32429]: daemon mds.d restarted 2024-03-24T00:32:11.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:11 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:32:11.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:11 smithi078 ceph-mon[32429]: osdmap e136: 12 total, 12 up, 12 in 2024-03-24T00:32:11.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:11 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6836/2239323994,v1:172.21.15.78:6837/2239323994] up:boot 2024-03-24T00:32:11.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:11 smithi078 ceph-mon[32429]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-03-24T00:32:11.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:11 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-03-24T00:32:11.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:11 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:32:11.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:11 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:11 smithi107 ceph-mon[36485]: daemon mds.d restarted 2024-03-24T00:32:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:11 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:32:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:11 smithi107 ceph-mon[36485]: osdmap e136: 12 total, 12 up, 12 in 2024-03-24T00:32:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:11 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6836/2239323994,v1:172.21.15.78:6837/2239323994] up:boot 2024-03-24T00:32:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:11 smithi107 ceph-mon[36485]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-03-24T00:32:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:11 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-03-24T00:32:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:11 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:32:11.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:11 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:11.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:11 smithi106 ceph-mon[36505]: daemon mds.d restarted 2024-03-24T00:32:11.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:11 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:32:11.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:11 smithi106 ceph-mon[36505]: osdmap e136: 12 total, 12 up, 12 in 2024-03-24T00:32:11.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:11 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6836/2239323994,v1:172.21.15.78:6837/2239323994] up:boot 2024-03-24T00:32:11.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:11 smithi106 ceph-mon[36505]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-03-24T00:32:11.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:11 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-03-24T00:32:11.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:11 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:32:11.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:11 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:12.706 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 461 sec latency 574297.287 ms 2024-03-24T00:32:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:12 smithi078 ceph-mon[32429]: pgmap v376: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:32:12.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:12 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6838/3377749691,v1:172.21.15.78:6839/3377749691] up:resolve 2024-03-24T00:32:12.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:12 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:12 smithi107 ceph-mon[36485]: pgmap v376: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:32:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:12 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6838/3377749691,v1:172.21.15.78:6839/3377749691] up:resolve 2024-03-24T00:32:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:12 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:12.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:12 smithi106 ceph-mon[36505]: pgmap v376: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:32:12.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:12 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6838/3377749691,v1:172.21.15.78:6839/3377749691] up:resolve 2024-03-24T00:32:12.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:12 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:13.706 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 462 sec latency 575297.342 ms 2024-03-24T00:32:14.088 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:13 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999981 2024-03-24T00:32:14.088 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:13 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6838/3377749691,v1:172.21.15.78:6839/3377749691] up:reconnect 2024-03-24T00:32:14.088 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:13 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:14.088 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:13 smithi078 ceph-mon[32429]: pgmap v377: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:32:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:13 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999981 2024-03-24T00:32:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:13 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6838/3377749691,v1:172.21.15.78:6839/3377749691] up:reconnect 2024-03-24T00:32:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:13 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:13 smithi107 ceph-mon[36485]: pgmap v377: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:32:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:13 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999981 2024-03-24T00:32:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:13 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6838/3377749691,v1:172.21.15.78:6839/3377749691] up:reconnect 2024-03-24T00:32:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:13 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:14.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:13 smithi106 ceph-mon[36505]: pgmap v377: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:32:14.445 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7efd86180700 time 2024-03-24T00:32:14.103984+0000 2024-03-24T00:32:14.445 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:32:14.445 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:32:14.445 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7efd907aa4a1] 2024-03-24T00:32:14.446 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7efd907aa667] 2024-03-24T00:32:14.446 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f114664223] 2024-03-24T00:32:14.446 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f1144de5d6] 2024-03-24T00:32:14.446 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f1144deda1] 2024-03-24T00:32:14.446 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f1144e12d5] 2024-03-24T00:32:14.446 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f114531233] 2024-03-24T00:32:14.446 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f114533197] 2024-03-24T00:32:14.446 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f11443211b] 2024-03-24T00:32:14.446 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f1144326bc] 2024-03-24T00:32:14.446 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f11441c1ff] 2024-03-24T00:32:14.446 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7efd90a3c488] 2024-03-24T00:32:14.446 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 13: (DispatchQueue::entry()+0x51f) [0x7efd90a3960f] 2024-03-24T00:32:14.446 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7efd90b0ce81] 2024-03-24T00:32:14.447 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7efd8f5271ca] 2024-03-24T00:32:14.447 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 16: clone() 2024-03-24T00:32:14.447 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 2024-03-24T00:32:14.105+0000 7efd86180700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7efd86180700 time 2024-03-24T00:32:14.103984+0000 2024-03-24T00:32:14.448 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:32:14.448 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 2024-03-24T00:32:14.448 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:32:14.448 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7efd907aa4a1] 2024-03-24T00:32:14.448 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7efd907aa667] 2024-03-24T00:32:14.448 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f114664223] 2024-03-24T00:32:14.448 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f1144de5d6] 2024-03-24T00:32:14.448 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f1144deda1] 2024-03-24T00:32:14.448 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f1144e12d5] 2024-03-24T00:32:14.448 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f114531233] 2024-03-24T00:32:14.448 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f114533197] 2024-03-24T00:32:14.448 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f11443211b] 2024-03-24T00:32:14.448 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f1144326bc] 2024-03-24T00:32:14.449 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f11441c1ff] 2024-03-24T00:32:14.449 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7efd90a3c488] 2024-03-24T00:32:14.449 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 13: (DispatchQueue::entry()+0x51f) [0x7efd90a3960f] 2024-03-24T00:32:14.449 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7efd90b0ce81] 2024-03-24T00:32:14.449 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7efd8f5271ca] 2024-03-24T00:32:14.449 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 16: clone() 2024-03-24T00:32:14.449 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 2024-03-24T00:32:14.449 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: *** Caught signal (Aborted) ** 2024-03-24T00:32:14.449 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: in thread 7efd86180700 thread_name:ms_dispatch 2024-03-24T00:32:14.449 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:32:14.449 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efd8f531d20] 2024-03-24T00:32:14.449 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 2: gsignal() 2024-03-24T00:32:14.449 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 3: abort() 2024-03-24T00:32:14.449 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7efd907aa4fb] 2024-03-24T00:32:14.450 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7efd907aa667] 2024-03-24T00:32:14.450 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f114664223] 2024-03-24T00:32:14.450 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f1144de5d6] 2024-03-24T00:32:14.450 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f1144deda1] 2024-03-24T00:32:14.450 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f1144e12d5] 2024-03-24T00:32:14.450 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f114531233] 2024-03-24T00:32:14.450 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f114533197] 2024-03-24T00:32:14.450 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f11443211b] 2024-03-24T00:32:14.450 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f1144326bc] 2024-03-24T00:32:14.450 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f11441c1ff] 2024-03-24T00:32:14.450 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7efd90a3c488] 2024-03-24T00:32:14.450 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 16: (DispatchQueue::entry()+0x51f) [0x7efd90a3960f] 2024-03-24T00:32:14.450 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7efd90b0ce81] 2024-03-24T00:32:14.450 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7efd8f5271ca] 2024-03-24T00:32:14.451 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 19: clone() 2024-03-24T00:32:14.451 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 2024-03-24T00:32:14.107+0000 7efd86180700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:32:14.451 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: in thread 7efd86180700 thread_name:ms_dispatch 2024-03-24T00:32:14.451 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 2024-03-24T00:32:14.451 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:32:14.451 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efd8f531d20] 2024-03-24T00:32:14.451 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 2: gsignal() 2024-03-24T00:32:14.451 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 3: abort() 2024-03-24T00:32:14.451 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7efd907aa4fb] 2024-03-24T00:32:14.451 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7efd907aa667] 2024-03-24T00:32:14.451 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f114664223] 2024-03-24T00:32:14.451 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f1144de5d6] 2024-03-24T00:32:14.451 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f1144deda1] 2024-03-24T00:32:14.451 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f1144e12d5] 2024-03-24T00:32:14.451 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f114531233] 2024-03-24T00:32:14.452 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f114533197] 2024-03-24T00:32:14.452 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f11443211b] 2024-03-24T00:32:14.452 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f1144326bc] 2024-03-24T00:32:14.452 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f11441c1ff] 2024-03-24T00:32:14.452 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7efd90a3c488] 2024-03-24T00:32:14.452 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 16: (DispatchQueue::entry()+0x51f) [0x7efd90a3960f] 2024-03-24T00:32:14.452 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7efd90b0ce81] 2024-03-24T00:32:14.452 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7efd8f5271ca] 2024-03-24T00:32:14.452 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 19: clone() 2024-03-24T00:32:14.452 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:32:14.452 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 2024-03-24T00:32:14.452 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: -1> 2024-03-24T00:32:14.105+0000 7efd86180700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7efd86180700 time 2024-03-24T00:32:14.103984+0000 2024-03-24T00:32:14.452 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:32:14.453 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 2024-03-24T00:32:14.453 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:32:14.453 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7efd907aa4a1] 2024-03-24T00:32:14.453 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7efd907aa667] 2024-03-24T00:32:14.453 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f114664223] 2024-03-24T00:32:14.453 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f1144de5d6] 2024-03-24T00:32:14.453 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f1144deda1] 2024-03-24T00:32:14.453 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f1144e12d5] 2024-03-24T00:32:14.453 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f114531233] 2024-03-24T00:32:14.453 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f114533197] 2024-03-24T00:32:14.453 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f11443211b] 2024-03-24T00:32:14.453 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f1144326bc] 2024-03-24T00:32:14.453 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f11441c1ff] 2024-03-24T00:32:14.453 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7efd90a3c488] 2024-03-24T00:32:14.454 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 13: (DispatchQueue::entry()+0x51f) [0x7efd90a3960f] 2024-03-24T00:32:14.454 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7efd90b0ce81] 2024-03-24T00:32:14.454 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7efd8f5271ca] 2024-03-24T00:32:14.454 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 16: clone() 2024-03-24T00:32:14.454 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 2024-03-24T00:32:14.454 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 0> 2024-03-24T00:32:14.107+0000 7efd86180700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:32:14.454 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: in thread 7efd86180700 thread_name:ms_dispatch 2024-03-24T00:32:14.454 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 2024-03-24T00:32:14.454 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:32:14.454 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efd8f531d20] 2024-03-24T00:32:14.454 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 2: gsignal() 2024-03-24T00:32:14.454 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 3: abort() 2024-03-24T00:32:14.454 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7efd907aa4fb] 2024-03-24T00:32:14.454 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7efd907aa667] 2024-03-24T00:32:14.455 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f114664223] 2024-03-24T00:32:14.455 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f1144de5d6] 2024-03-24T00:32:14.455 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f1144deda1] 2024-03-24T00:32:14.455 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f1144e12d5] 2024-03-24T00:32:14.456 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f114531233] 2024-03-24T00:32:14.456 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f114533197] 2024-03-24T00:32:14.456 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f11443211b] 2024-03-24T00:32:14.456 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f1144326bc] 2024-03-24T00:32:14.456 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f11441c1ff] 2024-03-24T00:32:14.456 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7efd90a3c488] 2024-03-24T00:32:14.456 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 16: (DispatchQueue::entry()+0x51f) [0x7efd90a3960f] 2024-03-24T00:32:14.456 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7efd90b0ce81] 2024-03-24T00:32:14.456 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7efd8f5271ca] 2024-03-24T00:32:14.456 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 19: clone() 2024-03-24T00:32:14.456 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:32:14.456 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 2024-03-24T00:32:14.456 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: -1> 2024-03-24T00:32:14.105+0000 7efd86180700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7efd86180700 time 2024-03-24T00:32:14.103984+0000 2024-03-24T00:32:14.457 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:32:14.457 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 2024-03-24T00:32:14.457 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:32:14.457 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7efd907aa4a1] 2024-03-24T00:32:14.457 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7efd907aa667] 2024-03-24T00:32:14.457 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f114664223] 2024-03-24T00:32:14.457 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f1144de5d6] 2024-03-24T00:32:14.457 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f1144deda1] 2024-03-24T00:32:14.457 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f1144e12d5] 2024-03-24T00:32:14.457 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f114531233] 2024-03-24T00:32:14.457 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f114533197] 2024-03-24T00:32:14.457 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f11443211b] 2024-03-24T00:32:14.457 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f1144326bc] 2024-03-24T00:32:14.457 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f11441c1ff] 2024-03-24T00:32:14.457 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7efd90a3c488] 2024-03-24T00:32:14.458 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 13: (DispatchQueue::entry()+0x51f) [0x7efd90a3960f] 2024-03-24T00:32:14.458 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7efd90b0ce81] 2024-03-24T00:32:14.458 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7efd8f5271ca] 2024-03-24T00:32:14.458 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 16: clone() 2024-03-24T00:32:14.458 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 2024-03-24T00:32:14.458 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 0> 2024-03-24T00:32:14.107+0000 7efd86180700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:32:14.458 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: in thread 7efd86180700 thread_name:ms_dispatch 2024-03-24T00:32:14.458 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 2024-03-24T00:32:14.458 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:32:14.458 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efd8f531d20] 2024-03-24T00:32:14.458 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 2: gsignal() 2024-03-24T00:32:14.458 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 3: abort() 2024-03-24T00:32:14.458 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7efd907aa4fb] 2024-03-24T00:32:14.458 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7efd907aa667] 2024-03-24T00:32:14.458 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f114664223] 2024-03-24T00:32:14.459 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f1144de5d6] 2024-03-24T00:32:14.459 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f1144deda1] 2024-03-24T00:32:14.459 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f1144e12d5] 2024-03-24T00:32:14.459 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f114531233] 2024-03-24T00:32:14.459 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f114533197] 2024-03-24T00:32:14.459 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f11443211b] 2024-03-24T00:32:14.459 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f1144326bc] 2024-03-24T00:32:14.459 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f11441c1ff] 2024-03-24T00:32:14.459 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7efd90a3c488] 2024-03-24T00:32:14.459 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 16: (DispatchQueue::entry()+0x51f) [0x7efd90a3960f] 2024-03-24T00:32:14.459 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7efd90b0ce81] 2024-03-24T00:32:14.459 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7efd8f5271ca] 2024-03-24T00:32:14.459 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 19: clone() 2024-03-24T00:32:14.459 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:32:14.460 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89143]: 2024-03-24T00:32:14.460 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 podman[89342]: 2024-03-24 00:32:14.16047737 +0000 UTC m=+0.023478331 container died 2d537e771551d8fae8aa9fc275867472001f13de1efcb4fcd1e2846161440880 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-24T00:32:14.706 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 463 sec latency 576297.484 ms 2024-03-24T00:32:14.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6838/3377749691,v1:172.21.15.78:6839/3377749691] up:rejoin 2024-03-24T00:32:14.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:14.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-mon[32429]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-24T00:32:14.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:32:14.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6838/3377749691,v1:172.21.15.78:6839/3377749691] up:active 2024-03-24T00:32:14.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:14 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 1 up:standby 2024-03-24T00:32:15.090 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 podman[89342]: 2024-03-24 00:32:14.822030177 +0000 UTC m=+0.685031180 container remove 2d537e771551d8fae8aa9fc275867472001f13de1efcb4fcd1e2846161440880 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:32:15.090 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:14 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:32:15.090 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:15 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.g.service: Failed with result 'exit-code'. 2024-03-24T00:32:15.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:14 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6838/3377749691,v1:172.21.15.78:6839/3377749691] up:rejoin 2024-03-24T00:32:15.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:14 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:15.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:14 smithi107 ceph-mon[36485]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-24T00:32:15.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:14 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:32:15.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:14 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6838/3377749691,v1:172.21.15.78:6839/3377749691] up:active 2024-03-24T00:32:15.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:14 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 1 up:standby 2024-03-24T00:32:15.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:14 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6838/3377749691,v1:172.21.15.78:6839/3377749691] up:rejoin 2024-03-24T00:32:15.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:14 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:15.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:14 smithi106 ceph-mon[36505]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-24T00:32:15.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:14 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:32:15.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:14 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6838/3377749691,v1:172.21.15.78:6839/3377749691] up:active 2024-03-24T00:32:15.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:14 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 1 up:standby 2024-03-24T00:32:15.706 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 464 sec latency 577297.617 ms 2024-03-24T00:32:16.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:15 smithi078 ceph-mon[32429]: pgmap v378: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:32:16.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:15 smithi107 ceph-mon[36485]: pgmap v378: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:32:16.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:15 smithi106 ceph-mon[36505]: pgmap v378: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:32:16.706 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 465 sec latency 578297.665 ms 2024-03-24T00:32:17.707 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 466 sec latency 579297.714 ms 2024-03-24T00:32:18.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:18 smithi078 ceph-mon[32429]: pgmap v379: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:32:18.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:18 smithi107 ceph-mon[36485]: pgmap v379: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:32:18.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:18 smithi106 ceph-mon[36505]: pgmap v379: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:32:18.707 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 467 sec latency 580297.808 ms 2024-03-24T00:32:19.707 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 468 sec latency 581297.928 ms 2024-03-24T00:32:20.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:20 smithi078 ceph-mon[32429]: pgmap v380: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:32:20.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:20 smithi107 ceph-mon[36485]: pgmap v380: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:32:20.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:20 smithi106 ceph-mon[36505]: pgmap v380: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 27 op/s 2024-03-24T00:32:20.707 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 469 sec latency 582298.018 ms 2024-03-24T00:32:21.707 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 470 sec latency 583298.146 ms 2024-03-24T00:32:22.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:22 smithi078 ceph-mon[32429]: pgmap v381: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 26 op/s 2024-03-24T00:32:22.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:22 smithi107 ceph-mon[36485]: pgmap v381: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 26 op/s 2024-03-24T00:32:22.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:22 smithi106 ceph-mon[36505]: pgmap v381: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 26 op/s 2024-03-24T00:32:22.707 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 471 sec latency 584298.282 ms 2024-03-24T00:32:23.707 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 472 sec latency 585298.402 ms 2024-03-24T00:32:24.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:24 smithi078 ceph-mon[32429]: pgmap v382: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T00:32:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:24 smithi107 ceph-mon[36485]: pgmap v382: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T00:32:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:24 smithi106 ceph-mon[36505]: pgmap v382: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T00:32:24.707 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 473 sec latency 586298.521 ms 2024-03-24T00:32:25.439 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:25 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.g.service: Scheduled restart job, restart counter is at 5. 2024-03-24T00:32:25.439 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:25 smithi078 systemd[1]: Stopped Ceph mds.g for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:32:25.439 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:25 smithi078 systemd[1]: Starting Ceph mds.g for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:32:25.439 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:25 smithi078 podman[89442]: 2024-03-24 00:32:25.231916617 +0000 UTC m=+0.018474753 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:32:25.691 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:25 smithi078 podman[89442]: 2024-03-24 00:32:25.439612937 +0000 UTC m=+0.226171054 container create bb815871cfc947409d8912c7976ef39f3cb9040f26af16d2b96372c8882e98df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:32:25.691 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:25 smithi078 podman[89442]: 2024-03-24 00:32:25.667062154 +0000 UTC m=+0.453620280 container init bb815871cfc947409d8912c7976ef39f3cb9040f26af16d2b96372c8882e98df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:32:25.691 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:25 smithi078 podman[89442]: 2024-03-24 00:32:25.669986302 +0000 UTC m=+0.456544429 container start bb815871cfc947409d8912c7976ef39f3cb9040f26af16d2b96372c8882e98df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-24T00:32:25.707 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 474 sec latency 587298.565 ms 2024-03-24T00:32:26.090 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:25 smithi078 ceph-mds[89527]: -- 172.21.15.78:0/1185198229 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55becbba3a40 con 0x55becbb77000 2024-03-24T00:32:26.090 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:25 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: starting mds.g at 2024-03-24T00:32:26.091 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:25 smithi078 bash[89442]: bb815871cfc947409d8912c7976ef39f3cb9040f26af16d2b96372c8882e98df 2024-03-24T00:32:26.091 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:25 smithi078 systemd[1]: Started Ceph mds.g for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:32:26.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:26 smithi078 ceph-mon[32429]: pgmap v383: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T00:32:26.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:26 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:32:26.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:26 smithi078 ceph-mon[32429]: daemon mds.g restarted 2024-03-24T00:32:26.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:26 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:32:26.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:26 smithi078 ceph-mon[32429]: osdmap e137: 12 total, 12 up, 12 in 2024-03-24T00:32:26.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:26 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6838/2480119933,v1:172.21.15.78:6839/2480119933] up:boot 2024-03-24T00:32:26.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:26 smithi078 ceph-mon[32429]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-24T00:32:26.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:26 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-03-24T00:32:26.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:26 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:32:26.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:26 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:26.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:26 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:32:26.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:26 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:32:26.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:26 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:32:26.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:26 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:32:26.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:26 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:32:26.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:26 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:32:26.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:26 smithi107 ceph-mon[36485]: pgmap v383: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T00:32:26.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:26 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:32:26.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:26 smithi107 ceph-mon[36485]: daemon mds.g restarted 2024-03-24T00:32:26.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:26 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:32:26.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:26 smithi107 ceph-mon[36485]: osdmap e137: 12 total, 12 up, 12 in 2024-03-24T00:32:26.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:26 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6838/2480119933,v1:172.21.15.78:6839/2480119933] up:boot 2024-03-24T00:32:26.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:26 smithi107 ceph-mon[36485]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-24T00:32:26.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:26 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-03-24T00:32:26.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:26 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:32:26.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:26 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:26.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:26 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:32:26.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:26 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:32:26.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:26 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:32:26.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:26 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:32:26.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:26 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:32:26.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:26 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:32:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:26 smithi106 ceph-mon[36505]: pgmap v383: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T00:32:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:26 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:32:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:26 smithi106 ceph-mon[36505]: daemon mds.g restarted 2024-03-24T00:32:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:26 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:32:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:26 smithi106 ceph-mon[36505]: osdmap e137: 12 total, 12 up, 12 in 2024-03-24T00:32:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:26 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6838/2480119933,v1:172.21.15.78:6839/2480119933] up:boot 2024-03-24T00:32:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:26 smithi106 ceph-mon[36505]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-24T00:32:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:26 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-03-24T00:32:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:26 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:32:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:26 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:26 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:32:26.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:26 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:32:26.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:26 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:32:26.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:26 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:32:26.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:26 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:32:26.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:26 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:32:26.707 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 475 sec latency 588298.618 ms 2024-03-24T00:32:27.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:27 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:32:27.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:27 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:32:27.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:27 smithi078 ceph-mon[32429]: pgmap v385: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 220 B/s wr, 6 op/s 2024-03-24T00:32:27.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:27 smithi078 ceph-mon[32429]: pgmap v386: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 280 B/s wr, 8 op/s 2024-03-24T00:32:27.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:27 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:32:27.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:27 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6836/2239323994,v1:172.21.15.78:6837/2239323994] up:resolve 2024-03-24T00:32:27.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:27 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:27.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:27 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:32:27.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:27 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:32:27.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:27 smithi107 ceph-mon[36485]: pgmap v385: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 220 B/s wr, 6 op/s 2024-03-24T00:32:27.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:27 smithi107 ceph-mon[36485]: pgmap v386: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 280 B/s wr, 8 op/s 2024-03-24T00:32:27.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:27 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:32:27.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:27 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6836/2239323994,v1:172.21.15.78:6837/2239323994] up:resolve 2024-03-24T00:32:27.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:27 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:27.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:27 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:32:27.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:27 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:32:27.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:27 smithi106 ceph-mon[36505]: pgmap v385: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 220 B/s wr, 6 op/s 2024-03-24T00:32:27.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:27 smithi106 ceph-mon[36505]: pgmap v386: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 280 B/s wr, 8 op/s 2024-03-24T00:32:27.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:27 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:32:27.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:27 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6836/2239323994,v1:172.21.15.78:6837/2239323994] up:resolve 2024-03-24T00:32:27.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:27 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:27.707 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 476 sec latency 589298.676 ms 2024-03-24T00:32:28.708 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 477 sec latency 590298.735 ms 2024-03-24T00:32:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:28 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:32:29.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:28 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6836/2239323994,v1:172.21.15.78:6837/2239323994] up:reconnect 2024-03-24T00:32:29.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:28 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:29.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:28 smithi106 ceph-mon[36505]: pgmap v387: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 140 B/s wr, 4 op/s 2024-03-24T00:32:29.161 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:28 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:32:29.161 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:28 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6836/2239323994,v1:172.21.15.78:6837/2239323994] up:reconnect 2024-03-24T00:32:29.161 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:28 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:29.161 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:28 smithi078 ceph-mon[32429]: pgmap v387: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 140 B/s wr, 4 op/s 2024-03-24T00:32:29.162 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3810a33700 time 2024-03-24T00:32:29.108133+0000 2024-03-24T00:32:29.162 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:32:29.162 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:32:29.162 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f381b05d4a1] 2024-03-24T00:32:29.162 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f381b05d667] 2024-03-24T00:32:29.162 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56220de35223] 2024-03-24T00:32:29.163 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56220dcaf5d6] 2024-03-24T00:32:29.163 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56220dcafda1] 2024-03-24T00:32:29.163 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56220dcb22d5] 2024-03-24T00:32:29.163 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56220dd02233] 2024-03-24T00:32:29.163 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56220dd04197] 2024-03-24T00:32:29.163 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56220dc0311b] 2024-03-24T00:32:29.163 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56220dc036bc] 2024-03-24T00:32:29.163 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56220dbed1ff] 2024-03-24T00:32:29.163 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f381b2ef488] 2024-03-24T00:32:29.163 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 13: (DispatchQueue::entry()+0x51f) [0x7f381b2ec60f] 2024-03-24T00:32:29.163 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f381b3bfe81] 2024-03-24T00:32:29.163 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3819dda1ca] 2024-03-24T00:32:29.163 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 16: clone() 2024-03-24T00:32:29.164 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 2024-03-24T00:32:29.109+0000 7f3810a33700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3810a33700 time 2024-03-24T00:32:29.108133+0000 2024-03-24T00:32:29.164 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:32:29.164 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 2024-03-24T00:32:29.164 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:32:29.164 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f381b05d4a1] 2024-03-24T00:32:29.164 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f381b05d667] 2024-03-24T00:32:29.164 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56220de35223] 2024-03-24T00:32:29.164 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56220dcaf5d6] 2024-03-24T00:32:29.164 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56220dcafda1] 2024-03-24T00:32:29.164 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56220dcb22d5] 2024-03-24T00:32:29.164 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56220dd02233] 2024-03-24T00:32:29.164 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56220dd04197] 2024-03-24T00:32:29.164 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56220dc0311b] 2024-03-24T00:32:29.164 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56220dc036bc] 2024-03-24T00:32:29.165 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56220dbed1ff] 2024-03-24T00:32:29.165 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f381b2ef488] 2024-03-24T00:32:29.165 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 13: (DispatchQueue::entry()+0x51f) [0x7f381b2ec60f] 2024-03-24T00:32:29.165 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f381b3bfe81] 2024-03-24T00:32:29.165 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3819dda1ca] 2024-03-24T00:32:29.165 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 16: clone() 2024-03-24T00:32:29.165 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 2024-03-24T00:32:29.165 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: *** Caught signal (Aborted) ** 2024-03-24T00:32:29.165 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: in thread 7f3810a33700 thread_name:ms_dispatch 2024-03-24T00:32:29.165 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:32:29.165 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3819de4d20] 2024-03-24T00:32:29.166 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 2: gsignal() 2024-03-24T00:32:29.166 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 3: abort() 2024-03-24T00:32:29.167 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f381b05d4fb] 2024-03-24T00:32:29.167 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f381b05d667] 2024-03-24T00:32:29.167 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56220de35223] 2024-03-24T00:32:29.167 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56220dcaf5d6] 2024-03-24T00:32:29.167 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56220dcafda1] 2024-03-24T00:32:29.167 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56220dcb22d5] 2024-03-24T00:32:29.167 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56220dd02233] 2024-03-24T00:32:29.167 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56220dd04197] 2024-03-24T00:32:29.167 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56220dc0311b] 2024-03-24T00:32:29.167 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56220dc036bc] 2024-03-24T00:32:29.167 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56220dbed1ff] 2024-03-24T00:32:29.167 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f381b2ef488] 2024-03-24T00:32:29.167 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 16: (DispatchQueue::entry()+0x51f) [0x7f381b2ec60f] 2024-03-24T00:32:29.168 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f381b3bfe81] 2024-03-24T00:32:29.168 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3819dda1ca] 2024-03-24T00:32:29.168 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 19: clone() 2024-03-24T00:32:29.168 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 2024-03-24T00:32:29.111+0000 7f3810a33700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:32:29.168 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: in thread 7f3810a33700 thread_name:ms_dispatch 2024-03-24T00:32:29.168 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 2024-03-24T00:32:29.168 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:32:29.168 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3819de4d20] 2024-03-24T00:32:29.168 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 2: gsignal() 2024-03-24T00:32:29.168 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 3: abort() 2024-03-24T00:32:29.168 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f381b05d4fb] 2024-03-24T00:32:29.168 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f381b05d667] 2024-03-24T00:32:29.168 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56220de35223] 2024-03-24T00:32:29.168 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56220dcaf5d6] 2024-03-24T00:32:29.169 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56220dcafda1] 2024-03-24T00:32:29.169 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56220dcb22d5] 2024-03-24T00:32:29.169 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56220dd02233] 2024-03-24T00:32:29.169 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56220dd04197] 2024-03-24T00:32:29.169 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56220dc0311b] 2024-03-24T00:32:29.169 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56220dc036bc] 2024-03-24T00:32:29.169 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56220dbed1ff] 2024-03-24T00:32:29.169 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f381b2ef488] 2024-03-24T00:32:29.169 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 16: (DispatchQueue::entry()+0x51f) [0x7f381b2ec60f] 2024-03-24T00:32:29.169 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f381b3bfe81] 2024-03-24T00:32:29.169 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3819dda1ca] 2024-03-24T00:32:29.169 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 19: clone() 2024-03-24T00:32:29.169 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:32:29.169 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 2024-03-24T00:32:29.169 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: -1> 2024-03-24T00:32:29.109+0000 7f3810a33700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3810a33700 time 2024-03-24T00:32:29.108133+0000 2024-03-24T00:32:29.170 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:32:29.170 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 2024-03-24T00:32:29.170 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:32:29.170 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f381b05d4a1] 2024-03-24T00:32:29.170 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f381b05d667] 2024-03-24T00:32:29.170 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56220de35223] 2024-03-24T00:32:29.170 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56220dcaf5d6] 2024-03-24T00:32:29.170 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56220dcafda1] 2024-03-24T00:32:29.170 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56220dcb22d5] 2024-03-24T00:32:29.170 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56220dd02233] 2024-03-24T00:32:29.170 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56220dd04197] 2024-03-24T00:32:29.170 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56220dc0311b] 2024-03-24T00:32:29.170 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56220dc036bc] 2024-03-24T00:32:29.171 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56220dbed1ff] 2024-03-24T00:32:29.171 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f381b2ef488] 2024-03-24T00:32:29.171 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 13: (DispatchQueue::entry()+0x51f) [0x7f381b2ec60f] 2024-03-24T00:32:29.171 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f381b3bfe81] 2024-03-24T00:32:29.171 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3819dda1ca] 2024-03-24T00:32:29.171 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 16: clone() 2024-03-24T00:32:29.171 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 2024-03-24T00:32:29.171 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 0> 2024-03-24T00:32:29.111+0000 7f3810a33700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:32:29.171 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: in thread 7f3810a33700 thread_name:ms_dispatch 2024-03-24T00:32:29.171 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 2024-03-24T00:32:29.171 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:32:29.171 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3819de4d20] 2024-03-24T00:32:29.171 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 2: gsignal() 2024-03-24T00:32:29.171 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 3: abort() 2024-03-24T00:32:29.172 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f381b05d4fb] 2024-03-24T00:32:29.172 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f381b05d667] 2024-03-24T00:32:29.172 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56220de35223] 2024-03-24T00:32:29.172 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56220dcaf5d6] 2024-03-24T00:32:29.172 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56220dcafda1] 2024-03-24T00:32:29.172 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56220dcb22d5] 2024-03-24T00:32:29.172 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56220dd02233] 2024-03-24T00:32:29.172 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56220dd04197] 2024-03-24T00:32:29.172 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56220dc0311b] 2024-03-24T00:32:29.172 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56220dc036bc] 2024-03-24T00:32:29.172 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56220dbed1ff] 2024-03-24T00:32:29.172 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f381b2ef488] 2024-03-24T00:32:29.172 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 16: (DispatchQueue::entry()+0x51f) [0x7f381b2ec60f] 2024-03-24T00:32:29.172 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f381b3bfe81] 2024-03-24T00:32:29.172 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3819dda1ca] 2024-03-24T00:32:29.173 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 19: clone() 2024-03-24T00:32:29.173 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:32:29.173 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 2024-03-24T00:32:29.173 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: -1> 2024-03-24T00:32:29.109+0000 7f3810a33700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3810a33700 time 2024-03-24T00:32:29.108133+0000 2024-03-24T00:32:29.173 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:32:29.173 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 2024-03-24T00:32:29.173 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:32:29.173 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f381b05d4a1] 2024-03-24T00:32:29.173 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f381b05d667] 2024-03-24T00:32:29.173 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56220de35223] 2024-03-24T00:32:29.173 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56220dcaf5d6] 2024-03-24T00:32:29.173 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56220dcafda1] 2024-03-24T00:32:29.173 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56220dcb22d5] 2024-03-24T00:32:29.173 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56220dd02233] 2024-03-24T00:32:29.174 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56220dd04197] 2024-03-24T00:32:29.174 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56220dc0311b] 2024-03-24T00:32:29.174 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56220dc036bc] 2024-03-24T00:32:29.174 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56220dbed1ff] 2024-03-24T00:32:29.174 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f381b2ef488] 2024-03-24T00:32:29.174 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 13: (DispatchQueue::entry()+0x51f) [0x7f381b2ec60f] 2024-03-24T00:32:29.174 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f381b3bfe81] 2024-03-24T00:32:29.174 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3819dda1ca] 2024-03-24T00:32:29.174 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 16: clone() 2024-03-24T00:32:29.174 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 2024-03-24T00:32:29.174 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 0> 2024-03-24T00:32:29.111+0000 7f3810a33700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:32:29.174 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: in thread 7f3810a33700 thread_name:ms_dispatch 2024-03-24T00:32:29.174 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 2024-03-24T00:32:29.174 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:32:29.175 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3819de4d20] 2024-03-24T00:32:29.175 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 2: gsignal() 2024-03-24T00:32:29.175 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 3: abort() 2024-03-24T00:32:29.175 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f381b05d4fb] 2024-03-24T00:32:29.175 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f381b05d667] 2024-03-24T00:32:29.175 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56220de35223] 2024-03-24T00:32:29.175 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56220dcaf5d6] 2024-03-24T00:32:29.175 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56220dcafda1] 2024-03-24T00:32:29.175 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56220dcb22d5] 2024-03-24T00:32:29.175 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56220dd02233] 2024-03-24T00:32:29.175 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56220dd04197] 2024-03-24T00:32:29.175 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56220dc0311b] 2024-03-24T00:32:29.175 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56220dc036bc] 2024-03-24T00:32:29.175 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56220dbed1ff] 2024-03-24T00:32:29.175 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f381b2ef488] 2024-03-24T00:32:29.176 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 16: (DispatchQueue::entry()+0x51f) [0x7f381b2ec60f] 2024-03-24T00:32:29.176 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f381b3bfe81] 2024-03-24T00:32:29.176 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3819dda1ca] 2024-03-24T00:32:29.176 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 19: clone() 2024-03-24T00:32:29.176 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:32:29.176 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89304]: 2024-03-24T00:32:29.176 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 podman[89710]: 2024-03-24 00:32:29.162211474 +0000 UTC m=+0.020089109 container died 5ce9b31bbc3d3b64eb7034a89ac6a41336a0f63c3282d078511d9235866d9ef0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-24T00:32:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:28 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:32:29.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:28 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6836/2239323994,v1:172.21.15.78:6837/2239323994] up:reconnect 2024-03-24T00:32:29.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:28 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:29.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:28 smithi107 ceph-mon[36485]: pgmap v387: 129 pgs: 129 active+clean; 27 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 140 B/s wr, 4 op/s 2024-03-24T00:32:29.708 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 478 sec latency 591298.788 ms 2024-03-24T00:32:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6836/2239323994,v1:172.21.15.78:6837/2239323994] up:rejoin 2024-03-24T00:32:30.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:30.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-mon[32429]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-24T00:32:30.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:32:30.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6836/2239323994,v1:172.21.15.78:6837/2239323994] up:active 2024-03-24T00:32:30.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:29 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=h=up:active} 3 up:standby-replay 1 up:standby 2024-03-24T00:32:30.091 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 podman[89710]: 2024-03-24 00:32:29.823081113 +0000 UTC m=+0.680958807 container remove 5ce9b31bbc3d3b64eb7034a89ac6a41336a0f63c3282d078511d9235866d9ef0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-24T00:32:30.092 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:29 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:32:30.092 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:30 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.d.service: Failed with result 'exit-code'. 2024-03-24T00:32:30.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:29 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6836/2239323994,v1:172.21.15.78:6837/2239323994] up:rejoin 2024-03-24T00:32:30.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:29 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:30.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:29 smithi106 ceph-mon[36505]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-24T00:32:30.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:29 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:32:30.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:29 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6836/2239323994,v1:172.21.15.78:6837/2239323994] up:active 2024-03-24T00:32:30.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:29 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=h=up:active} 3 up:standby-replay 1 up:standby 2024-03-24T00:32:30.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:29 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6836/2239323994,v1:172.21.15.78:6837/2239323994] up:rejoin 2024-03-24T00:32:30.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:29 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:30.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:29 smithi107 ceph-mon[36485]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-24T00:32:30.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:29 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:32:30.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:29 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6836/2239323994,v1:172.21.15.78:6837/2239323994] up:active 2024-03-24T00:32:30.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:29 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=h=up:active} 3 up:standby-replay 1 up:standby 2024-03-24T00:32:30.708 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 479 sec latency 592298.879 ms 2024-03-24T00:32:31.100 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:30 smithi078 ceph-mon[32429]: pgmap v388: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 37 KiB/s wr, 19 op/s 2024-03-24T00:32:31.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:30 smithi107 ceph-mon[36485]: pgmap v388: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 37 KiB/s wr, 19 op/s 2024-03-24T00:32:31.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:30 smithi106 ceph-mon[36505]: pgmap v388: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 37 KiB/s wr, 19 op/s 2024-03-24T00:32:31.708 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 480 sec latency 593298.928 ms 2024-03-24T00:32:32.708 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 481 sec latency 594298.984 ms 2024-03-24T00:32:32.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:32 smithi078 ceph-mon[32429]: pgmap v389: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 37 KiB/s wr, 19 op/s 2024-03-24T00:32:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:32 smithi106 ceph-mon[36505]: pgmap v389: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 37 KiB/s wr, 19 op/s 2024-03-24T00:32:33.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:32 smithi107 ceph-mon[36485]: pgmap v389: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 37 KiB/s wr, 19 op/s 2024-03-24T00:32:33.708 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 482 sec latency 595299.016 ms 2024-03-24T00:32:34.708 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 483 sec latency 596299.051 ms 2024-03-24T00:32:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:34 smithi078 ceph-mon[32429]: pgmap v390: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 30 KiB/s wr, 12 op/s 2024-03-24T00:32:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:34 smithi106 ceph-mon[36505]: pgmap v390: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 30 KiB/s wr, 12 op/s 2024-03-24T00:32:35.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:34 smithi107 ceph-mon[36485]: pgmap v390: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 30 KiB/s wr, 12 op/s 2024-03-24T00:32:35.708 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 484 sec latency 597299.163 ms 2024-03-24T00:32:36.708 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 485 sec latency 598299.257 ms 2024-03-24T00:32:36.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:36 smithi106 ceph-mon[36505]: pgmap v391: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:32:37.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:36 smithi078 ceph-mon[32429]: pgmap v391: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:32:37.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:36 smithi107 ceph-mon[36485]: pgmap v391: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:32:37.708 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 486 sec latency 599299.306 ms 2024-03-24T00:32:38.708 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 487 sec latency 600299.425 ms 2024-03-24T00:32:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:38 smithi078 ceph-mon[32429]: osdmap e138: 12 total, 12 up, 12 in 2024-03-24T00:32:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:38 smithi078 ceph-mon[32429]: pgmap v393: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:32:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:38 smithi107 ceph-mon[36485]: osdmap e138: 12 total, 12 up, 12 in 2024-03-24T00:32:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:38 smithi107 ceph-mon[36485]: pgmap v393: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:32:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:38 smithi106 ceph-mon[36505]: osdmap e138: 12 total, 12 up, 12 in 2024-03-24T00:32:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:38 smithi106 ceph-mon[36505]: pgmap v393: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:32:39.708 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 488 sec latency 601299.511 ms 2024-03-24T00:32:40.349 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:40 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.d.service: Scheduled restart job, restart counter is at 2. 2024-03-24T00:32:40.349 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:40 smithi078 systemd[1]: Stopped Ceph mds.d for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:32:40.350 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:40 smithi078 systemd[1]: Starting Ceph mds.d for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:32:40.350 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:40 smithi078 podman[89816]: 2024-03-24 00:32:40.230272145 +0000 UTC m=+0.017895565 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:32:40.610 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:40 smithi078 podman[89816]: 2024-03-24 00:32:40.349111632 +0000 UTC m=+0.136735057 container create 3be55a373ac667c1947343c7a47a56c472fa7d4e09d2eb66d5c350b0e4e74d03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-24T00:32:40.610 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:40 smithi078 podman[89816]: 2024-03-24 00:32:40.585199563 +0000 UTC m=+0.372822979 container init 3be55a373ac667c1947343c7a47a56c472fa7d4e09d2eb66d5c350b0e4e74d03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=) 2024-03-24T00:32:40.610 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:40 smithi078 podman[89816]: 2024-03-24 00:32:40.5883201 +0000 UTC m=+0.375943516 container start 3be55a373ac667c1947343c7a47a56c472fa7d4e09d2eb66d5c350b0e4e74d03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-24T00:32:40.708 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 489 sec latency 602299.619 ms 2024-03-24T00:32:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:40 smithi106 ceph-mon[36505]: pgmap v394: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:32:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:40 smithi078 ceph-mon[32429]: pgmap v394: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:32:41.091 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:40 smithi078 ceph-mds[89836]: -- 172.21.15.78:0/1093449613 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56315de79a40 con 0x56315de4cc00 2024-03-24T00:32:41.091 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:40 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: starting mds.d at 2024-03-24T00:32:41.091 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:40 smithi078 bash[89816]: 3be55a373ac667c1947343c7a47a56c472fa7d4e09d2eb66d5c350b0e4e74d03 2024-03-24T00:32:41.091 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:32:40 smithi078 systemd[1]: Started Ceph mds.d for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:32:41.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:40 smithi107 ceph-mon[36485]: pgmap v394: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:32:41.708 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 490 sec latency 603299.686 ms 2024-03-24T00:32:41.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:41 smithi106 ceph-mon[36505]: daemon mds.d restarted 2024-03-24T00:32:41.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:41 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:32:41.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:41 smithi106 ceph-mon[36505]: osdmap e139: 12 total, 12 up, 12 in 2024-03-24T00:32:41.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:41 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6836/1077781429,v1:172.21.15.78:6837/1077781429] up:boot 2024-03-24T00:32:41.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:41 smithi106 ceph-mon[36505]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-03-24T00:32:41.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:41 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-03-24T00:32:41.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:41 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:32:41.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:41 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:42.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:41 smithi078 ceph-mon[32429]: daemon mds.d restarted 2024-03-24T00:32:42.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:41 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:32:42.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:41 smithi078 ceph-mon[32429]: osdmap e139: 12 total, 12 up, 12 in 2024-03-24T00:32:42.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:41 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6836/1077781429,v1:172.21.15.78:6837/1077781429] up:boot 2024-03-24T00:32:42.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:41 smithi078 ceph-mon[32429]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-03-24T00:32:42.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:41 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-03-24T00:32:42.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:41 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:32:42.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:41 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:42.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:41 smithi107 ceph-mon[36485]: daemon mds.d restarted 2024-03-24T00:32:42.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:41 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:32:42.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:41 smithi107 ceph-mon[36485]: osdmap e139: 12 total, 12 up, 12 in 2024-03-24T00:32:42.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:41 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6836/1077781429,v1:172.21.15.78:6837/1077781429] up:boot 2024-03-24T00:32:42.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:41 smithi107 ceph-mon[36485]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-03-24T00:32:42.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:41 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-03-24T00:32:42.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:41 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:32:42.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:41 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:replay,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:42.708 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 491 sec latency 604299.737 ms 2024-03-24T00:32:42.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:42 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6838/2480119933,v1:172.21.15.78:6839/2480119933] up:resolve 2024-03-24T00:32:42.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:42 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:42.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:42 smithi106 ceph-mon[36505]: pgmap v396: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-24T00:32:43.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:42 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6838/2480119933,v1:172.21.15.78:6839/2480119933] up:resolve 2024-03-24T00:32:43.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:42 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:43.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:42 smithi078 ceph-mon[32429]: pgmap v396: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-24T00:32:43.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:42 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6838/2480119933,v1:172.21.15.78:6839/2480119933] up:resolve 2024-03-24T00:32:43.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:42 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:resolve,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:43.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:42 smithi107 ceph-mon[36485]: pgmap v396: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-24T00:32:43.709 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 492 sec latency 605299.821 ms 2024-03-24T00:32:43.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:43 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:32:43.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:43 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6838/2480119933,v1:172.21.15.78:6839/2480119933] up:reconnect 2024-03-24T00:32:43.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:43 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:43 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:32:43.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:43 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6838/2480119933,v1:172.21.15.78:6839/2480119933] up:reconnect 2024-03-24T00:32:43.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:43 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:43 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:32:44.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:43 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6838/2480119933,v1:172.21.15.78:6839/2480119933] up:reconnect 2024-03-24T00:32:44.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:43 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:reconnect,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:44.471 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f5a4d731700 time 2024-03-24T00:32:44.123382+0000 2024-03-24T00:32:44.471 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:32:44.471 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:32:44.471 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f5a57d5b4a1] 2024-03-24T00:32:44.471 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f5a57d5b667] 2024-03-24T00:32:44.472 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55beca157223] 2024-03-24T00:32:44.472 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bec9fd15d6] 2024-03-24T00:32:44.472 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bec9fd1da1] 2024-03-24T00:32:44.472 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bec9fd42d5] 2024-03-24T00:32:44.472 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55beca024233] 2024-03-24T00:32:44.472 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55beca026197] 2024-03-24T00:32:44.472 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bec9f2511b] 2024-03-24T00:32:44.472 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bec9f256bc] 2024-03-24T00:32:44.472 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bec9f0f1ff] 2024-03-24T00:32:44.472 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5a57fed488] 2024-03-24T00:32:44.472 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 13: (DispatchQueue::entry()+0x51f) [0x7f5a57fea60f] 2024-03-24T00:32:44.472 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5a580bde81] 2024-03-24T00:32:44.472 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5a56ad81ca] 2024-03-24T00:32:44.472 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 16: clone() 2024-03-24T00:32:44.473 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 2024-03-24T00:32:44.124+0000 7f5a4d731700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f5a4d731700 time 2024-03-24T00:32:44.123382+0000 2024-03-24T00:32:44.473 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:32:44.474 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 2024-03-24T00:32:44.474 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:32:44.474 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f5a57d5b4a1] 2024-03-24T00:32:44.474 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f5a57d5b667] 2024-03-24T00:32:44.474 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55beca157223] 2024-03-24T00:32:44.474 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bec9fd15d6] 2024-03-24T00:32:44.474 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bec9fd1da1] 2024-03-24T00:32:44.474 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bec9fd42d5] 2024-03-24T00:32:44.474 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55beca024233] 2024-03-24T00:32:44.474 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55beca026197] 2024-03-24T00:32:44.474 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bec9f2511b] 2024-03-24T00:32:44.474 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bec9f256bc] 2024-03-24T00:32:44.475 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bec9f0f1ff] 2024-03-24T00:32:44.475 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5a57fed488] 2024-03-24T00:32:44.475 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 13: (DispatchQueue::entry()+0x51f) [0x7f5a57fea60f] 2024-03-24T00:32:44.475 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5a580bde81] 2024-03-24T00:32:44.475 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5a56ad81ca] 2024-03-24T00:32:44.475 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 16: clone() 2024-03-24T00:32:44.475 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 2024-03-24T00:32:44.475 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: *** Caught signal (Aborted) ** 2024-03-24T00:32:44.475 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: in thread 7f5a4d731700 thread_name:ms_dispatch 2024-03-24T00:32:44.475 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:32:44.475 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5a56ae2d20] 2024-03-24T00:32:44.475 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 2: gsignal() 2024-03-24T00:32:44.475 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 3: abort() 2024-03-24T00:32:44.475 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f5a57d5b4fb] 2024-03-24T00:32:44.476 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f5a57d5b667] 2024-03-24T00:32:44.476 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55beca157223] 2024-03-24T00:32:44.476 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bec9fd15d6] 2024-03-24T00:32:44.476 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bec9fd1da1] 2024-03-24T00:32:44.476 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bec9fd42d5] 2024-03-24T00:32:44.476 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55beca024233] 2024-03-24T00:32:44.476 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55beca026197] 2024-03-24T00:32:44.476 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bec9f2511b] 2024-03-24T00:32:44.476 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bec9f256bc] 2024-03-24T00:32:44.476 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bec9f0f1ff] 2024-03-24T00:32:44.476 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5a57fed488] 2024-03-24T00:32:44.476 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 16: (DispatchQueue::entry()+0x51f) [0x7f5a57fea60f] 2024-03-24T00:32:44.476 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5a580bde81] 2024-03-24T00:32:44.476 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5a56ad81ca] 2024-03-24T00:32:44.477 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 19: clone() 2024-03-24T00:32:44.477 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 2024-03-24T00:32:44.126+0000 7f5a4d731700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:32:44.477 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: in thread 7f5a4d731700 thread_name:ms_dispatch 2024-03-24T00:32:44.477 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 2024-03-24T00:32:44.477 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:32:44.477 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5a56ae2d20] 2024-03-24T00:32:44.477 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 2: gsignal() 2024-03-24T00:32:44.477 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 3: abort() 2024-03-24T00:32:44.477 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f5a57d5b4fb] 2024-03-24T00:32:44.477 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f5a57d5b667] 2024-03-24T00:32:44.477 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55beca157223] 2024-03-24T00:32:44.477 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bec9fd15d6] 2024-03-24T00:32:44.477 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bec9fd1da1] 2024-03-24T00:32:44.477 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bec9fd42d5] 2024-03-24T00:32:44.478 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55beca024233] 2024-03-24T00:32:44.478 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55beca026197] 2024-03-24T00:32:44.478 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bec9f2511b] 2024-03-24T00:32:44.478 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bec9f256bc] 2024-03-24T00:32:44.478 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bec9f0f1ff] 2024-03-24T00:32:44.478 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5a57fed488] 2024-03-24T00:32:44.478 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 16: (DispatchQueue::entry()+0x51f) [0x7f5a57fea60f] 2024-03-24T00:32:44.478 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5a580bde81] 2024-03-24T00:32:44.478 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5a56ad81ca] 2024-03-24T00:32:44.478 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 19: clone() 2024-03-24T00:32:44.478 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:32:44.478 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 2024-03-24T00:32:44.478 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: -1> 2024-03-24T00:32:44.124+0000 7f5a4d731700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f5a4d731700 time 2024-03-24T00:32:44.123382+0000 2024-03-24T00:32:44.479 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:32:44.479 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 2024-03-24T00:32:44.479 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:32:44.479 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f5a57d5b4a1] 2024-03-24T00:32:44.479 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f5a57d5b667] 2024-03-24T00:32:44.479 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55beca157223] 2024-03-24T00:32:44.479 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bec9fd15d6] 2024-03-24T00:32:44.479 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bec9fd1da1] 2024-03-24T00:32:44.479 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bec9fd42d5] 2024-03-24T00:32:44.479 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55beca024233] 2024-03-24T00:32:44.479 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55beca026197] 2024-03-24T00:32:44.479 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bec9f2511b] 2024-03-24T00:32:44.479 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bec9f256bc] 2024-03-24T00:32:44.479 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bec9f0f1ff] 2024-03-24T00:32:44.479 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5a57fed488] 2024-03-24T00:32:44.480 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 13: (DispatchQueue::entry()+0x51f) [0x7f5a57fea60f] 2024-03-24T00:32:44.480 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5a580bde81] 2024-03-24T00:32:44.480 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5a56ad81ca] 2024-03-24T00:32:44.480 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 16: clone() 2024-03-24T00:32:44.480 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 2024-03-24T00:32:44.480 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 0> 2024-03-24T00:32:44.126+0000 7f5a4d731700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:32:44.480 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: in thread 7f5a4d731700 thread_name:ms_dispatch 2024-03-24T00:32:44.480 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 2024-03-24T00:32:44.480 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:32:44.480 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5a56ae2d20] 2024-03-24T00:32:44.480 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 2: gsignal() 2024-03-24T00:32:44.480 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 3: abort() 2024-03-24T00:32:44.480 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f5a57d5b4fb] 2024-03-24T00:32:44.480 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f5a57d5b667] 2024-03-24T00:32:44.480 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55beca157223] 2024-03-24T00:32:44.481 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bec9fd15d6] 2024-03-24T00:32:44.481 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bec9fd1da1] 2024-03-24T00:32:44.481 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bec9fd42d5] 2024-03-24T00:32:44.481 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55beca024233] 2024-03-24T00:32:44.481 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55beca026197] 2024-03-24T00:32:44.481 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bec9f2511b] 2024-03-24T00:32:44.481 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bec9f256bc] 2024-03-24T00:32:44.481 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bec9f0f1ff] 2024-03-24T00:32:44.481 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5a57fed488] 2024-03-24T00:32:44.481 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 16: (DispatchQueue::entry()+0x51f) [0x7f5a57fea60f] 2024-03-24T00:32:44.481 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5a580bde81] 2024-03-24T00:32:44.481 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5a56ad81ca] 2024-03-24T00:32:44.481 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 19: clone() 2024-03-24T00:32:44.481 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:32:44.482 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 2024-03-24T00:32:44.482 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: -1> 2024-03-24T00:32:44.124+0000 7f5a4d731700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f5a4d731700 time 2024-03-24T00:32:44.123382+0000 2024-03-24T00:32:44.482 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:32:44.482 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 2024-03-24T00:32:44.482 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:32:44.482 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f5a57d5b4a1] 2024-03-24T00:32:44.482 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f5a57d5b667] 2024-03-24T00:32:44.482 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55beca157223] 2024-03-24T00:32:44.482 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bec9fd15d6] 2024-03-24T00:32:44.483 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bec9fd1da1] 2024-03-24T00:32:44.483 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bec9fd42d5] 2024-03-24T00:32:44.483 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55beca024233] 2024-03-24T00:32:44.483 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55beca026197] 2024-03-24T00:32:44.483 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bec9f2511b] 2024-03-24T00:32:44.483 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bec9f256bc] 2024-03-24T00:32:44.483 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bec9f0f1ff] 2024-03-24T00:32:44.483 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5a57fed488] 2024-03-24T00:32:44.483 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 13: (DispatchQueue::entry()+0x51f) [0x7f5a57fea60f] 2024-03-24T00:32:44.484 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5a580bde81] 2024-03-24T00:32:44.484 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5a56ad81ca] 2024-03-24T00:32:44.484 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 16: clone() 2024-03-24T00:32:44.484 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 2024-03-24T00:32:44.484 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 0> 2024-03-24T00:32:44.126+0000 7f5a4d731700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:32:44.484 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: in thread 7f5a4d731700 thread_name:ms_dispatch 2024-03-24T00:32:44.484 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 2024-03-24T00:32:44.484 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:32:44.484 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5a56ae2d20] 2024-03-24T00:32:44.484 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 2: gsignal() 2024-03-24T00:32:44.484 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 3: abort() 2024-03-24T00:32:44.484 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f5a57d5b4fb] 2024-03-24T00:32:44.484 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f5a57d5b667] 2024-03-24T00:32:44.484 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55beca157223] 2024-03-24T00:32:44.484 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bec9fd15d6] 2024-03-24T00:32:44.485 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bec9fd1da1] 2024-03-24T00:32:44.485 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55bec9fd42d5] 2024-03-24T00:32:44.485 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55beca024233] 2024-03-24T00:32:44.485 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55beca026197] 2024-03-24T00:32:44.485 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bec9f2511b] 2024-03-24T00:32:44.485 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bec9f256bc] 2024-03-24T00:32:44.485 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bec9f0f1ff] 2024-03-24T00:32:44.485 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5a57fed488] 2024-03-24T00:32:44.485 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 16: (DispatchQueue::entry()+0x51f) [0x7f5a57fea60f] 2024-03-24T00:32:44.485 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5a580bde81] 2024-03-24T00:32:44.485 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5a56ad81ca] 2024-03-24T00:32:44.486 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 19: clone() 2024-03-24T00:32:44.486 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:32:44.486 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g[89523]: 2024-03-24T00:32:44.486 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 podman[89870]: 2024-03-24 00:32:44.175880452 +0000 UTC m=+0.019379917 container died bb815871cfc947409d8912c7976ef39f3cb9040f26af16d2b96372c8882e98df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-24T00:32:44.709 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 493 sec latency 606299.918 ms 2024-03-24T00:32:44.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6838/2480119933,v1:172.21.15.78:6839/2480119933] up:rejoin 2024-03-24T00:32:44.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:44.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-mon[32429]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-24T00:32:44.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:32:44.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6838/2480119933,v1:172.21.15.78:6839/2480119933] up:active 2024-03-24T00:32:44.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 1 up:standby 2024-03-24T00:32:44.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-mon[32429]: osdmap e140: 12 total, 12 up, 12 in 2024-03-24T00:32:44.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:44 smithi078 ceph-mon[32429]: pgmap v398: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 154 B/s wr, 4 op/s 2024-03-24T00:32:45.090 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 podman[89870]: 2024-03-24 00:32:44.798469014 +0000 UTC m=+0.641968476 container remove bb815871cfc947409d8912c7976ef39f3cb9040f26af16d2b96372c8882e98df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-24T00:32:45.090 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:44 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:32:45.091 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:45 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.g.service: Failed with result 'exit-code'. 2024-03-24T00:32:45.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:44 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6838/2480119933,v1:172.21.15.78:6839/2480119933] up:rejoin 2024-03-24T00:32:45.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:44 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:45.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:44 smithi107 ceph-mon[36485]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-24T00:32:45.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:44 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:32:45.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:44 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6838/2480119933,v1:172.21.15.78:6839/2480119933] up:active 2024-03-24T00:32:45.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:44 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 1 up:standby 2024-03-24T00:32:45.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:44 smithi107 ceph-mon[36485]: osdmap e140: 12 total, 12 up, 12 in 2024-03-24T00:32:45.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:44 smithi107 ceph-mon[36485]: pgmap v398: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 154 B/s wr, 4 op/s 2024-03-24T00:32:45.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:44 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6838/2480119933,v1:172.21.15.78:6839/2480119933] up:rejoin 2024-03-24T00:32:45.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:44 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=g=up:rejoin,2=h=up:active} 2 up:standby-replay 2 up:standby 2024-03-24T00:32:45.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:44 smithi106 ceph-mon[36505]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-24T00:32:45.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:44 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:32:45.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:44 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6838/2480119933,v1:172.21.15.78:6839/2480119933] up:active 2024-03-24T00:32:45.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:44 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 1 up:standby 2024-03-24T00:32:45.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:44 smithi106 ceph-mon[36505]: osdmap e140: 12 total, 12 up, 12 in 2024-03-24T00:32:45.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:44 smithi106 ceph-mon[36505]: pgmap v398: 129 pgs: 129 active+clean; 27 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 154 B/s wr, 4 op/s 2024-03-24T00:32:45.709 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 494 sec latency 607299.969 ms 2024-03-24T00:32:46.709 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 495 sec latency 608300.018 ms 2024-03-24T00:32:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:46 smithi106 ceph-mon[36505]: pgmap v399: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 511 B/s wr, 13 op/s 2024-03-24T00:32:46.972 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:46 smithi078 ceph-mon[32429]: pgmap v399: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 511 B/s wr, 13 op/s 2024-03-24T00:32:47.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:46 smithi107 ceph-mon[36485]: pgmap v399: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 511 B/s wr, 13 op/s 2024-03-24T00:32:47.709 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 496 sec latency 609300.054 ms 2024-03-24T00:32:48.709 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 497 sec latency 610300.139 ms 2024-03-24T00:32:48.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:48 smithi078 ceph-mon[32429]: pgmap v400: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 383 B/s wr, 9 op/s 2024-03-24T00:32:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:48 smithi106 ceph-mon[36505]: pgmap v400: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 383 B/s wr, 9 op/s 2024-03-24T00:32:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:48 smithi107 ceph-mon[36485]: pgmap v400: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 383 B/s wr, 9 op/s 2024-03-24T00:32:49.709 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 498 sec latency 611300.238 ms 2024-03-24T00:32:50.709 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 499 sec latency 612300.405 ms 2024-03-24T00:32:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:50 smithi078 ceph-mon[32429]: pgmap v401: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 309 B/s wr, 10 op/s 2024-03-24T00:32:50.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:50 smithi106 ceph-mon[36505]: pgmap v401: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 309 B/s wr, 10 op/s 2024-03-24T00:32:51.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:50 smithi107 ceph-mon[36485]: pgmap v401: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 309 B/s wr, 10 op/s 2024-03-24T00:32:51.709 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 500 sec latency 613300.524 ms 2024-03-24T00:32:52.709 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 501 sec latency 614300.581 ms 2024-03-24T00:32:52.896 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:52 smithi107 ceph-mon[36485]: pgmap v402: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-24T00:32:52.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:52 smithi106 ceph-mon[36505]: pgmap v402: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-24T00:32:53.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:52 smithi078 ceph-mon[32429]: pgmap v402: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-24T00:32:53.710 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 502 sec latency 615300.731 ms 2024-03-24T00:32:54.710 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 503 sec latency 616300.838 ms 2024-03-24T00:32:54.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:54 smithi078 ceph-mon[32429]: pgmap v403: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 293 B/s wr, 10 op/s 2024-03-24T00:32:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:54 smithi106 ceph-mon[36505]: pgmap v403: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 293 B/s wr, 10 op/s 2024-03-24T00:32:55.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:54 smithi107 ceph-mon[36485]: pgmap v403: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 293 B/s wr, 10 op/s 2024-03-24T00:32:55.590 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:55 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.g.service: Scheduled restart job, restart counter is at 6. 2024-03-24T00:32:55.590 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:55 smithi078 systemd[1]: Stopped Ceph mds.g for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:32:55.591 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:55 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.g.service: Start request repeated too quickly. 2024-03-24T00:32:55.591 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:55 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.g.service: Failed with result 'exit-code'. 2024-03-24T00:32:55.591 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 24 00:32:55 smithi078 systemd[1]: Failed to start Ceph mds.g for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:32:55.710 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 504 sec latency 617300.935 ms 2024-03-24T00:32:56.710 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 505 sec latency 618301.017 ms 2024-03-24T00:32:56.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:56 smithi106 ceph-mon[36505]: pgmap v404: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T00:32:57.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:56 smithi078 ceph-mon[32429]: pgmap v404: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T00:32:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:56 smithi107 ceph-mon[36485]: pgmap v404: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T00:32:57.710 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 506 sec latency 619301.125 ms 2024-03-24T00:32:58.710 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 507 sec latency 620301.196 ms 2024-03-24T00:32:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:58 smithi106 ceph-mon[36505]: pgmap v405: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:32:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:58 smithi078 ceph-mon[32429]: pgmap v405: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:32:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:58 smithi107 ceph-mon[36485]: pgmap v405: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:32:59.710 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 508 sec latency 621301.275 ms 2024-03-24T00:32:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:59 smithi106 ceph-mon[36505]: Replacing daemon mds.g as rank 1 with standby daemon mds.d 2024-03-24T00:32:59.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:59 smithi106 ceph-mon[36505]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-03-24T00:32:59.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:59 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:32:59.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:59 smithi106 ceph-mon[36505]: osdmap e141: 12 total, 12 up, 12 in 2024-03-24T00:32:59.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:32:59 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:59 smithi078 ceph-mon[32429]: Replacing daemon mds.g as rank 1 with standby daemon mds.d 2024-03-24T00:33:00.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:59 smithi078 ceph-mon[32429]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-03-24T00:33:00.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:59 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:33:00.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:59 smithi078 ceph-mon[32429]: osdmap e141: 12 total, 12 up, 12 in 2024-03-24T00:33:00.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:32:59 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:00.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:59 smithi107 ceph-mon[36485]: Replacing daemon mds.g as rank 1 with standby daemon mds.d 2024-03-24T00:33:00.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:59 smithi107 ceph-mon[36485]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-03-24T00:33:00.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:59 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:33:00.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:59 smithi107 ceph-mon[36485]: osdmap e141: 12 total, 12 up, 12 in 2024-03-24T00:33:00.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:32:59 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:00.710 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 509 sec latency 622301.342 ms 2024-03-24T00:33:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:01 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6836/1077781429,v1:172.21.15.78:6837/1077781429] up:resolve 2024-03-24T00:33:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:01 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:01.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:01 smithi106 ceph-mon[36505]: pgmap v407: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:33:01.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:01 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6836/1077781429,v1:172.21.15.78:6837/1077781429] up:resolve 2024-03-24T00:33:01.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:01 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:01.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:01 smithi078 ceph-mon[32429]: pgmap v407: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:33:01.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:01 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6836/1077781429,v1:172.21.15.78:6837/1077781429] up:resolve 2024-03-24T00:33:01.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:01 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:01.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:01 smithi107 ceph-mon[36485]: pgmap v407: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:33:01.710 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 510 sec latency 623301.460 ms 2024-03-24T00:33:02.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:02 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:33:02.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:02 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6836/1077781429,v1:172.21.15.78:6837/1077781429] up:reconnect 2024-03-24T00:33:02.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:02 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:02.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:02 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:33:02.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:02 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6836/1077781429,v1:172.21.15.78:6837/1077781429] up:reconnect 2024-03-24T00:33:02.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:02 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:02.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:02 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:33:02.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:02 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6836/1077781429,v1:172.21.15.78:6837/1077781429] up:reconnect 2024-03-24T00:33:02.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:02 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:02.710 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 511 sec latency 624301.505 ms 2024-03-24T00:33:03.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:03 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6836/1077781429,v1:172.21.15.78:6837/1077781429] up:rejoin 2024-03-24T00:33:03.405 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:03 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:03.405 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:03 smithi106 ceph-mon[36505]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-24T00:33:03.405 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:03 smithi106 ceph-mon[36505]: pgmap v408: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:33:03.406 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:03 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:33:03.406 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:03 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6836/1077781429,v1:172.21.15.78:6837/1077781429] up:active 2024-03-24T00:33:03.406 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:03 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:03.476 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6836/1077781429,v1:172.21.15.78:6837/1077781429] up:rejoin 2024-03-24T00:33:03.476 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:03.476 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-mon[32429]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-24T00:33:03.476 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-mon[32429]: pgmap v408: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:33:03.476 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:33:03.476 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6836/1077781429,v1:172.21.15.78:6837/1077781429] up:active 2024-03-24T00:33:03.476 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:03.477 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f05aedaf700 time 2024-03-24T00:33:03.129628+0000 2024-03-24T00:33:03.477 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:33:03.477 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:33:03.477 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f05b93d94a1] 2024-03-24T00:33:03.477 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f05b93d9667] 2024-03-24T00:33:03.477 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56315c747223] 2024-03-24T00:33:03.477 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56315c5c15d6] 2024-03-24T00:33:03.478 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56315c5c1da1] 2024-03-24T00:33:03.478 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56315c5c42d5] 2024-03-24T00:33:03.478 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56315c614233] 2024-03-24T00:33:03.478 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56315c616197] 2024-03-24T00:33:03.478 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56315c51511b] 2024-03-24T00:33:03.478 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56315c5156bc] 2024-03-24T00:33:03.478 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56315c4ff1ff] 2024-03-24T00:33:03.478 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f05b966b488] 2024-03-24T00:33:03.478 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 13: (DispatchQueue::entry()+0x51f) [0x7f05b966860f] 2024-03-24T00:33:03.478 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f05b973be81] 2024-03-24T00:33:03.478 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f05b81561ca] 2024-03-24T00:33:03.478 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 16: clone() 2024-03-24T00:33:03.478 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 2024-03-24T00:33:03.130+0000 7f05aedaf700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f05aedaf700 time 2024-03-24T00:33:03.129628+0000 2024-03-24T00:33:03.479 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:33:03.479 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 2024-03-24T00:33:03.479 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:33:03.479 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f05b93d94a1] 2024-03-24T00:33:03.479 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f05b93d9667] 2024-03-24T00:33:03.479 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56315c747223] 2024-03-24T00:33:03.479 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56315c5c15d6] 2024-03-24T00:33:03.479 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56315c5c1da1] 2024-03-24T00:33:03.479 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56315c5c42d5] 2024-03-24T00:33:03.479 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56315c614233] 2024-03-24T00:33:03.479 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56315c616197] 2024-03-24T00:33:03.479 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56315c51511b] 2024-03-24T00:33:03.479 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56315c5156bc] 2024-03-24T00:33:03.480 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56315c4ff1ff] 2024-03-24T00:33:03.480 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f05b966b488] 2024-03-24T00:33:03.480 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 13: (DispatchQueue::entry()+0x51f) [0x7f05b966860f] 2024-03-24T00:33:03.480 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f05b973be81] 2024-03-24T00:33:03.480 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f05b81561ca] 2024-03-24T00:33:03.480 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 16: clone() 2024-03-24T00:33:03.480 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 2024-03-24T00:33:03.480 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: *** Caught signal (Aborted) ** 2024-03-24T00:33:03.480 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: in thread 7f05aedaf700 thread_name:ms_dispatch 2024-03-24T00:33:03.480 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:33:03.480 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f05b8160d20] 2024-03-24T00:33:03.480 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 2: gsignal() 2024-03-24T00:33:03.480 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 3: abort() 2024-03-24T00:33:03.480 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f05b93d94fb] 2024-03-24T00:33:03.481 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f05b93d9667] 2024-03-24T00:33:03.481 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56315c747223] 2024-03-24T00:33:03.481 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56315c5c15d6] 2024-03-24T00:33:03.481 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56315c5c1da1] 2024-03-24T00:33:03.481 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56315c5c42d5] 2024-03-24T00:33:03.481 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56315c614233] 2024-03-24T00:33:03.481 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56315c616197] 2024-03-24T00:33:03.481 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56315c51511b] 2024-03-24T00:33:03.481 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56315c5156bc] 2024-03-24T00:33:03.481 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56315c4ff1ff] 2024-03-24T00:33:03.481 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f05b966b488] 2024-03-24T00:33:03.481 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 16: (DispatchQueue::entry()+0x51f) [0x7f05b966860f] 2024-03-24T00:33:03.481 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f05b973be81] 2024-03-24T00:33:03.481 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f05b81561ca] 2024-03-24T00:33:03.482 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 19: clone() 2024-03-24T00:33:03.482 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 2024-03-24T00:33:03.133+0000 7f05aedaf700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:33:03.482 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: in thread 7f05aedaf700 thread_name:ms_dispatch 2024-03-24T00:33:03.482 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 2024-03-24T00:33:03.482 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:33:03.482 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f05b8160d20] 2024-03-24T00:33:03.482 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 2: gsignal() 2024-03-24T00:33:03.482 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 3: abort() 2024-03-24T00:33:03.482 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f05b93d94fb] 2024-03-24T00:33:03.482 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f05b93d9667] 2024-03-24T00:33:03.483 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56315c747223] 2024-03-24T00:33:03.483 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56315c5c15d6] 2024-03-24T00:33:03.483 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56315c5c1da1] 2024-03-24T00:33:03.483 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56315c5c42d5] 2024-03-24T00:33:03.483 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56315c614233] 2024-03-24T00:33:03.484 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56315c616197] 2024-03-24T00:33:03.484 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56315c51511b] 2024-03-24T00:33:03.484 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56315c5156bc] 2024-03-24T00:33:03.484 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56315c4ff1ff] 2024-03-24T00:33:03.484 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f05b966b488] 2024-03-24T00:33:03.484 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 16: (DispatchQueue::entry()+0x51f) [0x7f05b966860f] 2024-03-24T00:33:03.484 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f05b973be81] 2024-03-24T00:33:03.484 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f05b81561ca] 2024-03-24T00:33:03.484 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 19: clone() 2024-03-24T00:33:03.484 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:33:03.484 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 2024-03-24T00:33:03.484 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: -1> 2024-03-24T00:33:03.130+0000 7f05aedaf700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f05aedaf700 time 2024-03-24T00:33:03.129628+0000 2024-03-24T00:33:03.484 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:33:03.484 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 2024-03-24T00:33:03.485 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:33:03.485 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f05b93d94a1] 2024-03-24T00:33:03.485 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f05b93d9667] 2024-03-24T00:33:03.485 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56315c747223] 2024-03-24T00:33:03.485 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56315c5c15d6] 2024-03-24T00:33:03.485 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56315c5c1da1] 2024-03-24T00:33:03.485 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56315c5c42d5] 2024-03-24T00:33:03.485 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56315c614233] 2024-03-24T00:33:03.485 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56315c616197] 2024-03-24T00:33:03.485 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56315c51511b] 2024-03-24T00:33:03.485 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56315c5156bc] 2024-03-24T00:33:03.485 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56315c4ff1ff] 2024-03-24T00:33:03.485 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f05b966b488] 2024-03-24T00:33:03.485 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 13: (DispatchQueue::entry()+0x51f) [0x7f05b966860f] 2024-03-24T00:33:03.485 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f05b973be81] 2024-03-24T00:33:03.486 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f05b81561ca] 2024-03-24T00:33:03.486 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 16: clone() 2024-03-24T00:33:03.486 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 2024-03-24T00:33:03.486 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 0> 2024-03-24T00:33:03.133+0000 7f05aedaf700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:33:03.486 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: in thread 7f05aedaf700 thread_name:ms_dispatch 2024-03-24T00:33:03.486 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 2024-03-24T00:33:03.486 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:33:03.486 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f05b8160d20] 2024-03-24T00:33:03.486 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 2: gsignal() 2024-03-24T00:33:03.486 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 3: abort() 2024-03-24T00:33:03.487 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f05b93d94fb] 2024-03-24T00:33:03.487 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f05b93d9667] 2024-03-24T00:33:03.487 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56315c747223] 2024-03-24T00:33:03.487 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56315c5c15d6] 2024-03-24T00:33:03.487 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56315c5c1da1] 2024-03-24T00:33:03.487 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56315c5c42d5] 2024-03-24T00:33:03.487 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56315c614233] 2024-03-24T00:33:03.487 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56315c616197] 2024-03-24T00:33:03.487 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56315c51511b] 2024-03-24T00:33:03.487 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56315c5156bc] 2024-03-24T00:33:03.487 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56315c4ff1ff] 2024-03-24T00:33:03.487 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f05b966b488] 2024-03-24T00:33:03.487 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 16: (DispatchQueue::entry()+0x51f) [0x7f05b966860f] 2024-03-24T00:33:03.487 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f05b973be81] 2024-03-24T00:33:03.488 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f05b81561ca] 2024-03-24T00:33:03.488 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 19: clone() 2024-03-24T00:33:03.488 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:33:03.488 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 2024-03-24T00:33:03.488 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: -1> 2024-03-24T00:33:03.130+0000 7f05aedaf700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f05aedaf700 time 2024-03-24T00:33:03.129628+0000 2024-03-24T00:33:03.488 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:33:03.488 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 2024-03-24T00:33:03.488 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:33:03.488 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f05b93d94a1] 2024-03-24T00:33:03.488 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f05b93d9667] 2024-03-24T00:33:03.488 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56315c747223] 2024-03-24T00:33:03.488 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56315c5c15d6] 2024-03-24T00:33:03.488 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56315c5c1da1] 2024-03-24T00:33:03.488 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56315c5c42d5] 2024-03-24T00:33:03.489 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56315c614233] 2024-03-24T00:33:03.489 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56315c616197] 2024-03-24T00:33:03.489 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56315c51511b] 2024-03-24T00:33:03.489 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56315c5156bc] 2024-03-24T00:33:03.489 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56315c4ff1ff] 2024-03-24T00:33:03.489 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f05b966b488] 2024-03-24T00:33:03.489 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 13: (DispatchQueue::entry()+0x51f) [0x7f05b966860f] 2024-03-24T00:33:03.489 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f05b973be81] 2024-03-24T00:33:03.489 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f05b81561ca] 2024-03-24T00:33:03.489 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 16: clone() 2024-03-24T00:33:03.489 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 2024-03-24T00:33:03.489 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 0> 2024-03-24T00:33:03.133+0000 7f05aedaf700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:33:03.489 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: in thread 7f05aedaf700 thread_name:ms_dispatch 2024-03-24T00:33:03.489 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 2024-03-24T00:33:03.489 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:33:03.490 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f05b8160d20] 2024-03-24T00:33:03.490 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 2: gsignal() 2024-03-24T00:33:03.490 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 3: abort() 2024-03-24T00:33:03.490 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f05b93d94fb] 2024-03-24T00:33:03.490 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f05b93d9667] 2024-03-24T00:33:03.490 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56315c747223] 2024-03-24T00:33:03.490 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56315c5c15d6] 2024-03-24T00:33:03.490 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56315c5c1da1] 2024-03-24T00:33:03.490 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56315c5c42d5] 2024-03-24T00:33:03.490 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56315c614233] 2024-03-24T00:33:03.490 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56315c616197] 2024-03-24T00:33:03.490 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56315c51511b] 2024-03-24T00:33:03.490 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56315c5156bc] 2024-03-24T00:33:03.491 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56315c4ff1ff] 2024-03-24T00:33:03.491 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f05b966b488] 2024-03-24T00:33:03.491 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 16: (DispatchQueue::entry()+0x51f) [0x7f05b966860f] 2024-03-24T00:33:03.491 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f05b973be81] 2024-03-24T00:33:03.491 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f05b81561ca] 2024-03-24T00:33:03.491 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 19: clone() 2024-03-24T00:33:03.491 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:33:03.491 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[89832]: 2024-03-24T00:33:03.491 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 podman[89926]: 2024-03-24 00:33:03.192843939 +0000 UTC m=+0.024835940 container died 3be55a373ac667c1947343c7a47a56c472fa7d4e09d2eb66d5c350b0e4e74d03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-24T00:33:03.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:03 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6836/1077781429,v1:172.21.15.78:6837/1077781429] up:rejoin 2024-03-24T00:33:03.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:03 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:03.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:03 smithi107 ceph-mon[36485]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-24T00:33:03.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:03 smithi107 ceph-mon[36485]: pgmap v408: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:33:03.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:03 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:33:03.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:03 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6836/1077781429,v1:172.21.15.78:6837/1077781429] up:active 2024-03-24T00:33:03.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:03 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:03.710 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 512 sec latency 625301.557 ms 2024-03-24T00:33:04.128 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 podman[89926]: 2024-03-24 00:33:03.861552585 +0000 UTC m=+0.693544572 container remove 3be55a373ac667c1947343c7a47a56c472fa7d4e09d2eb66d5c350b0e4e74d03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-24T00:33:04.128 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:03 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:33:04.128 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:04 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.d.service: Failed with result 'exit-code'. 2024-03-24T00:33:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:04 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=h=up:active} 3 up:standby-replay 2024-03-24T00:33:04.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:04 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=h=up:active} 3 up:standby-replay 2024-03-24T00:33:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:04 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=h=up:active} 3 up:standby-replay 2024-03-24T00:33:04.710 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 513 sec latency 626301.643 ms 2024-03-24T00:33:05.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:05 smithi106 ceph-mon[36505]: pgmap v409: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:33:05.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:05 smithi078 ceph-mon[32429]: pgmap v409: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:33:05.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:05 smithi107 ceph-mon[36485]: pgmap v409: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:33:05.710 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 514 sec latency 627301.746 ms 2024-03-24T00:33:06.711 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 515 sec latency 628301.806 ms 2024-03-24T00:33:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:06 smithi106 ceph-mon[36505]: pgmap v410: 129 pgs: 129 active+clean; 27 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-24T00:33:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:06 smithi078 ceph-mon[32429]: pgmap v410: 129 pgs: 129 active+clean; 27 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-24T00:33:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:06 smithi107 ceph-mon[36485]: pgmap v410: 129 pgs: 129 active+clean; 27 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-24T00:33:07.711 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 516 sec latency 629301.841 ms 2024-03-24T00:33:08.711 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 517 sec latency 630301.925 ms 2024-03-24T00:33:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:08 smithi106 ceph-mon[36505]: pgmap v411: 129 pgs: 129 active+clean; 27 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-24T00:33:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:08 smithi078 ceph-mon[32429]: pgmap v411: 129 pgs: 129 active+clean; 27 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-24T00:33:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:08 smithi107 ceph-mon[36485]: pgmap v411: 129 pgs: 129 active+clean; 27 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 28 op/s 2024-03-24T00:33:09.711 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 518 sec latency 631302.011 ms 2024-03-24T00:33:10.711 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 519 sec latency 632302.103 ms 2024-03-24T00:33:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:10 smithi106 ceph-mon[36505]: pgmap v412: 129 pgs: 129 active+clean; 27 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-24T00:33:11.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:10 smithi078 ceph-mon[32429]: pgmap v412: 129 pgs: 129 active+clean; 27 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-24T00:33:11.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:10 smithi107 ceph-mon[36485]: pgmap v412: 129 pgs: 129 active+clean; 27 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-24T00:33:11.711 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 520 sec latency 633302.229 ms 2024-03-24T00:33:12.711 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 521 sec latency 634302.316 ms 2024-03-24T00:33:12.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:12 smithi106 ceph-mon[36505]: pgmap v413: 129 pgs: 129 active+clean; 27 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-24T00:33:13.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:12 smithi078 ceph-mon[32429]: pgmap v413: 129 pgs: 129 active+clean; 27 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-24T00:33:13.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:12 smithi107 ceph-mon[36485]: pgmap v413: 129 pgs: 129 active+clean; 27 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-24T00:33:13.711 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 522 sec latency 635302.408 ms 2024-03-24T00:33:14.603 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:14 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.d.service: Scheduled restart job, restart counter is at 3. 2024-03-24T00:33:14.603 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:14 smithi078 systemd[1]: Stopped Ceph mds.d for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:33:14.603 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:14 smithi078 systemd[1]: Starting Ceph mds.d for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:33:14.604 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:14 smithi078 podman[90020]: 2024-03-24 00:33:14.485228778 +0000 UTC m=+0.019217703 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:33:14.604 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:14 smithi078 podman[90020]: 2024-03-24 00:33:14.603718337 +0000 UTC m=+0.137707306 container create 6607ebbdfbbd4bb6060752ce8db81ebf6399c2a02f3d6e37eca0695b9f69f66f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:33:14.711 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 523 sec latency 636302.467 ms 2024-03-24T00:33:14.863 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:14 smithi078 ceph-mon[32429]: pgmap v414: 129 pgs: 129 active+clean; 27 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-24T00:33:14.863 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:14 smithi078 podman[90020]: 2024-03-24 00:33:14.839523211 +0000 UTC m=+0.373512136 container init 6607ebbdfbbd4bb6060752ce8db81ebf6399c2a02f3d6e37eca0695b9f69f66f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-24T00:33:14.863 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:14 smithi078 podman[90020]: 2024-03-24 00:33:14.842105353 +0000 UTC m=+0.376094286 container start 6607ebbdfbbd4bb6060752ce8db81ebf6399c2a02f3d6e37eca0695b9f69f66f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-24T00:33:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:14 smithi106 ceph-mon[36505]: pgmap v414: 129 pgs: 129 active+clean; 27 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-24T00:33:15.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:14 smithi107 ceph-mon[36485]: pgmap v414: 129 pgs: 129 active+clean; 27 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-24T00:33:15.340 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:14 smithi078 ceph-mds[90039]: -- 172.21.15.78:0/3274608569 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5622835f1a40 con 0x5622835c5000 2024-03-24T00:33:15.340 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: starting mds.d at 2024-03-24T00:33:15.341 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:14 smithi078 bash[90020]: 6607ebbdfbbd4bb6060752ce8db81ebf6399c2a02f3d6e37eca0695b9f69f66f 2024-03-24T00:33:15.341 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:14 smithi078 systemd[1]: Started Ceph mds.d for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:33:15.711 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 524 sec latency 637302.551 ms 2024-03-24T00:33:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:15 smithi106 ceph-mon[36505]: daemon mds.d restarted 2024-03-24T00:33:15.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:15 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:33:15.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:15 smithi106 ceph-mon[36505]: osdmap e142: 12 total, 12 up, 12 in 2024-03-24T00:33:15.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:15 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6836/1186751954,v1:172.21.15.78:6837/1186751954] up:boot 2024-03-24T00:33:15.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:15 smithi106 ceph-mon[36505]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-03-24T00:33:15.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:15 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:33:15.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:33:15.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:15 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:16.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:15 smithi078 ceph-mon[32429]: daemon mds.d restarted 2024-03-24T00:33:16.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:15 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:33:16.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:15 smithi078 ceph-mon[32429]: osdmap e142: 12 total, 12 up, 12 in 2024-03-24T00:33:16.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:15 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6836/1186751954,v1:172.21.15.78:6837/1186751954] up:boot 2024-03-24T00:33:16.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:15 smithi078 ceph-mon[32429]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-03-24T00:33:16.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:15 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:33:16.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:33:16.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:15 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:16.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:15 smithi107 ceph-mon[36485]: daemon mds.d restarted 2024-03-24T00:33:16.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:15 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:33:16.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:15 smithi107 ceph-mon[36485]: osdmap e142: 12 total, 12 up, 12 in 2024-03-24T00:33:16.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:15 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6836/1186751954,v1:172.21.15.78:6837/1186751954] up:boot 2024-03-24T00:33:16.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:15 smithi107 ceph-mon[36485]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-03-24T00:33:16.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:15 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:33:16.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:33:16.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:15 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:16.711 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 525 sec latency 638302.638 ms 2024-03-24T00:33:17.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:16 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6834/1416685123,v1:172.21.15.78:6835/1416685123] up:resolve 2024-03-24T00:33:17.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:16 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:17.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:16 smithi106 ceph-mon[36505]: pgmap v416: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:33:17.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:16 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6834/1416685123,v1:172.21.15.78:6835/1416685123] up:resolve 2024-03-24T00:33:17.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:16 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:17.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:16 smithi078 ceph-mon[32429]: pgmap v416: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:33:17.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:16 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6834/1416685123,v1:172.21.15.78:6835/1416685123] up:resolve 2024-03-24T00:33:17.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:16 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:17.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:16 smithi107 ceph-mon[36485]: pgmap v416: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:33:17.712 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 526 sec latency 639302.724 ms 2024-03-24T00:33:18.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:17 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:33:18.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:17 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6834/1416685123,v1:172.21.15.78:6835/1416685123] up:reconnect 2024-03-24T00:33:18.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:17 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:17 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:33:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:17 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6834/1416685123,v1:172.21.15.78:6835/1416685123] up:reconnect 2024-03-24T00:33:18.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:17 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:17 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:33:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:17 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6834/1416685123,v1:172.21.15.78:6835/1416685123] up:reconnect 2024-03-24T00:33:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:17 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:18.712 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 527 sec latency 640302.838 ms 2024-03-24T00:33:19.327 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6834/1416685123,v1:172.21.15.78:6835/1416685123] up:rejoin 2024-03-24T00:33:19.328 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:19.328 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-mon[32429]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-24T00:33:19.328 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-mon[32429]: pgmap v417: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:33:19.328 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f862aafe700 time 2024-03-24T00:33:18.925640+0000 2024-03-24T00:33:19.329 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:33:19.329 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:33:19.329 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f86351284a1] 2024-03-24T00:33:19.329 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f8635128667] 2024-03-24T00:33:19.329 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55af9d59c223] 2024-03-24T00:33:19.329 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55af9d4165d6] 2024-03-24T00:33:19.329 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55af9d416da1] 2024-03-24T00:33:19.329 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55af9d4192d5] 2024-03-24T00:33:19.329 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55af9d469233] 2024-03-24T00:33:19.329 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55af9d46b197] 2024-03-24T00:33:19.329 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55af9d36a11b] 2024-03-24T00:33:19.330 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55af9d36a6bc] 2024-03-24T00:33:19.330 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55af9d3541ff] 2024-03-24T00:33:19.330 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f86353ba488] 2024-03-24T00:33:19.330 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 13: (DispatchQueue::entry()+0x51f) [0x7f86353b760f] 2024-03-24T00:33:19.330 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f863548ae81] 2024-03-24T00:33:19.330 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f8633ea51ca] 2024-03-24T00:33:19.330 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 16: clone() 2024-03-24T00:33:19.330 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 2024-03-24T00:33:18.926+0000 7f862aafe700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f862aafe700 time 2024-03-24T00:33:18.925640+0000 2024-03-24T00:33:19.330 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:33:19.330 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 2024-03-24T00:33:19.330 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:33:19.331 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f86351284a1] 2024-03-24T00:33:19.331 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f8635128667] 2024-03-24T00:33:19.331 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55af9d59c223] 2024-03-24T00:33:19.331 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55af9d4165d6] 2024-03-24T00:33:19.331 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55af9d416da1] 2024-03-24T00:33:19.331 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55af9d4192d5] 2024-03-24T00:33:19.331 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55af9d469233] 2024-03-24T00:33:19.331 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55af9d46b197] 2024-03-24T00:33:19.331 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55af9d36a11b] 2024-03-24T00:33:19.331 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55af9d36a6bc] 2024-03-24T00:33:19.331 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55af9d3541ff] 2024-03-24T00:33:19.331 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f86353ba488] 2024-03-24T00:33:19.332 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 13: (DispatchQueue::entry()+0x51f) [0x7f86353b760f] 2024-03-24T00:33:19.332 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f863548ae81] 2024-03-24T00:33:19.332 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f8633ea51ca] 2024-03-24T00:33:19.332 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 16: clone() 2024-03-24T00:33:19.332 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 2024-03-24T00:33:19.332 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: *** Caught signal (Aborted) ** 2024-03-24T00:33:19.332 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: in thread 7f862aafe700 thread_name:ms_dispatch 2024-03-24T00:33:19.332 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:33:19.332 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8633eafd20] 2024-03-24T00:33:19.332 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 2: gsignal() 2024-03-24T00:33:19.332 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 3: abort() 2024-03-24T00:33:19.332 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f86351284fb] 2024-03-24T00:33:19.332 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f8635128667] 2024-03-24T00:33:19.333 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55af9d59c223] 2024-03-24T00:33:19.333 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55af9d4165d6] 2024-03-24T00:33:19.333 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55af9d416da1] 2024-03-24T00:33:19.333 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55af9d4192d5] 2024-03-24T00:33:19.333 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55af9d469233] 2024-03-24T00:33:19.333 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55af9d46b197] 2024-03-24T00:33:19.333 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55af9d36a11b] 2024-03-24T00:33:19.333 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55af9d36a6bc] 2024-03-24T00:33:19.333 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55af9d3541ff] 2024-03-24T00:33:19.333 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f86353ba488] 2024-03-24T00:33:19.333 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 16: (DispatchQueue::entry()+0x51f) [0x7f86353b760f] 2024-03-24T00:33:19.333 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f863548ae81] 2024-03-24T00:33:19.333 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f8633ea51ca] 2024-03-24T00:33:19.334 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 19: clone() 2024-03-24T00:33:19.334 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 2024-03-24T00:33:18.928+0000 7f862aafe700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:33:19.334 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: in thread 7f862aafe700 thread_name:ms_dispatch 2024-03-24T00:33:19.334 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 2024-03-24T00:33:19.334 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:33:19.334 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8633eafd20] 2024-03-24T00:33:19.334 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 2: gsignal() 2024-03-24T00:33:19.334 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 3: abort() 2024-03-24T00:33:19.334 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f86351284fb] 2024-03-24T00:33:19.335 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f8635128667] 2024-03-24T00:33:19.336 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55af9d59c223] 2024-03-24T00:33:19.336 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55af9d4165d6] 2024-03-24T00:33:19.336 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55af9d416da1] 2024-03-24T00:33:19.336 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55af9d4192d5] 2024-03-24T00:33:19.336 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55af9d469233] 2024-03-24T00:33:19.336 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55af9d46b197] 2024-03-24T00:33:19.336 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55af9d36a11b] 2024-03-24T00:33:19.336 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55af9d36a6bc] 2024-03-24T00:33:19.336 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55af9d3541ff] 2024-03-24T00:33:19.336 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f86353ba488] 2024-03-24T00:33:19.337 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 16: (DispatchQueue::entry()+0x51f) [0x7f86353b760f] 2024-03-24T00:33:19.337 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f863548ae81] 2024-03-24T00:33:19.337 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f8633ea51ca] 2024-03-24T00:33:19.337 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 19: clone() 2024-03-24T00:33:19.337 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:33:19.337 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 2024-03-24T00:33:19.337 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: -1> 2024-03-24T00:33:18.926+0000 7f862aafe700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f862aafe700 time 2024-03-24T00:33:18.925640+0000 2024-03-24T00:33:19.337 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:33:19.337 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 2024-03-24T00:33:19.337 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:33:19.337 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f86351284a1] 2024-03-24T00:33:19.337 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f8635128667] 2024-03-24T00:33:19.337 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55af9d59c223] 2024-03-24T00:33:19.338 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55af9d4165d6] 2024-03-24T00:33:19.338 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55af9d416da1] 2024-03-24T00:33:19.338 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55af9d4192d5] 2024-03-24T00:33:19.338 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55af9d469233] 2024-03-24T00:33:19.338 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55af9d46b197] 2024-03-24T00:33:19.338 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55af9d36a11b] 2024-03-24T00:33:19.338 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55af9d36a6bc] 2024-03-24T00:33:19.338 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55af9d3541ff] 2024-03-24T00:33:19.338 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f86353ba488] 2024-03-24T00:33:19.338 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 13: (DispatchQueue::entry()+0x51f) [0x7f86353b760f] 2024-03-24T00:33:19.338 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f863548ae81] 2024-03-24T00:33:19.338 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f8633ea51ca] 2024-03-24T00:33:19.339 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 16: clone() 2024-03-24T00:33:19.339 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 2024-03-24T00:33:19.339 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 0> 2024-03-24T00:33:18.928+0000 7f862aafe700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:33:19.339 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: in thread 7f862aafe700 thread_name:ms_dispatch 2024-03-24T00:33:19.339 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 2024-03-24T00:33:19.339 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:33:19.339 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8633eafd20] 2024-03-24T00:33:19.339 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 2: gsignal() 2024-03-24T00:33:19.339 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 3: abort() 2024-03-24T00:33:19.339 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f86351284fb] 2024-03-24T00:33:19.339 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f8635128667] 2024-03-24T00:33:19.339 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55af9d59c223] 2024-03-24T00:33:19.339 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55af9d4165d6] 2024-03-24T00:33:19.340 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55af9d416da1] 2024-03-24T00:33:19.340 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55af9d4192d5] 2024-03-24T00:33:19.340 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55af9d469233] 2024-03-24T00:33:19.340 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55af9d46b197] 2024-03-24T00:33:19.340 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55af9d36a11b] 2024-03-24T00:33:19.340 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55af9d36a6bc] 2024-03-24T00:33:19.340 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55af9d3541ff] 2024-03-24T00:33:19.340 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f86353ba488] 2024-03-24T00:33:19.340 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 16: (DispatchQueue::entry()+0x51f) [0x7f86353b760f] 2024-03-24T00:33:19.340 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f863548ae81] 2024-03-24T00:33:19.340 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f8633ea51ca] 2024-03-24T00:33:19.340 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 19: clone() 2024-03-24T00:33:19.340 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:33:19.341 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 2024-03-24T00:33:19.341 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: -1> 2024-03-24T00:33:18.926+0000 7f862aafe700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f862aafe700 time 2024-03-24T00:33:18.925640+0000 2024-03-24T00:33:19.341 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:33:19.341 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 2024-03-24T00:33:19.341 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:33:19.341 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f86351284a1] 2024-03-24T00:33:19.341 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f8635128667] 2024-03-24T00:33:19.341 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55af9d59c223] 2024-03-24T00:33:19.341 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55af9d4165d6] 2024-03-24T00:33:19.341 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55af9d416da1] 2024-03-24T00:33:19.341 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55af9d4192d5] 2024-03-24T00:33:19.341 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55af9d469233] 2024-03-24T00:33:19.341 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55af9d46b197] 2024-03-24T00:33:19.342 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55af9d36a11b] 2024-03-24T00:33:19.342 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55af9d36a6bc] 2024-03-24T00:33:19.342 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55af9d3541ff] 2024-03-24T00:33:19.342 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f86353ba488] 2024-03-24T00:33:19.342 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 13: (DispatchQueue::entry()+0x51f) [0x7f86353b760f] 2024-03-24T00:33:19.342 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f863548ae81] 2024-03-24T00:33:19.342 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f8633ea51ca] 2024-03-24T00:33:19.342 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 16: clone() 2024-03-24T00:33:19.342 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 2024-03-24T00:33:19.342 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 0> 2024-03-24T00:33:18.928+0000 7f862aafe700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:33:19.342 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: in thread 7f862aafe700 thread_name:ms_dispatch 2024-03-24T00:33:19.342 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 2024-03-24T00:33:19.342 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:33:19.343 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8633eafd20] 2024-03-24T00:33:19.343 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 2: gsignal() 2024-03-24T00:33:19.343 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 3: abort() 2024-03-24T00:33:19.343 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f86351284fb] 2024-03-24T00:33:19.343 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f8635128667] 2024-03-24T00:33:19.343 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55af9d59c223] 2024-03-24T00:33:19.343 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55af9d4165d6] 2024-03-24T00:33:19.343 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55af9d416da1] 2024-03-24T00:33:19.343 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55af9d4192d5] 2024-03-24T00:33:19.343 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55af9d469233] 2024-03-24T00:33:19.345 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55af9d46b197] 2024-03-24T00:33:19.345 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55af9d36a11b] 2024-03-24T00:33:19.345 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55af9d36a6bc] 2024-03-24T00:33:19.346 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55af9d3541ff] 2024-03-24T00:33:19.346 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f86353ba488] 2024-03-24T00:33:19.346 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 16: (DispatchQueue::entry()+0x51f) [0x7f86353b760f] 2024-03-24T00:33:19.346 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f863548ae81] 2024-03-24T00:33:19.346 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f8633ea51ca] 2024-03-24T00:33:19.346 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 19: clone() 2024-03-24T00:33:19.346 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:33:19.346 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[70420]: 2024-03-24T00:33:19.346 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:18 smithi078 podman[90073]: 2024-03-24 00:33:18.995858181 +0000 UTC m=+0.019341792 container died 2bae411f70111c67297db78af224b5d61cd9eaca3e674ba15080643f82ffc3ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:33:19.347 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:18 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6834/1416685123,v1:172.21.15.78:6835/1416685123] up:rejoin 2024-03-24T00:33:19.347 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:18 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:19.347 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:18 smithi107 ceph-mon[36485]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-24T00:33:19.347 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:18 smithi107 ceph-mon[36485]: pgmap v417: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:33:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:18 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6834/1416685123,v1:172.21.15.78:6835/1416685123] up:rejoin 2024-03-24T00:33:19.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:18 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:19.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:18 smithi106 ceph-mon[36505]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-24T00:33:19.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:18 smithi106 ceph-mon[36505]: pgmap v417: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:33:19.712 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 528 sec latency 641302.912 ms 2024-03-24T00:33:20.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:19 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:33:20.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:19 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6834/1416685123,v1:172.21.15.78:6835/1416685123] up:active 2024-03-24T00:33:20.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:19 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:20.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:19 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=h=up:active} 3 up:standby-replay 2024-03-24T00:33:20.091 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:19 smithi078 podman[90073]: 2024-03-24 00:33:19.661527365 +0000 UTC m=+0.685011027 container remove 2bae411f70111c67297db78af224b5d61cd9eaca3e674ba15080643f82ffc3ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD) 2024-03-24T00:33:20.091 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:19 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:33:20.091 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:19 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.j.service: Failed with result 'exit-code'. 2024-03-24T00:33:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:19 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:33:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:19 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6834/1416685123,v1:172.21.15.78:6835/1416685123] up:active 2024-03-24T00:33:20.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:19 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:20.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:19 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=h=up:active} 3 up:standby-replay 2024-03-24T00:33:20.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:19 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:33:20.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:19 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6834/1416685123,v1:172.21.15.78:6835/1416685123] up:active 2024-03-24T00:33:20.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:19 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:20.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:19 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=h=up:active} 3 up:standby-replay 2024-03-24T00:33:20.712 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 529 sec latency 642303.031 ms 2024-03-24T00:33:20.916 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:20 smithi078 ceph-mon[32429]: pgmap v418: 129 pgs: 129 active+clean; 27 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:33:21.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:20 smithi107 ceph-mon[36485]: pgmap v418: 129 pgs: 129 active+clean; 27 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:33:21.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:20 smithi106 ceph-mon[36505]: pgmap v418: 129 pgs: 129 active+clean; 27 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:33:21.712 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 530 sec latency 643303.157 ms 2024-03-24T00:33:22.712 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 531 sec latency 644303.287 ms 2024-03-24T00:33:22.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:22 smithi078 ceph-mon[32429]: pgmap v419: 129 pgs: 129 active+clean; 27 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:33:22.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:22 smithi106 ceph-mon[36505]: pgmap v419: 129 pgs: 129 active+clean; 27 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:33:23.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:22 smithi107 ceph-mon[36485]: pgmap v419: 129 pgs: 129 active+clean; 27 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:33:23.712 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 532 sec latency 645303.384 ms 2024-03-24T00:33:24.712 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 533 sec latency 646303.432 ms 2024-03-24T00:33:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:24 smithi078 ceph-mon[32429]: pgmap v420: 129 pgs: 129 active+clean; 27 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:33:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:24 smithi106 ceph-mon[36505]: pgmap v420: 129 pgs: 129 active+clean; 27 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:33:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:24 smithi107 ceph-mon[36485]: pgmap v420: 129 pgs: 129 active+clean; 27 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:33:25.712 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 534 sec latency 647303.516 ms 2024-03-24T00:33:26.712 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 535 sec latency 648303.608 ms 2024-03-24T00:33:26.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:26 smithi106 ceph-mon[36505]: pgmap v421: 129 pgs: 129 active+clean; 27 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 23 KiB/s wr, 14 op/s 2024-03-24T00:33:27.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:26 smithi078 ceph-mon[32429]: pgmap v421: 129 pgs: 129 active+clean; 27 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 23 KiB/s wr, 14 op/s 2024-03-24T00:33:27.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:26 smithi107 ceph-mon[36485]: pgmap v421: 129 pgs: 129 active+clean; 27 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 23 KiB/s wr, 14 op/s 2024-03-24T00:33:27.712 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 536 sec latency 649303.679 ms 2024-03-24T00:33:27.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:27 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:33:27.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:27 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:33:27.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:27 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:33:27.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:27 smithi106 ceph-mon[36505]: pgmap v422: 129 pgs: 129 active+clean; 27 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 25 KiB/s wr, 12 op/s 2024-03-24T00:33:27.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:27 smithi106 ceph-mon[36505]: pgmap v423: 129 pgs: 129 active+clean; 27 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 31 KiB/s wr, 15 op/s 2024-03-24T00:33:27.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:27 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:33:28.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:27 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:33:28.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:27 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:33:28.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:27 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:33:28.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:27 smithi078 ceph-mon[32429]: pgmap v422: 129 pgs: 129 active+clean; 27 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 25 KiB/s wr, 12 op/s 2024-03-24T00:33:28.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:27 smithi078 ceph-mon[32429]: pgmap v423: 129 pgs: 129 active+clean; 27 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 31 KiB/s wr, 15 op/s 2024-03-24T00:33:28.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:27 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:33:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:27 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:33:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:27 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:33:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:27 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:33:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:27 smithi107 ceph-mon[36485]: pgmap v422: 129 pgs: 129 active+clean; 27 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 25 KiB/s wr, 12 op/s 2024-03-24T00:33:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:27 smithi107 ceph-mon[36485]: pgmap v423: 129 pgs: 129 active+clean; 27 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 31 KiB/s wr, 15 op/s 2024-03-24T00:33:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:27 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:33:28.713 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 537 sec latency 650303.763 ms 2024-03-24T00:33:29.713 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 538 sec latency 651303.853 ms 2024-03-24T00:33:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:29 smithi106 ceph-mon[36505]: osdmap e143: 12 total, 12 up, 12 in 2024-03-24T00:33:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:29 smithi106 ceph-mon[36505]: pgmap v425: 129 pgs: 129 active+clean; 27 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 151 B/s wr, 4 op/s 2024-03-24T00:33:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:29 smithi078 ceph-mon[32429]: osdmap e143: 12 total, 12 up, 12 in 2024-03-24T00:33:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:29 smithi078 ceph-mon[32429]: pgmap v425: 129 pgs: 129 active+clean; 27 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 151 B/s wr, 4 op/s 2024-03-24T00:33:30.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:29 smithi107 ceph-mon[36485]: osdmap e143: 12 total, 12 up, 12 in 2024-03-24T00:33:30.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:29 smithi107 ceph-mon[36485]: pgmap v425: 129 pgs: 129 active+clean; 27 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 151 B/s wr, 4 op/s 2024-03-24T00:33:30.454 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:30 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.j.service: Scheduled restart job, restart counter is at 1. 2024-03-24T00:33:30.454 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:30 smithi078 systemd[1]: Stopped Ceph mds.j for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:33:30.454 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:30 smithi078 systemd[1]: Starting Ceph mds.j for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:33:30.454 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:30 smithi078 podman[90301]: 2024-03-24 00:33:30.23234151 +0000 UTC m=+0.018758493 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:33:30.454 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:30 smithi078 podman[90301]: 2024-03-24 00:33:30.3372596 +0000 UTC m=+0.123676574 container create 78a0a33da52aa067b772376a25afe236c61fcde18a0469b8781847763013677a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-24T00:33:30.713 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 539 sec latency 652303.942 ms 2024-03-24T00:33:30.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:30 smithi078 ceph-mon[32429]: daemon mds.j restarted 2024-03-24T00:33:30.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:30 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:33:30.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:30 smithi078 ceph-mon[32429]: osdmap e144: 12 total, 12 up, 12 in 2024-03-24T00:33:30.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:30 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6834/2163967090,v1:172.21.15.78:6835/2163967090] up:boot 2024-03-24T00:33:30.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:30 smithi078 ceph-mon[32429]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-24T00:33:30.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:30 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:33:30.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:30 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:33:30.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:30 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:30.841 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:30 smithi078 podman[90301]: 2024-03-24 00:33:30.573280004 +0000 UTC m=+0.359696985 container init 78a0a33da52aa067b772376a25afe236c61fcde18a0469b8781847763013677a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-24T00:33:30.842 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:30 smithi078 podman[90301]: 2024-03-24 00:33:30.575847504 +0000 UTC m=+0.362264480 container start 78a0a33da52aa067b772376a25afe236c61fcde18a0469b8781847763013677a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-24T00:33:30.842 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:30 smithi078 ceph-mds[90320]: -- 172.21.15.78:0/3305132677 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fa273a5a40 con 0x55fa27378800 2024-03-24T00:33:30.842 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:30 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: starting mds.j at 2024-03-24T00:33:30.842 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:30 smithi078 bash[90301]: 78a0a33da52aa067b772376a25afe236c61fcde18a0469b8781847763013677a 2024-03-24T00:33:30.842 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:30 smithi078 systemd[1]: Started Ceph mds.j for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:33:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:30 smithi106 ceph-mon[36505]: daemon mds.j restarted 2024-03-24T00:33:30.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:30 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:33:30.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:30 smithi106 ceph-mon[36505]: osdmap e144: 12 total, 12 up, 12 in 2024-03-24T00:33:30.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:30 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6834/2163967090,v1:172.21.15.78:6835/2163967090] up:boot 2024-03-24T00:33:30.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:30 smithi106 ceph-mon[36505]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-24T00:33:30.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:30 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:33:30.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:30 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:33:30.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:30 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:31.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:30 smithi107 ceph-mon[36485]: daemon mds.j restarted 2024-03-24T00:33:31.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:30 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:33:31.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:30 smithi107 ceph-mon[36485]: osdmap e144: 12 total, 12 up, 12 in 2024-03-24T00:33:31.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:30 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6834/2163967090,v1:172.21.15.78:6835/2163967090] up:boot 2024-03-24T00:33:31.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:30 smithi107 ceph-mon[36485]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-24T00:33:31.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:30 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:33:31.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:30 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:33:31.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:30 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:31.713 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 540 sec latency 653304.057 ms 2024-03-24T00:33:32.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:31 smithi078 ceph-mon[32429]: pgmap v427: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 214 B/s wr, 6 op/s 2024-03-24T00:33:32.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:31 smithi107 ceph-mon[36485]: pgmap v427: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 214 B/s wr, 6 op/s 2024-03-24T00:33:32.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:31 smithi106 ceph-mon[36505]: pgmap v427: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 214 B/s wr, 6 op/s 2024-03-24T00:33:32.713 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 541 sec latency 654304.141 ms 2024-03-24T00:33:32.994 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:32 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6836/1186751954,v1:172.21.15.78:6837/1186751954] up:resolve 2024-03-24T00:33:32.994 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:32 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:33.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:32 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6836/1186751954,v1:172.21.15.78:6837/1186751954] up:resolve 2024-03-24T00:33:33.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:32 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:33.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:32 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6836/1186751954,v1:172.21.15.78:6837/1186751954] up:resolve 2024-03-24T00:33:33.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:32 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:33.713 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 542 sec latency 655304.188 ms 2024-03-24T00:33:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:33 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999981 2024-03-24T00:33:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:33 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6836/1186751954,v1:172.21.15.78:6837/1186751954] up:reconnect 2024-03-24T00:33:34.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:33 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:34.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:33 smithi078 ceph-mon[32429]: pgmap v428: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-24T00:33:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:33 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999981 2024-03-24T00:33:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:33 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6836/1186751954,v1:172.21.15.78:6837/1186751954] up:reconnect 2024-03-24T00:33:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:33 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:33 smithi107 ceph-mon[36485]: pgmap v428: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-24T00:33:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:33 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999981 2024-03-24T00:33:34.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:33 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6836/1186751954,v1:172.21.15.78:6837/1186751954] up:reconnect 2024-03-24T00:33:34.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:33 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:34.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:33 smithi106 ceph-mon[36505]: pgmap v428: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-24T00:33:34.527 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f837e0b3700 time 2024-03-24T00:33:34.129228+0000 2024-03-24T00:33:34.528 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:33:34.528 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:33:34.528 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f83886dd4a1] 2024-03-24T00:33:34.528 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f83886dd667] 2024-03-24T00:33:34.528 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x562280d5e223] 2024-03-24T00:33:34.528 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x562280bd85d6] 2024-03-24T00:33:34.528 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x562280bd8da1] 2024-03-24T00:33:34.528 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x562280bdb2d5] 2024-03-24T00:33:34.528 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x562280c2b233] 2024-03-24T00:33:34.528 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x562280c2d197] 2024-03-24T00:33:34.528 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x562280b2c11b] 2024-03-24T00:33:34.529 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x562280b2c6bc] 2024-03-24T00:33:34.529 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x562280b161ff] 2024-03-24T00:33:34.529 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f838896f488] 2024-03-24T00:33:34.529 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 13: (DispatchQueue::entry()+0x51f) [0x7f838896c60f] 2024-03-24T00:33:34.529 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8388a3fe81] 2024-03-24T00:33:34.529 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f838745a1ca] 2024-03-24T00:33:34.529 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 16: clone() 2024-03-24T00:33:34.529 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 2024-03-24T00:33:34.130+0000 7f837e0b3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f837e0b3700 time 2024-03-24T00:33:34.129228+0000 2024-03-24T00:33:34.529 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:33:34.529 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 2024-03-24T00:33:34.529 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:33:34.530 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f83886dd4a1] 2024-03-24T00:33:34.530 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f83886dd667] 2024-03-24T00:33:34.530 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x562280d5e223] 2024-03-24T00:33:34.530 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x562280bd85d6] 2024-03-24T00:33:34.530 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x562280bd8da1] 2024-03-24T00:33:34.530 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x562280bdb2d5] 2024-03-24T00:33:34.530 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x562280c2b233] 2024-03-24T00:33:34.530 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x562280c2d197] 2024-03-24T00:33:34.530 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x562280b2c11b] 2024-03-24T00:33:34.530 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x562280b2c6bc] 2024-03-24T00:33:34.530 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x562280b161ff] 2024-03-24T00:33:34.530 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f838896f488] 2024-03-24T00:33:34.531 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 13: (DispatchQueue::entry()+0x51f) [0x7f838896c60f] 2024-03-24T00:33:34.531 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8388a3fe81] 2024-03-24T00:33:34.531 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f838745a1ca] 2024-03-24T00:33:34.531 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 16: clone() 2024-03-24T00:33:34.531 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 2024-03-24T00:33:34.531 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: *** Caught signal (Aborted) ** 2024-03-24T00:33:34.531 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: in thread 7f837e0b3700 thread_name:ms_dispatch 2024-03-24T00:33:34.531 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:33:34.531 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8387464d20] 2024-03-24T00:33:34.532 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 2: gsignal() 2024-03-24T00:33:34.532 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 3: abort() 2024-03-24T00:33:34.533 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f83886dd4fb] 2024-03-24T00:33:34.533 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f83886dd667] 2024-03-24T00:33:34.533 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x562280d5e223] 2024-03-24T00:33:34.533 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x562280bd85d6] 2024-03-24T00:33:34.533 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x562280bd8da1] 2024-03-24T00:33:34.533 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x562280bdb2d5] 2024-03-24T00:33:34.533 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x562280c2b233] 2024-03-24T00:33:34.533 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x562280c2d197] 2024-03-24T00:33:34.533 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x562280b2c11b] 2024-03-24T00:33:34.533 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x562280b2c6bc] 2024-03-24T00:33:34.533 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x562280b161ff] 2024-03-24T00:33:34.534 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f838896f488] 2024-03-24T00:33:34.534 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 16: (DispatchQueue::entry()+0x51f) [0x7f838896c60f] 2024-03-24T00:33:34.534 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8388a3fe81] 2024-03-24T00:33:34.534 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f838745a1ca] 2024-03-24T00:33:34.534 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 19: clone() 2024-03-24T00:33:34.534 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 2024-03-24T00:33:34.132+0000 7f837e0b3700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:33:34.534 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: in thread 7f837e0b3700 thread_name:ms_dispatch 2024-03-24T00:33:34.534 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 2024-03-24T00:33:34.534 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:33:34.534 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8387464d20] 2024-03-24T00:33:34.534 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 2: gsignal() 2024-03-24T00:33:34.534 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 3: abort() 2024-03-24T00:33:34.535 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f83886dd4fb] 2024-03-24T00:33:34.535 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f83886dd667] 2024-03-24T00:33:34.535 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x562280d5e223] 2024-03-24T00:33:34.535 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x562280bd85d6] 2024-03-24T00:33:34.535 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x562280bd8da1] 2024-03-24T00:33:34.535 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x562280bdb2d5] 2024-03-24T00:33:34.535 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x562280c2b233] 2024-03-24T00:33:34.535 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x562280c2d197] 2024-03-24T00:33:34.535 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x562280b2c11b] 2024-03-24T00:33:34.535 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x562280b2c6bc] 2024-03-24T00:33:34.535 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x562280b161ff] 2024-03-24T00:33:34.535 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f838896f488] 2024-03-24T00:33:34.535 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 16: (DispatchQueue::entry()+0x51f) [0x7f838896c60f] 2024-03-24T00:33:34.535 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8388a3fe81] 2024-03-24T00:33:34.536 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f838745a1ca] 2024-03-24T00:33:34.536 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 19: clone() 2024-03-24T00:33:34.536 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:33:34.536 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 2024-03-24T00:33:34.536 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: -1> 2024-03-24T00:33:34.130+0000 7f837e0b3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f837e0b3700 time 2024-03-24T00:33:34.129228+0000 2024-03-24T00:33:34.536 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:33:34.536 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 2024-03-24T00:33:34.536 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:33:34.536 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f83886dd4a1] 2024-03-24T00:33:34.536 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f83886dd667] 2024-03-24T00:33:34.536 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x562280d5e223] 2024-03-24T00:33:34.536 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x562280bd85d6] 2024-03-24T00:33:34.536 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x562280bd8da1] 2024-03-24T00:33:34.537 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x562280bdb2d5] 2024-03-24T00:33:34.537 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x562280c2b233] 2024-03-24T00:33:34.537 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x562280c2d197] 2024-03-24T00:33:34.537 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x562280b2c11b] 2024-03-24T00:33:34.537 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x562280b2c6bc] 2024-03-24T00:33:34.537 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x562280b161ff] 2024-03-24T00:33:34.537 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f838896f488] 2024-03-24T00:33:34.537 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 13: (DispatchQueue::entry()+0x51f) [0x7f838896c60f] 2024-03-24T00:33:34.537 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8388a3fe81] 2024-03-24T00:33:34.537 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f838745a1ca] 2024-03-24T00:33:34.537 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 16: clone() 2024-03-24T00:33:34.537 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 2024-03-24T00:33:34.538 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 0> 2024-03-24T00:33:34.132+0000 7f837e0b3700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:33:34.538 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: in thread 7f837e0b3700 thread_name:ms_dispatch 2024-03-24T00:33:34.538 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 2024-03-24T00:33:34.538 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:33:34.538 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8387464d20] 2024-03-24T00:33:34.538 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 2: gsignal() 2024-03-24T00:33:34.538 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 3: abort() 2024-03-24T00:33:34.538 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f83886dd4fb] 2024-03-24T00:33:34.538 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f83886dd667] 2024-03-24T00:33:34.538 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x562280d5e223] 2024-03-24T00:33:34.538 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x562280bd85d6] 2024-03-24T00:33:34.538 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x562280bd8da1] 2024-03-24T00:33:34.538 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x562280bdb2d5] 2024-03-24T00:33:34.538 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x562280c2b233] 2024-03-24T00:33:34.539 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x562280c2d197] 2024-03-24T00:33:34.539 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x562280b2c11b] 2024-03-24T00:33:34.539 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x562280b2c6bc] 2024-03-24T00:33:34.539 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x562280b161ff] 2024-03-24T00:33:34.539 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f838896f488] 2024-03-24T00:33:34.539 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 16: (DispatchQueue::entry()+0x51f) [0x7f838896c60f] 2024-03-24T00:33:34.539 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8388a3fe81] 2024-03-24T00:33:34.539 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f838745a1ca] 2024-03-24T00:33:34.539 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 19: clone() 2024-03-24T00:33:34.539 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:33:34.539 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 2024-03-24T00:33:34.539 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: -1> 2024-03-24T00:33:34.130+0000 7f837e0b3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f837e0b3700 time 2024-03-24T00:33:34.129228+0000 2024-03-24T00:33:34.539 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:33:34.539 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 2024-03-24T00:33:34.540 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:33:34.540 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f83886dd4a1] 2024-03-24T00:33:34.540 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f83886dd667] 2024-03-24T00:33:34.540 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x562280d5e223] 2024-03-24T00:33:34.540 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x562280bd85d6] 2024-03-24T00:33:34.540 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x562280bd8da1] 2024-03-24T00:33:34.540 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x562280bdb2d5] 2024-03-24T00:33:34.540 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x562280c2b233] 2024-03-24T00:33:34.540 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x562280c2d197] 2024-03-24T00:33:34.540 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x562280b2c11b] 2024-03-24T00:33:34.541 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x562280b2c6bc] 2024-03-24T00:33:34.541 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x562280b161ff] 2024-03-24T00:33:34.541 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f838896f488] 2024-03-24T00:33:34.541 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 13: (DispatchQueue::entry()+0x51f) [0x7f838896c60f] 2024-03-24T00:33:34.541 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8388a3fe81] 2024-03-24T00:33:34.541 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f838745a1ca] 2024-03-24T00:33:34.541 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 16: clone() 2024-03-24T00:33:34.541 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 2024-03-24T00:33:34.541 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 0> 2024-03-24T00:33:34.132+0000 7f837e0b3700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:33:34.541 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: in thread 7f837e0b3700 thread_name:ms_dispatch 2024-03-24T00:33:34.541 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 2024-03-24T00:33:34.541 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:33:34.541 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8387464d20] 2024-03-24T00:33:34.541 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 2: gsignal() 2024-03-24T00:33:34.542 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 3: abort() 2024-03-24T00:33:34.542 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f83886dd4fb] 2024-03-24T00:33:34.542 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f83886dd667] 2024-03-24T00:33:34.542 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x562280d5e223] 2024-03-24T00:33:34.542 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x562280bd85d6] 2024-03-24T00:33:34.542 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x562280bd8da1] 2024-03-24T00:33:34.542 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x562280bdb2d5] 2024-03-24T00:33:34.542 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x562280c2b233] 2024-03-24T00:33:34.542 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x562280c2d197] 2024-03-24T00:33:34.542 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x562280b2c11b] 2024-03-24T00:33:34.542 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x562280b2c6bc] 2024-03-24T00:33:34.542 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x562280b161ff] 2024-03-24T00:33:34.542 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f838896f488] 2024-03-24T00:33:34.542 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 16: (DispatchQueue::entry()+0x51f) [0x7f838896c60f] 2024-03-24T00:33:34.543 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8388a3fe81] 2024-03-24T00:33:34.543 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f838745a1ca] 2024-03-24T00:33:34.543 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 19: clone() 2024-03-24T00:33:34.543 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:33:34.543 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90035]: 2024-03-24T00:33:34.543 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 podman[90365]: 2024-03-24 00:33:34.185289088 +0000 UTC m=+0.020791011 container died 6607ebbdfbbd4bb6060752ce8db81ebf6399c2a02f3d6e37eca0695b9f69f66f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-24T00:33:34.670 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:34 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6836/1186751954,v1:172.21.15.78:6837/1186751954] up:rejoin 2024-03-24T00:33:34.670 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:34 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:34.670 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:34 smithi106 ceph-mon[36505]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-24T00:33:34.670 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:34 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:33:34.671 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:34 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6836/1186751954,v1:172.21.15.78:6837/1186751954] up:active 2024-03-24T00:33:34.671 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:34 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=h=up:active} 3 up:standby-replay 2024-03-24T00:33:34.671 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:34 smithi106 ceph-mon[36505]: osdmap e145: 12 total, 12 up, 12 in 2024-03-24T00:33:34.713 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 543 sec latency 656304.275 ms 2024-03-24T00:33:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6836/1186751954,v1:172.21.15.78:6837/1186751954] up:rejoin 2024-03-24T00:33:34.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:34.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-mon[32429]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-24T00:33:34.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:33:34.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6836/1186751954,v1:172.21.15.78:6837/1186751954] up:active 2024-03-24T00:33:34.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=h=up:active} 3 up:standby-replay 2024-03-24T00:33:34.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:34 smithi078 ceph-mon[32429]: osdmap e145: 12 total, 12 up, 12 in 2024-03-24T00:33:35.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:34 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6836/1186751954,v1:172.21.15.78:6837/1186751954] up:rejoin 2024-03-24T00:33:35.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:34 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:35.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:34 smithi107 ceph-mon[36485]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-24T00:33:35.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:34 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:33:35.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:34 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6836/1186751954,v1:172.21.15.78:6837/1186751954] up:active 2024-03-24T00:33:35.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:34 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=h=up:active} 3 up:standby-replay 2024-03-24T00:33:35.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:34 smithi107 ceph-mon[36485]: osdmap e145: 12 total, 12 up, 12 in 2024-03-24T00:33:35.341 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 podman[90365]: 2024-03-24 00:33:34.872344521 +0000 UTC m=+0.707846474 container remove 6607ebbdfbbd4bb6060752ce8db81ebf6399c2a02f3d6e37eca0695b9f69f66f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-24T00:33:35.341 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:34 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:33:35.341 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:35 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.d.service: Failed with result 'exit-code'. 2024-03-24T00:33:35.713 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 544 sec latency 657304.378 ms 2024-03-24T00:33:36.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:35 smithi078 ceph-mon[32429]: pgmap v430: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 611 B/s wr, 16 op/s 2024-03-24T00:33:36.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:35 smithi107 ceph-mon[36485]: pgmap v430: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 611 B/s wr, 16 op/s 2024-03-24T00:33:36.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:35 smithi106 ceph-mon[36505]: pgmap v430: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 611 B/s wr, 16 op/s 2024-03-24T00:33:36.713 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 545 sec latency 658304.458 ms 2024-03-24T00:33:37.714 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 546 sec latency 659304.576 ms 2024-03-24T00:33:38.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:38 smithi106 ceph-mon[36505]: pgmap v431: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 511 B/s wr, 13 op/s 2024-03-24T00:33:38.714 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 547 sec latency 660304.698 ms 2024-03-24T00:33:38.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:38 smithi078 ceph-mon[32429]: pgmap v431: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 511 B/s wr, 13 op/s 2024-03-24T00:33:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:38 smithi107 ceph-mon[36485]: pgmap v431: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 511 B/s wr, 13 op/s 2024-03-24T00:33:39.714 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 548 sec latency 661304.800 ms 2024-03-24T00:33:40.714 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 549 sec latency 662304.941 ms 2024-03-24T00:33:40.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:40 smithi078 ceph-mon[32429]: pgmap v432: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 351 B/s wr, 8 op/s 2024-03-24T00:33:40.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:40 smithi107 ceph-mon[36485]: pgmap v432: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 351 B/s wr, 8 op/s 2024-03-24T00:33:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:40 smithi106 ceph-mon[36505]: pgmap v432: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 351 B/s wr, 8 op/s 2024-03-24T00:33:41.714 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 550 sec latency 663305.007 ms 2024-03-24T00:33:42.714 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 551 sec latency 664305.053 ms 2024-03-24T00:33:42.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:42 smithi078 ceph-mon[32429]: pgmap v433: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-24T00:33:42.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:42 smithi107 ceph-mon[36485]: pgmap v433: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-24T00:33:42.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:42 smithi106 ceph-mon[36505]: pgmap v433: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-24T00:33:43.714 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 552 sec latency 665305.108 ms 2024-03-24T00:33:44.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:44 smithi078 ceph-mon[32429]: pgmap v434: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-24T00:33:44.714 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 553 sec latency 666305.162 ms 2024-03-24T00:33:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:44 smithi107 ceph-mon[36485]: pgmap v434: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-24T00:33:44.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:44 smithi106 ceph-mon[36505]: pgmap v434: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-24T00:33:45.621 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:45 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.d.service: Scheduled restart job, restart counter is at 4. 2024-03-24T00:33:45.621 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:45 smithi078 systemd[1]: Stopped Ceph mds.d for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:33:45.621 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:45 smithi078 systemd[1]: Starting Ceph mds.d for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:33:45.621 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:45 smithi078 podman[90461]: 2024-03-24 00:33:45.482266685 +0000 UTC m=+0.019548133 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:33:45.714 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 554 sec latency 667305.201 ms 2024-03-24T00:33:45.882 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:45 smithi078 podman[90461]: 2024-03-24 00:33:45.621896052 +0000 UTC m=+0.159177533 container create 887af7276f3425c49fcb5cd15c01bca73168952c68e340503a0bf6ed198ce275 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-24T00:33:45.883 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:45 smithi078 podman[90461]: 2024-03-24 00:33:45.858063562 +0000 UTC m=+0.395345003 container init 887af7276f3425c49fcb5cd15c01bca73168952c68e340503a0bf6ed198ce275 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-24T00:33:45.883 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:45 smithi078 podman[90461]: 2024-03-24 00:33:45.860911783 +0000 UTC m=+0.398193227 container start 887af7276f3425c49fcb5cd15c01bca73168952c68e340503a0bf6ed198ce275 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-24T00:33:45.883 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:45 smithi078 ceph-mds[90480]: -- 172.21.15.78:0/960290064 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55696b971a40 con 0x55696b945000 2024-03-24T00:33:46.340 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:45 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: starting mds.d at 2024-03-24T00:33:46.340 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:45 smithi078 bash[90461]: 887af7276f3425c49fcb5cd15c01bca73168952c68e340503a0bf6ed198ce275 2024-03-24T00:33:46.341 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:33:45 smithi078 systemd[1]: Started Ceph mds.d for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:33:46.714 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 555 sec latency 668305.295 ms 2024-03-24T00:33:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:46 smithi078 ceph-mon[32429]: pgmap v435: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 273 B/s wr, 12 op/s 2024-03-24T00:33:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:46 smithi078 ceph-mon[32429]: daemon mds.d restarted 2024-03-24T00:33:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:46 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:33:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:46 smithi078 ceph-mon[32429]: osdmap e146: 12 total, 12 up, 12 in 2024-03-24T00:33:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:46 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6836/878237454,v1:172.21.15.78:6837/878237454] up:boot 2024-03-24T00:33:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:46 smithi078 ceph-mon[32429]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-03-24T00:33:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:46 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:33:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:46 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:33:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:46 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:46.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:46 smithi107 ceph-mon[36485]: pgmap v435: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 273 B/s wr, 12 op/s 2024-03-24T00:33:46.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:46 smithi107 ceph-mon[36485]: daemon mds.d restarted 2024-03-24T00:33:46.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:46 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:33:46.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:46 smithi107 ceph-mon[36485]: osdmap e146: 12 total, 12 up, 12 in 2024-03-24T00:33:46.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:46 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6836/878237454,v1:172.21.15.78:6837/878237454] up:boot 2024-03-24T00:33:46.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:46 smithi107 ceph-mon[36485]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-03-24T00:33:46.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:46 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:33:46.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:46 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:33:46.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:46 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:46 smithi106 ceph-mon[36505]: pgmap v435: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 273 B/s wr, 12 op/s 2024-03-24T00:33:46.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:46 smithi106 ceph-mon[36505]: daemon mds.d restarted 2024-03-24T00:33:46.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:46 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:33:46.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:46 smithi106 ceph-mon[36505]: osdmap e146: 12 total, 12 up, 12 in 2024-03-24T00:33:46.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:46 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6836/878237454,v1:172.21.15.78:6837/878237454] up:boot 2024-03-24T00:33:46.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:46 smithi106 ceph-mon[36505]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-03-24T00:33:46.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:46 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:33:46.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:46 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:33:46.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:46 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:47.714 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 556 sec latency 669305.409 ms 2024-03-24T00:33:48.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:47 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6834/2163967090,v1:172.21.15.78:6835/2163967090] up:resolve 2024-03-24T00:33:48.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:47 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:48.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:47 smithi078 ceph-mon[32429]: pgmap v437: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:33:48.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:47 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6834/2163967090,v1:172.21.15.78:6835/2163967090] up:resolve 2024-03-24T00:33:48.345 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:47 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:48.345 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:47 smithi107 ceph-mon[36485]: pgmap v437: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:33:48.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:47 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6834/2163967090,v1:172.21.15.78:6835/2163967090] up:resolve 2024-03-24T00:33:48.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:47 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:48.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:47 smithi106 ceph-mon[36505]: pgmap v437: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:33:48.714 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 557 sec latency 670305.513 ms 2024-03-24T00:33:49.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:48 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:33:49.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:48 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6834/2163967090,v1:172.21.15.78:6835/2163967090] up:reconnect 2024-03-24T00:33:49.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:48 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:49.186 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f1b1d786700 time 2024-03-24T00:33:49.130589+0000 2024-03-24T00:33:49.186 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:33:49.186 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:33:49.186 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1b27db04a1] 2024-03-24T00:33:49.186 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1b27db0667] 2024-03-24T00:33:49.186 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55fa247cf223] 2024-03-24T00:33:49.187 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55fa246495d6] 2024-03-24T00:33:49.187 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55fa24649da1] 2024-03-24T00:33:49.187 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55fa2464c2d5] 2024-03-24T00:33:49.187 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55fa2469c233] 2024-03-24T00:33:49.187 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55fa2469e197] 2024-03-24T00:33:49.187 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55fa2459d11b] 2024-03-24T00:33:49.187 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55fa2459d6bc] 2024-03-24T00:33:49.187 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55fa245871ff] 2024-03-24T00:33:49.187 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1b28042488] 2024-03-24T00:33:49.187 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 13: (DispatchQueue::entry()+0x51f) [0x7f1b2803f60f] 2024-03-24T00:33:49.187 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1b28112e81] 2024-03-24T00:33:49.187 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1b26b2d1ca] 2024-03-24T00:33:49.187 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 16: clone() 2024-03-24T00:33:49.187 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 2024-03-24T00:33:49.130+0000 7f1b1d786700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f1b1d786700 time 2024-03-24T00:33:49.130589+0000 2024-03-24T00:33:49.188 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:33:49.188 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 2024-03-24T00:33:49.188 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:33:49.188 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1b27db04a1] 2024-03-24T00:33:49.188 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1b27db0667] 2024-03-24T00:33:49.188 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55fa247cf223] 2024-03-24T00:33:49.188 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55fa246495d6] 2024-03-24T00:33:49.188 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55fa24649da1] 2024-03-24T00:33:49.188 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55fa2464c2d5] 2024-03-24T00:33:49.188 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55fa2469c233] 2024-03-24T00:33:49.188 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55fa2469e197] 2024-03-24T00:33:49.188 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55fa2459d11b] 2024-03-24T00:33:49.188 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55fa2459d6bc] 2024-03-24T00:33:49.189 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55fa245871ff] 2024-03-24T00:33:49.189 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1b28042488] 2024-03-24T00:33:49.189 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 13: (DispatchQueue::entry()+0x51f) [0x7f1b2803f60f] 2024-03-24T00:33:49.189 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1b28112e81] 2024-03-24T00:33:49.189 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1b26b2d1ca] 2024-03-24T00:33:49.189 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 16: clone() 2024-03-24T00:33:49.189 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 2024-03-24T00:33:49.189 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: *** Caught signal (Aborted) ** 2024-03-24T00:33:49.189 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: in thread 7f1b1d786700 thread_name:ms_dispatch 2024-03-24T00:33:49.189 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:33:49.189 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1b26b37d20] 2024-03-24T00:33:49.190 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 2: gsignal() 2024-03-24T00:33:49.190 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 3: abort() 2024-03-24T00:33:49.190 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1b27db04fb] 2024-03-24T00:33:49.190 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1b27db0667] 2024-03-24T00:33:49.191 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55fa247cf223] 2024-03-24T00:33:49.191 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55fa246495d6] 2024-03-24T00:33:49.191 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55fa24649da1] 2024-03-24T00:33:49.191 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55fa2464c2d5] 2024-03-24T00:33:49.191 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55fa2469c233] 2024-03-24T00:33:49.191 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55fa2469e197] 2024-03-24T00:33:49.191 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55fa2459d11b] 2024-03-24T00:33:49.191 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55fa2459d6bc] 2024-03-24T00:33:49.191 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55fa245871ff] 2024-03-24T00:33:49.191 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1b28042488] 2024-03-24T00:33:49.191 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 16: (DispatchQueue::entry()+0x51f) [0x7f1b2803f60f] 2024-03-24T00:33:49.191 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1b28112e81] 2024-03-24T00:33:49.191 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1b26b2d1ca] 2024-03-24T00:33:49.192 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 19: clone() 2024-03-24T00:33:49.192 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 2024-03-24T00:33:49.132+0000 7f1b1d786700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:33:49.192 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: in thread 7f1b1d786700 thread_name:ms_dispatch 2024-03-24T00:33:49.192 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 2024-03-24T00:33:49.192 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:33:49.192 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1b26b37d20] 2024-03-24T00:33:49.192 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 2: gsignal() 2024-03-24T00:33:49.192 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 3: abort() 2024-03-24T00:33:49.192 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1b27db04fb] 2024-03-24T00:33:49.192 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1b27db0667] 2024-03-24T00:33:49.192 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55fa247cf223] 2024-03-24T00:33:49.192 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55fa246495d6] 2024-03-24T00:33:49.192 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55fa24649da1] 2024-03-24T00:33:49.192 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55fa2464c2d5] 2024-03-24T00:33:49.193 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55fa2469c233] 2024-03-24T00:33:49.193 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55fa2469e197] 2024-03-24T00:33:49.193 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55fa2459d11b] 2024-03-24T00:33:49.193 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55fa2459d6bc] 2024-03-24T00:33:49.193 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55fa245871ff] 2024-03-24T00:33:49.193 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1b28042488] 2024-03-24T00:33:49.193 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 16: (DispatchQueue::entry()+0x51f) [0x7f1b2803f60f] 2024-03-24T00:33:49.193 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1b28112e81] 2024-03-24T00:33:49.193 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1b26b2d1ca] 2024-03-24T00:33:49.193 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 19: clone() 2024-03-24T00:33:49.193 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:33:49.193 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 2024-03-24T00:33:49.193 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: -1> 2024-03-24T00:33:49.130+0000 7f1b1d786700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f1b1d786700 time 2024-03-24T00:33:49.130589+0000 2024-03-24T00:33:49.193 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:33:49.193 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 2024-03-24T00:33:49.194 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:33:49.194 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1b27db04a1] 2024-03-24T00:33:49.194 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1b27db0667] 2024-03-24T00:33:49.194 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55fa247cf223] 2024-03-24T00:33:49.194 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55fa246495d6] 2024-03-24T00:33:49.194 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55fa24649da1] 2024-03-24T00:33:49.194 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55fa2464c2d5] 2024-03-24T00:33:49.194 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55fa2469c233] 2024-03-24T00:33:49.194 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55fa2469e197] 2024-03-24T00:33:49.194 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55fa2459d11b] 2024-03-24T00:33:49.194 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55fa2459d6bc] 2024-03-24T00:33:49.194 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55fa245871ff] 2024-03-24T00:33:49.194 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1b28042488] 2024-03-24T00:33:49.195 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 13: (DispatchQueue::entry()+0x51f) [0x7f1b2803f60f] 2024-03-24T00:33:49.195 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1b28112e81] 2024-03-24T00:33:49.195 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1b26b2d1ca] 2024-03-24T00:33:49.195 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 16: clone() 2024-03-24T00:33:49.195 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 2024-03-24T00:33:49.195 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 0> 2024-03-24T00:33:49.132+0000 7f1b1d786700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:33:49.195 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: in thread 7f1b1d786700 thread_name:ms_dispatch 2024-03-24T00:33:49.195 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 2024-03-24T00:33:49.195 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:33:49.195 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1b26b37d20] 2024-03-24T00:33:49.195 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 2: gsignal() 2024-03-24T00:33:49.195 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 3: abort() 2024-03-24T00:33:49.195 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1b27db04fb] 2024-03-24T00:33:49.195 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1b27db0667] 2024-03-24T00:33:49.196 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55fa247cf223] 2024-03-24T00:33:49.196 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55fa246495d6] 2024-03-24T00:33:49.196 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55fa24649da1] 2024-03-24T00:33:49.196 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55fa2464c2d5] 2024-03-24T00:33:49.196 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55fa2469c233] 2024-03-24T00:33:49.196 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55fa2469e197] 2024-03-24T00:33:49.196 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55fa2459d11b] 2024-03-24T00:33:49.196 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55fa2459d6bc] 2024-03-24T00:33:49.196 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55fa245871ff] 2024-03-24T00:33:49.196 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1b28042488] 2024-03-24T00:33:49.196 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 16: (DispatchQueue::entry()+0x51f) [0x7f1b2803f60f] 2024-03-24T00:33:49.196 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1b28112e81] 2024-03-24T00:33:49.196 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1b26b2d1ca] 2024-03-24T00:33:49.196 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 19: clone() 2024-03-24T00:33:49.196 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:33:49.197 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 2024-03-24T00:33:49.197 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: -1> 2024-03-24T00:33:49.130+0000 7f1b1d786700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f1b1d786700 time 2024-03-24T00:33:49.130589+0000 2024-03-24T00:33:49.197 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:33:49.197 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 2024-03-24T00:33:49.197 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:33:49.197 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1b27db04a1] 2024-03-24T00:33:49.197 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1b27db0667] 2024-03-24T00:33:49.197 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55fa247cf223] 2024-03-24T00:33:49.197 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55fa246495d6] 2024-03-24T00:33:49.197 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55fa24649da1] 2024-03-24T00:33:49.197 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55fa2464c2d5] 2024-03-24T00:33:49.197 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55fa2469c233] 2024-03-24T00:33:49.197 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55fa2469e197] 2024-03-24T00:33:49.197 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55fa2459d11b] 2024-03-24T00:33:49.198 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55fa2459d6bc] 2024-03-24T00:33:49.198 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55fa245871ff] 2024-03-24T00:33:49.198 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1b28042488] 2024-03-24T00:33:49.198 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 13: (DispatchQueue::entry()+0x51f) [0x7f1b2803f60f] 2024-03-24T00:33:49.198 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1b28112e81] 2024-03-24T00:33:49.198 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1b26b2d1ca] 2024-03-24T00:33:49.198 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 16: clone() 2024-03-24T00:33:49.198 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 2024-03-24T00:33:49.198 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 0> 2024-03-24T00:33:49.132+0000 7f1b1d786700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:33:49.198 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: in thread 7f1b1d786700 thread_name:ms_dispatch 2024-03-24T00:33:49.198 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 2024-03-24T00:33:49.198 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:33:49.198 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1b26b37d20] 2024-03-24T00:33:49.198 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 2: gsignal() 2024-03-24T00:33:49.199 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 3: abort() 2024-03-24T00:33:49.199 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1b27db04fb] 2024-03-24T00:33:49.199 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1b27db0667] 2024-03-24T00:33:49.199 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55fa247cf223] 2024-03-24T00:33:49.199 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55fa246495d6] 2024-03-24T00:33:49.199 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55fa24649da1] 2024-03-24T00:33:49.199 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55fa2464c2d5] 2024-03-24T00:33:49.199 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55fa2469c233] 2024-03-24T00:33:49.199 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55fa2469e197] 2024-03-24T00:33:49.199 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55fa2459d11b] 2024-03-24T00:33:49.199 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55fa2459d6bc] 2024-03-24T00:33:49.199 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55fa245871ff] 2024-03-24T00:33:49.199 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1b28042488] 2024-03-24T00:33:49.199 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 16: (DispatchQueue::entry()+0x51f) [0x7f1b2803f60f] 2024-03-24T00:33:49.199 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1b28112e81] 2024-03-24T00:33:49.200 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1b26b2d1ca] 2024-03-24T00:33:49.200 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 19: clone() 2024-03-24T00:33:49.200 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:33:49.200 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90316]: 2024-03-24T00:33:49.200 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 podman[90514]: 2024-03-24 00:33:49.186190053 +0000 UTC m=+0.019762096 container died 78a0a33da52aa067b772376a25afe236c61fcde18a0469b8781847763013677a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-24T00:33:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:48 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:33:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:48 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6834/2163967090,v1:172.21.15.78:6835/2163967090] up:reconnect 2024-03-24T00:33:49.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:48 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:48 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:33:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:48 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6834/2163967090,v1:172.21.15.78:6835/2163967090] up:reconnect 2024-03-24T00:33:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:48 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:49.714 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 558 sec latency 671305.626 ms 2024-03-24T00:33:50.340 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 podman[90514]: 2024-03-24 00:33:49.905068433 +0000 UTC m=+0.738640462 container remove 78a0a33da52aa067b772376a25afe236c61fcde18a0469b8781847763013677a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-24T00:33:50.341 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:49 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:33:50.341 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:33:50 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.j.service: Failed with result 'exit-code'. 2024-03-24T00:33:50.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6834/2163967090,v1:172.21.15.78:6835/2163967090] up:rejoin 2024-03-24T00:33:50.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:50.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-mon[32429]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-24T00:33:50.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:33:50.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6834/2163967090,v1:172.21.15.78:6835/2163967090] up:active 2024-03-24T00:33:50.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=h=up:active} 3 up:standby-replay 2024-03-24T00:33:50.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:49 smithi078 ceph-mon[32429]: pgmap v438: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:33:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:49 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6834/2163967090,v1:172.21.15.78:6835/2163967090] up:rejoin 2024-03-24T00:33:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:49 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:49 smithi107 ceph-mon[36485]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-24T00:33:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:49 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:33:50.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:49 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6834/2163967090,v1:172.21.15.78:6835/2163967090] up:active 2024-03-24T00:33:50.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:49 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=h=up:active} 3 up:standby-replay 2024-03-24T00:33:50.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:49 smithi107 ceph-mon[36485]: pgmap v438: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:33:50.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:49 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6834/2163967090,v1:172.21.15.78:6835/2163967090] up:rejoin 2024-03-24T00:33:50.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:49 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:33:50.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:49 smithi106 ceph-mon[36505]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-24T00:33:50.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:49 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:33:50.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:49 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6834/2163967090,v1:172.21.15.78:6835/2163967090] up:active 2024-03-24T00:33:50.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:49 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=h=up:active} 3 up:standby-replay 2024-03-24T00:33:50.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:49 smithi106 ceph-mon[36505]: pgmap v438: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:33:50.715 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 559 sec latency 672305.697 ms 2024-03-24T00:33:51.715 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 560 sec latency 673305.771 ms 2024-03-24T00:33:52.715 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 561 sec latency 674305.866 ms 2024-03-24T00:33:52.764 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:52 smithi078 ceph-mon[32429]: pgmap v439: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:33:52.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:52 smithi107 ceph-mon[36485]: pgmap v439: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:33:52.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:52 smithi106 ceph-mon[36505]: pgmap v439: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:33:53.715 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 562 sec latency 675305.963 ms 2024-03-24T00:33:54.715 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 563 sec latency 676306.055 ms 2024-03-24T00:33:54.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:54 smithi078 ceph-mon[32429]: pgmap v440: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:33:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:54 smithi107 ceph-mon[36485]: pgmap v440: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:33:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:54 smithi106 ceph-mon[36505]: pgmap v440: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:33:55.715 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 564 sec latency 677306.104 ms 2024-03-24T00:33:56.715 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 565 sec latency 678306.148 ms 2024-03-24T00:33:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:56 smithi078 ceph-mon[32429]: pgmap v441: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:33:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:56 smithi107 ceph-mon[36485]: pgmap v441: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:33:56.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:56 smithi106 ceph-mon[36505]: pgmap v441: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T00:33:57.715 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 566 sec latency 679306.264 ms 2024-03-24T00:33:58.715 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 567 sec latency 680306.358 ms 2024-03-24T00:33:58.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:33:58 smithi078 ceph-mon[32429]: pgmap v442: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 178 B/s wr, 9 op/s 2024-03-24T00:33:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:33:58 smithi107 ceph-mon[36485]: pgmap v442: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 178 B/s wr, 9 op/s 2024-03-24T00:33:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:33:58 smithi106 ceph-mon[36505]: pgmap v442: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 178 B/s wr, 9 op/s 2024-03-24T00:33:59.715 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 568 sec latency 681306.468 ms 2024-03-24T00:34:00.606 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:00 smithi078 ceph-mon[32429]: pgmap v443: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-24T00:34:00.606 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:00 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.j.service: Scheduled restart job, restart counter is at 2. 2024-03-24T00:34:00.607 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:00 smithi078 systemd[1]: Stopped Ceph mds.j for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:34:00.607 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:00 smithi078 systemd[1]: Starting Ceph mds.j for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:34:00.607 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:00 smithi078 podman[90604]: 2024-03-24 00:34:00.481829747 +0000 UTC m=+0.019592180 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:34:00.608 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:00 smithi078 podman[90604]: 2024-03-24 00:34:00.605947107 +0000 UTC m=+0.143709539 container create d3e87c5f3abb4eb0a6adc0d1865bf63521eb37f9655a044c59012e3c02f9b2d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-24T00:34:00.715 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 569 sec latency 682306.524 ms 2024-03-24T00:34:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:00 smithi107 ceph-mon[36485]: pgmap v443: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-24T00:34:00.874 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:00 smithi078 podman[90604]: 2024-03-24 00:34:00.850168638 +0000 UTC m=+0.387931079 container init d3e87c5f3abb4eb0a6adc0d1865bf63521eb37f9655a044c59012e3c02f9b2d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-24T00:34:00.874 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:00 smithi078 podman[90604]: 2024-03-24 00:34:00.852830824 +0000 UTC m=+0.390593266 container start d3e87c5f3abb4eb0a6adc0d1865bf63521eb37f9655a044c59012e3c02f9b2d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-24T00:34:00.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:00 smithi106 ceph-mon[36505]: pgmap v443: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-24T00:34:01.340 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:00 smithi078 ceph-mds[90623]: -- 172.21.15.78:0/625360448 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5626d3055a40 con 0x5626d3029000 2024-03-24T00:34:01.340 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:00 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: starting mds.j at 2024-03-24T00:34:01.341 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:00 smithi078 bash[90604]: d3e87c5f3abb4eb0a6adc0d1865bf63521eb37f9655a044c59012e3c02f9b2d9 2024-03-24T00:34:01.341 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:00 smithi078 systemd[1]: Started Ceph mds.j for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:34:01.716 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 570 sec latency 683306.655 ms 2024-03-24T00:34:01.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:01 smithi078 ceph-mon[32429]: daemon mds.j restarted 2024-03-24T00:34:01.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:01 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:34:01.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:01 smithi078 ceph-mon[32429]: osdmap e147: 12 total, 12 up, 12 in 2024-03-24T00:34:01.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:01 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6834/93340378,v1:172.21.15.78:6835/93340378] up:boot 2024-03-24T00:34:01.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:01 smithi078 ceph-mon[32429]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-24T00:34:01.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:01 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:34:01.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:01 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:34:01.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:01 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:01.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:01 smithi107 ceph-mon[36485]: daemon mds.j restarted 2024-03-24T00:34:01.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:01 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:34:01.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:01 smithi107 ceph-mon[36485]: osdmap e147: 12 total, 12 up, 12 in 2024-03-24T00:34:01.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:01 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6834/93340378,v1:172.21.15.78:6835/93340378] up:boot 2024-03-24T00:34:01.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:01 smithi107 ceph-mon[36485]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-24T00:34:01.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:01 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:34:01.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:01 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:34:01.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:01 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:01.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:01 smithi106 ceph-mon[36505]: daemon mds.j restarted 2024-03-24T00:34:01.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:01 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:34:01.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:01 smithi106 ceph-mon[36505]: osdmap e147: 12 total, 12 up, 12 in 2024-03-24T00:34:01.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:01 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6834/93340378,v1:172.21.15.78:6835/93340378] up:boot 2024-03-24T00:34:01.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:01 smithi106 ceph-mon[36505]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-24T00:34:01.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:01 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:34:01.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:01 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:34:01.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:01 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:02.716 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 571 sec latency 684306.753 ms 2024-03-24T00:34:02.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:02 smithi078 ceph-mon[32429]: pgmap v445: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:34:02.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:02 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6836/878237454,v1:172.21.15.78:6837/878237454] up:resolve 2024-03-24T00:34:02.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:02 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:02 smithi107 ceph-mon[36485]: pgmap v445: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:34:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:02 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6836/878237454,v1:172.21.15.78:6837/878237454] up:resolve 2024-03-24T00:34:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:02 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:02.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:02 smithi106 ceph-mon[36505]: pgmap v445: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:34:02.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:02 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6836/878237454,v1:172.21.15.78:6837/878237454] up:resolve 2024-03-24T00:34:02.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:02 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:03.716 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 572 sec latency 685306.886 ms 2024-03-24T00:34:04.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:03 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:34:04.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:03 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6836/878237454,v1:172.21.15.78:6837/878237454] up:reconnect 2024-03-24T00:34:04.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:03 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:04.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:03 smithi078 ceph-mon[32429]: pgmap v446: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:34:04.197 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f172fd08700 time 2024-03-24T00:34:04.133091+0000 2024-03-24T00:34:04.197 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:34:04.197 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:34:04.197 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f173a3324a1] 2024-03-24T00:34:04.197 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f173a332667] 2024-03-24T00:34:04.197 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556968cec223] 2024-03-24T00:34:04.197 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556968b665d6] 2024-03-24T00:34:04.197 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556968b66da1] 2024-03-24T00:34:04.197 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556968b692d5] 2024-03-24T00:34:04.197 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556968bb9233] 2024-03-24T00:34:04.197 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556968bbb197] 2024-03-24T00:34:04.198 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556968aba11b] 2024-03-24T00:34:04.198 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556968aba6bc] 2024-03-24T00:34:04.198 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556968aa41ff] 2024-03-24T00:34:04.198 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f173a5c4488] 2024-03-24T00:34:04.198 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 13: (DispatchQueue::entry()+0x51f) [0x7f173a5c160f] 2024-03-24T00:34:04.198 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f173a694e81] 2024-03-24T00:34:04.198 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f17390af1ca] 2024-03-24T00:34:04.198 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 16: clone() 2024-03-24T00:34:04.198 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 2024-03-24T00:34:04.134+0000 7f172fd08700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f172fd08700 time 2024-03-24T00:34:04.133091+0000 2024-03-24T00:34:04.198 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:34:04.198 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 2024-03-24T00:34:04.198 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:34:04.198 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f173a3324a1] 2024-03-24T00:34:04.199 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f173a332667] 2024-03-24T00:34:04.199 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556968cec223] 2024-03-24T00:34:04.199 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556968b665d6] 2024-03-24T00:34:04.199 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556968b66da1] 2024-03-24T00:34:04.199 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556968b692d5] 2024-03-24T00:34:04.199 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556968bb9233] 2024-03-24T00:34:04.199 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556968bbb197] 2024-03-24T00:34:04.199 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556968aba11b] 2024-03-24T00:34:04.199 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556968aba6bc] 2024-03-24T00:34:04.199 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556968aa41ff] 2024-03-24T00:34:04.199 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f173a5c4488] 2024-03-24T00:34:04.199 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 13: (DispatchQueue::entry()+0x51f) [0x7f173a5c160f] 2024-03-24T00:34:04.199 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f173a694e81] 2024-03-24T00:34:04.200 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f17390af1ca] 2024-03-24T00:34:04.200 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 16: clone() 2024-03-24T00:34:04.200 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 2024-03-24T00:34:04.200 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: *** Caught signal (Aborted) ** 2024-03-24T00:34:04.200 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: in thread 7f172fd08700 thread_name:ms_dispatch 2024-03-24T00:34:04.200 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:34:04.200 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f17390b9d20] 2024-03-24T00:34:04.201 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 2: gsignal() 2024-03-24T00:34:04.201 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 3: abort() 2024-03-24T00:34:04.201 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f173a3324fb] 2024-03-24T00:34:04.201 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f173a332667] 2024-03-24T00:34:04.201 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556968cec223] 2024-03-24T00:34:04.201 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556968b665d6] 2024-03-24T00:34:04.201 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556968b66da1] 2024-03-24T00:34:04.202 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556968b692d5] 2024-03-24T00:34:04.202 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556968bb9233] 2024-03-24T00:34:04.202 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556968bbb197] 2024-03-24T00:34:04.202 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556968aba11b] 2024-03-24T00:34:04.202 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556968aba6bc] 2024-03-24T00:34:04.202 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556968aa41ff] 2024-03-24T00:34:04.202 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f173a5c4488] 2024-03-24T00:34:04.202 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 16: (DispatchQueue::entry()+0x51f) [0x7f173a5c160f] 2024-03-24T00:34:04.202 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f173a694e81] 2024-03-24T00:34:04.202 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f17390af1ca] 2024-03-24T00:34:04.202 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 19: clone() 2024-03-24T00:34:04.202 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 2024-03-24T00:34:04.136+0000 7f172fd08700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:34:04.202 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: in thread 7f172fd08700 thread_name:ms_dispatch 2024-03-24T00:34:04.203 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 2024-03-24T00:34:04.203 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:34:04.203 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f17390b9d20] 2024-03-24T00:34:04.203 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 2: gsignal() 2024-03-24T00:34:04.203 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 3: abort() 2024-03-24T00:34:04.203 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f173a3324fb] 2024-03-24T00:34:04.203 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f173a332667] 2024-03-24T00:34:04.203 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556968cec223] 2024-03-24T00:34:04.203 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556968b665d6] 2024-03-24T00:34:04.203 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556968b66da1] 2024-03-24T00:34:04.203 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556968b692d5] 2024-03-24T00:34:04.203 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556968bb9233] 2024-03-24T00:34:04.203 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556968bbb197] 2024-03-24T00:34:04.203 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556968aba11b] 2024-03-24T00:34:04.203 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556968aba6bc] 2024-03-24T00:34:04.204 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556968aa41ff] 2024-03-24T00:34:04.204 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f173a5c4488] 2024-03-24T00:34:04.204 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 16: (DispatchQueue::entry()+0x51f) [0x7f173a5c160f] 2024-03-24T00:34:04.204 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f173a694e81] 2024-03-24T00:34:04.204 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f17390af1ca] 2024-03-24T00:34:04.204 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 19: clone() 2024-03-24T00:34:04.204 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:34:04.204 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 2024-03-24T00:34:04.204 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: -1> 2024-03-24T00:34:04.134+0000 7f172fd08700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f172fd08700 time 2024-03-24T00:34:04.133091+0000 2024-03-24T00:34:04.204 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:34:04.204 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 2024-03-24T00:34:04.204 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:34:04.204 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f173a3324a1] 2024-03-24T00:34:04.204 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f173a332667] 2024-03-24T00:34:04.204 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556968cec223] 2024-03-24T00:34:04.205 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556968b665d6] 2024-03-24T00:34:04.205 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556968b66da1] 2024-03-24T00:34:04.205 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556968b692d5] 2024-03-24T00:34:04.205 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556968bb9233] 2024-03-24T00:34:04.205 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556968bbb197] 2024-03-24T00:34:04.205 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556968aba11b] 2024-03-24T00:34:04.205 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556968aba6bc] 2024-03-24T00:34:04.205 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556968aa41ff] 2024-03-24T00:34:04.205 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f173a5c4488] 2024-03-24T00:34:04.205 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 13: (DispatchQueue::entry()+0x51f) [0x7f173a5c160f] 2024-03-24T00:34:04.205 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f173a694e81] 2024-03-24T00:34:04.205 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f17390af1ca] 2024-03-24T00:34:04.205 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 16: clone() 2024-03-24T00:34:04.205 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 2024-03-24T00:34:04.206 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 0> 2024-03-24T00:34:04.136+0000 7f172fd08700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:34:04.206 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: in thread 7f172fd08700 thread_name:ms_dispatch 2024-03-24T00:34:04.206 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 2024-03-24T00:34:04.206 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:34:04.206 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f17390b9d20] 2024-03-24T00:34:04.206 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 2: gsignal() 2024-03-24T00:34:04.206 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 3: abort() 2024-03-24T00:34:04.206 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f173a3324fb] 2024-03-24T00:34:04.206 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f173a332667] 2024-03-24T00:34:04.206 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556968cec223] 2024-03-24T00:34:04.206 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556968b665d6] 2024-03-24T00:34:04.206 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556968b66da1] 2024-03-24T00:34:04.206 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556968b692d5] 2024-03-24T00:34:04.207 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556968bb9233] 2024-03-24T00:34:04.207 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556968bbb197] 2024-03-24T00:34:04.207 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556968aba11b] 2024-03-24T00:34:04.207 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556968aba6bc] 2024-03-24T00:34:04.207 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556968aa41ff] 2024-03-24T00:34:04.207 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f173a5c4488] 2024-03-24T00:34:04.207 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 16: (DispatchQueue::entry()+0x51f) [0x7f173a5c160f] 2024-03-24T00:34:04.207 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f173a694e81] 2024-03-24T00:34:04.207 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f17390af1ca] 2024-03-24T00:34:04.207 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 19: clone() 2024-03-24T00:34:04.207 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:34:04.207 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 2024-03-24T00:34:04.207 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: -1> 2024-03-24T00:34:04.134+0000 7f172fd08700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f172fd08700 time 2024-03-24T00:34:04.133091+0000 2024-03-24T00:34:04.207 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:34:04.208 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 2024-03-24T00:34:04.208 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:34:04.208 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f173a3324a1] 2024-03-24T00:34:04.208 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f173a332667] 2024-03-24T00:34:04.208 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556968cec223] 2024-03-24T00:34:04.208 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556968b665d6] 2024-03-24T00:34:04.208 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556968b66da1] 2024-03-24T00:34:04.208 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556968b692d5] 2024-03-24T00:34:04.208 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556968bb9233] 2024-03-24T00:34:04.208 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556968bbb197] 2024-03-24T00:34:04.208 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556968aba11b] 2024-03-24T00:34:04.208 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556968aba6bc] 2024-03-24T00:34:04.208 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556968aa41ff] 2024-03-24T00:34:04.208 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f173a5c4488] 2024-03-24T00:34:04.209 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 13: (DispatchQueue::entry()+0x51f) [0x7f173a5c160f] 2024-03-24T00:34:04.209 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f173a694e81] 2024-03-24T00:34:04.209 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f17390af1ca] 2024-03-24T00:34:04.209 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 16: clone() 2024-03-24T00:34:04.209 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 2024-03-24T00:34:04.209 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 0> 2024-03-24T00:34:04.136+0000 7f172fd08700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:34:04.209 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: in thread 7f172fd08700 thread_name:ms_dispatch 2024-03-24T00:34:04.209 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 2024-03-24T00:34:04.209 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:34:04.209 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f17390b9d20] 2024-03-24T00:34:04.209 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 2: gsignal() 2024-03-24T00:34:04.209 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 3: abort() 2024-03-24T00:34:04.209 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f173a3324fb] 2024-03-24T00:34:04.210 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f173a332667] 2024-03-24T00:34:04.210 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556968cec223] 2024-03-24T00:34:04.210 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556968b665d6] 2024-03-24T00:34:04.210 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556968b66da1] 2024-03-24T00:34:04.210 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x556968b692d5] 2024-03-24T00:34:04.210 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556968bb9233] 2024-03-24T00:34:04.210 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556968bbb197] 2024-03-24T00:34:04.210 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556968aba11b] 2024-03-24T00:34:04.210 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556968aba6bc] 2024-03-24T00:34:04.210 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556968aa41ff] 2024-03-24T00:34:04.210 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f173a5c4488] 2024-03-24T00:34:04.210 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 16: (DispatchQueue::entry()+0x51f) [0x7f173a5c160f] 2024-03-24T00:34:04.210 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f173a694e81] 2024-03-24T00:34:04.211 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f17390af1ca] 2024-03-24T00:34:04.211 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 19: clone() 2024-03-24T00:34:04.211 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:34:04.211 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90476]: 2024-03-24T00:34:04.211 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 podman[90657]: 2024-03-24 00:34:04.19671297 +0000 UTC m=+0.023762715 container died 887af7276f3425c49fcb5cd15c01bca73168952c68e340503a0bf6ed198ce275 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-24T00:34:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:03 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:34:04.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:03 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6836/878237454,v1:172.21.15.78:6837/878237454] up:reconnect 2024-03-24T00:34:04.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:03 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:04.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:03 smithi107 ceph-mon[36485]: pgmap v446: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:34:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:03 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:34:04.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:03 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6836/878237454,v1:172.21.15.78:6837/878237454] up:reconnect 2024-03-24T00:34:04.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:03 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:04.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:03 smithi106 ceph-mon[36505]: pgmap v446: 129 pgs: 129 active+clean; 27 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:34:04.716 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 573 sec latency 686307.003 ms 2024-03-24T00:34:05.300 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 podman[90657]: 2024-03-24 00:34:04.898034754 +0000 UTC m=+0.725084499 container remove 887af7276f3425c49fcb5cd15c01bca73168952c68e340503a0bf6ed198ce275 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:34:05.300 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:04 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:34:05.300 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:05 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.d.service: Failed with result 'exit-code'. 2024-03-24T00:34:05.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6836/878237454,v1:172.21.15.78:6837/878237454] up:rejoin 2024-03-24T00:34:05.301 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:05.301 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-mon[32429]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-24T00:34:05.301 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:34:05.301 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6836/878237454,v1:172.21.15.78:6837/878237454] up:active 2024-03-24T00:34:05.301 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:04 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=h=up:active} 3 up:standby-replay 2024-03-24T00:34:05.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:04 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6836/878237454,v1:172.21.15.78:6837/878237454] up:rejoin 2024-03-24T00:34:05.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:04 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:05.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:04 smithi107 ceph-mon[36485]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-24T00:34:05.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:04 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:34:05.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:04 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6836/878237454,v1:172.21.15.78:6837/878237454] up:active 2024-03-24T00:34:05.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:04 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=h=up:active} 3 up:standby-replay 2024-03-24T00:34:05.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:04 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6836/878237454,v1:172.21.15.78:6837/878237454] up:rejoin 2024-03-24T00:34:05.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:04 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:05.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:04 smithi106 ceph-mon[36505]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-24T00:34:05.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:04 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:34:05.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:04 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6836/878237454,v1:172.21.15.78:6837/878237454] up:active 2024-03-24T00:34:05.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:04 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=h=up:active} 3 up:standby-replay 2024-03-24T00:34:05.716 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 574 sec latency 687307.060 ms 2024-03-24T00:34:06.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:05 smithi078 ceph-mon[32429]: pgmap v447: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 29 op/s 2024-03-24T00:34:06.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:05 smithi107 ceph-mon[36485]: pgmap v447: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 29 op/s 2024-03-24T00:34:06.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:05 smithi106 ceph-mon[36505]: pgmap v447: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 29 op/s 2024-03-24T00:34:06.716 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 575 sec latency 688307.187 ms 2024-03-24T00:34:07.716 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 576 sec latency 689307.281 ms 2024-03-24T00:34:08.716 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 577 sec latency 690307.373 ms 2024-03-24T00:34:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:08 smithi078 ceph-mon[32429]: pgmap v448: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 29 op/s 2024-03-24T00:34:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:08 smithi107 ceph-mon[36485]: pgmap v448: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 29 op/s 2024-03-24T00:34:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:08 smithi106 ceph-mon[36505]: pgmap v448: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 29 op/s 2024-03-24T00:34:09.716 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 578 sec latency 691307.409 ms 2024-03-24T00:34:10.716 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 579 sec latency 692307.525 ms 2024-03-24T00:34:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:10 smithi078 ceph-mon[32429]: pgmap v449: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 29 op/s 2024-03-24T00:34:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:10 smithi107 ceph-mon[36485]: pgmap v449: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 29 op/s 2024-03-24T00:34:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:10 smithi106 ceph-mon[36505]: pgmap v449: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 29 op/s 2024-03-24T00:34:11.717 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 580 sec latency 693307.630 ms 2024-03-24T00:34:12.717 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 581 sec latency 694307.749 ms 2024-03-24T00:34:12.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:12 smithi078 ceph-mon[32429]: pgmap v450: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 KiB/s wr, 26 op/s 2024-03-24T00:34:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:12 smithi107 ceph-mon[36485]: pgmap v450: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 KiB/s wr, 26 op/s 2024-03-24T00:34:12.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:12 smithi106 ceph-mon[36505]: pgmap v450: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 KiB/s wr, 26 op/s 2024-03-24T00:34:13.717 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 582 sec latency 695307.856 ms 2024-03-24T00:34:14.717 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 583 sec latency 696307.970 ms 2024-03-24T00:34:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:14 smithi078 ceph-mon[32429]: pgmap v451: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T00:34:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:14 smithi107 ceph-mon[36485]: pgmap v451: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T00:34:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:14 smithi106 ceph-mon[36505]: pgmap v451: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T00:34:15.693 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:15 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.d.service: Scheduled restart job, restart counter is at 5. 2024-03-24T00:34:15.693 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:15 smithi078 systemd[1]: Stopped Ceph mds.d for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:34:15.693 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:15 smithi078 systemd[1]: Starting Ceph mds.d for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:34:15.693 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:15 smithi078 podman[90756]: 2024-03-24 00:34:15.58210642 +0000 UTC m=+0.114602504 container create 96b8892d80223db1b744169ef1bc6c44d3c4fa5c7ce3b9b55f0c78d888fda473 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-24T00:34:15.693 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:15 smithi078 podman[90756]: 2024-03-24 00:34:15.485695279 +0000 UTC m=+0.018191333 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:34:15.717 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 584 sec latency 697308.012 ms 2024-03-24T00:34:16.090 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:15 smithi078 podman[90756]: 2024-03-24 00:34:15.81785503 +0000 UTC m=+0.350351085 container init 96b8892d80223db1b744169ef1bc6c44d3c4fa5c7ce3b9b55f0c78d888fda473 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True) 2024-03-24T00:34:16.091 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:15 smithi078 podman[90756]: 2024-03-24 00:34:15.820269244 +0000 UTC m=+0.352765291 container start 96b8892d80223db1b744169ef1bc6c44d3c4fa5c7ce3b9b55f0c78d888fda473 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:34:16.091 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:15 smithi078 ceph-mds[90775]: -- 172.21.15.78:0/475131657 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562860ad3a40 con 0x562860aa6800 2024-03-24T00:34:16.091 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:15 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: starting mds.d at 2024-03-24T00:34:16.091 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:15 smithi078 bash[90756]: 96b8892d80223db1b744169ef1bc6c44d3c4fa5c7ce3b9b55f0c78d888fda473 2024-03-24T00:34:16.091 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:15 smithi078 systemd[1]: Started Ceph mds.d for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:34:16.717 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 585 sec latency 698308.039 ms 2024-03-24T00:34:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:16 smithi078 ceph-mon[32429]: pgmap v452: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-03-24T00:34:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:16 smithi078 ceph-mon[32429]: daemon mds.d restarted 2024-03-24T00:34:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:16 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:34:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:16 smithi078 ceph-mon[32429]: osdmap e148: 12 total, 12 up, 12 in 2024-03-24T00:34:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:16 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6836/2612396245,v1:172.21.15.78:6837/2612396245] up:boot 2024-03-24T00:34:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:16 smithi078 ceph-mon[32429]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-03-24T00:34:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:16 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:34:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:34:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:16 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:16 smithi107 ceph-mon[36485]: pgmap v452: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-03-24T00:34:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:16 smithi107 ceph-mon[36485]: daemon mds.d restarted 2024-03-24T00:34:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:16 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:34:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:16 smithi107 ceph-mon[36485]: osdmap e148: 12 total, 12 up, 12 in 2024-03-24T00:34:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:16 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6836/2612396245,v1:172.21.15.78:6837/2612396245] up:boot 2024-03-24T00:34:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:16 smithi107 ceph-mon[36485]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-03-24T00:34:16.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:16 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:34:16.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:34:16.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:16 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:16.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:16 smithi106 ceph-mon[36505]: pgmap v452: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-03-24T00:34:16.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:16 smithi106 ceph-mon[36505]: daemon mds.d restarted 2024-03-24T00:34:16.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:16 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:34:16.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:16 smithi106 ceph-mon[36505]: osdmap e148: 12 total, 12 up, 12 in 2024-03-24T00:34:16.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:16 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6836/2612396245,v1:172.21.15.78:6837/2612396245] up:boot 2024-03-24T00:34:16.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:16 smithi106 ceph-mon[36505]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-03-24T00:34:16.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:16 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:34:16.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:34:16.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:16 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:17.717 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 586 sec latency 699308.072 ms 2024-03-24T00:34:18.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:17 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6834/93340378,v1:172.21.15.78:6835/93340378] up:resolve 2024-03-24T00:34:18.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:17 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:18.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:17 smithi106 ceph-mon[36505]: pgmap v454: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:34:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:17 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6834/93340378,v1:172.21.15.78:6835/93340378] up:resolve 2024-03-24T00:34:18.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:17 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:18.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:17 smithi078 ceph-mon[32429]: pgmap v454: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:34:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:17 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6834/93340378,v1:172.21.15.78:6835/93340378] up:resolve 2024-03-24T00:34:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:17 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:17 smithi107 ceph-mon[36485]: pgmap v454: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:34:18.717 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 587 sec latency 700308.184 ms 2024-03-24T00:34:19.137 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:18 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999981 2024-03-24T00:34:19.138 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:18 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6834/93340378,v1:172.21.15.78:6835/93340378] up:reconnect 2024-03-24T00:34:19.138 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:18 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:18 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999981 2024-03-24T00:34:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:18 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6834/93340378,v1:172.21.15.78:6835/93340378] up:reconnect 2024-03-24T00:34:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:18 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:18 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999981 2024-03-24T00:34:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:18 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6834/93340378,v1:172.21.15.78:6835/93340378] up:reconnect 2024-03-24T00:34:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:18 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:19.496 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f0f274d4700 time 2024-03-24T00:34:19.138363+0000 2024-03-24T00:34:19.496 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:34:19.496 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:34:19.497 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f0f31afe4a1] 2024-03-24T00:34:19.497 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0f31afe667] 2024-03-24T00:34:19.497 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5626d110d223] 2024-03-24T00:34:19.497 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5626d0f875d6] 2024-03-24T00:34:19.497 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5626d0f87da1] 2024-03-24T00:34:19.497 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5626d0f8a2d5] 2024-03-24T00:34:19.497 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5626d0fda233] 2024-03-24T00:34:19.497 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5626d0fdc197] 2024-03-24T00:34:19.497 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5626d0edb11b] 2024-03-24T00:34:19.497 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5626d0edb6bc] 2024-03-24T00:34:19.497 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5626d0ec51ff] 2024-03-24T00:34:19.497 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0f31d90488] 2024-03-24T00:34:19.498 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 13: (DispatchQueue::entry()+0x51f) [0x7f0f31d8d60f] 2024-03-24T00:34:19.498 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0f31e60e81] 2024-03-24T00:34:19.498 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0f3087b1ca] 2024-03-24T00:34:19.498 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 16: clone() 2024-03-24T00:34:19.499 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 2024-03-24T00:34:19.139+0000 7f0f274d4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f0f274d4700 time 2024-03-24T00:34:19.138363+0000 2024-03-24T00:34:19.499 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:34:19.499 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 2024-03-24T00:34:19.499 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:34:19.499 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f0f31afe4a1] 2024-03-24T00:34:19.499 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0f31afe667] 2024-03-24T00:34:19.499 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5626d110d223] 2024-03-24T00:34:19.499 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5626d0f875d6] 2024-03-24T00:34:19.499 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5626d0f87da1] 2024-03-24T00:34:19.499 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5626d0f8a2d5] 2024-03-24T00:34:19.500 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5626d0fda233] 2024-03-24T00:34:19.500 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5626d0fdc197] 2024-03-24T00:34:19.500 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5626d0edb11b] 2024-03-24T00:34:19.500 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5626d0edb6bc] 2024-03-24T00:34:19.500 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5626d0ec51ff] 2024-03-24T00:34:19.500 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0f31d90488] 2024-03-24T00:34:19.500 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 13: (DispatchQueue::entry()+0x51f) [0x7f0f31d8d60f] 2024-03-24T00:34:19.500 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0f31e60e81] 2024-03-24T00:34:19.500 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0f3087b1ca] 2024-03-24T00:34:19.500 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 16: clone() 2024-03-24T00:34:19.500 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 2024-03-24T00:34:19.500 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: *** Caught signal (Aborted) ** 2024-03-24T00:34:19.500 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: in thread 7f0f274d4700 thread_name:ms_dispatch 2024-03-24T00:34:19.501 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:34:19.501 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0f30885d20] 2024-03-24T00:34:19.501 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 2: gsignal() 2024-03-24T00:34:19.501 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 3: abort() 2024-03-24T00:34:19.501 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0f31afe4fb] 2024-03-24T00:34:19.501 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0f31afe667] 2024-03-24T00:34:19.501 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5626d110d223] 2024-03-24T00:34:19.501 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5626d0f875d6] 2024-03-24T00:34:19.501 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5626d0f87da1] 2024-03-24T00:34:19.501 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5626d0f8a2d5] 2024-03-24T00:34:19.501 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5626d0fda233] 2024-03-24T00:34:19.501 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5626d0fdc197] 2024-03-24T00:34:19.501 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5626d0edb11b] 2024-03-24T00:34:19.501 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5626d0edb6bc] 2024-03-24T00:34:19.501 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5626d0ec51ff] 2024-03-24T00:34:19.501 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0f31d90488] 2024-03-24T00:34:19.502 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 16: (DispatchQueue::entry()+0x51f) [0x7f0f31d8d60f] 2024-03-24T00:34:19.502 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0f31e60e81] 2024-03-24T00:34:19.502 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0f3087b1ca] 2024-03-24T00:34:19.502 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 19: clone() 2024-03-24T00:34:19.502 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 2024-03-24T00:34:19.142+0000 7f0f274d4700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:34:19.502 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: in thread 7f0f274d4700 thread_name:ms_dispatch 2024-03-24T00:34:19.502 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 2024-03-24T00:34:19.502 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:34:19.502 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0f30885d20] 2024-03-24T00:34:19.502 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 2: gsignal() 2024-03-24T00:34:19.502 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 3: abort() 2024-03-24T00:34:19.502 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0f31afe4fb] 2024-03-24T00:34:19.502 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0f31afe667] 2024-03-24T00:34:19.502 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5626d110d223] 2024-03-24T00:34:19.503 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5626d0f875d6] 2024-03-24T00:34:19.503 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5626d0f87da1] 2024-03-24T00:34:19.503 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5626d0f8a2d5] 2024-03-24T00:34:19.503 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5626d0fda233] 2024-03-24T00:34:19.503 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5626d0fdc197] 2024-03-24T00:34:19.503 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5626d0edb11b] 2024-03-24T00:34:19.503 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5626d0edb6bc] 2024-03-24T00:34:19.503 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5626d0ec51ff] 2024-03-24T00:34:19.503 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0f31d90488] 2024-03-24T00:34:19.503 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 16: (DispatchQueue::entry()+0x51f) [0x7f0f31d8d60f] 2024-03-24T00:34:19.503 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0f31e60e81] 2024-03-24T00:34:19.503 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0f3087b1ca] 2024-03-24T00:34:19.503 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 19: clone() 2024-03-24T00:34:19.503 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:34:19.504 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 2024-03-24T00:34:19.504 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: -1> 2024-03-24T00:34:19.139+0000 7f0f274d4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f0f274d4700 time 2024-03-24T00:34:19.138363+0000 2024-03-24T00:34:19.504 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:34:19.504 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 2024-03-24T00:34:19.504 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:34:19.504 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f0f31afe4a1] 2024-03-24T00:34:19.504 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0f31afe667] 2024-03-24T00:34:19.504 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5626d110d223] 2024-03-24T00:34:19.504 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5626d0f875d6] 2024-03-24T00:34:19.504 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5626d0f87da1] 2024-03-24T00:34:19.504 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5626d0f8a2d5] 2024-03-24T00:34:19.504 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5626d0fda233] 2024-03-24T00:34:19.504 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5626d0fdc197] 2024-03-24T00:34:19.504 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5626d0edb11b] 2024-03-24T00:34:19.504 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5626d0edb6bc] 2024-03-24T00:34:19.505 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5626d0ec51ff] 2024-03-24T00:34:19.505 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0f31d90488] 2024-03-24T00:34:19.505 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 13: (DispatchQueue::entry()+0x51f) [0x7f0f31d8d60f] 2024-03-24T00:34:19.505 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0f31e60e81] 2024-03-24T00:34:19.505 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0f3087b1ca] 2024-03-24T00:34:19.505 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 16: clone() 2024-03-24T00:34:19.505 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 2024-03-24T00:34:19.505 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 0> 2024-03-24T00:34:19.142+0000 7f0f274d4700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:34:19.505 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: in thread 7f0f274d4700 thread_name:ms_dispatch 2024-03-24T00:34:19.505 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 2024-03-24T00:34:19.505 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:34:19.505 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0f30885d20] 2024-03-24T00:34:19.505 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 2: gsignal() 2024-03-24T00:34:19.505 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 3: abort() 2024-03-24T00:34:19.506 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0f31afe4fb] 2024-03-24T00:34:19.506 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0f31afe667] 2024-03-24T00:34:19.506 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5626d110d223] 2024-03-24T00:34:19.506 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5626d0f875d6] 2024-03-24T00:34:19.506 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5626d0f87da1] 2024-03-24T00:34:19.506 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5626d0f8a2d5] 2024-03-24T00:34:19.506 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5626d0fda233] 2024-03-24T00:34:19.506 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5626d0fdc197] 2024-03-24T00:34:19.506 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5626d0edb11b] 2024-03-24T00:34:19.506 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5626d0edb6bc] 2024-03-24T00:34:19.506 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5626d0ec51ff] 2024-03-24T00:34:19.506 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0f31d90488] 2024-03-24T00:34:19.506 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 16: (DispatchQueue::entry()+0x51f) [0x7f0f31d8d60f] 2024-03-24T00:34:19.507 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0f31e60e81] 2024-03-24T00:34:19.507 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0f3087b1ca] 2024-03-24T00:34:19.507 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 19: clone() 2024-03-24T00:34:19.507 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:34:19.507 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 2024-03-24T00:34:19.507 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: -1> 2024-03-24T00:34:19.139+0000 7f0f274d4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f0f274d4700 time 2024-03-24T00:34:19.138363+0000 2024-03-24T00:34:19.507 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:34:19.507 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 2024-03-24T00:34:19.507 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:34:19.507 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f0f31afe4a1] 2024-03-24T00:34:19.507 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0f31afe667] 2024-03-24T00:34:19.508 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5626d110d223] 2024-03-24T00:34:19.508 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5626d0f875d6] 2024-03-24T00:34:19.508 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5626d0f87da1] 2024-03-24T00:34:19.508 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5626d0f8a2d5] 2024-03-24T00:34:19.508 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5626d0fda233] 2024-03-24T00:34:19.508 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5626d0fdc197] 2024-03-24T00:34:19.508 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5626d0edb11b] 2024-03-24T00:34:19.509 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5626d0edb6bc] 2024-03-24T00:34:19.509 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5626d0ec51ff] 2024-03-24T00:34:19.509 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0f31d90488] 2024-03-24T00:34:19.509 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 13: (DispatchQueue::entry()+0x51f) [0x7f0f31d8d60f] 2024-03-24T00:34:19.509 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0f31e60e81] 2024-03-24T00:34:19.509 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0f3087b1ca] 2024-03-24T00:34:19.509 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 16: clone() 2024-03-24T00:34:19.509 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 2024-03-24T00:34:19.509 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 0> 2024-03-24T00:34:19.142+0000 7f0f274d4700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:34:19.509 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: in thread 7f0f274d4700 thread_name:ms_dispatch 2024-03-24T00:34:19.509 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 2024-03-24T00:34:19.509 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:34:19.509 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0f30885d20] 2024-03-24T00:34:19.509 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 2: gsignal() 2024-03-24T00:34:19.509 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 3: abort() 2024-03-24T00:34:19.510 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0f31afe4fb] 2024-03-24T00:34:19.510 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f0f31afe667] 2024-03-24T00:34:19.510 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5626d110d223] 2024-03-24T00:34:19.510 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5626d0f875d6] 2024-03-24T00:34:19.510 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5626d0f87da1] 2024-03-24T00:34:19.510 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x5626d0f8a2d5] 2024-03-24T00:34:19.510 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5626d0fda233] 2024-03-24T00:34:19.510 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5626d0fdc197] 2024-03-24T00:34:19.510 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5626d0edb11b] 2024-03-24T00:34:19.510 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5626d0edb6bc] 2024-03-24T00:34:19.510 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5626d0ec51ff] 2024-03-24T00:34:19.510 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0f31d90488] 2024-03-24T00:34:19.510 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 16: (DispatchQueue::entry()+0x51f) [0x7f0f31d8d60f] 2024-03-24T00:34:19.511 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0f31e60e81] 2024-03-24T00:34:19.511 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0f3087b1ca] 2024-03-24T00:34:19.511 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 19: clone() 2024-03-24T00:34:19.511 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:34:19.511 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[90619]: 2024-03-24T00:34:19.511 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 podman[90809]: 2024-03-24 00:34:19.188190022 +0000 UTC m=+0.019450029 container died d3e87c5f3abb4eb0a6adc0d1865bf63521eb37f9655a044c59012e3c02f9b2d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:34:19.717 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 588 sec latency 701308.241 ms 2024-03-24T00:34:20.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:19 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6834/93340378,v1:172.21.15.78:6835/93340378] up:rejoin 2024-03-24T00:34:20.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:19 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:20.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:19 smithi106 ceph-mon[36505]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-24T00:34:20.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:19 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:34:20.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:19 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6834/93340378,v1:172.21.15.78:6835/93340378] up:active 2024-03-24T00:34:20.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:19 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=h=up:active} 3 up:standby-replay 2024-03-24T00:34:20.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:19 smithi106 ceph-mon[36505]: pgmap v455: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:34:20.176 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 podman[90809]: 2024-03-24 00:34:19.949048764 +0000 UTC m=+0.780308758 container remove d3e87c5f3abb4eb0a6adc0d1865bf63521eb37f9655a044c59012e3c02f9b2d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-24T00:34:20.176 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:19 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:34:20.177 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6834/93340378,v1:172.21.15.78:6835/93340378] up:rejoin 2024-03-24T00:34:20.177 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:20.177 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-mon[32429]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-24T00:34:20.177 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:34:20.177 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6834/93340378,v1:172.21.15.78:6835/93340378] up:active 2024-03-24T00:34:20.177 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=h=up:active} 3 up:standby-replay 2024-03-24T00:34:20.177 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:19 smithi078 ceph-mon[32429]: pgmap v455: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:34:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:19 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6834/93340378,v1:172.21.15.78:6835/93340378] up:rejoin 2024-03-24T00:34:20.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:19 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:20.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:19 smithi107 ceph-mon[36485]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-24T00:34:20.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:19 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:34:20.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:19 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6834/93340378,v1:172.21.15.78:6835/93340378] up:active 2024-03-24T00:34:20.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:19 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=h=up:active} 3 up:standby-replay 2024-03-24T00:34:20.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:19 smithi107 ceph-mon[36485]: pgmap v455: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T00:34:20.590 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:20 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.j.service: Failed with result 'exit-code'. 2024-03-24T00:34:20.717 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 589 sec latency 702308.356 ms 2024-03-24T00:34:21.717 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 590 sec latency 703308.441 ms 2024-03-24T00:34:22.717 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 591 sec latency 704308.566 ms 2024-03-24T00:34:22.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:22 smithi078 ceph-mon[32429]: pgmap v456: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:34:22.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:22 smithi107 ceph-mon[36485]: pgmap v456: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:34:22.854 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:22 smithi106 ceph-mon[36505]: pgmap v456: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:34:23.718 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 592 sec latency 705308.670 ms 2024-03-24T00:34:24.718 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 593 sec latency 706308.797 ms 2024-03-24T00:34:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:24 smithi078 ceph-mon[32429]: pgmap v457: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:34:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:24 smithi107 ceph-mon[36485]: pgmap v457: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:34:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:24 smithi106 ceph-mon[36505]: pgmap v457: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:34:25.718 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 594 sec latency 707308.845 ms 2024-03-24T00:34:26.718 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 595 sec latency 708308.958 ms 2024-03-24T00:34:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:26 smithi078 ceph-mon[32429]: pgmap v458: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:34:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:26 smithi107 ceph-mon[36485]: pgmap v458: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:34:26.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:26 smithi106 ceph-mon[36505]: pgmap v458: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-03-24T00:34:27.718 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 596 sec latency 709308.989 ms 2024-03-24T00:34:28.718 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 597 sec latency 710309.066 ms 2024-03-24T00:34:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:28 smithi078 ceph-mon[32429]: pgmap v459: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 23 KiB/s wr, 12 op/s 2024-03-24T00:34:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:28 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:34:28.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:28 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:28.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:28 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:34:28.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:28 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:34:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:28 smithi107 ceph-mon[36485]: pgmap v459: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 23 KiB/s wr, 12 op/s 2024-03-24T00:34:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:28 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:34:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:28 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:28 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:34:28.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:28 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:34:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:28 smithi106 ceph-mon[36505]: pgmap v459: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 23 KiB/s wr, 12 op/s 2024-03-24T00:34:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:28 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:34:28.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:28 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:28.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:28 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:34:28.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:28 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:34:29.718 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 598 sec latency 711309.163 ms 2024-03-24T00:34:29.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:29 smithi078 ceph-mon[32429]: pgmap v460: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 25 KiB/s wr, 13 op/s 2024-03-24T00:34:29.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:29 smithi078 ceph-mon[32429]: osdmap e149: 12 total, 12 up, 12 in 2024-03-24T00:34:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:29 smithi107 ceph-mon[36485]: pgmap v460: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 25 KiB/s wr, 13 op/s 2024-03-24T00:34:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:29 smithi107 ceph-mon[36485]: osdmap e149: 12 total, 12 up, 12 in 2024-03-24T00:34:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:29 smithi106 ceph-mon[36505]: pgmap v460: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 25 KiB/s wr, 13 op/s 2024-03-24T00:34:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:29 smithi106 ceph-mon[36505]: osdmap e149: 12 total, 12 up, 12 in 2024-03-24T00:34:30.616 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:30 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.j.service: Scheduled restart job, restart counter is at 3. 2024-03-24T00:34:30.616 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:30 smithi078 systemd[1]: Stopped Ceph mds.j for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:34:30.616 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:30 smithi078 systemd[1]: Starting Ceph mds.j for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:34:30.616 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:30 smithi078 podman[91034]: 2024-03-24 00:34:30.48428399 +0000 UTC m=+0.018519088 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:34:30.616 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:30 smithi078 podman[91034]: 2024-03-24 00:34:30.616320171 +0000 UTC m=+0.150555244 container create 9019ad3457304a74c4e4cbcd3e003e9f6c101b891e5f9bd5c5521457bd2130e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-24T00:34:30.718 INFO:tasks.workunit.client.0.smithi078.stdout: 1 15198 0.00 MB/sec execute 599 sec latency 712309.211 ms 2024-03-24T00:34:30.877 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:30 smithi078 podman[91034]: 2024-03-24 00:34:30.852800916 +0000 UTC m=+0.387036006 container init 9019ad3457304a74c4e4cbcd3e003e9f6c101b891e5f9bd5c5521457bd2130e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-24T00:34:30.877 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:30 smithi078 podman[91034]: 2024-03-24 00:34:30.855404045 +0000 UTC m=+0.389639130 container start 9019ad3457304a74c4e4cbcd3e003e9f6c101b891e5f9bd5c5521457bd2130e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-24T00:34:31.340 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:30 smithi078 ceph-mds[91054]: -- 172.21.15.78:0/2271731795 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f1cd20da40 con 0x55f1cd1e0800 2024-03-24T00:34:31.340 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:30 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: starting mds.j at 2024-03-24T00:34:31.341 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:30 smithi078 bash[91034]: 9019ad3457304a74c4e4cbcd3e003e9f6c101b891e5f9bd5c5521457bd2130e5 2024-03-24T00:34:31.341 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:30 smithi078 systemd[1]: Started Ceph mds.j for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:34:31.718 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 600 sec 2024-03-24T00:34:31.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:31 smithi078 ceph-mon[32429]: pgmap v462: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 116 B/s wr, 3 op/s 2024-03-24T00:34:31.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:31 smithi078 ceph-mon[32429]: daemon mds.j restarted 2024-03-24T00:34:31.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:31 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:34:31.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:31 smithi078 ceph-mon[32429]: osdmap e150: 12 total, 12 up, 12 in 2024-03-24T00:34:31.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:31 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6834/3414536872,v1:172.21.15.78:6835/3414536872] up:boot 2024-03-24T00:34:31.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:31 smithi078 ceph-mon[32429]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-24T00:34:31.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:31 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:34:31.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:31 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:34:31.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:31 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:31.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:31 smithi107 ceph-mon[36485]: pgmap v462: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 116 B/s wr, 3 op/s 2024-03-24T00:34:31.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:31 smithi107 ceph-mon[36485]: daemon mds.j restarted 2024-03-24T00:34:31.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:31 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:34:31.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:31 smithi107 ceph-mon[36485]: osdmap e150: 12 total, 12 up, 12 in 2024-03-24T00:34:31.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:31 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6834/3414536872,v1:172.21.15.78:6835/3414536872] up:boot 2024-03-24T00:34:31.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:31 smithi107 ceph-mon[36485]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-24T00:34:31.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:31 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:34:31.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:31 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:34:31.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:31 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:31.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:31 smithi106 ceph-mon[36505]: pgmap v462: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 116 B/s wr, 3 op/s 2024-03-24T00:34:31.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:31 smithi106 ceph-mon[36505]: daemon mds.j restarted 2024-03-24T00:34:31.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:31 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:34:31.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:31 smithi106 ceph-mon[36505]: osdmap e150: 12 total, 12 up, 12 in 2024-03-24T00:34:31.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:31 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6834/3414536872,v1:172.21.15.78:6835/3414536872] up:boot 2024-03-24T00:34:31.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:31 smithi106 ceph-mon[36505]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-24T00:34:31.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:31 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:34:31.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:31 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:34:31.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:31 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:replay,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:32.718 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 601 sec 2024-03-24T00:34:33.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:32 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6836/2612396245,v1:172.21.15.78:6837/2612396245] up:resolve 2024-03-24T00:34:33.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:32 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:33.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:32 smithi078 ceph-mon[32429]: pgmap v464: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 151 B/s wr, 4 op/s 2024-03-24T00:34:33.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:32 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6836/2612396245,v1:172.21.15.78:6837/2612396245] up:resolve 2024-03-24T00:34:33.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:32 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:33.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:32 smithi107 ceph-mon[36485]: pgmap v464: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 151 B/s wr, 4 op/s 2024-03-24T00:34:33.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:32 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6836/2612396245,v1:172.21.15.78:6837/2612396245] up:resolve 2024-03-24T00:34:33.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:32 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:resolve,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:33.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:32 smithi106 ceph-mon[36505]: pgmap v464: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 151 B/s wr, 4 op/s 2024-03-24T00:34:33.718 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 602 sec 2024-03-24T00:34:34.163 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:33 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:34:34.164 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:33 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6836/2612396245,v1:172.21.15.78:6837/2612396245] up:reconnect 2024-03-24T00:34:34.164 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:33 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:34.164 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f68a938c700 time 2024-03-24T00:34:34.158639+0000 2024-03-24T00:34:34.164 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:34:34.164 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:34:34.164 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f68b39b64a1] 2024-03-24T00:34:34.164 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f68b39b6667] 2024-03-24T00:34:34.165 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56285ed95223] 2024-03-24T00:34:34.165 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56285ec0f5d6] 2024-03-24T00:34:34.165 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56285ec0fda1] 2024-03-24T00:34:34.165 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56285ec122d5] 2024-03-24T00:34:34.165 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56285ec62233] 2024-03-24T00:34:34.165 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56285ec64197] 2024-03-24T00:34:34.165 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56285eb6311b] 2024-03-24T00:34:34.165 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56285eb636bc] 2024-03-24T00:34:34.165 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56285eb4d1ff] 2024-03-24T00:34:34.165 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f68b3c48488] 2024-03-24T00:34:34.165 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 13: (DispatchQueue::entry()+0x51f) [0x7f68b3c4560f] 2024-03-24T00:34:34.165 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f68b3d18e81] 2024-03-24T00:34:34.166 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f68b27331ca] 2024-03-24T00:34:34.166 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 16: clone() 2024-03-24T00:34:34.166 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 2024-03-24T00:34:34.159+0000 7f68a938c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f68a938c700 time 2024-03-24T00:34:34.158639+0000 2024-03-24T00:34:34.166 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:34:34.166 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 2024-03-24T00:34:34.166 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:34:34.166 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f68b39b64a1] 2024-03-24T00:34:34.166 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f68b39b6667] 2024-03-24T00:34:34.166 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56285ed95223] 2024-03-24T00:34:34.166 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56285ec0f5d6] 2024-03-24T00:34:34.166 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56285ec0fda1] 2024-03-24T00:34:34.166 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56285ec122d5] 2024-03-24T00:34:34.166 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56285ec62233] 2024-03-24T00:34:34.166 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56285ec64197] 2024-03-24T00:34:34.167 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56285eb6311b] 2024-03-24T00:34:34.167 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56285eb636bc] 2024-03-24T00:34:34.167 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56285eb4d1ff] 2024-03-24T00:34:34.167 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f68b3c48488] 2024-03-24T00:34:34.167 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 13: (DispatchQueue::entry()+0x51f) [0x7f68b3c4560f] 2024-03-24T00:34:34.167 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f68b3d18e81] 2024-03-24T00:34:34.167 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f68b27331ca] 2024-03-24T00:34:34.167 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 16: clone() 2024-03-24T00:34:34.167 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 2024-03-24T00:34:34.167 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: *** Caught signal (Aborted) ** 2024-03-24T00:34:34.167 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: in thread 7f68a938c700 thread_name:ms_dispatch 2024-03-24T00:34:34.167 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:34:34.167 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f68b273dd20] 2024-03-24T00:34:34.168 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 2: gsignal() 2024-03-24T00:34:34.168 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 3: abort() 2024-03-24T00:34:34.168 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f68b39b64fb] 2024-03-24T00:34:34.168 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f68b39b6667] 2024-03-24T00:34:34.168 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56285ed95223] 2024-03-24T00:34:34.168 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56285ec0f5d6] 2024-03-24T00:34:34.168 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56285ec0fda1] 2024-03-24T00:34:34.168 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56285ec122d5] 2024-03-24T00:34:34.168 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56285ec62233] 2024-03-24T00:34:34.168 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56285ec64197] 2024-03-24T00:34:34.169 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56285eb6311b] 2024-03-24T00:34:34.169 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56285eb636bc] 2024-03-24T00:34:34.169 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56285eb4d1ff] 2024-03-24T00:34:34.169 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f68b3c48488] 2024-03-24T00:34:34.169 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 16: (DispatchQueue::entry()+0x51f) [0x7f68b3c4560f] 2024-03-24T00:34:34.169 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f68b3d18e81] 2024-03-24T00:34:34.169 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f68b27331ca] 2024-03-24T00:34:34.169 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 19: clone() 2024-03-24T00:34:34.169 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 2024-03-24T00:34:34.161+0000 7f68a938c700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:34:34.169 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: in thread 7f68a938c700 thread_name:ms_dispatch 2024-03-24T00:34:34.169 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 2024-03-24T00:34:34.169 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:34:34.169 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f68b273dd20] 2024-03-24T00:34:34.170 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 2: gsignal() 2024-03-24T00:34:34.170 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 3: abort() 2024-03-24T00:34:34.170 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f68b39b64fb] 2024-03-24T00:34:34.170 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f68b39b6667] 2024-03-24T00:34:34.170 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56285ed95223] 2024-03-24T00:34:34.170 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56285ec0f5d6] 2024-03-24T00:34:34.170 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56285ec0fda1] 2024-03-24T00:34:34.170 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56285ec122d5] 2024-03-24T00:34:34.170 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56285ec62233] 2024-03-24T00:34:34.170 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56285ec64197] 2024-03-24T00:34:34.170 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56285eb6311b] 2024-03-24T00:34:34.170 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56285eb636bc] 2024-03-24T00:34:34.170 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56285eb4d1ff] 2024-03-24T00:34:34.171 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f68b3c48488] 2024-03-24T00:34:34.171 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 16: (DispatchQueue::entry()+0x51f) [0x7f68b3c4560f] 2024-03-24T00:34:34.171 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f68b3d18e81] 2024-03-24T00:34:34.171 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f68b27331ca] 2024-03-24T00:34:34.171 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 19: clone() 2024-03-24T00:34:34.171 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:34:34.171 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 2024-03-24T00:34:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:33 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:34:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:33 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6836/2612396245,v1:172.21.15.78:6837/2612396245] up:reconnect 2024-03-24T00:34:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:33 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:33 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:34:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:33 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6836/2612396245,v1:172.21.15.78:6837/2612396245] up:reconnect 2024-03-24T00:34:34.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:33 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:reconnect,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:34.457 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: -1> 2024-03-24T00:34:34.159+0000 7f68a938c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f68a938c700 time 2024-03-24T00:34:34.158639+0000 2024-03-24T00:34:34.457 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:34:34.457 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 2024-03-24T00:34:34.457 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:34:34.457 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f68b39b64a1] 2024-03-24T00:34:34.457 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f68b39b6667] 2024-03-24T00:34:34.457 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56285ed95223] 2024-03-24T00:34:34.457 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56285ec0f5d6] 2024-03-24T00:34:34.457 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56285ec0fda1] 2024-03-24T00:34:34.458 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56285ec122d5] 2024-03-24T00:34:34.458 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56285ec62233] 2024-03-24T00:34:34.458 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56285ec64197] 2024-03-24T00:34:34.458 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56285eb6311b] 2024-03-24T00:34:34.458 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56285eb636bc] 2024-03-24T00:34:34.458 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56285eb4d1ff] 2024-03-24T00:34:34.458 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f68b3c48488] 2024-03-24T00:34:34.458 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 13: (DispatchQueue::entry()+0x51f) [0x7f68b3c4560f] 2024-03-24T00:34:34.458 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f68b3d18e81] 2024-03-24T00:34:34.458 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f68b27331ca] 2024-03-24T00:34:34.458 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 16: clone() 2024-03-24T00:34:34.458 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 2024-03-24T00:34:34.459 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 0> 2024-03-24T00:34:34.161+0000 7f68a938c700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:34:34.459 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: in thread 7f68a938c700 thread_name:ms_dispatch 2024-03-24T00:34:34.459 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 2024-03-24T00:34:34.460 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:34:34.460 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f68b273dd20] 2024-03-24T00:34:34.460 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 2: gsignal() 2024-03-24T00:34:34.460 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 3: abort() 2024-03-24T00:34:34.460 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f68b39b64fb] 2024-03-24T00:34:34.460 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f68b39b6667] 2024-03-24T00:34:34.460 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56285ed95223] 2024-03-24T00:34:34.460 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56285ec0f5d6] 2024-03-24T00:34:34.460 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56285ec0fda1] 2024-03-24T00:34:34.460 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56285ec122d5] 2024-03-24T00:34:34.460 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56285ec62233] 2024-03-24T00:34:34.460 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56285ec64197] 2024-03-24T00:34:34.460 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56285eb6311b] 2024-03-24T00:34:34.461 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56285eb636bc] 2024-03-24T00:34:34.461 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56285eb4d1ff] 2024-03-24T00:34:34.461 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f68b3c48488] 2024-03-24T00:34:34.461 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 16: (DispatchQueue::entry()+0x51f) [0x7f68b3c4560f] 2024-03-24T00:34:34.461 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f68b3d18e81] 2024-03-24T00:34:34.461 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f68b27331ca] 2024-03-24T00:34:34.461 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 19: clone() 2024-03-24T00:34:34.461 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:34:34.461 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 2024-03-24T00:34:34.461 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: -1> 2024-03-24T00:34:34.159+0000 7f68a938c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f68a938c700 time 2024-03-24T00:34:34.158639+0000 2024-03-24T00:34:34.461 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:34:34.461 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 2024-03-24T00:34:34.461 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:34:34.461 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f68b39b64a1] 2024-03-24T00:34:34.461 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f68b39b6667] 2024-03-24T00:34:34.462 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56285ed95223] 2024-03-24T00:34:34.462 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56285ec0f5d6] 2024-03-24T00:34:34.462 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56285ec0fda1] 2024-03-24T00:34:34.462 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56285ec122d5] 2024-03-24T00:34:34.462 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56285ec62233] 2024-03-24T00:34:34.462 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56285ec64197] 2024-03-24T00:34:34.462 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56285eb6311b] 2024-03-24T00:34:34.462 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56285eb636bc] 2024-03-24T00:34:34.462 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56285eb4d1ff] 2024-03-24T00:34:34.462 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f68b3c48488] 2024-03-24T00:34:34.462 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 13: (DispatchQueue::entry()+0x51f) [0x7f68b3c4560f] 2024-03-24T00:34:34.462 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f68b3d18e81] 2024-03-24T00:34:34.462 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f68b27331ca] 2024-03-24T00:34:34.462 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 16: clone() 2024-03-24T00:34:34.463 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 2024-03-24T00:34:34.463 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 0> 2024-03-24T00:34:34.161+0000 7f68a938c700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:34:34.463 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: in thread 7f68a938c700 thread_name:ms_dispatch 2024-03-24T00:34:34.463 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 2024-03-24T00:34:34.463 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:34:34.463 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f68b273dd20] 2024-03-24T00:34:34.463 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 2: gsignal() 2024-03-24T00:34:34.463 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 3: abort() 2024-03-24T00:34:34.463 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f68b39b64fb] 2024-03-24T00:34:34.463 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f68b39b6667] 2024-03-24T00:34:34.463 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56285ed95223] 2024-03-24T00:34:34.463 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56285ec0f5d6] 2024-03-24T00:34:34.463 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56285ec0fda1] 2024-03-24T00:34:34.463 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x56285ec122d5] 2024-03-24T00:34:34.464 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56285ec62233] 2024-03-24T00:34:34.464 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56285ec64197] 2024-03-24T00:34:34.464 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56285eb6311b] 2024-03-24T00:34:34.464 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56285eb636bc] 2024-03-24T00:34:34.464 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56285eb4d1ff] 2024-03-24T00:34:34.464 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f68b3c48488] 2024-03-24T00:34:34.464 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 16: (DispatchQueue::entry()+0x51f) [0x7f68b3c4560f] 2024-03-24T00:34:34.464 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f68b3d18e81] 2024-03-24T00:34:34.464 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f68b27331ca] 2024-03-24T00:34:34.464 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 19: clone() 2024-03-24T00:34:34.464 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:34:34.464 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d[90771]: 2024-03-24T00:34:34.464 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 podman[91089]: 2024-03-24 00:34:34.208401068 +0000 UTC m=+0.020216906 container died 96b8892d80223db1b744169ef1bc6c44d3c4fa5c7ce3b9b55f0c78d888fda473 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-24T00:34:34.718 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 603 sec 2024-03-24T00:34:35.090 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 podman[91089]: 2024-03-24 00:34:34.824338469 +0000 UTC m=+0.636154297 container remove 96b8892d80223db1b744169ef1bc6c44d3c4fa5c7ce3b9b55f0c78d888fda473 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-d, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-24T00:34:35.090 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:34 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:34:35.090 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:35 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.d.service: Failed with result 'exit-code'. 2024-03-24T00:34:35.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6836/2612396245,v1:172.21.15.78:6837/2612396245] up:rejoin 2024-03-24T00:34:35.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:35.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-mon[32429]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-24T00:34:35.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:34:35.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6836/2612396245,v1:172.21.15.78:6837/2612396245] up:active 2024-03-24T00:34:35.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=h=up:active} 3 up:standby-replay 2024-03-24T00:34:35.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-mon[32429]: pgmap v465: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 151 B/s wr, 4 op/s 2024-03-24T00:34:35.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:34 smithi078 ceph-mon[32429]: osdmap e151: 12 total, 12 up, 12 in 2024-03-24T00:34:35.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:34 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6836/2612396245,v1:172.21.15.78:6837/2612396245] up:rejoin 2024-03-24T00:34:35.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:34 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:35.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:34 smithi107 ceph-mon[36485]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-24T00:34:35.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:34 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:34:35.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:34 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6836/2612396245,v1:172.21.15.78:6837/2612396245] up:active 2024-03-24T00:34:35.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:34 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=h=up:active} 3 up:standby-replay 2024-03-24T00:34:35.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:34 smithi107 ceph-mon[36485]: pgmap v465: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 151 B/s wr, 4 op/s 2024-03-24T00:34:35.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:34 smithi107 ceph-mon[36485]: osdmap e151: 12 total, 12 up, 12 in 2024-03-24T00:34:35.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:34 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6836/2612396245,v1:172.21.15.78:6837/2612396245] up:rejoin 2024-03-24T00:34:35.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:34 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=d=up:rejoin,2=h=up:active} 2 up:standby-replay 1 up:standby 2024-03-24T00:34:35.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:34 smithi106 ceph-mon[36505]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-24T00:34:35.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:34 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:34:35.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:34 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6836/2612396245,v1:172.21.15.78:6837/2612396245] up:active 2024-03-24T00:34:35.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:34 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=d=up:active,2=h=up:active} 3 up:standby-replay 2024-03-24T00:34:35.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:34 smithi106 ceph-mon[36505]: pgmap v465: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 151 B/s wr, 4 op/s 2024-03-24T00:34:35.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:34 smithi106 ceph-mon[36505]: osdmap e151: 12 total, 12 up, 12 in 2024-03-24T00:34:35.718 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 604 sec 2024-03-24T00:34:36.718 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 605 sec 2024-03-24T00:34:37.556 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:37 smithi078 ceph-mon[32429]: pgmap v467: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 531 B/s wr, 14 op/s 2024-03-24T00:34:37.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:37 smithi107 ceph-mon[36485]: pgmap v467: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 531 B/s wr, 14 op/s 2024-03-24T00:34:37.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:37 smithi106 ceph-mon[36505]: pgmap v467: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 531 B/s wr, 14 op/s 2024-03-24T00:34:37.718 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 606 sec 2024-03-24T00:34:38.719 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 607 sec 2024-03-24T00:34:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:39 smithi078 ceph-mon[32429]: pgmap v468: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 511 B/s wr, 13 op/s 2024-03-24T00:34:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:39 smithi107 ceph-mon[36485]: pgmap v468: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 511 B/s wr, 13 op/s 2024-03-24T00:34:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:39 smithi106 ceph-mon[36505]: pgmap v468: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 511 B/s wr, 13 op/s 2024-03-24T00:34:39.719 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 608 sec 2024-03-24T00:34:40.719 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 609 sec 2024-03-24T00:34:41.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:41 smithi078 ceph-mon[32429]: pgmap v469: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 332 B/s wr, 8 op/s 2024-03-24T00:34:41.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:41 smithi107 ceph-mon[36485]: pgmap v469: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 332 B/s wr, 8 op/s 2024-03-24T00:34:41.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:41 smithi106 ceph-mon[36505]: pgmap v469: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 332 B/s wr, 8 op/s 2024-03-24T00:34:41.719 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 610 sec 2024-03-24T00:34:42.719 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 611 sec 2024-03-24T00:34:43.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:43 smithi078 ceph-mon[32429]: pgmap v470: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-24T00:34:43.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:43 smithi107 ceph-mon[36485]: pgmap v470: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-24T00:34:43.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:43 smithi106 ceph-mon[36505]: pgmap v470: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-24T00:34:43.719 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 612 sec 2024-03-24T00:34:44.719 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 613 sec 2024-03-24T00:34:45.198 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:45 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.d.service: Scheduled restart job, restart counter is at 6. 2024-03-24T00:34:45.199 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:45 smithi078 systemd[1]: Stopped Ceph mds.d for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:34:45.199 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:45 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.d.service: Start request repeated too quickly. 2024-03-24T00:34:45.199 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:45 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.d.service: Failed with result 'exit-code'. 2024-03-24T00:34:45.199 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 24 00:34:45 smithi078 systemd[1]: Failed to start Ceph mds.d for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:34:45.483 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:45 smithi078 ceph-mon[32429]: pgmap v471: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-24T00:34:45.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:45 smithi107 ceph-mon[36485]: pgmap v471: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-24T00:34:45.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:45 smithi106 ceph-mon[36505]: pgmap v471: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-24T00:34:45.719 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 614 sec 2024-03-24T00:34:46.719 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 615 sec 2024-03-24T00:34:47.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:47 smithi078 ceph-mon[32429]: pgmap v472: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T00:34:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:47 smithi107 ceph-mon[36485]: pgmap v472: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T00:34:47.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:47 smithi106 ceph-mon[36505]: pgmap v472: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T00:34:47.719 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 616 sec 2024-03-24T00:34:48.720 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 617 sec 2024-03-24T00:34:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:49 smithi078 ceph-mon[32429]: pgmap v473: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:34:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:49 smithi078 ceph-mon[32429]: Replacing daemon mds.d as rank 1 with standby daemon mds.j 2024-03-24T00:34:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:49 smithi078 ceph-mon[32429]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-03-24T00:34:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:49 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:34:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:49 smithi078 ceph-mon[32429]: osdmap e152: 12 total, 12 up, 12 in 2024-03-24T00:34:49.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:49 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=h=up:active} 2 up:standby-replay 2024-03-24T00:34:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:49 smithi107 ceph-mon[36485]: pgmap v473: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:34:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:49 smithi107 ceph-mon[36485]: Replacing daemon mds.d as rank 1 with standby daemon mds.j 2024-03-24T00:34:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:49 smithi107 ceph-mon[36485]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-03-24T00:34:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:49 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:34:49.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:49 smithi107 ceph-mon[36485]: osdmap e152: 12 total, 12 up, 12 in 2024-03-24T00:34:49.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:49 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=h=up:active} 2 up:standby-replay 2024-03-24T00:34:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:49 smithi106 ceph-mon[36505]: pgmap v473: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:34:49.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:49 smithi106 ceph-mon[36505]: Replacing daemon mds.d as rank 1 with standby daemon mds.j 2024-03-24T00:34:49.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:49 smithi106 ceph-mon[36505]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-03-24T00:34:49.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:49 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:34:49.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:49 smithi106 ceph-mon[36505]: osdmap e152: 12 total, 12 up, 12 in 2024-03-24T00:34:49.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:49 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=h=up:active} 2 up:standby-replay 2024-03-24T00:34:49.720 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 618 sec 2024-03-24T00:34:50.720 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 619 sec 2024-03-24T00:34:51.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:51 smithi106 ceph-mon[36505]: pgmap v475: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:34:51.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:51 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6834/3414536872,v1:172.21.15.78:6835/3414536872] up:resolve 2024-03-24T00:34:51.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:51 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=h=up:active} 2 up:standby-replay 2024-03-24T00:34:51.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:51 smithi078 ceph-mon[32429]: pgmap v475: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:34:51.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:51 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6834/3414536872,v1:172.21.15.78:6835/3414536872] up:resolve 2024-03-24T00:34:51.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:51 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=h=up:active} 2 up:standby-replay 2024-03-24T00:34:51.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:51 smithi107 ceph-mon[36485]: pgmap v475: 129 pgs: 129 active+clean; 27 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:34:51.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:51 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6834/3414536872,v1:172.21.15.78:6835/3414536872] up:resolve 2024-03-24T00:34:51.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:51 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=h=up:active} 2 up:standby-replay 2024-03-24T00:34:51.720 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 620 sec 2024-03-24T00:34:52.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:52 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999981 2024-03-24T00:34:52.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:52 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6834/3414536872,v1:172.21.15.78:6835/3414536872] up:reconnect 2024-03-24T00:34:52.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:52 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=h=up:active} 2 up:standby-replay 2024-03-24T00:34:52.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:52 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999981 2024-03-24T00:34:52.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:52 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6834/3414536872,v1:172.21.15.78:6835/3414536872] up:reconnect 2024-03-24T00:34:52.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:52 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=h=up:active} 2 up:standby-replay 2024-03-24T00:34:52.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:52 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999981 2024-03-24T00:34:52.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:52 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6834/3414536872,v1:172.21.15.78:6835/3414536872] up:reconnect 2024-03-24T00:34:52.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:52 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=h=up:active} 2 up:standby-replay 2024-03-24T00:34:52.720 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 621 sec 2024-03-24T00:34:53.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-mon[32429]: pgmap v476: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:53.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6834/3414536872,v1:172.21.15.78:6835/3414536872] up:rejoin 2024-03-24T00:34:53.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=h=up:active} 2 up:standby-replay 2024-03-24T00:34:53.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-mon[32429]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-24T00:34:53.543 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fbd71ef9700 time 2024-03-24T00:34:53.173123+0000 2024-03-24T00:34:53.544 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:34:53.544 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:34:53.544 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fbd7c5234a1] 2024-03-24T00:34:53.544 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fbd7c523667] 2024-03-24T00:34:53.544 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f1cc3e2223] 2024-03-24T00:34:53.544 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f1cc25c5d6] 2024-03-24T00:34:53.544 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f1cc25cda1] 2024-03-24T00:34:53.544 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f1cc25f2d5] 2024-03-24T00:34:53.544 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f1cc2af233] 2024-03-24T00:34:53.544 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f1cc2b1197] 2024-03-24T00:34:53.544 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f1cc1b011b] 2024-03-24T00:34:53.545 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f1cc1b06bc] 2024-03-24T00:34:53.545 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f1cc19a1ff] 2024-03-24T00:34:53.545 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fbd7c7b5488] 2024-03-24T00:34:53.545 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 13: (DispatchQueue::entry()+0x51f) [0x7fbd7c7b260f] 2024-03-24T00:34:53.545 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fbd7c885e81] 2024-03-24T00:34:53.545 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fbd7b2a01ca] 2024-03-24T00:34:53.545 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 16: clone() 2024-03-24T00:34:53.545 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 2024-03-24T00:34:53.173+0000 7fbd71ef9700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fbd71ef9700 time 2024-03-24T00:34:53.173123+0000 2024-03-24T00:34:53.545 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:34:53.545 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 2024-03-24T00:34:53.545 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:34:53.546 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fbd7c5234a1] 2024-03-24T00:34:53.546 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fbd7c523667] 2024-03-24T00:34:53.546 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f1cc3e2223] 2024-03-24T00:34:53.546 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f1cc25c5d6] 2024-03-24T00:34:53.546 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f1cc25cda1] 2024-03-24T00:34:53.546 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f1cc25f2d5] 2024-03-24T00:34:53.546 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f1cc2af233] 2024-03-24T00:34:53.546 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f1cc2b1197] 2024-03-24T00:34:53.546 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f1cc1b011b] 2024-03-24T00:34:53.546 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f1cc1b06bc] 2024-03-24T00:34:53.546 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f1cc19a1ff] 2024-03-24T00:34:53.546 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fbd7c7b5488] 2024-03-24T00:34:53.546 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 13: (DispatchQueue::entry()+0x51f) [0x7fbd7c7b260f] 2024-03-24T00:34:53.547 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fbd7c885e81] 2024-03-24T00:34:53.547 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fbd7b2a01ca] 2024-03-24T00:34:53.547 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 16: clone() 2024-03-24T00:34:53.547 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 2024-03-24T00:34:53.547 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: *** Caught signal (Aborted) ** 2024-03-24T00:34:53.547 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: in thread 7fbd71ef9700 thread_name:ms_dispatch 2024-03-24T00:34:53.547 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:34:53.547 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbd7b2aad20] 2024-03-24T00:34:53.547 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 2: gsignal() 2024-03-24T00:34:53.547 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 3: abort() 2024-03-24T00:34:53.547 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fbd7c5234fb] 2024-03-24T00:34:53.547 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fbd7c523667] 2024-03-24T00:34:53.547 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f1cc3e2223] 2024-03-24T00:34:53.547 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f1cc25c5d6] 2024-03-24T00:34:53.547 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f1cc25cda1] 2024-03-24T00:34:53.548 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f1cc25f2d5] 2024-03-24T00:34:53.548 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f1cc2af233] 2024-03-24T00:34:53.548 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f1cc2b1197] 2024-03-24T00:34:53.548 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f1cc1b011b] 2024-03-24T00:34:53.548 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f1cc1b06bc] 2024-03-24T00:34:53.548 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f1cc19a1ff] 2024-03-24T00:34:53.548 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fbd7c7b5488] 2024-03-24T00:34:53.548 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 16: (DispatchQueue::entry()+0x51f) [0x7fbd7c7b260f] 2024-03-24T00:34:53.548 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fbd7c885e81] 2024-03-24T00:34:53.548 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fbd7b2a01ca] 2024-03-24T00:34:53.548 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 19: clone() 2024-03-24T00:34:53.548 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 2024-03-24T00:34:53.175+0000 7fbd71ef9700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:34:53.548 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: in thread 7fbd71ef9700 thread_name:ms_dispatch 2024-03-24T00:34:53.548 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 2024-03-24T00:34:53.549 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:34:53.549 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbd7b2aad20] 2024-03-24T00:34:53.549 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 2: gsignal() 2024-03-24T00:34:53.549 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 3: abort() 2024-03-24T00:34:53.549 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fbd7c5234fb] 2024-03-24T00:34:53.549 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fbd7c523667] 2024-03-24T00:34:53.550 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f1cc3e2223] 2024-03-24T00:34:53.550 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f1cc25c5d6] 2024-03-24T00:34:53.550 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f1cc25cda1] 2024-03-24T00:34:53.550 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f1cc25f2d5] 2024-03-24T00:34:53.550 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f1cc2af233] 2024-03-24T00:34:53.550 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f1cc2b1197] 2024-03-24T00:34:53.550 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f1cc1b011b] 2024-03-24T00:34:53.551 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f1cc1b06bc] 2024-03-24T00:34:53.551 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f1cc19a1ff] 2024-03-24T00:34:53.551 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fbd7c7b5488] 2024-03-24T00:34:53.551 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 16: (DispatchQueue::entry()+0x51f) [0x7fbd7c7b260f] 2024-03-24T00:34:53.551 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fbd7c885e81] 2024-03-24T00:34:53.551 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fbd7b2a01ca] 2024-03-24T00:34:53.551 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 19: clone() 2024-03-24T00:34:53.551 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:34:53.551 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 2024-03-24T00:34:53.551 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: -1> 2024-03-24T00:34:53.173+0000 7fbd71ef9700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fbd71ef9700 time 2024-03-24T00:34:53.173123+0000 2024-03-24T00:34:53.551 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:34:53.551 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 2024-03-24T00:34:53.551 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:34:53.552 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fbd7c5234a1] 2024-03-24T00:34:53.552 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fbd7c523667] 2024-03-24T00:34:53.552 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f1cc3e2223] 2024-03-24T00:34:53.552 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f1cc25c5d6] 2024-03-24T00:34:53.552 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f1cc25cda1] 2024-03-24T00:34:53.552 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f1cc25f2d5] 2024-03-24T00:34:53.552 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f1cc2af233] 2024-03-24T00:34:53.552 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f1cc2b1197] 2024-03-24T00:34:53.552 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f1cc1b011b] 2024-03-24T00:34:53.552 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f1cc1b06bc] 2024-03-24T00:34:53.552 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f1cc19a1ff] 2024-03-24T00:34:53.552 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fbd7c7b5488] 2024-03-24T00:34:53.552 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 13: (DispatchQueue::entry()+0x51f) [0x7fbd7c7b260f] 2024-03-24T00:34:53.552 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fbd7c885e81] 2024-03-24T00:34:53.553 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fbd7b2a01ca] 2024-03-24T00:34:53.553 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 16: clone() 2024-03-24T00:34:53.553 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 2024-03-24T00:34:53.553 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 0> 2024-03-24T00:34:53.175+0000 7fbd71ef9700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:34:53.553 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: in thread 7fbd71ef9700 thread_name:ms_dispatch 2024-03-24T00:34:53.553 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 2024-03-24T00:34:53.553 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:34:53.553 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbd7b2aad20] 2024-03-24T00:34:53.553 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 2: gsignal() 2024-03-24T00:34:53.553 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 3: abort() 2024-03-24T00:34:53.553 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fbd7c5234fb] 2024-03-24T00:34:53.553 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fbd7c523667] 2024-03-24T00:34:53.553 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f1cc3e2223] 2024-03-24T00:34:53.553 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f1cc25c5d6] 2024-03-24T00:34:53.554 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f1cc25cda1] 2024-03-24T00:34:53.554 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f1cc25f2d5] 2024-03-24T00:34:53.554 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f1cc2af233] 2024-03-24T00:34:53.554 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f1cc2b1197] 2024-03-24T00:34:53.554 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f1cc1b011b] 2024-03-24T00:34:53.554 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f1cc1b06bc] 2024-03-24T00:34:53.554 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f1cc19a1ff] 2024-03-24T00:34:53.554 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fbd7c7b5488] 2024-03-24T00:34:53.554 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 16: (DispatchQueue::entry()+0x51f) [0x7fbd7c7b260f] 2024-03-24T00:34:53.554 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fbd7c885e81] 2024-03-24T00:34:53.554 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fbd7b2a01ca] 2024-03-24T00:34:53.554 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 19: clone() 2024-03-24T00:34:53.555 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:34:53.555 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 2024-03-24T00:34:53.555 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: -1> 2024-03-24T00:34:53.173+0000 7fbd71ef9700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fbd71ef9700 time 2024-03-24T00:34:53.173123+0000 2024-03-24T00:34:53.555 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:34:53.555 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 2024-03-24T00:34:53.555 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:34:53.555 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fbd7c5234a1] 2024-03-24T00:34:53.555 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fbd7c523667] 2024-03-24T00:34:53.555 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f1cc3e2223] 2024-03-24T00:34:53.555 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f1cc25c5d6] 2024-03-24T00:34:53.555 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f1cc25cda1] 2024-03-24T00:34:53.555 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f1cc25f2d5] 2024-03-24T00:34:53.555 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f1cc2af233] 2024-03-24T00:34:53.555 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f1cc2b1197] 2024-03-24T00:34:53.556 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f1cc1b011b] 2024-03-24T00:34:53.556 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f1cc1b06bc] 2024-03-24T00:34:53.556 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f1cc19a1ff] 2024-03-24T00:34:53.556 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fbd7c7b5488] 2024-03-24T00:34:53.556 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 13: (DispatchQueue::entry()+0x51f) [0x7fbd7c7b260f] 2024-03-24T00:34:53.556 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fbd7c885e81] 2024-03-24T00:34:53.556 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fbd7b2a01ca] 2024-03-24T00:34:53.556 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 16: clone() 2024-03-24T00:34:53.556 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 2024-03-24T00:34:53.556 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 0> 2024-03-24T00:34:53.175+0000 7fbd71ef9700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:34:53.556 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: in thread 7fbd71ef9700 thread_name:ms_dispatch 2024-03-24T00:34:53.556 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 2024-03-24T00:34:53.556 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:34:53.556 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbd7b2aad20] 2024-03-24T00:34:53.556 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 2: gsignal() 2024-03-24T00:34:53.557 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 3: abort() 2024-03-24T00:34:53.557 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fbd7c5234fb] 2024-03-24T00:34:53.557 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fbd7c523667] 2024-03-24T00:34:53.557 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55f1cc3e2223] 2024-03-24T00:34:53.557 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55f1cc25c5d6] 2024-03-24T00:34:53.557 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55f1cc25cda1] 2024-03-24T00:34:53.557 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55f1cc25f2d5] 2024-03-24T00:34:53.557 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55f1cc2af233] 2024-03-24T00:34:53.557 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f1cc2b1197] 2024-03-24T00:34:53.557 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55f1cc1b011b] 2024-03-24T00:34:53.558 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f1cc1b06bc] 2024-03-24T00:34:53.558 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55f1cc19a1ff] 2024-03-24T00:34:53.558 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fbd7c7b5488] 2024-03-24T00:34:53.558 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 16: (DispatchQueue::entry()+0x51f) [0x7fbd7c7b260f] 2024-03-24T00:34:53.558 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fbd7c885e81] 2024-03-24T00:34:53.558 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fbd7b2a01ca] 2024-03-24T00:34:53.558 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 19: clone() 2024-03-24T00:34:53.558 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:34:53.558 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91050]: 2024-03-24T00:34:53.558 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 podman[91144]: 2024-03-24 00:34:53.225366683 +0000 UTC m=+0.020633371 container died 9019ad3457304a74c4e4cbcd3e003e9f6c101b891e5f9bd5c5521457bd2130e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:34:53.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:53 smithi107 ceph-mon[36485]: pgmap v476: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:53.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:53 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6834/3414536872,v1:172.21.15.78:6835/3414536872] up:rejoin 2024-03-24T00:34:53.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:53 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=h=up:active} 2 up:standby-replay 2024-03-24T00:34:53.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:53 smithi107 ceph-mon[36485]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-24T00:34:53.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:53 smithi106 ceph-mon[36505]: pgmap v476: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:53.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:53 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6834/3414536872,v1:172.21.15.78:6835/3414536872] up:rejoin 2024-03-24T00:34:53.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:53 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=h=up:active} 2 up:standby-replay 2024-03-24T00:34:53.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:53 smithi106 ceph-mon[36505]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-24T00:34:53.720 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 622 sec 2024-03-24T00:34:54.227 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:54 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:34:54.227 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:54 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6834/3414536872,v1:172.21.15.78:6835/3414536872] up:active 2024-03-24T00:34:54.227 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:54 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=h=up:active} 2 up:standby-replay 2024-03-24T00:34:54.227 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 podman[91144]: 2024-03-24 00:34:53.99199642 +0000 UTC m=+0.787263157 container remove 9019ad3457304a74c4e4cbcd3e003e9f6c101b891e5f9bd5c5521457bd2130e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:34:54.227 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:53 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:34:54.494 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:54 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:34:54.495 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:54 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6834/3414536872,v1:172.21.15.78:6835/3414536872] up:active 2024-03-24T00:34:54.495 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:54 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=h=up:active} 2 up:standby-replay 2024-03-24T00:34:54.590 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:34:54 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.j.service: Failed with result 'exit-code'. 2024-03-24T00:34:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:54 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:34:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:54 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6834/3414536872,v1:172.21.15.78:6835/3414536872] up:active 2024-03-24T00:34:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:54 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=h=up:active} 2 up:standby-replay 2024-03-24T00:34:54.720 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 623 sec 2024-03-24T00:34:55.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:55 smithi078 ceph-mon[32429]: pgmap v477: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:55.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:55 smithi107 ceph-mon[36485]: pgmap v477: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:55.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:55 smithi106 ceph-mon[36505]: pgmap v477: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:55.720 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 624 sec 2024-03-24T00:34:56.720 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 625 sec 2024-03-24T00:34:57.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:57 smithi078 ceph-mon[32429]: pgmap v478: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 204 B/s wr, 9 op/s 2024-03-24T00:34:57.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:57 smithi107 ceph-mon[36485]: pgmap v478: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 204 B/s wr, 9 op/s 2024-03-24T00:34:57.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:57 smithi106 ceph-mon[36505]: pgmap v478: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 204 B/s wr, 9 op/s 2024-03-24T00:34:57.720 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 626 sec 2024-03-24T00:34:58.720 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 627 sec 2024-03-24T00:34:59.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:34:59 smithi078 ceph-mon[32429]: pgmap v479: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 204 B/s wr, 9 op/s 2024-03-24T00:34:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:34:59 smithi107 ceph-mon[36485]: pgmap v479: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 204 B/s wr, 9 op/s 2024-03-24T00:34:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:34:59 smithi106 ceph-mon[36505]: pgmap v479: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 204 B/s wr, 9 op/s 2024-03-24T00:34:59.720 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 628 sec 2024-03-24T00:35:00.721 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 629 sec 2024-03-24T00:35:01.436 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:01 smithi078 ceph-mon[32429]: pgmap v480: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 186 B/s wr, 8 op/s 2024-03-24T00:35:01.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:01 smithi107 ceph-mon[36485]: pgmap v480: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 186 B/s wr, 8 op/s 2024-03-24T00:35:01.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:01 smithi106 ceph-mon[36505]: pgmap v480: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 186 B/s wr, 8 op/s 2024-03-24T00:35:01.721 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 630 sec 2024-03-24T00:35:02.721 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 631 sec 2024-03-24T00:35:03.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:03 smithi078 ceph-mon[32429]: pgmap v481: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-24T00:35:03.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:03 smithi107 ceph-mon[36485]: pgmap v481: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-24T00:35:03.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:03 smithi106 ceph-mon[36505]: pgmap v481: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-24T00:35:03.721 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 632 sec 2024-03-24T00:35:04.642 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:04 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.j.service: Scheduled restart job, restart counter is at 4. 2024-03-24T00:35:04.642 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:04 smithi078 systemd[1]: Stopped Ceph mds.j for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:35:04.643 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:04 smithi078 systemd[1]: Starting Ceph mds.j for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:35:04.643 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:04 smithi078 podman[91237]: 2024-03-24 00:35:04.481343426 +0000 UTC m=+0.019292254 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:35:04.643 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:04 smithi078 podman[91237]: 2024-03-24 00:35:04.64322104 +0000 UTC m=+0.181169893 container create dda7dff44c65804e22ea4a355c536db748853e8faed45dd3c028ca20e2232820 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True) 2024-03-24T00:35:04.721 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 633 sec 2024-03-24T00:35:04.904 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:04 smithi078 podman[91237]: 2024-03-24 00:35:04.879374854 +0000 UTC m=+0.417323684 container init dda7dff44c65804e22ea4a355c536db748853e8faed45dd3c028ca20e2232820 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-24T00:35:04.904 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:04 smithi078 podman[91237]: 2024-03-24 00:35:04.882111035 +0000 UTC m=+0.420059855 container start dda7dff44c65804e22ea4a355c536db748853e8faed45dd3c028ca20e2232820 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-24T00:35:05.188 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:04 smithi078 ceph-mds[91256]: -- 172.21.15.78:0/2632918712 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fcab233a40 con 0x55fcab207000 2024-03-24T00:35:05.188 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: starting mds.j at 2024-03-24T00:35:05.188 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:04 smithi078 bash[91237]: dda7dff44c65804e22ea4a355c536db748853e8faed45dd3c028ca20e2232820 2024-03-24T00:35:05.188 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:04 smithi078 systemd[1]: Started Ceph mds.j for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:35:05.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:05 smithi078 ceph-mon[32429]: pgmap v482: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T00:35:05.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:05 smithi078 ceph-mon[32429]: daemon mds.j restarted 2024-03-24T00:35:05.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:05 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:35:05.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:05 smithi078 ceph-mon[32429]: osdmap e153: 12 total, 12 up, 12 in 2024-03-24T00:35:05.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:05 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6834/1985874955,v1:172.21.15.78:6835/1985874955] up:boot 2024-03-24T00:35:05.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:05 smithi078 ceph-mon[32429]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-03-24T00:35:05.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:05 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:35:05.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:35:05.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:05 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=h=up:active} 2 up:standby-replay 2024-03-24T00:35:05.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:05 smithi107 ceph-mon[36485]: pgmap v482: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T00:35:05.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:05 smithi107 ceph-mon[36485]: daemon mds.j restarted 2024-03-24T00:35:05.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:05 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:35:05.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:05 smithi107 ceph-mon[36485]: osdmap e153: 12 total, 12 up, 12 in 2024-03-24T00:35:05.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:05 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6834/1985874955,v1:172.21.15.78:6835/1985874955] up:boot 2024-03-24T00:35:05.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:05 smithi107 ceph-mon[36485]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-03-24T00:35:05.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:05 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:35:05.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:35:05.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:05 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=h=up:active} 2 up:standby-replay 2024-03-24T00:35:05.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:05 smithi106 ceph-mon[36505]: pgmap v482: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T00:35:05.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:05 smithi106 ceph-mon[36505]: daemon mds.j restarted 2024-03-24T00:35:05.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:05 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:35:05.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:05 smithi106 ceph-mon[36505]: osdmap e153: 12 total, 12 up, 12 in 2024-03-24T00:35:05.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:05 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6834/1985874955,v1:172.21.15.78:6835/1985874955] up:boot 2024-03-24T00:35:05.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:05 smithi106 ceph-mon[36505]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-03-24T00:35:05.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:05 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:35:05.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:35:05.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:05 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=h=up:active} 2 up:standby-replay 2024-03-24T00:35:05.721 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 634 sec 2024-03-24T00:35:06.721 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 635 sec 2024-03-24T00:35:07.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:06 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6834/1985874955,v1:172.21.15.78:6835/1985874955] up:resolve 2024-03-24T00:35:07.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:06 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=h=up:active} 2 up:standby-replay 2024-03-24T00:35:07.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:06 smithi078 ceph-mon[32429]: pgmap v484: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:35:07.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:06 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6834/1985874955,v1:172.21.15.78:6835/1985874955] up:resolve 2024-03-24T00:35:07.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:06 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=h=up:active} 2 up:standby-replay 2024-03-24T00:35:07.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:06 smithi107 ceph-mon[36485]: pgmap v484: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:35:07.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:06 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6834/1985874955,v1:172.21.15.78:6835/1985874955] up:resolve 2024-03-24T00:35:07.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:06 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=h=up:active} 2 up:standby-replay 2024-03-24T00:35:07.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:06 smithi106 ceph-mon[36505]: pgmap v484: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:35:07.721 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 636 sec 2024-03-24T00:35:08.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:07 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:35:08.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:07 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6834/1985874955,v1:172.21.15.78:6835/1985874955] up:reconnect 2024-03-24T00:35:08.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:07 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=h=up:active} 2 up:standby-replay 2024-03-24T00:35:08.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:07 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:35:08.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:07 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6834/1985874955,v1:172.21.15.78:6835/1985874955] up:reconnect 2024-03-24T00:35:08.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:07 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=h=up:active} 2 up:standby-replay 2024-03-24T00:35:08.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:07 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0 2024-03-24T00:35:08.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:07 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6834/1985874955,v1:172.21.15.78:6835/1985874955] up:reconnect 2024-03-24T00:35:08.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:07 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=h=up:active} 2 up:standby-replay 2024-03-24T00:35:08.721 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 637 sec 2024-03-24T00:35:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6834/1985874955,v1:172.21.15.78:6835/1985874955] up:rejoin 2024-03-24T00:35:09.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=h=up:active} 2 up:standby-replay 2024-03-24T00:35:09.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-mon[32429]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-24T00:35:09.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-mon[32429]: pgmap v485: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:35:09.341 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fc4bfa88700 time 2024-03-24T00:35:08.972655+0000 2024-03-24T00:35:09.341 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:35:09.341 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:35:09.341 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fc4ca0b24a1] 2024-03-24T00:35:09.342 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc4ca0b2667] 2024-03-24T00:35:09.342 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55fcaa1f0223] 2024-03-24T00:35:09.342 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55fcaa06a5d6] 2024-03-24T00:35:09.342 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55fcaa06ada1] 2024-03-24T00:35:09.342 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55fcaa06d2d5] 2024-03-24T00:35:09.342 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55fcaa0bd233] 2024-03-24T00:35:09.342 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55fcaa0bf197] 2024-03-24T00:35:09.342 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55fca9fbe11b] 2024-03-24T00:35:09.342 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55fca9fbe6bc] 2024-03-24T00:35:09.342 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55fca9fa81ff] 2024-03-24T00:35:09.342 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc4ca344488] 2024-03-24T00:35:09.342 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 13: (DispatchQueue::entry()+0x51f) [0x7fc4ca34160f] 2024-03-24T00:35:09.342 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc4ca414e81] 2024-03-24T00:35:09.343 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc4c8e2f1ca] 2024-03-24T00:35:09.343 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 16: clone() 2024-03-24T00:35:09.343 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: *** Caught signal (Aborted) ** 2024-03-24T00:35:09.343 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: in thread 7fc4bfa88700 thread_name:ms_dispatch 2024-03-24T00:35:09.343 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 2024-03-24T00:35:08.973+0000 7fc4bfa88700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fc4bfa88700 time 2024-03-24T00:35:08.972655+0000 2024-03-24T00:35:09.343 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:35:09.343 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 2024-03-24T00:35:09.343 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:35:09.343 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fc4ca0b24a1] 2024-03-24T00:35:09.343 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc4ca0b2667] 2024-03-24T00:35:09.343 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55fcaa1f0223] 2024-03-24T00:35:09.343 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55fcaa06a5d6] 2024-03-24T00:35:09.343 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55fcaa06ada1] 2024-03-24T00:35:09.344 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55fcaa06d2d5] 2024-03-24T00:35:09.344 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55fcaa0bd233] 2024-03-24T00:35:09.344 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55fcaa0bf197] 2024-03-24T00:35:09.344 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55fca9fbe11b] 2024-03-24T00:35:09.344 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55fca9fbe6bc] 2024-03-24T00:35:09.344 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55fca9fa81ff] 2024-03-24T00:35:09.344 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc4ca344488] 2024-03-24T00:35:09.344 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 13: (DispatchQueue::entry()+0x51f) [0x7fc4ca34160f] 2024-03-24T00:35:09.344 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc4ca414e81] 2024-03-24T00:35:09.344 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc4c8e2f1ca] 2024-03-24T00:35:09.344 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 16: clone() 2024-03-24T00:35:09.344 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 2024-03-24T00:35:09.344 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:35:09.344 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc4c8e39d20] 2024-03-24T00:35:09.345 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:08 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6834/1985874955,v1:172.21.15.78:6835/1985874955] up:rejoin 2024-03-24T00:35:09.345 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:08 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=h=up:active} 2 up:standby-replay 2024-03-24T00:35:09.345 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:08 smithi107 ceph-mon[36485]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-24T00:35:09.345 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:08 smithi107 ceph-mon[36485]: pgmap v485: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:35:09.346 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 2: gsignal() 2024-03-24T00:35:09.346 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 3: abort() 2024-03-24T00:35:09.346 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fc4ca0b24fb] 2024-03-24T00:35:09.346 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc4ca0b2667] 2024-03-24T00:35:09.346 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55fcaa1f0223] 2024-03-24T00:35:09.346 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55fcaa06a5d6] 2024-03-24T00:35:09.347 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55fcaa06ada1] 2024-03-24T00:35:09.347 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55fcaa06d2d5] 2024-03-24T00:35:09.347 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55fcaa0bd233] 2024-03-24T00:35:09.347 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55fcaa0bf197] 2024-03-24T00:35:09.347 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55fca9fbe11b] 2024-03-24T00:35:09.347 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55fca9fbe6bc] 2024-03-24T00:35:09.347 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55fca9fa81ff] 2024-03-24T00:35:09.347 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc4ca344488] 2024-03-24T00:35:09.347 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 16: (DispatchQueue::entry()+0x51f) [0x7fc4ca34160f] 2024-03-24T00:35:09.348 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc4ca414e81] 2024-03-24T00:35:09.348 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc4c8e2f1ca] 2024-03-24T00:35:09.348 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 19: clone() 2024-03-24T00:35:09.348 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 2024-03-24T00:35:08.976+0000 7fc4bfa88700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:35:09.348 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: in thread 7fc4bfa88700 thread_name:ms_dispatch 2024-03-24T00:35:09.348 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 2024-03-24T00:35:09.348 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:35:09.348 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc4c8e39d20] 2024-03-24T00:35:09.348 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 2: gsignal() 2024-03-24T00:35:09.348 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 3: abort() 2024-03-24T00:35:09.348 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fc4ca0b24fb] 2024-03-24T00:35:09.348 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc4ca0b2667] 2024-03-24T00:35:09.348 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55fcaa1f0223] 2024-03-24T00:35:09.348 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55fcaa06a5d6] 2024-03-24T00:35:09.349 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55fcaa06ada1] 2024-03-24T00:35:09.349 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55fcaa06d2d5] 2024-03-24T00:35:09.349 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55fcaa0bd233] 2024-03-24T00:35:09.349 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55fcaa0bf197] 2024-03-24T00:35:09.349 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55fca9fbe11b] 2024-03-24T00:35:09.349 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55fca9fbe6bc] 2024-03-24T00:35:09.349 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55fca9fa81ff] 2024-03-24T00:35:09.349 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc4ca344488] 2024-03-24T00:35:09.349 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 16: (DispatchQueue::entry()+0x51f) [0x7fc4ca34160f] 2024-03-24T00:35:09.349 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc4ca414e81] 2024-03-24T00:35:09.349 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc4c8e2f1ca] 2024-03-24T00:35:09.349 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 19: clone() 2024-03-24T00:35:09.349 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:35:09.349 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 2024-03-24T00:35:09.349 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: -1> 2024-03-24T00:35:08.973+0000 7fc4bfa88700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fc4bfa88700 time 2024-03-24T00:35:08.972655+0000 2024-03-24T00:35:09.350 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:35:09.350 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 2024-03-24T00:35:09.350 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:35:09.350 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fc4ca0b24a1] 2024-03-24T00:35:09.350 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc4ca0b2667] 2024-03-24T00:35:09.350 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55fcaa1f0223] 2024-03-24T00:35:09.350 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55fcaa06a5d6] 2024-03-24T00:35:09.350 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55fcaa06ada1] 2024-03-24T00:35:09.350 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55fcaa06d2d5] 2024-03-24T00:35:09.350 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55fcaa0bd233] 2024-03-24T00:35:09.350 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55fcaa0bf197] 2024-03-24T00:35:09.350 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55fca9fbe11b] 2024-03-24T00:35:09.350 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55fca9fbe6bc] 2024-03-24T00:35:09.351 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55fca9fa81ff] 2024-03-24T00:35:09.351 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc4ca344488] 2024-03-24T00:35:09.351 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 13: (DispatchQueue::entry()+0x51f) [0x7fc4ca34160f] 2024-03-24T00:35:09.351 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc4ca414e81] 2024-03-24T00:35:09.351 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc4c8e2f1ca] 2024-03-24T00:35:09.351 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 16: clone() 2024-03-24T00:35:09.351 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 2024-03-24T00:35:09.351 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 0> 2024-03-24T00:35:08.976+0000 7fc4bfa88700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:35:09.351 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: in thread 7fc4bfa88700 thread_name:ms_dispatch 2024-03-24T00:35:09.351 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 2024-03-24T00:35:09.351 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:35:09.351 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc4c8e39d20] 2024-03-24T00:35:09.351 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 2: gsignal() 2024-03-24T00:35:09.351 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 3: abort() 2024-03-24T00:35:09.352 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fc4ca0b24fb] 2024-03-24T00:35:09.352 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc4ca0b2667] 2024-03-24T00:35:09.352 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55fcaa1f0223] 2024-03-24T00:35:09.352 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55fcaa06a5d6] 2024-03-24T00:35:09.352 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55fcaa06ada1] 2024-03-24T00:35:09.352 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55fcaa06d2d5] 2024-03-24T00:35:09.352 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55fcaa0bd233] 2024-03-24T00:35:09.352 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55fcaa0bf197] 2024-03-24T00:35:09.352 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55fca9fbe11b] 2024-03-24T00:35:09.352 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55fca9fbe6bc] 2024-03-24T00:35:09.352 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55fca9fa81ff] 2024-03-24T00:35:09.352 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc4ca344488] 2024-03-24T00:35:09.352 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 16: (DispatchQueue::entry()+0x51f) [0x7fc4ca34160f] 2024-03-24T00:35:09.353 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc4ca414e81] 2024-03-24T00:35:09.353 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc4c8e2f1ca] 2024-03-24T00:35:09.353 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 19: clone() 2024-03-24T00:35:09.353 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:35:09.353 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 2024-03-24T00:35:09.353 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: -1> 2024-03-24T00:35:08.973+0000 7fc4bfa88700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fc4bfa88700 time 2024-03-24T00:35:08.972655+0000 2024-03-24T00:35:09.353 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:35:09.353 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 2024-03-24T00:35:09.353 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:35:09.353 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fc4ca0b24a1] 2024-03-24T00:35:09.353 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc4ca0b2667] 2024-03-24T00:35:09.353 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55fcaa1f0223] 2024-03-24T00:35:09.353 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55fcaa06a5d6] 2024-03-24T00:35:09.353 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55fcaa06ada1] 2024-03-24T00:35:09.353 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55fcaa06d2d5] 2024-03-24T00:35:09.354 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55fcaa0bd233] 2024-03-24T00:35:09.354 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55fcaa0bf197] 2024-03-24T00:35:09.354 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55fca9fbe11b] 2024-03-24T00:35:09.354 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55fca9fbe6bc] 2024-03-24T00:35:09.354 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55fca9fa81ff] 2024-03-24T00:35:09.354 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc4ca344488] 2024-03-24T00:35:09.354 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 13: (DispatchQueue::entry()+0x51f) [0x7fc4ca34160f] 2024-03-24T00:35:09.354 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc4ca414e81] 2024-03-24T00:35:09.354 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc4c8e2f1ca] 2024-03-24T00:35:09.354 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 16: clone() 2024-03-24T00:35:09.354 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 2024-03-24T00:35:09.354 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 0> 2024-03-24T00:35:08.976+0000 7fc4bfa88700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:35:09.354 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: in thread 7fc4bfa88700 thread_name:ms_dispatch 2024-03-24T00:35:09.354 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 2024-03-24T00:35:09.355 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:35:09.355 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc4c8e39d20] 2024-03-24T00:35:09.355 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 2: gsignal() 2024-03-24T00:35:09.355 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 3: abort() 2024-03-24T00:35:09.355 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fc4ca0b24fb] 2024-03-24T00:35:09.355 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc4ca0b2667] 2024-03-24T00:35:09.355 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55fcaa1f0223] 2024-03-24T00:35:09.355 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55fcaa06a5d6] 2024-03-24T00:35:09.355 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55fcaa06ada1] 2024-03-24T00:35:09.355 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55fcaa06d2d5] 2024-03-24T00:35:09.355 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55fcaa0bd233] 2024-03-24T00:35:09.355 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55fcaa0bf197] 2024-03-24T00:35:09.355 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55fca9fbe11b] 2024-03-24T00:35:09.355 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55fca9fbe6bc] 2024-03-24T00:35:09.356 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55fca9fa81ff] 2024-03-24T00:35:09.356 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc4ca344488] 2024-03-24T00:35:09.356 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 16: (DispatchQueue::entry()+0x51f) [0x7fc4ca34160f] 2024-03-24T00:35:09.356 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc4ca414e81] 2024-03-24T00:35:09.356 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc4c8e2f1ca] 2024-03-24T00:35:09.356 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 19: clone() 2024-03-24T00:35:09.356 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:35:09.356 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91252]: 2024-03-24T00:35:09.356 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:09 smithi078 podman[91290]: 2024-03-24 00:35:09.02744807 +0000 UTC m=+0.022225813 container died dda7dff44c65804e22ea4a355c536db748853e8faed45dd3c028ca20e2232820 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:35:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:08 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6834/1985874955,v1:172.21.15.78:6835/1985874955] up:rejoin 2024-03-24T00:35:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:08 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=h=up:active} 2 up:standby-replay 2024-03-24T00:35:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:08 smithi106 ceph-mon[36505]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-24T00:35:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:08 smithi106 ceph-mon[36505]: pgmap v485: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:35:09.721 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 638 sec 2024-03-24T00:35:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:09 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:35:10.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:09 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6834/1985874955,v1:172.21.15.78:6835/1985874955] up:active 2024-03-24T00:35:10.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:09 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=h=up:active} 2 up:standby-replay 2024-03-24T00:35:10.091 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:09 smithi078 podman[91290]: 2024-03-24 00:35:09.818620726 +0000 UTC m=+0.813398514 container remove dda7dff44c65804e22ea4a355c536db748853e8faed45dd3c028ca20e2232820 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-24T00:35:10.091 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:09 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:35:10.091 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:10 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.j.service: Failed with result 'exit-code'. 2024-03-24T00:35:10.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:09 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:35:10.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:09 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6834/1985874955,v1:172.21.15.78:6835/1985874955] up:active 2024-03-24T00:35:10.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:09 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=h=up:active} 2 up:standby-replay 2024-03-24T00:35:10.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:09 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:35:10.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:09 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6834/1985874955,v1:172.21.15.78:6835/1985874955] up:active 2024-03-24T00:35:10.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:09 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=h=up:active} 2 up:standby-replay 2024-03-24T00:35:10.721 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 639 sec 2024-03-24T00:35:11.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:10 smithi078 ceph-mon[32429]: pgmap v486: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:35:11.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:10 smithi107 ceph-mon[36485]: pgmap v486: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:35:11.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:10 smithi106 ceph-mon[36505]: pgmap v486: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:35:11.721 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 640 sec 2024-03-24T00:35:12.722 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 641 sec 2024-03-24T00:35:13.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:13 smithi078 ceph-mon[32429]: pgmap v487: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 25 op/s 2024-03-24T00:35:13.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:13 smithi107 ceph-mon[36485]: pgmap v487: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 25 op/s 2024-03-24T00:35:13.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:13 smithi106 ceph-mon[36505]: pgmap v487: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 25 op/s 2024-03-24T00:35:13.722 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 642 sec 2024-03-24T00:35:14.722 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 643 sec 2024-03-24T00:35:15.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:15 smithi078 ceph-mon[32429]: pgmap v488: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 25 op/s 2024-03-24T00:35:15.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:15 smithi107 ceph-mon[36485]: pgmap v488: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 25 op/s 2024-03-24T00:35:15.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:15 smithi106 ceph-mon[36505]: pgmap v488: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 25 op/s 2024-03-24T00:35:15.722 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 644 sec 2024-03-24T00:35:16.722 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 645 sec 2024-03-24T00:35:17.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:17 smithi078 ceph-mon[32429]: pgmap v489: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-03-24T00:35:17.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:17 smithi107 ceph-mon[36485]: pgmap v489: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-03-24T00:35:17.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:17 smithi106 ceph-mon[36505]: pgmap v489: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-03-24T00:35:17.722 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 646 sec 2024-03-24T00:35:18.722 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 647 sec 2024-03-24T00:35:19.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:19 smithi078 ceph-mon[32429]: pgmap v490: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-24T00:35:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:19 smithi107 ceph-mon[36485]: pgmap v490: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-24T00:35:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:19 smithi106 ceph-mon[36505]: pgmap v490: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-24T00:35:19.722 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 648 sec 2024-03-24T00:35:20.415 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:20 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.j.service: Scheduled restart job, restart counter is at 5. 2024-03-24T00:35:20.415 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:20 smithi078 systemd[1]: Stopped Ceph mds.j for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:35:20.415 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:20 smithi078 systemd[1]: Starting Ceph mds.j for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T00:35:20.415 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:20 smithi078 podman[91380]: 2024-03-24 00:35:20.224951308 +0000 UTC m=+0.018555914 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:35:20.696 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:20 smithi078 podman[91380]: 2024-03-24 00:35:20.415581399 +0000 UTC m=+0.209186031 container create a79393ec3cf05f4ca97e98d3c53956c7cf103c3a739a30dd1a868477320b34a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:35:20.722 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 649 sec 2024-03-24T00:35:21.090 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:20 smithi078 podman[91380]: 2024-03-24 00:35:20.697199277 +0000 UTC m=+0.490803877 container init a79393ec3cf05f4ca97e98d3c53956c7cf103c3a739a30dd1a868477320b34a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-24T00:35:21.091 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:20 smithi078 podman[91380]: 2024-03-24 00:35:20.699790994 +0000 UTC m=+0.493395588 container start a79393ec3cf05f4ca97e98d3c53956c7cf103c3a739a30dd1a868477320b34a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-24T00:35:21.091 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:20 smithi078 ceph-mds[91399]: -- 172.21.15.78:0/461001480 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d9d0169a40 con 0x55d9d013c800 2024-03-24T00:35:21.091 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:20 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: starting mds.j at 2024-03-24T00:35:21.091 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:20 smithi078 bash[91380]: a79393ec3cf05f4ca97e98d3c53956c7cf103c3a739a30dd1a868477320b34a3 2024-03-24T00:35:21.091 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:20 smithi078 systemd[1]: Started Ceph mds.j for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:35:21.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:21 smithi078 ceph-mon[32429]: pgmap v491: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-24T00:35:21.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:21 smithi078 ceph-mon[32429]: daemon mds.j restarted 2024-03-24T00:35:21.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:21 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:35:21.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:21 smithi078 ceph-mon[32429]: osdmap e154: 12 total, 12 up, 12 in 2024-03-24T00:35:21.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:21 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6834/3027423976,v1:172.21.15.78:6835/3027423976] up:boot 2024-03-24T00:35:21.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:21 smithi078 ceph-mon[32429]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-03-24T00:35:21.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:21 smithi078 ceph-mon[32429]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:35:21.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:21 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:35:21.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:21 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=h=up:active} 2 up:standby-replay 2024-03-24T00:35:21.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:21 smithi107 ceph-mon[36485]: pgmap v491: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-24T00:35:21.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:21 smithi107 ceph-mon[36485]: daemon mds.j restarted 2024-03-24T00:35:21.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:21 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:35:21.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:21 smithi107 ceph-mon[36485]: osdmap e154: 12 total, 12 up, 12 in 2024-03-24T00:35:21.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:21 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6834/3027423976,v1:172.21.15.78:6835/3027423976] up:boot 2024-03-24T00:35:21.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:21 smithi107 ceph-mon[36485]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-03-24T00:35:21.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:21 smithi107 ceph-mon[36485]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:35:21.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:21 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:35:21.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:21 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=h=up:active} 2 up:standby-replay 2024-03-24T00:35:21.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:21 smithi106 ceph-mon[36505]: pgmap v491: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-24T00:35:21.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:21 smithi106 ceph-mon[36505]: daemon mds.j restarted 2024-03-24T00:35:21.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:21 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:35:21.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:21 smithi106 ceph-mon[36505]: osdmap e154: 12 total, 12 up, 12 in 2024-03-24T00:35:21.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:21 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6834/3027423976,v1:172.21.15.78:6835/3027423976] up:boot 2024-03-24T00:35:21.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:21 smithi106 ceph-mon[36505]: Standby daemon mds.j assigned to filesystem cephfs as rank 1 2024-03-24T00:35:21.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:21 smithi106 ceph-mon[36505]: fsmap cephfs:2/3 {0=b=up:active,2=h=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:35:21.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:21 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:35:21.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:21 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:replay,2=h=up:active} 2 up:standby-replay 2024-03-24T00:35:21.722 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 650 sec 2024-03-24T00:35:22.722 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 651 sec 2024-03-24T00:35:23.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:22 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6834/3027423976,v1:172.21.15.78:6835/3027423976] up:resolve 2024-03-24T00:35:23.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:22 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=h=up:active} 2 up:standby-replay 2024-03-24T00:35:23.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:22 smithi078 ceph-mon[32429]: pgmap v493: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 204 B/s wr, 4 op/s 2024-03-24T00:35:23.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:22 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6834/3027423976,v1:172.21.15.78:6835/3027423976] up:resolve 2024-03-24T00:35:23.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:22 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=h=up:active} 2 up:standby-replay 2024-03-24T00:35:23.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:22 smithi107 ceph-mon[36485]: pgmap v493: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 204 B/s wr, 4 op/s 2024-03-24T00:35:23.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:22 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6834/3027423976,v1:172.21.15.78:6835/3027423976] up:resolve 2024-03-24T00:35:23.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:22 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:resolve,2=h=up:active} 2 up:standby-replay 2024-03-24T00:35:23.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:22 smithi106 ceph-mon[36505]: pgmap v493: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 204 B/s wr, 4 op/s 2024-03-24T00:35:23.722 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 652 sec 2024-03-24T00:35:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:23 smithi078 ceph-mon[32429]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999982 2024-03-24T00:35:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:23 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6834/3027423976,v1:172.21.15.78:6835/3027423976] up:reconnect 2024-03-24T00:35:24.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:23 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=h=up:active} 2 up:standby-replay 2024-03-24T00:35:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:23 smithi107 ceph-mon[36485]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999982 2024-03-24T00:35:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:23 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6834/3027423976,v1:172.21.15.78:6835/3027423976] up:reconnect 2024-03-24T00:35:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:23 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=h=up:active} 2 up:standby-replay 2024-03-24T00:35:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:23 smithi106 ceph-mon[36505]: reconnect by client.15462 v1:192.168.0.1:0/3344586265 after 0.000999982 2024-03-24T00:35:24.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:23 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6834/3027423976,v1:172.21.15.78:6835/3027423976] up:reconnect 2024-03-24T00:35:24.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:23 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:reconnect,2=h=up:active} 2 up:standby-replay 2024-03-24T00:35:24.722 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 653 sec 2024-03-24T00:35:25.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6834/3027423976,v1:172.21.15.78:6835/3027423976] up:rejoin 2024-03-24T00:35:25.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-mon[32429]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=h=up:active} 2 up:standby-replay 2024-03-24T00:35:25.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-mon[32429]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-24T00:35:25.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-mon[32429]: pgmap v494: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 204 B/s wr, 4 op/s 2024-03-24T00:35:25.091 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f423cc64700 time 2024-03-24T00:35:24.777367+0000 2024-03-24T00:35:25.091 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:35:25.091 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:35:25.091 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f424728e4a1] 2024-03-24T00:35:25.091 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f424728e667] 2024-03-24T00:35:25.091 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55d9ceb2d223] 2024-03-24T00:35:25.091 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55d9ce9a75d6] 2024-03-24T00:35:25.091 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55d9ce9a7da1] 2024-03-24T00:35:25.092 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55d9ce9aa2d5] 2024-03-24T00:35:25.092 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55d9ce9fa233] 2024-03-24T00:35:25.092 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d9ce9fc197] 2024-03-24T00:35:25.092 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d9ce8fb11b] 2024-03-24T00:35:25.092 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d9ce8fb6bc] 2024-03-24T00:35:25.092 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55d9ce8e51ff] 2024-03-24T00:35:25.092 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f4247520488] 2024-03-24T00:35:25.092 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 13: (DispatchQueue::entry()+0x51f) [0x7f424751d60f] 2024-03-24T00:35:25.092 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f42475f0e81] 2024-03-24T00:35:25.092 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f424600b1ca] 2024-03-24T00:35:25.092 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 16: clone() 2024-03-24T00:35:25.093 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 2024-03-24T00:35:24.778+0000 7f423cc64700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f423cc64700 time 2024-03-24T00:35:24.777367+0000 2024-03-24T00:35:25.093 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:35:25.093 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 2024-03-24T00:35:25.093 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:35:25.093 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f424728e4a1] 2024-03-24T00:35:25.093 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f424728e667] 2024-03-24T00:35:25.093 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55d9ceb2d223] 2024-03-24T00:35:25.093 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55d9ce9a75d6] 2024-03-24T00:35:25.094 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55d9ce9a7da1] 2024-03-24T00:35:25.094 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55d9ce9aa2d5] 2024-03-24T00:35:25.094 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55d9ce9fa233] 2024-03-24T00:35:25.094 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d9ce9fc197] 2024-03-24T00:35:25.094 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d9ce8fb11b] 2024-03-24T00:35:25.094 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d9ce8fb6bc] 2024-03-24T00:35:25.094 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55d9ce8e51ff] 2024-03-24T00:35:25.094 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f4247520488] 2024-03-24T00:35:25.094 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 13: (DispatchQueue::entry()+0x51f) [0x7f424751d60f] 2024-03-24T00:35:25.094 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f42475f0e81] 2024-03-24T00:35:25.094 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f424600b1ca] 2024-03-24T00:35:25.094 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 16: clone() 2024-03-24T00:35:25.095 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 2024-03-24T00:35:25.095 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: *** Caught signal (Aborted) ** 2024-03-24T00:35:25.095 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: in thread 7f423cc64700 thread_name:ms_dispatch 2024-03-24T00:35:25.095 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:35:25.095 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4246015d20] 2024-03-24T00:35:25.095 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 2: gsignal() 2024-03-24T00:35:25.095 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 3: abort() 2024-03-24T00:35:25.095 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f424728e4fb] 2024-03-24T00:35:25.095 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f424728e667] 2024-03-24T00:35:25.095 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55d9ceb2d223] 2024-03-24T00:35:25.095 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55d9ce9a75d6] 2024-03-24T00:35:25.095 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55d9ce9a7da1] 2024-03-24T00:35:25.095 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55d9ce9aa2d5] 2024-03-24T00:35:25.096 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55d9ce9fa233] 2024-03-24T00:35:25.096 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d9ce9fc197] 2024-03-24T00:35:25.096 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d9ce8fb11b] 2024-03-24T00:35:25.096 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d9ce8fb6bc] 2024-03-24T00:35:25.096 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55d9ce8e51ff] 2024-03-24T00:35:25.096 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f4247520488] 2024-03-24T00:35:25.096 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 16: (DispatchQueue::entry()+0x51f) [0x7f424751d60f] 2024-03-24T00:35:25.096 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f42475f0e81] 2024-03-24T00:35:25.096 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f424600b1ca] 2024-03-24T00:35:25.096 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 19: clone() 2024-03-24T00:35:25.096 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 2024-03-24T00:35:24.780+0000 7f423cc64700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:35:25.096 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: in thread 7f423cc64700 thread_name:ms_dispatch 2024-03-24T00:35:25.096 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 2024-03-24T00:35:25.096 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:35:25.096 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4246015d20] 2024-03-24T00:35:25.096 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 2: gsignal() 2024-03-24T00:35:25.097 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 3: abort() 2024-03-24T00:35:25.097 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f424728e4fb] 2024-03-24T00:35:25.097 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f424728e667] 2024-03-24T00:35:25.097 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55d9ceb2d223] 2024-03-24T00:35:25.097 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55d9ce9a75d6] 2024-03-24T00:35:25.097 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55d9ce9a7da1] 2024-03-24T00:35:25.097 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55d9ce9aa2d5] 2024-03-24T00:35:25.097 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55d9ce9fa233] 2024-03-24T00:35:25.097 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d9ce9fc197] 2024-03-24T00:35:25.098 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d9ce8fb11b] 2024-03-24T00:35:25.098 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d9ce8fb6bc] 2024-03-24T00:35:25.098 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55d9ce8e51ff] 2024-03-24T00:35:25.098 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f4247520488] 2024-03-24T00:35:25.098 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 16: (DispatchQueue::entry()+0x51f) [0x7f424751d60f] 2024-03-24T00:35:25.098 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f42475f0e81] 2024-03-24T00:35:25.098 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f424600b1ca] 2024-03-24T00:35:25.099 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 19: clone() 2024-03-24T00:35:25.099 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:35:25.099 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 2024-03-24T00:35:25.099 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: -1> 2024-03-24T00:35:24.778+0000 7f423cc64700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f423cc64700 time 2024-03-24T00:35:24.777367+0000 2024-03-24T00:35:25.099 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:35:25.099 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 2024-03-24T00:35:25.099 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:35:25.099 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f424728e4a1] 2024-03-24T00:35:25.099 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f424728e667] 2024-03-24T00:35:25.100 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:24 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6834/3027423976,v1:172.21.15.78:6835/3027423976] up:rejoin 2024-03-24T00:35:25.100 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:24 smithi107 ceph-mon[36485]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=h=up:active} 2 up:standby-replay 2024-03-24T00:35:25.100 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:24 smithi107 ceph-mon[36485]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-24T00:35:25.100 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:24 smithi107 ceph-mon[36485]: pgmap v494: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 204 B/s wr, 4 op/s 2024-03-24T00:35:25.100 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55d9ceb2d223] 2024-03-24T00:35:25.101 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55d9ce9a75d6] 2024-03-24T00:35:25.101 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55d9ce9a7da1] 2024-03-24T00:35:25.101 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55d9ce9aa2d5] 2024-03-24T00:35:25.101 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55d9ce9fa233] 2024-03-24T00:35:25.101 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d9ce9fc197] 2024-03-24T00:35:25.101 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d9ce8fb11b] 2024-03-24T00:35:25.101 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d9ce8fb6bc] 2024-03-24T00:35:25.101 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55d9ce8e51ff] 2024-03-24T00:35:25.101 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f4247520488] 2024-03-24T00:35:25.101 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 13: (DispatchQueue::entry()+0x51f) [0x7f424751d60f] 2024-03-24T00:35:25.101 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f42475f0e81] 2024-03-24T00:35:25.101 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f424600b1ca] 2024-03-24T00:35:25.102 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 16: clone() 2024-03-24T00:35:25.102 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 2024-03-24T00:35:25.102 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 0> 2024-03-24T00:35:24.780+0000 7f423cc64700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:35:25.102 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: in thread 7f423cc64700 thread_name:ms_dispatch 2024-03-24T00:35:25.102 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 2024-03-24T00:35:25.102 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:35:25.102 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4246015d20] 2024-03-24T00:35:25.102 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 2: gsignal() 2024-03-24T00:35:25.102 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 3: abort() 2024-03-24T00:35:25.102 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f424728e4fb] 2024-03-24T00:35:25.102 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f424728e667] 2024-03-24T00:35:25.102 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55d9ceb2d223] 2024-03-24T00:35:25.102 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55d9ce9a75d6] 2024-03-24T00:35:25.102 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55d9ce9a7da1] 2024-03-24T00:35:25.103 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55d9ce9aa2d5] 2024-03-24T00:35:25.103 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55d9ce9fa233] 2024-03-24T00:35:25.103 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d9ce9fc197] 2024-03-24T00:35:25.103 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d9ce8fb11b] 2024-03-24T00:35:25.103 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d9ce8fb6bc] 2024-03-24T00:35:25.103 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55d9ce8e51ff] 2024-03-24T00:35:25.103 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f4247520488] 2024-03-24T00:35:25.103 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 16: (DispatchQueue::entry()+0x51f) [0x7f424751d60f] 2024-03-24T00:35:25.103 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f42475f0e81] 2024-03-24T00:35:25.103 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f424600b1ca] 2024-03-24T00:35:25.103 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 19: clone() 2024-03-24T00:35:25.103 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:35:25.103 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 2024-03-24T00:35:25.103 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: -1> 2024-03-24T00:35:24.778+0000 7f423cc64700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f423cc64700 time 2024-03-24T00:35:24.777367+0000 2024-03-24T00:35:25.104 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:35:25.104 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 2024-03-24T00:35:25.104 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:35:25.104 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f424728e4a1] 2024-03-24T00:35:25.104 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f424728e667] 2024-03-24T00:35:25.104 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55d9ceb2d223] 2024-03-24T00:35:25.104 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55d9ce9a75d6] 2024-03-24T00:35:25.104 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55d9ce9a7da1] 2024-03-24T00:35:25.104 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 6: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55d9ce9aa2d5] 2024-03-24T00:35:25.104 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55d9ce9fa233] 2024-03-24T00:35:25.105 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d9ce9fc197] 2024-03-24T00:35:25.105 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d9ce8fb11b] 2024-03-24T00:35:25.105 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d9ce8fb6bc] 2024-03-24T00:35:25.105 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55d9ce8e51ff] 2024-03-24T00:35:25.105 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f4247520488] 2024-03-24T00:35:25.105 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 13: (DispatchQueue::entry()+0x51f) [0x7f424751d60f] 2024-03-24T00:35:25.105 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f42475f0e81] 2024-03-24T00:35:25.105 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f424600b1ca] 2024-03-24T00:35:25.105 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 16: clone() 2024-03-24T00:35:25.105 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 2024-03-24T00:35:25.105 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 0> 2024-03-24T00:35:24.780+0000 7f423cc64700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:35:25.105 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: in thread 7f423cc64700 thread_name:ms_dispatch 2024-03-24T00:35:25.105 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 2024-03-24T00:35:25.105 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:35:25.105 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4246015d20] 2024-03-24T00:35:25.106 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 2: gsignal() 2024-03-24T00:35:25.106 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 3: abort() 2024-03-24T00:35:25.106 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f424728e4fb] 2024-03-24T00:35:25.106 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f424728e667] 2024-03-24T00:35:25.106 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55d9ceb2d223] 2024-03-24T00:35:25.106 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55d9ce9a75d6] 2024-03-24T00:35:25.106 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55d9ce9a7da1] 2024-03-24T00:35:25.106 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 9: (Server::_unlink_local(boost::intrusive_ptr const&, CDentry*, CDentry*)+0xa95) [0x55d9ce9aa2d5] 2024-03-24T00:35:25.106 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55d9ce9fa233] 2024-03-24T00:35:25.106 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d9ce9fc197] 2024-03-24T00:35:25.106 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d9ce8fb11b] 2024-03-24T00:35:25.106 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d9ce8fb6bc] 2024-03-24T00:35:25.107 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55d9ce8e51ff] 2024-03-24T00:35:25.107 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f4247520488] 2024-03-24T00:35:25.107 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 16: (DispatchQueue::entry()+0x51f) [0x7f424751d60f] 2024-03-24T00:35:25.107 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f42475f0e81] 2024-03-24T00:35:25.107 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f424600b1ca] 2024-03-24T00:35:25.107 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 19: clone() 2024-03-24T00:35:25.107 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:35:25.107 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j[91395]: 2024-03-24T00:35:25.107 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:24 smithi078 podman[91433]: 2024-03-24 00:35:24.835510475 +0000 UTC m=+0.019622655 container died a79393ec3cf05f4ca97e98d3c53956c7cf103c3a739a30dd1a868477320b34a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-24T00:35:25.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:24 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6834/3027423976,v1:172.21.15.78:6835/3027423976] up:rejoin 2024-03-24T00:35:25.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:24 smithi106 ceph-mon[36505]: fsmap cephfs:3/3 {0=b=up:active,1=j=up:rejoin,2=h=up:active} 2 up:standby-replay 2024-03-24T00:35:25.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:24 smithi106 ceph-mon[36505]: daemon mds.j is now active in filesystem cephfs as rank 1 2024-03-24T00:35:25.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:24 smithi106 ceph-mon[36505]: pgmap v494: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 204 B/s wr, 4 op/s 2024-03-24T00:35:25.722 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 654 sec 2024-03-24T00:35:26.090 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:25 smithi078 podman[91433]: 2024-03-24 00:35:25.753108972 +0000 UTC m=+0.937221155 container remove a79393ec3cf05f4ca97e98d3c53956c7cf103c3a739a30dd1a868477320b34a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-j, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-24T00:35:26.091 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:25 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:35:26.091 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:25 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.j.service: Failed with result 'exit-code'. 2024-03-24T00:35:26.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:25 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:35:26.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:25 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.78:6834/3027423976,v1:172.21.15.78:6835/3027423976] up:active 2024-03-24T00:35:26.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:25 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=h=up:active} 2 up:standby-replay 2024-03-24T00:35:26.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:25 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:35:26.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:25 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.78:6834/3027423976,v1:172.21.15.78:6835/3027423976] up:active 2024-03-24T00:35:26.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:25 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=h=up:active} 2 up:standby-replay 2024-03-24T00:35:26.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:25 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:35:26.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:25 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.78:6834/3027423976,v1:172.21.15.78:6835/3027423976] up:active 2024-03-24T00:35:26.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:25 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=j=up:active,2=h=up:active} 2 up:standby-replay 2024-03-24T00:35:26.723 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 655 sec 2024-03-24T00:35:27.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:26 smithi078 ceph-mon[32429]: pgmap v495: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 27 KiB/s wr, 12 op/s 2024-03-24T00:35:27.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:26 smithi107 ceph-mon[36485]: pgmap v495: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 27 KiB/s wr, 12 op/s 2024-03-24T00:35:27.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:26 smithi106 ceph-mon[36505]: pgmap v495: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 27 KiB/s wr, 12 op/s 2024-03-24T00:35:27.723 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 656 sec 2024-03-24T00:35:28.723 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 657 sec 2024-03-24T00:35:29.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:29 smithi078 ceph-mon[32429]: pgmap v496: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 27 KiB/s wr, 12 op/s 2024-03-24T00:35:29.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:29 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:35:29.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:29 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:29.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:29 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:35:29.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:29 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:35:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:29 smithi107 ceph-mon[36485]: pgmap v496: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 27 KiB/s wr, 12 op/s 2024-03-24T00:35:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:29 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:35:29.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:29 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:29.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:29 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:35:29.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:29 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:35:29.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:29 smithi106 ceph-mon[36505]: pgmap v496: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 27 KiB/s wr, 12 op/s 2024-03-24T00:35:29.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:29 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:35:29.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:29 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:29.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:29 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:35:29.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:29 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:35:29.723 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 658 sec 2024-03-24T00:35:30.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:30 smithi078 ceph-mon[32429]: pgmap v497: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 31 KiB/s wr, 14 op/s 2024-03-24T00:35:30.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:30 smithi078 ceph-mon[32429]: pgmap v498: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 33 KiB/s wr, 12 op/s 2024-03-24T00:35:30.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:30 smithi107 ceph-mon[36485]: pgmap v497: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 31 KiB/s wr, 14 op/s 2024-03-24T00:35:30.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:30 smithi107 ceph-mon[36485]: pgmap v498: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 33 KiB/s wr, 12 op/s 2024-03-24T00:35:30.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:30 smithi106 ceph-mon[36505]: pgmap v497: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 31 KiB/s wr, 14 op/s 2024-03-24T00:35:30.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:30 smithi106 ceph-mon[36505]: pgmap v498: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 33 KiB/s wr, 12 op/s 2024-03-24T00:35:30.723 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 659 sec 2024-03-24T00:35:31.723 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 660 sec 2024-03-24T00:35:32.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:32 smithi078 ceph-mon[32429]: pgmap v499: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 31 KiB/s wr, 14 op/s 2024-03-24T00:35:32.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:32 smithi107 ceph-mon[36485]: pgmap v499: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 31 KiB/s wr, 14 op/s 2024-03-24T00:35:32.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:32 smithi106 ceph-mon[36505]: pgmap v499: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 31 KiB/s wr, 14 op/s 2024-03-24T00:35:32.723 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 661 sec 2024-03-24T00:35:33.723 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 662 sec 2024-03-24T00:35:34.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:34 smithi078 ceph-mon[32429]: pgmap v500: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 31 KiB/s wr, 14 op/s 2024-03-24T00:35:34.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:34 smithi078 ceph-mon[32429]: osdmap e155: 12 total, 12 up, 12 in 2024-03-24T00:35:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:34 smithi107 ceph-mon[36485]: pgmap v500: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 31 KiB/s wr, 14 op/s 2024-03-24T00:35:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:34 smithi107 ceph-mon[36485]: osdmap e155: 12 total, 12 up, 12 in 2024-03-24T00:35:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:34 smithi106 ceph-mon[36505]: pgmap v500: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 31 KiB/s wr, 14 op/s 2024-03-24T00:35:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:34 smithi106 ceph-mon[36505]: osdmap e155: 12 total, 12 up, 12 in 2024-03-24T00:35:34.723 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 663 sec 2024-03-24T00:35:35.723 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 664 sec 2024-03-24T00:35:36.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:36 smithi078 ceph-mon[32429]: pgmap v502: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 151 B/s wr, 3 op/s 2024-03-24T00:35:36.591 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:36 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.j.service: Scheduled restart job, restart counter is at 6. 2024-03-24T00:35:36.591 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:36 smithi078 systemd[1]: Stopped Ceph mds.j for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:35:36.591 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:36 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.j.service: Start request repeated too quickly. 2024-03-24T00:35:36.591 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:36 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.j.service: Failed with result 'exit-code'. 2024-03-24T00:35:36.591 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 24 00:35:36 smithi078 systemd[1]: Failed to start Ceph mds.j for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T00:35:36.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:36 smithi107 ceph-mon[36485]: pgmap v502: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 151 B/s wr, 3 op/s 2024-03-24T00:35:36.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:36 smithi106 ceph-mon[36505]: pgmap v502: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 151 B/s wr, 3 op/s 2024-03-24T00:35:36.723 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 665 sec 2024-03-24T00:35:37.723 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 666 sec 2024-03-24T00:35:38.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:38 smithi078 ceph-mon[32429]: pgmap v503: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-24T00:35:38.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:38 smithi107 ceph-mon[36485]: pgmap v503: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-24T00:35:38.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:38 smithi106 ceph-mon[36505]: pgmap v503: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-24T00:35:38.723 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 667 sec 2024-03-24T00:35:39.724 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 668 sec 2024-03-24T00:35:40.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:40 smithi106 ceph-mon[36505]: pgmap v504: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 204 B/s wr, 4 op/s 2024-03-24T00:35:40.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:40 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=j=up:active(laggy or crashed),2=h=up:active} 2 up:standby-replay 2024-03-24T00:35:40.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:40 smithi106 ceph-mon[36505]: osdmap e156: 12 total, 12 up, 12 in 2024-03-24T00:35:40.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:40 smithi078 ceph-mon[32429]: pgmap v504: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 204 B/s wr, 4 op/s 2024-03-24T00:35:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:40 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=j=up:active(laggy or crashed),2=h=up:active} 2 up:standby-replay 2024-03-24T00:35:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:40 smithi078 ceph-mon[32429]: osdmap e156: 12 total, 12 up, 12 in 2024-03-24T00:35:40.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:40 smithi107 ceph-mon[36485]: pgmap v504: 129 pgs: 129 active+clean; 27 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 204 B/s wr, 4 op/s 2024-03-24T00:35:40.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:40 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=j=up:active(laggy or crashed),2=h=up:active} 2 up:standby-replay 2024-03-24T00:35:40.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:40 smithi107 ceph-mon[36485]: osdmap e156: 12 total, 12 up, 12 in 2024-03-24T00:35:40.723 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 669 sec 2024-03-24T00:35:41.724 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 670 sec 2024-03-24T00:35:42.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:42 smithi106 ceph-mon[36505]: pgmap v506: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-24T00:35:42.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:42 smithi078 ceph-mon[32429]: pgmap v506: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-24T00:35:42.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:42 smithi107 ceph-mon[36485]: pgmap v506: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-24T00:35:42.724 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 671 sec 2024-03-24T00:35:43.724 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 672 sec 2024-03-24T00:35:44.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:44 smithi078 ceph-mon[32429]: pgmap v507: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 211 B/s wr, 4 op/s 2024-03-24T00:35:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:44 smithi107 ceph-mon[36485]: pgmap v507: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 211 B/s wr, 4 op/s 2024-03-24T00:35:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:44 smithi106 ceph-mon[36505]: pgmap v507: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 211 B/s wr, 4 op/s 2024-03-24T00:35:44.724 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 673 sec 2024-03-24T00:35:45.724 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 674 sec 2024-03-24T00:35:46.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:46 smithi078 ceph-mon[32429]: pgmap v508: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 204 B/s wr, 4 op/s 2024-03-24T00:35:46.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:46 smithi107 ceph-mon[36485]: pgmap v508: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 204 B/s wr, 4 op/s 2024-03-24T00:35:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:46 smithi106 ceph-mon[36505]: pgmap v508: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 204 B/s wr, 4 op/s 2024-03-24T00:35:46.724 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 675 sec 2024-03-24T00:35:47.724 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 676 sec 2024-03-24T00:35:48.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:48 smithi078 ceph-mon[32429]: pgmap v509: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 102 B/s wr, 4 op/s 2024-03-24T00:35:48.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:48 smithi107 ceph-mon[36485]: pgmap v509: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 102 B/s wr, 4 op/s 2024-03-24T00:35:48.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:48 smithi106 ceph-mon[36505]: pgmap v509: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 102 B/s wr, 4 op/s 2024-03-24T00:35:48.724 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 677 sec 2024-03-24T00:35:49.724 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 678 sec 2024-03-24T00:35:50.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:50 smithi078 ceph-mon[32429]: pgmap v510: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 102 B/s wr, 4 op/s 2024-03-24T00:35:50.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:50 smithi107 ceph-mon[36485]: pgmap v510: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 102 B/s wr, 4 op/s 2024-03-24T00:35:50.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:50 smithi106 ceph-mon[36505]: pgmap v510: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 102 B/s wr, 4 op/s 2024-03-24T00:35:50.724 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 679 sec 2024-03-24T00:35:51.724 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 680 sec 2024-03-24T00:35:52.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:52 smithi078 ceph-mon[32429]: pgmap v511: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 87 B/s wr, 5 op/s 2024-03-24T00:35:52.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:52 smithi107 ceph-mon[36485]: pgmap v511: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 87 B/s wr, 5 op/s 2024-03-24T00:35:52.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:52 smithi106 ceph-mon[36505]: pgmap v511: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 87 B/s wr, 5 op/s 2024-03-24T00:35:52.725 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 681 sec 2024-03-24T00:35:53.725 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 682 sec 2024-03-24T00:35:54.454 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:54 smithi106 ceph-mon[36505]: pgmap v512: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:35:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:54 smithi078 ceph-mon[32429]: pgmap v512: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:35:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:54 smithi107 ceph-mon[36485]: pgmap v512: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:35:54.725 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 683 sec 2024-03-24T00:35:55.725 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 684 sec 2024-03-24T00:35:56.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:56 smithi078 ceph-mon[32429]: pgmap v513: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:35:56.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:56 smithi107 ceph-mon[36485]: pgmap v513: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:35:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:56 smithi106 ceph-mon[36505]: pgmap v513: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:35:56.725 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 685 sec 2024-03-24T00:35:57.725 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 686 sec 2024-03-24T00:35:58.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:35:58 smithi078 ceph-mon[32429]: pgmap v514: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:35:58.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:35:58 smithi107 ceph-mon[36485]: pgmap v514: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:35:58.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:35:58 smithi106 ceph-mon[36505]: pgmap v514: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:35:58.725 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 687 sec 2024-03-24T00:35:59.725 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 688 sec 2024-03-24T00:36:00.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:00 smithi078 ceph-mon[32429]: pgmap v515: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:00.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:00 smithi107 ceph-mon[36485]: pgmap v515: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:00.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:00 smithi106 ceph-mon[36505]: pgmap v515: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:00.725 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 689 sec 2024-03-24T00:36:01.726 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 690 sec 2024-03-24T00:36:02.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:02 smithi078 ceph-mon[32429]: pgmap v516: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:36:02.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:02 smithi107 ceph-mon[36485]: pgmap v516: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:36:02.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:02 smithi106 ceph-mon[36505]: pgmap v516: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:36:02.726 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 691 sec 2024-03-24T00:36:03.726 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 692 sec 2024-03-24T00:36:04.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:04 smithi078 ceph-mon[32429]: pgmap v517: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-24T00:36:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:04 smithi107 ceph-mon[36485]: pgmap v517: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-24T00:36:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:04 smithi106 ceph-mon[36505]: pgmap v517: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-24T00:36:04.726 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 693 sec 2024-03-24T00:36:05.726 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 694 sec 2024-03-24T00:36:06.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:06 smithi078 ceph-mon[32429]: pgmap v518: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-24T00:36:06.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:06 smithi107 ceph-mon[36485]: pgmap v518: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-24T00:36:06.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:06 smithi106 ceph-mon[36505]: pgmap v518: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-24T00:36:06.726 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 695 sec 2024-03-24T00:36:07.726 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 696 sec 2024-03-24T00:36:08.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:08 smithi078 ceph-mon[32429]: pgmap v519: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:36:08.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:08 smithi107 ceph-mon[36485]: pgmap v519: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:36:08.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:08 smithi106 ceph-mon[36505]: pgmap v519: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:36:08.726 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 697 sec 2024-03-24T00:36:09.726 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 698 sec 2024-03-24T00:36:10.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:10 smithi078 ceph-mon[32429]: pgmap v520: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-24T00:36:10.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:10 smithi107 ceph-mon[36485]: pgmap v520: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-24T00:36:10.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:10 smithi106 ceph-mon[36505]: pgmap v520: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-24T00:36:10.726 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 699 sec 2024-03-24T00:36:11.726 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 700 sec 2024-03-24T00:36:12.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:12 smithi078 ceph-mon[32429]: pgmap v521: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:36:12.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:12 smithi107 ceph-mon[36485]: pgmap v521: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:36:12.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:12 smithi106 ceph-mon[36505]: pgmap v521: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:36:12.726 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 701 sec 2024-03-24T00:36:13.727 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 702 sec 2024-03-24T00:36:14.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:14 smithi078 ceph-mon[32429]: pgmap v522: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:14 smithi107 ceph-mon[36485]: pgmap v522: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:14 smithi106 ceph-mon[36505]: pgmap v522: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:14.727 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 703 sec 2024-03-24T00:36:15.727 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 704 sec 2024-03-24T00:36:16.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:16 smithi078 ceph-mon[32429]: pgmap v523: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:16.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:16 smithi107 ceph-mon[36485]: pgmap v523: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:16 smithi106 ceph-mon[36505]: pgmap v523: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:16.727 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 705 sec 2024-03-24T00:36:17.727 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 706 sec 2024-03-24T00:36:18.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:18 smithi078 ceph-mon[32429]: pgmap v524: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:18.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:18 smithi107 ceph-mon[36485]: pgmap v524: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:18.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:18 smithi106 ceph-mon[36505]: pgmap v524: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:18.727 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 707 sec 2024-03-24T00:36:19.727 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 708 sec 2024-03-24T00:36:20.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:20 smithi078 ceph-mon[32429]: pgmap v525: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:20.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:20 smithi107 ceph-mon[36485]: pgmap v525: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:20.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:20 smithi106 ceph-mon[36505]: pgmap v525: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:20.727 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 709 sec 2024-03-24T00:36:21.727 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 710 sec 2024-03-24T00:36:22.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:22 smithi078 ceph-mon[32429]: pgmap v526: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:22.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:22 smithi107 ceph-mon[36485]: pgmap v526: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:22.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:22 smithi106 ceph-mon[36505]: pgmap v526: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:22.727 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 711 sec 2024-03-24T00:36:23.727 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 712 sec 2024-03-24T00:36:24.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:24 smithi078 ceph-mon[32429]: pgmap v527: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:24 smithi107 ceph-mon[36485]: pgmap v527: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:24 smithi106 ceph-mon[36505]: pgmap v527: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:24.727 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 713 sec 2024-03-24T00:36:25.728 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 714 sec 2024-03-24T00:36:26.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:26 smithi078 ceph-mon[32429]: pgmap v528: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:26.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:26 smithi107 ceph-mon[36485]: pgmap v528: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:26.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:26 smithi106 ceph-mon[36505]: pgmap v528: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:26.728 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 715 sec 2024-03-24T00:36:27.728 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 716 sec 2024-03-24T00:36:28.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:28 smithi078 ceph-mon[32429]: pgmap v529: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:28.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:28 smithi107 ceph-mon[36485]: pgmap v529: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:28.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:28 smithi106 ceph-mon[36505]: pgmap v529: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:28.728 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 717 sec 2024-03-24T00:36:29.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:29 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:36:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:29 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:36:29.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:29 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:36:29.728 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 718 sec 2024-03-24T00:36:30.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:30 smithi078 ceph-mon[32429]: pgmap v530: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:30.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:30 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:36:30.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:30 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:36:30.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:30 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:36:30.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:30 smithi107 ceph-mon[36485]: pgmap v530: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:30.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:30 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:36:30.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:30 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:36:30.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:30 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:36:30.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:30 smithi106 ceph-mon[36505]: pgmap v530: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:30.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:30 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:36:30.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:30 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:36:30.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:30 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:36:30.728 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 719 sec 2024-03-24T00:36:31.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:31 smithi078 ceph-mon[32429]: pgmap v531: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:36:31.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:31 smithi078 ceph-mon[32429]: pgmap v532: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:36:31.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:31 smithi107 ceph-mon[36485]: pgmap v531: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:36:31.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:31 smithi107 ceph-mon[36485]: pgmap v532: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:36:31.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:31 smithi106 ceph-mon[36505]: pgmap v531: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:36:31.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:31 smithi106 ceph-mon[36505]: pgmap v532: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:36:31.728 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 720 sec 2024-03-24T00:36:32.728 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 721 sec 2024-03-24T00:36:33.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:33 smithi078 ceph-mon[32429]: pgmap v533: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:36:33.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:33 smithi107 ceph-mon[36485]: pgmap v533: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:36:33.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:33 smithi106 ceph-mon[36505]: pgmap v533: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:36:33.728 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 722 sec 2024-03-24T00:36:34.728 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 723 sec 2024-03-24T00:36:35.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:35 smithi078 ceph-mon[32429]: pgmap v534: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:36:35.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:35 smithi107 ceph-mon[36485]: pgmap v534: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:36:35.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:35 smithi106 ceph-mon[36505]: pgmap v534: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:36:35.728 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 724 sec 2024-03-24T00:36:36.728 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 725 sec 2024-03-24T00:36:37.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:37 smithi078 ceph-mon[32429]: pgmap v535: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:36:37.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:37 smithi107 ceph-mon[36485]: pgmap v535: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:36:37.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:37 smithi106 ceph-mon[36505]: pgmap v535: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:36:37.728 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 726 sec 2024-03-24T00:36:38.729 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 727 sec 2024-03-24T00:36:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:39 smithi078 ceph-mon[32429]: pgmap v536: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:36:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:39 smithi107 ceph-mon[36485]: pgmap v536: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:36:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:39 smithi106 ceph-mon[36505]: pgmap v536: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:36:39.729 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 728 sec 2024-03-24T00:36:40.729 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 729 sec 2024-03-24T00:36:41.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:41 smithi078 ceph-mon[32429]: pgmap v537: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:36:41.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:41 smithi107 ceph-mon[36485]: pgmap v537: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:36:41.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:41 smithi106 ceph-mon[36505]: pgmap v537: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:36:41.729 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 730 sec 2024-03-24T00:36:42.729 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 731 sec 2024-03-24T00:36:43.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:43 smithi078 ceph-mon[32429]: pgmap v538: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:43.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:43 smithi107 ceph-mon[36485]: pgmap v538: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:43.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:43 smithi106 ceph-mon[36505]: pgmap v538: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:43.729 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 732 sec 2024-03-24T00:36:44.729 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 733 sec 2024-03-24T00:36:45.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:45 smithi078 ceph-mon[32429]: pgmap v539: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:45.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:45 smithi107 ceph-mon[36485]: pgmap v539: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:45.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:45 smithi106 ceph-mon[36505]: pgmap v539: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:45.729 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 734 sec 2024-03-24T00:36:46.729 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 735 sec 2024-03-24T00:36:47.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:47 smithi078 ceph-mon[32429]: pgmap v540: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:47 smithi107 ceph-mon[36485]: pgmap v540: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:47.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:47 smithi106 ceph-mon[36505]: pgmap v540: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:47.729 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 736 sec 2024-03-24T00:36:48.729 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 737 sec 2024-03-24T00:36:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:49 smithi078 ceph-mon[32429]: pgmap v541: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:49 smithi107 ceph-mon[36485]: pgmap v541: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:49 smithi106 ceph-mon[36505]: pgmap v541: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:49.729 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 738 sec 2024-03-24T00:36:50.730 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 739 sec 2024-03-24T00:36:51.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:51 smithi078 ceph-mon[32429]: pgmap v542: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:51.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:51 smithi107 ceph-mon[36485]: pgmap v542: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:51.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:51 smithi106 ceph-mon[36505]: pgmap v542: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:51.730 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 740 sec 2024-03-24T00:36:52.730 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 741 sec 2024-03-24T00:36:53.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:53 smithi078 ceph-mon[32429]: pgmap v543: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:53.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:53 smithi107 ceph-mon[36485]: pgmap v543: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:53.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:53 smithi106 ceph-mon[36505]: pgmap v543: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:53.730 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 742 sec 2024-03-24T00:36:54.730 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 743 sec 2024-03-24T00:36:55.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:55 smithi078 ceph-mon[32429]: pgmap v544: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:55.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:55 smithi107 ceph-mon[36485]: pgmap v544: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:55.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:55 smithi106 ceph-mon[36505]: pgmap v544: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:55.730 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 744 sec 2024-03-24T00:36:56.730 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 745 sec 2024-03-24T00:36:57.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:57 smithi078 ceph-mon[32429]: pgmap v545: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:57.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:57 smithi107 ceph-mon[36485]: pgmap v545: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:57.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:57 smithi106 ceph-mon[36505]: pgmap v545: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:57.730 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 746 sec 2024-03-24T00:36:58.730 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 747 sec 2024-03-24T00:36:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:36:59 smithi078 ceph-mon[32429]: pgmap v546: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:36:59 smithi107 ceph-mon[36485]: pgmap v546: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:36:59 smithi106 ceph-mon[36505]: pgmap v546: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:36:59.730 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 748 sec 2024-03-24T00:37:00.730 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 749 sec 2024-03-24T00:37:01.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:01 smithi078 ceph-mon[32429]: pgmap v547: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:01.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:01 smithi107 ceph-mon[36485]: pgmap v547: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:01.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:01 smithi106 ceph-mon[36505]: pgmap v547: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:01.730 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 750 sec 2024-03-24T00:37:02.731 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 751 sec 2024-03-24T00:37:03.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:03 smithi078 ceph-mon[32429]: pgmap v548: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:03.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:03 smithi107 ceph-mon[36485]: pgmap v548: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:03.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:03 smithi106 ceph-mon[36505]: pgmap v548: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:03.731 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 752 sec 2024-03-24T00:37:04.731 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 753 sec 2024-03-24T00:37:05.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:05 smithi078 ceph-mon[32429]: pgmap v549: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:05.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:05 smithi107 ceph-mon[36485]: pgmap v549: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:05.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:05 smithi106 ceph-mon[36505]: pgmap v549: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:05.731 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 754 sec 2024-03-24T00:37:06.731 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 755 sec 2024-03-24T00:37:07.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:07 smithi078 ceph-mon[32429]: pgmap v550: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:07.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:07 smithi107 ceph-mon[36485]: pgmap v550: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:07.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:07 smithi106 ceph-mon[36505]: pgmap v550: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:07.731 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 756 sec 2024-03-24T00:37:08.731 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 757 sec 2024-03-24T00:37:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:09 smithi078 ceph-mon[32429]: pgmap v551: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:09 smithi107 ceph-mon[36485]: pgmap v551: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:09 smithi106 ceph-mon[36505]: pgmap v551: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:09.731 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 758 sec 2024-03-24T00:37:10.731 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 759 sec 2024-03-24T00:37:11.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:11 smithi078 ceph-mon[32429]: pgmap v552: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:11.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:11 smithi107 ceph-mon[36485]: pgmap v552: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:11.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:11 smithi106 ceph-mon[36505]: pgmap v552: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:11.731 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 760 sec 2024-03-24T00:37:12.731 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 761 sec 2024-03-24T00:37:13.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:13 smithi078 ceph-mon[32429]: pgmap v553: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:13.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:13 smithi107 ceph-mon[36485]: pgmap v553: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:13.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:13 smithi106 ceph-mon[36505]: pgmap v553: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:13.731 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 762 sec 2024-03-24T00:37:14.732 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 763 sec 2024-03-24T00:37:15.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:15 smithi078 ceph-mon[32429]: pgmap v554: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:15.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:15 smithi107 ceph-mon[36485]: pgmap v554: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:15.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:15 smithi106 ceph-mon[36505]: pgmap v554: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:15.732 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 764 sec 2024-03-24T00:37:16.732 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 765 sec 2024-03-24T00:37:17.564 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:17 smithi078 ceph-mon[32429]: pgmap v555: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:17.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:17 smithi107 ceph-mon[36485]: pgmap v555: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:17.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:17 smithi106 ceph-mon[36505]: pgmap v555: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:17.732 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 766 sec 2024-03-24T00:37:18.732 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 767 sec 2024-03-24T00:37:19.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:19 smithi078 ceph-mon[32429]: pgmap v556: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:19 smithi107 ceph-mon[36485]: pgmap v556: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:19 smithi106 ceph-mon[36505]: pgmap v556: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:19.732 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 768 sec 2024-03-24T00:37:20.732 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 769 sec 2024-03-24T00:37:21.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:21 smithi078 ceph-mon[32429]: pgmap v557: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:21.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:21 smithi107 ceph-mon[36485]: pgmap v557: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:21.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:21 smithi106 ceph-mon[36505]: pgmap v557: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:21.732 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 770 sec 2024-03-24T00:37:22.732 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 771 sec 2024-03-24T00:37:23.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:23 smithi078 ceph-mon[32429]: pgmap v558: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:23.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:23 smithi107 ceph-mon[36485]: pgmap v558: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:23.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:23 smithi106 ceph-mon[36505]: pgmap v558: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:23.733 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 772 sec 2024-03-24T00:37:24.732 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 773 sec 2024-03-24T00:37:25.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:25 smithi078 ceph-mon[32429]: pgmap v559: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:25.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:25 smithi107 ceph-mon[36485]: pgmap v559: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:25.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:25 smithi106 ceph-mon[36505]: pgmap v559: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:25.733 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 774 sec 2024-03-24T00:37:26.733 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 775 sec 2024-03-24T00:37:27.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:27 smithi078 ceph-mon[32429]: pgmap v560: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:27.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:27 smithi107 ceph-mon[36485]: pgmap v560: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:27.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:27 smithi106 ceph-mon[36505]: pgmap v560: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:27.733 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 776 sec 2024-03-24T00:37:28.733 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 777 sec 2024-03-24T00:37:29.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:29 smithi078 ceph-mon[32429]: pgmap v561: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:29 smithi107 ceph-mon[36485]: pgmap v561: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:29.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:29 smithi106 ceph-mon[36505]: pgmap v561: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:29.733 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 778 sec 2024-03-24T00:37:30.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:30 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:37:30.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:30 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:37:30.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:30 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:37:30.733 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 779 sec 2024-03-24T00:37:31.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:31 smithi078 ceph-mon[32429]: pgmap v562: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:31.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:31 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:37:31.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:31 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:37:31.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:31 smithi078 ceph-mon[32429]: pgmap v563: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T00:37:31.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:31 smithi078 ceph-mon[32429]: pgmap v564: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:37:31.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:31 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:37:31.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:31 smithi107 ceph-mon[36485]: pgmap v562: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:31.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:31 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:37:31.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:31 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:37:31.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:31 smithi107 ceph-mon[36485]: pgmap v563: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T00:37:31.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:31 smithi107 ceph-mon[36485]: pgmap v564: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:37:31.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:31 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:37:31.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:31 smithi106 ceph-mon[36505]: pgmap v562: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:31.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:31 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:37:31.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:31 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:37:31.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:31 smithi106 ceph-mon[36505]: pgmap v563: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T00:37:31.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:31 smithi106 ceph-mon[36505]: pgmap v564: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:37:31.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:31 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:37:31.733 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 780 sec 2024-03-24T00:37:32.733 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 781 sec 2024-03-24T00:37:33.733 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 782 sec 2024-03-24T00:37:33.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:33 smithi078 ceph-mon[32429]: pgmap v565: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:37:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:33 smithi107 ceph-mon[36485]: pgmap v565: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:37:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:33 smithi106 ceph-mon[36505]: pgmap v565: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:37:34.733 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 783 sec 2024-03-24T00:37:35.733 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 784 sec 2024-03-24T00:37:35.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:35 smithi078 ceph-mon[32429]: pgmap v566: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:37:35.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:35 smithi107 ceph-mon[36485]: pgmap v566: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:37:35.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:35 smithi106 ceph-mon[36505]: pgmap v566: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:37:36.733 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 785 sec 2024-03-24T00:37:37.734 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 786 sec 2024-03-24T00:37:37.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:37 smithi078 ceph-mon[32429]: pgmap v567: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T00:37:37.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:37 smithi107 ceph-mon[36485]: pgmap v567: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T00:37:37.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:37 smithi106 ceph-mon[36505]: pgmap v567: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T00:37:38.734 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 787 sec 2024-03-24T00:37:39.734 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 788 sec 2024-03-24T00:37:39.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:39 smithi078 ceph-mon[32429]: pgmap v568: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T00:37:39.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:39 smithi107 ceph-mon[36485]: pgmap v568: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T00:37:39.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:39 smithi106 ceph-mon[36505]: pgmap v568: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T00:37:40.734 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 789 sec 2024-03-24T00:37:41.734 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 790 sec 2024-03-24T00:37:41.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:41 smithi078 ceph-mon[32429]: pgmap v569: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:37:41.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:41 smithi107 ceph-mon[36485]: pgmap v569: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:37:41.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:41 smithi106 ceph-mon[36505]: pgmap v569: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:37:42.734 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 791 sec 2024-03-24T00:37:43.734 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 792 sec 2024-03-24T00:37:43.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:43 smithi078 ceph-mon[32429]: pgmap v570: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:37:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:43 smithi107 ceph-mon[36485]: pgmap v570: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:37:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:43 smithi106 ceph-mon[36505]: pgmap v570: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:37:44.734 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 793 sec 2024-03-24T00:37:45.734 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 794 sec 2024-03-24T00:37:45.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:45 smithi078 ceph-mon[32429]: pgmap v571: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:37:45.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:45 smithi107 ceph-mon[36485]: pgmap v571: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:37:45.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:45 smithi106 ceph-mon[36505]: pgmap v571: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:37:46.734 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 795 sec 2024-03-24T00:37:47.734 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 796 sec 2024-03-24T00:37:47.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:47 smithi078 ceph-mon[32429]: pgmap v572: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:37:47.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:47 smithi107 ceph-mon[36485]: pgmap v572: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:37:47.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:47 smithi106 ceph-mon[36505]: pgmap v572: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:37:48.735 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 797 sec 2024-03-24T00:37:49.735 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 798 sec 2024-03-24T00:37:49.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:49 smithi078 ceph-mon[32429]: pgmap v573: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:49.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:49 smithi107 ceph-mon[36485]: pgmap v573: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:49.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:49 smithi106 ceph-mon[36505]: pgmap v573: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:50.735 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 799 sec 2024-03-24T00:37:51.735 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 800 sec 2024-03-24T00:37:51.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:51 smithi078 ceph-mon[32429]: pgmap v574: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:51.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:51 smithi107 ceph-mon[36485]: pgmap v574: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:51 smithi106 ceph-mon[36505]: pgmap v574: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:52.735 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 801 sec 2024-03-24T00:37:53.735 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 802 sec 2024-03-24T00:37:53.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:53 smithi078 ceph-mon[32429]: pgmap v575: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:53 smithi107 ceph-mon[36485]: pgmap v575: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:53.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:53 smithi106 ceph-mon[36505]: pgmap v575: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:54.735 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 803 sec 2024-03-24T00:37:55.735 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 804 sec 2024-03-24T00:37:55.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:55 smithi078 ceph-mon[32429]: pgmap v576: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:55.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:55 smithi107 ceph-mon[36485]: pgmap v576: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:55.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:55 smithi106 ceph-mon[36505]: pgmap v576: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:37:56.735 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 805 sec 2024-03-24T00:37:57.735 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 806 sec 2024-03-24T00:37:57.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:57 smithi078 ceph-mon[32429]: pgmap v577: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:37:57.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:57 smithi107 ceph-mon[36485]: pgmap v577: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:37:57.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:57 smithi106 ceph-mon[36505]: pgmap v577: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:37:58.735 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 807 sec 2024-03-24T00:37:59.735 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 808 sec 2024-03-24T00:37:59.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:37:59 smithi078 ceph-mon[32429]: pgmap v578: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:37:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:37:59 smithi107 ceph-mon[36485]: pgmap v578: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:37:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:37:59 smithi106 ceph-mon[36505]: pgmap v578: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:38:00.736 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 809 sec 2024-03-24T00:38:01.736 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 810 sec 2024-03-24T00:38:01.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:01 smithi078 ceph-mon[32429]: pgmap v579: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:38:01.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:01 smithi107 ceph-mon[36485]: pgmap v579: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:38:01.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:01 smithi106 ceph-mon[36505]: pgmap v579: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:38:02.735 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 811 sec 2024-03-24T00:38:03.736 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 812 sec 2024-03-24T00:38:03.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:03 smithi078 ceph-mon[32429]: pgmap v580: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:38:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:03 smithi107 ceph-mon[36485]: pgmap v580: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:38:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:03 smithi106 ceph-mon[36505]: pgmap v580: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:38:04.736 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 813 sec 2024-03-24T00:38:05.736 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 814 sec 2024-03-24T00:38:05.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:05 smithi078 ceph-mon[32429]: pgmap v581: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:38:05.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:05 smithi107 ceph-mon[36485]: pgmap v581: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:38:05.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:05 smithi106 ceph-mon[36505]: pgmap v581: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:38:06.736 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 815 sec 2024-03-24T00:38:07.736 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 816 sec 2024-03-24T00:38:07.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:07 smithi078 ceph-mon[32429]: pgmap v582: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:38:07.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:07 smithi107 ceph-mon[36485]: pgmap v582: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:38:07.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:07 smithi106 ceph-mon[36505]: pgmap v582: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:38:08.736 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 817 sec 2024-03-24T00:38:09.736 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 818 sec 2024-03-24T00:38:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:09 smithi078 ceph-mon[32429]: pgmap v583: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:09.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:09 smithi107 ceph-mon[36485]: pgmap v583: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:09.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:09 smithi106 ceph-mon[36505]: pgmap v583: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:10.737 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 819 sec 2024-03-24T00:38:11.736 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 820 sec 2024-03-24T00:38:11.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:11 smithi078 ceph-mon[32429]: pgmap v584: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:11 smithi107 ceph-mon[36485]: pgmap v584: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:11.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:11 smithi106 ceph-mon[36505]: pgmap v584: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:12.737 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 821 sec 2024-03-24T00:38:13.737 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 822 sec 2024-03-24T00:38:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:13 smithi078 ceph-mon[32429]: pgmap v585: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:13 smithi107 ceph-mon[36485]: pgmap v585: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:13 smithi106 ceph-mon[36505]: pgmap v585: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:14.737 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 823 sec 2024-03-24T00:38:15.737 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 824 sec 2024-03-24T00:38:15.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:15 smithi078 ceph-mon[32429]: pgmap v586: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:15 smithi107 ceph-mon[36485]: pgmap v586: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:15 smithi106 ceph-mon[36505]: pgmap v586: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:16.737 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 825 sec 2024-03-24T00:38:17.737 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 826 sec 2024-03-24T00:38:17.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:17 smithi106 ceph-mon[36505]: pgmap v587: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:18.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:17 smithi078 ceph-mon[32429]: pgmap v587: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:18.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:17 smithi107 ceph-mon[36485]: pgmap v587: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:18.737 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 827 sec 2024-03-24T00:38:19.737 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 828 sec 2024-03-24T00:38:19.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:19 smithi078 ceph-mon[32429]: pgmap v588: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:19.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:19 smithi106 ceph-mon[36505]: pgmap v588: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:20.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:19 smithi107 ceph-mon[36485]: pgmap v588: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:20.737 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 829 sec 2024-03-24T00:38:21.737 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 830 sec 2024-03-24T00:38:21.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:21 smithi106 ceph-mon[36505]: pgmap v589: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:22.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:21 smithi078 ceph-mon[32429]: pgmap v589: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:22.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:21 smithi107 ceph-mon[36485]: pgmap v589: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:22.737 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 831 sec 2024-03-24T00:38:23.737 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 832 sec 2024-03-24T00:38:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:23 smithi106 ceph-mon[36505]: pgmap v590: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:23 smithi078 ceph-mon[32429]: pgmap v590: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:23 smithi107 ceph-mon[36485]: pgmap v590: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:24.737 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 833 sec 2024-03-24T00:38:25.738 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 834 sec 2024-03-24T00:38:25.896 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:25 smithi107 ceph-mon[36485]: pgmap v591: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:25.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:25 smithi106 ceph-mon[36505]: pgmap v591: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:26.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:25 smithi078 ceph-mon[32429]: pgmap v591: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:26.738 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 835 sec 2024-03-24T00:38:27.738 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 836 sec 2024-03-24T00:38:27.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:27 smithi106 ceph-mon[36505]: pgmap v592: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:28.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:27 smithi078 ceph-mon[32429]: pgmap v592: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:27 smithi107 ceph-mon[36485]: pgmap v592: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:28.738 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 837 sec 2024-03-24T00:38:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:28 smithi106 ceph-mon[36505]: 1 slow requests, 1 included below; oldest blocked for > 183.111027 secs 2024-03-24T00:38:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:28 smithi106 ceph-mon[36505]: slow request 183.111027 seconds old, received at 2024-03-24T00:35:24.775244+0000: peer_request:client.15462:3775 currently acquired locks 2024-03-24T00:38:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:28 smithi078 ceph-mon[32429]: 1 slow requests, 1 included below; oldest blocked for > 183.111027 secs 2024-03-24T00:38:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:28 smithi078 ceph-mon[32429]: slow request 183.111027 seconds old, received at 2024-03-24T00:35:24.775244+0000: peer_request:client.15462:3775 currently acquired locks 2024-03-24T00:38:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:28 smithi107 ceph-mon[36485]: 1 slow requests, 1 included below; oldest blocked for > 183.111027 secs 2024-03-24T00:38:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:28 smithi107 ceph-mon[36485]: slow request 183.111027 seconds old, received at 2024-03-24T00:35:24.775244+0000: peer_request:client.15462:3775 currently acquired locks 2024-03-24T00:38:29.738 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 838 sec 2024-03-24T00:38:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:29 smithi106 ceph-mon[36505]: pgmap v593: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:29 smithi106 ceph-mon[36505]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-03-24T00:38:29.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:29 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] up:active 2024-03-24T00:38:29.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:29 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=j=up:active(laggy or crashed),2=h=up:active} 2 up:standby-replay 2024-03-24T00:38:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:29 smithi078 ceph-mon[32429]: pgmap v593: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:30.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:29 smithi078 ceph-mon[32429]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-03-24T00:38:30.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:29 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] up:active 2024-03-24T00:38:30.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:29 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=j=up:active(laggy or crashed),2=h=up:active} 2 up:standby-replay 2024-03-24T00:38:30.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:29 smithi107 ceph-mon[36485]: pgmap v593: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:30.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:29 smithi107 ceph-mon[36485]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-03-24T00:38:30.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:29 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] up:active 2024-03-24T00:38:30.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:29 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=j=up:active(laggy or crashed),2=h=up:active} 2 up:standby-replay 2024-03-24T00:38:30.738 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 839 sec 2024-03-24T00:38:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:30 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:38:31.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:30 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:38:31.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:30 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:38:31.738 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 840 sec 2024-03-24T00:38:31.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:31 smithi106 ceph-mon[36505]: pgmap v594: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:31.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:31 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:38:31.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:31 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:38:31.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:31 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:38:32.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:31 smithi078 ceph-mon[32429]: pgmap v594: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:32.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:31 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:38:32.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:31 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:38:32.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:31 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:38:32.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:31 smithi107 ceph-mon[36485]: pgmap v594: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:32.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:31 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:38:32.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:31 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:38:32.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:31 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:38:32.738 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 841 sec 2024-03-24T00:38:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:32 smithi106 ceph-mon[36505]: pgmap v595: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:38:32.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:32 smithi106 ceph-mon[36505]: pgmap v596: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:38:33.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:32 smithi078 ceph-mon[32429]: pgmap v595: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:38:33.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:32 smithi078 ceph-mon[32429]: pgmap v596: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:38:33.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:32 smithi107 ceph-mon[36485]: pgmap v595: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:38:33.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:32 smithi107 ceph-mon[36485]: pgmap v596: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:38:33.738 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 842 sec 2024-03-24T00:38:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:33 smithi106 ceph-mon[36505]: 1 slow requests, 0 included below; oldest blocked for > 188.111089 secs 2024-03-24T00:38:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:33 smithi078 ceph-mon[32429]: 1 slow requests, 0 included below; oldest blocked for > 188.111089 secs 2024-03-24T00:38:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:33 smithi107 ceph-mon[36485]: 1 slow requests, 0 included below; oldest blocked for > 188.111089 secs 2024-03-24T00:38:34.738 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 843 sec 2024-03-24T00:38:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:34 smithi106 ceph-mon[36505]: pgmap v597: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:38:35.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:34 smithi078 ceph-mon[32429]: pgmap v597: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:38:35.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:34 smithi107 ceph-mon[36485]: pgmap v597: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:38:35.738 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 844 sec 2024-03-24T00:38:35.992 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:35 smithi078 ceph-mon[32429]: pgmap v598: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:38:36.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:35 smithi107 ceph-mon[36485]: pgmap v598: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:38:36.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:35 smithi106 ceph-mon[36505]: pgmap v598: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:38:36.739 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 845 sec 2024-03-24T00:38:37.739 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 846 sec 2024-03-24T00:38:38.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:38 smithi078 ceph-mon[32429]: pgmap v599: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:38:38.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:38 smithi078 ceph-mon[32429]: 1 slow requests, 0 included below; oldest blocked for > 193.111151 secs 2024-03-24T00:38:38.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:38 smithi107 ceph-mon[36485]: pgmap v599: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:38:38.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:38 smithi107 ceph-mon[36485]: 1 slow requests, 0 included below; oldest blocked for > 193.111151 secs 2024-03-24T00:38:38.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:38 smithi106 ceph-mon[36505]: pgmap v599: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:38:38.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:38 smithi106 ceph-mon[36505]: 1 slow requests, 0 included below; oldest blocked for > 193.111151 secs 2024-03-24T00:38:38.739 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 847 sec 2024-03-24T00:38:39.739 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 848 sec 2024-03-24T00:38:40.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:40 smithi078 ceph-mon[32429]: pgmap v600: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:38:40.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:40 smithi107 ceph-mon[36485]: pgmap v600: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:38:40.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:40 smithi106 ceph-mon[36505]: pgmap v600: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:38:40.739 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 849 sec 2024-03-24T00:38:41.739 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 850 sec 2024-03-24T00:38:42.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:42 smithi078 ceph-mon[32429]: pgmap v601: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:38:42.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:42 smithi107 ceph-mon[36485]: pgmap v601: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:38:42.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:42 smithi106 ceph-mon[36505]: pgmap v601: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:38:42.739 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 851 sec 2024-03-24T00:38:43.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:43 smithi078 ceph-mon[32429]: 1 slow requests, 0 included below; oldest blocked for > 198.111238 secs 2024-03-24T00:38:43.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:43 smithi107 ceph-mon[36485]: 1 slow requests, 0 included below; oldest blocked for > 198.111238 secs 2024-03-24T00:38:43.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:43 smithi106 ceph-mon[36505]: 1 slow requests, 0 included below; oldest blocked for > 198.111238 secs 2024-03-24T00:38:43.739 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 852 sec 2024-03-24T00:38:44.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:44 smithi078 ceph-mon[32429]: pgmap v602: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:44 smithi107 ceph-mon[36485]: pgmap v602: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:44.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:44 smithi106 ceph-mon[36505]: pgmap v602: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:44.739 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 853 sec 2024-03-24T00:38:45.739 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 854 sec 2024-03-24T00:38:46.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:46 smithi078 ceph-mon[32429]: pgmap v603: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:46.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:46 smithi107 ceph-mon[36485]: pgmap v603: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:46 smithi106 ceph-mon[36505]: pgmap v603: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:46.739 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 855 sec 2024-03-24T00:38:47.739 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 856 sec 2024-03-24T00:38:48.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:48 smithi078 ceph-mon[32429]: pgmap v604: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:48.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:48 smithi078 ceph-mon[32429]: 1 slow requests, 0 included below; oldest blocked for > 203.111299 secs 2024-03-24T00:38:48.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:48 smithi107 ceph-mon[36485]: pgmap v604: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:48.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:48 smithi107 ceph-mon[36485]: 1 slow requests, 0 included below; oldest blocked for > 203.111299 secs 2024-03-24T00:38:48.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:48 smithi106 ceph-mon[36505]: pgmap v604: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:48.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:48 smithi106 ceph-mon[36505]: 1 slow requests, 0 included below; oldest blocked for > 203.111299 secs 2024-03-24T00:38:48.739 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 857 sec 2024-03-24T00:38:49.740 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 858 sec 2024-03-24T00:38:50.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:50 smithi078 ceph-mon[32429]: pgmap v605: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:50.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:50 smithi107 ceph-mon[36485]: pgmap v605: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:50.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:50 smithi106 ceph-mon[36505]: pgmap v605: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:50.740 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 859 sec 2024-03-24T00:38:51.740 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 860 sec 2024-03-24T00:38:52.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:52 smithi078 ceph-mon[32429]: pgmap v606: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:52.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:52 smithi107 ceph-mon[36485]: pgmap v606: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:52.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:52 smithi106 ceph-mon[36505]: pgmap v606: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:52.740 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 861 sec 2024-03-24T00:38:53.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:53 smithi078 ceph-mon[32429]: 1 slow requests, 0 included below; oldest blocked for > 208.111358 secs 2024-03-24T00:38:53.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:53 smithi107 ceph-mon[36485]: 1 slow requests, 0 included below; oldest blocked for > 208.111358 secs 2024-03-24T00:38:53.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:53 smithi106 ceph-mon[36505]: 1 slow requests, 0 included below; oldest blocked for > 208.111358 secs 2024-03-24T00:38:53.741 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 862 sec 2024-03-24T00:38:54.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:54 smithi078 ceph-mon[32429]: pgmap v607: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:54 smithi107 ceph-mon[36485]: pgmap v607: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:54 smithi106 ceph-mon[36505]: pgmap v607: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:54.740 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 863 sec 2024-03-24T00:38:55.740 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 864 sec 2024-03-24T00:38:56.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:56 smithi078 ceph-mon[32429]: pgmap v608: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:56.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:56 smithi107 ceph-mon[36485]: pgmap v608: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:56 smithi106 ceph-mon[36505]: pgmap v608: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:38:56.740 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 865 sec 2024-03-24T00:38:57.740 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 866 sec 2024-03-24T00:38:58.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:58 smithi078 ceph-mon[32429]: pgmap v609: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:58.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:38:58 smithi078 ceph-mon[32429]: 1 slow requests, 0 included below; oldest blocked for > 213.111418 secs 2024-03-24T00:38:58.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:58 smithi107 ceph-mon[36485]: pgmap v609: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:58.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:38:58 smithi107 ceph-mon[36485]: 1 slow requests, 0 included below; oldest blocked for > 213.111418 secs 2024-03-24T00:38:58.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:58 smithi106 ceph-mon[36505]: pgmap v609: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:58.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:38:58 smithi106 ceph-mon[36505]: 1 slow requests, 0 included below; oldest blocked for > 213.111418 secs 2024-03-24T00:38:58.740 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 867 sec 2024-03-24T00:38:59.741 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 868 sec 2024-03-24T00:39:00.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:00 smithi078 ceph-mon[32429]: pgmap v610: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:00.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:00 smithi107 ceph-mon[36485]: pgmap v610: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:00.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:00 smithi106 ceph-mon[36505]: pgmap v610: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:00.741 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 869 sec 2024-03-24T00:39:01.741 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 870 sec 2024-03-24T00:39:02.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:02 smithi078 ceph-mon[32429]: pgmap v611: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:02.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:02 smithi107 ceph-mon[36485]: pgmap v611: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:02.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:02 smithi106 ceph-mon[36505]: pgmap v611: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:02.741 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 871 sec 2024-03-24T00:39:03.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:03 smithi106 ceph-mon[36505]: 2 slow requests, 1 included below; oldest blocked for > 218.111476 secs 2024-03-24T00:39:03.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:03 smithi106 ceph-mon[36505]: slow request 182.844404 seconds old, received at 2024-03-24T00:36:00.042315+0000: client_request(client.25385:166 mksnap #0x1//scheduled-2024-03-24-00_36_00_UTC 2024-03-24T00:36:00.042488+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-24T00:39:03.741 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 872 sec 2024-03-24T00:39:03.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:03 smithi078 ceph-mon[32429]: 2 slow requests, 1 included below; oldest blocked for > 218.111476 secs 2024-03-24T00:39:03.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:03 smithi078 ceph-mon[32429]: slow request 182.844404 seconds old, received at 2024-03-24T00:36:00.042315+0000: client_request(client.25385:166 mksnap #0x1//scheduled-2024-03-24-00_36_00_UTC 2024-03-24T00:36:00.042488+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-24T00:39:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:03 smithi107 ceph-mon[36485]: 2 slow requests, 1 included below; oldest blocked for > 218.111476 secs 2024-03-24T00:39:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:03 smithi107 ceph-mon[36485]: slow request 182.844404 seconds old, received at 2024-03-24T00:36:00.042315+0000: client_request(client.25385:166 mksnap #0x1//scheduled-2024-03-24-00_36_00_UTC 2024-03-24T00:36:00.042488+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-24T00:39:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:04 smithi106 ceph-mon[36505]: pgmap v612: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:04.741 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 873 sec 2024-03-24T00:39:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:04 smithi078 ceph-mon[32429]: pgmap v612: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:04 smithi107 ceph-mon[36485]: pgmap v612: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:05.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:05 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] up:active 2024-03-24T00:39:05.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:05 smithi106 ceph-mon[36505]: fsmap cephfs:3 {0=b=up:active,1=j=up:active(laggy or crashed),2=h=up:active} 2 up:standby-replay 2024-03-24T00:39:05.741 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 874 sec 2024-03-24T00:39:05.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:05 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] up:active 2024-03-24T00:39:05.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:05 smithi078 ceph-mon[32429]: fsmap cephfs:3 {0=b=up:active,1=j=up:active(laggy or crashed),2=h=up:active} 2 up:standby-replay 2024-03-24T00:39:05.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:05 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6838/2482071304,v1:172.21.15.106:6839/2482071304] up:active 2024-03-24T00:39:05.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:05 smithi107 ceph-mon[36485]: fsmap cephfs:3 {0=b=up:active,1=j=up:active(laggy or crashed),2=h=up:active} 2 up:standby-replay 2024-03-24T00:39:06.620 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:06 smithi078 ceph-mon[32429]: pgmap v613: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:06.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:06 smithi106 ceph-mon[36505]: pgmap v613: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:06.741 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 875 sec 2024-03-24T00:39:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:06 smithi107 ceph-mon[36485]: pgmap v613: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:07.742 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 876 sec 2024-03-24T00:39:08.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:08 smithi106 ceph-mon[36505]: pgmap v614: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:08.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 223.111539 secs 2024-03-24T00:39:08.742 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 877 sec 2024-03-24T00:39:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:08 smithi078 ceph-mon[32429]: pgmap v614: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 223.111539 secs 2024-03-24T00:39:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:08 smithi107 ceph-mon[36485]: pgmap v614: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 223.111539 secs 2024-03-24T00:39:09.742 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 878 sec 2024-03-24T00:39:10.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:10 smithi106 ceph-mon[36505]: pgmap v615: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:10.742 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 879 sec 2024-03-24T00:39:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:10 smithi078 ceph-mon[32429]: pgmap v615: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:10 smithi107 ceph-mon[36485]: pgmap v615: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:11.742 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 880 sec 2024-03-24T00:39:12.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:12 smithi106 ceph-mon[36505]: pgmap v616: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:12.742 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 881 sec 2024-03-24T00:39:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:12 smithi078 ceph-mon[32429]: pgmap v616: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:12 smithi107 ceph-mon[36485]: pgmap v616: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:13.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 228.111576 secs 2024-03-24T00:39:13.742 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 882 sec 2024-03-24T00:39:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 228.111576 secs 2024-03-24T00:39:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 228.111576 secs 2024-03-24T00:39:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:14 smithi106 ceph-mon[36505]: pgmap v617: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:14.742 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 883 sec 2024-03-24T00:39:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:14 smithi078 ceph-mon[32429]: pgmap v617: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:14 smithi107 ceph-mon[36485]: pgmap v617: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:15.742 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 884 sec 2024-03-24T00:39:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:16 smithi106 ceph-mon[36505]: pgmap v618: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:16.742 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 885 sec 2024-03-24T00:39:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:16 smithi078 ceph-mon[32429]: pgmap v618: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:16 smithi107 ceph-mon[36485]: pgmap v618: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:17.742 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 886 sec 2024-03-24T00:39:18.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:18 smithi106 ceph-mon[36505]: pgmap v619: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:18.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 233.111659 secs 2024-03-24T00:39:18.743 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 887 sec 2024-03-24T00:39:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:18 smithi078 ceph-mon[32429]: pgmap v619: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 233.111659 secs 2024-03-24T00:39:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:18 smithi107 ceph-mon[36485]: pgmap v619: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 233.111659 secs 2024-03-24T00:39:19.743 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 888 sec 2024-03-24T00:39:20.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:20 smithi106 ceph-mon[36505]: pgmap v620: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:20.743 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 889 sec 2024-03-24T00:39:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:20 smithi078 ceph-mon[32429]: pgmap v620: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:20 smithi107 ceph-mon[36485]: pgmap v620: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:21.743 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 890 sec 2024-03-24T00:39:22.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:22 smithi106 ceph-mon[36505]: pgmap v621: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:22.743 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 891 sec 2024-03-24T00:39:22.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:22 smithi078 ceph-mon[32429]: pgmap v621: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:22.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:22 smithi107 ceph-mon[36485]: pgmap v621: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:23.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 238.111743 secs 2024-03-24T00:39:23.743 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 892 sec 2024-03-24T00:39:23.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 238.111743 secs 2024-03-24T00:39:23.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 238.111743 secs 2024-03-24T00:39:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:24 smithi106 ceph-mon[36505]: pgmap v622: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:24.743 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 893 sec 2024-03-24T00:39:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:24 smithi078 ceph-mon[32429]: pgmap v622: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:24 smithi107 ceph-mon[36485]: pgmap v622: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:25.743 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 894 sec 2024-03-24T00:39:26.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:26 smithi106 ceph-mon[36505]: pgmap v623: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:26.743 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 895 sec 2024-03-24T00:39:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:26 smithi078 ceph-mon[32429]: pgmap v623: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:26 smithi107 ceph-mon[36485]: pgmap v623: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:27.743 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 896 sec 2024-03-24T00:39:28.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:28 smithi106 ceph-mon[36505]: pgmap v624: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:28.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 243.111828 secs 2024-03-24T00:39:28.743 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 897 sec 2024-03-24T00:39:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:28 smithi078 ceph-mon[32429]: pgmap v624: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:28.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 243.111828 secs 2024-03-24T00:39:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:28 smithi107 ceph-mon[36485]: pgmap v624: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 243.111828 secs 2024-03-24T00:39:29.744 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 898 sec 2024-03-24T00:39:30.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:30 smithi106 ceph-mon[36505]: pgmap v625: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:30.744 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 899 sec 2024-03-24T00:39:30.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:30 smithi078 ceph-mon[32429]: pgmap v625: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:30 smithi107 ceph-mon[36485]: pgmap v625: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:31.744 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 900 sec 2024-03-24T00:39:31.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:31 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:39:31.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:31 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:39:31.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:31 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:39:32.744 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 901 sec 2024-03-24T00:39:32.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:32 smithi078 ceph-mon[32429]: pgmap v626: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:32.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:32 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:39:32.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:32 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:39:32.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:32 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:39:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:32 smithi107 ceph-mon[36485]: pgmap v626: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:32 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:39:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:32 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:39:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:32 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:39:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:32 smithi106 ceph-mon[36505]: pgmap v626: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:32.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:32 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:39:32.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:32 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:39:32.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:32 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:39:33.744 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 902 sec 2024-03-24T00:39:33.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:33 smithi078 ceph-mon[32429]: pgmap v627: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:39:33.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:33 smithi078 ceph-mon[32429]: pgmap v628: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:39:33.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 248.111888 secs 2024-03-24T00:39:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:33 smithi107 ceph-mon[36485]: pgmap v627: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:39:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:33 smithi107 ceph-mon[36485]: pgmap v628: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:39:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 248.111888 secs 2024-03-24T00:39:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:33 smithi106 ceph-mon[36505]: pgmap v627: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:39:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:33 smithi106 ceph-mon[36505]: pgmap v628: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:39:33.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 248.111888 secs 2024-03-24T00:39:34.744 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 903 sec 2024-03-24T00:39:35.744 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 904 sec 2024-03-24T00:39:35.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:35 smithi078 ceph-mon[32429]: pgmap v629: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:39:35.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:35 smithi107 ceph-mon[36485]: pgmap v629: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:39:35.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:35 smithi106 ceph-mon[36505]: pgmap v629: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:39:36.744 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 905 sec 2024-03-24T00:39:37.744 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 906 sec 2024-03-24T00:39:37.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:37 smithi078 ceph-mon[32429]: pgmap v630: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:39:37.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:37 smithi107 ceph-mon[36485]: pgmap v630: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:39:37.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:37 smithi106 ceph-mon[36505]: pgmap v630: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:39:38.744 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 907 sec 2024-03-24T00:39:38.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 253.111925 secs 2024-03-24T00:39:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 253.111925 secs 2024-03-24T00:39:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 253.111925 secs 2024-03-24T00:39:39.744 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 908 sec 2024-03-24T00:39:39.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:39 smithi078 ceph-mon[32429]: pgmap v631: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:39:39.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:39 smithi107 ceph-mon[36485]: pgmap v631: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:39:39.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:39 smithi106 ceph-mon[36505]: pgmap v631: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:39:40.744 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 909 sec 2024-03-24T00:39:41.745 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 910 sec 2024-03-24T00:39:41.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:41 smithi078 ceph-mon[32429]: pgmap v632: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:39:41.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:41 smithi107 ceph-mon[36485]: pgmap v632: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:39:41.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:41 smithi106 ceph-mon[36505]: pgmap v632: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:39:42.745 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 911 sec 2024-03-24T00:39:43.745 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 912 sec 2024-03-24T00:39:43.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:43 smithi078 ceph-mon[32429]: pgmap v633: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:39:43.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 258.112006 secs 2024-03-24T00:39:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:43 smithi107 ceph-mon[36485]: pgmap v633: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:39:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 258.112006 secs 2024-03-24T00:39:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:43 smithi106 ceph-mon[36505]: pgmap v633: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:39:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 258.112006 secs 2024-03-24T00:39:44.745 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 913 sec 2024-03-24T00:39:45.745 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 914 sec 2024-03-24T00:39:45.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:45 smithi078 ceph-mon[32429]: pgmap v634: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:45.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:45 smithi107 ceph-mon[36485]: pgmap v634: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:45.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:45 smithi106 ceph-mon[36505]: pgmap v634: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:46.745 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 915 sec 2024-03-24T00:39:47.745 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 916 sec 2024-03-24T00:39:47.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:47 smithi078 ceph-mon[32429]: pgmap v635: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:47.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:47 smithi107 ceph-mon[36485]: pgmap v635: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:47.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:47 smithi106 ceph-mon[36505]: pgmap v635: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:48.745 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 917 sec 2024-03-24T00:39:48.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 263.112090 secs 2024-03-24T00:39:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 263.112090 secs 2024-03-24T00:39:48.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 263.112090 secs 2024-03-24T00:39:49.745 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 918 sec 2024-03-24T00:39:49.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:49 smithi078 ceph-mon[32429]: pgmap v636: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:49.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:49 smithi107 ceph-mon[36485]: pgmap v636: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:49.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:49 smithi106 ceph-mon[36505]: pgmap v636: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:50.745 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 919 sec 2024-03-24T00:39:51.745 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 920 sec 2024-03-24T00:39:51.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:51 smithi078 ceph-mon[32429]: pgmap v637: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:51.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:51 smithi107 ceph-mon[36485]: pgmap v637: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:51 smithi106 ceph-mon[36505]: pgmap v637: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:52.745 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 921 sec 2024-03-24T00:39:53.746 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 922 sec 2024-03-24T00:39:53.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:53 smithi078 ceph-mon[32429]: pgmap v638: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:53.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 268.112149 secs 2024-03-24T00:39:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:53 smithi107 ceph-mon[36485]: pgmap v638: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 268.112149 secs 2024-03-24T00:39:53.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:53 smithi106 ceph-mon[36505]: pgmap v638: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:53.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 268.112149 secs 2024-03-24T00:39:54.746 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 923 sec 2024-03-24T00:39:55.746 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 924 sec 2024-03-24T00:39:55.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:55 smithi078 ceph-mon[32429]: pgmap v639: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:55.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:55 smithi107 ceph-mon[36485]: pgmap v639: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:55.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:55 smithi106 ceph-mon[36505]: pgmap v639: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:56.746 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 925 sec 2024-03-24T00:39:57.746 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 926 sec 2024-03-24T00:39:57.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:57 smithi078 ceph-mon[32429]: pgmap v640: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:57.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:57 smithi107 ceph-mon[36485]: pgmap v640: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:57.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:57 smithi106 ceph-mon[36505]: pgmap v640: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:58.746 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 927 sec 2024-03-24T00:39:58.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 273.112234 secs 2024-03-24T00:39:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 273.112234 secs 2024-03-24T00:39:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 273.112234 secs 2024-03-24T00:39:59.746 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 928 sec 2024-03-24T00:39:59.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:39:59 smithi078 ceph-mon[32429]: pgmap v641: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:39:59 smithi107 ceph-mon[36485]: pgmap v641: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:39:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:39:59 smithi106 ceph-mon[36505]: pgmap v641: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:00.746 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 929 sec 2024-03-24T00:40:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:00 smithi078 ceph-mon[32429]: Health detail: HEALTH_WARN 5 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T00:40:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:00 smithi078 ceph-mon[32429]: [WRN] CEPHADM_FAILED_DAEMON: 5 failed cephadm daemon(s) 2024-03-24T00:40:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:00 smithi078 ceph-mon[32429]: daemon mds.a on smithi078 is in unknown state 2024-03-24T00:40:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:00 smithi078 ceph-mon[32429]: daemon mds.k on smithi106 is in error state 2024-03-24T00:40:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:00 smithi078 ceph-mon[32429]: daemon mds.l on smithi107 is in error state 2024-03-24T00:40:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:00 smithi078 ceph-mon[32429]: daemon mds.f on smithi107 is in error state 2024-03-24T00:40:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:00 smithi078 ceph-mon[32429]: daemon mds.i on smithi107 is in error state 2024-03-24T00:40:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:00 smithi078 ceph-mon[32429]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-03-24T00:40:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:00 smithi078 ceph-mon[32429]: mds.b(mds.0): 2 slow requests are blocked > 180 secs 2024-03-24T00:40:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:00 smithi107 ceph-mon[36485]: Health detail: HEALTH_WARN 5 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T00:40:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:00 smithi107 ceph-mon[36485]: [WRN] CEPHADM_FAILED_DAEMON: 5 failed cephadm daemon(s) 2024-03-24T00:40:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:00 smithi107 ceph-mon[36485]: daemon mds.a on smithi078 is in unknown state 2024-03-24T00:40:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:00 smithi107 ceph-mon[36485]: daemon mds.k on smithi106 is in error state 2024-03-24T00:40:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:00 smithi107 ceph-mon[36485]: daemon mds.l on smithi107 is in error state 2024-03-24T00:40:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:00 smithi107 ceph-mon[36485]: daemon mds.f on smithi107 is in error state 2024-03-24T00:40:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:00 smithi107 ceph-mon[36485]: daemon mds.i on smithi107 is in error state 2024-03-24T00:40:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:00 smithi107 ceph-mon[36485]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-03-24T00:40:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:00 smithi107 ceph-mon[36485]: mds.b(mds.0): 2 slow requests are blocked > 180 secs 2024-03-24T00:40:00.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:00 smithi106 ceph-mon[36505]: Health detail: HEALTH_WARN 5 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T00:40:00.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:00 smithi106 ceph-mon[36505]: [WRN] CEPHADM_FAILED_DAEMON: 5 failed cephadm daemon(s) 2024-03-24T00:40:00.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:00 smithi106 ceph-mon[36505]: daemon mds.a on smithi078 is in unknown state 2024-03-24T00:40:00.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:00 smithi106 ceph-mon[36505]: daemon mds.k on smithi106 is in error state 2024-03-24T00:40:00.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:00 smithi106 ceph-mon[36505]: daemon mds.l on smithi107 is in error state 2024-03-24T00:40:00.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:00 smithi106 ceph-mon[36505]: daemon mds.f on smithi107 is in error state 2024-03-24T00:40:00.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:00 smithi106 ceph-mon[36505]: daemon mds.i on smithi107 is in error state 2024-03-24T00:40:00.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:00 smithi106 ceph-mon[36505]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-03-24T00:40:00.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:00 smithi106 ceph-mon[36505]: mds.b(mds.0): 2 slow requests are blocked > 180 secs 2024-03-24T00:40:01.746 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 930 sec 2024-03-24T00:40:01.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:01 smithi078 ceph-mon[32429]: pgmap v642: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:01.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:01 smithi107 ceph-mon[36485]: pgmap v642: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:01.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:01 smithi106 ceph-mon[36505]: pgmap v642: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:02.746 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 931 sec 2024-03-24T00:40:03.746 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 932 sec 2024-03-24T00:40:03.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:03 smithi078 ceph-mon[32429]: pgmap v643: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:03.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 278.112310 secs 2024-03-24T00:40:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:03 smithi107 ceph-mon[36485]: pgmap v643: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 278.112310 secs 2024-03-24T00:40:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:03 smithi106 ceph-mon[36505]: pgmap v643: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 278.112310 secs 2024-03-24T00:40:04.746 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 933 sec 2024-03-24T00:40:05.746 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 934 sec 2024-03-24T00:40:05.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:05 smithi078 ceph-mon[32429]: pgmap v644: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:05.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:05 smithi107 ceph-mon[36485]: pgmap v644: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:05.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:05 smithi106 ceph-mon[36505]: pgmap v644: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:06.747 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 935 sec 2024-03-24T00:40:07.747 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 936 sec 2024-03-24T00:40:07.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:07 smithi078 ceph-mon[32429]: pgmap v645: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:07.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:07 smithi107 ceph-mon[36485]: pgmap v645: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:07.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:07 smithi106 ceph-mon[36505]: pgmap v645: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:08.747 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 937 sec 2024-03-24T00:40:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 283.112356 secs 2024-03-24T00:40:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 283.112356 secs 2024-03-24T00:40:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 283.112356 secs 2024-03-24T00:40:09.747 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 938 sec 2024-03-24T00:40:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:09 smithi078 ceph-mon[32429]: pgmap v646: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:09.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:09 smithi107 ceph-mon[36485]: pgmap v646: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:09.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:09 smithi106 ceph-mon[36505]: pgmap v646: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:10.747 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 939 sec 2024-03-24T00:40:11.747 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 940 sec 2024-03-24T00:40:11.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:11 smithi078 ceph-mon[32429]: pgmap v647: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:11 smithi107 ceph-mon[36485]: pgmap v647: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:11.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:11 smithi106 ceph-mon[36505]: pgmap v647: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:12.747 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 941 sec 2024-03-24T00:40:13.747 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 942 sec 2024-03-24T00:40:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:13 smithi078 ceph-mon[32429]: pgmap v648: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:13.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 288.112412 secs 2024-03-24T00:40:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:13 smithi107 ceph-mon[36485]: pgmap v648: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 288.112412 secs 2024-03-24T00:40:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:13 smithi106 ceph-mon[36505]: pgmap v648: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 288.112412 secs 2024-03-24T00:40:14.747 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 943 sec 2024-03-24T00:40:15.747 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 944 sec 2024-03-24T00:40:15.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:15 smithi078 ceph-mon[32429]: pgmap v649: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:15 smithi107 ceph-mon[36485]: pgmap v649: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:15 smithi106 ceph-mon[36505]: pgmap v649: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:16.747 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 945 sec 2024-03-24T00:40:17.748 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 946 sec 2024-03-24T00:40:17.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:17 smithi078 ceph-mon[32429]: pgmap v650: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:17.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:17 smithi107 ceph-mon[36485]: pgmap v650: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:17.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:17 smithi106 ceph-mon[36505]: pgmap v650: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:18.748 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 947 sec 2024-03-24T00:40:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 293.112473 secs 2024-03-24T00:40:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 293.112473 secs 2024-03-24T00:40:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 293.112473 secs 2024-03-24T00:40:19.748 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 948 sec 2024-03-24T00:40:19.836 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:19 smithi078 ceph-mon[32429]: pgmap v651: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:19 smithi107 ceph-mon[36485]: pgmap v651: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:19.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:19 smithi106 ceph-mon[36505]: pgmap v651: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:20.748 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 949 sec 2024-03-24T00:40:21.748 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 950 sec 2024-03-24T00:40:21.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:21 smithi078 ceph-mon[32429]: pgmap v652: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:21 smithi107 ceph-mon[36485]: pgmap v652: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:21.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:21 smithi106 ceph-mon[36505]: pgmap v652: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:22.750 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 951 sec 2024-03-24T00:40:23.748 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 952 sec 2024-03-24T00:40:23.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:23 smithi078 ceph-mon[32429]: pgmap v653: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:23.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 298.112559 secs 2024-03-24T00:40:23.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:23 smithi107 ceph-mon[36485]: pgmap v653: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:23.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 298.112559 secs 2024-03-24T00:40:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:23 smithi106 ceph-mon[36505]: pgmap v653: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 298.112559 secs 2024-03-24T00:40:24.748 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 953 sec 2024-03-24T00:40:25.748 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 954 sec 2024-03-24T00:40:25.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:25 smithi078 ceph-mon[32429]: pgmap v654: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:25.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:25 smithi107 ceph-mon[36485]: pgmap v654: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:25.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:25 smithi106 ceph-mon[36505]: pgmap v654: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:26.748 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 955 sec 2024-03-24T00:40:27.748 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 956 sec 2024-03-24T00:40:27.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:27 smithi078 ceph-mon[32429]: pgmap v655: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:27.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:27 smithi107 ceph-mon[36485]: pgmap v655: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:27.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:27 smithi106 ceph-mon[36505]: pgmap v655: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:28.749 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 957 sec 2024-03-24T00:40:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 303.112649 secs 2024-03-24T00:40:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 303.112649 secs 2024-03-24T00:40:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 303.112649 secs 2024-03-24T00:40:29.749 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 958 sec 2024-03-24T00:40:29.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:29 smithi078 ceph-mon[32429]: pgmap v656: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:29 smithi107 ceph-mon[36485]: pgmap v656: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:29 smithi106 ceph-mon[36505]: pgmap v656: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:30.749 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 959 sec 2024-03-24T00:40:31.749 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 960 sec 2024-03-24T00:40:31.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:31 smithi078 ceph-mon[32429]: pgmap v657: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:31.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:31 smithi107 ceph-mon[36485]: pgmap v657: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:31.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:31 smithi106 ceph-mon[36505]: pgmap v657: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:32.749 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 961 sec 2024-03-24T00:40:32.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:32 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:40:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:32 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:40:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:32 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:40:33.749 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 962 sec 2024-03-24T00:40:33.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:33 smithi078 ceph-mon[32429]: pgmap v658: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:33.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:33 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:40:33.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:33 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:40:33.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:33 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:40:33.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 308.112713 secs 2024-03-24T00:40:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:33 smithi107 ceph-mon[36485]: pgmap v658: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:33 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:40:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:33 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:40:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:33 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:40:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 308.112713 secs 2024-03-24T00:40:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:33 smithi106 ceph-mon[36505]: pgmap v658: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:33 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:40:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:33 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:40:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:33 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:40:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 308.112713 secs 2024-03-24T00:40:34.749 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 963 sec 2024-03-24T00:40:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:34 smithi078 ceph-mon[32429]: pgmap v659: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:40:34.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:34 smithi078 ceph-mon[32429]: pgmap v660: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:40:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:34 smithi107 ceph-mon[36485]: pgmap v659: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:40:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:34 smithi107 ceph-mon[36485]: pgmap v660: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:40:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:34 smithi106 ceph-mon[36505]: pgmap v659: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:40:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:34 smithi106 ceph-mon[36505]: pgmap v660: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:40:35.749 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 964 sec 2024-03-24T00:40:36.749 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 965 sec 2024-03-24T00:40:36.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:36 smithi078 ceph-mon[32429]: pgmap v661: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:40:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:36 smithi107 ceph-mon[36485]: pgmap v661: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:40:36.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:36 smithi106 ceph-mon[36505]: pgmap v661: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:40:37.750 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 966 sec 2024-03-24T00:40:38.750 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 967 sec 2024-03-24T00:40:38.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:38 smithi078 ceph-mon[32429]: pgmap v662: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:40:38.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 313.112769 secs 2024-03-24T00:40:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:38 smithi107 ceph-mon[36485]: pgmap v662: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:40:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 313.112769 secs 2024-03-24T00:40:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:38 smithi106 ceph-mon[36505]: pgmap v662: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:40:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 313.112769 secs 2024-03-24T00:40:39.750 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 968 sec 2024-03-24T00:40:40.750 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 969 sec 2024-03-24T00:40:40.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:40 smithi078 ceph-mon[32429]: pgmap v663: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:40:40.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:40 smithi107 ceph-mon[36485]: pgmap v663: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:40:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:40 smithi106 ceph-mon[36505]: pgmap v663: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:40:41.750 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 970 sec 2024-03-24T00:40:42.750 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 971 sec 2024-03-24T00:40:42.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:42 smithi078 ceph-mon[32429]: pgmap v664: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:40:42.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:42 smithi107 ceph-mon[36485]: pgmap v664: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:40:42.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:42 smithi106 ceph-mon[36505]: pgmap v664: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:40:43.750 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 972 sec 2024-03-24T00:40:43.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 318.112852 secs 2024-03-24T00:40:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 318.112852 secs 2024-03-24T00:40:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 318.112852 secs 2024-03-24T00:40:44.750 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 973 sec 2024-03-24T00:40:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:44 smithi107 ceph-mon[36485]: pgmap v665: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:40:44.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:44 smithi106 ceph-mon[36505]: pgmap v665: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:40:45.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:44 smithi078 ceph-mon[32429]: pgmap v665: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:40:45.751 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 974 sec 2024-03-24T00:40:46.750 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 975 sec 2024-03-24T00:40:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:46 smithi106 ceph-mon[36505]: pgmap v666: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:47.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:46 smithi078 ceph-mon[32429]: pgmap v666: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:47.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:46 smithi107 ceph-mon[36485]: pgmap v666: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:47.750 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 976 sec 2024-03-24T00:40:48.750 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 977 sec 2024-03-24T00:40:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:48 smithi106 ceph-mon[36505]: pgmap v667: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:48.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 323.112943 secs 2024-03-24T00:40:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:48 smithi078 ceph-mon[32429]: pgmap v667: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:49.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 323.112943 secs 2024-03-24T00:40:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:48 smithi107 ceph-mon[36485]: pgmap v667: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 323.112943 secs 2024-03-24T00:40:49.751 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 978 sec 2024-03-24T00:40:50.751 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 979 sec 2024-03-24T00:40:50.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:50 smithi106 ceph-mon[36505]: pgmap v668: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:50 smithi078 ceph-mon[32429]: pgmap v668: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:51.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:50 smithi107 ceph-mon[36485]: pgmap v668: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:51.751 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 980 sec 2024-03-24T00:40:52.751 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 981 sec 2024-03-24T00:40:52.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:52 smithi106 ceph-mon[36505]: pgmap v669: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:53.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:52 smithi078 ceph-mon[32429]: pgmap v669: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:53.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:52 smithi107 ceph-mon[36485]: pgmap v669: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:53.751 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 982 sec 2024-03-24T00:40:53.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 328.113004 secs 2024-03-24T00:40:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 328.113004 secs 2024-03-24T00:40:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 328.113004 secs 2024-03-24T00:40:54.751 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 983 sec 2024-03-24T00:40:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:54 smithi106 ceph-mon[36505]: pgmap v670: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:55.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:54 smithi078 ceph-mon[32429]: pgmap v670: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:55.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:54 smithi107 ceph-mon[36485]: pgmap v670: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:55.752 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 984 sec 2024-03-24T00:40:56.751 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 985 sec 2024-03-24T00:40:56.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:56 smithi106 ceph-mon[36505]: pgmap v671: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:57.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:56 smithi078 ceph-mon[32429]: pgmap v671: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:56 smithi107 ceph-mon[36485]: pgmap v671: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:40:57.752 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 986 sec 2024-03-24T00:40:58.751 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 987 sec 2024-03-24T00:40:58.896 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:58 smithi107 ceph-mon[36485]: pgmap v672: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:58.896 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:40:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 333.113063 secs 2024-03-24T00:40:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:58 smithi106 ceph-mon[36505]: pgmap v672: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:40:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 333.113063 secs 2024-03-24T00:40:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:58 smithi078 ceph-mon[32429]: pgmap v672: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:59.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:40:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 333.113063 secs 2024-03-24T00:40:59.751 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 988 sec 2024-03-24T00:41:00.751 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 989 sec 2024-03-24T00:41:00.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:00 smithi106 ceph-mon[36505]: pgmap v673: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:01.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:00 smithi078 ceph-mon[32429]: pgmap v673: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:01.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:00 smithi107 ceph-mon[36485]: pgmap v673: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:01.752 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 990 sec 2024-03-24T00:41:02.752 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 991 sec 2024-03-24T00:41:02.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:02 smithi106 ceph-mon[36505]: pgmap v674: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:03.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:02 smithi078 ceph-mon[32429]: pgmap v674: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:03.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:02 smithi107 ceph-mon[36485]: pgmap v674: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:03.752 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 992 sec 2024-03-24T00:41:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 338.113123 secs 2024-03-24T00:41:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 338.113123 secs 2024-03-24T00:41:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 338.113123 secs 2024-03-24T00:41:04.752 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 993 sec 2024-03-24T00:41:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:04 smithi106 ceph-mon[36505]: pgmap v675: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:05.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:04 smithi078 ceph-mon[32429]: pgmap v675: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:05.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:04 smithi107 ceph-mon[36485]: pgmap v675: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:05.752 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 994 sec 2024-03-24T00:41:06.752 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 995 sec 2024-03-24T00:41:06.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:06 smithi078 ceph-mon[32429]: pgmap v676: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:06 smithi106 ceph-mon[36505]: pgmap v676: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:06 smithi107 ceph-mon[36485]: pgmap v676: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:07.752 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 996 sec 2024-03-24T00:41:08.752 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 997 sec 2024-03-24T00:41:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:08 smithi106 ceph-mon[36505]: pgmap v677: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:08.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 343.113211 secs 2024-03-24T00:41:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:08 smithi078 ceph-mon[32429]: pgmap v677: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 343.113211 secs 2024-03-24T00:41:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:08 smithi107 ceph-mon[36485]: pgmap v677: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 343.113211 secs 2024-03-24T00:41:09.752 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 998 sec 2024-03-24T00:41:10.752 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 999 sec 2024-03-24T00:41:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:10 smithi078 ceph-mon[32429]: pgmap v678: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:10 smithi106 ceph-mon[36505]: pgmap v678: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:11.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:10 smithi107 ceph-mon[36485]: pgmap v678: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:11.753 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1000 sec 2024-03-24T00:41:12.753 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1001 sec 2024-03-24T00:41:12.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:12 smithi106 ceph-mon[36505]: pgmap v679: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:13.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:12 smithi078 ceph-mon[32429]: pgmap v679: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:13.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:12 smithi107 ceph-mon[36485]: pgmap v679: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:13.753 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1002 sec 2024-03-24T00:41:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 348.113273 secs 2024-03-24T00:41:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 348.113273 secs 2024-03-24T00:41:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 348.113273 secs 2024-03-24T00:41:14.753 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1003 sec 2024-03-24T00:41:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:14 smithi106 ceph-mon[36505]: pgmap v680: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:15.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:14 smithi078 ceph-mon[32429]: pgmap v680: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:15.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:14 smithi107 ceph-mon[36485]: pgmap v680: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:15.753 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1004 sec 2024-03-24T00:41:16.753 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1005 sec 2024-03-24T00:41:16.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:16 smithi106 ceph-mon[36505]: pgmap v681: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:17.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:16 smithi078 ceph-mon[32429]: pgmap v681: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:17.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:16 smithi107 ceph-mon[36485]: pgmap v681: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:17.753 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1006 sec 2024-03-24T00:41:18.753 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1007 sec 2024-03-24T00:41:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:18 smithi106 ceph-mon[36505]: pgmap v682: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:18.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 353.113325 secs 2024-03-24T00:41:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:18 smithi078 ceph-mon[32429]: pgmap v682: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 353.113325 secs 2024-03-24T00:41:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:18 smithi107 ceph-mon[36485]: pgmap v682: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 353.113325 secs 2024-03-24T00:41:19.753 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1008 sec 2024-03-24T00:41:20.753 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1009 sec 2024-03-24T00:41:21.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:20 smithi078 ceph-mon[32429]: pgmap v683: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:21.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:20 smithi107 ceph-mon[36485]: pgmap v683: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:21.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:20 smithi106 ceph-mon[36505]: pgmap v683: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:21.753 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1010 sec 2024-03-24T00:41:22.753 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1011 sec 2024-03-24T00:41:23.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:22 smithi078 ceph-mon[32429]: pgmap v684: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:23.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:22 smithi107 ceph-mon[36485]: pgmap v684: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:23.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:22 smithi106 ceph-mon[36505]: pgmap v684: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:23.753 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1012 sec 2024-03-24T00:41:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 358.113342 secs 2024-03-24T00:41:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 358.113342 secs 2024-03-24T00:41:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 358.113342 secs 2024-03-24T00:41:24.753 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1013 sec 2024-03-24T00:41:25.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:24 smithi078 ceph-mon[32429]: pgmap v685: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:24 smithi107 ceph-mon[36485]: pgmap v685: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:25.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:24 smithi106 ceph-mon[36505]: pgmap v685: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:25.754 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1014 sec 2024-03-24T00:41:25.992 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:25 smithi078 ceph-mon[32429]: pgmap v686: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:26.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:25 smithi107 ceph-mon[36485]: pgmap v686: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:26.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:25 smithi106 ceph-mon[36505]: pgmap v686: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:26.755 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1015 sec 2024-03-24T00:41:27.754 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1016 sec 2024-03-24T00:41:28.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:27 smithi106 ceph-mon[36505]: pgmap v687: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:28.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:27 smithi078 ceph-mon[32429]: pgmap v687: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:28.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:27 smithi107 ceph-mon[36485]: pgmap v687: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:28.754 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1017 sec 2024-03-24T00:41:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:28 smithi106 ceph-mon[36505]: 2 slow requests, 1 included below; oldest blocked for > 363.113406 secs 2024-03-24T00:41:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:28 smithi106 ceph-mon[36505]: slow request 363.113405 seconds old, received at 2024-03-24T00:35:24.775244+0000: peer_request:client.15462:3775 currently acquired locks 2024-03-24T00:41:29.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:28 smithi107 ceph-mon[36485]: 2 slow requests, 1 included below; oldest blocked for > 363.113406 secs 2024-03-24T00:41:29.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:28 smithi107 ceph-mon[36485]: slow request 363.113405 seconds old, received at 2024-03-24T00:35:24.775244+0000: peer_request:client.15462:3775 currently acquired locks 2024-03-24T00:41:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:28 smithi078 ceph-mon[32429]: 2 slow requests, 1 included below; oldest blocked for > 363.113406 secs 2024-03-24T00:41:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:28 smithi078 ceph-mon[32429]: slow request 363.113405 seconds old, received at 2024-03-24T00:35:24.775244+0000: peer_request:client.15462:3775 currently acquired locks 2024-03-24T00:41:29.754 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1018 sec 2024-03-24T00:41:30.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:29 smithi106 ceph-mon[36505]: pgmap v688: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:30.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:29 smithi078 ceph-mon[32429]: pgmap v688: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:30.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:29 smithi107 ceph-mon[36485]: pgmap v688: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:30.754 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1019 sec 2024-03-24T00:41:31.754 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1020 sec 2024-03-24T00:41:32.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:31 smithi078 ceph-mon[32429]: pgmap v689: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:32.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:31 smithi107 ceph-mon[36485]: pgmap v689: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:32.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:31 smithi106 ceph-mon[36505]: pgmap v689: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:32.754 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1021 sec 2024-03-24T00:41:33.754 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1022 sec 2024-03-24T00:41:34.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:33 smithi078 ceph-mon[32429]: pgmap v690: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:34.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 368.113464 secs 2024-03-24T00:41:34.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:33 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:41:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:33 smithi107 ceph-mon[36485]: pgmap v690: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 368.113464 secs 2024-03-24T00:41:34.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:33 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:41:34.190 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:33 smithi106 ceph-mon[36505]: pgmap v690: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:34.190 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 368.113464 secs 2024-03-24T00:41:34.191 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:33 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:41:34.754 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1023 sec 2024-03-24T00:41:35.755 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1024 sec 2024-03-24T00:41:36.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:36 smithi106 ceph-mon[36505]: pgmap v691: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:36.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:41:36.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:41:36.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:41:36.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:41:36.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:41:36.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:41:36.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:41:36.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:41:36.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:41:36.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:36 smithi078 ceph-mon[32429]: pgmap v691: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:36.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:41:36.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:41:36.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:41:36.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:41:36.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:41:36.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:41:36.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:41:36.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:41:36.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:41:36.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:36 smithi107 ceph-mon[36485]: pgmap v691: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:36.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:41:36.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:41:36.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:41:36.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:41:36.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:41:36.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:41:36.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:41:36.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:41:36.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:41:36.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:41:36.754 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1025 sec 2024-03-24T00:41:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:37 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T00:41:37.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:37 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T00:41:37.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:37 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T00:41:37.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:37 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T00:41:37.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:37 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T00:41:37.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:37 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T00:41:37.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:37 smithi106 ceph-mon[36505]: pgmap v692: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T00:41:37.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:37 smithi106 ceph-mon[36505]: pgmap v693: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T00:41:37.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:37 smithi106 ceph-mon[36505]: Health check update: 8 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-24T00:41:37.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:37 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T00:41:37.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:37 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T00:41:37.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:37 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T00:41:37.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:37 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T00:41:37.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:37 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T00:41:37.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:37 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T00:41:37.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:37 smithi078 ceph-mon[32429]: pgmap v692: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T00:41:37.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:37 smithi078 ceph-mon[32429]: pgmap v693: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T00:41:37.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:37 smithi078 ceph-mon[32429]: Health check update: 8 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-24T00:41:37.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:37 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T00:41:37.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:37 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T00:41:37.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:37 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T00:41:37.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:37 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T00:41:37.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:37 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T00:41:37.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:37 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T00:41:37.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:37 smithi107 ceph-mon[36485]: pgmap v692: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T00:41:37.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:37 smithi107 ceph-mon[36485]: pgmap v693: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T00:41:37.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:37 smithi107 ceph-mon[36485]: Health check update: 8 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-24T00:41:37.754 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1026 sec 2024-03-24T00:41:38.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 373.113527 secs 2024-03-24T00:41:38.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 373.113527 secs 2024-03-24T00:41:38.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 373.113527 secs 2024-03-24T00:41:38.754 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1027 sec 2024-03-24T00:41:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:39 smithi078 ceph-mon[32429]: pgmap v694: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T00:41:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:39 smithi107 ceph-mon[36485]: pgmap v694: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T00:41:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:39 smithi106 ceph-mon[36505]: pgmap v694: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T00:41:39.755 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1028 sec 2024-03-24T00:41:40.755 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1029 sec 2024-03-24T00:41:41.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:41 smithi078 ceph-mon[32429]: pgmap v695: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:41:41.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:41 smithi107 ceph-mon[36485]: pgmap v695: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:41:41.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:41 smithi106 ceph-mon[36505]: pgmap v695: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:41:41.755 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1030 sec 2024-03-24T00:41:42.755 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1031 sec 2024-03-24T00:41:43.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:43 smithi078 ceph-mon[32429]: pgmap v696: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T00:41:43.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 378.113587 secs 2024-03-24T00:41:43.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:43 smithi107 ceph-mon[36485]: pgmap v696: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T00:41:43.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 378.113587 secs 2024-03-24T00:41:43.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:43 smithi106 ceph-mon[36505]: pgmap v696: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T00:41:43.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 378.113587 secs 2024-03-24T00:41:43.755 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1032 sec 2024-03-24T00:41:44.755 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1033 sec 2024-03-24T00:41:45.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:45 smithi078 ceph-mon[32429]: pgmap v697: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T00:41:45.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:45 smithi107 ceph-mon[36485]: pgmap v697: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T00:41:45.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:45 smithi106 ceph-mon[36505]: pgmap v697: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T00:41:45.755 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1034 sec 2024-03-24T00:41:46.755 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1035 sec 2024-03-24T00:41:47.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:47 smithi078 ceph-mon[32429]: pgmap v698: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:41:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:47 smithi107 ceph-mon[36485]: pgmap v698: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:41:47.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:47 smithi106 ceph-mon[36505]: pgmap v698: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:41:47.755 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1036 sec 2024-03-24T00:41:48.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 383.113648 secs 2024-03-24T00:41:48.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 383.113648 secs 2024-03-24T00:41:48.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 383.113648 secs 2024-03-24T00:41:48.755 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1037 sec 2024-03-24T00:41:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:49 smithi078 ceph-mon[32429]: pgmap v699: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:49 smithi107 ceph-mon[36485]: pgmap v699: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:49 smithi106 ceph-mon[36505]: pgmap v699: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:49.755 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1038 sec 2024-03-24T00:41:50.756 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1039 sec 2024-03-24T00:41:51.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:51 smithi078 ceph-mon[32429]: pgmap v700: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:51.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:51 smithi107 ceph-mon[36485]: pgmap v700: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:51.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:51 smithi106 ceph-mon[36505]: pgmap v700: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:51.756 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1040 sec 2024-03-24T00:41:52.756 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1041 sec 2024-03-24T00:41:53.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:53 smithi078 ceph-mon[32429]: pgmap v701: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:53.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 388.113686 secs 2024-03-24T00:41:53.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:53 smithi107 ceph-mon[36485]: pgmap v701: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:53.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 388.113686 secs 2024-03-24T00:41:53.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:53 smithi106 ceph-mon[36505]: pgmap v701: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:53.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 388.113686 secs 2024-03-24T00:41:53.756 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1042 sec 2024-03-24T00:41:54.756 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1043 sec 2024-03-24T00:41:55.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:55 smithi078 ceph-mon[32429]: pgmap v702: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:55.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:55 smithi107 ceph-mon[36485]: pgmap v702: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:55.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:55 smithi106 ceph-mon[36505]: pgmap v702: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:55.756 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1044 sec 2024-03-24T00:41:56.756 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1045 sec 2024-03-24T00:41:57.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:57 smithi078 ceph-mon[32429]: pgmap v703: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:57.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:57 smithi107 ceph-mon[36485]: pgmap v703: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:57.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:57 smithi106 ceph-mon[36505]: pgmap v703: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:57.756 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1046 sec 2024-03-24T00:41:58.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 393.113748 secs 2024-03-24T00:41:58.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 393.113748 secs 2024-03-24T00:41:58.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 393.113748 secs 2024-03-24T00:41:58.756 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1047 sec 2024-03-24T00:41:59.495 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:41:59 smithi107 ceph-mon[36485]: pgmap v704: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:59.553 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:41:59 smithi106 ceph-mon[36505]: pgmap v704: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:41:59 smithi078 ceph-mon[32429]: pgmap v704: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:41:59.757 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1048 sec 2024-03-24T00:42:00.757 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1049 sec 2024-03-24T00:42:01.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:01 smithi078 ceph-mon[32429]: pgmap v705: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:01.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:01 smithi107 ceph-mon[36485]: pgmap v705: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:01.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:01 smithi106 ceph-mon[36505]: pgmap v705: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:01.757 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1050 sec 2024-03-24T00:42:02.757 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1051 sec 2024-03-24T00:42:03.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:03 smithi078 ceph-mon[32429]: pgmap v706: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:03.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:03 smithi078 ceph-mon[32429]: 2 slow requests, 1 included below; oldest blocked for > 398.113810 secs 2024-03-24T00:42:03.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:03 smithi078 ceph-mon[32429]: slow request 362.846739 seconds old, received at 2024-03-24T00:36:00.042315+0000: client_request(client.25385:166 mksnap #0x1//scheduled-2024-03-24-00_36_00_UTC 2024-03-24T00:36:00.042488+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-24T00:42:03.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:03 smithi107 ceph-mon[36485]: pgmap v706: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:03.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:03 smithi107 ceph-mon[36485]: 2 slow requests, 1 included below; oldest blocked for > 398.113810 secs 2024-03-24T00:42:03.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:03 smithi107 ceph-mon[36485]: slow request 362.846739 seconds old, received at 2024-03-24T00:36:00.042315+0000: client_request(client.25385:166 mksnap #0x1//scheduled-2024-03-24-00_36_00_UTC 2024-03-24T00:36:00.042488+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-24T00:42:03.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:03 smithi106 ceph-mon[36505]: pgmap v706: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:03.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:03 smithi106 ceph-mon[36505]: 2 slow requests, 1 included below; oldest blocked for > 398.113810 secs 2024-03-24T00:42:03.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:03 smithi106 ceph-mon[36505]: slow request 362.846739 seconds old, received at 2024-03-24T00:36:00.042315+0000: client_request(client.25385:166 mksnap #0x1//scheduled-2024-03-24-00_36_00_UTC 2024-03-24T00:36:00.042488+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-24T00:42:03.757 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1052 sec 2024-03-24T00:42:04.757 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1053 sec 2024-03-24T00:42:05.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:05 smithi078 ceph-mon[32429]: pgmap v707: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:05.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:05 smithi107 ceph-mon[36485]: pgmap v707: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:05.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:05 smithi106 ceph-mon[36505]: pgmap v707: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:05.757 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1054 sec 2024-03-24T00:42:06.757 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1055 sec 2024-03-24T00:42:07.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:07 smithi078 ceph-mon[32429]: pgmap v708: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:07.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:07 smithi107 ceph-mon[36485]: pgmap v708: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:07.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:07 smithi106 ceph-mon[36505]: pgmap v708: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:07.757 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1056 sec 2024-03-24T00:42:08.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 403.113897 secs 2024-03-24T00:42:08.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 403.113897 secs 2024-03-24T00:42:08.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 403.113897 secs 2024-03-24T00:42:08.757 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1057 sec 2024-03-24T00:42:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:09 smithi078 ceph-mon[32429]: pgmap v709: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:09 smithi107 ceph-mon[36485]: pgmap v709: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:09 smithi106 ceph-mon[36505]: pgmap v709: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:09.757 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1058 sec 2024-03-24T00:42:10.757 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1059 sec 2024-03-24T00:42:11.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:11 smithi078 ceph-mon[32429]: pgmap v710: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:11.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:11 smithi107 ceph-mon[36485]: pgmap v710: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:11.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:11 smithi106 ceph-mon[36505]: pgmap v710: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:11.757 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1060 sec 2024-03-24T00:42:12.758 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1061 sec 2024-03-24T00:42:13.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:13 smithi078 ceph-mon[32429]: pgmap v711: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:13.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 408.113957 secs 2024-03-24T00:42:13.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:13 smithi107 ceph-mon[36485]: pgmap v711: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:13.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 408.113957 secs 2024-03-24T00:42:13.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:13 smithi106 ceph-mon[36505]: pgmap v711: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:13.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 408.113957 secs 2024-03-24T00:42:13.758 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1062 sec 2024-03-24T00:42:14.758 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1063 sec 2024-03-24T00:42:15.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:15 smithi078 ceph-mon[32429]: pgmap v712: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:15.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:15 smithi107 ceph-mon[36485]: pgmap v712: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:15.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:15 smithi106 ceph-mon[36505]: pgmap v712: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:15.758 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1064 sec 2024-03-24T00:42:16.758 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1065 sec 2024-03-24T00:42:17.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:17 smithi078 ceph-mon[32429]: pgmap v713: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:17.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:17 smithi107 ceph-mon[36485]: pgmap v713: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:17.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:17 smithi106 ceph-mon[36505]: pgmap v713: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:17.758 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1066 sec 2024-03-24T00:42:18.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 413.114040 secs 2024-03-24T00:42:18.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 413.114040 secs 2024-03-24T00:42:18.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 413.114040 secs 2024-03-24T00:42:18.758 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1067 sec 2024-03-24T00:42:19.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:19 smithi078 ceph-mon[32429]: pgmap v714: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:19 smithi107 ceph-mon[36485]: pgmap v714: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:19 smithi106 ceph-mon[36505]: pgmap v714: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:19.758 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1068 sec 2024-03-24T00:42:20.759 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1069 sec 2024-03-24T00:42:21.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:21 smithi078 ceph-mon[32429]: pgmap v715: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:21.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:21 smithi107 ceph-mon[36485]: pgmap v715: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:21.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:21 smithi106 ceph-mon[36505]: pgmap v715: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:21.758 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1070 sec 2024-03-24T00:42:22.758 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1071 sec 2024-03-24T00:42:23.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:23 smithi078 ceph-mon[32429]: pgmap v716: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:23.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 418.114100 secs 2024-03-24T00:42:23.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:23 smithi107 ceph-mon[36485]: pgmap v716: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:23.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 418.114100 secs 2024-03-24T00:42:23.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:23 smithi106 ceph-mon[36505]: pgmap v716: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:23.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 418.114100 secs 2024-03-24T00:42:23.758 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1072 sec 2024-03-24T00:42:24.759 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1073 sec 2024-03-24T00:42:25.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:25 smithi078 ceph-mon[32429]: pgmap v717: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:25.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:25 smithi107 ceph-mon[36485]: pgmap v717: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:25.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:25 smithi106 ceph-mon[36505]: pgmap v717: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:25.759 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1074 sec 2024-03-24T00:42:26.759 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1075 sec 2024-03-24T00:42:27.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:27 smithi078 ceph-mon[32429]: pgmap v718: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:27.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:27 smithi107 ceph-mon[36485]: pgmap v718: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:27.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:27 smithi106 ceph-mon[36505]: pgmap v718: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:27.759 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1076 sec 2024-03-24T00:42:28.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 423.114178 secs 2024-03-24T00:42:28.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 423.114178 secs 2024-03-24T00:42:28.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 423.114178 secs 2024-03-24T00:42:28.759 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1077 sec 2024-03-24T00:42:29.553 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:29 smithi106 ceph-mon[36505]: pgmap v719: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:29.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:29 smithi078 ceph-mon[32429]: pgmap v719: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:29 smithi107 ceph-mon[36485]: pgmap v719: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:29.759 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1078 sec 2024-03-24T00:42:30.759 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1079 sec 2024-03-24T00:42:31.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:31 smithi078 ceph-mon[32429]: pgmap v720: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:31.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:31 smithi107 ceph-mon[36485]: pgmap v720: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:31.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:31 smithi106 ceph-mon[36505]: pgmap v720: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:31.759 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1080 sec 2024-03-24T00:42:32.759 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1081 sec 2024-03-24T00:42:33.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:33 smithi078 ceph-mon[32429]: pgmap v721: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:33.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 428.114221 secs 2024-03-24T00:42:33.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:33 smithi107 ceph-mon[36485]: pgmap v721: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:33.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 428.114221 secs 2024-03-24T00:42:33.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:33 smithi106 ceph-mon[36505]: pgmap v721: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:33.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 428.114221 secs 2024-03-24T00:42:33.760 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1082 sec 2024-03-24T00:42:34.759 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1083 sec 2024-03-24T00:42:35.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:35 smithi078 ceph-mon[32429]: pgmap v722: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:35.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:35 smithi107 ceph-mon[36485]: pgmap v722: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:35.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:35 smithi106 ceph-mon[36505]: pgmap v722: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:35.759 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1084 sec 2024-03-24T00:42:36.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:42:36.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:42:36.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:42:36.759 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1085 sec 2024-03-24T00:42:37.760 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1086 sec 2024-03-24T00:42:37.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:37 smithi106 ceph-mon[36505]: pgmap v723: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:37.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:42:37.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:42:37.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:42:37.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:42:37.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:42:37.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:42:37.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:42:37.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:42:37.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:42:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:37 smithi078 ceph-mon[32429]: pgmap v723: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:38.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:42:38.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:42:38.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:42:38.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:42:38.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:42:38.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:42:38.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:42:38.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:42:38.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:42:38.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:37 smithi107 ceph-mon[36485]: pgmap v723: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:38.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:42:38.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:42:38.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:42:38.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:42:38.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:42:38.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:42:38.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:42:38.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:42:38.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:42:38.760 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1087 sec 2024-03-24T00:42:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:38 smithi078 ceph-mon[32429]: pgmap v724: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T00:42:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:38 smithi078 ceph-mon[32429]: pgmap v725: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:42:39.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 433.114288 secs 2024-03-24T00:42:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:38 smithi107 ceph-mon[36485]: pgmap v724: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T00:42:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:38 smithi107 ceph-mon[36485]: pgmap v725: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:42:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 433.114288 secs 2024-03-24T00:42:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:38 smithi106 ceph-mon[36505]: pgmap v724: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T00:42:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:38 smithi106 ceph-mon[36505]: pgmap v725: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:42:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 433.114288 secs 2024-03-24T00:42:39.760 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1088 sec 2024-03-24T00:42:40.760 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1089 sec 2024-03-24T00:42:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:40 smithi078 ceph-mon[32429]: pgmap v726: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:42:41.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:40 smithi107 ceph-mon[36485]: pgmap v726: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:42:41.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:40 smithi106 ceph-mon[36505]: pgmap v726: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:42:41.760 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1090 sec 2024-03-24T00:42:42.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:41 smithi078 ceph-mon[32429]: pgmap v727: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:42:42.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:41 smithi107 ceph-mon[36485]: pgmap v727: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:42:42.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:41 smithi106 ceph-mon[36505]: pgmap v727: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:42:42.760 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1091 sec 2024-03-24T00:42:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 438.114364 secs 2024-03-24T00:42:43.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 438.114364 secs 2024-03-24T00:42:43.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 438.114364 secs 2024-03-24T00:42:43.760 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1092 sec 2024-03-24T00:42:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:44 smithi078 ceph-mon[32429]: pgmap v728: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:42:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:44 smithi107 ceph-mon[36485]: pgmap v728: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:42:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:44 smithi106 ceph-mon[36505]: pgmap v728: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:42:44.762 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1093 sec 2024-03-24T00:42:45.760 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1094 sec 2024-03-24T00:42:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:46 smithi106 ceph-mon[36505]: pgmap v729: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:42:46.761 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1095 sec 2024-03-24T00:42:46.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:46 smithi078 ceph-mon[32429]: pgmap v729: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:42:46.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:46 smithi107 ceph-mon[36485]: pgmap v729: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:42:47.761 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1096 sec 2024-03-24T00:42:48.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:48 smithi106 ceph-mon[36505]: pgmap v730: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:42:48.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 443.114401 secs 2024-03-24T00:42:48.761 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1097 sec 2024-03-24T00:42:48.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:48 smithi078 ceph-mon[32429]: pgmap v730: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:42:48.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 443.114401 secs 2024-03-24T00:42:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:48 smithi107 ceph-mon[36485]: pgmap v730: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:42:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 443.114401 secs 2024-03-24T00:42:49.761 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1098 sec 2024-03-24T00:42:50.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:50 smithi078 ceph-mon[32429]: pgmap v731: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:50.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:50 smithi107 ceph-mon[36485]: pgmap v731: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:50.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:50 smithi106 ceph-mon[36505]: pgmap v731: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:50.761 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1099 sec 2024-03-24T00:42:51.761 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1100 sec 2024-03-24T00:42:52.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:52 smithi106 ceph-mon[36505]: pgmap v732: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:52.761 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1101 sec 2024-03-24T00:42:52.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:52 smithi078 ceph-mon[32429]: pgmap v732: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:52.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:52 smithi107 ceph-mon[36485]: pgmap v732: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:53.761 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1102 sec 2024-03-24T00:42:53.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 448.114486 secs 2024-03-24T00:42:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 448.114486 secs 2024-03-24T00:42:53.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 448.114486 secs 2024-03-24T00:42:54.761 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1103 sec 2024-03-24T00:42:54.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:54 smithi078 ceph-mon[32429]: pgmap v733: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:54 smithi107 ceph-mon[36485]: pgmap v733: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:54 smithi106 ceph-mon[36505]: pgmap v733: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:42:55.761 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1104 sec 2024-03-24T00:42:56.761 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1105 sec 2024-03-24T00:42:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:56 smithi078 ceph-mon[32429]: pgmap v734: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:42:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:56 smithi107 ceph-mon[36485]: pgmap v734: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:42:56.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:56 smithi106 ceph-mon[36505]: pgmap v734: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:42:57.761 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1106 sec 2024-03-24T00:42:58.762 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1107 sec 2024-03-24T00:42:58.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:58 smithi078 ceph-mon[32429]: pgmap v735: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:42:58.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:42:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 453.114569 secs 2024-03-24T00:42:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:58 smithi107 ceph-mon[36485]: pgmap v735: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:42:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:42:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 453.114569 secs 2024-03-24T00:42:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:58 smithi106 ceph-mon[36505]: pgmap v735: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:42:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:42:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 453.114569 secs 2024-03-24T00:42:59.762 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1108 sec 2024-03-24T00:43:00.704 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:00 smithi106 ceph-mon[36505]: pgmap v736: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:00.762 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1109 sec 2024-03-24T00:43:00.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:00 smithi078 ceph-mon[32429]: pgmap v736: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:00 smithi107 ceph-mon[36485]: pgmap v736: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:01.762 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1110 sec 2024-03-24T00:43:02.762 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1111 sec 2024-03-24T00:43:02.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:02 smithi078 ceph-mon[32429]: pgmap v737: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:02 smithi107 ceph-mon[36485]: pgmap v737: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:02.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:02 smithi106 ceph-mon[36505]: pgmap v737: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:03.762 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1112 sec 2024-03-24T00:43:03.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 458.114656 secs 2024-03-24T00:43:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 458.114656 secs 2024-03-24T00:43:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 458.114656 secs 2024-03-24T00:43:04.762 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1113 sec 2024-03-24T00:43:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:04 smithi078 ceph-mon[32429]: pgmap v738: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:04 smithi107 ceph-mon[36485]: pgmap v738: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:04 smithi106 ceph-mon[36505]: pgmap v738: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:05.762 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1114 sec 2024-03-24T00:43:06.762 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1115 sec 2024-03-24T00:43:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:06 smithi078 ceph-mon[32429]: pgmap v739: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:43:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:06 smithi107 ceph-mon[36485]: pgmap v739: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:43:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:06 smithi106 ceph-mon[36505]: pgmap v739: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:43:07.762 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1116 sec 2024-03-24T00:43:08.762 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1117 sec 2024-03-24T00:43:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:08 smithi078 ceph-mon[32429]: pgmap v740: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:43:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 463.114697 secs 2024-03-24T00:43:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:08 smithi107 ceph-mon[36485]: pgmap v740: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:43:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 463.114697 secs 2024-03-24T00:43:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:08 smithi106 ceph-mon[36505]: pgmap v740: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:43:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 463.114697 secs 2024-03-24T00:43:09.762 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1118 sec 2024-03-24T00:43:10.762 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1119 sec 2024-03-24T00:43:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:10 smithi078 ceph-mon[32429]: pgmap v741: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:10 smithi107 ceph-mon[36485]: pgmap v741: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:10.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:10 smithi106 ceph-mon[36505]: pgmap v741: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:11.763 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1120 sec 2024-03-24T00:43:12.763 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1121 sec 2024-03-24T00:43:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:12 smithi078 ceph-mon[32429]: pgmap v742: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:12 smithi107 ceph-mon[36485]: pgmap v742: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:12.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:12 smithi106 ceph-mon[36505]: pgmap v742: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:13.763 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1122 sec 2024-03-24T00:43:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 468.114713 secs 2024-03-24T00:43:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 468.114713 secs 2024-03-24T00:43:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 468.114713 secs 2024-03-24T00:43:14.763 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1123 sec 2024-03-24T00:43:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:14 smithi078 ceph-mon[32429]: pgmap v743: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:14 smithi107 ceph-mon[36485]: pgmap v743: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:14 smithi106 ceph-mon[36505]: pgmap v743: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:15.763 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1124 sec 2024-03-24T00:43:16.763 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1125 sec 2024-03-24T00:43:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:16 smithi078 ceph-mon[32429]: pgmap v744: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:43:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:16 smithi107 ceph-mon[36485]: pgmap v744: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:43:16.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:16 smithi106 ceph-mon[36505]: pgmap v744: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:43:17.764 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1126 sec 2024-03-24T00:43:18.763 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1127 sec 2024-03-24T00:43:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:18 smithi078 ceph-mon[32429]: pgmap v745: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:43:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 473.114770 secs 2024-03-24T00:43:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:18 smithi107 ceph-mon[36485]: pgmap v745: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:43:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 473.114770 secs 2024-03-24T00:43:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:18 smithi106 ceph-mon[36505]: pgmap v745: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:43:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 473.114770 secs 2024-03-24T00:43:19.763 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1128 sec 2024-03-24T00:43:20.764 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1129 sec 2024-03-24T00:43:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:20 smithi078 ceph-mon[32429]: pgmap v746: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:20 smithi107 ceph-mon[36485]: pgmap v746: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:20.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:20 smithi106 ceph-mon[36505]: pgmap v746: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:21.764 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1130 sec 2024-03-24T00:43:22.764 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1131 sec 2024-03-24T00:43:22.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:22 smithi078 ceph-mon[32429]: pgmap v747: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:22.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:22 smithi107 ceph-mon[36485]: pgmap v747: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:22.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:22 smithi106 ceph-mon[36505]: pgmap v747: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:23.764 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1132 sec 2024-03-24T00:43:23.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 478.114826 secs 2024-03-24T00:43:23.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 478.114826 secs 2024-03-24T00:43:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 478.114826 secs 2024-03-24T00:43:24.764 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1133 sec 2024-03-24T00:43:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:24 smithi078 ceph-mon[32429]: pgmap v748: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:24 smithi107 ceph-mon[36485]: pgmap v748: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:24 smithi106 ceph-mon[36505]: pgmap v748: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:25.764 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1134 sec 2024-03-24T00:43:26.764 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1135 sec 2024-03-24T00:43:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:26 smithi078 ceph-mon[32429]: pgmap v749: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:43:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:26 smithi107 ceph-mon[36485]: pgmap v749: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:43:26.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:26 smithi106 ceph-mon[36505]: pgmap v749: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:43:27.764 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1136 sec 2024-03-24T00:43:28.764 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1137 sec 2024-03-24T00:43:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:28 smithi078 ceph-mon[32429]: pgmap v750: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:43:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 483.114888 secs 2024-03-24T00:43:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:28 smithi107 ceph-mon[36485]: pgmap v750: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:43:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 483.114888 secs 2024-03-24T00:43:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:28 smithi106 ceph-mon[36505]: pgmap v750: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:43:28.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 483.114888 secs 2024-03-24T00:43:29.764 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1138 sec 2024-03-24T00:43:30.764 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1139 sec 2024-03-24T00:43:30.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:30 smithi078 ceph-mon[32429]: pgmap v751: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:30 smithi107 ceph-mon[36485]: pgmap v751: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:30 smithi106 ceph-mon[36505]: pgmap v751: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:31.765 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1140 sec 2024-03-24T00:43:32.765 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1141 sec 2024-03-24T00:43:32.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:32 smithi078 ceph-mon[32429]: pgmap v752: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:32 smithi107 ceph-mon[36485]: pgmap v752: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:32 smithi106 ceph-mon[36505]: pgmap v752: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:33.765 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1142 sec 2024-03-24T00:43:33.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 488.114949 secs 2024-03-24T00:43:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 488.114949 secs 2024-03-24T00:43:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 488.114949 secs 2024-03-24T00:43:34.765 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1143 sec 2024-03-24T00:43:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:34 smithi078 ceph-mon[32429]: pgmap v753: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:34 smithi107 ceph-mon[36485]: pgmap v753: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:34 smithi106 ceph-mon[36505]: pgmap v753: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:35.765 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1144 sec 2024-03-24T00:43:36.765 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1145 sec 2024-03-24T00:43:36.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:36 smithi078 ceph-mon[32429]: pgmap v754: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:43:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:36 smithi107 ceph-mon[36485]: pgmap v754: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:43:36.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:36 smithi106 ceph-mon[36505]: pgmap v754: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:43:37.765 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1146 sec 2024-03-24T00:43:37.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:43:37.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:43:37.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:43:38.765 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1147 sec 2024-03-24T00:43:38.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:38 smithi078 ceph-mon[32429]: pgmap v755: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:43:38.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 493.115013 secs 2024-03-24T00:43:38.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:38 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:43:38.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:38 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:43:38.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:38 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:43:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:38 smithi107 ceph-mon[36485]: pgmap v755: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:43:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 493.115013 secs 2024-03-24T00:43:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:38 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:43:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:38 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:43:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:38 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:43:38.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:38 smithi106 ceph-mon[36505]: pgmap v755: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:43:38.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 493.115013 secs 2024-03-24T00:43:38.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:38 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:43:38.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:38 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:43:38.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:38 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:43:39.765 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1148 sec 2024-03-24T00:43:39.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:39 smithi078 ceph-mon[32429]: pgmap v756: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:43:39.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:39 smithi078 ceph-mon[32429]: pgmap v757: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:43:39.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:39 smithi107 ceph-mon[36485]: pgmap v756: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:43:39.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:39 smithi107 ceph-mon[36485]: pgmap v757: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:43:39.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:39 smithi106 ceph-mon[36505]: pgmap v756: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:43:39.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:39 smithi106 ceph-mon[36505]: pgmap v757: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:43:40.766 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1149 sec 2024-03-24T00:43:41.765 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1150 sec 2024-03-24T00:43:41.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:41 smithi078 ceph-mon[32429]: pgmap v758: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:43:41.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:41 smithi107 ceph-mon[36485]: pgmap v758: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:43:41.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:41 smithi106 ceph-mon[36505]: pgmap v758: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:43:42.765 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1151 sec 2024-03-24T00:43:43.766 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1152 sec 2024-03-24T00:43:43.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:43 smithi078 ceph-mon[32429]: pgmap v759: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:43:43.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 498.115072 secs 2024-03-24T00:43:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:43 smithi107 ceph-mon[36485]: pgmap v759: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:43:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 498.115072 secs 2024-03-24T00:43:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:43 smithi106 ceph-mon[36505]: pgmap v759: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:43:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 498.115072 secs 2024-03-24T00:43:44.766 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1153 sec 2024-03-24T00:43:45.766 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1154 sec 2024-03-24T00:43:45.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:45 smithi078 ceph-mon[32429]: pgmap v760: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:45.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:45 smithi107 ceph-mon[36485]: pgmap v760: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:45.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:45 smithi106 ceph-mon[36505]: pgmap v760: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:46.766 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1155 sec 2024-03-24T00:43:47.766 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1156 sec 2024-03-24T00:43:47.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:47 smithi078 ceph-mon[32429]: pgmap v761: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:43:47.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:47 smithi107 ceph-mon[36485]: pgmap v761: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:43:47.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:47 smithi106 ceph-mon[36505]: pgmap v761: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:43:48.766 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1157 sec 2024-03-24T00:43:48.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 503.115158 secs 2024-03-24T00:43:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 503.115158 secs 2024-03-24T00:43:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 503.115158 secs 2024-03-24T00:43:49.766 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1158 sec 2024-03-24T00:43:49.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:49 smithi078 ceph-mon[32429]: pgmap v762: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:43:49.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:49 smithi107 ceph-mon[36485]: pgmap v762: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:43:49.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:49 smithi106 ceph-mon[36505]: pgmap v762: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:43:50.766 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1159 sec 2024-03-24T00:43:51.766 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1160 sec 2024-03-24T00:43:51.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:51 smithi078 ceph-mon[32429]: pgmap v763: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:51.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:51 smithi107 ceph-mon[36485]: pgmap v763: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:51 smithi106 ceph-mon[36505]: pgmap v763: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:52.766 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1161 sec 2024-03-24T00:43:53.766 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1162 sec 2024-03-24T00:43:53.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:53 smithi078 ceph-mon[32429]: pgmap v764: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:53.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 508.115241 secs 2024-03-24T00:43:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:53 smithi107 ceph-mon[36485]: pgmap v764: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 508.115241 secs 2024-03-24T00:43:53.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:53 smithi106 ceph-mon[36505]: pgmap v764: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:53.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 508.115241 secs 2024-03-24T00:43:54.770 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1163 sec 2024-03-24T00:43:55.767 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1164 sec 2024-03-24T00:43:55.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:55 smithi078 ceph-mon[32429]: pgmap v765: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:55.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:55 smithi107 ceph-mon[36485]: pgmap v765: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:55.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:55 smithi106 ceph-mon[36505]: pgmap v765: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:56.767 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1165 sec 2024-03-24T00:43:57.767 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1166 sec 2024-03-24T00:43:57.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:57 smithi078 ceph-mon[32429]: pgmap v766: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:57.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:57 smithi107 ceph-mon[36485]: pgmap v766: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:57.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:57 smithi106 ceph-mon[36505]: pgmap v766: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:58.767 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1167 sec 2024-03-24T00:43:58.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 513.115292 secs 2024-03-24T00:43:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 513.115292 secs 2024-03-24T00:43:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 513.115292 secs 2024-03-24T00:43:59.767 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1168 sec 2024-03-24T00:43:59.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:43:59 smithi078 ceph-mon[32429]: pgmap v767: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:43:59 smithi107 ceph-mon[36485]: pgmap v767: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:43:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:43:59 smithi106 ceph-mon[36505]: pgmap v767: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:00.767 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1169 sec 2024-03-24T00:44:01.767 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1170 sec 2024-03-24T00:44:01.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:01 smithi078 ceph-mon[32429]: pgmap v768: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:01.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:01 smithi107 ceph-mon[36485]: pgmap v768: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:01.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:01 smithi106 ceph-mon[36505]: pgmap v768: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:02.767 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1171 sec 2024-03-24T00:44:03.768 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1172 sec 2024-03-24T00:44:03.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:03 smithi078 ceph-mon[32429]: pgmap v769: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:03.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 518.115353 secs 2024-03-24T00:44:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:03 smithi107 ceph-mon[36485]: pgmap v769: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 518.115353 secs 2024-03-24T00:44:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:03 smithi106 ceph-mon[36505]: pgmap v769: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 518.115353 secs 2024-03-24T00:44:04.768 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1173 sec 2024-03-24T00:44:05.768 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1174 sec 2024-03-24T00:44:05.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:05 smithi078 ceph-mon[32429]: pgmap v770: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:05.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:05 smithi107 ceph-mon[36485]: pgmap v770: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:05.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:05 smithi106 ceph-mon[36505]: pgmap v770: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:06.768 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1175 sec 2024-03-24T00:44:07.768 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1176 sec 2024-03-24T00:44:07.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:07 smithi078 ceph-mon[32429]: pgmap v771: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:07.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:07 smithi107 ceph-mon[36485]: pgmap v771: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:07.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:07 smithi106 ceph-mon[36505]: pgmap v771: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:08.768 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1177 sec 2024-03-24T00:44:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 523.115416 secs 2024-03-24T00:44:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 523.115416 secs 2024-03-24T00:44:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 523.115416 secs 2024-03-24T00:44:09.768 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1178 sec 2024-03-24T00:44:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:09 smithi078 ceph-mon[32429]: pgmap v772: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:09.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:09 smithi107 ceph-mon[36485]: pgmap v772: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:09.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:09 smithi106 ceph-mon[36505]: pgmap v772: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:10.768 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1179 sec 2024-03-24T00:44:11.768 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1180 sec 2024-03-24T00:44:11.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:11 smithi078 ceph-mon[32429]: pgmap v773: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:11 smithi107 ceph-mon[36485]: pgmap v773: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:11.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:11 smithi106 ceph-mon[36505]: pgmap v773: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:12.768 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1181 sec 2024-03-24T00:44:13.768 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1182 sec 2024-03-24T00:44:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:13 smithi078 ceph-mon[32429]: pgmap v774: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 528.115477 secs 2024-03-24T00:44:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:13 smithi107 ceph-mon[36485]: pgmap v774: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 528.115477 secs 2024-03-24T00:44:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:13 smithi106 ceph-mon[36505]: pgmap v774: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 528.115477 secs 2024-03-24T00:44:14.768 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1183 sec 2024-03-24T00:44:15.768 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1184 sec 2024-03-24T00:44:15.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:15 smithi078 ceph-mon[32429]: pgmap v775: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:15 smithi107 ceph-mon[36485]: pgmap v775: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:15 smithi106 ceph-mon[36505]: pgmap v775: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:16.769 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1185 sec 2024-03-24T00:44:17.769 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1186 sec 2024-03-24T00:44:17.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:17 smithi078 ceph-mon[32429]: pgmap v776: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:17.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:17 smithi107 ceph-mon[36485]: pgmap v776: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:17.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:17 smithi106 ceph-mon[36505]: pgmap v776: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:18.769 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1187 sec 2024-03-24T00:44:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 533.115537 secs 2024-03-24T00:44:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 533.115537 secs 2024-03-24T00:44:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 533.115537 secs 2024-03-24T00:44:19.769 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1188 sec 2024-03-24T00:44:19.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:19 smithi078 ceph-mon[32429]: pgmap v777: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:19 smithi107 ceph-mon[36485]: pgmap v777: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:19.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:19 smithi106 ceph-mon[36505]: pgmap v777: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:20.769 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1189 sec 2024-03-24T00:44:21.769 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1190 sec 2024-03-24T00:44:21.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:21 smithi078 ceph-mon[32429]: pgmap v778: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:21 smithi107 ceph-mon[36485]: pgmap v778: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:21.905 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:21 smithi106 ceph-mon[36505]: pgmap v778: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:22.769 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1191 sec 2024-03-24T00:44:23.769 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1192 sec 2024-03-24T00:44:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:23 smithi106 ceph-mon[36505]: pgmap v779: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:23.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 538.115598 secs 2024-03-24T00:44:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:23 smithi078 ceph-mon[32429]: pgmap v779: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:24.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 538.115598 secs 2024-03-24T00:44:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:23 smithi107 ceph-mon[36485]: pgmap v779: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 538.115598 secs 2024-03-24T00:44:24.769 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1193 sec 2024-03-24T00:44:25.769 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1194 sec 2024-03-24T00:44:25.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:25 smithi106 ceph-mon[36505]: pgmap v780: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:26.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:25 smithi078 ceph-mon[32429]: pgmap v780: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:26.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:25 smithi107 ceph-mon[36485]: pgmap v780: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:26.770 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1195 sec 2024-03-24T00:44:27.770 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1196 sec 2024-03-24T00:44:27.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:27 smithi106 ceph-mon[36505]: pgmap v781: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:28.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:27 smithi078 ceph-mon[32429]: pgmap v781: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:27 smithi107 ceph-mon[36485]: pgmap v781: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:28.770 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1197 sec 2024-03-24T00:44:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 543.115659 secs 2024-03-24T00:44:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 543.115659 secs 2024-03-24T00:44:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 543.115659 secs 2024-03-24T00:44:29.770 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1198 sec 2024-03-24T00:44:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:29 smithi106 ceph-mon[36505]: pgmap v782: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:29 smithi078 ceph-mon[32429]: pgmap v782: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:30.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:29 smithi107 ceph-mon[36485]: pgmap v782: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:30.770 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1199 sec 2024-03-24T00:44:31.770 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1200 sec 2024-03-24T00:44:31.896 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:31 smithi107 ceph-mon[36485]: pgmap v783: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:31.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:31 smithi106 ceph-mon[36505]: pgmap v783: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:32.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:31 smithi078 ceph-mon[32429]: pgmap v783: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:32.770 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1201 sec 2024-03-24T00:44:33.770 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1202 sec 2024-03-24T00:44:33.893 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:33 smithi078 ceph-mon[32429]: pgmap v784: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:33.893 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 548.115720 secs 2024-03-24T00:44:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:33 smithi106 ceph-mon[36505]: pgmap v784: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 548.115720 secs 2024-03-24T00:44:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:33 smithi107 ceph-mon[36485]: pgmap v784: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:34.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 548.115720 secs 2024-03-24T00:44:34.770 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1203 sec 2024-03-24T00:44:35.770 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1204 sec 2024-03-24T00:44:35.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:35 smithi106 ceph-mon[36505]: pgmap v785: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:36.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:35 smithi078 ceph-mon[32429]: pgmap v785: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:36.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:35 smithi107 ceph-mon[36485]: pgmap v785: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:36.770 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1205 sec 2024-03-24T00:44:37.771 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1206 sec 2024-03-24T00:44:37.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:37 smithi106 ceph-mon[36505]: pgmap v786: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:37 smithi078 ceph-mon[32429]: pgmap v786: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:38.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:37 smithi107 ceph-mon[36485]: pgmap v786: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:38.770 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1207 sec 2024-03-24T00:44:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 553.115810 secs 2024-03-24T00:44:38.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:38 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:44:38.992 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 553.115810 secs 2024-03-24T00:44:38.992 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:38 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:44:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 553.115810 secs 2024-03-24T00:44:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:38 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:44:39.771 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1208 sec 2024-03-24T00:44:39.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:39 smithi106 ceph-mon[36505]: pgmap v787: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:39.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:39 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:44:39.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:39 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:44:39.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:39 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:44:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:39 smithi078 ceph-mon[32429]: pgmap v787: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:39 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:44:40.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:39 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:44:40.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:39 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:44:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:39 smithi107 ceph-mon[36485]: pgmap v787: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:39 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:44:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:39 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:44:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:39 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:44:40.771 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1209 sec 2024-03-24T00:44:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:40 smithi106 ceph-mon[36505]: pgmap v788: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:44:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:40 smithi106 ceph-mon[36505]: pgmap v789: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:44:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:40 smithi078 ceph-mon[32429]: pgmap v788: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:44:41.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:40 smithi078 ceph-mon[32429]: pgmap v789: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:44:41.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:40 smithi107 ceph-mon[36485]: pgmap v788: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:44:41.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:40 smithi107 ceph-mon[36485]: pgmap v789: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:44:41.771 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1210 sec 2024-03-24T00:44:42.771 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1211 sec 2024-03-24T00:44:42.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:42 smithi106 ceph-mon[36505]: pgmap v790: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:44:43.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:42 smithi078 ceph-mon[32429]: pgmap v790: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:44:43.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:42 smithi107 ceph-mon[36485]: pgmap v790: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:44:43.771 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1212 sec 2024-03-24T00:44:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 558.115901 secs 2024-03-24T00:44:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 558.115901 secs 2024-03-24T00:44:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 558.115901 secs 2024-03-24T00:44:44.771 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1213 sec 2024-03-24T00:44:44.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:44 smithi106 ceph-mon[36505]: pgmap v791: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:44:45.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:44 smithi078 ceph-mon[32429]: pgmap v791: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:44:45.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:44 smithi107 ceph-mon[36485]: pgmap v791: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:44:45.771 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1214 sec 2024-03-24T00:44:46.771 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1215 sec 2024-03-24T00:44:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:46 smithi106 ceph-mon[36505]: pgmap v792: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:44:47.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:46 smithi078 ceph-mon[32429]: pgmap v792: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:44:47.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:46 smithi107 ceph-mon[36485]: pgmap v792: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:44:47.771 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1216 sec 2024-03-24T00:44:48.772 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1217 sec 2024-03-24T00:44:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:48 smithi106 ceph-mon[36505]: pgmap v793: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:44:48.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 563.115989 secs 2024-03-24T00:44:49.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:48 smithi078 ceph-mon[32429]: pgmap v793: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:44:49.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 563.115989 secs 2024-03-24T00:44:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:48 smithi107 ceph-mon[36485]: pgmap v793: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:44:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 563.115989 secs 2024-03-24T00:44:49.772 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1218 sec 2024-03-24T00:44:50.772 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1219 sec 2024-03-24T00:44:50.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:50 smithi106 ceph-mon[36505]: pgmap v794: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:44:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:50 smithi078 ceph-mon[32429]: pgmap v794: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:44:51.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:50 smithi107 ceph-mon[36485]: pgmap v794: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:44:51.772 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1220 sec 2024-03-24T00:44:52.772 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1221 sec 2024-03-24T00:44:53.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:52 smithi078 ceph-mon[32429]: pgmap v795: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:53.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:52 smithi107 ceph-mon[36485]: pgmap v795: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:53.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:52 smithi106 ceph-mon[36505]: pgmap v795: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:53.772 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1222 sec 2024-03-24T00:44:54.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 568.116031 secs 2024-03-24T00:44:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 568.116031 secs 2024-03-24T00:44:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 568.116031 secs 2024-03-24T00:44:54.772 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1223 sec 2024-03-24T00:44:55.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:54 smithi078 ceph-mon[32429]: pgmap v796: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:55.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:54 smithi107 ceph-mon[36485]: pgmap v796: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:55.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:54 smithi106 ceph-mon[36505]: pgmap v796: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:55.772 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1224 sec 2024-03-24T00:44:56.772 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1225 sec 2024-03-24T00:44:57.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:56 smithi078 ceph-mon[32429]: pgmap v797: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:56 smithi107 ceph-mon[36485]: pgmap v797: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:57.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:56 smithi106 ceph-mon[36505]: pgmap v797: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:44:57.772 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1226 sec 2024-03-24T00:44:58.772 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1227 sec 2024-03-24T00:44:59.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:58 smithi078 ceph-mon[32429]: pgmap v798: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:59.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:44:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 573.116070 secs 2024-03-24T00:44:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:58 smithi107 ceph-mon[36485]: pgmap v798: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:44:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 573.116070 secs 2024-03-24T00:44:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:58 smithi106 ceph-mon[36505]: pgmap v798: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:44:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 573.116070 secs 2024-03-24T00:44:59.773 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1228 sec 2024-03-24T00:45:00.772 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1229 sec 2024-03-24T00:45:01.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:00 smithi078 ceph-mon[32429]: pgmap v799: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:01.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:00 smithi107 ceph-mon[36485]: pgmap v799: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:01.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:00 smithi106 ceph-mon[36505]: pgmap v799: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:01.773 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1230 sec 2024-03-24T00:45:02.773 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1231 sec 2024-03-24T00:45:02.954 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:02 smithi106 ceph-mon[36505]: pgmap v800: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:03.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:02 smithi078 ceph-mon[32429]: pgmap v800: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:03.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:02 smithi107 ceph-mon[36485]: pgmap v800: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:03.773 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1232 sec 2024-03-24T00:45:04.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 578.116131 secs 2024-03-24T00:45:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 578.116131 secs 2024-03-24T00:45:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 578.116131 secs 2024-03-24T00:45:04.773 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1233 sec 2024-03-24T00:45:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:04 smithi078 ceph-mon[32429]: pgmap v801: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:05.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:04 smithi107 ceph-mon[36485]: pgmap v801: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:05.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:04 smithi106 ceph-mon[36505]: pgmap v801: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:05.773 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1234 sec 2024-03-24T00:45:06.773 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1235 sec 2024-03-24T00:45:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:06 smithi078 ceph-mon[32429]: pgmap v802: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:06 smithi107 ceph-mon[36485]: pgmap v802: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:07.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:06 smithi106 ceph-mon[36505]: pgmap v802: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:07.773 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1236 sec 2024-03-24T00:45:08.773 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1237 sec 2024-03-24T00:45:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:08 smithi078 ceph-mon[32429]: pgmap v803: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 583.116196 secs 2024-03-24T00:45:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:08 smithi107 ceph-mon[36485]: pgmap v803: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 583.116196 secs 2024-03-24T00:45:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:08 smithi106 ceph-mon[36505]: pgmap v803: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 583.116196 secs 2024-03-24T00:45:09.773 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1238 sec 2024-03-24T00:45:10.773 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1239 sec 2024-03-24T00:45:11.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:10 smithi078 ceph-mon[32429]: pgmap v804: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:11.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:10 smithi107 ceph-mon[36485]: pgmap v804: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:11.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:10 smithi106 ceph-mon[36505]: pgmap v804: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:11.773 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1240 sec 2024-03-24T00:45:12.774 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1241 sec 2024-03-24T00:45:13.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:12 smithi078 ceph-mon[32429]: pgmap v805: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:13.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:12 smithi107 ceph-mon[36485]: pgmap v805: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:13.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:12 smithi106 ceph-mon[36505]: pgmap v805: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:13.774 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1242 sec 2024-03-24T00:45:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 588.116292 secs 2024-03-24T00:45:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 588.116292 secs 2024-03-24T00:45:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 588.116292 secs 2024-03-24T00:45:14.774 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1243 sec 2024-03-24T00:45:15.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:14 smithi078 ceph-mon[32429]: pgmap v806: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:15.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:14 smithi107 ceph-mon[36485]: pgmap v806: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:15.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:14 smithi106 ceph-mon[36505]: pgmap v806: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:15.774 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1244 sec 2024-03-24T00:45:16.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:15 smithi078 ceph-mon[32429]: pgmap v807: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:16.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:15 smithi107 ceph-mon[36485]: pgmap v807: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:16.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:15 smithi106 ceph-mon[36505]: pgmap v807: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:16.774 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1245 sec 2024-03-24T00:45:17.774 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1246 sec 2024-03-24T00:45:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:17 smithi078 ceph-mon[32429]: pgmap v808: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:17 smithi107 ceph-mon[36485]: pgmap v808: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:18.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:17 smithi106 ceph-mon[36505]: pgmap v808: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:18.774 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1247 sec 2024-03-24T00:45:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 593.116376 secs 2024-03-24T00:45:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 593.116376 secs 2024-03-24T00:45:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 593.116376 secs 2024-03-24T00:45:19.774 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1248 sec 2024-03-24T00:45:20.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:19 smithi078 ceph-mon[32429]: pgmap v809: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:19 smithi107 ceph-mon[36485]: pgmap v809: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:20.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:19 smithi106 ceph-mon[36505]: pgmap v809: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:20.775 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1249 sec 2024-03-24T00:45:21.775 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1250 sec 2024-03-24T00:45:22.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:21 smithi078 ceph-mon[32429]: pgmap v810: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:22.349 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:21 smithi107 ceph-mon[36485]: pgmap v810: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:22.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:21 smithi106 ceph-mon[36505]: pgmap v810: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:22.774 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1251 sec 2024-03-24T00:45:23.775 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1252 sec 2024-03-24T00:45:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 598.116438 secs 2024-03-24T00:45:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:23 smithi078 ceph-mon[32429]: pgmap v811: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 598.116438 secs 2024-03-24T00:45:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:23 smithi107 ceph-mon[36485]: pgmap v811: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 598.116438 secs 2024-03-24T00:45:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:23 smithi106 ceph-mon[36505]: pgmap v811: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:24.775 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1253 sec 2024-03-24T00:45:25.775 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1254 sec 2024-03-24T00:45:26.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:25 smithi078 ceph-mon[32429]: pgmap v812: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:26.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:25 smithi107 ceph-mon[36485]: pgmap v812: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:26.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:25 smithi106 ceph-mon[36505]: pgmap v812: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:26.775 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1255 sec 2024-03-24T00:45:27.775 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1256 sec 2024-03-24T00:45:28.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:27 smithi078 ceph-mon[32429]: pgmap v813: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:28.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:27 smithi107 ceph-mon[36485]: pgmap v813: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:28.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:27 smithi106 ceph-mon[36505]: pgmap v813: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:28.776 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1257 sec 2024-03-24T00:45:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 603.116500 secs 2024-03-24T00:45:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 603.116500 secs 2024-03-24T00:45:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 603.116500 secs 2024-03-24T00:45:29.775 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1258 sec 2024-03-24T00:45:30.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:30 smithi078 ceph-mon[32429]: pgmap v814: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:30.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:30 smithi107 ceph-mon[36485]: pgmap v814: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:30.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:30 smithi106 ceph-mon[36505]: pgmap v814: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:30.775 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1259 sec 2024-03-24T00:45:31.775 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1260 sec 2024-03-24T00:45:32.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:32 smithi078 ceph-mon[32429]: pgmap v815: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:32.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:32 smithi107 ceph-mon[36485]: pgmap v815: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:32.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:32 smithi106 ceph-mon[36505]: pgmap v815: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:32.775 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1261 sec 2024-03-24T00:45:33.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 608.116589 secs 2024-03-24T00:45:33.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 608.116589 secs 2024-03-24T00:45:33.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 608.116589 secs 2024-03-24T00:45:33.776 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1262 sec 2024-03-24T00:45:34.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:34 smithi078 ceph-mon[32429]: pgmap v816: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:34 smithi107 ceph-mon[36485]: pgmap v816: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:34 smithi106 ceph-mon[36505]: pgmap v816: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:34.776 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1263 sec 2024-03-24T00:45:35.776 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1264 sec 2024-03-24T00:45:36.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:36 smithi078 ceph-mon[32429]: pgmap v817: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:36.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:36 smithi107 ceph-mon[36485]: pgmap v817: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:36.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:36 smithi106 ceph-mon[36505]: pgmap v817: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:36.776 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1265 sec 2024-03-24T00:45:37.776 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1266 sec 2024-03-24T00:45:38.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:38 smithi078 ceph-mon[32429]: pgmap v818: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:38.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 613.116676 secs 2024-03-24T00:45:38.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:38 smithi107 ceph-mon[36485]: pgmap v818: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:38.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 613.116676 secs 2024-03-24T00:45:38.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:38 smithi106 ceph-mon[36505]: pgmap v818: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:38.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 613.116676 secs 2024-03-24T00:45:38.776 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1267 sec 2024-03-24T00:45:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:39 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:45:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:39 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:45:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:39 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:45:39.776 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1268 sec 2024-03-24T00:45:40.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:40 smithi078 ceph-mon[32429]: pgmap v819: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:40 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:45:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:40 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:45:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:40 smithi078 ceph-mon[32429]: pgmap v820: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:45:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:40 smithi078 ceph-mon[32429]: pgmap v821: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:45:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:40 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:45:40.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:40 smithi107 ceph-mon[36485]: pgmap v819: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:40.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:40 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:45:40.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:40 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:45:40.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:40 smithi107 ceph-mon[36485]: pgmap v820: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:45:40.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:40 smithi107 ceph-mon[36485]: pgmap v821: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:45:40.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:40 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:45:40.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:40 smithi106 ceph-mon[36505]: pgmap v819: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:40.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:40 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:45:40.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:40 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:45:40.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:40 smithi106 ceph-mon[36505]: pgmap v820: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:45:40.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:40 smithi106 ceph-mon[36505]: pgmap v821: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:45:40.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:40 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:45:40.776 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1269 sec 2024-03-24T00:45:41.776 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1270 sec 2024-03-24T00:45:42.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:41 smithi078 ceph-mon[32429]: pgmap v822: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:45:42.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:41 smithi107 ceph-mon[36485]: pgmap v822: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:45:42.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:41 smithi106 ceph-mon[36505]: pgmap v822: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:45:42.776 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1271 sec 2024-03-24T00:45:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 618.116736 secs 2024-03-24T00:45:43.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 618.116736 secs 2024-03-24T00:45:43.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 618.116736 secs 2024-03-24T00:45:43.776 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1272 sec 2024-03-24T00:45:44.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:44 smithi078 ceph-mon[32429]: pgmap v823: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:45:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:44 smithi107 ceph-mon[36485]: pgmap v823: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:45:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:44 smithi106 ceph-mon[36505]: pgmap v823: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:45:44.777 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1273 sec 2024-03-24T00:45:45.777 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1274 sec 2024-03-24T00:45:46.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:45 smithi078 ceph-mon[32429]: pgmap v824: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:45:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:45 smithi107 ceph-mon[36485]: pgmap v824: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:45:46.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:45 smithi106 ceph-mon[36505]: pgmap v824: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:45:46.777 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1275 sec 2024-03-24T00:45:47.777 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1276 sec 2024-03-24T00:45:48.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:47 smithi078 ceph-mon[32429]: pgmap v825: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:45:48.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:47 smithi107 ceph-mon[36485]: pgmap v825: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:45:48.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:47 smithi106 ceph-mon[36505]: pgmap v825: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:45:48.777 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1277 sec 2024-03-24T00:45:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 623.116800 secs 2024-03-24T00:45:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 623.116800 secs 2024-03-24T00:45:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 623.116800 secs 2024-03-24T00:45:49.777 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1278 sec 2024-03-24T00:45:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:49 smithi078 ceph-mon[32429]: pgmap v826: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:45:50.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:49 smithi107 ceph-mon[36485]: pgmap v826: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:45:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:49 smithi106 ceph-mon[36505]: pgmap v826: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:45:50.777 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1279 sec 2024-03-24T00:45:51.777 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1280 sec 2024-03-24T00:45:52.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:51 smithi078 ceph-mon[32429]: pgmap v827: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:52.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:51 smithi107 ceph-mon[36485]: pgmap v827: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:52.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:51 smithi106 ceph-mon[36505]: pgmap v827: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:52.778 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1281 sec 2024-03-24T00:45:53.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 628.116887 secs 2024-03-24T00:45:53.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 628.116887 secs 2024-03-24T00:45:53.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 628.116887 secs 2024-03-24T00:45:53.777 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1282 sec 2024-03-24T00:45:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:54 smithi078 ceph-mon[32429]: pgmap v828: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:54 smithi107 ceph-mon[36485]: pgmap v828: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:54 smithi106 ceph-mon[36505]: pgmap v828: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:54.777 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1283 sec 2024-03-24T00:45:55.777 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1284 sec 2024-03-24T00:45:56.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:55 smithi078 ceph-mon[32429]: pgmap v829: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:56.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:55 smithi107 ceph-mon[36485]: pgmap v829: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:56.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:55 smithi106 ceph-mon[36505]: pgmap v829: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:56.778 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1285 sec 2024-03-24T00:45:57.778 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1286 sec 2024-03-24T00:45:58.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:57 smithi078 ceph-mon[32429]: pgmap v830: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:58.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:57 smithi107 ceph-mon[36485]: pgmap v830: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:58.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:57 smithi106 ceph-mon[36505]: pgmap v830: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:45:58.778 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1287 sec 2024-03-24T00:45:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 633.116976 secs 2024-03-24T00:45:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 633.116976 secs 2024-03-24T00:45:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 633.116976 secs 2024-03-24T00:45:59.778 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1288 sec 2024-03-24T00:46:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:45:59 smithi078 ceph-mon[32429]: pgmap v831: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:00.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:45:59 smithi107 ceph-mon[36485]: pgmap v831: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:00.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:45:59 smithi106 ceph-mon[36505]: pgmap v831: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:00.778 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1289 sec 2024-03-24T00:46:01.778 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1290 sec 2024-03-24T00:46:02.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:01 smithi078 ceph-mon[32429]: pgmap v832: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:02.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:01 smithi107 ceph-mon[36485]: pgmap v832: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:02.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:01 smithi106 ceph-mon[36505]: pgmap v832: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:02.778 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1291 sec 2024-03-24T00:46:03.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 638.117059 secs 2024-03-24T00:46:03.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 638.117059 secs 2024-03-24T00:46:03.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 638.117059 secs 2024-03-24T00:46:03.778 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1292 sec 2024-03-24T00:46:04.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:04 smithi078 ceph-mon[32429]: pgmap v833: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:04 smithi107 ceph-mon[36485]: pgmap v833: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:04 smithi106 ceph-mon[36505]: pgmap v833: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:04.778 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1293 sec 2024-03-24T00:46:05.778 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1294 sec 2024-03-24T00:46:06.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:05 smithi078 ceph-mon[32429]: pgmap v834: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:05 smithi107 ceph-mon[36485]: pgmap v834: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:06.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:05 smithi106 ceph-mon[36505]: pgmap v834: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:06.779 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1295 sec 2024-03-24T00:46:07.778 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1296 sec 2024-03-24T00:46:08.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:07 smithi078 ceph-mon[32429]: pgmap v835: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:08.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:07 smithi107 ceph-mon[36485]: pgmap v835: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:08.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:07 smithi106 ceph-mon[36505]: pgmap v835: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:08.779 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1297 sec 2024-03-24T00:46:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 643.117150 secs 2024-03-24T00:46:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 643.117150 secs 2024-03-24T00:46:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 643.117150 secs 2024-03-24T00:46:09.779 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1298 sec 2024-03-24T00:46:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:09 smithi078 ceph-mon[32429]: pgmap v836: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:10.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:09 smithi107 ceph-mon[36485]: pgmap v836: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:10.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:09 smithi106 ceph-mon[36505]: pgmap v836: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:10.779 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1299 sec 2024-03-24T00:46:11.779 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1300 sec 2024-03-24T00:46:12.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:11 smithi078 ceph-mon[32429]: pgmap v837: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:12.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:11 smithi107 ceph-mon[36485]: pgmap v837: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:12.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:11 smithi106 ceph-mon[36505]: pgmap v837: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:12.779 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1301 sec 2024-03-24T00:46:13.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 648.117233 secs 2024-03-24T00:46:13.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 648.117233 secs 2024-03-24T00:46:13.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 648.117233 secs 2024-03-24T00:46:13.779 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1302 sec 2024-03-24T00:46:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:14 smithi078 ceph-mon[32429]: pgmap v838: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:14 smithi107 ceph-mon[36485]: pgmap v838: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:14 smithi106 ceph-mon[36505]: pgmap v838: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:14.779 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1303 sec 2024-03-24T00:46:15.779 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1304 sec 2024-03-24T00:46:16.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:15 smithi078 ceph-mon[32429]: pgmap v839: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:16.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:15 smithi107 ceph-mon[36485]: pgmap v839: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:16.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:15 smithi106 ceph-mon[36505]: pgmap v839: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:16.779 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1305 sec 2024-03-24T00:46:17.779 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1306 sec 2024-03-24T00:46:18.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:17 smithi078 ceph-mon[32429]: pgmap v840: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:18.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:17 smithi107 ceph-mon[36485]: pgmap v840: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:18.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:17 smithi106 ceph-mon[36505]: pgmap v840: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:18.780 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1307 sec 2024-03-24T00:46:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 653.117298 secs 2024-03-24T00:46:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 653.117298 secs 2024-03-24T00:46:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 653.117298 secs 2024-03-24T00:46:19.780 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1308 sec 2024-03-24T00:46:20.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:19 smithi078 ceph-mon[32429]: pgmap v841: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:20.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:19 smithi107 ceph-mon[36485]: pgmap v841: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:20.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:19 smithi106 ceph-mon[36505]: pgmap v841: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:20.780 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1309 sec 2024-03-24T00:46:21.780 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1310 sec 2024-03-24T00:46:22.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:21 smithi078 ceph-mon[32429]: pgmap v842: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:22.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:21 smithi107 ceph-mon[36485]: pgmap v842: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:22.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:21 smithi106 ceph-mon[36505]: pgmap v842: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:22.780 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1311 sec 2024-03-24T00:46:23.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 658.117385 secs 2024-03-24T00:46:23.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 658.117385 secs 2024-03-24T00:46:23.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 658.117385 secs 2024-03-24T00:46:23.780 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1312 sec 2024-03-24T00:46:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:24 smithi078 ceph-mon[32429]: pgmap v843: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:24 smithi107 ceph-mon[36485]: pgmap v843: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:24 smithi106 ceph-mon[36505]: pgmap v843: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:24.780 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1313 sec 2024-03-24T00:46:25.780 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1314 sec 2024-03-24T00:46:26.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:25 smithi078 ceph-mon[32429]: pgmap v844: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:26.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:25 smithi107 ceph-mon[36485]: pgmap v844: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:26.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:25 smithi106 ceph-mon[36505]: pgmap v844: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:26.780 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1315 sec 2024-03-24T00:46:27.780 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1316 sec 2024-03-24T00:46:28.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:27 smithi078 ceph-mon[32429]: pgmap v845: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:27 smithi107 ceph-mon[36485]: pgmap v845: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:28.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:27 smithi106 ceph-mon[36505]: pgmap v845: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:28.780 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1317 sec 2024-03-24T00:46:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 663.117447 secs 2024-03-24T00:46:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 663.117447 secs 2024-03-24T00:46:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 663.117447 secs 2024-03-24T00:46:29.781 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1318 sec 2024-03-24T00:46:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:29 smithi078 ceph-mon[32429]: pgmap v846: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:30.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:29 smithi107 ceph-mon[36485]: pgmap v846: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:30.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:29 smithi106 ceph-mon[36505]: pgmap v846: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:30.781 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1319 sec 2024-03-24T00:46:31.781 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1320 sec 2024-03-24T00:46:32.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:31 smithi078 ceph-mon[32429]: pgmap v847: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:32.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:31 smithi107 ceph-mon[36485]: pgmap v847: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:32.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:31 smithi106 ceph-mon[36505]: pgmap v847: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:32.781 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1321 sec 2024-03-24T00:46:33.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 668.117505 secs 2024-03-24T00:46:33.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 668.117505 secs 2024-03-24T00:46:33.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 668.117505 secs 2024-03-24T00:46:33.781 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1322 sec 2024-03-24T00:46:34.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:34 smithi078 ceph-mon[32429]: pgmap v848: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:34 smithi107 ceph-mon[36485]: pgmap v848: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:34 smithi106 ceph-mon[36505]: pgmap v848: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:34.781 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1323 sec 2024-03-24T00:46:35.781 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1324 sec 2024-03-24T00:46:36.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:35 smithi078 ceph-mon[32429]: pgmap v849: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:36.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:35 smithi107 ceph-mon[36485]: pgmap v849: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:36.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:35 smithi106 ceph-mon[36505]: pgmap v849: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:36.781 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1325 sec 2024-03-24T00:46:37.781 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1326 sec 2024-03-24T00:46:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:37 smithi078 ceph-mon[32429]: pgmap v850: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:38.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:37 smithi107 ceph-mon[36485]: pgmap v850: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:38.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:37 smithi106 ceph-mon[36505]: pgmap v850: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:38.781 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1327 sec 2024-03-24T00:46:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 673.117583 secs 2024-03-24T00:46:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 673.117583 secs 2024-03-24T00:46:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 673.117583 secs 2024-03-24T00:46:39.781 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1328 sec 2024-03-24T00:46:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:39 smithi078 ceph-mon[32429]: pgmap v851: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:39 smithi107 ceph-mon[36485]: pgmap v851: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:39 smithi106 ceph-mon[36505]: pgmap v851: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:40.781 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1329 sec 2024-03-24T00:46:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:40 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:46:41.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:40 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:46:41.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:40 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:46:41.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:40 smithi078 ceph-mon[32429]: pgmap v852: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T00:46:41.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:40 smithi078 ceph-mon[32429]: pgmap v853: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:46:41.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:40 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:46:41.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:40 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:46:41.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:40 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:46:41.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:40 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:46:41.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:40 smithi107 ceph-mon[36485]: pgmap v852: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T00:46:41.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:40 smithi107 ceph-mon[36485]: pgmap v853: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:46:41.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:40 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:46:41.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:40 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:46:41.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:40 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:46:41.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:40 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:46:41.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:40 smithi106 ceph-mon[36505]: pgmap v852: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T00:46:41.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:40 smithi106 ceph-mon[36505]: pgmap v853: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:46:41.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:40 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:46:41.781 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1330 sec 2024-03-24T00:46:42.782 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1331 sec 2024-03-24T00:46:43.782 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1332 sec 2024-03-24T00:46:43.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:43 smithi078 ceph-mon[32429]: pgmap v854: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:46:43.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 678.117642 secs 2024-03-24T00:46:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:43 smithi107 ceph-mon[36485]: pgmap v854: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:46:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 678.117642 secs 2024-03-24T00:46:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:43 smithi106 ceph-mon[36505]: pgmap v854: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:46:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 678.117642 secs 2024-03-24T00:46:44.782 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1333 sec 2024-03-24T00:46:45.782 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1334 sec 2024-03-24T00:46:45.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:45 smithi078 ceph-mon[32429]: pgmap v855: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:46:45.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:45 smithi107 ceph-mon[36485]: pgmap v855: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:46:45.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:45 smithi106 ceph-mon[36505]: pgmap v855: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:46:46.782 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1335 sec 2024-03-24T00:46:47.782 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1336 sec 2024-03-24T00:46:47.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:47 smithi078 ceph-mon[32429]: pgmap v856: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:46:47.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:47 smithi107 ceph-mon[36485]: pgmap v856: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:46:47.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:47 smithi106 ceph-mon[36505]: pgmap v856: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:46:48.782 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1337 sec 2024-03-24T00:46:48.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 683.117701 secs 2024-03-24T00:46:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 683.117701 secs 2024-03-24T00:46:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 683.117701 secs 2024-03-24T00:46:49.782 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1338 sec 2024-03-24T00:46:49.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:49 smithi078 ceph-mon[32429]: pgmap v857: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:46:49.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:49 smithi107 ceph-mon[36485]: pgmap v857: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:46:49.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:49 smithi106 ceph-mon[36505]: pgmap v857: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:46:50.782 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1339 sec 2024-03-24T00:46:51.782 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1340 sec 2024-03-24T00:46:51.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:51 smithi078 ceph-mon[32429]: pgmap v858: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:46:51.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:51 smithi107 ceph-mon[36485]: pgmap v858: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:46:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:51 smithi106 ceph-mon[36505]: pgmap v858: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:46:52.783 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1341 sec 2024-03-24T00:46:53.783 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1342 sec 2024-03-24T00:46:53.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:53 smithi078 ceph-mon[32429]: pgmap v859: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:53.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 688.117761 secs 2024-03-24T00:46:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:53 smithi107 ceph-mon[36485]: pgmap v859: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 688.117761 secs 2024-03-24T00:46:53.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:53 smithi106 ceph-mon[36505]: pgmap v859: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:53.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 688.117761 secs 2024-03-24T00:46:54.783 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1343 sec 2024-03-24T00:46:55.783 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1344 sec 2024-03-24T00:46:55.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:55 smithi078 ceph-mon[32429]: pgmap v860: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:55.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:55 smithi107 ceph-mon[36485]: pgmap v860: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:55.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:55 smithi106 ceph-mon[36505]: pgmap v860: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:56.783 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1345 sec 2024-03-24T00:46:57.783 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1346 sec 2024-03-24T00:46:57.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:57 smithi078 ceph-mon[32429]: pgmap v861: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:57.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:57 smithi107 ceph-mon[36485]: pgmap v861: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:57.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:57 smithi106 ceph-mon[36505]: pgmap v861: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:58.783 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1347 sec 2024-03-24T00:46:58.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 693.117801 secs 2024-03-24T00:46:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 693.117801 secs 2024-03-24T00:46:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 693.117801 secs 2024-03-24T00:46:59.783 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1348 sec 2024-03-24T00:46:59.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:46:59 smithi078 ceph-mon[32429]: pgmap v862: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:46:59 smithi107 ceph-mon[36485]: pgmap v862: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:46:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:46:59 smithi106 ceph-mon[36505]: pgmap v862: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:00.783 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1349 sec 2024-03-24T00:47:01.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:01 smithi106 ceph-mon[36505]: pgmap v863: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:01.783 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1350 sec 2024-03-24T00:47:01.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:01 smithi078 ceph-mon[32429]: pgmap v863: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:01.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:01 smithi107 ceph-mon[36485]: pgmap v863: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:02.784 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1351 sec 2024-03-24T00:47:03.784 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1352 sec 2024-03-24T00:47:03.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:03 smithi078 ceph-mon[32429]: pgmap v864: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:03.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 698.117885 secs 2024-03-24T00:47:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:03 smithi107 ceph-mon[36485]: pgmap v864: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 698.117885 secs 2024-03-24T00:47:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:03 smithi106 ceph-mon[36505]: pgmap v864: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 698.117885 secs 2024-03-24T00:47:04.784 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1353 sec 2024-03-24T00:47:05.784 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1354 sec 2024-03-24T00:47:05.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:05 smithi078 ceph-mon[32429]: pgmap v865: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:05.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:05 smithi107 ceph-mon[36485]: pgmap v865: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:05.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:05 smithi106 ceph-mon[36505]: pgmap v865: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:06.784 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1355 sec 2024-03-24T00:47:07.784 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1356 sec 2024-03-24T00:47:07.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:07 smithi078 ceph-mon[32429]: pgmap v866: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:07.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:07 smithi107 ceph-mon[36485]: pgmap v866: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:07.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:07 smithi106 ceph-mon[36505]: pgmap v866: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:08.785 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1357 sec 2024-03-24T00:47:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 703.117926 secs 2024-03-24T00:47:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 703.117926 secs 2024-03-24T00:47:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 703.117926 secs 2024-03-24T00:47:09.784 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1358 sec 2024-03-24T00:47:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:09 smithi078 ceph-mon[32429]: pgmap v867: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:09.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:09 smithi107 ceph-mon[36485]: pgmap v867: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:09.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:09 smithi106 ceph-mon[36505]: pgmap v867: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:10.784 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1359 sec 2024-03-24T00:47:11.784 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1360 sec 2024-03-24T00:47:11.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:11 smithi078 ceph-mon[32429]: pgmap v868: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:11 smithi107 ceph-mon[36485]: pgmap v868: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:11.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:11 smithi106 ceph-mon[36505]: pgmap v868: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:12.785 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1361 sec 2024-03-24T00:47:13.785 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1362 sec 2024-03-24T00:47:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:13 smithi078 ceph-mon[32429]: pgmap v869: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:13.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 708.118011 secs 2024-03-24T00:47:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:13 smithi107 ceph-mon[36485]: pgmap v869: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 708.118011 secs 2024-03-24T00:47:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:13 smithi106 ceph-mon[36505]: pgmap v869: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 708.118011 secs 2024-03-24T00:47:14.785 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1363 sec 2024-03-24T00:47:15.785 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1364 sec 2024-03-24T00:47:15.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:15 smithi078 ceph-mon[32429]: pgmap v870: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:15 smithi107 ceph-mon[36485]: pgmap v870: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:15 smithi106 ceph-mon[36505]: pgmap v870: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:16.785 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1365 sec 2024-03-24T00:47:17.785 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1366 sec 2024-03-24T00:47:17.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:17 smithi078 ceph-mon[32429]: pgmap v871: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:17.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:17 smithi107 ceph-mon[36485]: pgmap v871: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:17.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:17 smithi106 ceph-mon[36505]: pgmap v871: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:18.785 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1367 sec 2024-03-24T00:47:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 713.118100 secs 2024-03-24T00:47:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 713.118100 secs 2024-03-24T00:47:18.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 713.118100 secs 2024-03-24T00:47:19.785 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1368 sec 2024-03-24T00:47:19.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:19 smithi078 ceph-mon[32429]: pgmap v872: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:19 smithi107 ceph-mon[36485]: pgmap v872: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:19.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:19 smithi106 ceph-mon[36505]: pgmap v872: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:20.785 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1369 sec 2024-03-24T00:47:21.785 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1370 sec 2024-03-24T00:47:21.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:21 smithi078 ceph-mon[32429]: pgmap v873: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:21 smithi107 ceph-mon[36485]: pgmap v873: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:21.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:21 smithi106 ceph-mon[36505]: pgmap v873: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:22.785 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1371 sec 2024-03-24T00:47:23.785 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1372 sec 2024-03-24T00:47:23.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:23 smithi078 ceph-mon[32429]: pgmap v874: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:23.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 718.118160 secs 2024-03-24T00:47:23.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:23 smithi107 ceph-mon[36485]: pgmap v874: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:23.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 718.118160 secs 2024-03-24T00:47:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:23 smithi106 ceph-mon[36505]: pgmap v874: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:23.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 718.118160 secs 2024-03-24T00:47:24.786 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1373 sec 2024-03-24T00:47:25.786 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1374 sec 2024-03-24T00:47:25.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:25 smithi078 ceph-mon[32429]: pgmap v875: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:25.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:25 smithi107 ceph-mon[36485]: pgmap v875: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:25.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:25 smithi106 ceph-mon[36505]: pgmap v875: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:26.786 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1375 sec 2024-03-24T00:47:27.786 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1376 sec 2024-03-24T00:47:27.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:27 smithi107 ceph-mon[36485]: pgmap v876: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:27.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:27 smithi106 ceph-mon[36505]: pgmap v876: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:28.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:27 smithi078 ceph-mon[32429]: pgmap v876: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:28.786 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1377 sec 2024-03-24T00:47:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:28 smithi106 ceph-mon[36505]: 2 slow requests, 1 included below; oldest blocked for > 723.118248 secs 2024-03-24T00:47:28.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:28 smithi106 ceph-mon[36505]: slow request 723.118247 seconds old, received at 2024-03-24T00:35:24.775244+0000: peer_request:client.15462:3775 currently acquired locks 2024-03-24T00:47:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:28 smithi078 ceph-mon[32429]: 2 slow requests, 1 included below; oldest blocked for > 723.118248 secs 2024-03-24T00:47:29.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:28 smithi078 ceph-mon[32429]: slow request 723.118247 seconds old, received at 2024-03-24T00:35:24.775244+0000: peer_request:client.15462:3775 currently acquired locks 2024-03-24T00:47:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:28 smithi107 ceph-mon[36485]: 2 slow requests, 1 included below; oldest blocked for > 723.118248 secs 2024-03-24T00:47:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:28 smithi107 ceph-mon[36485]: slow request 723.118247 seconds old, received at 2024-03-24T00:35:24.775244+0000: peer_request:client.15462:3775 currently acquired locks 2024-03-24T00:47:29.786 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1378 sec 2024-03-24T00:47:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:29 smithi106 ceph-mon[36505]: pgmap v877: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:29 smithi078 ceph-mon[32429]: pgmap v877: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:30.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:29 smithi107 ceph-mon[36485]: pgmap v877: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:30.786 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1379 sec 2024-03-24T00:47:31.786 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1380 sec 2024-03-24T00:47:31.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:31 smithi106 ceph-mon[36505]: pgmap v878: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:32.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:31 smithi078 ceph-mon[32429]: pgmap v878: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:32.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:31 smithi107 ceph-mon[36485]: pgmap v878: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:32.786 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1381 sec 2024-03-24T00:47:33.786 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1382 sec 2024-03-24T00:47:33.895 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:33 smithi107 ceph-mon[36485]: pgmap v879: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:33.895 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 728.118295 secs 2024-03-24T00:47:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:33 smithi106 ceph-mon[36505]: pgmap v879: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 728.118295 secs 2024-03-24T00:47:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:33 smithi078 ceph-mon[32429]: pgmap v879: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 728.118295 secs 2024-03-24T00:47:34.786 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1383 sec 2024-03-24T00:47:35.787 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1384 sec 2024-03-24T00:47:35.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:35 smithi106 ceph-mon[36505]: pgmap v880: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:36.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:35 smithi078 ceph-mon[32429]: pgmap v880: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:36.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:35 smithi107 ceph-mon[36485]: pgmap v880: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:36.787 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1385 sec 2024-03-24T00:47:37.787 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1386 sec 2024-03-24T00:47:37.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:37 smithi106 ceph-mon[36505]: pgmap v881: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:37 smithi078 ceph-mon[32429]: pgmap v881: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:38.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:37 smithi107 ceph-mon[36485]: pgmap v881: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:38.787 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1387 sec 2024-03-24T00:47:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 733.118321 secs 2024-03-24T00:47:38.992 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 733.118321 secs 2024-03-24T00:47:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 733.118321 secs 2024-03-24T00:47:39.787 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1388 sec 2024-03-24T00:47:39.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:39 smithi106 ceph-mon[36505]: pgmap v882: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:39 smithi078 ceph-mon[32429]: pgmap v882: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:39 smithi107 ceph-mon[36485]: pgmap v882: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:40.787 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1389 sec 2024-03-24T00:47:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:40 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:47:40.992 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:40 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:47:41.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:40 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:47:41.787 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1390 sec 2024-03-24T00:47:41.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:41 smithi106 ceph-mon[36505]: pgmap v883: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:41.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:41 smithi078 ceph-mon[32429]: pgmap v883: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:42.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:41 smithi107 ceph-mon[36485]: pgmap v883: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:42.787 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1391 sec 2024-03-24T00:47:43.787 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1392 sec 2024-03-24T00:47:43.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:43 smithi078 ceph-mon[32429]: pgmap v884: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:43.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 738.118379 secs 2024-03-24T00:47:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:43 smithi106 ceph-mon[36505]: pgmap v884: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:43.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 738.118379 secs 2024-03-24T00:47:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:43 smithi107 ceph-mon[36485]: pgmap v884: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:44.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 738.118379 secs 2024-03-24T00:47:44.787 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1393 sec 2024-03-24T00:47:45.788 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1394 sec 2024-03-24T00:47:45.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:45 smithi106 ceph-mon[36505]: pgmap v885: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:45.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:47:45.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:47:45.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:47:45.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:47:45.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:47:45.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:47:45.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:47:46.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:45 smithi078 ceph-mon[32429]: pgmap v885: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:46.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:47:46.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:47:46.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:47:46.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:47:46.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:47:46.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:47:46.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:47:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:45 smithi107 ceph-mon[36485]: pgmap v885: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:47:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:47:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:47:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:47:46.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:47:46.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:47:46.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:47:46.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:47:46.787 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1395 sec 2024-03-24T00:47:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:46 smithi106 ceph-mon[36505]: pgmap v886: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:47:46.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:46 smithi106 ceph-mon[36505]: pgmap v887: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:47:47.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:46 smithi078 ceph-mon[32429]: pgmap v886: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:47:47.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:46 smithi078 ceph-mon[32429]: pgmap v887: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:47:47.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:46 smithi107 ceph-mon[36485]: pgmap v886: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:47:47.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:46 smithi107 ceph-mon[36485]: pgmap v887: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:47:47.789 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1396 sec 2024-03-24T00:47:48.788 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1397 sec 2024-03-24T00:47:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:48 smithi106 ceph-mon[36505]: pgmap v888: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:47:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 743.118437 secs 2024-03-24T00:47:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:48 smithi078 ceph-mon[32429]: pgmap v888: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:47:49.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 743.118437 secs 2024-03-24T00:47:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:48 smithi107 ceph-mon[36485]: pgmap v888: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:47:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 743.118437 secs 2024-03-24T00:47:49.788 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1398 sec 2024-03-24T00:47:50.788 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1399 sec 2024-03-24T00:47:50.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:50 smithi106 ceph-mon[36505]: pgmap v889: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:47:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:50 smithi078 ceph-mon[32429]: pgmap v889: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:47:51.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:50 smithi107 ceph-mon[36485]: pgmap v889: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:47:51.788 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1400 sec 2024-03-24T00:47:52.788 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1401 sec 2024-03-24T00:47:52.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:52 smithi106 ceph-mon[36505]: pgmap v890: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:47:53.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:52 smithi078 ceph-mon[32429]: pgmap v890: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:47:53.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:52 smithi107 ceph-mon[36485]: pgmap v890: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:47:53.788 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1402 sec 2024-03-24T00:47:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 748.118514 secs 2024-03-24T00:47:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 748.118514 secs 2024-03-24T00:47:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 748.118514 secs 2024-03-24T00:47:54.788 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1403 sec 2024-03-24T00:47:54.992 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:54 smithi106 ceph-mon[36505]: pgmap v891: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:47:55.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:54 smithi078 ceph-mon[32429]: pgmap v891: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:47:55.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:54 smithi107 ceph-mon[36485]: pgmap v891: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:47:55.788 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1404 sec 2024-03-24T00:47:56.788 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1405 sec 2024-03-24T00:47:57.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:56 smithi078 ceph-mon[32429]: pgmap v892: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:47:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:56 smithi107 ceph-mon[36485]: pgmap v892: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:47:57.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:56 smithi106 ceph-mon[36505]: pgmap v892: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:47:57.788 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1406 sec 2024-03-24T00:47:58.789 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1407 sec 2024-03-24T00:47:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:58 smithi078 ceph-mon[32429]: pgmap v893: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:59.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:47:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 753.118588 secs 2024-03-24T00:47:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:58 smithi107 ceph-mon[36485]: pgmap v893: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:47:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 753.118588 secs 2024-03-24T00:47:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:58 smithi106 ceph-mon[36505]: pgmap v893: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:59.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:47:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 753.118588 secs 2024-03-24T00:47:59.788 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1408 sec 2024-03-24T00:48:00.789 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1409 sec 2024-03-24T00:48:01.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:00 smithi078 ceph-mon[32429]: pgmap v894: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:01.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:00 smithi107 ceph-mon[36485]: pgmap v894: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:01.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:00 smithi106 ceph-mon[36505]: pgmap v894: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:01.789 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1410 sec 2024-03-24T00:48:02.789 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1411 sec 2024-03-24T00:48:03.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:02 smithi078 ceph-mon[32429]: pgmap v895: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:03.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:02 smithi107 ceph-mon[36485]: pgmap v895: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:03.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:02 smithi106 ceph-mon[36505]: pgmap v895: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:03.790 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1412 sec 2024-03-24T00:48:03.998 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:03 smithi107 ceph-mon[36485]: 2 slow requests, 1 included below; oldest blocked for > 758.118675 secs 2024-03-24T00:48:03.998 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:03 smithi107 ceph-mon[36485]: slow request 722.851603 seconds old, received at 2024-03-24T00:36:00.042315+0000: client_request(client.25385:166 mksnap #0x1//scheduled-2024-03-24-00_36_00_UTC 2024-03-24T00:36:00.042488+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-24T00:48:04.053 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:03 smithi106 ceph-mon[36505]: 2 slow requests, 1 included below; oldest blocked for > 758.118675 secs 2024-03-24T00:48:04.054 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:03 smithi106 ceph-mon[36505]: slow request 722.851603 seconds old, received at 2024-03-24T00:36:00.042315+0000: client_request(client.25385:166 mksnap #0x1//scheduled-2024-03-24-00_36_00_UTC 2024-03-24T00:36:00.042488+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-24T00:48:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:03 smithi078 ceph-mon[32429]: 2 slow requests, 1 included below; oldest blocked for > 758.118675 secs 2024-03-24T00:48:04.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:03 smithi078 ceph-mon[32429]: slow request 722.851603 seconds old, received at 2024-03-24T00:36:00.042315+0000: client_request(client.25385:166 mksnap #0x1//scheduled-2024-03-24-00_36_00_UTC 2024-03-24T00:36:00.042488+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-24T00:48:04.789 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1413 sec 2024-03-24T00:48:05.066 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:04 smithi106 ceph-mon[36505]: pgmap v896: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:05.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:04 smithi078 ceph-mon[32429]: pgmap v896: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:05.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:04 smithi107 ceph-mon[36485]: pgmap v896: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:05.789 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1414 sec 2024-03-24T00:48:06.789 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1415 sec 2024-03-24T00:48:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:06 smithi078 ceph-mon[32429]: pgmap v897: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:06 smithi107 ceph-mon[36485]: pgmap v897: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:07.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:06 smithi106 ceph-mon[36505]: pgmap v897: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:07.789 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1416 sec 2024-03-24T00:48:08.789 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1417 sec 2024-03-24T00:48:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:08 smithi078 ceph-mon[32429]: pgmap v898: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 763.118761 secs 2024-03-24T00:48:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:08 smithi107 ceph-mon[36485]: pgmap v898: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 763.118761 secs 2024-03-24T00:48:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:08 smithi106 ceph-mon[36505]: pgmap v898: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 763.118761 secs 2024-03-24T00:48:09.790 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1418 sec 2024-03-24T00:48:10.790 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1419 sec 2024-03-24T00:48:11.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:10 smithi078 ceph-mon[32429]: pgmap v899: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:11.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:10 smithi107 ceph-mon[36485]: pgmap v899: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:11.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:10 smithi106 ceph-mon[36505]: pgmap v899: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:11.790 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1420 sec 2024-03-24T00:48:12.790 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1421 sec 2024-03-24T00:48:13.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:12 smithi078 ceph-mon[32429]: pgmap v900: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:13.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:12 smithi107 ceph-mon[36485]: pgmap v900: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:13.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:12 smithi106 ceph-mon[36505]: pgmap v900: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:13.790 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1422 sec 2024-03-24T00:48:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 768.118848 secs 2024-03-24T00:48:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 768.118848 secs 2024-03-24T00:48:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 768.118848 secs 2024-03-24T00:48:14.790 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1423 sec 2024-03-24T00:48:15.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:14 smithi078 ceph-mon[32429]: pgmap v901: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:15.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:14 smithi107 ceph-mon[36485]: pgmap v901: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:15.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:14 smithi106 ceph-mon[36505]: pgmap v901: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:15.790 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1424 sec 2024-03-24T00:48:16.790 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1425 sec 2024-03-24T00:48:17.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:16 smithi078 ceph-mon[32429]: pgmap v902: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:17.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:16 smithi107 ceph-mon[36485]: pgmap v902: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:17.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:16 smithi106 ceph-mon[36505]: pgmap v902: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:17.790 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1426 sec 2024-03-24T00:48:18.790 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1427 sec 2024-03-24T00:48:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:18 smithi078 ceph-mon[32429]: pgmap v903: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:19.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 773.118937 secs 2024-03-24T00:48:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:18 smithi107 ceph-mon[36485]: pgmap v903: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 773.118937 secs 2024-03-24T00:48:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:18 smithi106 ceph-mon[36505]: pgmap v903: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 773.118937 secs 2024-03-24T00:48:19.790 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1428 sec 2024-03-24T00:48:20.790 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1429 sec 2024-03-24T00:48:21.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:20 smithi078 ceph-mon[32429]: pgmap v904: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:21.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:20 smithi107 ceph-mon[36485]: pgmap v904: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:21.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:20 smithi106 ceph-mon[36505]: pgmap v904: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:21.790 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1430 sec 2024-03-24T00:48:22.790 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1431 sec 2024-03-24T00:48:23.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:22 smithi078 ceph-mon[32429]: pgmap v905: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:23.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:22 smithi107 ceph-mon[36485]: pgmap v905: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:23.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:22 smithi106 ceph-mon[36505]: pgmap v905: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:23.791 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1432 sec 2024-03-24T00:48:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 778.118972 secs 2024-03-24T00:48:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 778.118972 secs 2024-03-24T00:48:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 778.118972 secs 2024-03-24T00:48:24.791 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1433 sec 2024-03-24T00:48:25.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:24 smithi078 ceph-mon[32429]: pgmap v906: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:24 smithi107 ceph-mon[36485]: pgmap v906: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:25.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:24 smithi106 ceph-mon[36505]: pgmap v906: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:25.791 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1434 sec 2024-03-24T00:48:26.791 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1435 sec 2024-03-24T00:48:27.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:26 smithi078 ceph-mon[32429]: pgmap v907: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:27.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:26 smithi107 ceph-mon[36485]: pgmap v907: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:27.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:26 smithi106 ceph-mon[36505]: pgmap v907: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:27.791 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1436 sec 2024-03-24T00:48:28.791 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1437 sec 2024-03-24T00:48:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:28 smithi078 ceph-mon[32429]: pgmap v908: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:29.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 783.119055 secs 2024-03-24T00:48:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:28 smithi107 ceph-mon[36485]: pgmap v908: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 783.119055 secs 2024-03-24T00:48:29.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:28 smithi106 ceph-mon[36505]: pgmap v908: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:29.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 783.119055 secs 2024-03-24T00:48:29.791 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1438 sec 2024-03-24T00:48:30.791 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1439 sec 2024-03-24T00:48:31.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:30 smithi078 ceph-mon[32429]: pgmap v909: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:31.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:30 smithi107 ceph-mon[36485]: pgmap v909: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:31.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:30 smithi106 ceph-mon[36505]: pgmap v909: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:31.792 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1440 sec 2024-03-24T00:48:32.792 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1441 sec 2024-03-24T00:48:33.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:32 smithi078 ceph-mon[32429]: pgmap v910: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:33.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:32 smithi107 ceph-mon[36485]: pgmap v910: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:33.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:32 smithi106 ceph-mon[36505]: pgmap v910: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:33.792 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1442 sec 2024-03-24T00:48:33.995 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 788.119109 secs 2024-03-24T00:48:33.995 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:33 smithi107 ceph-mon[36485]: pgmap v911: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:34.053 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 788.119109 secs 2024-03-24T00:48:34.054 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:33 smithi106 ceph-mon[36505]: pgmap v911: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 788.119109 secs 2024-03-24T00:48:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:33 smithi078 ceph-mon[32429]: pgmap v911: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:34.792 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1443 sec 2024-03-24T00:48:35.792 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1444 sec 2024-03-24T00:48:36.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:36 smithi078 ceph-mon[32429]: pgmap v912: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:36.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:36 smithi107 ceph-mon[36485]: pgmap v912: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:36.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:36 smithi106 ceph-mon[36505]: pgmap v912: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:36.792 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1445 sec 2024-03-24T00:48:37.792 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1446 sec 2024-03-24T00:48:38.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:38 smithi078 ceph-mon[32429]: pgmap v913: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:38.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 793.119197 secs 2024-03-24T00:48:38.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:38 smithi107 ceph-mon[36485]: pgmap v913: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:38.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 793.119197 secs 2024-03-24T00:48:38.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:38 smithi106 ceph-mon[36505]: pgmap v913: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:38.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 793.119197 secs 2024-03-24T00:48:38.792 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1447 sec 2024-03-24T00:48:39.792 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1448 sec 2024-03-24T00:48:40.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:40 smithi078 ceph-mon[32429]: pgmap v914: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:40 smithi107 ceph-mon[36485]: pgmap v914: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:40.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:40 smithi106 ceph-mon[36505]: pgmap v914: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:40.792 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1449 sec 2024-03-24T00:48:41.792 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1450 sec 2024-03-24T00:48:42.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:42 smithi078 ceph-mon[32429]: pgmap v915: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:42.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:42 smithi107 ceph-mon[36485]: pgmap v915: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:42.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:42 smithi106 ceph-mon[36505]: pgmap v915: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:42.793 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1451 sec 2024-03-24T00:48:43.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 798.119292 secs 2024-03-24T00:48:43.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 798.119292 secs 2024-03-24T00:48:43.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 798.119292 secs 2024-03-24T00:48:43.793 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1452 sec 2024-03-24T00:48:44.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:44 smithi078 ceph-mon[32429]: pgmap v916: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:44 smithi107 ceph-mon[36485]: pgmap v916: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:44 smithi106 ceph-mon[36505]: pgmap v916: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:44.793 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1453 sec 2024-03-24T00:48:45.793 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1454 sec 2024-03-24T00:48:46.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:46 smithi078 ceph-mon[32429]: pgmap v917: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:46.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:46 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:48:46.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:46 smithi107 ceph-mon[36485]: pgmap v917: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:46.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:46 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:48:46.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:46 smithi106 ceph-mon[36505]: pgmap v917: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:46.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:46 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:48:46.793 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1455 sec 2024-03-24T00:48:47.793 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1456 sec 2024-03-24T00:48:48.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:48 smithi078 ceph-mon[32429]: pgmap v918: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:48.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 803.119352 secs 2024-03-24T00:48:48.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:48 smithi107 ceph-mon[36485]: pgmap v918: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:48.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 803.119352 secs 2024-03-24T00:48:48.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:48 smithi106 ceph-mon[36505]: pgmap v918: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:48.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 803.119352 secs 2024-03-24T00:48:48.793 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1457 sec 2024-03-24T00:48:49.793 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1458 sec 2024-03-24T00:48:50.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:50 smithi078 ceph-mon[32429]: pgmap v919: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:50.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:50 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:48:50.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:50 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:48:50.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:50 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:48:50.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:50 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:48:50.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:50 smithi078 ceph-mon[32429]: pgmap v920: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:48:50.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:50 smithi078 ceph-mon[32429]: pgmap v921: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:48:50.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:50 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:48:50.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:50 smithi107 ceph-mon[36485]: pgmap v919: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:50.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:50 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:48:50.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:50 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:48:50.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:50 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:48:50.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:50 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:48:50.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:50 smithi107 ceph-mon[36485]: pgmap v920: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:48:50.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:50 smithi107 ceph-mon[36485]: pgmap v921: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:48:50.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:50 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:48:50.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:50 smithi106 ceph-mon[36505]: pgmap v919: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:48:50.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:50 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:48:50.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:50 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:48:50.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:50 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:48:50.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:50 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:48:50.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:50 smithi106 ceph-mon[36505]: pgmap v920: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:48:50.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:50 smithi106 ceph-mon[36505]: pgmap v921: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:48:50.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:50 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:48:50.793 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1459 sec 2024-03-24T00:48:51.794 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1460 sec 2024-03-24T00:48:52.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:52 smithi106 ceph-mon[36505]: pgmap v922: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:48:52.794 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1461 sec 2024-03-24T00:48:52.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:52 smithi078 ceph-mon[32429]: pgmap v922: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:48:52.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:52 smithi107 ceph-mon[36485]: pgmap v922: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:48:53.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 808.119364 secs 2024-03-24T00:48:53.794 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1462 sec 2024-03-24T00:48:53.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 808.119364 secs 2024-03-24T00:48:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 808.119364 secs 2024-03-24T00:48:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:54 smithi106 ceph-mon[36505]: pgmap v923: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:48:54.794 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1463 sec 2024-03-24T00:48:54.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:54 smithi078 ceph-mon[32429]: pgmap v923: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:48:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:54 smithi107 ceph-mon[36485]: pgmap v923: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:48:55.794 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1464 sec 2024-03-24T00:48:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:56 smithi106 ceph-mon[36505]: pgmap v924: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:48:56.794 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1465 sec 2024-03-24T00:48:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:56 smithi078 ceph-mon[32429]: pgmap v924: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:48:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:56 smithi107 ceph-mon[36485]: pgmap v924: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T00:48:57.794 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1466 sec 2024-03-24T00:48:58.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:58 smithi106 ceph-mon[36505]: pgmap v925: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:48:58.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:48:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 813.119425 secs 2024-03-24T00:48:58.794 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1467 sec 2024-03-24T00:48:58.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:58 smithi078 ceph-mon[32429]: pgmap v925: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:48:58.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:48:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 813.119425 secs 2024-03-24T00:48:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:58 smithi107 ceph-mon[36485]: pgmap v925: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:48:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:48:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 813.119425 secs 2024-03-24T00:48:59.794 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1468 sec 2024-03-24T00:49:00.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:00 smithi106 ceph-mon[36505]: pgmap v926: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:49:00.794 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1469 sec 2024-03-24T00:49:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:00 smithi078 ceph-mon[32429]: pgmap v926: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:49:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:00 smithi107 ceph-mon[36485]: pgmap v926: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:49:01.794 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1470 sec 2024-03-24T00:49:02.795 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1471 sec 2024-03-24T00:49:02.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:02 smithi078 ceph-mon[32429]: pgmap v927: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:02 smithi107 ceph-mon[36485]: pgmap v927: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:02.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:02 smithi106 ceph-mon[36505]: pgmap v927: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:03.745 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 818.119485 secs 2024-03-24T00:49:03.794 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1472 sec 2024-03-24T00:49:03.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 818.119485 secs 2024-03-24T00:49:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 818.119485 secs 2024-03-24T00:49:04.764 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:04 smithi107 ceph-mon[36485]: pgmap v928: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:04.794 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1473 sec 2024-03-24T00:49:04.803 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:04 smithi106 ceph-mon[36505]: pgmap v928: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:04 smithi078 ceph-mon[32429]: pgmap v928: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:05.795 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1474 sec 2024-03-24T00:49:06.795 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1475 sec 2024-03-24T00:49:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:06 smithi078 ceph-mon[32429]: pgmap v929: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:06 smithi107 ceph-mon[36485]: pgmap v929: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:06 smithi106 ceph-mon[36505]: pgmap v929: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:07.795 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1476 sec 2024-03-24T00:49:08.795 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1477 sec 2024-03-24T00:49:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:08 smithi078 ceph-mon[32429]: pgmap v930: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 823.119544 secs 2024-03-24T00:49:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:08 smithi107 ceph-mon[36485]: pgmap v930: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 823.119544 secs 2024-03-24T00:49:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:08 smithi106 ceph-mon[36505]: pgmap v930: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:08.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 823.119544 secs 2024-03-24T00:49:09.795 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1478 sec 2024-03-24T00:49:10.795 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1479 sec 2024-03-24T00:49:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:10 smithi078 ceph-mon[32429]: pgmap v931: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:10 smithi107 ceph-mon[36485]: pgmap v931: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:10 smithi106 ceph-mon[36505]: pgmap v931: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:11.795 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1480 sec 2024-03-24T00:49:12.795 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1481 sec 2024-03-24T00:49:12.828 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:12 smithi078 ceph-mon[32429]: pgmap v932: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:12 smithi107 ceph-mon[36485]: pgmap v932: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:12.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:12 smithi106 ceph-mon[36505]: pgmap v932: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:13.795 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1482 sec 2024-03-24T00:49:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 828.119602 secs 2024-03-24T00:49:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 828.119602 secs 2024-03-24T00:49:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 828.119602 secs 2024-03-24T00:49:14.795 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1483 sec 2024-03-24T00:49:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:14 smithi078 ceph-mon[32429]: pgmap v933: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:14 smithi107 ceph-mon[36485]: pgmap v933: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:14 smithi106 ceph-mon[36505]: pgmap v933: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:15.795 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1484 sec 2024-03-24T00:49:16.796 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1485 sec 2024-03-24T00:49:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:16 smithi078 ceph-mon[32429]: pgmap v934: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:16 smithi107 ceph-mon[36485]: pgmap v934: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:16.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:16 smithi106 ceph-mon[36505]: pgmap v934: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:17.796 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1486 sec 2024-03-24T00:49:18.796 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1487 sec 2024-03-24T00:49:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:18 smithi078 ceph-mon[32429]: pgmap v935: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:18.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 833.119660 secs 2024-03-24T00:49:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:18 smithi107 ceph-mon[36485]: pgmap v935: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 833.119660 secs 2024-03-24T00:49:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:18 smithi106 ceph-mon[36505]: pgmap v935: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 833.119660 secs 2024-03-24T00:49:19.796 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1488 sec 2024-03-24T00:49:20.796 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1489 sec 2024-03-24T00:49:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:20 smithi078 ceph-mon[32429]: pgmap v936: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:20 smithi107 ceph-mon[36485]: pgmap v936: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:20.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:20 smithi106 ceph-mon[36505]: pgmap v936: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:21.796 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1490 sec 2024-03-24T00:49:22.796 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1491 sec 2024-03-24T00:49:22.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:22 smithi078 ceph-mon[32429]: pgmap v937: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:22.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:22 smithi107 ceph-mon[36485]: pgmap v937: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:22.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:22 smithi106 ceph-mon[36505]: pgmap v937: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:23.796 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1492 sec 2024-03-24T00:49:23.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 838.119717 secs 2024-03-24T00:49:23.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 838.119717 secs 2024-03-24T00:49:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 838.119717 secs 2024-03-24T00:49:24.796 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1493 sec 2024-03-24T00:49:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:24 smithi078 ceph-mon[32429]: pgmap v938: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:24 smithi107 ceph-mon[36485]: pgmap v938: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:24 smithi106 ceph-mon[36505]: pgmap v938: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:25.796 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1494 sec 2024-03-24T00:49:26.796 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1495 sec 2024-03-24T00:49:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:26 smithi078 ceph-mon[32429]: pgmap v939: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:26 smithi107 ceph-mon[36485]: pgmap v939: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:26.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:26 smithi106 ceph-mon[36505]: pgmap v939: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:27.797 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1496 sec 2024-03-24T00:49:28.797 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1497 sec 2024-03-24T00:49:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:28 smithi078 ceph-mon[32429]: pgmap v940: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:28.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 843.119776 secs 2024-03-24T00:49:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:28 smithi107 ceph-mon[36485]: pgmap v940: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 843.119776 secs 2024-03-24T00:49:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:28 smithi106 ceph-mon[36505]: pgmap v940: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:28.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 843.119776 secs 2024-03-24T00:49:29.797 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1498 sec 2024-03-24T00:49:30.797 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1499 sec 2024-03-24T00:49:30.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:30 smithi078 ceph-mon[32429]: pgmap v941: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:30 smithi107 ceph-mon[36485]: pgmap v941: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:30 smithi106 ceph-mon[36505]: pgmap v941: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:31.797 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1500 sec 2024-03-24T00:49:32.797 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1501 sec 2024-03-24T00:49:32.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:32 smithi078 ceph-mon[32429]: pgmap v942: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:32 smithi107 ceph-mon[36485]: pgmap v942: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:32 smithi106 ceph-mon[36505]: pgmap v942: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:33.745 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 848.119884 secs 2024-03-24T00:49:33.797 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1502 sec 2024-03-24T00:49:33.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 848.119884 secs 2024-03-24T00:49:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 848.119884 secs 2024-03-24T00:49:34.797 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1503 sec 2024-03-24T00:49:34.803 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:34 smithi106 ceph-mon[36505]: pgmap v943: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:34 smithi078 ceph-mon[32429]: pgmap v943: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:34 smithi107 ceph-mon[36485]: pgmap v943: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:35.797 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1504 sec 2024-03-24T00:49:36.797 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1505 sec 2024-03-24T00:49:36.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:36 smithi078 ceph-mon[32429]: pgmap v944: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:36 smithi107 ceph-mon[36485]: pgmap v944: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:36.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:36 smithi106 ceph-mon[36505]: pgmap v944: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:37.797 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1506 sec 2024-03-24T00:49:38.798 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1507 sec 2024-03-24T00:49:38.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:38 smithi078 ceph-mon[32429]: pgmap v945: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:38.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 853.119970 secs 2024-03-24T00:49:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:38 smithi107 ceph-mon[36485]: pgmap v945: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 853.119970 secs 2024-03-24T00:49:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:38 smithi106 ceph-mon[36505]: pgmap v945: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 853.119970 secs 2024-03-24T00:49:39.798 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1508 sec 2024-03-24T00:49:40.798 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1509 sec 2024-03-24T00:49:40.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:40 smithi078 ceph-mon[32429]: pgmap v946: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:40.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:40 smithi107 ceph-mon[36485]: pgmap v946: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:40 smithi106 ceph-mon[36505]: pgmap v946: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:41.798 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1510 sec 2024-03-24T00:49:42.798 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1511 sec 2024-03-24T00:49:42.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:42 smithi078 ceph-mon[32429]: pgmap v947: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:42.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:42 smithi107 ceph-mon[36485]: pgmap v947: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:42.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:42 smithi106 ceph-mon[36505]: pgmap v947: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:43.798 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1512 sec 2024-03-24T00:49:43.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 858.120028 secs 2024-03-24T00:49:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 858.120028 secs 2024-03-24T00:49:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 858.120028 secs 2024-03-24T00:49:44.798 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1513 sec 2024-03-24T00:49:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:44 smithi078 ceph-mon[32429]: pgmap v948: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:44 smithi107 ceph-mon[36485]: pgmap v948: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:44.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:44 smithi106 ceph-mon[36505]: pgmap v948: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:45.798 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1514 sec 2024-03-24T00:49:46.798 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1515 sec 2024-03-24T00:49:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:46 smithi078 ceph-mon[32429]: pgmap v949: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:46.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:46 smithi107 ceph-mon[36485]: pgmap v949: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:46 smithi106 ceph-mon[36505]: pgmap v949: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:47.798 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1516 sec 2024-03-24T00:49:48.798 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1517 sec 2024-03-24T00:49:48.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:48 smithi078 ceph-mon[32429]: pgmap v950: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:48.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 863.120116 secs 2024-03-24T00:49:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:48 smithi107 ceph-mon[36485]: pgmap v950: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 863.120116 secs 2024-03-24T00:49:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:48 smithi106 ceph-mon[36505]: pgmap v950: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:48.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 863.120116 secs 2024-03-24T00:49:49.798 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1518 sec 2024-03-24T00:49:50.799 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1519 sec 2024-03-24T00:49:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:50 smithi078 ceph-mon[32429]: pgmap v951: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:50.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:50 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:49:50.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:50 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:49:50.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:50 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:49:50.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:50 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:49:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:50 smithi107 ceph-mon[36485]: pgmap v951: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:50 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:49:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:50 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:49:50.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:50 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:49:50.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:50 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:49:50.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:50 smithi106 ceph-mon[36505]: pgmap v951: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:49:50.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:50 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:49:50.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:50 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:49:50.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:50 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:49:50.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:50 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:49:51.799 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1520 sec 2024-03-24T00:49:51.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:51 smithi078 ceph-mon[32429]: pgmap v952: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T00:49:51.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:51 smithi078 ceph-mon[32429]: pgmap v953: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:49:51.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:51 smithi107 ceph-mon[36485]: pgmap v952: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T00:49:51.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:51 smithi107 ceph-mon[36485]: pgmap v953: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:49:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:51 smithi106 ceph-mon[36505]: pgmap v952: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T00:49:51.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:51 smithi106 ceph-mon[36505]: pgmap v953: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:49:52.799 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1521 sec 2024-03-24T00:49:53.799 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1522 sec 2024-03-24T00:49:53.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:53 smithi078 ceph-mon[32429]: pgmap v954: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:49:53.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 868.120174 secs 2024-03-24T00:49:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:53 smithi107 ceph-mon[36485]: pgmap v954: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:49:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 868.120174 secs 2024-03-24T00:49:53.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:53 smithi106 ceph-mon[36505]: pgmap v954: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:49:53.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 868.120174 secs 2024-03-24T00:49:54.799 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1523 sec 2024-03-24T00:49:55.799 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1524 sec 2024-03-24T00:49:55.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:55 smithi078 ceph-mon[32429]: pgmap v955: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:49:55.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:55 smithi107 ceph-mon[36485]: pgmap v955: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:49:55.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:55 smithi106 ceph-mon[36505]: pgmap v955: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:49:56.803 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1525 sec 2024-03-24T00:49:57.799 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1526 sec 2024-03-24T00:49:57.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:57 smithi078 ceph-mon[32429]: pgmap v956: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:49:57.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:57 smithi107 ceph-mon[36485]: pgmap v956: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:49:57.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:57 smithi106 ceph-mon[36505]: pgmap v956: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:49:58.799 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1527 sec 2024-03-24T00:49:58.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 873.120232 secs 2024-03-24T00:49:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 873.120232 secs 2024-03-24T00:49:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 873.120232 secs 2024-03-24T00:49:59.799 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1528 sec 2024-03-24T00:49:59.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:49:59 smithi078 ceph-mon[32429]: pgmap v957: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:49:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:49:59 smithi107 ceph-mon[36485]: pgmap v957: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:49:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:49:59 smithi106 ceph-mon[36505]: pgmap v957: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:50:00.799 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1529 sec 2024-03-24T00:50:00.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:00 smithi078 ceph-mon[32429]: Health detail: HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T00:50:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:00 smithi078 ceph-mon[32429]: [WRN] CEPHADM_FAILED_DAEMON: 8 failed cephadm daemon(s) 2024-03-24T00:50:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:00 smithi078 ceph-mon[32429]: daemon mds.j on smithi078 is in error state 2024-03-24T00:50:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:00 smithi078 ceph-mon[32429]: daemon mds.d on smithi078 is in error state 2024-03-24T00:50:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:00 smithi078 ceph-mon[32429]: daemon mds.g on smithi078 is in error state 2024-03-24T00:50:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:00 smithi078 ceph-mon[32429]: daemon mds.a on smithi078 is in error state 2024-03-24T00:50:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:00 smithi078 ceph-mon[32429]: daemon mds.k on smithi106 is in error state 2024-03-24T00:50:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:00 smithi078 ceph-mon[32429]: daemon mds.l on smithi107 is in error state 2024-03-24T00:50:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:00 smithi078 ceph-mon[32429]: daemon mds.f on smithi107 is in error state 2024-03-24T00:50:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:00 smithi078 ceph-mon[32429]: daemon mds.i on smithi107 is in error state 2024-03-24T00:50:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:00 smithi078 ceph-mon[32429]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-03-24T00:50:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:00 smithi078 ceph-mon[32429]: mds.b(mds.0): 2 slow requests are blocked > 180 secs 2024-03-24T00:50:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:00 smithi107 ceph-mon[36485]: Health detail: HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T00:50:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:00 smithi107 ceph-mon[36485]: [WRN] CEPHADM_FAILED_DAEMON: 8 failed cephadm daemon(s) 2024-03-24T00:50:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:00 smithi107 ceph-mon[36485]: daemon mds.j on smithi078 is in error state 2024-03-24T00:50:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:00 smithi107 ceph-mon[36485]: daemon mds.d on smithi078 is in error state 2024-03-24T00:50:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:00 smithi107 ceph-mon[36485]: daemon mds.g on smithi078 is in error state 2024-03-24T00:50:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:00 smithi107 ceph-mon[36485]: daemon mds.a on smithi078 is in error state 2024-03-24T00:50:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:00 smithi107 ceph-mon[36485]: daemon mds.k on smithi106 is in error state 2024-03-24T00:50:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:00 smithi107 ceph-mon[36485]: daemon mds.l on smithi107 is in error state 2024-03-24T00:50:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:00 smithi107 ceph-mon[36485]: daemon mds.f on smithi107 is in error state 2024-03-24T00:50:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:00 smithi107 ceph-mon[36485]: daemon mds.i on smithi107 is in error state 2024-03-24T00:50:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:00 smithi107 ceph-mon[36485]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-03-24T00:50:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:00 smithi107 ceph-mon[36485]: mds.b(mds.0): 2 slow requests are blocked > 180 secs 2024-03-24T00:50:00.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:00 smithi106 ceph-mon[36505]: Health detail: HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T00:50:00.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:00 smithi106 ceph-mon[36505]: [WRN] CEPHADM_FAILED_DAEMON: 8 failed cephadm daemon(s) 2024-03-24T00:50:00.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:00 smithi106 ceph-mon[36505]: daemon mds.j on smithi078 is in error state 2024-03-24T00:50:00.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:00 smithi106 ceph-mon[36505]: daemon mds.d on smithi078 is in error state 2024-03-24T00:50:00.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:00 smithi106 ceph-mon[36505]: daemon mds.g on smithi078 is in error state 2024-03-24T00:50:00.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:00 smithi106 ceph-mon[36505]: daemon mds.a on smithi078 is in error state 2024-03-24T00:50:00.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:00 smithi106 ceph-mon[36505]: daemon mds.k on smithi106 is in error state 2024-03-24T00:50:00.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:00 smithi106 ceph-mon[36505]: daemon mds.l on smithi107 is in error state 2024-03-24T00:50:00.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:00 smithi106 ceph-mon[36505]: daemon mds.f on smithi107 is in error state 2024-03-24T00:50:00.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:00 smithi106 ceph-mon[36505]: daemon mds.i on smithi107 is in error state 2024-03-24T00:50:00.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:00 smithi106 ceph-mon[36505]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-03-24T00:50:00.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:00 smithi106 ceph-mon[36505]: mds.b(mds.0): 2 slow requests are blocked > 180 secs 2024-03-24T00:50:01.799 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1530 sec 2024-03-24T00:50:01.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:01 smithi078 ceph-mon[32429]: pgmap v958: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:50:01.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:01 smithi107 ceph-mon[36485]: pgmap v958: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:50:01.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:01 smithi106 ceph-mon[36505]: pgmap v958: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:50:02.800 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1531 sec 2024-03-24T00:50:03.802 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1532 sec 2024-03-24T00:50:03.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:03 smithi078 ceph-mon[32429]: pgmap v959: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:03.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 878.120269 secs 2024-03-24T00:50:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:03 smithi107 ceph-mon[36485]: pgmap v959: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 878.120269 secs 2024-03-24T00:50:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:03 smithi106 ceph-mon[36505]: pgmap v959: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 878.120269 secs 2024-03-24T00:50:04.800 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1533 sec 2024-03-24T00:50:05.800 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1534 sec 2024-03-24T00:50:05.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:05 smithi078 ceph-mon[32429]: pgmap v960: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:05.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:05 smithi107 ceph-mon[36485]: pgmap v960: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:05.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:05 smithi106 ceph-mon[36505]: pgmap v960: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:06.800 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1535 sec 2024-03-24T00:50:07.800 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1536 sec 2024-03-24T00:50:07.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:07 smithi078 ceph-mon[32429]: pgmap v961: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:07.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:07 smithi107 ceph-mon[36485]: pgmap v961: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:07.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:07 smithi106 ceph-mon[36505]: pgmap v961: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:08.800 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1537 sec 2024-03-24T00:50:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 883.120318 secs 2024-03-24T00:50:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 883.120318 secs 2024-03-24T00:50:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 883.120318 secs 2024-03-24T00:50:09.800 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1538 sec 2024-03-24T00:50:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:09 smithi078 ceph-mon[32429]: pgmap v962: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:09.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:09 smithi107 ceph-mon[36485]: pgmap v962: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:09.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:09 smithi106 ceph-mon[36505]: pgmap v962: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:10.800 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1539 sec 2024-03-24T00:50:11.800 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1540 sec 2024-03-24T00:50:11.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:11 smithi078 ceph-mon[32429]: pgmap v963: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:11 smithi107 ceph-mon[36485]: pgmap v963: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:11.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:11 smithi106 ceph-mon[36505]: pgmap v963: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:12.800 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1541 sec 2024-03-24T00:50:13.800 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1542 sec 2024-03-24T00:50:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:13 smithi078 ceph-mon[32429]: pgmap v964: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:13.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 888.120379 secs 2024-03-24T00:50:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:13 smithi107 ceph-mon[36485]: pgmap v964: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 888.120379 secs 2024-03-24T00:50:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:13 smithi106 ceph-mon[36505]: pgmap v964: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 888.120379 secs 2024-03-24T00:50:14.801 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1543 sec 2024-03-24T00:50:15.801 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1544 sec 2024-03-24T00:50:15.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:15 smithi078 ceph-mon[32429]: pgmap v965: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:15 smithi107 ceph-mon[36485]: pgmap v965: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:15 smithi106 ceph-mon[36505]: pgmap v965: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:16.801 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1545 sec 2024-03-24T00:50:17.801 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1546 sec 2024-03-24T00:50:17.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:17 smithi078 ceph-mon[32429]: pgmap v966: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:17.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:17 smithi107 ceph-mon[36485]: pgmap v966: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:17.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:17 smithi106 ceph-mon[36505]: pgmap v966: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:18.801 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1547 sec 2024-03-24T00:50:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 893.120441 secs 2024-03-24T00:50:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 893.120441 secs 2024-03-24T00:50:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 893.120441 secs 2024-03-24T00:50:19.801 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1548 sec 2024-03-24T00:50:19.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:19 smithi078 ceph-mon[32429]: pgmap v967: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:19 smithi107 ceph-mon[36485]: pgmap v967: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:19.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:19 smithi106 ceph-mon[36505]: pgmap v967: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:20.801 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1549 sec 2024-03-24T00:50:21.801 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1550 sec 2024-03-24T00:50:21.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:21 smithi078 ceph-mon[32429]: pgmap v968: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:21 smithi107 ceph-mon[36485]: pgmap v968: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:21.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:21 smithi106 ceph-mon[36505]: pgmap v968: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:22.801 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1551 sec 2024-03-24T00:50:23.801 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1552 sec 2024-03-24T00:50:23.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:23 smithi078 ceph-mon[32429]: pgmap v969: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:23.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 898.120501 secs 2024-03-24T00:50:23.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:23 smithi107 ceph-mon[36485]: pgmap v969: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:23.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 898.120501 secs 2024-03-24T00:50:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:23 smithi106 ceph-mon[36505]: pgmap v969: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 898.120501 secs 2024-03-24T00:50:24.802 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1553 sec 2024-03-24T00:50:25.801 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1554 sec 2024-03-24T00:50:25.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:25 smithi078 ceph-mon[32429]: pgmap v970: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:25.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:25 smithi107 ceph-mon[36485]: pgmap v970: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:25.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:25 smithi106 ceph-mon[36505]: pgmap v970: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:26.802 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1555 sec 2024-03-24T00:50:27.802 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1556 sec 2024-03-24T00:50:27.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:27 smithi078 ceph-mon[32429]: pgmap v971: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:27.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:27 smithi107 ceph-mon[36485]: pgmap v971: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:27.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:27 smithi106 ceph-mon[36505]: pgmap v971: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:28.802 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1557 sec 2024-03-24T00:50:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 903.120558 secs 2024-03-24T00:50:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 903.120558 secs 2024-03-24T00:50:28.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 903.120558 secs 2024-03-24T00:50:29.802 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1558 sec 2024-03-24T00:50:29.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:29 smithi078 ceph-mon[32429]: pgmap v972: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:29 smithi107 ceph-mon[36485]: pgmap v972: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:29 smithi106 ceph-mon[36505]: pgmap v972: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:30.802 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1559 sec 2024-03-24T00:50:31.802 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1560 sec 2024-03-24T00:50:31.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:31 smithi078 ceph-mon[32429]: pgmap v973: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:31.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:31 smithi107 ceph-mon[36485]: pgmap v973: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:31.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:31 smithi106 ceph-mon[36505]: pgmap v973: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:32.802 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1561 sec 2024-03-24T00:50:33.802 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1562 sec 2024-03-24T00:50:33.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:33 smithi078 ceph-mon[32429]: pgmap v974: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:33.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 908.120616 secs 2024-03-24T00:50:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:33 smithi107 ceph-mon[36485]: pgmap v974: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 908.120616 secs 2024-03-24T00:50:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:33 smithi106 ceph-mon[36505]: pgmap v974: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 908.120616 secs 2024-03-24T00:50:34.802 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1563 sec 2024-03-24T00:50:35.802 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1564 sec 2024-03-24T00:50:35.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:35 smithi078 ceph-mon[32429]: pgmap v975: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:35.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:35 smithi107 ceph-mon[36485]: pgmap v975: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:35.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:35 smithi106 ceph-mon[36505]: pgmap v975: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:36.802 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1565 sec 2024-03-24T00:50:37.803 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1566 sec 2024-03-24T00:50:37.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:37 smithi107 ceph-mon[36485]: pgmap v976: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:37.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:37 smithi106 ceph-mon[36505]: pgmap v976: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:37 smithi078 ceph-mon[32429]: pgmap v976: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:38.803 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1567 sec 2024-03-24T00:50:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 913.120627 secs 2024-03-24T00:50:38.992 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 913.120627 secs 2024-03-24T00:50:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 913.120627 secs 2024-03-24T00:50:39.803 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1568 sec 2024-03-24T00:50:39.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:39 smithi106 ceph-mon[36505]: pgmap v977: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:39 smithi078 ceph-mon[32429]: pgmap v977: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:39 smithi107 ceph-mon[36485]: pgmap v977: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:40.803 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1569 sec 2024-03-24T00:50:41.803 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1570 sec 2024-03-24T00:50:41.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:41 smithi106 ceph-mon[36505]: pgmap v978: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:42.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:41 smithi078 ceph-mon[32429]: pgmap v978: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:42.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:41 smithi107 ceph-mon[36485]: pgmap v978: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:50:42.803 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1571 sec 2024-03-24T00:50:43.803 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1572 sec 2024-03-24T00:50:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:43 smithi106 ceph-mon[36505]: pgmap v979: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:50:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 918.120685 secs 2024-03-24T00:50:44.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:43 smithi078 ceph-mon[32429]: pgmap v979: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:50:44.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 918.120685 secs 2024-03-24T00:50:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:43 smithi107 ceph-mon[36485]: pgmap v979: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:50:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 918.120685 secs 2024-03-24T00:50:44.803 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1573 sec 2024-03-24T00:50:45.803 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1574 sec 2024-03-24T00:50:45.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:45 smithi106 ceph-mon[36505]: pgmap v980: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:50:46.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:45 smithi078 ceph-mon[32429]: pgmap v980: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:50:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:45 smithi107 ceph-mon[36485]: pgmap v980: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:50:46.803 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1575 sec 2024-03-24T00:50:47.804 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1576 sec 2024-03-24T00:50:47.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:47 smithi106 ceph-mon[36505]: pgmap v981: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:50:48.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:47 smithi078 ceph-mon[32429]: pgmap v981: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:50:48.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:47 smithi107 ceph-mon[36485]: pgmap v981: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:50:48.804 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1577 sec 2024-03-24T00:50:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 923.120743 secs 2024-03-24T00:50:49.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 923.120743 secs 2024-03-24T00:50:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 923.120743 secs 2024-03-24T00:50:49.804 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1578 sec 2024-03-24T00:50:49.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:49 smithi106 ceph-mon[36505]: pgmap v982: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:50:50.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:49 smithi078 ceph-mon[32429]: pgmap v982: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:50:50.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:49 smithi107 ceph-mon[36485]: pgmap v982: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:50:50.804 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1579 sec 2024-03-24T00:50:50.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:50 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:50:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:50 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:50:51.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:50 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:50:51.804 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1580 sec 2024-03-24T00:50:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:51 smithi106 ceph-mon[36505]: pgmap v983: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:50:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:51 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:50:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:51 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:50:51.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:51 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:50:52.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:51 smithi078 ceph-mon[32429]: pgmap v983: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:50:52.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:51 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:50:52.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:51 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:50:52.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:51 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:50:52.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:51 smithi107 ceph-mon[36485]: pgmap v983: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:50:52.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:51 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:50:52.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:51 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:50:52.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:51 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:50:52.804 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1581 sec 2024-03-24T00:50:52.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:52 smithi106 ceph-mon[36505]: pgmap v984: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T00:50:52.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:52 smithi106 ceph-mon[36505]: pgmap v985: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:50:53.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:52 smithi078 ceph-mon[32429]: pgmap v984: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T00:50:53.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:52 smithi078 ceph-mon[32429]: pgmap v985: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:50:53.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:52 smithi107 ceph-mon[36485]: pgmap v984: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T00:50:53.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:52 smithi107 ceph-mon[36485]: pgmap v985: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:50:53.804 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1582 sec 2024-03-24T00:50:53.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 928.120801 secs 2024-03-24T00:50:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 928.120801 secs 2024-03-24T00:50:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 928.120801 secs 2024-03-24T00:50:54.804 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1583 sec 2024-03-24T00:50:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:54 smithi106 ceph-mon[36505]: pgmap v986: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:50:55.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:54 smithi078 ceph-mon[32429]: pgmap v986: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:50:55.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:54 smithi107 ceph-mon[36485]: pgmap v986: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:50:55.804 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1584 sec 2024-03-24T00:50:56.804 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1585 sec 2024-03-24T00:50:56.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:56 smithi106 ceph-mon[36505]: pgmap v987: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:50:57.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:56 smithi078 ceph-mon[32429]: pgmap v987: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:50:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:56 smithi107 ceph-mon[36485]: pgmap v987: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:50:57.804 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1586 sec 2024-03-24T00:50:58.805 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1587 sec 2024-03-24T00:50:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:58 smithi106 ceph-mon[36505]: pgmap v988: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:50:58.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:50:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 933.120859 secs 2024-03-24T00:50:59.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:58 smithi078 ceph-mon[32429]: pgmap v988: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:50:59.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:50:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 933.120859 secs 2024-03-24T00:50:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:58 smithi107 ceph-mon[36485]: pgmap v988: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:50:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:50:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 933.120859 secs 2024-03-24T00:50:59.805 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1588 sec 2024-03-24T00:51:00.805 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1589 sec 2024-03-24T00:51:00.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:00 smithi106 ceph-mon[36505]: pgmap v989: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:51:01.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:00 smithi078 ceph-mon[32429]: pgmap v989: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:51:01.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:00 smithi107 ceph-mon[36485]: pgmap v989: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:51:01.805 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1590 sec 2024-03-24T00:51:02.805 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1591 sec 2024-03-24T00:51:03.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:02 smithi078 ceph-mon[32429]: pgmap v990: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:51:03.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:02 smithi107 ceph-mon[36485]: pgmap v990: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:51:03.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:02 smithi106 ceph-mon[36505]: pgmap v990: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:51:03.805 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1592 sec 2024-03-24T00:51:03.995 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 938.120916 secs 2024-03-24T00:51:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 938.120916 secs 2024-03-24T00:51:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 938.120916 secs 2024-03-24T00:51:04.805 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1593 sec 2024-03-24T00:51:05.053 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:04 smithi106 ceph-mon[36505]: pgmap v991: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:05.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:04 smithi078 ceph-mon[32429]: pgmap v991: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:05.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:04 smithi107 ceph-mon[36485]: pgmap v991: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:05.805 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1594 sec 2024-03-24T00:51:06.805 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1595 sec 2024-03-24T00:51:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:06 smithi078 ceph-mon[32429]: pgmap v992: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:06 smithi107 ceph-mon[36485]: pgmap v992: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:07.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:06 smithi106 ceph-mon[36505]: pgmap v992: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:07.805 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1596 sec 2024-03-24T00:51:08.806 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1597 sec 2024-03-24T00:51:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:08 smithi078 ceph-mon[32429]: pgmap v993: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 943.120975 secs 2024-03-24T00:51:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:08 smithi107 ceph-mon[36485]: pgmap v993: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 943.120975 secs 2024-03-24T00:51:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:08 smithi106 ceph-mon[36505]: pgmap v993: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 943.120975 secs 2024-03-24T00:51:09.806 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1598 sec 2024-03-24T00:51:10.806 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1599 sec 2024-03-24T00:51:11.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:10 smithi078 ceph-mon[32429]: pgmap v994: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:11.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:10 smithi107 ceph-mon[36485]: pgmap v994: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:11.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:10 smithi106 ceph-mon[36505]: pgmap v994: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:11.806 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1600 sec 2024-03-24T00:51:12.806 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1601 sec 2024-03-24T00:51:13.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:12 smithi078 ceph-mon[32429]: pgmap v995: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:51:13.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:12 smithi107 ceph-mon[36485]: pgmap v995: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:51:13.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:12 smithi106 ceph-mon[36505]: pgmap v995: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:51:13.806 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1602 sec 2024-03-24T00:51:14.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 948.121033 secs 2024-03-24T00:51:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 948.121033 secs 2024-03-24T00:51:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 948.121033 secs 2024-03-24T00:51:14.806 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1603 sec 2024-03-24T00:51:15.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:14 smithi078 ceph-mon[32429]: pgmap v996: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:51:15.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:14 smithi107 ceph-mon[36485]: pgmap v996: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:51:15.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:14 smithi106 ceph-mon[36505]: pgmap v996: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:51:15.806 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1604 sec 2024-03-24T00:51:16.806 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1605 sec 2024-03-24T00:51:17.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:16 smithi078 ceph-mon[32429]: pgmap v997: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:51:17.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:16 smithi107 ceph-mon[36485]: pgmap v997: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:51:17.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:16 smithi106 ceph-mon[36505]: pgmap v997: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:51:17.806 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1606 sec 2024-03-24T00:51:18.807 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1607 sec 2024-03-24T00:51:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:18 smithi078 ceph-mon[32429]: pgmap v998: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:51:19.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 953.121105 secs 2024-03-24T00:51:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:18 smithi107 ceph-mon[36485]: pgmap v998: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:51:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 953.121105 secs 2024-03-24T00:51:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:18 smithi106 ceph-mon[36505]: pgmap v998: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:51:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 953.121105 secs 2024-03-24T00:51:19.807 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1608 sec 2024-03-24T00:51:20.807 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1609 sec 2024-03-24T00:51:21.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:20 smithi078 ceph-mon[32429]: pgmap v999: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:51:21.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:20 smithi107 ceph-mon[36485]: pgmap v999: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:51:21.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:20 smithi106 ceph-mon[36505]: pgmap v999: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T00:51:21.807 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1610 sec 2024-03-24T00:51:22.807 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1611 sec 2024-03-24T00:51:23.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:22 smithi078 ceph-mon[32429]: pgmap v1000: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:51:23.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:22 smithi107 ceph-mon[36485]: pgmap v1000: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:51:23.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:22 smithi106 ceph-mon[36505]: pgmap v1000: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:51:23.807 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1612 sec 2024-03-24T00:51:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 958.121144 secs 2024-03-24T00:51:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 958.121144 secs 2024-03-24T00:51:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 958.121144 secs 2024-03-24T00:51:24.807 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1613 sec 2024-03-24T00:51:25.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:24 smithi078 ceph-mon[32429]: pgmap v1001: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:24 smithi107 ceph-mon[36485]: pgmap v1001: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:25.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:24 smithi106 ceph-mon[36505]: pgmap v1001: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:25.807 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1614 sec 2024-03-24T00:51:26.807 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1615 sec 2024-03-24T00:51:27.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:26 smithi078 ceph-mon[32429]: pgmap v1002: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:27.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:26 smithi107 ceph-mon[36485]: pgmap v1002: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:27.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:26 smithi106 ceph-mon[36505]: pgmap v1002: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:27.807 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1616 sec 2024-03-24T00:51:28.808 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1617 sec 2024-03-24T00:51:29.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:28 smithi078 ceph-mon[32429]: pgmap v1003: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:29.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 963.121206 secs 2024-03-24T00:51:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:28 smithi107 ceph-mon[36485]: pgmap v1003: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 963.121206 secs 2024-03-24T00:51:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:28 smithi106 ceph-mon[36505]: pgmap v1003: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:29.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 963.121206 secs 2024-03-24T00:51:29.808 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1618 sec 2024-03-24T00:51:30.808 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1619 sec 2024-03-24T00:51:31.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:30 smithi078 ceph-mon[32429]: pgmap v1004: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:31.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:30 smithi107 ceph-mon[36485]: pgmap v1004: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:31.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:30 smithi106 ceph-mon[36505]: pgmap v1004: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:31.808 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1620 sec 2024-03-24T00:51:32.808 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1621 sec 2024-03-24T00:51:33.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:32 smithi078 ceph-mon[32429]: pgmap v1005: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:33.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:32 smithi107 ceph-mon[36485]: pgmap v1005: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:33.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:32 smithi106 ceph-mon[36505]: pgmap v1005: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:33.808 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1622 sec 2024-03-24T00:51:33.992 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 968.121247 secs 2024-03-24T00:51:33.994 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 968.121247 secs 2024-03-24T00:51:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 968.121247 secs 2024-03-24T00:51:34.808 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1623 sec 2024-03-24T00:51:35.053 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:34 smithi106 ceph-mon[36505]: pgmap v1006: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:35.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:34 smithi078 ceph-mon[32429]: pgmap v1006: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:35.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:34 smithi107 ceph-mon[36485]: pgmap v1006: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:35.808 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1624 sec 2024-03-24T00:51:36.808 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1625 sec 2024-03-24T00:51:37.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:36 smithi078 ceph-mon[32429]: pgmap v1007: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:37.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:36 smithi107 ceph-mon[36485]: pgmap v1007: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:37.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:36 smithi106 ceph-mon[36505]: pgmap v1007: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:37.808 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1626 sec 2024-03-24T00:51:38.808 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1627 sec 2024-03-24T00:51:39.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:38 smithi078 ceph-mon[32429]: pgmap v1008: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:39.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 973.121292 secs 2024-03-24T00:51:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:38 smithi107 ceph-mon[36485]: pgmap v1008: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 973.121292 secs 2024-03-24T00:51:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:38 smithi106 ceph-mon[36505]: pgmap v1008: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 973.121292 secs 2024-03-24T00:51:39.809 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1628 sec 2024-03-24T00:51:40.809 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1629 sec 2024-03-24T00:51:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:40 smithi078 ceph-mon[32429]: pgmap v1009: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:41.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:40 smithi107 ceph-mon[36485]: pgmap v1009: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:41.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:40 smithi106 ceph-mon[36505]: pgmap v1009: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:41.809 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1630 sec 2024-03-24T00:51:42.809 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1631 sec 2024-03-24T00:51:43.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:42 smithi078 ceph-mon[32429]: pgmap v1010: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:43.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:42 smithi107 ceph-mon[36485]: pgmap v1010: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:43.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:42 smithi106 ceph-mon[36505]: pgmap v1010: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:43.809 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1632 sec 2024-03-24T00:51:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 978.121326 secs 2024-03-24T00:51:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 978.121326 secs 2024-03-24T00:51:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 978.121326 secs 2024-03-24T00:51:44.809 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1633 sec 2024-03-24T00:51:45.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:44 smithi078 ceph-mon[32429]: pgmap v1011: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:45.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:44 smithi107 ceph-mon[36485]: pgmap v1011: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:45.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:44 smithi106 ceph-mon[36505]: pgmap v1011: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:45.809 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1634 sec 2024-03-24T00:51:46.809 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1635 sec 2024-03-24T00:51:47.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:46 smithi078 ceph-mon[32429]: pgmap v1012: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:47.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:46 smithi107 ceph-mon[36485]: pgmap v1012: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:47.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:46 smithi106 ceph-mon[36505]: pgmap v1012: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:47.809 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1636 sec 2024-03-24T00:51:48.809 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1637 sec 2024-03-24T00:51:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:48 smithi078 ceph-mon[32429]: pgmap v1013: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:49.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 983.121388 secs 2024-03-24T00:51:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:48 smithi107 ceph-mon[36485]: pgmap v1013: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 983.121388 secs 2024-03-24T00:51:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:48 smithi106 ceph-mon[36505]: pgmap v1013: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 983.121388 secs 2024-03-24T00:51:49.810 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1638 sec 2024-03-24T00:51:50.810 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1639 sec 2024-03-24T00:51:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:50 smithi078 ceph-mon[32429]: pgmap v1014: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:51.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:50 smithi107 ceph-mon[36485]: pgmap v1014: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:51.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:50 smithi106 ceph-mon[36505]: pgmap v1014: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:51.809 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1640 sec 2024-03-24T00:51:52.042 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:51 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:51:52.064 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:51 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:51:52.070 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:51 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:51:52.810 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1641 sec 2024-03-24T00:51:53.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:52 smithi078 ceph-mon[32429]: pgmap v1015: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:53.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:52 smithi107 ceph-mon[36485]: pgmap v1015: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:53.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:52 smithi106 ceph-mon[36505]: pgmap v1015: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:53.810 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1642 sec 2024-03-24T00:51:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 988.121452 secs 2024-03-24T00:51:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:54 smithi078 ceph-mon[32429]: pgmap v1016: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:54 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:54 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:54 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:54 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:54 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:54 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 988.121452 secs 2024-03-24T00:51:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:54 smithi107 ceph-mon[36485]: pgmap v1016: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:54 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:54 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:54.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:54 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:54.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:54 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:54.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:54 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:54.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:54 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 988.121452 secs 2024-03-24T00:51:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:54 smithi106 ceph-mon[36505]: pgmap v1016: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:54.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:54 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:54.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:54 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:54.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:54 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:54.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:54 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:54.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:54 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:54.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:54 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:54.810 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1643 sec 2024-03-24T00:51:55.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:55.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:55.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:55 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T00:51:55.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:55 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T00:51:55.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:55.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:55.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:55 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T00:51:55.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:55 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T00:51:55.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:55.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:55.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:55 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T00:51:55.405 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:55 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T00:51:55.405 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:51:55.405 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:51:55.405 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:55 smithi106 ceph-mon[36505]: pgmap v1017: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:55.405 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:55 smithi106 ceph-mon[36505]: pgmap v1018: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:51:55.405 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:55.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:55.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:55.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:55 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T00:51:55.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:55 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T00:51:55.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:55.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:55.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:55 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T00:51:55.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:55 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T00:51:55.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:55.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:55.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:55 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T00:51:55.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:55 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T00:51:55.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:51:55.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:51:55.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:55 smithi078 ceph-mon[32429]: pgmap v1017: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:55.594 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:55 smithi078 ceph-mon[32429]: pgmap v1018: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:51:55.594 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:55.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:55.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:55.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:55 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T00:51:55.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:55 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T00:51:55.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:55.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:55.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:55 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T00:51:55.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:55 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T00:51:55.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:55.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:55.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.596 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.596 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.596 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:51:55.596 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:55 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T00:51:55.596 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:55 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T00:51:55.596 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:51:55.596 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:51:55.596 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:55 smithi107 ceph-mon[36485]: pgmap v1017: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:51:55.596 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:55 smithi107 ceph-mon[36485]: pgmap v1018: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:51:55.596 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:51:55.810 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1644 sec 2024-03-24T00:51:56.810 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1645 sec 2024-03-24T00:51:57.810 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1646 sec 2024-03-24T00:51:58.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:57 smithi078 ceph-mon[32429]: pgmap v1019: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:51:58.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:57 smithi107 ceph-mon[36485]: pgmap v1019: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:51:58.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:57 smithi106 ceph-mon[36505]: pgmap v1019: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:51:58.810 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1647 sec 2024-03-24T00:51:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 993.121514 secs 2024-03-24T00:51:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 993.121514 secs 2024-03-24T00:51:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 993.121514 secs 2024-03-24T00:51:59.810 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1648 sec 2024-03-24T00:52:00.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:51:59 smithi078 ceph-mon[32429]: pgmap v1020: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:52:00.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:51:59 smithi107 ceph-mon[36485]: pgmap v1020: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:52:00.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:51:59 smithi106 ceph-mon[36505]: pgmap v1020: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:52:00.810 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1649 sec 2024-03-24T00:52:01.810 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1650 sec 2024-03-24T00:52:02.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:01 smithi078 ceph-mon[32429]: pgmap v1021: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:52:02.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:01 smithi107 ceph-mon[36485]: pgmap v1021: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:52:02.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:01 smithi106 ceph-mon[36505]: pgmap v1021: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:52:02.810 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1651 sec 2024-03-24T00:52:03.811 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1652 sec 2024-03-24T00:52:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:03 smithi078 ceph-mon[32429]: pgmap v1022: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:52:04.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 998.121597 secs 2024-03-24T00:52:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:03 smithi107 ceph-mon[36485]: pgmap v1022: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:52:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 998.121597 secs 2024-03-24T00:52:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:03 smithi106 ceph-mon[36505]: pgmap v1022: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T00:52:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 998.121597 secs 2024-03-24T00:52:04.811 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1653 sec 2024-03-24T00:52:05.811 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1654 sec 2024-03-24T00:52:06.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:05 smithi078 ceph-mon[32429]: pgmap v1023: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:52:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:05 smithi107 ceph-mon[36485]: pgmap v1023: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:52:06.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:05 smithi106 ceph-mon[36505]: pgmap v1023: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:52:06.811 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1655 sec 2024-03-24T00:52:07.811 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1656 sec 2024-03-24T00:52:08.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:07 smithi078 ceph-mon[32429]: pgmap v1024: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:08.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:07 smithi107 ceph-mon[36485]: pgmap v1024: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:08.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:07 smithi106 ceph-mon[36505]: pgmap v1024: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:08.811 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1657 sec 2024-03-24T00:52:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1003.121662 secs 2024-03-24T00:52:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1003.121662 secs 2024-03-24T00:52:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1003.121662 secs 2024-03-24T00:52:09.811 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1658 sec 2024-03-24T00:52:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:09 smithi078 ceph-mon[32429]: pgmap v1025: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:10.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:09 smithi107 ceph-mon[36485]: pgmap v1025: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:10.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:09 smithi106 ceph-mon[36505]: pgmap v1025: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:10.811 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1659 sec 2024-03-24T00:52:11.811 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1660 sec 2024-03-24T00:52:12.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:11 smithi078 ceph-mon[32429]: pgmap v1026: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:12.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:11 smithi107 ceph-mon[36485]: pgmap v1026: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:12.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:11 smithi106 ceph-mon[36505]: pgmap v1026: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:12.811 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1661 sec 2024-03-24T00:52:13.811 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1662 sec 2024-03-24T00:52:14.094 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:13 smithi078 ceph-mon[32429]: pgmap v1027: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:14.094 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1008.121676 secs 2024-03-24T00:52:14.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:13 smithi107 ceph-mon[36485]: pgmap v1027: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:14.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1008.121676 secs 2024-03-24T00:52:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:13 smithi106 ceph-mon[36505]: pgmap v1027: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:14.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1008.121676 secs 2024-03-24T00:52:14.811 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1663 sec 2024-03-24T00:52:15.812 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1664 sec 2024-03-24T00:52:16.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:15 smithi078 ceph-mon[32429]: pgmap v1028: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:16.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:15 smithi107 ceph-mon[36485]: pgmap v1028: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:16.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:15 smithi106 ceph-mon[36505]: pgmap v1028: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:16.812 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1665 sec 2024-03-24T00:52:17.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:16 smithi078 ceph-mon[32429]: pgmap v1029: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:17.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:16 smithi107 ceph-mon[36485]: pgmap v1029: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:17.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:16 smithi106 ceph-mon[36505]: pgmap v1029: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:17.812 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1666 sec 2024-03-24T00:52:18.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1013.121739 secs 2024-03-24T00:52:18.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1013.121739 secs 2024-03-24T00:52:18.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1013.121739 secs 2024-03-24T00:52:18.812 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1667 sec 2024-03-24T00:52:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:19 smithi106 ceph-mon[36505]: pgmap v1030: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:19.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:19 smithi078 ceph-mon[32429]: pgmap v1030: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:19 smithi107 ceph-mon[36485]: pgmap v1030: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:19.812 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1668 sec 2024-03-24T00:52:20.812 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1669 sec 2024-03-24T00:52:21.813 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1670 sec 2024-03-24T00:52:22.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:21 smithi078 ceph-mon[32429]: pgmap v1031: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:22.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:21 smithi107 ceph-mon[36485]: pgmap v1031: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:22.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:21 smithi106 ceph-mon[36505]: pgmap v1031: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:22.812 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1671 sec 2024-03-24T00:52:23.812 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1672 sec 2024-03-24T00:52:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:23 smithi078 ceph-mon[32429]: pgmap v1032: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:24.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1018.121800 secs 2024-03-24T00:52:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:23 smithi107 ceph-mon[36485]: pgmap v1032: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1018.121800 secs 2024-03-24T00:52:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:23 smithi106 ceph-mon[36505]: pgmap v1032: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1018.121800 secs 2024-03-24T00:52:24.812 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1673 sec 2024-03-24T00:52:25.813 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1674 sec 2024-03-24T00:52:26.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:25 smithi078 ceph-mon[32429]: pgmap v1033: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:26.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:25 smithi107 ceph-mon[36485]: pgmap v1033: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:26.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:25 smithi106 ceph-mon[36505]: pgmap v1033: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:26.813 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1675 sec 2024-03-24T00:52:27.813 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1676 sec 2024-03-24T00:52:28.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:27 smithi078 ceph-mon[32429]: pgmap v1034: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:27 smithi107 ceph-mon[36485]: pgmap v1034: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:28.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:27 smithi106 ceph-mon[36505]: pgmap v1034: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:28.813 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1677 sec 2024-03-24T00:52:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1023.121890 secs 2024-03-24T00:52:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1023.121890 secs 2024-03-24T00:52:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1023.121890 secs 2024-03-24T00:52:29.813 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1678 sec 2024-03-24T00:52:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:29 smithi078 ceph-mon[32429]: pgmap v1035: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:30.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:29 smithi107 ceph-mon[36485]: pgmap v1035: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:30.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:29 smithi106 ceph-mon[36505]: pgmap v1035: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:30.813 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1679 sec 2024-03-24T00:52:31.813 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1680 sec 2024-03-24T00:52:32.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:31 smithi078 ceph-mon[32429]: pgmap v1036: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:32.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:31 smithi107 ceph-mon[36485]: pgmap v1036: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:32.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:31 smithi106 ceph-mon[36505]: pgmap v1036: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:32.813 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1681 sec 2024-03-24T00:52:33.813 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1682 sec 2024-03-24T00:52:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:33 smithi078 ceph-mon[32429]: pgmap v1037: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:34.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1028.121978 secs 2024-03-24T00:52:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:33 smithi107 ceph-mon[36485]: pgmap v1037: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1028.121978 secs 2024-03-24T00:52:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:33 smithi106 ceph-mon[36505]: pgmap v1037: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1028.121978 secs 2024-03-24T00:52:34.813 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1683 sec 2024-03-24T00:52:35.813 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1684 sec 2024-03-24T00:52:35.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:35 smithi078 ceph-mon[32429]: pgmap v1038: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:36.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:35 smithi107 ceph-mon[36485]: pgmap v1038: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:36.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:35 smithi106 ceph-mon[36505]: pgmap v1038: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:36.813 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1685 sec 2024-03-24T00:52:37.814 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1686 sec 2024-03-24T00:52:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:37 smithi078 ceph-mon[32429]: pgmap v1039: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:38.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:37 smithi107 ceph-mon[36485]: pgmap v1039: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:38.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:37 smithi106 ceph-mon[36505]: pgmap v1039: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:38.814 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1687 sec 2024-03-24T00:52:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1033.122068 secs 2024-03-24T00:52:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1033.122068 secs 2024-03-24T00:52:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1033.122068 secs 2024-03-24T00:52:39.814 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1688 sec 2024-03-24T00:52:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:39 smithi078 ceph-mon[32429]: pgmap v1040: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:39 smithi107 ceph-mon[36485]: pgmap v1040: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:39 smithi106 ceph-mon[36505]: pgmap v1040: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:40.814 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1689 sec 2024-03-24T00:52:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:40 smithi078 ceph-mon[32429]: pgmap v1041: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:41.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:40 smithi107 ceph-mon[36485]: pgmap v1041: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:41.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:40 smithi106 ceph-mon[36505]: pgmap v1041: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:41.814 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1690 sec 2024-03-24T00:52:42.814 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1691 sec 2024-03-24T00:52:43.814 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1692 sec 2024-03-24T00:52:44.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:43 smithi078 ceph-mon[32429]: pgmap v1042: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:44.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1038.122159 secs 2024-03-24T00:52:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:43 smithi107 ceph-mon[36485]: pgmap v1042: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1038.122159 secs 2024-03-24T00:52:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:43 smithi106 ceph-mon[36505]: pgmap v1042: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1038.122159 secs 2024-03-24T00:52:44.814 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1693 sec 2024-03-24T00:52:45.814 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1694 sec 2024-03-24T00:52:46.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:45 smithi078 ceph-mon[32429]: pgmap v1043: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:45 smithi107 ceph-mon[36485]: pgmap v1043: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:46.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:45 smithi106 ceph-mon[36505]: pgmap v1043: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:46.815 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1695 sec 2024-03-24T00:52:47.815 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1696 sec 2024-03-24T00:52:48.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:47 smithi078 ceph-mon[32429]: pgmap v1044: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:48.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:47 smithi107 ceph-mon[36485]: pgmap v1044: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:48.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:47 smithi106 ceph-mon[36505]: pgmap v1044: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:48.815 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1697 sec 2024-03-24T00:52:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1043.122249 secs 2024-03-24T00:52:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1043.122249 secs 2024-03-24T00:52:49.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1043.122249 secs 2024-03-24T00:52:49.815 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1698 sec 2024-03-24T00:52:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:49 smithi078 ceph-mon[32429]: pgmap v1045: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:50.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:49 smithi107 ceph-mon[36485]: pgmap v1045: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:49 smithi106 ceph-mon[36505]: pgmap v1045: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:50.815 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1699 sec 2024-03-24T00:52:51.815 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1700 sec 2024-03-24T00:52:52.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:51 smithi078 ceph-mon[32429]: pgmap v1046: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:52.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:51 smithi107 ceph-mon[36485]: pgmap v1046: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:52.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:51 smithi106 ceph-mon[36505]: pgmap v1046: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:52.815 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1701 sec 2024-03-24T00:52:53.815 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1702 sec 2024-03-24T00:52:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:53 smithi078 ceph-mon[32429]: pgmap v1047: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:54.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1048.122344 secs 2024-03-24T00:52:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:53 smithi107 ceph-mon[36485]: pgmap v1047: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1048.122344 secs 2024-03-24T00:52:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:53 smithi106 ceph-mon[36505]: pgmap v1047: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1048.122344 secs 2024-03-24T00:52:54.815 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1703 sec 2024-03-24T00:52:55.815 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1704 sec 2024-03-24T00:52:56.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:55 smithi078 ceph-mon[32429]: pgmap v1048: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:56.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:52:56.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:52:56.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:52:56.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:52:56.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:52:56.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:52:56.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:52:56.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:55 smithi107 ceph-mon[36485]: pgmap v1048: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:56.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:52:56.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:52:56.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:52:56.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:52:56.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:52:56.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:52:56.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:52:56.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:55 smithi106 ceph-mon[36505]: pgmap v1048: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:52:56.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:52:56.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:52:56.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:52:56.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:52:56.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:52:56.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:52:56.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:52:56.815 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1705 sec 2024-03-24T00:52:57.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:52:57.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:52:57.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:52:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:52:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:52:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:52:57.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:52:57.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:52:57.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:52:57.815 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1706 sec 2024-03-24T00:52:58.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:57 smithi078 ceph-mon[32429]: pgmap v1049: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T00:52:58.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:57 smithi078 ceph-mon[32429]: pgmap v1050: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T00:52:58.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:57 smithi107 ceph-mon[36485]: pgmap v1049: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T00:52:58.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:57 smithi107 ceph-mon[36485]: pgmap v1050: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T00:52:58.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:57 smithi106 ceph-mon[36505]: pgmap v1049: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T00:52:58.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:57 smithi106 ceph-mon[36505]: pgmap v1050: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T00:52:58.816 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1707 sec 2024-03-24T00:52:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1053.122403 secs 2024-03-24T00:52:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1053.122403 secs 2024-03-24T00:52:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1053.122403 secs 2024-03-24T00:52:59.816 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1708 sec 2024-03-24T00:53:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:52:59 smithi078 ceph-mon[32429]: pgmap v1051: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T00:53:00.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:52:59 smithi107 ceph-mon[36485]: pgmap v1051: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T00:53:00.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:52:59 smithi106 ceph-mon[36505]: pgmap v1051: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T00:53:00.816 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1709 sec 2024-03-24T00:53:01.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:00 smithi078 ceph-mon[32429]: pgmap v1052: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:53:01.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:00 smithi107 ceph-mon[36485]: pgmap v1052: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:53:01.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:00 smithi106 ceph-mon[36505]: pgmap v1052: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:53:01.816 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1710 sec 2024-03-24T00:53:02.816 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1711 sec 2024-03-24T00:53:03.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:02 smithi078 ceph-mon[32429]: pgmap v1053: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T00:53:03.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:02 smithi107 ceph-mon[36485]: pgmap v1053: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T00:53:03.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:02 smithi106 ceph-mon[36505]: pgmap v1053: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T00:53:03.816 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1712 sec 2024-03-24T00:53:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1058.122458 secs 2024-03-24T00:53:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1058.122458 secs 2024-03-24T00:53:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1058.122458 secs 2024-03-24T00:53:04.816 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1713 sec 2024-03-24T00:53:05.303 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:05 smithi106 ceph-mon[36505]: pgmap v1054: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T00:53:05.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:05 smithi078 ceph-mon[32429]: pgmap v1054: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T00:53:05.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:05 smithi107 ceph-mon[36485]: pgmap v1054: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T00:53:05.816 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1714 sec 2024-03-24T00:53:06.816 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1715 sec 2024-03-24T00:53:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:07 smithi078 ceph-mon[32429]: pgmap v1055: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:53:07.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:07 smithi107 ceph-mon[36485]: pgmap v1055: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:53:07.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:07 smithi106 ceph-mon[36505]: pgmap v1055: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:53:07.816 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1716 sec 2024-03-24T00:53:08.816 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1717 sec 2024-03-24T00:53:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1063.122546 secs 2024-03-24T00:53:09.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:09 smithi078 ceph-mon[32429]: pgmap v1056: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1063.122546 secs 2024-03-24T00:53:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:09 smithi107 ceph-mon[36485]: pgmap v1056: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1063.122546 secs 2024-03-24T00:53:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:09 smithi106 ceph-mon[36505]: pgmap v1056: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:09.816 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1718 sec 2024-03-24T00:53:10.816 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1719 sec 2024-03-24T00:53:11.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:11 smithi078 ceph-mon[32429]: pgmap v1057: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:11.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:11 smithi107 ceph-mon[36485]: pgmap v1057: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:11.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:11 smithi106 ceph-mon[36505]: pgmap v1057: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:11.816 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1720 sec 2024-03-24T00:53:12.817 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1721 sec 2024-03-24T00:53:13.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:13 smithi078 ceph-mon[32429]: pgmap v1058: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:13.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:13 smithi107 ceph-mon[36485]: pgmap v1058: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:13.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:13 smithi106 ceph-mon[36505]: pgmap v1058: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:13.817 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1722 sec 2024-03-24T00:53:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1068.122644 secs 2024-03-24T00:53:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1068.122644 secs 2024-03-24T00:53:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1068.122644 secs 2024-03-24T00:53:14.817 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1723 sec 2024-03-24T00:53:15.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:15 smithi078 ceph-mon[32429]: pgmap v1059: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:15.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:15 smithi107 ceph-mon[36485]: pgmap v1059: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:15.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:15 smithi106 ceph-mon[36505]: pgmap v1059: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:15.817 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1724 sec 2024-03-24T00:53:16.817 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1725 sec 2024-03-24T00:53:17.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:17 smithi078 ceph-mon[32429]: pgmap v1060: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:17.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:17 smithi107 ceph-mon[36485]: pgmap v1060: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:17.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:17 smithi106 ceph-mon[36505]: pgmap v1060: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:17.817 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1726 sec 2024-03-24T00:53:18.817 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1727 sec 2024-03-24T00:53:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1073.122728 secs 2024-03-24T00:53:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:19 smithi078 ceph-mon[32429]: pgmap v1061: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1073.122728 secs 2024-03-24T00:53:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:19 smithi107 ceph-mon[36485]: pgmap v1061: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1073.122728 secs 2024-03-24T00:53:19.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:19 smithi106 ceph-mon[36505]: pgmap v1061: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:19.817 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1728 sec 2024-03-24T00:53:20.817 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1729 sec 2024-03-24T00:53:21.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:21 smithi078 ceph-mon[32429]: pgmap v1062: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:21.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:21 smithi107 ceph-mon[36485]: pgmap v1062: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:21.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:21 smithi106 ceph-mon[36505]: pgmap v1062: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:21.817 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1730 sec 2024-03-24T00:53:22.817 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1731 sec 2024-03-24T00:53:23.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:23 smithi078 ceph-mon[32429]: pgmap v1063: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:23.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:23 smithi107 ceph-mon[36485]: pgmap v1063: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:23.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:23 smithi106 ceph-mon[36505]: pgmap v1063: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:23.818 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1732 sec 2024-03-24T00:53:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1078.122813 secs 2024-03-24T00:53:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1078.122813 secs 2024-03-24T00:53:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1078.122813 secs 2024-03-24T00:53:24.818 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1733 sec 2024-03-24T00:53:25.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:25 smithi078 ceph-mon[32429]: pgmap v1064: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:25.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:25 smithi107 ceph-mon[36485]: pgmap v1064: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:25.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:25 smithi106 ceph-mon[36505]: pgmap v1064: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:25.818 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1734 sec 2024-03-24T00:53:26.818 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1735 sec 2024-03-24T00:53:27.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:27 smithi078 ceph-mon[32429]: pgmap v1065: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:27.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:27 smithi107 ceph-mon[36485]: pgmap v1065: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:27.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:27 smithi106 ceph-mon[36505]: pgmap v1065: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:27.818 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1736 sec 2024-03-24T00:53:28.818 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1737 sec 2024-03-24T00:53:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1083.122874 secs 2024-03-24T00:53:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:29 smithi078 ceph-mon[32429]: pgmap v1066: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1083.122874 secs 2024-03-24T00:53:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:29 smithi107 ceph-mon[36485]: pgmap v1066: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1083.122874 secs 2024-03-24T00:53:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:29 smithi106 ceph-mon[36505]: pgmap v1066: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:29.818 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1738 sec 2024-03-24T00:53:30.818 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1739 sec 2024-03-24T00:53:31.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:31 smithi078 ceph-mon[32429]: pgmap v1067: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:31.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:31 smithi107 ceph-mon[36485]: pgmap v1067: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:31.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:31 smithi106 ceph-mon[36505]: pgmap v1067: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:31.818 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1740 sec 2024-03-24T00:53:32.818 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1741 sec 2024-03-24T00:53:33.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:33 smithi078 ceph-mon[32429]: pgmap v1068: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:33.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1088.122962 secs 2024-03-24T00:53:33.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:33 smithi107 ceph-mon[36485]: pgmap v1068: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:33.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1088.122962 secs 2024-03-24T00:53:33.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:33 smithi106 ceph-mon[36505]: pgmap v1068: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:33.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1088.122962 secs 2024-03-24T00:53:33.818 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1742 sec 2024-03-24T00:53:34.818 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1743 sec 2024-03-24T00:53:35.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:35 smithi078 ceph-mon[32429]: pgmap v1069: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:35.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:35 smithi107 ceph-mon[36485]: pgmap v1069: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:35.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:35 smithi106 ceph-mon[36505]: pgmap v1069: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:35.818 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1744 sec 2024-03-24T00:53:36.818 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1745 sec 2024-03-24T00:53:37.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:37 smithi078 ceph-mon[32429]: pgmap v1070: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:37 smithi107 ceph-mon[36485]: pgmap v1070: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:37 smithi106 ceph-mon[36505]: pgmap v1070: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:37.818 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1746 sec 2024-03-24T00:53:38.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1093.123004 secs 2024-03-24T00:53:38.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1093.123004 secs 2024-03-24T00:53:38.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1093.123004 secs 2024-03-24T00:53:38.819 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1747 sec 2024-03-24T00:53:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:39 smithi078 ceph-mon[32429]: pgmap v1071: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:39 smithi107 ceph-mon[36485]: pgmap v1071: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:39 smithi106 ceph-mon[36505]: pgmap v1071: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:39.819 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1748 sec 2024-03-24T00:53:40.819 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1749 sec 2024-03-24T00:53:41.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:41 smithi078 ceph-mon[32429]: pgmap v1072: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:41 smithi107 ceph-mon[36485]: pgmap v1072: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:41.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:41 smithi106 ceph-mon[36505]: pgmap v1072: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:41.819 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1750 sec 2024-03-24T00:53:42.819 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1751 sec 2024-03-24T00:53:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:43 smithi078 ceph-mon[32429]: pgmap v1073: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:43.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1098.123092 secs 2024-03-24T00:53:43.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:43 smithi107 ceph-mon[36485]: pgmap v1073: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:43.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1098.123092 secs 2024-03-24T00:53:43.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:43 smithi106 ceph-mon[36505]: pgmap v1073: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:43.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1098.123092 secs 2024-03-24T00:53:43.819 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1752 sec 2024-03-24T00:53:44.819 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1753 sec 2024-03-24T00:53:45.819 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1754 sec 2024-03-24T00:53:45.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:45 smithi078 ceph-mon[32429]: pgmap v1074: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:45.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:45 smithi107 ceph-mon[36485]: pgmap v1074: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:45.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:45 smithi106 ceph-mon[36505]: pgmap v1074: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:46.819 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1755 sec 2024-03-24T00:53:47.820 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1756 sec 2024-03-24T00:53:47.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:47 smithi078 ceph-mon[32429]: pgmap v1075: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:47.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:47 smithi107 ceph-mon[36485]: pgmap v1075: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:47.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:47 smithi106 ceph-mon[36505]: pgmap v1075: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:48.820 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1757 sec 2024-03-24T00:53:48.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1103.123186 secs 2024-03-24T00:53:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1103.123186 secs 2024-03-24T00:53:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1103.123186 secs 2024-03-24T00:53:49.820 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1758 sec 2024-03-24T00:53:49.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:49 smithi078 ceph-mon[32429]: pgmap v1076: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:49.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:49 smithi107 ceph-mon[36485]: pgmap v1076: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:49.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:49 smithi106 ceph-mon[36505]: pgmap v1076: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:50.820 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1759 sec 2024-03-24T00:53:51.820 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1760 sec 2024-03-24T00:53:51.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:51 smithi078 ceph-mon[32429]: pgmap v1077: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:51.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:51 smithi107 ceph-mon[36485]: pgmap v1077: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:51 smithi106 ceph-mon[36505]: pgmap v1077: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:52.820 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1761 sec 2024-03-24T00:53:53.820 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1762 sec 2024-03-24T00:53:53.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:53 smithi078 ceph-mon[32429]: pgmap v1078: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:53.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1108.123276 secs 2024-03-24T00:53:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:53 smithi107 ceph-mon[36485]: pgmap v1078: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1108.123276 secs 2024-03-24T00:53:53.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:53 smithi106 ceph-mon[36505]: pgmap v1078: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:53.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1108.123276 secs 2024-03-24T00:53:54.820 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1763 sec 2024-03-24T00:53:55.820 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1764 sec 2024-03-24T00:53:55.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:55 smithi078 ceph-mon[32429]: pgmap v1079: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:55.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:55 smithi107 ceph-mon[36485]: pgmap v1079: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:55.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:55 smithi106 ceph-mon[36505]: pgmap v1079: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:53:56.820 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1765 sec 2024-03-24T00:53:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:53:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:53:56.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:53:57.820 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1766 sec 2024-03-24T00:53:57.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:57 smithi078 ceph-mon[32429]: pgmap v1080: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:57.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:57 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:53:57.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:57 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:53:57.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:57 smithi078 ceph-mon[32429]: pgmap v1081: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:53:57.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:57 smithi078 ceph-mon[32429]: pgmap v1082: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:53:57.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:57 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:53:57.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:57 smithi107 ceph-mon[36485]: pgmap v1080: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:57.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:57 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:53:57.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:57 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:53:57.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:57 smithi107 ceph-mon[36485]: pgmap v1081: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:53:57.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:57 smithi107 ceph-mon[36485]: pgmap v1082: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:53:57.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:57 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:53:57.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:57 smithi106 ceph-mon[36505]: pgmap v1080: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:57.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:57 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:53:57.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:57 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:53:57.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:57 smithi106 ceph-mon[36505]: pgmap v1081: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:53:57.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:57 smithi106 ceph-mon[36505]: pgmap v1082: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:53:57.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:57 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:53:58.820 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1767 sec 2024-03-24T00:53:58.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1113.123313 secs 2024-03-24T00:53:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1113.123313 secs 2024-03-24T00:53:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1113.123313 secs 2024-03-24T00:53:59.821 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1768 sec 2024-03-24T00:53:59.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:53:59 smithi078 ceph-mon[32429]: pgmap v1083: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:53:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:53:59 smithi107 ceph-mon[36485]: pgmap v1083: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:53:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:53:59 smithi106 ceph-mon[36505]: pgmap v1083: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:54:00.821 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1769 sec 2024-03-24T00:54:01.821 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1770 sec 2024-03-24T00:54:02.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:01 smithi078 ceph-mon[32429]: pgmap v1084: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:54:02.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:01 smithi107 ceph-mon[36485]: pgmap v1084: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:54:02.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:01 smithi106 ceph-mon[36505]: pgmap v1084: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:54:02.821 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1771 sec 2024-03-24T00:54:03.821 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1772 sec 2024-03-24T00:54:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:03 smithi078 ceph-mon[32429]: pgmap v1085: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:54:04.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1118.123373 secs 2024-03-24T00:54:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:03 smithi107 ceph-mon[36485]: pgmap v1085: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:54:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1118.123373 secs 2024-03-24T00:54:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:03 smithi106 ceph-mon[36505]: pgmap v1085: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:54:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1118.123373 secs 2024-03-24T00:54:04.821 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1773 sec 2024-03-24T00:54:05.821 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1774 sec 2024-03-24T00:54:06.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:05 smithi078 ceph-mon[32429]: pgmap v1086: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:54:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:05 smithi107 ceph-mon[36485]: pgmap v1086: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:54:06.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:05 smithi106 ceph-mon[36505]: pgmap v1086: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:54:06.821 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1775 sec 2024-03-24T00:54:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:06 smithi078 ceph-mon[32429]: pgmap v1087: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:54:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:06 smithi107 ceph-mon[36485]: pgmap v1087: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:54:07.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:06 smithi106 ceph-mon[36505]: pgmap v1087: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:54:07.821 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1776 sec 2024-03-24T00:54:08.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1123.123435 secs 2024-03-24T00:54:08.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1123.123435 secs 2024-03-24T00:54:08.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1123.123435 secs 2024-03-24T00:54:08.821 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1777 sec 2024-03-24T00:54:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:09 smithi106 ceph-mon[36505]: pgmap v1088: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:09.492 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:09 smithi078 ceph-mon[32429]: pgmap v1088: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:09 smithi107 ceph-mon[36485]: pgmap v1088: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:09.821 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1778 sec 2024-03-24T00:54:10.821 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1779 sec 2024-03-24T00:54:11.822 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1780 sec 2024-03-24T00:54:12.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:11 smithi078 ceph-mon[32429]: pgmap v1089: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:12.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:11 smithi107 ceph-mon[36485]: pgmap v1089: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:12.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:11 smithi106 ceph-mon[36505]: pgmap v1089: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:12.822 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1781 sec 2024-03-24T00:54:13.822 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1782 sec 2024-03-24T00:54:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:13 smithi078 ceph-mon[32429]: pgmap v1090: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:14.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1128.123497 secs 2024-03-24T00:54:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:13 smithi107 ceph-mon[36485]: pgmap v1090: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1128.123497 secs 2024-03-24T00:54:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:13 smithi106 ceph-mon[36505]: pgmap v1090: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1128.123497 secs 2024-03-24T00:54:14.822 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1783 sec 2024-03-24T00:54:15.822 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1784 sec 2024-03-24T00:54:16.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:15 smithi078 ceph-mon[32429]: pgmap v1091: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:16.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:15 smithi107 ceph-mon[36485]: pgmap v1091: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:16.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:15 smithi106 ceph-mon[36505]: pgmap v1091: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:16.822 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1785 sec 2024-03-24T00:54:17.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:16 smithi078 ceph-mon[32429]: pgmap v1092: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:17.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:16 smithi107 ceph-mon[36485]: pgmap v1092: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:17.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:16 smithi106 ceph-mon[36505]: pgmap v1092: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:17.822 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1786 sec 2024-03-24T00:54:18.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1133.123558 secs 2024-03-24T00:54:18.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1133.123558 secs 2024-03-24T00:54:18.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1133.123558 secs 2024-03-24T00:54:18.822 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1787 sec 2024-03-24T00:54:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:19 smithi106 ceph-mon[36505]: pgmap v1093: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:19.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:19 smithi078 ceph-mon[32429]: pgmap v1093: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:19 smithi107 ceph-mon[36485]: pgmap v1093: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:19.822 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1788 sec 2024-03-24T00:54:20.822 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1789 sec 2024-03-24T00:54:21.823 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1790 sec 2024-03-24T00:54:22.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:21 smithi078 ceph-mon[32429]: pgmap v1094: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:22.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:21 smithi107 ceph-mon[36485]: pgmap v1094: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:22.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:21 smithi106 ceph-mon[36505]: pgmap v1094: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:22.823 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1791 sec 2024-03-24T00:54:23.823 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1792 sec 2024-03-24T00:54:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:23 smithi078 ceph-mon[32429]: pgmap v1095: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:24.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1138.123622 secs 2024-03-24T00:54:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:23 smithi107 ceph-mon[36485]: pgmap v1095: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1138.123622 secs 2024-03-24T00:54:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:23 smithi106 ceph-mon[36505]: pgmap v1095: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1138.123622 secs 2024-03-24T00:54:24.823 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1793 sec 2024-03-24T00:54:25.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:24 smithi078 ceph-mon[32429]: pgmap v1096: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:24 smithi107 ceph-mon[36485]: pgmap v1096: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:25.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:24 smithi106 ceph-mon[36505]: pgmap v1096: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:25.823 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1794 sec 2024-03-24T00:54:26.823 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1795 sec 2024-03-24T00:54:27.823 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1796 sec 2024-03-24T00:54:28.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:27 smithi078 ceph-mon[32429]: pgmap v1097: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:27 smithi107 ceph-mon[36485]: pgmap v1097: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:28.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:27 smithi106 ceph-mon[36505]: pgmap v1097: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:28.823 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1797 sec 2024-03-24T00:54:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1143.123684 secs 2024-03-24T00:54:29.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:28 smithi078 ceph-mon[32429]: pgmap v1098: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1143.123684 secs 2024-03-24T00:54:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:28 smithi107 ceph-mon[36485]: pgmap v1098: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1143.123684 secs 2024-03-24T00:54:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:28 smithi106 ceph-mon[36505]: pgmap v1098: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:29.823 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1798 sec 2024-03-24T00:54:30.823 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1799 sec 2024-03-24T00:54:31.823 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1800 sec 2024-03-24T00:54:32.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:31 smithi078 ceph-mon[32429]: pgmap v1099: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:32.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:31 smithi107 ceph-mon[36485]: pgmap v1099: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:32.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:31 smithi106 ceph-mon[36505]: pgmap v1099: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:32.823 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1801 sec 2024-03-24T00:54:33.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:32 smithi078 ceph-mon[32429]: pgmap v1100: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:33.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:32 smithi107 ceph-mon[36485]: pgmap v1100: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:33.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:32 smithi106 ceph-mon[36505]: pgmap v1100: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:33.823 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1802 sec 2024-03-24T00:54:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1148.123746 secs 2024-03-24T00:54:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1148.123746 secs 2024-03-24T00:54:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1148.123746 secs 2024-03-24T00:54:34.825 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1803 sec 2024-03-24T00:54:35.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:34 smithi106 ceph-mon[36505]: pgmap v1101: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:35.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:34 smithi078 ceph-mon[32429]: pgmap v1101: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:35.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:34 smithi107 ceph-mon[36485]: pgmap v1101: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:35.824 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1804 sec 2024-03-24T00:54:36.824 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1805 sec 2024-03-24T00:54:37.824 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1806 sec 2024-03-24T00:54:38.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:37 smithi078 ceph-mon[32429]: pgmap v1102: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:38.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:37 smithi107 ceph-mon[36485]: pgmap v1102: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:38.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:37 smithi106 ceph-mon[36505]: pgmap v1102: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:38.824 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1807 sec 2024-03-24T00:54:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1153.123831 secs 2024-03-24T00:54:39.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:38 smithi078 ceph-mon[32429]: pgmap v1103: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1153.123831 secs 2024-03-24T00:54:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:38 smithi107 ceph-mon[36485]: pgmap v1103: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1153.123831 secs 2024-03-24T00:54:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:38 smithi106 ceph-mon[36505]: pgmap v1103: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:39.824 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1808 sec 2024-03-24T00:54:40.824 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1809 sec 2024-03-24T00:54:41.824 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1810 sec 2024-03-24T00:54:42.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:41 smithi078 ceph-mon[32429]: pgmap v1104: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:42.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:41 smithi107 ceph-mon[36485]: pgmap v1104: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:42.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:41 smithi106 ceph-mon[36505]: pgmap v1104: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:42.824 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1811 sec 2024-03-24T00:54:43.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:42 smithi078 ceph-mon[32429]: pgmap v1105: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:43.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:42 smithi107 ceph-mon[36485]: pgmap v1105: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:43.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:42 smithi106 ceph-mon[36505]: pgmap v1105: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:43.824 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1812 sec 2024-03-24T00:54:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1158.123920 secs 2024-03-24T00:54:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1158.123920 secs 2024-03-24T00:54:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1158.123920 secs 2024-03-24T00:54:44.824 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1813 sec 2024-03-24T00:54:45.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:44 smithi078 ceph-mon[32429]: pgmap v1106: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:45.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:44 smithi107 ceph-mon[36485]: pgmap v1106: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:45.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:44 smithi106 ceph-mon[36505]: pgmap v1106: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:45.824 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1814 sec 2024-03-24T00:54:46.825 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1815 sec 2024-03-24T00:54:47.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:46 smithi107 ceph-mon[36485]: pgmap v1107: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:47.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:46 smithi106 ceph-mon[36505]: pgmap v1107: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:47.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:46 smithi078 ceph-mon[32429]: pgmap v1107: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:47.825 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1816 sec 2024-03-24T00:54:48.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1163.124008 secs 2024-03-24T00:54:48.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1163.124008 secs 2024-03-24T00:54:48.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1163.124008 secs 2024-03-24T00:54:48.825 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1817 sec 2024-03-24T00:54:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:49 smithi106 ceph-mon[36505]: pgmap v1108: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:49 smithi078 ceph-mon[32429]: pgmap v1108: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:49 smithi107 ceph-mon[36485]: pgmap v1108: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:49.825 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1818 sec 2024-03-24T00:54:50.825 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1819 sec 2024-03-24T00:54:51.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:50 smithi107 ceph-mon[36485]: pgmap v1109: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:51.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:50 smithi106 ceph-mon[36505]: pgmap v1109: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:51.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:50 smithi078 ceph-mon[32429]: pgmap v1109: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:51.825 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1820 sec 2024-03-24T00:54:52.825 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1821 sec 2024-03-24T00:54:53.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:52 smithi107 ceph-mon[36485]: pgmap v1110: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:53.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:52 smithi106 ceph-mon[36505]: pgmap v1110: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:53.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:52 smithi078 ceph-mon[32429]: pgmap v1110: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:53.825 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1822 sec 2024-03-24T00:54:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1168.124071 secs 2024-03-24T00:54:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1168.124071 secs 2024-03-24T00:54:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1168.124071 secs 2024-03-24T00:54:54.825 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1823 sec 2024-03-24T00:54:55.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:54 smithi106 ceph-mon[36505]: pgmap v1111: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:55.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:54 smithi078 ceph-mon[32429]: pgmap v1111: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:55.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:54 smithi107 ceph-mon[36485]: pgmap v1111: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:54:55.825 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1824 sec 2024-03-24T00:54:56.826 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1825 sec 2024-03-24T00:54:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:56 smithi107 ceph-mon[36485]: pgmap v1112: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:54:57.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:56 smithi106 ceph-mon[36505]: pgmap v1112: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:57.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:54:57.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:56 smithi078 ceph-mon[32429]: pgmap v1112: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:57.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:54:57.825 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1826 sec 2024-03-24T00:54:58.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:57 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:54:58.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:57 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:54:58.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:57 smithi106 ceph-mon[36505]: pgmap v1113: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:54:58.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:57 smithi106 ceph-mon[36505]: pgmap v1114: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:54:58.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:57 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:54:58.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:57 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:54:58.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:57 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:54:58.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:57 smithi078 ceph-mon[32429]: pgmap v1113: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:54:58.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:57 smithi078 ceph-mon[32429]: pgmap v1114: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:54:58.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:57 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:54:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:57 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:54:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:57 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:54:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:57 smithi107 ceph-mon[36485]: pgmap v1113: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:54:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:57 smithi107 ceph-mon[36485]: pgmap v1114: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:54:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:57 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:54:58.826 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1827 sec 2024-03-24T00:54:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1173.124111 secs 2024-03-24T00:54:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1173.124111 secs 2024-03-24T00:54:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1173.124111 secs 2024-03-24T00:54:59.826 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1828 sec 2024-03-24T00:55:00.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:54:59 smithi106 ceph-mon[36505]: pgmap v1115: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:55:00.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:54:59 smithi078 ceph-mon[32429]: pgmap v1115: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:55:00.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:54:59 smithi107 ceph-mon[36485]: pgmap v1115: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:55:00.825 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1829 sec 2024-03-24T00:55:01.826 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1830 sec 2024-03-24T00:55:02.826 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1831 sec 2024-03-24T00:55:02.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:02 smithi078 ceph-mon[32429]: pgmap v1116: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:55:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:02 smithi107 ceph-mon[36485]: pgmap v1116: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:55:02.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:02 smithi106 ceph-mon[36505]: pgmap v1116: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:55:03.826 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1832 sec 2024-03-24T00:55:03.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1178.124172 secs 2024-03-24T00:55:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1178.124172 secs 2024-03-24T00:55:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1178.124172 secs 2024-03-24T00:55:04.826 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1833 sec 2024-03-24T00:55:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:04 smithi078 ceph-mon[32429]: pgmap v1117: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:55:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:04 smithi107 ceph-mon[36485]: pgmap v1117: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:55:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:04 smithi106 ceph-mon[36505]: pgmap v1117: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:55:05.826 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1834 sec 2024-03-24T00:55:06.826 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1835 sec 2024-03-24T00:55:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:06 smithi078 ceph-mon[32429]: pgmap v1118: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:55:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:06 smithi107 ceph-mon[36485]: pgmap v1118: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:55:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:06 smithi106 ceph-mon[36505]: pgmap v1118: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:55:07.826 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1836 sec 2024-03-24T00:55:08.826 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1837 sec 2024-03-24T00:55:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:08 smithi078 ceph-mon[32429]: pgmap v1119: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:55:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1183.124213 secs 2024-03-24T00:55:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:08 smithi107 ceph-mon[36485]: pgmap v1119: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:55:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1183.124213 secs 2024-03-24T00:55:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:08 smithi106 ceph-mon[36505]: pgmap v1119: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:55:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1183.124213 secs 2024-03-24T00:55:09.827 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1838 sec 2024-03-24T00:55:10.827 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1839 sec 2024-03-24T00:55:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:10 smithi078 ceph-mon[32429]: pgmap v1120: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:10 smithi107 ceph-mon[36485]: pgmap v1120: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:10 smithi106 ceph-mon[36505]: pgmap v1120: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:11.827 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1840 sec 2024-03-24T00:55:12.827 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1841 sec 2024-03-24T00:55:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:12 smithi078 ceph-mon[32429]: pgmap v1121: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:12 smithi107 ceph-mon[36485]: pgmap v1121: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:12.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:12 smithi106 ceph-mon[36505]: pgmap v1121: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:13.827 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1842 sec 2024-03-24T00:55:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1188.124301 secs 2024-03-24T00:55:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1188.124301 secs 2024-03-24T00:55:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1188.124301 secs 2024-03-24T00:55:14.827 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1843 sec 2024-03-24T00:55:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:14 smithi078 ceph-mon[32429]: pgmap v1122: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:14 smithi107 ceph-mon[36485]: pgmap v1122: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:14 smithi106 ceph-mon[36505]: pgmap v1122: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:15.827 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1844 sec 2024-03-24T00:55:16.827 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1845 sec 2024-03-24T00:55:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:16 smithi078 ceph-mon[32429]: pgmap v1123: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:16 smithi107 ceph-mon[36485]: pgmap v1123: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:16.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:16 smithi106 ceph-mon[36505]: pgmap v1123: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:17.827 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1846 sec 2024-03-24T00:55:18.827 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1847 sec 2024-03-24T00:55:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:18 smithi106 ceph-mon[36505]: pgmap v1124: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1193.124360 secs 2024-03-24T00:55:19.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:18 smithi078 ceph-mon[32429]: pgmap v1124: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:19.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1193.124360 secs 2024-03-24T00:55:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:18 smithi107 ceph-mon[36485]: pgmap v1124: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1193.124360 secs 2024-03-24T00:55:19.827 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1848 sec 2024-03-24T00:55:20.827 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1849 sec 2024-03-24T00:55:20.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:20 smithi106 ceph-mon[36505]: pgmap v1125: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:21.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:20 smithi078 ceph-mon[32429]: pgmap v1125: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:21.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:20 smithi107 ceph-mon[36485]: pgmap v1125: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:21.827 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1850 sec 2024-03-24T00:55:22.827 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1851 sec 2024-03-24T00:55:22.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:22 smithi106 ceph-mon[36505]: pgmap v1126: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:23.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:22 smithi078 ceph-mon[32429]: pgmap v1126: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:23.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:22 smithi107 ceph-mon[36485]: pgmap v1126: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:23.828 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1852 sec 2024-03-24T00:55:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1198.124420 secs 2024-03-24T00:55:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1198.124420 secs 2024-03-24T00:55:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1198.124420 secs 2024-03-24T00:55:24.828 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1853 sec 2024-03-24T00:55:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:24 smithi106 ceph-mon[36505]: pgmap v1127: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:25.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:24 smithi078 ceph-mon[32429]: pgmap v1127: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:24 smithi107 ceph-mon[36485]: pgmap v1127: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:25.828 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1854 sec 2024-03-24T00:55:26.828 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1855 sec 2024-03-24T00:55:26.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:26 smithi106 ceph-mon[36505]: pgmap v1128: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:27.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:26 smithi078 ceph-mon[32429]: pgmap v1128: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:27.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:26 smithi107 ceph-mon[36485]: pgmap v1128: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:27.828 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1856 sec 2024-03-24T00:55:28.828 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1857 sec 2024-03-24T00:55:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:28 smithi106 ceph-mon[36505]: pgmap v1129: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1203.124501 secs 2024-03-24T00:55:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:28 smithi078 ceph-mon[32429]: pgmap v1129: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:29.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1203.124501 secs 2024-03-24T00:55:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:28 smithi107 ceph-mon[36485]: pgmap v1129: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1203.124501 secs 2024-03-24T00:55:29.828 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1858 sec 2024-03-24T00:55:30.828 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1859 sec 2024-03-24T00:55:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:30 smithi106 ceph-mon[36505]: pgmap v1130: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:31.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:30 smithi078 ceph-mon[32429]: pgmap v1130: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:31.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:30 smithi107 ceph-mon[36485]: pgmap v1130: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:31.828 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1860 sec 2024-03-24T00:55:32.829 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1861 sec 2024-03-24T00:55:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:32 smithi106 ceph-mon[36505]: pgmap v1131: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:33.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:32 smithi078 ceph-mon[32429]: pgmap v1131: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:33.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:32 smithi107 ceph-mon[36485]: pgmap v1131: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:33.829 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1862 sec 2024-03-24T00:55:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1208.124561 secs 2024-03-24T00:55:33.995 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1208.124561 secs 2024-03-24T00:55:34.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1208.124561 secs 2024-03-24T00:55:34.829 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1863 sec 2024-03-24T00:55:34.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:34 smithi106 ceph-mon[36505]: pgmap v1132: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:35.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:34 smithi078 ceph-mon[32429]: pgmap v1132: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:35.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:34 smithi107 ceph-mon[36485]: pgmap v1132: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:35.829 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1864 sec 2024-03-24T00:55:36.829 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1865 sec 2024-03-24T00:55:36.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:36 smithi106 ceph-mon[36505]: pgmap v1133: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:37.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:36 smithi078 ceph-mon[32429]: pgmap v1133: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:37.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:36 smithi107 ceph-mon[36485]: pgmap v1133: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:37.829 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1866 sec 2024-03-24T00:55:38.829 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1867 sec 2024-03-24T00:55:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:38 smithi106 ceph-mon[36505]: pgmap v1134: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:38.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1213.124592 secs 2024-03-24T00:55:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:38 smithi078 ceph-mon[32429]: pgmap v1134: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:39.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1213.124592 secs 2024-03-24T00:55:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:38 smithi107 ceph-mon[36485]: pgmap v1134: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1213.124592 secs 2024-03-24T00:55:39.829 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1868 sec 2024-03-24T00:55:40.829 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1869 sec 2024-03-24T00:55:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:40 smithi106 ceph-mon[36505]: pgmap v1135: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:40 smithi078 ceph-mon[32429]: pgmap v1135: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:41.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:40 smithi107 ceph-mon[36485]: pgmap v1135: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:41.829 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1870 sec 2024-03-24T00:55:42.829 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1871 sec 2024-03-24T00:55:42.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:42 smithi078 ceph-mon[32429]: pgmap v1136: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:42.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:42 smithi106 ceph-mon[36505]: pgmap v1136: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:43.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:42 smithi107 ceph-mon[36485]: pgmap v1136: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:43.830 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1872 sec 2024-03-24T00:55:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1218.124678 secs 2024-03-24T00:55:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1218.124678 secs 2024-03-24T00:55:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1218.124678 secs 2024-03-24T00:55:44.829 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1873 sec 2024-03-24T00:55:44.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:44 smithi106 ceph-mon[36505]: pgmap v1137: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:45.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:44 smithi078 ceph-mon[32429]: pgmap v1137: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:45.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:44 smithi107 ceph-mon[36485]: pgmap v1137: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:45.830 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1874 sec 2024-03-24T00:55:46.830 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1875 sec 2024-03-24T00:55:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:46 smithi106 ceph-mon[36505]: pgmap v1138: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:47.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:46 smithi078 ceph-mon[32429]: pgmap v1138: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:47.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:46 smithi107 ceph-mon[36485]: pgmap v1138: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:47.830 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1876 sec 2024-03-24T00:55:48.830 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1877 sec 2024-03-24T00:55:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:48 smithi078 ceph-mon[32429]: pgmap v1139: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:49.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1223.124767 secs 2024-03-24T00:55:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:48 smithi107 ceph-mon[36485]: pgmap v1139: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1223.124767 secs 2024-03-24T00:55:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:48 smithi106 ceph-mon[36505]: pgmap v1139: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1223.124767 secs 2024-03-24T00:55:49.830 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1878 sec 2024-03-24T00:55:50.830 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1879 sec 2024-03-24T00:55:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:50 smithi078 ceph-mon[32429]: pgmap v1140: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:51.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:50 smithi107 ceph-mon[36485]: pgmap v1140: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:51.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:50 smithi106 ceph-mon[36505]: pgmap v1140: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:51.830 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1880 sec 2024-03-24T00:55:52.830 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1881 sec 2024-03-24T00:55:53.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:52 smithi078 ceph-mon[32429]: pgmap v1141: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:53.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:52 smithi107 ceph-mon[36485]: pgmap v1141: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:53.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:52 smithi106 ceph-mon[36505]: pgmap v1141: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:53.830 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1882 sec 2024-03-24T00:55:54.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1228.124852 secs 2024-03-24T00:55:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1228.124852 secs 2024-03-24T00:55:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1228.124852 secs 2024-03-24T00:55:54.830 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1883 sec 2024-03-24T00:55:55.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:54 smithi078 ceph-mon[32429]: pgmap v1142: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:55.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:54 smithi107 ceph-mon[36485]: pgmap v1142: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:55.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:54 smithi106 ceph-mon[36505]: pgmap v1142: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:55.830 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1884 sec 2024-03-24T00:55:56.831 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1885 sec 2024-03-24T00:55:57.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:56 smithi078 ceph-mon[32429]: pgmap v1143: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:56 smithi107 ceph-mon[36485]: pgmap v1143: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:57.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:56 smithi106 ceph-mon[36505]: pgmap v1143: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:57.831 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1886 sec 2024-03-24T00:55:58.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:57 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:55:58.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:57 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:55:58.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:57 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:55:58.831 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1887 sec 2024-03-24T00:55:59.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:58 smithi078 ceph-mon[32429]: pgmap v1144: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:59.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1233.124921 secs 2024-03-24T00:55:59.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:58 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:55:59.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:58 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:55:59.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:58 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:55:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:58 smithi107 ceph-mon[36485]: pgmap v1144: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1233.124921 secs 2024-03-24T00:55:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:58 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:55:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:58 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:55:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:58 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:55:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:58 smithi106 ceph-mon[36505]: pgmap v1144: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:55:59.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1233.124921 secs 2024-03-24T00:55:59.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:58 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:55:59.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:58 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:55:59.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:58 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:55:59.831 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1888 sec 2024-03-24T00:56:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:59 smithi078 ceph-mon[32429]: pgmap v1145: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:56:00.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:55:59 smithi078 ceph-mon[32429]: pgmap v1146: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:56:00.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:59 smithi107 ceph-mon[36485]: pgmap v1145: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:56:00.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:55:59 smithi107 ceph-mon[36485]: pgmap v1146: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:56:00.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:59 smithi106 ceph-mon[36505]: pgmap v1145: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:56:00.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:55:59 smithi106 ceph-mon[36505]: pgmap v1146: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:56:00.831 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1889 sec 2024-03-24T00:56:01.831 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1890 sec 2024-03-24T00:56:02.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:01 smithi078 ceph-mon[32429]: pgmap v1147: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:56:02.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:01 smithi107 ceph-mon[36485]: pgmap v1147: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:56:02.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:01 smithi106 ceph-mon[36505]: pgmap v1147: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:56:02.831 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1891 sec 2024-03-24T00:56:03.831 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1892 sec 2024-03-24T00:56:04.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:03 smithi078 ceph-mon[32429]: pgmap v1148: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:56:04.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1238.124978 secs 2024-03-24T00:56:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:03 smithi107 ceph-mon[36485]: pgmap v1148: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:56:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1238.124978 secs 2024-03-24T00:56:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:03 smithi106 ceph-mon[36505]: pgmap v1148: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:56:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1238.124978 secs 2024-03-24T00:56:04.831 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1893 sec 2024-03-24T00:56:05.831 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1894 sec 2024-03-24T00:56:06.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:05 smithi078 ceph-mon[32429]: pgmap v1149: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:56:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:05 smithi107 ceph-mon[36485]: pgmap v1149: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:56:06.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:05 smithi106 ceph-mon[36505]: pgmap v1149: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:56:06.832 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1895 sec 2024-03-24T00:56:07.832 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1896 sec 2024-03-24T00:56:07.954 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:07 smithi106 ceph-mon[36505]: pgmap v1150: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:56:08.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:07 smithi078 ceph-mon[32429]: pgmap v1150: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:56:08.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:07 smithi107 ceph-mon[36485]: pgmap v1150: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:56:08.832 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1897 sec 2024-03-24T00:56:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1243.125056 secs 2024-03-24T00:56:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1243.125056 secs 2024-03-24T00:56:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1243.125056 secs 2024-03-24T00:56:09.832 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1898 sec 2024-03-24T00:56:10.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:09 smithi078 ceph-mon[32429]: pgmap v1151: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:56:10.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:09 smithi107 ceph-mon[36485]: pgmap v1151: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:56:10.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:09 smithi106 ceph-mon[36505]: pgmap v1151: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:56:10.832 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1899 sec 2024-03-24T00:56:11.832 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1900 sec 2024-03-24T00:56:12.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:11 smithi078 ceph-mon[32429]: pgmap v1152: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:12.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:11 smithi107 ceph-mon[36485]: pgmap v1152: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:12.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:11 smithi106 ceph-mon[36505]: pgmap v1152: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:12.832 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1901 sec 2024-03-24T00:56:13.832 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1902 sec 2024-03-24T00:56:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:13 smithi078 ceph-mon[32429]: pgmap v1153: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:14.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1248.125117 secs 2024-03-24T00:56:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:13 smithi107 ceph-mon[36485]: pgmap v1153: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1248.125117 secs 2024-03-24T00:56:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:13 smithi106 ceph-mon[36505]: pgmap v1153: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1248.125117 secs 2024-03-24T00:56:14.832 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1903 sec 2024-03-24T00:56:15.832 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1904 sec 2024-03-24T00:56:16.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:15 smithi078 ceph-mon[32429]: pgmap v1154: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:16.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:15 smithi107 ceph-mon[36485]: pgmap v1154: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:16.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:15 smithi106 ceph-mon[36505]: pgmap v1154: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:16.833 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1905 sec 2024-03-24T00:56:17.833 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1906 sec 2024-03-24T00:56:18.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:17 smithi078 ceph-mon[32429]: pgmap v1155: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:18.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:17 smithi107 ceph-mon[36485]: pgmap v1155: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:18.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:17 smithi106 ceph-mon[36505]: pgmap v1155: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:18.833 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1907 sec 2024-03-24T00:56:19.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1253.125206 secs 2024-03-24T00:56:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1253.125206 secs 2024-03-24T00:56:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1253.125206 secs 2024-03-24T00:56:19.833 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1908 sec 2024-03-24T00:56:20.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:19 smithi078 ceph-mon[32429]: pgmap v1156: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:20.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:19 smithi107 ceph-mon[36485]: pgmap v1156: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:20.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:19 smithi106 ceph-mon[36505]: pgmap v1156: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:20.833 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1909 sec 2024-03-24T00:56:21.833 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1910 sec 2024-03-24T00:56:22.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:21 smithi078 ceph-mon[32429]: pgmap v1157: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:22.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:21 smithi107 ceph-mon[36485]: pgmap v1157: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:22.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:21 smithi106 ceph-mon[36505]: pgmap v1157: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:22.833 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1911 sec 2024-03-24T00:56:23.833 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1912 sec 2024-03-24T00:56:24.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:23 smithi078 ceph-mon[32429]: pgmap v1158: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:24.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1258.125250 secs 2024-03-24T00:56:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:23 smithi107 ceph-mon[36485]: pgmap v1158: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1258.125250 secs 2024-03-24T00:56:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:23 smithi106 ceph-mon[36505]: pgmap v1158: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1258.125250 secs 2024-03-24T00:56:24.833 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1913 sec 2024-03-24T00:56:25.833 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1914 sec 2024-03-24T00:56:26.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:25 smithi078 ceph-mon[32429]: pgmap v1159: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:26.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:25 smithi107 ceph-mon[36485]: pgmap v1159: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:26.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:25 smithi106 ceph-mon[36505]: pgmap v1159: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:26.833 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1915 sec 2024-03-24T00:56:27.833 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1916 sec 2024-03-24T00:56:28.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:27 smithi078 ceph-mon[32429]: pgmap v1160: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:27 smithi107 ceph-mon[36485]: pgmap v1160: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:28.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:27 smithi106 ceph-mon[36505]: pgmap v1160: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:28.834 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1917 sec 2024-03-24T00:56:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1263.125317 secs 2024-03-24T00:56:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1263.125317 secs 2024-03-24T00:56:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1263.125317 secs 2024-03-24T00:56:29.834 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1918 sec 2024-03-24T00:56:29.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:29 smithi078 ceph-mon[32429]: pgmap v1161: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:30.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:29 smithi107 ceph-mon[36485]: pgmap v1161: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:30.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:29 smithi106 ceph-mon[36505]: pgmap v1161: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:30.834 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1919 sec 2024-03-24T00:56:31.834 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1920 sec 2024-03-24T00:56:32.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:31 smithi078 ceph-mon[32429]: pgmap v1162: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:32.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:31 smithi107 ceph-mon[36485]: pgmap v1162: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:32.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:31 smithi106 ceph-mon[36505]: pgmap v1162: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:32.834 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1921 sec 2024-03-24T00:56:33.834 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1922 sec 2024-03-24T00:56:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:33 smithi078 ceph-mon[32429]: pgmap v1163: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:34.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1268.125353 secs 2024-03-24T00:56:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:33 smithi107 ceph-mon[36485]: pgmap v1163: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1268.125353 secs 2024-03-24T00:56:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:33 smithi106 ceph-mon[36505]: pgmap v1163: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1268.125353 secs 2024-03-24T00:56:34.834 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1923 sec 2024-03-24T00:56:35.834 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1924 sec 2024-03-24T00:56:36.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:35 smithi078 ceph-mon[32429]: pgmap v1164: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:36.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:35 smithi107 ceph-mon[36485]: pgmap v1164: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:36.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:35 smithi106 ceph-mon[36505]: pgmap v1164: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:36.834 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1925 sec 2024-03-24T00:56:37.834 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1926 sec 2024-03-24T00:56:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:37 smithi078 ceph-mon[32429]: pgmap v1165: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:38.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:37 smithi107 ceph-mon[36485]: pgmap v1165: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:38.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:37 smithi106 ceph-mon[36505]: pgmap v1165: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:38.834 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1927 sec 2024-03-24T00:56:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1273.125439 secs 2024-03-24T00:56:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1273.125439 secs 2024-03-24T00:56:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1273.125439 secs 2024-03-24T00:56:39.835 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1928 sec 2024-03-24T00:56:40.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:39 smithi078 ceph-mon[32429]: pgmap v1166: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:39 smithi107 ceph-mon[36485]: pgmap v1166: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:39 smithi106 ceph-mon[36505]: pgmap v1166: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:40.834 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1929 sec 2024-03-24T00:56:41.835 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1930 sec 2024-03-24T00:56:42.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:41 smithi078 ceph-mon[32429]: pgmap v1167: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:42.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:41 smithi107 ceph-mon[36485]: pgmap v1167: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:42.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:41 smithi106 ceph-mon[36505]: pgmap v1167: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:42.835 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1931 sec 2024-03-24T00:56:43.836 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1932 sec 2024-03-24T00:56:44.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:43 smithi078 ceph-mon[32429]: pgmap v1168: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:44.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1278.125500 secs 2024-03-24T00:56:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:43 smithi107 ceph-mon[36485]: pgmap v1168: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1278.125500 secs 2024-03-24T00:56:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:43 smithi106 ceph-mon[36505]: pgmap v1168: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:44.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1278.125500 secs 2024-03-24T00:56:44.835 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1933 sec 2024-03-24T00:56:45.835 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1934 sec 2024-03-24T00:56:46.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:45 smithi078 ceph-mon[32429]: pgmap v1169: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:45 smithi107 ceph-mon[36485]: pgmap v1169: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:46.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:45 smithi106 ceph-mon[36505]: pgmap v1169: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:46.835 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1935 sec 2024-03-24T00:56:47.835 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1936 sec 2024-03-24T00:56:48.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:47 smithi078 ceph-mon[32429]: pgmap v1170: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:48.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:47 smithi107 ceph-mon[36485]: pgmap v1170: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:48.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:47 smithi106 ceph-mon[36505]: pgmap v1170: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:48.835 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1937 sec 2024-03-24T00:56:49.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1283.125585 secs 2024-03-24T00:56:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1283.125585 secs 2024-03-24T00:56:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1283.125585 secs 2024-03-24T00:56:49.835 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1938 sec 2024-03-24T00:56:50.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:49 smithi078 ceph-mon[32429]: pgmap v1171: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:50.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:49 smithi107 ceph-mon[36485]: pgmap v1171: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:49 smithi106 ceph-mon[36505]: pgmap v1171: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:50.835 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1939 sec 2024-03-24T00:56:51.835 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1940 sec 2024-03-24T00:56:52.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:51 smithi078 ceph-mon[32429]: pgmap v1172: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:52.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:51 smithi107 ceph-mon[36485]: pgmap v1172: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:52.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:51 smithi106 ceph-mon[36505]: pgmap v1172: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:52.835 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1941 sec 2024-03-24T00:56:53.836 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1942 sec 2024-03-24T00:56:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:53 smithi078 ceph-mon[32429]: pgmap v1173: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:54.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1288.125639 secs 2024-03-24T00:56:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:53 smithi107 ceph-mon[36485]: pgmap v1173: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1288.125639 secs 2024-03-24T00:56:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:53 smithi106 ceph-mon[36505]: pgmap v1173: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1288.125639 secs 2024-03-24T00:56:54.836 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1943 sec 2024-03-24T00:56:55.836 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1944 sec 2024-03-24T00:56:56.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:55 smithi078 ceph-mon[32429]: pgmap v1174: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:56.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:55 smithi107 ceph-mon[36485]: pgmap v1174: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:56.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:55 smithi106 ceph-mon[36505]: pgmap v1174: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:56:56.836 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1945 sec 2024-03-24T00:56:57.836 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1946 sec 2024-03-24T00:56:58.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:57 smithi078 ceph-mon[32429]: pgmap v1175: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:58.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:57 smithi107 ceph-mon[36485]: pgmap v1175: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:58.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:57 smithi106 ceph-mon[36505]: pgmap v1175: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:58.836 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1947 sec 2024-03-24T00:56:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1293.125699 secs 2024-03-24T00:56:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:58 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:56:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1293.125699 secs 2024-03-24T00:56:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:58 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:56:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1293.125699 secs 2024-03-24T00:56:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:58 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:56:59.836 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1948 sec 2024-03-24T00:57:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:59 smithi078 ceph-mon[32429]: pgmap v1176: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:00.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:59 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:57:00.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:59 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:57:00.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:56:59 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:57:00.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:59 smithi107 ceph-mon[36485]: pgmap v1176: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:00.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:59 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:57:00.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:59 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:57:00.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:56:59 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:57:00.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:59 smithi106 ceph-mon[36505]: pgmap v1176: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:00.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:59 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:57:00.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:59 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:57:00.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:56:59 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:57:00.836 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1949 sec 2024-03-24T00:57:01.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:00 smithi078 ceph-mon[32429]: pgmap v1177: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:57:01.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:00 smithi078 ceph-mon[32429]: pgmap v1178: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:57:01.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:00 smithi107 ceph-mon[36485]: pgmap v1177: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:57:01.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:00 smithi107 ceph-mon[36485]: pgmap v1178: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:57:01.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:00 smithi106 ceph-mon[36505]: pgmap v1177: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:57:01.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:00 smithi106 ceph-mon[36505]: pgmap v1178: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:57:01.836 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1950 sec 2024-03-24T00:57:02.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:01 smithi078 ceph-mon[32429]: pgmap v1179: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:57:02.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:01 smithi107 ceph-mon[36485]: pgmap v1179: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:57:02.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:01 smithi106 ceph-mon[36505]: pgmap v1179: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:57:02.836 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1951 sec 2024-03-24T00:57:03.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1298.125737 secs 2024-03-24T00:57:03.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1298.125737 secs 2024-03-24T00:57:03.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1298.125737 secs 2024-03-24T00:57:03.836 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1952 sec 2024-03-24T00:57:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:04 smithi106 ceph-mon[36505]: pgmap v1180: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:57:04.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:04 smithi078 ceph-mon[32429]: pgmap v1180: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:57:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:04 smithi107 ceph-mon[36485]: pgmap v1180: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:57:04.836 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1953 sec 2024-03-24T00:57:05.837 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1954 sec 2024-03-24T00:57:06.837 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1955 sec 2024-03-24T00:57:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:06 smithi078 ceph-mon[32429]: pgmap v1181: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:57:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:06 smithi107 ceph-mon[36485]: pgmap v1181: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:57:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:06 smithi106 ceph-mon[36505]: pgmap v1181: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:57:07.837 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1956 sec 2024-03-24T00:57:08.837 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1957 sec 2024-03-24T00:57:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:08 smithi078 ceph-mon[32429]: pgmap v1182: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:57:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1303.125821 secs 2024-03-24T00:57:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:08 smithi107 ceph-mon[36485]: pgmap v1182: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:57:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1303.125821 secs 2024-03-24T00:57:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:08 smithi106 ceph-mon[36505]: pgmap v1182: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:57:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1303.125821 secs 2024-03-24T00:57:09.837 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1958 sec 2024-03-24T00:57:10.837 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1959 sec 2024-03-24T00:57:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:10 smithi078 ceph-mon[32429]: pgmap v1183: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:57:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:10 smithi107 ceph-mon[36485]: pgmap v1183: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:57:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:10 smithi106 ceph-mon[36505]: pgmap v1183: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:57:11.837 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1960 sec 2024-03-24T00:57:12.837 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1961 sec 2024-03-24T00:57:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:12 smithi078 ceph-mon[32429]: pgmap v1184: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:12 smithi107 ceph-mon[36485]: pgmap v1184: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:12.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:12 smithi106 ceph-mon[36505]: pgmap v1184: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:13.837 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1962 sec 2024-03-24T00:57:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1308.125883 secs 2024-03-24T00:57:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1308.125883 secs 2024-03-24T00:57:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1308.125883 secs 2024-03-24T00:57:14.837 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1963 sec 2024-03-24T00:57:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:14 smithi078 ceph-mon[32429]: pgmap v1185: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:14 smithi107 ceph-mon[36485]: pgmap v1185: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:14 smithi106 ceph-mon[36505]: pgmap v1185: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:15.837 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1964 sec 2024-03-24T00:57:16.837 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1965 sec 2024-03-24T00:57:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:16 smithi078 ceph-mon[32429]: pgmap v1186: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:16 smithi107 ceph-mon[36485]: pgmap v1186: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:16.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:16 smithi106 ceph-mon[36505]: pgmap v1186: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:17.837 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1966 sec 2024-03-24T00:57:18.838 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1967 sec 2024-03-24T00:57:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:18 smithi078 ceph-mon[32429]: pgmap v1187: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1313.125944 secs 2024-03-24T00:57:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:18 smithi107 ceph-mon[36485]: pgmap v1187: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1313.125944 secs 2024-03-24T00:57:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:18 smithi106 ceph-mon[36505]: pgmap v1187: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:18.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1313.125944 secs 2024-03-24T00:57:19.838 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1968 sec 2024-03-24T00:57:20.838 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1969 sec 2024-03-24T00:57:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:20 smithi078 ceph-mon[32429]: pgmap v1188: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:20 smithi107 ceph-mon[36485]: pgmap v1188: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:20.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:20 smithi106 ceph-mon[36505]: pgmap v1188: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:21.838 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1970 sec 2024-03-24T00:57:22.838 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1971 sec 2024-03-24T00:57:22.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:22 smithi078 ceph-mon[32429]: pgmap v1189: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:22.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:22 smithi107 ceph-mon[36485]: pgmap v1189: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:22.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:22 smithi106 ceph-mon[36505]: pgmap v1189: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:23.838 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1972 sec 2024-03-24T00:57:23.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1318.126032 secs 2024-03-24T00:57:23.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1318.126032 secs 2024-03-24T00:57:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1318.126032 secs 2024-03-24T00:57:24.838 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1973 sec 2024-03-24T00:57:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:24 smithi078 ceph-mon[32429]: pgmap v1190: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:24 smithi107 ceph-mon[36485]: pgmap v1190: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:24 smithi106 ceph-mon[36505]: pgmap v1190: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:25.838 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1974 sec 2024-03-24T00:57:26.838 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1975 sec 2024-03-24T00:57:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:26 smithi078 ceph-mon[32429]: pgmap v1191: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:26 smithi107 ceph-mon[36485]: pgmap v1191: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:26.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:26 smithi106 ceph-mon[36505]: pgmap v1191: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:27.838 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1976 sec 2024-03-24T00:57:28.839 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1977 sec 2024-03-24T00:57:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:28 smithi078 ceph-mon[32429]: pgmap v1192: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1323.126121 secs 2024-03-24T00:57:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:28 smithi107 ceph-mon[36485]: pgmap v1192: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1323.126121 secs 2024-03-24T00:57:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:28 smithi106 ceph-mon[36505]: pgmap v1192: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1323.126121 secs 2024-03-24T00:57:29.839 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1978 sec 2024-03-24T00:57:30.839 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1979 sec 2024-03-24T00:57:30.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:30 smithi078 ceph-mon[32429]: pgmap v1193: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:30 smithi107 ceph-mon[36485]: pgmap v1193: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:30 smithi106 ceph-mon[36505]: pgmap v1193: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:31.839 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1980 sec 2024-03-24T00:57:32.839 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1981 sec 2024-03-24T00:57:32.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:32 smithi078 ceph-mon[32429]: pgmap v1194: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:32 smithi107 ceph-mon[36485]: pgmap v1194: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:32 smithi106 ceph-mon[36505]: pgmap v1194: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:33.839 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1982 sec 2024-03-24T00:57:33.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1328.126182 secs 2024-03-24T00:57:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1328.126182 secs 2024-03-24T00:57:33.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1328.126182 secs 2024-03-24T00:57:34.839 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1983 sec 2024-03-24T00:57:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:34 smithi078 ceph-mon[32429]: pgmap v1195: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:34 smithi107 ceph-mon[36485]: pgmap v1195: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:34 smithi106 ceph-mon[36505]: pgmap v1195: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:35.839 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1984 sec 2024-03-24T00:57:36.839 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1985 sec 2024-03-24T00:57:36.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:36 smithi078 ceph-mon[32429]: pgmap v1196: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:36 smithi107 ceph-mon[36485]: pgmap v1196: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:36.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:36 smithi106 ceph-mon[36505]: pgmap v1196: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:37.839 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1986 sec 2024-03-24T00:57:38.839 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1987 sec 2024-03-24T00:57:38.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:38 smithi078 ceph-mon[32429]: pgmap v1197: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:38.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1333.126243 secs 2024-03-24T00:57:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:38 smithi107 ceph-mon[36485]: pgmap v1197: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1333.126243 secs 2024-03-24T00:57:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:38 smithi106 ceph-mon[36505]: pgmap v1197: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1333.126243 secs 2024-03-24T00:57:39.840 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1988 sec 2024-03-24T00:57:40.840 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1989 sec 2024-03-24T00:57:40.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:40 smithi078 ceph-mon[32429]: pgmap v1198: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:40.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:40 smithi107 ceph-mon[36485]: pgmap v1198: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:40 smithi106 ceph-mon[36505]: pgmap v1198: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:41.840 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1990 sec 2024-03-24T00:57:42.840 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1991 sec 2024-03-24T00:57:42.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:42 smithi078 ceph-mon[32429]: pgmap v1199: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:42.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:42 smithi107 ceph-mon[36485]: pgmap v1199: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:42.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:42 smithi106 ceph-mon[36505]: pgmap v1199: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:43.841 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1992 sec 2024-03-24T00:57:43.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1338.126311 secs 2024-03-24T00:57:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1338.126311 secs 2024-03-24T00:57:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1338.126311 secs 2024-03-24T00:57:44.840 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1993 sec 2024-03-24T00:57:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:44 smithi078 ceph-mon[32429]: pgmap v1200: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:44 smithi107 ceph-mon[36485]: pgmap v1200: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:44.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:44 smithi106 ceph-mon[36505]: pgmap v1200: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:45.840 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1994 sec 2024-03-24T00:57:46.840 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1995 sec 2024-03-24T00:57:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:46 smithi078 ceph-mon[32429]: pgmap v1201: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:46.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:46 smithi107 ceph-mon[36485]: pgmap v1201: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:46 smithi106 ceph-mon[36505]: pgmap v1201: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:47.840 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1996 sec 2024-03-24T00:57:48.840 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1997 sec 2024-03-24T00:57:48.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:48 smithi078 ceph-mon[32429]: pgmap v1202: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:48.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1343.126397 secs 2024-03-24T00:57:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:48 smithi107 ceph-mon[36485]: pgmap v1202: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1343.126397 secs 2024-03-24T00:57:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:48 smithi106 ceph-mon[36505]: pgmap v1202: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1343.126397 secs 2024-03-24T00:57:49.840 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1998 sec 2024-03-24T00:57:50.840 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 1999 sec 2024-03-24T00:57:50.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:50 smithi078 ceph-mon[32429]: pgmap v1203: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:50 smithi107 ceph-mon[36485]: pgmap v1203: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:50.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:50 smithi106 ceph-mon[36505]: pgmap v1203: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:51.841 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2000 sec 2024-03-24T00:57:52.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:52 smithi078 ceph-mon[32429]: pgmap v1204: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:52.841 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2001 sec 2024-03-24T00:57:52.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:52 smithi107 ceph-mon[36485]: pgmap v1204: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:52.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:52 smithi106 ceph-mon[36505]: pgmap v1204: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:53.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1348.126458 secs 2024-03-24T00:57:53.841 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2002 sec 2024-03-24T00:57:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1348.126458 secs 2024-03-24T00:57:53.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1348.126458 secs 2024-03-24T00:57:54.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:54 smithi078 ceph-mon[32429]: pgmap v1205: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:54.841 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2003 sec 2024-03-24T00:57:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:54 smithi107 ceph-mon[36485]: pgmap v1205: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:54 smithi106 ceph-mon[36505]: pgmap v1205: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:55.841 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2004 sec 2024-03-24T00:57:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:56 smithi078 ceph-mon[32429]: pgmap v1206: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:56.841 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2005 sec 2024-03-24T00:57:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:56 smithi107 ceph-mon[36485]: pgmap v1206: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:56.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:56 smithi106 ceph-mon[36505]: pgmap v1206: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:57:57.841 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2006 sec 2024-03-24T00:57:58.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:58 smithi078 ceph-mon[32429]: pgmap v1207: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:58.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1353.126518 secs 2024-03-24T00:57:58.841 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2007 sec 2024-03-24T00:57:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:58 smithi107 ceph-mon[36485]: pgmap v1207: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1353.126518 secs 2024-03-24T00:57:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:58 smithi106 ceph-mon[36505]: pgmap v1207: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1353.126518 secs 2024-03-24T00:57:59.841 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2008 sec 2024-03-24T00:57:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:57:59 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:57:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:57:59 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:58:00.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:57:59 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:58:00.842 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2009 sec 2024-03-24T00:58:00.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:00 smithi106 ceph-mon[36505]: pgmap v1208: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:00.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:00 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:58:00.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:00 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:58:00.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:00 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:58:01.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:00 smithi078 ceph-mon[32429]: pgmap v1208: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:01.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:00 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:58:01.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:00 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:58:01.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:00 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:58:01.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:00 smithi107 ceph-mon[36485]: pgmap v1208: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:01.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:00 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:58:01.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:00 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:58:01.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:00 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:58:01.841 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2010 sec 2024-03-24T00:58:01.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:01 smithi106 ceph-mon[36505]: pgmap v1209: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:58:01.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:01 smithi106 ceph-mon[36505]: pgmap v1210: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:58:02.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:01 smithi078 ceph-mon[32429]: pgmap v1209: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:58:02.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:01 smithi078 ceph-mon[32429]: pgmap v1210: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:58:02.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:01 smithi107 ceph-mon[36485]: pgmap v1209: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T00:58:02.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:01 smithi107 ceph-mon[36485]: pgmap v1210: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:58:02.841 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2011 sec 2024-03-24T00:58:03.841 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2012 sec 2024-03-24T00:58:03.895 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:03 smithi107 ceph-mon[36485]: pgmap v1211: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:58:03.895 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1358.126607 secs 2024-03-24T00:58:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:03 smithi106 ceph-mon[36505]: pgmap v1211: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:58:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1358.126607 secs 2024-03-24T00:58:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:03 smithi078 ceph-mon[32429]: pgmap v1211: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:58:04.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1358.126607 secs 2024-03-24T00:58:04.841 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2013 sec 2024-03-24T00:58:05.842 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2014 sec 2024-03-24T00:58:05.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:05 smithi106 ceph-mon[36505]: pgmap v1212: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:58:06.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:05 smithi078 ceph-mon[32429]: pgmap v1212: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:58:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:05 smithi107 ceph-mon[36485]: pgmap v1212: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:58:06.842 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2015 sec 2024-03-24T00:58:07.842 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2016 sec 2024-03-24T00:58:07.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:07 smithi106 ceph-mon[36505]: pgmap v1213: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:58:08.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:07 smithi078 ceph-mon[32429]: pgmap v1213: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:58:08.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:07 smithi107 ceph-mon[36485]: pgmap v1213: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:58:08.842 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2017 sec 2024-03-24T00:58:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1363.126665 secs 2024-03-24T00:58:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1363.126665 secs 2024-03-24T00:58:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1363.126665 secs 2024-03-24T00:58:09.842 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2018 sec 2024-03-24T00:58:09.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:09 smithi106 ceph-mon[36505]: pgmap v1214: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:58:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:09 smithi078 ceph-mon[32429]: pgmap v1214: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:58:10.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:09 smithi107 ceph-mon[36485]: pgmap v1214: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:58:10.842 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2019 sec 2024-03-24T00:58:11.842 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2020 sec 2024-03-24T00:58:11.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:11 smithi106 ceph-mon[36505]: pgmap v1215: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:58:12.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:11 smithi078 ceph-mon[32429]: pgmap v1215: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:58:12.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:11 smithi107 ceph-mon[36485]: pgmap v1215: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:58:12.842 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2021 sec 2024-03-24T00:58:13.842 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2022 sec 2024-03-24T00:58:13.893 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:13 smithi078 ceph-mon[32429]: pgmap v1216: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:13.893 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1368.126678 secs 2024-03-24T00:58:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:13 smithi106 ceph-mon[36505]: pgmap v1216: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1368.126678 secs 2024-03-24T00:58:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:13 smithi107 ceph-mon[36485]: pgmap v1216: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1368.126678 secs 2024-03-24T00:58:14.842 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2023 sec 2024-03-24T00:58:15.842 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2024 sec 2024-03-24T00:58:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:15 smithi106 ceph-mon[36505]: pgmap v1217: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:16.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:15 smithi078 ceph-mon[32429]: pgmap v1217: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:16.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:15 smithi107 ceph-mon[36485]: pgmap v1217: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:16.842 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2025 sec 2024-03-24T00:58:17.843 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2026 sec 2024-03-24T00:58:17.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:17 smithi106 ceph-mon[36505]: pgmap v1218: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:18.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:17 smithi078 ceph-mon[32429]: pgmap v1218: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:18.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:17 smithi107 ceph-mon[36485]: pgmap v1218: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:18.843 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2027 sec 2024-03-24T00:58:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1373.126768 secs 2024-03-24T00:58:18.992 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1373.126768 secs 2024-03-24T00:58:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1373.126768 secs 2024-03-24T00:58:19.843 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2028 sec 2024-03-24T00:58:19.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:19 smithi106 ceph-mon[36505]: pgmap v1219: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:20.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:19 smithi078 ceph-mon[32429]: pgmap v1219: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:20.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:19 smithi107 ceph-mon[36485]: pgmap v1219: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:20.843 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2029 sec 2024-03-24T00:58:21.843 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2030 sec 2024-03-24T00:58:21.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:21 smithi106 ceph-mon[36505]: pgmap v1220: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:22.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:21 smithi078 ceph-mon[32429]: pgmap v1220: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:22.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:21 smithi107 ceph-mon[36485]: pgmap v1220: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:22.843 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2031 sec 2024-03-24T00:58:23.843 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2032 sec 2024-03-24T00:58:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:23 smithi106 ceph-mon[36505]: pgmap v1221: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1378.126850 secs 2024-03-24T00:58:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:23 smithi078 ceph-mon[32429]: pgmap v1221: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:24.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1378.126850 secs 2024-03-24T00:58:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:23 smithi107 ceph-mon[36485]: pgmap v1221: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1378.126850 secs 2024-03-24T00:58:24.843 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2033 sec 2024-03-24T00:58:25.843 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2034 sec 2024-03-24T00:58:25.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:25 smithi106 ceph-mon[36505]: pgmap v1222: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:26.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:25 smithi078 ceph-mon[32429]: pgmap v1222: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:26.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:25 smithi107 ceph-mon[36485]: pgmap v1222: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:26.843 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2035 sec 2024-03-24T00:58:27.843 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2036 sec 2024-03-24T00:58:27.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:27 smithi106 ceph-mon[36505]: pgmap v1223: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:28.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:27 smithi078 ceph-mon[32429]: pgmap v1223: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:27 smithi107 ceph-mon[36485]: pgmap v1223: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:28.843 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2037 sec 2024-03-24T00:58:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1383.126929 secs 2024-03-24T00:58:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1383.126929 secs 2024-03-24T00:58:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1383.126929 secs 2024-03-24T00:58:29.844 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2038 sec 2024-03-24T00:58:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:29 smithi078 ceph-mon[32429]: pgmap v1224: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:30.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:29 smithi107 ceph-mon[36485]: pgmap v1224: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:30.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:29 smithi106 ceph-mon[36505]: pgmap v1224: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:30.844 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2039 sec 2024-03-24T00:58:31.844 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2040 sec 2024-03-24T00:58:32.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:31 smithi078 ceph-mon[32429]: pgmap v1225: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:32.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:31 smithi107 ceph-mon[36485]: pgmap v1225: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:32.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:31 smithi106 ceph-mon[36505]: pgmap v1225: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:32.844 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2041 sec 2024-03-24T00:58:33.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:33 smithi078 ceph-mon[32429]: pgmap v1226: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:33.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1388.126989 secs 2024-03-24T00:58:33.844 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2042 sec 2024-03-24T00:58:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:33 smithi107 ceph-mon[36485]: pgmap v1226: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1388.126989 secs 2024-03-24T00:58:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:33 smithi106 ceph-mon[36505]: pgmap v1226: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1388.126989 secs 2024-03-24T00:58:34.844 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2043 sec 2024-03-24T00:58:35.844 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2044 sec 2024-03-24T00:58:36.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:35 smithi078 ceph-mon[32429]: pgmap v1227: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:36.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:35 smithi107 ceph-mon[36485]: pgmap v1227: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:36.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:35 smithi106 ceph-mon[36505]: pgmap v1227: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:36.844 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2045 sec 2024-03-24T00:58:37.844 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2046 sec 2024-03-24T00:58:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:37 smithi078 ceph-mon[32429]: pgmap v1228: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:38.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:37 smithi107 ceph-mon[36485]: pgmap v1228: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:38.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:37 smithi106 ceph-mon[36505]: pgmap v1228: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:38.844 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2047 sec 2024-03-24T00:58:39.053 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1393.127081 secs 2024-03-24T00:58:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1393.127081 secs 2024-03-24T00:58:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1393.127081 secs 2024-03-24T00:58:39.844 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2048 sec 2024-03-24T00:58:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:39 smithi078 ceph-mon[32429]: pgmap v1229: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:39 smithi107 ceph-mon[36485]: pgmap v1229: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:39 smithi106 ceph-mon[36505]: pgmap v1229: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:40.844 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2049 sec 2024-03-24T00:58:41.845 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2050 sec 2024-03-24T00:58:42.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:41 smithi078 ceph-mon[32429]: pgmap v1230: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:42.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:41 smithi107 ceph-mon[36485]: pgmap v1230: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:42.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:41 smithi106 ceph-mon[36505]: pgmap v1230: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:42.845 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2051 sec 2024-03-24T00:58:43.845 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2052 sec 2024-03-24T00:58:44.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:43 smithi078 ceph-mon[32429]: pgmap v1231: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:44.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1398.127140 secs 2024-03-24T00:58:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:43 smithi107 ceph-mon[36485]: pgmap v1231: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1398.127140 secs 2024-03-24T00:58:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:43 smithi106 ceph-mon[36505]: pgmap v1231: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1398.127140 secs 2024-03-24T00:58:44.845 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2053 sec 2024-03-24T00:58:45.845 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2054 sec 2024-03-24T00:58:46.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:45 smithi078 ceph-mon[32429]: pgmap v1232: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:45 smithi107 ceph-mon[36485]: pgmap v1232: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:46.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:45 smithi106 ceph-mon[36505]: pgmap v1232: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:46.845 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2055 sec 2024-03-24T00:58:47.845 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2056 sec 2024-03-24T00:58:48.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:47 smithi078 ceph-mon[32429]: pgmap v1233: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:48.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:47 smithi107 ceph-mon[36485]: pgmap v1233: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:48.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:47 smithi106 ceph-mon[36505]: pgmap v1233: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:48.845 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2057 sec 2024-03-24T00:58:48.991 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1403.127200 secs 2024-03-24T00:58:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1403.127200 secs 2024-03-24T00:58:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1403.127200 secs 2024-03-24T00:58:49.845 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2058 sec 2024-03-24T00:58:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:49 smithi078 ceph-mon[32429]: pgmap v1234: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:50.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:49 smithi107 ceph-mon[36485]: pgmap v1234: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:49 smithi106 ceph-mon[36505]: pgmap v1234: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:50.846 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2059 sec 2024-03-24T00:58:51.846 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2060 sec 2024-03-24T00:58:52.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:51 smithi078 ceph-mon[32429]: pgmap v1235: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:52.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:51 smithi107 ceph-mon[36485]: pgmap v1235: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:52.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:51 smithi106 ceph-mon[36505]: pgmap v1235: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:52.846 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2061 sec 2024-03-24T00:58:53.846 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2062 sec 2024-03-24T00:58:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:53 smithi078 ceph-mon[32429]: pgmap v1236: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:54.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1408.127290 secs 2024-03-24T00:58:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:53 smithi107 ceph-mon[36485]: pgmap v1236: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1408.127290 secs 2024-03-24T00:58:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:53 smithi106 ceph-mon[36505]: pgmap v1236: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1408.127290 secs 2024-03-24T00:58:54.846 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2063 sec 2024-03-24T00:58:55.846 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2064 sec 2024-03-24T00:58:56.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:55 smithi078 ceph-mon[32429]: pgmap v1237: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:56.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:55 smithi107 ceph-mon[36485]: pgmap v1237: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:56.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:55 smithi106 ceph-mon[36505]: pgmap v1237: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:58:56.846 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2065 sec 2024-03-24T00:58:57.846 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2066 sec 2024-03-24T00:58:58.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:57 smithi078 ceph-mon[32429]: pgmap v1238: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:58.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:57 smithi107 ceph-mon[36485]: pgmap v1238: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:58.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:57 smithi106 ceph-mon[36505]: pgmap v1238: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:58.846 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2067 sec 2024-03-24T00:58:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1413.127375 secs 2024-03-24T00:58:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1413.127375 secs 2024-03-24T00:58:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1413.127375 secs 2024-03-24T00:58:59.846 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2068 sec 2024-03-24T00:59:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:58:59 smithi078 ceph-mon[32429]: pgmap v1239: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:00.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:58:59 smithi107 ceph-mon[36485]: pgmap v1239: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:00.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:58:59 smithi106 ceph-mon[36505]: pgmap v1239: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:00.846 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2069 sec 2024-03-24T00:59:01.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:00 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:59:01.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:00 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:59:01.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:00 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:59:01.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:00 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:59:01.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:00 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:59:01.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:00 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:59:01.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:00 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:59:01.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:00 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:59:01.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:00 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:59:01.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:00 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:59:01.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:00 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:59:01.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:00 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T00:59:01.847 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2070 sec 2024-03-24T00:59:02.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:01 smithi078 ceph-mon[32429]: pgmap v1240: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:02.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:01 smithi078 ceph-mon[32429]: pgmap v1241: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T00:59:02.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:01 smithi078 ceph-mon[32429]: pgmap v1242: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:59:02.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:01 smithi107 ceph-mon[36485]: pgmap v1240: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:02.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:01 smithi107 ceph-mon[36485]: pgmap v1241: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T00:59:02.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:01 smithi107 ceph-mon[36485]: pgmap v1242: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:59:02.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:01 smithi106 ceph-mon[36505]: pgmap v1240: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:02.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:01 smithi106 ceph-mon[36505]: pgmap v1241: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T00:59:02.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:01 smithi106 ceph-mon[36505]: pgmap v1242: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:59:02.846 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2071 sec 2024-03-24T00:59:03.846 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2072 sec 2024-03-24T00:59:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:03 smithi078 ceph-mon[32429]: pgmap v1243: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:59:04.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1418.127459 secs 2024-03-24T00:59:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:03 smithi107 ceph-mon[36485]: pgmap v1243: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:59:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1418.127459 secs 2024-03-24T00:59:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:03 smithi106 ceph-mon[36505]: pgmap v1243: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:59:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1418.127459 secs 2024-03-24T00:59:04.847 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2073 sec 2024-03-24T00:59:05.847 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2074 sec 2024-03-24T00:59:06.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:05 smithi078 ceph-mon[32429]: pgmap v1244: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:59:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:05 smithi107 ceph-mon[36485]: pgmap v1244: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:59:06.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:05 smithi106 ceph-mon[36505]: pgmap v1244: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T00:59:06.847 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2075 sec 2024-03-24T00:59:07.847 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2076 sec 2024-03-24T00:59:08.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:07 smithi078 ceph-mon[32429]: pgmap v1245: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:59:08.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:07 smithi107 ceph-mon[36485]: pgmap v1245: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:59:08.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:07 smithi106 ceph-mon[36505]: pgmap v1245: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:59:08.847 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2077 sec 2024-03-24T00:59:09.053 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1423.127525 secs 2024-03-24T00:59:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1423.127525 secs 2024-03-24T00:59:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1423.127525 secs 2024-03-24T00:59:09.847 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2078 sec 2024-03-24T00:59:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:09 smithi078 ceph-mon[32429]: pgmap v1246: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:59:10.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:09 smithi107 ceph-mon[36485]: pgmap v1246: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:59:10.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:09 smithi106 ceph-mon[36505]: pgmap v1246: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T00:59:10.847 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2079 sec 2024-03-24T00:59:11.847 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2080 sec 2024-03-24T00:59:12.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:11 smithi078 ceph-mon[32429]: pgmap v1247: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:59:12.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:11 smithi107 ceph-mon[36485]: pgmap v1247: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:59:12.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:11 smithi106 ceph-mon[36505]: pgmap v1247: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T00:59:12.847 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2081 sec 2024-03-24T00:59:13.848 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2082 sec 2024-03-24T00:59:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:13 smithi078 ceph-mon[32429]: pgmap v1248: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:14.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1428.127592 secs 2024-03-24T00:59:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:13 smithi107 ceph-mon[36485]: pgmap v1248: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1428.127592 secs 2024-03-24T00:59:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:13 smithi106 ceph-mon[36505]: pgmap v1248: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1428.127592 secs 2024-03-24T00:59:14.848 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2083 sec 2024-03-24T00:59:15.848 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2084 sec 2024-03-24T00:59:16.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:15 smithi078 ceph-mon[32429]: pgmap v1249: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:16.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:15 smithi107 ceph-mon[36485]: pgmap v1249: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:16.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:15 smithi106 ceph-mon[36505]: pgmap v1249: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:16.848 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2085 sec 2024-03-24T00:59:17.848 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2086 sec 2024-03-24T00:59:18.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:17 smithi078 ceph-mon[32429]: pgmap v1250: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:18.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:17 smithi107 ceph-mon[36485]: pgmap v1250: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:18.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:17 smithi106 ceph-mon[36505]: pgmap v1250: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:18.848 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2087 sec 2024-03-24T00:59:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1433.127607 secs 2024-03-24T00:59:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1433.127607 secs 2024-03-24T00:59:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1433.127607 secs 2024-03-24T00:59:19.848 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2088 sec 2024-03-24T00:59:20.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:19 smithi078 ceph-mon[32429]: pgmap v1251: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:20.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:19 smithi107 ceph-mon[36485]: pgmap v1251: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:20.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:19 smithi106 ceph-mon[36505]: pgmap v1251: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:20.848 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2089 sec 2024-03-24T00:59:21.848 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2090 sec 2024-03-24T00:59:22.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:21 smithi078 ceph-mon[32429]: pgmap v1252: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:22.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:21 smithi107 ceph-mon[36485]: pgmap v1252: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:22.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:21 smithi106 ceph-mon[36505]: pgmap v1252: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:22.848 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2091 sec 2024-03-24T00:59:23.848 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2092 sec 2024-03-24T00:59:24.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:23 smithi078 ceph-mon[32429]: pgmap v1253: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:24.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1438.127692 secs 2024-03-24T00:59:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:23 smithi107 ceph-mon[36485]: pgmap v1253: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1438.127692 secs 2024-03-24T00:59:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:23 smithi106 ceph-mon[36505]: pgmap v1253: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1438.127692 secs 2024-03-24T00:59:24.848 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2093 sec 2024-03-24T00:59:25.848 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2094 sec 2024-03-24T00:59:26.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:25 smithi078 ceph-mon[32429]: pgmap v1254: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:26.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:25 smithi107 ceph-mon[36485]: pgmap v1254: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:26.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:25 smithi106 ceph-mon[36505]: pgmap v1254: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:26.849 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2095 sec 2024-03-24T00:59:27.849 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2096 sec 2024-03-24T00:59:28.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:27 smithi078 ceph-mon[32429]: pgmap v1255: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:27 smithi107 ceph-mon[36485]: pgmap v1255: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:28.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:27 smithi106 ceph-mon[36505]: pgmap v1255: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:28.849 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2097 sec 2024-03-24T00:59:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:28 smithi078 ceph-mon[32429]: 2 slow requests, 1 included below; oldest blocked for > 1443.127753 secs 2024-03-24T00:59:29.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:28 smithi078 ceph-mon[32429]: slow request 1443.127752 seconds old, received at 2024-03-24T00:35:24.775244+0000: peer_request:client.15462:3775 currently acquired locks 2024-03-24T00:59:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:28 smithi107 ceph-mon[36485]: 2 slow requests, 1 included below; oldest blocked for > 1443.127753 secs 2024-03-24T00:59:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:28 smithi107 ceph-mon[36485]: slow request 1443.127752 seconds old, received at 2024-03-24T00:35:24.775244+0000: peer_request:client.15462:3775 currently acquired locks 2024-03-24T00:59:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:28 smithi106 ceph-mon[36505]: 2 slow requests, 1 included below; oldest blocked for > 1443.127753 secs 2024-03-24T00:59:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:28 smithi106 ceph-mon[36505]: slow request 1443.127752 seconds old, received at 2024-03-24T00:35:24.775244+0000: peer_request:client.15462:3775 currently acquired locks 2024-03-24T00:59:29.849 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2098 sec 2024-03-24T00:59:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:29 smithi078 ceph-mon[32429]: pgmap v1256: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:30.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:29 smithi107 ceph-mon[36485]: pgmap v1256: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:30.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:29 smithi106 ceph-mon[36505]: pgmap v1256: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:30.849 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2099 sec 2024-03-24T00:59:31.849 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2100 sec 2024-03-24T00:59:32.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:31 smithi078 ceph-mon[32429]: pgmap v1257: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:32.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:31 smithi107 ceph-mon[36485]: pgmap v1257: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:32.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:31 smithi106 ceph-mon[36505]: pgmap v1257: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:32.849 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2101 sec 2024-03-24T00:59:33.849 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2102 sec 2024-03-24T00:59:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:33 smithi078 ceph-mon[32429]: pgmap v1258: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:34.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1448.127812 secs 2024-03-24T00:59:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:33 smithi107 ceph-mon[36485]: pgmap v1258: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1448.127812 secs 2024-03-24T00:59:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:33 smithi106 ceph-mon[36505]: pgmap v1258: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1448.127812 secs 2024-03-24T00:59:34.849 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2103 sec 2024-03-24T00:59:35.849 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2104 sec 2024-03-24T00:59:36.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:35 smithi078 ceph-mon[32429]: pgmap v1259: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:36.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:35 smithi107 ceph-mon[36485]: pgmap v1259: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:36.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:35 smithi106 ceph-mon[36505]: pgmap v1259: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:36.850 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2105 sec 2024-03-24T00:59:37.850 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2106 sec 2024-03-24T00:59:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:37 smithi078 ceph-mon[32429]: pgmap v1260: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:38.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:37 smithi107 ceph-mon[36485]: pgmap v1260: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:38.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:37 smithi106 ceph-mon[36505]: pgmap v1260: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:38.850 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2107 sec 2024-03-24T00:59:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1453.127870 secs 2024-03-24T00:59:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1453.127870 secs 2024-03-24T00:59:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1453.127870 secs 2024-03-24T00:59:39.850 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2108 sec 2024-03-24T00:59:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:39 smithi078 ceph-mon[32429]: pgmap v1261: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:39 smithi107 ceph-mon[36485]: pgmap v1261: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:39 smithi106 ceph-mon[36505]: pgmap v1261: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:40.850 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2109 sec 2024-03-24T00:59:41.850 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2110 sec 2024-03-24T00:59:42.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:41 smithi078 ceph-mon[32429]: pgmap v1262: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:42.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:41 smithi107 ceph-mon[36485]: pgmap v1262: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:42.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:41 smithi106 ceph-mon[36505]: pgmap v1262: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:42.850 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2111 sec 2024-03-24T00:59:43.850 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2112 sec 2024-03-24T00:59:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:43 smithi078 ceph-mon[32429]: pgmap v1263: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:44.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1458.127959 secs 2024-03-24T00:59:44.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:43 smithi107 ceph-mon[36485]: pgmap v1263: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:44.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1458.127959 secs 2024-03-24T00:59:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:43 smithi106 ceph-mon[36505]: pgmap v1263: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1458.127959 secs 2024-03-24T00:59:44.850 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2113 sec 2024-03-24T00:59:45.850 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2114 sec 2024-03-24T00:59:46.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:45 smithi078 ceph-mon[32429]: pgmap v1264: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:45 smithi107 ceph-mon[36485]: pgmap v1264: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:46.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:45 smithi106 ceph-mon[36505]: pgmap v1264: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:46.851 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2115 sec 2024-03-24T00:59:47.851 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2116 sec 2024-03-24T00:59:48.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:47 smithi078 ceph-mon[32429]: pgmap v1265: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:48.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:47 smithi107 ceph-mon[36485]: pgmap v1265: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:48.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:47 smithi106 ceph-mon[36505]: pgmap v1265: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:48.851 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2117 sec 2024-03-24T00:59:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1463.128048 secs 2024-03-24T00:59:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1463.128048 secs 2024-03-24T00:59:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1463.128048 secs 2024-03-24T00:59:49.851 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2118 sec 2024-03-24T00:59:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:49 smithi078 ceph-mon[32429]: pgmap v1266: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:50.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:49 smithi107 ceph-mon[36485]: pgmap v1266: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:49 smithi106 ceph-mon[36505]: pgmap v1266: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:50.851 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2119 sec 2024-03-24T00:59:51.852 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2120 sec 2024-03-24T00:59:52.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:51 smithi078 ceph-mon[32429]: pgmap v1267: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:52.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:51 smithi107 ceph-mon[36485]: pgmap v1267: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:52.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:51 smithi106 ceph-mon[36505]: pgmap v1267: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:52.851 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2121 sec 2024-03-24T00:59:53.851 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2122 sec 2024-03-24T00:59:54.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:53 smithi078 ceph-mon[32429]: pgmap v1268: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:54.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1468.128134 secs 2024-03-24T00:59:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:53 smithi107 ceph-mon[36485]: pgmap v1268: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1468.128134 secs 2024-03-24T00:59:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:53 smithi106 ceph-mon[36505]: pgmap v1268: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1468.128134 secs 2024-03-24T00:59:54.851 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2123 sec 2024-03-24T00:59:55.851 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2124 sec 2024-03-24T00:59:56.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:55 smithi078 ceph-mon[32429]: pgmap v1269: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:56.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:55 smithi107 ceph-mon[36485]: pgmap v1269: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:56.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:55 smithi106 ceph-mon[36505]: pgmap v1269: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:56.851 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2125 sec 2024-03-24T00:59:57.852 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2126 sec 2024-03-24T00:59:58.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:57 smithi078 ceph-mon[32429]: pgmap v1270: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:58.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:57 smithi107 ceph-mon[36485]: pgmap v1270: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:58.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:57 smithi106 ceph-mon[36505]: pgmap v1270: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:58.851 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2127 sec 2024-03-24T00:59:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1473.128173 secs 2024-03-24T00:59:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 00:59:58 smithi106 ceph-mon[36505]: pgmap v1271: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1473.128173 secs 2024-03-24T00:59:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 00:59:58 smithi078 ceph-mon[32429]: pgmap v1271: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1473.128173 secs 2024-03-24T00:59:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 00:59:58 smithi107 ceph-mon[36485]: pgmap v1271: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T00:59:59.852 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2128 sec 2024-03-24T01:00:00.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:00 smithi078 ceph-mon[32429]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:00:00.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:00 smithi107 ceph-mon[36485]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:00:00.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:00 smithi106 ceph-mon[36505]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:00:00.852 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2129 sec 2024-03-24T01:00:01.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:01 smithi078 ceph-mon[32429]: pgmap v1272: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:01.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:01 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:00:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:01 smithi107 ceph-mon[36485]: pgmap v1272: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:01 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:00:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:01 smithi106 ceph-mon[36505]: pgmap v1272: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:01 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:00:01.852 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2130 sec 2024-03-24T01:00:02.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:02 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:00:02.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:02 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:00:02.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:02 smithi078 ceph-mon[32429]: pgmap v1273: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:00:02.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:02 smithi078 ceph-mon[32429]: pgmap v1274: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:00:02.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:02 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:00:02.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:02 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:00:02.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:02 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:00:02.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:02 smithi107 ceph-mon[36485]: pgmap v1273: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:00:02.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:02 smithi107 ceph-mon[36485]: pgmap v1274: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:00:02.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:02 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:00:02.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:02 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:00:02.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:02 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:00:02.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:02 smithi106 ceph-mon[36505]: pgmap v1273: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:00:02.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:02 smithi106 ceph-mon[36505]: pgmap v1274: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:00:02.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:02 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:00:02.852 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2131 sec 2024-03-24T01:00:03.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:03 smithi106 ceph-mon[36505]: 2 slow requests, 1 included below; oldest blocked for > 1478.128259 secs 2024-03-24T01:00:03.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:03 smithi106 ceph-mon[36505]: slow request 1442.861188 seconds old, received at 2024-03-24T00:36:00.042315+0000: client_request(client.25385:166 mksnap #0x1//scheduled-2024-03-24-00_36_00_UTC 2024-03-24T00:36:00.042488+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-24T01:00:03.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:03 smithi078 ceph-mon[32429]: 2 slow requests, 1 included below; oldest blocked for > 1478.128259 secs 2024-03-24T01:00:03.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:03 smithi078 ceph-mon[32429]: slow request 1442.861188 seconds old, received at 2024-03-24T00:36:00.042315+0000: client_request(client.25385:166 mksnap #0x1//scheduled-2024-03-24-00_36_00_UTC 2024-03-24T00:36:00.042488+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-24T01:00:03.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:03 smithi107 ceph-mon[36485]: 2 slow requests, 1 included below; oldest blocked for > 1478.128259 secs 2024-03-24T01:00:03.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:03 smithi107 ceph-mon[36485]: slow request 1442.861188 seconds old, received at 2024-03-24T00:36:00.042315+0000: client_request(client.25385:166 mksnap #0x1//scheduled-2024-03-24-00_36_00_UTC 2024-03-24T00:36:00.042488+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-24T01:00:03.852 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2132 sec 2024-03-24T01:00:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:04 smithi106 ceph-mon[36505]: pgmap v1275: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:00:04.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:04 smithi078 ceph-mon[32429]: pgmap v1275: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:00:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:04 smithi107 ceph-mon[36485]: pgmap v1275: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:00:04.852 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2133 sec 2024-03-24T01:00:05.852 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2134 sec 2024-03-24T01:00:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:06 smithi078 ceph-mon[32429]: pgmap v1276: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:00:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:06 smithi107 ceph-mon[36485]: pgmap v1276: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:00:06.852 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2135 sec 2024-03-24T01:00:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:06 smithi106 ceph-mon[36505]: pgmap v1276: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:00:07.852 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2136 sec 2024-03-24T01:00:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:08 smithi078 ceph-mon[32429]: pgmap v1277: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:00:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1483.128321 secs 2024-03-24T01:00:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:08 smithi107 ceph-mon[36485]: pgmap v1277: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:00:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1483.128321 secs 2024-03-24T01:00:08.852 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2137 sec 2024-03-24T01:00:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:08 smithi106 ceph-mon[36505]: pgmap v1277: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:00:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1483.128321 secs 2024-03-24T01:00:09.853 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2138 sec 2024-03-24T01:00:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:10 smithi078 ceph-mon[32429]: pgmap v1278: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:00:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:10 smithi107 ceph-mon[36485]: pgmap v1278: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:00:10.853 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2139 sec 2024-03-24T01:00:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:10 smithi106 ceph-mon[36505]: pgmap v1278: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:00:11.853 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2140 sec 2024-03-24T01:00:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:12 smithi078 ceph-mon[32429]: pgmap v1279: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:00:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:12 smithi107 ceph-mon[36485]: pgmap v1279: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:00:12.853 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2141 sec 2024-03-24T01:00:12.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:12 smithi106 ceph-mon[36505]: pgmap v1279: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:00:13.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1488.128332 secs 2024-03-24T01:00:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1488.128332 secs 2024-03-24T01:00:13.853 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2142 sec 2024-03-24T01:00:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1488.128332 secs 2024-03-24T01:00:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:14 smithi078 ceph-mon[32429]: pgmap v1280: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:14 smithi107 ceph-mon[36485]: pgmap v1280: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:14.853 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2143 sec 2024-03-24T01:00:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:14 smithi106 ceph-mon[36505]: pgmap v1280: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:15.853 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2144 sec 2024-03-24T01:00:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:16 smithi078 ceph-mon[32429]: pgmap v1281: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:16 smithi107 ceph-mon[36485]: pgmap v1281: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:16.853 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2145 sec 2024-03-24T01:00:16.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:16 smithi106 ceph-mon[36505]: pgmap v1281: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:17.853 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2146 sec 2024-03-24T01:00:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:18 smithi078 ceph-mon[32429]: pgmap v1282: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:18.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1493.128348 secs 2024-03-24T01:00:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:18 smithi107 ceph-mon[36485]: pgmap v1282: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1493.128348 secs 2024-03-24T01:00:18.853 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2147 sec 2024-03-24T01:00:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:18 smithi106 ceph-mon[36505]: pgmap v1282: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:18.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1493.128348 secs 2024-03-24T01:00:19.853 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2148 sec 2024-03-24T01:00:20.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:20 smithi078 ceph-mon[32429]: pgmap v1283: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:20 smithi107 ceph-mon[36485]: pgmap v1283: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:20.854 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2149 sec 2024-03-24T01:00:20.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:20 smithi106 ceph-mon[36505]: pgmap v1283: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:21.854 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2150 sec 2024-03-24T01:00:22.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:22 smithi078 ceph-mon[32429]: pgmap v1284: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:22.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:22 smithi107 ceph-mon[36485]: pgmap v1284: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:22.854 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2151 sec 2024-03-24T01:00:22.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:22 smithi106 ceph-mon[36505]: pgmap v1284: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:23.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1498.128407 secs 2024-03-24T01:00:23.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1498.128407 secs 2024-03-24T01:00:23.854 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2152 sec 2024-03-24T01:00:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1498.128407 secs 2024-03-24T01:00:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:24 smithi078 ceph-mon[32429]: pgmap v1285: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:24 smithi107 ceph-mon[36485]: pgmap v1285: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:24.854 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2153 sec 2024-03-24T01:00:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:24 smithi106 ceph-mon[36505]: pgmap v1285: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:25.854 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2154 sec 2024-03-24T01:00:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:26 smithi078 ceph-mon[32429]: pgmap v1286: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:26 smithi107 ceph-mon[36485]: pgmap v1286: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:26.854 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2155 sec 2024-03-24T01:00:26.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:26 smithi106 ceph-mon[36505]: pgmap v1286: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:27.854 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2156 sec 2024-03-24T01:00:28.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:28 smithi078 ceph-mon[32429]: pgmap v1287: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:28.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1503.128459 secs 2024-03-24T01:00:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:28 smithi107 ceph-mon[36485]: pgmap v1287: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1503.128459 secs 2024-03-24T01:00:28.854 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2157 sec 2024-03-24T01:00:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:28 smithi106 ceph-mon[36505]: pgmap v1287: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1503.128459 secs 2024-03-24T01:00:29.854 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2158 sec 2024-03-24T01:00:30.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:30 smithi078 ceph-mon[32429]: pgmap v1288: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:30 smithi107 ceph-mon[36485]: pgmap v1288: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:30.854 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2159 sec 2024-03-24T01:00:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:30 smithi106 ceph-mon[36505]: pgmap v1288: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:31.855 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2160 sec 2024-03-24T01:00:32.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:32 smithi078 ceph-mon[32429]: pgmap v1289: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:32 smithi107 ceph-mon[36485]: pgmap v1289: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:32.855 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2161 sec 2024-03-24T01:00:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:32 smithi106 ceph-mon[36505]: pgmap v1289: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:33.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1508.128519 secs 2024-03-24T01:00:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1508.128519 secs 2024-03-24T01:00:33.857 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2162 sec 2024-03-24T01:00:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1508.128519 secs 2024-03-24T01:00:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:34 smithi078 ceph-mon[32429]: pgmap v1290: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:34 smithi107 ceph-mon[36485]: pgmap v1290: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:34.855 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2163 sec 2024-03-24T01:00:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:34 smithi106 ceph-mon[36505]: pgmap v1290: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:35.855 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2164 sec 2024-03-24T01:00:36.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:36 smithi078 ceph-mon[32429]: pgmap v1291: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:36 smithi107 ceph-mon[36485]: pgmap v1291: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:36.855 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2165 sec 2024-03-24T01:00:36.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:36 smithi106 ceph-mon[36505]: pgmap v1291: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:37.855 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2166 sec 2024-03-24T01:00:38.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:38 smithi078 ceph-mon[32429]: pgmap v1292: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:38.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1513.128580 secs 2024-03-24T01:00:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:38 smithi107 ceph-mon[36485]: pgmap v1292: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1513.128580 secs 2024-03-24T01:00:38.855 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2167 sec 2024-03-24T01:00:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:38 smithi106 ceph-mon[36505]: pgmap v1292: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1513.128580 secs 2024-03-24T01:00:39.855 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2168 sec 2024-03-24T01:00:40.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:40 smithi078 ceph-mon[32429]: pgmap v1293: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:40.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:40 smithi107 ceph-mon[36485]: pgmap v1293: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:40.855 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2169 sec 2024-03-24T01:00:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:40 smithi106 ceph-mon[36505]: pgmap v1293: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:41.855 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2170 sec 2024-03-24T01:00:42.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:42 smithi078 ceph-mon[32429]: pgmap v1294: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:42.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:42 smithi107 ceph-mon[36485]: pgmap v1294: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:42.855 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2171 sec 2024-03-24T01:00:42.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:42 smithi106 ceph-mon[36505]: pgmap v1294: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:43.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1518.128642 secs 2024-03-24T01:00:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1518.128642 secs 2024-03-24T01:00:43.855 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2172 sec 2024-03-24T01:00:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1518.128642 secs 2024-03-24T01:00:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:44 smithi078 ceph-mon[32429]: pgmap v1295: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:44 smithi107 ceph-mon[36485]: pgmap v1295: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:44.855 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2173 sec 2024-03-24T01:00:44.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:44 smithi106 ceph-mon[36505]: pgmap v1295: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:45.856 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2174 sec 2024-03-24T01:00:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:46 smithi078 ceph-mon[32429]: pgmap v1296: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:46.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:46 smithi107 ceph-mon[36485]: pgmap v1296: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:46.856 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2175 sec 2024-03-24T01:00:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:46 smithi106 ceph-mon[36505]: pgmap v1296: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:47.856 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2176 sec 2024-03-24T01:00:48.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:48 smithi078 ceph-mon[32429]: pgmap v1297: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:48.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1523.128703 secs 2024-03-24T01:00:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:48 smithi107 ceph-mon[36485]: pgmap v1297: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1523.128703 secs 2024-03-24T01:00:48.856 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2177 sec 2024-03-24T01:00:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:48 smithi106 ceph-mon[36505]: pgmap v1297: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:48.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1523.128703 secs 2024-03-24T01:00:49.856 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2178 sec 2024-03-24T01:00:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:50 smithi078 ceph-mon[32429]: pgmap v1298: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:50 smithi107 ceph-mon[36485]: pgmap v1298: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:50.856 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2179 sec 2024-03-24T01:00:50.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:50 smithi106 ceph-mon[36505]: pgmap v1298: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:51.856 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2180 sec 2024-03-24T01:00:52.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:52 smithi078 ceph-mon[32429]: pgmap v1299: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:52.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:52 smithi107 ceph-mon[36485]: pgmap v1299: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:52.856 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2181 sec 2024-03-24T01:00:52.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:52 smithi106 ceph-mon[36505]: pgmap v1299: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:53.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1528.128764 secs 2024-03-24T01:00:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1528.128764 secs 2024-03-24T01:00:53.856 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2182 sec 2024-03-24T01:00:53.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1528.128764 secs 2024-03-24T01:00:54.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:54 smithi078 ceph-mon[32429]: pgmap v1300: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:54 smithi107 ceph-mon[36485]: pgmap v1300: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:54.856 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2183 sec 2024-03-24T01:00:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:54 smithi106 ceph-mon[36505]: pgmap v1300: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:55.856 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2184 sec 2024-03-24T01:00:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:56 smithi078 ceph-mon[32429]: pgmap v1301: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:56 smithi107 ceph-mon[36485]: pgmap v1301: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:56.856 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2185 sec 2024-03-24T01:00:56.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:56 smithi106 ceph-mon[36505]: pgmap v1301: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:57.857 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2186 sec 2024-03-24T01:00:58.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:58 smithi078 ceph-mon[32429]: pgmap v1302: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:58.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:00:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1533.128852 secs 2024-03-24T01:00:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:58 smithi107 ceph-mon[36485]: pgmap v1302: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:00:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1533.128852 secs 2024-03-24T01:00:58.857 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2187 sec 2024-03-24T01:00:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:58 smithi106 ceph-mon[36505]: pgmap v1302: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:00:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:00:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1533.128852 secs 2024-03-24T01:00:59.857 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2188 sec 2024-03-24T01:01:00.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:00 smithi078 ceph-mon[32429]: pgmap v1303: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:01:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:00 smithi107 ceph-mon[36485]: pgmap v1303: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:01:00.857 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2189 sec 2024-03-24T01:01:00.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:00 smithi106 ceph-mon[36505]: pgmap v1303: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:01:01.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:01 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:01:01.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:01 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:01:01.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:01 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:01:01.857 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2190 sec 2024-03-24T01:01:02.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:02 smithi078 ceph-mon[32429]: pgmap v1304: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:02.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:02 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:01:02.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:02 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:01:02.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:02 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:01:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:02 smithi107 ceph-mon[36485]: pgmap v1304: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:02 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:01:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:02 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:01:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:02 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:01:02.857 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2191 sec 2024-03-24T01:01:02.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:02 smithi106 ceph-mon[36505]: pgmap v1304: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:02.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:02 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:01:02.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:02 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:01:02.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:02 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:01:03.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:03 smithi078 ceph-mon[32429]: pgmap v1305: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:01:03.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:03 smithi078 ceph-mon[32429]: pgmap v1306: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:01:03.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1538.128943 secs 2024-03-24T01:01:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:03 smithi107 ceph-mon[36485]: pgmap v1305: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:01:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:03 smithi107 ceph-mon[36485]: pgmap v1306: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:01:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1538.128943 secs 2024-03-24T01:01:03.857 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2192 sec 2024-03-24T01:01:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:03 smithi106 ceph-mon[36505]: pgmap v1305: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:01:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:03 smithi106 ceph-mon[36505]: pgmap v1306: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:01:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1538.128943 secs 2024-03-24T01:01:04.857 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2193 sec 2024-03-24T01:01:05.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:05 smithi078 ceph-mon[32429]: pgmap v1307: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:01:05.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:05 smithi107 ceph-mon[36485]: pgmap v1307: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:01:05.857 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2194 sec 2024-03-24T01:01:05.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:05 smithi106 ceph-mon[36505]: pgmap v1307: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:01:06.857 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2195 sec 2024-03-24T01:01:07.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:07 smithi078 ceph-mon[32429]: pgmap v1308: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:01:07.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:07 smithi107 ceph-mon[36485]: pgmap v1308: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:01:07.857 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2196 sec 2024-03-24T01:01:07.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:07 smithi106 ceph-mon[36505]: pgmap v1308: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:01:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1543.129028 secs 2024-03-24T01:01:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1543.129028 secs 2024-03-24T01:01:08.858 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2197 sec 2024-03-24T01:01:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1543.129028 secs 2024-03-24T01:01:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:09 smithi078 ceph-mon[32429]: pgmap v1309: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:01:09.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:09 smithi107 ceph-mon[36485]: pgmap v1309: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:01:09.858 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2198 sec 2024-03-24T01:01:09.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:09 smithi106 ceph-mon[36505]: pgmap v1309: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:01:10.859 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2199 sec 2024-03-24T01:01:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:11 smithi107 ceph-mon[36485]: pgmap v1310: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:01:11.858 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2200 sec 2024-03-24T01:01:11.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:11 smithi106 ceph-mon[36505]: pgmap v1310: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:01:12.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:11 smithi078 ceph-mon[32429]: pgmap v1310: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:01:12.858 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2201 sec 2024-03-24T01:01:13.858 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2202 sec 2024-03-24T01:01:13.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:13 smithi078 ceph-mon[32429]: pgmap v1311: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:01:13.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1548.129065 secs 2024-03-24T01:01:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:13 smithi106 ceph-mon[36505]: pgmap v1311: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:01:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1548.129065 secs 2024-03-24T01:01:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:13 smithi107 ceph-mon[36485]: pgmap v1311: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:01:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1548.129065 secs 2024-03-24T01:01:14.862 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2203 sec 2024-03-24T01:01:15.858 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2204 sec 2024-03-24T01:01:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:15 smithi106 ceph-mon[36505]: pgmap v1312: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:01:16.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:15 smithi078 ceph-mon[32429]: pgmap v1312: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:01:16.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:15 smithi107 ceph-mon[36485]: pgmap v1312: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:01:16.859 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2205 sec 2024-03-24T01:01:17.859 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2206 sec 2024-03-24T01:01:17.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:17 smithi106 ceph-mon[36505]: pgmap v1313: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:18.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:17 smithi078 ceph-mon[32429]: pgmap v1313: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:18.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:17 smithi107 ceph-mon[36485]: pgmap v1313: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:18.859 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2207 sec 2024-03-24T01:01:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1553.129153 secs 2024-03-24T01:01:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1553.129153 secs 2024-03-24T01:01:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1553.129153 secs 2024-03-24T01:01:19.859 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2208 sec 2024-03-24T01:01:19.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:19 smithi106 ceph-mon[36505]: pgmap v1314: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:01:20.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:19 smithi078 ceph-mon[32429]: pgmap v1314: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:01:20.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:19 smithi107 ceph-mon[36485]: pgmap v1314: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:01:20.859 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2209 sec 2024-03-24T01:01:21.859 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2210 sec 2024-03-24T01:01:21.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:21 smithi106 ceph-mon[36505]: pgmap v1315: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:01:22.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:21 smithi078 ceph-mon[32429]: pgmap v1315: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:01:22.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:21 smithi107 ceph-mon[36485]: pgmap v1315: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:01:22.859 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2211 sec 2024-03-24T01:01:23.859 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2212 sec 2024-03-24T01:01:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:23 smithi106 ceph-mon[36505]: pgmap v1316: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1558.129244 secs 2024-03-24T01:01:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:23 smithi078 ceph-mon[32429]: pgmap v1316: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:24.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1558.129244 secs 2024-03-24T01:01:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:23 smithi107 ceph-mon[36485]: pgmap v1316: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1558.129244 secs 2024-03-24T01:01:24.859 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2213 sec 2024-03-24T01:01:25.859 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2214 sec 2024-03-24T01:01:25.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:25 smithi106 ceph-mon[36505]: pgmap v1317: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:01:26.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:25 smithi078 ceph-mon[32429]: pgmap v1317: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:01:26.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:25 smithi107 ceph-mon[36485]: pgmap v1317: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:01:26.860 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2215 sec 2024-03-24T01:01:27.859 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2216 sec 2024-03-24T01:01:27.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:27 smithi106 ceph-mon[36505]: pgmap v1318: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:28.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:27 smithi078 ceph-mon[32429]: pgmap v1318: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:27 smithi107 ceph-mon[36485]: pgmap v1318: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:28.860 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2217 sec 2024-03-24T01:01:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1563.129289 secs 2024-03-24T01:01:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1563.129289 secs 2024-03-24T01:01:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1563.129289 secs 2024-03-24T01:01:29.860 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2218 sec 2024-03-24T01:01:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:29 smithi106 ceph-mon[36505]: pgmap v1319: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:01:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:29 smithi078 ceph-mon[32429]: pgmap v1319: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:01:30.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:29 smithi107 ceph-mon[36485]: pgmap v1319: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:01:30.860 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2219 sec 2024-03-24T01:01:31.860 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2220 sec 2024-03-24T01:01:31.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:31 smithi106 ceph-mon[36505]: pgmap v1320: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:01:32.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:31 smithi078 ceph-mon[32429]: pgmap v1320: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:01:32.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:31 smithi107 ceph-mon[36485]: pgmap v1320: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:01:32.860 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2221 sec 2024-03-24T01:01:33.860 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2222 sec 2024-03-24T01:01:33.896 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:33 smithi107 ceph-mon[36485]: pgmap v1321: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:33.896 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1568.129352 secs 2024-03-24T01:01:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:33 smithi106 ceph-mon[36505]: pgmap v1321: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1568.129352 secs 2024-03-24T01:01:34.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:33 smithi078 ceph-mon[32429]: pgmap v1321: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:34.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1568.129352 secs 2024-03-24T01:01:34.860 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2223 sec 2024-03-24T01:01:35.860 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2224 sec 2024-03-24T01:01:35.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:35 smithi106 ceph-mon[36505]: pgmap v1322: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:01:36.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:35 smithi078 ceph-mon[32429]: pgmap v1322: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:01:36.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:35 smithi107 ceph-mon[36485]: pgmap v1322: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:01:36.860 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2225 sec 2024-03-24T01:01:37.860 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2226 sec 2024-03-24T01:01:37.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:37 smithi106 ceph-mon[36505]: pgmap v1323: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:37 smithi078 ceph-mon[32429]: pgmap v1323: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:38.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:37 smithi107 ceph-mon[36485]: pgmap v1323: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:38.861 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2227 sec 2024-03-24T01:01:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1573.129413 secs 2024-03-24T01:01:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1573.129413 secs 2024-03-24T01:01:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1573.129413 secs 2024-03-24T01:01:39.861 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2228 sec 2024-03-24T01:01:39.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:39 smithi106 ceph-mon[36505]: pgmap v1324: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:01:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:39 smithi078 ceph-mon[32429]: pgmap v1324: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:01:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:39 smithi107 ceph-mon[36485]: pgmap v1324: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:01:40.861 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2229 sec 2024-03-24T01:01:41.861 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2230 sec 2024-03-24T01:01:41.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:41 smithi106 ceph-mon[36505]: pgmap v1325: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:01:42.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:41 smithi078 ceph-mon[32429]: pgmap v1325: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:01:42.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:41 smithi107 ceph-mon[36485]: pgmap v1325: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:01:42.861 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2231 sec 2024-03-24T01:01:43.861 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2232 sec 2024-03-24T01:01:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:43 smithi078 ceph-mon[32429]: pgmap v1326: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:01:44.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1578.129501 secs 2024-03-24T01:01:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:43 smithi107 ceph-mon[36485]: pgmap v1326: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:01:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1578.129501 secs 2024-03-24T01:01:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:43 smithi106 ceph-mon[36505]: pgmap v1326: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:01:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1578.129501 secs 2024-03-24T01:01:44.861 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2233 sec 2024-03-24T01:01:45.861 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2234 sec 2024-03-24T01:01:46.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:45 smithi078 ceph-mon[32429]: pgmap v1327: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:01:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:45 smithi107 ceph-mon[36485]: pgmap v1327: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:01:46.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:45 smithi106 ceph-mon[36505]: pgmap v1327: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:01:46.861 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2235 sec 2024-03-24T01:01:47.861 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2236 sec 2024-03-24T01:01:48.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:47 smithi078 ceph-mon[32429]: pgmap v1328: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:01:48.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:47 smithi107 ceph-mon[36485]: pgmap v1328: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:01:48.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:47 smithi106 ceph-mon[36505]: pgmap v1328: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:01:48.861 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2237 sec 2024-03-24T01:01:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1583.129563 secs 2024-03-24T01:01:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1583.129563 secs 2024-03-24T01:01:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1583.129563 secs 2024-03-24T01:01:49.862 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2238 sec 2024-03-24T01:01:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:49 smithi078 ceph-mon[32429]: pgmap v1329: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:01:50.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:49 smithi107 ceph-mon[36485]: pgmap v1329: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:01:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:49 smithi106 ceph-mon[36505]: pgmap v1329: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:01:50.861 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2239 sec 2024-03-24T01:01:51.862 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2240 sec 2024-03-24T01:01:52.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:51 smithi078 ceph-mon[32429]: pgmap v1330: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:01:52.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:51 smithi107 ceph-mon[36485]: pgmap v1330: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:01:52.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:51 smithi106 ceph-mon[36505]: pgmap v1330: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:01:52.862 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2241 sec 2024-03-24T01:01:53.862 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2242 sec 2024-03-24T01:01:54.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:53 smithi078 ceph-mon[32429]: pgmap v1331: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:01:54.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1588.129603 secs 2024-03-24T01:01:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:53 smithi107 ceph-mon[36485]: pgmap v1331: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:01:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1588.129603 secs 2024-03-24T01:01:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:53 smithi106 ceph-mon[36505]: pgmap v1331: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:01:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1588.129603 secs 2024-03-24T01:01:54.862 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2243 sec 2024-03-24T01:01:55.862 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2244 sec 2024-03-24T01:01:56.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:55 smithi078 ceph-mon[32429]: pgmap v1332: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:01:56.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:55 smithi107 ceph-mon[36485]: pgmap v1332: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:01:56.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:55 smithi106 ceph-mon[36505]: pgmap v1332: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:01:56.862 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2245 sec 2024-03-24T01:01:57.862 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2246 sec 2024-03-24T01:01:58.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:57 smithi078 ceph-mon[32429]: pgmap v1333: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:58.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:57 smithi107 ceph-mon[36485]: pgmap v1333: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:58.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:57 smithi106 ceph-mon[36505]: pgmap v1333: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:58.862 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2247 sec 2024-03-24T01:01:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1593.129665 secs 2024-03-24T01:01:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1593.129665 secs 2024-03-24T01:01:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1593.129665 secs 2024-03-24T01:01:59.862 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2248 sec 2024-03-24T01:02:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:01:59 smithi078 ceph-mon[32429]: pgmap v1334: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:00.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:01:59 smithi107 ceph-mon[36485]: pgmap v1334: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:00.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:01:59 smithi106 ceph-mon[36505]: pgmap v1334: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:00.862 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2249 sec 2024-03-24T01:02:01.863 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2250 sec 2024-03-24T01:02:02.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:01 smithi078 ceph-mon[32429]: pgmap v1335: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:02.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:01 smithi107 ceph-mon[36485]: pgmap v1335: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:02.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:01 smithi106 ceph-mon[36505]: pgmap v1335: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:02.862 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2251 sec 2024-03-24T01:02:03.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:02 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:02:03.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:02 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:02:03.148 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:02 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:02:03.863 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2252 sec 2024-03-24T01:02:03.884 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:03 smithi107 ceph-mon[36485]: pgmap v1336: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:03.885 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1598.129730 secs 2024-03-24T01:02:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:03 smithi078 ceph-mon[32429]: pgmap v1336: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:04.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1598.129730 secs 2024-03-24T01:02:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:03 smithi106 ceph-mon[36505]: pgmap v1336: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1598.129730 secs 2024-03-24T01:02:04.862 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2253 sec 2024-03-24T01:02:05.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:05 smithi078 ceph-mon[32429]: pgmap v1337: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:05.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:02:05.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:02:05.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:02:05.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:02:05.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:02:05.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:02:05.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:05.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:05.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:05.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:05.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:05.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:05.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:05.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:05.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:05 smithi107 ceph-mon[36485]: pgmap v1337: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:05.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:02:05.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:02:05.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:02:05.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:02:05.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:02:05.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:02:05.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:05.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:05.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:05.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:05.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:05.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:05.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:05.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:05.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:05 smithi106 ceph-mon[36505]: pgmap v1337: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:05.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:02:05.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:02:05.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:02:05.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:02:05.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:02:05.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:02:05.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:05.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:05.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:05.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:05.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:05.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:05.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:05.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:05.863 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2254 sec 2024-03-24T01:02:06.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:06 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T01:02:06.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:06 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T01:02:06.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:06 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T01:02:06.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:06 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T01:02:06.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:06 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:06.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:06 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:06.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:06 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:06.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:06 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:06.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:06 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T01:02:06.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:06 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T01:02:06.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:06 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:02:06.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:06 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:02:06.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:06 smithi078 ceph-mon[32429]: pgmap v1338: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T01:02:06.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:06 smithi078 ceph-mon[32429]: pgmap v1339: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:02:06.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:06 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:02:06.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:06 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T01:02:06.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:06 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T01:02:06.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:06 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T01:02:06.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:06 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T01:02:06.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:06 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:06.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:06 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:06.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:06 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:06.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:06 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:06.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:06 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T01:02:06.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:06 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T01:02:06.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:06 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:02:06.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:06 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:02:06.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:06 smithi107 ceph-mon[36485]: pgmap v1338: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T01:02:06.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:06 smithi107 ceph-mon[36485]: pgmap v1339: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:02:06.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:06 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:02:06.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:06 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T01:02:06.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:06 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T01:02:06.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:06 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T01:02:06.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:06 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T01:02:06.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:06 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:06.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:06 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:06.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:06 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:06.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:06 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:02:06.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:06 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T01:02:06.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:06 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T01:02:06.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:06 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:02:06.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:06 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:02:06.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:06 smithi106 ceph-mon[36505]: pgmap v1338: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T01:02:06.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:06 smithi106 ceph-mon[36505]: pgmap v1339: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:02:06.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:06 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:02:06.863 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2255 sec 2024-03-24T01:02:07.863 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2256 sec 2024-03-24T01:02:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:08 smithi078 ceph-mon[32429]: pgmap v1340: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:02:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1603.129820 secs 2024-03-24T01:02:08.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:08 smithi107 ceph-mon[36485]: pgmap v1340: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:02:08.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1603.129820 secs 2024-03-24T01:02:08.863 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2257 sec 2024-03-24T01:02:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:08 smithi106 ceph-mon[36505]: pgmap v1340: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:02:08.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1603.129820 secs 2024-03-24T01:02:09.863 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2258 sec 2024-03-24T01:02:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:10 smithi078 ceph-mon[32429]: pgmap v1341: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:02:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:10 smithi107 ceph-mon[36485]: pgmap v1341: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:02:10.863 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2259 sec 2024-03-24T01:02:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:10 smithi106 ceph-mon[36505]: pgmap v1341: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:02:11.863 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2260 sec 2024-03-24T01:02:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:12 smithi078 ceph-mon[32429]: pgmap v1342: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:02:12.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:12 smithi107 ceph-mon[36485]: pgmap v1342: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:02:12.863 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2261 sec 2024-03-24T01:02:12.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:12 smithi106 ceph-mon[36505]: pgmap v1342: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:02:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1608.129886 secs 2024-03-24T01:02:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1608.129886 secs 2024-03-24T01:02:13.863 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2262 sec 2024-03-24T01:02:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1608.129886 secs 2024-03-24T01:02:14.764 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:14 smithi078 ceph-mon[32429]: pgmap v1343: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:02:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:14 smithi107 ceph-mon[36485]: pgmap v1343: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:02:14.863 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2263 sec 2024-03-24T01:02:14.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:14 smithi106 ceph-mon[36505]: pgmap v1343: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:02:15.864 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2264 sec 2024-03-24T01:02:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:16 smithi078 ceph-mon[32429]: pgmap v1344: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:02:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:16 smithi107 ceph-mon[36485]: pgmap v1344: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:02:16.864 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2265 sec 2024-03-24T01:02:16.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:16 smithi106 ceph-mon[36505]: pgmap v1344: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:02:17.864 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2266 sec 2024-03-24T01:02:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:18 smithi078 ceph-mon[32429]: pgmap v1345: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:18.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1613.129973 secs 2024-03-24T01:02:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:18 smithi107 ceph-mon[36485]: pgmap v1345: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1613.129973 secs 2024-03-24T01:02:18.864 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2267 sec 2024-03-24T01:02:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:18 smithi106 ceph-mon[36505]: pgmap v1345: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1613.129973 secs 2024-03-24T01:02:19.864 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2268 sec 2024-03-24T01:02:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:20 smithi078 ceph-mon[32429]: pgmap v1346: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:20 smithi107 ceph-mon[36485]: pgmap v1346: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:20.864 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2269 sec 2024-03-24T01:02:20.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:20 smithi106 ceph-mon[36505]: pgmap v1346: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:21.864 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2270 sec 2024-03-24T01:02:22.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:22 smithi078 ceph-mon[32429]: pgmap v1347: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:22.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:22 smithi107 ceph-mon[36485]: pgmap v1347: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:22.864 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2271 sec 2024-03-24T01:02:22.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:22 smithi106 ceph-mon[36505]: pgmap v1347: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:23.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1618.130058 secs 2024-03-24T01:02:23.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1618.130058 secs 2024-03-24T01:02:23.864 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2272 sec 2024-03-24T01:02:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1618.130058 secs 2024-03-24T01:02:24.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:24 smithi078 ceph-mon[32429]: pgmap v1348: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:24 smithi107 ceph-mon[36485]: pgmap v1348: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:24.864 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2273 sec 2024-03-24T01:02:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:24 smithi106 ceph-mon[36505]: pgmap v1348: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:25.864 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2274 sec 2024-03-24T01:02:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:26 smithi078 ceph-mon[32429]: pgmap v1349: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:26 smithi107 ceph-mon[36485]: pgmap v1349: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:26.864 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2275 sec 2024-03-24T01:02:26.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:26 smithi106 ceph-mon[36505]: pgmap v1349: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:27.864 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2276 sec 2024-03-24T01:02:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:28 smithi078 ceph-mon[32429]: pgmap v1350: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:28.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1623.130149 secs 2024-03-24T01:02:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:28 smithi107 ceph-mon[36485]: pgmap v1350: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1623.130149 secs 2024-03-24T01:02:28.864 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2277 sec 2024-03-24T01:02:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:28 smithi106 ceph-mon[36505]: pgmap v1350: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1623.130149 secs 2024-03-24T01:02:29.865 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2278 sec 2024-03-24T01:02:30.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:30 smithi078 ceph-mon[32429]: pgmap v1351: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:30 smithi107 ceph-mon[36485]: pgmap v1351: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:30.865 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2279 sec 2024-03-24T01:02:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:30 smithi106 ceph-mon[36505]: pgmap v1351: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:31.865 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2280 sec 2024-03-24T01:02:32.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:32 smithi078 ceph-mon[32429]: pgmap v1352: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:32 smithi107 ceph-mon[36485]: pgmap v1352: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:32.865 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2281 sec 2024-03-24T01:02:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:32 smithi106 ceph-mon[36505]: pgmap v1352: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:33.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1628.130209 secs 2024-03-24T01:02:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1628.130209 secs 2024-03-24T01:02:33.865 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2282 sec 2024-03-24T01:02:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1628.130209 secs 2024-03-24T01:02:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:34 smithi078 ceph-mon[32429]: pgmap v1353: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:34 smithi107 ceph-mon[36485]: pgmap v1353: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:34.865 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2283 sec 2024-03-24T01:02:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:34 smithi106 ceph-mon[36505]: pgmap v1353: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:35.865 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2284 sec 2024-03-24T01:02:36.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:36 smithi078 ceph-mon[32429]: pgmap v1354: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:36 smithi107 ceph-mon[36485]: pgmap v1354: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:36.865 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2285 sec 2024-03-24T01:02:36.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:36 smithi106 ceph-mon[36505]: pgmap v1354: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:37.865 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2286 sec 2024-03-24T01:02:38.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:38 smithi078 ceph-mon[32429]: pgmap v1355: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:38.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1633.130297 secs 2024-03-24T01:02:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:38 smithi107 ceph-mon[36485]: pgmap v1355: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1633.130297 secs 2024-03-24T01:02:38.865 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2287 sec 2024-03-24T01:02:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:38 smithi106 ceph-mon[36505]: pgmap v1355: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1633.130297 secs 2024-03-24T01:02:39.866 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2288 sec 2024-03-24T01:02:40.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:40 smithi078 ceph-mon[32429]: pgmap v1356: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:40.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:40 smithi107 ceph-mon[36485]: pgmap v1356: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:40.865 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2289 sec 2024-03-24T01:02:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:40 smithi106 ceph-mon[36505]: pgmap v1356: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:41.866 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2290 sec 2024-03-24T01:02:42.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:42 smithi078 ceph-mon[32429]: pgmap v1357: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:42.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:42 smithi107 ceph-mon[36485]: pgmap v1357: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:42.866 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2291 sec 2024-03-24T01:02:42.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:42 smithi106 ceph-mon[36505]: pgmap v1357: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:43.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1638.130358 secs 2024-03-24T01:02:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1638.130358 secs 2024-03-24T01:02:43.866 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2292 sec 2024-03-24T01:02:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1638.130358 secs 2024-03-24T01:02:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:44 smithi078 ceph-mon[32429]: pgmap v1358: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:44 smithi107 ceph-mon[36485]: pgmap v1358: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:44.866 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2293 sec 2024-03-24T01:02:44.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:44 smithi106 ceph-mon[36505]: pgmap v1358: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:45.866 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2294 sec 2024-03-24T01:02:46.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:46 smithi078 ceph-mon[32429]: pgmap v1359: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:46.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:46 smithi107 ceph-mon[36485]: pgmap v1359: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:46.866 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2295 sec 2024-03-24T01:02:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:46 smithi106 ceph-mon[36505]: pgmap v1359: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:47.866 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2296 sec 2024-03-24T01:02:48.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:48 smithi078 ceph-mon[32429]: pgmap v1360: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:48.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1643.130373 secs 2024-03-24T01:02:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:48 smithi107 ceph-mon[36485]: pgmap v1360: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1643.130373 secs 2024-03-24T01:02:48.866 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2297 sec 2024-03-24T01:02:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:48 smithi106 ceph-mon[36505]: pgmap v1360: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1643.130373 secs 2024-03-24T01:02:49.866 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2298 sec 2024-03-24T01:02:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:50 smithi078 ceph-mon[32429]: pgmap v1361: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:50 smithi107 ceph-mon[36485]: pgmap v1361: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:50.866 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2299 sec 2024-03-24T01:02:50.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:50 smithi106 ceph-mon[36505]: pgmap v1361: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:51.867 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2300 sec 2024-03-24T01:02:52.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:52 smithi078 ceph-mon[32429]: pgmap v1362: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:52.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:52 smithi107 ceph-mon[36485]: pgmap v1362: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:52.866 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2301 sec 2024-03-24T01:02:52.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:52 smithi106 ceph-mon[36505]: pgmap v1362: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:53.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1648.130434 secs 2024-03-24T01:02:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1648.130434 secs 2024-03-24T01:02:53.867 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2302 sec 2024-03-24T01:02:53.906 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1648.130434 secs 2024-03-24T01:02:54.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:54 smithi078 ceph-mon[32429]: pgmap v1363: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:54 smithi107 ceph-mon[36485]: pgmap v1363: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:54.867 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2303 sec 2024-03-24T01:02:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:54 smithi106 ceph-mon[36505]: pgmap v1363: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:55.867 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2304 sec 2024-03-24T01:02:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:56 smithi078 ceph-mon[32429]: pgmap v1364: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:56 smithi107 ceph-mon[36485]: pgmap v1364: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:56.867 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2305 sec 2024-03-24T01:02:56.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:56 smithi106 ceph-mon[36505]: pgmap v1364: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:57.867 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2306 sec 2024-03-24T01:02:58.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:58 smithi107 ceph-mon[36485]: pgmap v1365: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:58.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:02:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1653.130492 secs 2024-03-24T01:02:58.867 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2307 sec 2024-03-24T01:02:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:58 smithi106 ceph-mon[36505]: pgmap v1365: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:02:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1653.130492 secs 2024-03-24T01:02:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:58 smithi078 ceph-mon[32429]: pgmap v1365: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:02:59.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:02:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1653.130492 secs 2024-03-24T01:02:59.867 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2308 sec 2024-03-24T01:03:00.867 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2309 sec 2024-03-24T01:03:00.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:00 smithi106 ceph-mon[36505]: pgmap v1366: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:01.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:00 smithi078 ceph-mon[32429]: pgmap v1366: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:01.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:00 smithi107 ceph-mon[36485]: pgmap v1366: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:01.867 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2310 sec 2024-03-24T01:03:02.877 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2311 sec 2024-03-24T01:03:02.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:02 smithi106 ceph-mon[36505]: pgmap v1367: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:03.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:02 smithi078 ceph-mon[32429]: pgmap v1367: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:03.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:02 smithi107 ceph-mon[36485]: pgmap v1367: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:03.867 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2312 sec 2024-03-24T01:03:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1658.130553 secs 2024-03-24T01:03:03.994 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1658.130553 secs 2024-03-24T01:03:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1658.130553 secs 2024-03-24T01:03:04.868 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2313 sec 2024-03-24T01:03:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:04 smithi106 ceph-mon[36505]: pgmap v1368: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:05.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:04 smithi078 ceph-mon[32429]: pgmap v1368: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:05.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:04 smithi107 ceph-mon[36485]: pgmap v1368: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:05.868 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2314 sec 2024-03-24T01:03:05.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:03:06.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:03:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:03:06.868 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2315 sec 2024-03-24T01:03:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:06 smithi106 ceph-mon[36505]: pgmap v1369: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:06 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:03:06.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:06 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:03:06.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:06 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:03:06.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:06 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:03:06.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:06 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:03:06.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:06 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:03:07.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:06 smithi078 ceph-mon[32429]: pgmap v1369: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:07.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:06 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:03:07.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:06 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:03:07.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:06 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:03:07.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:06 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:03:07.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:06 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:03:07.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:06 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:03:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:06 smithi107 ceph-mon[36485]: pgmap v1369: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:06 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:03:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:06 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:03:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:06 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:03:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:06 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:03:07.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:06 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:03:07.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:06 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:03:07.868 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2316 sec 2024-03-24T01:03:07.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:07 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:03:07.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:07 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:03:07.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:07 smithi106 ceph-mon[36505]: pgmap v1370: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T01:03:07.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:07 smithi106 ceph-mon[36505]: pgmap v1371: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:03:07.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:07 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:03:08.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:07 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:03:08.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:07 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:03:08.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:07 smithi078 ceph-mon[32429]: pgmap v1370: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T01:03:08.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:07 smithi078 ceph-mon[32429]: pgmap v1371: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:03:08.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:07 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:03:08.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:07 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:03:08.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:07 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:03:08.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:07 smithi107 ceph-mon[36485]: pgmap v1370: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T01:03:08.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:07 smithi107 ceph-mon[36485]: pgmap v1371: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:03:08.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:07 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:03:08.868 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2317 sec 2024-03-24T01:03:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1663.130614 secs 2024-03-24T01:03:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1663.130614 secs 2024-03-24T01:03:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1663.130614 secs 2024-03-24T01:03:09.868 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2318 sec 2024-03-24T01:03:09.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:09 smithi106 ceph-mon[36505]: pgmap v1372: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:03:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:09 smithi078 ceph-mon[32429]: pgmap v1372: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:03:10.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:09 smithi107 ceph-mon[36485]: pgmap v1372: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:03:10.868 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2319 sec 2024-03-24T01:03:11.868 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2320 sec 2024-03-24T01:03:12.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:11 smithi078 ceph-mon[32429]: pgmap v1373: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:03:12.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:11 smithi107 ceph-mon[36485]: pgmap v1373: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:03:12.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:11 smithi106 ceph-mon[36505]: pgmap v1373: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:03:12.868 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2321 sec 2024-03-24T01:03:13.868 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2322 sec 2024-03-24T01:03:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:13 smithi078 ceph-mon[32429]: pgmap v1374: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:03:14.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1668.130672 secs 2024-03-24T01:03:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:13 smithi107 ceph-mon[36485]: pgmap v1374: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:03:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1668.130672 secs 2024-03-24T01:03:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:13 smithi106 ceph-mon[36505]: pgmap v1374: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:03:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1668.130672 secs 2024-03-24T01:03:14.868 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2323 sec 2024-03-24T01:03:15.868 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2324 sec 2024-03-24T01:03:16.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:15 smithi078 ceph-mon[32429]: pgmap v1375: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 995 B/s rd, 2 op/s 2024-03-24T01:03:16.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:15 smithi107 ceph-mon[36485]: pgmap v1375: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 995 B/s rd, 2 op/s 2024-03-24T01:03:16.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:15 smithi106 ceph-mon[36505]: pgmap v1375: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 995 B/s rd, 2 op/s 2024-03-24T01:03:16.868 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2325 sec 2024-03-24T01:03:17.868 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2326 sec 2024-03-24T01:03:18.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:17 smithi078 ceph-mon[32429]: pgmap v1376: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:03:18.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:17 smithi107 ceph-mon[36485]: pgmap v1376: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:03:18.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:17 smithi106 ceph-mon[36505]: pgmap v1376: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:03:18.869 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2327 sec 2024-03-24T01:03:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1673.130734 secs 2024-03-24T01:03:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1673.130734 secs 2024-03-24T01:03:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1673.130734 secs 2024-03-24T01:03:19.869 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2328 sec 2024-03-24T01:03:20.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:19 smithi078 ceph-mon[32429]: pgmap v1377: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:03:20.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:19 smithi107 ceph-mon[36485]: pgmap v1377: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:03:20.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:19 smithi106 ceph-mon[36505]: pgmap v1377: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:03:20.869 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2329 sec 2024-03-24T01:03:21.869 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2330 sec 2024-03-24T01:03:22.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:21 smithi078 ceph-mon[32429]: pgmap v1378: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:03:22.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:21 smithi107 ceph-mon[36485]: pgmap v1378: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:03:22.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:21 smithi106 ceph-mon[36505]: pgmap v1378: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:03:22.869 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2331 sec 2024-03-24T01:03:23.869 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2332 sec 2024-03-24T01:03:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:23 smithi078 ceph-mon[32429]: pgmap v1379: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:24.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1678.130817 secs 2024-03-24T01:03:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:23 smithi107 ceph-mon[36485]: pgmap v1379: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1678.130817 secs 2024-03-24T01:03:24.155 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:23 smithi106 ceph-mon[36505]: pgmap v1379: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:24.155 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1678.130817 secs 2024-03-24T01:03:24.869 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2333 sec 2024-03-24T01:03:25.869 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2334 sec 2024-03-24T01:03:26.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:25 smithi078 ceph-mon[32429]: pgmap v1380: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:26.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:25 smithi107 ceph-mon[36485]: pgmap v1380: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:26.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:25 smithi106 ceph-mon[36505]: pgmap v1380: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:26.869 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2335 sec 2024-03-24T01:03:27.869 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2336 sec 2024-03-24T01:03:28.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:27 smithi078 ceph-mon[32429]: pgmap v1381: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:27 smithi107 ceph-mon[36485]: pgmap v1381: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:28.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:27 smithi106 ceph-mon[36505]: pgmap v1381: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:28.869 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2337 sec 2024-03-24T01:03:29.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1683.130860 secs 2024-03-24T01:03:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1683.130860 secs 2024-03-24T01:03:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1683.130860 secs 2024-03-24T01:03:29.870 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2338 sec 2024-03-24T01:03:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:29 smithi078 ceph-mon[32429]: pgmap v1382: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:30.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:29 smithi107 ceph-mon[36485]: pgmap v1382: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:30.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:29 smithi106 ceph-mon[36505]: pgmap v1382: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:30.870 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2339 sec 2024-03-24T01:03:31.870 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2340 sec 2024-03-24T01:03:32.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:31 smithi078 ceph-mon[32429]: pgmap v1383: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:32.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:31 smithi107 ceph-mon[36485]: pgmap v1383: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:32.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:31 smithi106 ceph-mon[36505]: pgmap v1383: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:32.870 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2341 sec 2024-03-24T01:03:33.870 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2342 sec 2024-03-24T01:03:34.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:33 smithi078 ceph-mon[32429]: pgmap v1384: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:34.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1688.130949 secs 2024-03-24T01:03:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:33 smithi107 ceph-mon[36485]: pgmap v1384: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1688.130949 secs 2024-03-24T01:03:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:33 smithi106 ceph-mon[36505]: pgmap v1384: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:34.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1688.130949 secs 2024-03-24T01:03:34.870 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2343 sec 2024-03-24T01:03:35.870 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2344 sec 2024-03-24T01:03:36.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:35 smithi078 ceph-mon[32429]: pgmap v1385: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:36.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:35 smithi107 ceph-mon[36485]: pgmap v1385: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:36.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:35 smithi106 ceph-mon[36505]: pgmap v1385: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:36.871 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2345 sec 2024-03-24T01:03:37.870 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2346 sec 2024-03-24T01:03:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:37 smithi078 ceph-mon[32429]: pgmap v1386: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:38.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:37 smithi107 ceph-mon[36485]: pgmap v1386: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:38.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:37 smithi106 ceph-mon[36505]: pgmap v1386: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:38.870 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2347 sec 2024-03-24T01:03:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1693.131036 secs 2024-03-24T01:03:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1693.131036 secs 2024-03-24T01:03:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1693.131036 secs 2024-03-24T01:03:39.870 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2348 sec 2024-03-24T01:03:40.093 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:39 smithi078 ceph-mon[32429]: pgmap v1387: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:40.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:39 smithi107 ceph-mon[36485]: pgmap v1387: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:39 smithi106 ceph-mon[36505]: pgmap v1387: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:40.870 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2349 sec 2024-03-24T01:03:41.871 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2350 sec 2024-03-24T01:03:42.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:41 smithi078 ceph-mon[32429]: pgmap v1388: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:42.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:41 smithi107 ceph-mon[36485]: pgmap v1388: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:42.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:41 smithi106 ceph-mon[36505]: pgmap v1388: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:42.871 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2351 sec 2024-03-24T01:03:43.871 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2352 sec 2024-03-24T01:03:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:43 smithi078 ceph-mon[32429]: pgmap v1389: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:44.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1698.131123 secs 2024-03-24T01:03:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:43 smithi107 ceph-mon[36485]: pgmap v1389: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1698.131123 secs 2024-03-24T01:03:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:43 smithi106 ceph-mon[36505]: pgmap v1389: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1698.131123 secs 2024-03-24T01:03:44.871 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2353 sec 2024-03-24T01:03:45.871 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2354 sec 2024-03-24T01:03:46.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:45 smithi078 ceph-mon[32429]: pgmap v1390: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:45 smithi107 ceph-mon[36485]: pgmap v1390: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:46.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:45 smithi106 ceph-mon[36505]: pgmap v1390: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:46.871 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2355 sec 2024-03-24T01:03:47.871 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2356 sec 2024-03-24T01:03:48.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:47 smithi078 ceph-mon[32429]: pgmap v1391: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:48.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:47 smithi107 ceph-mon[36485]: pgmap v1391: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:48.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:47 smithi106 ceph-mon[36505]: pgmap v1391: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:48.874 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2357 sec 2024-03-24T01:03:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1703.131165 secs 2024-03-24T01:03:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1703.131165 secs 2024-03-24T01:03:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1703.131165 secs 2024-03-24T01:03:49.871 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2358 sec 2024-03-24T01:03:50.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:49 smithi107 ceph-mon[36485]: pgmap v1392: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:49 smithi106 ceph-mon[36505]: pgmap v1392: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:50.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:49 smithi078 ceph-mon[32429]: pgmap v1392: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:50.872 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2359 sec 2024-03-24T01:03:51.871 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2360 sec 2024-03-24T01:03:52.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:51 smithi106 ceph-mon[36505]: pgmap v1393: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:51 smithi078 ceph-mon[32429]: pgmap v1393: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:52.345 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:51 smithi107 ceph-mon[36485]: pgmap v1393: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:52.871 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2361 sec 2024-03-24T01:03:53.872 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2362 sec 2024-03-24T01:03:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:53 smithi106 ceph-mon[36505]: pgmap v1394: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:54.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1708.131228 secs 2024-03-24T01:03:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:53 smithi078 ceph-mon[32429]: pgmap v1394: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1708.131228 secs 2024-03-24T01:03:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:53 smithi107 ceph-mon[36485]: pgmap v1394: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1708.131228 secs 2024-03-24T01:03:54.872 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2363 sec 2024-03-24T01:03:55.872 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2364 sec 2024-03-24T01:03:55.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:55 smithi106 ceph-mon[36505]: pgmap v1395: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:56.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:55 smithi078 ceph-mon[32429]: pgmap v1395: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:56.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:55 smithi107 ceph-mon[36485]: pgmap v1395: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:03:56.872 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2365 sec 2024-03-24T01:03:57.872 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2366 sec 2024-03-24T01:03:58.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:57 smithi078 ceph-mon[32429]: pgmap v1396: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:58.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:57 smithi107 ceph-mon[36485]: pgmap v1396: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:58.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:57 smithi106 ceph-mon[36505]: pgmap v1396: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:58.872 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2367 sec 2024-03-24T01:03:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1713.131269 secs 2024-03-24T01:03:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1713.131269 secs 2024-03-24T01:03:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1713.131269 secs 2024-03-24T01:03:59.872 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2368 sec 2024-03-24T01:04:00.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:03:59 smithi078 ceph-mon[32429]: pgmap v1397: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:00.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:03:59 smithi107 ceph-mon[36485]: pgmap v1397: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:00.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:03:59 smithi106 ceph-mon[36505]: pgmap v1397: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:00.872 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2369 sec 2024-03-24T01:04:01.872 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2370 sec 2024-03-24T01:04:02.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:01 smithi078 ceph-mon[32429]: pgmap v1398: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:02.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:01 smithi107 ceph-mon[36485]: pgmap v1398: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:02.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:01 smithi106 ceph-mon[36505]: pgmap v1398: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:02.872 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2371 sec 2024-03-24T01:04:03.873 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2372 sec 2024-03-24T01:04:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:03 smithi078 ceph-mon[32429]: pgmap v1399: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:04.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1718.131330 secs 2024-03-24T01:04:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:03 smithi107 ceph-mon[36485]: pgmap v1399: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1718.131330 secs 2024-03-24T01:04:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:03 smithi106 ceph-mon[36505]: pgmap v1399: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1718.131330 secs 2024-03-24T01:04:04.873 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2373 sec 2024-03-24T01:04:05.873 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2374 sec 2024-03-24T01:04:06.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:05 smithi078 ceph-mon[32429]: pgmap v1400: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:05 smithi107 ceph-mon[36485]: pgmap v1400: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:06.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:05 smithi106 ceph-mon[36505]: pgmap v1400: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:06.873 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2375 sec 2024-03-24T01:04:07.873 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2376 sec 2024-03-24T01:04:08.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:07 smithi078 ceph-mon[32429]: pgmap v1401: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:08.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:07 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:04:08.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:07 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:04:08.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:07 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:04:08.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:07 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:04:08.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:07 smithi107 ceph-mon[36485]: pgmap v1401: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:08.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:07 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:04:08.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:07 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:04:08.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:07 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:04:08.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:07 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:04:08.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:07 smithi106 ceph-mon[36505]: pgmap v1401: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:08.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:07 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:04:08.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:07 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:04:08.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:07 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:04:08.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:07 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:04:08.873 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2377 sec 2024-03-24T01:04:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:08 smithi078 ceph-mon[32429]: pgmap v1402: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:04:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:08 smithi078 ceph-mon[32429]: pgmap v1403: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:04:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1723.131393 secs 2024-03-24T01:04:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:08 smithi107 ceph-mon[36485]: pgmap v1402: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:04:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:08 smithi107 ceph-mon[36485]: pgmap v1403: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:04:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1723.131393 secs 2024-03-24T01:04:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:08 smithi106 ceph-mon[36505]: pgmap v1402: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:04:09.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:08 smithi106 ceph-mon[36505]: pgmap v1403: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:04:09.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1723.131393 secs 2024-03-24T01:04:09.873 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2378 sec 2024-03-24T01:04:10.873 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2379 sec 2024-03-24T01:04:11.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:10 smithi078 ceph-mon[32429]: pgmap v1404: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:04:11.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:10 smithi107 ceph-mon[36485]: pgmap v1404: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:04:11.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:10 smithi106 ceph-mon[36505]: pgmap v1404: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:04:11.873 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2380 sec 2024-03-24T01:04:12.873 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2381 sec 2024-03-24T01:04:13.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:12 smithi078 ceph-mon[32429]: pgmap v1405: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:04:13.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:12 smithi107 ceph-mon[36485]: pgmap v1405: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:04:13.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:12 smithi106 ceph-mon[36505]: pgmap v1405: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:04:13.873 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2382 sec 2024-03-24T01:04:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1728.131454 secs 2024-03-24T01:04:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1728.131454 secs 2024-03-24T01:04:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1728.131454 secs 2024-03-24T01:04:14.873 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2383 sec 2024-03-24T01:04:15.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:14 smithi078 ceph-mon[32429]: pgmap v1406: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:04:15.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:14 smithi107 ceph-mon[36485]: pgmap v1406: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:04:15.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:14 smithi106 ceph-mon[36505]: pgmap v1406: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:04:15.873 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2384 sec 2024-03-24T01:04:16.874 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2385 sec 2024-03-24T01:04:17.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:16 smithi078 ceph-mon[32429]: pgmap v1407: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:04:17.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:16 smithi107 ceph-mon[36485]: pgmap v1407: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:04:17.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:16 smithi106 ceph-mon[36505]: pgmap v1407: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:04:17.874 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2386 sec 2024-03-24T01:04:18.874 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2387 sec 2024-03-24T01:04:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:18 smithi078 ceph-mon[32429]: pgmap v1408: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:04:19.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1733.131517 secs 2024-03-24T01:04:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:18 smithi107 ceph-mon[36485]: pgmap v1408: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:04:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1733.131517 secs 2024-03-24T01:04:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:18 smithi106 ceph-mon[36505]: pgmap v1408: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:04:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1733.131517 secs 2024-03-24T01:04:19.874 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2388 sec 2024-03-24T01:04:20.874 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2389 sec 2024-03-24T01:04:21.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:20 smithi078 ceph-mon[32429]: pgmap v1409: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:21.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:20 smithi107 ceph-mon[36485]: pgmap v1409: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:21.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:20 smithi106 ceph-mon[36505]: pgmap v1409: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:21.874 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2390 sec 2024-03-24T01:04:22.874 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2391 sec 2024-03-24T01:04:23.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:22 smithi078 ceph-mon[32429]: pgmap v1410: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:23.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:22 smithi107 ceph-mon[36485]: pgmap v1410: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:23.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:22 smithi106 ceph-mon[36505]: pgmap v1410: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:23.874 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2392 sec 2024-03-24T01:04:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1738.131588 secs 2024-03-24T01:04:24.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1738.131588 secs 2024-03-24T01:04:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1738.131588 secs 2024-03-24T01:04:24.874 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2393 sec 2024-03-24T01:04:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:24 smithi107 ceph-mon[36485]: pgmap v1411: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:25.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:24 smithi106 ceph-mon[36505]: pgmap v1411: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:24 smithi078 ceph-mon[32429]: pgmap v1411: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:25.875 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2394 sec 2024-03-24T01:04:26.875 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2395 sec 2024-03-24T01:04:27.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:26 smithi106 ceph-mon[36505]: pgmap v1412: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:27.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:26 smithi078 ceph-mon[32429]: pgmap v1412: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:27.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:26 smithi107 ceph-mon[36485]: pgmap v1412: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:27.875 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2396 sec 2024-03-24T01:04:28.875 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2397 sec 2024-03-24T01:04:29.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:28 smithi106 ceph-mon[36505]: pgmap v1413: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:29.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1743.131677 secs 2024-03-24T01:04:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:28 smithi078 ceph-mon[32429]: pgmap v1413: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1743.131677 secs 2024-03-24T01:04:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:28 smithi107 ceph-mon[36485]: pgmap v1413: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1743.131677 secs 2024-03-24T01:04:29.875 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2398 sec 2024-03-24T01:04:30.875 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2399 sec 2024-03-24T01:04:31.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:30 smithi106 ceph-mon[36505]: pgmap v1414: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:31.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:30 smithi078 ceph-mon[32429]: pgmap v1414: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:31.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:30 smithi107 ceph-mon[36485]: pgmap v1414: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:31.875 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2400 sec 2024-03-24T01:04:32.875 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2401 sec 2024-03-24T01:04:33.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:32 smithi107 ceph-mon[36485]: pgmap v1415: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:33.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:32 smithi106 ceph-mon[36505]: pgmap v1415: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:33.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:32 smithi078 ceph-mon[32429]: pgmap v1415: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:33.875 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2402 sec 2024-03-24T01:04:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1748.131717 secs 2024-03-24T01:04:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1748.131717 secs 2024-03-24T01:04:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1748.131717 secs 2024-03-24T01:04:34.877 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2403 sec 2024-03-24T01:04:35.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:34 smithi106 ceph-mon[36505]: pgmap v1416: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:35.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:34 smithi078 ceph-mon[32429]: pgmap v1416: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:35.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:34 smithi107 ceph-mon[36485]: pgmap v1416: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:35.875 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2404 sec 2024-03-24T01:04:36.876 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2405 sec 2024-03-24T01:04:37.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:36 smithi106 ceph-mon[36505]: pgmap v1417: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:37.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:36 smithi078 ceph-mon[32429]: pgmap v1417: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:36 smithi107 ceph-mon[36485]: pgmap v1417: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:37.876 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2406 sec 2024-03-24T01:04:38.876 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2407 sec 2024-03-24T01:04:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:38 smithi106 ceph-mon[36505]: pgmap v1418: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1753.131778 secs 2024-03-24T01:04:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:38 smithi078 ceph-mon[32429]: pgmap v1418: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:39.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1753.131778 secs 2024-03-24T01:04:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:38 smithi107 ceph-mon[36485]: pgmap v1418: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1753.131778 secs 2024-03-24T01:04:39.876 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2408 sec 2024-03-24T01:04:40.876 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2409 sec 2024-03-24T01:04:41.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:40 smithi106 ceph-mon[36505]: pgmap v1419: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:41.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:40 smithi078 ceph-mon[32429]: pgmap v1419: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:40 smithi107 ceph-mon[36485]: pgmap v1419: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:41.876 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2410 sec 2024-03-24T01:04:42.876 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2411 sec 2024-03-24T01:04:43.141 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:42 smithi078 ceph-mon[32429]: pgmap v1420: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:43.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:42 smithi106 ceph-mon[36505]: pgmap v1420: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:43.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:42 smithi107 ceph-mon[36485]: pgmap v1420: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:43.876 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2412 sec 2024-03-24T01:04:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1758.131866 secs 2024-03-24T01:04:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1758.131866 secs 2024-03-24T01:04:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1758.131866 secs 2024-03-24T01:04:44.876 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2413 sec 2024-03-24T01:04:45.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:44 smithi106 ceph-mon[36505]: pgmap v1421: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:45.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:44 smithi078 ceph-mon[32429]: pgmap v1421: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:45.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:44 smithi107 ceph-mon[36485]: pgmap v1421: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:45.876 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2414 sec 2024-03-24T01:04:46.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:45 smithi078 ceph-mon[32429]: pgmap v1422: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:46.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:45 smithi107 ceph-mon[36485]: pgmap v1422: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:46.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:45 smithi106 ceph-mon[36505]: pgmap v1422: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:46.876 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2415 sec 2024-03-24T01:04:47.876 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2416 sec 2024-03-24T01:04:48.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:48 smithi078 ceph-mon[32429]: pgmap v1423: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:48.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1763.131967 secs 2024-03-24T01:04:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:48 smithi107 ceph-mon[36485]: pgmap v1423: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1763.131967 secs 2024-03-24T01:04:48.876 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2417 sec 2024-03-24T01:04:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:48 smithi106 ceph-mon[36505]: pgmap v1423: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1763.131967 secs 2024-03-24T01:04:49.876 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2418 sec 2024-03-24T01:04:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:50 smithi078 ceph-mon[32429]: pgmap v1424: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:50 smithi107 ceph-mon[36485]: pgmap v1424: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:50.876 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2419 sec 2024-03-24T01:04:50.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:50 smithi106 ceph-mon[36505]: pgmap v1424: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:51.877 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2420 sec 2024-03-24T01:04:52.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:52 smithi078 ceph-mon[32429]: pgmap v1425: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:52.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:52 smithi107 ceph-mon[36485]: pgmap v1425: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:52.877 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2421 sec 2024-03-24T01:04:52.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:52 smithi106 ceph-mon[36505]: pgmap v1425: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:53.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1768.132076 secs 2024-03-24T01:04:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1768.132076 secs 2024-03-24T01:04:53.877 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2422 sec 2024-03-24T01:04:53.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1768.132076 secs 2024-03-24T01:04:54.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:54 smithi078 ceph-mon[32429]: pgmap v1426: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:54 smithi107 ceph-mon[36485]: pgmap v1426: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:54.877 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2423 sec 2024-03-24T01:04:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:54 smithi106 ceph-mon[36505]: pgmap v1426: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:55.877 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2424 sec 2024-03-24T01:04:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:56 smithi078 ceph-mon[32429]: pgmap v1427: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:56 smithi107 ceph-mon[36485]: pgmap v1427: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:56.877 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2425 sec 2024-03-24T01:04:56.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:56 smithi106 ceph-mon[36505]: pgmap v1427: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:57.877 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2426 sec 2024-03-24T01:04:58.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:58 smithi078 ceph-mon[32429]: pgmap v1428: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:58.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:04:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1773.132161 secs 2024-03-24T01:04:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:58 smithi107 ceph-mon[36485]: pgmap v1428: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:04:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1773.132161 secs 2024-03-24T01:04:58.877 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2427 sec 2024-03-24T01:04:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:58 smithi106 ceph-mon[36505]: pgmap v1428: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:04:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:04:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1773.132161 secs 2024-03-24T01:04:59.877 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2428 sec 2024-03-24T01:05:00.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:00 smithi078 ceph-mon[32429]: pgmap v1429: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:00 smithi107 ceph-mon[36485]: pgmap v1429: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:00.877 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2429 sec 2024-03-24T01:05:00.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:00 smithi106 ceph-mon[36505]: pgmap v1429: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:01.877 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2430 sec 2024-03-24T01:05:02.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:02 smithi078 ceph-mon[32429]: pgmap v1430: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:02 smithi107 ceph-mon[36485]: pgmap v1430: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:02.877 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2431 sec 2024-03-24T01:05:02.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:02 smithi106 ceph-mon[36505]: pgmap v1430: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:03.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1778.132200 secs 2024-03-24T01:05:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1778.132200 secs 2024-03-24T01:05:03.878 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2432 sec 2024-03-24T01:05:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1778.132200 secs 2024-03-24T01:05:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:04 smithi078 ceph-mon[32429]: pgmap v1431: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:04 smithi107 ceph-mon[36485]: pgmap v1431: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:04.878 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2433 sec 2024-03-24T01:05:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:04 smithi106 ceph-mon[36505]: pgmap v1431: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:05.878 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2434 sec 2024-03-24T01:05:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:06 smithi078 ceph-mon[32429]: pgmap v1432: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:06 smithi107 ceph-mon[36485]: pgmap v1432: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:06.878 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2435 sec 2024-03-24T01:05:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:06 smithi106 ceph-mon[36505]: pgmap v1432: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:07.878 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2436 sec 2024-03-24T01:05:08.878 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2437 sec 2024-03-24T01:05:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:08 smithi106 ceph-mon[36505]: pgmap v1433: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:08 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:05:08.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1783.132269 secs 2024-03-24T01:05:08.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:08 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:05:08.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:08 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:05:08.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:08 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:05:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:08 smithi078 ceph-mon[32429]: pgmap v1433: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:08 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:05:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1783.132269 secs 2024-03-24T01:05:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:08 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:05:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:08 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:05:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:08 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:05:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:08 smithi107 ceph-mon[36485]: pgmap v1433: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:08 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:05:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1783.132269 secs 2024-03-24T01:05:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:08 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:05:09.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:08 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:05:09.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:08 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:05:09.879 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2438 sec 2024-03-24T01:05:09.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:09 smithi106 ceph-mon[36505]: pgmap v1434: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:05:09.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:09 smithi106 ceph-mon[36505]: pgmap v1435: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:05:10.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:09 smithi078 ceph-mon[32429]: pgmap v1434: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:05:10.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:09 smithi078 ceph-mon[32429]: pgmap v1435: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:05:10.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:09 smithi107 ceph-mon[36485]: pgmap v1434: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:05:10.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:09 smithi107 ceph-mon[36485]: pgmap v1435: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:05:10.878 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2439 sec 2024-03-24T01:05:11.879 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2440 sec 2024-03-24T01:05:11.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:11 smithi106 ceph-mon[36505]: pgmap v1436: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:05:12.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:11 smithi078 ceph-mon[32429]: pgmap v1436: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:05:12.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:11 smithi107 ceph-mon[36485]: pgmap v1436: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:05:12.878 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2441 sec 2024-03-24T01:05:13.879 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2442 sec 2024-03-24T01:05:13.893 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:13 smithi078 ceph-mon[32429]: pgmap v1437: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:05:13.893 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1788.132355 secs 2024-03-24T01:05:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:13 smithi106 ceph-mon[36505]: pgmap v1437: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:05:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1788.132355 secs 2024-03-24T01:05:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:13 smithi107 ceph-mon[36485]: pgmap v1437: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:05:14.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1788.132355 secs 2024-03-24T01:05:14.879 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2443 sec 2024-03-24T01:05:15.879 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2444 sec 2024-03-24T01:05:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:15 smithi106 ceph-mon[36505]: pgmap v1438: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:05:16.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:15 smithi078 ceph-mon[32429]: pgmap v1438: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:05:16.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:15 smithi107 ceph-mon[36485]: pgmap v1438: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:05:16.879 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2445 sec 2024-03-24T01:05:17.879 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2446 sec 2024-03-24T01:05:17.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:17 smithi106 ceph-mon[36505]: pgmap v1439: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:05:18.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:17 smithi078 ceph-mon[32429]: pgmap v1439: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:05:18.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:17 smithi107 ceph-mon[36485]: pgmap v1439: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:05:18.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1793.132416 secs 2024-03-24T01:05:18.879 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2447 sec 2024-03-24T01:05:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1793.132416 secs 2024-03-24T01:05:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1793.132416 secs 2024-03-24T01:05:19.879 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2448 sec 2024-03-24T01:05:19.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:19 smithi106 ceph-mon[36505]: pgmap v1440: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:05:20.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:19 smithi078 ceph-mon[32429]: pgmap v1440: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:05:20.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:19 smithi107 ceph-mon[36485]: pgmap v1440: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:05:20.879 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2449 sec 2024-03-24T01:05:21.879 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2450 sec 2024-03-24T01:05:21.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:21 smithi106 ceph-mon[36505]: pgmap v1441: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:22.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:21 smithi078 ceph-mon[32429]: pgmap v1441: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:22.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:21 smithi107 ceph-mon[36485]: pgmap v1441: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:22.879 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2451 sec 2024-03-24T01:05:23.879 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2452 sec 2024-03-24T01:05:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:23 smithi106 ceph-mon[36505]: pgmap v1442: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:23.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1798.132476 secs 2024-03-24T01:05:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:23 smithi078 ceph-mon[32429]: pgmap v1442: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:24.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1798.132476 secs 2024-03-24T01:05:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:23 smithi107 ceph-mon[36485]: pgmap v1442: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1798.132476 secs 2024-03-24T01:05:24.880 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2453 sec 2024-03-24T01:05:25.880 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2454 sec 2024-03-24T01:05:25.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:25 smithi106 ceph-mon[36505]: pgmap v1443: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:26.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:25 smithi078 ceph-mon[32429]: pgmap v1443: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:26.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:25 smithi107 ceph-mon[36485]: pgmap v1443: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:26.880 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2455 sec 2024-03-24T01:05:27.881 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2456 sec 2024-03-24T01:05:27.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:27 smithi106 ceph-mon[36505]: pgmap v1444: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:28.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:27 smithi078 ceph-mon[32429]: pgmap v1444: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:27 smithi107 ceph-mon[36485]: pgmap v1444: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:28.880 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2457 sec 2024-03-24T01:05:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1803.132494 secs 2024-03-24T01:05:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1803.132494 secs 2024-03-24T01:05:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1803.132494 secs 2024-03-24T01:05:29.880 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2458 sec 2024-03-24T01:05:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:29 smithi106 ceph-mon[36505]: pgmap v1445: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:30.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:29 smithi078 ceph-mon[32429]: pgmap v1445: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:30.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:29 smithi107 ceph-mon[36485]: pgmap v1445: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:30.880 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2459 sec 2024-03-24T01:05:31.880 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2460 sec 2024-03-24T01:05:31.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:31 smithi106 ceph-mon[36505]: pgmap v1446: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:32.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:31 smithi078 ceph-mon[32429]: pgmap v1446: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:32.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:31 smithi107 ceph-mon[36485]: pgmap v1446: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:32.880 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2461 sec 2024-03-24T01:05:33.880 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2462 sec 2024-03-24T01:05:33.893 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:33 smithi078 ceph-mon[32429]: pgmap v1447: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:33.893 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1808.132554 secs 2024-03-24T01:05:33.896 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:33 smithi107 ceph-mon[36485]: pgmap v1447: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:33.896 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1808.132554 secs 2024-03-24T01:05:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:33 smithi106 ceph-mon[36505]: pgmap v1447: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1808.132554 secs 2024-03-24T01:05:34.880 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2463 sec 2024-03-24T01:05:35.881 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2464 sec 2024-03-24T01:05:35.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:35 smithi106 ceph-mon[36505]: pgmap v1448: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:36.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:35 smithi078 ceph-mon[32429]: pgmap v1448: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:36.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:35 smithi107 ceph-mon[36485]: pgmap v1448: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:36.881 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2465 sec 2024-03-24T01:05:37.881 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2466 sec 2024-03-24T01:05:37.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:37 smithi106 ceph-mon[36505]: pgmap v1449: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:37 smithi078 ceph-mon[32429]: pgmap v1449: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:38.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:37 smithi107 ceph-mon[36485]: pgmap v1449: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:38.881 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2467 sec 2024-03-24T01:05:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1813.132616 secs 2024-03-24T01:05:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1813.132616 secs 2024-03-24T01:05:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1813.132616 secs 2024-03-24T01:05:39.881 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2468 sec 2024-03-24T01:05:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:39 smithi078 ceph-mon[32429]: pgmap v1450: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:39 smithi107 ceph-mon[36485]: pgmap v1450: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:39 smithi106 ceph-mon[36505]: pgmap v1450: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:40.881 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2469 sec 2024-03-24T01:05:41.881 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2470 sec 2024-03-24T01:05:42.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:41 smithi078 ceph-mon[32429]: pgmap v1451: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:42.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:41 smithi107 ceph-mon[36485]: pgmap v1451: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:42.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:41 smithi106 ceph-mon[36505]: pgmap v1451: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:42.881 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2471 sec 2024-03-24T01:05:43.881 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2472 sec 2024-03-24T01:05:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:43 smithi078 ceph-mon[32429]: pgmap v1452: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1818.132628 secs 2024-03-24T01:05:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:43 smithi107 ceph-mon[36485]: pgmap v1452: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1818.132628 secs 2024-03-24T01:05:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:43 smithi106 ceph-mon[36505]: pgmap v1452: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1818.132628 secs 2024-03-24T01:05:44.881 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2473 sec 2024-03-24T01:05:45.881 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2474 sec 2024-03-24T01:05:45.954 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:45 smithi106 ceph-mon[36505]: pgmap v1453: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:46.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:45 smithi078 ceph-mon[32429]: pgmap v1453: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:45 smithi107 ceph-mon[36485]: pgmap v1453: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:46.882 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2475 sec 2024-03-24T01:05:47.882 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2476 sec 2024-03-24T01:05:48.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:47 smithi078 ceph-mon[32429]: pgmap v1454: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:48.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:47 smithi107 ceph-mon[36485]: pgmap v1454: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:48.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:47 smithi106 ceph-mon[36505]: pgmap v1454: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:48.882 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2477 sec 2024-03-24T01:05:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1823.132716 secs 2024-03-24T01:05:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1823.132716 secs 2024-03-24T01:05:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1823.132716 secs 2024-03-24T01:05:49.882 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2478 sec 2024-03-24T01:05:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:49 smithi078 ceph-mon[32429]: pgmap v1455: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:50.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:49 smithi107 ceph-mon[36485]: pgmap v1455: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:49 smithi106 ceph-mon[36505]: pgmap v1455: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:50.882 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2479 sec 2024-03-24T01:05:51.882 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2480 sec 2024-03-24T01:05:52.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:51 smithi078 ceph-mon[32429]: pgmap v1456: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:52.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:51 smithi107 ceph-mon[36485]: pgmap v1456: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:52.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:51 smithi106 ceph-mon[36505]: pgmap v1456: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:52.882 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2481 sec 2024-03-24T01:05:53.882 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2482 sec 2024-03-24T01:05:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:53 smithi078 ceph-mon[32429]: pgmap v1457: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1828.132758 secs 2024-03-24T01:05:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:53 smithi107 ceph-mon[36485]: pgmap v1457: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1828.132758 secs 2024-03-24T01:05:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:53 smithi106 ceph-mon[36505]: pgmap v1457: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1828.132758 secs 2024-03-24T01:05:54.882 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2483 sec 2024-03-24T01:05:55.882 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2484 sec 2024-03-24T01:05:56.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:55 smithi078 ceph-mon[32429]: pgmap v1458: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:56.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:55 smithi107 ceph-mon[36485]: pgmap v1458: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:56.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:55 smithi106 ceph-mon[36505]: pgmap v1458: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:05:56.882 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2485 sec 2024-03-24T01:05:57.882 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2486 sec 2024-03-24T01:05:58.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:57 smithi078 ceph-mon[32429]: pgmap v1459: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:58.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:57 smithi107 ceph-mon[36485]: pgmap v1459: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:58.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:57 smithi106 ceph-mon[36505]: pgmap v1459: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:58.883 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2487 sec 2024-03-24T01:05:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1833.132819 secs 2024-03-24T01:05:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1833.132819 secs 2024-03-24T01:05:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1833.132819 secs 2024-03-24T01:05:59.883 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2488 sec 2024-03-24T01:06:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:05:59 smithi078 ceph-mon[32429]: pgmap v1460: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:00.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:05:59 smithi107 ceph-mon[36485]: pgmap v1460: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:00.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:05:59 smithi106 ceph-mon[36505]: pgmap v1460: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:00.883 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2489 sec 2024-03-24T01:06:01.883 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2490 sec 2024-03-24T01:06:02.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:01 smithi078 ceph-mon[32429]: pgmap v1461: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:02.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:01 smithi107 ceph-mon[36485]: pgmap v1461: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:02.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:01 smithi106 ceph-mon[36505]: pgmap v1461: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:02.883 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2491 sec 2024-03-24T01:06:03.883 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2492 sec 2024-03-24T01:06:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:03 smithi078 ceph-mon[32429]: pgmap v1462: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:04.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1838.132910 secs 2024-03-24T01:06:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:03 smithi107 ceph-mon[36485]: pgmap v1462: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1838.132910 secs 2024-03-24T01:06:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:03 smithi106 ceph-mon[36505]: pgmap v1462: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1838.132910 secs 2024-03-24T01:06:04.883 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2493 sec 2024-03-24T01:06:05.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:05 smithi078 ceph-mon[32429]: pgmap v1463: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:05.883 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2494 sec 2024-03-24T01:06:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:05 smithi107 ceph-mon[36485]: pgmap v1463: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:06.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:05 smithi106 ceph-mon[36505]: pgmap v1463: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:06.883 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2495 sec 2024-03-24T01:06:07.883 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2496 sec 2024-03-24T01:06:08.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:07 smithi078 ceph-mon[32429]: pgmap v1464: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:08.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:07 smithi107 ceph-mon[36485]: pgmap v1464: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:08.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:07 smithi106 ceph-mon[36505]: pgmap v1464: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:08.884 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2497 sec 2024-03-24T01:06:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1843.132998 secs 2024-03-24T01:06:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:08 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:06:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1843.132998 secs 2024-03-24T01:06:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:08 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:06:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1843.132998 secs 2024-03-24T01:06:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:08 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:06:09.883 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2498 sec 2024-03-24T01:06:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:09 smithi078 ceph-mon[32429]: pgmap v1465: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:10.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:06:10.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:06:10.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:06:10.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:09 smithi107 ceph-mon[36485]: pgmap v1465: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:10.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:06:10.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:06:10.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:06:10.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:09 smithi106 ceph-mon[36505]: pgmap v1465: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:10.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:06:10.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:06:10.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:06:10.884 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2499 sec 2024-03-24T01:06:11.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:10 smithi078 ceph-mon[32429]: pgmap v1466: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:06:11.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:10 smithi078 ceph-mon[32429]: pgmap v1467: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:06:11.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:10 smithi107 ceph-mon[36485]: pgmap v1466: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:06:11.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:10 smithi107 ceph-mon[36485]: pgmap v1467: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:06:11.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:10 smithi106 ceph-mon[36505]: pgmap v1466: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:06:11.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:10 smithi106 ceph-mon[36505]: pgmap v1467: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:06:11.884 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2500 sec 2024-03-24T01:06:12.884 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2501 sec 2024-03-24T01:06:13.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:12 smithi078 ceph-mon[32429]: pgmap v1468: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:06:13.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:12 smithi107 ceph-mon[36485]: pgmap v1468: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:06:13.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:12 smithi106 ceph-mon[36505]: pgmap v1468: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:06:13.884 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2502 sec 2024-03-24T01:06:13.992 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1848.133059 secs 2024-03-24T01:06:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1848.133059 secs 2024-03-24T01:06:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1848.133059 secs 2024-03-24T01:06:14.884 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2503 sec 2024-03-24T01:06:15.053 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:14 smithi106 ceph-mon[36505]: pgmap v1469: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:06:15.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:14 smithi078 ceph-mon[32429]: pgmap v1469: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:06:15.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:14 smithi107 ceph-mon[36485]: pgmap v1469: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:06:15.884 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2504 sec 2024-03-24T01:06:16.884 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2505 sec 2024-03-24T01:06:17.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:16 smithi078 ceph-mon[32429]: pgmap v1470: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:06:17.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:16 smithi107 ceph-mon[36485]: pgmap v1470: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:06:17.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:16 smithi106 ceph-mon[36505]: pgmap v1470: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:06:17.884 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2506 sec 2024-03-24T01:06:18.884 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2507 sec 2024-03-24T01:06:19.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:18 smithi078 ceph-mon[32429]: pgmap v1471: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:06:19.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1853.133121 secs 2024-03-24T01:06:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:18 smithi107 ceph-mon[36485]: pgmap v1471: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:06:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1853.133121 secs 2024-03-24T01:06:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:18 smithi106 ceph-mon[36505]: pgmap v1471: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:06:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1853.133121 secs 2024-03-24T01:06:19.884 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2508 sec 2024-03-24T01:06:20.884 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2509 sec 2024-03-24T01:06:21.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:20 smithi078 ceph-mon[32429]: pgmap v1472: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:06:21.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:20 smithi107 ceph-mon[36485]: pgmap v1472: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:06:21.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:20 smithi106 ceph-mon[36505]: pgmap v1472: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:06:21.885 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2510 sec 2024-03-24T01:06:22.885 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2511 sec 2024-03-24T01:06:23.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:22 smithi078 ceph-mon[32429]: pgmap v1473: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:23.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:22 smithi107 ceph-mon[36485]: pgmap v1473: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:23.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:22 smithi106 ceph-mon[36505]: pgmap v1473: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:23.885 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2512 sec 2024-03-24T01:06:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1858.133179 secs 2024-03-24T01:06:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1858.133179 secs 2024-03-24T01:06:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1858.133179 secs 2024-03-24T01:06:24.885 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2513 sec 2024-03-24T01:06:25.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:24 smithi078 ceph-mon[32429]: pgmap v1474: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:24 smithi107 ceph-mon[36485]: pgmap v1474: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:25.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:24 smithi106 ceph-mon[36505]: pgmap v1474: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:25.885 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2514 sec 2024-03-24T01:06:26.885 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2515 sec 2024-03-24T01:06:27.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:26 smithi078 ceph-mon[32429]: pgmap v1475: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:27.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:26 smithi107 ceph-mon[36485]: pgmap v1475: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:27.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:26 smithi106 ceph-mon[36505]: pgmap v1475: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:27.885 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2516 sec 2024-03-24T01:06:28.885 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2517 sec 2024-03-24T01:06:29.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:28 smithi078 ceph-mon[32429]: pgmap v1476: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:29.093 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1863.133267 secs 2024-03-24T01:06:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:28 smithi107 ceph-mon[36485]: pgmap v1476: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1863.133267 secs 2024-03-24T01:06:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:28 smithi106 ceph-mon[36505]: pgmap v1476: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:29.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1863.133267 secs 2024-03-24T01:06:29.885 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2518 sec 2024-03-24T01:06:30.886 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2519 sec 2024-03-24T01:06:31.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:30 smithi078 ceph-mon[32429]: pgmap v1477: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:31.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:30 smithi107 ceph-mon[36485]: pgmap v1477: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:31.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:30 smithi106 ceph-mon[36505]: pgmap v1477: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:31.886 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2520 sec 2024-03-24T01:06:32.886 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2521 sec 2024-03-24T01:06:33.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:32 smithi078 ceph-mon[32429]: pgmap v1478: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:33.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:32 smithi107 ceph-mon[36485]: pgmap v1478: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:33.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:32 smithi106 ceph-mon[36505]: pgmap v1478: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:33.886 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2522 sec 2024-03-24T01:06:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1868.133353 secs 2024-03-24T01:06:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1868.133353 secs 2024-03-24T01:06:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1868.133353 secs 2024-03-24T01:06:34.886 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2523 sec 2024-03-24T01:06:35.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:34 smithi078 ceph-mon[32429]: pgmap v1479: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:35.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:34 smithi107 ceph-mon[36485]: pgmap v1479: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:35.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:34 smithi106 ceph-mon[36505]: pgmap v1479: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:35.886 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2524 sec 2024-03-24T01:06:36.886 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2525 sec 2024-03-24T01:06:37.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:36 smithi078 ceph-mon[32429]: pgmap v1480: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:37.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:36 smithi107 ceph-mon[36485]: pgmap v1480: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:37.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:36 smithi106 ceph-mon[36505]: pgmap v1480: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:37.886 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2526 sec 2024-03-24T01:06:38.886 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2527 sec 2024-03-24T01:06:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:38 smithi078 ceph-mon[32429]: pgmap v1481: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:39.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1873.133372 secs 2024-03-24T01:06:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:38 smithi107 ceph-mon[36485]: pgmap v1481: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1873.133372 secs 2024-03-24T01:06:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:38 smithi106 ceph-mon[36505]: pgmap v1481: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1873.133372 secs 2024-03-24T01:06:39.886 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2528 sec 2024-03-24T01:06:40.886 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2529 sec 2024-03-24T01:06:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:40 smithi078 ceph-mon[32429]: pgmap v1482: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:41.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:40 smithi107 ceph-mon[36485]: pgmap v1482: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:41.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:40 smithi106 ceph-mon[36505]: pgmap v1482: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:41.886 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2530 sec 2024-03-24T01:06:42.886 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2531 sec 2024-03-24T01:06:43.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:42 smithi078 ceph-mon[32429]: pgmap v1483: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:43.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:42 smithi107 ceph-mon[36485]: pgmap v1483: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:43.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:42 smithi106 ceph-mon[36505]: pgmap v1483: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:43.887 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2532 sec 2024-03-24T01:06:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1878.133434 secs 2024-03-24T01:06:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1878.133434 secs 2024-03-24T01:06:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1878.133434 secs 2024-03-24T01:06:44.887 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2533 sec 2024-03-24T01:06:45.053 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:44 smithi106 ceph-mon[36505]: pgmap v1484: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:45.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:44 smithi078 ceph-mon[32429]: pgmap v1484: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:45.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:44 smithi107 ceph-mon[36485]: pgmap v1484: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:45.887 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2534 sec 2024-03-24T01:06:46.887 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2535 sec 2024-03-24T01:06:47.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:46 smithi078 ceph-mon[32429]: pgmap v1485: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:47.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:46 smithi107 ceph-mon[36485]: pgmap v1485: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:47.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:46 smithi106 ceph-mon[36505]: pgmap v1485: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:47.887 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2536 sec 2024-03-24T01:06:48.887 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2537 sec 2024-03-24T01:06:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:48 smithi078 ceph-mon[32429]: pgmap v1486: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:49.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1883.133448 secs 2024-03-24T01:06:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:48 smithi107 ceph-mon[36485]: pgmap v1486: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1883.133448 secs 2024-03-24T01:06:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:48 smithi106 ceph-mon[36505]: pgmap v1486: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1883.133448 secs 2024-03-24T01:06:49.887 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2538 sec 2024-03-24T01:06:50.887 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2539 sec 2024-03-24T01:06:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:50 smithi078 ceph-mon[32429]: pgmap v1487: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:51.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:50 smithi107 ceph-mon[36485]: pgmap v1487: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:51.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:50 smithi106 ceph-mon[36505]: pgmap v1487: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:51.887 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2540 sec 2024-03-24T01:06:52.887 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2541 sec 2024-03-24T01:06:53.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:52 smithi078 ceph-mon[32429]: pgmap v1488: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:53.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:52 smithi107 ceph-mon[36485]: pgmap v1488: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:53.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:52 smithi106 ceph-mon[36505]: pgmap v1488: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:53.887 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2542 sec 2024-03-24T01:06:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1888.133510 secs 2024-03-24T01:06:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1888.133510 secs 2024-03-24T01:06:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1888.133510 secs 2024-03-24T01:06:54.888 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2543 sec 2024-03-24T01:06:55.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:54 smithi078 ceph-mon[32429]: pgmap v1489: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:55.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:54 smithi107 ceph-mon[36485]: pgmap v1489: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:55.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:54 smithi106 ceph-mon[36505]: pgmap v1489: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:55.888 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2544 sec 2024-03-24T01:06:56.888 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2545 sec 2024-03-24T01:06:57.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:56 smithi078 ceph-mon[32429]: pgmap v1490: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:56 smithi107 ceph-mon[36485]: pgmap v1490: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:57.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:56 smithi106 ceph-mon[36505]: pgmap v1490: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:06:57.888 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2546 sec 2024-03-24T01:06:58.888 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2547 sec 2024-03-24T01:06:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:58 smithi078 ceph-mon[32429]: pgmap v1491: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:06:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1893.133570 secs 2024-03-24T01:06:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:58 smithi107 ceph-mon[36485]: pgmap v1491: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:06:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1893.133570 secs 2024-03-24T01:06:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:58 smithi106 ceph-mon[36505]: pgmap v1491: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:06:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1893.133570 secs 2024-03-24T01:06:59.888 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2548 sec 2024-03-24T01:07:00.888 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2549 sec 2024-03-24T01:07:01.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:00 smithi078 ceph-mon[32429]: pgmap v1492: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:01.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:00 smithi107 ceph-mon[36485]: pgmap v1492: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:01.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:00 smithi106 ceph-mon[36505]: pgmap v1492: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:01.888 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2550 sec 2024-03-24T01:07:02.888 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2551 sec 2024-03-24T01:07:03.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:02 smithi078 ceph-mon[32429]: pgmap v1493: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:03.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:02 smithi107 ceph-mon[36485]: pgmap v1493: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:03.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:02 smithi106 ceph-mon[36505]: pgmap v1493: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:03.888 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2552 sec 2024-03-24T01:07:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1898.133656 secs 2024-03-24T01:07:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1898.133656 secs 2024-03-24T01:07:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1898.133656 secs 2024-03-24T01:07:04.889 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2553 sec 2024-03-24T01:07:05.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:04 smithi078 ceph-mon[32429]: pgmap v1494: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:05.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:04 smithi107 ceph-mon[36485]: pgmap v1494: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:05.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:04 smithi106 ceph-mon[36505]: pgmap v1494: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:05.889 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2554 sec 2024-03-24T01:07:06.889 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2555 sec 2024-03-24T01:07:07.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:06 smithi078 ceph-mon[32429]: pgmap v1495: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:06 smithi107 ceph-mon[36485]: pgmap v1495: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:07.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:06 smithi106 ceph-mon[36505]: pgmap v1495: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:07.889 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2556 sec 2024-03-24T01:07:08.889 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2557 sec 2024-03-24T01:07:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:08 smithi078 ceph-mon[32429]: pgmap v1496: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1903.133718 secs 2024-03-24T01:07:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:08 smithi107 ceph-mon[36485]: pgmap v1496: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1903.133718 secs 2024-03-24T01:07:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:08 smithi106 ceph-mon[36505]: pgmap v1496: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1903.133718 secs 2024-03-24T01:07:09.889 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2558 sec 2024-03-24T01:07:10.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:07:10.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:07:10.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:07:10.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:07:10.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:07:10.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:07:10.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:07:10.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:07:10.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:07:10.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:07:10.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:07:10.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:07:10.889 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2559 sec 2024-03-24T01:07:11.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:10 smithi106 ceph-mon[36505]: pgmap v1497: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:11.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:10 smithi106 ceph-mon[36505]: pgmap v1498: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:07:11.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:10 smithi106 ceph-mon[36505]: pgmap v1499: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:07:11.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:10 smithi078 ceph-mon[32429]: pgmap v1497: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:11.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:10 smithi078 ceph-mon[32429]: pgmap v1498: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:07:11.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:10 smithi078 ceph-mon[32429]: pgmap v1499: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:07:11.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:10 smithi107 ceph-mon[36485]: pgmap v1497: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:11.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:10 smithi107 ceph-mon[36485]: pgmap v1498: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:07:11.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:10 smithi107 ceph-mon[36485]: pgmap v1499: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:07:11.889 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2560 sec 2024-03-24T01:07:12.889 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2561 sec 2024-03-24T01:07:13.143 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:12 smithi078 ceph-mon[32429]: pgmap v1500: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:07:13.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:12 smithi106 ceph-mon[36505]: pgmap v1500: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:07:13.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:12 smithi107 ceph-mon[36485]: pgmap v1500: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:07:13.889 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2562 sec 2024-03-24T01:07:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1908.133782 secs 2024-03-24T01:07:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1908.133782 secs 2024-03-24T01:07:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1908.133782 secs 2024-03-24T01:07:14.889 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2563 sec 2024-03-24T01:07:15.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:14 smithi106 ceph-mon[36505]: pgmap v1501: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:07:15.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:14 smithi078 ceph-mon[32429]: pgmap v1501: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:07:15.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:14 smithi107 ceph-mon[36485]: pgmap v1501: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:07:15.889 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2564 sec 2024-03-24T01:07:16.889 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2565 sec 2024-03-24T01:07:17.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:16 smithi106 ceph-mon[36505]: pgmap v1502: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:07:17.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:16 smithi078 ceph-mon[32429]: pgmap v1502: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:07:17.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:16 smithi107 ceph-mon[36485]: pgmap v1502: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:07:17.890 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2566 sec 2024-03-24T01:07:18.890 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2567 sec 2024-03-24T01:07:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:18 smithi106 ceph-mon[36505]: pgmap v1503: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:07:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1913.133843 secs 2024-03-24T01:07:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:18 smithi078 ceph-mon[32429]: pgmap v1503: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:07:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1913.133843 secs 2024-03-24T01:07:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:18 smithi107 ceph-mon[36485]: pgmap v1503: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:07:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1913.133843 secs 2024-03-24T01:07:19.890 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2568 sec 2024-03-24T01:07:20.890 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2569 sec 2024-03-24T01:07:21.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:20 smithi106 ceph-mon[36505]: pgmap v1504: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:07:21.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:20 smithi078 ceph-mon[32429]: pgmap v1504: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:07:21.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:20 smithi107 ceph-mon[36485]: pgmap v1504: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:07:21.890 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2570 sec 2024-03-24T01:07:22.890 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2571 sec 2024-03-24T01:07:23.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:22 smithi106 ceph-mon[36505]: pgmap v1505: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:23.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:22 smithi078 ceph-mon[32429]: pgmap v1505: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:23.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:22 smithi107 ceph-mon[36485]: pgmap v1505: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:23.890 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2572 sec 2024-03-24T01:07:24.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1918.133903 secs 2024-03-24T01:07:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1918.133903 secs 2024-03-24T01:07:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1918.133903 secs 2024-03-24T01:07:24.890 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2573 sec 2024-03-24T01:07:25.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:24 smithi106 ceph-mon[36505]: pgmap v1506: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:24 smithi078 ceph-mon[32429]: pgmap v1506: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:25.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:24 smithi107 ceph-mon[36485]: pgmap v1506: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:25.891 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2574 sec 2024-03-24T01:07:26.890 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2575 sec 2024-03-24T01:07:27.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:26 smithi106 ceph-mon[36505]: pgmap v1507: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:27.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:26 smithi078 ceph-mon[32429]: pgmap v1507: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:27.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:26 smithi107 ceph-mon[36485]: pgmap v1507: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:27.890 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2576 sec 2024-03-24T01:07:28.891 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2577 sec 2024-03-24T01:07:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:28 smithi106 ceph-mon[36505]: pgmap v1508: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1923.133941 secs 2024-03-24T01:07:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:28 smithi078 ceph-mon[32429]: pgmap v1508: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1923.133941 secs 2024-03-24T01:07:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:28 smithi107 ceph-mon[36485]: pgmap v1508: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1923.133941 secs 2024-03-24T01:07:29.891 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2578 sec 2024-03-24T01:07:30.892 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2579 sec 2024-03-24T01:07:31.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:30 smithi106 ceph-mon[36505]: pgmap v1509: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:31.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:30 smithi078 ceph-mon[32429]: pgmap v1509: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:31.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:30 smithi107 ceph-mon[36485]: pgmap v1509: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:31.891 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2580 sec 2024-03-24T01:07:32.891 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2581 sec 2024-03-24T01:07:33.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:32 smithi107 ceph-mon[36485]: pgmap v1510: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:33.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:32 smithi106 ceph-mon[36505]: pgmap v1510: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:33.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:32 smithi078 ceph-mon[32429]: pgmap v1510: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:33.891 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2582 sec 2024-03-24T01:07:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1928.133981 secs 2024-03-24T01:07:34.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1928.133981 secs 2024-03-24T01:07:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1928.133981 secs 2024-03-24T01:07:34.891 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2583 sec 2024-03-24T01:07:35.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:34 smithi106 ceph-mon[36505]: pgmap v1511: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:35.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:34 smithi078 ceph-mon[32429]: pgmap v1511: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:35.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:34 smithi107 ceph-mon[36485]: pgmap v1511: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:35.891 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2584 sec 2024-03-24T01:07:36.891 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2585 sec 2024-03-24T01:07:37.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:36 smithi106 ceph-mon[36505]: pgmap v1512: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:37.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:36 smithi078 ceph-mon[32429]: pgmap v1512: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:36 smithi107 ceph-mon[36485]: pgmap v1512: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:37.891 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2586 sec 2024-03-24T01:07:38.891 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2587 sec 2024-03-24T01:07:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:38 smithi078 ceph-mon[32429]: pgmap v1513: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:39.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1933.134069 secs 2024-03-24T01:07:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:38 smithi107 ceph-mon[36485]: pgmap v1513: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:39.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1933.134069 secs 2024-03-24T01:07:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:38 smithi106 ceph-mon[36505]: pgmap v1513: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1933.134069 secs 2024-03-24T01:07:39.891 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2588 sec 2024-03-24T01:07:40.891 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2589 sec 2024-03-24T01:07:41.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:40 smithi078 ceph-mon[32429]: pgmap v1514: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:40 smithi107 ceph-mon[36485]: pgmap v1514: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:41.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:40 smithi106 ceph-mon[36505]: pgmap v1514: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:41.892 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2590 sec 2024-03-24T01:07:42.892 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2591 sec 2024-03-24T01:07:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:42 smithi078 ceph-mon[32429]: pgmap v1515: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:43.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:42 smithi107 ceph-mon[36485]: pgmap v1515: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:43.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:42 smithi106 ceph-mon[36505]: pgmap v1515: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:43.892 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2592 sec 2024-03-24T01:07:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1938.134133 secs 2024-03-24T01:07:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1938.134133 secs 2024-03-24T01:07:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1938.134133 secs 2024-03-24T01:07:44.892 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2593 sec 2024-03-24T01:07:45.303 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:44 smithi106 ceph-mon[36505]: pgmap v1516: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:45.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:44 smithi078 ceph-mon[32429]: pgmap v1516: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:45.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:44 smithi107 ceph-mon[36485]: pgmap v1516: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:45.892 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2594 sec 2024-03-24T01:07:46.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:45 smithi078 ceph-mon[32429]: pgmap v1517: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:46.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:45 smithi107 ceph-mon[36485]: pgmap v1517: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:46.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:45 smithi106 ceph-mon[36505]: pgmap v1517: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:46.892 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2595 sec 2024-03-24T01:07:47.892 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2596 sec 2024-03-24T01:07:48.892 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2597 sec 2024-03-24T01:07:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:48 smithi106 ceph-mon[36505]: pgmap v1518: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1943.134222 secs 2024-03-24T01:07:49.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:48 smithi078 ceph-mon[32429]: pgmap v1518: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:49.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1943.134222 secs 2024-03-24T01:07:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:48 smithi107 ceph-mon[36485]: pgmap v1518: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1943.134222 secs 2024-03-24T01:07:49.893 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2598 sec 2024-03-24T01:07:50.892 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2599 sec 2024-03-24T01:07:51.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:50 smithi106 ceph-mon[36505]: pgmap v1519: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:51.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:50 smithi078 ceph-mon[32429]: pgmap v1519: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:51.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:50 smithi107 ceph-mon[36485]: pgmap v1519: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:51.892 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2600 sec 2024-03-24T01:07:52.893 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2601 sec 2024-03-24T01:07:53.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:52 smithi078 ceph-mon[32429]: pgmap v1520: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:53.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:52 smithi107 ceph-mon[36485]: pgmap v1520: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:53.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:52 smithi106 ceph-mon[36505]: pgmap v1520: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:53.893 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2602 sec 2024-03-24T01:07:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1948.134326 secs 2024-03-24T01:07:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1948.134326 secs 2024-03-24T01:07:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1948.134326 secs 2024-03-24T01:07:54.893 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2603 sec 2024-03-24T01:07:55.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:54 smithi078 ceph-mon[32429]: pgmap v1521: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:55.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:54 smithi107 ceph-mon[36485]: pgmap v1521: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:55.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:54 smithi106 ceph-mon[36505]: pgmap v1521: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:55.893 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2604 sec 2024-03-24T01:07:56.893 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2605 sec 2024-03-24T01:07:57.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:56 smithi078 ceph-mon[32429]: pgmap v1522: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:57.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:56 smithi107 ceph-mon[36485]: pgmap v1522: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:57.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:56 smithi106 ceph-mon[36505]: pgmap v1522: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:57.893 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2606 sec 2024-03-24T01:07:58.893 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2607 sec 2024-03-24T01:07:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:58 smithi078 ceph-mon[32429]: pgmap v1523: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1953.134409 secs 2024-03-24T01:07:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:58 smithi107 ceph-mon[36485]: pgmap v1523: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1953.134409 secs 2024-03-24T01:07:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:58 smithi106 ceph-mon[36505]: pgmap v1523: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:07:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1953.134409 secs 2024-03-24T01:07:59.893 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2608 sec 2024-03-24T01:08:00.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:07:59 smithi078 ceph-mon[32429]: pgmap v1524: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:00.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:07:59 smithi107 ceph-mon[36485]: pgmap v1524: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:00.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:07:59 smithi106 ceph-mon[36505]: pgmap v1524: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:00.893 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2609 sec 2024-03-24T01:08:01.893 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2610 sec 2024-03-24T01:08:02.894 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2611 sec 2024-03-24T01:08:03.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:02 smithi078 ceph-mon[32429]: pgmap v1525: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:03.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:02 smithi107 ceph-mon[36485]: pgmap v1525: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:03.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:02 smithi106 ceph-mon[36505]: pgmap v1525: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:03.894 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2612 sec 2024-03-24T01:08:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1958.134470 secs 2024-03-24T01:08:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1958.134470 secs 2024-03-24T01:08:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1958.134470 secs 2024-03-24T01:08:04.894 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2613 sec 2024-03-24T01:08:05.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:04 smithi078 ceph-mon[32429]: pgmap v1526: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:05.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:04 smithi107 ceph-mon[36485]: pgmap v1526: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:05.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:04 smithi106 ceph-mon[36505]: pgmap v1526: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:05.894 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2614 sec 2024-03-24T01:08:06.894 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2615 sec 2024-03-24T01:08:07.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:06 smithi078 ceph-mon[32429]: pgmap v1527: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:07.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:06 smithi107 ceph-mon[36485]: pgmap v1527: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:07.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:06 smithi106 ceph-mon[36505]: pgmap v1527: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:07.894 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2616 sec 2024-03-24T01:08:08.894 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2617 sec 2024-03-24T01:08:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:08 smithi078 ceph-mon[32429]: pgmap v1528: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1963.134533 secs 2024-03-24T01:08:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:08 smithi107 ceph-mon[36485]: pgmap v1528: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1963.134533 secs 2024-03-24T01:08:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:08 smithi106 ceph-mon[36505]: pgmap v1528: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1963.134533 secs 2024-03-24T01:08:09.894 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2618 sec 2024-03-24T01:08:10.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:09 smithi078 ceph-mon[32429]: pgmap v1529: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:10.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:09 smithi107 ceph-mon[36485]: pgmap v1529: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:10.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:09 smithi106 ceph-mon[36505]: pgmap v1529: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:10.894 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2619 sec 2024-03-24T01:08:11.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:08:11.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:08:11.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:08:11.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:10 smithi078 ceph-mon[32429]: pgmap v1530: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:08:11.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:10 smithi078 ceph-mon[32429]: pgmap v1531: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:08:11.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:08:11.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:08:11.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:08:11.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:08:11.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:10 smithi107 ceph-mon[36485]: pgmap v1530: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:08:11.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:10 smithi107 ceph-mon[36485]: pgmap v1531: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:08:11.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:08:11.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:08:11.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:08:11.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:08:11.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:10 smithi106 ceph-mon[36505]: pgmap v1530: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:08:11.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:10 smithi106 ceph-mon[36505]: pgmap v1531: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:08:11.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:08:11.894 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2620 sec 2024-03-24T01:08:12.895 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2621 sec 2024-03-24T01:08:13.895 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2622 sec 2024-03-24T01:08:14.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:13 smithi078 ceph-mon[32429]: pgmap v1532: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:08:14.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1968.134596 secs 2024-03-24T01:08:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:13 smithi107 ceph-mon[36485]: pgmap v1532: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:08:14.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1968.134596 secs 2024-03-24T01:08:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:13 smithi106 ceph-mon[36505]: pgmap v1532: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:08:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1968.134596 secs 2024-03-24T01:08:14.895 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2623 sec 2024-03-24T01:08:15.895 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2624 sec 2024-03-24T01:08:15.954 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:15 smithi106 ceph-mon[36505]: pgmap v1533: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:08:16.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:15 smithi078 ceph-mon[32429]: pgmap v1533: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:08:16.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:15 smithi107 ceph-mon[36485]: pgmap v1533: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:08:16.895 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2625 sec 2024-03-24T01:08:17.895 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2626 sec 2024-03-24T01:08:18.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:17 smithi078 ceph-mon[32429]: pgmap v1534: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:08:18.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:17 smithi107 ceph-mon[36485]: pgmap v1534: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:08:18.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:17 smithi106 ceph-mon[36505]: pgmap v1534: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:08:18.895 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2627 sec 2024-03-24T01:08:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1973.134656 secs 2024-03-24T01:08:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1973.134656 secs 2024-03-24T01:08:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1973.134656 secs 2024-03-24T01:08:19.895 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2628 sec 2024-03-24T01:08:20.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:19 smithi078 ceph-mon[32429]: pgmap v1535: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:08:20.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:19 smithi107 ceph-mon[36485]: pgmap v1535: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:08:20.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:19 smithi106 ceph-mon[36505]: pgmap v1535: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:08:20.895 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2629 sec 2024-03-24T01:08:21.895 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2630 sec 2024-03-24T01:08:22.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:21 smithi078 ceph-mon[32429]: pgmap v1536: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:08:22.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:21 smithi107 ceph-mon[36485]: pgmap v1536: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:08:22.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:21 smithi106 ceph-mon[36505]: pgmap v1536: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:08:22.895 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2631 sec 2024-03-24T01:08:23.896 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2632 sec 2024-03-24T01:08:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:23 smithi078 ceph-mon[32429]: pgmap v1537: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1978.134718 secs 2024-03-24T01:08:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:23 smithi107 ceph-mon[36485]: pgmap v1537: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1978.134718 secs 2024-03-24T01:08:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:23 smithi106 ceph-mon[36505]: pgmap v1537: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1978.134718 secs 2024-03-24T01:08:24.896 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2633 sec 2024-03-24T01:08:25.896 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2634 sec 2024-03-24T01:08:26.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:25 smithi078 ceph-mon[32429]: pgmap v1538: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:26.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:25 smithi107 ceph-mon[36485]: pgmap v1538: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:26.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:25 smithi106 ceph-mon[36505]: pgmap v1538: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:26.896 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2635 sec 2024-03-24T01:08:27.896 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2636 sec 2024-03-24T01:08:28.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:27 smithi078 ceph-mon[32429]: pgmap v1539: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:27 smithi107 ceph-mon[36485]: pgmap v1539: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:28.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:27 smithi106 ceph-mon[36505]: pgmap v1539: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:28.896 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2637 sec 2024-03-24T01:08:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1983.134780 secs 2024-03-24T01:08:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1983.134780 secs 2024-03-24T01:08:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1983.134780 secs 2024-03-24T01:08:29.896 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2638 sec 2024-03-24T01:08:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:29 smithi078 ceph-mon[32429]: pgmap v1540: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:30.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:29 smithi107 ceph-mon[36485]: pgmap v1540: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:30.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:29 smithi106 ceph-mon[36505]: pgmap v1540: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:30.897 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2639 sec 2024-03-24T01:08:31.896 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2640 sec 2024-03-24T01:08:32.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:31 smithi078 ceph-mon[32429]: pgmap v1541: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:32.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:31 smithi107 ceph-mon[36485]: pgmap v1541: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:32.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:31 smithi106 ceph-mon[36505]: pgmap v1541: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:32.896 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2641 sec 2024-03-24T01:08:33.897 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2642 sec 2024-03-24T01:08:34.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:33 smithi078 ceph-mon[32429]: pgmap v1542: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:34.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1988.134869 secs 2024-03-24T01:08:34.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:33 smithi107 ceph-mon[36485]: pgmap v1542: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:34.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1988.134869 secs 2024-03-24T01:08:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:33 smithi106 ceph-mon[36505]: pgmap v1542: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1988.134869 secs 2024-03-24T01:08:34.897 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2643 sec 2024-03-24T01:08:35.897 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2644 sec 2024-03-24T01:08:36.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:35 smithi078 ceph-mon[32429]: pgmap v1543: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:36.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:35 smithi107 ceph-mon[36485]: pgmap v1543: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:36.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:35 smithi106 ceph-mon[36505]: pgmap v1543: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:36.897 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2645 sec 2024-03-24T01:08:37.897 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2646 sec 2024-03-24T01:08:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:37 smithi078 ceph-mon[32429]: pgmap v1544: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:38.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:37 smithi107 ceph-mon[36485]: pgmap v1544: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:38.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:37 smithi106 ceph-mon[36505]: pgmap v1544: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:38.897 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2647 sec 2024-03-24T01:08:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1993.134959 secs 2024-03-24T01:08:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1993.134959 secs 2024-03-24T01:08:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1993.134959 secs 2024-03-24T01:08:39.897 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2648 sec 2024-03-24T01:08:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:39 smithi078 ceph-mon[32429]: pgmap v1545: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:39 smithi107 ceph-mon[36485]: pgmap v1545: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:39 smithi106 ceph-mon[36505]: pgmap v1545: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:40.897 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2649 sec 2024-03-24T01:08:41.897 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2650 sec 2024-03-24T01:08:42.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:41 smithi078 ceph-mon[32429]: pgmap v1546: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:42.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:41 smithi107 ceph-mon[36485]: pgmap v1546: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:42.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:41 smithi106 ceph-mon[36505]: pgmap v1546: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:42.897 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2651 sec 2024-03-24T01:08:43.897 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2652 sec 2024-03-24T01:08:44.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:43 smithi078 ceph-mon[32429]: pgmap v1547: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:44.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 1998.135047 secs 2024-03-24T01:08:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:43 smithi107 ceph-mon[36485]: pgmap v1547: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 1998.135047 secs 2024-03-24T01:08:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:43 smithi106 ceph-mon[36505]: pgmap v1547: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 1998.135047 secs 2024-03-24T01:08:44.897 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2653 sec 2024-03-24T01:08:45.898 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2654 sec 2024-03-24T01:08:46.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:45 smithi078 ceph-mon[32429]: pgmap v1548: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:45 smithi107 ceph-mon[36485]: pgmap v1548: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:46.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:45 smithi106 ceph-mon[36505]: pgmap v1548: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:46.898 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2655 sec 2024-03-24T01:08:47.898 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2656 sec 2024-03-24T01:08:48.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:47 smithi078 ceph-mon[32429]: pgmap v1549: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:48.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:47 smithi107 ceph-mon[36485]: pgmap v1549: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:48.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:47 smithi106 ceph-mon[36505]: pgmap v1549: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:48.898 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2657 sec 2024-03-24T01:08:49.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2003.135137 secs 2024-03-24T01:08:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2003.135137 secs 2024-03-24T01:08:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2003.135137 secs 2024-03-24T01:08:49.898 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2658 sec 2024-03-24T01:08:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:49 smithi078 ceph-mon[32429]: pgmap v1550: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:50.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:49 smithi107 ceph-mon[36485]: pgmap v1550: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:49 smithi106 ceph-mon[36505]: pgmap v1550: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:50.898 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2659 sec 2024-03-24T01:08:51.898 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2660 sec 2024-03-24T01:08:52.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:51 smithi078 ceph-mon[32429]: pgmap v1551: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:52.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:51 smithi107 ceph-mon[36485]: pgmap v1551: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:52.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:51 smithi106 ceph-mon[36505]: pgmap v1551: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:52.898 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2661 sec 2024-03-24T01:08:53.899 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2662 sec 2024-03-24T01:08:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:53 smithi078 ceph-mon[32429]: pgmap v1552: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2008.135176 secs 2024-03-24T01:08:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:53 smithi107 ceph-mon[36485]: pgmap v1552: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2008.135176 secs 2024-03-24T01:08:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:53 smithi106 ceph-mon[36505]: pgmap v1552: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2008.135176 secs 2024-03-24T01:08:54.898 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2663 sec 2024-03-24T01:08:55.898 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2664 sec 2024-03-24T01:08:56.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:55 smithi078 ceph-mon[32429]: pgmap v1553: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:56.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:55 smithi107 ceph-mon[36485]: pgmap v1553: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:56.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:55 smithi106 ceph-mon[36505]: pgmap v1553: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:08:56.899 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2665 sec 2024-03-24T01:08:57.898 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2666 sec 2024-03-24T01:08:58.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:57 smithi078 ceph-mon[32429]: pgmap v1554: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:58.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:57 smithi107 ceph-mon[36485]: pgmap v1554: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:58.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:57 smithi106 ceph-mon[36505]: pgmap v1554: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:58.899 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2667 sec 2024-03-24T01:08:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2013.135265 secs 2024-03-24T01:08:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2013.135265 secs 2024-03-24T01:08:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2013.135265 secs 2024-03-24T01:08:59.899 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2668 sec 2024-03-24T01:09:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:08:59 smithi078 ceph-mon[32429]: pgmap v1555: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:00.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:08:59 smithi107 ceph-mon[36485]: pgmap v1555: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:00.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:08:59 smithi106 ceph-mon[36505]: pgmap v1555: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:00.899 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2669 sec 2024-03-24T01:09:01.899 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2670 sec 2024-03-24T01:09:02.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:01 smithi078 ceph-mon[32429]: pgmap v1556: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:02.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:01 smithi107 ceph-mon[36485]: pgmap v1556: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:02.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:01 smithi106 ceph-mon[36505]: pgmap v1556: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:02.899 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2671 sec 2024-03-24T01:09:03.899 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2672 sec 2024-03-24T01:09:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:03 smithi078 ceph-mon[32429]: pgmap v1557: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:04.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2018.135355 secs 2024-03-24T01:09:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:03 smithi107 ceph-mon[36485]: pgmap v1557: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2018.135355 secs 2024-03-24T01:09:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:03 smithi106 ceph-mon[36505]: pgmap v1557: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2018.135355 secs 2024-03-24T01:09:04.899 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2673 sec 2024-03-24T01:09:05.899 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2674 sec 2024-03-24T01:09:06.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:05 smithi078 ceph-mon[32429]: pgmap v1558: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:05 smithi107 ceph-mon[36485]: pgmap v1558: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:06.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:05 smithi106 ceph-mon[36505]: pgmap v1558: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:06.899 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2675 sec 2024-03-24T01:09:07.899 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2676 sec 2024-03-24T01:09:08.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:07 smithi078 ceph-mon[32429]: pgmap v1559: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:08.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:07 smithi107 ceph-mon[36485]: pgmap v1559: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:08.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:07 smithi106 ceph-mon[36505]: pgmap v1559: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:08.900 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2677 sec 2024-03-24T01:09:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2023.135416 secs 2024-03-24T01:09:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2023.135416 secs 2024-03-24T01:09:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2023.135416 secs 2024-03-24T01:09:09.900 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2678 sec 2024-03-24T01:09:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:09 smithi078 ceph-mon[32429]: pgmap v1560: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:10.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:09 smithi107 ceph-mon[36485]: pgmap v1560: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:10.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:09 smithi106 ceph-mon[36505]: pgmap v1560: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:10.900 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2679 sec 2024-03-24T01:09:11.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:09:11.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:09:11.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:09:11.900 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2680 sec 2024-03-24T01:09:12.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:11 smithi078 ceph-mon[32429]: pgmap v1561: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:12.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:11 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:09:12.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:11 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:09:12.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:11 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:09:12.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:11 smithi107 ceph-mon[36485]: pgmap v1561: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:12.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:11 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:09:12.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:11 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:09:12.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:11 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:09:12.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:11 smithi106 ceph-mon[36505]: pgmap v1561: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:12.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:11 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:09:12.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:11 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:09:12.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:11 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:09:12.900 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2681 sec 2024-03-24T01:09:13.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:12 smithi078 ceph-mon[32429]: pgmap v1562: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:09:13.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:12 smithi078 ceph-mon[32429]: pgmap v1563: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:09:13.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:12 smithi107 ceph-mon[36485]: pgmap v1562: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:09:13.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:12 smithi107 ceph-mon[36485]: pgmap v1563: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:09:13.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:12 smithi106 ceph-mon[36505]: pgmap v1562: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:09:13.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:12 smithi106 ceph-mon[36505]: pgmap v1563: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:09:13.900 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2682 sec 2024-03-24T01:09:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2028.135477 secs 2024-03-24T01:09:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2028.135477 secs 2024-03-24T01:09:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2028.135477 secs 2024-03-24T01:09:14.900 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2683 sec 2024-03-24T01:09:15.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:14 smithi107 ceph-mon[36485]: pgmap v1564: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:09:15.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:14 smithi106 ceph-mon[36505]: pgmap v1564: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:09:15.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:14 smithi078 ceph-mon[32429]: pgmap v1564: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:09:15.900 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2684 sec 2024-03-24T01:09:16.900 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2685 sec 2024-03-24T01:09:17.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:16 smithi106 ceph-mon[36505]: pgmap v1565: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:09:17.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:16 smithi078 ceph-mon[32429]: pgmap v1565: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:09:17.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:16 smithi107 ceph-mon[36485]: pgmap v1565: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:09:17.900 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2686 sec 2024-03-24T01:09:18.900 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2687 sec 2024-03-24T01:09:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:18 smithi106 ceph-mon[36505]: pgmap v1566: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:09:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2033.135540 secs 2024-03-24T01:09:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:18 smithi078 ceph-mon[32429]: pgmap v1566: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:09:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2033.135540 secs 2024-03-24T01:09:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:18 smithi107 ceph-mon[36485]: pgmap v1566: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:09:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2033.135540 secs 2024-03-24T01:09:19.900 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2688 sec 2024-03-24T01:09:20.900 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2689 sec 2024-03-24T01:09:21.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:20 smithi106 ceph-mon[36505]: pgmap v1567: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:09:21.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:20 smithi078 ceph-mon[32429]: pgmap v1567: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:09:21.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:20 smithi107 ceph-mon[36485]: pgmap v1567: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:09:21.901 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2690 sec 2024-03-24T01:09:22.901 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2691 sec 2024-03-24T01:09:23.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:22 smithi106 ceph-mon[36505]: pgmap v1568: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:09:23.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:22 smithi078 ceph-mon[32429]: pgmap v1568: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:09:23.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:22 smithi107 ceph-mon[36485]: pgmap v1568: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:09:23.901 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2692 sec 2024-03-24T01:09:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2038.135583 secs 2024-03-24T01:09:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2038.135583 secs 2024-03-24T01:09:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2038.135583 secs 2024-03-24T01:09:24.901 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2693 sec 2024-03-24T01:09:25.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:24 smithi106 ceph-mon[36505]: pgmap v1569: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:24 smithi078 ceph-mon[32429]: pgmap v1569: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:25.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:24 smithi107 ceph-mon[36485]: pgmap v1569: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:25.901 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2694 sec 2024-03-24T01:09:26.901 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2695 sec 2024-03-24T01:09:27.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:26 smithi106 ceph-mon[36505]: pgmap v1570: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:27.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:26 smithi078 ceph-mon[32429]: pgmap v1570: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:27.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:26 smithi107 ceph-mon[36485]: pgmap v1570: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:27.901 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2696 sec 2024-03-24T01:09:28.901 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2697 sec 2024-03-24T01:09:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:28 smithi106 ceph-mon[36505]: pgmap v1571: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2043.135622 secs 2024-03-24T01:09:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:28 smithi078 ceph-mon[32429]: pgmap v1571: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2043.135622 secs 2024-03-24T01:09:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:28 smithi107 ceph-mon[36485]: pgmap v1571: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2043.135622 secs 2024-03-24T01:09:29.901 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2698 sec 2024-03-24T01:09:30.901 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2699 sec 2024-03-24T01:09:31.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:30 smithi106 ceph-mon[36505]: pgmap v1572: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:31.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:30 smithi078 ceph-mon[32429]: pgmap v1572: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:31.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:30 smithi107 ceph-mon[36485]: pgmap v1572: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:31.901 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2700 sec 2024-03-24T01:09:32.901 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2701 sec 2024-03-24T01:09:33.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:32 smithi107 ceph-mon[36485]: pgmap v1573: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:33.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:32 smithi106 ceph-mon[36505]: pgmap v1573: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:33.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:32 smithi078 ceph-mon[32429]: pgmap v1573: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:33.902 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2702 sec 2024-03-24T01:09:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2048.135685 secs 2024-03-24T01:09:34.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2048.135685 secs 2024-03-24T01:09:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2048.135685 secs 2024-03-24T01:09:34.902 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2703 sec 2024-03-24T01:09:35.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:34 smithi106 ceph-mon[36505]: pgmap v1574: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:35.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:34 smithi078 ceph-mon[32429]: pgmap v1574: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:35.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:34 smithi107 ceph-mon[36485]: pgmap v1574: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:35.902 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2704 sec 2024-03-24T01:09:36.902 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2705 sec 2024-03-24T01:09:37.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:36 smithi106 ceph-mon[36505]: pgmap v1575: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:37.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:36 smithi078 ceph-mon[32429]: pgmap v1575: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:36 smithi107 ceph-mon[36485]: pgmap v1575: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:37.902 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2706 sec 2024-03-24T01:09:38.902 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2707 sec 2024-03-24T01:09:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:38 smithi106 ceph-mon[36505]: pgmap v1576: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2053.135774 secs 2024-03-24T01:09:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:38 smithi078 ceph-mon[32429]: pgmap v1576: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2053.135774 secs 2024-03-24T01:09:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:38 smithi107 ceph-mon[36485]: pgmap v1576: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2053.135774 secs 2024-03-24T01:09:39.902 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2708 sec 2024-03-24T01:09:40.902 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2709 sec 2024-03-24T01:09:41.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:40 smithi106 ceph-mon[36505]: pgmap v1577: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:41.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:40 smithi078 ceph-mon[32429]: pgmap v1577: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:40 smithi107 ceph-mon[36485]: pgmap v1577: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:41.902 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2710 sec 2024-03-24T01:09:42.902 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2711 sec 2024-03-24T01:09:43.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:42 smithi106 ceph-mon[36505]: pgmap v1578: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:42 smithi078 ceph-mon[32429]: pgmap v1578: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:43.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:42 smithi107 ceph-mon[36485]: pgmap v1578: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:43.902 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2712 sec 2024-03-24T01:09:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2058.135835 secs 2024-03-24T01:09:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2058.135835 secs 2024-03-24T01:09:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2058.135835 secs 2024-03-24T01:09:44.902 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2713 sec 2024-03-24T01:09:45.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:44 smithi106 ceph-mon[36505]: pgmap v1579: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:45.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:44 smithi078 ceph-mon[32429]: pgmap v1579: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:45.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:44 smithi107 ceph-mon[36485]: pgmap v1579: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:45.902 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2714 sec 2024-03-24T01:09:46.902 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2715 sec 2024-03-24T01:09:47.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:46 smithi078 ceph-mon[32429]: pgmap v1580: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:47.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:46 smithi107 ceph-mon[36485]: pgmap v1580: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:47.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:46 smithi106 ceph-mon[36505]: pgmap v1580: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:47.902 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2716 sec 2024-03-24T01:09:48.903 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2717 sec 2024-03-24T01:09:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:48 smithi078 ceph-mon[32429]: pgmap v1581: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:49.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2063.135875 secs 2024-03-24T01:09:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:48 smithi107 ceph-mon[36485]: pgmap v1581: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:49.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2063.135875 secs 2024-03-24T01:09:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:48 smithi106 ceph-mon[36505]: pgmap v1581: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2063.135875 secs 2024-03-24T01:09:49.903 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2718 sec 2024-03-24T01:09:50.903 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2719 sec 2024-03-24T01:09:51.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:50 smithi078 ceph-mon[32429]: pgmap v1582: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:51.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:50 smithi107 ceph-mon[36485]: pgmap v1582: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:51.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:50 smithi106 ceph-mon[36505]: pgmap v1582: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:51.903 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2720 sec 2024-03-24T01:09:52.903 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2721 sec 2024-03-24T01:09:53.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:52 smithi078 ceph-mon[32429]: pgmap v1583: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:53.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:52 smithi107 ceph-mon[36485]: pgmap v1583: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:53.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:52 smithi106 ceph-mon[36505]: pgmap v1583: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:53.903 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2722 sec 2024-03-24T01:09:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2068.135962 secs 2024-03-24T01:09:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2068.135962 secs 2024-03-24T01:09:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2068.135962 secs 2024-03-24T01:09:54.903 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2723 sec 2024-03-24T01:09:55.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:54 smithi078 ceph-mon[32429]: pgmap v1584: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:55.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:54 smithi107 ceph-mon[36485]: pgmap v1584: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:55.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:54 smithi106 ceph-mon[36505]: pgmap v1584: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:55.903 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2724 sec 2024-03-24T01:09:56.903 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2725 sec 2024-03-24T01:09:57.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:56 smithi078 ceph-mon[32429]: pgmap v1585: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:57.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:56 smithi107 ceph-mon[36485]: pgmap v1585: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:57.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:56 smithi106 ceph-mon[36505]: pgmap v1585: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:57.903 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2726 sec 2024-03-24T01:09:58.904 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2727 sec 2024-03-24T01:09:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:58 smithi078 ceph-mon[32429]: pgmap v1586: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:09:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2073.136024 secs 2024-03-24T01:09:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:58 smithi107 ceph-mon[36485]: pgmap v1586: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:09:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2073.136024 secs 2024-03-24T01:09:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:58 smithi106 ceph-mon[36505]: pgmap v1586: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:09:59.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:09:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2073.136024 secs 2024-03-24T01:09:59.904 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2728 sec 2024-03-24T01:10:00.904 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2729 sec 2024-03-24T01:10:01.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:00 smithi078 ceph-mon[32429]: pgmap v1587: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:01.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:00 smithi078 ceph-mon[32429]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:10:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:00 smithi107 ceph-mon[36485]: pgmap v1587: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:00 smithi107 ceph-mon[36485]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:10:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:00 smithi106 ceph-mon[36505]: pgmap v1587: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:00 smithi106 ceph-mon[36505]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:10:01.904 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2730 sec 2024-03-24T01:10:02.904 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2731 sec 2024-03-24T01:10:03.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:02 smithi078 ceph-mon[32429]: pgmap v1588: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:03.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:02 smithi107 ceph-mon[36485]: pgmap v1588: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:03.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:02 smithi106 ceph-mon[36505]: pgmap v1588: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:03.904 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2732 sec 2024-03-24T01:10:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2078.136112 secs 2024-03-24T01:10:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2078.136112 secs 2024-03-24T01:10:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2078.136112 secs 2024-03-24T01:10:04.904 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2733 sec 2024-03-24T01:10:05.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:04 smithi078 ceph-mon[32429]: pgmap v1589: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:05.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:04 smithi107 ceph-mon[36485]: pgmap v1589: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:05.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:04 smithi106 ceph-mon[36505]: pgmap v1589: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:05.904 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2734 sec 2024-03-24T01:10:06.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:05 smithi078 ceph-mon[32429]: pgmap v1590: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:06.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:05 smithi107 ceph-mon[36485]: pgmap v1590: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:06.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:05 smithi106 ceph-mon[36505]: pgmap v1590: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:06.904 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2735 sec 2024-03-24T01:10:07.904 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2736 sec 2024-03-24T01:10:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:08 smithi078 ceph-mon[32429]: pgmap v1591: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2083.136203 secs 2024-03-24T01:10:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:08 smithi107 ceph-mon[36485]: pgmap v1591: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2083.136203 secs 2024-03-24T01:10:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:08 smithi106 ceph-mon[36505]: pgmap v1591: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2083.136203 secs 2024-03-24T01:10:08.904 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2737 sec 2024-03-24T01:10:09.904 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2738 sec 2024-03-24T01:10:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:10 smithi078 ceph-mon[32429]: pgmap v1592: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:10 smithi107 ceph-mon[36485]: pgmap v1592: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:10 smithi106 ceph-mon[36505]: pgmap v1592: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:10.904 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2739 sec 2024-03-24T01:10:11.905 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2740 sec 2024-03-24T01:10:12.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:12 smithi078 ceph-mon[32429]: pgmap v1593: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:12.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:12 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:10:12.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:12 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:10:12.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:12 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:10:12.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:12 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:10:12.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:12 smithi107 ceph-mon[36485]: pgmap v1593: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:12.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:12 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:10:12.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:12 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:10:12.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:12 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:10:12.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:12 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:10:12.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:12 smithi106 ceph-mon[36505]: pgmap v1593: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:12.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:12 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:10:12.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:12 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:10:12.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:12 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:10:12.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:12 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:10:12.904 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2741 sec 2024-03-24T01:10:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:13 smithi078 ceph-mon[32429]: pgmap v1594: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:10:13.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:13 smithi078 ceph-mon[32429]: pgmap v1595: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:10:13.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2088.136294 secs 2024-03-24T01:10:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:13 smithi107 ceph-mon[36485]: pgmap v1594: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:10:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:13 smithi107 ceph-mon[36485]: pgmap v1595: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:10:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2088.136294 secs 2024-03-24T01:10:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:13 smithi106 ceph-mon[36505]: pgmap v1594: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:10:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:13 smithi106 ceph-mon[36505]: pgmap v1595: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:10:13.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2088.136294 secs 2024-03-24T01:10:13.904 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2742 sec 2024-03-24T01:10:14.905 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2743 sec 2024-03-24T01:10:15.905 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2744 sec 2024-03-24T01:10:16.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:15 smithi078 ceph-mon[32429]: pgmap v1596: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:10:16.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:15 smithi107 ceph-mon[36485]: pgmap v1596: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:10:16.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:15 smithi106 ceph-mon[36505]: pgmap v1596: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:10:16.905 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2745 sec 2024-03-24T01:10:17.905 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2746 sec 2024-03-24T01:10:18.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:17 smithi078 ceph-mon[32429]: pgmap v1597: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:10:18.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:17 smithi107 ceph-mon[36485]: pgmap v1597: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:10:18.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:17 smithi106 ceph-mon[36505]: pgmap v1597: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:10:18.905 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2747 sec 2024-03-24T01:10:19.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2093.136335 secs 2024-03-24T01:10:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2093.136335 secs 2024-03-24T01:10:19.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2093.136335 secs 2024-03-24T01:10:19.905 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2748 sec 2024-03-24T01:10:20.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:19 smithi078 ceph-mon[32429]: pgmap v1598: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:10:20.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:19 smithi107 ceph-mon[36485]: pgmap v1598: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:10:20.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:19 smithi106 ceph-mon[36505]: pgmap v1598: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:10:20.905 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2749 sec 2024-03-24T01:10:21.906 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2750 sec 2024-03-24T01:10:22.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:21 smithi078 ceph-mon[32429]: pgmap v1599: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:10:22.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:21 smithi107 ceph-mon[36485]: pgmap v1599: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:10:22.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:21 smithi106 ceph-mon[36505]: pgmap v1599: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:10:22.905 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2751 sec 2024-03-24T01:10:23.906 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2752 sec 2024-03-24T01:10:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:23 smithi078 ceph-mon[32429]: pgmap v1600: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:10:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2098.136426 secs 2024-03-24T01:10:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:23 smithi107 ceph-mon[36485]: pgmap v1600: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:10:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2098.136426 secs 2024-03-24T01:10:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:23 smithi106 ceph-mon[36505]: pgmap v1600: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:10:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2098.136426 secs 2024-03-24T01:10:24.906 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2753 sec 2024-03-24T01:10:25.906 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2754 sec 2024-03-24T01:10:26.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:25 smithi078 ceph-mon[32429]: pgmap v1601: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:26.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:25 smithi107 ceph-mon[36485]: pgmap v1601: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:26.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:25 smithi106 ceph-mon[36505]: pgmap v1601: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:26.906 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2755 sec 2024-03-24T01:10:27.906 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2756 sec 2024-03-24T01:10:28.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:27 smithi078 ceph-mon[32429]: pgmap v1602: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:27 smithi107 ceph-mon[36485]: pgmap v1602: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:28.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:27 smithi106 ceph-mon[36505]: pgmap v1602: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:28.906 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2757 sec 2024-03-24T01:10:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2103.136445 secs 2024-03-24T01:10:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2103.136445 secs 2024-03-24T01:10:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2103.136445 secs 2024-03-24T01:10:29.906 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2758 sec 2024-03-24T01:10:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:29 smithi078 ceph-mon[32429]: pgmap v1603: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:30.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:29 smithi107 ceph-mon[36485]: pgmap v1603: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:30.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:29 smithi106 ceph-mon[36505]: pgmap v1603: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:30.906 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2759 sec 2024-03-24T01:10:31.906 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2760 sec 2024-03-24T01:10:32.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:31 smithi078 ceph-mon[32429]: pgmap v1604: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:32.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:31 smithi107 ceph-mon[36485]: pgmap v1604: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:32.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:31 smithi106 ceph-mon[36505]: pgmap v1604: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:32.906 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2761 sec 2024-03-24T01:10:33.907 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2762 sec 2024-03-24T01:10:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:33 smithi078 ceph-mon[32429]: pgmap v1605: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:34.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2108.136507 secs 2024-03-24T01:10:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:33 smithi107 ceph-mon[36485]: pgmap v1605: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2108.136507 secs 2024-03-24T01:10:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:33 smithi106 ceph-mon[36505]: pgmap v1605: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2108.136507 secs 2024-03-24T01:10:34.907 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2763 sec 2024-03-24T01:10:35.907 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2764 sec 2024-03-24T01:10:36.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:35 smithi078 ceph-mon[32429]: pgmap v1606: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:36.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:35 smithi107 ceph-mon[36485]: pgmap v1606: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:36.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:35 smithi106 ceph-mon[36505]: pgmap v1606: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:36.907 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2765 sec 2024-03-24T01:10:37.907 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2766 sec 2024-03-24T01:10:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:37 smithi078 ceph-mon[32429]: pgmap v1607: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:38.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:37 smithi107 ceph-mon[36485]: pgmap v1607: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:38.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:37 smithi106 ceph-mon[36505]: pgmap v1607: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:38.907 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2767 sec 2024-03-24T01:10:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2113.136586 secs 2024-03-24T01:10:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2113.136586 secs 2024-03-24T01:10:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2113.136586 secs 2024-03-24T01:10:39.907 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2768 sec 2024-03-24T01:10:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:39 smithi078 ceph-mon[32429]: pgmap v1608: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:39 smithi107 ceph-mon[36485]: pgmap v1608: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:39 smithi106 ceph-mon[36505]: pgmap v1608: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:40.907 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2769 sec 2024-03-24T01:10:41.907 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2770 sec 2024-03-24T01:10:42.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:41 smithi078 ceph-mon[32429]: pgmap v1609: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:42.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:41 smithi107 ceph-mon[36485]: pgmap v1609: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:42.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:41 smithi106 ceph-mon[36505]: pgmap v1609: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:42.907 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2771 sec 2024-03-24T01:10:43.907 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2772 sec 2024-03-24T01:10:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:43 smithi078 ceph-mon[32429]: pgmap v1610: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:44.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2118.136647 secs 2024-03-24T01:10:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:43 smithi107 ceph-mon[36485]: pgmap v1610: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:44.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2118.136647 secs 2024-03-24T01:10:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:43 smithi106 ceph-mon[36505]: pgmap v1610: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2118.136647 secs 2024-03-24T01:10:44.908 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2773 sec 2024-03-24T01:10:45.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:45 smithi078 ceph-mon[32429]: pgmap v1611: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:45.907 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2774 sec 2024-03-24T01:10:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:45 smithi107 ceph-mon[36485]: pgmap v1611: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:46.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:45 smithi106 ceph-mon[36505]: pgmap v1611: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:46.908 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2775 sec 2024-03-24T01:10:47.908 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2776 sec 2024-03-24T01:10:48.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:47 smithi078 ceph-mon[32429]: pgmap v1612: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:48.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:47 smithi107 ceph-mon[36485]: pgmap v1612: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:48.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:47 smithi106 ceph-mon[36505]: pgmap v1612: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:48.908 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2777 sec 2024-03-24T01:10:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2123.136710 secs 2024-03-24T01:10:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2123.136710 secs 2024-03-24T01:10:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2123.136710 secs 2024-03-24T01:10:49.908 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2778 sec 2024-03-24T01:10:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:49 smithi078 ceph-mon[32429]: pgmap v1613: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:50.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:49 smithi107 ceph-mon[36485]: pgmap v1613: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:49 smithi106 ceph-mon[36505]: pgmap v1613: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:50.908 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2779 sec 2024-03-24T01:10:51.908 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2780 sec 2024-03-24T01:10:52.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:51 smithi078 ceph-mon[32429]: pgmap v1614: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:52.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:51 smithi107 ceph-mon[36485]: pgmap v1614: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:52.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:51 smithi106 ceph-mon[36505]: pgmap v1614: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:52.908 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2781 sec 2024-03-24T01:10:53.908 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2782 sec 2024-03-24T01:10:54.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:53 smithi078 ceph-mon[32429]: pgmap v1615: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:54.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2128.136802 secs 2024-03-24T01:10:54.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:53 smithi107 ceph-mon[36485]: pgmap v1615: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:54.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2128.136802 secs 2024-03-24T01:10:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:53 smithi106 ceph-mon[36505]: pgmap v1615: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2128.136802 secs 2024-03-24T01:10:54.908 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2783 sec 2024-03-24T01:10:55.908 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2784 sec 2024-03-24T01:10:56.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:55 smithi078 ceph-mon[32429]: pgmap v1616: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:56.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:55 smithi107 ceph-mon[36485]: pgmap v1616: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:56.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:55 smithi106 ceph-mon[36505]: pgmap v1616: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:10:56.908 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2785 sec 2024-03-24T01:10:57.910 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2786 sec 2024-03-24T01:10:58.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:57 smithi078 ceph-mon[32429]: pgmap v1617: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:58.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:57 smithi107 ceph-mon[36485]: pgmap v1617: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:58.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:57 smithi106 ceph-mon[36505]: pgmap v1617: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:58.909 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2787 sec 2024-03-24T01:10:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2133.136888 secs 2024-03-24T01:10:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2133.136888 secs 2024-03-24T01:10:59.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2133.136888 secs 2024-03-24T01:10:59.909 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2788 sec 2024-03-24T01:11:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:10:59 smithi078 ceph-mon[32429]: pgmap v1618: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:00.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:10:59 smithi107 ceph-mon[36485]: pgmap v1618: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:00.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:10:59 smithi106 ceph-mon[36505]: pgmap v1618: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:00.909 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2789 sec 2024-03-24T01:11:01.909 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2790 sec 2024-03-24T01:11:02.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:01 smithi078 ceph-mon[32429]: pgmap v1619: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:02.100 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:01 smithi107 ceph-mon[36485]: pgmap v1619: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:02.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:01 smithi106 ceph-mon[36505]: pgmap v1619: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:02.909 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2791 sec 2024-03-24T01:11:03.909 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2792 sec 2024-03-24T01:11:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:03 smithi078 ceph-mon[32429]: pgmap v1620: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2138.136978 secs 2024-03-24T01:11:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:03 smithi107 ceph-mon[36485]: pgmap v1620: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2138.136978 secs 2024-03-24T01:11:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:03 smithi106 ceph-mon[36505]: pgmap v1620: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2138.136978 secs 2024-03-24T01:11:04.909 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2793 sec 2024-03-24T01:11:05.909 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2794 sec 2024-03-24T01:11:06.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:05 smithi078 ceph-mon[32429]: pgmap v1621: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:05 smithi107 ceph-mon[36485]: pgmap v1621: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:06.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:05 smithi106 ceph-mon[36505]: pgmap v1621: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:06.909 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2795 sec 2024-03-24T01:11:07.909 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2796 sec 2024-03-24T01:11:08.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:07 smithi078 ceph-mon[32429]: pgmap v1622: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:08.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:07 smithi107 ceph-mon[36485]: pgmap v1622: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:08.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:07 smithi106 ceph-mon[36505]: pgmap v1622: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:08.909 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2797 sec 2024-03-24T01:11:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2143.137042 secs 2024-03-24T01:11:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2143.137042 secs 2024-03-24T01:11:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2143.137042 secs 2024-03-24T01:11:09.909 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2798 sec 2024-03-24T01:11:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:09 smithi078 ceph-mon[32429]: pgmap v1623: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:10.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:09 smithi107 ceph-mon[36485]: pgmap v1623: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:10.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:09 smithi106 ceph-mon[36505]: pgmap v1623: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:10.910 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2799 sec 2024-03-24T01:11:11.910 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2800 sec 2024-03-24T01:11:12.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:11 smithi078 ceph-mon[32429]: pgmap v1624: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:12.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:11 smithi107 ceph-mon[36485]: pgmap v1624: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:12.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:11 smithi106 ceph-mon[36505]: pgmap v1624: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:12.910 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2801 sec 2024-03-24T01:11:13.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:12 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:11:13.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:12 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:11:13.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:12 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:11:13.910 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2802 sec 2024-03-24T01:11:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:13 smithi078 ceph-mon[32429]: pgmap v1625: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:14.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2148.137133 secs 2024-03-24T01:11:14.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:13 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:11:14.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:13 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:11:14.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:13 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:11:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:13 smithi107 ceph-mon[36485]: pgmap v1625: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2148.137133 secs 2024-03-24T01:11:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:13 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:11:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:13 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:11:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:13 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:11:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:13 smithi106 ceph-mon[36505]: pgmap v1625: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:14.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2148.137133 secs 2024-03-24T01:11:14.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:13 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:11:14.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:13 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:11:14.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:13 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:11:14.910 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2803 sec 2024-03-24T01:11:15.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:14 smithi078 ceph-mon[32429]: pgmap v1626: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:11:15.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:14 smithi078 ceph-mon[32429]: pgmap v1627: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:11:15.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:14 smithi107 ceph-mon[36485]: pgmap v1626: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:11:15.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:14 smithi107 ceph-mon[36485]: pgmap v1627: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:11:15.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:14 smithi106 ceph-mon[36505]: pgmap v1626: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:11:15.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:14 smithi106 ceph-mon[36505]: pgmap v1627: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:11:15.910 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2804 sec 2024-03-24T01:11:16.910 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2805 sec 2024-03-24T01:11:17.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:16 smithi078 ceph-mon[32429]: pgmap v1628: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:11:17.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:16 smithi107 ceph-mon[36485]: pgmap v1628: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:11:17.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:16 smithi106 ceph-mon[36505]: pgmap v1628: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:11:17.911 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2806 sec 2024-03-24T01:11:18.910 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2807 sec 2024-03-24T01:11:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:18 smithi078 ceph-mon[32429]: pgmap v1629: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:11:19.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2153.137149 secs 2024-03-24T01:11:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:18 smithi107 ceph-mon[36485]: pgmap v1629: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:11:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2153.137149 secs 2024-03-24T01:11:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:18 smithi106 ceph-mon[36505]: pgmap v1629: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:11:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2153.137149 secs 2024-03-24T01:11:19.910 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2808 sec 2024-03-24T01:11:20.910 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2809 sec 2024-03-24T01:11:21.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:20 smithi078 ceph-mon[32429]: pgmap v1630: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:11:21.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:20 smithi107 ceph-mon[36485]: pgmap v1630: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:11:21.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:20 smithi106 ceph-mon[36505]: pgmap v1630: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:11:21.910 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2810 sec 2024-03-24T01:11:22.911 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2811 sec 2024-03-24T01:11:23.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:22 smithi078 ceph-mon[32429]: pgmap v1631: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:11:23.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:22 smithi107 ceph-mon[36485]: pgmap v1631: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:11:23.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:22 smithi106 ceph-mon[36505]: pgmap v1631: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:11:23.911 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2812 sec 2024-03-24T01:11:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2158.137190 secs 2024-03-24T01:11:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2158.137190 secs 2024-03-24T01:11:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2158.137190 secs 2024-03-24T01:11:24.911 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2813 sec 2024-03-24T01:11:25.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:24 smithi078 ceph-mon[32429]: pgmap v1632: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:11:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:24 smithi107 ceph-mon[36485]: pgmap v1632: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:11:25.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:24 smithi106 ceph-mon[36505]: pgmap v1632: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:11:25.914 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2814 sec 2024-03-24T01:11:26.911 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2815 sec 2024-03-24T01:11:27.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:26 smithi078 ceph-mon[32429]: pgmap v1633: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:27.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:26 smithi107 ceph-mon[36485]: pgmap v1633: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:27.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:26 smithi106 ceph-mon[36505]: pgmap v1633: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:27.911 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2816 sec 2024-03-24T01:11:28.912 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2817 sec 2024-03-24T01:11:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:28 smithi078 ceph-mon[32429]: pgmap v1634: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:29.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2163.137279 secs 2024-03-24T01:11:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:28 smithi107 ceph-mon[36485]: pgmap v1634: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2163.137279 secs 2024-03-24T01:11:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:28 smithi106 ceph-mon[36505]: pgmap v1634: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2163.137279 secs 2024-03-24T01:11:29.911 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2818 sec 2024-03-24T01:11:30.911 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2819 sec 2024-03-24T01:11:31.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:30 smithi106 ceph-mon[36505]: pgmap v1635: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:31.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:30 smithi078 ceph-mon[32429]: pgmap v1635: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:31.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:30 smithi107 ceph-mon[36485]: pgmap v1635: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:31.911 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2820 sec 2024-03-24T01:11:32.912 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2821 sec 2024-03-24T01:11:33.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:32 smithi107 ceph-mon[36485]: pgmap v1636: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:33.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:32 smithi106 ceph-mon[36505]: pgmap v1636: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:33.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:32 smithi078 ceph-mon[32429]: pgmap v1636: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:33.912 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2822 sec 2024-03-24T01:11:34.161 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2168.137342 secs 2024-03-24T01:11:34.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2168.137342 secs 2024-03-24T01:11:34.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2168.137342 secs 2024-03-24T01:11:34.912 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2823 sec 2024-03-24T01:11:35.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:34 smithi106 ceph-mon[36505]: pgmap v1637: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:35.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:34 smithi078 ceph-mon[32429]: pgmap v1637: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:35.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:34 smithi107 ceph-mon[36485]: pgmap v1637: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:35.912 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2824 sec 2024-03-24T01:11:36.912 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2825 sec 2024-03-24T01:11:37.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:36 smithi106 ceph-mon[36505]: pgmap v1638: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:37.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:36 smithi078 ceph-mon[32429]: pgmap v1638: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:36 smithi107 ceph-mon[36485]: pgmap v1638: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:37.912 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2826 sec 2024-03-24T01:11:38.912 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2827 sec 2024-03-24T01:11:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:38 smithi106 ceph-mon[36505]: pgmap v1639: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2173.137404 secs 2024-03-24T01:11:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:38 smithi078 ceph-mon[32429]: pgmap v1639: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:39.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2173.137404 secs 2024-03-24T01:11:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:38 smithi107 ceph-mon[36485]: pgmap v1639: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2173.137404 secs 2024-03-24T01:11:39.912 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2828 sec 2024-03-24T01:11:40.912 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2829 sec 2024-03-24T01:11:41.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:40 smithi106 ceph-mon[36505]: pgmap v1640: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:41.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:40 smithi078 ceph-mon[32429]: pgmap v1640: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:40 smithi107 ceph-mon[36485]: pgmap v1640: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:41.912 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2830 sec 2024-03-24T01:11:42.912 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2831 sec 2024-03-24T01:11:43.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:42 smithi106 ceph-mon[36505]: pgmap v1641: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:42 smithi078 ceph-mon[32429]: pgmap v1641: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:43.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:42 smithi107 ceph-mon[36485]: pgmap v1641: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:43.912 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2832 sec 2024-03-24T01:11:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2178.137463 secs 2024-03-24T01:11:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2178.137463 secs 2024-03-24T01:11:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2178.137463 secs 2024-03-24T01:11:44.913 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2833 sec 2024-03-24T01:11:45.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:44 smithi106 ceph-mon[36505]: pgmap v1642: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:45.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:44 smithi078 ceph-mon[32429]: pgmap v1642: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:45.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:44 smithi107 ceph-mon[36485]: pgmap v1642: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:45.913 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2834 sec 2024-03-24T01:11:46.913 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2835 sec 2024-03-24T01:11:47.143 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:46 smithi078 ceph-mon[32429]: pgmap v1643: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:47.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:46 smithi106 ceph-mon[36505]: pgmap v1643: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:47.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:46 smithi107 ceph-mon[36485]: pgmap v1643: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:47.913 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2836 sec 2024-03-24T01:11:48.913 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2837 sec 2024-03-24T01:11:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:48 smithi106 ceph-mon[36505]: pgmap v1644: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2183.137479 secs 2024-03-24T01:11:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:48 smithi078 ceph-mon[32429]: pgmap v1644: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:49.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2183.137479 secs 2024-03-24T01:11:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:48 smithi107 ceph-mon[36485]: pgmap v1644: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2183.137479 secs 2024-03-24T01:11:49.913 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2838 sec 2024-03-24T01:11:50.913 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2839 sec 2024-03-24T01:11:51.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:50 smithi106 ceph-mon[36505]: pgmap v1645: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:51.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:50 smithi078 ceph-mon[32429]: pgmap v1645: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:51.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:50 smithi107 ceph-mon[36485]: pgmap v1645: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:51.913 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2840 sec 2024-03-24T01:11:52.913 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2841 sec 2024-03-24T01:11:53.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:52 smithi106 ceph-mon[36505]: pgmap v1646: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:53.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:52 smithi078 ceph-mon[32429]: pgmap v1646: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:53.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:52 smithi107 ceph-mon[36485]: pgmap v1646: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:53.913 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2842 sec 2024-03-24T01:11:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2188.137564 secs 2024-03-24T01:11:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2188.137564 secs 2024-03-24T01:11:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2188.137564 secs 2024-03-24T01:11:54.913 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2843 sec 2024-03-24T01:11:55.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:54 smithi106 ceph-mon[36505]: pgmap v1647: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:55.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:54 smithi078 ceph-mon[32429]: pgmap v1647: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:55.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:54 smithi107 ceph-mon[36485]: pgmap v1647: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:55.914 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2844 sec 2024-03-24T01:11:56.914 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2845 sec 2024-03-24T01:11:57.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:56 smithi106 ceph-mon[36505]: pgmap v1648: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:57.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:56 smithi078 ceph-mon[32429]: pgmap v1648: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:57.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:56 smithi107 ceph-mon[36485]: pgmap v1648: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:11:57.914 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2846 sec 2024-03-24T01:11:58.914 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2847 sec 2024-03-24T01:11:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:58 smithi106 ceph-mon[36505]: pgmap v1649: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:11:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2193.137653 secs 2024-03-24T01:11:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:58 smithi078 ceph-mon[32429]: pgmap v1649: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:11:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2193.137653 secs 2024-03-24T01:11:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:58 smithi107 ceph-mon[36485]: pgmap v1649: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:11:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2193.137653 secs 2024-03-24T01:11:59.914 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2848 sec 2024-03-24T01:12:00.914 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2849 sec 2024-03-24T01:12:01.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:00 smithi078 ceph-mon[32429]: pgmap v1650: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:00 smithi107 ceph-mon[36485]: pgmap v1650: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:00 smithi106 ceph-mon[36505]: pgmap v1650: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:01.914 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2850 sec 2024-03-24T01:12:02.914 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2851 sec 2024-03-24T01:12:03.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:02 smithi078 ceph-mon[32429]: pgmap v1651: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:03.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:02 smithi107 ceph-mon[36485]: pgmap v1651: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:03.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:02 smithi106 ceph-mon[36505]: pgmap v1651: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:03.914 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2852 sec 2024-03-24T01:12:04.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2198.137716 secs 2024-03-24T01:12:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2198.137716 secs 2024-03-24T01:12:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2198.137716 secs 2024-03-24T01:12:04.915 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2853 sec 2024-03-24T01:12:05.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:04 smithi078 ceph-mon[32429]: pgmap v1652: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:05.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:04 smithi107 ceph-mon[36485]: pgmap v1652: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:05.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:04 smithi106 ceph-mon[36505]: pgmap v1652: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:05.914 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2854 sec 2024-03-24T01:12:06.915 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2855 sec 2024-03-24T01:12:07.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:06 smithi078 ceph-mon[32429]: pgmap v1653: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:07.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:06 smithi107 ceph-mon[36485]: pgmap v1653: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:07.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:06 smithi106 ceph-mon[36505]: pgmap v1653: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:07.915 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2856 sec 2024-03-24T01:12:08.915 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2857 sec 2024-03-24T01:12:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:08 smithi078 ceph-mon[32429]: pgmap v1654: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:09.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2203.137776 secs 2024-03-24T01:12:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:08 smithi107 ceph-mon[36485]: pgmap v1654: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2203.137776 secs 2024-03-24T01:12:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:08 smithi106 ceph-mon[36505]: pgmap v1654: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2203.137776 secs 2024-03-24T01:12:09.915 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2858 sec 2024-03-24T01:12:10.916 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2859 sec 2024-03-24T01:12:11.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:10 smithi078 ceph-mon[32429]: pgmap v1655: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:11.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:10 smithi107 ceph-mon[36485]: pgmap v1655: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:11.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:10 smithi106 ceph-mon[36505]: pgmap v1655: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:11.915 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2860 sec 2024-03-24T01:12:12.915 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2861 sec 2024-03-24T01:12:13.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:12 smithi078 ceph-mon[32429]: pgmap v1656: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:13.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:12 smithi107 ceph-mon[36485]: pgmap v1656: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:13.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:12 smithi106 ceph-mon[36505]: pgmap v1656: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:13.915 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2862 sec 2024-03-24T01:12:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2208.137837 secs 2024-03-24T01:12:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:13 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:12:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2208.137837 secs 2024-03-24T01:12:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:13 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:12:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2208.137837 secs 2024-03-24T01:12:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:13 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:12:14.915 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2863 sec 2024-03-24T01:12:15.303 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:14 smithi106 ceph-mon[36505]: pgmap v1657: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:15.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:14 smithi078 ceph-mon[32429]: pgmap v1657: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:15.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:14 smithi107 ceph-mon[36485]: pgmap v1657: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:15.915 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2864 sec 2024-03-24T01:12:16.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:16 smithi106 ceph-mon[36505]: pgmap v1658: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:16.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:12:16.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:12:16.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:12:16.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:12:16.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:12:16.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:12:16.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:16 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T01:12:16.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:16 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T01:12:16.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:16 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T01:12:16.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:16 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T01:12:16.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:12:16.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:12:16.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:12:16.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:16 smithi078 ceph-mon[32429]: pgmap v1658: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:16.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:12:16.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:12:16.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:12:16.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:12:16.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:12:16.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:12:16.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:16 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T01:12:16.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:16 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T01:12:16.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:16 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T01:12:16.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:16 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T01:12:16.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:12:16.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:12:16.343 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:12:16.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:16 smithi107 ceph-mon[36485]: pgmap v1658: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:16.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:12:16.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:12:16.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:12:16.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:12:16.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:12:16.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:12:16.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:16 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T01:12:16.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:16 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T01:12:16.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.345 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:16 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T01:12:16.345 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:16 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T01:12:16.345 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.345 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.345 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.345 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:12:16.345 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:12:16.345 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:12:16.345 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:12:16.915 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2865 sec 2024-03-24T01:12:17.915 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2866 sec 2024-03-24T01:12:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:17 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T01:12:18.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:17 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T01:12:18.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:17 smithi078 ceph-mon[32429]: pgmap v1659: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T01:12:18.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:17 smithi078 ceph-mon[32429]: pgmap v1660: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:12:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:17 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T01:12:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:17 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T01:12:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:17 smithi107 ceph-mon[36485]: pgmap v1659: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T01:12:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:17 smithi107 ceph-mon[36485]: pgmap v1660: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:12:18.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:17 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T01:12:18.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:17 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T01:12:18.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:17 smithi106 ceph-mon[36505]: pgmap v1659: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T01:12:18.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:17 smithi106 ceph-mon[36505]: pgmap v1660: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:12:18.916 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2867 sec 2024-03-24T01:12:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2213.137877 secs 2024-03-24T01:12:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2213.137877 secs 2024-03-24T01:12:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2213.137877 secs 2024-03-24T01:12:19.916 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2868 sec 2024-03-24T01:12:20.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:19 smithi078 ceph-mon[32429]: pgmap v1661: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:12:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:19 smithi107 ceph-mon[36485]: pgmap v1661: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:12:20.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:19 smithi106 ceph-mon[36505]: pgmap v1661: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:12:20.916 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2869 sec 2024-03-24T01:12:21.916 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2870 sec 2024-03-24T01:12:22.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:21 smithi078 ceph-mon[32429]: pgmap v1662: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:12:22.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:21 smithi107 ceph-mon[36485]: pgmap v1662: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:12:22.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:21 smithi106 ceph-mon[36505]: pgmap v1662: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:12:22.916 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2871 sec 2024-03-24T01:12:23.916 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2872 sec 2024-03-24T01:12:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:23 smithi078 ceph-mon[32429]: pgmap v1663: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:12:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2218.137966 secs 2024-03-24T01:12:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:23 smithi107 ceph-mon[36485]: pgmap v1663: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:12:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2218.137966 secs 2024-03-24T01:12:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:23 smithi106 ceph-mon[36505]: pgmap v1663: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:12:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2218.137966 secs 2024-03-24T01:12:24.917 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2873 sec 2024-03-24T01:12:25.917 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2874 sec 2024-03-24T01:12:26.336 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:25 smithi106 ceph-mon[36505]: pgmap v1664: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:12:26.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:25 smithi078 ceph-mon[32429]: pgmap v1664: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:12:26.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:25 smithi107 ceph-mon[36485]: pgmap v1664: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:12:26.917 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2875 sec 2024-03-24T01:12:27.917 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2876 sec 2024-03-24T01:12:28.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:27 smithi078 ceph-mon[32429]: pgmap v1665: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:12:28.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:27 smithi107 ceph-mon[36485]: pgmap v1665: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:12:28.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:27 smithi106 ceph-mon[36505]: pgmap v1665: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:12:28.917 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2877 sec 2024-03-24T01:12:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2223.138055 secs 2024-03-24T01:12:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:28 smithi078 ceph-mon[32429]: pgmap v1666: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2223.138055 secs 2024-03-24T01:12:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:28 smithi107 ceph-mon[36485]: pgmap v1666: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2223.138055 secs 2024-03-24T01:12:29.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:28 smithi106 ceph-mon[36505]: pgmap v1666: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:29.917 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2878 sec 2024-03-24T01:12:30.917 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2879 sec 2024-03-24T01:12:31.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:31 smithi078 ceph-mon[32429]: pgmap v1667: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:31.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:31 smithi107 ceph-mon[36485]: pgmap v1667: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:31.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:31 smithi106 ceph-mon[36505]: pgmap v1667: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:31.917 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2880 sec 2024-03-24T01:12:32.917 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2881 sec 2024-03-24T01:12:33.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:33 smithi078 ceph-mon[32429]: pgmap v1668: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:33.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:33 smithi107 ceph-mon[36485]: pgmap v1668: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:33.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:33 smithi106 ceph-mon[36505]: pgmap v1668: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:33.917 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2882 sec 2024-03-24T01:12:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2228.138142 secs 2024-03-24T01:12:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2228.138142 secs 2024-03-24T01:12:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2228.138142 secs 2024-03-24T01:12:34.917 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2883 sec 2024-03-24T01:12:35.337 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:35 smithi107 ceph-mon[36485]: pgmap v1669: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:35.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:35 smithi078 ceph-mon[32429]: pgmap v1669: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:35.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:35 smithi106 ceph-mon[36505]: pgmap v1669: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:35.917 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2884 sec 2024-03-24T01:12:36.917 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2885 sec 2024-03-24T01:12:37.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:37 smithi078 ceph-mon[32429]: pgmap v1670: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:37 smithi107 ceph-mon[36485]: pgmap v1670: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:37 smithi106 ceph-mon[36505]: pgmap v1670: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:37.917 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2886 sec 2024-03-24T01:12:38.917 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2887 sec 2024-03-24T01:12:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2233.138232 secs 2024-03-24T01:12:39.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:39 smithi078 ceph-mon[32429]: pgmap v1671: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2233.138232 secs 2024-03-24T01:12:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:39 smithi107 ceph-mon[36485]: pgmap v1671: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2233.138232 secs 2024-03-24T01:12:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:39 smithi106 ceph-mon[36505]: pgmap v1671: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:39.918 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2888 sec 2024-03-24T01:12:40.918 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2889 sec 2024-03-24T01:12:41.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:41 smithi078 ceph-mon[32429]: pgmap v1672: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:41 smithi107 ceph-mon[36485]: pgmap v1672: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:41.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:41 smithi106 ceph-mon[36505]: pgmap v1672: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:41.918 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2890 sec 2024-03-24T01:12:42.918 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2891 sec 2024-03-24T01:12:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:43 smithi078 ceph-mon[32429]: pgmap v1673: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:43.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:43 smithi107 ceph-mon[36485]: pgmap v1673: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:43.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:43 smithi106 ceph-mon[36505]: pgmap v1673: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:43.918 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2892 sec 2024-03-24T01:12:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2238.138337 secs 2024-03-24T01:12:44.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2238.138337 secs 2024-03-24T01:12:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2238.138337 secs 2024-03-24T01:12:44.918 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2893 sec 2024-03-24T01:12:45.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:45 smithi106 ceph-mon[36505]: pgmap v1674: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:45.403 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:45 smithi107 ceph-mon[36485]: pgmap v1674: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:45.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:45 smithi078 ceph-mon[32429]: pgmap v1674: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:45.918 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2894 sec 2024-03-24T01:12:46.918 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2895 sec 2024-03-24T01:12:47.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:47 smithi106 ceph-mon[36505]: pgmap v1675: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:47.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:47 smithi078 ceph-mon[32429]: pgmap v1675: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:47 smithi107 ceph-mon[36485]: pgmap v1675: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:47.918 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2896 sec 2024-03-24T01:12:48.918 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2897 sec 2024-03-24T01:12:49.393 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2243.138421 secs 2024-03-24T01:12:49.393 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:49 smithi078 ceph-mon[32429]: pgmap v1676: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2243.138421 secs 2024-03-24T01:12:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:49 smithi106 ceph-mon[36505]: pgmap v1676: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2243.138421 secs 2024-03-24T01:12:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:49 smithi107 ceph-mon[36485]: pgmap v1676: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:49.918 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2898 sec 2024-03-24T01:12:50.918 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2899 sec 2024-03-24T01:12:51.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:51 smithi106 ceph-mon[36505]: pgmap v1677: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:51.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:51 smithi078 ceph-mon[32429]: pgmap v1677: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:51.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:51 smithi107 ceph-mon[36485]: pgmap v1677: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:51.919 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2900 sec 2024-03-24T01:12:52.919 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2901 sec 2024-03-24T01:12:53.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:53 smithi106 ceph-mon[36505]: pgmap v1678: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:53.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:53 smithi078 ceph-mon[32429]: pgmap v1678: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:53.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:53 smithi107 ceph-mon[36485]: pgmap v1678: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:53.919 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2902 sec 2024-03-24T01:12:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2248.138482 secs 2024-03-24T01:12:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2248.138482 secs 2024-03-24T01:12:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2248.138482 secs 2024-03-24T01:12:54.919 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2903 sec 2024-03-24T01:12:55.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:55 smithi106 ceph-mon[36505]: pgmap v1679: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:55.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:55 smithi078 ceph-mon[32429]: pgmap v1679: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:55.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:55 smithi107 ceph-mon[36485]: pgmap v1679: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:55.919 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2904 sec 2024-03-24T01:12:56.919 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2905 sec 2024-03-24T01:12:57.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:57 smithi106 ceph-mon[36505]: pgmap v1680: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:57.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:57 smithi078 ceph-mon[32429]: pgmap v1680: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:57.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:57 smithi107 ceph-mon[36485]: pgmap v1680: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:57.919 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2906 sec 2024-03-24T01:12:58.919 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2907 sec 2024-03-24T01:12:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2253.138567 secs 2024-03-24T01:12:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:12:59 smithi106 ceph-mon[36505]: pgmap v1681: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:59.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2253.138567 secs 2024-03-24T01:12:59.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:12:59 smithi078 ceph-mon[32429]: pgmap v1681: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2253.138567 secs 2024-03-24T01:12:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:12:59 smithi107 ceph-mon[36485]: pgmap v1681: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:12:59.919 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2908 sec 2024-03-24T01:13:00.919 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2909 sec 2024-03-24T01:13:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:01 smithi106 ceph-mon[36505]: pgmap v1682: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:01.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:01 smithi078 ceph-mon[32429]: pgmap v1682: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:01.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:01 smithi107 ceph-mon[36485]: pgmap v1682: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:01.919 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2910 sec 2024-03-24T01:13:02.920 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2911 sec 2024-03-24T01:13:03.396 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:03 smithi107 ceph-mon[36485]: pgmap v1683: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:03.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:03 smithi106 ceph-mon[36505]: pgmap v1683: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:03.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:03 smithi078 ceph-mon[32429]: pgmap v1683: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:03.920 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2912 sec 2024-03-24T01:13:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2258.138622 secs 2024-03-24T01:13:04.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2258.138622 secs 2024-03-24T01:13:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2258.138622 secs 2024-03-24T01:13:04.920 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2913 sec 2024-03-24T01:13:05.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:05 smithi106 ceph-mon[36505]: pgmap v1684: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:05.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:05 smithi078 ceph-mon[32429]: pgmap v1684: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:05.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:05 smithi107 ceph-mon[36485]: pgmap v1684: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:05.920 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2914 sec 2024-03-24T01:13:06.920 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2915 sec 2024-03-24T01:13:07.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:07 smithi106 ceph-mon[36505]: pgmap v1685: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:07.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:07 smithi078 ceph-mon[32429]: pgmap v1685: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:07.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:07 smithi107 ceph-mon[36485]: pgmap v1685: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:07.920 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2916 sec 2024-03-24T01:13:08.920 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2917 sec 2024-03-24T01:13:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2263.138684 secs 2024-03-24T01:13:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:09 smithi106 ceph-mon[36505]: pgmap v1686: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2263.138684 secs 2024-03-24T01:13:09.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:09 smithi078 ceph-mon[32429]: pgmap v1686: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2263.138684 secs 2024-03-24T01:13:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:09 smithi107 ceph-mon[36485]: pgmap v1686: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:09.920 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2918 sec 2024-03-24T01:13:10.920 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2919 sec 2024-03-24T01:13:11.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:11 smithi106 ceph-mon[36505]: pgmap v1687: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:11.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:11 smithi078 ceph-mon[32429]: pgmap v1687: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:11.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:11 smithi107 ceph-mon[36485]: pgmap v1687: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:11.920 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2920 sec 2024-03-24T01:13:12.921 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2921 sec 2024-03-24T01:13:13.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:13 smithi106 ceph-mon[36505]: pgmap v1688: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:13.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:13 smithi078 ceph-mon[32429]: pgmap v1688: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:13.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:13 smithi107 ceph-mon[36485]: pgmap v1688: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:13.920 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2922 sec 2024-03-24T01:13:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2268.138744 secs 2024-03-24T01:13:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2268.138744 secs 2024-03-24T01:13:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2268.138744 secs 2024-03-24T01:13:14.920 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2923 sec 2024-03-24T01:13:15.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:15 smithi078 ceph-mon[32429]: pgmap v1689: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:15.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:15 smithi107 ceph-mon[36485]: pgmap v1689: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:15.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:15 smithi106 ceph-mon[36505]: pgmap v1689: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:15.921 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2924 sec 2024-03-24T01:13:16.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:13:16.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:13:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:13:16.921 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2925 sec 2024-03-24T01:13:17.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:17 smithi106 ceph-mon[36505]: pgmap v1690: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:17.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:17 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:13:17.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:17 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:13:17.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:17 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:13:17.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:17 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:13:17.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:17 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:13:17.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:17 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:13:17.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:17 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:13:17.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:17 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:13:17.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:17 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:13:17.921 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2926 sec 2024-03-24T01:13:18.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:17 smithi078 ceph-mon[32429]: pgmap v1690: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:18.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:17 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:13:18.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:17 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:13:18.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:17 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:13:18.093 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:17 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:13:18.093 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:17 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:13:18.093 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:17 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:13:18.093 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:17 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:13:18.093 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:17 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:13:18.093 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:17 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:13:18.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:17 smithi107 ceph-mon[36485]: pgmap v1690: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:18.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:17 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:13:18.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:17 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:13:18.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:17 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:13:18.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:17 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:13:18.096 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:17 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:13:18.096 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:17 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:13:18.096 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:17 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:13:18.096 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:17 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:13:18.096 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:17 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:13:18.921 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2927 sec 2024-03-24T01:13:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:18 smithi078 ceph-mon[32429]: pgmap v1691: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T01:13:19.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:18 smithi078 ceph-mon[32429]: pgmap v1692: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:13:19.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2273.138808 secs 2024-03-24T01:13:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:18 smithi107 ceph-mon[36485]: pgmap v1691: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T01:13:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:18 smithi107 ceph-mon[36485]: pgmap v1692: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:13:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2273.138808 secs 2024-03-24T01:13:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:18 smithi106 ceph-mon[36505]: pgmap v1691: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T01:13:19.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:18 smithi106 ceph-mon[36505]: pgmap v1692: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:13:19.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2273.138808 secs 2024-03-24T01:13:19.921 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2928 sec 2024-03-24T01:13:20.921 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2929 sec 2024-03-24T01:13:21.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:20 smithi078 ceph-mon[32429]: pgmap v1693: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:13:21.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:20 smithi107 ceph-mon[36485]: pgmap v1693: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:13:21.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:20 smithi106 ceph-mon[36505]: pgmap v1693: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:13:21.921 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2930 sec 2024-03-24T01:13:22.921 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2931 sec 2024-03-24T01:13:23.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:22 smithi078 ceph-mon[32429]: pgmap v1694: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:13:23.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:22 smithi107 ceph-mon[36485]: pgmap v1694: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:13:23.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:22 smithi106 ceph-mon[36505]: pgmap v1694: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:13:23.921 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2932 sec 2024-03-24T01:13:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2278.138896 secs 2024-03-24T01:13:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2278.138896 secs 2024-03-24T01:13:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2278.138896 secs 2024-03-24T01:13:24.921 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2933 sec 2024-03-24T01:13:25.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:24 smithi078 ceph-mon[32429]: pgmap v1695: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:13:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:24 smithi107 ceph-mon[36485]: pgmap v1695: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:13:25.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:24 smithi106 ceph-mon[36505]: pgmap v1695: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:13:25.922 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2934 sec 2024-03-24T01:13:26.922 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2935 sec 2024-03-24T01:13:27.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:26 smithi078 ceph-mon[32429]: pgmap v1696: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:13:27.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:26 smithi107 ceph-mon[36485]: pgmap v1696: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:13:27.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:26 smithi106 ceph-mon[36505]: pgmap v1696: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:13:27.922 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2936 sec 2024-03-24T01:13:28.922 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2937 sec 2024-03-24T01:13:29.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:28 smithi078 ceph-mon[32429]: pgmap v1697: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:13:29.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2283.138957 secs 2024-03-24T01:13:29.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:28 smithi107 ceph-mon[36485]: pgmap v1697: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:13:29.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2283.138957 secs 2024-03-24T01:13:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:28 smithi106 ceph-mon[36505]: pgmap v1697: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:13:29.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2283.138957 secs 2024-03-24T01:13:29.923 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2938 sec 2024-03-24T01:13:30.922 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2939 sec 2024-03-24T01:13:31.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:30 smithi078 ceph-mon[32429]: pgmap v1698: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:31.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:30 smithi107 ceph-mon[36485]: pgmap v1698: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:31.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:30 smithi106 ceph-mon[36505]: pgmap v1698: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:31.922 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2940 sec 2024-03-24T01:13:32.922 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2941 sec 2024-03-24T01:13:33.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:32 smithi078 ceph-mon[32429]: pgmap v1699: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:33.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:32 smithi107 ceph-mon[36485]: pgmap v1699: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:33.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:32 smithi106 ceph-mon[36505]: pgmap v1699: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:33.922 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2942 sec 2024-03-24T01:13:33.994 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2288.139045 secs 2024-03-24T01:13:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2288.139045 secs 2024-03-24T01:13:34.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2288.139045 secs 2024-03-24T01:13:34.922 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2943 sec 2024-03-24T01:13:35.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:34 smithi078 ceph-mon[32429]: pgmap v1700: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:35.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:34 smithi107 ceph-mon[36485]: pgmap v1700: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:35.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:34 smithi106 ceph-mon[36505]: pgmap v1700: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:35.922 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2944 sec 2024-03-24T01:13:36.923 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2945 sec 2024-03-24T01:13:37.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:36 smithi078 ceph-mon[32429]: pgmap v1701: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:37.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:36 smithi107 ceph-mon[36485]: pgmap v1701: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:37.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:36 smithi106 ceph-mon[36505]: pgmap v1701: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:37.923 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2946 sec 2024-03-24T01:13:38.923 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2947 sec 2024-03-24T01:13:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:38 smithi078 ceph-mon[32429]: pgmap v1702: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:39.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2293.139132 secs 2024-03-24T01:13:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:38 smithi107 ceph-mon[36485]: pgmap v1702: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2293.139132 secs 2024-03-24T01:13:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:38 smithi106 ceph-mon[36505]: pgmap v1702: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2293.139132 secs 2024-03-24T01:13:39.923 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2948 sec 2024-03-24T01:13:40.923 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2949 sec 2024-03-24T01:13:41.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:40 smithi078 ceph-mon[32429]: pgmap v1703: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:41.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:40 smithi107 ceph-mon[36485]: pgmap v1703: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:41.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:40 smithi106 ceph-mon[36505]: pgmap v1703: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:41.923 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2950 sec 2024-03-24T01:13:42.923 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2951 sec 2024-03-24T01:13:43.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:42 smithi078 ceph-mon[32429]: pgmap v1704: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:43.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:42 smithi107 ceph-mon[36485]: pgmap v1704: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:43.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:42 smithi106 ceph-mon[36505]: pgmap v1704: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:43.923 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2952 sec 2024-03-24T01:13:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2298.139219 secs 2024-03-24T01:13:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2298.139219 secs 2024-03-24T01:13:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2298.139219 secs 2024-03-24T01:13:44.923 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2953 sec 2024-03-24T01:13:45.053 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:44 smithi106 ceph-mon[36505]: pgmap v1705: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:45.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:44 smithi078 ceph-mon[32429]: pgmap v1705: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:45.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:44 smithi107 ceph-mon[36485]: pgmap v1705: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:45.923 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2954 sec 2024-03-24T01:13:46.923 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2955 sec 2024-03-24T01:13:47.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:46 smithi078 ceph-mon[32429]: pgmap v1706: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:47.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:46 smithi107 ceph-mon[36485]: pgmap v1706: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:47.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:46 smithi106 ceph-mon[36505]: pgmap v1706: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:47.924 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2956 sec 2024-03-24T01:13:48.924 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2957 sec 2024-03-24T01:13:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:48 smithi078 ceph-mon[32429]: pgmap v1707: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:49.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2303.139309 secs 2024-03-24T01:13:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:48 smithi107 ceph-mon[36485]: pgmap v1707: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2303.139309 secs 2024-03-24T01:13:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:48 smithi106 ceph-mon[36505]: pgmap v1707: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2303.139309 secs 2024-03-24T01:13:49.924 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2958 sec 2024-03-24T01:13:50.924 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2959 sec 2024-03-24T01:13:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:50 smithi078 ceph-mon[32429]: pgmap v1708: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:51.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:50 smithi107 ceph-mon[36485]: pgmap v1708: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:51.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:50 smithi106 ceph-mon[36505]: pgmap v1708: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:51.924 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2960 sec 2024-03-24T01:13:52.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:52 smithi078 ceph-mon[32429]: pgmap v1709: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:52.924 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2961 sec 2024-03-24T01:13:53.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:52 smithi107 ceph-mon[36485]: pgmap v1709: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:53.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:52 smithi106 ceph-mon[36505]: pgmap v1709: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:53.924 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2962 sec 2024-03-24T01:13:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2308.139393 secs 2024-03-24T01:13:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2308.139393 secs 2024-03-24T01:13:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2308.139393 secs 2024-03-24T01:13:54.924 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2963 sec 2024-03-24T01:13:55.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:54 smithi078 ceph-mon[32429]: pgmap v1710: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:55.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:54 smithi107 ceph-mon[36485]: pgmap v1710: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:55.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:54 smithi106 ceph-mon[36505]: pgmap v1710: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:55.924 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2964 sec 2024-03-24T01:13:56.925 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2965 sec 2024-03-24T01:13:57.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:56 smithi078 ceph-mon[32429]: pgmap v1711: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:56 smithi107 ceph-mon[36485]: pgmap v1711: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:57.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:56 smithi106 ceph-mon[36505]: pgmap v1711: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:13:57.925 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2966 sec 2024-03-24T01:13:58.925 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2967 sec 2024-03-24T01:13:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:58 smithi078 ceph-mon[32429]: pgmap v1712: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:59.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:13:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2313.139453 secs 2024-03-24T01:13:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:58 smithi107 ceph-mon[36485]: pgmap v1712: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:13:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2313.139453 secs 2024-03-24T01:13:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:58 smithi106 ceph-mon[36505]: pgmap v1712: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:13:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2313.139453 secs 2024-03-24T01:13:59.925 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2968 sec 2024-03-24T01:14:00.925 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2969 sec 2024-03-24T01:14:01.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:00 smithi078 ceph-mon[32429]: pgmap v1713: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:01.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:00 smithi107 ceph-mon[36485]: pgmap v1713: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:01.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:00 smithi106 ceph-mon[36505]: pgmap v1713: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:01.925 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2970 sec 2024-03-24T01:14:02.925 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2971 sec 2024-03-24T01:14:03.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:02 smithi078 ceph-mon[32429]: pgmap v1714: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:03.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:02 smithi107 ceph-mon[36485]: pgmap v1714: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:03.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:02 smithi106 ceph-mon[36505]: pgmap v1714: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:03.925 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2972 sec 2024-03-24T01:14:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2318.139514 secs 2024-03-24T01:14:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2318.139514 secs 2024-03-24T01:14:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2318.139514 secs 2024-03-24T01:14:04.925 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2973 sec 2024-03-24T01:14:05.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:04 smithi078 ceph-mon[32429]: pgmap v1715: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:05.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:04 smithi107 ceph-mon[36485]: pgmap v1715: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:05.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:04 smithi106 ceph-mon[36505]: pgmap v1715: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:05.925 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2974 sec 2024-03-24T01:14:06.925 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2975 sec 2024-03-24T01:14:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:06 smithi078 ceph-mon[32429]: pgmap v1716: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:06 smithi107 ceph-mon[36485]: pgmap v1716: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:07.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:06 smithi106 ceph-mon[36505]: pgmap v1716: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:07.925 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2976 sec 2024-03-24T01:14:08.926 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2977 sec 2024-03-24T01:14:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:08 smithi078 ceph-mon[32429]: pgmap v1717: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2323.139607 secs 2024-03-24T01:14:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:08 smithi107 ceph-mon[36485]: pgmap v1717: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2323.139607 secs 2024-03-24T01:14:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:08 smithi106 ceph-mon[36505]: pgmap v1717: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2323.139607 secs 2024-03-24T01:14:09.925 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2978 sec 2024-03-24T01:14:10.925 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2979 sec 2024-03-24T01:14:11.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:10 smithi078 ceph-mon[32429]: pgmap v1718: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:11.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:10 smithi107 ceph-mon[36485]: pgmap v1718: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:11.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:10 smithi106 ceph-mon[36505]: pgmap v1718: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:11.926 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2980 sec 2024-03-24T01:14:12.926 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2981 sec 2024-03-24T01:14:13.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:12 smithi078 ceph-mon[32429]: pgmap v1719: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:13.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:12 smithi107 ceph-mon[36485]: pgmap v1719: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:13.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:12 smithi106 ceph-mon[36505]: pgmap v1719: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:13.926 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2982 sec 2024-03-24T01:14:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2328.139667 secs 2024-03-24T01:14:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2328.139667 secs 2024-03-24T01:14:14.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2328.139667 secs 2024-03-24T01:14:14.926 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2983 sec 2024-03-24T01:14:15.053 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:14 smithi106 ceph-mon[36505]: pgmap v1720: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:15.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:14 smithi078 ceph-mon[32429]: pgmap v1720: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:15.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:14 smithi107 ceph-mon[36485]: pgmap v1720: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:15.926 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2984 sec 2024-03-24T01:14:16.926 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2985 sec 2024-03-24T01:14:17.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:16 smithi078 ceph-mon[32429]: pgmap v1721: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:17.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:16 smithi107 ceph-mon[36485]: pgmap v1721: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:17.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:16 smithi106 ceph-mon[36505]: pgmap v1721: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:17.926 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2986 sec 2024-03-24T01:14:18.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:17 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:14:18.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:17 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:14:18.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:17 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:14:18.926 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2987 sec 2024-03-24T01:14:19.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:18 smithi078 ceph-mon[32429]: pgmap v1722: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:19.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2333.139737 secs 2024-03-24T01:14:19.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:18 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:14:19.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:18 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:14:19.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:18 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:14:19.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:18 smithi107 ceph-mon[36485]: pgmap v1722: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:19.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2333.139737 secs 2024-03-24T01:14:19.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:18 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:14:19.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:18 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:14:19.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:18 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:14:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:18 smithi106 ceph-mon[36505]: pgmap v1722: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2333.139737 secs 2024-03-24T01:14:19.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:18 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:14:19.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:18 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:14:19.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:18 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:14:19.926 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2988 sec 2024-03-24T01:14:20.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:19 smithi078 ceph-mon[32429]: pgmap v1723: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:14:20.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:19 smithi078 ceph-mon[32429]: pgmap v1724: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:14:20.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:19 smithi107 ceph-mon[36485]: pgmap v1723: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:14:20.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:19 smithi107 ceph-mon[36485]: pgmap v1724: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:14:20.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:19 smithi106 ceph-mon[36505]: pgmap v1723: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:14:20.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:19 smithi106 ceph-mon[36505]: pgmap v1724: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:14:20.926 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2989 sec 2024-03-24T01:14:21.926 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2990 sec 2024-03-24T01:14:22.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:21 smithi078 ceph-mon[32429]: pgmap v1725: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:14:22.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:21 smithi107 ceph-mon[36485]: pgmap v1725: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:14:22.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:21 smithi106 ceph-mon[36505]: pgmap v1725: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:14:22.926 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2991 sec 2024-03-24T01:14:23.927 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2992 sec 2024-03-24T01:14:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:23 smithi078 ceph-mon[32429]: pgmap v1726: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:14:24.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2338.139798 secs 2024-03-24T01:14:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:23 smithi107 ceph-mon[36485]: pgmap v1726: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:14:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2338.139798 secs 2024-03-24T01:14:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:23 smithi106 ceph-mon[36505]: pgmap v1726: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:14:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2338.139798 secs 2024-03-24T01:14:24.926 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2993 sec 2024-03-24T01:14:25.926 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2994 sec 2024-03-24T01:14:26.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:25 smithi078 ceph-mon[32429]: pgmap v1727: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:14:26.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:25 smithi107 ceph-mon[36485]: pgmap v1727: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:14:26.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:25 smithi106 ceph-mon[36505]: pgmap v1727: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:14:26.927 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2995 sec 2024-03-24T01:14:27.927 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2996 sec 2024-03-24T01:14:28.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:27 smithi078 ceph-mon[32429]: pgmap v1728: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:14:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:27 smithi107 ceph-mon[36485]: pgmap v1728: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:14:28.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:27 smithi106 ceph-mon[36505]: pgmap v1728: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:14:28.927 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2997 sec 2024-03-24T01:14:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2343.139851 secs 2024-03-24T01:14:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2343.139851 secs 2024-03-24T01:14:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2343.139851 secs 2024-03-24T01:14:29.927 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2998 sec 2024-03-24T01:14:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:29 smithi078 ceph-mon[32429]: pgmap v1729: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:14:30.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:29 smithi107 ceph-mon[36485]: pgmap v1729: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:14:30.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:29 smithi106 ceph-mon[36505]: pgmap v1729: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:14:30.927 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 2999 sec 2024-03-24T01:14:31.927 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3000 sec 2024-03-24T01:14:32.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:31 smithi078 ceph-mon[32429]: pgmap v1730: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:14:32.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:31 smithi107 ceph-mon[36485]: pgmap v1730: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:14:32.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:31 smithi106 ceph-mon[36505]: pgmap v1730: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:14:32.927 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3001 sec 2024-03-24T01:14:33.927 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3002 sec 2024-03-24T01:14:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:33 smithi078 ceph-mon[32429]: pgmap v1731: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:14:34.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2348.139937 secs 2024-03-24T01:14:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:33 smithi107 ceph-mon[36485]: pgmap v1731: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:14:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2348.139937 secs 2024-03-24T01:14:34.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:33 smithi106 ceph-mon[36505]: pgmap v1731: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:14:34.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2348.139937 secs 2024-03-24T01:14:34.927 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3003 sec 2024-03-24T01:14:35.927 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3004 sec 2024-03-24T01:14:36.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:35 smithi078 ceph-mon[32429]: pgmap v1732: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:14:36.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:35 smithi107 ceph-mon[36485]: pgmap v1732: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:14:36.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:35 smithi106 ceph-mon[36505]: pgmap v1732: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:14:36.927 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3005 sec 2024-03-24T01:14:37.927 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3006 sec 2024-03-24T01:14:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:37 smithi078 ceph-mon[32429]: pgmap v1733: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:14:38.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:37 smithi107 ceph-mon[36485]: pgmap v1733: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:14:38.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:37 smithi106 ceph-mon[36505]: pgmap v1733: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:14:38.927 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3007 sec 2024-03-24T01:14:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2353.140025 secs 2024-03-24T01:14:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2353.140025 secs 2024-03-24T01:14:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2353.140025 secs 2024-03-24T01:14:39.927 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3008 sec 2024-03-24T01:14:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:39 smithi078 ceph-mon[32429]: pgmap v1734: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:39 smithi107 ceph-mon[36485]: pgmap v1734: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:39 smithi106 ceph-mon[36505]: pgmap v1734: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:40.928 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3009 sec 2024-03-24T01:14:41.928 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3010 sec 2024-03-24T01:14:42.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:41 smithi078 ceph-mon[32429]: pgmap v1735: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:42.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:41 smithi107 ceph-mon[36485]: pgmap v1735: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:42.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:41 smithi106 ceph-mon[36505]: pgmap v1735: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:42.928 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3011 sec 2024-03-24T01:14:43.928 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3012 sec 2024-03-24T01:14:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:43 smithi078 ceph-mon[32429]: pgmap v1736: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2358.140118 secs 2024-03-24T01:14:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:43 smithi107 ceph-mon[36485]: pgmap v1736: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2358.140118 secs 2024-03-24T01:14:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:43 smithi106 ceph-mon[36505]: pgmap v1736: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2358.140118 secs 2024-03-24T01:14:44.928 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3013 sec 2024-03-24T01:14:45.928 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3014 sec 2024-03-24T01:14:46.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:45 smithi078 ceph-mon[32429]: pgmap v1737: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:45 smithi107 ceph-mon[36485]: pgmap v1737: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:46.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:45 smithi106 ceph-mon[36505]: pgmap v1737: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:46.928 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3015 sec 2024-03-24T01:14:47.928 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3016 sec 2024-03-24T01:14:48.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:47 smithi078 ceph-mon[32429]: pgmap v1738: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:48.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:47 smithi107 ceph-mon[36485]: pgmap v1738: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:48.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:47 smithi106 ceph-mon[36505]: pgmap v1738: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:48.929 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3017 sec 2024-03-24T01:14:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2363.140161 secs 2024-03-24T01:14:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2363.140161 secs 2024-03-24T01:14:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2363.140161 secs 2024-03-24T01:14:49.929 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3018 sec 2024-03-24T01:14:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:49 smithi106 ceph-mon[36505]: pgmap v1739: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:50.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:49 smithi078 ceph-mon[32429]: pgmap v1739: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:49 smithi107 ceph-mon[36485]: pgmap v1739: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:50.929 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3019 sec 2024-03-24T01:14:51.929 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3020 sec 2024-03-24T01:14:52.143 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:51 smithi078 ceph-mon[32429]: pgmap v1740: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:52.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:51 smithi106 ceph-mon[36505]: pgmap v1740: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:52.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:51 smithi107 ceph-mon[36485]: pgmap v1740: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:52.929 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3021 sec 2024-03-24T01:14:53.929 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3022 sec 2024-03-24T01:14:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:53 smithi106 ceph-mon[36505]: pgmap v1741: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2368.140249 secs 2024-03-24T01:14:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:53 smithi078 ceph-mon[32429]: pgmap v1741: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2368.140249 secs 2024-03-24T01:14:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:53 smithi107 ceph-mon[36485]: pgmap v1741: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2368.140249 secs 2024-03-24T01:14:54.929 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3023 sec 2024-03-24T01:14:55.929 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3024 sec 2024-03-24T01:14:56.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:55 smithi106 ceph-mon[36505]: pgmap v1742: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:56.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:55 smithi078 ceph-mon[32429]: pgmap v1742: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:56.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:55 smithi107 ceph-mon[36485]: pgmap v1742: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:14:56.929 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3025 sec 2024-03-24T01:14:57.929 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3026 sec 2024-03-24T01:14:58.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:57 smithi106 ceph-mon[36505]: pgmap v1743: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:58.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:57 smithi078 ceph-mon[32429]: pgmap v1743: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:57 smithi107 ceph-mon[36485]: pgmap v1743: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:58.929 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3027 sec 2024-03-24T01:14:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2373.140322 secs 2024-03-24T01:14:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2373.140322 secs 2024-03-24T01:14:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2373.140322 secs 2024-03-24T01:14:59.930 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3028 sec 2024-03-24T01:15:00.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:14:59 smithi106 ceph-mon[36505]: pgmap v1744: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:00.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:14:59 smithi078 ceph-mon[32429]: pgmap v1744: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:00.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:14:59 smithi107 ceph-mon[36485]: pgmap v1744: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:00.930 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3029 sec 2024-03-24T01:15:01.930 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3030 sec 2024-03-24T01:15:02.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:01 smithi106 ceph-mon[36505]: pgmap v1745: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:02.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:01 smithi078 ceph-mon[32429]: pgmap v1745: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:02.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:01 smithi107 ceph-mon[36485]: pgmap v1745: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:02.930 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3031 sec 2024-03-24T01:15:03.930 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3032 sec 2024-03-24T01:15:04.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:03 smithi107 ceph-mon[36485]: pgmap v1746: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:04.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2378.140405 secs 2024-03-24T01:15:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:03 smithi106 ceph-mon[36505]: pgmap v1746: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2378.140405 secs 2024-03-24T01:15:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:03 smithi078 ceph-mon[32429]: pgmap v1746: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:04.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2378.140405 secs 2024-03-24T01:15:04.930 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3033 sec 2024-03-24T01:15:05.930 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3034 sec 2024-03-24T01:15:06.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:05 smithi106 ceph-mon[36505]: pgmap v1747: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:06.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:05 smithi078 ceph-mon[32429]: pgmap v1747: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:06.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:05 smithi107 ceph-mon[36485]: pgmap v1747: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:06.930 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3035 sec 2024-03-24T01:15:07.930 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3036 sec 2024-03-24T01:15:08.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:07 smithi106 ceph-mon[36505]: pgmap v1748: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:08.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:07 smithi078 ceph-mon[32429]: pgmap v1748: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:08.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:07 smithi107 ceph-mon[36485]: pgmap v1748: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:08.930 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3037 sec 2024-03-24T01:15:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2383.140490 secs 2024-03-24T01:15:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2383.140490 secs 2024-03-24T01:15:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2383.140490 secs 2024-03-24T01:15:09.930 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3038 sec 2024-03-24T01:15:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:09 smithi078 ceph-mon[32429]: pgmap v1749: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:10.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:09 smithi106 ceph-mon[36505]: pgmap v1749: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:10.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:09 smithi107 ceph-mon[36485]: pgmap v1749: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:10.931 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3039 sec 2024-03-24T01:15:11.931 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3040 sec 2024-03-24T01:15:12.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:11 smithi106 ceph-mon[36505]: pgmap v1750: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:12.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:11 smithi078 ceph-mon[32429]: pgmap v1750: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:12.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:11 smithi107 ceph-mon[36485]: pgmap v1750: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:12.933 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3041 sec 2024-03-24T01:15:13.931 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3042 sec 2024-03-24T01:15:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:13 smithi106 ceph-mon[36505]: pgmap v1751: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2388.140576 secs 2024-03-24T01:15:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:13 smithi078 ceph-mon[32429]: pgmap v1751: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2388.140576 secs 2024-03-24T01:15:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:13 smithi107 ceph-mon[36485]: pgmap v1751: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2388.140576 secs 2024-03-24T01:15:14.931 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3043 sec 2024-03-24T01:15:15.931 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3044 sec 2024-03-24T01:15:16.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:15 smithi106 ceph-mon[36505]: pgmap v1752: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:16.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:15 smithi078 ceph-mon[32429]: pgmap v1752: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:16.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:15 smithi107 ceph-mon[36485]: pgmap v1752: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:16.931 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3045 sec 2024-03-24T01:15:17.931 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3046 sec 2024-03-24T01:15:18.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:17 smithi106 ceph-mon[36505]: pgmap v1753: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:17 smithi078 ceph-mon[32429]: pgmap v1753: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:17 smithi107 ceph-mon[36485]: pgmap v1753: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:18.931 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3047 sec 2024-03-24T01:15:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2393.140637 secs 2024-03-24T01:15:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:18 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:15:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:18 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:15:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:18 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:15:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:18 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:15:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2393.140637 secs 2024-03-24T01:15:19.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:18 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:15:19.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:18 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:15:19.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:18 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:15:19.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:18 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:15:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2393.140637 secs 2024-03-24T01:15:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:18 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:15:19.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:18 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:15:19.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:18 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:15:19.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:18 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:15:19.931 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3048 sec 2024-03-24T01:15:20.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:19 smithi078 ceph-mon[32429]: pgmap v1754: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:20.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:19 smithi078 ceph-mon[32429]: pgmap v1755: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:15:20.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:19 smithi078 ceph-mon[32429]: pgmap v1756: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:15:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:19 smithi107 ceph-mon[36485]: pgmap v1754: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:19 smithi107 ceph-mon[36485]: pgmap v1755: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:15:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:19 smithi107 ceph-mon[36485]: pgmap v1756: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:15:20.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:19 smithi106 ceph-mon[36505]: pgmap v1754: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:20.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:19 smithi106 ceph-mon[36505]: pgmap v1755: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:15:20.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:19 smithi106 ceph-mon[36505]: pgmap v1756: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:15:20.931 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3049 sec 2024-03-24T01:15:21.931 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3050 sec 2024-03-24T01:15:22.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:21 smithi078 ceph-mon[32429]: pgmap v1757: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:15:22.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:21 smithi107 ceph-mon[36485]: pgmap v1757: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:15:22.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:21 smithi106 ceph-mon[36505]: pgmap v1757: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:15:22.931 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3051 sec 2024-03-24T01:15:23.931 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3052 sec 2024-03-24T01:15:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:23 smithi078 ceph-mon[32429]: pgmap v1758: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:15:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2398.140687 secs 2024-03-24T01:15:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:23 smithi107 ceph-mon[36485]: pgmap v1758: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:15:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2398.140687 secs 2024-03-24T01:15:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:23 smithi106 ceph-mon[36505]: pgmap v1758: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:15:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2398.140687 secs 2024-03-24T01:15:24.931 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3053 sec 2024-03-24T01:15:25.932 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3054 sec 2024-03-24T01:15:26.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:25 smithi078 ceph-mon[32429]: pgmap v1759: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:15:26.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:25 smithi107 ceph-mon[36485]: pgmap v1759: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:15:26.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:25 smithi106 ceph-mon[36505]: pgmap v1759: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:15:26.932 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3055 sec 2024-03-24T01:15:27.932 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3056 sec 2024-03-24T01:15:28.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:27 smithi078 ceph-mon[32429]: pgmap v1760: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:15:28.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:27 smithi107 ceph-mon[36485]: pgmap v1760: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:15:28.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:27 smithi106 ceph-mon[36505]: pgmap v1760: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:15:28.932 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3057 sec 2024-03-24T01:15:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2403.140748 secs 2024-03-24T01:15:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2403.140748 secs 2024-03-24T01:15:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2403.140748 secs 2024-03-24T01:15:29.932 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3058 sec 2024-03-24T01:15:30.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:29 smithi078 ceph-mon[32429]: pgmap v1761: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:15:30.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:29 smithi107 ceph-mon[36485]: pgmap v1761: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:15:30.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:29 smithi106 ceph-mon[36505]: pgmap v1761: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:15:30.932 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3059 sec 2024-03-24T01:15:31.932 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3060 sec 2024-03-24T01:15:32.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:31 smithi078 ceph-mon[32429]: pgmap v1762: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:32.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:31 smithi107 ceph-mon[36485]: pgmap v1762: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:32.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:31 smithi106 ceph-mon[36505]: pgmap v1762: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:32.932 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3061 sec 2024-03-24T01:15:33.932 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3062 sec 2024-03-24T01:15:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:33 smithi078 ceph-mon[32429]: pgmap v1763: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:34.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2408.140789 secs 2024-03-24T01:15:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:33 smithi107 ceph-mon[36485]: pgmap v1763: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2408.140789 secs 2024-03-24T01:15:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:33 smithi106 ceph-mon[36505]: pgmap v1763: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:34.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2408.140789 secs 2024-03-24T01:15:34.932 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3063 sec 2024-03-24T01:15:35.932 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3064 sec 2024-03-24T01:15:36.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:35 smithi078 ceph-mon[32429]: pgmap v1764: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:36.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:35 smithi107 ceph-mon[36485]: pgmap v1764: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:36.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:35 smithi106 ceph-mon[36505]: pgmap v1764: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:36.932 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3065 sec 2024-03-24T01:15:37.933 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3066 sec 2024-03-24T01:15:38.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:37 smithi078 ceph-mon[32429]: pgmap v1765: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:38.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:37 smithi107 ceph-mon[36485]: pgmap v1765: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:38.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:37 smithi106 ceph-mon[36505]: pgmap v1765: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:38.933 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3067 sec 2024-03-24T01:15:39.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2413.140874 secs 2024-03-24T01:15:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2413.140874 secs 2024-03-24T01:15:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2413.140874 secs 2024-03-24T01:15:39.933 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3068 sec 2024-03-24T01:15:40.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:39 smithi078 ceph-mon[32429]: pgmap v1766: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:39 smithi107 ceph-mon[36485]: pgmap v1766: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:40.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:39 smithi106 ceph-mon[36505]: pgmap v1766: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:40.933 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3069 sec 2024-03-24T01:15:41.933 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3070 sec 2024-03-24T01:15:42.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:41 smithi078 ceph-mon[32429]: pgmap v1767: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:42.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:41 smithi107 ceph-mon[36485]: pgmap v1767: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:42.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:41 smithi106 ceph-mon[36505]: pgmap v1767: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:42.933 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3071 sec 2024-03-24T01:15:43.933 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3072 sec 2024-03-24T01:15:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:43 smithi078 ceph-mon[32429]: pgmap v1768: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:44.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2418.140964 secs 2024-03-24T01:15:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:43 smithi107 ceph-mon[36485]: pgmap v1768: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2418.140964 secs 2024-03-24T01:15:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:43 smithi106 ceph-mon[36505]: pgmap v1768: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2418.140964 secs 2024-03-24T01:15:44.933 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3073 sec 2024-03-24T01:15:45.933 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3074 sec 2024-03-24T01:15:46.204 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:45 smithi106 ceph-mon[36505]: pgmap v1769: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:46.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:45 smithi078 ceph-mon[32429]: pgmap v1769: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:46.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:45 smithi107 ceph-mon[36485]: pgmap v1769: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:46.933 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3075 sec 2024-03-24T01:15:47.933 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3076 sec 2024-03-24T01:15:48.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:47 smithi078 ceph-mon[32429]: pgmap v1770: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:48.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:47 smithi107 ceph-mon[36485]: pgmap v1770: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:48.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:47 smithi106 ceph-mon[36505]: pgmap v1770: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:48.933 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3077 sec 2024-03-24T01:15:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2423.141043 secs 2024-03-24T01:15:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2423.141043 secs 2024-03-24T01:15:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2423.141043 secs 2024-03-24T01:15:49.934 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3078 sec 2024-03-24T01:15:50.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:49 smithi078 ceph-mon[32429]: pgmap v1771: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:49 smithi107 ceph-mon[36485]: pgmap v1771: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:50.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:49 smithi106 ceph-mon[36505]: pgmap v1771: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:50.934 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3079 sec 2024-03-24T01:15:51.934 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3080 sec 2024-03-24T01:15:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:51 smithi078 ceph-mon[32429]: pgmap v1772: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:52.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:51 smithi107 ceph-mon[36485]: pgmap v1772: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:52.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:51 smithi106 ceph-mon[36505]: pgmap v1772: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:52.934 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3081 sec 2024-03-24T01:15:53.934 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3082 sec 2024-03-24T01:15:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:53 smithi078 ceph-mon[32429]: pgmap v1773: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2428.141082 secs 2024-03-24T01:15:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:53 smithi107 ceph-mon[36485]: pgmap v1773: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2428.141082 secs 2024-03-24T01:15:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:53 smithi106 ceph-mon[36505]: pgmap v1773: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2428.141082 secs 2024-03-24T01:15:54.934 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3083 sec 2024-03-24T01:15:55.934 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3084 sec 2024-03-24T01:15:56.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:55 smithi078 ceph-mon[32429]: pgmap v1774: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:56.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:55 smithi107 ceph-mon[36485]: pgmap v1774: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:56.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:55 smithi106 ceph-mon[36505]: pgmap v1774: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:15:56.934 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3085 sec 2024-03-24T01:15:57.934 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3086 sec 2024-03-24T01:15:58.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:57 smithi078 ceph-mon[32429]: pgmap v1775: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:57 smithi107 ceph-mon[36485]: pgmap v1775: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:58.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:57 smithi106 ceph-mon[36505]: pgmap v1775: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:58.934 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3087 sec 2024-03-24T01:15:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2433.141141 secs 2024-03-24T01:15:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2433.141141 secs 2024-03-24T01:15:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2433.141141 secs 2024-03-24T01:15:59.934 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3088 sec 2024-03-24T01:16:00.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:15:59 smithi078 ceph-mon[32429]: pgmap v1776: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:00.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:15:59 smithi107 ceph-mon[36485]: pgmap v1776: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:00.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:15:59 smithi106 ceph-mon[36505]: pgmap v1776: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:00.934 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3089 sec 2024-03-24T01:16:01.934 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3090 sec 2024-03-24T01:16:02.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:01 smithi078 ceph-mon[32429]: pgmap v1777: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:02.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:01 smithi107 ceph-mon[36485]: pgmap v1777: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:02.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:01 smithi106 ceph-mon[36505]: pgmap v1777: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:02.934 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3091 sec 2024-03-24T01:16:03.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:03 smithi078 ceph-mon[32429]: pgmap v1778: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:03.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:03 smithi107 ceph-mon[36485]: pgmap v1778: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:03.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:03 smithi106 ceph-mon[36505]: pgmap v1778: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:03.934 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3092 sec 2024-03-24T01:16:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2438.141223 secs 2024-03-24T01:16:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2438.141223 secs 2024-03-24T01:16:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2438.141223 secs 2024-03-24T01:16:04.935 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3093 sec 2024-03-24T01:16:05.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:05 smithi078 ceph-mon[32429]: pgmap v1779: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:05.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:05 smithi107 ceph-mon[36485]: pgmap v1779: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:05.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:05 smithi106 ceph-mon[36505]: pgmap v1779: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:05.935 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3094 sec 2024-03-24T01:16:06.935 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3095 sec 2024-03-24T01:16:07.935 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3096 sec 2024-03-24T01:16:08.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:07 smithi078 ceph-mon[32429]: pgmap v1780: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:08.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:07 smithi107 ceph-mon[36485]: pgmap v1780: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:08.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:07 smithi106 ceph-mon[36505]: pgmap v1780: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:08.935 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3097 sec 2024-03-24T01:16:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2443.141317 secs 2024-03-24T01:16:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2443.141317 secs 2024-03-24T01:16:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2443.141317 secs 2024-03-24T01:16:09.935 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3098 sec 2024-03-24T01:16:10.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:09 smithi078 ceph-mon[32429]: pgmap v1781: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:10.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:09 smithi107 ceph-mon[36485]: pgmap v1781: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:10.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:09 smithi106 ceph-mon[36505]: pgmap v1781: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:10.935 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3099 sec 2024-03-24T01:16:11.935 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3100 sec 2024-03-24T01:16:12.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:11 smithi078 ceph-mon[32429]: pgmap v1782: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:12.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:11 smithi107 ceph-mon[36485]: pgmap v1782: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:12.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:11 smithi106 ceph-mon[36505]: pgmap v1782: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:12.935 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3101 sec 2024-03-24T01:16:13.935 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3102 sec 2024-03-24T01:16:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:13 smithi078 ceph-mon[32429]: pgmap v1783: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2448.141377 secs 2024-03-24T01:16:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:13 smithi107 ceph-mon[36485]: pgmap v1783: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2448.141377 secs 2024-03-24T01:16:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:13 smithi106 ceph-mon[36505]: pgmap v1783: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2448.141377 secs 2024-03-24T01:16:14.936 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3103 sec 2024-03-24T01:16:15.936 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3104 sec 2024-03-24T01:16:16.204 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:15 smithi106 ceph-mon[36505]: pgmap v1784: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:16.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:15 smithi078 ceph-mon[32429]: pgmap v1784: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:16.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:15 smithi107 ceph-mon[36485]: pgmap v1784: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:16.938 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3105 sec 2024-03-24T01:16:17.936 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3106 sec 2024-03-24T01:16:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:17 smithi078 ceph-mon[32429]: pgmap v1785: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:17 smithi107 ceph-mon[36485]: pgmap v1785: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:18.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:17 smithi106 ceph-mon[36505]: pgmap v1785: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:18.936 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3107 sec 2024-03-24T01:16:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2453.141441 secs 2024-03-24T01:16:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2453.141441 secs 2024-03-24T01:16:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2453.141441 secs 2024-03-24T01:16:19.936 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3108 sec 2024-03-24T01:16:20.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:19 smithi078 ceph-mon[32429]: pgmap v1786: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:20.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:19 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:16:20.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:19 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:16:20.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:19 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:16:20.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:19 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:16:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:19 smithi107 ceph-mon[36485]: pgmap v1786: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:19 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:16:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:19 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:16:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:19 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:16:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:19 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:16:20.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:19 smithi106 ceph-mon[36505]: pgmap v1786: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:20.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:19 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:16:20.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:19 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:16:20.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:19 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:16:20.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:19 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:16:20.936 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3109 sec 2024-03-24T01:16:21.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:20 smithi078 ceph-mon[32429]: pgmap v1787: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:16:21.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:20 smithi078 ceph-mon[32429]: pgmap v1788: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:16:21.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:20 smithi107 ceph-mon[36485]: pgmap v1787: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:16:21.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:20 smithi107 ceph-mon[36485]: pgmap v1788: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:16:21.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:20 smithi106 ceph-mon[36505]: pgmap v1787: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:16:21.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:20 smithi106 ceph-mon[36505]: pgmap v1788: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:16:21.936 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3110 sec 2024-03-24T01:16:22.936 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3111 sec 2024-03-24T01:16:23.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:22 smithi078 ceph-mon[32429]: pgmap v1789: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:16:23.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:22 smithi107 ceph-mon[36485]: pgmap v1789: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:16:23.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:22 smithi106 ceph-mon[36505]: pgmap v1789: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:16:23.936 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3112 sec 2024-03-24T01:16:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2458.141501 secs 2024-03-24T01:16:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2458.141501 secs 2024-03-24T01:16:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2458.141501 secs 2024-03-24T01:16:24.936 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3113 sec 2024-03-24T01:16:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:24 smithi078 ceph-mon[32429]: pgmap v1790: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:16:25.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:24 smithi107 ceph-mon[36485]: pgmap v1790: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:16:25.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:24 smithi106 ceph-mon[36505]: pgmap v1790: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:16:25.936 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3114 sec 2024-03-24T01:16:26.937 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3115 sec 2024-03-24T01:16:27.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:26 smithi078 ceph-mon[32429]: pgmap v1791: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:16:27.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:26 smithi107 ceph-mon[36485]: pgmap v1791: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:16:27.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:26 smithi106 ceph-mon[36505]: pgmap v1791: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:16:27.937 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3116 sec 2024-03-24T01:16:28.937 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3117 sec 2024-03-24T01:16:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:28 smithi078 ceph-mon[32429]: pgmap v1792: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:16:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2463.141565 secs 2024-03-24T01:16:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:28 smithi107 ceph-mon[36485]: pgmap v1792: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:16:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2463.141565 secs 2024-03-24T01:16:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:28 smithi106 ceph-mon[36505]: pgmap v1792: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:16:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2463.141565 secs 2024-03-24T01:16:29.937 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3118 sec 2024-03-24T01:16:30.937 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3119 sec 2024-03-24T01:16:31.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:30 smithi078 ceph-mon[32429]: pgmap v1793: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:16:31.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:30 smithi107 ceph-mon[36485]: pgmap v1793: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:16:31.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:30 smithi106 ceph-mon[36505]: pgmap v1793: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:16:31.937 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3120 sec 2024-03-24T01:16:32.937 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3121 sec 2024-03-24T01:16:33.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:32 smithi078 ceph-mon[32429]: pgmap v1794: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:33.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:32 smithi107 ceph-mon[36485]: pgmap v1794: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:33.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:32 smithi106 ceph-mon[36505]: pgmap v1794: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:33.937 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3122 sec 2024-03-24T01:16:34.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2468.141625 secs 2024-03-24T01:16:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2468.141625 secs 2024-03-24T01:16:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2468.141625 secs 2024-03-24T01:16:34.937 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3123 sec 2024-03-24T01:16:35.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:34 smithi078 ceph-mon[32429]: pgmap v1795: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:35.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:34 smithi107 ceph-mon[36485]: pgmap v1795: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:35.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:34 smithi106 ceph-mon[36505]: pgmap v1795: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:35.937 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3124 sec 2024-03-24T01:16:36.937 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3125 sec 2024-03-24T01:16:37.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:36 smithi078 ceph-mon[32429]: pgmap v1796: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:36 smithi107 ceph-mon[36485]: pgmap v1796: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:36 smithi106 ceph-mon[36505]: pgmap v1796: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:37.938 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3126 sec 2024-03-24T01:16:38.938 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3127 sec 2024-03-24T01:16:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:38 smithi078 ceph-mon[32429]: pgmap v1797: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2473.141685 secs 2024-03-24T01:16:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:38 smithi107 ceph-mon[36485]: pgmap v1797: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2473.141685 secs 2024-03-24T01:16:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:38 smithi106 ceph-mon[36505]: pgmap v1797: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2473.141685 secs 2024-03-24T01:16:39.938 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3128 sec 2024-03-24T01:16:40.938 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3129 sec 2024-03-24T01:16:41.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:40 smithi078 ceph-mon[32429]: pgmap v1798: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:40 smithi107 ceph-mon[36485]: pgmap v1798: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:41.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:40 smithi106 ceph-mon[36505]: pgmap v1798: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:41.938 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3130 sec 2024-03-24T01:16:42.938 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3131 sec 2024-03-24T01:16:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:42 smithi078 ceph-mon[32429]: pgmap v1799: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:43.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:42 smithi107 ceph-mon[36485]: pgmap v1799: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:43.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:42 smithi106 ceph-mon[36505]: pgmap v1799: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:43.938 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3132 sec 2024-03-24T01:16:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2478.141767 secs 2024-03-24T01:16:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2478.141767 secs 2024-03-24T01:16:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2478.141767 secs 2024-03-24T01:16:44.938 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3133 sec 2024-03-24T01:16:45.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:44 smithi078 ceph-mon[32429]: pgmap v1800: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:45.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:44 smithi107 ceph-mon[36485]: pgmap v1800: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:45.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:44 smithi106 ceph-mon[36505]: pgmap v1800: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:45.938 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3134 sec 2024-03-24T01:16:46.939 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3135 sec 2024-03-24T01:16:47.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:46 smithi078 ceph-mon[32429]: pgmap v1801: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:47.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:46 smithi107 ceph-mon[36485]: pgmap v1801: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:47.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:46 smithi106 ceph-mon[36505]: pgmap v1801: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:47.938 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3136 sec 2024-03-24T01:16:48.940 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3137 sec 2024-03-24T01:16:49.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:48 smithi078 ceph-mon[32429]: pgmap v1802: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:49.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2483.141824 secs 2024-03-24T01:16:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:48 smithi107 ceph-mon[36485]: pgmap v1802: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2483.141824 secs 2024-03-24T01:16:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:48 smithi106 ceph-mon[36505]: pgmap v1802: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2483.141824 secs 2024-03-24T01:16:49.939 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3138 sec 2024-03-24T01:16:50.939 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3139 sec 2024-03-24T01:16:51.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:51 smithi078 ceph-mon[32429]: pgmap v1803: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:51.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:51 smithi107 ceph-mon[36485]: pgmap v1803: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:51.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:51 smithi106 ceph-mon[36505]: pgmap v1803: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:51.939 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3140 sec 2024-03-24T01:16:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:52 smithi078 ceph-mon[32429]: pgmap v1804: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:52.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:52 smithi107 ceph-mon[36485]: pgmap v1804: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:52.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:52 smithi106 ceph-mon[36505]: pgmap v1804: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:52.939 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3141 sec 2024-03-24T01:16:53.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2488.141908 secs 2024-03-24T01:16:53.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2488.141908 secs 2024-03-24T01:16:53.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2488.141908 secs 2024-03-24T01:16:53.939 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3142 sec 2024-03-24T01:16:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:54 smithi078 ceph-mon[32429]: pgmap v1805: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:54 smithi107 ceph-mon[36485]: pgmap v1805: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:54 smithi106 ceph-mon[36505]: pgmap v1805: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:54.939 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3143 sec 2024-03-24T01:16:55.939 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3144 sec 2024-03-24T01:16:56.939 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3145 sec 2024-03-24T01:16:57.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:56 smithi078 ceph-mon[32429]: pgmap v1806: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:56 smithi107 ceph-mon[36485]: pgmap v1806: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:57.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:56 smithi106 ceph-mon[36505]: pgmap v1806: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:57.939 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3146 sec 2024-03-24T01:16:58.940 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3147 sec 2024-03-24T01:16:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:58 smithi078 ceph-mon[32429]: pgmap v1807: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:59.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:16:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2493.141993 secs 2024-03-24T01:16:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:58 smithi107 ceph-mon[36485]: pgmap v1807: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:16:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2493.141993 secs 2024-03-24T01:16:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:58 smithi106 ceph-mon[36505]: pgmap v1807: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:16:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:16:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2493.141993 secs 2024-03-24T01:16:59.940 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3148 sec 2024-03-24T01:17:00.940 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3149 sec 2024-03-24T01:17:01.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:00 smithi078 ceph-mon[32429]: pgmap v1808: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:01.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:00 smithi107 ceph-mon[36485]: pgmap v1808: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:01.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:00 smithi106 ceph-mon[36505]: pgmap v1808: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:01.940 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3150 sec 2024-03-24T01:17:02.940 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3151 sec 2024-03-24T01:17:03.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:02 smithi078 ceph-mon[32429]: pgmap v1809: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:03.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:02 smithi107 ceph-mon[36485]: pgmap v1809: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:03.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:02 smithi106 ceph-mon[36505]: pgmap v1809: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:03.940 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3152 sec 2024-03-24T01:17:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2498.142082 secs 2024-03-24T01:17:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2498.142082 secs 2024-03-24T01:17:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2498.142082 secs 2024-03-24T01:17:04.940 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3153 sec 2024-03-24T01:17:04.994 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:04 smithi107 ceph-mon[36485]: pgmap v1810: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:05.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:04 smithi078 ceph-mon[32429]: pgmap v1810: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:05.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:04 smithi106 ceph-mon[36505]: pgmap v1810: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:05.940 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3154 sec 2024-03-24T01:17:06.940 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3155 sec 2024-03-24T01:17:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:06 smithi078 ceph-mon[32429]: pgmap v1811: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:06 smithi107 ceph-mon[36485]: pgmap v1811: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:07.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:06 smithi106 ceph-mon[36505]: pgmap v1811: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:07.941 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3156 sec 2024-03-24T01:17:08.940 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3157 sec 2024-03-24T01:17:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:08 smithi078 ceph-mon[32429]: pgmap v1812: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2503.142141 secs 2024-03-24T01:17:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:08 smithi107 ceph-mon[36485]: pgmap v1812: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2503.142141 secs 2024-03-24T01:17:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:08 smithi106 ceph-mon[36505]: pgmap v1812: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2503.142141 secs 2024-03-24T01:17:09.940 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3158 sec 2024-03-24T01:17:10.941 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3159 sec 2024-03-24T01:17:11.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:10 smithi078 ceph-mon[32429]: pgmap v1813: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:11.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:10 smithi107 ceph-mon[36485]: pgmap v1813: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:11.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:10 smithi106 ceph-mon[36505]: pgmap v1813: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:11.941 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3160 sec 2024-03-24T01:17:12.941 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3161 sec 2024-03-24T01:17:13.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:12 smithi078 ceph-mon[32429]: pgmap v1814: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:13.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:12 smithi107 ceph-mon[36485]: pgmap v1814: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:13.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:12 smithi106 ceph-mon[36505]: pgmap v1814: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:13.941 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3162 sec 2024-03-24T01:17:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2508.142232 secs 2024-03-24T01:17:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2508.142232 secs 2024-03-24T01:17:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2508.142232 secs 2024-03-24T01:17:14.941 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3163 sec 2024-03-24T01:17:15.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:14 smithi078 ceph-mon[32429]: pgmap v1815: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:15.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:14 smithi107 ceph-mon[36485]: pgmap v1815: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:15.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:14 smithi106 ceph-mon[36505]: pgmap v1815: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:15.941 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3164 sec 2024-03-24T01:17:16.941 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3165 sec 2024-03-24T01:17:17.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:16 smithi078 ceph-mon[32429]: pgmap v1816: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:17.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:16 smithi107 ceph-mon[36485]: pgmap v1816: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:17.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:16 smithi106 ceph-mon[36505]: pgmap v1816: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:17.941 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3166 sec 2024-03-24T01:17:18.941 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3167 sec 2024-03-24T01:17:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:18 smithi078 ceph-mon[32429]: pgmap v1817: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:19.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2513.142324 secs 2024-03-24T01:17:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:18 smithi107 ceph-mon[36485]: pgmap v1817: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2513.142324 secs 2024-03-24T01:17:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:18 smithi106 ceph-mon[36505]: pgmap v1817: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2513.142324 secs 2024-03-24T01:17:19.941 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3168 sec 2024-03-24T01:17:20.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:19 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:17:20.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:19 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:17:20.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:19 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:17:20.941 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3169 sec 2024-03-24T01:17:21.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:20 smithi078 ceph-mon[32429]: pgmap v1818: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:21.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:20 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:17:21.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:20 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:17:21.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:20 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:17:21.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:20 smithi107 ceph-mon[36485]: pgmap v1818: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:21.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:20 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:17:21.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:20 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:17:21.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:20 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:17:21.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:20 smithi106 ceph-mon[36505]: pgmap v1818: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:21.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:20 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:17:21.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:20 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:17:21.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:20 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:17:21.942 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3170 sec 2024-03-24T01:17:22.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:21 smithi078 ceph-mon[32429]: pgmap v1819: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:17:22.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:21 smithi078 ceph-mon[32429]: pgmap v1820: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:22.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:21 smithi107 ceph-mon[36485]: pgmap v1819: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:17:22.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:21 smithi107 ceph-mon[36485]: pgmap v1820: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:22.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:21 smithi106 ceph-mon[36505]: pgmap v1819: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:17:22.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:21 smithi106 ceph-mon[36505]: pgmap v1820: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:22.942 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3171 sec 2024-03-24T01:17:23.942 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3172 sec 2024-03-24T01:17:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:23 smithi078 ceph-mon[32429]: pgmap v1821: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:17:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2518.142385 secs 2024-03-24T01:17:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:23 smithi107 ceph-mon[36485]: pgmap v1821: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:17:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2518.142385 secs 2024-03-24T01:17:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:23 smithi106 ceph-mon[36505]: pgmap v1821: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:17:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2518.142385 secs 2024-03-24T01:17:24.942 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3173 sec 2024-03-24T01:17:25.942 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3174 sec 2024-03-24T01:17:26.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:25 smithi078 ceph-mon[32429]: pgmap v1822: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:17:26.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:25 smithi107 ceph-mon[36485]: pgmap v1822: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:17:26.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:25 smithi106 ceph-mon[36505]: pgmap v1822: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:17:26.942 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3175 sec 2024-03-24T01:17:27.942 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3176 sec 2024-03-24T01:17:28.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:27 smithi078 ceph-mon[32429]: pgmap v1823: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:17:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:27 smithi107 ceph-mon[36485]: pgmap v1823: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:17:28.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:27 smithi106 ceph-mon[36505]: pgmap v1823: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:17:28.942 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3177 sec 2024-03-24T01:17:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2523.142447 secs 2024-03-24T01:17:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2523.142447 secs 2024-03-24T01:17:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2523.142447 secs 2024-03-24T01:17:29.942 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3178 sec 2024-03-24T01:17:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:29 smithi078 ceph-mon[32429]: pgmap v1824: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:17:30.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:29 smithi107 ceph-mon[36485]: pgmap v1824: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:17:30.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:29 smithi106 ceph-mon[36505]: pgmap v1824: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:17:30.942 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3179 sec 2024-03-24T01:17:31.942 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3180 sec 2024-03-24T01:17:32.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:31 smithi078 ceph-mon[32429]: pgmap v1825: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:17:32.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:31 smithi107 ceph-mon[36485]: pgmap v1825: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:17:32.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:31 smithi106 ceph-mon[36505]: pgmap v1825: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:17:32.943 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3181 sec 2024-03-24T01:17:33.943 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3182 sec 2024-03-24T01:17:34.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:33 smithi078 ceph-mon[32429]: pgmap v1826: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-24T01:17:34.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2528.142531 secs 2024-03-24T01:17:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:33 smithi107 ceph-mon[36485]: pgmap v1826: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-24T01:17:34.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2528.142531 secs 2024-03-24T01:17:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:33 smithi106 ceph-mon[36505]: pgmap v1826: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-24T01:17:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2528.142531 secs 2024-03-24T01:17:34.943 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3183 sec 2024-03-24T01:17:35.943 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3184 sec 2024-03-24T01:17:36.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:35 smithi078 ceph-mon[32429]: pgmap v1827: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:17:36.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:35 smithi107 ceph-mon[36485]: pgmap v1827: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:17:36.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:35 smithi106 ceph-mon[36505]: pgmap v1827: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:17:36.943 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3185 sec 2024-03-24T01:17:37.943 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3186 sec 2024-03-24T01:17:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:37 smithi078 ceph-mon[32429]: pgmap v1828: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:17:38.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:37 smithi107 ceph-mon[36485]: pgmap v1828: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:17:38.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:37 smithi106 ceph-mon[36505]: pgmap v1828: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:17:38.943 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3187 sec 2024-03-24T01:17:39.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2533.142623 secs 2024-03-24T01:17:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2533.142623 secs 2024-03-24T01:17:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2533.142623 secs 2024-03-24T01:17:39.943 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3188 sec 2024-03-24T01:17:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:39 smithi078 ceph-mon[32429]: pgmap v1829: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:39 smithi107 ceph-mon[36485]: pgmap v1829: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:39 smithi106 ceph-mon[36505]: pgmap v1829: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:40.943 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3189 sec 2024-03-24T01:17:41.943 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3190 sec 2024-03-24T01:17:42.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:41 smithi078 ceph-mon[32429]: pgmap v1830: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:17:42.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:41 smithi107 ceph-mon[36485]: pgmap v1830: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:17:42.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:41 smithi106 ceph-mon[36505]: pgmap v1830: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:17:42.943 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3191 sec 2024-03-24T01:17:43.943 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3192 sec 2024-03-24T01:17:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:43 smithi078 ceph-mon[32429]: pgmap v1831: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:17:44.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2538.142683 secs 2024-03-24T01:17:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:43 smithi107 ceph-mon[36485]: pgmap v1831: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:17:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2538.142683 secs 2024-03-24T01:17:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:43 smithi106 ceph-mon[36505]: pgmap v1831: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:17:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2538.142683 secs 2024-03-24T01:17:44.944 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3193 sec 2024-03-24T01:17:45.944 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3194 sec 2024-03-24T01:17:46.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:45 smithi078 ceph-mon[32429]: pgmap v1832: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:45 smithi107 ceph-mon[36485]: pgmap v1832: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:46.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:45 smithi106 ceph-mon[36505]: pgmap v1832: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:46.944 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3195 sec 2024-03-24T01:17:47.944 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3196 sec 2024-03-24T01:17:48.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:47 smithi078 ceph-mon[32429]: pgmap v1833: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:48.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:47 smithi107 ceph-mon[36485]: pgmap v1833: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:48.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:47 smithi106 ceph-mon[36505]: pgmap v1833: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:48.944 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3197 sec 2024-03-24T01:17:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2543.142756 secs 2024-03-24T01:17:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2543.142756 secs 2024-03-24T01:17:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2543.142756 secs 2024-03-24T01:17:49.944 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3198 sec 2024-03-24T01:17:50.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:49 smithi106 ceph-mon[36505]: pgmap v1834: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:50.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:49 smithi078 ceph-mon[32429]: pgmap v1834: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:49 smithi107 ceph-mon[36485]: pgmap v1834: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:50.944 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3199 sec 2024-03-24T01:17:51.944 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3200 sec 2024-03-24T01:17:52.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:51 smithi106 ceph-mon[36505]: pgmap v1835: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:17:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:51 smithi078 ceph-mon[32429]: pgmap v1835: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:17:52.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:51 smithi107 ceph-mon[36485]: pgmap v1835: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:17:52.944 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3201 sec 2024-03-24T01:17:53.944 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3202 sec 2024-03-24T01:17:54.143 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:53 smithi078 ceph-mon[32429]: pgmap v1836: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:17:54.143 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2548.142828 secs 2024-03-24T01:17:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:53 smithi106 ceph-mon[36505]: pgmap v1836: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:17:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2548.142828 secs 2024-03-24T01:17:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:53 smithi107 ceph-mon[36485]: pgmap v1836: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:17:54.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2548.142828 secs 2024-03-24T01:17:54.945 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3203 sec 2024-03-24T01:17:55.944 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3204 sec 2024-03-24T01:17:56.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:55 smithi106 ceph-mon[36505]: pgmap v1837: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:56.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:55 smithi078 ceph-mon[32429]: pgmap v1837: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:56.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:55 smithi107 ceph-mon[36485]: pgmap v1837: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:17:56.945 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3205 sec 2024-03-24T01:17:57.945 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3206 sec 2024-03-24T01:17:58.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:57 smithi106 ceph-mon[36505]: pgmap v1838: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:58.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:57 smithi078 ceph-mon[32429]: pgmap v1838: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:57 smithi107 ceph-mon[36485]: pgmap v1838: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:58.945 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3207 sec 2024-03-24T01:17:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2553.142888 secs 2024-03-24T01:17:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2553.142888 secs 2024-03-24T01:17:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2553.142888 secs 2024-03-24T01:17:59.945 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3208 sec 2024-03-24T01:18:00.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:17:59 smithi106 ceph-mon[36505]: pgmap v1839: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:00.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:17:59 smithi078 ceph-mon[32429]: pgmap v1839: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:00.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:17:59 smithi107 ceph-mon[36485]: pgmap v1839: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:00.945 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3209 sec 2024-03-24T01:18:01.945 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3210 sec 2024-03-24T01:18:02.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:01 smithi106 ceph-mon[36505]: pgmap v1840: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:18:02.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:01 smithi078 ceph-mon[32429]: pgmap v1840: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:18:02.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:01 smithi107 ceph-mon[36485]: pgmap v1840: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:18:02.945 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3211 sec 2024-03-24T01:18:03.945 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3212 sec 2024-03-24T01:18:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:03 smithi106 ceph-mon[36505]: pgmap v1841: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:18:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2558.142949 secs 2024-03-24T01:18:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:03 smithi078 ceph-mon[32429]: pgmap v1841: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:18:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2558.142949 secs 2024-03-24T01:18:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:03 smithi107 ceph-mon[36485]: pgmap v1841: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:18:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2558.142949 secs 2024-03-24T01:18:04.945 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3213 sec 2024-03-24T01:18:05.945 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3214 sec 2024-03-24T01:18:06.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:05 smithi107 ceph-mon[36485]: pgmap v1842: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:06.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:05 smithi106 ceph-mon[36505]: pgmap v1842: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:06.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:05 smithi078 ceph-mon[32429]: pgmap v1842: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:06.945 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3215 sec 2024-03-24T01:18:07.945 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3216 sec 2024-03-24T01:18:08.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:07 smithi106 ceph-mon[36505]: pgmap v1843: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:08.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:07 smithi078 ceph-mon[32429]: pgmap v1843: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:08.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:07 smithi107 ceph-mon[36485]: pgmap v1843: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:08.946 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3217 sec 2024-03-24T01:18:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2563.143012 secs 2024-03-24T01:18:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2563.143012 secs 2024-03-24T01:18:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2563.143012 secs 2024-03-24T01:18:09.946 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3218 sec 2024-03-24T01:18:10.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:09 smithi106 ceph-mon[36505]: pgmap v1844: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:10.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:09 smithi078 ceph-mon[32429]: pgmap v1844: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:10.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:09 smithi107 ceph-mon[36485]: pgmap v1844: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:10.946 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3219 sec 2024-03-24T01:18:11.946 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3220 sec 2024-03-24T01:18:12.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:11 smithi106 ceph-mon[36505]: pgmap v1845: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:18:12.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:11 smithi078 ceph-mon[32429]: pgmap v1845: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:18:12.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:11 smithi107 ceph-mon[36485]: pgmap v1845: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:18:12.946 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3221 sec 2024-03-24T01:18:13.946 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3222 sec 2024-03-24T01:18:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:13 smithi106 ceph-mon[36505]: pgmap v1846: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:18:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2568.143073 secs 2024-03-24T01:18:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:13 smithi078 ceph-mon[32429]: pgmap v1846: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:18:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2568.143073 secs 2024-03-24T01:18:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:13 smithi107 ceph-mon[36485]: pgmap v1846: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:18:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2568.143073 secs 2024-03-24T01:18:14.946 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3223 sec 2024-03-24T01:18:15.946 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3224 sec 2024-03-24T01:18:16.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:15 smithi106 ceph-mon[36505]: pgmap v1847: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:16.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:15 smithi078 ceph-mon[32429]: pgmap v1847: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:16.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:15 smithi107 ceph-mon[36485]: pgmap v1847: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:16.946 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3225 sec 2024-03-24T01:18:17.946 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3226 sec 2024-03-24T01:18:18.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:17 smithi106 ceph-mon[36505]: pgmap v1848: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:17 smithi078 ceph-mon[32429]: pgmap v1848: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:17 smithi107 ceph-mon[36485]: pgmap v1848: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:18.947 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3227 sec 2024-03-24T01:18:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2573.143116 secs 2024-03-24T01:18:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2573.143116 secs 2024-03-24T01:18:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2573.143116 secs 2024-03-24T01:18:19.947 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3228 sec 2024-03-24T01:18:20.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:19 smithi078 ceph-mon[32429]: pgmap v1849: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:19 smithi107 ceph-mon[36485]: pgmap v1849: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:20.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:19 smithi106 ceph-mon[36505]: pgmap v1849: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:20.947 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3229 sec 2024-03-24T01:18:21.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:20 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:18:21.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:20 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:18:21.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:20 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:18:21.947 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3230 sec 2024-03-24T01:18:22.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:21 smithi078 ceph-mon[32429]: pgmap v1850: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:18:22.133 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:21 smithi106 ceph-mon[36505]: pgmap v1850: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:18:22.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:21 smithi107 ceph-mon[36485]: pgmap v1850: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:18:22.947 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3231 sec 2024-03-24T01:18:23.947 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3232 sec 2024-03-24T01:18:24.322 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:23 smithi078 ceph-mon[32429]: pgmap v1851: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:18:24.322 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2578.143205 secs 2024-03-24T01:18:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:23 smithi107 ceph-mon[36485]: pgmap v1851: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:18:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2578.143205 secs 2024-03-24T01:18:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:23 smithi106 ceph-mon[36505]: pgmap v1851: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:18:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2578.143205 secs 2024-03-24T01:18:24.947 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3233 sec 2024-03-24T01:18:25.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:25 smithi106 ceph-mon[36505]: pgmap v1852: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:18:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:18:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:18:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:18:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:18:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:18:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:25 smithi106 ceph-mon[36505]: pgmap v1853: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:18:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:25 smithi106 ceph-mon[36505]: pgmap v1854: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T01:18:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:18:25.947 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3234 sec 2024-03-24T01:18:25.962 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:25 smithi078 ceph-mon[32429]: pgmap v1852: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:25.962 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:18:25.962 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:18:25.962 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:18:25.962 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:18:25.962 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:18:25.962 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:18:25.962 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:25 smithi078 ceph-mon[32429]: pgmap v1853: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:18:25.962 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:25 smithi078 ceph-mon[32429]: pgmap v1854: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T01:18:25.962 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:18:26.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:25 smithi107 ceph-mon[36485]: pgmap v1852: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:26.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:18:26.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:18:26.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:18:26.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:18:26.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:18:26.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:18:26.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:25 smithi107 ceph-mon[36485]: pgmap v1853: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:18:26.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:25 smithi107 ceph-mon[36485]: pgmap v1854: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T01:18:26.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:18:26.947 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3235 sec 2024-03-24T01:18:27.947 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3236 sec 2024-03-24T01:18:28.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:27 smithi078 ceph-mon[32429]: pgmap v1855: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:18:28.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:27 smithi107 ceph-mon[36485]: pgmap v1855: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:18:28.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:27 smithi106 ceph-mon[36505]: pgmap v1855: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:18:28.947 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3237 sec 2024-03-24T01:18:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2583.143264 secs 2024-03-24T01:18:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2583.143264 secs 2024-03-24T01:18:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2583.143264 secs 2024-03-24T01:18:29.947 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3238 sec 2024-03-24T01:18:30.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:29 smithi078 ceph-mon[32429]: pgmap v1856: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T01:18:30.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:29 smithi107 ceph-mon[36485]: pgmap v1856: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T01:18:30.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:29 smithi106 ceph-mon[36505]: pgmap v1856: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T01:18:30.947 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3239 sec 2024-03-24T01:18:31.947 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3240 sec 2024-03-24T01:18:32.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:31 smithi078 ceph-mon[32429]: pgmap v1857: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:18:32.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:31 smithi107 ceph-mon[36485]: pgmap v1857: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:18:32.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:31 smithi106 ceph-mon[36505]: pgmap v1857: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:18:32.948 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3241 sec 2024-03-24T01:18:33.948 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3242 sec 2024-03-24T01:18:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:33 smithi078 ceph-mon[32429]: pgmap v1858: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:18:34.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2588.143295 secs 2024-03-24T01:18:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:33 smithi107 ceph-mon[36485]: pgmap v1858: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:18:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2588.143295 secs 2024-03-24T01:18:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:33 smithi106 ceph-mon[36505]: pgmap v1858: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:18:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2588.143295 secs 2024-03-24T01:18:34.948 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3243 sec 2024-03-24T01:18:35.948 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3244 sec 2024-03-24T01:18:36.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:35 smithi078 ceph-mon[32429]: pgmap v1859: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:18:36.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:35 smithi107 ceph-mon[36485]: pgmap v1859: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:18:36.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:35 smithi106 ceph-mon[36505]: pgmap v1859: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:18:36.948 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3245 sec 2024-03-24T01:18:37.948 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3246 sec 2024-03-24T01:18:38.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:37 smithi078 ceph-mon[32429]: pgmap v1860: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:38.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:37 smithi107 ceph-mon[36485]: pgmap v1860: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:38.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:37 smithi106 ceph-mon[36505]: pgmap v1860: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:38.948 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3247 sec 2024-03-24T01:18:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2593.143376 secs 2024-03-24T01:18:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2593.143376 secs 2024-03-24T01:18:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2593.143376 secs 2024-03-24T01:18:39.948 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3248 sec 2024-03-24T01:18:40.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:39 smithi078 ceph-mon[32429]: pgmap v1861: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:39 smithi107 ceph-mon[36485]: pgmap v1861: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:40.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:39 smithi106 ceph-mon[36505]: pgmap v1861: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:40.948 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3249 sec 2024-03-24T01:18:41.948 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3250 sec 2024-03-24T01:18:42.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:41 smithi078 ceph-mon[32429]: pgmap v1862: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:42.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:41 smithi107 ceph-mon[36485]: pgmap v1862: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:42.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:41 smithi106 ceph-mon[36505]: pgmap v1862: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:42.948 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3251 sec 2024-03-24T01:18:43.948 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3252 sec 2024-03-24T01:18:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:43 smithi078 ceph-mon[32429]: pgmap v1863: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2598.143413 secs 2024-03-24T01:18:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:44 smithi107 ceph-mon[36485]: pgmap v1863: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2598.143413 secs 2024-03-24T01:18:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:43 smithi106 ceph-mon[36505]: pgmap v1863: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2598.143413 secs 2024-03-24T01:18:44.948 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3253 sec 2024-03-24T01:18:45.949 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3254 sec 2024-03-24T01:18:46.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:46 smithi078 ceph-mon[32429]: pgmap v1864: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:46.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:46 smithi107 ceph-mon[36485]: pgmap v1864: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:46.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:46 smithi106 ceph-mon[36505]: pgmap v1864: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:46.949 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3255 sec 2024-03-24T01:18:47.949 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3256 sec 2024-03-24T01:18:48.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:48 smithi078 ceph-mon[32429]: pgmap v1865: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:48.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:48 smithi107 ceph-mon[36485]: pgmap v1865: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:48.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:48 smithi106 ceph-mon[36505]: pgmap v1865: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:48.949 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3257 sec 2024-03-24T01:18:49.303 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2603.143504 secs 2024-03-24T01:18:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2603.143504 secs 2024-03-24T01:18:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2603.143504 secs 2024-03-24T01:18:49.949 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3258 sec 2024-03-24T01:18:50.316 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:50 smithi106 ceph-mon[36505]: pgmap v1866: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:50.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:50 smithi078 ceph-mon[32429]: pgmap v1866: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:50 smithi107 ceph-mon[36485]: pgmap v1866: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:50.949 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3259 sec 2024-03-24T01:18:51.949 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3260 sec 2024-03-24T01:18:52.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:52 smithi106 ceph-mon[36505]: pgmap v1867: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:52 smithi078 ceph-mon[32429]: pgmap v1867: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:52.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:52 smithi107 ceph-mon[36485]: pgmap v1867: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:52.949 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3261 sec 2024-03-24T01:18:53.949 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3262 sec 2024-03-24T01:18:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2608.143563 secs 2024-03-24T01:18:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:54 smithi078 ceph-mon[32429]: pgmap v1868: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2608.143563 secs 2024-03-24T01:18:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:54 smithi107 ceph-mon[36485]: pgmap v1868: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2608.143563 secs 2024-03-24T01:18:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:54 smithi106 ceph-mon[36505]: pgmap v1868: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:54.950 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3263 sec 2024-03-24T01:18:55.950 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3264 sec 2024-03-24T01:18:56.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:56 smithi078 ceph-mon[32429]: pgmap v1869: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:56.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:56 smithi107 ceph-mon[36485]: pgmap v1869: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:56.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:56 smithi106 ceph-mon[36505]: pgmap v1869: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:56.950 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3265 sec 2024-03-24T01:18:57.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:57 smithi078 ceph-mon[32429]: pgmap v1870: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:57.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:57 smithi107 ceph-mon[36485]: pgmap v1870: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:57.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:57 smithi106 ceph-mon[36505]: pgmap v1870: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:57.950 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3266 sec 2024-03-24T01:18:58.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2613.143622 secs 2024-03-24T01:18:58.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2613.143622 secs 2024-03-24T01:18:58.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2613.143622 secs 2024-03-24T01:18:58.950 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3267 sec 2024-03-24T01:18:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:18:59 smithi078 ceph-mon[32429]: pgmap v1871: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:18:59 smithi107 ceph-mon[36485]: pgmap v1871: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:18:59 smithi106 ceph-mon[36505]: pgmap v1871: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:18:59.950 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3268 sec 2024-03-24T01:19:00.950 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3269 sec 2024-03-24T01:19:01.950 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3270 sec 2024-03-24T01:19:02.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:01 smithi106 ceph-mon[36505]: pgmap v1872: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:02.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:01 smithi078 ceph-mon[32429]: pgmap v1872: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:02.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:01 smithi107 ceph-mon[36485]: pgmap v1872: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:02.950 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3271 sec 2024-03-24T01:19:03.950 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3272 sec 2024-03-24T01:19:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2618.143681 secs 2024-03-24T01:19:04.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:03 smithi078 ceph-mon[32429]: pgmap v1873: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2618.143681 secs 2024-03-24T01:19:04.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:03 smithi107 ceph-mon[36485]: pgmap v1873: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2618.143681 secs 2024-03-24T01:19:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:03 smithi106 ceph-mon[36505]: pgmap v1873: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:04.950 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3273 sec 2024-03-24T01:19:05.950 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3274 sec 2024-03-24T01:19:06.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:05 smithi078 ceph-mon[32429]: pgmap v1874: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:06.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:05 smithi107 ceph-mon[36485]: pgmap v1874: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:06.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:05 smithi106 ceph-mon[36505]: pgmap v1874: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:06.951 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3275 sec 2024-03-24T01:19:07.951 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3276 sec 2024-03-24T01:19:08.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:07 smithi078 ceph-mon[32429]: pgmap v1875: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:08.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:07 smithi107 ceph-mon[36485]: pgmap v1875: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:08.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:07 smithi106 ceph-mon[36505]: pgmap v1875: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:08.951 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3277 sec 2024-03-24T01:19:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2623.143739 secs 2024-03-24T01:19:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2623.143739 secs 2024-03-24T01:19:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2623.143739 secs 2024-03-24T01:19:09.951 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3278 sec 2024-03-24T01:19:10.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:09 smithi078 ceph-mon[32429]: pgmap v1876: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:10.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:09 smithi107 ceph-mon[36485]: pgmap v1876: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:10.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:09 smithi106 ceph-mon[36505]: pgmap v1876: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:10.951 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3279 sec 2024-03-24T01:19:11.951 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3280 sec 2024-03-24T01:19:12.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:11 smithi078 ceph-mon[32429]: pgmap v1877: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:12.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:11 smithi107 ceph-mon[36485]: pgmap v1877: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:12.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:11 smithi106 ceph-mon[36505]: pgmap v1877: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:12.951 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3281 sec 2024-03-24T01:19:13.951 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3282 sec 2024-03-24T01:19:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2628.143796 secs 2024-03-24T01:19:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:13 smithi078 ceph-mon[32429]: pgmap v1878: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2628.143796 secs 2024-03-24T01:19:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:13 smithi107 ceph-mon[36485]: pgmap v1878: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2628.143796 secs 2024-03-24T01:19:14.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:13 smithi106 ceph-mon[36505]: pgmap v1878: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:14.952 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3283 sec 2024-03-24T01:19:15.951 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3284 sec 2024-03-24T01:19:16.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:16 smithi078 ceph-mon[32429]: pgmap v1879: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:16.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:16 smithi107 ceph-mon[36485]: pgmap v1879: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:16.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:16 smithi106 ceph-mon[36505]: pgmap v1879: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:16.951 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3285 sec 2024-03-24T01:19:17.952 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3286 sec 2024-03-24T01:19:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:18 smithi078 ceph-mon[32429]: pgmap v1880: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:18 smithi107 ceph-mon[36485]: pgmap v1880: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:18.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:18 smithi106 ceph-mon[36505]: pgmap v1880: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:18.952 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3287 sec 2024-03-24T01:19:19.303 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2633.143856 secs 2024-03-24T01:19:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2633.143856 secs 2024-03-24T01:19:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2633.143856 secs 2024-03-24T01:19:19.952 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3288 sec 2024-03-24T01:19:20.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:20 smithi078 ceph-mon[32429]: pgmap v1881: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:20 smithi107 ceph-mon[36485]: pgmap v1881: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:20.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:20 smithi106 ceph-mon[36505]: pgmap v1881: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:20.952 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3289 sec 2024-03-24T01:19:21.952 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3290 sec 2024-03-24T01:19:22.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:22 smithi078 ceph-mon[32429]: pgmap v1882: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:22.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:22 smithi107 ceph-mon[36485]: pgmap v1882: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:22.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:22 smithi106 ceph-mon[36505]: pgmap v1882: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:22.952 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3291 sec 2024-03-24T01:19:23.952 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3292 sec 2024-03-24T01:19:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2638.143944 secs 2024-03-24T01:19:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:24 smithi078 ceph-mon[32429]: pgmap v1883: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2638.143944 secs 2024-03-24T01:19:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:24 smithi107 ceph-mon[36485]: pgmap v1883: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2638.143944 secs 2024-03-24T01:19:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:24 smithi106 ceph-mon[36505]: pgmap v1883: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:24.953 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3293 sec 2024-03-24T01:19:25.952 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3294 sec 2024-03-24T01:19:26.311 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:26 smithi107 ceph-mon[36485]: pgmap v1884: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:26.312 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:26 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:19:26.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:26 smithi078 ceph-mon[32429]: pgmap v1884: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:26.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:26 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:19:26.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:26 smithi106 ceph-mon[36505]: pgmap v1884: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:26.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:26 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:19:26.953 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3295 sec 2024-03-24T01:19:27.953 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3296 sec 2024-03-24T01:19:28.266 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:28 smithi107 ceph-mon[36485]: pgmap v1885: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:28.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:28 smithi078 ceph-mon[32429]: pgmap v1885: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:28.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:28 smithi106 ceph-mon[36505]: pgmap v1885: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:28.953 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3297 sec 2024-03-24T01:19:29.330 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2643.144031 secs 2024-03-24T01:19:29.330 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:29 smithi107 ceph-mon[36485]: pgmap v1886: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2643.144031 secs 2024-03-24T01:19:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:29 smithi078 ceph-mon[32429]: pgmap v1886: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2643.144031 secs 2024-03-24T01:19:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:29 smithi106 ceph-mon[36505]: pgmap v1886: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:29.953 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3298 sec 2024-03-24T01:19:30.953 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3299 sec 2024-03-24T01:19:31.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:30 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:19:31.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:30 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:19:31.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:30 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:19:31.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:30 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:19:31.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:30 smithi078 ceph-mon[32429]: pgmap v1887: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:19:31.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:30 smithi078 ceph-mon[32429]: pgmap v1888: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:19:31.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:30 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:19:31.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:30 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:19:31.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:30 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:19:31.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:30 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:19:31.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:30 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:19:31.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:30 smithi107 ceph-mon[36485]: pgmap v1887: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:19:31.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:30 smithi107 ceph-mon[36485]: pgmap v1888: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:19:31.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:30 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:19:31.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:30 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:19:31.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:30 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:19:31.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:30 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:19:31.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:30 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:19:31.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:30 smithi106 ceph-mon[36505]: pgmap v1887: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:19:31.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:30 smithi106 ceph-mon[36505]: pgmap v1888: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:19:31.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:30 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:19:31.953 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3300 sec 2024-03-24T01:19:32.953 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3301 sec 2024-03-24T01:19:33.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:32 smithi078 ceph-mon[32429]: pgmap v1889: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:19:33.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:32 smithi107 ceph-mon[36485]: pgmap v1889: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:19:33.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:32 smithi106 ceph-mon[36505]: pgmap v1889: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:19:33.953 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3302 sec 2024-03-24T01:19:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2648.144124 secs 2024-03-24T01:19:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2648.144124 secs 2024-03-24T01:19:34.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2648.144124 secs 2024-03-24T01:19:34.953 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3303 sec 2024-03-24T01:19:35.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:34 smithi078 ceph-mon[32429]: pgmap v1890: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:19:35.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:34 smithi107 ceph-mon[36485]: pgmap v1890: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:19:35.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:34 smithi106 ceph-mon[36505]: pgmap v1890: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:19:35.953 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3304 sec 2024-03-24T01:19:36.953 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3305 sec 2024-03-24T01:19:37.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:36 smithi078 ceph-mon[32429]: pgmap v1891: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:19:37.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:36 smithi107 ceph-mon[36485]: pgmap v1891: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:19:37.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:36 smithi106 ceph-mon[36505]: pgmap v1891: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:19:37.953 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3306 sec 2024-03-24T01:19:38.953 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3307 sec 2024-03-24T01:19:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:38 smithi078 ceph-mon[32429]: pgmap v1892: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:19:39.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2653.144206 secs 2024-03-24T01:19:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:38 smithi107 ceph-mon[36485]: pgmap v1892: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:19:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2653.144206 secs 2024-03-24T01:19:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:38 smithi106 ceph-mon[36505]: pgmap v1892: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:19:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2653.144206 secs 2024-03-24T01:19:39.954 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3308 sec 2024-03-24T01:19:40.954 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3309 sec 2024-03-24T01:19:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:40 smithi078 ceph-mon[32429]: pgmap v1893: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:19:41.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:40 smithi107 ceph-mon[36485]: pgmap v1893: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:19:41.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:40 smithi106 ceph-mon[36505]: pgmap v1893: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:19:41.954 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3310 sec 2024-03-24T01:19:42.954 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3311 sec 2024-03-24T01:19:43.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:42 smithi078 ceph-mon[32429]: pgmap v1894: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:43.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:42 smithi107 ceph-mon[36485]: pgmap v1894: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:43.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:42 smithi106 ceph-mon[36505]: pgmap v1894: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:43.954 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3312 sec 2024-03-24T01:19:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2658.144299 secs 2024-03-24T01:19:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2658.144299 secs 2024-03-24T01:19:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2658.144299 secs 2024-03-24T01:19:44.954 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3313 sec 2024-03-24T01:19:45.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:44 smithi078 ceph-mon[32429]: pgmap v1895: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:45.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:44 smithi107 ceph-mon[36485]: pgmap v1895: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:45.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:44 smithi106 ceph-mon[36505]: pgmap v1895: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:45.954 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3314 sec 2024-03-24T01:19:46.954 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3315 sec 2024-03-24T01:19:47.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:46 smithi078 ceph-mon[32429]: pgmap v1896: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:47.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:46 smithi107 ceph-mon[36485]: pgmap v1896: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:47.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:46 smithi106 ceph-mon[36505]: pgmap v1896: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:47.954 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3316 sec 2024-03-24T01:19:48.954 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3317 sec 2024-03-24T01:19:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:48 smithi078 ceph-mon[32429]: pgmap v1897: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:49.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2663.144324 secs 2024-03-24T01:19:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:48 smithi107 ceph-mon[36485]: pgmap v1897: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2663.144324 secs 2024-03-24T01:19:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:48 smithi106 ceph-mon[36505]: pgmap v1897: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2663.144324 secs 2024-03-24T01:19:49.954 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3318 sec 2024-03-24T01:19:50.954 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3319 sec 2024-03-24T01:19:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:50 smithi078 ceph-mon[32429]: pgmap v1898: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:51.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:50 smithi107 ceph-mon[36485]: pgmap v1898: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:51.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:50 smithi106 ceph-mon[36505]: pgmap v1898: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:51.955 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3320 sec 2024-03-24T01:19:52.955 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3321 sec 2024-03-24T01:19:53.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:52 smithi078 ceph-mon[32429]: pgmap v1899: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:53.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:52 smithi107 ceph-mon[36485]: pgmap v1899: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:53.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:52 smithi106 ceph-mon[36505]: pgmap v1899: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:53.955 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3322 sec 2024-03-24T01:19:54.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2668.144336 secs 2024-03-24T01:19:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2668.144336 secs 2024-03-24T01:19:54.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2668.144336 secs 2024-03-24T01:19:54.955 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3323 sec 2024-03-24T01:19:55.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:54 smithi078 ceph-mon[32429]: pgmap v1900: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:55.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:54 smithi107 ceph-mon[36485]: pgmap v1900: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:55.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:54 smithi106 ceph-mon[36505]: pgmap v1900: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:55.955 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3324 sec 2024-03-24T01:19:56.955 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3325 sec 2024-03-24T01:19:57.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:56 smithi078 ceph-mon[32429]: pgmap v1901: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:56 smithi107 ceph-mon[36485]: pgmap v1901: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:57.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:56 smithi106 ceph-mon[36505]: pgmap v1901: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:57.955 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3326 sec 2024-03-24T01:19:58.955 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3327 sec 2024-03-24T01:19:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:58 smithi078 ceph-mon[32429]: pgmap v1902: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:19:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2673.144396 secs 2024-03-24T01:19:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:58 smithi107 ceph-mon[36485]: pgmap v1902: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:19:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2673.144396 secs 2024-03-24T01:19:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:58 smithi106 ceph-mon[36505]: pgmap v1902: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:19:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:19:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2673.144396 secs 2024-03-24T01:19:59.955 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3328 sec 2024-03-24T01:20:00.955 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3329 sec 2024-03-24T01:20:01.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:00 smithi078 ceph-mon[32429]: pgmap v1903: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:01.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:00 smithi078 ceph-mon[32429]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:20:01.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:00 smithi107 ceph-mon[36485]: pgmap v1903: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:01.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:00 smithi107 ceph-mon[36485]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:20:01.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:00 smithi106 ceph-mon[36505]: pgmap v1903: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:01.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:00 smithi106 ceph-mon[36505]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:20:01.956 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3330 sec 2024-03-24T01:20:02.955 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3331 sec 2024-03-24T01:20:03.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:02 smithi078 ceph-mon[32429]: pgmap v1904: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:03.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:02 smithi107 ceph-mon[36485]: pgmap v1904: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:03.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:02 smithi106 ceph-mon[36505]: pgmap v1904: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:03.955 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3332 sec 2024-03-24T01:20:04.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2678.144476 secs 2024-03-24T01:20:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2678.144476 secs 2024-03-24T01:20:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2678.144476 secs 2024-03-24T01:20:04.956 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3333 sec 2024-03-24T01:20:05.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:04 smithi078 ceph-mon[32429]: pgmap v1905: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:05.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:04 smithi107 ceph-mon[36485]: pgmap v1905: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:05.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:04 smithi106 ceph-mon[36505]: pgmap v1905: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:05.956 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3334 sec 2024-03-24T01:20:06.956 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3335 sec 2024-03-24T01:20:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:06 smithi078 ceph-mon[32429]: pgmap v1906: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:06 smithi107 ceph-mon[36485]: pgmap v1906: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:07.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:06 smithi106 ceph-mon[36505]: pgmap v1906: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:07.956 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3336 sec 2024-03-24T01:20:08.956 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3337 sec 2024-03-24T01:20:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:08 smithi078 ceph-mon[32429]: pgmap v1907: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2683.144561 secs 2024-03-24T01:20:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:08 smithi107 ceph-mon[36485]: pgmap v1907: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2683.144561 secs 2024-03-24T01:20:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:08 smithi106 ceph-mon[36505]: pgmap v1907: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2683.144561 secs 2024-03-24T01:20:09.956 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3338 sec 2024-03-24T01:20:10.956 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3339 sec 2024-03-24T01:20:11.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:10 smithi078 ceph-mon[32429]: pgmap v1908: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:11.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:10 smithi107 ceph-mon[36485]: pgmap v1908: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:11.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:10 smithi106 ceph-mon[36505]: pgmap v1908: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:11.956 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3340 sec 2024-03-24T01:20:12.956 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3341 sec 2024-03-24T01:20:13.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:12 smithi107 ceph-mon[36485]: pgmap v1909: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:13.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:12 smithi106 ceph-mon[36505]: pgmap v1909: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:13.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:12 smithi078 ceph-mon[32429]: pgmap v1909: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:13.957 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3342 sec 2024-03-24T01:20:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2688.144637 secs 2024-03-24T01:20:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2688.144637 secs 2024-03-24T01:20:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2688.144637 secs 2024-03-24T01:20:14.957 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3343 sec 2024-03-24T01:20:15.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:14 smithi106 ceph-mon[36505]: pgmap v1910: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:15.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:14 smithi078 ceph-mon[32429]: pgmap v1910: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:15.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:14 smithi107 ceph-mon[36485]: pgmap v1910: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:15.957 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3344 sec 2024-03-24T01:20:16.957 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3345 sec 2024-03-24T01:20:17.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:16 smithi106 ceph-mon[36505]: pgmap v1911: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:17.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:16 smithi078 ceph-mon[32429]: pgmap v1911: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:17.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:16 smithi107 ceph-mon[36485]: pgmap v1911: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:17.957 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3346 sec 2024-03-24T01:20:18.957 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3347 sec 2024-03-24T01:20:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:18 smithi106 ceph-mon[36505]: pgmap v1912: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:19.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2693.144698 secs 2024-03-24T01:20:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:18 smithi078 ceph-mon[32429]: pgmap v1912: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2693.144698 secs 2024-03-24T01:20:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:18 smithi107 ceph-mon[36485]: pgmap v1912: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2693.144698 secs 2024-03-24T01:20:19.957 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3348 sec 2024-03-24T01:20:20.957 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3349 sec 2024-03-24T01:20:21.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:20 smithi106 ceph-mon[36505]: pgmap v1913: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:21.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:20 smithi078 ceph-mon[32429]: pgmap v1913: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:21.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:20 smithi107 ceph-mon[36485]: pgmap v1913: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:21.957 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3350 sec 2024-03-24T01:20:22.957 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3351 sec 2024-03-24T01:20:23.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:22 smithi106 ceph-mon[36505]: pgmap v1914: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:23.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:22 smithi078 ceph-mon[32429]: pgmap v1914: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:23.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:22 smithi107 ceph-mon[36485]: pgmap v1914: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:23.958 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3352 sec 2024-03-24T01:20:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2698.144784 secs 2024-03-24T01:20:24.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2698.144784 secs 2024-03-24T01:20:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2698.144784 secs 2024-03-24T01:20:24.958 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3353 sec 2024-03-24T01:20:25.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:24 smithi106 ceph-mon[36505]: pgmap v1915: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:24 smithi078 ceph-mon[32429]: pgmap v1915: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:25.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:24 smithi107 ceph-mon[36485]: pgmap v1915: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:25.958 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3354 sec 2024-03-24T01:20:26.958 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3355 sec 2024-03-24T01:20:27.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:26 smithi106 ceph-mon[36505]: pgmap v1916: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:27.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:26 smithi078 ceph-mon[32429]: pgmap v1916: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:27.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:26 smithi107 ceph-mon[36485]: pgmap v1916: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:27.958 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3356 sec 2024-03-24T01:20:28.958 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3357 sec 2024-03-24T01:20:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:28 smithi106 ceph-mon[36505]: pgmap v1917: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2703.144873 secs 2024-03-24T01:20:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:28 smithi078 ceph-mon[32429]: pgmap v1917: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2703.144873 secs 2024-03-24T01:20:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:28 smithi107 ceph-mon[36485]: pgmap v1917: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2703.144873 secs 2024-03-24T01:20:29.958 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3358 sec 2024-03-24T01:20:30.958 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3359 sec 2024-03-24T01:20:31.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:30 smithi106 ceph-mon[36505]: pgmap v1918: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:31.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:30 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:20:31.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:30 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:20:31.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:30 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:20:31.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:30 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:20:31.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:30 smithi078 ceph-mon[32429]: pgmap v1918: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:31.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:30 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:20:31.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:30 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:20:31.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:30 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:20:31.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:30 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:20:31.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:30 smithi107 ceph-mon[36485]: pgmap v1918: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:31.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:30 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:20:31.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:30 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:20:31.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:30 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:20:31.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:30 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:20:31.958 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3360 sec 2024-03-24T01:20:32.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:31 smithi106 ceph-mon[36505]: pgmap v1919: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:20:32.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:31 smithi106 ceph-mon[36505]: pgmap v1920: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:20:32.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:31 smithi078 ceph-mon[32429]: pgmap v1919: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:20:32.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:31 smithi078 ceph-mon[32429]: pgmap v1920: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:20:32.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:31 smithi107 ceph-mon[36485]: pgmap v1919: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:20:32.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:31 smithi107 ceph-mon[36485]: pgmap v1920: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:20:32.958 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3361 sec 2024-03-24T01:20:33.958 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3362 sec 2024-03-24T01:20:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:33 smithi106 ceph-mon[36505]: pgmap v1921: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:20:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2708.144912 secs 2024-03-24T01:20:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:33 smithi078 ceph-mon[32429]: pgmap v1921: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:20:34.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2708.144912 secs 2024-03-24T01:20:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:33 smithi107 ceph-mon[36485]: pgmap v1921: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:20:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2708.144912 secs 2024-03-24T01:20:34.958 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3363 sec 2024-03-24T01:20:35.958 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3364 sec 2024-03-24T01:20:36.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:35 smithi107 ceph-mon[36485]: pgmap v1922: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:20:36.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:35 smithi106 ceph-mon[36505]: pgmap v1922: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:20:36.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:35 smithi078 ceph-mon[32429]: pgmap v1922: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:20:36.959 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3365 sec 2024-03-24T01:20:37.959 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3366 sec 2024-03-24T01:20:38.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:37 smithi106 ceph-mon[36505]: pgmap v1923: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:20:38.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:37 smithi078 ceph-mon[32429]: pgmap v1923: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:20:38.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:37 smithi107 ceph-mon[36485]: pgmap v1923: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:20:38.959 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3367 sec 2024-03-24T01:20:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2713.144951 secs 2024-03-24T01:20:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2713.144951 secs 2024-03-24T01:20:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2713.144951 secs 2024-03-24T01:20:39.959 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3368 sec 2024-03-24T01:20:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:39 smithi106 ceph-mon[36505]: pgmap v1924: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:20:40.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:39 smithi078 ceph-mon[32429]: pgmap v1924: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:20:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:39 smithi107 ceph-mon[36485]: pgmap v1924: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:20:40.959 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3369 sec 2024-03-24T01:20:41.959 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3370 sec 2024-03-24T01:20:42.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:41 smithi078 ceph-mon[32429]: pgmap v1925: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:20:42.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:41 smithi107 ceph-mon[36485]: pgmap v1925: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:20:42.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:41 smithi106 ceph-mon[36505]: pgmap v1925: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:20:42.959 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3371 sec 2024-03-24T01:20:43.959 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3372 sec 2024-03-24T01:20:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:43 smithi078 ceph-mon[32429]: pgmap v1926: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2718.145040 secs 2024-03-24T01:20:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:43 smithi107 ceph-mon[36485]: pgmap v1926: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2718.145040 secs 2024-03-24T01:20:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:43 smithi106 ceph-mon[36505]: pgmap v1926: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2718.145040 secs 2024-03-24T01:20:44.959 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3373 sec 2024-03-24T01:20:45.959 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3374 sec 2024-03-24T01:20:46.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:45 smithi078 ceph-mon[32429]: pgmap v1927: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:46.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:45 smithi107 ceph-mon[36485]: pgmap v1927: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:46.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:45 smithi106 ceph-mon[36505]: pgmap v1927: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:46.960 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3375 sec 2024-03-24T01:20:47.960 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3376 sec 2024-03-24T01:20:48.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:47 smithi078 ceph-mon[32429]: pgmap v1928: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:48.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:47 smithi107 ceph-mon[36485]: pgmap v1928: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:48.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:47 smithi106 ceph-mon[36505]: pgmap v1928: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:48.960 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3377 sec 2024-03-24T01:20:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2723.145129 secs 2024-03-24T01:20:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2723.145129 secs 2024-03-24T01:20:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2723.145129 secs 2024-03-24T01:20:49.960 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3378 sec 2024-03-24T01:20:50.303 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:49 smithi106 ceph-mon[36505]: pgmap v1929: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:50.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:49 smithi078 ceph-mon[32429]: pgmap v1929: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:49 smithi107 ceph-mon[36485]: pgmap v1929: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:50.960 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3379 sec 2024-03-24T01:20:51.960 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3380 sec 2024-03-24T01:20:52.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:51 smithi078 ceph-mon[32429]: pgmap v1930: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:52.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:51 smithi107 ceph-mon[36485]: pgmap v1930: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:52.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:51 smithi106 ceph-mon[36505]: pgmap v1930: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:52.960 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3381 sec 2024-03-24T01:20:53.960 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3382 sec 2024-03-24T01:20:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:53 smithi078 ceph-mon[32429]: pgmap v1931: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2728.145211 secs 2024-03-24T01:20:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:53 smithi107 ceph-mon[36485]: pgmap v1931: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2728.145211 secs 2024-03-24T01:20:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:53 smithi106 ceph-mon[36505]: pgmap v1931: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2728.145211 secs 2024-03-24T01:20:54.960 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3383 sec 2024-03-24T01:20:55.960 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3384 sec 2024-03-24T01:20:56.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:55 smithi078 ceph-mon[32429]: pgmap v1932: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:56.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:55 smithi107 ceph-mon[36485]: pgmap v1932: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:56.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:55 smithi106 ceph-mon[36505]: pgmap v1932: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:20:56.960 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3385 sec 2024-03-24T01:20:57.960 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3386 sec 2024-03-24T01:20:58.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:57 smithi078 ceph-mon[32429]: pgmap v1933: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:57 smithi107 ceph-mon[36485]: pgmap v1933: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:58.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:57 smithi106 ceph-mon[36505]: pgmap v1933: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:58.961 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3387 sec 2024-03-24T01:20:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2733.145272 secs 2024-03-24T01:20:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2733.145272 secs 2024-03-24T01:20:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2733.145272 secs 2024-03-24T01:20:59.961 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3388 sec 2024-03-24T01:21:00.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:20:59 smithi078 ceph-mon[32429]: pgmap v1934: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:00.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:20:59 smithi107 ceph-mon[36485]: pgmap v1934: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:00.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:20:59 smithi106 ceph-mon[36505]: pgmap v1934: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:00.961 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3389 sec 2024-03-24T01:21:01.961 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3390 sec 2024-03-24T01:21:02.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:01 smithi078 ceph-mon[32429]: pgmap v1935: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:02.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:01 smithi107 ceph-mon[36485]: pgmap v1935: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:02.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:01 smithi106 ceph-mon[36505]: pgmap v1935: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:02.961 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3391 sec 2024-03-24T01:21:03.961 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3392 sec 2024-03-24T01:21:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:03 smithi078 ceph-mon[32429]: pgmap v1936: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:04.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2738.145360 secs 2024-03-24T01:21:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:03 smithi107 ceph-mon[36485]: pgmap v1936: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2738.145360 secs 2024-03-24T01:21:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:03 smithi106 ceph-mon[36505]: pgmap v1936: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2738.145360 secs 2024-03-24T01:21:04.961 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3393 sec 2024-03-24T01:21:05.961 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3394 sec 2024-03-24T01:21:06.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:05 smithi078 ceph-mon[32429]: pgmap v1937: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:06.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:05 smithi107 ceph-mon[36485]: pgmap v1937: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:06.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:05 smithi106 ceph-mon[36505]: pgmap v1937: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:06.961 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3395 sec 2024-03-24T01:21:07.961 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3396 sec 2024-03-24T01:21:08.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:07 smithi078 ceph-mon[32429]: pgmap v1938: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:08.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:07 smithi107 ceph-mon[36485]: pgmap v1938: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:08.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:07 smithi106 ceph-mon[36505]: pgmap v1938: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:08.961 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3397 sec 2024-03-24T01:21:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2743.145456 secs 2024-03-24T01:21:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2743.145456 secs 2024-03-24T01:21:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2743.145456 secs 2024-03-24T01:21:09.961 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3398 sec 2024-03-24T01:21:10.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:09 smithi078 ceph-mon[32429]: pgmap v1939: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:10.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:09 smithi107 ceph-mon[36485]: pgmap v1939: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:10.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:09 smithi106 ceph-mon[36505]: pgmap v1939: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:10.961 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3399 sec 2024-03-24T01:21:11.962 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3400 sec 2024-03-24T01:21:12.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:11 smithi078 ceph-mon[32429]: pgmap v1940: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:12.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:11 smithi107 ceph-mon[36485]: pgmap v1940: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:12.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:11 smithi106 ceph-mon[36505]: pgmap v1940: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:12.962 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3401 sec 2024-03-24T01:21:13.962 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3402 sec 2024-03-24T01:21:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:13 smithi078 ceph-mon[32429]: pgmap v1941: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2748.145560 secs 2024-03-24T01:21:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:13 smithi107 ceph-mon[36485]: pgmap v1941: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2748.145560 secs 2024-03-24T01:21:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:13 smithi106 ceph-mon[36505]: pgmap v1941: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:14.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2748.145560 secs 2024-03-24T01:21:14.962 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3403 sec 2024-03-24T01:21:15.962 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3404 sec 2024-03-24T01:21:16.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:15 smithi078 ceph-mon[32429]: pgmap v1942: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:16.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:15 smithi107 ceph-mon[36485]: pgmap v1942: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:16.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:15 smithi106 ceph-mon[36505]: pgmap v1942: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:16.962 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3405 sec 2024-03-24T01:21:17.962 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3406 sec 2024-03-24T01:21:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:17 smithi078 ceph-mon[32429]: pgmap v1943: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:17 smithi107 ceph-mon[36485]: pgmap v1943: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:18.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:17 smithi106 ceph-mon[36505]: pgmap v1943: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:18.962 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3407 sec 2024-03-24T01:21:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2753.145620 secs 2024-03-24T01:21:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2753.145620 secs 2024-03-24T01:21:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2753.145620 secs 2024-03-24T01:21:19.962 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3408 sec 2024-03-24T01:21:20.303 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:19 smithi106 ceph-mon[36505]: pgmap v1944: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:20.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:19 smithi078 ceph-mon[32429]: pgmap v1944: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:19 smithi107 ceph-mon[36485]: pgmap v1944: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:20.963 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3409 sec 2024-03-24T01:21:21.963 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3410 sec 2024-03-24T01:21:22.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:21 smithi078 ceph-mon[32429]: pgmap v1945: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:22.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:21 smithi107 ceph-mon[36485]: pgmap v1945: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:22.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:21 smithi106 ceph-mon[36505]: pgmap v1945: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:22.963 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3411 sec 2024-03-24T01:21:23.963 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3412 sec 2024-03-24T01:21:24.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:23 smithi078 ceph-mon[32429]: pgmap v1946: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:24.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2758.145635 secs 2024-03-24T01:21:24.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:23 smithi107 ceph-mon[36485]: pgmap v1946: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:24.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2758.145635 secs 2024-03-24T01:21:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:23 smithi106 ceph-mon[36505]: pgmap v1946: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2758.145635 secs 2024-03-24T01:21:24.963 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3413 sec 2024-03-24T01:21:25.963 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3414 sec 2024-03-24T01:21:26.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:25 smithi078 ceph-mon[32429]: pgmap v1947: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:26.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:25 smithi107 ceph-mon[36485]: pgmap v1947: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:26.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:25 smithi106 ceph-mon[36505]: pgmap v1947: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:26.963 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3415 sec 2024-03-24T01:21:27.963 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3416 sec 2024-03-24T01:21:28.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:27 smithi078 ceph-mon[32429]: pgmap v1948: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:28.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:27 smithi107 ceph-mon[36485]: pgmap v1948: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:28.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:27 smithi106 ceph-mon[36505]: pgmap v1948: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:28.963 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3417 sec 2024-03-24T01:21:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2763.145726 secs 2024-03-24T01:21:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2763.145726 secs 2024-03-24T01:21:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2763.145726 secs 2024-03-24T01:21:29.963 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3418 sec 2024-03-24T01:21:30.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:30 smithi078 ceph-mon[32429]: pgmap v1949: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:30.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:30 smithi107 ceph-mon[36485]: pgmap v1949: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:30.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:30 smithi106 ceph-mon[36505]: pgmap v1949: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:30.963 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3419 sec 2024-03-24T01:21:31.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:31 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:21:31.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:31 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:21:31.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:31 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:21:31.963 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3420 sec 2024-03-24T01:21:32.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:32 smithi078 ceph-mon[32429]: pgmap v1950: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:32.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:32 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:21:32.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:32 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:21:32.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:32 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:21:32.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:32 smithi107 ceph-mon[36485]: pgmap v1950: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:32.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:32 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:21:32.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:32 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:21:32.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:32 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:21:32.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:32 smithi106 ceph-mon[36505]: pgmap v1950: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:32.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:32 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:21:32.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:32 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:21:32.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:32 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:21:32.963 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3421 sec 2024-03-24T01:21:33.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:33 smithi078 ceph-mon[32429]: pgmap v1951: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:21:33.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:33 smithi078 ceph-mon[32429]: pgmap v1952: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:21:33.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:33 smithi107 ceph-mon[36485]: pgmap v1951: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:21:33.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:33 smithi107 ceph-mon[36485]: pgmap v1952: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:21:33.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:33 smithi106 ceph-mon[36505]: pgmap v1951: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:21:33.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:33 smithi106 ceph-mon[36505]: pgmap v1952: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:21:33.964 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3422 sec 2024-03-24T01:21:34.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2768.145809 secs 2024-03-24T01:21:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2768.145809 secs 2024-03-24T01:21:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2768.145809 secs 2024-03-24T01:21:34.964 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3423 sec 2024-03-24T01:21:35.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:35 smithi078 ceph-mon[32429]: pgmap v1953: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:21:35.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:35 smithi107 ceph-mon[36485]: pgmap v1953: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:21:35.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:35 smithi106 ceph-mon[36505]: pgmap v1953: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:21:35.964 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3424 sec 2024-03-24T01:21:36.967 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3425 sec 2024-03-24T01:21:37.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:37 smithi078 ceph-mon[32429]: pgmap v1954: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:21:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:37 smithi107 ceph-mon[36485]: pgmap v1954: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:21:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:37 smithi106 ceph-mon[36505]: pgmap v1954: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:21:37.964 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3426 sec 2024-03-24T01:21:38.964 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3427 sec 2024-03-24T01:21:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:39 smithi078 ceph-mon[32429]: pgmap v1955: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:21:39.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2773.145871 secs 2024-03-24T01:21:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:39 smithi107 ceph-mon[36485]: pgmap v1955: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:21:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2773.145871 secs 2024-03-24T01:21:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:39 smithi106 ceph-mon[36505]: pgmap v1955: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:21:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2773.145871 secs 2024-03-24T01:21:39.964 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3428 sec 2024-03-24T01:21:40.964 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3429 sec 2024-03-24T01:21:41.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:41 smithi078 ceph-mon[32429]: pgmap v1956: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:21:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:41 smithi107 ceph-mon[36485]: pgmap v1956: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:21:41.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:41 smithi106 ceph-mon[36505]: pgmap v1956: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:21:41.964 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3430 sec 2024-03-24T01:21:42.965 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3431 sec 2024-03-24T01:21:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:43 smithi078 ceph-mon[32429]: pgmap v1957: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:21:43.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:43 smithi107 ceph-mon[36485]: pgmap v1957: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:21:43.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:43 smithi106 ceph-mon[36505]: pgmap v1957: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:21:43.965 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3432 sec 2024-03-24T01:21:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2778.145960 secs 2024-03-24T01:21:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2778.145960 secs 2024-03-24T01:21:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2778.145960 secs 2024-03-24T01:21:44.965 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3433 sec 2024-03-24T01:21:45.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:45 smithi078 ceph-mon[32429]: pgmap v1958: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:45.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:45 smithi107 ceph-mon[36485]: pgmap v1958: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:45.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:45 smithi106 ceph-mon[36505]: pgmap v1958: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:45.965 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3434 sec 2024-03-24T01:21:46.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:46 smithi078 ceph-mon[32429]: pgmap v1959: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:46.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:46 smithi107 ceph-mon[36485]: pgmap v1959: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:46.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:46 smithi106 ceph-mon[36505]: pgmap v1959: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:46.965 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3435 sec 2024-03-24T01:21:47.965 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3436 sec 2024-03-24T01:21:48.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:48 smithi106 ceph-mon[36505]: pgmap v1960: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:48.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2783.146048 secs 2024-03-24T01:21:48.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:48 smithi078 ceph-mon[32429]: pgmap v1960: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:48.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2783.146048 secs 2024-03-24T01:21:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:48 smithi107 ceph-mon[36485]: pgmap v1960: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2783.146048 secs 2024-03-24T01:21:48.965 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3437 sec 2024-03-24T01:21:49.965 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3438 sec 2024-03-24T01:21:50.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:50 smithi106 ceph-mon[36505]: pgmap v1961: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:50 smithi078 ceph-mon[32429]: pgmap v1961: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:50 smithi107 ceph-mon[36485]: pgmap v1961: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:50.965 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3439 sec 2024-03-24T01:21:51.966 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3440 sec 2024-03-24T01:21:52.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:52 smithi078 ceph-mon[32429]: pgmap v1962: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:52.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:52 smithi107 ceph-mon[36485]: pgmap v1962: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:52.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:52 smithi106 ceph-mon[36505]: pgmap v1962: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:52.965 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3441 sec 2024-03-24T01:21:53.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2788.146136 secs 2024-03-24T01:21:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2788.146136 secs 2024-03-24T01:21:53.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2788.146136 secs 2024-03-24T01:21:53.966 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3442 sec 2024-03-24T01:21:54.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:54 smithi078 ceph-mon[32429]: pgmap v1963: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:54 smithi107 ceph-mon[36485]: pgmap v1963: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:54 smithi106 ceph-mon[36505]: pgmap v1963: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:54.966 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3443 sec 2024-03-24T01:21:55.966 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3444 sec 2024-03-24T01:21:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:56 smithi078 ceph-mon[32429]: pgmap v1964: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:56 smithi107 ceph-mon[36485]: pgmap v1964: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:56.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:56 smithi106 ceph-mon[36505]: pgmap v1964: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:56.966 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3445 sec 2024-03-24T01:21:57.966 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3446 sec 2024-03-24T01:21:58.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:58 smithi078 ceph-mon[32429]: pgmap v1965: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:58.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:21:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2793.146221 secs 2024-03-24T01:21:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:58 smithi107 ceph-mon[36485]: pgmap v1965: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:21:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2793.146221 secs 2024-03-24T01:21:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:58 smithi106 ceph-mon[36505]: pgmap v1965: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:21:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:21:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2793.146221 secs 2024-03-24T01:21:58.966 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3447 sec 2024-03-24T01:21:59.966 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3448 sec 2024-03-24T01:22:00.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:00 smithi078 ceph-mon[32429]: pgmap v1966: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:00 smithi107 ceph-mon[36485]: pgmap v1966: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:00.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:00 smithi106 ceph-mon[36505]: pgmap v1966: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:00.966 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3449 sec 2024-03-24T01:22:01.966 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3450 sec 2024-03-24T01:22:02.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:02 smithi078 ceph-mon[32429]: pgmap v1967: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:02 smithi107 ceph-mon[36485]: pgmap v1967: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:02.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:02 smithi106 ceph-mon[36505]: pgmap v1967: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:02.966 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3451 sec 2024-03-24T01:22:03.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2798.146286 secs 2024-03-24T01:22:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2798.146286 secs 2024-03-24T01:22:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2798.146286 secs 2024-03-24T01:22:03.966 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3452 sec 2024-03-24T01:22:04.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:04 smithi078 ceph-mon[32429]: pgmap v1968: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:04 smithi107 ceph-mon[36485]: pgmap v1968: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:04 smithi106 ceph-mon[36505]: pgmap v1968: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:04.967 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3453 sec 2024-03-24T01:22:05.967 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3454 sec 2024-03-24T01:22:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:06 smithi078 ceph-mon[32429]: pgmap v1969: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:06 smithi107 ceph-mon[36485]: pgmap v1969: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:06 smithi106 ceph-mon[36505]: pgmap v1969: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:06.967 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3455 sec 2024-03-24T01:22:07.967 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3456 sec 2024-03-24T01:22:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:08 smithi078 ceph-mon[32429]: pgmap v1970: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2803.146341 secs 2024-03-24T01:22:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:08 smithi107 ceph-mon[36485]: pgmap v1970: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2803.146341 secs 2024-03-24T01:22:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:08 smithi106 ceph-mon[36505]: pgmap v1970: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2803.146341 secs 2024-03-24T01:22:08.967 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3457 sec 2024-03-24T01:22:09.967 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3458 sec 2024-03-24T01:22:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:10 smithi078 ceph-mon[32429]: pgmap v1971: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:10 smithi107 ceph-mon[36485]: pgmap v1971: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:10 smithi106 ceph-mon[36505]: pgmap v1971: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:10.967 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3459 sec 2024-03-24T01:22:11.967 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3460 sec 2024-03-24T01:22:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:12 smithi078 ceph-mon[32429]: pgmap v1972: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:12 smithi107 ceph-mon[36485]: pgmap v1972: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:12.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:12 smithi106 ceph-mon[36505]: pgmap v1972: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:12.967 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3461 sec 2024-03-24T01:22:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2808.146402 secs 2024-03-24T01:22:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2808.146402 secs 2024-03-24T01:22:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2808.146402 secs 2024-03-24T01:22:13.967 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3462 sec 2024-03-24T01:22:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:14 smithi078 ceph-mon[32429]: pgmap v1973: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:14 smithi107 ceph-mon[36485]: pgmap v1973: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:14 smithi106 ceph-mon[36505]: pgmap v1973: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:14.968 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3463 sec 2024-03-24T01:22:15.968 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3464 sec 2024-03-24T01:22:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:16 smithi078 ceph-mon[32429]: pgmap v1974: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:16 smithi107 ceph-mon[36485]: pgmap v1974: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:16.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:16 smithi106 ceph-mon[36505]: pgmap v1974: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:16.968 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3465 sec 2024-03-24T01:22:17.968 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3466 sec 2024-03-24T01:22:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:18 smithi078 ceph-mon[32429]: pgmap v1975: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2813.146484 secs 2024-03-24T01:22:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:18 smithi107 ceph-mon[36485]: pgmap v1975: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2813.146484 secs 2024-03-24T01:22:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:18 smithi106 ceph-mon[36505]: pgmap v1975: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:18.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2813.146484 secs 2024-03-24T01:22:18.968 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3467 sec 2024-03-24T01:22:19.968 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3468 sec 2024-03-24T01:22:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:20 smithi078 ceph-mon[32429]: pgmap v1976: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:20 smithi107 ceph-mon[36485]: pgmap v1976: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:20.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:20 smithi106 ceph-mon[36505]: pgmap v1976: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:20.968 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3469 sec 2024-03-24T01:22:21.969 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3470 sec 2024-03-24T01:22:22.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:22 smithi078 ceph-mon[32429]: pgmap v1977: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:22.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:22 smithi107 ceph-mon[36485]: pgmap v1977: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:22.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:22 smithi106 ceph-mon[36505]: pgmap v1977: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:22.968 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3471 sec 2024-03-24T01:22:23.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2818.146544 secs 2024-03-24T01:22:23.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2818.146544 secs 2024-03-24T01:22:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2818.146544 secs 2024-03-24T01:22:23.968 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3472 sec 2024-03-24T01:22:24.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:24 smithi078 ceph-mon[32429]: pgmap v1978: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:24 smithi107 ceph-mon[36485]: pgmap v1978: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:24 smithi106 ceph-mon[36505]: pgmap v1978: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:24.968 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3473 sec 2024-03-24T01:22:25.969 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3474 sec 2024-03-24T01:22:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:26 smithi078 ceph-mon[32429]: pgmap v1979: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:26 smithi107 ceph-mon[36485]: pgmap v1979: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:26.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:26 smithi106 ceph-mon[36505]: pgmap v1979: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:26.969 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3475 sec 2024-03-24T01:22:27.969 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3476 sec 2024-03-24T01:22:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:28 smithi078 ceph-mon[32429]: pgmap v1980: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:28.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2823.146632 secs 2024-03-24T01:22:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:28 smithi107 ceph-mon[36485]: pgmap v1980: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2823.146632 secs 2024-03-24T01:22:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:28 smithi106 ceph-mon[36505]: pgmap v1980: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2823.146632 secs 2024-03-24T01:22:28.969 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3477 sec 2024-03-24T01:22:29.969 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3478 sec 2024-03-24T01:22:30.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:30 smithi078 ceph-mon[32429]: pgmap v1981: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:30 smithi107 ceph-mon[36485]: pgmap v1981: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:30 smithi106 ceph-mon[36505]: pgmap v1981: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:30.969 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3479 sec 2024-03-24T01:22:31.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:31 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:22:31.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:31 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:22:31.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:31 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:22:31.969 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3480 sec 2024-03-24T01:22:32.537 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:32 smithi107 ceph-mon[36485]: pgmap v1982: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:32.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:32 smithi078 ceph-mon[32429]: pgmap v1982: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:32.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:32 smithi106 ceph-mon[36505]: pgmap v1982: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:32.969 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3481 sec 2024-03-24T01:22:33.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2828.146702 secs 2024-03-24T01:22:33.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:33 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:33.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:33 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2828.146702 secs 2024-03-24T01:22:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:33 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:33 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:33.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2828.146702 secs 2024-03-24T01:22:33.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:33 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:33.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:33 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:33.969 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3482 sec 2024-03-24T01:22:34.970 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3483 sec 2024-03-24T01:22:35.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:34 smithi078 ceph-mon[32429]: pgmap v1983: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:35.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:34 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:35.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:34 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:35.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:34 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:35.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:34 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:35.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:34 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:35.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:34 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:35.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:34 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:35.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:34 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:35.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:34 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:35.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:34 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:35.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:34 smithi107 ceph-mon[36485]: pgmap v1983: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:35.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:34 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:35.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:34 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:35.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:34 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:35.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:34 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:35.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:34 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:35.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:34 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:35.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:34 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:35.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:34 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:35.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:34 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:35.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:34 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:35.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:34 smithi106 ceph-mon[36505]: pgmap v1983: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:35.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:34 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:35.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:34 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:35.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:34 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:35.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:34 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:35.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:34 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:35.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:34 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:35.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:34 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:35.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:34 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:35.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:34 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:35.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:34 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:35.969 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3484 sec 2024-03-24T01:22:36.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:35 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T01:22:36.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:35 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T01:22:36.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:36.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:36.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:36.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:36.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:36.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:36.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:35 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T01:22:36.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:35 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T01:22:36.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:36.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:36.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:36.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:36.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:36.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:36.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:35 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T01:22:36.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:35 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T01:22:36.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:22:36.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:22:36.093 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:35 smithi078 ceph-mon[32429]: pgmap v1984: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:36.093 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:35 smithi078 ceph-mon[32429]: pgmap v1985: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-24T01:22:36.093 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:36.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:35 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T01:22:36.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:35 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T01:22:36.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:36.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:36.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:36.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:36.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:36.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:36.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:35 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T01:22:36.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:35 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T01:22:36.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:36.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:36.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:36.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:36.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:36.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:36.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:35 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T01:22:36.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:35 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T01:22:36.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:22:36.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:22:36.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:35 smithi107 ceph-mon[36485]: pgmap v1984: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:36.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:35 smithi107 ceph-mon[36485]: pgmap v1985: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-24T01:22:36.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:36.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:35 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T01:22:36.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:35 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T01:22:36.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:36.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:36.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:36.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:36.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:36.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:36.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:35 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T01:22:36.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:35 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T01:22:36.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:36.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:36.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:36.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:36.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:36.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:22:36.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:35 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T01:22:36.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:35 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T01:22:36.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:22:36.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:22:36.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:35 smithi106 ceph-mon[36505]: pgmap v1984: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:36.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:35 smithi106 ceph-mon[36505]: pgmap v1985: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-24T01:22:36.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:22:36.969 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3485 sec 2024-03-24T01:22:37.969 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3486 sec 2024-03-24T01:22:38.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:37 smithi078 ceph-mon[32429]: pgmap v1986: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:22:38.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:37 smithi107 ceph-mon[36485]: pgmap v1986: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:22:38.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:37 smithi106 ceph-mon[36505]: pgmap v1986: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:22:38.969 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3487 sec 2024-03-24T01:22:39.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2833.146759 secs 2024-03-24T01:22:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2833.146759 secs 2024-03-24T01:22:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2833.146759 secs 2024-03-24T01:22:39.969 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3488 sec 2024-03-24T01:22:40.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:39 smithi078 ceph-mon[32429]: pgmap v1987: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:22:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:39 smithi107 ceph-mon[36485]: pgmap v1987: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:22:40.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:39 smithi106 ceph-mon[36505]: pgmap v1987: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:22:40.970 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3489 sec 2024-03-24T01:22:41.970 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3490 sec 2024-03-24T01:22:42.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:42 smithi078 ceph-mon[32429]: pgmap v1988: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:22:42.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:42 smithi107 ceph-mon[36485]: pgmap v1988: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:22:42.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:42 smithi106 ceph-mon[36505]: pgmap v1988: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:22:42.970 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3491 sec 2024-03-24T01:22:43.970 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3492 sec 2024-03-24T01:22:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2838.146846 secs 2024-03-24T01:22:44.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:44 smithi078 ceph-mon[32429]: pgmap v1989: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:22:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2838.146846 secs 2024-03-24T01:22:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:44 smithi107 ceph-mon[36485]: pgmap v1989: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:22:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2838.146846 secs 2024-03-24T01:22:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:44 smithi106 ceph-mon[36505]: pgmap v1989: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:22:44.970 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3493 sec 2024-03-24T01:22:45.970 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3494 sec 2024-03-24T01:22:46.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:46 smithi078 ceph-mon[32429]: pgmap v1990: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:22:46.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:46 smithi107 ceph-mon[36485]: pgmap v1990: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:22:46.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:46 smithi106 ceph-mon[36505]: pgmap v1990: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:22:46.970 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3495 sec 2024-03-24T01:22:47.970 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3496 sec 2024-03-24T01:22:48.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:48 smithi078 ceph-mon[32429]: pgmap v1991: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:48.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:48 smithi107 ceph-mon[36485]: pgmap v1991: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:48.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:48 smithi106 ceph-mon[36505]: pgmap v1991: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:48.970 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3497 sec 2024-03-24T01:22:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2843.146934 secs 2024-03-24T01:22:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2843.146934 secs 2024-03-24T01:22:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2843.146934 secs 2024-03-24T01:22:49.971 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3498 sec 2024-03-24T01:22:50.305 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:50 smithi106 ceph-mon[36505]: pgmap v1992: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:50.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:50 smithi078 ceph-mon[32429]: pgmap v1992: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:50 smithi107 ceph-mon[36485]: pgmap v1992: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:50.971 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3499 sec 2024-03-24T01:22:51.971 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3500 sec 2024-03-24T01:22:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:52 smithi078 ceph-mon[32429]: pgmap v1993: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:52.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:52 smithi107 ceph-mon[36485]: pgmap v1993: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:52.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:52 smithi106 ceph-mon[36505]: pgmap v1993: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:52.971 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3501 sec 2024-03-24T01:22:53.971 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3502 sec 2024-03-24T01:22:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2848.146975 secs 2024-03-24T01:22:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:54 smithi078 ceph-mon[32429]: pgmap v1994: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2848.146975 secs 2024-03-24T01:22:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:54 smithi107 ceph-mon[36485]: pgmap v1994: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2848.146975 secs 2024-03-24T01:22:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:54 smithi106 ceph-mon[36505]: pgmap v1994: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:54.971 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3503 sec 2024-03-24T01:22:55.971 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3504 sec 2024-03-24T01:22:56.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:56 smithi078 ceph-mon[32429]: pgmap v1995: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:56.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:56 smithi107 ceph-mon[36485]: pgmap v1995: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:56.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:56 smithi106 ceph-mon[36505]: pgmap v1995: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:22:56.975 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3505 sec 2024-03-24T01:22:57.971 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3506 sec 2024-03-24T01:22:58.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:58 smithi078 ceph-mon[32429]: pgmap v1996: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:58 smithi107 ceph-mon[36485]: pgmap v1996: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:58.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:58 smithi106 ceph-mon[36505]: pgmap v1996: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:58.971 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3507 sec 2024-03-24T01:22:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:22:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2853.147065 secs 2024-03-24T01:22:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:22:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2853.147065 secs 2024-03-24T01:22:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:22:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2853.147065 secs 2024-03-24T01:22:59.971 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3508 sec 2024-03-24T01:23:00.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:00 smithi078 ceph-mon[32429]: pgmap v1997: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:00.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:00 smithi107 ceph-mon[36485]: pgmap v1997: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:00.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:00 smithi106 ceph-mon[36505]: pgmap v1997: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:00.971 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3509 sec 2024-03-24T01:23:01.971 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3510 sec 2024-03-24T01:23:02.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:02 smithi078 ceph-mon[32429]: pgmap v1998: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:02.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:02 smithi107 ceph-mon[36485]: pgmap v1998: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:02.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:02 smithi106 ceph-mon[36505]: pgmap v1998: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:02.972 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3511 sec 2024-03-24T01:23:03.972 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3512 sec 2024-03-24T01:23:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2858.147151 secs 2024-03-24T01:23:04.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:04 smithi078 ceph-mon[32429]: pgmap v1999: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2858.147151 secs 2024-03-24T01:23:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:04 smithi107 ceph-mon[36485]: pgmap v1999: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2858.147151 secs 2024-03-24T01:23:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:04 smithi106 ceph-mon[36505]: pgmap v1999: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:04.972 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3513 sec 2024-03-24T01:23:05.972 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3514 sec 2024-03-24T01:23:06.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:06 smithi078 ceph-mon[32429]: pgmap v2000: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:06.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:06 smithi107 ceph-mon[36485]: pgmap v2000: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:06.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:06 smithi106 ceph-mon[36505]: pgmap v2000: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:06.972 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3515 sec 2024-03-24T01:23:07.972 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3516 sec 2024-03-24T01:23:08.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:08 smithi078 ceph-mon[32429]: pgmap v2001: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:08.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:08 smithi107 ceph-mon[36485]: pgmap v2001: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:08.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:08 smithi106 ceph-mon[36505]: pgmap v2001: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:08.972 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3517 sec 2024-03-24T01:23:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2863.147238 secs 2024-03-24T01:23:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2863.147238 secs 2024-03-24T01:23:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2863.147238 secs 2024-03-24T01:23:09.972 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3518 sec 2024-03-24T01:23:10.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:10 smithi078 ceph-mon[32429]: pgmap v2002: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:10.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:10 smithi107 ceph-mon[36485]: pgmap v2002: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:10.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:10 smithi106 ceph-mon[36505]: pgmap v2002: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:10.972 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3519 sec 2024-03-24T01:23:11.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:11 smithi078 ceph-mon[32429]: pgmap v2003: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:11.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:11 smithi107 ceph-mon[36485]: pgmap v2003: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:11.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:11 smithi106 ceph-mon[36505]: pgmap v2003: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:11.972 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3520 sec 2024-03-24T01:23:12.972 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3521 sec 2024-03-24T01:23:13.972 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3522 sec 2024-03-24T01:23:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2868.147319 secs 2024-03-24T01:23:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:14 smithi078 ceph-mon[32429]: pgmap v2004: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2868.147319 secs 2024-03-24T01:23:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:14 smithi107 ceph-mon[36485]: pgmap v2004: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2868.147319 secs 2024-03-24T01:23:14.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:14 smithi106 ceph-mon[36505]: pgmap v2004: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:14.973 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3523 sec 2024-03-24T01:23:15.973 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3524 sec 2024-03-24T01:23:16.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:16 smithi078 ceph-mon[32429]: pgmap v2005: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:16.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:16 smithi107 ceph-mon[36485]: pgmap v2005: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:16.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:16 smithi106 ceph-mon[36505]: pgmap v2005: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:16.973 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3525 sec 2024-03-24T01:23:17.973 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3526 sec 2024-03-24T01:23:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:18 smithi078 ceph-mon[32429]: pgmap v2006: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:18 smithi107 ceph-mon[36485]: pgmap v2006: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:18.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:18 smithi106 ceph-mon[36505]: pgmap v2006: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:18.973 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3527 sec 2024-03-24T01:23:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2873.147407 secs 2024-03-24T01:23:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2873.147407 secs 2024-03-24T01:23:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2873.147407 secs 2024-03-24T01:23:19.973 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3528 sec 2024-03-24T01:23:20.303 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:20 smithi106 ceph-mon[36505]: pgmap v2007: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:20.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:20 smithi078 ceph-mon[32429]: pgmap v2007: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:20 smithi107 ceph-mon[36485]: pgmap v2007: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:20.973 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3529 sec 2024-03-24T01:23:21.973 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3530 sec 2024-03-24T01:23:22.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:22 smithi078 ceph-mon[32429]: pgmap v2008: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:22.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:22 smithi107 ceph-mon[36485]: pgmap v2008: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:22.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:22 smithi106 ceph-mon[36505]: pgmap v2008: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:22.973 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3531 sec 2024-03-24T01:23:23.973 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3532 sec 2024-03-24T01:23:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2878.147466 secs 2024-03-24T01:23:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:24 smithi078 ceph-mon[32429]: pgmap v2009: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2878.147466 secs 2024-03-24T01:23:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:24 smithi107 ceph-mon[36485]: pgmap v2009: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2878.147466 secs 2024-03-24T01:23:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:24 smithi106 ceph-mon[36505]: pgmap v2009: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:24.973 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3533 sec 2024-03-24T01:23:25.973 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3534 sec 2024-03-24T01:23:26.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:26 smithi078 ceph-mon[32429]: pgmap v2010: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:26.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:26 smithi107 ceph-mon[36485]: pgmap v2010: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:26.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:26 smithi106 ceph-mon[36505]: pgmap v2010: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:26.974 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3535 sec 2024-03-24T01:23:27.974 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3536 sec 2024-03-24T01:23:28.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:28 smithi078 ceph-mon[32429]: pgmap v2011: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:28.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:28 smithi107 ceph-mon[36485]: pgmap v2011: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:28.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:28 smithi106 ceph-mon[36505]: pgmap v2011: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:28.974 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3537 sec 2024-03-24T01:23:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:29 smithi078 ceph-mon[32429]: 2 slow requests, 1 included below; oldest blocked for > 2883.147551 secs 2024-03-24T01:23:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:29 smithi078 ceph-mon[32429]: slow request 2883.147551 seconds old, received at 2024-03-24T00:35:24.775244+0000: peer_request:client.15462:3775 currently acquired locks 2024-03-24T01:23:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:29 smithi107 ceph-mon[36485]: 2 slow requests, 1 included below; oldest blocked for > 2883.147551 secs 2024-03-24T01:23:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:29 smithi107 ceph-mon[36485]: slow request 2883.147551 seconds old, received at 2024-03-24T00:35:24.775244+0000: peer_request:client.15462:3775 currently acquired locks 2024-03-24T01:23:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:29 smithi106 ceph-mon[36505]: 2 slow requests, 1 included below; oldest blocked for > 2883.147551 secs 2024-03-24T01:23:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:29 smithi106 ceph-mon[36505]: slow request 2883.147551 seconds old, received at 2024-03-24T00:35:24.775244+0000: peer_request:client.15462:3775 currently acquired locks 2024-03-24T01:23:29.974 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3538 sec 2024-03-24T01:23:30.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:30 smithi078 ceph-mon[32429]: pgmap v2012: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:30.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:30 smithi107 ceph-mon[36485]: pgmap v2012: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:30.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:30 smithi106 ceph-mon[36505]: pgmap v2012: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:30.974 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3539 sec 2024-03-24T01:23:31.974 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3540 sec 2024-03-24T01:23:32.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:32 smithi078 ceph-mon[32429]: pgmap v2013: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:32.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:32 smithi107 ceph-mon[36485]: pgmap v2013: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:32.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:32 smithi106 ceph-mon[36505]: pgmap v2013: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:32.974 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3541 sec 2024-03-24T01:23:33.974 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3542 sec 2024-03-24T01:23:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2888.147612 secs 2024-03-24T01:23:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:34 smithi078 ceph-mon[32429]: pgmap v2014: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:34.345 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2888.147612 secs 2024-03-24T01:23:34.345 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:34 smithi107 ceph-mon[36485]: pgmap v2014: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2888.147612 secs 2024-03-24T01:23:34.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:34 smithi106 ceph-mon[36505]: pgmap v2014: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:34.977 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3543 sec 2024-03-24T01:23:35.974 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3544 sec 2024-03-24T01:23:36.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:36 smithi078 ceph-mon[32429]: pgmap v2015: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:36.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:23:36.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:23:36.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:23:36.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:23:36.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:23:36.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:23:36.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:23:36.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:36 smithi107 ceph-mon[36485]: pgmap v2015: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:36.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:23:36.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:23:36.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:23:36.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:23:36.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:23:36.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:23:36.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:23:36.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:36 smithi106 ceph-mon[36505]: pgmap v2015: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:36.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:23:36.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:23:36.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:23:36.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:23:36.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:23:36.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:23:36.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:23:36.974 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3545 sec 2024-03-24T01:23:37.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:23:37.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:23:37.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:23:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:23:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:23:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:23:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:23:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:23:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:23:37.974 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3546 sec 2024-03-24T01:23:38.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:38 smithi078 ceph-mon[32429]: pgmap v2016: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T01:23:38.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:38 smithi078 ceph-mon[32429]: pgmap v2017: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:23:38.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:38 smithi107 ceph-mon[36485]: pgmap v2016: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T01:23:38.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:38 smithi107 ceph-mon[36485]: pgmap v2017: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:23:38.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:38 smithi106 ceph-mon[36505]: pgmap v2016: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T01:23:38.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:38 smithi106 ceph-mon[36505]: pgmap v2017: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:23:38.974 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3547 sec 2024-03-24T01:23:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2893.147674 secs 2024-03-24T01:23:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2893.147674 secs 2024-03-24T01:23:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2893.147674 secs 2024-03-24T01:23:39.974 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3548 sec 2024-03-24T01:23:40.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:40 smithi078 ceph-mon[32429]: pgmap v2018: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:23:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:40 smithi107 ceph-mon[36485]: pgmap v2018: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:23:40.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:40 smithi106 ceph-mon[36505]: pgmap v2018: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:23:40.975 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3549 sec 2024-03-24T01:23:41.975 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3550 sec 2024-03-24T01:23:42.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:42 smithi078 ceph-mon[32429]: pgmap v2019: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:23:42.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:42 smithi107 ceph-mon[36485]: pgmap v2019: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:23:42.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:42 smithi106 ceph-mon[36505]: pgmap v2019: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:23:42.975 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3551 sec 2024-03-24T01:23:43.975 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3552 sec 2024-03-24T01:23:44.143 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:44 smithi078 ceph-mon[32429]: pgmap v2020: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:23:44.143 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2898.147734 secs 2024-03-24T01:23:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:44 smithi107 ceph-mon[36485]: pgmap v2020: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:23:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2898.147734 secs 2024-03-24T01:23:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:44 smithi106 ceph-mon[36505]: pgmap v2020: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:23:44.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2898.147734 secs 2024-03-24T01:23:44.975 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3553 sec 2024-03-24T01:23:45.975 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3554 sec 2024-03-24T01:23:46.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:46 smithi078 ceph-mon[32429]: pgmap v2021: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:23:46.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:46 smithi107 ceph-mon[36485]: pgmap v2021: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:23:46.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:46 smithi106 ceph-mon[36505]: pgmap v2021: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:23:46.975 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3555 sec 2024-03-24T01:23:47.975 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3556 sec 2024-03-24T01:23:48.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:48 smithi078 ceph-mon[32429]: pgmap v2022: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:23:48.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:48 smithi107 ceph-mon[36485]: pgmap v2022: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:23:48.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:48 smithi106 ceph-mon[36505]: pgmap v2022: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:23:48.976 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3557 sec 2024-03-24T01:23:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2903.147821 secs 2024-03-24T01:23:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2903.147821 secs 2024-03-24T01:23:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2903.147821 secs 2024-03-24T01:23:49.976 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3558 sec 2024-03-24T01:23:50.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:50 smithi078 ceph-mon[32429]: pgmap v2023: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:50 smithi107 ceph-mon[36485]: pgmap v2023: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:50.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:50 smithi106 ceph-mon[36505]: pgmap v2023: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:50.976 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3559 sec 2024-03-24T01:23:51.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:51 smithi078 ceph-mon[32429]: pgmap v2024: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:51.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:51 smithi107 ceph-mon[36485]: pgmap v2024: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:51.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:51 smithi106 ceph-mon[36505]: pgmap v2024: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:51.976 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3560 sec 2024-03-24T01:23:52.976 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3561 sec 2024-03-24T01:23:53.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:53 smithi078 ceph-mon[32429]: pgmap v2025: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:53.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2908.147904 secs 2024-03-24T01:23:53.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:53 smithi107 ceph-mon[36485]: pgmap v2025: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:53.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2908.147904 secs 2024-03-24T01:23:53.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:53 smithi106 ceph-mon[36505]: pgmap v2025: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:53.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2908.147904 secs 2024-03-24T01:23:53.976 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3562 sec 2024-03-24T01:23:54.976 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3563 sec 2024-03-24T01:23:55.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:55 smithi078 ceph-mon[32429]: pgmap v2026: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:55.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:55 smithi107 ceph-mon[36485]: pgmap v2026: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:55.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:55 smithi106 ceph-mon[36505]: pgmap v2026: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:55.976 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3564 sec 2024-03-24T01:23:56.976 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3565 sec 2024-03-24T01:23:57.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:57 smithi078 ceph-mon[32429]: pgmap v2027: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:57.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:57 smithi107 ceph-mon[36485]: pgmap v2027: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:57.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:57 smithi106 ceph-mon[36505]: pgmap v2027: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:57.976 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3566 sec 2024-03-24T01:23:58.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2913.147962 secs 2024-03-24T01:23:58.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2913.147962 secs 2024-03-24T01:23:58.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2913.147962 secs 2024-03-24T01:23:58.976 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3567 sec 2024-03-24T01:23:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:23:59 smithi078 ceph-mon[32429]: pgmap v2028: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:23:59 smithi107 ceph-mon[36485]: pgmap v2028: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:23:59 smithi106 ceph-mon[36505]: pgmap v2028: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:23:59.976 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3568 sec 2024-03-24T01:24:00.976 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3569 sec 2024-03-24T01:24:01.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:01 smithi078 ceph-mon[32429]: pgmap v2029: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:24:01.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:01 smithi107 ceph-mon[36485]: pgmap v2029: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:24:01.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:01 smithi106 ceph-mon[36505]: pgmap v2029: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:24:01.977 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3570 sec 2024-03-24T01:24:02.977 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3571 sec 2024-03-24T01:24:03.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:03 smithi078 ceph-mon[32429]: pgmap v2030: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:03.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:03 smithi078 ceph-mon[32429]: 2 slow requests, 1 included below; oldest blocked for > 2918.148047 secs 2024-03-24T01:24:03.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:03 smithi078 ceph-mon[32429]: slow request 2882.880975 seconds old, received at 2024-03-24T00:36:00.042315+0000: client_request(client.25385:166 mksnap #0x1//scheduled-2024-03-24-00_36_00_UTC 2024-03-24T00:36:00.042488+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-24T01:24:03.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:03 smithi107 ceph-mon[36485]: pgmap v2030: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:03.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:03 smithi107 ceph-mon[36485]: 2 slow requests, 1 included below; oldest blocked for > 2918.148047 secs 2024-03-24T01:24:03.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:03 smithi107 ceph-mon[36485]: slow request 2882.880975 seconds old, received at 2024-03-24T00:36:00.042315+0000: client_request(client.25385:166 mksnap #0x1//scheduled-2024-03-24-00_36_00_UTC 2024-03-24T00:36:00.042488+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-24T01:24:03.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:03 smithi106 ceph-mon[36505]: pgmap v2030: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:03.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:03 smithi106 ceph-mon[36505]: 2 slow requests, 1 included below; oldest blocked for > 2918.148047 secs 2024-03-24T01:24:03.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:03 smithi106 ceph-mon[36505]: slow request 2882.880975 seconds old, received at 2024-03-24T00:36:00.042315+0000: client_request(client.25385:166 mksnap #0x1//scheduled-2024-03-24-00_36_00_UTC 2024-03-24T00:36:00.042488+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-24T01:24:03.977 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3572 sec 2024-03-24T01:24:04.977 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3573 sec 2024-03-24T01:24:05.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:05 smithi078 ceph-mon[32429]: pgmap v2031: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:24:05.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:05 smithi107 ceph-mon[36485]: pgmap v2031: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:24:05.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:05 smithi106 ceph-mon[36505]: pgmap v2031: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:24:05.977 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3574 sec 2024-03-24T01:24:06.977 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3575 sec 2024-03-24T01:24:07.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:07 smithi078 ceph-mon[32429]: pgmap v2032: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:07.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:07 smithi107 ceph-mon[36485]: pgmap v2032: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:07.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:07 smithi106 ceph-mon[36505]: pgmap v2032: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:07.977 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3576 sec 2024-03-24T01:24:08.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2923.148088 secs 2024-03-24T01:24:08.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2923.148088 secs 2024-03-24T01:24:08.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2923.148088 secs 2024-03-24T01:24:08.977 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3577 sec 2024-03-24T01:24:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:09 smithi078 ceph-mon[32429]: pgmap v2033: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:24:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:09 smithi107 ceph-mon[36485]: pgmap v2033: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:24:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:09 smithi106 ceph-mon[36505]: pgmap v2033: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:24:09.977 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3578 sec 2024-03-24T01:24:10.977 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3579 sec 2024-03-24T01:24:11.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:11 smithi078 ceph-mon[32429]: pgmap v2034: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:24:11.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:11 smithi107 ceph-mon[36485]: pgmap v2034: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:24:11.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:11 smithi106 ceph-mon[36505]: pgmap v2034: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:24:11.977 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3580 sec 2024-03-24T01:24:12.978 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3581 sec 2024-03-24T01:24:13.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:13 smithi078 ceph-mon[32429]: pgmap v2035: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:13.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2928.148175 secs 2024-03-24T01:24:13.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:13 smithi107 ceph-mon[36485]: pgmap v2035: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:13.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2928.148175 secs 2024-03-24T01:24:13.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:13 smithi106 ceph-mon[36505]: pgmap v2035: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:13.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2928.148175 secs 2024-03-24T01:24:13.978 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3582 sec 2024-03-24T01:24:14.978 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3583 sec 2024-03-24T01:24:15.978 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3584 sec 2024-03-24T01:24:16.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:15 smithi078 ceph-mon[32429]: pgmap v2036: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:24:16.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:15 smithi078 ceph-mon[32429]: osdmap e157: 12 total, 12 up, 12 in 2024-03-24T01:24:16.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:15 smithi107 ceph-mon[36485]: pgmap v2036: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:24:16.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:15 smithi107 ceph-mon[36485]: osdmap e157: 12 total, 12 up, 12 in 2024-03-24T01:24:16.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:15 smithi106 ceph-mon[36505]: pgmap v2036: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:24:16.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:15 smithi106 ceph-mon[36505]: osdmap e157: 12 total, 12 up, 12 in 2024-03-24T01:24:16.978 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3585 sec 2024-03-24T01:24:17.978 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3586 sec 2024-03-24T01:24:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:17 smithi078 ceph-mon[32429]: pgmap v2038: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:24:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:17 smithi107 ceph-mon[36485]: pgmap v2038: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:24:18.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:17 smithi106 ceph-mon[36505]: pgmap v2038: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:24:18.978 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3587 sec 2024-03-24T01:24:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2933.148264 secs 2024-03-24T01:24:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2933.148264 secs 2024-03-24T01:24:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2933.148264 secs 2024-03-24T01:24:19.978 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3588 sec 2024-03-24T01:24:20.303 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:19 smithi106 ceph-mon[36505]: pgmap v2039: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:24:20.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:19 smithi078 ceph-mon[32429]: pgmap v2039: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:24:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:19 smithi107 ceph-mon[36485]: pgmap v2039: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:24:20.978 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3589 sec 2024-03-24T01:24:21.978 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3590 sec 2024-03-24T01:24:22.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:21 smithi078 ceph-mon[32429]: pgmap v2040: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:24:22.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:21 smithi107 ceph-mon[36485]: pgmap v2040: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:24:22.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:21 smithi106 ceph-mon[36505]: pgmap v2040: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:24:22.979 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3591 sec 2024-03-24T01:24:23.979 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3592 sec 2024-03-24T01:24:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:23 smithi078 ceph-mon[32429]: pgmap v2041: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:24:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2938.148347 secs 2024-03-24T01:24:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:23 smithi107 ceph-mon[36485]: pgmap v2041: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:24:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2938.148347 secs 2024-03-24T01:24:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:23 smithi106 ceph-mon[36505]: pgmap v2041: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:24:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2938.148347 secs 2024-03-24T01:24:24.979 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3593 sec 2024-03-24T01:24:25.979 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3594 sec 2024-03-24T01:24:26.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:25 smithi078 ceph-mon[32429]: pgmap v2042: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:24:26.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:25 smithi107 ceph-mon[36485]: pgmap v2042: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:24:26.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:25 smithi106 ceph-mon[36505]: pgmap v2042: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:24:26.979 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3595 sec 2024-03-24T01:24:27.979 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3596 sec 2024-03-24T01:24:28.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:27 smithi078 ceph-mon[32429]: pgmap v2043: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T01:24:28.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:27 smithi107 ceph-mon[36485]: pgmap v2043: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T01:24:28.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:27 smithi106 ceph-mon[36505]: pgmap v2043: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T01:24:28.979 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3597 sec 2024-03-24T01:24:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2943.148362 secs 2024-03-24T01:24:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2943.148362 secs 2024-03-24T01:24:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2943.148362 secs 2024-03-24T01:24:29.979 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3598 sec 2024-03-24T01:24:30.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:29 smithi078 ceph-mon[32429]: pgmap v2044: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:24:30.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:29 smithi107 ceph-mon[36485]: pgmap v2044: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:24:30.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:29 smithi106 ceph-mon[36505]: pgmap v2044: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:24:30.979 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3599 sec 2024-03-24T01:24:31.979 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3600 sec 2024-03-24T01:24:32.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:31 smithi078 ceph-mon[32429]: pgmap v2045: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:24:32.345 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:31 smithi107 ceph-mon[36485]: pgmap v2045: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:24:32.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:31 smithi106 ceph-mon[36505]: pgmap v2045: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:24:32.979 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3601 sec 2024-03-24T01:24:33.979 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3602 sec 2024-03-24T01:24:34.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:33 smithi078 ceph-mon[32429]: pgmap v2046: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:34.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2948.148449 secs 2024-03-24T01:24:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:33 smithi107 ceph-mon[36485]: pgmap v2046: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2948.148449 secs 2024-03-24T01:24:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:33 smithi106 ceph-mon[36505]: pgmap v2046: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:34.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2948.148449 secs 2024-03-24T01:24:34.980 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3603 sec 2024-03-24T01:24:35.979 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3604 sec 2024-03-24T01:24:36.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:35 smithi078 ceph-mon[32429]: pgmap v2047: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:24:36.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:35 smithi107 ceph-mon[36485]: pgmap v2047: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:24:36.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:35 smithi106 ceph-mon[36505]: pgmap v2047: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:24:36.979 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3605 sec 2024-03-24T01:24:37.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:24:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:24:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:24:37.980 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3606 sec 2024-03-24T01:24:38.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:37 smithi107 ceph-mon[36485]: pgmap v2048: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:38.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:24:38.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:24:38.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:37 smithi107 ceph-mon[36485]: pgmap v2049: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:24:38.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:37 smithi107 ceph-mon[36485]: pgmap v2050: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:24:38.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:24:38.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:37 smithi078 ceph-mon[32429]: pgmap v2048: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:38.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:24:38.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:24:38.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:37 smithi078 ceph-mon[32429]: pgmap v2049: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:24:38.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:37 smithi078 ceph-mon[32429]: pgmap v2050: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:24:38.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:24:38.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:37 smithi106 ceph-mon[36505]: pgmap v2048: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:38.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:24:38.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:24:38.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:37 smithi106 ceph-mon[36505]: pgmap v2049: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:24:38.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:37 smithi106 ceph-mon[36505]: pgmap v2050: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:24:38.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:24:38.980 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3607 sec 2024-03-24T01:24:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2953.148512 secs 2024-03-24T01:24:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2953.148512 secs 2024-03-24T01:24:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2953.148512 secs 2024-03-24T01:24:39.980 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3608 sec 2024-03-24T01:24:40.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:39 smithi078 ceph-mon[32429]: pgmap v2051: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:24:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:39 smithi107 ceph-mon[36485]: pgmap v2051: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:24:40.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:39 smithi106 ceph-mon[36505]: pgmap v2051: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:24:40.980 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3609 sec 2024-03-24T01:24:41.980 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3610 sec 2024-03-24T01:24:42.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:42 smithi078 ceph-mon[32429]: pgmap v2052: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:24:42.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:42 smithi107 ceph-mon[36485]: pgmap v2052: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:24:42.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:42 smithi106 ceph-mon[36505]: pgmap v2052: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:24:42.980 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3611 sec 2024-03-24T01:24:43.980 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3612 sec 2024-03-24T01:24:44.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2958.148572 secs 2024-03-24T01:24:44.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:44 smithi078 ceph-mon[32429]: pgmap v2053: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:24:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2958.148572 secs 2024-03-24T01:24:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:44 smithi107 ceph-mon[36485]: pgmap v2053: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:24:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2958.148572 secs 2024-03-24T01:24:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:44 smithi106 ceph-mon[36505]: pgmap v2053: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:24:44.980 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3613 sec 2024-03-24T01:24:45.980 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3614 sec 2024-03-24T01:24:46.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:46 smithi078 ceph-mon[32429]: pgmap v2054: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:24:46.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:46 smithi107 ceph-mon[36485]: pgmap v2054: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:24:46.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:46 smithi106 ceph-mon[36505]: pgmap v2054: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:24:46.980 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3615 sec 2024-03-24T01:24:47.980 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3616 sec 2024-03-24T01:24:48.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:48 smithi078 ceph-mon[32429]: pgmap v2055: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:24:48.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:48 smithi107 ceph-mon[36485]: pgmap v2055: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:24:48.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:48 smithi106 ceph-mon[36505]: pgmap v2055: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:24:48.980 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3617 sec 2024-03-24T01:24:49.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2963.148654 secs 2024-03-24T01:24:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2963.148654 secs 2024-03-24T01:24:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2963.148654 secs 2024-03-24T01:24:49.981 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3618 sec 2024-03-24T01:24:50.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:50 smithi078 ceph-mon[32429]: pgmap v2056: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:24:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:50 smithi107 ceph-mon[36485]: pgmap v2056: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:24:50.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:50 smithi106 ceph-mon[36505]: pgmap v2056: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:24:50.981 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3619 sec 2024-03-24T01:24:51.982 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3620 sec 2024-03-24T01:24:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:52 smithi078 ceph-mon[32429]: pgmap v2057: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:52.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:52 smithi107 ceph-mon[36485]: pgmap v2057: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:52.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:52 smithi106 ceph-mon[36505]: pgmap v2057: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:52.981 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3621 sec 2024-03-24T01:24:53.981 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3622 sec 2024-03-24T01:24:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2968.148715 secs 2024-03-24T01:24:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:54 smithi078 ceph-mon[32429]: pgmap v2058: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:24:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2968.148715 secs 2024-03-24T01:24:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:54 smithi107 ceph-mon[36485]: pgmap v2058: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:24:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2968.148715 secs 2024-03-24T01:24:54.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:54 smithi106 ceph-mon[36505]: pgmap v2058: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:24:54.981 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3623 sec 2024-03-24T01:24:55.981 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3624 sec 2024-03-24T01:24:56.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:56 smithi078 ceph-mon[32429]: pgmap v2059: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:24:56.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:56 smithi107 ceph-mon[36485]: pgmap v2059: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:24:56.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:56 smithi106 ceph-mon[36505]: pgmap v2059: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:24:56.981 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3625 sec 2024-03-24T01:24:57.981 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3626 sec 2024-03-24T01:24:58.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:58 smithi078 ceph-mon[32429]: pgmap v2060: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:58 smithi107 ceph-mon[36485]: pgmap v2060: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:58.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:58 smithi106 ceph-mon[36505]: pgmap v2060: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:58.981 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3627 sec 2024-03-24T01:24:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:24:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2973.148805 secs 2024-03-24T01:24:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:24:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2973.148805 secs 2024-03-24T01:24:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:24:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2973.148805 secs 2024-03-24T01:24:59.981 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3628 sec 2024-03-24T01:25:00.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:00 smithi078 ceph-mon[32429]: pgmap v2061: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:00.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:00 smithi107 ceph-mon[36485]: pgmap v2061: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:00.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:00 smithi106 ceph-mon[36505]: pgmap v2061: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:00.982 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3629 sec 2024-03-24T01:25:01.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:01 smithi078 ceph-mon[32429]: pgmap v2062: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:25:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:01 smithi107 ceph-mon[36485]: pgmap v2062: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:25:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:01 smithi106 ceph-mon[36505]: pgmap v2062: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:25:01.982 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3630 sec 2024-03-24T01:25:02.982 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3631 sec 2024-03-24T01:25:03.982 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3632 sec 2024-03-24T01:25:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2978.148839 secs 2024-03-24T01:25:04.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:04 smithi078 ceph-mon[32429]: pgmap v2063: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2978.148839 secs 2024-03-24T01:25:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:04 smithi107 ceph-mon[36485]: pgmap v2063: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2978.148839 secs 2024-03-24T01:25:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:04 smithi106 ceph-mon[36505]: pgmap v2063: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:04.982 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3633 sec 2024-03-24T01:25:05.982 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3634 sec 2024-03-24T01:25:06.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:06 smithi078 ceph-mon[32429]: pgmap v2064: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:06.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:06 smithi107 ceph-mon[36485]: pgmap v2064: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:06.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:06 smithi106 ceph-mon[36505]: pgmap v2064: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:06.982 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3635 sec 2024-03-24T01:25:07.982 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3636 sec 2024-03-24T01:25:08.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:08 smithi078 ceph-mon[32429]: pgmap v2065: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:25:08.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:08 smithi107 ceph-mon[36485]: pgmap v2065: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:25:08.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:08 smithi106 ceph-mon[36505]: pgmap v2065: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:25:08.982 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3637 sec 2024-03-24T01:25:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2983.148898 secs 2024-03-24T01:25:09.346 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2983.148898 secs 2024-03-24T01:25:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2983.148898 secs 2024-03-24T01:25:09.982 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3638 sec 2024-03-24T01:25:10.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:10 smithi078 ceph-mon[32429]: pgmap v2066: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:10.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:10 smithi107 ceph-mon[36485]: pgmap v2066: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:10.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:10 smithi106 ceph-mon[36505]: pgmap v2066: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:10.982 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3639 sec 2024-03-24T01:25:11.982 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3640 sec 2024-03-24T01:25:12.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:12 smithi078 ceph-mon[32429]: pgmap v2067: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:25:12.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:12 smithi107 ceph-mon[36485]: pgmap v2067: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:25:12.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:12 smithi106 ceph-mon[36505]: pgmap v2067: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:25:12.982 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3641 sec 2024-03-24T01:25:13.983 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3642 sec 2024-03-24T01:25:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2988.148984 secs 2024-03-24T01:25:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:14 smithi078 ceph-mon[32429]: pgmap v2068: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2988.148984 secs 2024-03-24T01:25:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:14 smithi107 ceph-mon[36485]: pgmap v2068: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2988.148984 secs 2024-03-24T01:25:14.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:14 smithi106 ceph-mon[36505]: pgmap v2068: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:14.983 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3643 sec 2024-03-24T01:25:15.983 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3644 sec 2024-03-24T01:25:16.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:16 smithi078 ceph-mon[32429]: pgmap v2069: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:16.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:16 smithi107 ceph-mon[36485]: pgmap v2069: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:16.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:16 smithi106 ceph-mon[36505]: pgmap v2069: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:16.983 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3645 sec 2024-03-24T01:25:17.983 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3646 sec 2024-03-24T01:25:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:18 smithi078 ceph-mon[32429]: pgmap v2070: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:25:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:18 smithi107 ceph-mon[36485]: pgmap v2070: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:25:18.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:18 smithi106 ceph-mon[36505]: pgmap v2070: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:25:18.984 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3647 sec 2024-03-24T01:25:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2993.149043 secs 2024-03-24T01:25:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2993.149043 secs 2024-03-24T01:25:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2993.149043 secs 2024-03-24T01:25:19.983 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3648 sec 2024-03-24T01:25:20.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:20 smithi078 ceph-mon[32429]: pgmap v2071: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:20 smithi107 ceph-mon[36485]: pgmap v2071: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:20.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:20 smithi106 ceph-mon[36505]: pgmap v2071: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:20.983 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3649 sec 2024-03-24T01:25:21.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:21 smithi107 ceph-mon[36485]: pgmap v2072: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:25:21.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:21 smithi106 ceph-mon[36505]: pgmap v2072: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:25:21.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:21 smithi078 ceph-mon[32429]: pgmap v2072: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:25:21.983 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3650 sec 2024-03-24T01:25:22.983 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3651 sec 2024-03-24T01:25:23.983 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3652 sec 2024-03-24T01:25:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 2998.149129 secs 2024-03-24T01:25:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:24 smithi078 ceph-mon[32429]: pgmap v2073: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:25:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 2998.149129 secs 2024-03-24T01:25:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:24 smithi107 ceph-mon[36485]: pgmap v2073: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:25:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 2998.149129 secs 2024-03-24T01:25:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:24 smithi106 ceph-mon[36505]: pgmap v2073: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:25:24.984 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3653 sec 2024-03-24T01:25:25.984 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3654 sec 2024-03-24T01:25:26.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:26 smithi078 ceph-mon[32429]: pgmap v2074: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:25:26.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:26 smithi107 ceph-mon[36485]: pgmap v2074: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:25:26.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:26 smithi106 ceph-mon[36505]: pgmap v2074: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:25:26.984 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3655 sec 2024-03-24T01:25:27.984 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3656 sec 2024-03-24T01:25:28.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:28 smithi078 ceph-mon[32429]: pgmap v2075: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:25:28.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:28 smithi107 ceph-mon[36485]: pgmap v2075: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:25:28.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:28 smithi106 ceph-mon[36505]: pgmap v2075: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:25:28.984 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3657 sec 2024-03-24T01:25:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3003.149187 secs 2024-03-24T01:25:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3003.149187 secs 2024-03-24T01:25:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3003.149187 secs 2024-03-24T01:25:29.984 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3658 sec 2024-03-24T01:25:30.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:30 smithi078 ceph-mon[32429]: pgmap v2076: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:25:30.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:30 smithi107 ceph-mon[36485]: pgmap v2076: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:25:30.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:30 smithi106 ceph-mon[36505]: pgmap v2076: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:25:30.984 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3659 sec 2024-03-24T01:25:31.984 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3660 sec 2024-03-24T01:25:32.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:32 smithi078 ceph-mon[32429]: pgmap v2077: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:25:32.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:32 smithi107 ceph-mon[36485]: pgmap v2077: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:25:32.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:32 smithi106 ceph-mon[36505]: pgmap v2077: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:25:32.984 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3661 sec 2024-03-24T01:25:33.984 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3662 sec 2024-03-24T01:25:34.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3008.149273 secs 2024-03-24T01:25:34.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:34 smithi078 ceph-mon[32429]: pgmap v2078: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3008.149273 secs 2024-03-24T01:25:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:34 smithi107 ceph-mon[36485]: pgmap v2078: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3008.149273 secs 2024-03-24T01:25:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:34 smithi106 ceph-mon[36505]: pgmap v2078: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:34.985 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3663 sec 2024-03-24T01:25:35.984 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3664 sec 2024-03-24T01:25:36.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:36 smithi078 ceph-mon[32429]: pgmap v2079: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:36.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:36 smithi107 ceph-mon[36485]: pgmap v2079: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:36.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:36 smithi106 ceph-mon[36505]: pgmap v2079: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:36.985 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3665 sec 2024-03-24T01:25:37.985 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3666 sec 2024-03-24T01:25:38.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:38 smithi078 ceph-mon[32429]: pgmap v2080: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:25:38.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:38 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:25:38.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:38 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:25:38.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:38 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:25:38.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:38 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:25:38.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:38 smithi107 ceph-mon[36485]: pgmap v2080: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:25:38.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:38 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:25:38.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:38 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:25:38.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:38 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:25:38.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:38 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:25:38.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:38 smithi106 ceph-mon[36505]: pgmap v2080: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:25:38.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:38 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:25:38.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:38 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:25:38.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:38 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:25:38.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:38 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:25:38.985 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3667 sec 2024-03-24T01:25:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:39 smithi078 ceph-mon[32429]: pgmap v2081: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:25:39.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:39 smithi078 ceph-mon[32429]: pgmap v2082: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:25:39.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3013.149350 secs 2024-03-24T01:25:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:39 smithi107 ceph-mon[36485]: pgmap v2081: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:25:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:39 smithi107 ceph-mon[36485]: pgmap v2082: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:25:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3013.149350 secs 2024-03-24T01:25:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:39 smithi106 ceph-mon[36505]: pgmap v2081: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:25:39.405 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:39 smithi106 ceph-mon[36505]: pgmap v2082: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:25:39.405 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3013.149350 secs 2024-03-24T01:25:39.985 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3668 sec 2024-03-24T01:25:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:40 smithi107 ceph-mon[36485]: pgmap v2083: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:25:40.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:40 smithi106 ceph-mon[36505]: pgmap v2083: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:25:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:40 smithi078 ceph-mon[32429]: pgmap v2083: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:25:40.985 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3669 sec 2024-03-24T01:25:41.985 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3670 sec 2024-03-24T01:25:42.985 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3671 sec 2024-03-24T01:25:43.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:42 smithi078 ceph-mon[32429]: pgmap v2084: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:25:43.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:42 smithi107 ceph-mon[36485]: pgmap v2084: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:25:43.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:42 smithi106 ceph-mon[36505]: pgmap v2084: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:25:43.985 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3672 sec 2024-03-24T01:25:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3018.149434 secs 2024-03-24T01:25:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3018.149434 secs 2024-03-24T01:25:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3018.149434 secs 2024-03-24T01:25:44.985 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3673 sec 2024-03-24T01:25:45.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:44 smithi078 ceph-mon[32429]: pgmap v2085: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:25:45.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:44 smithi107 ceph-mon[36485]: pgmap v2085: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:25:45.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:44 smithi106 ceph-mon[36505]: pgmap v2085: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:25:45.986 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3674 sec 2024-03-24T01:25:46.986 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3675 sec 2024-03-24T01:25:47.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:46 smithi107 ceph-mon[36485]: pgmap v2086: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:25:47.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:46 smithi106 ceph-mon[36505]: pgmap v2086: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:25:47.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:46 smithi078 ceph-mon[32429]: pgmap v2086: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:25:47.986 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3676 sec 2024-03-24T01:25:48.986 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3677 sec 2024-03-24T01:25:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:48 smithi078 ceph-mon[32429]: pgmap v2087: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:25:49.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3023.149520 secs 2024-03-24T01:25:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:48 smithi106 ceph-mon[36505]: pgmap v2087: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:25:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3023.149520 secs 2024-03-24T01:25:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:48 smithi107 ceph-mon[36485]: pgmap v2087: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:25:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3023.149520 secs 2024-03-24T01:25:49.986 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3678 sec 2024-03-24T01:25:50.986 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3679 sec 2024-03-24T01:25:51.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:50 smithi106 ceph-mon[36505]: pgmap v2088: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:51.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:50 smithi078 ceph-mon[32429]: pgmap v2088: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:51.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:50 smithi107 ceph-mon[36485]: pgmap v2088: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:51.986 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3680 sec 2024-03-24T01:25:52.986 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3681 sec 2024-03-24T01:25:53.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:52 smithi106 ceph-mon[36505]: pgmap v2089: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:25:53.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:52 smithi078 ceph-mon[32429]: pgmap v2089: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:25:53.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:52 smithi107 ceph-mon[36485]: pgmap v2089: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:25:53.986 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3682 sec 2024-03-24T01:25:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3028.149624 secs 2024-03-24T01:25:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3028.149624 secs 2024-03-24T01:25:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3028.149624 secs 2024-03-24T01:25:54.986 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3683 sec 2024-03-24T01:25:55.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:54 smithi106 ceph-mon[36505]: pgmap v2090: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:55.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:54 smithi078 ceph-mon[32429]: pgmap v2090: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:55.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:54 smithi107 ceph-mon[36485]: pgmap v2090: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:55.986 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3684 sec 2024-03-24T01:25:56.986 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3685 sec 2024-03-24T01:25:57.143 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:56 smithi078 ceph-mon[32429]: pgmap v2091: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:25:57.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:56 smithi106 ceph-mon[36505]: pgmap v2091: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:25:57.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:56 smithi107 ceph-mon[36485]: pgmap v2091: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:25:57.987 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3686 sec 2024-03-24T01:25:58.987 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3687 sec 2024-03-24T01:25:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:58 smithi106 ceph-mon[36505]: pgmap v2092: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:59.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:25:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3033.149715 secs 2024-03-24T01:25:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:58 smithi078 ceph-mon[32429]: pgmap v2092: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:25:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3033.149715 secs 2024-03-24T01:25:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:58 smithi107 ceph-mon[36485]: pgmap v2092: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:25:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:25:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3033.149715 secs 2024-03-24T01:25:59.987 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3688 sec 2024-03-24T01:26:00.987 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3689 sec 2024-03-24T01:26:01.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:00 smithi106 ceph-mon[36505]: pgmap v2093: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:01.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:00 smithi078 ceph-mon[32429]: pgmap v2093: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:00 smithi107 ceph-mon[36485]: pgmap v2093: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:01.987 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3690 sec 2024-03-24T01:26:02.987 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3691 sec 2024-03-24T01:26:03.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:02 smithi106 ceph-mon[36505]: pgmap v2094: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:03.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:02 smithi078 ceph-mon[32429]: pgmap v2094: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:03.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:02 smithi107 ceph-mon[36485]: pgmap v2094: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:03.987 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3692 sec 2024-03-24T01:26:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3038.149776 secs 2024-03-24T01:26:04.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3038.149776 secs 2024-03-24T01:26:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3038.149776 secs 2024-03-24T01:26:04.987 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3693 sec 2024-03-24T01:26:05.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:04 smithi078 ceph-mon[32429]: pgmap v2095: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:05.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:04 smithi106 ceph-mon[36505]: pgmap v2095: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:05.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:04 smithi107 ceph-mon[36485]: pgmap v2095: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:05.987 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3694 sec 2024-03-24T01:26:06.987 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3695 sec 2024-03-24T01:26:07.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:06 smithi106 ceph-mon[36505]: pgmap v2096: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:07.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:06 smithi078 ceph-mon[32429]: pgmap v2096: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:07.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:06 smithi107 ceph-mon[36485]: pgmap v2096: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:07.987 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3696 sec 2024-03-24T01:26:08.987 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3697 sec 2024-03-24T01:26:09.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:08 smithi107 ceph-mon[36485]: pgmap v2097: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:09.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3043.149835 secs 2024-03-24T01:26:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:08 smithi106 ceph-mon[36505]: pgmap v2097: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3043.149835 secs 2024-03-24T01:26:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:08 smithi078 ceph-mon[32429]: pgmap v2097: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:09.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3043.149835 secs 2024-03-24T01:26:09.988 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3698 sec 2024-03-24T01:26:10.988 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3699 sec 2024-03-24T01:26:11.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:10 smithi106 ceph-mon[36505]: pgmap v2098: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:11.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:10 smithi078 ceph-mon[32429]: pgmap v2098: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:11.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:10 smithi107 ceph-mon[36485]: pgmap v2098: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:11.988 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3700 sec 2024-03-24T01:26:12.988 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3701 sec 2024-03-24T01:26:13.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:12 smithi106 ceph-mon[36505]: pgmap v2099: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:13.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:12 smithi078 ceph-mon[32429]: pgmap v2099: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:13.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:12 smithi107 ceph-mon[36485]: pgmap v2099: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:13.988 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3702 sec 2024-03-24T01:26:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3048.149923 secs 2024-03-24T01:26:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3048.149923 secs 2024-03-24T01:26:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3048.149923 secs 2024-03-24T01:26:14.988 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3703 sec 2024-03-24T01:26:15.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:14 smithi106 ceph-mon[36505]: pgmap v2100: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:15.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:14 smithi078 ceph-mon[32429]: pgmap v2100: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:15.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:14 smithi107 ceph-mon[36485]: pgmap v2100: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:15.988 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3704 sec 2024-03-24T01:26:16.988 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3705 sec 2024-03-24T01:26:17.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:16 smithi106 ceph-mon[36505]: pgmap v2101: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:17.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:16 smithi078 ceph-mon[32429]: pgmap v2101: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:17.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:16 smithi107 ceph-mon[36485]: pgmap v2101: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:17.988 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3706 sec 2024-03-24T01:26:18.988 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3707 sec 2024-03-24T01:26:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:18 smithi078 ceph-mon[32429]: pgmap v2102: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3053.150012 secs 2024-03-24T01:26:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:18 smithi107 ceph-mon[36485]: pgmap v2102: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3053.150012 secs 2024-03-24T01:26:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:18 smithi106 ceph-mon[36505]: pgmap v2102: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:19.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3053.150012 secs 2024-03-24T01:26:19.988 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3708 sec 2024-03-24T01:26:20.988 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3709 sec 2024-03-24T01:26:21.203 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:20 smithi106 ceph-mon[36505]: pgmap v2103: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:21.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:20 smithi078 ceph-mon[32429]: pgmap v2103: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:21.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:20 smithi107 ceph-mon[36485]: pgmap v2103: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:21.988 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3710 sec 2024-03-24T01:26:22.988 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3711 sec 2024-03-24T01:26:23.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:22 smithi078 ceph-mon[32429]: pgmap v2104: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:23.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:22 smithi107 ceph-mon[36485]: pgmap v2104: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:23.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:22 smithi106 ceph-mon[36505]: pgmap v2104: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:23.988 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3712 sec 2024-03-24T01:26:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3058.150103 secs 2024-03-24T01:26:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3058.150103 secs 2024-03-24T01:26:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3058.150103 secs 2024-03-24T01:26:24.989 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3713 sec 2024-03-24T01:26:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:24 smithi078 ceph-mon[32429]: pgmap v2105: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:25.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:24 smithi107 ceph-mon[36485]: pgmap v2105: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:25.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:24 smithi106 ceph-mon[36505]: pgmap v2105: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:25.989 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3714 sec 2024-03-24T01:26:26.989 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3715 sec 2024-03-24T01:26:27.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:26 smithi078 ceph-mon[32429]: pgmap v2106: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:27.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:26 smithi107 ceph-mon[36485]: pgmap v2106: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:27.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:26 smithi106 ceph-mon[36505]: pgmap v2106: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:27.989 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3716 sec 2024-03-24T01:26:28.989 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3717 sec 2024-03-24T01:26:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:28 smithi078 ceph-mon[32429]: pgmap v2107: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3063.150192 secs 2024-03-24T01:26:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:28 smithi107 ceph-mon[36485]: pgmap v2107: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:29.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3063.150192 secs 2024-03-24T01:26:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:28 smithi106 ceph-mon[36505]: pgmap v2107: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3063.150192 secs 2024-03-24T01:26:29.989 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3718 sec 2024-03-24T01:26:30.989 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3719 sec 2024-03-24T01:26:31.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:30 smithi078 ceph-mon[32429]: pgmap v2108: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:31.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:30 smithi107 ceph-mon[36485]: pgmap v2108: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:31.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:30 smithi106 ceph-mon[36505]: pgmap v2108: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:31.989 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3720 sec 2024-03-24T01:26:32.989 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3721 sec 2024-03-24T01:26:33.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:32 smithi078 ceph-mon[32429]: pgmap v2109: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:33.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:32 smithi107 ceph-mon[36485]: pgmap v2109: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:33.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:32 smithi106 ceph-mon[36505]: pgmap v2109: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:33.989 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3722 sec 2024-03-24T01:26:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3068.150250 secs 2024-03-24T01:26:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3068.150250 secs 2024-03-24T01:26:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3068.150250 secs 2024-03-24T01:26:34.990 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3723 sec 2024-03-24T01:26:35.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:34 smithi078 ceph-mon[32429]: pgmap v2110: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:35.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:34 smithi107 ceph-mon[36485]: pgmap v2110: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:35.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:34 smithi106 ceph-mon[36505]: pgmap v2110: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:35.990 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3724 sec 2024-03-24T01:26:36.990 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3725 sec 2024-03-24T01:26:37.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:36 smithi078 ceph-mon[32429]: pgmap v2111: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:36 smithi107 ceph-mon[36485]: pgmap v2111: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:36 smithi106 ceph-mon[36505]: pgmap v2111: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:37.990 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3726 sec 2024-03-24T01:26:38.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:26:38.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:26:38.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:26:38.990 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3727 sec 2024-03-24T01:26:39.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:38 smithi107 ceph-mon[36485]: pgmap v2112: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:39.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3073.150268 secs 2024-03-24T01:26:39.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:38 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:26:39.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:38 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:26:39.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:38 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:26:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:38 smithi078 ceph-mon[32429]: pgmap v2112: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:39.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3073.150268 secs 2024-03-24T01:26:39.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:38 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:26:39.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:38 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:26:39.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:38 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:26:39.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:38 smithi106 ceph-mon[36505]: pgmap v2112: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:39.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3073.150268 secs 2024-03-24T01:26:39.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:38 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:26:39.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:38 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:26:39.405 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:38 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:26:39.990 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3728 sec 2024-03-24T01:26:40.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:39 smithi078 ceph-mon[32429]: pgmap v2113: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:26:40.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:39 smithi078 ceph-mon[32429]: pgmap v2114: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:26:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:39 smithi107 ceph-mon[36485]: pgmap v2113: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:26:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:39 smithi107 ceph-mon[36485]: pgmap v2114: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:26:40.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:39 smithi106 ceph-mon[36505]: pgmap v2113: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:26:40.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:39 smithi106 ceph-mon[36505]: pgmap v2114: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:26:40.990 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3729 sec 2024-03-24T01:26:41.990 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3730 sec 2024-03-24T01:26:42.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:41 smithi078 ceph-mon[32429]: pgmap v2115: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:26:42.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:41 smithi107 ceph-mon[36485]: pgmap v2115: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:26:42.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:41 smithi106 ceph-mon[36505]: pgmap v2115: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:26:42.990 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3731 sec 2024-03-24T01:26:43.990 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3732 sec 2024-03-24T01:26:44.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:43 smithi078 ceph-mon[32429]: pgmap v2116: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:26:44.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3078.150351 secs 2024-03-24T01:26:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:43 smithi107 ceph-mon[36485]: pgmap v2116: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:26:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3078.150351 secs 2024-03-24T01:26:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:43 smithi106 ceph-mon[36505]: pgmap v2116: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:26:44.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3078.150351 secs 2024-03-24T01:26:44.990 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3733 sec 2024-03-24T01:26:45.990 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3734 sec 2024-03-24T01:26:46.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:45 smithi078 ceph-mon[32429]: pgmap v2117: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:26:46.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:45 smithi107 ceph-mon[36485]: pgmap v2117: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:26:46.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:45 smithi106 ceph-mon[36505]: pgmap v2117: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:26:46.995 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3735 sec 2024-03-24T01:26:47.990 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3736 sec 2024-03-24T01:26:48.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:47 smithi078 ceph-mon[32429]: pgmap v2118: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:26:48.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:47 smithi107 ceph-mon[36485]: pgmap v2118: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:26:48.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:47 smithi106 ceph-mon[36505]: pgmap v2118: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:26:48.991 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3737 sec 2024-03-24T01:26:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3083.150439 secs 2024-03-24T01:26:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3083.150439 secs 2024-03-24T01:26:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3083.150439 secs 2024-03-24T01:26:49.991 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3738 sec 2024-03-24T01:26:50.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:49 smithi078 ceph-mon[32429]: pgmap v2119: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:26:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:49 smithi107 ceph-mon[36485]: pgmap v2119: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:26:50.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:49 smithi106 ceph-mon[36505]: pgmap v2119: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:26:50.991 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3739 sec 2024-03-24T01:26:51.991 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3740 sec 2024-03-24T01:26:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:51 smithi078 ceph-mon[32429]: pgmap v2120: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:52.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:51 smithi107 ceph-mon[36485]: pgmap v2120: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:52.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:51 smithi106 ceph-mon[36505]: pgmap v2120: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:52.991 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3741 sec 2024-03-24T01:26:53.991 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3742 sec 2024-03-24T01:26:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:53 smithi078 ceph-mon[32429]: pgmap v2121: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3088.150503 secs 2024-03-24T01:26:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:53 smithi107 ceph-mon[36485]: pgmap v2121: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3088.150503 secs 2024-03-24T01:26:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:53 smithi106 ceph-mon[36505]: pgmap v2121: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3088.150503 secs 2024-03-24T01:26:54.991 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3743 sec 2024-03-24T01:26:55.991 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3744 sec 2024-03-24T01:26:56.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:55 smithi078 ceph-mon[32429]: pgmap v2122: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:56.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:55 smithi107 ceph-mon[36485]: pgmap v2122: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:56.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:55 smithi106 ceph-mon[36505]: pgmap v2122: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:26:56.991 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3745 sec 2024-03-24T01:26:57.991 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3746 sec 2024-03-24T01:26:58.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:57 smithi078 ceph-mon[32429]: pgmap v2123: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:57 smithi107 ceph-mon[36485]: pgmap v2123: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:58.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:57 smithi106 ceph-mon[36505]: pgmap v2123: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:58.991 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3747 sec 2024-03-24T01:26:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3093.150567 secs 2024-03-24T01:26:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3093.150567 secs 2024-03-24T01:26:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3093.150567 secs 2024-03-24T01:26:59.991 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3748 sec 2024-03-24T01:27:00.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:26:59 smithi078 ceph-mon[32429]: pgmap v2124: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:00.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:26:59 smithi107 ceph-mon[36485]: pgmap v2124: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:00.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:26:59 smithi106 ceph-mon[36505]: pgmap v2124: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:00.991 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3749 sec 2024-03-24T01:27:01.991 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3750 sec 2024-03-24T01:27:02.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:01 smithi078 ceph-mon[32429]: pgmap v2125: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:02.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:01 smithi107 ceph-mon[36485]: pgmap v2125: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:02.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:01 smithi106 ceph-mon[36505]: pgmap v2125: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:02.991 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3751 sec 2024-03-24T01:27:03.992 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3752 sec 2024-03-24T01:27:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:03 smithi078 ceph-mon[32429]: pgmap v2126: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:04.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3098.150624 secs 2024-03-24T01:27:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:03 smithi107 ceph-mon[36485]: pgmap v2126: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3098.150624 secs 2024-03-24T01:27:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:03 smithi106 ceph-mon[36505]: pgmap v2126: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3098.150624 secs 2024-03-24T01:27:04.992 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3753 sec 2024-03-24T01:27:05.992 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3754 sec 2024-03-24T01:27:06.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:06 smithi078 ceph-mon[32429]: pgmap v2127: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:06.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:06 smithi107 ceph-mon[36485]: pgmap v2127: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:06.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:06 smithi106 ceph-mon[36505]: pgmap v2127: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:06.992 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3755 sec 2024-03-24T01:27:07.997 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3756 sec 2024-03-24T01:27:08.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:08 smithi078 ceph-mon[32429]: pgmap v2128: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:08.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:08 smithi107 ceph-mon[36485]: pgmap v2128: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:08.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:08 smithi106 ceph-mon[36505]: pgmap v2128: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:08.992 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3757 sec 2024-03-24T01:27:09.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3103.150686 secs 2024-03-24T01:27:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3103.150686 secs 2024-03-24T01:27:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3103.150686 secs 2024-03-24T01:27:09.992 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3758 sec 2024-03-24T01:27:10.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:10 smithi078 ceph-mon[32429]: pgmap v2129: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:10.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:10 smithi107 ceph-mon[36485]: pgmap v2129: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:10.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:10 smithi106 ceph-mon[36505]: pgmap v2129: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:10.992 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3759 sec 2024-03-24T01:27:11.993 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3760 sec 2024-03-24T01:27:12.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:12 smithi078 ceph-mon[32429]: pgmap v2130: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:12.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:12 smithi107 ceph-mon[36485]: pgmap v2130: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:12.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:12 smithi106 ceph-mon[36505]: pgmap v2130: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:12.992 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3761 sec 2024-03-24T01:27:13.992 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3762 sec 2024-03-24T01:27:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:14 smithi078 ceph-mon[32429]: pgmap v2131: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3108.150745 secs 2024-03-24T01:27:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:14 smithi107 ceph-mon[36485]: pgmap v2131: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3108.150745 secs 2024-03-24T01:27:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:14 smithi106 ceph-mon[36505]: pgmap v2131: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:14.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3108.150745 secs 2024-03-24T01:27:14.992 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3763 sec 2024-03-24T01:27:15.994 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3764 sec 2024-03-24T01:27:16.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:16 smithi078 ceph-mon[32429]: pgmap v2132: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:16.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:16 smithi107 ceph-mon[36485]: pgmap v2132: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:16.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:16 smithi106 ceph-mon[36505]: pgmap v2132: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:16.993 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3765 sec 2024-03-24T01:27:17.993 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3766 sec 2024-03-24T01:27:18.343 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:18 smithi078 ceph-mon[32429]: pgmap v2133: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:18.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:18 smithi107 ceph-mon[36485]: pgmap v2133: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:18.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:18 smithi106 ceph-mon[36505]: pgmap v2133: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:18.993 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3767 sec 2024-03-24T01:27:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3113.150806 secs 2024-03-24T01:27:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3113.150806 secs 2024-03-24T01:27:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3113.150806 secs 2024-03-24T01:27:19.993 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3768 sec 2024-03-24T01:27:20.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:20 smithi078 ceph-mon[32429]: pgmap v2134: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:20 smithi107 ceph-mon[36485]: pgmap v2134: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:20.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:20 smithi106 ceph-mon[36505]: pgmap v2134: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:20.993 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3769 sec 2024-03-24T01:27:21.993 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3770 sec 2024-03-24T01:27:22.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:22 smithi078 ceph-mon[32429]: pgmap v2135: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:22.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:22 smithi107 ceph-mon[36485]: pgmap v2135: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:22.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:22 smithi106 ceph-mon[36505]: pgmap v2135: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:22.993 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3771 sec 2024-03-24T01:27:23.993 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3772 sec 2024-03-24T01:27:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:24 smithi078 ceph-mon[32429]: pgmap v2136: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3118.150818 secs 2024-03-24T01:27:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:24 smithi107 ceph-mon[36485]: pgmap v2136: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3118.150818 secs 2024-03-24T01:27:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:24 smithi106 ceph-mon[36505]: pgmap v2136: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3118.150818 secs 2024-03-24T01:27:24.993 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3773 sec 2024-03-24T01:27:25.993 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3774 sec 2024-03-24T01:27:26.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:26 smithi078 ceph-mon[32429]: pgmap v2137: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:26.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:26 smithi107 ceph-mon[36485]: pgmap v2137: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:26.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:26 smithi106 ceph-mon[36505]: pgmap v2137: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:26.993 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3775 sec 2024-03-24T01:27:27.994 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3776 sec 2024-03-24T01:27:28.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:28 smithi078 ceph-mon[32429]: pgmap v2138: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:28.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:28 smithi107 ceph-mon[36485]: pgmap v2138: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:28.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:28 smithi106 ceph-mon[36505]: pgmap v2138: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:28.994 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3777 sec 2024-03-24T01:27:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3123.150908 secs 2024-03-24T01:27:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3123.150908 secs 2024-03-24T01:27:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3123.150908 secs 2024-03-24T01:27:29.994 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3778 sec 2024-03-24T01:27:30.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:30 smithi078 ceph-mon[32429]: pgmap v2139: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:30.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:30 smithi107 ceph-mon[36485]: pgmap v2139: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:30.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:30 smithi106 ceph-mon[36505]: pgmap v2139: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:30.994 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3779 sec 2024-03-24T01:27:31.994 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3780 sec 2024-03-24T01:27:32.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:32 smithi078 ceph-mon[32429]: pgmap v2140: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:32.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:32 smithi107 ceph-mon[36485]: pgmap v2140: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:32.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:32 smithi106 ceph-mon[36505]: pgmap v2140: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:32.994 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3781 sec 2024-03-24T01:27:33.994 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3782 sec 2024-03-24T01:27:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:34 smithi078 ceph-mon[32429]: pgmap v2141: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:34.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3128.150963 secs 2024-03-24T01:27:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:34 smithi107 ceph-mon[36485]: pgmap v2141: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3128.150963 secs 2024-03-24T01:27:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:34 smithi106 ceph-mon[36505]: pgmap v2141: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3128.150963 secs 2024-03-24T01:27:34.994 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3783 sec 2024-03-24T01:27:35.994 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3784 sec 2024-03-24T01:27:36.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:36 smithi078 ceph-mon[32429]: pgmap v2142: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:36.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:36 smithi107 ceph-mon[36485]: pgmap v2142: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:36.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:36 smithi106 ceph-mon[36505]: pgmap v2142: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:36.994 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3785 sec 2024-03-24T01:27:37.994 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3786 sec 2024-03-24T01:27:38.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:38 smithi078 ceph-mon[32429]: pgmap v2143: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:38.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:38 smithi107 ceph-mon[36485]: pgmap v2143: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:38.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:38 smithi106 ceph-mon[36505]: pgmap v2143: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:38.994 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3787 sec 2024-03-24T01:27:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3133.151021 secs 2024-03-24T01:27:39.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:39 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:27:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3133.151021 secs 2024-03-24T01:27:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:39 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:27:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3133.151021 secs 2024-03-24T01:27:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:39 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:27:39.994 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3788 sec 2024-03-24T01:27:40.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:40 smithi078 ceph-mon[32429]: pgmap v2144: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:40.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:40 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:27:40.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:40 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:27:40.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:40 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:27:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:40 smithi107 ceph-mon[36485]: pgmap v2144: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:40 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:27:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:40 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:27:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:40 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:27:40.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:40 smithi106 ceph-mon[36505]: pgmap v2144: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:40.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:40 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:27:40.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:40 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:27:40.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:40 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:27:40.995 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3789 sec 2024-03-24T01:27:41.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:41 smithi078 ceph-mon[32429]: pgmap v2145: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:27:41.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:41 smithi078 ceph-mon[32429]: pgmap v2146: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:27:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:41 smithi107 ceph-mon[36485]: pgmap v2145: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:27:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:41 smithi107 ceph-mon[36485]: pgmap v2146: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:27:41.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:41 smithi106 ceph-mon[36505]: pgmap v2145: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:27:41.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:41 smithi106 ceph-mon[36505]: pgmap v2146: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:27:41.995 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3790 sec 2024-03-24T01:27:42.995 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3791 sec 2024-03-24T01:27:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:43 smithi078 ceph-mon[32429]: pgmap v2147: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:27:43.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:43 smithi107 ceph-mon[36485]: pgmap v2147: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:27:43.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:43 smithi106 ceph-mon[36505]: pgmap v2147: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:27:43.995 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3792 sec 2024-03-24T01:27:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3138.151107 secs 2024-03-24T01:27:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3138.151107 secs 2024-03-24T01:27:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3138.151107 secs 2024-03-24T01:27:44.995 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3793 sec 2024-03-24T01:27:45.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:45 smithi078 ceph-mon[32429]: pgmap v2148: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:27:45.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:45 smithi107 ceph-mon[36485]: pgmap v2148: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:27:45.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:45 smithi106 ceph-mon[36505]: pgmap v2148: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:27:45.995 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3794 sec 2024-03-24T01:27:46.995 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3795 sec 2024-03-24T01:27:47.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:47 smithi078 ceph-mon[32429]: pgmap v2149: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:27:47.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:47 smithi107 ceph-mon[36485]: pgmap v2149: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:27:47.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:47 smithi106 ceph-mon[36505]: pgmap v2149: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:27:47.995 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3796 sec 2024-03-24T01:27:48.995 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3797 sec 2024-03-24T01:27:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:49 smithi078 ceph-mon[32429]: pgmap v2150: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:27:49.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3143.151195 secs 2024-03-24T01:27:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:49 smithi107 ceph-mon[36485]: pgmap v2150: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:27:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3143.151195 secs 2024-03-24T01:27:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:49 smithi106 ceph-mon[36505]: pgmap v2150: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:27:49.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3143.151195 secs 2024-03-24T01:27:49.995 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3798 sec 2024-03-24T01:27:50.996 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3799 sec 2024-03-24T01:27:51.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:51 smithi107 ceph-mon[36485]: pgmap v2151: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:27:51.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:51 smithi106 ceph-mon[36505]: pgmap v2151: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:27:51.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:51 smithi078 ceph-mon[32429]: pgmap v2151: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:27:51.996 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3800 sec 2024-03-24T01:27:52.996 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3801 sec 2024-03-24T01:27:53.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:53 smithi106 ceph-mon[36505]: pgmap v2152: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:53.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:53 smithi078 ceph-mon[32429]: pgmap v2152: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:53.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:53 smithi107 ceph-mon[36485]: pgmap v2152: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:53.996 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3802 sec 2024-03-24T01:27:54.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3148.151231 secs 2024-03-24T01:27:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3148.151231 secs 2024-03-24T01:27:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3148.151231 secs 2024-03-24T01:27:54.996 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3803 sec 2024-03-24T01:27:55.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:55 smithi106 ceph-mon[36505]: pgmap v2153: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:55.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:55 smithi078 ceph-mon[32429]: pgmap v2153: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:55.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:55 smithi107 ceph-mon[36485]: pgmap v2153: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:55.996 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3804 sec 2024-03-24T01:27:56.996 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3805 sec 2024-03-24T01:27:57.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:57 smithi106 ceph-mon[36505]: pgmap v2154: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:57.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:57 smithi078 ceph-mon[32429]: pgmap v2154: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:57.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:57 smithi107 ceph-mon[36485]: pgmap v2154: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:57.996 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3806 sec 2024-03-24T01:27:58.492 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:58 smithi078 ceph-mon[32429]: pgmap v2155: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:58.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:58 smithi107 ceph-mon[36485]: pgmap v2155: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:58.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:58 smithi106 ceph-mon[36505]: pgmap v2155: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:27:58.996 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3807 sec 2024-03-24T01:27:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:27:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3153.151309 secs 2024-03-24T01:27:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:27:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3153.151309 secs 2024-03-24T01:27:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:27:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3153.151309 secs 2024-03-24T01:27:59.996 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3808 sec 2024-03-24T01:28:00.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:00 smithi078 ceph-mon[32429]: pgmap v2156: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:00.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:00 smithi107 ceph-mon[36485]: pgmap v2156: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:00.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:00 smithi106 ceph-mon[36505]: pgmap v2156: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:00.996 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3809 sec 2024-03-24T01:28:01.997 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3810 sec 2024-03-24T01:28:02.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:02 smithi106 ceph-mon[36505]: pgmap v2157: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:02.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:02 smithi078 ceph-mon[32429]: pgmap v2157: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:02 smithi107 ceph-mon[36485]: pgmap v2157: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:02.997 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3811 sec 2024-03-24T01:28:03.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3158.151370 secs 2024-03-24T01:28:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3158.151370 secs 2024-03-24T01:28:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3158.151370 secs 2024-03-24T01:28:03.996 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3812 sec 2024-03-24T01:28:04.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:04 smithi078 ceph-mon[32429]: pgmap v2158: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:04 smithi107 ceph-mon[36485]: pgmap v2158: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:04 smithi106 ceph-mon[36505]: pgmap v2158: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:04.997 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3813 sec 2024-03-24T01:28:05.997 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3814 sec 2024-03-24T01:28:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:06 smithi078 ceph-mon[32429]: pgmap v2159: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:06 smithi107 ceph-mon[36485]: pgmap v2159: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:06 smithi106 ceph-mon[36505]: pgmap v2159: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:06.997 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3815 sec 2024-03-24T01:28:07.997 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3816 sec 2024-03-24T01:28:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:08 smithi078 ceph-mon[32429]: pgmap v2160: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3163.151432 secs 2024-03-24T01:28:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:08 smithi107 ceph-mon[36485]: pgmap v2160: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3163.151432 secs 2024-03-24T01:28:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:08 smithi106 ceph-mon[36505]: pgmap v2160: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3163.151432 secs 2024-03-24T01:28:08.997 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3817 sec 2024-03-24T01:28:09.997 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3818 sec 2024-03-24T01:28:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:10 smithi078 ceph-mon[32429]: pgmap v2161: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:10 smithi107 ceph-mon[36485]: pgmap v2161: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:10 smithi106 ceph-mon[36505]: pgmap v2161: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:10.997 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3819 sec 2024-03-24T01:28:11.997 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3820 sec 2024-03-24T01:28:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:12 smithi078 ceph-mon[32429]: pgmap v2162: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:12 smithi107 ceph-mon[36485]: pgmap v2162: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:12.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:12 smithi106 ceph-mon[36505]: pgmap v2162: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:12.997 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3821 sec 2024-03-24T01:28:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3168.151492 secs 2024-03-24T01:28:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3168.151492 secs 2024-03-24T01:28:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3168.151492 secs 2024-03-24T01:28:13.997 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3822 sec 2024-03-24T01:28:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:14 smithi078 ceph-mon[32429]: pgmap v2163: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:14 smithi107 ceph-mon[36485]: pgmap v2163: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:14 smithi106 ceph-mon[36505]: pgmap v2163: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:14.997 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3823 sec 2024-03-24T01:28:15.998 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3824 sec 2024-03-24T01:28:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:16 smithi078 ceph-mon[32429]: pgmap v2164: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:16 smithi107 ceph-mon[36485]: pgmap v2164: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:16.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:16 smithi106 ceph-mon[36505]: pgmap v2164: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:16.998 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3825 sec 2024-03-24T01:28:17.998 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3826 sec 2024-03-24T01:28:18.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:18 smithi078 ceph-mon[32429]: pgmap v2165: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:18.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3173.151553 secs 2024-03-24T01:28:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:18 smithi107 ceph-mon[36485]: pgmap v2165: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3173.151553 secs 2024-03-24T01:28:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:18 smithi106 ceph-mon[36505]: pgmap v2165: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:18.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3173.151553 secs 2024-03-24T01:28:18.998 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3827 sec 2024-03-24T01:28:19.998 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3828 sec 2024-03-24T01:28:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:20 smithi078 ceph-mon[32429]: pgmap v2166: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:20 smithi107 ceph-mon[36485]: pgmap v2166: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:20.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:20 smithi106 ceph-mon[36505]: pgmap v2166: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:20.998 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3829 sec 2024-03-24T01:28:21.998 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3830 sec 2024-03-24T01:28:22.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:22 smithi078 ceph-mon[32429]: pgmap v2167: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:22.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:22 smithi107 ceph-mon[36485]: pgmap v2167: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:22.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:22 smithi106 ceph-mon[36505]: pgmap v2167: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:22.998 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3831 sec 2024-03-24T01:28:23.803 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3178.151615 secs 2024-03-24T01:28:23.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3178.151615 secs 2024-03-24T01:28:23.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3178.151615 secs 2024-03-24T01:28:23.998 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3832 sec 2024-03-24T01:28:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:24 smithi078 ceph-mon[32429]: pgmap v2168: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:24 smithi107 ceph-mon[36485]: pgmap v2168: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:24 smithi106 ceph-mon[36505]: pgmap v2168: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:24.998 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3833 sec 2024-03-24T01:28:25.998 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3834 sec 2024-03-24T01:28:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:26 smithi078 ceph-mon[32429]: pgmap v2169: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:26 smithi107 ceph-mon[36485]: pgmap v2169: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:26.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:26 smithi106 ceph-mon[36505]: pgmap v2169: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:26.999 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3835 sec 2024-03-24T01:28:27.999 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3836 sec 2024-03-24T01:28:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:28 smithi078 ceph-mon[32429]: pgmap v2170: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:28.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3183.151674 secs 2024-03-24T01:28:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:28 smithi107 ceph-mon[36485]: pgmap v2170: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3183.151674 secs 2024-03-24T01:28:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:28 smithi106 ceph-mon[36505]: pgmap v2170: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:28.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3183.151674 secs 2024-03-24T01:28:28.999 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3837 sec 2024-03-24T01:28:29.999 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3838 sec 2024-03-24T01:28:30.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:30 smithi078 ceph-mon[32429]: pgmap v2171: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:30.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:30 smithi107 ceph-mon[36485]: pgmap v2171: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:30.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:30 smithi106 ceph-mon[36505]: pgmap v2171: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:30.999 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3839 sec 2024-03-24T01:28:31.999 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3840 sec 2024-03-24T01:28:32.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:32 smithi078 ceph-mon[32429]: pgmap v2172: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:32 smithi107 ceph-mon[36485]: pgmap v2172: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:32 smithi106 ceph-mon[36505]: pgmap v2172: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:32.999 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3841 sec 2024-03-24T01:28:33.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3188.151715 secs 2024-03-24T01:28:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3188.151715 secs 2024-03-24T01:28:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3188.151715 secs 2024-03-24T01:28:33.999 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3842 sec 2024-03-24T01:28:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:34 smithi078 ceph-mon[32429]: pgmap v2173: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:34 smithi107 ceph-mon[36485]: pgmap v2173: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:34 smithi106 ceph-mon[36505]: pgmap v2173: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:34.999 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3843 sec 2024-03-24T01:28:35.999 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3844 sec 2024-03-24T01:28:36.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:36 smithi078 ceph-mon[32429]: pgmap v2174: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:36 smithi107 ceph-mon[36485]: pgmap v2174: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:36.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:36 smithi106 ceph-mon[36505]: pgmap v2174: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:36.999 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3845 sec 2024-03-24T01:28:38.000 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3846 sec 2024-03-24T01:28:38.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:38 smithi078 ceph-mon[32429]: pgmap v2175: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:38.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3193.151800 secs 2024-03-24T01:28:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:38 smithi107 ceph-mon[36485]: pgmap v2175: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3193.151800 secs 2024-03-24T01:28:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:38 smithi106 ceph-mon[36505]: pgmap v2175: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3193.151800 secs 2024-03-24T01:28:38.999 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3847 sec 2024-03-24T01:28:39.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:39 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:28:39.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:39 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:28:39.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:39 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:28:40.000 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3848 sec 2024-03-24T01:28:40.745 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:40 smithi107 ceph-mon[36485]: pgmap v2176: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:40.745 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:40 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:28:40.745 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:40 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:28:40.745 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:40 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:28:40.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:40 smithi078 ceph-mon[32429]: pgmap v2176: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:40.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:40 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:28:40.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:40 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:28:40.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:40 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:28:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:40 smithi106 ceph-mon[36505]: pgmap v2176: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:40 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:28:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:40 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:28:40.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:40 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:28:41.000 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3849 sec 2024-03-24T01:28:41.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:41 smithi078 ceph-mon[32429]: pgmap v2177: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:28:41.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:41 smithi078 ceph-mon[32429]: pgmap v2178: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:28:41.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:41 smithi107 ceph-mon[36485]: pgmap v2177: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:28:41.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:41 smithi107 ceph-mon[36485]: pgmap v2178: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:28:41.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:41 smithi106 ceph-mon[36505]: pgmap v2177: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:28:41.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:41 smithi106 ceph-mon[36505]: pgmap v2178: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:28:42.000 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3850 sec 2024-03-24T01:28:43.000 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3851 sec 2024-03-24T01:28:43.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:43 smithi078 ceph-mon[32429]: pgmap v2179: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:28:43.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3198.151889 secs 2024-03-24T01:28:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:43 smithi107 ceph-mon[36485]: pgmap v2179: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:28:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3198.151889 secs 2024-03-24T01:28:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:43 smithi106 ceph-mon[36505]: pgmap v2179: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:28:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3198.151889 secs 2024-03-24T01:28:44.000 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3852 sec 2024-03-24T01:28:45.000 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3853 sec 2024-03-24T01:28:45.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:45 smithi078 ceph-mon[32429]: pgmap v2180: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:28:45.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:45 smithi107 ceph-mon[36485]: pgmap v2180: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:28:45.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:45 smithi106 ceph-mon[36505]: pgmap v2180: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:28:46.000 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3854 sec 2024-03-24T01:28:47.000 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3855 sec 2024-03-24T01:28:47.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:47 smithi078 ceph-mon[32429]: pgmap v2181: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:28:47.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:47 smithi107 ceph-mon[36485]: pgmap v2181: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:28:47.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:47 smithi106 ceph-mon[36505]: pgmap v2181: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:28:48.000 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3856 sec 2024-03-24T01:28:48.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3203.151929 secs 2024-03-24T01:28:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3203.151929 secs 2024-03-24T01:28:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3203.151929 secs 2024-03-24T01:28:49.000 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3857 sec 2024-03-24T01:28:49.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:49 smithi078 ceph-mon[32429]: pgmap v2182: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:28:49.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:49 smithi107 ceph-mon[36485]: pgmap v2182: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:28:49.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:49 smithi106 ceph-mon[36505]: pgmap v2182: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:28:50.001 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3858 sec 2024-03-24T01:28:51.002 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3859 sec 2024-03-24T01:28:51.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:51 smithi078 ceph-mon[32429]: pgmap v2183: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:28:51.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:51 smithi107 ceph-mon[36485]: pgmap v2183: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:28:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:51 smithi106 ceph-mon[36505]: pgmap v2183: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:28:52.001 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3860 sec 2024-03-24T01:28:53.001 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3861 sec 2024-03-24T01:28:53.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:53 smithi078 ceph-mon[32429]: pgmap v2184: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:28:53.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3208.152018 secs 2024-03-24T01:28:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:53 smithi107 ceph-mon[36485]: pgmap v2184: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:28:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3208.152018 secs 2024-03-24T01:28:53.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:53 smithi106 ceph-mon[36505]: pgmap v2184: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:28:53.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3208.152018 secs 2024-03-24T01:28:54.001 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3862 sec 2024-03-24T01:28:55.001 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3863 sec 2024-03-24T01:28:55.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:55 smithi078 ceph-mon[32429]: pgmap v2185: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:28:55.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:55 smithi107 ceph-mon[36485]: pgmap v2185: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:28:55.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:55 smithi106 ceph-mon[36505]: pgmap v2185: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:28:56.001 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3864 sec 2024-03-24T01:28:57.001 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3865 sec 2024-03-24T01:28:57.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:57 smithi078 ceph-mon[32429]: pgmap v2186: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:28:57.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:57 smithi107 ceph-mon[36485]: pgmap v2186: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:28:57.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:57 smithi106 ceph-mon[36505]: pgmap v2186: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:28:58.001 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3866 sec 2024-03-24T01:28:58.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3213.152080 secs 2024-03-24T01:28:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3213.152080 secs 2024-03-24T01:28:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3213.152080 secs 2024-03-24T01:28:59.001 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3867 sec 2024-03-24T01:28:59.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:28:59 smithi078 ceph-mon[32429]: pgmap v2187: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:28:59 smithi107 ceph-mon[36485]: pgmap v2187: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:28:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:28:59 smithi106 ceph-mon[36505]: pgmap v2187: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:00.001 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3868 sec 2024-03-24T01:29:01.001 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3869 sec 2024-03-24T01:29:01.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:01 smithi078 ceph-mon[32429]: pgmap v2188: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:01.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:01 smithi107 ceph-mon[36485]: pgmap v2188: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:01.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:01 smithi106 ceph-mon[36505]: pgmap v2188: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:02.001 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3870 sec 2024-03-24T01:29:03.002 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3871 sec 2024-03-24T01:29:03.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:03 smithi078 ceph-mon[32429]: pgmap v2189: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:03.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3218.152168 secs 2024-03-24T01:29:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:03 smithi107 ceph-mon[36485]: pgmap v2189: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3218.152168 secs 2024-03-24T01:29:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:03 smithi106 ceph-mon[36505]: pgmap v2189: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3218.152168 secs 2024-03-24T01:29:04.001 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3872 sec 2024-03-24T01:29:05.002 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3873 sec 2024-03-24T01:29:05.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:05 smithi078 ceph-mon[32429]: pgmap v2190: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:05.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:05 smithi107 ceph-mon[36485]: pgmap v2190: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:05.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:05 smithi106 ceph-mon[36505]: pgmap v2190: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:06.002 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3874 sec 2024-03-24T01:29:07.002 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3875 sec 2024-03-24T01:29:07.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:07 smithi078 ceph-mon[32429]: pgmap v2191: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:07.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:07 smithi107 ceph-mon[36485]: pgmap v2191: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:07.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:07 smithi106 ceph-mon[36505]: pgmap v2191: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:08.002 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3876 sec 2024-03-24T01:29:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3223.152259 secs 2024-03-24T01:29:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3223.152259 secs 2024-03-24T01:29:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3223.152259 secs 2024-03-24T01:29:09.002 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3877 sec 2024-03-24T01:29:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:09 smithi078 ceph-mon[32429]: pgmap v2192: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:09.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:09 smithi107 ceph-mon[36485]: pgmap v2192: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:09.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:09 smithi106 ceph-mon[36505]: pgmap v2192: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:10.002 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3878 sec 2024-03-24T01:29:11.002 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3879 sec 2024-03-24T01:29:11.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:11 smithi078 ceph-mon[32429]: pgmap v2193: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:11 smithi107 ceph-mon[36485]: pgmap v2193: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:11.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:11 smithi106 ceph-mon[36505]: pgmap v2193: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:12.002 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3880 sec 2024-03-24T01:29:13.002 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3881 sec 2024-03-24T01:29:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:13 smithi078 ceph-mon[32429]: pgmap v2194: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:13.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3228.152322 secs 2024-03-24T01:29:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:13 smithi107 ceph-mon[36485]: pgmap v2194: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3228.152322 secs 2024-03-24T01:29:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:13 smithi106 ceph-mon[36505]: pgmap v2194: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3228.152322 secs 2024-03-24T01:29:14.002 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3882 sec 2024-03-24T01:29:15.003 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3883 sec 2024-03-24T01:29:15.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:15 smithi078 ceph-mon[32429]: pgmap v2195: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:15 smithi107 ceph-mon[36485]: pgmap v2195: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:15 smithi106 ceph-mon[36505]: pgmap v2195: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:16.002 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3884 sec 2024-03-24T01:29:17.003 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3885 sec 2024-03-24T01:29:17.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:17 smithi078 ceph-mon[32429]: pgmap v2196: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:17.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:17 smithi107 ceph-mon[36485]: pgmap v2196: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:17.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:17 smithi106 ceph-mon[36505]: pgmap v2196: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:18.003 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3886 sec 2024-03-24T01:29:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3233.152383 secs 2024-03-24T01:29:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3233.152383 secs 2024-03-24T01:29:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3233.152383 secs 2024-03-24T01:29:19.003 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3887 sec 2024-03-24T01:29:19.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:19 smithi078 ceph-mon[32429]: pgmap v2197: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:19 smithi107 ceph-mon[36485]: pgmap v2197: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:19.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:19 smithi106 ceph-mon[36505]: pgmap v2197: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:20.003 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3888 sec 2024-03-24T01:29:21.003 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3889 sec 2024-03-24T01:29:21.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:21 smithi078 ceph-mon[32429]: pgmap v2198: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:21 smithi107 ceph-mon[36485]: pgmap v2198: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:21.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:21 smithi106 ceph-mon[36505]: pgmap v2198: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:22.003 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3890 sec 2024-03-24T01:29:23.003 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3891 sec 2024-03-24T01:29:23.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:23 smithi078 ceph-mon[32429]: pgmap v2199: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:23.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3238.152444 secs 2024-03-24T01:29:23.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:23 smithi107 ceph-mon[36485]: pgmap v2199: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:23.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3238.152444 secs 2024-03-24T01:29:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:23 smithi106 ceph-mon[36505]: pgmap v2199: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3238.152444 secs 2024-03-24T01:29:24.003 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3892 sec 2024-03-24T01:29:25.003 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3893 sec 2024-03-24T01:29:25.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:25 smithi078 ceph-mon[32429]: pgmap v2200: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:25.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:25 smithi107 ceph-mon[36485]: pgmap v2200: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:25.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:25 smithi106 ceph-mon[36505]: pgmap v2200: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:26.003 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3894 sec 2024-03-24T01:29:27.003 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3895 sec 2024-03-24T01:29:27.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:27 smithi078 ceph-mon[32429]: pgmap v2201: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:27.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:27 smithi107 ceph-mon[36485]: pgmap v2201: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:27.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:27 smithi106 ceph-mon[36505]: pgmap v2201: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:28.004 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3896 sec 2024-03-24T01:29:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3243.152502 secs 2024-03-24T01:29:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3243.152502 secs 2024-03-24T01:29:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3243.152502 secs 2024-03-24T01:29:29.003 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3897 sec 2024-03-24T01:29:29.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:29 smithi078 ceph-mon[32429]: pgmap v2202: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:29 smithi107 ceph-mon[36485]: pgmap v2202: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:29 smithi106 ceph-mon[36505]: pgmap v2202: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:30.004 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3898 sec 2024-03-24T01:29:31.004 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3899 sec 2024-03-24T01:29:31.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:31 smithi078 ceph-mon[32429]: pgmap v2203: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:31.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:31 smithi107 ceph-mon[36485]: pgmap v2203: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:31.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:31 smithi106 ceph-mon[36505]: pgmap v2203: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:32.004 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3900 sec 2024-03-24T01:29:33.004 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3901 sec 2024-03-24T01:29:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:33 smithi106 ceph-mon[36505]: pgmap v2204: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3248.152559 secs 2024-03-24T01:29:34.004 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3902 sec 2024-03-24T01:29:34.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:33 smithi078 ceph-mon[32429]: pgmap v2204: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:34.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3248.152559 secs 2024-03-24T01:29:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:33 smithi107 ceph-mon[36485]: pgmap v2204: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3248.152559 secs 2024-03-24T01:29:35.004 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3903 sec 2024-03-24T01:29:35.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:35 smithi106 ceph-mon[36505]: pgmap v2205: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:36.004 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3904 sec 2024-03-24T01:29:36.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:35 smithi078 ceph-mon[32429]: pgmap v2205: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:36.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:35 smithi107 ceph-mon[36485]: pgmap v2205: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:37.004 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3905 sec 2024-03-24T01:29:37.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:37 smithi106 ceph-mon[36505]: pgmap v2206: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:38.004 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3906 sec 2024-03-24T01:29:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:37 smithi078 ceph-mon[32429]: pgmap v2206: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:38.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:37 smithi107 ceph-mon[36485]: pgmap v2206: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3253.152597 secs 2024-03-24T01:29:39.004 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3907 sec 2024-03-24T01:29:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3253.152597 secs 2024-03-24T01:29:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3253.152597 secs 2024-03-24T01:29:39.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:39 smithi106 ceph-mon[36505]: pgmap v2207: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:40.005 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3908 sec 2024-03-24T01:29:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:39 smithi078 ceph-mon[32429]: pgmap v2207: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:39 smithi107 ceph-mon[36485]: pgmap v2207: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:40 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:29:41.004 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3909 sec 2024-03-24T01:29:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:40 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:29:41.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:40 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:29:41.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:41 smithi106 ceph-mon[36505]: pgmap v2208: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:41.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:41 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:29:41.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:41 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:29:41.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:41 smithi106 ceph-mon[36505]: pgmap v2209: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:29:41.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:41 smithi106 ceph-mon[36505]: pgmap v2210: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:29:41.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:41 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:29:41.995 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:41 smithi107 ceph-mon[36485]: pgmap v2208: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:41.995 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:41 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:29:41.995 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:41 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:29:41.995 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:41 smithi107 ceph-mon[36485]: pgmap v2209: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:29:41.995 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:41 smithi107 ceph-mon[36485]: pgmap v2210: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:29:41.995 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:41 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:29:42.005 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3910 sec 2024-03-24T01:29:42.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:41 smithi078 ceph-mon[32429]: pgmap v2208: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:42.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:41 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:29:42.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:41 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:29:42.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:41 smithi078 ceph-mon[32429]: pgmap v2209: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:29:42.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:41 smithi078 ceph-mon[32429]: pgmap v2210: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:29:42.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:41 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:29:43.005 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3911 sec 2024-03-24T01:29:44.005 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3912 sec 2024-03-24T01:29:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:43 smithi078 ceph-mon[32429]: pgmap v2211: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:29:44.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3258.152658 secs 2024-03-24T01:29:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:43 smithi107 ceph-mon[36485]: pgmap v2211: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:29:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3258.152658 secs 2024-03-24T01:29:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:43 smithi106 ceph-mon[36505]: pgmap v2211: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:29:44.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3258.152658 secs 2024-03-24T01:29:45.005 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3913 sec 2024-03-24T01:29:46.005 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3914 sec 2024-03-24T01:29:46.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:45 smithi078 ceph-mon[32429]: pgmap v2212: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:29:46.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:45 smithi107 ceph-mon[36485]: pgmap v2212: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:29:46.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:45 smithi106 ceph-mon[36505]: pgmap v2212: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:29:47.005 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3915 sec 2024-03-24T01:29:48.005 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3916 sec 2024-03-24T01:29:48.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:47 smithi078 ceph-mon[32429]: pgmap v2213: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:29:48.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:47 smithi107 ceph-mon[36485]: pgmap v2213: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:29:48.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:47 smithi106 ceph-mon[36505]: pgmap v2213: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:29:49.005 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3917 sec 2024-03-24T01:29:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3263.152706 secs 2024-03-24T01:29:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3263.152706 secs 2024-03-24T01:29:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3263.152706 secs 2024-03-24T01:29:50.005 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3918 sec 2024-03-24T01:29:50.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:49 smithi078 ceph-mon[32429]: pgmap v2214: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:29:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:49 smithi107 ceph-mon[36485]: pgmap v2214: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:29:50.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:49 smithi106 ceph-mon[36505]: pgmap v2214: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:29:51.005 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3919 sec 2024-03-24T01:29:52.005 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3920 sec 2024-03-24T01:29:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:51 smithi078 ceph-mon[32429]: pgmap v2215: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:29:52.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:51 smithi107 ceph-mon[36485]: pgmap v2215: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:29:52.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:51 smithi106 ceph-mon[36505]: pgmap v2215: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:29:53.005 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3921 sec 2024-03-24T01:29:54.006 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3922 sec 2024-03-24T01:29:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:53 smithi078 ceph-mon[32429]: pgmap v2216: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3268.152763 secs 2024-03-24T01:29:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:53 smithi107 ceph-mon[36485]: pgmap v2216: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3268.152763 secs 2024-03-24T01:29:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:53 smithi106 ceph-mon[36505]: pgmap v2216: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3268.152763 secs 2024-03-24T01:29:55.006 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3923 sec 2024-03-24T01:29:56.006 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3924 sec 2024-03-24T01:29:56.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:55 smithi078 ceph-mon[32429]: pgmap v2217: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:56.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:55 smithi107 ceph-mon[36485]: pgmap v2217: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:56.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:55 smithi106 ceph-mon[36505]: pgmap v2217: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:29:57.006 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3925 sec 2024-03-24T01:29:58.006 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3926 sec 2024-03-24T01:29:58.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:57 smithi078 ceph-mon[32429]: pgmap v2218: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:57 smithi107 ceph-mon[36485]: pgmap v2218: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:58.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:57 smithi106 ceph-mon[36505]: pgmap v2218: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:59.006 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3927 sec 2024-03-24T01:29:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3273.152821 secs 2024-03-24T01:29:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3273.152821 secs 2024-03-24T01:29:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3273.152821 secs 2024-03-24T01:30:00.006 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3928 sec 2024-03-24T01:30:00.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:29:59 smithi078 ceph-mon[32429]: pgmap v2219: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:00.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:29:59 smithi107 ceph-mon[36485]: pgmap v2219: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:00.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:29:59 smithi106 ceph-mon[36505]: pgmap v2219: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:01.006 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3929 sec 2024-03-24T01:30:01.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:01 smithi078 ceph-mon[32429]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:30:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:01 smithi107 ceph-mon[36485]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:30:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:01 smithi106 ceph-mon[36505]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:30:02.006 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3930 sec 2024-03-24T01:30:02.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:02 smithi078 ceph-mon[32429]: pgmap v2220: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:02.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:02 smithi107 ceph-mon[36485]: pgmap v2220: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:02.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:02 smithi106 ceph-mon[36505]: pgmap v2220: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:03.006 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3931 sec 2024-03-24T01:30:04.006 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3932 sec 2024-03-24T01:30:04.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:04 smithi078 ceph-mon[32429]: pgmap v2221: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:04.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3278.152908 secs 2024-03-24T01:30:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:04 smithi107 ceph-mon[36485]: pgmap v2221: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3278.152908 secs 2024-03-24T01:30:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:04 smithi106 ceph-mon[36505]: pgmap v2221: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3278.152908 secs 2024-03-24T01:30:05.006 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3933 sec 2024-03-24T01:30:06.007 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3934 sec 2024-03-24T01:30:06.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:06 smithi078 ceph-mon[32429]: pgmap v2222: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:06.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:06 smithi107 ceph-mon[36485]: pgmap v2222: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:06.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:06 smithi106 ceph-mon[36505]: pgmap v2222: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:07.006 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3935 sec 2024-03-24T01:30:08.006 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3936 sec 2024-03-24T01:30:08.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:08 smithi078 ceph-mon[32429]: pgmap v2223: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:08.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:08 smithi107 ceph-mon[36485]: pgmap v2223: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:08.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:08 smithi106 ceph-mon[36505]: pgmap v2223: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:09.007 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3937 sec 2024-03-24T01:30:09.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3283.152992 secs 2024-03-24T01:30:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3283.152992 secs 2024-03-24T01:30:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3283.152992 secs 2024-03-24T01:30:10.006 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3938 sec 2024-03-24T01:30:10.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:10 smithi078 ceph-mon[32429]: pgmap v2224: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:10.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:10 smithi107 ceph-mon[36485]: pgmap v2224: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:10.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:10 smithi106 ceph-mon[36505]: pgmap v2224: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:11.010 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3939 sec 2024-03-24T01:30:12.007 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3940 sec 2024-03-24T01:30:12.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:12 smithi078 ceph-mon[32429]: pgmap v2225: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:12.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:12 smithi107 ceph-mon[36485]: pgmap v2225: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:12.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:12 smithi106 ceph-mon[36505]: pgmap v2225: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:13.007 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3941 sec 2024-03-24T01:30:14.007 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3942 sec 2024-03-24T01:30:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:14 smithi078 ceph-mon[32429]: pgmap v2226: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3288.153079 secs 2024-03-24T01:30:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:14 smithi107 ceph-mon[36485]: pgmap v2226: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3288.153079 secs 2024-03-24T01:30:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:14 smithi106 ceph-mon[36505]: pgmap v2226: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:14.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3288.153079 secs 2024-03-24T01:30:15.007 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3943 sec 2024-03-24T01:30:16.007 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3944 sec 2024-03-24T01:30:16.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:16 smithi078 ceph-mon[32429]: pgmap v2227: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:16.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:16 smithi107 ceph-mon[36485]: pgmap v2227: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:16.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:16 smithi106 ceph-mon[36505]: pgmap v2227: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:17.007 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3945 sec 2024-03-24T01:30:18.007 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3946 sec 2024-03-24T01:30:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:18 smithi078 ceph-mon[32429]: pgmap v2228: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:18 smithi107 ceph-mon[36485]: pgmap v2228: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:18.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:18 smithi106 ceph-mon[36505]: pgmap v2228: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:19.007 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3947 sec 2024-03-24T01:30:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3293.153140 secs 2024-03-24T01:30:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3293.153140 secs 2024-03-24T01:30:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3293.153140 secs 2024-03-24T01:30:20.007 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3948 sec 2024-03-24T01:30:20.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:20 smithi078 ceph-mon[32429]: pgmap v2229: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:20 smithi107 ceph-mon[36485]: pgmap v2229: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:20.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:20 smithi106 ceph-mon[36505]: pgmap v2229: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:21.007 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3949 sec 2024-03-24T01:30:22.007 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3950 sec 2024-03-24T01:30:22.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:22 smithi078 ceph-mon[32429]: pgmap v2230: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:22.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:22 smithi107 ceph-mon[36485]: pgmap v2230: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:22.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:22 smithi106 ceph-mon[36505]: pgmap v2230: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:23.007 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3951 sec 2024-03-24T01:30:24.007 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3952 sec 2024-03-24T01:30:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:24 smithi078 ceph-mon[32429]: pgmap v2231: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3298.153181 secs 2024-03-24T01:30:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:24 smithi107 ceph-mon[36485]: pgmap v2231: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3298.153181 secs 2024-03-24T01:30:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:24 smithi106 ceph-mon[36505]: pgmap v2231: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3298.153181 secs 2024-03-24T01:30:25.008 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3953 sec 2024-03-24T01:30:26.008 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3954 sec 2024-03-24T01:30:26.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:26 smithi078 ceph-mon[32429]: pgmap v2232: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:26.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:26 smithi107 ceph-mon[36485]: pgmap v2232: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:26.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:26 smithi106 ceph-mon[36505]: pgmap v2232: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:27.008 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3955 sec 2024-03-24T01:30:28.008 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3956 sec 2024-03-24T01:30:28.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:28 smithi078 ceph-mon[32429]: pgmap v2233: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:28.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:28 smithi107 ceph-mon[36485]: pgmap v2233: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:28.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:28 smithi106 ceph-mon[36505]: pgmap v2233: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:29.008 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3957 sec 2024-03-24T01:30:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3303.153241 secs 2024-03-24T01:30:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3303.153241 secs 2024-03-24T01:30:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3303.153241 secs 2024-03-24T01:30:30.008 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3958 sec 2024-03-24T01:30:30.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:30 smithi078 ceph-mon[32429]: pgmap v2234: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:30.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:30 smithi107 ceph-mon[36485]: pgmap v2234: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:30.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:30 smithi106 ceph-mon[36505]: pgmap v2234: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:31.008 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3959 sec 2024-03-24T01:30:32.008 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3960 sec 2024-03-24T01:30:32.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:32 smithi078 ceph-mon[32429]: pgmap v2235: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:32.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:32 smithi107 ceph-mon[36485]: pgmap v2235: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:32.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:32 smithi106 ceph-mon[36505]: pgmap v2235: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:33.008 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3961 sec 2024-03-24T01:30:34.008 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3962 sec 2024-03-24T01:30:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:34 smithi078 ceph-mon[32429]: pgmap v2236: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:34.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3308.153287 secs 2024-03-24T01:30:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:34 smithi107 ceph-mon[36485]: pgmap v2236: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3308.153287 secs 2024-03-24T01:30:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:34 smithi106 ceph-mon[36505]: pgmap v2236: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3308.153287 secs 2024-03-24T01:30:35.009 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3963 sec 2024-03-24T01:30:36.009 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3964 sec 2024-03-24T01:30:36.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:36 smithi078 ceph-mon[32429]: pgmap v2237: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:36.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:36 smithi107 ceph-mon[36485]: pgmap v2237: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:36.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:36 smithi106 ceph-mon[36505]: pgmap v2237: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:37.009 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3965 sec 2024-03-24T01:30:38.009 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3966 sec 2024-03-24T01:30:38.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:38 smithi078 ceph-mon[32429]: pgmap v2238: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:38.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:38 smithi107 ceph-mon[36485]: pgmap v2238: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:38.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:38 smithi106 ceph-mon[36505]: pgmap v2238: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:39.009 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3967 sec 2024-03-24T01:30:39.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3313.153367 secs 2024-03-24T01:30:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3313.153367 secs 2024-03-24T01:30:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3313.153367 secs 2024-03-24T01:30:40.009 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3968 sec 2024-03-24T01:30:40.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:40 smithi078 ceph-mon[32429]: pgmap v2239: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:40 smithi107 ceph-mon[36485]: pgmap v2239: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:40.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:40 smithi106 ceph-mon[36505]: pgmap v2239: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:41.009 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3969 sec 2024-03-24T01:30:41.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:41 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:30:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:41 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:30:41.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:41 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:30:42.009 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3970 sec 2024-03-24T01:30:42.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:42 smithi078 ceph-mon[32429]: pgmap v2240: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:42.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:42 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:30:42.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:42 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:30:42.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:42 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:30:42.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:42 smithi107 ceph-mon[36485]: pgmap v2240: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:42.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:42 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:30:42.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:42 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:30:42.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:42 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:30:42.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:42 smithi106 ceph-mon[36505]: pgmap v2240: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:42.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:42 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:30:42.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:42 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:30:42.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:42 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:30:43.009 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3971 sec 2024-03-24T01:30:43.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:43 smithi078 ceph-mon[32429]: pgmap v2241: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:30:43.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:43 smithi078 ceph-mon[32429]: pgmap v2242: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:30:43.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:43 smithi107 ceph-mon[36485]: pgmap v2241: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:30:43.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:43 smithi107 ceph-mon[36485]: pgmap v2242: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:30:43.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:43 smithi106 ceph-mon[36505]: pgmap v2241: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:30:43.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:43 smithi106 ceph-mon[36505]: pgmap v2242: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:30:44.009 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3972 sec 2024-03-24T01:30:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3318.153451 secs 2024-03-24T01:30:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3318.153451 secs 2024-03-24T01:30:44.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3318.153451 secs 2024-03-24T01:30:45.009 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3973 sec 2024-03-24T01:30:45.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:45 smithi106 ceph-mon[36505]: pgmap v2243: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:30:45.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:45 smithi078 ceph-mon[32429]: pgmap v2243: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:30:45.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:45 smithi107 ceph-mon[36485]: pgmap v2243: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:30:46.009 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3974 sec 2024-03-24T01:30:47.009 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3975 sec 2024-03-24T01:30:47.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:47 smithi106 ceph-mon[36505]: pgmap v2244: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:30:47.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:47 smithi078 ceph-mon[32429]: pgmap v2244: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:30:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:47 smithi107 ceph-mon[36485]: pgmap v2244: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:30:48.009 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3976 sec 2024-03-24T01:30:49.010 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3977 sec 2024-03-24T01:30:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:49 smithi106 ceph-mon[36505]: pgmap v2245: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:30:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3323.153538 secs 2024-03-24T01:30:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:49 smithi078 ceph-mon[32429]: pgmap v2245: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:30:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3323.153538 secs 2024-03-24T01:30:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:49 smithi107 ceph-mon[36485]: pgmap v2245: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:30:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3323.153538 secs 2024-03-24T01:30:50.010 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3978 sec 2024-03-24T01:30:51.010 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3979 sec 2024-03-24T01:30:51.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:51 smithi106 ceph-mon[36505]: pgmap v2246: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:30:51.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:51 smithi078 ceph-mon[32429]: pgmap v2246: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:30:51.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:51 smithi107 ceph-mon[36485]: pgmap v2246: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:30:52.010 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3980 sec 2024-03-24T01:30:53.010 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3981 sec 2024-03-24T01:30:53.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:53 smithi106 ceph-mon[36505]: pgmap v2247: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:30:53.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:53 smithi078 ceph-mon[32429]: pgmap v2247: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:30:53.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:53 smithi107 ceph-mon[36485]: pgmap v2247: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:30:54.010 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3982 sec 2024-03-24T01:30:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3328.153598 secs 2024-03-24T01:30:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3328.153598 secs 2024-03-24T01:30:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3328.153598 secs 2024-03-24T01:30:55.010 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3983 sec 2024-03-24T01:30:55.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:55 smithi106 ceph-mon[36505]: pgmap v2248: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:55.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:55 smithi078 ceph-mon[32429]: pgmap v2248: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:55.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:55 smithi107 ceph-mon[36485]: pgmap v2248: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:56.010 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3984 sec 2024-03-24T01:30:57.010 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3985 sec 2024-03-24T01:30:57.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:57 smithi106 ceph-mon[36505]: pgmap v2249: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:57.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:57 smithi078 ceph-mon[32429]: pgmap v2249: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:57.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:57 smithi107 ceph-mon[36485]: pgmap v2249: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:58.010 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3986 sec 2024-03-24T01:30:59.011 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3987 sec 2024-03-24T01:30:59.392 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:59 smithi078 ceph-mon[32429]: pgmap v2250: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:59.392 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:30:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3333.153633 secs 2024-03-24T01:30:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:59 smithi106 ceph-mon[36505]: pgmap v2250: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:30:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3333.153633 secs 2024-03-24T01:30:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:59 smithi107 ceph-mon[36485]: pgmap v2250: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:30:59.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:30:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3333.153633 secs 2024-03-24T01:31:00.011 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3988 sec 2024-03-24T01:31:01.011 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3989 sec 2024-03-24T01:31:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:01 smithi106 ceph-mon[36505]: pgmap v2251: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:01.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:01 smithi078 ceph-mon[32429]: pgmap v2251: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:01.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:01 smithi107 ceph-mon[36485]: pgmap v2251: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:02.011 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3990 sec 2024-03-24T01:31:03.011 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3991 sec 2024-03-24T01:31:03.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:03 smithi106 ceph-mon[36505]: pgmap v2252: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:03.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:03 smithi078 ceph-mon[32429]: pgmap v2252: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:03.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:03 smithi107 ceph-mon[36485]: pgmap v2252: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:04.011 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3992 sec 2024-03-24T01:31:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3338.153693 secs 2024-03-24T01:31:04.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3338.153693 secs 2024-03-24T01:31:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3338.153693 secs 2024-03-24T01:31:05.011 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3993 sec 2024-03-24T01:31:05.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:05 smithi106 ceph-mon[36505]: pgmap v2253: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:05.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:05 smithi078 ceph-mon[32429]: pgmap v2253: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:05.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:05 smithi107 ceph-mon[36485]: pgmap v2253: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:06.012 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3994 sec 2024-03-24T01:31:06.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:06 smithi078 ceph-mon[32429]: pgmap v2254: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:06.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:06 smithi107 ceph-mon[36485]: pgmap v2254: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:06.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:06 smithi106 ceph-mon[36505]: pgmap v2254: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:07.011 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3995 sec 2024-03-24T01:31:08.011 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3996 sec 2024-03-24T01:31:09.011 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3997 sec 2024-03-24T01:31:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:08 smithi078 ceph-mon[32429]: pgmap v2255: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3343.153782 secs 2024-03-24T01:31:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:08 smithi107 ceph-mon[36485]: pgmap v2255: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3343.153782 secs 2024-03-24T01:31:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:08 smithi106 ceph-mon[36505]: pgmap v2255: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3343.153782 secs 2024-03-24T01:31:10.011 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3998 sec 2024-03-24T01:31:11.011 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 3999 sec 2024-03-24T01:31:11.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:10 smithi078 ceph-mon[32429]: pgmap v2256: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:11.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:10 smithi107 ceph-mon[36485]: pgmap v2256: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:11.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:10 smithi106 ceph-mon[36505]: pgmap v2256: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:12.011 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4000 sec 2024-03-24T01:31:13.011 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4001 sec 2024-03-24T01:31:13.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:12 smithi078 ceph-mon[32429]: pgmap v2257: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:13.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:12 smithi107 ceph-mon[36485]: pgmap v2257: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:13.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:12 smithi106 ceph-mon[36505]: pgmap v2257: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:14.011 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4002 sec 2024-03-24T01:31:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3348.153873 secs 2024-03-24T01:31:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3348.153873 secs 2024-03-24T01:31:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3348.153873 secs 2024-03-24T01:31:15.011 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4003 sec 2024-03-24T01:31:15.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:14 smithi078 ceph-mon[32429]: pgmap v2258: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:15.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:14 smithi107 ceph-mon[36485]: pgmap v2258: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:15.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:14 smithi106 ceph-mon[36505]: pgmap v2258: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:16.012 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4004 sec 2024-03-24T01:31:17.011 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4005 sec 2024-03-24T01:31:17.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:16 smithi078 ceph-mon[32429]: pgmap v2259: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:17.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:16 smithi107 ceph-mon[36485]: pgmap v2259: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:17.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:16 smithi106 ceph-mon[36505]: pgmap v2259: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:18.012 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4006 sec 2024-03-24T01:31:19.012 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4007 sec 2024-03-24T01:31:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:18 smithi078 ceph-mon[32429]: pgmap v2260: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:19.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3353.153932 secs 2024-03-24T01:31:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:18 smithi107 ceph-mon[36485]: pgmap v2260: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3353.153932 secs 2024-03-24T01:31:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:18 smithi106 ceph-mon[36505]: pgmap v2260: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3353.153932 secs 2024-03-24T01:31:20.012 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4008 sec 2024-03-24T01:31:21.012 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4009 sec 2024-03-24T01:31:21.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:20 smithi078 ceph-mon[32429]: pgmap v2261: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:21.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:20 smithi107 ceph-mon[36485]: pgmap v2261: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:21.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:20 smithi106 ceph-mon[36505]: pgmap v2261: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:22.012 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4010 sec 2024-03-24T01:31:23.012 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4011 sec 2024-03-24T01:31:23.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:22 smithi078 ceph-mon[32429]: pgmap v2262: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:23.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:22 smithi107 ceph-mon[36485]: pgmap v2262: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:23.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:22 smithi106 ceph-mon[36505]: pgmap v2262: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:24.012 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4012 sec 2024-03-24T01:31:24.053 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3358.153992 secs 2024-03-24T01:31:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3358.153992 secs 2024-03-24T01:31:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3358.153992 secs 2024-03-24T01:31:25.012 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4013 sec 2024-03-24T01:31:25.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:24 smithi078 ceph-mon[32429]: pgmap v2263: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:24 smithi107 ceph-mon[36485]: pgmap v2263: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:25.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:24 smithi106 ceph-mon[36505]: pgmap v2263: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:26.012 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4014 sec 2024-03-24T01:31:27.013 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4015 sec 2024-03-24T01:31:27.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:26 smithi078 ceph-mon[32429]: pgmap v2264: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:27.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:26 smithi107 ceph-mon[36485]: pgmap v2264: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:27.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:26 smithi106 ceph-mon[36505]: pgmap v2264: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:28.013 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4016 sec 2024-03-24T01:31:29.013 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4017 sec 2024-03-24T01:31:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:28 smithi078 ceph-mon[32429]: pgmap v2265: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:29.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3363.154075 secs 2024-03-24T01:31:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:28 smithi107 ceph-mon[36485]: pgmap v2265: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3363.154075 secs 2024-03-24T01:31:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:28 smithi106 ceph-mon[36505]: pgmap v2265: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3363.154075 secs 2024-03-24T01:31:30.013 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4018 sec 2024-03-24T01:31:31.013 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4019 sec 2024-03-24T01:31:31.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:30 smithi078 ceph-mon[32429]: pgmap v2266: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:31.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:30 smithi107 ceph-mon[36485]: pgmap v2266: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:31.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:30 smithi106 ceph-mon[36505]: pgmap v2266: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:32.013 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4020 sec 2024-03-24T01:31:33.013 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4021 sec 2024-03-24T01:31:33.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:32 smithi078 ceph-mon[32429]: pgmap v2267: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:33.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:32 smithi107 ceph-mon[36485]: pgmap v2267: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:33.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:32 smithi106 ceph-mon[36505]: pgmap v2267: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:34.013 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4022 sec 2024-03-24T01:31:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3368.154136 secs 2024-03-24T01:31:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3368.154136 secs 2024-03-24T01:31:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3368.154136 secs 2024-03-24T01:31:35.013 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4023 sec 2024-03-24T01:31:35.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:34 smithi078 ceph-mon[32429]: pgmap v2268: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:35.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:34 smithi107 ceph-mon[36485]: pgmap v2268: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:35.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:34 smithi106 ceph-mon[36505]: pgmap v2268: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:36.013 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4024 sec 2024-03-24T01:31:37.013 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4025 sec 2024-03-24T01:31:37.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:36 smithi078 ceph-mon[32429]: pgmap v2269: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:37.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:36 smithi107 ceph-mon[36485]: pgmap v2269: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:37.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:36 smithi106 ceph-mon[36505]: pgmap v2269: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:38.013 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4026 sec 2024-03-24T01:31:39.013 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4027 sec 2024-03-24T01:31:39.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:38 smithi078 ceph-mon[32429]: pgmap v2270: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:39.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3373.154197 secs 2024-03-24T01:31:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:38 smithi107 ceph-mon[36485]: pgmap v2270: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3373.154197 secs 2024-03-24T01:31:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:38 smithi106 ceph-mon[36505]: pgmap v2270: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3373.154197 secs 2024-03-24T01:31:40.014 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4028 sec 2024-03-24T01:31:41.014 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4029 sec 2024-03-24T01:31:41.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:40 smithi078 ceph-mon[32429]: pgmap v2271: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:41.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:40 smithi107 ceph-mon[36485]: pgmap v2271: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:41.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:40 smithi106 ceph-mon[36505]: pgmap v2271: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:42.013 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4030 sec 2024-03-24T01:31:42.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:41 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:31:42.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:41 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:31:42.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:41 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:31:43.014 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4031 sec 2024-03-24T01:31:43.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:42 smithi078 ceph-mon[32429]: pgmap v2272: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:43.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:42 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:31:43.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:42 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:31:43.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:42 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:31:43.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:42 smithi107 ceph-mon[36485]: pgmap v2272: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:43.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:42 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:31:43.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:42 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:31:43.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:42 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:31:43.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:42 smithi106 ceph-mon[36505]: pgmap v2272: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:43.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:42 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:31:43.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:42 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:31:43.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:42 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:31:44.014 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4032 sec 2024-03-24T01:31:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:43 smithi078 ceph-mon[32429]: pgmap v2273: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:31:44.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:43 smithi078 ceph-mon[32429]: pgmap v2274: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:31:44.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3378.154295 secs 2024-03-24T01:31:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:43 smithi107 ceph-mon[36485]: pgmap v2273: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:31:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:43 smithi107 ceph-mon[36485]: pgmap v2274: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:31:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3378.154295 secs 2024-03-24T01:31:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:43 smithi106 ceph-mon[36505]: pgmap v2273: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:31:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:43 smithi106 ceph-mon[36505]: pgmap v2274: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:31:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3378.154295 secs 2024-03-24T01:31:45.014 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4033 sec 2024-03-24T01:31:46.014 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4034 sec 2024-03-24T01:31:46.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:45 smithi078 ceph-mon[32429]: pgmap v2275: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:31:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:45 smithi107 ceph-mon[36485]: pgmap v2275: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:31:46.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:45 smithi106 ceph-mon[36505]: pgmap v2275: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:31:47.014 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4035 sec 2024-03-24T01:31:48.014 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4036 sec 2024-03-24T01:31:48.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:47 smithi078 ceph-mon[32429]: pgmap v2276: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:31:48.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:47 smithi107 ceph-mon[36485]: pgmap v2276: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:31:48.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:47 smithi106 ceph-mon[36505]: pgmap v2276: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:31:49.014 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4037 sec 2024-03-24T01:31:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3383.154321 secs 2024-03-24T01:31:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3383.154321 secs 2024-03-24T01:31:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3383.154321 secs 2024-03-24T01:31:50.014 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4038 sec 2024-03-24T01:31:50.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:49 smithi078 ceph-mon[32429]: pgmap v2277: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:31:50.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:49 smithi107 ceph-mon[36485]: pgmap v2277: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:31:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:49 smithi106 ceph-mon[36505]: pgmap v2277: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:31:51.014 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4039 sec 2024-03-24T01:31:52.014 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4040 sec 2024-03-24T01:31:52.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:51 smithi078 ceph-mon[32429]: pgmap v2278: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:31:52.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:51 smithi107 ceph-mon[36485]: pgmap v2278: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:31:52.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:51 smithi106 ceph-mon[36505]: pgmap v2278: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:31:53.014 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4041 sec 2024-03-24T01:31:54.015 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4042 sec 2024-03-24T01:31:54.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:53 smithi078 ceph-mon[32429]: pgmap v2279: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:31:54.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3388.154360 secs 2024-03-24T01:31:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:53 smithi107 ceph-mon[36485]: pgmap v2279: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:31:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3388.154360 secs 2024-03-24T01:31:54.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:53 smithi106 ceph-mon[36505]: pgmap v2279: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:31:54.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3388.154360 secs 2024-03-24T01:31:55.015 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4043 sec 2024-03-24T01:31:56.015 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4044 sec 2024-03-24T01:31:56.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:55 smithi078 ceph-mon[32429]: pgmap v2280: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:56.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:55 smithi107 ceph-mon[36485]: pgmap v2280: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:56.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:55 smithi106 ceph-mon[36505]: pgmap v2280: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:31:57.015 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4045 sec 2024-03-24T01:31:58.015 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4046 sec 2024-03-24T01:31:58.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:57 smithi078 ceph-mon[32429]: pgmap v2281: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:58.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:57 smithi107 ceph-mon[36485]: pgmap v2281: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:58.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:57 smithi106 ceph-mon[36505]: pgmap v2281: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:59.015 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4047 sec 2024-03-24T01:31:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3393.154422 secs 2024-03-24T01:31:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3393.154422 secs 2024-03-24T01:31:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3393.154422 secs 2024-03-24T01:32:00.015 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4048 sec 2024-03-24T01:32:00.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:31:59 smithi106 ceph-mon[36505]: pgmap v2282: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:00.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:31:59 smithi078 ceph-mon[32429]: pgmap v2282: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:00.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:31:59 smithi107 ceph-mon[36485]: pgmap v2282: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:01.016 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4049 sec 2024-03-24T01:32:02.015 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4050 sec 2024-03-24T01:32:02.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:01 smithi106 ceph-mon[36505]: pgmap v2283: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:02.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:01 smithi078 ceph-mon[32429]: pgmap v2283: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:02.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:01 smithi107 ceph-mon[36485]: pgmap v2283: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:03.015 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4051 sec 2024-03-24T01:32:04.015 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4052 sec 2024-03-24T01:32:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:03 smithi106 ceph-mon[36505]: pgmap v2284: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3398.154485 secs 2024-03-24T01:32:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:03 smithi078 ceph-mon[32429]: pgmap v2284: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:04.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3398.154485 secs 2024-03-24T01:32:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:03 smithi107 ceph-mon[36485]: pgmap v2284: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3398.154485 secs 2024-03-24T01:32:05.016 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4053 sec 2024-03-24T01:32:06.016 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4054 sec 2024-03-24T01:32:06.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:05 smithi106 ceph-mon[36505]: pgmap v2285: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:06.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:05 smithi078 ceph-mon[32429]: pgmap v2285: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:06.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:05 smithi107 ceph-mon[36485]: pgmap v2285: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:07.016 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4055 sec 2024-03-24T01:32:08.016 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4056 sec 2024-03-24T01:32:08.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:07 smithi106 ceph-mon[36505]: pgmap v2286: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:08.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:07 smithi078 ceph-mon[32429]: pgmap v2286: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:08.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:07 smithi107 ceph-mon[36485]: pgmap v2286: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:09.016 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4057 sec 2024-03-24T01:32:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3403.154549 secs 2024-03-24T01:32:09.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3403.154549 secs 2024-03-24T01:32:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3403.154549 secs 2024-03-24T01:32:10.016 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4058 sec 2024-03-24T01:32:10.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:09 smithi106 ceph-mon[36505]: pgmap v2287: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:10.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:09 smithi078 ceph-mon[32429]: pgmap v2287: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:10.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:09 smithi107 ceph-mon[36485]: pgmap v2287: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:11.016 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4059 sec 2024-03-24T01:32:12.016 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4060 sec 2024-03-24T01:32:12.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:11 smithi107 ceph-mon[36485]: pgmap v2288: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:12.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:11 smithi106 ceph-mon[36505]: pgmap v2288: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:12.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:11 smithi078 ceph-mon[32429]: pgmap v2288: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:13.016 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4061 sec 2024-03-24T01:32:14.016 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4062 sec 2024-03-24T01:32:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:13 smithi106 ceph-mon[36505]: pgmap v2289: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3408.154613 secs 2024-03-24T01:32:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:13 smithi078 ceph-mon[32429]: pgmap v2289: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3408.154613 secs 2024-03-24T01:32:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:13 smithi107 ceph-mon[36485]: pgmap v2289: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3408.154613 secs 2024-03-24T01:32:15.016 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4063 sec 2024-03-24T01:32:16.016 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4064 sec 2024-03-24T01:32:16.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:15 smithi106 ceph-mon[36505]: pgmap v2290: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:16.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:15 smithi078 ceph-mon[32429]: pgmap v2290: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:16.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:15 smithi107 ceph-mon[36485]: pgmap v2290: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:17.016 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4065 sec 2024-03-24T01:32:18.017 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4066 sec 2024-03-24T01:32:18.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:17 smithi106 ceph-mon[36505]: pgmap v2291: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:17 smithi078 ceph-mon[32429]: pgmap v2291: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:17 smithi107 ceph-mon[36485]: pgmap v2291: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:19.017 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4067 sec 2024-03-24T01:32:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3413.154676 secs 2024-03-24T01:32:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3413.154676 secs 2024-03-24T01:32:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3413.154676 secs 2024-03-24T01:32:20.017 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4068 sec 2024-03-24T01:32:20.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:19 smithi106 ceph-mon[36505]: pgmap v2292: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:20.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:19 smithi078 ceph-mon[32429]: pgmap v2292: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:19 smithi107 ceph-mon[36485]: pgmap v2292: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:21.017 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4069 sec 2024-03-24T01:32:22.017 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4070 sec 2024-03-24T01:32:22.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:21 smithi106 ceph-mon[36505]: pgmap v2293: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:22.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:21 smithi078 ceph-mon[32429]: pgmap v2293: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:22.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:21 smithi107 ceph-mon[36485]: pgmap v2293: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:23.017 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4071 sec 2024-03-24T01:32:24.017 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4072 sec 2024-03-24T01:32:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:23 smithi106 ceph-mon[36505]: pgmap v2294: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3418.154737 secs 2024-03-24T01:32:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:23 smithi078 ceph-mon[32429]: pgmap v2294: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3418.154737 secs 2024-03-24T01:32:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:23 smithi107 ceph-mon[36485]: pgmap v2294: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3418.154737 secs 2024-03-24T01:32:25.017 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4073 sec 2024-03-24T01:32:26.017 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4074 sec 2024-03-24T01:32:26.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:25 smithi106 ceph-mon[36505]: pgmap v2295: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:26.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:25 smithi078 ceph-mon[32429]: pgmap v2295: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:26.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:25 smithi107 ceph-mon[36485]: pgmap v2295: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:27.017 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4075 sec 2024-03-24T01:32:28.017 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4076 sec 2024-03-24T01:32:28.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:27 smithi106 ceph-mon[36505]: pgmap v2296: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:28.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:27 smithi078 ceph-mon[32429]: pgmap v2296: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:28.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:27 smithi107 ceph-mon[36485]: pgmap v2296: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:29.018 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4077 sec 2024-03-24T01:32:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3423.154800 secs 2024-03-24T01:32:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3423.154800 secs 2024-03-24T01:32:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3423.154800 secs 2024-03-24T01:32:30.018 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4078 sec 2024-03-24T01:32:30.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:29 smithi106 ceph-mon[36505]: pgmap v2297: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:30.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:29 smithi078 ceph-mon[32429]: pgmap v2297: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:30.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:29 smithi107 ceph-mon[36485]: pgmap v2297: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:31.018 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4079 sec 2024-03-24T01:32:32.018 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4080 sec 2024-03-24T01:32:32.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:31 smithi106 ceph-mon[36505]: pgmap v2298: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:32.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:31 smithi078 ceph-mon[32429]: pgmap v2298: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:32.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:31 smithi107 ceph-mon[36485]: pgmap v2298: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:33.018 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4081 sec 2024-03-24T01:32:34.018 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4082 sec 2024-03-24T01:32:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:33 smithi078 ceph-mon[32429]: pgmap v2299: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:34.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3428.154889 secs 2024-03-24T01:32:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:33 smithi107 ceph-mon[36485]: pgmap v2299: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3428.154889 secs 2024-03-24T01:32:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:33 smithi106 ceph-mon[36505]: pgmap v2299: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3428.154889 secs 2024-03-24T01:32:35.018 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4083 sec 2024-03-24T01:32:36.018 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4084 sec 2024-03-24T01:32:36.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:35 smithi078 ceph-mon[32429]: pgmap v2300: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:36.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:35 smithi107 ceph-mon[36485]: pgmap v2300: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:36.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:35 smithi106 ceph-mon[36505]: pgmap v2300: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:37.018 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4085 sec 2024-03-24T01:32:38.018 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4086 sec 2024-03-24T01:32:38.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:37 smithi078 ceph-mon[32429]: pgmap v2301: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:38.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:37 smithi107 ceph-mon[36485]: pgmap v2301: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:38.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:37 smithi106 ceph-mon[36505]: pgmap v2301: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:39.018 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4087 sec 2024-03-24T01:32:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3433.154906 secs 2024-03-24T01:32:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3433.154906 secs 2024-03-24T01:32:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3433.154906 secs 2024-03-24T01:32:40.018 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4088 sec 2024-03-24T01:32:40.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:39 smithi078 ceph-mon[32429]: pgmap v2302: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:40.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:39 smithi107 ceph-mon[36485]: pgmap v2302: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:40.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:39 smithi106 ceph-mon[36505]: pgmap v2302: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:41.018 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4089 sec 2024-03-24T01:32:42.018 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4090 sec 2024-03-24T01:32:42.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:41 smithi078 ceph-mon[32429]: pgmap v2303: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:42.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:41 smithi107 ceph-mon[36485]: pgmap v2303: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:42.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:41 smithi106 ceph-mon[36505]: pgmap v2303: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:43.018 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4091 sec 2024-03-24T01:32:43.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:42 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:32:43.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:42 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:32:43.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:42 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:32:44.019 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4092 sec 2024-03-24T01:32:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:43 smithi078 ceph-mon[32429]: pgmap v2304: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:44.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3438.154971 secs 2024-03-24T01:32:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:43 smithi107 ceph-mon[36485]: pgmap v2304: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3438.154971 secs 2024-03-24T01:32:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:43 smithi106 ceph-mon[36505]: pgmap v2304: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:44.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3438.154971 secs 2024-03-24T01:32:45.019 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4093 sec 2024-03-24T01:32:45.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:32:45.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:32:45.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:45 smithi078 ceph-mon[32429]: pgmap v2305: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:45.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:32:45.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:32:45.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:32:45.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:32:45.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:45.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:45.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:45.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:45.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:45.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:45.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:45.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:45.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:32:45.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:32:45.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:45 smithi107 ceph-mon[36485]: pgmap v2305: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:45.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:32:45.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:32:45.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:32:45.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:32:45.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:45.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:45.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:45.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:45.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:45.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:45.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:45.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:45.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:32:45.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:32:45.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:45 smithi106 ceph-mon[36505]: pgmap v2305: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:45.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:32:45.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:32:45.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:32:45.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:32:45.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:45.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:45.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:45.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:45.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:45.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:45.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:45.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:46.019 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4094 sec 2024-03-24T01:32:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:46 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T01:32:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:46 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T01:32:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:46 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T01:32:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:46 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T01:32:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:46 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:46 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:46 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:46 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:46 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T01:32:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:46 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T01:32:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:46 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:32:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:46 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:32:46.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:46 smithi078 ceph-mon[32429]: pgmap v2306: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T01:32:46.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:46 smithi078 ceph-mon[32429]: pgmap v2307: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:32:46.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:46 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:32:46.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:46 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T01:32:46.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:46 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T01:32:46.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:46 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T01:32:46.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:46 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T01:32:46.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:46 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:46.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:46 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:46.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:46 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:46.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:46 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:46.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:46 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T01:32:46.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:46 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T01:32:46.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:46 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:32:46.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:46 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:32:46.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:46 smithi107 ceph-mon[36485]: pgmap v2306: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T01:32:46.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:46 smithi107 ceph-mon[36485]: pgmap v2307: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:32:46.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:46 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:32:46.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:46 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T01:32:46.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:46 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T01:32:46.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:46 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T01:32:46.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:46 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T01:32:46.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:46 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:46.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:46 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:46.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:46 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:46.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:46 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:32:46.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:46 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T01:32:46.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:46 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T01:32:46.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:46 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:32:46.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:46 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:32:46.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:46 smithi106 ceph-mon[36505]: pgmap v2306: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T01:32:46.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:46 smithi106 ceph-mon[36505]: pgmap v2307: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:32:46.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:46 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:32:47.021 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4095 sec 2024-03-24T01:32:48.019 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4096 sec 2024-03-24T01:32:48.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:48 smithi078 ceph-mon[32429]: pgmap v2308: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:32:48.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3443.155060 secs 2024-03-24T01:32:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:48 smithi107 ceph-mon[36485]: pgmap v2308: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:32:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3443.155060 secs 2024-03-24T01:32:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:48 smithi106 ceph-mon[36505]: pgmap v2308: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:32:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3443.155060 secs 2024-03-24T01:32:49.019 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4097 sec 2024-03-24T01:32:50.019 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4098 sec 2024-03-24T01:32:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:50 smithi078 ceph-mon[32429]: pgmap v2309: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:32:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:50 smithi107 ceph-mon[36485]: pgmap v2309: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:32:50.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:50 smithi106 ceph-mon[36505]: pgmap v2309: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:32:51.019 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4099 sec 2024-03-24T01:32:52.019 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4100 sec 2024-03-24T01:32:52.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:52 smithi078 ceph-mon[32429]: pgmap v2310: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:32:52.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:52 smithi107 ceph-mon[36485]: pgmap v2310: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:32:52.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:52 smithi106 ceph-mon[36505]: pgmap v2310: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:32:53.020 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4101 sec 2024-03-24T01:32:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3448.155150 secs 2024-03-24T01:32:53.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3448.155150 secs 2024-03-24T01:32:54.019 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4102 sec 2024-03-24T01:32:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3448.155150 secs 2024-03-24T01:32:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:54 smithi107 ceph-mon[36485]: pgmap v2311: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:32:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:54 smithi106 ceph-mon[36505]: pgmap v2311: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:32:55.019 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4103 sec 2024-03-24T01:32:55.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:54 smithi078 ceph-mon[32429]: pgmap v2311: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:32:56.019 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4104 sec 2024-03-24T01:32:56.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:56 smithi106 ceph-mon[36505]: pgmap v2312: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:32:57.019 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4105 sec 2024-03-24T01:32:57.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:56 smithi078 ceph-mon[32429]: pgmap v2312: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:32:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:56 smithi107 ceph-mon[36485]: pgmap v2312: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:32:58.019 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4106 sec 2024-03-24T01:32:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:58 smithi106 ceph-mon[36505]: pgmap v2313: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:32:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3453.155238 secs 2024-03-24T01:32:59.020 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4107 sec 2024-03-24T01:32:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:58 smithi078 ceph-mon[32429]: pgmap v2313: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:59.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:32:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3453.155238 secs 2024-03-24T01:32:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:58 smithi107 ceph-mon[36485]: pgmap v2313: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:32:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:32:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3453.155238 secs 2024-03-24T01:33:00.020 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4108 sec 2024-03-24T01:33:00.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:00 smithi078 ceph-mon[32429]: pgmap v2314: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:00.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:00 smithi106 ceph-mon[36505]: pgmap v2314: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:01.020 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4109 sec 2024-03-24T01:33:01.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:00 smithi107 ceph-mon[36485]: pgmap v2314: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:02.020 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4110 sec 2024-03-24T01:33:02.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:02 smithi106 ceph-mon[36505]: pgmap v2315: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:03.020 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4111 sec 2024-03-24T01:33:03.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:02 smithi078 ceph-mon[32429]: pgmap v2315: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:03.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:02 smithi107 ceph-mon[36485]: pgmap v2315: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3458.155330 secs 2024-03-24T01:33:04.020 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4112 sec 2024-03-24T01:33:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3458.155330 secs 2024-03-24T01:33:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3458.155330 secs 2024-03-24T01:33:04.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:04 smithi078 ceph-mon[32429]: pgmap v2316: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:04 smithi106 ceph-mon[36505]: pgmap v2316: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:05.020 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4113 sec 2024-03-24T01:33:05.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:04 smithi107 ceph-mon[36485]: pgmap v2316: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:06.020 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4114 sec 2024-03-24T01:33:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:06 smithi106 ceph-mon[36505]: pgmap v2317: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:07.020 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4115 sec 2024-03-24T01:33:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:06 smithi078 ceph-mon[32429]: pgmap v2317: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:06 smithi107 ceph-mon[36485]: pgmap v2317: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:08.020 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4116 sec 2024-03-24T01:33:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:08 smithi106 ceph-mon[36505]: pgmap v2318: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3463.155390 secs 2024-03-24T01:33:09.020 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4117 sec 2024-03-24T01:33:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:08 smithi078 ceph-mon[32429]: pgmap v2318: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3463.155390 secs 2024-03-24T01:33:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:08 smithi107 ceph-mon[36485]: pgmap v2318: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3463.155390 secs 2024-03-24T01:33:10.020 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4118 sec 2024-03-24T01:33:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:10 smithi106 ceph-mon[36505]: pgmap v2319: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:11.020 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4119 sec 2024-03-24T01:33:11.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:10 smithi078 ceph-mon[32429]: pgmap v2319: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:11.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:10 smithi107 ceph-mon[36485]: pgmap v2319: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:12.020 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4120 sec 2024-03-24T01:33:12.895 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:12 smithi107 ceph-mon[36485]: pgmap v2320: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:12.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:12 smithi106 ceph-mon[36505]: pgmap v2320: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:13.021 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4121 sec 2024-03-24T01:33:13.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:12 smithi078 ceph-mon[32429]: pgmap v2320: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3468.155450 secs 2024-03-24T01:33:14.020 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4122 sec 2024-03-24T01:33:14.089 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3468.155450 secs 2024-03-24T01:33:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3468.155450 secs 2024-03-24T01:33:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:14 smithi106 ceph-mon[36505]: pgmap v2321: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:15.021 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4123 sec 2024-03-24T01:33:15.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:14 smithi078 ceph-mon[32429]: pgmap v2321: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:15.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:14 smithi107 ceph-mon[36485]: pgmap v2321: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:16.021 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4124 sec 2024-03-24T01:33:16.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:16 smithi106 ceph-mon[36505]: pgmap v2322: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:17.021 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4125 sec 2024-03-24T01:33:17.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:16 smithi078 ceph-mon[32429]: pgmap v2322: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:17.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:16 smithi107 ceph-mon[36485]: pgmap v2322: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:18.021 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4126 sec 2024-03-24T01:33:18.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:18 smithi106 ceph-mon[36505]: pgmap v2323: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:18.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3473.155505 secs 2024-03-24T01:33:19.021 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4127 sec 2024-03-24T01:33:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:18 smithi078 ceph-mon[32429]: pgmap v2323: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3473.155505 secs 2024-03-24T01:33:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:18 smithi107 ceph-mon[36485]: pgmap v2323: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3473.155505 secs 2024-03-24T01:33:20.021 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4128 sec 2024-03-24T01:33:20.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:20 smithi106 ceph-mon[36505]: pgmap v2324: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:21.021 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4129 sec 2024-03-24T01:33:21.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:20 smithi078 ceph-mon[32429]: pgmap v2324: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:21.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:20 smithi107 ceph-mon[36485]: pgmap v2324: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:22.021 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4130 sec 2024-03-24T01:33:22.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:22 smithi106 ceph-mon[36505]: pgmap v2325: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:23.022 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4131 sec 2024-03-24T01:33:23.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:22 smithi078 ceph-mon[32429]: pgmap v2325: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:23.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:22 smithi107 ceph-mon[36485]: pgmap v2325: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3478.155519 secs 2024-03-24T01:33:24.021 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4132 sec 2024-03-24T01:33:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3478.155519 secs 2024-03-24T01:33:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3478.155519 secs 2024-03-24T01:33:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:24 smithi106 ceph-mon[36505]: pgmap v2326: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:25.021 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4133 sec 2024-03-24T01:33:25.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:24 smithi078 ceph-mon[32429]: pgmap v2326: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:24 smithi107 ceph-mon[36485]: pgmap v2326: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:26.022 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4134 sec 2024-03-24T01:33:26.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:26 smithi106 ceph-mon[36505]: pgmap v2327: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:27.022 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4135 sec 2024-03-24T01:33:27.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:26 smithi078 ceph-mon[32429]: pgmap v2327: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:27.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:26 smithi107 ceph-mon[36485]: pgmap v2327: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:28.022 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4136 sec 2024-03-24T01:33:29.022 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4137 sec 2024-03-24T01:33:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:28 smithi078 ceph-mon[32429]: pgmap v2328: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:29.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3483.155624 secs 2024-03-24T01:33:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:28 smithi107 ceph-mon[36485]: pgmap v2328: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3483.155624 secs 2024-03-24T01:33:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:28 smithi106 ceph-mon[36505]: pgmap v2328: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3483.155624 secs 2024-03-24T01:33:30.022 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4138 sec 2024-03-24T01:33:31.022 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4139 sec 2024-03-24T01:33:31.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:30 smithi078 ceph-mon[32429]: pgmap v2329: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:31.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:30 smithi107 ceph-mon[36485]: pgmap v2329: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:31.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:30 smithi106 ceph-mon[36505]: pgmap v2329: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:32.022 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4140 sec 2024-03-24T01:33:33.022 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4141 sec 2024-03-24T01:33:33.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:32 smithi078 ceph-mon[32429]: pgmap v2330: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:33.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:32 smithi107 ceph-mon[36485]: pgmap v2330: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:33.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:32 smithi106 ceph-mon[36505]: pgmap v2330: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:34.022 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4142 sec 2024-03-24T01:33:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3488.155712 secs 2024-03-24T01:33:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3488.155712 secs 2024-03-24T01:33:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3488.155712 secs 2024-03-24T01:33:35.022 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4143 sec 2024-03-24T01:33:35.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:34 smithi078 ceph-mon[32429]: pgmap v2331: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:35.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:34 smithi107 ceph-mon[36485]: pgmap v2331: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:35.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:34 smithi106 ceph-mon[36505]: pgmap v2331: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:36.023 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4144 sec 2024-03-24T01:33:37.026 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4145 sec 2024-03-24T01:33:37.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:36 smithi078 ceph-mon[32429]: pgmap v2332: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:37.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:36 smithi107 ceph-mon[36485]: pgmap v2332: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:37.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:36 smithi106 ceph-mon[36505]: pgmap v2332: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:38.023 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4146 sec 2024-03-24T01:33:39.023 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4147 sec 2024-03-24T01:33:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:38 smithi078 ceph-mon[32429]: pgmap v2333: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3493.155727 secs 2024-03-24T01:33:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:38 smithi107 ceph-mon[36485]: pgmap v2333: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3493.155727 secs 2024-03-24T01:33:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:38 smithi106 ceph-mon[36505]: pgmap v2333: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3493.155727 secs 2024-03-24T01:33:40.023 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4148 sec 2024-03-24T01:33:41.023 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4149 sec 2024-03-24T01:33:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:40 smithi078 ceph-mon[32429]: pgmap v2334: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:41.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:40 smithi107 ceph-mon[36485]: pgmap v2334: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:41.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:40 smithi106 ceph-mon[36505]: pgmap v2334: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:42.023 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4150 sec 2024-03-24T01:33:43.023 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4151 sec 2024-03-24T01:33:43.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:42 smithi078 ceph-mon[32429]: pgmap v2335: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:43.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:42 smithi107 ceph-mon[36485]: pgmap v2335: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:43.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:42 smithi106 ceph-mon[36505]: pgmap v2335: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:44.023 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4152 sec 2024-03-24T01:33:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3498.155817 secs 2024-03-24T01:33:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3498.155817 secs 2024-03-24T01:33:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3498.155817 secs 2024-03-24T01:33:45.024 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4153 sec 2024-03-24T01:33:45.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:44 smithi078 ceph-mon[32429]: pgmap v2336: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:45.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:44 smithi107 ceph-mon[36485]: pgmap v2336: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:45.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:44 smithi106 ceph-mon[36505]: pgmap v2336: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:33:46.023 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4154 sec 2024-03-24T01:33:46.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:33:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:33:46.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:33:47.024 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4155 sec 2024-03-24T01:33:47.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:47 smithi106 ceph-mon[36505]: pgmap v2337: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:47.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:47 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:33:47.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:47 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:33:47.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:47 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:33:47.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:47 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:33:47.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:47 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:33:47.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:47 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:33:47.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:47 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:33:47.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:47 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:33:47.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:47 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:33:47.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:47 smithi078 ceph-mon[32429]: pgmap v2337: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:47.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:47 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:33:47.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:47 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:33:47.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:47 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:33:47.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:47 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:33:47.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:47 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:33:47.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:47 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:33:47.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:47 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:33:47.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:47 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:33:47.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:47 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:33:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:47 smithi107 ceph-mon[36485]: pgmap v2337: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:47 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:33:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:47 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:33:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:47 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:33:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:47 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:33:47.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:47 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:33:47.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:47 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:33:47.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:47 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:33:47.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:47 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:33:47.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:47 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:33:48.023 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4156 sec 2024-03-24T01:33:48.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:48 smithi106 ceph-mon[36505]: pgmap v2338: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T01:33:48.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:48 smithi106 ceph-mon[36505]: pgmap v2339: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:33:48.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:48 smithi078 ceph-mon[32429]: pgmap v2338: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T01:33:48.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:48 smithi078 ceph-mon[32429]: pgmap v2339: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:33:48.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:48 smithi107 ceph-mon[36485]: pgmap v2338: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T01:33:48.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:48 smithi107 ceph-mon[36485]: pgmap v2339: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:33:49.024 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4157 sec 2024-03-24T01:33:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3503.155929 secs 2024-03-24T01:33:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3503.155929 secs 2024-03-24T01:33:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3503.155929 secs 2024-03-24T01:33:50.024 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4158 sec 2024-03-24T01:33:50.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:50 smithi078 ceph-mon[32429]: pgmap v2340: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:33:50.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:50 smithi107 ceph-mon[36485]: pgmap v2340: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:33:50.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:50 smithi106 ceph-mon[36505]: pgmap v2340: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:33:51.024 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4159 sec 2024-03-24T01:33:52.024 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4160 sec 2024-03-24T01:33:52.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:52 smithi078 ceph-mon[32429]: pgmap v2341: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:33:52.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:52 smithi107 ceph-mon[36485]: pgmap v2341: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:33:52.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:52 smithi106 ceph-mon[36505]: pgmap v2341: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:33:53.024 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4161 sec 2024-03-24T01:33:54.024 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4162 sec 2024-03-24T01:33:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:54 smithi078 ceph-mon[32429]: pgmap v2342: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:33:54.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3508.155990 secs 2024-03-24T01:33:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:54 smithi107 ceph-mon[36485]: pgmap v2342: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:33:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3508.155990 secs 2024-03-24T01:33:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:54 smithi106 ceph-mon[36505]: pgmap v2342: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:33:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3508.155990 secs 2024-03-24T01:33:55.024 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4163 sec 2024-03-24T01:33:55.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:55 smithi078 ceph-mon[32429]: pgmap v2343: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:33:55.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:55 smithi107 ceph-mon[36485]: pgmap v2343: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:33:55.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:55 smithi106 ceph-mon[36505]: pgmap v2343: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:33:56.024 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4164 sec 2024-03-24T01:33:57.024 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4165 sec 2024-03-24T01:33:58.024 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4166 sec 2024-03-24T01:33:58.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:57 smithi106 ceph-mon[36505]: pgmap v2344: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:33:58.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:57 smithi078 ceph-mon[32429]: pgmap v2344: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:33:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:57 smithi107 ceph-mon[36485]: pgmap v2344: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:33:59.024 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4167 sec 2024-03-24T01:33:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3513.156078 secs 2024-03-24T01:33:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3513.156078 secs 2024-03-24T01:33:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3513.156078 secs 2024-03-24T01:34:00.024 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4168 sec 2024-03-24T01:34:00.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:33:59 smithi106 ceph-mon[36505]: pgmap v2345: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:00.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:33:59 smithi078 ceph-mon[32429]: pgmap v2345: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:00.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:33:59 smithi107 ceph-mon[36485]: pgmap v2345: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:01.025 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4169 sec 2024-03-24T01:34:02.025 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4170 sec 2024-03-24T01:34:02.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:01 smithi106 ceph-mon[36505]: pgmap v2346: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:02.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:01 smithi078 ceph-mon[32429]: pgmap v2346: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:02.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:01 smithi107 ceph-mon[36485]: pgmap v2346: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:03.025 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4171 sec 2024-03-24T01:34:04.025 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4172 sec 2024-03-24T01:34:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:03 smithi106 ceph-mon[36505]: pgmap v2347: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3518.156167 secs 2024-03-24T01:34:04.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:03 smithi078 ceph-mon[32429]: pgmap v2347: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:04.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3518.156167 secs 2024-03-24T01:34:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:03 smithi107 ceph-mon[36485]: pgmap v2347: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3518.156167 secs 2024-03-24T01:34:05.025 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4173 sec 2024-03-24T01:34:06.025 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4174 sec 2024-03-24T01:34:06.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:05 smithi106 ceph-mon[36505]: pgmap v2348: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:06.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:05 smithi078 ceph-mon[32429]: pgmap v2348: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:06.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:05 smithi107 ceph-mon[36485]: pgmap v2348: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:07.025 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4175 sec 2024-03-24T01:34:08.025 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4176 sec 2024-03-24T01:34:08.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:07 smithi106 ceph-mon[36505]: pgmap v2349: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:08.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:07 smithi078 ceph-mon[32429]: pgmap v2349: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:08.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:07 smithi107 ceph-mon[36485]: pgmap v2349: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:09.025 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4177 sec 2024-03-24T01:34:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3523.156257 secs 2024-03-24T01:34:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3523.156257 secs 2024-03-24T01:34:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3523.156257 secs 2024-03-24T01:34:10.025 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4178 sec 2024-03-24T01:34:10.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:09 smithi106 ceph-mon[36505]: pgmap v2350: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:10.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:09 smithi078 ceph-mon[32429]: pgmap v2350: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:10.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:09 smithi107 ceph-mon[36485]: pgmap v2350: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:11.025 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4179 sec 2024-03-24T01:34:12.026 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4180 sec 2024-03-24T01:34:12.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:11 smithi107 ceph-mon[36485]: pgmap v2351: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:12.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:11 smithi106 ceph-mon[36505]: pgmap v2351: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:12.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:11 smithi078 ceph-mon[32429]: pgmap v2351: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:13.026 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4181 sec 2024-03-24T01:34:14.026 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4182 sec 2024-03-24T01:34:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:13 smithi106 ceph-mon[36505]: pgmap v2352: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3528.156298 secs 2024-03-24T01:34:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:13 smithi078 ceph-mon[32429]: pgmap v2352: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3528.156298 secs 2024-03-24T01:34:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:13 smithi107 ceph-mon[36485]: pgmap v2352: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3528.156298 secs 2024-03-24T01:34:15.026 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4183 sec 2024-03-24T01:34:16.026 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4184 sec 2024-03-24T01:34:16.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:15 smithi106 ceph-mon[36505]: pgmap v2353: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:16.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:15 smithi078 ceph-mon[32429]: pgmap v2353: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:16.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:15 smithi107 ceph-mon[36485]: pgmap v2353: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:17.026 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4185 sec 2024-03-24T01:34:18.026 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4186 sec 2024-03-24T01:34:18.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:17 smithi106 ceph-mon[36505]: pgmap v2354: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:17 smithi078 ceph-mon[32429]: pgmap v2354: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:17 smithi107 ceph-mon[36485]: pgmap v2354: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:19.026 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4187 sec 2024-03-24T01:34:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3533.156359 secs 2024-03-24T01:34:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3533.156359 secs 2024-03-24T01:34:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3533.156359 secs 2024-03-24T01:34:20.026 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4188 sec 2024-03-24T01:34:20.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:19 smithi106 ceph-mon[36505]: pgmap v2355: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:20.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:19 smithi078 ceph-mon[32429]: pgmap v2355: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:19 smithi107 ceph-mon[36485]: pgmap v2355: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:21.026 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4189 sec 2024-03-24T01:34:22.026 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4190 sec 2024-03-24T01:34:22.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:21 smithi078 ceph-mon[32429]: pgmap v2356: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:22.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:21 smithi107 ceph-mon[36485]: pgmap v2356: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:22.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:21 smithi106 ceph-mon[36505]: pgmap v2356: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:23.026 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4191 sec 2024-03-24T01:34:24.027 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4192 sec 2024-03-24T01:34:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:23 smithi078 ceph-mon[32429]: pgmap v2357: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3538.156419 secs 2024-03-24T01:34:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:23 smithi107 ceph-mon[36485]: pgmap v2357: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3538.156419 secs 2024-03-24T01:34:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:23 smithi106 ceph-mon[36505]: pgmap v2357: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3538.156419 secs 2024-03-24T01:34:25.027 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4193 sec 2024-03-24T01:34:26.027 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4194 sec 2024-03-24T01:34:26.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:25 smithi078 ceph-mon[32429]: pgmap v2358: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:26.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:25 smithi107 ceph-mon[36485]: pgmap v2358: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:26.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:25 smithi106 ceph-mon[36505]: pgmap v2358: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:27.027 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4195 sec 2024-03-24T01:34:28.027 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4196 sec 2024-03-24T01:34:28.204 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:27 smithi106 ceph-mon[36505]: pgmap v2359: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:28.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:27 smithi078 ceph-mon[32429]: pgmap v2359: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:28.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:27 smithi107 ceph-mon[36485]: pgmap v2359: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:29.027 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4197 sec 2024-03-24T01:34:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3543.156479 secs 2024-03-24T01:34:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3543.156479 secs 2024-03-24T01:34:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3543.156479 secs 2024-03-24T01:34:30.028 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4198 sec 2024-03-24T01:34:30.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:29 smithi078 ceph-mon[32429]: pgmap v2360: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:30.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:29 smithi107 ceph-mon[36485]: pgmap v2360: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:30.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:29 smithi106 ceph-mon[36505]: pgmap v2360: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:31.027 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4199 sec 2024-03-24T01:34:32.027 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4200 sec 2024-03-24T01:34:32.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:31 smithi078 ceph-mon[32429]: pgmap v2361: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:32.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:31 smithi107 ceph-mon[36485]: pgmap v2361: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:32.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:31 smithi106 ceph-mon[36505]: pgmap v2361: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:33.027 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4201 sec 2024-03-24T01:34:34.027 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4202 sec 2024-03-24T01:34:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:33 smithi078 ceph-mon[32429]: pgmap v2362: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:34.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3548.156539 secs 2024-03-24T01:34:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:33 smithi107 ceph-mon[36485]: pgmap v2362: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3548.156539 secs 2024-03-24T01:34:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:33 smithi106 ceph-mon[36505]: pgmap v2362: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:34.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3548.156539 secs 2024-03-24T01:34:35.027 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4203 sec 2024-03-24T01:34:36.027 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4204 sec 2024-03-24T01:34:36.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:35 smithi078 ceph-mon[32429]: pgmap v2363: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:36.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:35 smithi107 ceph-mon[36485]: pgmap v2363: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:36.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:35 smithi106 ceph-mon[36505]: pgmap v2363: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:37.028 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4205 sec 2024-03-24T01:34:38.028 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4206 sec 2024-03-24T01:34:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:37 smithi078 ceph-mon[32429]: pgmap v2364: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:38.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:37 smithi107 ceph-mon[36485]: pgmap v2364: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:38.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:37 smithi106 ceph-mon[36505]: pgmap v2364: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:39.028 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4207 sec 2024-03-24T01:34:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3553.156603 secs 2024-03-24T01:34:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3553.156603 secs 2024-03-24T01:34:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3553.156603 secs 2024-03-24T01:34:40.028 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4208 sec 2024-03-24T01:34:40.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:39 smithi078 ceph-mon[32429]: pgmap v2365: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:39 smithi107 ceph-mon[36485]: pgmap v2365: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:40.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:39 smithi106 ceph-mon[36505]: pgmap v2365: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:41.028 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4209 sec 2024-03-24T01:34:42.028 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4210 sec 2024-03-24T01:34:42.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:41 smithi078 ceph-mon[32429]: pgmap v2366: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:42.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:41 smithi107 ceph-mon[36485]: pgmap v2366: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:42.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:41 smithi106 ceph-mon[36505]: pgmap v2366: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:43.028 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4211 sec 2024-03-24T01:34:44.028 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4212 sec 2024-03-24T01:34:44.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:43 smithi078 ceph-mon[32429]: pgmap v2367: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:44.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3558.156685 secs 2024-03-24T01:34:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:43 smithi107 ceph-mon[36485]: pgmap v2367: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3558.156685 secs 2024-03-24T01:34:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:43 smithi106 ceph-mon[36505]: pgmap v2367: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3558.156685 secs 2024-03-24T01:34:45.028 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4213 sec 2024-03-24T01:34:46.028 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4214 sec 2024-03-24T01:34:46.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:45 smithi078 ceph-mon[32429]: pgmap v2368: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:46.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:45 smithi107 ceph-mon[36485]: pgmap v2368: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:46.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:45 smithi106 ceph-mon[36505]: pgmap v2368: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:34:47.028 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4215 sec 2024-03-24T01:34:47.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:46 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:34:47.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:46 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:34:47.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:46 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:34:48.028 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4216 sec 2024-03-24T01:34:48.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:47 smithi078 ceph-mon[32429]: pgmap v2369: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:48.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:47 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:34:48.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:47 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:34:48.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:47 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:34:48.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:47 smithi107 ceph-mon[36485]: pgmap v2369: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:48.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:47 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:34:48.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:47 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:34:48.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:47 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:34:48.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:47 smithi106 ceph-mon[36505]: pgmap v2369: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:48.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:47 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:34:48.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:47 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:34:48.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:47 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:34:49.029 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4217 sec 2024-03-24T01:34:49.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:48 smithi078 ceph-mon[32429]: pgmap v2370: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:34:49.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:48 smithi078 ceph-mon[32429]: pgmap v2371: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:34:49.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3563.156729 secs 2024-03-24T01:34:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:48 smithi107 ceph-mon[36485]: pgmap v2370: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:34:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:48 smithi107 ceph-mon[36485]: pgmap v2371: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:34:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3563.156729 secs 2024-03-24T01:34:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:48 smithi106 ceph-mon[36505]: pgmap v2370: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:34:49.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:48 smithi106 ceph-mon[36505]: pgmap v2371: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:34:49.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3563.156729 secs 2024-03-24T01:34:50.029 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4218 sec 2024-03-24T01:34:51.029 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4219 sec 2024-03-24T01:34:51.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:51 smithi078 ceph-mon[32429]: pgmap v2372: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:34:51.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:51 smithi107 ceph-mon[36485]: pgmap v2372: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:34:51.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:51 smithi106 ceph-mon[36505]: pgmap v2372: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:34:52.029 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4220 sec 2024-03-24T01:34:53.029 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4221 sec 2024-03-24T01:34:53.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:53 smithi078 ceph-mon[32429]: pgmap v2373: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:34:53.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:53 smithi107 ceph-mon[36485]: pgmap v2373: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:34:53.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:53 smithi106 ceph-mon[36505]: pgmap v2373: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:34:54.029 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4222 sec 2024-03-24T01:34:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3568.156788 secs 2024-03-24T01:34:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3568.156788 secs 2024-03-24T01:34:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3568.156788 secs 2024-03-24T01:34:55.029 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4223 sec 2024-03-24T01:34:55.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:55 smithi078 ceph-mon[32429]: pgmap v2374: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:34:55.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:55 smithi107 ceph-mon[36485]: pgmap v2374: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:34:55.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:55 smithi106 ceph-mon[36505]: pgmap v2374: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:34:56.029 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4224 sec 2024-03-24T01:34:57.029 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4225 sec 2024-03-24T01:34:57.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:57 smithi078 ceph-mon[32429]: pgmap v2375: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:34:57.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:57 smithi107 ceph-mon[36485]: pgmap v2375: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:34:57.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:57 smithi106 ceph-mon[36505]: pgmap v2375: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:34:58.029 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4226 sec 2024-03-24T01:34:59.030 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4227 sec 2024-03-24T01:34:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:59 smithi107 ceph-mon[36485]: pgmap v2376: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:34:59.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:34:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3573.156846 secs 2024-03-24T01:34:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:59 smithi106 ceph-mon[36505]: pgmap v2376: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:34:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:34:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3573.156846 secs 2024-03-24T01:34:59.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:59 smithi078 ceph-mon[32429]: pgmap v2376: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:34:59.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:34:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3573.156846 secs 2024-03-24T01:35:00.030 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4228 sec 2024-03-24T01:35:01.030 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4229 sec 2024-03-24T01:35:01.393 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:01 smithi078 ceph-mon[32429]: pgmap v2377: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:01 smithi106 ceph-mon[36505]: pgmap v2377: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:01.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:01 smithi107 ceph-mon[36485]: pgmap v2377: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:02.030 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4230 sec 2024-03-24T01:35:03.030 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4231 sec 2024-03-24T01:35:03.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:03 smithi106 ceph-mon[36505]: pgmap v2378: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:03.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:03 smithi078 ceph-mon[32429]: pgmap v2378: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:03.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:03 smithi107 ceph-mon[36485]: pgmap v2378: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:04.030 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4232 sec 2024-03-24T01:35:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3578.156931 secs 2024-03-24T01:35:04.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3578.156931 secs 2024-03-24T01:35:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3578.156931 secs 2024-03-24T01:35:05.030 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4233 sec 2024-03-24T01:35:05.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:05 smithi106 ceph-mon[36505]: pgmap v2379: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:05.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:05 smithi078 ceph-mon[32429]: pgmap v2379: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:05.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:05 smithi107 ceph-mon[36485]: pgmap v2379: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:06.030 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4234 sec 2024-03-24T01:35:07.030 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4235 sec 2024-03-24T01:35:07.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:07 smithi106 ceph-mon[36505]: pgmap v2380: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:07.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:07 smithi078 ceph-mon[32429]: pgmap v2380: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:07.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:07 smithi107 ceph-mon[36485]: pgmap v2380: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:08.030 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4236 sec 2024-03-24T01:35:09.030 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4237 sec 2024-03-24T01:35:09.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:09 smithi078 ceph-mon[32429]: pgmap v2381: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:09.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3583.157018 secs 2024-03-24T01:35:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:09 smithi106 ceph-mon[36505]: pgmap v2381: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3583.157018 secs 2024-03-24T01:35:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:09 smithi107 ceph-mon[36485]: pgmap v2381: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:09.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3583.157018 secs 2024-03-24T01:35:10.030 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4238 sec 2024-03-24T01:35:11.030 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4239 sec 2024-03-24T01:35:11.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:11 smithi106 ceph-mon[36505]: pgmap v2382: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:11.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:11 smithi078 ceph-mon[32429]: pgmap v2382: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:11.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:11 smithi107 ceph-mon[36485]: pgmap v2382: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:12.031 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4240 sec 2024-03-24T01:35:13.031 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4241 sec 2024-03-24T01:35:13.396 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:13 smithi107 ceph-mon[36485]: pgmap v2383: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:13.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:13 smithi106 ceph-mon[36505]: pgmap v2383: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:13.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:13 smithi078 ceph-mon[32429]: pgmap v2383: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:14.031 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4242 sec 2024-03-24T01:35:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3588.157075 secs 2024-03-24T01:35:14.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3588.157075 secs 2024-03-24T01:35:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3588.157075 secs 2024-03-24T01:35:15.031 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4243 sec 2024-03-24T01:35:15.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:15 smithi106 ceph-mon[36505]: pgmap v2384: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:15.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:15 smithi078 ceph-mon[32429]: pgmap v2384: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:15.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:15 smithi107 ceph-mon[36485]: pgmap v2384: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:16.031 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4244 sec 2024-03-24T01:35:17.031 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4245 sec 2024-03-24T01:35:17.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:17 smithi106 ceph-mon[36505]: pgmap v2385: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:17.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:17 smithi078 ceph-mon[32429]: pgmap v2385: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:17.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:17 smithi107 ceph-mon[36485]: pgmap v2385: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:18.031 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4246 sec 2024-03-24T01:35:19.031 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4247 sec 2024-03-24T01:35:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:19 smithi106 ceph-mon[36505]: pgmap v2386: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3593.157163 secs 2024-03-24T01:35:19.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:19 smithi078 ceph-mon[32429]: pgmap v2386: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:19.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3593.157163 secs 2024-03-24T01:35:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:19 smithi107 ceph-mon[36485]: pgmap v2386: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3593.157163 secs 2024-03-24T01:35:20.031 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4248 sec 2024-03-24T01:35:21.031 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4249 sec 2024-03-24T01:35:21.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:21 smithi106 ceph-mon[36505]: pgmap v2387: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:21.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:21 smithi078 ceph-mon[32429]: pgmap v2387: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:21.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:21 smithi107 ceph-mon[36485]: pgmap v2387: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:22.031 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4250 sec 2024-03-24T01:35:23.031 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4251 sec 2024-03-24T01:35:23.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:23 smithi106 ceph-mon[36505]: pgmap v2388: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:23.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:23 smithi078 ceph-mon[32429]: pgmap v2388: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:23.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:23 smithi107 ceph-mon[36485]: pgmap v2388: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:24.031 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4252 sec 2024-03-24T01:35:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3598.157221 secs 2024-03-24T01:35:24.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3598.157221 secs 2024-03-24T01:35:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3598.157221 secs 2024-03-24T01:35:25.034 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4253 sec 2024-03-24T01:35:25.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:25 smithi106 ceph-mon[36505]: pgmap v2389: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:25.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:25 smithi078 ceph-mon[32429]: pgmap v2389: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:25.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:25 smithi107 ceph-mon[36485]: pgmap v2389: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:26.032 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4254 sec 2024-03-24T01:35:27.032 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4255 sec 2024-03-24T01:35:27.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:27 smithi106 ceph-mon[36505]: pgmap v2390: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:27.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:27 smithi078 ceph-mon[32429]: pgmap v2390: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:27.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:27 smithi107 ceph-mon[36485]: pgmap v2390: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:28.032 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4256 sec 2024-03-24T01:35:29.032 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4257 sec 2024-03-24T01:35:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:29 smithi106 ceph-mon[36505]: pgmap v2391: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:29.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3603.157286 secs 2024-03-24T01:35:29.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:29 smithi078 ceph-mon[32429]: pgmap v2391: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:29.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3603.157286 secs 2024-03-24T01:35:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:29 smithi107 ceph-mon[36485]: pgmap v2391: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3603.157286 secs 2024-03-24T01:35:30.032 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4258 sec 2024-03-24T01:35:31.032 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4259 sec 2024-03-24T01:35:31.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:31 smithi078 ceph-mon[32429]: pgmap v2392: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:31.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:31 smithi107 ceph-mon[36485]: pgmap v2392: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:31.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:31 smithi106 ceph-mon[36505]: pgmap v2392: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:32.032 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4260 sec 2024-03-24T01:35:33.032 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4261 sec 2024-03-24T01:35:33.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:33 smithi078 ceph-mon[32429]: pgmap v2393: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:33.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:33 smithi107 ceph-mon[36485]: pgmap v2393: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:33.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:33 smithi106 ceph-mon[36505]: pgmap v2393: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:34.032 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4262 sec 2024-03-24T01:35:34.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3608.157295 secs 2024-03-24T01:35:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3608.157295 secs 2024-03-24T01:35:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3608.157295 secs 2024-03-24T01:35:35.033 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4263 sec 2024-03-24T01:35:35.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:35 smithi078 ceph-mon[32429]: pgmap v2394: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:35.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:35 smithi107 ceph-mon[36485]: pgmap v2394: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:35.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:35 smithi106 ceph-mon[36505]: pgmap v2394: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:36.033 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4264 sec 2024-03-24T01:35:37.033 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4265 sec 2024-03-24T01:35:37.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:37 smithi078 ceph-mon[32429]: pgmap v2395: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:37.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:37 smithi107 ceph-mon[36485]: pgmap v2395: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:37.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:37 smithi106 ceph-mon[36505]: pgmap v2395: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:38.033 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4266 sec 2024-03-24T01:35:39.033 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4267 sec 2024-03-24T01:35:39.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:39 smithi078 ceph-mon[32429]: pgmap v2396: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:39.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3613.157355 secs 2024-03-24T01:35:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:39 smithi107 ceph-mon[36485]: pgmap v2396: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:39.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3613.157355 secs 2024-03-24T01:35:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:39 smithi106 ceph-mon[36505]: pgmap v2396: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:39.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3613.157355 secs 2024-03-24T01:35:40.033 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4268 sec 2024-03-24T01:35:41.033 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4269 sec 2024-03-24T01:35:41.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:41 smithi078 ceph-mon[32429]: pgmap v2397: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:41.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:41 smithi107 ceph-mon[36485]: pgmap v2397: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:41.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:41 smithi106 ceph-mon[36505]: pgmap v2397: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:42.033 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4270 sec 2024-03-24T01:35:43.033 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4271 sec 2024-03-24T01:35:43.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:43 smithi078 ceph-mon[32429]: pgmap v2398: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:43.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:43 smithi107 ceph-mon[36485]: pgmap v2398: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:43.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:43 smithi106 ceph-mon[36505]: pgmap v2398: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:44.033 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4272 sec 2024-03-24T01:35:44.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3618.157442 secs 2024-03-24T01:35:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3618.157442 secs 2024-03-24T01:35:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3618.157442 secs 2024-03-24T01:35:45.033 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4273 sec 2024-03-24T01:35:45.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:45 smithi078 ceph-mon[32429]: pgmap v2399: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:45.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:45 smithi107 ceph-mon[36485]: pgmap v2399: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:45.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:45 smithi106 ceph-mon[36505]: pgmap v2399: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:46.033 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4274 sec 2024-03-24T01:35:46.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:46 smithi078 ceph-mon[32429]: pgmap v2400: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:46.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:46 smithi107 ceph-mon[36485]: pgmap v2400: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:46 smithi106 ceph-mon[36505]: pgmap v2400: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:47.033 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4275 sec 2024-03-24T01:35:48.033 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4276 sec 2024-03-24T01:35:49.033 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4277 sec 2024-03-24T01:35:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:48 smithi078 ceph-mon[32429]: pgmap v2401: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:49.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:48 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:35:49.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3623.157510 secs 2024-03-24T01:35:49.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:48 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:35:49.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:48 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:35:49.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:48 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:35:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:48 smithi107 ceph-mon[36485]: pgmap v2401: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:48 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:35:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3623.157510 secs 2024-03-24T01:35:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:48 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:35:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:48 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:35:49.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:48 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:35:49.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:48 smithi106 ceph-mon[36505]: pgmap v2401: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:35:49.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:48 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:35:49.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3623.157510 secs 2024-03-24T01:35:49.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:48 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:35:49.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:48 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:35:49.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:48 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:35:50.034 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4278 sec 2024-03-24T01:35:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:49 smithi078 ceph-mon[32429]: pgmap v2402: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:35:50.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:49 smithi078 ceph-mon[32429]: pgmap v2403: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:35:50.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:49 smithi107 ceph-mon[36485]: pgmap v2402: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:35:50.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:49 smithi107 ceph-mon[36485]: pgmap v2403: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:35:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:49 smithi106 ceph-mon[36505]: pgmap v2402: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:35:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:49 smithi106 ceph-mon[36505]: pgmap v2403: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:35:51.034 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4279 sec 2024-03-24T01:35:52.034 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4280 sec 2024-03-24T01:35:52.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:51 smithi078 ceph-mon[32429]: pgmap v2404: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:35:52.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:51 smithi107 ceph-mon[36485]: pgmap v2404: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:35:52.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:51 smithi106 ceph-mon[36505]: pgmap v2404: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:35:53.034 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4281 sec 2024-03-24T01:35:54.034 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4282 sec 2024-03-24T01:35:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:53 smithi078 ceph-mon[32429]: pgmap v2405: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:35:54.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3628.157596 secs 2024-03-24T01:35:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:53 smithi107 ceph-mon[36485]: pgmap v2405: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:35:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3628.157596 secs 2024-03-24T01:35:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:53 smithi106 ceph-mon[36505]: pgmap v2405: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:35:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3628.157596 secs 2024-03-24T01:35:55.034 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4283 sec 2024-03-24T01:35:55.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:55 smithi078 ceph-mon[32429]: pgmap v2406: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:35:56.034 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4284 sec 2024-03-24T01:35:56.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:55 smithi107 ceph-mon[36485]: pgmap v2406: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:35:56.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:55 smithi106 ceph-mon[36505]: pgmap v2406: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:35:57.034 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4285 sec 2024-03-24T01:35:58.034 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4286 sec 2024-03-24T01:35:58.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:57 smithi078 ceph-mon[32429]: pgmap v2407: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:35:58.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:57 smithi107 ceph-mon[36485]: pgmap v2407: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:35:58.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:57 smithi106 ceph-mon[36505]: pgmap v2407: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:35:59.034 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4287 sec 2024-03-24T01:35:59.054 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3633.157656 secs 2024-03-24T01:35:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3633.157656 secs 2024-03-24T01:35:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3633.157656 secs 2024-03-24T01:36:00.034 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4288 sec 2024-03-24T01:36:00.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:35:59 smithi078 ceph-mon[32429]: pgmap v2408: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:36:00.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:35:59 smithi107 ceph-mon[36485]: pgmap v2408: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:36:00.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:35:59 smithi106 ceph-mon[36505]: pgmap v2408: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:36:01.034 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4289 sec 2024-03-24T01:36:02.034 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4290 sec 2024-03-24T01:36:02.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:01 smithi078 ceph-mon[32429]: pgmap v2409: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:02.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:01 smithi107 ceph-mon[36485]: pgmap v2409: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:02.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:01 smithi106 ceph-mon[36505]: pgmap v2409: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:03.035 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4291 sec 2024-03-24T01:36:04.035 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4292 sec 2024-03-24T01:36:04.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:03 smithi078 ceph-mon[32429]: pgmap v2410: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:04.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3638.157716 secs 2024-03-24T01:36:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:03 smithi107 ceph-mon[36485]: pgmap v2410: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3638.157716 secs 2024-03-24T01:36:04.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:03 smithi106 ceph-mon[36505]: pgmap v2410: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:04.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3638.157716 secs 2024-03-24T01:36:05.035 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4293 sec 2024-03-24T01:36:06.035 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4294 sec 2024-03-24T01:36:06.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:05 smithi078 ceph-mon[32429]: pgmap v2411: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:05 smithi107 ceph-mon[36485]: pgmap v2411: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:06.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:05 smithi106 ceph-mon[36505]: pgmap v2411: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:07.035 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4295 sec 2024-03-24T01:36:08.035 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4296 sec 2024-03-24T01:36:08.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:07 smithi078 ceph-mon[32429]: pgmap v2412: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:08.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:07 smithi107 ceph-mon[36485]: pgmap v2412: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:08.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:07 smithi106 ceph-mon[36505]: pgmap v2412: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:09.035 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4297 sec 2024-03-24T01:36:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3643.157777 secs 2024-03-24T01:36:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3643.157777 secs 2024-03-24T01:36:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3643.157777 secs 2024-03-24T01:36:10.035 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4298 sec 2024-03-24T01:36:10.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:09 smithi078 ceph-mon[32429]: pgmap v2413: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:10.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:09 smithi107 ceph-mon[36485]: pgmap v2413: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:10.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:09 smithi106 ceph-mon[36505]: pgmap v2413: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:11.035 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4299 sec 2024-03-24T01:36:12.035 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4300 sec 2024-03-24T01:36:12.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:11 smithi078 ceph-mon[32429]: pgmap v2414: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:12.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:11 smithi107 ceph-mon[36485]: pgmap v2414: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:12.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:11 smithi106 ceph-mon[36505]: pgmap v2414: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:13.036 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4301 sec 2024-03-24T01:36:14.036 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4302 sec 2024-03-24T01:36:14.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:13 smithi078 ceph-mon[32429]: pgmap v2415: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:14.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3648.157834 secs 2024-03-24T01:36:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:13 smithi107 ceph-mon[36485]: pgmap v2415: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3648.157834 secs 2024-03-24T01:36:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:13 smithi106 ceph-mon[36505]: pgmap v2415: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:14.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3648.157834 secs 2024-03-24T01:36:15.036 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4303 sec 2024-03-24T01:36:16.036 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4304 sec 2024-03-24T01:36:16.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:15 smithi078 ceph-mon[32429]: pgmap v2416: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:16.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:15 smithi107 ceph-mon[36485]: pgmap v2416: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:16.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:15 smithi106 ceph-mon[36505]: pgmap v2416: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:17.036 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4305 sec 2024-03-24T01:36:18.036 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4306 sec 2024-03-24T01:36:18.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:17 smithi078 ceph-mon[32429]: pgmap v2417: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:18.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:17 smithi107 ceph-mon[36485]: pgmap v2417: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:18.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:17 smithi106 ceph-mon[36505]: pgmap v2417: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:19.036 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4307 sec 2024-03-24T01:36:19.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3653.157919 secs 2024-03-24T01:36:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3653.157919 secs 2024-03-24T01:36:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3653.157919 secs 2024-03-24T01:36:20.036 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4308 sec 2024-03-24T01:36:20.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:19 smithi078 ceph-mon[32429]: pgmap v2418: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:20.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:19 smithi107 ceph-mon[36485]: pgmap v2418: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:20.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:19 smithi106 ceph-mon[36505]: pgmap v2418: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:21.036 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4309 sec 2024-03-24T01:36:22.036 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4310 sec 2024-03-24T01:36:22.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:21 smithi078 ceph-mon[32429]: pgmap v2419: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:22.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:21 smithi107 ceph-mon[36485]: pgmap v2419: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:22.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:21 smithi106 ceph-mon[36505]: pgmap v2419: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:23.036 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4311 sec 2024-03-24T01:36:24.037 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4312 sec 2024-03-24T01:36:24.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:23 smithi078 ceph-mon[32429]: pgmap v2420: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:24.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3658.157958 secs 2024-03-24T01:36:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:23 smithi107 ceph-mon[36485]: pgmap v2420: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3658.157958 secs 2024-03-24T01:36:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:23 smithi106 ceph-mon[36505]: pgmap v2420: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:24.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3658.157958 secs 2024-03-24T01:36:25.037 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4313 sec 2024-03-24T01:36:26.037 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4314 sec 2024-03-24T01:36:26.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:25 smithi078 ceph-mon[32429]: pgmap v2421: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:26.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:25 smithi107 ceph-mon[36485]: pgmap v2421: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:26.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:25 smithi106 ceph-mon[36505]: pgmap v2421: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:27.037 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4315 sec 2024-03-24T01:36:28.037 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4316 sec 2024-03-24T01:36:28.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:27 smithi078 ceph-mon[32429]: pgmap v2422: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:28.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:27 smithi107 ceph-mon[36485]: pgmap v2422: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:28.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:27 smithi106 ceph-mon[36505]: pgmap v2422: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:29.037 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4317 sec 2024-03-24T01:36:29.053 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3663.158017 secs 2024-03-24T01:36:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3663.158017 secs 2024-03-24T01:36:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3663.158017 secs 2024-03-24T01:36:30.037 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4318 sec 2024-03-24T01:36:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:29 smithi078 ceph-mon[32429]: pgmap v2423: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:30.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:29 smithi107 ceph-mon[36485]: pgmap v2423: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:30.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:29 smithi106 ceph-mon[36505]: pgmap v2423: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:31.037 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4319 sec 2024-03-24T01:36:32.037 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4320 sec 2024-03-24T01:36:32.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:31 smithi078 ceph-mon[32429]: pgmap v2424: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:32.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:31 smithi107 ceph-mon[36485]: pgmap v2424: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:32.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:31 smithi106 ceph-mon[36505]: pgmap v2424: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:33.037 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4321 sec 2024-03-24T01:36:34.037 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4322 sec 2024-03-24T01:36:34.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:33 smithi078 ceph-mon[32429]: pgmap v2425: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:34.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3668.158076 secs 2024-03-24T01:36:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:33 smithi107 ceph-mon[36485]: pgmap v2425: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3668.158076 secs 2024-03-24T01:36:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:33 smithi106 ceph-mon[36505]: pgmap v2425: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:34.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3668.158076 secs 2024-03-24T01:36:35.037 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4323 sec 2024-03-24T01:36:36.038 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4324 sec 2024-03-24T01:36:36.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:35 smithi078 ceph-mon[32429]: pgmap v2426: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:36.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:35 smithi107 ceph-mon[36485]: pgmap v2426: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:36.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:35 smithi106 ceph-mon[36505]: pgmap v2426: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:37.038 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4325 sec 2024-03-24T01:36:38.038 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4326 sec 2024-03-24T01:36:38.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:37 smithi078 ceph-mon[32429]: pgmap v2427: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:38.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:37 smithi107 ceph-mon[36485]: pgmap v2427: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:38.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:37 smithi106 ceph-mon[36505]: pgmap v2427: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:39.038 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4327 sec 2024-03-24T01:36:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3673.158163 secs 2024-03-24T01:36:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3673.158163 secs 2024-03-24T01:36:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3673.158163 secs 2024-03-24T01:36:40.038 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4328 sec 2024-03-24T01:36:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:39 smithi078 ceph-mon[32429]: pgmap v2428: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:39 smithi107 ceph-mon[36485]: pgmap v2428: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:39 smithi106 ceph-mon[36505]: pgmap v2428: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:41.038 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4329 sec 2024-03-24T01:36:42.038 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4330 sec 2024-03-24T01:36:42.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:41 smithi078 ceph-mon[32429]: pgmap v2429: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:42.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:41 smithi107 ceph-mon[36485]: pgmap v2429: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:42.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:41 smithi106 ceph-mon[36505]: pgmap v2429: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:43.038 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4331 sec 2024-03-24T01:36:44.038 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4332 sec 2024-03-24T01:36:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:43 smithi078 ceph-mon[32429]: pgmap v2430: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:44.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3678.158251 secs 2024-03-24T01:36:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:43 smithi107 ceph-mon[36485]: pgmap v2430: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3678.158251 secs 2024-03-24T01:36:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:43 smithi106 ceph-mon[36505]: pgmap v2430: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:44.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3678.158251 secs 2024-03-24T01:36:45.038 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4333 sec 2024-03-24T01:36:46.039 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4334 sec 2024-03-24T01:36:46.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:45 smithi078 ceph-mon[32429]: pgmap v2431: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:45 smithi107 ceph-mon[36485]: pgmap v2431: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:46.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:45 smithi106 ceph-mon[36505]: pgmap v2431: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:47.039 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4335 sec 2024-03-24T01:36:48.039 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4336 sec 2024-03-24T01:36:48.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:47 smithi078 ceph-mon[32429]: pgmap v2432: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:48.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:47 smithi107 ceph-mon[36485]: pgmap v2432: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:48.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:47 smithi106 ceph-mon[36505]: pgmap v2432: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:49.039 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4337 sec 2024-03-24T01:36:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3683.158321 secs 2024-03-24T01:36:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:48 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:36:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3683.158321 secs 2024-03-24T01:36:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:48 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:36:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3683.158321 secs 2024-03-24T01:36:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:48 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:36:50.039 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4338 sec 2024-03-24T01:36:50.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:49 smithi078 ceph-mon[32429]: pgmap v2433: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:50.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:49 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:36:50.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:49 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:36:50.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:49 smithi078 ceph-mon[32429]: pgmap v2434: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:36:50.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:49 smithi078 ceph-mon[32429]: pgmap v2435: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:36:50.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:49 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:36:50.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:49 smithi107 ceph-mon[36485]: pgmap v2433: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:50.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:49 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:36:50.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:49 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:36:50.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:49 smithi107 ceph-mon[36485]: pgmap v2434: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:36:50.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:49 smithi107 ceph-mon[36485]: pgmap v2435: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:36:50.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:49 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:36:50.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:49 smithi106 ceph-mon[36505]: pgmap v2433: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:36:50.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:49 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:36:50.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:49 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:36:50.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:49 smithi106 ceph-mon[36505]: pgmap v2434: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:36:50.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:49 smithi106 ceph-mon[36505]: pgmap v2435: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:36:50.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:49 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:36:51.039 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4339 sec 2024-03-24T01:36:51.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:51 smithi078 ceph-mon[32429]: pgmap v2436: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:36:51.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:51 smithi107 ceph-mon[36485]: pgmap v2436: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:36:51.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:51 smithi106 ceph-mon[36505]: pgmap v2436: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:36:52.039 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4340 sec 2024-03-24T01:36:53.039 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4341 sec 2024-03-24T01:36:53.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:53 smithi078 ceph-mon[32429]: pgmap v2437: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:36:53.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:53 smithi107 ceph-mon[36485]: pgmap v2437: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:36:53.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:53 smithi106 ceph-mon[36505]: pgmap v2437: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:36:54.039 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4342 sec 2024-03-24T01:36:54.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3688.158382 secs 2024-03-24T01:36:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3688.158382 secs 2024-03-24T01:36:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3688.158382 secs 2024-03-24T01:36:55.039 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4343 sec 2024-03-24T01:36:55.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:55 smithi078 ceph-mon[32429]: pgmap v2438: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:36:55.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:55 smithi107 ceph-mon[36485]: pgmap v2438: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:36:55.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:55 smithi106 ceph-mon[36505]: pgmap v2438: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:36:56.039 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4344 sec 2024-03-24T01:36:57.039 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4345 sec 2024-03-24T01:36:57.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:57 smithi078 ceph-mon[32429]: pgmap v2439: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:36:57.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:57 smithi107 ceph-mon[36485]: pgmap v2439: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:36:57.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:57 smithi106 ceph-mon[36505]: pgmap v2439: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:36:58.039 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4346 sec 2024-03-24T01:36:58.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3693.158439 secs 2024-03-24T01:36:58.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3693.158439 secs 2024-03-24T01:36:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3693.158439 secs 2024-03-24T01:36:59.040 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4347 sec 2024-03-24T01:36:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:36:59 smithi106 ceph-mon[36505]: pgmap v2440: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:36:59.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:36:59 smithi078 ceph-mon[32429]: pgmap v2440: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:36:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:36:59 smithi107 ceph-mon[36485]: pgmap v2440: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:37:00.040 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4348 sec 2024-03-24T01:37:01.040 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4349 sec 2024-03-24T01:37:01.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:01 smithi078 ceph-mon[32429]: pgmap v2441: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:01.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:01 smithi107 ceph-mon[36485]: pgmap v2441: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:01.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:01 smithi106 ceph-mon[36505]: pgmap v2441: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:02.040 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4350 sec 2024-03-24T01:37:03.040 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4351 sec 2024-03-24T01:37:03.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:03 smithi078 ceph-mon[32429]: pgmap v2442: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:03.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:03 smithi107 ceph-mon[36485]: pgmap v2442: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:03.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:03 smithi106 ceph-mon[36505]: pgmap v2442: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:04.040 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4352 sec 2024-03-24T01:37:04.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3698.158523 secs 2024-03-24T01:37:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3698.158523 secs 2024-03-24T01:37:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3698.158523 secs 2024-03-24T01:37:05.040 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4353 sec 2024-03-24T01:37:05.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:05 smithi078 ceph-mon[32429]: pgmap v2443: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:05.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:05 smithi107 ceph-mon[36485]: pgmap v2443: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:05.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:05 smithi106 ceph-mon[36505]: pgmap v2443: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:06.040 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4354 sec 2024-03-24T01:37:07.040 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4355 sec 2024-03-24T01:37:07.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:07 smithi078 ceph-mon[32429]: pgmap v2444: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:07.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:07 smithi107 ceph-mon[36485]: pgmap v2444: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:07.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:07 smithi106 ceph-mon[36505]: pgmap v2444: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:08.040 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4356 sec 2024-03-24T01:37:08.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3703.158583 secs 2024-03-24T01:37:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3703.158583 secs 2024-03-24T01:37:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3703.158583 secs 2024-03-24T01:37:09.040 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4357 sec 2024-03-24T01:37:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:09 smithi106 ceph-mon[36505]: pgmap v2445: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:09.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:09 smithi078 ceph-mon[32429]: pgmap v2445: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:09.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:09 smithi107 ceph-mon[36485]: pgmap v2445: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:10.040 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4358 sec 2024-03-24T01:37:11.041 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4359 sec 2024-03-24T01:37:11.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:11 smithi078 ceph-mon[32429]: pgmap v2446: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:11.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:11 smithi107 ceph-mon[36485]: pgmap v2446: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:11.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:11 smithi106 ceph-mon[36505]: pgmap v2446: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:12.041 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4360 sec 2024-03-24T01:37:13.041 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4361 sec 2024-03-24T01:37:13.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:13 smithi078 ceph-mon[32429]: pgmap v2447: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:13.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:13 smithi107 ceph-mon[36485]: pgmap v2447: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:13.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:13 smithi106 ceph-mon[36505]: pgmap v2447: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:14.041 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4362 sec 2024-03-24T01:37:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3708.158627 secs 2024-03-24T01:37:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3708.158627 secs 2024-03-24T01:37:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3708.158627 secs 2024-03-24T01:37:15.041 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4363 sec 2024-03-24T01:37:15.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:15 smithi078 ceph-mon[32429]: pgmap v2448: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:15.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:15 smithi107 ceph-mon[36485]: pgmap v2448: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:15.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:15 smithi106 ceph-mon[36505]: pgmap v2448: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:16.041 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4364 sec 2024-03-24T01:37:17.041 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4365 sec 2024-03-24T01:37:17.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:17 smithi078 ceph-mon[32429]: pgmap v2449: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:17.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:17 smithi107 ceph-mon[36485]: pgmap v2449: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:17.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:17 smithi106 ceph-mon[36505]: pgmap v2449: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:18.041 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4366 sec 2024-03-24T01:37:18.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3713.158690 secs 2024-03-24T01:37:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3713.158690 secs 2024-03-24T01:37:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3713.158690 secs 2024-03-24T01:37:19.041 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4367 sec 2024-03-24T01:37:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:19 smithi106 ceph-mon[36505]: pgmap v2450: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:19.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:19 smithi078 ceph-mon[32429]: pgmap v2450: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:19 smithi107 ceph-mon[36485]: pgmap v2450: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:20.042 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4368 sec 2024-03-24T01:37:21.042 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4369 sec 2024-03-24T01:37:21.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:21 smithi078 ceph-mon[32429]: pgmap v2451: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:21.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:21 smithi107 ceph-mon[36485]: pgmap v2451: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:21.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:21 smithi106 ceph-mon[36505]: pgmap v2451: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:22.042 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4370 sec 2024-03-24T01:37:23.042 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4371 sec 2024-03-24T01:37:23.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:23 smithi078 ceph-mon[32429]: pgmap v2452: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:23.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:23 smithi107 ceph-mon[36485]: pgmap v2452: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:23.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:23 smithi106 ceph-mon[36505]: pgmap v2452: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:24.042 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4372 sec 2024-03-24T01:37:24.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3718.158778 secs 2024-03-24T01:37:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3718.158778 secs 2024-03-24T01:37:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3718.158778 secs 2024-03-24T01:37:25.042 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4373 sec 2024-03-24T01:37:25.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:25 smithi078 ceph-mon[32429]: pgmap v2453: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:25.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:25 smithi107 ceph-mon[36485]: pgmap v2453: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:25.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:25 smithi106 ceph-mon[36505]: pgmap v2453: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:26.042 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4374 sec 2024-03-24T01:37:27.042 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4375 sec 2024-03-24T01:37:27.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:27 smithi078 ceph-mon[32429]: pgmap v2454: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:27.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:27 smithi107 ceph-mon[36485]: pgmap v2454: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:27.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:27 smithi106 ceph-mon[36505]: pgmap v2454: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:28.042 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4376 sec 2024-03-24T01:37:28.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3723.158865 secs 2024-03-24T01:37:28.700 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3723.158865 secs 2024-03-24T01:37:28.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3723.158865 secs 2024-03-24T01:37:29.042 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4377 sec 2024-03-24T01:37:29.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:29 smithi106 ceph-mon[36505]: pgmap v2455: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:29.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:29 smithi078 ceph-mon[32429]: pgmap v2455: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:29 smithi107 ceph-mon[36485]: pgmap v2455: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:30.042 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4378 sec 2024-03-24T01:37:31.042 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4379 sec 2024-03-24T01:37:31.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:31 smithi078 ceph-mon[32429]: pgmap v2456: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:31.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:31 smithi107 ceph-mon[36485]: pgmap v2456: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:31.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:31 smithi106 ceph-mon[36505]: pgmap v2456: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:32.043 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4380 sec 2024-03-24T01:37:33.043 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4381 sec 2024-03-24T01:37:33.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:33 smithi078 ceph-mon[32429]: pgmap v2457: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:33.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:33 smithi107 ceph-mon[36485]: pgmap v2457: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:33.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:33 smithi106 ceph-mon[36505]: pgmap v2457: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:34.043 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4382 sec 2024-03-24T01:37:34.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3728.158924 secs 2024-03-24T01:37:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3728.158924 secs 2024-03-24T01:37:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3728.158924 secs 2024-03-24T01:37:35.043 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4383 sec 2024-03-24T01:37:35.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:35 smithi078 ceph-mon[32429]: pgmap v2458: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:35.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:35 smithi107 ceph-mon[36485]: pgmap v2458: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:35.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:35 smithi106 ceph-mon[36505]: pgmap v2458: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:36.043 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4384 sec 2024-03-24T01:37:37.043 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4385 sec 2024-03-24T01:37:37.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:37 smithi078 ceph-mon[32429]: pgmap v2459: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:37.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:37 smithi107 ceph-mon[36485]: pgmap v2459: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:37.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:37 smithi106 ceph-mon[36505]: pgmap v2459: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:38.043 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4386 sec 2024-03-24T01:37:38.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3733.159011 secs 2024-03-24T01:37:38.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3733.159011 secs 2024-03-24T01:37:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3733.159011 secs 2024-03-24T01:37:39.043 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4387 sec 2024-03-24T01:37:39.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:39 smithi106 ceph-mon[36505]: pgmap v2460: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:39.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:39 smithi078 ceph-mon[32429]: pgmap v2460: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:39.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:39 smithi107 ceph-mon[36485]: pgmap v2460: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:40.043 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4388 sec 2024-03-24T01:37:41.043 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4389 sec 2024-03-24T01:37:41.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:41 smithi078 ceph-mon[32429]: pgmap v2461: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:41.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:41 smithi107 ceph-mon[36485]: pgmap v2461: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:41.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:41 smithi106 ceph-mon[36505]: pgmap v2461: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:42.043 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4390 sec 2024-03-24T01:37:43.044 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4391 sec 2024-03-24T01:37:43.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:43 smithi078 ceph-mon[32429]: pgmap v2462: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:43.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:43 smithi107 ceph-mon[36485]: pgmap v2462: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:43.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:43 smithi106 ceph-mon[36505]: pgmap v2462: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:44.044 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4392 sec 2024-03-24T01:37:44.572 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3738.159025 secs 2024-03-24T01:37:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3738.159025 secs 2024-03-24T01:37:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3738.159025 secs 2024-03-24T01:37:45.044 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4393 sec 2024-03-24T01:37:46.044 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4394 sec 2024-03-24T01:37:46.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:46 smithi106 ceph-mon[36505]: pgmap v2463: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:46.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:46 smithi078 ceph-mon[32429]: pgmap v2463: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:46.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:46 smithi107 ceph-mon[36485]: pgmap v2463: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:47.044 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4395 sec 2024-03-24T01:37:47.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:47 smithi078 ceph-mon[32429]: pgmap v2464: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:47 smithi107 ceph-mon[36485]: pgmap v2464: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:47.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:47 smithi106 ceph-mon[36505]: pgmap v2464: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:48.044 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4396 sec 2024-03-24T01:37:48.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3743.159088 secs 2024-03-24T01:37:48.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3743.159088 secs 2024-03-24T01:37:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3743.159088 secs 2024-03-24T01:37:49.044 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4397 sec 2024-03-24T01:37:49.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:49 smithi106 ceph-mon[36505]: pgmap v2465: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:49.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:49 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:37:49.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:49 smithi078 ceph-mon[32429]: pgmap v2465: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:49.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:49 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:37:49.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:49 smithi107 ceph-mon[36485]: pgmap v2465: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:37:49.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:49 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:37:50.044 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4398 sec 2024-03-24T01:37:50.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:50 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:37:50.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:50 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:37:50.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:50 smithi106 ceph-mon[36505]: pgmap v2466: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:37:50.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:50 smithi106 ceph-mon[36505]: pgmap v2467: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:37:50.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:50 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:37:50.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:50 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:37:50.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:50 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:37:50.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:50 smithi078 ceph-mon[32429]: pgmap v2466: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:37:50.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:50 smithi078 ceph-mon[32429]: pgmap v2467: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:37:50.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:50 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:37:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:50 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:37:50.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:50 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:37:50.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:50 smithi107 ceph-mon[36485]: pgmap v2466: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:37:50.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:50 smithi107 ceph-mon[36485]: pgmap v2467: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:37:50.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:50 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:37:51.044 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4399 sec 2024-03-24T01:37:52.044 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4400 sec 2024-03-24T01:37:53.044 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4401 sec 2024-03-24T01:37:53.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:52 smithi078 ceph-mon[32429]: pgmap v2468: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:37:53.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:52 smithi107 ceph-mon[36485]: pgmap v2468: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:37:53.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:52 smithi106 ceph-mon[36505]: pgmap v2468: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:37:54.044 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4402 sec 2024-03-24T01:37:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3748.159142 secs 2024-03-24T01:37:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3748.159142 secs 2024-03-24T01:37:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3748.159142 secs 2024-03-24T01:37:55.045 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4403 sec 2024-03-24T01:37:55.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:54 smithi078 ceph-mon[32429]: pgmap v2469: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:37:55.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:54 smithi107 ceph-mon[36485]: pgmap v2469: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:37:55.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:54 smithi106 ceph-mon[36505]: pgmap v2469: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:37:56.045 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4404 sec 2024-03-24T01:37:57.045 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4405 sec 2024-03-24T01:37:57.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:56 smithi078 ceph-mon[32429]: pgmap v2470: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:37:57.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:56 smithi107 ceph-mon[36485]: pgmap v2470: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:37:57.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:56 smithi106 ceph-mon[36505]: pgmap v2470: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:37:58.045 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4406 sec 2024-03-24T01:37:59.045 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4407 sec 2024-03-24T01:37:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:58 smithi078 ceph-mon[32429]: pgmap v2471: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:37:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:37:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3753.159221 secs 2024-03-24T01:37:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:58 smithi107 ceph-mon[36485]: pgmap v2471: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:37:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:37:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3753.159221 secs 2024-03-24T01:37:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:58 smithi106 ceph-mon[36505]: pgmap v2471: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:37:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:37:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3753.159221 secs 2024-03-24T01:38:00.045 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4408 sec 2024-03-24T01:38:01.045 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4409 sec 2024-03-24T01:38:01.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:00 smithi078 ceph-mon[32429]: pgmap v2472: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:38:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:00 smithi107 ceph-mon[36485]: pgmap v2472: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:38:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:00 smithi106 ceph-mon[36505]: pgmap v2472: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:38:02.045 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4410 sec 2024-03-24T01:38:03.045 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4411 sec 2024-03-24T01:38:03.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:03 smithi078 ceph-mon[32429]: pgmap v2473: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:38:03.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:03 smithi107 ceph-mon[36485]: pgmap v2473: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:38:03.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:03 smithi106 ceph-mon[36505]: pgmap v2473: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:38:04.045 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4412 sec 2024-03-24T01:38:04.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3758.159308 secs 2024-03-24T01:38:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3758.159308 secs 2024-03-24T01:38:04.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3758.159308 secs 2024-03-24T01:38:05.045 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4413 sec 2024-03-24T01:38:05.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:05 smithi078 ceph-mon[32429]: pgmap v2474: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:38:05.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:05 smithi107 ceph-mon[36485]: pgmap v2474: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:38:05.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:05 smithi106 ceph-mon[36505]: pgmap v2474: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:38:06.045 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4414 sec 2024-03-24T01:38:07.045 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4415 sec 2024-03-24T01:38:07.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:07 smithi078 ceph-mon[32429]: pgmap v2475: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:38:07.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:07 smithi107 ceph-mon[36485]: pgmap v2475: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:38:07.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:07 smithi106 ceph-mon[36505]: pgmap v2475: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:38:08.046 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4416 sec 2024-03-24T01:38:09.046 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4417 sec 2024-03-24T01:38:09.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3763.159369 secs 2024-03-24T01:38:09.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:09 smithi078 ceph-mon[32429]: pgmap v2476: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:38:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3763.159369 secs 2024-03-24T01:38:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:09 smithi107 ceph-mon[36485]: pgmap v2476: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:38:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3763.159369 secs 2024-03-24T01:38:09.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:09 smithi106 ceph-mon[36505]: pgmap v2476: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:38:10.046 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4418 sec 2024-03-24T01:38:11.046 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4419 sec 2024-03-24T01:38:11.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:11 smithi078 ceph-mon[32429]: pgmap v2477: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:38:11.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:11 smithi107 ceph-mon[36485]: pgmap v2477: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:38:11.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:11 smithi106 ceph-mon[36505]: pgmap v2477: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:38:12.046 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4420 sec 2024-03-24T01:38:13.046 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4421 sec 2024-03-24T01:38:13.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:13 smithi078 ceph-mon[32429]: pgmap v2478: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:38:13.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:13 smithi107 ceph-mon[36485]: pgmap v2478: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:38:13.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:13 smithi106 ceph-mon[36505]: pgmap v2478: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:38:14.048 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4422 sec 2024-03-24T01:38:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3768.159431 secs 2024-03-24T01:38:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3768.159431 secs 2024-03-24T01:38:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3768.159431 secs 2024-03-24T01:38:15.046 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4423 sec 2024-03-24T01:38:15.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:15 smithi078 ceph-mon[32429]: pgmap v2479: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:15.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:15 smithi107 ceph-mon[36485]: pgmap v2479: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:15.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:15 smithi106 ceph-mon[36505]: pgmap v2479: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:16.046 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4424 sec 2024-03-24T01:38:17.047 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4425 sec 2024-03-24T01:38:17.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:17 smithi078 ceph-mon[32429]: pgmap v2480: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:38:17.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:17 smithi107 ceph-mon[36485]: pgmap v2480: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:38:17.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:17 smithi106 ceph-mon[36505]: pgmap v2480: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:38:18.050 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4426 sec 2024-03-24T01:38:19.047 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4427 sec 2024-03-24T01:38:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3773.159490 secs 2024-03-24T01:38:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:19 smithi078 ceph-mon[32429]: pgmap v2481: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3773.159490 secs 2024-03-24T01:38:19.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:19 smithi107 ceph-mon[36485]: pgmap v2481: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3773.159490 secs 2024-03-24T01:38:19.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:19 smithi106 ceph-mon[36505]: pgmap v2481: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:20.047 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4428 sec 2024-03-24T01:38:21.047 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4429 sec 2024-03-24T01:38:21.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:21 smithi078 ceph-mon[32429]: pgmap v2482: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:21.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:21 smithi107 ceph-mon[36485]: pgmap v2482: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:21.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:21 smithi106 ceph-mon[36505]: pgmap v2482: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:22.047 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4430 sec 2024-03-24T01:38:23.047 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4431 sec 2024-03-24T01:38:23.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:23 smithi078 ceph-mon[32429]: pgmap v2483: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:38:23.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:23 smithi107 ceph-mon[36485]: pgmap v2483: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:38:23.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:23 smithi106 ceph-mon[36505]: pgmap v2483: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:38:24.047 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4432 sec 2024-03-24T01:38:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3778.159575 secs 2024-03-24T01:38:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3778.159575 secs 2024-03-24T01:38:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3778.159575 secs 2024-03-24T01:38:25.047 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4433 sec 2024-03-24T01:38:25.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:25 smithi078 ceph-mon[32429]: pgmap v2484: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:25.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:25 smithi107 ceph-mon[36485]: pgmap v2484: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:25.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:25 smithi106 ceph-mon[36505]: pgmap v2484: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:26.047 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4434 sec 2024-03-24T01:38:27.047 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4435 sec 2024-03-24T01:38:27.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:27 smithi078 ceph-mon[32429]: pgmap v2485: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:38:27.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:27 smithi107 ceph-mon[36485]: pgmap v2485: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:38:27.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:27 smithi106 ceph-mon[36505]: pgmap v2485: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:38:28.047 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4436 sec 2024-03-24T01:38:29.047 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4437 sec 2024-03-24T01:38:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3783.159663 secs 2024-03-24T01:38:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:29 smithi078 ceph-mon[32429]: pgmap v2486: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3783.159663 secs 2024-03-24T01:38:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:29 smithi107 ceph-mon[36485]: pgmap v2486: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3783.159663 secs 2024-03-24T01:38:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:29 smithi106 ceph-mon[36505]: pgmap v2486: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:30.048 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4438 sec 2024-03-24T01:38:31.048 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4439 sec 2024-03-24T01:38:31.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:31 smithi078 ceph-mon[32429]: pgmap v2487: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:31.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:31 smithi107 ceph-mon[36485]: pgmap v2487: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:31.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:31 smithi106 ceph-mon[36505]: pgmap v2487: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:32.048 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4440 sec 2024-03-24T01:38:33.048 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4441 sec 2024-03-24T01:38:33.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:33 smithi078 ceph-mon[32429]: pgmap v2488: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:38:33.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:33 smithi107 ceph-mon[36485]: pgmap v2488: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:38:33.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:33 smithi106 ceph-mon[36505]: pgmap v2488: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:38:34.048 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4442 sec 2024-03-24T01:38:34.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3788.159750 secs 2024-03-24T01:38:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3788.159750 secs 2024-03-24T01:38:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3788.159750 secs 2024-03-24T01:38:35.048 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4443 sec 2024-03-24T01:38:35.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:35 smithi078 ceph-mon[32429]: pgmap v2489: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:35.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:35 smithi107 ceph-mon[36485]: pgmap v2489: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:35.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:35 smithi106 ceph-mon[36505]: pgmap v2489: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:36.048 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4444 sec 2024-03-24T01:38:37.048 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4445 sec 2024-03-24T01:38:37.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:37 smithi078 ceph-mon[32429]: pgmap v2490: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:38:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:37 smithi107 ceph-mon[36485]: pgmap v2490: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:38:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:37 smithi106 ceph-mon[36505]: pgmap v2490: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:38:38.048 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4446 sec 2024-03-24T01:38:39.048 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4447 sec 2024-03-24T01:38:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3793.159806 secs 2024-03-24T01:38:39.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:39 smithi078 ceph-mon[32429]: pgmap v2491: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3793.159806 secs 2024-03-24T01:38:39.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:39 smithi107 ceph-mon[36485]: pgmap v2491: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3793.159806 secs 2024-03-24T01:38:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:39 smithi106 ceph-mon[36505]: pgmap v2491: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:40.048 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4448 sec 2024-03-24T01:38:41.049 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4449 sec 2024-03-24T01:38:41.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:41 smithi078 ceph-mon[32429]: pgmap v2492: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:41 smithi107 ceph-mon[36485]: pgmap v2492: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:41.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:41 smithi106 ceph-mon[36505]: pgmap v2492: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:42.048 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4450 sec 2024-03-24T01:38:43.049 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4451 sec 2024-03-24T01:38:43.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:43 smithi078 ceph-mon[32429]: pgmap v2493: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:38:43.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:43 smithi107 ceph-mon[36485]: pgmap v2493: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:38:43.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:43 smithi106 ceph-mon[36505]: pgmap v2493: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:38:44.049 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4452 sec 2024-03-24T01:38:44.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3798.159894 secs 2024-03-24T01:38:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3798.159894 secs 2024-03-24T01:38:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3798.159894 secs 2024-03-24T01:38:45.049 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4453 sec 2024-03-24T01:38:45.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:45 smithi078 ceph-mon[32429]: pgmap v2494: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:45.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:45 smithi107 ceph-mon[36485]: pgmap v2494: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:45.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:45 smithi106 ceph-mon[36505]: pgmap v2494: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:46.049 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4454 sec 2024-03-24T01:38:47.049 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4455 sec 2024-03-24T01:38:47.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:47 smithi078 ceph-mon[32429]: pgmap v2495: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:38:47.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:47 smithi107 ceph-mon[36485]: pgmap v2495: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:38:47.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:47 smithi106 ceph-mon[36505]: pgmap v2495: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:38:48.049 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4456 sec 2024-03-24T01:38:49.049 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4457 sec 2024-03-24T01:38:49.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3803.159983 secs 2024-03-24T01:38:49.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:49 smithi107 ceph-mon[36485]: pgmap v2496: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3803.159983 secs 2024-03-24T01:38:49.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:49 smithi106 ceph-mon[36505]: pgmap v2496: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3803.159983 secs 2024-03-24T01:38:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:49 smithi078 ceph-mon[32429]: pgmap v2496: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:50.049 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4458 sec 2024-03-24T01:38:50.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:50 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:38:50.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:50 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:38:50.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:50 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:38:51.049 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4459 sec 2024-03-24T01:38:51.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:51 smithi106 ceph-mon[36505]: pgmap v2497: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:51.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:51 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:38:51.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:51 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:38:51.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:51 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:38:51.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:51 smithi078 ceph-mon[32429]: pgmap v2497: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:51.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:51 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:38:51.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:51 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:38:51.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:51 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:38:51.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:51 smithi107 ceph-mon[36485]: pgmap v2497: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:38:51.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:51 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:38:51.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:51 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:38:51.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:51 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:38:52.049 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4460 sec 2024-03-24T01:38:52.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:52 smithi106 ceph-mon[36505]: pgmap v2498: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:38:52.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:52 smithi106 ceph-mon[36505]: pgmap v2499: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:38:52.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:52 smithi078 ceph-mon[32429]: pgmap v2498: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:38:52.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:52 smithi078 ceph-mon[32429]: pgmap v2499: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:38:52.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:52 smithi107 ceph-mon[36485]: pgmap v2498: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:38:52.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:52 smithi107 ceph-mon[36485]: pgmap v2499: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:38:53.049 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4461 sec 2024-03-24T01:38:54.050 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4462 sec 2024-03-24T01:38:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:54 smithi106 ceph-mon[36505]: pgmap v2500: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:38:54.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3808.160041 secs 2024-03-24T01:38:54.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:54 smithi078 ceph-mon[32429]: pgmap v2500: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:38:54.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3808.160041 secs 2024-03-24T01:38:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:54 smithi107 ceph-mon[36485]: pgmap v2500: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:38:54.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3808.160041 secs 2024-03-24T01:38:55.050 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4463 sec 2024-03-24T01:38:56.050 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4464 sec 2024-03-24T01:38:56.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:56 smithi078 ceph-mon[32429]: pgmap v2501: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:38:56.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:56 smithi107 ceph-mon[36485]: pgmap v2501: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:38:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:56 smithi106 ceph-mon[36505]: pgmap v2501: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:38:57.050 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4465 sec 2024-03-24T01:38:58.050 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4466 sec 2024-03-24T01:38:58.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:58 smithi078 ceph-mon[32429]: pgmap v2502: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:38:58.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:58 smithi107 ceph-mon[36485]: pgmap v2502: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:38:58.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:58 smithi106 ceph-mon[36505]: pgmap v2502: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:38:59.050 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4467 sec 2024-03-24T01:38:59.554 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:38:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3813.160080 secs 2024-03-24T01:38:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:38:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3813.160080 secs 2024-03-24T01:38:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:38:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3813.160080 secs 2024-03-24T01:39:00.050 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4468 sec 2024-03-24T01:39:00.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:00 smithi078 ceph-mon[32429]: pgmap v2503: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:39:00.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:00 smithi107 ceph-mon[36485]: pgmap v2503: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:39:00.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:00 smithi106 ceph-mon[36505]: pgmap v2503: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:39:01.050 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4469 sec 2024-03-24T01:39:02.050 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4470 sec 2024-03-24T01:39:02.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:02 smithi078 ceph-mon[32429]: pgmap v2504: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:39:02.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:02 smithi107 ceph-mon[36485]: pgmap v2504: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:39:02.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:02 smithi106 ceph-mon[36505]: pgmap v2504: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:39:03.050 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4471 sec 2024-03-24T01:39:04.050 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4472 sec 2024-03-24T01:39:04.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:04 smithi078 ceph-mon[32429]: pgmap v2505: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:04.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3818.160167 secs 2024-03-24T01:39:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:04 smithi107 ceph-mon[36485]: pgmap v2505: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3818.160167 secs 2024-03-24T01:39:04.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:04 smithi106 ceph-mon[36505]: pgmap v2505: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:04.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3818.160167 secs 2024-03-24T01:39:05.050 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4473 sec 2024-03-24T01:39:06.050 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4474 sec 2024-03-24T01:39:06.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:06 smithi078 ceph-mon[32429]: pgmap v2506: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:06.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:06 smithi107 ceph-mon[36485]: pgmap v2506: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:06.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:06 smithi106 ceph-mon[36505]: pgmap v2506: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:07.050 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4475 sec 2024-03-24T01:39:08.051 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4476 sec 2024-03-24T01:39:08.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:08 smithi078 ceph-mon[32429]: pgmap v2507: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:08.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:08 smithi107 ceph-mon[36485]: pgmap v2507: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:08.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:08 smithi106 ceph-mon[36505]: pgmap v2507: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:09.051 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4477 sec 2024-03-24T01:39:09.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3823.160252 secs 2024-03-24T01:39:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3823.160252 secs 2024-03-24T01:39:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3823.160252 secs 2024-03-24T01:39:10.051 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4478 sec 2024-03-24T01:39:10.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:10 smithi078 ceph-mon[32429]: pgmap v2508: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:10.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:10 smithi107 ceph-mon[36485]: pgmap v2508: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:10.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:10 smithi106 ceph-mon[36505]: pgmap v2508: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:11.051 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4479 sec 2024-03-24T01:39:12.051 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4480 sec 2024-03-24T01:39:12.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:12 smithi078 ceph-mon[32429]: pgmap v2509: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:12.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:12 smithi107 ceph-mon[36485]: pgmap v2509: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:12.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:12 smithi106 ceph-mon[36505]: pgmap v2509: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:13.051 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4481 sec 2024-03-24T01:39:14.051 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4482 sec 2024-03-24T01:39:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:14 smithi078 ceph-mon[32429]: pgmap v2510: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3828.160312 secs 2024-03-24T01:39:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:14 smithi107 ceph-mon[36485]: pgmap v2510: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:14.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3828.160312 secs 2024-03-24T01:39:14.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:14 smithi106 ceph-mon[36505]: pgmap v2510: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:14.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3828.160312 secs 2024-03-24T01:39:15.051 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4483 sec 2024-03-24T01:39:16.051 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4484 sec 2024-03-24T01:39:16.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:16 smithi078 ceph-mon[32429]: pgmap v2511: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:16.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:16 smithi107 ceph-mon[36485]: pgmap v2511: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:16 smithi106 ceph-mon[36505]: pgmap v2511: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:17.052 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4485 sec 2024-03-24T01:39:18.051 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4486 sec 2024-03-24T01:39:18.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:18 smithi078 ceph-mon[32429]: pgmap v2512: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:18.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:18 smithi107 ceph-mon[36485]: pgmap v2512: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:18.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:18 smithi106 ceph-mon[36505]: pgmap v2512: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:19.052 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4487 sec 2024-03-24T01:39:19.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3833.160348 secs 2024-03-24T01:39:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3833.160348 secs 2024-03-24T01:39:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3833.160348 secs 2024-03-24T01:39:20.052 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4488 sec 2024-03-24T01:39:20.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:20 smithi078 ceph-mon[32429]: pgmap v2513: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:20.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:20 smithi107 ceph-mon[36485]: pgmap v2513: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:20.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:20 smithi106 ceph-mon[36505]: pgmap v2513: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:21.052 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4489 sec 2024-03-24T01:39:22.052 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4490 sec 2024-03-24T01:39:22.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:22 smithi078 ceph-mon[32429]: pgmap v2514: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:22.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:22 smithi107 ceph-mon[36485]: pgmap v2514: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:22.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:22 smithi106 ceph-mon[36505]: pgmap v2514: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:23.052 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4491 sec 2024-03-24T01:39:24.052 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4492 sec 2024-03-24T01:39:24.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:24 smithi078 ceph-mon[32429]: pgmap v2515: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:24.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3838.160435 secs 2024-03-24T01:39:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:24 smithi107 ceph-mon[36485]: pgmap v2515: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3838.160435 secs 2024-03-24T01:39:24.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:24 smithi106 ceph-mon[36505]: pgmap v2515: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:24.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3838.160435 secs 2024-03-24T01:39:25.052 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4493 sec 2024-03-24T01:39:26.052 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4494 sec 2024-03-24T01:39:26.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:26 smithi078 ceph-mon[32429]: pgmap v2516: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:26.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:26 smithi107 ceph-mon[36485]: pgmap v2516: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:26 smithi106 ceph-mon[36505]: pgmap v2516: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:27.052 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4495 sec 2024-03-24T01:39:28.052 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4496 sec 2024-03-24T01:39:28.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:28 smithi078 ceph-mon[32429]: pgmap v2517: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:28.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:28 smithi107 ceph-mon[36485]: pgmap v2517: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:28.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:28 smithi106 ceph-mon[36505]: pgmap v2517: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:29.052 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4497 sec 2024-03-24T01:39:29.554 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3843.160493 secs 2024-03-24T01:39:29.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3843.160493 secs 2024-03-24T01:39:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3843.160493 secs 2024-03-24T01:39:30.052 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4498 sec 2024-03-24T01:39:30.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:30 smithi078 ceph-mon[32429]: pgmap v2518: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:30.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:30 smithi107 ceph-mon[36485]: pgmap v2518: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:30.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:30 smithi106 ceph-mon[36505]: pgmap v2518: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:31.053 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4499 sec 2024-03-24T01:39:31.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:31 smithi078 ceph-mon[32429]: pgmap v2519: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:31.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:31 smithi107 ceph-mon[36485]: pgmap v2519: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:31.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:31 smithi106 ceph-mon[36505]: pgmap v2519: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:32.052 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4500 sec 2024-03-24T01:39:33.053 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4501 sec 2024-03-24T01:39:34.053 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4502 sec 2024-03-24T01:39:34.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:33 smithi078 ceph-mon[32429]: pgmap v2520: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:34.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3848.160574 secs 2024-03-24T01:39:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:33 smithi107 ceph-mon[36485]: pgmap v2520: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3848.160574 secs 2024-03-24T01:39:34.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:33 smithi106 ceph-mon[36505]: pgmap v2520: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:34.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3848.160574 secs 2024-03-24T01:39:35.053 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4503 sec 2024-03-24T01:39:36.053 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4504 sec 2024-03-24T01:39:36.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:35 smithi078 ceph-mon[32429]: pgmap v2521: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:36.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:35 smithi107 ceph-mon[36485]: pgmap v2521: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:36.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:35 smithi106 ceph-mon[36505]: pgmap v2521: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:37.053 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4505 sec 2024-03-24T01:39:38.053 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4506 sec 2024-03-24T01:39:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:37 smithi078 ceph-mon[32429]: pgmap v2522: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:38.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:37 smithi107 ceph-mon[36485]: pgmap v2522: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:38.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:37 smithi106 ceph-mon[36505]: pgmap v2522: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:39.053 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4507 sec 2024-03-24T01:39:39.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3853.160637 secs 2024-03-24T01:39:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3853.160637 secs 2024-03-24T01:39:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3853.160637 secs 2024-03-24T01:39:40.053 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4508 sec 2024-03-24T01:39:40.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:39 smithi078 ceph-mon[32429]: pgmap v2523: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:39 smithi107 ceph-mon[36485]: pgmap v2523: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:39 smithi106 ceph-mon[36505]: pgmap v2523: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:41.053 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4509 sec 2024-03-24T01:39:42.053 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4510 sec 2024-03-24T01:39:42.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:41 smithi078 ceph-mon[32429]: pgmap v2524: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:42.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:41 smithi107 ceph-mon[36485]: pgmap v2524: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:42.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:41 smithi106 ceph-mon[36505]: pgmap v2524: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:43.053 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4511 sec 2024-03-24T01:39:44.054 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4512 sec 2024-03-24T01:39:44.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:43 smithi078 ceph-mon[32429]: pgmap v2525: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:44.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3858.160698 secs 2024-03-24T01:39:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:43 smithi107 ceph-mon[36485]: pgmap v2525: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:44.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3858.160698 secs 2024-03-24T01:39:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:43 smithi106 ceph-mon[36505]: pgmap v2525: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:44.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3858.160698 secs 2024-03-24T01:39:45.054 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4513 sec 2024-03-24T01:39:46.054 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4514 sec 2024-03-24T01:39:46.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:45 smithi078 ceph-mon[32429]: pgmap v2526: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:45 smithi107 ceph-mon[36485]: pgmap v2526: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:46.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:45 smithi106 ceph-mon[36505]: pgmap v2526: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:47.054 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4515 sec 2024-03-24T01:39:48.054 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4516 sec 2024-03-24T01:39:48.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:47 smithi078 ceph-mon[32429]: pgmap v2527: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:48.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:47 smithi107 ceph-mon[36485]: pgmap v2527: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:48.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:47 smithi106 ceph-mon[36505]: pgmap v2527: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:49.054 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4517 sec 2024-03-24T01:39:49.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3863.160786 secs 2024-03-24T01:39:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3863.160786 secs 2024-03-24T01:39:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3863.160786 secs 2024-03-24T01:39:50.054 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4518 sec 2024-03-24T01:39:50.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:49 smithi078 ceph-mon[32429]: pgmap v2528: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:50.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:49 smithi107 ceph-mon[36485]: pgmap v2528: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:50.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:49 smithi106 ceph-mon[36505]: pgmap v2528: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:39:51.054 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4519 sec 2024-03-24T01:39:52.054 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4520 sec 2024-03-24T01:39:52.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:51 smithi078 ceph-mon[32429]: pgmap v2529: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:52.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:51 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:39:52.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:51 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:39:52.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:51 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:39:52.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:51 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:39:52.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:51 smithi107 ceph-mon[36485]: pgmap v2529: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:52.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:51 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:39:52.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:51 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:39:52.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:51 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:39:52.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:51 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:39:52.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:51 smithi106 ceph-mon[36505]: pgmap v2529: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:52.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:51 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:39:52.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:51 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:39:52.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:51 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:39:52.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:51 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:39:53.054 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4521 sec 2024-03-24T01:39:53.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:52 smithi078 ceph-mon[32429]: pgmap v2530: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:39:53.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:52 smithi078 ceph-mon[32429]: pgmap v2531: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:39:53.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:52 smithi107 ceph-mon[36485]: pgmap v2530: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:39:53.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:52 smithi107 ceph-mon[36485]: pgmap v2531: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:39:53.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:52 smithi106 ceph-mon[36505]: pgmap v2530: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:39:53.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:52 smithi106 ceph-mon[36505]: pgmap v2531: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:39:54.054 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4522 sec 2024-03-24T01:39:54.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3868.160875 secs 2024-03-24T01:39:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3868.160875 secs 2024-03-24T01:39:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3868.160875 secs 2024-03-24T01:39:55.054 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4523 sec 2024-03-24T01:39:55.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:54 smithi078 ceph-mon[32429]: pgmap v2532: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:39:55.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:54 smithi107 ceph-mon[36485]: pgmap v2532: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:39:55.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:54 smithi106 ceph-mon[36505]: pgmap v2532: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:39:56.054 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4524 sec 2024-03-24T01:39:57.055 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4525 sec 2024-03-24T01:39:57.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:56 smithi078 ceph-mon[32429]: pgmap v2533: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:39:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:56 smithi107 ceph-mon[36485]: pgmap v2533: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:39:57.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:56 smithi106 ceph-mon[36505]: pgmap v2533: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:39:58.055 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4526 sec 2024-03-24T01:39:59.055 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4527 sec 2024-03-24T01:39:59.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:58 smithi078 ceph-mon[32429]: pgmap v2534: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:39:59.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:39:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3873.160963 secs 2024-03-24T01:39:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:58 smithi107 ceph-mon[36485]: pgmap v2534: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:39:59.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:39:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3873.160963 secs 2024-03-24T01:39:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:58 smithi106 ceph-mon[36505]: pgmap v2534: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:39:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:39:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3873.160963 secs 2024-03-24T01:40:00.055 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4528 sec 2024-03-24T01:40:01.055 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4529 sec 2024-03-24T01:40:01.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:00 smithi078 ceph-mon[32429]: pgmap v2535: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:40:01.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:00 smithi078 ceph-mon[32429]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:40:01.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:00 smithi107 ceph-mon[36485]: pgmap v2535: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:40:01.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:00 smithi107 ceph-mon[36485]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:40:01.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:00 smithi106 ceph-mon[36505]: pgmap v2535: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:40:01.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:00 smithi106 ceph-mon[36505]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:40:02.056 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4530 sec 2024-03-24T01:40:03.055 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4531 sec 2024-03-24T01:40:03.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:02 smithi078 ceph-mon[32429]: pgmap v2536: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:40:03.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:02 smithi107 ceph-mon[36485]: pgmap v2536: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:40:03.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:02 smithi106 ceph-mon[36505]: pgmap v2536: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:40:04.055 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4532 sec 2024-03-24T01:40:04.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3878.161002 secs 2024-03-24T01:40:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3878.161002 secs 2024-03-24T01:40:04.155 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3878.161002 secs 2024-03-24T01:40:05.055 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4533 sec 2024-03-24T01:40:05.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:04 smithi078 ceph-mon[32429]: pgmap v2537: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:05.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:04 smithi107 ceph-mon[36485]: pgmap v2537: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:05.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:04 smithi106 ceph-mon[36505]: pgmap v2537: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:06.055 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4534 sec 2024-03-24T01:40:07.055 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4535 sec 2024-03-24T01:40:07.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:06 smithi078 ceph-mon[32429]: pgmap v2538: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:06 smithi107 ceph-mon[36485]: pgmap v2538: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:07.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:06 smithi106 ceph-mon[36505]: pgmap v2538: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:08.055 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4536 sec 2024-03-24T01:40:09.055 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4537 sec 2024-03-24T01:40:09.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:08 smithi107 ceph-mon[36485]: pgmap v2539: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:09.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3883.161107 secs 2024-03-24T01:40:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:08 smithi106 ceph-mon[36505]: pgmap v2539: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3883.161107 secs 2024-03-24T01:40:09.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:08 smithi078 ceph-mon[32429]: pgmap v2539: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:09.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3883.161107 secs 2024-03-24T01:40:10.055 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4538 sec 2024-03-24T01:40:11.055 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4539 sec 2024-03-24T01:40:11.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:10 smithi106 ceph-mon[36505]: pgmap v2540: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:11.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:10 smithi078 ceph-mon[32429]: pgmap v2540: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:11.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:10 smithi107 ceph-mon[36485]: pgmap v2540: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:12.055 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4540 sec 2024-03-24T01:40:13.056 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4541 sec 2024-03-24T01:40:13.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:12 smithi106 ceph-mon[36505]: pgmap v2541: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:13.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:12 smithi078 ceph-mon[32429]: pgmap v2541: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:13.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:12 smithi107 ceph-mon[36485]: pgmap v2541: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:14.056 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4542 sec 2024-03-24T01:40:14.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3888.161199 secs 2024-03-24T01:40:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3888.161199 secs 2024-03-24T01:40:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3888.161199 secs 2024-03-24T01:40:15.056 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4543 sec 2024-03-24T01:40:15.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:14 smithi106 ceph-mon[36505]: pgmap v2542: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:15.244 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:14 smithi107 ceph-mon[36485]: pgmap v2542: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:15.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:14 smithi078 ceph-mon[32429]: pgmap v2542: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:16.056 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4544 sec 2024-03-24T01:40:17.056 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4545 sec 2024-03-24T01:40:17.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:16 smithi106 ceph-mon[36505]: pgmap v2543: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:17.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:16 smithi078 ceph-mon[32429]: pgmap v2543: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:17.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:16 smithi107 ceph-mon[36485]: pgmap v2543: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:18.056 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4546 sec 2024-03-24T01:40:19.056 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4547 sec 2024-03-24T01:40:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:18 smithi106 ceph-mon[36505]: pgmap v2544: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:19.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3893.161238 secs 2024-03-24T01:40:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:18 smithi078 ceph-mon[32429]: pgmap v2544: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3893.161238 secs 2024-03-24T01:40:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:18 smithi107 ceph-mon[36485]: pgmap v2544: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3893.161238 secs 2024-03-24T01:40:20.056 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4548 sec 2024-03-24T01:40:21.056 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4549 sec 2024-03-24T01:40:21.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:20 smithi106 ceph-mon[36505]: pgmap v2545: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:21.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:20 smithi078 ceph-mon[32429]: pgmap v2545: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:21.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:20 smithi107 ceph-mon[36485]: pgmap v2545: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:22.056 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4550 sec 2024-03-24T01:40:23.056 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4551 sec 2024-03-24T01:40:23.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:22 smithi106 ceph-mon[36505]: pgmap v2546: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:23.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:22 smithi078 ceph-mon[32429]: pgmap v2546: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:23.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:22 smithi107 ceph-mon[36485]: pgmap v2546: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:24.056 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4552 sec 2024-03-24T01:40:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3898.161338 secs 2024-03-24T01:40:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3898.161338 secs 2024-03-24T01:40:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3898.161338 secs 2024-03-24T01:40:25.057 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4553 sec 2024-03-24T01:40:25.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:24 smithi106 ceph-mon[36505]: pgmap v2547: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:25.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:24 smithi078 ceph-mon[32429]: pgmap v2547: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:25.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:24 smithi107 ceph-mon[36485]: pgmap v2547: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:26.057 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4554 sec 2024-03-24T01:40:27.057 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4555 sec 2024-03-24T01:40:27.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:26 smithi106 ceph-mon[36505]: pgmap v2548: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:27.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:26 smithi078 ceph-mon[32429]: pgmap v2548: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:27.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:26 smithi107 ceph-mon[36485]: pgmap v2548: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:28.057 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4556 sec 2024-03-24T01:40:29.057 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4557 sec 2024-03-24T01:40:29.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:28 smithi106 ceph-mon[36505]: pgmap v2549: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:29.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3903.161418 secs 2024-03-24T01:40:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:28 smithi078 ceph-mon[32429]: pgmap v2549: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3903.161418 secs 2024-03-24T01:40:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:28 smithi107 ceph-mon[36485]: pgmap v2549: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3903.161418 secs 2024-03-24T01:40:30.057 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4558 sec 2024-03-24T01:40:31.057 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4559 sec 2024-03-24T01:40:31.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:30 smithi106 ceph-mon[36505]: pgmap v2550: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:31.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:30 smithi078 ceph-mon[32429]: pgmap v2550: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:31.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:30 smithi107 ceph-mon[36485]: pgmap v2550: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:32.057 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4560 sec 2024-03-24T01:40:33.057 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4561 sec 2024-03-24T01:40:33.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:32 smithi106 ceph-mon[36505]: pgmap v2551: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:33.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:32 smithi078 ceph-mon[32429]: pgmap v2551: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:33.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:32 smithi107 ceph-mon[36485]: pgmap v2551: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:34.057 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4562 sec 2024-03-24T01:40:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3908.161478 secs 2024-03-24T01:40:34.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3908.161478 secs 2024-03-24T01:40:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3908.161478 secs 2024-03-24T01:40:35.057 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4563 sec 2024-03-24T01:40:35.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:34 smithi078 ceph-mon[32429]: pgmap v2552: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:35.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:34 smithi106 ceph-mon[36505]: pgmap v2552: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:35.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:34 smithi107 ceph-mon[36485]: pgmap v2552: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:36.058 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4564 sec 2024-03-24T01:40:37.058 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4565 sec 2024-03-24T01:40:37.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:36 smithi106 ceph-mon[36505]: pgmap v2553: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:37.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:36 smithi078 ceph-mon[32429]: pgmap v2553: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:36 smithi107 ceph-mon[36485]: pgmap v2553: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:38.058 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4566 sec 2024-03-24T01:40:39.058 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4567 sec 2024-03-24T01:40:39.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:38 smithi078 ceph-mon[32429]: pgmap v2554: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:39.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3913.161538 secs 2024-03-24T01:40:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:38 smithi107 ceph-mon[36485]: pgmap v2554: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3913.161538 secs 2024-03-24T01:40:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:38 smithi106 ceph-mon[36505]: pgmap v2554: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3913.161538 secs 2024-03-24T01:40:40.058 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4568 sec 2024-03-24T01:40:41.058 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4569 sec 2024-03-24T01:40:41.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:40 smithi078 ceph-mon[32429]: pgmap v2555: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:40 smithi107 ceph-mon[36485]: pgmap v2555: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:41.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:40 smithi106 ceph-mon[36505]: pgmap v2555: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:42.058 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4570 sec 2024-03-24T01:40:43.058 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4571 sec 2024-03-24T01:40:43.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:42 smithi078 ceph-mon[32429]: pgmap v2556: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:43.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:42 smithi107 ceph-mon[36485]: pgmap v2556: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:43.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:42 smithi106 ceph-mon[36505]: pgmap v2556: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:44.058 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4572 sec 2024-03-24T01:40:44.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3918.161625 secs 2024-03-24T01:40:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3918.161625 secs 2024-03-24T01:40:44.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3918.161625 secs 2024-03-24T01:40:45.058 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4573 sec 2024-03-24T01:40:45.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:44 smithi107 ceph-mon[36485]: pgmap v2557: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:45.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:44 smithi078 ceph-mon[32429]: pgmap v2557: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:45.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:44 smithi106 ceph-mon[36505]: pgmap v2557: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:46.058 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4574 sec 2024-03-24T01:40:47.058 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4575 sec 2024-03-24T01:40:47.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:46 smithi078 ceph-mon[32429]: pgmap v2558: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:47.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:46 smithi107 ceph-mon[36485]: pgmap v2558: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:47.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:46 smithi106 ceph-mon[36505]: pgmap v2558: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:48.058 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4576 sec 2024-03-24T01:40:49.059 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4577 sec 2024-03-24T01:40:49.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:48 smithi078 ceph-mon[32429]: pgmap v2559: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:49.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3923.161687 secs 2024-03-24T01:40:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:48 smithi107 ceph-mon[36485]: pgmap v2559: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3923.161687 secs 2024-03-24T01:40:49.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:48 smithi106 ceph-mon[36505]: pgmap v2559: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:49.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3923.161687 secs 2024-03-24T01:40:50.059 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4578 sec 2024-03-24T01:40:51.059 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4579 sec 2024-03-24T01:40:51.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:50 smithi078 ceph-mon[32429]: pgmap v2560: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:51.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:50 smithi107 ceph-mon[36485]: pgmap v2560: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:51.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:50 smithi106 ceph-mon[36505]: pgmap v2560: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:40:52.059 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4580 sec 2024-03-24T01:40:52.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:51 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:40:52.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:51 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:40:52.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:51 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:40:53.059 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4581 sec 2024-03-24T01:40:53.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:52 smithi078 ceph-mon[32429]: pgmap v2561: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:53.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:52 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:40:53.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:52 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:40:53.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:52 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:40:53.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:52 smithi107 ceph-mon[36485]: pgmap v2561: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:53.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:52 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:40:53.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:52 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:40:53.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:52 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:40:53.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:52 smithi106 ceph-mon[36505]: pgmap v2561: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:53.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:52 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:40:53.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:52 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:40:53.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:52 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:40:54.059 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4582 sec 2024-03-24T01:40:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:53 smithi078 ceph-mon[32429]: pgmap v2562: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:40:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:53 smithi078 ceph-mon[32429]: pgmap v2563: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:40:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3928.161748 secs 2024-03-24T01:40:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:53 smithi107 ceph-mon[36485]: pgmap v2562: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:40:54.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:53 smithi107 ceph-mon[36485]: pgmap v2563: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:40:54.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3928.161748 secs 2024-03-24T01:40:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:53 smithi106 ceph-mon[36505]: pgmap v2562: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:40:54.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:53 smithi106 ceph-mon[36505]: pgmap v2563: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:40:54.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3928.161748 secs 2024-03-24T01:40:55.059 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4583 sec 2024-03-24T01:40:56.059 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4584 sec 2024-03-24T01:40:56.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:55 smithi078 ceph-mon[32429]: pgmap v2564: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:40:56.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:55 smithi107 ceph-mon[36485]: pgmap v2564: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:40:56.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:55 smithi106 ceph-mon[36505]: pgmap v2564: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:40:57.059 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4585 sec 2024-03-24T01:40:58.059 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4586 sec 2024-03-24T01:40:58.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:57 smithi078 ceph-mon[32429]: pgmap v2565: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:40:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:57 smithi107 ceph-mon[36485]: pgmap v2565: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:40:58.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:57 smithi106 ceph-mon[36505]: pgmap v2565: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:40:59.060 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4587 sec 2024-03-24T01:40:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3933.161809 secs 2024-03-24T01:40:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3933.161809 secs 2024-03-24T01:40:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3933.161809 secs 2024-03-24T01:41:00.060 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4588 sec 2024-03-24T01:41:00.304 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:40:59 smithi106 ceph-mon[36505]: pgmap v2566: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:41:00.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:40:59 smithi078 ceph-mon[32429]: pgmap v2566: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:41:00.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:40:59 smithi107 ceph-mon[36485]: pgmap v2566: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:41:01.060 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4589 sec 2024-03-24T01:41:02.060 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4590 sec 2024-03-24T01:41:02.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:01 smithi078 ceph-mon[32429]: pgmap v2567: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:41:02.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:01 smithi107 ceph-mon[36485]: pgmap v2567: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:41:02.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:01 smithi106 ceph-mon[36505]: pgmap v2567: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:41:03.060 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4591 sec 2024-03-24T01:41:04.060 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4592 sec 2024-03-24T01:41:04.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:03 smithi078 ceph-mon[32429]: pgmap v2568: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:41:04.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3938.161866 secs 2024-03-24T01:41:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:03 smithi107 ceph-mon[36485]: pgmap v2568: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:41:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3938.161866 secs 2024-03-24T01:41:04.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:03 smithi106 ceph-mon[36505]: pgmap v2568: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:41:04.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3938.161866 secs 2024-03-24T01:41:05.060 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4593 sec 2024-03-24T01:41:06.060 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4594 sec 2024-03-24T01:41:06.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:05 smithi078 ceph-mon[32429]: pgmap v2569: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:06.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:05 smithi107 ceph-mon[36485]: pgmap v2569: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:06.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:05 smithi106 ceph-mon[36505]: pgmap v2569: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:07.060 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4595 sec 2024-03-24T01:41:08.060 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4596 sec 2024-03-24T01:41:08.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:07 smithi078 ceph-mon[32429]: pgmap v2570: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:08.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:07 smithi107 ceph-mon[36485]: pgmap v2570: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:08.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:07 smithi106 ceph-mon[36505]: pgmap v2570: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:09.060 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4597 sec 2024-03-24T01:41:09.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3943.161928 secs 2024-03-24T01:41:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3943.161928 secs 2024-03-24T01:41:09.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3943.161928 secs 2024-03-24T01:41:10.061 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4598 sec 2024-03-24T01:41:10.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:09 smithi078 ceph-mon[32429]: pgmap v2571: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:10.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:09 smithi107 ceph-mon[36485]: pgmap v2571: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:10.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:09 smithi106 ceph-mon[36505]: pgmap v2571: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:11.061 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4599 sec 2024-03-24T01:41:12.061 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4600 sec 2024-03-24T01:41:12.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:11 smithi078 ceph-mon[32429]: pgmap v2572: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:12.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:11 smithi107 ceph-mon[36485]: pgmap v2572: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:12.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:11 smithi106 ceph-mon[36505]: pgmap v2572: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:13.061 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4601 sec 2024-03-24T01:41:14.061 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4602 sec 2024-03-24T01:41:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:13 smithi078 ceph-mon[32429]: pgmap v2573: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3948.161986 secs 2024-03-24T01:41:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:13 smithi107 ceph-mon[36485]: pgmap v2573: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3948.161986 secs 2024-03-24T01:41:14.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:13 smithi106 ceph-mon[36505]: pgmap v2573: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:14.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3948.161986 secs 2024-03-24T01:41:15.061 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4603 sec 2024-03-24T01:41:16.061 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4604 sec 2024-03-24T01:41:16.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:15 smithi078 ceph-mon[32429]: pgmap v2574: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:16.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:15 smithi107 ceph-mon[36485]: pgmap v2574: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:16.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:15 smithi106 ceph-mon[36505]: pgmap v2574: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:17.061 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4605 sec 2024-03-24T01:41:18.061 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4606 sec 2024-03-24T01:41:18.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:17 smithi078 ceph-mon[32429]: pgmap v2575: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:17 smithi107 ceph-mon[36485]: pgmap v2575: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:18.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:17 smithi106 ceph-mon[36505]: pgmap v2575: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:19.061 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4607 sec 2024-03-24T01:41:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3953.162074 secs 2024-03-24T01:41:19.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3953.162074 secs 2024-03-24T01:41:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3953.162074 secs 2024-03-24T01:41:20.062 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4608 sec 2024-03-24T01:41:20.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:19 smithi078 ceph-mon[32429]: pgmap v2576: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:19 smithi107 ceph-mon[36485]: pgmap v2576: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:20.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:19 smithi106 ceph-mon[36505]: pgmap v2576: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:21.062 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4609 sec 2024-03-24T01:41:22.062 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4610 sec 2024-03-24T01:41:22.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:21 smithi078 ceph-mon[32429]: pgmap v2577: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:22.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:21 smithi107 ceph-mon[36485]: pgmap v2577: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:22.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:21 smithi106 ceph-mon[36505]: pgmap v2577: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:23.062 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4611 sec 2024-03-24T01:41:24.062 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4612 sec 2024-03-24T01:41:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:23 smithi078 ceph-mon[32429]: pgmap v2578: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3958.162161 secs 2024-03-24T01:41:24.346 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:23 smithi107 ceph-mon[36485]: pgmap v2578: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:24.346 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3958.162161 secs 2024-03-24T01:41:24.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:23 smithi106 ceph-mon[36505]: pgmap v2578: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:24.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3958.162161 secs 2024-03-24T01:41:25.062 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4613 sec 2024-03-24T01:41:26.062 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4614 sec 2024-03-24T01:41:26.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:25 smithi078 ceph-mon[32429]: pgmap v2579: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:26.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:25 smithi107 ceph-mon[36485]: pgmap v2579: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:26.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:25 smithi106 ceph-mon[36505]: pgmap v2579: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:27.062 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4615 sec 2024-03-24T01:41:28.062 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4616 sec 2024-03-24T01:41:28.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:28 smithi078 ceph-mon[32429]: pgmap v2580: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:28.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:28 smithi107 ceph-mon[36485]: pgmap v2580: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:28.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:28 smithi106 ceph-mon[36505]: pgmap v2580: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:29.062 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4617 sec 2024-03-24T01:41:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3963.162222 secs 2024-03-24T01:41:29.346 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3963.162222 secs 2024-03-24T01:41:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3963.162222 secs 2024-03-24T01:41:30.062 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4618 sec 2024-03-24T01:41:30.304 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:30 smithi106 ceph-mon[36505]: pgmap v2581: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:30.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:30 smithi078 ceph-mon[32429]: pgmap v2581: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:30.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:30 smithi107 ceph-mon[36485]: pgmap v2581: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:31.062 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4619 sec 2024-03-24T01:41:32.062 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4620 sec 2024-03-24T01:41:32.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:32 smithi078 ceph-mon[32429]: pgmap v2582: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:32.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:32 smithi107 ceph-mon[36485]: pgmap v2582: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:32.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:32 smithi106 ceph-mon[36505]: pgmap v2582: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:33.062 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4621 sec 2024-03-24T01:41:34.063 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4622 sec 2024-03-24T01:41:34.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:34 smithi078 ceph-mon[32429]: pgmap v2583: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:34.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3968.162313 secs 2024-03-24T01:41:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:34 smithi107 ceph-mon[36485]: pgmap v2583: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:34.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3968.162313 secs 2024-03-24T01:41:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:34 smithi106 ceph-mon[36505]: pgmap v2583: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:34.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3968.162313 secs 2024-03-24T01:41:35.063 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4623 sec 2024-03-24T01:41:36.063 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4624 sec 2024-03-24T01:41:36.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:36 smithi078 ceph-mon[32429]: pgmap v2584: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:36.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:36 smithi107 ceph-mon[36485]: pgmap v2584: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:36.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:36 smithi106 ceph-mon[36505]: pgmap v2584: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:37.063 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4625 sec 2024-03-24T01:41:38.063 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4626 sec 2024-03-24T01:41:38.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:38 smithi078 ceph-mon[32429]: pgmap v2585: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:38.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:38 smithi107 ceph-mon[36485]: pgmap v2585: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:38.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:38 smithi106 ceph-mon[36505]: pgmap v2585: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:39.063 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4627 sec 2024-03-24T01:41:39.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3973.162328 secs 2024-03-24T01:41:39.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3973.162328 secs 2024-03-24T01:41:39.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3973.162328 secs 2024-03-24T01:41:40.063 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4628 sec 2024-03-24T01:41:40.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:40 smithi078 ceph-mon[32429]: pgmap v2586: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:40.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:40 smithi107 ceph-mon[36485]: pgmap v2586: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:40.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:40 smithi106 ceph-mon[36505]: pgmap v2586: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:41.063 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4629 sec 2024-03-24T01:41:42.063 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4630 sec 2024-03-24T01:41:42.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:42 smithi078 ceph-mon[32429]: pgmap v2587: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:42.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:42 smithi107 ceph-mon[36485]: pgmap v2587: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:42.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:42 smithi106 ceph-mon[36505]: pgmap v2587: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:43.063 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4631 sec 2024-03-24T01:41:44.063 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4632 sec 2024-03-24T01:41:44.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:44 smithi078 ceph-mon[32429]: pgmap v2588: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:44.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3978.162386 secs 2024-03-24T01:41:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:44 smithi107 ceph-mon[36485]: pgmap v2588: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:44.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3978.162386 secs 2024-03-24T01:41:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:44 smithi106 ceph-mon[36505]: pgmap v2588: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:44.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3978.162386 secs 2024-03-24T01:41:45.064 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4633 sec 2024-03-24T01:41:46.064 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4634 sec 2024-03-24T01:41:46.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:46 smithi078 ceph-mon[32429]: pgmap v2589: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:46.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:46 smithi107 ceph-mon[36485]: pgmap v2589: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:46.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:46 smithi106 ceph-mon[36505]: pgmap v2589: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:47.064 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4635 sec 2024-03-24T01:41:48.064 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4636 sec 2024-03-24T01:41:48.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:48 smithi078 ceph-mon[32429]: pgmap v2590: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:48.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:48 smithi107 ceph-mon[36485]: pgmap v2590: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:48.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:48 smithi106 ceph-mon[36505]: pgmap v2590: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:49.064 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4637 sec 2024-03-24T01:41:49.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3983.162470 secs 2024-03-24T01:41:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3983.162470 secs 2024-03-24T01:41:49.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3983.162470 secs 2024-03-24T01:41:50.064 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4638 sec 2024-03-24T01:41:50.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:50 smithi078 ceph-mon[32429]: pgmap v2591: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:50 smithi107 ceph-mon[36485]: pgmap v2591: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:50.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:50 smithi106 ceph-mon[36505]: pgmap v2591: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:51.064 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4639 sec 2024-03-24T01:41:52.064 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4640 sec 2024-03-24T01:41:52.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:52 smithi078 ceph-mon[32429]: pgmap v2592: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:52.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:52 smithi107 ceph-mon[36485]: pgmap v2592: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:52.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:52 smithi106 ceph-mon[36505]: pgmap v2592: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:41:53.064 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4641 sec 2024-03-24T01:41:53.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:53 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:41:53.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:53 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:41:53.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:53 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:41:53.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:53 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:41:53.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:53 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:41:53.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:53 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:41:53.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:53 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:41:53.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:53 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:41:53.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:53 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:41:54.064 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4642 sec 2024-03-24T01:41:54.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:54 smithi078 ceph-mon[32429]: pgmap v2593: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:54.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3988.162536 secs 2024-03-24T01:41:54.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:54 smithi078 ceph-mon[32429]: pgmap v2594: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:41:54.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:54 smithi078 ceph-mon[32429]: pgmap v2595: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:41:54.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:54 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:41:54.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:54 smithi107 ceph-mon[36485]: pgmap v2593: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:54.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3988.162536 secs 2024-03-24T01:41:54.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:54 smithi107 ceph-mon[36485]: pgmap v2594: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:41:54.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:54 smithi107 ceph-mon[36485]: pgmap v2595: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:41:54.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:54 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:41:54.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:54 smithi106 ceph-mon[36505]: pgmap v2593: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:54.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3988.162536 secs 2024-03-24T01:41:54.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:54 smithi106 ceph-mon[36505]: pgmap v2594: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:41:54.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:54 smithi106 ceph-mon[36505]: pgmap v2595: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:41:54.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:54 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:41:55.064 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4643 sec 2024-03-24T01:41:56.064 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4644 sec 2024-03-24T01:41:56.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:56 smithi078 ceph-mon[32429]: pgmap v2596: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:41:56.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:56 smithi107 ceph-mon[36485]: pgmap v2596: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:41:56.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:56 smithi106 ceph-mon[36505]: pgmap v2596: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:41:57.064 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4645 sec 2024-03-24T01:41:58.065 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4646 sec 2024-03-24T01:41:58.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:58 smithi078 ceph-mon[32429]: pgmap v2597: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:41:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:58 smithi107 ceph-mon[36485]: pgmap v2597: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:41:58.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:58 smithi106 ceph-mon[36505]: pgmap v2597: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:41:59.065 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4647 sec 2024-03-24T01:41:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:41:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3993.162620 secs 2024-03-24T01:41:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:41:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3993.162620 secs 2024-03-24T01:41:59.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:41:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3993.162620 secs 2024-03-24T01:42:00.065 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4648 sec 2024-03-24T01:42:00.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:00 smithi078 ceph-mon[32429]: pgmap v2598: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:42:00.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:00 smithi107 ceph-mon[36485]: pgmap v2598: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:42:00.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:00 smithi106 ceph-mon[36505]: pgmap v2598: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:42:01.065 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4649 sec 2024-03-24T01:42:02.065 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4650 sec 2024-03-24T01:42:02.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:02 smithi078 ceph-mon[32429]: pgmap v2599: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:42:02.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:02 smithi107 ceph-mon[36485]: pgmap v2599: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:42:02.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:02 smithi106 ceph-mon[36505]: pgmap v2599: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:42:03.065 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4651 sec 2024-03-24T01:42:04.065 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4652 sec 2024-03-24T01:42:04.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 3998.162658 secs 2024-03-24T01:42:04.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:04 smithi078 ceph-mon[32429]: pgmap v2600: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:42:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 3998.162658 secs 2024-03-24T01:42:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:04 smithi107 ceph-mon[36485]: pgmap v2600: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:42:04.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 3998.162658 secs 2024-03-24T01:42:04.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:04 smithi106 ceph-mon[36505]: pgmap v2600: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:42:05.065 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4653 sec 2024-03-24T01:42:06.065 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4654 sec 2024-03-24T01:42:06.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:06 smithi078 ceph-mon[32429]: pgmap v2601: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:06.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:06 smithi107 ceph-mon[36485]: pgmap v2601: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:06.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:06 smithi106 ceph-mon[36505]: pgmap v2601: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:07.065 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4655 sec 2024-03-24T01:42:08.065 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4656 sec 2024-03-24T01:42:08.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:08 smithi078 ceph-mon[32429]: pgmap v2602: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:08.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:08 smithi107 ceph-mon[36485]: pgmap v2602: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:08.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:08 smithi106 ceph-mon[36505]: pgmap v2602: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:09.065 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4657 sec 2024-03-24T01:42:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4003.162742 secs 2024-03-24T01:42:09.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4003.162742 secs 2024-03-24T01:42:09.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4003.162742 secs 2024-03-24T01:42:10.066 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4658 sec 2024-03-24T01:42:10.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:10 smithi106 ceph-mon[36505]: pgmap v2603: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:10.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:10 smithi078 ceph-mon[32429]: pgmap v2603: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:10.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:10 smithi107 ceph-mon[36485]: pgmap v2603: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:11.066 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4659 sec 2024-03-24T01:42:12.066 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4660 sec 2024-03-24T01:42:12.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:12 smithi106 ceph-mon[36505]: pgmap v2604: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:12.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:12 smithi078 ceph-mon[32429]: pgmap v2604: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:12.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:12 smithi107 ceph-mon[36485]: pgmap v2604: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:13.066 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4661 sec 2024-03-24T01:42:14.066 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4662 sec 2024-03-24T01:42:14.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4008.162832 secs 2024-03-24T01:42:14.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:14 smithi106 ceph-mon[36505]: pgmap v2605: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4008.162832 secs 2024-03-24T01:42:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:14 smithi078 ceph-mon[32429]: pgmap v2605: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4008.162832 secs 2024-03-24T01:42:14.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:14 smithi107 ceph-mon[36485]: pgmap v2605: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:15.066 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4663 sec 2024-03-24T01:42:16.067 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4664 sec 2024-03-24T01:42:16.396 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:16 smithi107 ceph-mon[36485]: pgmap v2606: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:16.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:16 smithi106 ceph-mon[36505]: pgmap v2606: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:16.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:16 smithi078 ceph-mon[32429]: pgmap v2606: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:17.066 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4665 sec 2024-03-24T01:42:18.066 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4666 sec 2024-03-24T01:42:18.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:18 smithi106 ceph-mon[36505]: pgmap v2607: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:18.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:18 smithi078 ceph-mon[32429]: pgmap v2607: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:18.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:18 smithi107 ceph-mon[36485]: pgmap v2607: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:19.066 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4667 sec 2024-03-24T01:42:19.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4013.162891 secs 2024-03-24T01:42:19.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4013.162891 secs 2024-03-24T01:42:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4013.162891 secs 2024-03-24T01:42:20.066 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4668 sec 2024-03-24T01:42:20.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:20 smithi106 ceph-mon[36505]: pgmap v2608: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:20.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:20 smithi078 ceph-mon[32429]: pgmap v2608: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:20.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:20 smithi107 ceph-mon[36485]: pgmap v2608: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:21.066 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4669 sec 2024-03-24T01:42:22.067 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4670 sec 2024-03-24T01:42:22.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:22 smithi106 ceph-mon[36505]: pgmap v2609: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:22.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:22 smithi078 ceph-mon[32429]: pgmap v2609: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:22.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:22 smithi107 ceph-mon[36485]: pgmap v2609: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:23.067 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4671 sec 2024-03-24T01:42:24.067 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4672 sec 2024-03-24T01:42:24.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4018.162948 secs 2024-03-24T01:42:24.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:24 smithi106 ceph-mon[36505]: pgmap v2610: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:24.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4018.162948 secs 2024-03-24T01:42:24.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:24 smithi078 ceph-mon[32429]: pgmap v2610: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4018.162948 secs 2024-03-24T01:42:24.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:24 smithi107 ceph-mon[36485]: pgmap v2610: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:25.067 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4673 sec 2024-03-24T01:42:26.067 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4674 sec 2024-03-24T01:42:26.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:26 smithi106 ceph-mon[36505]: pgmap v2611: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:26.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:26 smithi078 ceph-mon[32429]: pgmap v2611: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:26.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:26 smithi107 ceph-mon[36485]: pgmap v2611: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:27.067 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4675 sec 2024-03-24T01:42:28.067 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4676 sec 2024-03-24T01:42:28.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:28 smithi106 ceph-mon[36505]: pgmap v2612: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:28.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:28 smithi078 ceph-mon[32429]: pgmap v2612: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:28.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:28 smithi107 ceph-mon[36485]: pgmap v2612: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:29.067 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4677 sec 2024-03-24T01:42:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4023.163036 secs 2024-03-24T01:42:29.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4023.163036 secs 2024-03-24T01:42:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4023.163036 secs 2024-03-24T01:42:30.067 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4678 sec 2024-03-24T01:42:30.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:30 smithi106 ceph-mon[36505]: pgmap v2613: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:30.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:30 smithi078 ceph-mon[32429]: pgmap v2613: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:30.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:30 smithi107 ceph-mon[36485]: pgmap v2613: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:31.070 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4679 sec 2024-03-24T01:42:32.067 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4680 sec 2024-03-24T01:42:32.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:32 smithi106 ceph-mon[36505]: pgmap v2614: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:32.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:32 smithi078 ceph-mon[32429]: pgmap v2614: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:32.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:32 smithi107 ceph-mon[36485]: pgmap v2614: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:33.068 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4681 sec 2024-03-24T01:42:34.068 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4682 sec 2024-03-24T01:42:34.393 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4028.163123 secs 2024-03-24T01:42:34.393 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:34 smithi078 ceph-mon[32429]: pgmap v2615: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4028.163123 secs 2024-03-24T01:42:34.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:34 smithi106 ceph-mon[36505]: pgmap v2615: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:34.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4028.163123 secs 2024-03-24T01:42:34.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:34 smithi107 ceph-mon[36485]: pgmap v2615: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:35.068 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4683 sec 2024-03-24T01:42:36.068 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4684 sec 2024-03-24T01:42:36.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:36 smithi106 ceph-mon[36505]: pgmap v2616: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:36.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:36 smithi078 ceph-mon[32429]: pgmap v2616: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:36.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:36 smithi107 ceph-mon[36485]: pgmap v2616: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:37.068 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4685 sec 2024-03-24T01:42:38.068 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4686 sec 2024-03-24T01:42:38.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:38 smithi106 ceph-mon[36505]: pgmap v2617: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:38.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:38 smithi078 ceph-mon[32429]: pgmap v2617: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:38.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:38 smithi107 ceph-mon[36485]: pgmap v2617: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:39.068 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4687 sec 2024-03-24T01:42:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4033.163184 secs 2024-03-24T01:42:39.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4033.163184 secs 2024-03-24T01:42:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4033.163184 secs 2024-03-24T01:42:40.068 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4688 sec 2024-03-24T01:42:40.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:40 smithi106 ceph-mon[36505]: pgmap v2618: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:40.508 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:40 smithi078 ceph-mon[32429]: pgmap v2618: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:40.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:40 smithi107 ceph-mon[36485]: pgmap v2618: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:41.068 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4689 sec 2024-03-24T01:42:42.068 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4690 sec 2024-03-24T01:42:42.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:42 smithi078 ceph-mon[32429]: pgmap v2619: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:42.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:42 smithi107 ceph-mon[36485]: pgmap v2619: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:42.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:42 smithi106 ceph-mon[36505]: pgmap v2619: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:43.068 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4691 sec 2024-03-24T01:42:44.068 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4692 sec 2024-03-24T01:42:44.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4038.163242 secs 2024-03-24T01:42:44.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:44 smithi078 ceph-mon[32429]: pgmap v2620: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4038.163242 secs 2024-03-24T01:42:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:44 smithi107 ceph-mon[36485]: pgmap v2620: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4038.163242 secs 2024-03-24T01:42:44.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:44 smithi106 ceph-mon[36505]: pgmap v2620: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:45.068 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4693 sec 2024-03-24T01:42:46.068 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4694 sec 2024-03-24T01:42:46.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:46 smithi078 ceph-mon[32429]: pgmap v2621: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:46.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:46 smithi107 ceph-mon[36485]: pgmap v2621: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:46.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:46 smithi106 ceph-mon[36505]: pgmap v2621: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:47.069 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4695 sec 2024-03-24T01:42:48.069 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4696 sec 2024-03-24T01:42:48.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:48 smithi078 ceph-mon[32429]: pgmap v2622: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:48.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:48 smithi107 ceph-mon[36485]: pgmap v2622: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:48.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:48 smithi106 ceph-mon[36505]: pgmap v2622: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:49.069 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4697 sec 2024-03-24T01:42:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4043.163323 secs 2024-03-24T01:42:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4043.163323 secs 2024-03-24T01:42:49.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4043.163323 secs 2024-03-24T01:42:50.069 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4698 sec 2024-03-24T01:42:50.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:50 smithi078 ceph-mon[32429]: pgmap v2623: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:50.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:50 smithi107 ceph-mon[36485]: pgmap v2623: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:50.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:50 smithi106 ceph-mon[36505]: pgmap v2623: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:42:51.070 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4699 sec 2024-03-24T01:42:52.069 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4700 sec 2024-03-24T01:42:52.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:52 smithi078 ceph-mon[32429]: pgmap v2624: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:42:52.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:52 smithi107 ceph-mon[36485]: pgmap v2624: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:42:52.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:52 smithi106 ceph-mon[36505]: pgmap v2624: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:42:53.069 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4701 sec 2024-03-24T01:42:53.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:53 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:42:53.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:53 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:42:53.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:53 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:42:54.069 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4702 sec 2024-03-24T01:42:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4048.163382 secs 2024-03-24T01:42:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:54 smithi078 ceph-mon[32429]: pgmap v2625: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:42:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4048.163382 secs 2024-03-24T01:42:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:54 smithi107 ceph-mon[36485]: pgmap v2625: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:42:54.352 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4048.163382 secs 2024-03-24T01:42:54.352 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:54 smithi106 ceph-mon[36505]: pgmap v2625: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:42:55.069 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4703 sec 2024-03-24T01:42:56.069 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4704 sec 2024-03-24T01:42:56.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:56 smithi078 ceph-mon[32429]: pgmap v2626: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:42:56.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:42:56.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:42:56.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:42:56.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:42:56.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:42:56.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:42:56.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:42:56.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:42:56.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:42:56.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:56 smithi107 ceph-mon[36485]: pgmap v2626: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:42:56.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:42:56.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:42:56.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:42:56.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:42:56.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:42:56.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:42:56.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:42:56.596 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:42:56.596 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:42:56.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:56 smithi106 ceph-mon[36505]: pgmap v2626: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:42:56.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:42:56.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:42:56.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:42:56.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:42:56.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:42:56.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:42:56.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.655 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:42:56.655 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:42:56.655 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:42:56.655 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:42:57.070 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4705 sec 2024-03-24T01:42:57.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:57 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T01:42:57.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:57 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T01:42:57.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:57 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T01:42:57.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:57 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T01:42:57.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:57 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T01:42:57.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:57 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T01:42:57.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:57 smithi078 ceph-mon[32429]: pgmap v2627: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:42:57.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:57 smithi078 ceph-mon[32429]: pgmap v2628: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:42:57.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:57 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T01:42:57.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:57 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T01:42:57.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:57 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T01:42:57.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:57 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T01:42:57.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:57 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T01:42:57.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:57 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T01:42:57.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:57 smithi107 ceph-mon[36485]: pgmap v2627: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:42:57.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:57 smithi107 ceph-mon[36485]: pgmap v2628: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:42:57.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:57 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T01:42:57.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:57 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T01:42:57.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:57 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T01:42:57.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:57 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T01:42:57.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:57 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T01:42:57.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:57 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T01:42:57.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:57 smithi106 ceph-mon[36505]: pgmap v2627: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:42:57.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:57 smithi106 ceph-mon[36505]: pgmap v2628: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:42:58.070 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4706 sec 2024-03-24T01:42:59.070 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4707 sec 2024-03-24T01:42:59.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4053.163441 secs 2024-03-24T01:42:59.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:42:59 smithi078 ceph-mon[32429]: pgmap v2629: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:42:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4053.163441 secs 2024-03-24T01:42:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:42:59 smithi107 ceph-mon[36485]: pgmap v2629: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:42:59.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4053.163441 secs 2024-03-24T01:42:59.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:42:59 smithi106 ceph-mon[36505]: pgmap v2629: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:43:00.070 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4708 sec 2024-03-24T01:43:01.070 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4709 sec 2024-03-24T01:43:01.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:01 smithi078 ceph-mon[32429]: pgmap v2630: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:43:01.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:01 smithi107 ceph-mon[36485]: pgmap v2630: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:43:01.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:01 smithi106 ceph-mon[36505]: pgmap v2630: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:43:02.070 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4710 sec 2024-03-24T01:43:03.070 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4711 sec 2024-03-24T01:43:03.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:03 smithi078 ceph-mon[32429]: pgmap v2631: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:43:03.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:03 smithi107 ceph-mon[36485]: pgmap v2631: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:43:03.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:03 smithi106 ceph-mon[36505]: pgmap v2631: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:43:04.070 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4712 sec 2024-03-24T01:43:04.492 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4058.163528 secs 2024-03-24T01:43:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4058.163528 secs 2024-03-24T01:43:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4058.163528 secs 2024-03-24T01:43:05.070 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4713 sec 2024-03-24T01:43:05.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:05 smithi078 ceph-mon[32429]: pgmap v2632: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:43:05.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:05 smithi107 ceph-mon[36485]: pgmap v2632: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:43:05.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:05 smithi106 ceph-mon[36505]: pgmap v2632: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:43:06.070 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4714 sec 2024-03-24T01:43:07.071 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4715 sec 2024-03-24T01:43:07.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:07 smithi078 ceph-mon[32429]: pgmap v2633: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:43:07.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:07 smithi107 ceph-mon[36485]: pgmap v2633: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:43:07.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:07 smithi106 ceph-mon[36505]: pgmap v2633: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:43:08.071 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4716 sec 2024-03-24T01:43:09.071 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4717 sec 2024-03-24T01:43:09.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4063.163614 secs 2024-03-24T01:43:09.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:09 smithi078 ceph-mon[32429]: pgmap v2634: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4063.163614 secs 2024-03-24T01:43:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:09 smithi107 ceph-mon[36485]: pgmap v2634: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4063.163614 secs 2024-03-24T01:43:09.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:09 smithi106 ceph-mon[36505]: pgmap v2634: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:10.071 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4718 sec 2024-03-24T01:43:11.071 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4719 sec 2024-03-24T01:43:11.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:11 smithi078 ceph-mon[32429]: pgmap v2635: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:11.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:11 smithi107 ceph-mon[36485]: pgmap v2635: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:11.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:11 smithi106 ceph-mon[36505]: pgmap v2635: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:12.071 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4720 sec 2024-03-24T01:43:13.071 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4721 sec 2024-03-24T01:43:13.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:13 smithi078 ceph-mon[32429]: pgmap v2636: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:13.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:13 smithi107 ceph-mon[36485]: pgmap v2636: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:13.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:13 smithi106 ceph-mon[36505]: pgmap v2636: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:14.071 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4722 sec 2024-03-24T01:43:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4068.163700 secs 2024-03-24T01:43:14.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4068.163700 secs 2024-03-24T01:43:14.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4068.163700 secs 2024-03-24T01:43:15.071 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4723 sec 2024-03-24T01:43:15.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:15 smithi078 ceph-mon[32429]: pgmap v2637: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:15.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:15 smithi107 ceph-mon[36485]: pgmap v2637: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:15.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:15 smithi106 ceph-mon[36505]: pgmap v2637: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:16.071 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4724 sec 2024-03-24T01:43:16.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:16 smithi078 ceph-mon[32429]: pgmap v2638: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:16.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:16 smithi107 ceph-mon[36485]: pgmap v2638: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:16 smithi106 ceph-mon[36505]: pgmap v2638: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:17.071 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4725 sec 2024-03-24T01:43:18.071 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4726 sec 2024-03-24T01:43:19.072 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4727 sec 2024-03-24T01:43:19.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4073.163782 secs 2024-03-24T01:43:19.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:19 smithi078 ceph-mon[32429]: pgmap v2639: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4073.163782 secs 2024-03-24T01:43:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:19 smithi107 ceph-mon[36485]: pgmap v2639: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4073.163782 secs 2024-03-24T01:43:19.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:19 smithi106 ceph-mon[36505]: pgmap v2639: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:20.072 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4728 sec 2024-03-24T01:43:21.072 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4729 sec 2024-03-24T01:43:21.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:21 smithi078 ceph-mon[32429]: pgmap v2640: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:21.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:21 smithi107 ceph-mon[36485]: pgmap v2640: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:21.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:21 smithi106 ceph-mon[36505]: pgmap v2640: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:22.072 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4730 sec 2024-03-24T01:43:23.072 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4731 sec 2024-03-24T01:43:23.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:23 smithi078 ceph-mon[32429]: pgmap v2641: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:23.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:23 smithi107 ceph-mon[36485]: pgmap v2641: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:23.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:23 smithi106 ceph-mon[36505]: pgmap v2641: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:24.072 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4732 sec 2024-03-24T01:43:24.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4078.163870 secs 2024-03-24T01:43:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4078.163870 secs 2024-03-24T01:43:24.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4078.163870 secs 2024-03-24T01:43:25.072 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4733 sec 2024-03-24T01:43:25.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:25 smithi078 ceph-mon[32429]: pgmap v2642: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:25.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:25 smithi107 ceph-mon[36485]: pgmap v2642: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:25.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:25 smithi106 ceph-mon[36505]: pgmap v2642: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:26.072 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4734 sec 2024-03-24T01:43:27.072 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4735 sec 2024-03-24T01:43:27.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:27 smithi078 ceph-mon[32429]: pgmap v2643: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:27.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:27 smithi107 ceph-mon[36485]: pgmap v2643: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:27.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:27 smithi106 ceph-mon[36505]: pgmap v2643: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:28.072 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4736 sec 2024-03-24T01:43:29.072 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4737 sec 2024-03-24T01:43:29.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4083.163956 secs 2024-03-24T01:43:29.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:29 smithi078 ceph-mon[32429]: pgmap v2644: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4083.163956 secs 2024-03-24T01:43:29.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:29 smithi107 ceph-mon[36485]: pgmap v2644: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:29.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4083.163956 secs 2024-03-24T01:43:29.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:29 smithi106 ceph-mon[36505]: pgmap v2644: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:30.073 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4738 sec 2024-03-24T01:43:31.073 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4739 sec 2024-03-24T01:43:31.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:31 smithi078 ceph-mon[32429]: pgmap v2645: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:31.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:31 smithi107 ceph-mon[36485]: pgmap v2645: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:31.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:31 smithi106 ceph-mon[36505]: pgmap v2645: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:32.073 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4740 sec 2024-03-24T01:43:33.073 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4741 sec 2024-03-24T01:43:33.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:33 smithi078 ceph-mon[32429]: pgmap v2646: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:33.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:33 smithi107 ceph-mon[36485]: pgmap v2646: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:33.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:33 smithi106 ceph-mon[36505]: pgmap v2646: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:34.073 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4742 sec 2024-03-24T01:43:34.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4088.164045 secs 2024-03-24T01:43:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4088.164045 secs 2024-03-24T01:43:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4088.164045 secs 2024-03-24T01:43:35.073 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4743 sec 2024-03-24T01:43:35.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:35 smithi078 ceph-mon[32429]: pgmap v2647: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:35.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:35 smithi107 ceph-mon[36485]: pgmap v2647: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:35.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:35 smithi106 ceph-mon[36505]: pgmap v2647: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:36.073 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4744 sec 2024-03-24T01:43:37.073 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4745 sec 2024-03-24T01:43:37.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:37 smithi078 ceph-mon[32429]: pgmap v2648: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:37.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:37 smithi107 ceph-mon[36485]: pgmap v2648: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:37.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:37 smithi106 ceph-mon[36505]: pgmap v2648: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:38.073 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4746 sec 2024-03-24T01:43:38.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:38 smithi078 ceph-mon[32429]: pgmap v2649: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:38.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:38 smithi107 ceph-mon[36485]: pgmap v2649: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:38.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:38 smithi106 ceph-mon[36505]: pgmap v2649: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:39.074 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4747 sec 2024-03-24T01:43:39.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4093.164137 secs 2024-03-24T01:43:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4093.164137 secs 2024-03-24T01:43:39.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4093.164137 secs 2024-03-24T01:43:40.073 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4748 sec 2024-03-24T01:43:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:40 smithi078 ceph-mon[32429]: pgmap v2650: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:40.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:40 smithi107 ceph-mon[36485]: pgmap v2650: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:40.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:40 smithi106 ceph-mon[36505]: pgmap v2650: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:41.073 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4749 sec 2024-03-24T01:43:42.073 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4750 sec 2024-03-24T01:43:43.074 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4751 sec 2024-03-24T01:43:43.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:43 smithi078 ceph-mon[32429]: pgmap v2651: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:43.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:43 smithi107 ceph-mon[36485]: pgmap v2651: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:43.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:43 smithi106 ceph-mon[36505]: pgmap v2651: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:44.074 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4752 sec 2024-03-24T01:43:44.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4098.164223 secs 2024-03-24T01:43:44.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4098.164223 secs 2024-03-24T01:43:44.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4098.164223 secs 2024-03-24T01:43:45.074 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4753 sec 2024-03-24T01:43:45.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:45 smithi078 ceph-mon[32429]: pgmap v2652: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:45.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:45 smithi107 ceph-mon[36485]: pgmap v2652: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:45.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:45 smithi106 ceph-mon[36505]: pgmap v2652: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:46.074 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4754 sec 2024-03-24T01:43:47.074 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4755 sec 2024-03-24T01:43:47.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:47 smithi078 ceph-mon[32429]: pgmap v2653: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:47 smithi107 ceph-mon[36485]: pgmap v2653: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:47.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:47 smithi106 ceph-mon[36505]: pgmap v2653: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:48.074 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4756 sec 2024-03-24T01:43:49.074 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4757 sec 2024-03-24T01:43:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4103.164312 secs 2024-03-24T01:43:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:49 smithi078 ceph-mon[32429]: pgmap v2654: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4103.164312 secs 2024-03-24T01:43:49.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:49 smithi107 ceph-mon[36485]: pgmap v2654: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:49.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4103.164312 secs 2024-03-24T01:43:49.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:49 smithi106 ceph-mon[36505]: pgmap v2654: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:50.074 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4758 sec 2024-03-24T01:43:51.074 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4759 sec 2024-03-24T01:43:51.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:51 smithi078 ceph-mon[32429]: pgmap v2655: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:51.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:51 smithi107 ceph-mon[36485]: pgmap v2655: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:51.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:51 smithi106 ceph-mon[36505]: pgmap v2655: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:52.074 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4760 sec 2024-03-24T01:43:53.075 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4761 sec 2024-03-24T01:43:53.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:53 smithi078 ceph-mon[32429]: pgmap v2656: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:53.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:53 smithi107 ceph-mon[36485]: pgmap v2656: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:53.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:53 smithi106 ceph-mon[36505]: pgmap v2656: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:54.075 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4762 sec 2024-03-24T01:43:54.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4108.164370 secs 2024-03-24T01:43:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4108.164370 secs 2024-03-24T01:43:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4108.164370 secs 2024-03-24T01:43:55.075 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4763 sec 2024-03-24T01:43:55.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:55 smithi078 ceph-mon[32429]: pgmap v2657: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:55.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:55 smithi107 ceph-mon[36485]: pgmap v2657: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:55.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:55 smithi106 ceph-mon[36505]: pgmap v2657: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:43:56.075 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4764 sec 2024-03-24T01:43:57.075 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4765 sec 2024-03-24T01:43:57.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:57 smithi078 ceph-mon[32429]: pgmap v2658: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:57.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:57 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:43:57.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:57 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:43:57.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:57 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:43:57.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:57 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:43:57.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:57 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:43:57.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:57 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:43:57.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:57 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:43:57.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:57 smithi107 ceph-mon[36485]: pgmap v2658: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:57.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:57 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:43:57.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:57 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:43:57.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:57 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:43:57.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:57 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:43:57.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:57 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:43:57.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:57 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:43:57.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:57 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:43:57.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:57 smithi106 ceph-mon[36505]: pgmap v2658: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:57.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:57 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:43:57.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:57 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:43:57.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:57 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:43:57.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:57 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:43:57.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:57 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:43:57.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:57 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:43:57.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:57 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:43:58.075 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4766 sec 2024-03-24T01:43:58.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:58 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:43:58.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:58 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:43:58.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:58 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:43:58.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:58 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:43:58.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:58 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:43:58.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:58 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:43:58.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:58 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:43:58.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:58 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:43:58.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:58 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:43:59.075 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4767 sec 2024-03-24T01:43:59.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:59 smithi078 ceph-mon[32429]: pgmap v2659: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T01:43:59.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:59 smithi078 ceph-mon[32429]: pgmap v2660: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:43:59.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:43:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4113.164380 secs 2024-03-24T01:43:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:59 smithi107 ceph-mon[36485]: pgmap v2659: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T01:43:59.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:59 smithi107 ceph-mon[36485]: pgmap v2660: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:43:59.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:43:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4113.164380 secs 2024-03-24T01:43:59.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:59 smithi106 ceph-mon[36505]: pgmap v2659: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T01:43:59.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:59 smithi106 ceph-mon[36505]: pgmap v2660: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:43:59.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:43:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4113.164380 secs 2024-03-24T01:44:00.075 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4768 sec 2024-03-24T01:44:00.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:00 smithi078 ceph-mon[32429]: pgmap v2661: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:44:00.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:00 smithi107 ceph-mon[36485]: pgmap v2661: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:44:00.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:00 smithi106 ceph-mon[36505]: pgmap v2661: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:44:01.075 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4769 sec 2024-03-24T01:44:02.075 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4770 sec 2024-03-24T01:44:02.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:02 smithi078 ceph-mon[32429]: pgmap v2662: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:44:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:02 smithi107 ceph-mon[36485]: pgmap v2662: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:44:02.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:02 smithi106 ceph-mon[36505]: pgmap v2662: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:44:03.075 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4771 sec 2024-03-24T01:44:03.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4118.164465 secs 2024-03-24T01:44:03.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4118.164465 secs 2024-03-24T01:44:03.905 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4118.164465 secs 2024-03-24T01:44:04.075 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4772 sec 2024-03-24T01:44:04.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:04 smithi078 ceph-mon[32429]: pgmap v2663: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:44:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:04 smithi107 ceph-mon[36485]: pgmap v2663: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:44:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:04 smithi106 ceph-mon[36505]: pgmap v2663: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:44:05.075 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4773 sec 2024-03-24T01:44:06.075 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4774 sec 2024-03-24T01:44:06.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:06 smithi078 ceph-mon[32429]: pgmap v2664: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:44:06.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:06 smithi107 ceph-mon[36485]: pgmap v2664: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:44:06.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:06 smithi106 ceph-mon[36505]: pgmap v2664: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T01:44:07.076 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4775 sec 2024-03-24T01:44:08.076 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4776 sec 2024-03-24T01:44:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:08 smithi078 ceph-mon[32429]: pgmap v2665: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:44:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4123.164527 secs 2024-03-24T01:44:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:08 smithi107 ceph-mon[36485]: pgmap v2665: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:44:08.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4123.164527 secs 2024-03-24T01:44:08.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:08 smithi106 ceph-mon[36505]: pgmap v2665: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:44:08.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4123.164527 secs 2024-03-24T01:44:09.077 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4777 sec 2024-03-24T01:44:10.076 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4778 sec 2024-03-24T01:44:10.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:10 smithi078 ceph-mon[32429]: pgmap v2666: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:10.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:10 smithi107 ceph-mon[36485]: pgmap v2666: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:10.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:10 smithi106 ceph-mon[36505]: pgmap v2666: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:11.076 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4779 sec 2024-03-24T01:44:12.077 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4780 sec 2024-03-24T01:44:12.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:12 smithi078 ceph-mon[32429]: pgmap v2667: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:12 smithi107 ceph-mon[36485]: pgmap v2667: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:12.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:12 smithi106 ceph-mon[36505]: pgmap v2667: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:13.076 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4781 sec 2024-03-24T01:44:13.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4128.164587 secs 2024-03-24T01:44:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4128.164587 secs 2024-03-24T01:44:13.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4128.164587 secs 2024-03-24T01:44:14.076 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4782 sec 2024-03-24T01:44:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:14 smithi078 ceph-mon[32429]: pgmap v2668: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:14 smithi107 ceph-mon[36485]: pgmap v2668: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:14.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:14 smithi106 ceph-mon[36505]: pgmap v2668: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:15.076 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4783 sec 2024-03-24T01:44:16.076 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4784 sec 2024-03-24T01:44:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:16 smithi078 ceph-mon[32429]: pgmap v2669: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:16.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:16 smithi107 ceph-mon[36485]: pgmap v2669: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:16.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:16 smithi106 ceph-mon[36505]: pgmap v2669: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:17.077 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4785 sec 2024-03-24T01:44:18.077 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4786 sec 2024-03-24T01:44:18.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:18 smithi078 ceph-mon[32429]: pgmap v2670: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:18.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4133.164689 secs 2024-03-24T01:44:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:18 smithi107 ceph-mon[36485]: pgmap v2670: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4133.164689 secs 2024-03-24T01:44:18.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:18 smithi106 ceph-mon[36505]: pgmap v2670: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:18.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4133.164689 secs 2024-03-24T01:44:19.077 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4787 sec 2024-03-24T01:44:20.077 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4788 sec 2024-03-24T01:44:20.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:20 smithi078 ceph-mon[32429]: pgmap v2671: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:20 smithi107 ceph-mon[36485]: pgmap v2671: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:20.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:20 smithi106 ceph-mon[36505]: pgmap v2671: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:21.077 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4789 sec 2024-03-24T01:44:22.077 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4790 sec 2024-03-24T01:44:22.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:22 smithi078 ceph-mon[32429]: pgmap v2672: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:22.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:22 smithi107 ceph-mon[36485]: pgmap v2672: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:22.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:22 smithi106 ceph-mon[36505]: pgmap v2672: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:23.077 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4791 sec 2024-03-24T01:44:23.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4138.164750 secs 2024-03-24T01:44:23.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4138.164750 secs 2024-03-24T01:44:23.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4138.164750 secs 2024-03-24T01:44:24.077 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4792 sec 2024-03-24T01:44:24.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:24 smithi078 ceph-mon[32429]: pgmap v2673: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:24 smithi107 ceph-mon[36485]: pgmap v2673: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:24 smithi106 ceph-mon[36505]: pgmap v2673: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:25.077 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4793 sec 2024-03-24T01:44:26.077 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4794 sec 2024-03-24T01:44:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:26 smithi078 ceph-mon[32429]: pgmap v2674: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:26 smithi107 ceph-mon[36485]: pgmap v2674: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:26.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:26 smithi106 ceph-mon[36505]: pgmap v2674: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:27.077 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4795 sec 2024-03-24T01:44:28.077 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4796 sec 2024-03-24T01:44:28.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:28 smithi078 ceph-mon[32429]: pgmap v2675: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:28.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4143.164809 secs 2024-03-24T01:44:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:28 smithi107 ceph-mon[36485]: pgmap v2675: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4143.164809 secs 2024-03-24T01:44:28.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:28 smithi106 ceph-mon[36505]: pgmap v2675: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:28.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4143.164809 secs 2024-03-24T01:44:29.078 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4797 sec 2024-03-24T01:44:30.078 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4798 sec 2024-03-24T01:44:31.078 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4799 sec 2024-03-24T01:44:31.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:31 smithi078 ceph-mon[32429]: pgmap v2676: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:31.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:31 smithi107 ceph-mon[36485]: pgmap v2676: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:31.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:31 smithi106 ceph-mon[36505]: pgmap v2676: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:32.078 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4800 sec 2024-03-24T01:44:33.078 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4801 sec 2024-03-24T01:44:33.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:33 smithi078 ceph-mon[32429]: pgmap v2677: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:33.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:33 smithi107 ceph-mon[36485]: pgmap v2677: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:33.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:33 smithi106 ceph-mon[36505]: pgmap v2677: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:34.078 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4802 sec 2024-03-24T01:44:34.492 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4148.164868 secs 2024-03-24T01:44:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4148.164868 secs 2024-03-24T01:44:34.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4148.164868 secs 2024-03-24T01:44:35.078 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4803 sec 2024-03-24T01:44:35.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:35 smithi078 ceph-mon[32429]: pgmap v2678: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:35.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:35 smithi107 ceph-mon[36485]: pgmap v2678: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:35.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:35 smithi106 ceph-mon[36505]: pgmap v2678: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:36.078 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4804 sec 2024-03-24T01:44:37.078 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4805 sec 2024-03-24T01:44:37.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:37 smithi078 ceph-mon[32429]: pgmap v2679: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:37.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:37 smithi107 ceph-mon[36485]: pgmap v2679: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:37.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:37 smithi106 ceph-mon[36505]: pgmap v2679: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:38.078 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4806 sec 2024-03-24T01:44:39.078 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4807 sec 2024-03-24T01:44:39.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:39 smithi078 ceph-mon[32429]: pgmap v2680: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:39.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4153.164929 secs 2024-03-24T01:44:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:39 smithi107 ceph-mon[36485]: pgmap v2680: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:39.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4153.164929 secs 2024-03-24T01:44:39.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:39 smithi106 ceph-mon[36505]: pgmap v2680: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:39.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4153.164929 secs 2024-03-24T01:44:40.078 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4808 sec 2024-03-24T01:44:41.078 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4809 sec 2024-03-24T01:44:41.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:41 smithi078 ceph-mon[32429]: pgmap v2681: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:41.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:41 smithi107 ceph-mon[36485]: pgmap v2681: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:41.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:41 smithi106 ceph-mon[36505]: pgmap v2681: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:42.079 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4810 sec 2024-03-24T01:44:43.079 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4811 sec 2024-03-24T01:44:43.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:43 smithi078 ceph-mon[32429]: pgmap v2682: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:43.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:43 smithi107 ceph-mon[36485]: pgmap v2682: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:43.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:43 smithi106 ceph-mon[36505]: pgmap v2682: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:44.079 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4812 sec 2024-03-24T01:44:44.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4158.165020 secs 2024-03-24T01:44:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4158.165020 secs 2024-03-24T01:44:44.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4158.165020 secs 2024-03-24T01:44:45.079 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4813 sec 2024-03-24T01:44:45.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:45 smithi078 ceph-mon[32429]: pgmap v2683: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:45.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:45 smithi107 ceph-mon[36485]: pgmap v2683: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:45.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:45 smithi106 ceph-mon[36505]: pgmap v2683: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:46.079 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4814 sec 2024-03-24T01:44:47.079 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4815 sec 2024-03-24T01:44:47.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:47 smithi078 ceph-mon[32429]: pgmap v2684: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:47.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:47 smithi107 ceph-mon[36485]: pgmap v2684: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:47.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:47 smithi106 ceph-mon[36505]: pgmap v2684: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:48.079 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4816 sec 2024-03-24T01:44:49.079 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4817 sec 2024-03-24T01:44:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:49 smithi078 ceph-mon[32429]: pgmap v2685: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4163.165109 secs 2024-03-24T01:44:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:49 smithi107 ceph-mon[36485]: pgmap v2685: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:49.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4163.165109 secs 2024-03-24T01:44:49.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:49 smithi106 ceph-mon[36505]: pgmap v2685: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:49.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4163.165109 secs 2024-03-24T01:44:50.079 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4818 sec 2024-03-24T01:44:51.079 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4819 sec 2024-03-24T01:44:51.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:51 smithi078 ceph-mon[32429]: pgmap v2686: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:51.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:51 smithi107 ceph-mon[36485]: pgmap v2686: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:51.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:51 smithi106 ceph-mon[36505]: pgmap v2686: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:52.079 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4820 sec 2024-03-24T01:44:53.080 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4821 sec 2024-03-24T01:44:53.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:53 smithi078 ceph-mon[32429]: pgmap v2687: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:53.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:53 smithi107 ceph-mon[36485]: pgmap v2687: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:53.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:53 smithi106 ceph-mon[36505]: pgmap v2687: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:54.080 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4822 sec 2024-03-24T01:44:54.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4168.165197 secs 2024-03-24T01:44:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4168.165197 secs 2024-03-24T01:44:54.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4168.165197 secs 2024-03-24T01:44:55.080 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4823 sec 2024-03-24T01:44:55.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:55 smithi078 ceph-mon[32429]: pgmap v2688: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:55.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:55 smithi107 ceph-mon[36485]: pgmap v2688: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:55.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:55 smithi106 ceph-mon[36505]: pgmap v2688: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:56.080 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4824 sec 2024-03-24T01:44:56.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:56 smithi078 ceph-mon[32429]: pgmap v2689: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:56.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:56 smithi107 ceph-mon[36485]: pgmap v2689: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:56.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:56 smithi106 ceph-mon[36505]: pgmap v2689: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:57.080 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4825 sec 2024-03-24T01:44:58.080 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4826 sec 2024-03-24T01:44:58.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:58 smithi078 ceph-mon[32429]: pgmap v2690: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:58.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:58 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:44:58.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4173.165263 secs 2024-03-24T01:44:58.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:58 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:44:58.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:58 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:44:58.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:58 smithi078 ceph-mon[32429]: pgmap v2691: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:44:58.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:58 smithi078 ceph-mon[32429]: pgmap v2692: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:44:58.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:44:58 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:44:58.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:58 smithi107 ceph-mon[36485]: pgmap v2690: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:58.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:58 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:44:58.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4173.165263 secs 2024-03-24T01:44:58.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:58 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:44:58.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:58 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:44:58.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:58 smithi107 ceph-mon[36485]: pgmap v2691: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:44:58.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:58 smithi107 ceph-mon[36485]: pgmap v2692: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:44:58.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:44:58 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:44:58.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:58 smithi106 ceph-mon[36505]: pgmap v2690: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:44:58.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:58 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:44:58.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4173.165263 secs 2024-03-24T01:44:58.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:58 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:44:58.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:58 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:44:58.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:58 smithi106 ceph-mon[36505]: pgmap v2691: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:44:58.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:58 smithi106 ceph-mon[36505]: pgmap v2692: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:44:58.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:44:58 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:44:59.080 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4827 sec 2024-03-24T01:45:00.081 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4828 sec 2024-03-24T01:45:01.080 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4829 sec 2024-03-24T01:45:01.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:01 smithi078 ceph-mon[32429]: pgmap v2693: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:45:01.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:01 smithi107 ceph-mon[36485]: pgmap v2693: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:45:01.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:01 smithi106 ceph-mon[36505]: pgmap v2693: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:45:02.080 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4830 sec 2024-03-24T01:45:03.080 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4831 sec 2024-03-24T01:45:03.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:03 smithi078 ceph-mon[32429]: pgmap v2694: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:45:03.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:03 smithi107 ceph-mon[36485]: pgmap v2694: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:45:03.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:03 smithi106 ceph-mon[36505]: pgmap v2694: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:45:04.080 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4832 sec 2024-03-24T01:45:04.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4178.165320 secs 2024-03-24T01:45:04.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4178.165320 secs 2024-03-24T01:45:04.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4178.165320 secs 2024-03-24T01:45:05.080 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4833 sec 2024-03-24T01:45:05.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:05 smithi078 ceph-mon[32429]: pgmap v2695: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:45:05.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:05 smithi107 ceph-mon[36485]: pgmap v2695: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:45:05.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:05 smithi106 ceph-mon[36505]: pgmap v2695: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:45:06.081 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4834 sec 2024-03-24T01:45:07.081 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4835 sec 2024-03-24T01:45:07.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:07 smithi078 ceph-mon[32429]: pgmap v2696: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:45:07.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:07 smithi107 ceph-mon[36485]: pgmap v2696: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:45:07.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:07 smithi106 ceph-mon[36505]: pgmap v2696: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:45:08.081 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4836 sec 2024-03-24T01:45:09.081 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4837 sec 2024-03-24T01:45:09.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4183.165383 secs 2024-03-24T01:45:09.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:09 smithi078 ceph-mon[32429]: pgmap v2697: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:45:09.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4183.165383 secs 2024-03-24T01:45:09.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:09 smithi107 ceph-mon[36485]: pgmap v2697: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:45:09.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4183.165383 secs 2024-03-24T01:45:09.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:09 smithi106 ceph-mon[36505]: pgmap v2697: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:45:10.081 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4838 sec 2024-03-24T01:45:11.081 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4839 sec 2024-03-24T01:45:11.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:11 smithi078 ceph-mon[32429]: pgmap v2698: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:11.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:11 smithi107 ceph-mon[36485]: pgmap v2698: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:11.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:11 smithi106 ceph-mon[36505]: pgmap v2698: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:12.081 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4840 sec 2024-03-24T01:45:12.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:12 smithi078 ceph-mon[32429]: pgmap v2699: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:12.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:12 smithi107 ceph-mon[36485]: pgmap v2699: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:12.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:12 smithi106 ceph-mon[36505]: pgmap v2699: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:13.081 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4841 sec 2024-03-24T01:45:13.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4188.165442 secs 2024-03-24T01:45:13.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4188.165442 secs 2024-03-24T01:45:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4188.165442 secs 2024-03-24T01:45:14.081 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4842 sec 2024-03-24T01:45:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:14 smithi078 ceph-mon[32429]: pgmap v2700: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:14 smithi107 ceph-mon[36485]: pgmap v2700: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:14 smithi106 ceph-mon[36505]: pgmap v2700: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:15.082 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4843 sec 2024-03-24T01:45:16.081 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4844 sec 2024-03-24T01:45:17.082 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4845 sec 2024-03-24T01:45:17.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:17 smithi078 ceph-mon[32429]: pgmap v2701: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:17.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:17 smithi107 ceph-mon[36485]: pgmap v2701: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:17.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:17 smithi106 ceph-mon[36505]: pgmap v2701: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:18.082 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4846 sec 2024-03-24T01:45:19.082 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4847 sec 2024-03-24T01:45:19.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4193.165502 secs 2024-03-24T01:45:19.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:19 smithi078 ceph-mon[32429]: pgmap v2702: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:19.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4193.165502 secs 2024-03-24T01:45:19.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:19 smithi107 ceph-mon[36485]: pgmap v2702: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:19.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4193.165502 secs 2024-03-24T01:45:19.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:19 smithi106 ceph-mon[36505]: pgmap v2702: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:20.082 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4848 sec 2024-03-24T01:45:21.082 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4849 sec 2024-03-24T01:45:21.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:21 smithi078 ceph-mon[32429]: pgmap v2703: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:21.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:21 smithi107 ceph-mon[36485]: pgmap v2703: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:21.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:21 smithi106 ceph-mon[36505]: pgmap v2703: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:22.082 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4850 sec 2024-03-24T01:45:23.082 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4851 sec 2024-03-24T01:45:23.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:23 smithi078 ceph-mon[32429]: pgmap v2704: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:23.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:23 smithi107 ceph-mon[36485]: pgmap v2704: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:23.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:23 smithi106 ceph-mon[36505]: pgmap v2704: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:24.082 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4852 sec 2024-03-24T01:45:24.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4198.165585 secs 2024-03-24T01:45:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4198.165585 secs 2024-03-24T01:45:24.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4198.165585 secs 2024-03-24T01:45:25.082 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4853 sec 2024-03-24T01:45:25.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:25 smithi078 ceph-mon[32429]: pgmap v2705: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:25.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:25 smithi107 ceph-mon[36485]: pgmap v2705: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:25.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:25 smithi106 ceph-mon[36505]: pgmap v2705: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:26.082 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4854 sec 2024-03-24T01:45:26.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:26 smithi078 ceph-mon[32429]: pgmap v2706: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:26.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:26 smithi107 ceph-mon[36485]: pgmap v2706: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:26 smithi106 ceph-mon[36505]: pgmap v2706: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:27.082 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4855 sec 2024-03-24T01:45:28.082 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4856 sec 2024-03-24T01:45:29.082 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4857 sec 2024-03-24T01:45:29.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4203.165636 secs 2024-03-24T01:45:29.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:29 smithi078 ceph-mon[32429]: pgmap v2707: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4203.165636 secs 2024-03-24T01:45:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:29 smithi107 ceph-mon[36485]: pgmap v2707: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:29.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4203.165636 secs 2024-03-24T01:45:29.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:29 smithi106 ceph-mon[36505]: pgmap v2707: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:30.082 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4858 sec 2024-03-24T01:45:31.083 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4859 sec 2024-03-24T01:45:31.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:31 smithi078 ceph-mon[32429]: pgmap v2708: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:31.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:31 smithi107 ceph-mon[36485]: pgmap v2708: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:31.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:31 smithi106 ceph-mon[36505]: pgmap v2708: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:32.083 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4860 sec 2024-03-24T01:45:33.083 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4861 sec 2024-03-24T01:45:33.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:33 smithi078 ceph-mon[32429]: pgmap v2709: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:33.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:33 smithi107 ceph-mon[36485]: pgmap v2709: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:33.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:33 smithi106 ceph-mon[36505]: pgmap v2709: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:34.083 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4862 sec 2024-03-24T01:45:34.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4208.165723 secs 2024-03-24T01:45:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4208.165723 secs 2024-03-24T01:45:34.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4208.165723 secs 2024-03-24T01:45:35.083 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4863 sec 2024-03-24T01:45:35.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:35 smithi078 ceph-mon[32429]: pgmap v2710: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:35.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:35 smithi107 ceph-mon[36485]: pgmap v2710: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:35.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:35 smithi106 ceph-mon[36505]: pgmap v2710: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:36.083 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4864 sec 2024-03-24T01:45:36.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:36 smithi078 ceph-mon[32429]: pgmap v2711: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:36.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:36 smithi107 ceph-mon[36485]: pgmap v2711: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:36.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:36 smithi106 ceph-mon[36505]: pgmap v2711: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:37.083 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4865 sec 2024-03-24T01:45:38.083 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4866 sec 2024-03-24T01:45:39.083 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4867 sec 2024-03-24T01:45:39.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4213.165811 secs 2024-03-24T01:45:39.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:39 smithi078 ceph-mon[32429]: pgmap v2712: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4213.165811 secs 2024-03-24T01:45:39.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:39 smithi107 ceph-mon[36485]: pgmap v2712: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:39.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4213.165811 secs 2024-03-24T01:45:39.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:39 smithi106 ceph-mon[36505]: pgmap v2712: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:40.083 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4868 sec 2024-03-24T01:45:41.083 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4869 sec 2024-03-24T01:45:41.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:41 smithi078 ceph-mon[32429]: pgmap v2713: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:41.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:41 smithi107 ceph-mon[36485]: pgmap v2713: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:41.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:41 smithi106 ceph-mon[36505]: pgmap v2713: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:42.084 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4870 sec 2024-03-24T01:45:43.084 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4871 sec 2024-03-24T01:45:43.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:43 smithi078 ceph-mon[32429]: pgmap v2714: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:43.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:43 smithi107 ceph-mon[36485]: pgmap v2714: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:43.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:43 smithi106 ceph-mon[36505]: pgmap v2714: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:44.084 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4872 sec 2024-03-24T01:45:44.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4218.165868 secs 2024-03-24T01:45:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4218.165868 secs 2024-03-24T01:45:44.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4218.165868 secs 2024-03-24T01:45:45.084 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4873 sec 2024-03-24T01:45:45.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:45 smithi078 ceph-mon[32429]: pgmap v2715: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:45.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:45 smithi107 ceph-mon[36485]: pgmap v2715: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:45.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:45 smithi106 ceph-mon[36505]: pgmap v2715: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:46.084 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4874 sec 2024-03-24T01:45:46.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:46 smithi078 ceph-mon[32429]: pgmap v2716: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:46.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:46 smithi107 ceph-mon[36485]: pgmap v2716: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:46.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:46 smithi106 ceph-mon[36505]: pgmap v2716: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:47.084 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4875 sec 2024-03-24T01:45:48.084 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4876 sec 2024-03-24T01:45:49.084 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4877 sec 2024-03-24T01:45:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4223.165925 secs 2024-03-24T01:45:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:49 smithi078 ceph-mon[32429]: pgmap v2717: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4223.165925 secs 2024-03-24T01:45:49.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:49 smithi107 ceph-mon[36485]: pgmap v2717: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:49.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4223.165925 secs 2024-03-24T01:45:49.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:49 smithi106 ceph-mon[36505]: pgmap v2717: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:50.084 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4878 sec 2024-03-24T01:45:51.084 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4879 sec 2024-03-24T01:45:51.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:51 smithi078 ceph-mon[32429]: pgmap v2718: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:51.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:51 smithi107 ceph-mon[36485]: pgmap v2718: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:51.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:51 smithi106 ceph-mon[36505]: pgmap v2718: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:52.084 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4880 sec 2024-03-24T01:45:53.084 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4881 sec 2024-03-24T01:45:53.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:53 smithi078 ceph-mon[32429]: pgmap v2719: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:53.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:53 smithi107 ceph-mon[36485]: pgmap v2719: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:53.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:53 smithi106 ceph-mon[36505]: pgmap v2719: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:54.085 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4882 sec 2024-03-24T01:45:54.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4228.166013 secs 2024-03-24T01:45:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4228.166013 secs 2024-03-24T01:45:54.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4228.166013 secs 2024-03-24T01:45:55.085 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4883 sec 2024-03-24T01:45:55.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:55 smithi078 ceph-mon[32429]: pgmap v2720: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:55.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:55 smithi107 ceph-mon[36485]: pgmap v2720: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:55.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:55 smithi106 ceph-mon[36505]: pgmap v2720: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:56.085 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4884 sec 2024-03-24T01:45:56.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:56 smithi078 ceph-mon[32429]: pgmap v2721: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:56.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:56 smithi107 ceph-mon[36485]: pgmap v2721: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:56.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:56 smithi106 ceph-mon[36505]: pgmap v2721: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:57.085 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4885 sec 2024-03-24T01:45:58.085 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4886 sec 2024-03-24T01:45:59.085 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4887 sec 2024-03-24T01:45:59.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4233.166073 secs 2024-03-24T01:45:59.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:59 smithi078 ceph-mon[32429]: pgmap v2722: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:59.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:59 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:45:59.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:59 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:45:59.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:59 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:45:59.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:45:59 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:45:59.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4233.166073 secs 2024-03-24T01:45:59.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:59 smithi107 ceph-mon[36485]: pgmap v2722: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:59.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:59 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:45:59.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:59 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:45:59.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:59 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:45:59.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:45:59 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:45:59.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4233.166073 secs 2024-03-24T01:45:59.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:59 smithi106 ceph-mon[36505]: pgmap v2722: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:45:59.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:59 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:45:59.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:59 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:45:59.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:59 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:45:59.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:45:59 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:46:00.085 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4888 sec 2024-03-24T01:46:00.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:00 smithi078 ceph-mon[32429]: pgmap v2723: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:46:00.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:00 smithi078 ceph-mon[32429]: pgmap v2724: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:46:00.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:00 smithi107 ceph-mon[36485]: pgmap v2723: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:46:00.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:00 smithi107 ceph-mon[36485]: pgmap v2724: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:46:00.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:00 smithi106 ceph-mon[36505]: pgmap v2723: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:46:00.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:00 smithi106 ceph-mon[36505]: pgmap v2724: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:46:01.085 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4889 sec 2024-03-24T01:46:01.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:01 smithi078 ceph-mon[32429]: pgmap v2725: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:46:01.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:01 smithi107 ceph-mon[36485]: pgmap v2725: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:46:01.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:01 smithi106 ceph-mon[36505]: pgmap v2725: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:46:02.085 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4890 sec 2024-03-24T01:46:03.085 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4891 sec 2024-03-24T01:46:04.085 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4892 sec 2024-03-24T01:46:04.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4238.166159 secs 2024-03-24T01:46:04.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:04 smithi078 ceph-mon[32429]: pgmap v2726: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:46:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4238.166159 secs 2024-03-24T01:46:04.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:04 smithi107 ceph-mon[36485]: pgmap v2726: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:46:04.355 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4238.166159 secs 2024-03-24T01:46:04.355 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:04 smithi106 ceph-mon[36505]: pgmap v2726: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:46:05.086 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4893 sec 2024-03-24T01:46:06.086 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4894 sec 2024-03-24T01:46:06.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:06 smithi078 ceph-mon[32429]: pgmap v2727: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:46:06.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:06 smithi107 ceph-mon[36485]: pgmap v2727: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:46:06.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:06 smithi106 ceph-mon[36505]: pgmap v2727: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:46:07.086 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4895 sec 2024-03-24T01:46:08.086 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4896 sec 2024-03-24T01:46:08.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:08 smithi078 ceph-mon[32429]: pgmap v2728: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:46:08.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:08 smithi107 ceph-mon[36485]: pgmap v2728: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:46:08.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:08 smithi106 ceph-mon[36505]: pgmap v2728: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:46:09.086 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4897 sec 2024-03-24T01:46:09.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4243.166249 secs 2024-03-24T01:46:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4243.166249 secs 2024-03-24T01:46:09.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4243.166249 secs 2024-03-24T01:46:10.086 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4898 sec 2024-03-24T01:46:10.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:10 smithi078 ceph-mon[32429]: pgmap v2729: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:46:10.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:10 smithi107 ceph-mon[36485]: pgmap v2729: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:46:10.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:10 smithi106 ceph-mon[36505]: pgmap v2729: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:46:11.086 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4899 sec 2024-03-24T01:46:12.086 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4900 sec 2024-03-24T01:46:12.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:12 smithi078 ceph-mon[32429]: pgmap v2730: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:12.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:12 smithi107 ceph-mon[36485]: pgmap v2730: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:12.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:12 smithi106 ceph-mon[36505]: pgmap v2730: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:13.086 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4901 sec 2024-03-24T01:46:14.086 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4902 sec 2024-03-24T01:46:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4248.166322 secs 2024-03-24T01:46:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:14 smithi078 ceph-mon[32429]: pgmap v2731: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:14.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4248.166322 secs 2024-03-24T01:46:14.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:14 smithi107 ceph-mon[36485]: pgmap v2731: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:14.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4248.166322 secs 2024-03-24T01:46:14.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:14 smithi106 ceph-mon[36505]: pgmap v2731: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:15.087 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4903 sec 2024-03-24T01:46:16.087 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4904 sec 2024-03-24T01:46:16.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:16 smithi078 ceph-mon[32429]: pgmap v2732: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:16.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:16 smithi107 ceph-mon[36485]: pgmap v2732: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:16.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:16 smithi106 ceph-mon[36505]: pgmap v2732: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:17.087 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4905 sec 2024-03-24T01:46:18.087 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4906 sec 2024-03-24T01:46:18.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:18 smithi078 ceph-mon[32429]: pgmap v2733: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:18.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:18 smithi107 ceph-mon[36485]: pgmap v2733: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:18.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:18 smithi106 ceph-mon[36505]: pgmap v2733: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:19.087 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4907 sec 2024-03-24T01:46:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4253.166381 secs 2024-03-24T01:46:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4253.166381 secs 2024-03-24T01:46:19.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4253.166381 secs 2024-03-24T01:46:20.087 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4908 sec 2024-03-24T01:46:20.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:20 smithi078 ceph-mon[32429]: pgmap v2734: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:20 smithi107 ceph-mon[36485]: pgmap v2734: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:20.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:20 smithi106 ceph-mon[36505]: pgmap v2734: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:21.087 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4909 sec 2024-03-24T01:46:22.087 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4910 sec 2024-03-24T01:46:22.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:22 smithi078 ceph-mon[32429]: pgmap v2735: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:22.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:22 smithi107 ceph-mon[36485]: pgmap v2735: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:22.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:22 smithi106 ceph-mon[36505]: pgmap v2735: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:23.088 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4911 sec 2024-03-24T01:46:24.088 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4912 sec 2024-03-24T01:46:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4258.166463 secs 2024-03-24T01:46:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:24 smithi078 ceph-mon[32429]: pgmap v2736: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4258.166463 secs 2024-03-24T01:46:24.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:24 smithi107 ceph-mon[36485]: pgmap v2736: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:24.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4258.166463 secs 2024-03-24T01:46:24.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:24 smithi106 ceph-mon[36505]: pgmap v2736: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:25.087 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4913 sec 2024-03-24T01:46:26.087 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4914 sec 2024-03-24T01:46:26.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:26 smithi078 ceph-mon[32429]: pgmap v2737: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:26.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:26 smithi107 ceph-mon[36485]: pgmap v2737: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:26.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:26 smithi106 ceph-mon[36505]: pgmap v2737: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:27.088 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4915 sec 2024-03-24T01:46:28.088 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4916 sec 2024-03-24T01:46:28.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:28 smithi078 ceph-mon[32429]: pgmap v2738: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:28.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:28 smithi107 ceph-mon[36485]: pgmap v2738: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:28.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:28 smithi106 ceph-mon[36505]: pgmap v2738: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:29.088 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4917 sec 2024-03-24T01:46:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4263.166551 secs 2024-03-24T01:46:29.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4263.166551 secs 2024-03-24T01:46:29.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4263.166551 secs 2024-03-24T01:46:30.088 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4918 sec 2024-03-24T01:46:30.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:30 smithi078 ceph-mon[32429]: pgmap v2739: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:30.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:30 smithi107 ceph-mon[36485]: pgmap v2739: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:30.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:30 smithi106 ceph-mon[36505]: pgmap v2739: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:31.088 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4919 sec 2024-03-24T01:46:32.088 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4920 sec 2024-03-24T01:46:32.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:32 smithi078 ceph-mon[32429]: pgmap v2740: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:32.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:32 smithi107 ceph-mon[36485]: pgmap v2740: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:32.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:32 smithi106 ceph-mon[36505]: pgmap v2740: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:33.088 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4921 sec 2024-03-24T01:46:34.088 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4922 sec 2024-03-24T01:46:34.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4268.166640 secs 2024-03-24T01:46:34.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:34 smithi107 ceph-mon[36485]: pgmap v2741: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:34.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4268.166640 secs 2024-03-24T01:46:34.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:34 smithi106 ceph-mon[36505]: pgmap v2741: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:34.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4268.166640 secs 2024-03-24T01:46:34.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:34 smithi078 ceph-mon[32429]: pgmap v2741: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:35.088 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4923 sec 2024-03-24T01:46:36.088 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4924 sec 2024-03-24T01:46:36.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:36 smithi106 ceph-mon[36505]: pgmap v2742: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:36.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:36 smithi078 ceph-mon[32429]: pgmap v2742: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:36.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:36 smithi107 ceph-mon[36485]: pgmap v2742: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:37.088 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4925 sec 2024-03-24T01:46:38.088 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4926 sec 2024-03-24T01:46:38.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:38 smithi106 ceph-mon[36505]: pgmap v2743: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:38.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:38 smithi078 ceph-mon[32429]: pgmap v2743: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:38.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:38 smithi107 ceph-mon[36485]: pgmap v2743: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:39.089 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4927 sec 2024-03-24T01:46:39.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4273.166701 secs 2024-03-24T01:46:39.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4273.166701 secs 2024-03-24T01:46:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4273.166701 secs 2024-03-24T01:46:40.089 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4928 sec 2024-03-24T01:46:40.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:40 smithi106 ceph-mon[36505]: pgmap v2744: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:40 smithi078 ceph-mon[32429]: pgmap v2744: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:40.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:40 smithi107 ceph-mon[36485]: pgmap v2744: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:41.089 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4929 sec 2024-03-24T01:46:42.089 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4930 sec 2024-03-24T01:46:42.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:42 smithi106 ceph-mon[36505]: pgmap v2745: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:42.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:42 smithi078 ceph-mon[32429]: pgmap v2745: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:42.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:42 smithi107 ceph-mon[36485]: pgmap v2745: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:43.089 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4931 sec 2024-03-24T01:46:44.089 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4932 sec 2024-03-24T01:46:44.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4278.166762 secs 2024-03-24T01:46:44.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:44 smithi106 ceph-mon[36505]: pgmap v2746: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:44.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4278.166762 secs 2024-03-24T01:46:44.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:44 smithi078 ceph-mon[32429]: pgmap v2746: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:44.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4278.166762 secs 2024-03-24T01:46:44.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:44 smithi107 ceph-mon[36485]: pgmap v2746: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:45.089 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4933 sec 2024-03-24T01:46:46.089 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4934 sec 2024-03-24T01:46:46.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:46 smithi106 ceph-mon[36505]: pgmap v2747: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:46.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:46 smithi078 ceph-mon[32429]: pgmap v2747: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:46.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:46 smithi107 ceph-mon[36485]: pgmap v2747: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:47.089 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4935 sec 2024-03-24T01:46:48.089 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4936 sec 2024-03-24T01:46:48.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:48 smithi106 ceph-mon[36505]: pgmap v2748: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:48.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:48 smithi078 ceph-mon[32429]: pgmap v2748: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:48.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:48 smithi107 ceph-mon[36485]: pgmap v2748: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:49.090 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4937 sec 2024-03-24T01:46:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4283.166848 secs 2024-03-24T01:46:49.495 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4283.166848 secs 2024-03-24T01:46:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4283.166848 secs 2024-03-24T01:46:50.090 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4938 sec 2024-03-24T01:46:50.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:50 smithi106 ceph-mon[36505]: pgmap v2749: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:50.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:50 smithi078 ceph-mon[32429]: pgmap v2749: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:50.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:50 smithi107 ceph-mon[36485]: pgmap v2749: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:51.090 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4939 sec 2024-03-24T01:46:52.090 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4940 sec 2024-03-24T01:46:52.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:52 smithi078 ceph-mon[32429]: pgmap v2750: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:52.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:52 smithi107 ceph-mon[36485]: pgmap v2750: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:52.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:52 smithi106 ceph-mon[36505]: pgmap v2750: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:53.090 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4941 sec 2024-03-24T01:46:54.090 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4942 sec 2024-03-24T01:46:54.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4288.166936 secs 2024-03-24T01:46:54.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:54 smithi078 ceph-mon[32429]: pgmap v2751: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4288.166936 secs 2024-03-24T01:46:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:54 smithi107 ceph-mon[36485]: pgmap v2751: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:54.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4288.166936 secs 2024-03-24T01:46:54.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:54 smithi106 ceph-mon[36505]: pgmap v2751: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:55.090 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4943 sec 2024-03-24T01:46:56.090 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4944 sec 2024-03-24T01:46:56.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:56 smithi078 ceph-mon[32429]: pgmap v2752: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:56.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:56 smithi107 ceph-mon[36485]: pgmap v2752: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:56.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:56 smithi106 ceph-mon[36505]: pgmap v2752: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:46:57.090 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4945 sec 2024-03-24T01:46:58.090 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4946 sec 2024-03-24T01:46:58.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:58 smithi078 ceph-mon[32429]: pgmap v2753: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:58.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:58 smithi107 ceph-mon[36485]: pgmap v2753: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:58.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:58 smithi106 ceph-mon[36505]: pgmap v2753: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:59.090 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4947 sec 2024-03-24T01:46:59.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4293.167026 secs 2024-03-24T01:46:59.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:46:59 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:46:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4293.167026 secs 2024-03-24T01:46:59.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:46:59 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:46:59.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4293.167026 secs 2024-03-24T01:46:59.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:46:59 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:47:00.090 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4948 sec 2024-03-24T01:47:00.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:00 smithi078 ceph-mon[32429]: pgmap v2754: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:00.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:00 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:47:00.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:00 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:47:00.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:00 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:47:00.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:00 smithi107 ceph-mon[36485]: pgmap v2754: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:00.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:00 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:47:00.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:00 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:47:00.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:00 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:47:00.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:00 smithi106 ceph-mon[36505]: pgmap v2754: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:00.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:00 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:47:00.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:00 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:47:00.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:00 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:47:01.090 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4949 sec 2024-03-24T01:47:01.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:01 smithi078 ceph-mon[32429]: pgmap v2755: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:47:01.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:01 smithi107 ceph-mon[36485]: pgmap v2755: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:47:01.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:01 smithi106 ceph-mon[36505]: pgmap v2755: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:47:02.091 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4950 sec 2024-03-24T01:47:02.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:02 smithi106 ceph-mon[36505]: pgmap v2756: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:47:02.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:02 smithi078 ceph-mon[32429]: pgmap v2756: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:47:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:02 smithi107 ceph-mon[36485]: pgmap v2756: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:47:03.091 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4951 sec 2024-03-24T01:47:03.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4298.167112 secs 2024-03-24T01:47:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4298.167112 secs 2024-03-24T01:47:03.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4298.167112 secs 2024-03-24T01:47:04.091 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4952 sec 2024-03-24T01:47:04.700 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:04 smithi078 ceph-mon[32429]: pgmap v2757: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:47:04.803 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:04 smithi106 ceph-mon[36505]: pgmap v2757: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:47:04.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:04 smithi107 ceph-mon[36485]: pgmap v2757: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:47:05.091 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4953 sec 2024-03-24T01:47:06.091 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4954 sec 2024-03-24T01:47:07.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:06 smithi078 ceph-mon[32429]: pgmap v2758: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:47:07.092 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4955 sec 2024-03-24T01:47:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:06 smithi107 ceph-mon[36485]: pgmap v2758: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:47:07.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:06 smithi106 ceph-mon[36505]: pgmap v2758: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:47:08.091 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4956 sec 2024-03-24T01:47:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:08 smithi078 ceph-mon[32429]: pgmap v2759: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:47:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4303.167174 secs 2024-03-24T01:47:09.091 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4957 sec 2024-03-24T01:47:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:08 smithi107 ceph-mon[36485]: pgmap v2759: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:47:09.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4303.167174 secs 2024-03-24T01:47:09.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:08 smithi106 ceph-mon[36505]: pgmap v2759: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:47:09.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4303.167174 secs 2024-03-24T01:47:10.091 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4958 sec 2024-03-24T01:47:11.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:10 smithi078 ceph-mon[32429]: pgmap v2760: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:47:11.091 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4959 sec 2024-03-24T01:47:11.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:10 smithi107 ceph-mon[36485]: pgmap v2760: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:47:11.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:10 smithi106 ceph-mon[36505]: pgmap v2760: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:47:12.091 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4960 sec 2024-03-24T01:47:13.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:12 smithi078 ceph-mon[32429]: pgmap v2761: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:13.092 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4961 sec 2024-03-24T01:47:13.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:12 smithi107 ceph-mon[36485]: pgmap v2761: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:13.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:12 smithi106 ceph-mon[36505]: pgmap v2761: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:14.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4308.167237 secs 2024-03-24T01:47:14.091 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4962 sec 2024-03-24T01:47:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4308.167237 secs 2024-03-24T01:47:14.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4308.167237 secs 2024-03-24T01:47:15.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:14 smithi078 ceph-mon[32429]: pgmap v2762: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:15.091 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4963 sec 2024-03-24T01:47:15.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:14 smithi107 ceph-mon[36485]: pgmap v2762: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:15.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:14 smithi106 ceph-mon[36505]: pgmap v2762: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:16.092 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4964 sec 2024-03-24T01:47:17.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:16 smithi078 ceph-mon[32429]: pgmap v2763: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:17.091 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4965 sec 2024-03-24T01:47:17.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:16 smithi107 ceph-mon[36485]: pgmap v2763: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:17.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:16 smithi106 ceph-mon[36505]: pgmap v2763: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:18.092 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4966 sec 2024-03-24T01:47:19.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:18 smithi078 ceph-mon[32429]: pgmap v2764: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:19.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4313.167314 secs 2024-03-24T01:47:19.092 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4967 sec 2024-03-24T01:47:19.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:18 smithi107 ceph-mon[36485]: pgmap v2764: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:19.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4313.167314 secs 2024-03-24T01:47:19.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:18 smithi106 ceph-mon[36505]: pgmap v2764: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:19.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4313.167314 secs 2024-03-24T01:47:20.092 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4968 sec 2024-03-24T01:47:21.092 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4969 sec 2024-03-24T01:47:21.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:20 smithi107 ceph-mon[36485]: pgmap v2765: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:21.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:20 smithi106 ceph-mon[36505]: pgmap v2765: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:21.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:20 smithi078 ceph-mon[32429]: pgmap v2765: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:22.092 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4970 sec 2024-03-24T01:47:23.092 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4971 sec 2024-03-24T01:47:23.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:22 smithi106 ceph-mon[36505]: pgmap v2766: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:23.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:22 smithi078 ceph-mon[32429]: pgmap v2766: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:23.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:22 smithi107 ceph-mon[36485]: pgmap v2766: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:24.092 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4972 sec 2024-03-24T01:47:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4318.167403 secs 2024-03-24T01:47:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4318.167403 secs 2024-03-24T01:47:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4318.167403 secs 2024-03-24T01:47:25.092 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4973 sec 2024-03-24T01:47:25.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:24 smithi106 ceph-mon[36505]: pgmap v2767: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:25.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:24 smithi078 ceph-mon[32429]: pgmap v2767: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:25.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:24 smithi107 ceph-mon[36485]: pgmap v2767: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:26.092 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4974 sec 2024-03-24T01:47:27.092 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4975 sec 2024-03-24T01:47:27.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:26 smithi106 ceph-mon[36505]: pgmap v2768: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:27.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:26 smithi078 ceph-mon[32429]: pgmap v2768: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:27.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:26 smithi107 ceph-mon[36485]: pgmap v2768: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:28.092 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4976 sec 2024-03-24T01:47:29.092 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4977 sec 2024-03-24T01:47:29.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:28 smithi106 ceph-mon[36505]: pgmap v2769: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:29.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4323.167493 secs 2024-03-24T01:47:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:28 smithi078 ceph-mon[32429]: pgmap v2769: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4323.167493 secs 2024-03-24T01:47:29.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:28 smithi107 ceph-mon[36485]: pgmap v2769: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:29.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4323.167493 secs 2024-03-24T01:47:30.093 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4978 sec 2024-03-24T01:47:31.093 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4979 sec 2024-03-24T01:47:31.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:30 smithi106 ceph-mon[36505]: pgmap v2770: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:31.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:30 smithi078 ceph-mon[32429]: pgmap v2770: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:31.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:30 smithi107 ceph-mon[36485]: pgmap v2770: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:32.093 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4980 sec 2024-03-24T01:47:33.093 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4981 sec 2024-03-24T01:47:33.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:32 smithi106 ceph-mon[36505]: pgmap v2771: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:33.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:32 smithi078 ceph-mon[32429]: pgmap v2771: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:33.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:32 smithi107 ceph-mon[36485]: pgmap v2771: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:34.093 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4982 sec 2024-03-24T01:47:34.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4328.167553 secs 2024-03-24T01:47:34.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4328.167553 secs 2024-03-24T01:47:34.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4328.167553 secs 2024-03-24T01:47:35.093 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4983 sec 2024-03-24T01:47:35.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:34 smithi106 ceph-mon[36505]: pgmap v2772: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:35.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:34 smithi078 ceph-mon[32429]: pgmap v2772: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:35.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:34 smithi107 ceph-mon[36485]: pgmap v2772: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:36.093 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4984 sec 2024-03-24T01:47:37.093 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4985 sec 2024-03-24T01:47:37.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:36 smithi106 ceph-mon[36505]: pgmap v2773: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:37.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:36 smithi078 ceph-mon[32429]: pgmap v2773: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:36 smithi107 ceph-mon[36485]: pgmap v2773: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:38.093 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4986 sec 2024-03-24T01:47:39.093 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4987 sec 2024-03-24T01:47:39.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:38 smithi106 ceph-mon[36505]: pgmap v2774: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:39.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4333.167641 secs 2024-03-24T01:47:39.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:38 smithi078 ceph-mon[32429]: pgmap v2774: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:39.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4333.167641 secs 2024-03-24T01:47:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:38 smithi107 ceph-mon[36485]: pgmap v2774: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:39.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4333.167641 secs 2024-03-24T01:47:40.093 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4988 sec 2024-03-24T01:47:41.094 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4989 sec 2024-03-24T01:47:41.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:40 smithi106 ceph-mon[36505]: pgmap v2775: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:41.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:40 smithi078 ceph-mon[32429]: pgmap v2775: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:40 smithi107 ceph-mon[36485]: pgmap v2775: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:42.094 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4990 sec 2024-03-24T01:47:43.094 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4991 sec 2024-03-24T01:47:43.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:42 smithi106 ceph-mon[36505]: pgmap v2776: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:43.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:42 smithi078 ceph-mon[32429]: pgmap v2776: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:43.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:42 smithi107 ceph-mon[36485]: pgmap v2776: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:44.094 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4992 sec 2024-03-24T01:47:44.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4338.167730 secs 2024-03-24T01:47:44.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4338.167730 secs 2024-03-24T01:47:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4338.167730 secs 2024-03-24T01:47:45.094 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4993 sec 2024-03-24T01:47:45.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:44 smithi106 ceph-mon[36505]: pgmap v2777: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:45.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:44 smithi078 ceph-mon[32429]: pgmap v2777: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:45.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:44 smithi107 ceph-mon[36485]: pgmap v2777: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:46.094 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4994 sec 2024-03-24T01:47:47.094 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4995 sec 2024-03-24T01:47:47.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:46 smithi106 ceph-mon[36505]: pgmap v2778: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:47.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:46 smithi078 ceph-mon[32429]: pgmap v2778: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:47.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:46 smithi107 ceph-mon[36485]: pgmap v2778: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:48.094 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4996 sec 2024-03-24T01:47:49.094 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4997 sec 2024-03-24T01:47:49.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:48 smithi078 ceph-mon[32429]: pgmap v2779: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:49.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4343.167792 secs 2024-03-24T01:47:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:48 smithi107 ceph-mon[36485]: pgmap v2779: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:49.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4343.167792 secs 2024-03-24T01:47:49.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:48 smithi106 ceph-mon[36505]: pgmap v2779: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:49.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4343.167792 secs 2024-03-24T01:47:50.094 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4998 sec 2024-03-24T01:47:51.094 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 4999 sec 2024-03-24T01:47:51.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:50 smithi078 ceph-mon[32429]: pgmap v2780: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:51.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:50 smithi107 ceph-mon[36485]: pgmap v2780: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:51.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:50 smithi106 ceph-mon[36505]: pgmap v2780: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:52.094 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5000 sec 2024-03-24T01:47:53.095 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5001 sec 2024-03-24T01:47:53.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:52 smithi078 ceph-mon[32429]: pgmap v2781: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:53.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:52 smithi107 ceph-mon[36485]: pgmap v2781: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:53.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:52 smithi106 ceph-mon[36505]: pgmap v2781: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:54.095 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5002 sec 2024-03-24T01:47:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4348.167850 secs 2024-03-24T01:47:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4348.167850 secs 2024-03-24T01:47:54.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4348.167850 secs 2024-03-24T01:47:55.095 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5003 sec 2024-03-24T01:47:55.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:54 smithi078 ceph-mon[32429]: pgmap v2782: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:55.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:54 smithi107 ceph-mon[36485]: pgmap v2782: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:55.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:54 smithi106 ceph-mon[36505]: pgmap v2782: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:56.095 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5004 sec 2024-03-24T01:47:57.095 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5005 sec 2024-03-24T01:47:57.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:56 smithi078 ceph-mon[32429]: pgmap v2783: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:57.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:56 smithi107 ceph-mon[36485]: pgmap v2783: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:57.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:56 smithi106 ceph-mon[36505]: pgmap v2783: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:58.095 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5006 sec 2024-03-24T01:47:59.095 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5007 sec 2024-03-24T01:47:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:58 smithi078 ceph-mon[32429]: pgmap v2784: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4353.167938 secs 2024-03-24T01:47:59.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:58 smithi107 ceph-mon[36485]: pgmap v2784: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:59.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4353.167938 secs 2024-03-24T01:47:59.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:58 smithi106 ceph-mon[36505]: pgmap v2784: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:47:59.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4353.167938 secs 2024-03-24T01:48:00.095 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5008 sec 2024-03-24T01:48:00.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:47:59 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:48:00.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:47:59 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:48:00.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:47:59 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:48:01.095 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5009 sec 2024-03-24T01:48:01.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:00 smithi078 ceph-mon[32429]: pgmap v2785: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:01.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:00 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:48:01.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:00 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:48:01.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:00 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:48:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:00 smithi107 ceph-mon[36485]: pgmap v2785: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:01.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:00 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:48:01.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:00 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:48:01.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:00 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:48:01.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:00 smithi106 ceph-mon[36505]: pgmap v2785: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:01.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:00 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:48:01.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:00 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:48:01.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:00 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:48:02.095 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5010 sec 2024-03-24T01:48:02.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:01 smithi078 ceph-mon[32429]: pgmap v2786: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:48:02.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:01 smithi078 ceph-mon[32429]: pgmap v2787: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:48:02.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:01 smithi107 ceph-mon[36485]: pgmap v2786: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:48:02.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:01 smithi107 ceph-mon[36485]: pgmap v2787: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:48:02.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:01 smithi106 ceph-mon[36505]: pgmap v2786: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:48:02.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:01 smithi106 ceph-mon[36505]: pgmap v2787: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:48:03.096 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5011 sec 2024-03-24T01:48:04.096 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5012 sec 2024-03-24T01:48:04.205 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:03 smithi106 ceph-mon[36505]: pgmap v2788: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:48:04.205 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4358.167998 secs 2024-03-24T01:48:04.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:03 smithi078 ceph-mon[32429]: pgmap v2788: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:48:04.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4358.167998 secs 2024-03-24T01:48:04.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:03 smithi107 ceph-mon[36485]: pgmap v2788: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:48:04.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4358.167998 secs 2024-03-24T01:48:05.096 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5013 sec 2024-03-24T01:48:06.096 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5014 sec 2024-03-24T01:48:06.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:05 smithi078 ceph-mon[32429]: pgmap v2789: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:48:06.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:05 smithi107 ceph-mon[36485]: pgmap v2789: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:48:06.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:05 smithi106 ceph-mon[36505]: pgmap v2789: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:48:07.096 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5015 sec 2024-03-24T01:48:08.096 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5016 sec 2024-03-24T01:48:08.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:07 smithi078 ceph-mon[32429]: pgmap v2790: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:48:08.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:07 smithi107 ceph-mon[36485]: pgmap v2790: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:48:08.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:07 smithi106 ceph-mon[36505]: pgmap v2790: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:48:09.096 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5017 sec 2024-03-24T01:48:09.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4363.168089 secs 2024-03-24T01:48:09.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4363.168089 secs 2024-03-24T01:48:09.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4363.168089 secs 2024-03-24T01:48:10.096 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5018 sec 2024-03-24T01:48:10.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:09 smithi078 ceph-mon[32429]: pgmap v2791: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:48:10.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:09 smithi107 ceph-mon[36485]: pgmap v2791: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:48:10.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:09 smithi106 ceph-mon[36505]: pgmap v2791: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:48:11.096 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5019 sec 2024-03-24T01:48:12.096 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5020 sec 2024-03-24T01:48:12.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:11 smithi078 ceph-mon[32429]: pgmap v2792: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:48:12.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:11 smithi107 ceph-mon[36485]: pgmap v2792: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:48:12.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:11 smithi106 ceph-mon[36505]: pgmap v2792: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:48:13.096 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5021 sec 2024-03-24T01:48:14.096 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5022 sec 2024-03-24T01:48:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:13 smithi078 ceph-mon[32429]: pgmap v2793: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4368.168147 secs 2024-03-24T01:48:14.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:13 smithi107 ceph-mon[36485]: pgmap v2793: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:14.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4368.168147 secs 2024-03-24T01:48:14.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:13 smithi106 ceph-mon[36505]: pgmap v2793: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:14.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4368.168147 secs 2024-03-24T01:48:15.096 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5023 sec 2024-03-24T01:48:16.097 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5024 sec 2024-03-24T01:48:16.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:15 smithi078 ceph-mon[32429]: pgmap v2794: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:16.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:15 smithi107 ceph-mon[36485]: pgmap v2794: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:16.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:15 smithi106 ceph-mon[36505]: pgmap v2794: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:17.097 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5025 sec 2024-03-24T01:48:18.097 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5026 sec 2024-03-24T01:48:18.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:17 smithi078 ceph-mon[32429]: pgmap v2795: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:18.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:17 smithi107 ceph-mon[36485]: pgmap v2795: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:18.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:17 smithi106 ceph-mon[36505]: pgmap v2795: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:19.097 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5027 sec 2024-03-24T01:48:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4373.168205 secs 2024-03-24T01:48:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4373.168205 secs 2024-03-24T01:48:19.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4373.168205 secs 2024-03-24T01:48:20.097 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5028 sec 2024-03-24T01:48:20.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:19 smithi078 ceph-mon[32429]: pgmap v2796: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:20.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:19 smithi107 ceph-mon[36485]: pgmap v2796: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:20.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:19 smithi106 ceph-mon[36505]: pgmap v2796: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:21.097 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5029 sec 2024-03-24T01:48:22.097 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5030 sec 2024-03-24T01:48:22.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:21 smithi078 ceph-mon[32429]: pgmap v2797: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:22.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:21 smithi107 ceph-mon[36485]: pgmap v2797: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:22.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:21 smithi106 ceph-mon[36505]: pgmap v2797: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:23.097 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5031 sec 2024-03-24T01:48:24.097 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5032 sec 2024-03-24T01:48:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:23 smithi078 ceph-mon[32429]: pgmap v2798: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4378.168292 secs 2024-03-24T01:48:24.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:23 smithi107 ceph-mon[36485]: pgmap v2798: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:24.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4378.168292 secs 2024-03-24T01:48:24.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:23 smithi106 ceph-mon[36505]: pgmap v2798: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:24.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4378.168292 secs 2024-03-24T01:48:25.098 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5033 sec 2024-03-24T01:48:26.098 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5034 sec 2024-03-24T01:48:26.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:25 smithi078 ceph-mon[32429]: pgmap v2799: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:26.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:25 smithi107 ceph-mon[36485]: pgmap v2799: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:26.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:25 smithi106 ceph-mon[36505]: pgmap v2799: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:27.098 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5035 sec 2024-03-24T01:48:28.098 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5036 sec 2024-03-24T01:48:28.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:27 smithi078 ceph-mon[32429]: pgmap v2800: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:28.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:27 smithi107 ceph-mon[36485]: pgmap v2800: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:28.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:27 smithi106 ceph-mon[36505]: pgmap v2800: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:29.098 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5037 sec 2024-03-24T01:48:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4383.168381 secs 2024-03-24T01:48:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4383.168381 secs 2024-03-24T01:48:29.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4383.168381 secs 2024-03-24T01:48:30.098 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5038 sec 2024-03-24T01:48:30.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:29 smithi078 ceph-mon[32429]: pgmap v2801: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:30.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:29 smithi107 ceph-mon[36485]: pgmap v2801: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:30.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:29 smithi106 ceph-mon[36505]: pgmap v2801: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:31.098 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5039 sec 2024-03-24T01:48:32.098 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5040 sec 2024-03-24T01:48:32.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:31 smithi078 ceph-mon[32429]: pgmap v2802: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:32.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:31 smithi107 ceph-mon[36485]: pgmap v2802: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:32.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:31 smithi106 ceph-mon[36505]: pgmap v2802: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:33.098 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5041 sec 2024-03-24T01:48:34.098 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5042 sec 2024-03-24T01:48:34.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:33 smithi078 ceph-mon[32429]: pgmap v2803: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:34.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4388.168470 secs 2024-03-24T01:48:34.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:33 smithi107 ceph-mon[36485]: pgmap v2803: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:34.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4388.168470 secs 2024-03-24T01:48:34.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:33 smithi106 ceph-mon[36505]: pgmap v2803: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:34.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4388.168470 secs 2024-03-24T01:48:35.098 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5043 sec 2024-03-24T01:48:36.098 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5044 sec 2024-03-24T01:48:36.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:36 smithi078 ceph-mon[32429]: pgmap v2804: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:36.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:36 smithi107 ceph-mon[36485]: pgmap v2804: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:36.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:36 smithi106 ceph-mon[36505]: pgmap v2804: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:37.099 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5045 sec 2024-03-24T01:48:38.099 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5046 sec 2024-03-24T01:48:38.143 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:38 smithi078 ceph-mon[32429]: pgmap v2805: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:38.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:38 smithi107 ceph-mon[36485]: pgmap v2805: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:38.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:38 smithi106 ceph-mon[36505]: pgmap v2805: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:39.099 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5047 sec 2024-03-24T01:48:39.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4393.168557 secs 2024-03-24T01:48:39.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4393.168557 secs 2024-03-24T01:48:39.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4393.168557 secs 2024-03-24T01:48:40.099 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5048 sec 2024-03-24T01:48:40.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:40 smithi078 ceph-mon[32429]: pgmap v2806: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:40 smithi107 ceph-mon[36485]: pgmap v2806: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:40.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:40 smithi106 ceph-mon[36505]: pgmap v2806: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:41.099 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5049 sec 2024-03-24T01:48:42.099 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5050 sec 2024-03-24T01:48:42.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:42 smithi078 ceph-mon[32429]: pgmap v2807: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:42.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:42 smithi107 ceph-mon[36485]: pgmap v2807: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:42.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:42 smithi106 ceph-mon[36505]: pgmap v2807: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:43.099 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5051 sec 2024-03-24T01:48:44.099 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5052 sec 2024-03-24T01:48:44.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:44 smithi078 ceph-mon[32429]: pgmap v2808: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:44.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4398.168619 secs 2024-03-24T01:48:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:44 smithi107 ceph-mon[36485]: pgmap v2808: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:44.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4398.168619 secs 2024-03-24T01:48:44.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:44 smithi106 ceph-mon[36505]: pgmap v2808: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:44.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4398.168619 secs 2024-03-24T01:48:45.099 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5053 sec 2024-03-24T01:48:46.099 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5054 sec 2024-03-24T01:48:46.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:46 smithi078 ceph-mon[32429]: pgmap v2809: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:46.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:46 smithi107 ceph-mon[36485]: pgmap v2809: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:46.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:46 smithi106 ceph-mon[36505]: pgmap v2809: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:47.100 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5055 sec 2024-03-24T01:48:48.100 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5056 sec 2024-03-24T01:48:48.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:48 smithi078 ceph-mon[32429]: pgmap v2810: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:48.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:48 smithi107 ceph-mon[36485]: pgmap v2810: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:48.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:48 smithi106 ceph-mon[36505]: pgmap v2810: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:49.100 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5057 sec 2024-03-24T01:48:49.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4403.168676 secs 2024-03-24T01:48:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4403.168676 secs 2024-03-24T01:48:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4403.168676 secs 2024-03-24T01:48:50.100 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5058 sec 2024-03-24T01:48:50.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:50 smithi078 ceph-mon[32429]: pgmap v2811: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:50 smithi107 ceph-mon[36485]: pgmap v2811: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:50.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:50 smithi106 ceph-mon[36505]: pgmap v2811: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:51.100 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5059 sec 2024-03-24T01:48:52.100 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5060 sec 2024-03-24T01:48:52.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:52 smithi078 ceph-mon[32429]: pgmap v2812: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:52.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:52 smithi107 ceph-mon[36485]: pgmap v2812: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:52.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:52 smithi106 ceph-mon[36505]: pgmap v2812: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:53.100 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5061 sec 2024-03-24T01:48:54.100 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5062 sec 2024-03-24T01:48:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:54 smithi078 ceph-mon[32429]: pgmap v2813: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4408.168712 secs 2024-03-24T01:48:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:54 smithi107 ceph-mon[36485]: pgmap v2813: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:54.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4408.168712 secs 2024-03-24T01:48:54.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:54 smithi106 ceph-mon[36505]: pgmap v2813: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:54.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4408.168712 secs 2024-03-24T01:48:55.100 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5063 sec 2024-03-24T01:48:56.101 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5064 sec 2024-03-24T01:48:56.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:56 smithi078 ceph-mon[32429]: pgmap v2814: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:56.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:56 smithi107 ceph-mon[36485]: pgmap v2814: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:56.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:56 smithi106 ceph-mon[36505]: pgmap v2814: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:48:57.100 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5065 sec 2024-03-24T01:48:58.101 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5066 sec 2024-03-24T01:48:58.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:58 smithi078 ceph-mon[32429]: pgmap v2815: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:58 smithi107 ceph-mon[36485]: pgmap v2815: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:58.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:58 smithi106 ceph-mon[36505]: pgmap v2815: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:59.101 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5067 sec 2024-03-24T01:48:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:48:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4413.168799 secs 2024-03-24T01:48:59.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:48:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4413.168799 secs 2024-03-24T01:48:59.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:48:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4413.168799 secs 2024-03-24T01:49:00.101 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5068 sec 2024-03-24T01:49:00.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:00 smithi078 ceph-mon[32429]: pgmap v2816: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:00.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:00 smithi107 ceph-mon[36485]: pgmap v2816: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:00.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:00 smithi106 ceph-mon[36505]: pgmap v2816: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:01.101 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5069 sec 2024-03-24T01:49:01.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:01 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:49:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:01 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:49:01.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:01 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:49:02.101 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5070 sec 2024-03-24T01:49:02.177 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:02 smithi078 ceph-mon[32429]: pgmap v2817: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:02.256 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:02 smithi106 ceph-mon[36505]: pgmap v2817: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:02.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:02 smithi107 ceph-mon[36485]: pgmap v2817: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:03.101 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5071 sec 2024-03-24T01:49:04.101 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5072 sec 2024-03-24T01:49:04.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:04 smithi078 ceph-mon[32429]: pgmap v2818: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:04.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4418.168809 secs 2024-03-24T01:49:04.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:04 smithi107 ceph-mon[36485]: pgmap v2818: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:04.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4418.168809 secs 2024-03-24T01:49:04.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:04 smithi106 ceph-mon[36505]: pgmap v2818: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:04.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4418.168809 secs 2024-03-24T01:49:05.101 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5073 sec 2024-03-24T01:49:05.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:05 smithi106 ceph-mon[36505]: pgmap v2819: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:05.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:49:05.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:49:05.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:49:05.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:49:05.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:49:05.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:49:05.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:05 smithi106 ceph-mon[36505]: pgmap v2820: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:49:05.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:05 smithi106 ceph-mon[36505]: pgmap v2821: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T01:49:05.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:49:06.101 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5074 sec 2024-03-24T01:49:06.121 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:05 smithi078 ceph-mon[32429]: pgmap v2819: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:06.121 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:49:06.121 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:49:06.121 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:49:06.121 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:49:06.121 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:49:06.121 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:49:06.121 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:05 smithi078 ceph-mon[32429]: pgmap v2820: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:49:06.122 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:05 smithi078 ceph-mon[32429]: pgmap v2821: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T01:49:06.122 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:49:06.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:05 smithi107 ceph-mon[36485]: pgmap v2819: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:06.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:49:06.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:49:06.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:49:06.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:49:06.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:49:06.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:49:06.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:05 smithi107 ceph-mon[36485]: pgmap v2820: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:49:06.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:05 smithi107 ceph-mon[36485]: pgmap v2821: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T01:49:06.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:49:07.101 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5075 sec 2024-03-24T01:49:08.101 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5076 sec 2024-03-24T01:49:08.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:08 smithi106 ceph-mon[36505]: pgmap v2822: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:49:08.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:08 smithi078 ceph-mon[32429]: pgmap v2822: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:49:08.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:08 smithi107 ceph-mon[36485]: pgmap v2822: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:49:09.101 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5077 sec 2024-03-24T01:49:09.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4423.168870 secs 2024-03-24T01:49:09.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4423.168870 secs 2024-03-24T01:49:09.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4423.168870 secs 2024-03-24T01:49:10.101 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5078 sec 2024-03-24T01:49:10.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:10 smithi106 ceph-mon[36505]: pgmap v2823: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T01:49:10.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:10 smithi078 ceph-mon[32429]: pgmap v2823: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T01:49:10.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:10 smithi107 ceph-mon[36485]: pgmap v2823: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T01:49:11.101 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5079 sec 2024-03-24T01:49:12.101 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5080 sec 2024-03-24T01:49:12.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:12 smithi106 ceph-mon[36505]: pgmap v2824: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:49:12.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:12 smithi078 ceph-mon[32429]: pgmap v2824: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:49:12.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:12 smithi107 ceph-mon[36485]: pgmap v2824: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:49:13.101 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5081 sec 2024-03-24T01:49:14.101 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5082 sec 2024-03-24T01:49:14.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4428.168908 secs 2024-03-24T01:49:14.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:14 smithi106 ceph-mon[36505]: pgmap v2825: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:49:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4428.168908 secs 2024-03-24T01:49:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:14 smithi078 ceph-mon[32429]: pgmap v2825: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:49:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4428.168908 secs 2024-03-24T01:49:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:14 smithi107 ceph-mon[36485]: pgmap v2825: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:49:15.102 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5083 sec 2024-03-24T01:49:16.102 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5084 sec 2024-03-24T01:49:16.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:16 smithi106 ceph-mon[36505]: pgmap v2826: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:49:16.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:16 smithi078 ceph-mon[32429]: pgmap v2826: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:49:16.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:16 smithi107 ceph-mon[36485]: pgmap v2826: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:49:17.102 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5085 sec 2024-03-24T01:49:18.102 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5086 sec 2024-03-24T01:49:18.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:18 smithi106 ceph-mon[36505]: pgmap v2827: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:18.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:18 smithi078 ceph-mon[32429]: pgmap v2827: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:18.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:18 smithi107 ceph-mon[36485]: pgmap v2827: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:19.102 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5087 sec 2024-03-24T01:49:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4433.168948 secs 2024-03-24T01:49:19.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4433.168948 secs 2024-03-24T01:49:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4433.168948 secs 2024-03-24T01:49:20.102 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5088 sec 2024-03-24T01:49:20.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:20 smithi106 ceph-mon[36505]: pgmap v2828: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:20.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:20 smithi078 ceph-mon[32429]: pgmap v2828: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:20.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:20 smithi107 ceph-mon[36485]: pgmap v2828: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:21.102 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5089 sec 2024-03-24T01:49:22.102 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5090 sec 2024-03-24T01:49:22.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:22 smithi078 ceph-mon[32429]: pgmap v2829: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:22.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:22 smithi107 ceph-mon[36485]: pgmap v2829: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:22.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:22 smithi106 ceph-mon[36505]: pgmap v2829: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:23.102 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5091 sec 2024-03-24T01:49:24.102 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5092 sec 2024-03-24T01:49:24.476 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4438.169006 secs 2024-03-24T01:49:24.477 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:24 smithi078 ceph-mon[32429]: pgmap v2830: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:24.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4438.169006 secs 2024-03-24T01:49:24.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:24 smithi107 ceph-mon[36485]: pgmap v2830: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:24.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4438.169006 secs 2024-03-24T01:49:24.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:24 smithi106 ceph-mon[36505]: pgmap v2830: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:25.102 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5093 sec 2024-03-24T01:49:26.102 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5094 sec 2024-03-24T01:49:26.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:26 smithi078 ceph-mon[32429]: pgmap v2831: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:26.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:26 smithi107 ceph-mon[36485]: pgmap v2831: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:26 smithi106 ceph-mon[36505]: pgmap v2831: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:27.103 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5095 sec 2024-03-24T01:49:28.103 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5096 sec 2024-03-24T01:49:28.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:28 smithi078 ceph-mon[32429]: pgmap v2832: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:28.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:28 smithi107 ceph-mon[36485]: pgmap v2832: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:28.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:28 smithi106 ceph-mon[36505]: pgmap v2832: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:29.103 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5097 sec 2024-03-24T01:49:29.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4443.169048 secs 2024-03-24T01:49:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4443.169048 secs 2024-03-24T01:49:29.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4443.169048 secs 2024-03-24T01:49:30.103 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5098 sec 2024-03-24T01:49:30.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:30 smithi078 ceph-mon[32429]: pgmap v2833: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:30.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:30 smithi107 ceph-mon[36485]: pgmap v2833: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:30.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:30 smithi106 ceph-mon[36505]: pgmap v2833: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:31.104 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5099 sec 2024-03-24T01:49:32.103 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5100 sec 2024-03-24T01:49:32.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:32 smithi078 ceph-mon[32429]: pgmap v2834: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:32.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:32 smithi107 ceph-mon[36485]: pgmap v2834: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:32.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:32 smithi106 ceph-mon[36505]: pgmap v2834: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:33.103 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5101 sec 2024-03-24T01:49:34.103 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5102 sec 2024-03-24T01:49:34.454 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4448.169102 secs 2024-03-24T01:49:34.454 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:34 smithi106 ceph-mon[36505]: pgmap v2835: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:34.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4448.169102 secs 2024-03-24T01:49:34.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:34 smithi078 ceph-mon[32429]: pgmap v2835: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4448.169102 secs 2024-03-24T01:49:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:34 smithi107 ceph-mon[36485]: pgmap v2835: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:35.103 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5103 sec 2024-03-24T01:49:36.103 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5104 sec 2024-03-24T01:49:36.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:36 smithi078 ceph-mon[32429]: pgmap v2836: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:36.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:36 smithi107 ceph-mon[36485]: pgmap v2836: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:36.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:36 smithi106 ceph-mon[36505]: pgmap v2836: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:37.103 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5105 sec 2024-03-24T01:49:38.104 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5106 sec 2024-03-24T01:49:38.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:38 smithi078 ceph-mon[32429]: pgmap v2837: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:38.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:38 smithi107 ceph-mon[36485]: pgmap v2837: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:38.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:38 smithi106 ceph-mon[36505]: pgmap v2837: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:39.104 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5107 sec 2024-03-24T01:49:39.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4453.169191 secs 2024-03-24T01:49:39.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4453.169191 secs 2024-03-24T01:49:39.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4453.169191 secs 2024-03-24T01:49:40.104 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5108 sec 2024-03-24T01:49:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:40 smithi078 ceph-mon[32429]: pgmap v2838: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:40.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:40 smithi107 ceph-mon[36485]: pgmap v2838: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:40.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:40 smithi106 ceph-mon[36505]: pgmap v2838: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:41.104 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5109 sec 2024-03-24T01:49:42.104 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5110 sec 2024-03-24T01:49:42.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:42 smithi078 ceph-mon[32429]: pgmap v2839: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:42.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:42 smithi107 ceph-mon[36485]: pgmap v2839: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:42.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:42 smithi106 ceph-mon[36505]: pgmap v2839: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:43.104 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5111 sec 2024-03-24T01:49:44.104 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5112 sec 2024-03-24T01:49:44.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4458.169274 secs 2024-03-24T01:49:44.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:44 smithi078 ceph-mon[32429]: pgmap v2840: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4458.169274 secs 2024-03-24T01:49:44.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:44 smithi107 ceph-mon[36485]: pgmap v2840: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:44.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4458.169274 secs 2024-03-24T01:49:44.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:44 smithi106 ceph-mon[36505]: pgmap v2840: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:45.104 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5113 sec 2024-03-24T01:49:46.104 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5114 sec 2024-03-24T01:49:46.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:46 smithi078 ceph-mon[32429]: pgmap v2841: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:46.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:46 smithi107 ceph-mon[36485]: pgmap v2841: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:46 smithi106 ceph-mon[36505]: pgmap v2841: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:47.104 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5115 sec 2024-03-24T01:49:48.105 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5116 sec 2024-03-24T01:49:48.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:48 smithi078 ceph-mon[32429]: pgmap v2842: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:48.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:48 smithi107 ceph-mon[36485]: pgmap v2842: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:48.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:48 smithi106 ceph-mon[36505]: pgmap v2842: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:49.105 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5117 sec 2024-03-24T01:49:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4463.169334 secs 2024-03-24T01:49:49.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4463.169334 secs 2024-03-24T01:49:49.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4463.169334 secs 2024-03-24T01:49:50.105 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5118 sec 2024-03-24T01:49:50.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:50 smithi078 ceph-mon[32429]: pgmap v2843: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:50.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:50 smithi107 ceph-mon[36485]: pgmap v2843: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:50.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:50 smithi106 ceph-mon[36505]: pgmap v2843: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:51.105 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5119 sec 2024-03-24T01:49:52.105 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5120 sec 2024-03-24T01:49:52.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:52 smithi078 ceph-mon[32429]: pgmap v2844: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:52.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:52 smithi107 ceph-mon[36485]: pgmap v2844: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:52.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:52 smithi106 ceph-mon[36505]: pgmap v2844: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:53.106 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5121 sec 2024-03-24T01:49:54.105 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5122 sec 2024-03-24T01:49:54.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4468.169394 secs 2024-03-24T01:49:54.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:54 smithi078 ceph-mon[32429]: pgmap v2845: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4468.169394 secs 2024-03-24T01:49:54.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:54 smithi107 ceph-mon[36485]: pgmap v2845: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4468.169394 secs 2024-03-24T01:49:54.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:54 smithi106 ceph-mon[36505]: pgmap v2845: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:55.105 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5123 sec 2024-03-24T01:49:56.105 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5124 sec 2024-03-24T01:49:56.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:56 smithi078 ceph-mon[32429]: pgmap v2846: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:56.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:56 smithi107 ceph-mon[36485]: pgmap v2846: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:56.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:56 smithi106 ceph-mon[36505]: pgmap v2846: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:49:57.105 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5125 sec 2024-03-24T01:49:58.106 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5126 sec 2024-03-24T01:49:58.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:58 smithi078 ceph-mon[32429]: pgmap v2847: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:58.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:58 smithi107 ceph-mon[36485]: pgmap v2847: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:58.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:58 smithi106 ceph-mon[36505]: pgmap v2847: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:59.106 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5127 sec 2024-03-24T01:49:59.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:49:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4473.169478 secs 2024-03-24T01:49:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:49:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4473.169478 secs 2024-03-24T01:49:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:49:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4473.169478 secs 2024-03-24T01:50:00.106 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5128 sec 2024-03-24T01:50:00.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:00 smithi078 ceph-mon[32429]: pgmap v2848: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:00.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:00 smithi078 ceph-mon[32429]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:50:00.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:00 smithi107 ceph-mon[36485]: pgmap v2848: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:00.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:00 smithi107 ceph-mon[36485]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:50:00.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:00 smithi106 ceph-mon[36505]: pgmap v2848: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:00.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:00 smithi106 ceph-mon[36505]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:50:01.106 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5129 sec 2024-03-24T01:50:01.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:01 smithi078 ceph-mon[32429]: pgmap v2849: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:01.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:01 smithi107 ceph-mon[36485]: pgmap v2849: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:01.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:01 smithi106 ceph-mon[36505]: pgmap v2849: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:02.106 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5130 sec 2024-03-24T01:50:03.106 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5131 sec 2024-03-24T01:50:04.106 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5132 sec 2024-03-24T01:50:04.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4478.169537 secs 2024-03-24T01:50:04.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:04 smithi106 ceph-mon[36505]: pgmap v2850: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:04.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4478.169537 secs 2024-03-24T01:50:04.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:04 smithi078 ceph-mon[32429]: pgmap v2850: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:04.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4478.169537 secs 2024-03-24T01:50:04.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:04 smithi107 ceph-mon[36485]: pgmap v2850: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:05.106 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5133 sec 2024-03-24T01:50:06.106 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5134 sec 2024-03-24T01:50:06.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:06 smithi106 ceph-mon[36505]: pgmap v2851: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:06.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:06 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:50:06.499 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:06 smithi107 ceph-mon[36485]: pgmap v2851: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:06.499 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:06 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:50:06.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:06 smithi078 ceph-mon[32429]: pgmap v2851: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:06.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:06 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:50:07.106 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5135 sec 2024-03-24T01:50:08.107 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5136 sec 2024-03-24T01:50:08.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:08 smithi107 ceph-mon[36485]: pgmap v2852: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:08.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:08 smithi106 ceph-mon[36505]: pgmap v2852: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:08.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:08 smithi078 ceph-mon[32429]: pgmap v2852: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:09.107 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5137 sec 2024-03-24T01:50:09.478 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4483.169578 secs 2024-03-24T01:50:09.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4483.169578 secs 2024-03-24T01:50:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4483.169578 secs 2024-03-24T01:50:10.107 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5138 sec 2024-03-24T01:50:10.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:10 smithi106 ceph-mon[36505]: pgmap v2853: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:10.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:50:10.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:50:10.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:50:10.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:50:10.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:50:10.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:10 smithi078 ceph-mon[32429]: pgmap v2853: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:10.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:50:10.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:50:10.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:50:10.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:50:10.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:50:10.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:10 smithi107 ceph-mon[36485]: pgmap v2853: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:10.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:50:10.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:50:10.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:50:10.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:50:10.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:50:11.107 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5139 sec 2024-03-24T01:50:11.580 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:11 smithi078 ceph-mon[32429]: pgmap v2854: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:50:11.580 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:11 smithi078 ceph-mon[32429]: pgmap v2855: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T01:50:11.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:11 smithi107 ceph-mon[36485]: pgmap v2854: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:50:11.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:11 smithi107 ceph-mon[36485]: pgmap v2855: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T01:50:11.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:11 smithi106 ceph-mon[36505]: pgmap v2854: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:50:11.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:11 smithi106 ceph-mon[36505]: pgmap v2855: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T01:50:12.107 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5140 sec 2024-03-24T01:50:13.107 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5141 sec 2024-03-24T01:50:13.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:13 smithi078 ceph-mon[32429]: pgmap v2856: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:50:13.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:13 smithi107 ceph-mon[36485]: pgmap v2856: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:50:13.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:13 smithi106 ceph-mon[36505]: pgmap v2856: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:50:14.107 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5142 sec 2024-03-24T01:50:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4488.169666 secs 2024-03-24T01:50:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4488.169666 secs 2024-03-24T01:50:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4488.169666 secs 2024-03-24T01:50:15.107 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5143 sec 2024-03-24T01:50:15.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:15 smithi078 ceph-mon[32429]: pgmap v2857: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:50:15.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:15 smithi107 ceph-mon[36485]: pgmap v2857: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:50:15.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:15 smithi106 ceph-mon[36505]: pgmap v2857: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:50:16.107 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5144 sec 2024-03-24T01:50:17.107 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5145 sec 2024-03-24T01:50:17.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:17 smithi078 ceph-mon[32429]: pgmap v2858: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:50:17.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:17 smithi107 ceph-mon[36485]: pgmap v2858: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:50:17.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:17 smithi106 ceph-mon[36505]: pgmap v2858: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:50:18.107 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5146 sec 2024-03-24T01:50:19.108 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5147 sec 2024-03-24T01:50:19.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:19 smithi078 ceph-mon[32429]: pgmap v2859: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:50:19.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4493.169752 secs 2024-03-24T01:50:19.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:19 smithi107 ceph-mon[36485]: pgmap v2859: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:50:19.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4493.169752 secs 2024-03-24T01:50:19.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:19 smithi106 ceph-mon[36505]: pgmap v2859: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:50:19.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4493.169752 secs 2024-03-24T01:50:20.108 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5148 sec 2024-03-24T01:50:21.108 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5149 sec 2024-03-24T01:50:21.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:21 smithi078 ceph-mon[32429]: pgmap v2860: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:50:21.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:21 smithi107 ceph-mon[36485]: pgmap v2860: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:50:21.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:21 smithi106 ceph-mon[36505]: pgmap v2860: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:50:22.108 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5150 sec 2024-03-24T01:50:23.108 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5151 sec 2024-03-24T01:50:23.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:23 smithi078 ceph-mon[32429]: pgmap v2861: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:23.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:23 smithi107 ceph-mon[36485]: pgmap v2861: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:23.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:23 smithi106 ceph-mon[36505]: pgmap v2861: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:24.108 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5152 sec 2024-03-24T01:50:24.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4498.169809 secs 2024-03-24T01:50:24.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4498.169809 secs 2024-03-24T01:50:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4498.169809 secs 2024-03-24T01:50:25.108 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5153 sec 2024-03-24T01:50:25.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:25 smithi078 ceph-mon[32429]: pgmap v2862: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:25.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:25 smithi107 ceph-mon[36485]: pgmap v2862: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:25.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:25 smithi106 ceph-mon[36505]: pgmap v2862: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:26.108 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5154 sec 2024-03-24T01:50:27.108 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5155 sec 2024-03-24T01:50:27.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:27 smithi078 ceph-mon[32429]: pgmap v2863: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:27.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:27 smithi107 ceph-mon[36485]: pgmap v2863: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:27.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:27 smithi106 ceph-mon[36505]: pgmap v2863: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:28.108 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5156 sec 2024-03-24T01:50:29.108 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5157 sec 2024-03-24T01:50:29.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:29 smithi078 ceph-mon[32429]: pgmap v2864: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:29.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4503.169870 secs 2024-03-24T01:50:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:29 smithi107 ceph-mon[36485]: pgmap v2864: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:29.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4503.169870 secs 2024-03-24T01:50:29.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:29 smithi106 ceph-mon[36505]: pgmap v2864: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:29.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4503.169870 secs 2024-03-24T01:50:30.108 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5158 sec 2024-03-24T01:50:31.108 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5159 sec 2024-03-24T01:50:31.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:31 smithi078 ceph-mon[32429]: pgmap v2865: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:31.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:31 smithi107 ceph-mon[36485]: pgmap v2865: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:31.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:31 smithi106 ceph-mon[36505]: pgmap v2865: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:32.108 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5160 sec 2024-03-24T01:50:33.108 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5161 sec 2024-03-24T01:50:33.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:33 smithi078 ceph-mon[32429]: pgmap v2866: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:33.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:33 smithi107 ceph-mon[36485]: pgmap v2866: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:33.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:33 smithi106 ceph-mon[36505]: pgmap v2866: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:34.109 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5162 sec 2024-03-24T01:50:34.492 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4508.169957 secs 2024-03-24T01:50:34.554 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4508.169957 secs 2024-03-24T01:50:34.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4508.169957 secs 2024-03-24T01:50:35.109 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5163 sec 2024-03-24T01:50:35.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:35 smithi078 ceph-mon[32429]: pgmap v2867: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:35.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:35 smithi107 ceph-mon[36485]: pgmap v2867: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:35.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:35 smithi106 ceph-mon[36505]: pgmap v2867: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:36.109 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5164 sec 2024-03-24T01:50:37.109 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5165 sec 2024-03-24T01:50:37.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:37 smithi078 ceph-mon[32429]: pgmap v2868: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:37.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:37 smithi107 ceph-mon[36485]: pgmap v2868: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:37.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:37 smithi106 ceph-mon[36505]: pgmap v2868: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:38.109 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5166 sec 2024-03-24T01:50:39.109 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5167 sec 2024-03-24T01:50:39.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:39 smithi078 ceph-mon[32429]: pgmap v2869: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:39.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4513.170044 secs 2024-03-24T01:50:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:39 smithi107 ceph-mon[36485]: pgmap v2869: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4513.170044 secs 2024-03-24T01:50:39.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:39 smithi106 ceph-mon[36505]: pgmap v2869: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:39.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4513.170044 secs 2024-03-24T01:50:40.109 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5168 sec 2024-03-24T01:50:41.109 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5169 sec 2024-03-24T01:50:41.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:41 smithi078 ceph-mon[32429]: pgmap v2870: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:41.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:41 smithi107 ceph-mon[36485]: pgmap v2870: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:41.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:41 smithi106 ceph-mon[36505]: pgmap v2870: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:42.109 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5170 sec 2024-03-24T01:50:43.109 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5171 sec 2024-03-24T01:50:43.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:43 smithi078 ceph-mon[32429]: pgmap v2871: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:50:43.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:43 smithi107 ceph-mon[36485]: pgmap v2871: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:50:43.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:43 smithi106 ceph-mon[36505]: pgmap v2871: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:50:44.110 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5172 sec 2024-03-24T01:50:44.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4518.170132 secs 2024-03-24T01:50:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4518.170132 secs 2024-03-24T01:50:44.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4518.170132 secs 2024-03-24T01:50:45.110 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5173 sec 2024-03-24T01:50:45.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:45 smithi078 ceph-mon[32429]: pgmap v2872: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:50:45.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:45 smithi107 ceph-mon[36485]: pgmap v2872: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:50:45.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:45 smithi106 ceph-mon[36505]: pgmap v2872: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:50:46.110 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5174 sec 2024-03-24T01:50:47.110 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5175 sec 2024-03-24T01:50:47.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:47 smithi078 ceph-mon[32429]: pgmap v2873: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:50:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:47 smithi107 ceph-mon[36485]: pgmap v2873: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:50:47.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:47 smithi106 ceph-mon[36505]: pgmap v2873: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:50:48.110 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5176 sec 2024-03-24T01:50:49.110 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5177 sec 2024-03-24T01:50:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:49 smithi078 ceph-mon[32429]: pgmap v2874: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:50:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4523.170166 secs 2024-03-24T01:50:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:49 smithi107 ceph-mon[36485]: pgmap v2874: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:50:49.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4523.170166 secs 2024-03-24T01:50:49.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:49 smithi106 ceph-mon[36505]: pgmap v2874: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:50:49.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4523.170166 secs 2024-03-24T01:50:50.110 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5178 sec 2024-03-24T01:50:51.110 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5179 sec 2024-03-24T01:50:51.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:51 smithi078 ceph-mon[32429]: pgmap v2875: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:50:51.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:51 smithi107 ceph-mon[36485]: pgmap v2875: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:50:51.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:51 smithi106 ceph-mon[36505]: pgmap v2875: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T01:50:52.110 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5180 sec 2024-03-24T01:50:53.110 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5181 sec 2024-03-24T01:50:53.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:53 smithi078 ceph-mon[32429]: pgmap v2876: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:50:53.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:53 smithi107 ceph-mon[36485]: pgmap v2876: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:50:53.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:53 smithi106 ceph-mon[36505]: pgmap v2876: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:50:54.110 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5182 sec 2024-03-24T01:50:54.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4528.170252 secs 2024-03-24T01:50:54.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4528.170252 secs 2024-03-24T01:50:54.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4528.170252 secs 2024-03-24T01:50:55.110 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5183 sec 2024-03-24T01:50:55.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:55 smithi078 ceph-mon[32429]: pgmap v2877: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:55.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:55 smithi107 ceph-mon[36485]: pgmap v2877: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:55.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:55 smithi106 ceph-mon[36505]: pgmap v2877: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:56.110 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5184 sec 2024-03-24T01:50:56.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:56 smithi078 ceph-mon[32429]: pgmap v2878: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:56.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:56 smithi107 ceph-mon[36485]: pgmap v2878: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:56.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:56 smithi106 ceph-mon[36505]: pgmap v2878: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:57.111 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5185 sec 2024-03-24T01:50:58.111 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5186 sec 2024-03-24T01:50:58.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:58 smithi106 ceph-mon[36505]: pgmap v2879: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:58.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:50:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4533.170323 secs 2024-03-24T01:50:59.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:58 smithi078 ceph-mon[32429]: pgmap v2879: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:59.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:50:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4533.170323 secs 2024-03-24T01:50:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:58 smithi107 ceph-mon[36485]: pgmap v2879: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:50:59.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:50:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4533.170323 secs 2024-03-24T01:50:59.111 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5187 sec 2024-03-24T01:51:00.111 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5188 sec 2024-03-24T01:51:00.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:00 smithi106 ceph-mon[36505]: pgmap v2880: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:01.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:00 smithi078 ceph-mon[32429]: pgmap v2880: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:01.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:00 smithi107 ceph-mon[36485]: pgmap v2880: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:01.111 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5189 sec 2024-03-24T01:51:02.111 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5190 sec 2024-03-24T01:51:02.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:02 smithi106 ceph-mon[36505]: pgmap v2881: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:03.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:02 smithi078 ceph-mon[32429]: pgmap v2881: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:03.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:02 smithi107 ceph-mon[36485]: pgmap v2881: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:03.111 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5191 sec 2024-03-24T01:51:03.906 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4538.170384 secs 2024-03-24T01:51:04.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4538.170384 secs 2024-03-24T01:51:04.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4538.170384 secs 2024-03-24T01:51:04.111 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5192 sec 2024-03-24T01:51:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:04 smithi106 ceph-mon[36505]: pgmap v2882: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:04.992 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:04 smithi078 ceph-mon[32429]: pgmap v2882: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:05.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:04 smithi107 ceph-mon[36485]: pgmap v2882: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:05.111 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5193 sec 2024-03-24T01:51:06.111 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5194 sec 2024-03-24T01:51:06.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:06 smithi106 ceph-mon[36505]: pgmap v2883: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:07.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:06 smithi078 ceph-mon[32429]: pgmap v2883: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:06 smithi107 ceph-mon[36485]: pgmap v2883: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:07.111 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5195 sec 2024-03-24T01:51:08.111 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5196 sec 2024-03-24T01:51:08.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:08 smithi106 ceph-mon[36505]: pgmap v2884: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:08.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4543.170445 secs 2024-03-24T01:51:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:08 smithi078 ceph-mon[32429]: pgmap v2884: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:09.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4543.170445 secs 2024-03-24T01:51:09.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:08 smithi107 ceph-mon[36485]: pgmap v2884: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:09.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4543.170445 secs 2024-03-24T01:51:09.111 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5197 sec 2024-03-24T01:51:10.112 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5198 sec 2024-03-24T01:51:10.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:10 smithi106 ceph-mon[36505]: pgmap v2885: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:10.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:51:10.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:51:10.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:51:10.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:51:11.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:10 smithi078 ceph-mon[32429]: pgmap v2885: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:11.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:51:11.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:51:11.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:51:11.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:51:11.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:10 smithi107 ceph-mon[36485]: pgmap v2885: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:11.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:51:11.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:51:11.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:51:11.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:51:11.112 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5199 sec 2024-03-24T01:51:11.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:11 smithi106 ceph-mon[36505]: pgmap v2886: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:51:11.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:11 smithi106 ceph-mon[36505]: pgmap v2887: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:51:12.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:11 smithi078 ceph-mon[32429]: pgmap v2886: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:51:12.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:11 smithi078 ceph-mon[32429]: pgmap v2887: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:51:12.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:11 smithi107 ceph-mon[36485]: pgmap v2886: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:51:12.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:11 smithi107 ceph-mon[36485]: pgmap v2887: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:51:12.112 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5200 sec 2024-03-24T01:51:13.112 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5201 sec 2024-03-24T01:51:13.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:13 smithi106 ceph-mon[36505]: pgmap v2888: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:51:13.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4548.170534 secs 2024-03-24T01:51:14.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:13 smithi078 ceph-mon[32429]: pgmap v2888: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:51:14.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4548.170534 secs 2024-03-24T01:51:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:13 smithi107 ceph-mon[36485]: pgmap v2888: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:51:14.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4548.170534 secs 2024-03-24T01:51:14.112 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5202 sec 2024-03-24T01:51:15.112 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5203 sec 2024-03-24T01:51:16.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:15 smithi078 ceph-mon[32429]: pgmap v2889: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:51:16.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:15 smithi107 ceph-mon[36485]: pgmap v2889: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:51:16.112 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5204 sec 2024-03-24T01:51:16.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:15 smithi106 ceph-mon[36505]: pgmap v2889: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:51:17.112 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5205 sec 2024-03-24T01:51:18.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:17 smithi078 ceph-mon[32429]: pgmap v2890: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:51:18.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:17 smithi107 ceph-mon[36485]: pgmap v2890: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:51:18.112 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5206 sec 2024-03-24T01:51:18.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:17 smithi106 ceph-mon[36505]: pgmap v2890: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:51:19.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4553.170622 secs 2024-03-24T01:51:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4553.170622 secs 2024-03-24T01:51:19.112 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5207 sec 2024-03-24T01:51:19.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4553.170622 secs 2024-03-24T01:51:20.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:19 smithi078 ceph-mon[32429]: pgmap v2891: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:51:20.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:19 smithi107 ceph-mon[36485]: pgmap v2891: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:51:20.113 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5208 sec 2024-03-24T01:51:20.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:19 smithi106 ceph-mon[36505]: pgmap v2891: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:51:21.113 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5209 sec 2024-03-24T01:51:21.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:21 smithi106 ceph-mon[36505]: pgmap v2892: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:51:21.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:21 smithi078 ceph-mon[32429]: pgmap v2892: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:51:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:21 smithi107 ceph-mon[36485]: pgmap v2892: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:51:22.113 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5210 sec 2024-03-24T01:51:23.113 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5211 sec 2024-03-24T01:51:23.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:23 smithi078 ceph-mon[32429]: pgmap v2893: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:23.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4558.170680 secs 2024-03-24T01:51:23.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:23 smithi107 ceph-mon[36485]: pgmap v2893: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:23.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4558.170680 secs 2024-03-24T01:51:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:23 smithi106 ceph-mon[36505]: pgmap v2893: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:23.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4558.170680 secs 2024-03-24T01:51:24.113 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5212 sec 2024-03-24T01:51:25.113 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5213 sec 2024-03-24T01:51:25.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:25 smithi078 ceph-mon[32429]: pgmap v2894: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:25.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:25 smithi107 ceph-mon[36485]: pgmap v2894: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:25 smithi106 ceph-mon[36505]: pgmap v2894: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:26.113 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5214 sec 2024-03-24T01:51:27.113 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5215 sec 2024-03-24T01:51:27.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:27 smithi078 ceph-mon[32429]: pgmap v2895: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:27.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:27 smithi107 ceph-mon[36485]: pgmap v2895: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:27.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:27 smithi106 ceph-mon[36505]: pgmap v2895: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:28.113 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5216 sec 2024-03-24T01:51:28.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4563.170722 secs 2024-03-24T01:51:28.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4563.170722 secs 2024-03-24T01:51:28.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4563.170722 secs 2024-03-24T01:51:29.113 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5217 sec 2024-03-24T01:51:29.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:29 smithi078 ceph-mon[32429]: pgmap v2896: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:29.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:29 smithi107 ceph-mon[36485]: pgmap v2896: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:29 smithi106 ceph-mon[36505]: pgmap v2896: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:30.114 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5218 sec 2024-03-24T01:51:31.114 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5219 sec 2024-03-24T01:51:31.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:31 smithi078 ceph-mon[32429]: pgmap v2897: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:31.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:31 smithi107 ceph-mon[36485]: pgmap v2897: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:31.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:31 smithi106 ceph-mon[36505]: pgmap v2897: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:32.114 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5220 sec 2024-03-24T01:51:33.114 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5221 sec 2024-03-24T01:51:33.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:33 smithi078 ceph-mon[32429]: pgmap v2898: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:33.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4568.170804 secs 2024-03-24T01:51:33.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:33 smithi107 ceph-mon[36485]: pgmap v2898: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:33.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4568.170804 secs 2024-03-24T01:51:33.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:33 smithi106 ceph-mon[36505]: pgmap v2898: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:33.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4568.170804 secs 2024-03-24T01:51:34.114 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5222 sec 2024-03-24T01:51:35.114 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5223 sec 2024-03-24T01:51:35.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:35 smithi078 ceph-mon[32429]: pgmap v2899: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:35.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:35 smithi107 ceph-mon[36485]: pgmap v2899: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:35.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:35 smithi106 ceph-mon[36505]: pgmap v2899: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:36.114 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5224 sec 2024-03-24T01:51:37.114 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5225 sec 2024-03-24T01:51:37.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:37 smithi078 ceph-mon[32429]: pgmap v2900: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:37.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:37 smithi107 ceph-mon[36485]: pgmap v2900: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:37.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:37 smithi106 ceph-mon[36505]: pgmap v2900: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:38.114 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5226 sec 2024-03-24T01:51:38.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4573.170866 secs 2024-03-24T01:51:38.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4573.170866 secs 2024-03-24T01:51:38.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4573.170866 secs 2024-03-24T01:51:39.114 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5227 sec 2024-03-24T01:51:39.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:39 smithi078 ceph-mon[32429]: pgmap v2901: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:39.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:39 smithi107 ceph-mon[36485]: pgmap v2901: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:39.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:39 smithi106 ceph-mon[36505]: pgmap v2901: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:40.114 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5228 sec 2024-03-24T01:51:41.115 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5229 sec 2024-03-24T01:51:41.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:41 smithi078 ceph-mon[32429]: pgmap v2902: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:41.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:41 smithi107 ceph-mon[36485]: pgmap v2902: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:41.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:41 smithi106 ceph-mon[36505]: pgmap v2902: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:42.114 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5230 sec 2024-03-24T01:51:43.115 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5231 sec 2024-03-24T01:51:43.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:43 smithi078 ceph-mon[32429]: pgmap v2903: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:43.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4578.170950 secs 2024-03-24T01:51:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:43 smithi107 ceph-mon[36485]: pgmap v2903: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4578.170950 secs 2024-03-24T01:51:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:43 smithi106 ceph-mon[36505]: pgmap v2903: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:43.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4578.170950 secs 2024-03-24T01:51:44.115 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5232 sec 2024-03-24T01:51:45.115 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5233 sec 2024-03-24T01:51:45.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:45 smithi078 ceph-mon[32429]: pgmap v2904: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:45.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:45 smithi107 ceph-mon[36485]: pgmap v2904: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:45.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:45 smithi106 ceph-mon[36505]: pgmap v2904: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:46.115 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5234 sec 2024-03-24T01:51:47.115 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5235 sec 2024-03-24T01:51:47.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:47 smithi078 ceph-mon[32429]: pgmap v2905: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:47.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:47 smithi107 ceph-mon[36485]: pgmap v2905: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:47.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:47 smithi106 ceph-mon[36505]: pgmap v2905: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:48.115 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5236 sec 2024-03-24T01:51:48.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4583.171039 secs 2024-03-24T01:51:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4583.171039 secs 2024-03-24T01:51:48.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4583.171039 secs 2024-03-24T01:51:49.115 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5237 sec 2024-03-24T01:51:49.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:49 smithi078 ceph-mon[32429]: pgmap v2906: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:49.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:49 smithi107 ceph-mon[36485]: pgmap v2906: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:49.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:49 smithi106 ceph-mon[36505]: pgmap v2906: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:50.115 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5238 sec 2024-03-24T01:51:51.115 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5239 sec 2024-03-24T01:51:51.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:51 smithi078 ceph-mon[32429]: pgmap v2907: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:51.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:51 smithi107 ceph-mon[36485]: pgmap v2907: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:51.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:51 smithi106 ceph-mon[36505]: pgmap v2907: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:52.115 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5240 sec 2024-03-24T01:51:53.115 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5241 sec 2024-03-24T01:51:53.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:53 smithi078 ceph-mon[32429]: pgmap v2908: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:53.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4588.171126 secs 2024-03-24T01:51:53.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:53 smithi107 ceph-mon[36485]: pgmap v2908: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:53.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4588.171126 secs 2024-03-24T01:51:53.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:53 smithi106 ceph-mon[36505]: pgmap v2908: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:53.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4588.171126 secs 2024-03-24T01:51:54.115 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5242 sec 2024-03-24T01:51:55.116 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5243 sec 2024-03-24T01:51:55.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:55 smithi078 ceph-mon[32429]: pgmap v2909: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:55.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:55 smithi107 ceph-mon[36485]: pgmap v2909: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:55.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:55 smithi106 ceph-mon[36505]: pgmap v2909: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:56.116 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5244 sec 2024-03-24T01:51:57.116 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5245 sec 2024-03-24T01:51:57.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:57 smithi078 ceph-mon[32429]: pgmap v2910: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:57.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:57 smithi107 ceph-mon[36485]: pgmap v2910: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:57.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:57 smithi106 ceph-mon[36505]: pgmap v2910: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:58.116 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5246 sec 2024-03-24T01:51:58.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4593.171203 secs 2024-03-24T01:51:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4593.171203 secs 2024-03-24T01:51:58.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4593.171203 secs 2024-03-24T01:51:59.116 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5247 sec 2024-03-24T01:51:59.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:51:59 smithi078 ceph-mon[32429]: pgmap v2911: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:51:59 smithi107 ceph-mon[36485]: pgmap v2911: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:51:59.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:51:59 smithi106 ceph-mon[36505]: pgmap v2911: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:00.116 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5248 sec 2024-03-24T01:52:01.116 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5249 sec 2024-03-24T01:52:01.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:01 smithi078 ceph-mon[32429]: pgmap v2912: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:01.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:01 smithi107 ceph-mon[36485]: pgmap v2912: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:01.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:01 smithi106 ceph-mon[36505]: pgmap v2912: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:02.116 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5250 sec 2024-03-24T01:52:03.116 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5251 sec 2024-03-24T01:52:03.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:03 smithi078 ceph-mon[32429]: pgmap v2913: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:03.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4598.171273 secs 2024-03-24T01:52:03.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:03 smithi107 ceph-mon[36485]: pgmap v2913: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:03.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4598.171273 secs 2024-03-24T01:52:03.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:03 smithi106 ceph-mon[36505]: pgmap v2913: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:03.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4598.171273 secs 2024-03-24T01:52:04.117 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5252 sec 2024-03-24T01:52:05.117 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5253 sec 2024-03-24T01:52:05.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:05 smithi078 ceph-mon[32429]: pgmap v2914: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:05.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:05 smithi107 ceph-mon[36485]: pgmap v2914: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:05.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:05 smithi106 ceph-mon[36505]: pgmap v2914: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:06.117 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5254 sec 2024-03-24T01:52:07.117 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5255 sec 2024-03-24T01:52:07.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:07 smithi078 ceph-mon[32429]: pgmap v2915: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:07.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:07 smithi107 ceph-mon[36485]: pgmap v2915: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:07.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:07 smithi106 ceph-mon[36505]: pgmap v2915: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:08.117 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5256 sec 2024-03-24T01:52:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4603.171357 secs 2024-03-24T01:52:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4603.171357 secs 2024-03-24T01:52:08.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4603.171357 secs 2024-03-24T01:52:09.117 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5257 sec 2024-03-24T01:52:09.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:09 smithi078 ceph-mon[32429]: pgmap v2916: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:09.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:09 smithi107 ceph-mon[36485]: pgmap v2916: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:09.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:09 smithi106 ceph-mon[36505]: pgmap v2916: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:10.117 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5258 sec 2024-03-24T01:52:10.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:52:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:52:10.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:52:11.117 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5259 sec 2024-03-24T01:52:11.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:11 smithi078 ceph-mon[32429]: pgmap v2917: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:11.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:11 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:52:11.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:11 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:52:11.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:11 smithi078 ceph-mon[32429]: pgmap v2918: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:52:11.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:11 smithi078 ceph-mon[32429]: pgmap v2919: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:52:11.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:11 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:52:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:11 smithi107 ceph-mon[36485]: pgmap v2917: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:11.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:11 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:52:11.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:11 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:52:11.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:11 smithi107 ceph-mon[36485]: pgmap v2918: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:52:11.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:11 smithi107 ceph-mon[36485]: pgmap v2919: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:52:11.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:11 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:52:11.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:11 smithi106 ceph-mon[36505]: pgmap v2917: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:11.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:11 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:52:11.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:11 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:52:11.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:11 smithi106 ceph-mon[36505]: pgmap v2918: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:52:11.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:11 smithi106 ceph-mon[36505]: pgmap v2919: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:52:11.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:11 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:52:12.117 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5260 sec 2024-03-24T01:52:13.118 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5261 sec 2024-03-24T01:52:14.117 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5262 sec 2024-03-24T01:52:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4608.171442 secs 2024-03-24T01:52:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:14 smithi078 ceph-mon[32429]: pgmap v2920: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:52:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4608.171442 secs 2024-03-24T01:52:14.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:14 smithi107 ceph-mon[36485]: pgmap v2920: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:52:14.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4608.171442 secs 2024-03-24T01:52:14.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:14 smithi106 ceph-mon[36505]: pgmap v2920: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:52:15.117 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5263 sec 2024-03-24T01:52:16.117 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5264 sec 2024-03-24T01:52:16.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:16 smithi078 ceph-mon[32429]: pgmap v2921: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:52:16.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:16 smithi107 ceph-mon[36485]: pgmap v2921: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:52:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:16 smithi106 ceph-mon[36505]: pgmap v2921: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:52:17.118 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5265 sec 2024-03-24T01:52:18.118 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5266 sec 2024-03-24T01:52:18.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:18 smithi078 ceph-mon[32429]: pgmap v2922: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:52:18.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:18 smithi107 ceph-mon[36485]: pgmap v2922: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:52:18.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:18 smithi106 ceph-mon[36505]: pgmap v2922: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:52:19.118 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5267 sec 2024-03-24T01:52:19.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4613.171530 secs 2024-03-24T01:52:19.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4613.171530 secs 2024-03-24T01:52:19.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4613.171530 secs 2024-03-24T01:52:20.118 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5268 sec 2024-03-24T01:52:20.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:20 smithi078 ceph-mon[32429]: pgmap v2923: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:52:20.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:20 smithi107 ceph-mon[36485]: pgmap v2923: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:52:20.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:20 smithi106 ceph-mon[36505]: pgmap v2923: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:52:21.118 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5269 sec 2024-03-24T01:52:22.118 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5270 sec 2024-03-24T01:52:22.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:22 smithi078 ceph-mon[32429]: pgmap v2924: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:52:22.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:22 smithi107 ceph-mon[36485]: pgmap v2924: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:52:22.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:22 smithi106 ceph-mon[36505]: pgmap v2924: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:52:23.118 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5271 sec 2024-03-24T01:52:24.118 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5272 sec 2024-03-24T01:52:24.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4618.171593 secs 2024-03-24T01:52:24.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:24 smithi078 ceph-mon[32429]: pgmap v2925: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4618.171593 secs 2024-03-24T01:52:24.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:24 smithi107 ceph-mon[36485]: pgmap v2925: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:24.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4618.171593 secs 2024-03-24T01:52:24.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:24 smithi106 ceph-mon[36505]: pgmap v2925: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:25.118 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5273 sec 2024-03-24T01:52:26.118 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5274 sec 2024-03-24T01:52:26.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:26 smithi078 ceph-mon[32429]: pgmap v2926: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:26.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:26 smithi107 ceph-mon[36485]: pgmap v2926: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:26 smithi106 ceph-mon[36505]: pgmap v2926: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:27.118 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5275 sec 2024-03-24T01:52:28.118 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5276 sec 2024-03-24T01:52:28.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:28 smithi078 ceph-mon[32429]: pgmap v2927: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:28.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:28 smithi107 ceph-mon[36485]: pgmap v2927: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:28.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:28 smithi106 ceph-mon[36505]: pgmap v2927: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:29.118 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5277 sec 2024-03-24T01:52:29.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4623.171652 secs 2024-03-24T01:52:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4623.171652 secs 2024-03-24T01:52:29.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4623.171652 secs 2024-03-24T01:52:30.119 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5278 sec 2024-03-24T01:52:30.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:30 smithi078 ceph-mon[32429]: pgmap v2928: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:30.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:30 smithi107 ceph-mon[36485]: pgmap v2928: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:30.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:30 smithi106 ceph-mon[36505]: pgmap v2928: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:31.119 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5279 sec 2024-03-24T01:52:32.119 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5280 sec 2024-03-24T01:52:32.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:32 smithi078 ceph-mon[32429]: pgmap v2929: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:32.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:32 smithi107 ceph-mon[36485]: pgmap v2929: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:32.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:32 smithi106 ceph-mon[36505]: pgmap v2929: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:33.119 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5281 sec 2024-03-24T01:52:34.119 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5282 sec 2024-03-24T01:52:34.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4628.171712 secs 2024-03-24T01:52:34.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:34 smithi078 ceph-mon[32429]: pgmap v2930: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4628.171712 secs 2024-03-24T01:52:34.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:34 smithi107 ceph-mon[36485]: pgmap v2930: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:34.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4628.171712 secs 2024-03-24T01:52:34.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:34 smithi106 ceph-mon[36505]: pgmap v2930: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:35.119 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5283 sec 2024-03-24T01:52:36.119 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5284 sec 2024-03-24T01:52:36.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:36 smithi078 ceph-mon[32429]: pgmap v2931: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:36.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:36 smithi107 ceph-mon[36485]: pgmap v2931: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:36.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:36 smithi106 ceph-mon[36505]: pgmap v2931: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:37.119 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5285 sec 2024-03-24T01:52:38.119 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5286 sec 2024-03-24T01:52:38.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:38 smithi078 ceph-mon[32429]: pgmap v2932: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:38.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:38 smithi107 ceph-mon[36485]: pgmap v2932: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:38.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:38 smithi106 ceph-mon[36505]: pgmap v2932: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:39.119 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5287 sec 2024-03-24T01:52:39.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4633.171773 secs 2024-03-24T01:52:39.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4633.171773 secs 2024-03-24T01:52:39.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4633.171773 secs 2024-03-24T01:52:40.119 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5288 sec 2024-03-24T01:52:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:40 smithi078 ceph-mon[32429]: pgmap v2933: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:40.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:40 smithi107 ceph-mon[36485]: pgmap v2933: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:40.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:40 smithi106 ceph-mon[36505]: pgmap v2933: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:41.120 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5289 sec 2024-03-24T01:52:42.120 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5290 sec 2024-03-24T01:52:42.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:42 smithi078 ceph-mon[32429]: pgmap v2934: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:42.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:42 smithi107 ceph-mon[36485]: pgmap v2934: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:42.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:42 smithi106 ceph-mon[36505]: pgmap v2934: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:43.120 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5291 sec 2024-03-24T01:52:44.120 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5292 sec 2024-03-24T01:52:44.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4638.171832 secs 2024-03-24T01:52:44.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:44 smithi078 ceph-mon[32429]: pgmap v2935: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4638.171832 secs 2024-03-24T01:52:44.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:44 smithi107 ceph-mon[36485]: pgmap v2935: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:44.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4638.171832 secs 2024-03-24T01:52:44.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:44 smithi106 ceph-mon[36505]: pgmap v2935: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:45.120 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5293 sec 2024-03-24T01:52:46.120 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5294 sec 2024-03-24T01:52:46.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:46 smithi078 ceph-mon[32429]: pgmap v2936: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:46.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:46 smithi107 ceph-mon[36485]: pgmap v2936: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:46.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:46 smithi106 ceph-mon[36505]: pgmap v2936: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:47.120 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5295 sec 2024-03-24T01:52:48.120 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5296 sec 2024-03-24T01:52:48.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:48 smithi078 ceph-mon[32429]: pgmap v2937: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:48.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:48 smithi107 ceph-mon[36485]: pgmap v2937: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:48.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:48 smithi106 ceph-mon[36505]: pgmap v2937: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:49.120 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5297 sec 2024-03-24T01:52:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4643.171919 secs 2024-03-24T01:52:49.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4643.171919 secs 2024-03-24T01:52:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4643.171919 secs 2024-03-24T01:52:50.121 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5298 sec 2024-03-24T01:52:50.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:50 smithi078 ceph-mon[32429]: pgmap v2938: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:50.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:50 smithi107 ceph-mon[36485]: pgmap v2938: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:50.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:50 smithi106 ceph-mon[36505]: pgmap v2938: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:51.121 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5299 sec 2024-03-24T01:52:52.121 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5300 sec 2024-03-24T01:52:52.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:52 smithi078 ceph-mon[32429]: pgmap v2939: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:52.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:52 smithi107 ceph-mon[36485]: pgmap v2939: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:52.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:52 smithi106 ceph-mon[36505]: pgmap v2939: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:53.121 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5301 sec 2024-03-24T01:52:54.121 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5302 sec 2024-03-24T01:52:54.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4648.171979 secs 2024-03-24T01:52:54.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:54 smithi078 ceph-mon[32429]: pgmap v2940: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:54.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4648.171979 secs 2024-03-24T01:52:54.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:54 smithi107 ceph-mon[36485]: pgmap v2940: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:54.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4648.171979 secs 2024-03-24T01:52:54.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:54 smithi106 ceph-mon[36505]: pgmap v2940: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:55.121 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5303 sec 2024-03-24T01:52:56.121 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5304 sec 2024-03-24T01:52:56.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:56 smithi078 ceph-mon[32429]: pgmap v2941: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:56.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:56 smithi107 ceph-mon[36485]: pgmap v2941: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:56 smithi106 ceph-mon[36505]: pgmap v2941: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:52:57.121 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5305 sec 2024-03-24T01:52:58.122 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5306 sec 2024-03-24T01:52:58.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:58 smithi107 ceph-mon[36485]: pgmap v2942: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:58.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:58 smithi106 ceph-mon[36505]: pgmap v2942: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:58.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:58 smithi078 ceph-mon[32429]: pgmap v2942: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:59.121 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5307 sec 2024-03-24T01:52:59.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:52:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4653.172061 secs 2024-03-24T01:52:59.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:52:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4653.172061 secs 2024-03-24T01:52:59.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:52:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4653.172061 secs 2024-03-24T01:53:00.121 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5308 sec 2024-03-24T01:53:00.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:00 smithi106 ceph-mon[36505]: pgmap v2943: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:00 smithi078 ceph-mon[32429]: pgmap v2943: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:00 smithi107 ceph-mon[36485]: pgmap v2943: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:01.121 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5309 sec 2024-03-24T01:53:01.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:01 smithi106 ceph-mon[36505]: pgmap v2944: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:01.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:01 smithi078 ceph-mon[32429]: pgmap v2944: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:01.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:01 smithi107 ceph-mon[36485]: pgmap v2944: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:02.122 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5310 sec 2024-03-24T01:53:03.122 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5311 sec 2024-03-24T01:53:04.122 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5312 sec 2024-03-24T01:53:04.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4658.172151 secs 2024-03-24T01:53:04.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:04 smithi078 ceph-mon[32429]: pgmap v2945: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:04.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4658.172151 secs 2024-03-24T01:53:04.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:04 smithi107 ceph-mon[36485]: pgmap v2945: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:04.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4658.172151 secs 2024-03-24T01:53:04.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:04 smithi106 ceph-mon[36505]: pgmap v2945: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:05.122 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5313 sec 2024-03-24T01:53:06.122 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5314 sec 2024-03-24T01:53:06.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:06 smithi078 ceph-mon[32429]: pgmap v2946: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:06.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:06 smithi107 ceph-mon[36485]: pgmap v2946: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:06.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:06 smithi106 ceph-mon[36505]: pgmap v2946: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:07.122 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5315 sec 2024-03-24T01:53:08.122 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5316 sec 2024-03-24T01:53:08.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:08 smithi078 ceph-mon[32429]: pgmap v2947: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:08.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:08 smithi107 ceph-mon[36485]: pgmap v2947: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:08.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:08 smithi106 ceph-mon[36505]: pgmap v2947: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:09.122 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5317 sec 2024-03-24T01:53:09.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4663.172241 secs 2024-03-24T01:53:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4663.172241 secs 2024-03-24T01:53:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4663.172241 secs 2024-03-24T01:53:10.122 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5318 sec 2024-03-24T01:53:10.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:10 smithi078 ceph-mon[32429]: pgmap v2948: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:10.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:10 smithi107 ceph-mon[36485]: pgmap v2948: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:10.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:10 smithi106 ceph-mon[36505]: pgmap v2948: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:11.122 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5319 sec 2024-03-24T01:53:12.122 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5320 sec 2024-03-24T01:53:12.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:12 smithi106 ceph-mon[36505]: pgmap v2949: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:12.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:12 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:53:12.408 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:12 smithi107 ceph-mon[36485]: pgmap v2949: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:12.408 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:12 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:53:12.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:12 smithi078 ceph-mon[32429]: pgmap v2949: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:12.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:12 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:53:13.123 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5321 sec 2024-03-24T01:53:14.122 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5322 sec 2024-03-24T01:53:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4668.172299 secs 2024-03-24T01:53:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:14 smithi078 ceph-mon[32429]: pgmap v2950: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:14 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:14 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:14 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:14 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:14 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:14 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4668.172299 secs 2024-03-24T01:53:14.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:14 smithi107 ceph-mon[36485]: pgmap v2950: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:14.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:14 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:14.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:14 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:14.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:14 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:14.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:14 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:14.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:14 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:14.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:14 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:14.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4668.172299 secs 2024-03-24T01:53:14.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:14 smithi106 ceph-mon[36505]: pgmap v2950: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:14.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:14 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:14.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:14 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:14.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:14 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:14.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:14 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:14.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:14 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:14.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:14 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:15.123 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5323 sec 2024-03-24T01:53:15.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:15.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:15.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.655 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.655 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.655 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:15.655 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:15.655 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.655 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.655 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.655 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.655 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:15.655 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:15.655 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.655 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.655 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.655 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.656 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:53:15.656 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:53:15.656 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:15.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:15.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:15.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:15.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:15.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:15.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:15.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:53:15.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:53:15.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:15.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:15.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:15.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:15.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:15.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:15.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:15.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:53:15.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:53:15.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:53:15.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:53:16.123 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5324 sec 2024-03-24T01:53:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:16 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T01:53:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:16 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T01:53:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:16 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T01:53:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:16 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T01:53:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:16 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T01:53:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:16 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T01:53:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:16 smithi106 ceph-mon[36505]: pgmap v2951: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:16 smithi106 ceph-mon[36505]: pgmap v2952: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:53:16.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:16 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T01:53:16.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:16 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T01:53:16.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:16 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T01:53:16.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:16 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T01:53:16.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:16 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T01:53:16.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:16 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T01:53:16.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:16 smithi078 ceph-mon[32429]: pgmap v2951: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:16.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:16 smithi078 ceph-mon[32429]: pgmap v2952: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:53:16.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:16 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T01:53:16.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:16 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T01:53:16.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:16 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T01:53:16.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:16 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T01:53:16.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:16 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T01:53:16.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:16 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T01:53:16.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:16 smithi107 ceph-mon[36485]: pgmap v2951: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:16.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:16 smithi107 ceph-mon[36485]: pgmap v2952: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:53:17.123 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5325 sec 2024-03-24T01:53:18.123 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5326 sec 2024-03-24T01:53:18.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:18 smithi078 ceph-mon[32429]: pgmap v2953: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:53:18.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:18 smithi106 ceph-mon[36505]: pgmap v2953: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:53:18.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:18 smithi107 ceph-mon[36485]: pgmap v2953: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:53:19.123 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5327 sec 2024-03-24T01:53:19.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4673.172388 secs 2024-03-24T01:53:19.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4673.172388 secs 2024-03-24T01:53:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4673.172388 secs 2024-03-24T01:53:20.123 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5328 sec 2024-03-24T01:53:20.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:20 smithi106 ceph-mon[36505]: pgmap v2954: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:53:20.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:20 smithi078 ceph-mon[32429]: pgmap v2954: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:53:20.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:20 smithi107 ceph-mon[36485]: pgmap v2954: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:53:21.123 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5329 sec 2024-03-24T01:53:21.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:21 smithi106 ceph-mon[36505]: pgmap v2955: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:53:21.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:21 smithi078 ceph-mon[32429]: pgmap v2955: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:53:21.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:21 smithi107 ceph-mon[36485]: pgmap v2955: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:53:22.123 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5330 sec 2024-03-24T01:53:23.123 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5331 sec 2024-03-24T01:53:24.124 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5332 sec 2024-03-24T01:53:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:23 smithi106 ceph-mon[36505]: pgmap v2956: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:53:24.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4678.172447 secs 2024-03-24T01:53:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:23 smithi078 ceph-mon[32429]: pgmap v2956: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:53:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4678.172447 secs 2024-03-24T01:53:24.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:23 smithi107 ceph-mon[36485]: pgmap v2956: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T01:53:24.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4678.172447 secs 2024-03-24T01:53:25.124 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5333 sec 2024-03-24T01:53:26.124 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5334 sec 2024-03-24T01:53:26.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:25 smithi106 ceph-mon[36505]: pgmap v2957: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:53:26.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:25 smithi078 ceph-mon[32429]: pgmap v2957: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:53:26.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:25 smithi107 ceph-mon[36485]: pgmap v2957: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:53:27.124 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5335 sec 2024-03-24T01:53:28.124 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5336 sec 2024-03-24T01:53:28.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:27 smithi106 ceph-mon[36505]: pgmap v2958: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:28.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:27 smithi078 ceph-mon[32429]: pgmap v2958: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:28.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:27 smithi107 ceph-mon[36485]: pgmap v2958: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:29.124 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5337 sec 2024-03-24T01:53:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4683.172533 secs 2024-03-24T01:53:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4683.172533 secs 2024-03-24T01:53:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4683.172533 secs 2024-03-24T01:53:30.124 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5338 sec 2024-03-24T01:53:30.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:29 smithi106 ceph-mon[36505]: pgmap v2959: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:30.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:29 smithi078 ceph-mon[32429]: pgmap v2959: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:30.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:29 smithi107 ceph-mon[36485]: pgmap v2959: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:31.124 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5339 sec 2024-03-24T01:53:32.124 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5340 sec 2024-03-24T01:53:32.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:31 smithi106 ceph-mon[36505]: pgmap v2960: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:32.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:31 smithi078 ceph-mon[32429]: pgmap v2960: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:32.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:31 smithi107 ceph-mon[36485]: pgmap v2960: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:33.124 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5341 sec 2024-03-24T01:53:34.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:33 smithi078 ceph-mon[32429]: pgmap v2961: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:34.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4688.172623 secs 2024-03-24T01:53:34.124 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5342 sec 2024-03-24T01:53:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:33 smithi106 ceph-mon[36505]: pgmap v2961: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:34.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4688.172623 secs 2024-03-24T01:53:34.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:33 smithi107 ceph-mon[36485]: pgmap v2961: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:34.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4688.172623 secs 2024-03-24T01:53:35.125 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5343 sec 2024-03-24T01:53:36.125 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5344 sec 2024-03-24T01:53:36.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:35 smithi106 ceph-mon[36505]: pgmap v2962: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:36.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:35 smithi078 ceph-mon[32429]: pgmap v2962: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:36.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:35 smithi107 ceph-mon[36485]: pgmap v2962: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:37.125 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5345 sec 2024-03-24T01:53:38.125 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5346 sec 2024-03-24T01:53:38.143 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:37 smithi078 ceph-mon[32429]: pgmap v2963: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:38.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:37 smithi106 ceph-mon[36505]: pgmap v2963: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:38.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:37 smithi107 ceph-mon[36485]: pgmap v2963: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:39.125 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5347 sec 2024-03-24T01:53:39.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4693.172683 secs 2024-03-24T01:53:39.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4693.172683 secs 2024-03-24T01:53:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4693.172683 secs 2024-03-24T01:53:40.125 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5348 sec 2024-03-24T01:53:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:39 smithi106 ceph-mon[36505]: pgmap v2964: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:40.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:39 smithi078 ceph-mon[32429]: pgmap v2964: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:39 smithi107 ceph-mon[36485]: pgmap v2964: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:41.125 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5349 sec 2024-03-24T01:53:42.125 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5350 sec 2024-03-24T01:53:42.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:41 smithi106 ceph-mon[36505]: pgmap v2965: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:42.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:41 smithi078 ceph-mon[32429]: pgmap v2965: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:42.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:41 smithi107 ceph-mon[36485]: pgmap v2965: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:43.125 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5351 sec 2024-03-24T01:53:44.125 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5352 sec 2024-03-24T01:53:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:43 smithi106 ceph-mon[36505]: pgmap v2966: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:44.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4698.172769 secs 2024-03-24T01:53:44.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:43 smithi078 ceph-mon[32429]: pgmap v2966: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:44.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4698.172769 secs 2024-03-24T01:53:44.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:43 smithi107 ceph-mon[36485]: pgmap v2966: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:44.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4698.172769 secs 2024-03-24T01:53:45.125 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5353 sec 2024-03-24T01:53:46.126 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5354 sec 2024-03-24T01:53:46.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:45 smithi106 ceph-mon[36505]: pgmap v2967: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:46.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:45 smithi078 ceph-mon[32429]: pgmap v2967: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:46.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:45 smithi107 ceph-mon[36485]: pgmap v2967: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:47.125 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5355 sec 2024-03-24T01:53:48.126 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5356 sec 2024-03-24T01:53:48.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:47 smithi106 ceph-mon[36505]: pgmap v2968: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:48.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:47 smithi078 ceph-mon[32429]: pgmap v2968: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:48.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:47 smithi107 ceph-mon[36485]: pgmap v2968: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:49.126 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5357 sec 2024-03-24T01:53:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4703.172812 secs 2024-03-24T01:53:49.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4703.172812 secs 2024-03-24T01:53:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4703.172812 secs 2024-03-24T01:53:50.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:49 smithi078 ceph-mon[32429]: pgmap v2969: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:50.126 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5358 sec 2024-03-24T01:53:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:49 smithi106 ceph-mon[36505]: pgmap v2969: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:50.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:49 smithi107 ceph-mon[36485]: pgmap v2969: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:51.126 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5359 sec 2024-03-24T01:53:52.126 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5360 sec 2024-03-24T01:53:52.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:51 smithi078 ceph-mon[32429]: pgmap v2970: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:52.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:51 smithi107 ceph-mon[36485]: pgmap v2970: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:52.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:51 smithi106 ceph-mon[36505]: pgmap v2970: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:53.126 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5361 sec 2024-03-24T01:53:54.126 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5362 sec 2024-03-24T01:53:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:53 smithi078 ceph-mon[32429]: pgmap v2971: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4708.172870 secs 2024-03-24T01:53:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:53 smithi107 ceph-mon[36485]: pgmap v2971: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:54.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4708.172870 secs 2024-03-24T01:53:54.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:53 smithi106 ceph-mon[36505]: pgmap v2971: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:54.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4708.172870 secs 2024-03-24T01:53:55.126 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5363 sec 2024-03-24T01:53:56.126 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5364 sec 2024-03-24T01:53:56.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:55 smithi078 ceph-mon[32429]: pgmap v2972: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:56.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:55 smithi107 ceph-mon[36485]: pgmap v2972: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:56.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:55 smithi106 ceph-mon[36505]: pgmap v2972: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:53:57.126 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5365 sec 2024-03-24T01:53:58.127 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5366 sec 2024-03-24T01:53:58.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:57 smithi078 ceph-mon[32429]: pgmap v2973: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:58.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:57 smithi107 ceph-mon[36485]: pgmap v2973: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:58.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:57 smithi106 ceph-mon[36505]: pgmap v2973: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:59.127 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5367 sec 2024-03-24T01:53:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4713.172914 secs 2024-03-24T01:53:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4713.172914 secs 2024-03-24T01:53:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4713.172914 secs 2024-03-24T01:54:00.127 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5368 sec 2024-03-24T01:54:00.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:53:59 smithi078 ceph-mon[32429]: pgmap v2974: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:00.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:53:59 smithi107 ceph-mon[36485]: pgmap v2974: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:00.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:53:59 smithi106 ceph-mon[36505]: pgmap v2974: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:01.127 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5369 sec 2024-03-24T01:54:02.127 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5370 sec 2024-03-24T01:54:02.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:01 smithi078 ceph-mon[32429]: pgmap v2975: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:02.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:01 smithi107 ceph-mon[36485]: pgmap v2975: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:02.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:01 smithi106 ceph-mon[36505]: pgmap v2975: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:03.127 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5371 sec 2024-03-24T01:54:04.127 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5372 sec 2024-03-24T01:54:04.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:03 smithi078 ceph-mon[32429]: pgmap v2976: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:04.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4718.173000 secs 2024-03-24T01:54:04.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:03 smithi107 ceph-mon[36485]: pgmap v2976: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:04.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4718.173000 secs 2024-03-24T01:54:04.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:03 smithi106 ceph-mon[36505]: pgmap v2976: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:04.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4718.173000 secs 2024-03-24T01:54:05.127 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5373 sec 2024-03-24T01:54:06.127 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5374 sec 2024-03-24T01:54:06.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:05 smithi078 ceph-mon[32429]: pgmap v2977: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:06.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:05 smithi107 ceph-mon[36485]: pgmap v2977: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:06.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:05 smithi106 ceph-mon[36505]: pgmap v2977: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:07.127 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5375 sec 2024-03-24T01:54:08.127 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5376 sec 2024-03-24T01:54:08.204 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:07 smithi106 ceph-mon[36505]: pgmap v2978: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:08.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:07 smithi078 ceph-mon[32429]: pgmap v2978: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:08.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:07 smithi107 ceph-mon[36485]: pgmap v2978: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:09.128 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5377 sec 2024-03-24T01:54:09.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4723.173041 secs 2024-03-24T01:54:09.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4723.173041 secs 2024-03-24T01:54:09.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4723.173041 secs 2024-03-24T01:54:10.128 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5378 sec 2024-03-24T01:54:10.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:09 smithi078 ceph-mon[32429]: pgmap v2979: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:10.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:09 smithi107 ceph-mon[36485]: pgmap v2979: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:10.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:09 smithi106 ceph-mon[36505]: pgmap v2979: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:11.128 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5379 sec 2024-03-24T01:54:12.128 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5380 sec 2024-03-24T01:54:12.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:11 smithi078 ceph-mon[32429]: pgmap v2980: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:12.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:11 smithi107 ceph-mon[36485]: pgmap v2980: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:12.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:11 smithi106 ceph-mon[36505]: pgmap v2980: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:13.128 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5381 sec 2024-03-24T01:54:14.128 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5382 sec 2024-03-24T01:54:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:13 smithi078 ceph-mon[32429]: pgmap v2981: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4728.173100 secs 2024-03-24T01:54:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:13 smithi107 ceph-mon[36485]: pgmap v2981: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:14.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4728.173100 secs 2024-03-24T01:54:14.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:13 smithi106 ceph-mon[36505]: pgmap v2981: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:14.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4728.173100 secs 2024-03-24T01:54:15.128 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5383 sec 2024-03-24T01:54:15.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:14 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:54:15.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:14 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:54:15.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:14 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:54:16.128 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5384 sec 2024-03-24T01:54:16.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:15 smithi078 ceph-mon[32429]: pgmap v2982: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:16.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:54:16.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:54:16.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:54:16.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:54:16.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:54:16.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:54:16.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:15 smithi107 ceph-mon[36485]: pgmap v2982: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:16.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:54:16.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:54:16.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:54:16.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:54:16.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:54:16.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:54:16.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:15 smithi106 ceph-mon[36505]: pgmap v2982: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:16.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:54:16.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:54:16.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:54:16.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:54:16.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:54:16.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:54:17.128 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5385 sec 2024-03-24T01:54:17.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:54:17.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:54:17.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:16 smithi078 ceph-mon[32429]: pgmap v2983: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:54:17.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:16 smithi078 ceph-mon[32429]: pgmap v2984: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-24T01:54:17.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:54:17.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:54:17.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:54:17.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:16 smithi107 ceph-mon[36485]: pgmap v2983: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:54:17.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:16 smithi107 ceph-mon[36485]: pgmap v2984: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-24T01:54:17.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:54:17.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:54:17.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:54:17.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:16 smithi106 ceph-mon[36505]: pgmap v2983: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:54:17.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:16 smithi106 ceph-mon[36505]: pgmap v2984: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-24T01:54:17.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:54:18.128 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5386 sec 2024-03-24T01:54:19.128 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5387 sec 2024-03-24T01:54:19.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4733.173179 secs 2024-03-24T01:54:19.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:19 smithi078 ceph-mon[32429]: pgmap v2985: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-24T01:54:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4733.173179 secs 2024-03-24T01:54:19.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:19 smithi107 ceph-mon[36485]: pgmap v2985: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-24T01:54:19.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4733.173179 secs 2024-03-24T01:54:19.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:19 smithi106 ceph-mon[36505]: pgmap v2985: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-24T01:54:20.128 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5388 sec 2024-03-24T01:54:21.129 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5389 sec 2024-03-24T01:54:21.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:21 smithi078 ceph-mon[32429]: pgmap v2986: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 990 B/s rd, 2 op/s 2024-03-24T01:54:21.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:21 smithi107 ceph-mon[36485]: pgmap v2986: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 990 B/s rd, 2 op/s 2024-03-24T01:54:21.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:21 smithi106 ceph-mon[36505]: pgmap v2986: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 990 B/s rd, 2 op/s 2024-03-24T01:54:22.129 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5390 sec 2024-03-24T01:54:23.129 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5391 sec 2024-03-24T01:54:23.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:23 smithi078 ceph-mon[32429]: pgmap v2987: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-24T01:54:23.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:23 smithi107 ceph-mon[36485]: pgmap v2987: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-24T01:54:23.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:23 smithi106 ceph-mon[36505]: pgmap v2987: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-24T01:54:24.129 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5392 sec 2024-03-24T01:54:24.495 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4738.173238 secs 2024-03-24T01:54:24.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4738.173238 secs 2024-03-24T01:54:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4738.173238 secs 2024-03-24T01:54:25.129 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5393 sec 2024-03-24T01:54:25.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:25 smithi078 ceph-mon[32429]: pgmap v2988: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-24T01:54:25.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:25 smithi107 ceph-mon[36485]: pgmap v2988: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-24T01:54:25.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:25 smithi106 ceph-mon[36505]: pgmap v2988: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-24T01:54:26.129 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5394 sec 2024-03-24T01:54:27.129 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5395 sec 2024-03-24T01:54:27.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:27 smithi078 ceph-mon[32429]: pgmap v2989: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:54:27.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:27 smithi107 ceph-mon[36485]: pgmap v2989: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:54:27.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:27 smithi106 ceph-mon[36505]: pgmap v2989: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:54:28.129 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5396 sec 2024-03-24T01:54:29.129 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5397 sec 2024-03-24T01:54:29.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4743.173290 secs 2024-03-24T01:54:29.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:29 smithi078 ceph-mon[32429]: pgmap v2990: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4743.173290 secs 2024-03-24T01:54:29.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:29 smithi107 ceph-mon[36485]: pgmap v2990: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:29.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4743.173290 secs 2024-03-24T01:54:29.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:29 smithi106 ceph-mon[36505]: pgmap v2990: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:30.129 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5398 sec 2024-03-24T01:54:31.130 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5399 sec 2024-03-24T01:54:31.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:31 smithi078 ceph-mon[32429]: pgmap v2991: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:31.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:31 smithi107 ceph-mon[36485]: pgmap v2991: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:31.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:31 smithi106 ceph-mon[36505]: pgmap v2991: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:32.130 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5400 sec 2024-03-24T01:54:33.130 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5401 sec 2024-03-24T01:54:33.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:33 smithi078 ceph-mon[32429]: pgmap v2992: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:33.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:33 smithi107 ceph-mon[36485]: pgmap v2992: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:33.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:33 smithi106 ceph-mon[36505]: pgmap v2992: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:34.130 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5402 sec 2024-03-24T01:54:34.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4748.173350 secs 2024-03-24T01:54:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4748.173350 secs 2024-03-24T01:54:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4748.173350 secs 2024-03-24T01:54:35.130 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5403 sec 2024-03-24T01:54:35.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:35 smithi078 ceph-mon[32429]: pgmap v2993: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:35.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:35 smithi107 ceph-mon[36485]: pgmap v2993: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:35.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:35 smithi106 ceph-mon[36505]: pgmap v2993: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:36.130 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5404 sec 2024-03-24T01:54:37.130 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5405 sec 2024-03-24T01:54:37.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:37 smithi078 ceph-mon[32429]: pgmap v2994: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:37.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:37 smithi107 ceph-mon[36485]: pgmap v2994: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:37.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:37 smithi106 ceph-mon[36505]: pgmap v2994: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:38.130 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5406 sec 2024-03-24T01:54:39.130 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5407 sec 2024-03-24T01:54:39.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4753.173411 secs 2024-03-24T01:54:39.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:39 smithi078 ceph-mon[32429]: pgmap v2995: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4753.173411 secs 2024-03-24T01:54:39.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:39 smithi107 ceph-mon[36485]: pgmap v2995: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:39.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4753.173411 secs 2024-03-24T01:54:39.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:39 smithi106 ceph-mon[36505]: pgmap v2995: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:40.130 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5408 sec 2024-03-24T01:54:41.130 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5409 sec 2024-03-24T01:54:41.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:41 smithi078 ceph-mon[32429]: pgmap v2996: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:41.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:41 smithi107 ceph-mon[36485]: pgmap v2996: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:41.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:41 smithi106 ceph-mon[36505]: pgmap v2996: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:42.130 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5410 sec 2024-03-24T01:54:43.130 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5411 sec 2024-03-24T01:54:43.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:43 smithi078 ceph-mon[32429]: pgmap v2997: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:43.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:43 smithi107 ceph-mon[36485]: pgmap v2997: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:43.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:43 smithi106 ceph-mon[36505]: pgmap v2997: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:44.131 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5412 sec 2024-03-24T01:54:44.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4758.173471 secs 2024-03-24T01:54:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4758.173471 secs 2024-03-24T01:54:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4758.173471 secs 2024-03-24T01:54:45.131 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5413 sec 2024-03-24T01:54:45.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:45 smithi078 ceph-mon[32429]: pgmap v2998: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:45.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:45 smithi107 ceph-mon[36485]: pgmap v2998: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:45.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:45 smithi106 ceph-mon[36505]: pgmap v2998: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:46.131 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5414 sec 2024-03-24T01:54:47.131 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5415 sec 2024-03-24T01:54:47.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:47 smithi078 ceph-mon[32429]: pgmap v2999: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:47 smithi107 ceph-mon[36485]: pgmap v2999: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:47.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:47 smithi106 ceph-mon[36505]: pgmap v2999: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:48.131 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5416 sec 2024-03-24T01:54:49.131 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5417 sec 2024-03-24T01:54:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4763.173554 secs 2024-03-24T01:54:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:49 smithi078 ceph-mon[32429]: pgmap v3000: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:49.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4763.173554 secs 2024-03-24T01:54:49.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:49 smithi107 ceph-mon[36485]: pgmap v3000: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4763.173554 secs 2024-03-24T01:54:49.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:49 smithi106 ceph-mon[36505]: pgmap v3000: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:50.131 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5418 sec 2024-03-24T01:54:51.131 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5419 sec 2024-03-24T01:54:51.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:51 smithi078 ceph-mon[32429]: pgmap v3001: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:51.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:51 smithi107 ceph-mon[36485]: pgmap v3001: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:51.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:51 smithi106 ceph-mon[36505]: pgmap v3001: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:52.131 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5420 sec 2024-03-24T01:54:53.132 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5421 sec 2024-03-24T01:54:53.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:53 smithi078 ceph-mon[32429]: pgmap v3002: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:53.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:53 smithi107 ceph-mon[36485]: pgmap v3002: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:53.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:53 smithi106 ceph-mon[36505]: pgmap v3002: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:54.132 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5422 sec 2024-03-24T01:54:54.495 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4768.173615 secs 2024-03-24T01:54:54.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4768.173615 secs 2024-03-24T01:54:54.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4768.173615 secs 2024-03-24T01:54:55.132 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5423 sec 2024-03-24T01:54:55.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:55 smithi078 ceph-mon[32429]: pgmap v3003: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:55.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:55 smithi107 ceph-mon[36485]: pgmap v3003: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:55.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:55 smithi106 ceph-mon[36505]: pgmap v3003: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:56.132 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5424 sec 2024-03-24T01:54:57.132 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5425 sec 2024-03-24T01:54:57.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:57 smithi078 ceph-mon[32429]: pgmap v3004: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:57.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:57 smithi107 ceph-mon[36485]: pgmap v3004: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:57.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:57 smithi106 ceph-mon[36505]: pgmap v3004: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:58.132 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5426 sec 2024-03-24T01:54:59.132 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5427 sec 2024-03-24T01:54:59.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4773.173676 secs 2024-03-24T01:54:59.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:54:59 smithi078 ceph-mon[32429]: pgmap v3005: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4773.173676 secs 2024-03-24T01:54:59.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:54:59 smithi107 ceph-mon[36485]: pgmap v3005: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:54:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4773.173676 secs 2024-03-24T01:54:59.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:54:59 smithi106 ceph-mon[36505]: pgmap v3005: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:00.132 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5428 sec 2024-03-24T01:55:01.132 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5429 sec 2024-03-24T01:55:01.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:01 smithi078 ceph-mon[32429]: pgmap v3006: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:01.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:01 smithi107 ceph-mon[36485]: pgmap v3006: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:01.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:01 smithi106 ceph-mon[36505]: pgmap v3006: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:02.132 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5430 sec 2024-03-24T01:55:03.132 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5431 sec 2024-03-24T01:55:03.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:03 smithi078 ceph-mon[32429]: pgmap v3007: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:03.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:03 smithi107 ceph-mon[36485]: pgmap v3007: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:03.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:03 smithi106 ceph-mon[36505]: pgmap v3007: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:04.132 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5432 sec 2024-03-24T01:55:04.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4778.173735 secs 2024-03-24T01:55:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4778.173735 secs 2024-03-24T01:55:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4778.173735 secs 2024-03-24T01:55:05.133 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5433 sec 2024-03-24T01:55:05.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:05 smithi078 ceph-mon[32429]: pgmap v3008: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:05.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:05 smithi107 ceph-mon[36485]: pgmap v3008: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:05.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:05 smithi106 ceph-mon[36505]: pgmap v3008: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:06.133 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5434 sec 2024-03-24T01:55:07.133 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5435 sec 2024-03-24T01:55:07.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:07 smithi078 ceph-mon[32429]: pgmap v3009: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:07.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:07 smithi107 ceph-mon[36485]: pgmap v3009: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:07.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:07 smithi106 ceph-mon[36505]: pgmap v3009: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:08.133 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5436 sec 2024-03-24T01:55:09.133 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5437 sec 2024-03-24T01:55:09.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4783.173823 secs 2024-03-24T01:55:09.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:09 smithi078 ceph-mon[32429]: pgmap v3010: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4783.173823 secs 2024-03-24T01:55:09.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:09 smithi107 ceph-mon[36485]: pgmap v3010: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:09.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4783.173823 secs 2024-03-24T01:55:09.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:09 smithi106 ceph-mon[36505]: pgmap v3010: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:10.133 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5438 sec 2024-03-24T01:55:11.133 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5439 sec 2024-03-24T01:55:11.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:11 smithi078 ceph-mon[32429]: pgmap v3011: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:11.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:11 smithi107 ceph-mon[36485]: pgmap v3011: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:11.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:11 smithi106 ceph-mon[36505]: pgmap v3011: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:12.133 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5440 sec 2024-03-24T01:55:13.133 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5441 sec 2024-03-24T01:55:13.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:13 smithi078 ceph-mon[32429]: pgmap v3012: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:13.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:13 smithi107 ceph-mon[36485]: pgmap v3012: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:13.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:13 smithi106 ceph-mon[36505]: pgmap v3012: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:14.133 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5442 sec 2024-03-24T01:55:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4788.173910 secs 2024-03-24T01:55:14.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4788.173910 secs 2024-03-24T01:55:14.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4788.173910 secs 2024-03-24T01:55:15.134 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5443 sec 2024-03-24T01:55:15.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:15 smithi078 ceph-mon[32429]: pgmap v3013: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:15.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:15 smithi107 ceph-mon[36485]: pgmap v3013: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:15.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:15 smithi106 ceph-mon[36505]: pgmap v3013: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:16.133 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5444 sec 2024-03-24T01:55:16.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:55:16.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:55:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:55:17.134 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5445 sec 2024-03-24T01:55:17.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:17 smithi078 ceph-mon[32429]: pgmap v3014: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:17.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:17 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:55:17.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:17 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:55:17.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:17 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:55:17.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:17 smithi107 ceph-mon[36485]: pgmap v3014: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:17.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:17 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:55:17.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:17 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:55:17.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:17 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:55:17.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:17 smithi106 ceph-mon[36505]: pgmap v3014: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:17.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:17 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:55:17.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:17 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:55:17.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:17 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:55:18.134 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5446 sec 2024-03-24T01:55:18.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:18 smithi078 ceph-mon[32429]: pgmap v3015: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:55:18.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:18 smithi078 ceph-mon[32429]: pgmap v3016: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:55:18.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:18 smithi107 ceph-mon[36485]: pgmap v3015: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:55:18.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:18 smithi107 ceph-mon[36485]: pgmap v3016: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:55:18.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:18 smithi106 ceph-mon[36505]: pgmap v3015: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:55:18.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:18 smithi106 ceph-mon[36505]: pgmap v3016: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:55:19.134 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5447 sec 2024-03-24T01:55:19.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4793.173970 secs 2024-03-24T01:55:19.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4793.173970 secs 2024-03-24T01:55:19.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4793.173970 secs 2024-03-24T01:55:20.134 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5448 sec 2024-03-24T01:55:20.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:20 smithi078 ceph-mon[32429]: pgmap v3017: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:55:20.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:20 smithi107 ceph-mon[36485]: pgmap v3017: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:55:20.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:20 smithi106 ceph-mon[36505]: pgmap v3017: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:55:21.134 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5449 sec 2024-03-24T01:55:22.134 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5450 sec 2024-03-24T01:55:22.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:22 smithi078 ceph-mon[32429]: pgmap v3018: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:55:22.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:22 smithi107 ceph-mon[36485]: pgmap v3018: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:55:22.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:22 smithi106 ceph-mon[36505]: pgmap v3018: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:55:23.134 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5451 sec 2024-03-24T01:55:24.134 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5452 sec 2024-03-24T01:55:24.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:24 smithi078 ceph-mon[32429]: pgmap v3019: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:55:24.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4798.174008 secs 2024-03-24T01:55:24.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:24 smithi107 ceph-mon[36485]: pgmap v3019: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:55:24.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4798.174008 secs 2024-03-24T01:55:24.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:24 smithi106 ceph-mon[36505]: pgmap v3019: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:55:24.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4798.174008 secs 2024-03-24T01:55:25.134 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5453 sec 2024-03-24T01:55:26.134 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5454 sec 2024-03-24T01:55:26.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:26 smithi078 ceph-mon[32429]: pgmap v3020: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:55:26.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:26 smithi107 ceph-mon[36485]: pgmap v3020: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:55:26.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:26 smithi106 ceph-mon[36505]: pgmap v3020: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:55:27.134 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5455 sec 2024-03-24T01:55:28.135 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5456 sec 2024-03-24T01:55:28.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:28 smithi078 ceph-mon[32429]: pgmap v3021: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:55:28.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:28 smithi107 ceph-mon[36485]: pgmap v3021: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:55:28.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:28 smithi106 ceph-mon[36505]: pgmap v3021: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:55:29.135 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5457 sec 2024-03-24T01:55:29.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4803.174098 secs 2024-03-24T01:55:29.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4803.174098 secs 2024-03-24T01:55:29.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4803.174098 secs 2024-03-24T01:55:30.135 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5458 sec 2024-03-24T01:55:30.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:30 smithi078 ceph-mon[32429]: pgmap v3022: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:30.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:30 smithi107 ceph-mon[36485]: pgmap v3022: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:30.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:30 smithi106 ceph-mon[36505]: pgmap v3022: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:31.135 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5459 sec 2024-03-24T01:55:32.135 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5460 sec 2024-03-24T01:55:32.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:32 smithi078 ceph-mon[32429]: pgmap v3023: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:32.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:32 smithi107 ceph-mon[36485]: pgmap v3023: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:32.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:32 smithi106 ceph-mon[36505]: pgmap v3023: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:33.135 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5461 sec 2024-03-24T01:55:34.135 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5462 sec 2024-03-24T01:55:34.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:34 smithi078 ceph-mon[32429]: pgmap v3024: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:34.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4808.174186 secs 2024-03-24T01:55:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:34 smithi107 ceph-mon[36485]: pgmap v3024: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:34.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4808.174186 secs 2024-03-24T01:55:34.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:34 smithi106 ceph-mon[36505]: pgmap v3024: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:34.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4808.174186 secs 2024-03-24T01:55:35.135 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5463 sec 2024-03-24T01:55:36.135 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5464 sec 2024-03-24T01:55:36.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:36 smithi078 ceph-mon[32429]: pgmap v3025: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:36.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:36 smithi107 ceph-mon[36485]: pgmap v3025: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:36.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:36 smithi106 ceph-mon[36505]: pgmap v3025: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:37.135 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5465 sec 2024-03-24T01:55:38.136 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5466 sec 2024-03-24T01:55:38.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:38 smithi078 ceph-mon[32429]: pgmap v3026: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:38.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:38 smithi107 ceph-mon[36485]: pgmap v3026: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:38.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:38 smithi106 ceph-mon[36505]: pgmap v3026: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:39.136 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5467 sec 2024-03-24T01:55:39.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4813.174248 secs 2024-03-24T01:55:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4813.174248 secs 2024-03-24T01:55:39.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4813.174248 secs 2024-03-24T01:55:40.136 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5468 sec 2024-03-24T01:55:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:40 smithi078 ceph-mon[32429]: pgmap v3027: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:40.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:40 smithi107 ceph-mon[36485]: pgmap v3027: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:40.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:40 smithi106 ceph-mon[36505]: pgmap v3027: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:41.136 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5469 sec 2024-03-24T01:55:42.136 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5470 sec 2024-03-24T01:55:42.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:42 smithi078 ceph-mon[32429]: pgmap v3028: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:42.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:42 smithi107 ceph-mon[36485]: pgmap v3028: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:42.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:42 smithi106 ceph-mon[36505]: pgmap v3028: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:43.136 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5471 sec 2024-03-24T01:55:44.136 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5472 sec 2024-03-24T01:55:44.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:44 smithi107 ceph-mon[36485]: pgmap v3029: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:44.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4818.174311 secs 2024-03-24T01:55:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:44 smithi106 ceph-mon[36505]: pgmap v3029: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4818.174311 secs 2024-03-24T01:55:44.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:44 smithi078 ceph-mon[32429]: pgmap v3029: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:44.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4818.174311 secs 2024-03-24T01:55:45.136 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5473 sec 2024-03-24T01:55:46.136 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5474 sec 2024-03-24T01:55:46.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:46 smithi106 ceph-mon[36505]: pgmap v3030: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:46 smithi078 ceph-mon[32429]: pgmap v3030: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:46.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:46 smithi107 ceph-mon[36485]: pgmap v3030: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:47.137 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5475 sec 2024-03-24T01:55:48.136 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5476 sec 2024-03-24T01:55:48.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:48 smithi106 ceph-mon[36505]: pgmap v3031: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:48.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:48 smithi078 ceph-mon[32429]: pgmap v3031: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:48 smithi107 ceph-mon[36485]: pgmap v3031: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:49.136 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5477 sec 2024-03-24T01:55:49.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4823.174373 secs 2024-03-24T01:55:49.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4823.174373 secs 2024-03-24T01:55:49.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4823.174373 secs 2024-03-24T01:55:50.137 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5478 sec 2024-03-24T01:55:50.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:50 smithi106 ceph-mon[36505]: pgmap v3032: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:50.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:50 smithi078 ceph-mon[32429]: pgmap v3032: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:50 smithi107 ceph-mon[36485]: pgmap v3032: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:51.137 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5479 sec 2024-03-24T01:55:52.137 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5480 sec 2024-03-24T01:55:52.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:52 smithi106 ceph-mon[36505]: pgmap v3033: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:52.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:52 smithi078 ceph-mon[32429]: pgmap v3033: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:52.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:52 smithi107 ceph-mon[36485]: pgmap v3033: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:53.137 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5481 sec 2024-03-24T01:55:54.137 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5482 sec 2024-03-24T01:55:54.620 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:54 smithi078 ceph-mon[32429]: pgmap v3034: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:54.620 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4828.174434 secs 2024-03-24T01:55:54.646 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:54 smithi107 ceph-mon[36485]: pgmap v3034: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:54.646 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4828.174434 secs 2024-03-24T01:55:54.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:54 smithi106 ceph-mon[36505]: pgmap v3034: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:54.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4828.174434 secs 2024-03-24T01:55:55.137 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5483 sec 2024-03-24T01:55:56.137 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5484 sec 2024-03-24T01:55:56.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:56 smithi106 ceph-mon[36505]: pgmap v3035: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:56.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:56 smithi078 ceph-mon[32429]: pgmap v3035: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:56 smithi107 ceph-mon[36485]: pgmap v3035: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:55:57.137 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5485 sec 2024-03-24T01:55:58.137 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5486 sec 2024-03-24T01:55:58.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:58 smithi106 ceph-mon[36505]: pgmap v3036: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:58.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:58 smithi078 ceph-mon[32429]: pgmap v3036: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:58 smithi107 ceph-mon[36485]: pgmap v3036: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:59.137 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5487 sec 2024-03-24T01:55:59.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:55:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4833.174523 secs 2024-03-24T01:55:59.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:55:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4833.174523 secs 2024-03-24T01:55:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:55:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4833.174523 secs 2024-03-24T01:56:00.137 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5488 sec 2024-03-24T01:56:00.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:00 smithi106 ceph-mon[36505]: pgmap v3037: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:00 smithi078 ceph-mon[32429]: pgmap v3037: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:00 smithi107 ceph-mon[36485]: pgmap v3037: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:01.138 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5489 sec 2024-03-24T01:56:01.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:01 smithi078 ceph-mon[32429]: pgmap v3038: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:01.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:01 smithi107 ceph-mon[36485]: pgmap v3038: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:01.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:01 smithi106 ceph-mon[36505]: pgmap v3038: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:02.138 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5490 sec 2024-03-24T01:56:03.138 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5491 sec 2024-03-24T01:56:04.138 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5492 sec 2024-03-24T01:56:04.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:03 smithi078 ceph-mon[32429]: pgmap v3039: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:04.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4838.174537 secs 2024-03-24T01:56:04.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:03 smithi107 ceph-mon[36485]: pgmap v3039: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:04.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4838.174537 secs 2024-03-24T01:56:04.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:03 smithi106 ceph-mon[36505]: pgmap v3039: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:04.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4838.174537 secs 2024-03-24T01:56:05.138 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5493 sec 2024-03-24T01:56:06.138 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5494 sec 2024-03-24T01:56:06.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:05 smithi078 ceph-mon[32429]: pgmap v3040: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:06.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:05 smithi107 ceph-mon[36485]: pgmap v3040: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:06.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:05 smithi106 ceph-mon[36505]: pgmap v3040: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:07.138 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5495 sec 2024-03-24T01:56:08.138 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5496 sec 2024-03-24T01:56:08.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:07 smithi078 ceph-mon[32429]: pgmap v3041: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:08.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:07 smithi107 ceph-mon[36485]: pgmap v3041: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:08.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:07 smithi106 ceph-mon[36505]: pgmap v3041: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:09.138 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5497 sec 2024-03-24T01:56:09.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4843.174597 secs 2024-03-24T01:56:09.303 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4843.174597 secs 2024-03-24T01:56:09.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4843.174597 secs 2024-03-24T01:56:10.138 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5498 sec 2024-03-24T01:56:10.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:09 smithi078 ceph-mon[32429]: pgmap v3042: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:10.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:09 smithi107 ceph-mon[36485]: pgmap v3042: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:10.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:09 smithi106 ceph-mon[36505]: pgmap v3042: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:11.138 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5499 sec 2024-03-24T01:56:12.139 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5500 sec 2024-03-24T01:56:12.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:11 smithi078 ceph-mon[32429]: pgmap v3043: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:12.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:11 smithi107 ceph-mon[36485]: pgmap v3043: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:12.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:11 smithi106 ceph-mon[36505]: pgmap v3043: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:13.139 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5501 sec 2024-03-24T01:56:14.139 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5502 sec 2024-03-24T01:56:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:13 smithi078 ceph-mon[32429]: pgmap v3044: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4848.174684 secs 2024-03-24T01:56:14.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:13 smithi107 ceph-mon[36485]: pgmap v3044: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:14.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4848.174684 secs 2024-03-24T01:56:14.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:13 smithi106 ceph-mon[36505]: pgmap v3044: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:14.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4848.174684 secs 2024-03-24T01:56:15.140 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5503 sec 2024-03-24T01:56:16.139 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5504 sec 2024-03-24T01:56:16.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:15 smithi078 ceph-mon[32429]: pgmap v3045: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:16.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:15 smithi107 ceph-mon[36485]: pgmap v3045: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:16.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:15 smithi106 ceph-mon[36505]: pgmap v3045: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:17.139 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5505 sec 2024-03-24T01:56:18.139 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5506 sec 2024-03-24T01:56:18.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:17 smithi078 ceph-mon[32429]: pgmap v3046: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:18.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:17 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:56:18.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:17 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:56:18.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:17 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:56:18.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:17 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:56:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:17 smithi107 ceph-mon[36485]: pgmap v3046: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:17 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:56:18.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:17 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:56:18.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:17 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:56:18.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:17 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:56:18.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:17 smithi106 ceph-mon[36505]: pgmap v3046: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:18.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:17 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:56:18.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:17 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:56:18.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:17 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:56:18.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:17 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:56:19.139 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5507 sec 2024-03-24T01:56:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:18 smithi078 ceph-mon[32429]: pgmap v3047: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:56:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:18 smithi078 ceph-mon[32429]: pgmap v3048: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:56:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4853.174768 secs 2024-03-24T01:56:19.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:18 smithi107 ceph-mon[36485]: pgmap v3047: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:56:19.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:18 smithi107 ceph-mon[36485]: pgmap v3048: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:56:19.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4853.174768 secs 2024-03-24T01:56:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:18 smithi106 ceph-mon[36505]: pgmap v3047: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:56:19.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:18 smithi106 ceph-mon[36505]: pgmap v3048: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:56:19.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4853.174768 secs 2024-03-24T01:56:20.139 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5508 sec 2024-03-24T01:56:21.139 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5509 sec 2024-03-24T01:56:21.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:20 smithi078 ceph-mon[32429]: pgmap v3049: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:56:21.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:20 smithi107 ceph-mon[36485]: pgmap v3049: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:56:21.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:20 smithi106 ceph-mon[36505]: pgmap v3049: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:56:22.140 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5510 sec 2024-03-24T01:56:23.140 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5511 sec 2024-03-24T01:56:23.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:22 smithi078 ceph-mon[32429]: pgmap v3050: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:56:23.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:22 smithi107 ceph-mon[36485]: pgmap v3050: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:56:23.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:22 smithi106 ceph-mon[36505]: pgmap v3050: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:56:24.140 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5512 sec 2024-03-24T01:56:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4858.174828 secs 2024-03-24T01:56:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4858.174828 secs 2024-03-24T01:56:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4858.174828 secs 2024-03-24T01:56:25.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:25 smithi078 ceph-mon[32429]: pgmap v3051: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:56:25.141 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5513 sec 2024-03-24T01:56:25.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:25 smithi107 ceph-mon[36485]: pgmap v3051: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:56:25.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:25 smithi106 ceph-mon[36505]: pgmap v3051: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:56:26.140 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5514 sec 2024-03-24T01:56:27.140 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5515 sec 2024-03-24T01:56:27.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:27 smithi078 ceph-mon[32429]: pgmap v3052: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:56:27.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:27 smithi107 ceph-mon[36485]: pgmap v3052: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:56:27.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:27 smithi106 ceph-mon[36505]: pgmap v3052: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:56:28.140 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5516 sec 2024-03-24T01:56:29.140 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5517 sec 2024-03-24T01:56:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:29 smithi078 ceph-mon[32429]: pgmap v3053: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:56:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4863.174917 secs 2024-03-24T01:56:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:29 smithi107 ceph-mon[36485]: pgmap v3053: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:56:29.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4863.174917 secs 2024-03-24T01:56:29.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:29 smithi106 ceph-mon[36505]: pgmap v3053: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:56:29.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4863.174917 secs 2024-03-24T01:56:30.140 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5518 sec 2024-03-24T01:56:31.140 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5519 sec 2024-03-24T01:56:31.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:31 smithi078 ceph-mon[32429]: pgmap v3054: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:31.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:31 smithi107 ceph-mon[36485]: pgmap v3054: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:31.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:31 smithi106 ceph-mon[36505]: pgmap v3054: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:32.140 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5520 sec 2024-03-24T01:56:33.141 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5521 sec 2024-03-24T01:56:33.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:33 smithi078 ceph-mon[32429]: pgmap v3055: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:33.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:33 smithi107 ceph-mon[36485]: pgmap v3055: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:33.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:33 smithi106 ceph-mon[36505]: pgmap v3055: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:34.141 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5522 sec 2024-03-24T01:56:34.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4868.174976 secs 2024-03-24T01:56:34.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4868.174976 secs 2024-03-24T01:56:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4868.174976 secs 2024-03-24T01:56:35.141 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5523 sec 2024-03-24T01:56:35.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:35 smithi078 ceph-mon[32429]: pgmap v3056: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:35.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:35 smithi107 ceph-mon[36485]: pgmap v3056: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:35.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:35 smithi106 ceph-mon[36505]: pgmap v3056: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:36.141 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5524 sec 2024-03-24T01:56:37.141 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5525 sec 2024-03-24T01:56:37.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:37 smithi078 ceph-mon[32429]: pgmap v3057: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:37.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:37 smithi107 ceph-mon[36485]: pgmap v3057: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:37.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:37 smithi106 ceph-mon[36505]: pgmap v3057: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:38.141 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5526 sec 2024-03-24T01:56:39.141 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5527 sec 2024-03-24T01:56:39.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:39 smithi078 ceph-mon[32429]: pgmap v3058: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:39.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4873.175063 secs 2024-03-24T01:56:39.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:39 smithi107 ceph-mon[36485]: pgmap v3058: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:39.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4873.175063 secs 2024-03-24T01:56:39.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:39 smithi106 ceph-mon[36505]: pgmap v3058: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:39.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4873.175063 secs 2024-03-24T01:56:40.141 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5528 sec 2024-03-24T01:56:41.141 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5529 sec 2024-03-24T01:56:41.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:41 smithi078 ceph-mon[32429]: pgmap v3059: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:41.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:41 smithi107 ceph-mon[36485]: pgmap v3059: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:41.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:41 smithi106 ceph-mon[36505]: pgmap v3059: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:42.141 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5530 sec 2024-03-24T01:56:43.141 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5531 sec 2024-03-24T01:56:43.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:43 smithi078 ceph-mon[32429]: pgmap v3060: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:43.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:43 smithi107 ceph-mon[36485]: pgmap v3060: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:43.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:43 smithi106 ceph-mon[36505]: pgmap v3060: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:44.142 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5532 sec 2024-03-24T01:56:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4878.175152 secs 2024-03-24T01:56:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4878.175152 secs 2024-03-24T01:56:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4878.175152 secs 2024-03-24T01:56:45.142 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5533 sec 2024-03-24T01:56:45.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:45 smithi078 ceph-mon[32429]: pgmap v3061: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:45.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:45 smithi107 ceph-mon[36485]: pgmap v3061: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:45.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:45 smithi106 ceph-mon[36505]: pgmap v3061: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:46.142 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5534 sec 2024-03-24T01:56:47.142 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5535 sec 2024-03-24T01:56:47.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:47 smithi078 ceph-mon[32429]: pgmap v3062: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:47.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:47 smithi107 ceph-mon[36485]: pgmap v3062: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:47.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:47 smithi106 ceph-mon[36505]: pgmap v3062: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:48.142 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5536 sec 2024-03-24T01:56:49.142 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5537 sec 2024-03-24T01:56:49.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:49 smithi078 ceph-mon[32429]: pgmap v3063: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:49.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4883.175212 secs 2024-03-24T01:56:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:49 smithi107 ceph-mon[36485]: pgmap v3063: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4883.175212 secs 2024-03-24T01:56:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:49 smithi106 ceph-mon[36505]: pgmap v3063: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:49.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4883.175212 secs 2024-03-24T01:56:50.142 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5538 sec 2024-03-24T01:56:51.142 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5539 sec 2024-03-24T01:56:51.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:51 smithi078 ceph-mon[32429]: pgmap v3064: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:51.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:51 smithi107 ceph-mon[36485]: pgmap v3064: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:51.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:51 smithi106 ceph-mon[36505]: pgmap v3064: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:52.142 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5540 sec 2024-03-24T01:56:53.142 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5541 sec 2024-03-24T01:56:53.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:53 smithi078 ceph-mon[32429]: pgmap v3065: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:53.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:53 smithi107 ceph-mon[36485]: pgmap v3065: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:53.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:53 smithi106 ceph-mon[36505]: pgmap v3065: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:54.142 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5542 sec 2024-03-24T01:56:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4888.175273 secs 2024-03-24T01:56:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4888.175273 secs 2024-03-24T01:56:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4888.175273 secs 2024-03-24T01:56:55.142 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5543 sec 2024-03-24T01:56:55.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:55 smithi078 ceph-mon[32429]: pgmap v3066: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:55.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:55 smithi107 ceph-mon[36485]: pgmap v3066: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:55.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:55 smithi106 ceph-mon[36505]: pgmap v3066: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:56.143 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5544 sec 2024-03-24T01:56:57.142 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5545 sec 2024-03-24T01:56:57.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:57 smithi078 ceph-mon[32429]: pgmap v3067: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:57.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:57 smithi107 ceph-mon[36485]: pgmap v3067: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:57.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:57 smithi106 ceph-mon[36505]: pgmap v3067: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:58.143 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5546 sec 2024-03-24T01:56:59.142 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5547 sec 2024-03-24T01:56:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:59 smithi078 ceph-mon[32429]: pgmap v3068: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:56:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4893.175327 secs 2024-03-24T01:56:59.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:59 smithi107 ceph-mon[36485]: pgmap v3068: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:59.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:56:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4893.175327 secs 2024-03-24T01:56:59.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:59 smithi106 ceph-mon[36505]: pgmap v3068: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:56:59.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:56:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4893.175327 secs 2024-03-24T01:57:00.143 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5548 sec 2024-03-24T01:57:01.143 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5549 sec 2024-03-24T01:57:01.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:01 smithi078 ceph-mon[32429]: pgmap v3069: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:01 smithi107 ceph-mon[36485]: pgmap v3069: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:01 smithi106 ceph-mon[36505]: pgmap v3069: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:02.143 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5550 sec 2024-03-24T01:57:03.143 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5551 sec 2024-03-24T01:57:03.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:03 smithi078 ceph-mon[32429]: pgmap v3070: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:03.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:03 smithi107 ceph-mon[36485]: pgmap v3070: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:03.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:03 smithi106 ceph-mon[36505]: pgmap v3070: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:04.143 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5552 sec 2024-03-24T01:57:04.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4898.175414 secs 2024-03-24T01:57:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4898.175414 secs 2024-03-24T01:57:04.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4898.175414 secs 2024-03-24T01:57:05.143 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5553 sec 2024-03-24T01:57:05.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:05 smithi078 ceph-mon[32429]: pgmap v3071: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:05.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:05 smithi107 ceph-mon[36485]: pgmap v3071: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:05.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:05 smithi106 ceph-mon[36505]: pgmap v3071: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:06.143 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5554 sec 2024-03-24T01:57:07.143 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5555 sec 2024-03-24T01:57:07.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:07 smithi078 ceph-mon[32429]: pgmap v3072: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:07.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:07 smithi107 ceph-mon[36485]: pgmap v3072: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:07.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:07 smithi106 ceph-mon[36505]: pgmap v3072: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:08.143 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5556 sec 2024-03-24T01:57:09.143 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5557 sec 2024-03-24T01:57:09.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:09 smithi078 ceph-mon[32429]: pgmap v3073: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:09.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4903.175479 secs 2024-03-24T01:57:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:09 smithi107 ceph-mon[36485]: pgmap v3073: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:09.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4903.175479 secs 2024-03-24T01:57:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:09 smithi106 ceph-mon[36505]: pgmap v3073: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:09.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4903.175479 secs 2024-03-24T01:57:10.144 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5558 sec 2024-03-24T01:57:11.143 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5559 sec 2024-03-24T01:57:11.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:11 smithi078 ceph-mon[32429]: pgmap v3074: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:11.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:11 smithi107 ceph-mon[36485]: pgmap v3074: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:11.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:11 smithi106 ceph-mon[36505]: pgmap v3074: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:12.144 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5560 sec 2024-03-24T01:57:13.144 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5561 sec 2024-03-24T01:57:13.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:13 smithi078 ceph-mon[32429]: pgmap v3075: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:13.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:13 smithi107 ceph-mon[36485]: pgmap v3075: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:13.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:13 smithi106 ceph-mon[36505]: pgmap v3075: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:14.144 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5562 sec 2024-03-24T01:57:14.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4908.175537 secs 2024-03-24T01:57:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4908.175537 secs 2024-03-24T01:57:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4908.175537 secs 2024-03-24T01:57:15.144 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5563 sec 2024-03-24T01:57:15.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:15 smithi106 ceph-mon[36505]: pgmap v3076: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:15.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:15 smithi078 ceph-mon[32429]: pgmap v3076: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:15.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:15 smithi107 ceph-mon[36485]: pgmap v3076: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:16.144 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5564 sec 2024-03-24T01:57:17.144 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5565 sec 2024-03-24T01:57:17.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:17 smithi106 ceph-mon[36505]: pgmap v3077: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:17.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:17 smithi078 ceph-mon[32429]: pgmap v3077: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:17.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:17 smithi107 ceph-mon[36485]: pgmap v3077: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:18.144 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5566 sec 2024-03-24T01:57:18.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:18 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:57:18.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:18 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:57:18.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:18 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:57:19.144 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5567 sec 2024-03-24T01:57:19.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:19 smithi106 ceph-mon[36505]: pgmap v3078: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:19.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4913.175604 secs 2024-03-24T01:57:19.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:19 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:57:19.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:19 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:57:19.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:19 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:57:19.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:19 smithi078 ceph-mon[32429]: pgmap v3078: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:19.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4913.175604 secs 2024-03-24T01:57:19.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:19 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:57:19.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:19 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:57:19.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:19 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:57:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:19 smithi107 ceph-mon[36485]: pgmap v3078: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:19.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4913.175604 secs 2024-03-24T01:57:19.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:19 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:57:19.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:19 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:57:19.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:19 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:57:20.144 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5568 sec 2024-03-24T01:57:20.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:20 smithi106 ceph-mon[36505]: pgmap v3079: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:57:20.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:20 smithi106 ceph-mon[36505]: pgmap v3080: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:57:20.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:20 smithi078 ceph-mon[32429]: pgmap v3079: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:57:20.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:20 smithi078 ceph-mon[32429]: pgmap v3080: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:57:20.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:20 smithi107 ceph-mon[36485]: pgmap v3079: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:57:20.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:20 smithi107 ceph-mon[36485]: pgmap v3080: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:57:21.144 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5569 sec 2024-03-24T01:57:22.145 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5570 sec 2024-03-24T01:57:22.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:22 smithi106 ceph-mon[36505]: pgmap v3081: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:57:22.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:22 smithi078 ceph-mon[32429]: pgmap v3081: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:57:22.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:22 smithi107 ceph-mon[36485]: pgmap v3081: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:57:23.144 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5571 sec 2024-03-24T01:57:24.145 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5572 sec 2024-03-24T01:57:24.396 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:24 smithi107 ceph-mon[36485]: pgmap v3082: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:57:24.396 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4918.175663 secs 2024-03-24T01:57:24.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:24 smithi106 ceph-mon[36505]: pgmap v3082: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:57:24.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4918.175663 secs 2024-03-24T01:57:24.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:24 smithi078 ceph-mon[32429]: pgmap v3082: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:57:24.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4918.175663 secs 2024-03-24T01:57:25.145 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5573 sec 2024-03-24T01:57:26.145 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5574 sec 2024-03-24T01:57:26.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:26 smithi106 ceph-mon[36505]: pgmap v3083: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:57:26.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:26 smithi078 ceph-mon[32429]: pgmap v3083: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:57:26.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:26 smithi107 ceph-mon[36485]: pgmap v3083: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:57:27.145 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5575 sec 2024-03-24T01:57:28.145 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5576 sec 2024-03-24T01:57:28.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:28 smithi106 ceph-mon[36505]: pgmap v3084: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:57:28.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:28 smithi078 ceph-mon[32429]: pgmap v3084: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:57:28.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:28 smithi107 ceph-mon[36485]: pgmap v3084: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:57:29.145 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5577 sec 2024-03-24T01:57:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4923.175704 secs 2024-03-24T01:57:29.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4923.175704 secs 2024-03-24T01:57:29.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4923.175704 secs 2024-03-24T01:57:30.145 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5578 sec 2024-03-24T01:57:30.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:30 smithi106 ceph-mon[36505]: pgmap v3085: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:57:30.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:30 smithi078 ceph-mon[32429]: pgmap v3085: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:57:30.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:30 smithi107 ceph-mon[36485]: pgmap v3085: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:57:31.145 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5579 sec 2024-03-24T01:57:32.145 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5580 sec 2024-03-24T01:57:32.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:32 smithi106 ceph-mon[36505]: pgmap v3086: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:32.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:32 smithi078 ceph-mon[32429]: pgmap v3086: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:32.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:32 smithi107 ceph-mon[36485]: pgmap v3086: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:33.146 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5581 sec 2024-03-24T01:57:34.146 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5582 sec 2024-03-24T01:57:34.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:34 smithi106 ceph-mon[36505]: pgmap v3087: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:34.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4928.175795 secs 2024-03-24T01:57:34.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:34 smithi078 ceph-mon[32429]: pgmap v3087: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:34.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4928.175795 secs 2024-03-24T01:57:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:34 smithi107 ceph-mon[36485]: pgmap v3087: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:34.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4928.175795 secs 2024-03-24T01:57:35.146 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5583 sec 2024-03-24T01:57:36.146 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5584 sec 2024-03-24T01:57:36.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:36 smithi106 ceph-mon[36505]: pgmap v3088: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:36.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:36 smithi078 ceph-mon[32429]: pgmap v3088: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:36.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:36 smithi107 ceph-mon[36485]: pgmap v3088: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:37.146 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5585 sec 2024-03-24T01:57:38.146 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5586 sec 2024-03-24T01:57:38.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:38 smithi106 ceph-mon[36505]: pgmap v3089: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:38.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:38 smithi078 ceph-mon[32429]: pgmap v3089: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:38.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:38 smithi107 ceph-mon[36485]: pgmap v3089: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:39.146 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5587 sec 2024-03-24T01:57:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4933.175884 secs 2024-03-24T01:57:39.491 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4933.175884 secs 2024-03-24T01:57:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4933.175884 secs 2024-03-24T01:57:40.146 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5588 sec 2024-03-24T01:57:40.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:40 smithi106 ceph-mon[36505]: pgmap v3090: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:40 smithi078 ceph-mon[32429]: pgmap v3090: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:40.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:40 smithi107 ceph-mon[36485]: pgmap v3090: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:41.146 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5589 sec 2024-03-24T01:57:42.146 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5590 sec 2024-03-24T01:57:42.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:42 smithi078 ceph-mon[32429]: pgmap v3091: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:42.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:42 smithi107 ceph-mon[36485]: pgmap v3091: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:42.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:42 smithi106 ceph-mon[36505]: pgmap v3091: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:43.146 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5591 sec 2024-03-24T01:57:44.146 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5592 sec 2024-03-24T01:57:44.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:44 smithi078 ceph-mon[32429]: pgmap v3092: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:44.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4938.175974 secs 2024-03-24T01:57:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:44 smithi107 ceph-mon[36485]: pgmap v3092: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:44.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4938.175974 secs 2024-03-24T01:57:44.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:44 smithi106 ceph-mon[36505]: pgmap v3092: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:44.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4938.175974 secs 2024-03-24T01:57:45.146 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5593 sec 2024-03-24T01:57:46.146 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5594 sec 2024-03-24T01:57:46.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:46 smithi078 ceph-mon[32429]: pgmap v3093: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:46.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:46 smithi107 ceph-mon[36485]: pgmap v3093: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:46 smithi106 ceph-mon[36505]: pgmap v3093: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:47.146 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5595 sec 2024-03-24T01:57:48.146 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5596 sec 2024-03-24T01:57:48.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:48 smithi078 ceph-mon[32429]: pgmap v3094: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:48.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:48 smithi107 ceph-mon[36485]: pgmap v3094: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:48.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:48 smithi106 ceph-mon[36505]: pgmap v3094: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:49.147 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5597 sec 2024-03-24T01:57:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4943.176012 secs 2024-03-24T01:57:49.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4943.176012 secs 2024-03-24T01:57:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4943.176012 secs 2024-03-24T01:57:50.147 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5598 sec 2024-03-24T01:57:50.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:50 smithi078 ceph-mon[32429]: pgmap v3095: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:50.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:50 smithi107 ceph-mon[36485]: pgmap v3095: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:50.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:50 smithi106 ceph-mon[36505]: pgmap v3095: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:51.147 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5599 sec 2024-03-24T01:57:52.147 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5600 sec 2024-03-24T01:57:52.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:52 smithi078 ceph-mon[32429]: pgmap v3096: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:52.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:52 smithi107 ceph-mon[36485]: pgmap v3096: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:52.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:52 smithi106 ceph-mon[36505]: pgmap v3096: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:53.147 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5601 sec 2024-03-24T01:57:54.147 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5602 sec 2024-03-24T01:57:54.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:54 smithi078 ceph-mon[32429]: pgmap v3097: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:54.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4948.176102 secs 2024-03-24T01:57:54.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:54 smithi107 ceph-mon[36485]: pgmap v3097: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:54.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4948.176102 secs 2024-03-24T01:57:54.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:54 smithi106 ceph-mon[36505]: pgmap v3097: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:54.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4948.176102 secs 2024-03-24T01:57:55.147 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5603 sec 2024-03-24T01:57:56.147 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5604 sec 2024-03-24T01:57:56.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:56 smithi078 ceph-mon[32429]: pgmap v3098: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:56.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:56 smithi107 ceph-mon[36485]: pgmap v3098: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:56 smithi106 ceph-mon[36505]: pgmap v3098: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:57:57.147 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5605 sec 2024-03-24T01:57:58.147 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5606 sec 2024-03-24T01:57:58.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:58 smithi078 ceph-mon[32429]: pgmap v3099: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:58.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:58 smithi107 ceph-mon[36485]: pgmap v3099: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:58.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:58 smithi106 ceph-mon[36505]: pgmap v3099: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:59.147 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5607 sec 2024-03-24T01:57:59.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:57:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4953.176191 secs 2024-03-24T01:57:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:57:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4953.176191 secs 2024-03-24T01:57:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:57:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4953.176191 secs 2024-03-24T01:58:00.148 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5608 sec 2024-03-24T01:58:00.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:00 smithi078 ceph-mon[32429]: pgmap v3100: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:00.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:00 smithi107 ceph-mon[36485]: pgmap v3100: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:00.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:00 smithi106 ceph-mon[36505]: pgmap v3100: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:01.148 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5609 sec 2024-03-24T01:58:02.148 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5610 sec 2024-03-24T01:58:02.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:02 smithi078 ceph-mon[32429]: pgmap v3101: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:02.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:02 smithi107 ceph-mon[36485]: pgmap v3101: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:02.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:02 smithi106 ceph-mon[36505]: pgmap v3101: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:03.148 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5611 sec 2024-03-24T01:58:04.148 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5612 sec 2024-03-24T01:58:04.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:04 smithi078 ceph-mon[32429]: pgmap v3102: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:04.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4958.176301 secs 2024-03-24T01:58:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:04 smithi107 ceph-mon[36485]: pgmap v3102: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:04.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4958.176301 secs 2024-03-24T01:58:04.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:04 smithi106 ceph-mon[36505]: pgmap v3102: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:04.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4958.176301 secs 2024-03-24T01:58:05.148 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5613 sec 2024-03-24T01:58:06.148 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5614 sec 2024-03-24T01:58:06.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:06 smithi078 ceph-mon[32429]: pgmap v3103: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:06.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:06 smithi107 ceph-mon[36485]: pgmap v3103: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:06.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:06 smithi106 ceph-mon[36505]: pgmap v3103: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:07.148 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5615 sec 2024-03-24T01:58:08.148 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5616 sec 2024-03-24T01:58:08.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:08 smithi078 ceph-mon[32429]: pgmap v3104: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:08.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:08 smithi107 ceph-mon[36485]: pgmap v3104: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:08.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:08 smithi106 ceph-mon[36505]: pgmap v3104: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:09.148 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5617 sec 2024-03-24T01:58:09.492 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4963.176325 secs 2024-03-24T01:58:09.554 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4963.176325 secs 2024-03-24T01:58:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4963.176325 secs 2024-03-24T01:58:10.148 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5618 sec 2024-03-24T01:58:10.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:10 smithi078 ceph-mon[32429]: pgmap v3105: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:10.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:10 smithi107 ceph-mon[36485]: pgmap v3105: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:10.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:10 smithi106 ceph-mon[36505]: pgmap v3105: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:11.148 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5619 sec 2024-03-24T01:58:11.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:11 smithi078 ceph-mon[32429]: pgmap v3106: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:11.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:11 smithi107 ceph-mon[36485]: pgmap v3106: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:11.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:11 smithi106 ceph-mon[36505]: pgmap v3106: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:12.148 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5620 sec 2024-03-24T01:58:13.148 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5621 sec 2024-03-24T01:58:13.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:13 smithi078 ceph-mon[32429]: pgmap v3107: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:13.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4968.176414 secs 2024-03-24T01:58:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:13 smithi107 ceph-mon[36485]: pgmap v3107: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:13.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4968.176414 secs 2024-03-24T01:58:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:13 smithi106 ceph-mon[36505]: pgmap v3107: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4968.176414 secs 2024-03-24T01:58:14.148 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5622 sec 2024-03-24T01:58:15.149 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5623 sec 2024-03-24T01:58:15.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:15 smithi078 ceph-mon[32429]: pgmap v3108: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:15 smithi107 ceph-mon[36485]: pgmap v3108: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:15 smithi106 ceph-mon[36505]: pgmap v3108: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:16.149 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5624 sec 2024-03-24T01:58:17.149 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5625 sec 2024-03-24T01:58:17.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:17 smithi078 ceph-mon[32429]: pgmap v3109: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:17.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:17 smithi107 ceph-mon[36485]: pgmap v3109: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:17.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:17 smithi106 ceph-mon[36505]: pgmap v3109: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:18.149 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5626 sec 2024-03-24T01:58:18.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4973.176494 secs 2024-03-24T01:58:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4973.176494 secs 2024-03-24T01:58:18.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4973.176494 secs 2024-03-24T01:58:19.149 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5627 sec 2024-03-24T01:58:19.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:19 smithi078 ceph-mon[32429]: pgmap v3110: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:19.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:19 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:58:19.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:19 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:58:19.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:19 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:58:19.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:19 smithi078 ceph-mon[32429]: pgmap v3111: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:58:19.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:19 smithi078 ceph-mon[32429]: pgmap v3112: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:58:19.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:19 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:58:19.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:19 smithi107 ceph-mon[36485]: pgmap v3110: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:19.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:19 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:58:19.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:19 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:58:19.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:19 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:58:19.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:19 smithi107 ceph-mon[36485]: pgmap v3111: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:58:19.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:19 smithi107 ceph-mon[36485]: pgmap v3112: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:58:19.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:19 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:58:19.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:19 smithi106 ceph-mon[36505]: pgmap v3110: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:19.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:19 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:58:19.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:19 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:58:19.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:19 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:58:19.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:19 smithi106 ceph-mon[36505]: pgmap v3111: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:58:19.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:19 smithi106 ceph-mon[36505]: pgmap v3112: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:58:19.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:19 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:58:20.149 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5628 sec 2024-03-24T01:58:21.149 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5629 sec 2024-03-24T01:58:22.149 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5630 sec 2024-03-24T01:58:22.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:22 smithi078 ceph-mon[32429]: pgmap v3113: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:58:22.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:22 smithi107 ceph-mon[36485]: pgmap v3113: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:58:22.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:22 smithi106 ceph-mon[36505]: pgmap v3113: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:58:23.149 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5631 sec 2024-03-24T01:58:24.149 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5632 sec 2024-03-24T01:58:24.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4978.176536 secs 2024-03-24T01:58:24.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:24 smithi078 ceph-mon[32429]: pgmap v3114: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:58:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4978.176536 secs 2024-03-24T01:58:24.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:24 smithi107 ceph-mon[36485]: pgmap v3114: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:58:24.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4978.176536 secs 2024-03-24T01:58:24.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:24 smithi106 ceph-mon[36505]: pgmap v3114: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:58:25.149 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5633 sec 2024-03-24T01:58:26.150 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5634 sec 2024-03-24T01:58:26.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:26 smithi078 ceph-mon[32429]: pgmap v3115: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:58:26.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:26 smithi107 ceph-mon[36485]: pgmap v3115: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:58:26.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:26 smithi106 ceph-mon[36505]: pgmap v3115: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:58:27.150 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5635 sec 2024-03-24T01:58:28.150 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5636 sec 2024-03-24T01:58:28.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:28 smithi078 ceph-mon[32429]: pgmap v3116: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:58:28.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:28 smithi107 ceph-mon[36485]: pgmap v3116: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:58:28.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:28 smithi106 ceph-mon[36505]: pgmap v3116: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:58:29.150 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5637 sec 2024-03-24T01:58:29.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4983.176625 secs 2024-03-24T01:58:29.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4983.176625 secs 2024-03-24T01:58:29.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4983.176625 secs 2024-03-24T01:58:30.150 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5638 sec 2024-03-24T01:58:30.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:30 smithi078 ceph-mon[32429]: pgmap v3117: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:58:30.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:30 smithi107 ceph-mon[36485]: pgmap v3117: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:58:30.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:30 smithi106 ceph-mon[36505]: pgmap v3117: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:58:31.150 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5639 sec 2024-03-24T01:58:32.150 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5640 sec 2024-03-24T01:58:32.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:32 smithi078 ceph-mon[32429]: pgmap v3118: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:32.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:32 smithi107 ceph-mon[36485]: pgmap v3118: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:32.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:32 smithi106 ceph-mon[36505]: pgmap v3118: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:33.150 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5641 sec 2024-03-24T01:58:34.150 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5642 sec 2024-03-24T01:58:34.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4988.176686 secs 2024-03-24T01:58:34.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:34 smithi078 ceph-mon[32429]: pgmap v3119: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4988.176686 secs 2024-03-24T01:58:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:34 smithi107 ceph-mon[36485]: pgmap v3119: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:34.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4988.176686 secs 2024-03-24T01:58:34.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:34 smithi106 ceph-mon[36505]: pgmap v3119: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:35.150 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5643 sec 2024-03-24T01:58:36.150 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5644 sec 2024-03-24T01:58:36.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:36 smithi078 ceph-mon[32429]: pgmap v3120: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:36.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:36 smithi107 ceph-mon[36485]: pgmap v3120: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:36.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:36 smithi106 ceph-mon[36505]: pgmap v3120: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:37.151 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5645 sec 2024-03-24T01:58:38.151 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5646 sec 2024-03-24T01:58:38.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:38 smithi078 ceph-mon[32429]: pgmap v3121: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:38.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:38 smithi107 ceph-mon[36485]: pgmap v3121: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:38.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:38 smithi106 ceph-mon[36505]: pgmap v3121: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:39.151 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5647 sec 2024-03-24T01:58:39.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4993.176743 secs 2024-03-24T01:58:39.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4993.176743 secs 2024-03-24T01:58:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4993.176743 secs 2024-03-24T01:58:40.151 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5648 sec 2024-03-24T01:58:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:40 smithi078 ceph-mon[32429]: pgmap v3122: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:40.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:40 smithi107 ceph-mon[36485]: pgmap v3122: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:40.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:40 smithi106 ceph-mon[36505]: pgmap v3122: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:41.151 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5649 sec 2024-03-24T01:58:42.151 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5650 sec 2024-03-24T01:58:42.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:42 smithi078 ceph-mon[32429]: pgmap v3123: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:42.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:42 smithi107 ceph-mon[36485]: pgmap v3123: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:42.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:42 smithi106 ceph-mon[36505]: pgmap v3123: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:43.151 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5651 sec 2024-03-24T01:58:44.151 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5652 sec 2024-03-24T01:58:44.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 4998.176802 secs 2024-03-24T01:58:44.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:44 smithi078 ceph-mon[32429]: pgmap v3124: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 4998.176802 secs 2024-03-24T01:58:44.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:44 smithi107 ceph-mon[36485]: pgmap v3124: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:44.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 4998.176802 secs 2024-03-24T01:58:44.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:44 smithi106 ceph-mon[36505]: pgmap v3124: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:45.151 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5653 sec 2024-03-24T01:58:46.151 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5654 sec 2024-03-24T01:58:46.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:46 smithi078 ceph-mon[32429]: pgmap v3125: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:46.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:46 smithi107 ceph-mon[36485]: pgmap v3125: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:46 smithi106 ceph-mon[36505]: pgmap v3125: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:47.151 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5655 sec 2024-03-24T01:58:48.152 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5656 sec 2024-03-24T01:58:48.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:48 smithi078 ceph-mon[32429]: pgmap v3126: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:48.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:48 smithi107 ceph-mon[36485]: pgmap v3126: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:48.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:48 smithi106 ceph-mon[36505]: pgmap v3126: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:49.152 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5657 sec 2024-03-24T01:58:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5003.176861 secs 2024-03-24T01:58:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5003.176861 secs 2024-03-24T01:58:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5003.176861 secs 2024-03-24T01:58:50.152 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5658 sec 2024-03-24T01:58:50.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:50 smithi078 ceph-mon[32429]: pgmap v3127: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:50.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:50 smithi107 ceph-mon[36485]: pgmap v3127: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:50.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:50 smithi106 ceph-mon[36505]: pgmap v3127: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:51.152 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5659 sec 2024-03-24T01:58:51.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:51 smithi078 ceph-mon[32429]: pgmap v3128: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:51.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:51 smithi107 ceph-mon[36485]: pgmap v3128: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:51 smithi106 ceph-mon[36505]: pgmap v3128: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:52.152 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5660 sec 2024-03-24T01:58:53.152 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5661 sec 2024-03-24T01:58:54.152 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5662 sec 2024-03-24T01:58:54.548 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5008.176922 secs 2024-03-24T01:58:54.548 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:54 smithi107 ceph-mon[36485]: pgmap v3129: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:54.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5008.176922 secs 2024-03-24T01:58:54.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:54 smithi078 ceph-mon[32429]: pgmap v3129: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5008.176922 secs 2024-03-24T01:58:54.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:54 smithi106 ceph-mon[36505]: pgmap v3129: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:55.153 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5663 sec 2024-03-24T01:58:56.152 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5664 sec 2024-03-24T01:58:56.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:56 smithi078 ceph-mon[32429]: pgmap v3130: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:56.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:56 smithi107 ceph-mon[36485]: pgmap v3130: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:56 smithi106 ceph-mon[36505]: pgmap v3130: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:58:57.152 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5665 sec 2024-03-24T01:58:58.153 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5666 sec 2024-03-24T01:58:58.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:58 smithi078 ceph-mon[32429]: pgmap v3131: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:58.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:58 smithi107 ceph-mon[36485]: pgmap v3131: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:58.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:58 smithi106 ceph-mon[36505]: pgmap v3131: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:59.153 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5667 sec 2024-03-24T01:58:59.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:58:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5013.176983 secs 2024-03-24T01:58:59.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:58:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5013.176983 secs 2024-03-24T01:58:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:58:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5013.176983 secs 2024-03-24T01:59:00.153 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5668 sec 2024-03-24T01:59:00.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:00 smithi078 ceph-mon[32429]: pgmap v3132: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:00.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:00 smithi107 ceph-mon[36485]: pgmap v3132: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:00.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:00 smithi106 ceph-mon[36505]: pgmap v3132: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:01.153 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5669 sec 2024-03-24T01:59:02.153 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5670 sec 2024-03-24T01:59:02.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:02 smithi078 ceph-mon[32429]: pgmap v3133: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:02.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:02 smithi107 ceph-mon[36485]: pgmap v3133: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:02.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:02 smithi106 ceph-mon[36505]: pgmap v3133: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:03.153 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5671 sec 2024-03-24T01:59:04.153 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5672 sec 2024-03-24T01:59:04.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5018.177069 secs 2024-03-24T01:59:04.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:04 smithi078 ceph-mon[32429]: pgmap v3134: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:04.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5018.177069 secs 2024-03-24T01:59:04.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:04 smithi107 ceph-mon[36485]: pgmap v3134: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5018.177069 secs 2024-03-24T01:59:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:04 smithi106 ceph-mon[36505]: pgmap v3134: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:05.153 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5673 sec 2024-03-24T01:59:06.153 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5674 sec 2024-03-24T01:59:06.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:06 smithi078 ceph-mon[32429]: pgmap v3135: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:06.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:06 smithi107 ceph-mon[36485]: pgmap v3135: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:06.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:06 smithi106 ceph-mon[36505]: pgmap v3135: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:07.154 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5675 sec 2024-03-24T01:59:08.153 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5676 sec 2024-03-24T01:59:08.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:08 smithi078 ceph-mon[32429]: pgmap v3136: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:08.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:08 smithi107 ceph-mon[36485]: pgmap v3136: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:08.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:08 smithi106 ceph-mon[36505]: pgmap v3136: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:09.154 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5677 sec 2024-03-24T01:59:09.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5023.177156 secs 2024-03-24T01:59:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5023.177156 secs 2024-03-24T01:59:09.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5023.177156 secs 2024-03-24T01:59:10.154 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5678 sec 2024-03-24T01:59:10.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:10 smithi078 ceph-mon[32429]: pgmap v3137: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:10.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:10 smithi107 ceph-mon[36485]: pgmap v3137: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:10.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:10 smithi106 ceph-mon[36505]: pgmap v3137: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:11.154 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5679 sec 2024-03-24T01:59:12.154 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5680 sec 2024-03-24T01:59:12.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:12 smithi078 ceph-mon[32429]: pgmap v3138: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:12.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:12 smithi107 ceph-mon[36485]: pgmap v3138: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:12.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:12 smithi106 ceph-mon[36505]: pgmap v3138: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:13.154 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5681 sec 2024-03-24T01:59:14.154 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5682 sec 2024-03-24T01:59:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5028.177240 secs 2024-03-24T01:59:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:14 smithi078 ceph-mon[32429]: pgmap v3139: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:14.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5028.177240 secs 2024-03-24T01:59:14.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:14 smithi107 ceph-mon[36485]: pgmap v3139: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5028.177240 secs 2024-03-24T01:59:14.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:14 smithi106 ceph-mon[36505]: pgmap v3139: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:15.154 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5683 sec 2024-03-24T01:59:16.154 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5684 sec 2024-03-24T01:59:16.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:16 smithi078 ceph-mon[32429]: pgmap v3140: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:16.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:16 smithi107 ceph-mon[36485]: pgmap v3140: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:16 smithi106 ceph-mon[36505]: pgmap v3140: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:17.154 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5685 sec 2024-03-24T01:59:18.154 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5686 sec 2024-03-24T01:59:18.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:18 smithi107 ceph-mon[36485]: pgmap v3141: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:18.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:18 smithi106 ceph-mon[36505]: pgmap v3141: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:18.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:18 smithi078 ceph-mon[32429]: pgmap v3141: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:19.155 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5687 sec 2024-03-24T01:59:19.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5033.177309 secs 2024-03-24T01:59:19.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:19 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:59:19.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5033.177309 secs 2024-03-24T01:59:19.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:19 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:59:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5033.177309 secs 2024-03-24T01:59:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:19 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:59:20.154 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5688 sec 2024-03-24T01:59:20.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:20 smithi106 ceph-mon[36505]: pgmap v3142: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:20.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:20 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:59:20.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:20 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:59:20.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:20 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:59:20.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:20 smithi078 ceph-mon[32429]: pgmap v3142: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:20.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:20 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:59:20.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:20 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:59:20.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:20 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:59:20.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:20 smithi107 ceph-mon[36485]: pgmap v3142: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:20.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:20 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:59:20.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:20 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:59:20.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:20 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T01:59:21.155 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5689 sec 2024-03-24T01:59:21.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:21 smithi106 ceph-mon[36505]: pgmap v3143: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:59:21.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:21 smithi106 ceph-mon[36505]: pgmap v3144: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:59:21.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:21 smithi078 ceph-mon[32429]: pgmap v3143: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:59:21.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:21 smithi078 ceph-mon[32429]: pgmap v3144: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:59:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:21 smithi107 ceph-mon[36485]: pgmap v3143: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T01:59:21.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:21 smithi107 ceph-mon[36485]: pgmap v3144: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T01:59:22.154 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5690 sec 2024-03-24T01:59:23.155 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5691 sec 2024-03-24T01:59:23.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:23 smithi106 ceph-mon[36505]: pgmap v3145: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:59:23.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:23 smithi078 ceph-mon[32429]: pgmap v3145: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:59:23.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:23 smithi107 ceph-mon[36485]: pgmap v3145: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:59:24.155 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5692 sec 2024-03-24T01:59:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5038.177393 secs 2024-03-24T01:59:24.745 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5038.177393 secs 2024-03-24T01:59:24.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5038.177393 secs 2024-03-24T01:59:25.155 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5693 sec 2024-03-24T01:59:25.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:25 smithi106 ceph-mon[36505]: pgmap v3146: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:59:25.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:25 smithi078 ceph-mon[32429]: pgmap v3146: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:59:25.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:25 smithi107 ceph-mon[36485]: pgmap v3146: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:59:26.155 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5694 sec 2024-03-24T01:59:27.155 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5695 sec 2024-03-24T01:59:27.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:27 smithi106 ceph-mon[36505]: pgmap v3147: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:59:27.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:27 smithi078 ceph-mon[32429]: pgmap v3147: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:59:27.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:27 smithi107 ceph-mon[36485]: pgmap v3147: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:59:28.155 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5696 sec 2024-03-24T01:59:29.157 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5697 sec 2024-03-24T01:59:29.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5043.177473 secs 2024-03-24T01:59:29.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:29 smithi106 ceph-mon[36505]: pgmap v3148: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:59:29.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5043.177473 secs 2024-03-24T01:59:29.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:29 smithi078 ceph-mon[32429]: pgmap v3148: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:59:29.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5043.177473 secs 2024-03-24T01:59:29.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:29 smithi107 ceph-mon[36485]: pgmap v3148: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T01:59:30.155 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5698 sec 2024-03-24T01:59:31.155 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5699 sec 2024-03-24T01:59:31.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:31 smithi106 ceph-mon[36505]: pgmap v3149: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:59:31.708 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:31 smithi078 ceph-mon[32429]: pgmap v3149: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:59:31.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:31 smithi107 ceph-mon[36485]: pgmap v3149: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T01:59:32.156 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5700 sec 2024-03-24T01:59:33.156 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5701 sec 2024-03-24T01:59:33.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:33 smithi106 ceph-mon[36505]: pgmap v3150: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:33.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:33 smithi078 ceph-mon[32429]: pgmap v3150: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:33.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:33 smithi107 ceph-mon[36485]: pgmap v3150: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:34.156 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5702 sec 2024-03-24T01:59:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5048.177510 secs 2024-03-24T01:59:34.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5048.177510 secs 2024-03-24T01:59:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5048.177510 secs 2024-03-24T01:59:35.156 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5703 sec 2024-03-24T01:59:35.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:35 smithi106 ceph-mon[36505]: pgmap v3151: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:35.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:35 smithi078 ceph-mon[32429]: pgmap v3151: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:35.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:35 smithi107 ceph-mon[36485]: pgmap v3151: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:36.156 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5704 sec 2024-03-24T01:59:37.156 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5705 sec 2024-03-24T01:59:37.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:37 smithi106 ceph-mon[36505]: pgmap v3152: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:37.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:37 smithi078 ceph-mon[32429]: pgmap v3152: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:37.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:37 smithi107 ceph-mon[36485]: pgmap v3152: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:38.156 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5706 sec 2024-03-24T01:59:39.156 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5707 sec 2024-03-24T01:59:39.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5053.177570 secs 2024-03-24T01:59:39.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:39 smithi078 ceph-mon[32429]: pgmap v3153: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5053.177570 secs 2024-03-24T01:59:39.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:39 smithi106 ceph-mon[36505]: pgmap v3153: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:39.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5053.177570 secs 2024-03-24T01:59:39.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:39 smithi107 ceph-mon[36485]: pgmap v3153: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:40.156 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5708 sec 2024-03-24T01:59:41.156 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5709 sec 2024-03-24T01:59:41.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:41 smithi106 ceph-mon[36505]: pgmap v3154: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:41.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:41 smithi078 ceph-mon[32429]: pgmap v3154: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:41.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:41 smithi107 ceph-mon[36485]: pgmap v3154: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:42.156 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5710 sec 2024-03-24T01:59:43.156 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5711 sec 2024-03-24T01:59:43.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:43 smithi106 ceph-mon[36505]: pgmap v3155: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:43.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:43 smithi078 ceph-mon[32429]: pgmap v3155: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:43 smithi107 ceph-mon[36485]: pgmap v3155: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:44.157 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5712 sec 2024-03-24T01:59:44.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5058.177630 secs 2024-03-24T01:59:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5058.177630 secs 2024-03-24T01:59:44.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5058.177630 secs 2024-03-24T01:59:45.157 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5713 sec 2024-03-24T01:59:45.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:45 smithi078 ceph-mon[32429]: pgmap v3156: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:45.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:45 smithi107 ceph-mon[36485]: pgmap v3156: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:45.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:45 smithi106 ceph-mon[36505]: pgmap v3156: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:46.157 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5714 sec 2024-03-24T01:59:47.157 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5715 sec 2024-03-24T01:59:47.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:47 smithi078 ceph-mon[32429]: pgmap v3157: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:47.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:47 smithi107 ceph-mon[36485]: pgmap v3157: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:47.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:47 smithi106 ceph-mon[36505]: pgmap v3157: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:48.157 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5716 sec 2024-03-24T01:59:49.157 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5717 sec 2024-03-24T01:59:49.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5063.177668 secs 2024-03-24T01:59:49.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:49 smithi078 ceph-mon[32429]: pgmap v3158: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:49.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5063.177668 secs 2024-03-24T01:59:49.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:49 smithi107 ceph-mon[36485]: pgmap v3158: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:49.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5063.177668 secs 2024-03-24T01:59:49.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:49 smithi106 ceph-mon[36505]: pgmap v3158: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:50.157 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5718 sec 2024-03-24T01:59:50.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:50 smithi078 ceph-mon[32429]: pgmap v3159: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:50.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:50 smithi107 ceph-mon[36485]: pgmap v3159: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:50.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:50 smithi106 ceph-mon[36505]: pgmap v3159: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:51.157 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5719 sec 2024-03-24T01:59:52.158 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5720 sec 2024-03-24T01:59:53.157 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5721 sec 2024-03-24T01:59:53.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:53 smithi078 ceph-mon[32429]: pgmap v3160: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:53.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:53 smithi107 ceph-mon[36485]: pgmap v3160: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:53.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:53 smithi106 ceph-mon[36505]: pgmap v3160: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:54.158 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5722 sec 2024-03-24T01:59:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5068.177728 secs 2024-03-24T01:59:54.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5068.177728 secs 2024-03-24T01:59:54.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5068.177728 secs 2024-03-24T01:59:55.158 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5723 sec 2024-03-24T01:59:55.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:55 smithi078 ceph-mon[32429]: pgmap v3161: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:55.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:55 smithi107 ceph-mon[36485]: pgmap v3161: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:55.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:55 smithi106 ceph-mon[36505]: pgmap v3161: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:56.158 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5724 sec 2024-03-24T01:59:57.158 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5725 sec 2024-03-24T01:59:57.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:57 smithi078 ceph-mon[32429]: pgmap v3162: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:57.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:57 smithi107 ceph-mon[36485]: pgmap v3162: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:57.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:57 smithi106 ceph-mon[36505]: pgmap v3162: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:58.158 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5726 sec 2024-03-24T01:59:59.158 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5727 sec 2024-03-24T01:59:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5073.177787 secs 2024-03-24T01:59:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 01:59:59 smithi078 ceph-mon[32429]: pgmap v3163: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5073.177787 secs 2024-03-24T01:59:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 01:59:59 smithi107 ceph-mon[36485]: pgmap v3163: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T01:59:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5073.177787 secs 2024-03-24T01:59:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 01:59:59 smithi106 ceph-mon[36505]: pgmap v3163: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:00.158 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5728 sec 2024-03-24T02:00:00.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:00 smithi106 ceph-mon[36505]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:00:00.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:00 smithi078 ceph-mon[32429]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:00:00.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:00 smithi107 ceph-mon[36485]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:00:01.158 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5729 sec 2024-03-24T02:00:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:01 smithi106 ceph-mon[36505]: pgmap v3164: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:01.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:01 smithi078 ceph-mon[32429]: pgmap v3164: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:01.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:01 smithi107 ceph-mon[36485]: pgmap v3164: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:02.158 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5730 sec 2024-03-24T02:00:03.158 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5731 sec 2024-03-24T02:00:03.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:03 smithi078 ceph-mon[32429]: pgmap v3165: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:03.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:03 smithi106 ceph-mon[36505]: pgmap v3165: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:03.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:03 smithi107 ceph-mon[36485]: pgmap v3165: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:04.159 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5732 sec 2024-03-24T02:00:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5078.177877 secs 2024-03-24T02:00:04.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5078.177877 secs 2024-03-24T02:00:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5078.177877 secs 2024-03-24T02:00:05.159 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5733 sec 2024-03-24T02:00:05.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:05 smithi106 ceph-mon[36505]: pgmap v3166: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:05.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:05 smithi078 ceph-mon[32429]: pgmap v3166: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:05.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:05 smithi107 ceph-mon[36485]: pgmap v3166: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:06.159 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5734 sec 2024-03-24T02:00:07.159 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5735 sec 2024-03-24T02:00:07.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:07 smithi106 ceph-mon[36505]: pgmap v3167: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:07.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:07 smithi078 ceph-mon[32429]: pgmap v3167: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:07.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:07 smithi107 ceph-mon[36485]: pgmap v3167: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:08.159 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5736 sec 2024-03-24T02:00:09.159 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5737 sec 2024-03-24T02:00:09.392 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5083.177961 secs 2024-03-24T02:00:09.393 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:09 smithi078 ceph-mon[32429]: pgmap v3168: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:09.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5083.177961 secs 2024-03-24T02:00:09.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:09 smithi106 ceph-mon[36505]: pgmap v3168: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:09.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5083.177961 secs 2024-03-24T02:00:09.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:09 smithi107 ceph-mon[36485]: pgmap v3168: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:10.159 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5738 sec 2024-03-24T02:00:11.159 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5739 sec 2024-03-24T02:00:11.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:11 smithi106 ceph-mon[36505]: pgmap v3169: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:11.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:11 smithi078 ceph-mon[32429]: pgmap v3169: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:11.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:11 smithi107 ceph-mon[36485]: pgmap v3169: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:12.159 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5740 sec 2024-03-24T02:00:13.159 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5741 sec 2024-03-24T02:00:13.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:13 smithi106 ceph-mon[36505]: pgmap v3170: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:13.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:13 smithi078 ceph-mon[32429]: pgmap v3170: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:13.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:13 smithi107 ceph-mon[36485]: pgmap v3170: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:14.159 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5742 sec 2024-03-24T02:00:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5088.178020 secs 2024-03-24T02:00:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5088.178020 secs 2024-03-24T02:00:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5088.178020 secs 2024-03-24T02:00:15.159 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5743 sec 2024-03-24T02:00:15.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:15 smithi106 ceph-mon[36505]: pgmap v3171: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:15.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:15 smithi078 ceph-mon[32429]: pgmap v3171: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:15.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:15 smithi107 ceph-mon[36485]: pgmap v3171: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:16.160 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5744 sec 2024-03-24T02:00:17.159 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5745 sec 2024-03-24T02:00:17.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:17 smithi106 ceph-mon[36505]: pgmap v3172: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:17.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:17 smithi078 ceph-mon[32429]: pgmap v3172: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:17.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:17 smithi107 ceph-mon[36485]: pgmap v3172: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:18.160 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5746 sec 2024-03-24T02:00:19.160 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5747 sec 2024-03-24T02:00:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5093.178081 secs 2024-03-24T02:00:19.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:19 smithi106 ceph-mon[36505]: pgmap v3173: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:19.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5093.178081 secs 2024-03-24T02:00:19.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:19 smithi078 ceph-mon[32429]: pgmap v3173: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:19.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5093.178081 secs 2024-03-24T02:00:19.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:19 smithi107 ceph-mon[36485]: pgmap v3173: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:20.160 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5748 sec 2024-03-24T02:00:20.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:20 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:00:20.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:20 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:00:20.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:20 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:00:21.160 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5749 sec 2024-03-24T02:00:21.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:21 smithi106 ceph-mon[36505]: pgmap v3174: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:21.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:21 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:00:21.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:21 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:00:21.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:21 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:00:21.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:21 smithi078 ceph-mon[32429]: pgmap v3174: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:21.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:21 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:00:21.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:21 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:00:21.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:21 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:00:21.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:21 smithi107 ceph-mon[36485]: pgmap v3174: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:21.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:21 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:00:21.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:21 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:00:21.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:21 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:00:22.160 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5750 sec 2024-03-24T02:00:22.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:22 smithi106 ceph-mon[36505]: pgmap v3175: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:00:22.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:22 smithi106 ceph-mon[36505]: pgmap v3176: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:00:22.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:22 smithi078 ceph-mon[32429]: pgmap v3175: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:00:22.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:22 smithi078 ceph-mon[32429]: pgmap v3176: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:00:22.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:22 smithi107 ceph-mon[36485]: pgmap v3175: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:00:22.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:22 smithi107 ceph-mon[36485]: pgmap v3176: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:00:23.160 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5751 sec 2024-03-24T02:00:24.160 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5752 sec 2024-03-24T02:00:24.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:24 smithi106 ceph-mon[36505]: pgmap v3177: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:00:24.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5098.178139 secs 2024-03-24T02:00:24.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:24 smithi078 ceph-mon[32429]: pgmap v3177: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:00:24.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5098.178139 secs 2024-03-24T02:00:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:24 smithi107 ceph-mon[36485]: pgmap v3177: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:00:24.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5098.178139 secs 2024-03-24T02:00:25.160 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5753 sec 2024-03-24T02:00:26.160 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5754 sec 2024-03-24T02:00:26.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:26 smithi106 ceph-mon[36505]: pgmap v3178: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:00:26.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:26 smithi078 ceph-mon[32429]: pgmap v3178: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:00:26.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:26 smithi107 ceph-mon[36485]: pgmap v3178: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:00:27.160 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5755 sec 2024-03-24T02:00:28.160 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5756 sec 2024-03-24T02:00:28.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:28 smithi078 ceph-mon[32429]: pgmap v3179: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:00:28.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:28 smithi107 ceph-mon[36485]: pgmap v3179: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:00:28.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:28 smithi106 ceph-mon[36505]: pgmap v3179: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:00:29.160 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5757 sec 2024-03-24T02:00:29.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5103.178199 secs 2024-03-24T02:00:29.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5103.178199 secs 2024-03-24T02:00:29.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5103.178199 secs 2024-03-24T02:00:30.160 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5758 sec 2024-03-24T02:00:30.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:30 smithi078 ceph-mon[32429]: pgmap v3180: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:00:30.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:30 smithi107 ceph-mon[36485]: pgmap v3180: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:00:30.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:30 smithi106 ceph-mon[36505]: pgmap v3180: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:00:31.161 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5759 sec 2024-03-24T02:00:32.161 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5760 sec 2024-03-24T02:00:32.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:32 smithi078 ceph-mon[32429]: pgmap v3181: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:00:32.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:32 smithi107 ceph-mon[36485]: pgmap v3181: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:00:32.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:32 smithi106 ceph-mon[36505]: pgmap v3181: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:00:33.161 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5761 sec 2024-03-24T02:00:34.161 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5762 sec 2024-03-24T02:00:34.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:34 smithi078 ceph-mon[32429]: pgmap v3182: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:34.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5108.178254 secs 2024-03-24T02:00:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:34 smithi107 ceph-mon[36485]: pgmap v3182: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:34.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5108.178254 secs 2024-03-24T02:00:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:34 smithi106 ceph-mon[36505]: pgmap v3182: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:34.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5108.178254 secs 2024-03-24T02:00:35.161 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5763 sec 2024-03-24T02:00:36.161 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5764 sec 2024-03-24T02:00:36.373 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:36 smithi078 ceph-mon[32429]: pgmap v3183: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:36.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:36 smithi107 ceph-mon[36485]: pgmap v3183: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:36.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:36 smithi106 ceph-mon[36505]: pgmap v3183: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:37.161 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5765 sec 2024-03-24T02:00:38.161 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5766 sec 2024-03-24T02:00:38.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:38 smithi078 ceph-mon[32429]: pgmap v3184: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:38.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:38 smithi107 ceph-mon[36485]: pgmap v3184: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:38.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:38 smithi106 ceph-mon[36505]: pgmap v3184: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:39.161 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5768 sec 2024-03-24T02:00:39.492 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5113.178321 secs 2024-03-24T02:00:39.553 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5113.178321 secs 2024-03-24T02:00:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5113.178321 secs 2024-03-24T02:00:40.161 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5769 sec 2024-03-24T02:00:40.565 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:40 smithi106 ceph-mon[36505]: pgmap v3185: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:40 smithi078 ceph-mon[32429]: pgmap v3185: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:40.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:40 smithi107 ceph-mon[36485]: pgmap v3185: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:41.161 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5770 sec 2024-03-24T02:00:42.161 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5771 sec 2024-03-24T02:00:42.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:42 smithi078 ceph-mon[32429]: pgmap v3186: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:42.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:42 smithi107 ceph-mon[36485]: pgmap v3186: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:42.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:42 smithi106 ceph-mon[36505]: pgmap v3186: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:43.162 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5772 sec 2024-03-24T02:00:44.162 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5773 sec 2024-03-24T02:00:44.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:44 smithi078 ceph-mon[32429]: pgmap v3187: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:44.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5118.178409 secs 2024-03-24T02:00:44.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:44 smithi107 ceph-mon[36485]: pgmap v3187: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:44.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5118.178409 secs 2024-03-24T02:00:44.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:44 smithi106 ceph-mon[36505]: pgmap v3187: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:44.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5118.178409 secs 2024-03-24T02:00:45.162 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5774 sec 2024-03-24T02:00:46.162 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5775 sec 2024-03-24T02:00:46.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:46 smithi078 ceph-mon[32429]: pgmap v3188: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:46.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:46 smithi107 ceph-mon[36485]: pgmap v3188: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:46 smithi106 ceph-mon[36505]: pgmap v3188: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:47.162 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5776 sec 2024-03-24T02:00:48.162 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5777 sec 2024-03-24T02:00:48.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:48 smithi078 ceph-mon[32429]: pgmap v3189: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:48.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:48 smithi107 ceph-mon[36485]: pgmap v3189: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:48.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:48 smithi106 ceph-mon[36505]: pgmap v3189: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:49.162 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5778 sec 2024-03-24T02:00:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5123.178460 secs 2024-03-24T02:00:49.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5123.178460 secs 2024-03-24T02:00:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5123.178460 secs 2024-03-24T02:00:50.162 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5779 sec 2024-03-24T02:00:50.556 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:50 smithi078 ceph-mon[32429]: pgmap v3190: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:50.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:50 smithi107 ceph-mon[36485]: pgmap v3190: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:50.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:50 smithi106 ceph-mon[36505]: pgmap v3190: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:51.162 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5780 sec 2024-03-24T02:00:51.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:51 smithi106 ceph-mon[36505]: pgmap v3191: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:51.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:51 smithi078 ceph-mon[32429]: pgmap v3191: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:51.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:51 smithi107 ceph-mon[36485]: pgmap v3191: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:52.162 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5781 sec 2024-03-24T02:00:53.163 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5782 sec 2024-03-24T02:00:54.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:53 smithi106 ceph-mon[36505]: pgmap v3192: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:54.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5128.178524 secs 2024-03-24T02:00:54.163 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5783 sec 2024-03-24T02:00:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:53 smithi078 ceph-mon[32429]: pgmap v3192: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5128.178524 secs 2024-03-24T02:00:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:53 smithi107 ceph-mon[36485]: pgmap v3192: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:54.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5128.178524 secs 2024-03-24T02:00:55.163 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5784 sec 2024-03-24T02:00:56.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:55 smithi106 ceph-mon[36505]: pgmap v3193: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:56.163 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5785 sec 2024-03-24T02:00:56.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:55 smithi078 ceph-mon[32429]: pgmap v3193: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:56.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:55 smithi107 ceph-mon[36485]: pgmap v3193: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:00:57.163 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5786 sec 2024-03-24T02:00:58.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:57 smithi106 ceph-mon[36505]: pgmap v3194: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:58.163 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5787 sec 2024-03-24T02:00:58.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:57 smithi078 ceph-mon[32429]: pgmap v3194: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:58.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:57 smithi107 ceph-mon[36485]: pgmap v3194: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5133.178583 secs 2024-03-24T02:00:59.163 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5788 sec 2024-03-24T02:00:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5133.178583 secs 2024-03-24T02:00:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5133.178583 secs 2024-03-24T02:01:00.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:00:59 smithi106 ceph-mon[36505]: pgmap v3195: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:00.163 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5789 sec 2024-03-24T02:01:00.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:00:59 smithi078 ceph-mon[32429]: pgmap v3195: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:00.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:00:59 smithi107 ceph-mon[36485]: pgmap v3195: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:01.163 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5790 sec 2024-03-24T02:01:02.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:01 smithi106 ceph-mon[36505]: pgmap v3196: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:02.164 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5791 sec 2024-03-24T02:01:02.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:01 smithi078 ceph-mon[32429]: pgmap v3196: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:02.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:01 smithi107 ceph-mon[36485]: pgmap v3196: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:03.164 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5792 sec 2024-03-24T02:01:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:03 smithi106 ceph-mon[36505]: pgmap v3197: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:04.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5138.178646 secs 2024-03-24T02:01:04.164 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5793 sec 2024-03-24T02:01:04.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:03 smithi078 ceph-mon[32429]: pgmap v3197: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:04.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5138.178646 secs 2024-03-24T02:01:04.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:03 smithi107 ceph-mon[36485]: pgmap v3197: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:04.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5138.178646 secs 2024-03-24T02:01:05.164 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5794 sec 2024-03-24T02:01:06.142 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:05 smithi078 ceph-mon[32429]: pgmap v3198: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:06.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:05 smithi106 ceph-mon[36505]: pgmap v3198: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:06.164 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5795 sec 2024-03-24T02:01:06.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:05 smithi107 ceph-mon[36485]: pgmap v3198: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:07.164 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5796 sec 2024-03-24T02:01:08.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:07 smithi106 ceph-mon[36505]: pgmap v3199: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:08.164 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5797 sec 2024-03-24T02:01:08.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:07 smithi078 ceph-mon[32429]: pgmap v3199: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:08.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:07 smithi107 ceph-mon[36485]: pgmap v3199: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5143.178705 secs 2024-03-24T02:01:09.164 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5798 sec 2024-03-24T02:01:09.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5143.178705 secs 2024-03-24T02:01:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5143.178705 secs 2024-03-24T02:01:10.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:09 smithi106 ceph-mon[36505]: pgmap v3200: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:10.164 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5799 sec 2024-03-24T02:01:10.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:09 smithi078 ceph-mon[32429]: pgmap v3200: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:10.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:09 smithi107 ceph-mon[36485]: pgmap v3200: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:11.164 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5800 sec 2024-03-24T02:01:12.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:11 smithi106 ceph-mon[36505]: pgmap v3201: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:12.164 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5801 sec 2024-03-24T02:01:12.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:11 smithi078 ceph-mon[32429]: pgmap v3201: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:12.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:11 smithi107 ceph-mon[36485]: pgmap v3201: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:13.165 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5802 sec 2024-03-24T02:01:14.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:13 smithi106 ceph-mon[36505]: pgmap v3202: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:14.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5148.178791 secs 2024-03-24T02:01:14.165 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5803 sec 2024-03-24T02:01:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:13 smithi078 ceph-mon[32429]: pgmap v3202: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5148.178791 secs 2024-03-24T02:01:14.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:13 smithi107 ceph-mon[36485]: pgmap v3202: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:14.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5148.178791 secs 2024-03-24T02:01:15.165 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5804 sec 2024-03-24T02:01:16.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:15 smithi106 ceph-mon[36505]: pgmap v3203: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:16.165 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5805 sec 2024-03-24T02:01:16.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:15 smithi078 ceph-mon[32429]: pgmap v3203: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:16.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:15 smithi107 ceph-mon[36485]: pgmap v3203: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:17.165 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5806 sec 2024-03-24T02:01:18.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:17 smithi106 ceph-mon[36505]: pgmap v3204: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:18.165 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5807 sec 2024-03-24T02:01:18.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:17 smithi078 ceph-mon[32429]: pgmap v3204: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:18.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:17 smithi107 ceph-mon[36485]: pgmap v3204: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:19.165 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5808 sec 2024-03-24T02:01:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5153.178850 secs 2024-03-24T02:01:19.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5153.178850 secs 2024-03-24T02:01:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5153.178850 secs 2024-03-24T02:01:20.165 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5809 sec 2024-03-24T02:01:20.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:19 smithi078 ceph-mon[32429]: pgmap v3205: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:19 smithi107 ceph-mon[36485]: pgmap v3205: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:20.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:19 smithi106 ceph-mon[36505]: pgmap v3205: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:21.165 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5810 sec 2024-03-24T02:01:21.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:20 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:01:21.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:20 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:01:21.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:20 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:01:22.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:21 smithi078 ceph-mon[32429]: pgmap v3206: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:22.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:21 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:01:22.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:21 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:01:22.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:21 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:01:22.165 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5811 sec 2024-03-24T02:01:22.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:21 smithi107 ceph-mon[36485]: pgmap v3206: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:22.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:21 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:01:22.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:21 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:01:22.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:21 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:01:22.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:21 smithi106 ceph-mon[36505]: pgmap v3206: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:22.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:21 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:01:22.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:21 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:01:22.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:21 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:01:23.165 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5812 sec 2024-03-24T02:01:23.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:22 smithi078 ceph-mon[32429]: pgmap v3207: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:01:23.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:22 smithi078 ceph-mon[32429]: pgmap v3208: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:01:23.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:22 smithi107 ceph-mon[36485]: pgmap v3207: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:01:23.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:22 smithi107 ceph-mon[36485]: pgmap v3208: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:01:23.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:22 smithi106 ceph-mon[36505]: pgmap v3207: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:01:23.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:22 smithi106 ceph-mon[36505]: pgmap v3208: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:01:24.165 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5813 sec 2024-03-24T02:01:24.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5158.178938 secs 2024-03-24T02:01:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5158.178938 secs 2024-03-24T02:01:24.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5158.178938 secs 2024-03-24T02:01:25.165 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5814 sec 2024-03-24T02:01:25.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:24 smithi078 ceph-mon[32429]: pgmap v3209: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:01:25.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:24 smithi107 ceph-mon[36485]: pgmap v3209: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:01:25.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:24 smithi106 ceph-mon[36505]: pgmap v3209: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:01:26.166 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5815 sec 2024-03-24T02:01:27.166 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5816 sec 2024-03-24T02:01:27.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:26 smithi078 ceph-mon[32429]: pgmap v3210: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:01:27.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:26 smithi107 ceph-mon[36485]: pgmap v3210: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:01:27.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:26 smithi106 ceph-mon[36505]: pgmap v3210: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:01:28.166 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5817 sec 2024-03-24T02:01:29.167 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5818 sec 2024-03-24T02:01:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:28 smithi078 ceph-mon[32429]: pgmap v3211: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:01:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5163.179026 secs 2024-03-24T02:01:29.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:28 smithi107 ceph-mon[36485]: pgmap v3211: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:01:29.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5163.179026 secs 2024-03-24T02:01:29.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:28 smithi106 ceph-mon[36505]: pgmap v3211: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:01:29.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5163.179026 secs 2024-03-24T02:01:30.166 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5819 sec 2024-03-24T02:01:31.166 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5820 sec 2024-03-24T02:01:31.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:30 smithi078 ceph-mon[32429]: pgmap v3212: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:01:31.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:30 smithi107 ceph-mon[36485]: pgmap v3212: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:01:31.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:30 smithi106 ceph-mon[36505]: pgmap v3212: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:01:32.166 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5821 sec 2024-03-24T02:01:33.166 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5822 sec 2024-03-24T02:01:33.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:32 smithi078 ceph-mon[32429]: pgmap v3213: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:01:33.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:32 smithi107 ceph-mon[36485]: pgmap v3213: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:01:33.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:32 smithi106 ceph-mon[36505]: pgmap v3213: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:01:34.166 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5823 sec 2024-03-24T02:01:34.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5168.179086 secs 2024-03-24T02:01:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5168.179086 secs 2024-03-24T02:01:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5168.179086 secs 2024-03-24T02:01:35.167 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5824 sec 2024-03-24T02:01:35.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:34 smithi078 ceph-mon[32429]: pgmap v3214: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:35.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:34 smithi107 ceph-mon[36485]: pgmap v3214: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:35.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:34 smithi106 ceph-mon[36505]: pgmap v3214: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:36.167 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5825 sec 2024-03-24T02:01:37.167 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5826 sec 2024-03-24T02:01:37.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:36 smithi078 ceph-mon[32429]: pgmap v3215: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:01:37.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:36 smithi107 ceph-mon[36485]: pgmap v3215: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:01:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:36 smithi106 ceph-mon[36505]: pgmap v3215: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:01:38.167 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5827 sec 2024-03-24T02:01:39.167 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5828 sec 2024-03-24T02:01:39.204 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:38 smithi106 ceph-mon[36505]: pgmap v3216: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:01:39.205 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5173.179127 secs 2024-03-24T02:01:39.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:38 smithi078 ceph-mon[32429]: pgmap v3216: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:01:39.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5173.179127 secs 2024-03-24T02:01:39.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:38 smithi107 ceph-mon[36485]: pgmap v3216: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:01:39.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5173.179127 secs 2024-03-24T02:01:40.167 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5829 sec 2024-03-24T02:01:41.167 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5830 sec 2024-03-24T02:01:41.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:40 smithi078 ceph-mon[32429]: pgmap v3217: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:01:41.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:40 smithi107 ceph-mon[36485]: pgmap v3217: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:01:41.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:40 smithi106 ceph-mon[36505]: pgmap v3217: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:01:42.167 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5831 sec 2024-03-24T02:01:43.169 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5832 sec 2024-03-24T02:01:43.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:42 smithi078 ceph-mon[32429]: pgmap v3218: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:01:43.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:42 smithi107 ceph-mon[36485]: pgmap v3218: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:01:43.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:42 smithi106 ceph-mon[36505]: pgmap v3218: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:01:44.167 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5833 sec 2024-03-24T02:01:44.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5178.179214 secs 2024-03-24T02:01:44.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5178.179214 secs 2024-03-24T02:01:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5178.179214 secs 2024-03-24T02:01:45.167 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5834 sec 2024-03-24T02:01:45.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:44 smithi078 ceph-mon[32429]: pgmap v3219: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:01:45.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:44 smithi107 ceph-mon[36485]: pgmap v3219: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:01:45.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:44 smithi106 ceph-mon[36505]: pgmap v3219: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:01:46.167 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5835 sec 2024-03-24T02:01:47.168 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5836 sec 2024-03-24T02:01:47.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:46 smithi078 ceph-mon[32429]: pgmap v3220: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:01:47.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:46 smithi107 ceph-mon[36485]: pgmap v3220: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:01:47.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:46 smithi106 ceph-mon[36505]: pgmap v3220: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:01:48.168 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5837 sec 2024-03-24T02:01:49.168 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5838 sec 2024-03-24T02:01:49.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:48 smithi078 ceph-mon[32429]: pgmap v3221: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:49.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5183.179275 secs 2024-03-24T02:01:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:48 smithi107 ceph-mon[36485]: pgmap v3221: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5183.179275 secs 2024-03-24T02:01:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:48 smithi106 ceph-mon[36505]: pgmap v3221: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5183.179275 secs 2024-03-24T02:01:50.168 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5839 sec 2024-03-24T02:01:51.168 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5840 sec 2024-03-24T02:01:51.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:50 smithi078 ceph-mon[32429]: pgmap v3222: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:51.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:50 smithi107 ceph-mon[36485]: pgmap v3222: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:51.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:50 smithi106 ceph-mon[36505]: pgmap v3222: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:52.168 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5841 sec 2024-03-24T02:01:53.168 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5842 sec 2024-03-24T02:01:53.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:52 smithi078 ceph-mon[32429]: pgmap v3223: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:53.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:52 smithi107 ceph-mon[36485]: pgmap v3223: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:53.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:52 smithi106 ceph-mon[36505]: pgmap v3223: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:54.168 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5843 sec 2024-03-24T02:01:54.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5188.179333 secs 2024-03-24T02:01:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5188.179333 secs 2024-03-24T02:01:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5188.179333 secs 2024-03-24T02:01:55.168 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5844 sec 2024-03-24T02:01:55.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:54 smithi078 ceph-mon[32429]: pgmap v3224: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:55.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:54 smithi107 ceph-mon[36485]: pgmap v3224: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:55.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:54 smithi106 ceph-mon[36505]: pgmap v3224: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:56.168 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5845 sec 2024-03-24T02:01:57.169 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5846 sec 2024-03-24T02:01:57.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:56 smithi078 ceph-mon[32429]: pgmap v3225: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:57.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:56 smithi107 ceph-mon[36485]: pgmap v3225: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:57.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:56 smithi106 ceph-mon[36505]: pgmap v3225: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:58.168 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5847 sec 2024-03-24T02:01:59.169 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5848 sec 2024-03-24T02:01:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:59 smithi078 ceph-mon[32429]: pgmap v3226: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:01:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5193.179416 secs 2024-03-24T02:01:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:59 smithi107 ceph-mon[36485]: pgmap v3226: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:01:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5193.179416 secs 2024-03-24T02:01:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:59 smithi106 ceph-mon[36505]: pgmap v3226: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:01:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:01:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5193.179416 secs 2024-03-24T02:02:00.169 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5849 sec 2024-03-24T02:02:01.169 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5850 sec 2024-03-24T02:02:01.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:01 smithi078 ceph-mon[32429]: pgmap v3227: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:01 smithi107 ceph-mon[36485]: pgmap v3227: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:01 smithi106 ceph-mon[36505]: pgmap v3227: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:02.169 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5851 sec 2024-03-24T02:02:03.169 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5852 sec 2024-03-24T02:02:03.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:03 smithi078 ceph-mon[32429]: pgmap v3228: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:03.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:03 smithi107 ceph-mon[36485]: pgmap v3228: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:03.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:03 smithi106 ceph-mon[36505]: pgmap v3228: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:04.169 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5853 sec 2024-03-24T02:02:04.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5198.179504 secs 2024-03-24T02:02:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5198.179504 secs 2024-03-24T02:02:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5198.179504 secs 2024-03-24T02:02:05.169 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5854 sec 2024-03-24T02:02:05.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:05 smithi078 ceph-mon[32429]: pgmap v3229: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:05.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:05 smithi107 ceph-mon[36485]: pgmap v3229: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:05.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:05 smithi106 ceph-mon[36505]: pgmap v3229: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:06.169 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5855 sec 2024-03-24T02:02:07.170 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5856 sec 2024-03-24T02:02:07.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:07 smithi078 ceph-mon[32429]: pgmap v3230: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:07.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:07 smithi107 ceph-mon[36485]: pgmap v3230: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:07.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:07 smithi106 ceph-mon[36505]: pgmap v3230: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:08.169 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5857 sec 2024-03-24T02:02:09.169 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5858 sec 2024-03-24T02:02:09.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:09 smithi078 ceph-mon[32429]: pgmap v3231: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:09.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5203.179566 secs 2024-03-24T02:02:09.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:09 smithi107 ceph-mon[36485]: pgmap v3231: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:09.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5203.179566 secs 2024-03-24T02:02:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:09 smithi106 ceph-mon[36505]: pgmap v3231: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:09.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5203.179566 secs 2024-03-24T02:02:10.169 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5859 sec 2024-03-24T02:02:11.170 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5860 sec 2024-03-24T02:02:11.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:11 smithi078 ceph-mon[32429]: pgmap v3232: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:11.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:11 smithi107 ceph-mon[36485]: pgmap v3232: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:11.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:11 smithi106 ceph-mon[36505]: pgmap v3232: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:12.170 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5861 sec 2024-03-24T02:02:13.170 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5862 sec 2024-03-24T02:02:13.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:13 smithi078 ceph-mon[32429]: pgmap v3233: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:13.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:13 smithi107 ceph-mon[36485]: pgmap v3233: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:13.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:13 smithi106 ceph-mon[36505]: pgmap v3233: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:14.170 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5863 sec 2024-03-24T02:02:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5208.179628 secs 2024-03-24T02:02:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5208.179628 secs 2024-03-24T02:02:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5208.179628 secs 2024-03-24T02:02:15.170 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5864 sec 2024-03-24T02:02:15.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:15 smithi078 ceph-mon[32429]: pgmap v3234: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:15.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:15 smithi107 ceph-mon[36485]: pgmap v3234: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:15.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:15 smithi106 ceph-mon[36505]: pgmap v3234: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:16.170 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5865 sec 2024-03-24T02:02:17.170 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5866 sec 2024-03-24T02:02:17.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:17 smithi078 ceph-mon[32429]: pgmap v3235: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:17.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:17 smithi107 ceph-mon[36485]: pgmap v3235: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:17.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:17 smithi106 ceph-mon[36505]: pgmap v3235: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:18.170 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5867 sec 2024-03-24T02:02:19.170 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5868 sec 2024-03-24T02:02:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:19 smithi078 ceph-mon[32429]: pgmap v3236: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5213.179717 secs 2024-03-24T02:02:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:19 smithi107 ceph-mon[36485]: pgmap v3236: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:19.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5213.179717 secs 2024-03-24T02:02:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:19 smithi106 ceph-mon[36505]: pgmap v3236: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5213.179717 secs 2024-03-24T02:02:20.171 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5869 sec 2024-03-24T02:02:21.171 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5870 sec 2024-03-24T02:02:21.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:21 smithi078 ceph-mon[32429]: pgmap v3237: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:21.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:21 smithi107 ceph-mon[36485]: pgmap v3237: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:21.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:21 smithi106 ceph-mon[36505]: pgmap v3237: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:22.171 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5871 sec 2024-03-24T02:02:22.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:22 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:02:22.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:22 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:02:22.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:22 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:02:23.171 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5872 sec 2024-03-24T02:02:23.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:23 smithi078 ceph-mon[32429]: pgmap v3238: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:23.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:23 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:02:23.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:23 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:02:23.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:23 smithi078 ceph-mon[32429]: pgmap v3239: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:02:23.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:23 smithi078 ceph-mon[32429]: pgmap v3240: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:02:23.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:23 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:02:23.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:23 smithi107 ceph-mon[36485]: pgmap v3238: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:23.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:23 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:02:23.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:23 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:02:23.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:23 smithi107 ceph-mon[36485]: pgmap v3239: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:02:23.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:23 smithi107 ceph-mon[36485]: pgmap v3240: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:02:23.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:23 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:02:23.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:23 smithi106 ceph-mon[36505]: pgmap v3238: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:23.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:23 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:02:23.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:23 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:02:23.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:23 smithi106 ceph-mon[36505]: pgmap v3239: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:02:23.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:23 smithi106 ceph-mon[36505]: pgmap v3240: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:02:23.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:23 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:02:24.171 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5873 sec 2024-03-24T02:02:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5218.179754 secs 2024-03-24T02:02:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5218.179754 secs 2024-03-24T02:02:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5218.179754 secs 2024-03-24T02:02:25.171 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5874 sec 2024-03-24T02:02:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:25 smithi078 ceph-mon[32429]: pgmap v3241: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:02:25.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:25 smithi107 ceph-mon[36485]: pgmap v3241: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:02:25.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:25 smithi106 ceph-mon[36505]: pgmap v3241: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:02:26.171 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5875 sec 2024-03-24T02:02:27.171 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5876 sec 2024-03-24T02:02:27.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:27 smithi106 ceph-mon[36505]: pgmap v3242: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:02:27.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:27 smithi078 ceph-mon[32429]: pgmap v3242: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:02:27.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:27 smithi107 ceph-mon[36485]: pgmap v3242: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:02:28.171 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5877 sec 2024-03-24T02:02:29.171 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5878 sec 2024-03-24T02:02:29.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5223.179836 secs 2024-03-24T02:02:29.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:29 smithi106 ceph-mon[36505]: pgmap v3243: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:02:29.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5223.179836 secs 2024-03-24T02:02:29.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:29 smithi078 ceph-mon[32429]: pgmap v3243: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:02:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5223.179836 secs 2024-03-24T02:02:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:29 smithi107 ceph-mon[36485]: pgmap v3243: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:02:30.171 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5879 sec 2024-03-24T02:02:31.171 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5880 sec 2024-03-24T02:02:31.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:31 smithi078 ceph-mon[32429]: pgmap v3244: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:02:31.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:31 smithi107 ceph-mon[36485]: pgmap v3244: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:02:31.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:31 smithi106 ceph-mon[36505]: pgmap v3244: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:02:32.172 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5881 sec 2024-03-24T02:02:33.172 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5882 sec 2024-03-24T02:02:33.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:33 smithi078 ceph-mon[32429]: pgmap v3245: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:02:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:33 smithi107 ceph-mon[36485]: pgmap v3245: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:02:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:33 smithi106 ceph-mon[36505]: pgmap v3245: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:02:34.172 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5883 sec 2024-03-24T02:02:34.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5228.179875 secs 2024-03-24T02:02:34.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5228.179875 secs 2024-03-24T02:02:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5228.179875 secs 2024-03-24T02:02:35.172 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5884 sec 2024-03-24T02:02:35.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:35 smithi078 ceph-mon[32429]: pgmap v3246: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:35.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:35 smithi107 ceph-mon[36485]: pgmap v3246: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:35.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:35 smithi106 ceph-mon[36505]: pgmap v3246: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:36.172 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5885 sec 2024-03-24T02:02:37.172 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5886 sec 2024-03-24T02:02:37.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:37 smithi078 ceph-mon[32429]: pgmap v3247: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:37.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:37 smithi107 ceph-mon[36485]: pgmap v3247: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:37.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:37 smithi106 ceph-mon[36505]: pgmap v3247: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:38.172 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5887 sec 2024-03-24T02:02:39.172 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5888 sec 2024-03-24T02:02:39.705 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5233.179965 secs 2024-03-24T02:02:39.705 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:39 smithi106 ceph-mon[36505]: pgmap v3248: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:39.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5233.179965 secs 2024-03-24T02:02:39.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:39 smithi078 ceph-mon[32429]: pgmap v3248: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:39.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5233.179965 secs 2024-03-24T02:02:39.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:39 smithi107 ceph-mon[36485]: pgmap v3248: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:40.172 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5889 sec 2024-03-24T02:02:41.173 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5890 sec 2024-03-24T02:02:41.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:41 smithi078 ceph-mon[32429]: pgmap v3249: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:41.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:41 smithi107 ceph-mon[36485]: pgmap v3249: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:41.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:41 smithi106 ceph-mon[36505]: pgmap v3249: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:42.172 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5891 sec 2024-03-24T02:02:43.173 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5892 sec 2024-03-24T02:02:43.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:43 smithi078 ceph-mon[32429]: pgmap v3250: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:43 smithi107 ceph-mon[36485]: pgmap v3250: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:43 smithi106 ceph-mon[36505]: pgmap v3250: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:44.173 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5893 sec 2024-03-24T02:02:44.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5238.180047 secs 2024-03-24T02:02:44.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5238.180047 secs 2024-03-24T02:02:44.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5238.180047 secs 2024-03-24T02:02:45.172 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5894 sec 2024-03-24T02:02:45.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:45 smithi078 ceph-mon[32429]: pgmap v3251: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:45.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:45 smithi107 ceph-mon[36485]: pgmap v3251: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:45.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:45 smithi106 ceph-mon[36505]: pgmap v3251: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:46.173 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5895 sec 2024-03-24T02:02:47.174 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5896 sec 2024-03-24T02:02:47.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:47 smithi078 ceph-mon[32429]: pgmap v3252: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:47.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:47 smithi107 ceph-mon[36485]: pgmap v3252: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:47.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:47 smithi106 ceph-mon[36505]: pgmap v3252: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:48.174 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5897 sec 2024-03-24T02:02:49.173 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5898 sec 2024-03-24T02:02:49.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5243.180107 secs 2024-03-24T02:02:49.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:49 smithi078 ceph-mon[32429]: pgmap v3253: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:49.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5243.180107 secs 2024-03-24T02:02:49.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:49 smithi107 ceph-mon[36485]: pgmap v3253: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:49.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5243.180107 secs 2024-03-24T02:02:49.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:49 smithi106 ceph-mon[36505]: pgmap v3253: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:50.174 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5899 sec 2024-03-24T02:02:51.174 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5900 sec 2024-03-24T02:02:51.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:51 smithi078 ceph-mon[32429]: pgmap v3254: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:51.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:51 smithi107 ceph-mon[36485]: pgmap v3254: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:51 smithi106 ceph-mon[36505]: pgmap v3254: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:52.174 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5901 sec 2024-03-24T02:02:52.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:52 smithi078 ceph-mon[32429]: pgmap v3255: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:52.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:52 smithi107 ceph-mon[36485]: pgmap v3255: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:52.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:52 smithi106 ceph-mon[36505]: pgmap v3255: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:53.174 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5902 sec 2024-03-24T02:02:53.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5248.180195 secs 2024-03-24T02:02:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5248.180195 secs 2024-03-24T02:02:53.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5248.180195 secs 2024-03-24T02:02:54.174 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5903 sec 2024-03-24T02:02:54.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:54 smithi078 ceph-mon[32429]: pgmap v3256: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:54 smithi107 ceph-mon[36485]: pgmap v3256: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:54 smithi106 ceph-mon[36505]: pgmap v3256: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:55.174 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5904 sec 2024-03-24T02:02:56.175 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5905 sec 2024-03-24T02:02:57.174 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5906 sec 2024-03-24T02:02:57.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:57 smithi078 ceph-mon[32429]: pgmap v3257: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:57.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:57 smithi107 ceph-mon[36485]: pgmap v3257: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:57.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:57 smithi106 ceph-mon[36505]: pgmap v3257: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:58.175 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5907 sec 2024-03-24T02:02:59.174 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5908 sec 2024-03-24T02:02:59.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5253.180291 secs 2024-03-24T02:02:59.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:02:59 smithi078 ceph-mon[32429]: pgmap v3258: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:59.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5253.180291 secs 2024-03-24T02:02:59.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:02:59 smithi107 ceph-mon[36485]: pgmap v3258: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:02:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5253.180291 secs 2024-03-24T02:02:59.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:02:59 smithi106 ceph-mon[36505]: pgmap v3258: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:00.174 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5909 sec 2024-03-24T02:03:01.175 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5910 sec 2024-03-24T02:03:01.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:01 smithi078 ceph-mon[32429]: pgmap v3259: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:01.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:01 smithi107 ceph-mon[36485]: pgmap v3259: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:01.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:01 smithi106 ceph-mon[36505]: pgmap v3259: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:02.175 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5911 sec 2024-03-24T02:03:03.175 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5912 sec 2024-03-24T02:03:03.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:03 smithi078 ceph-mon[32429]: pgmap v3260: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:03.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:03 smithi107 ceph-mon[36485]: pgmap v3260: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:03 smithi106 ceph-mon[36505]: pgmap v3260: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:04.175 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5913 sec 2024-03-24T02:03:04.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5258.180351 secs 2024-03-24T02:03:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5258.180351 secs 2024-03-24T02:03:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5258.180351 secs 2024-03-24T02:03:05.175 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5914 sec 2024-03-24T02:03:05.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:05 smithi078 ceph-mon[32429]: pgmap v3261: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:05.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:05 smithi107 ceph-mon[36485]: pgmap v3261: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:05.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:05 smithi106 ceph-mon[36505]: pgmap v3261: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:06.175 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5915 sec 2024-03-24T02:03:07.175 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5916 sec 2024-03-24T02:03:07.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:07 smithi078 ceph-mon[32429]: pgmap v3262: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:07.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:07 smithi107 ceph-mon[36485]: pgmap v3262: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:07.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:07 smithi106 ceph-mon[36505]: pgmap v3262: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:08.175 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5917 sec 2024-03-24T02:03:09.176 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5918 sec 2024-03-24T02:03:09.705 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5263.180413 secs 2024-03-24T02:03:09.705 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:09 smithi106 ceph-mon[36505]: pgmap v3263: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:09.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5263.180413 secs 2024-03-24T02:03:09.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:09 smithi078 ceph-mon[32429]: pgmap v3263: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:09.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5263.180413 secs 2024-03-24T02:03:09.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:09 smithi107 ceph-mon[36485]: pgmap v3263: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:10.176 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5919 sec 2024-03-24T02:03:11.176 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5920 sec 2024-03-24T02:03:11.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:11 smithi078 ceph-mon[32429]: pgmap v3264: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:11.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:11 smithi107 ceph-mon[36485]: pgmap v3264: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:11.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:11 smithi106 ceph-mon[36505]: pgmap v3264: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:12.176 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5921 sec 2024-03-24T02:03:13.176 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5922 sec 2024-03-24T02:03:13.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:13 smithi078 ceph-mon[32429]: pgmap v3265: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:13 smithi107 ceph-mon[36485]: pgmap v3265: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:13 smithi106 ceph-mon[36505]: pgmap v3265: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:14.176 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5923 sec 2024-03-24T02:03:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5268.180447 secs 2024-03-24T02:03:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5268.180447 secs 2024-03-24T02:03:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5268.180447 secs 2024-03-24T02:03:15.176 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5924 sec 2024-03-24T02:03:15.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:15 smithi078 ceph-mon[32429]: pgmap v3266: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:15.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:15 smithi107 ceph-mon[36485]: pgmap v3266: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:15 smithi106 ceph-mon[36505]: pgmap v3266: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:16.177 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5925 sec 2024-03-24T02:03:17.177 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5926 sec 2024-03-24T02:03:17.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:17 smithi078 ceph-mon[32429]: pgmap v3267: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:17.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:17 smithi107 ceph-mon[36485]: pgmap v3267: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:17.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:17 smithi106 ceph-mon[36505]: pgmap v3267: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:18.176 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5927 sec 2024-03-24T02:03:18.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5273.180509 secs 2024-03-24T02:03:18.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:18 smithi078 ceph-mon[32429]: pgmap v3268: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:18.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5273.180509 secs 2024-03-24T02:03:18.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:18 smithi107 ceph-mon[36485]: pgmap v3268: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:18.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5273.180509 secs 2024-03-24T02:03:18.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:18 smithi106 ceph-mon[36505]: pgmap v3268: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:19.176 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5928 sec 2024-03-24T02:03:20.177 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5929 sec 2024-03-24T02:03:21.177 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5930 sec 2024-03-24T02:03:21.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:21 smithi078 ceph-mon[32429]: pgmap v3269: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:21 smithi107 ceph-mon[36485]: pgmap v3269: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:21.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:21 smithi106 ceph-mon[36505]: pgmap v3269: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:22.177 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5931 sec 2024-03-24T02:03:23.177 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5932 sec 2024-03-24T02:03:23.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:23 smithi078 ceph-mon[32429]: pgmap v3270: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:23.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:23 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:03:23.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:23 smithi107 ceph-mon[36485]: pgmap v3270: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:23.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:23 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:03:23.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:23 smithi106 ceph-mon[36505]: pgmap v3270: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:23.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:23 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:03:24.177 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5933 sec 2024-03-24T02:03:24.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5278.180576 secs 2024-03-24T02:03:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5278.180576 secs 2024-03-24T02:03:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5278.180576 secs 2024-03-24T02:03:25.177 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5934 sec 2024-03-24T02:03:25.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:25 smithi078 ceph-mon[32429]: pgmap v3271: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:25.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:03:25.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:03:25.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:03:25.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:03:25.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:03:25.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:03:25.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:03:25.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:03:25.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:03:25.780 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:25 smithi107 ceph-mon[36485]: pgmap v3271: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:25.780 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:03:25.780 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:03:25.780 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:03:25.780 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:03:25.781 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:03:25.781 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:03:25.781 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.781 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.781 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.781 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.781 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.781 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.781 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.781 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.781 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.781 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.781 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.781 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.781 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:03:25.781 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:03:25.782 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:03:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:25 smithi106 ceph-mon[36505]: pgmap v3271: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:03:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:03:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:03:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:03:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:03:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:03:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:03:25.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:03:25.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:03:25.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:03:26.177 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5935 sec 2024-03-24T02:03:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:26 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T02:03:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:26 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T02:03:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:26 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T02:03:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:26 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T02:03:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:26 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T02:03:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:26 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T02:03:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:26 smithi078 ceph-mon[32429]: pgmap v3272: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T02:03:26.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:26 smithi078 ceph-mon[32429]: pgmap v3273: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:03:26.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:26 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T02:03:26.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:26 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T02:03:26.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:26 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T02:03:26.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:26 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T02:03:26.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:26 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T02:03:26.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:26 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T02:03:26.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:26 smithi107 ceph-mon[36485]: pgmap v3272: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T02:03:26.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:26 smithi107 ceph-mon[36485]: pgmap v3273: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:03:26.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:26 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T02:03:26.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:26 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T02:03:26.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:26 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T02:03:26.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:26 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T02:03:26.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:26 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T02:03:26.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:26 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T02:03:26.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:26 smithi106 ceph-mon[36505]: pgmap v3272: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T02:03:26.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:26 smithi106 ceph-mon[36505]: pgmap v3273: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:03:27.177 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5936 sec 2024-03-24T02:03:28.178 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5937 sec 2024-03-24T02:03:28.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:28 smithi078 ceph-mon[32429]: pgmap v3274: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:03:28.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:28 smithi107 ceph-mon[36485]: pgmap v3274: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:03:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:28 smithi106 ceph-mon[36505]: pgmap v3274: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:03:29.177 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5938 sec 2024-03-24T02:03:29.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5283.180635 secs 2024-03-24T02:03:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5283.180635 secs 2024-03-24T02:03:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5283.180635 secs 2024-03-24T02:03:30.177 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5939 sec 2024-03-24T02:03:30.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:30 smithi078 ceph-mon[32429]: pgmap v3275: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:03:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:30 smithi107 ceph-mon[36485]: pgmap v3275: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:03:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:30 smithi106 ceph-mon[36505]: pgmap v3275: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:03:31.177 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5940 sec 2024-03-24T02:03:32.178 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5941 sec 2024-03-24T02:03:32.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:32 smithi078 ceph-mon[32429]: pgmap v3276: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:03:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:32 smithi107 ceph-mon[36485]: pgmap v3276: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:03:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:32 smithi106 ceph-mon[36505]: pgmap v3276: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:03:33.178 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5942 sec 2024-03-24T02:03:33.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5288.180693 secs 2024-03-24T02:03:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5288.180693 secs 2024-03-24T02:03:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5288.180693 secs 2024-03-24T02:03:34.178 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5943 sec 2024-03-24T02:03:34.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:34 smithi078 ceph-mon[32429]: pgmap v3277: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:03:34.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:34 smithi107 ceph-mon[36485]: pgmap v3277: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:03:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:34 smithi106 ceph-mon[36505]: pgmap v3277: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:03:35.178 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5944 sec 2024-03-24T02:03:35.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:35 smithi107 ceph-mon[36485]: pgmap v3278: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:03:35.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:35 smithi078 ceph-mon[32429]: pgmap v3278: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:03:35.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:35 smithi106 ceph-mon[36505]: pgmap v3278: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:03:36.178 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5945 sec 2024-03-24T02:03:37.178 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5946 sec 2024-03-24T02:03:38.178 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5947 sec 2024-03-24T02:03:38.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:38 smithi078 ceph-mon[32429]: pgmap v3279: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:38 smithi107 ceph-mon[36485]: pgmap v3279: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:38 smithi106 ceph-mon[36505]: pgmap v3279: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:39.178 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5948 sec 2024-03-24T02:03:39.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5293.180782 secs 2024-03-24T02:03:39.803 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5293.180782 secs 2024-03-24T02:03:39.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5293.180782 secs 2024-03-24T02:03:40.178 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5949 sec 2024-03-24T02:03:40.756 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:40 smithi078 ceph-mon[32429]: pgmap v3280: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:40.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:40 smithi107 ceph-mon[36485]: pgmap v3280: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:40 smithi106 ceph-mon[36505]: pgmap v3280: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:41.178 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5950 sec 2024-03-24T02:03:42.178 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5951 sec 2024-03-24T02:03:42.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:42 smithi078 ceph-mon[32429]: pgmap v3281: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:42.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:42 smithi107 ceph-mon[36485]: pgmap v3281: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:42.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:42 smithi106 ceph-mon[36505]: pgmap v3281: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:43.178 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5952 sec 2024-03-24T02:03:44.178 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5953 sec 2024-03-24T02:03:44.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5298.180843 secs 2024-03-24T02:03:44.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:44 smithi078 ceph-mon[32429]: pgmap v3282: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:44.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5298.180843 secs 2024-03-24T02:03:44.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:44 smithi107 ceph-mon[36485]: pgmap v3282: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:44.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5298.180843 secs 2024-03-24T02:03:44.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:44 smithi106 ceph-mon[36505]: pgmap v3282: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:45.179 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5954 sec 2024-03-24T02:03:46.179 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5955 sec 2024-03-24T02:03:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:46 smithi078 ceph-mon[32429]: pgmap v3283: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:46.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:46 smithi107 ceph-mon[36485]: pgmap v3283: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:46 smithi106 ceph-mon[36505]: pgmap v3283: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:47.179 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5956 sec 2024-03-24T02:03:48.179 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5957 sec 2024-03-24T02:03:48.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:48 smithi078 ceph-mon[32429]: pgmap v3284: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:48 smithi107 ceph-mon[36485]: pgmap v3284: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:48.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:48 smithi106 ceph-mon[36505]: pgmap v3284: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:49.179 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5958 sec 2024-03-24T02:03:49.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5303.180903 secs 2024-03-24T02:03:49.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5303.180903 secs 2024-03-24T02:03:49.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5303.180903 secs 2024-03-24T02:03:50.179 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5959 sec 2024-03-24T02:03:50.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:50 smithi078 ceph-mon[32429]: pgmap v3285: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:50 smithi107 ceph-mon[36485]: pgmap v3285: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:50.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:50 smithi106 ceph-mon[36505]: pgmap v3285: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:51.179 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5960 sec 2024-03-24T02:03:52.179 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5961 sec 2024-03-24T02:03:52.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:52 smithi078 ceph-mon[32429]: pgmap v3286: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:52.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:52 smithi107 ceph-mon[36485]: pgmap v3286: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:52.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:52 smithi106 ceph-mon[36505]: pgmap v3286: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:53.179 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5962 sec 2024-03-24T02:03:53.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5308.180959 secs 2024-03-24T02:03:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5308.180959 secs 2024-03-24T02:03:53.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5308.180959 secs 2024-03-24T02:03:54.179 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5963 sec 2024-03-24T02:03:54.745 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:54 smithi107 ceph-mon[36485]: pgmap v3287: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:54.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:54 smithi078 ceph-mon[32429]: pgmap v3287: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:54 smithi106 ceph-mon[36505]: pgmap v3287: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:55.179 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5964 sec 2024-03-24T02:03:55.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:55 smithi078 ceph-mon[32429]: pgmap v3288: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:55.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:55 smithi107 ceph-mon[36485]: pgmap v3288: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:55.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:55 smithi106 ceph-mon[36505]: pgmap v3288: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:56.179 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5965 sec 2024-03-24T02:03:57.180 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5966 sec 2024-03-24T02:03:58.180 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5967 sec 2024-03-24T02:03:58.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:58 smithi078 ceph-mon[32429]: pgmap v3289: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:58 smithi107 ceph-mon[36485]: pgmap v3289: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:58 smithi106 ceph-mon[36505]: pgmap v3289: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:03:59.180 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5968 sec 2024-03-24T02:03:59.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:03:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5313.181013 secs 2024-03-24T02:03:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:03:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5313.181013 secs 2024-03-24T02:03:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:03:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5313.181013 secs 2024-03-24T02:04:00.180 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5969 sec 2024-03-24T02:04:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:00 smithi078 ceph-mon[32429]: pgmap v3290: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:00.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:00 smithi107 ceph-mon[36485]: pgmap v3290: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:00.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:00 smithi106 ceph-mon[36505]: pgmap v3290: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:01.180 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5970 sec 2024-03-24T02:04:02.180 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5971 sec 2024-03-24T02:04:02.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:02 smithi078 ceph-mon[32429]: pgmap v3291: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:02.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:02 smithi107 ceph-mon[36485]: pgmap v3291: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:02.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:02 smithi106 ceph-mon[36505]: pgmap v3291: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:03.180 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5972 sec 2024-03-24T02:04:04.180 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5973 sec 2024-03-24T02:04:04.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5318.181102 secs 2024-03-24T02:04:04.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:04 smithi078 ceph-mon[32429]: pgmap v3292: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5318.181102 secs 2024-03-24T02:04:04.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:04 smithi107 ceph-mon[36485]: pgmap v3292: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5318.181102 secs 2024-03-24T02:04:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:04 smithi106 ceph-mon[36505]: pgmap v3292: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:05.180 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5974 sec 2024-03-24T02:04:06.180 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5975 sec 2024-03-24T02:04:06.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:06 smithi078 ceph-mon[32429]: pgmap v3293: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:06.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:06 smithi107 ceph-mon[36485]: pgmap v3293: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:06 smithi106 ceph-mon[36505]: pgmap v3293: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:07.180 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5976 sec 2024-03-24T02:04:08.181 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5977 sec 2024-03-24T02:04:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:08 smithi078 ceph-mon[32429]: pgmap v3294: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:08 smithi107 ceph-mon[36485]: pgmap v3294: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:08 smithi106 ceph-mon[36505]: pgmap v3294: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:09.181 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5978 sec 2024-03-24T02:04:09.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5323.181145 secs 2024-03-24T02:04:09.804 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5323.181145 secs 2024-03-24T02:04:09.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5323.181145 secs 2024-03-24T02:04:10.181 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5979 sec 2024-03-24T02:04:10.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:10 smithi078 ceph-mon[32429]: pgmap v3295: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:10.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:10 smithi107 ceph-mon[36485]: pgmap v3295: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:10 smithi106 ceph-mon[36505]: pgmap v3295: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:11.181 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5980 sec 2024-03-24T02:04:12.181 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5981 sec 2024-03-24T02:04:12.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:12 smithi078 ceph-mon[32429]: pgmap v3296: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:12.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:12 smithi107 ceph-mon[36485]: pgmap v3296: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:12.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:12 smithi106 ceph-mon[36505]: pgmap v3296: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:13.181 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5982 sec 2024-03-24T02:04:13.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5328.181160 secs 2024-03-24T02:04:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5328.181160 secs 2024-03-24T02:04:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5328.181160 secs 2024-03-24T02:04:14.181 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5983 sec 2024-03-24T02:04:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:14 smithi078 ceph-mon[32429]: pgmap v3297: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:14 smithi107 ceph-mon[36485]: pgmap v3297: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:14 smithi106 ceph-mon[36505]: pgmap v3297: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:15.181 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5984 sec 2024-03-24T02:04:15.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:15 smithi078 ceph-mon[32429]: pgmap v3298: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:15 smithi107 ceph-mon[36485]: pgmap v3298: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:15 smithi106 ceph-mon[36505]: pgmap v3298: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:16.181 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5985 sec 2024-03-24T02:04:17.181 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5986 sec 2024-03-24T02:04:18.182 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5987 sec 2024-03-24T02:04:18.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:18 smithi078 ceph-mon[32429]: pgmap v3299: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:18 smithi107 ceph-mon[36485]: pgmap v3299: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:18.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:18 smithi106 ceph-mon[36505]: pgmap v3299: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:19.182 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5988 sec 2024-03-24T02:04:19.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5333.181245 secs 2024-03-24T02:04:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5333.181245 secs 2024-03-24T02:04:19.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5333.181245 secs 2024-03-24T02:04:20.182 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5989 sec 2024-03-24T02:04:20.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:20 smithi078 ceph-mon[32429]: pgmap v3300: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:20.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:20 smithi107 ceph-mon[36485]: pgmap v3300: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:20.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:20 smithi106 ceph-mon[36505]: pgmap v3300: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:21.182 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5990 sec 2024-03-24T02:04:22.182 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5991 sec 2024-03-24T02:04:22.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:22 smithi078 ceph-mon[32429]: pgmap v3301: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:22.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:22 smithi107 ceph-mon[36485]: pgmap v3301: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:22.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:22 smithi106 ceph-mon[36505]: pgmap v3301: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:23.182 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5992 sec 2024-03-24T02:04:24.182 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5993 sec 2024-03-24T02:04:24.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5338.181342 secs 2024-03-24T02:04:24.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:24 smithi078 ceph-mon[32429]: pgmap v3302: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:24.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5338.181342 secs 2024-03-24T02:04:24.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:24 smithi107 ceph-mon[36485]: pgmap v3302: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5338.181342 secs 2024-03-24T02:04:24.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:24 smithi106 ceph-mon[36505]: pgmap v3302: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:25.182 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5994 sec 2024-03-24T02:04:26.182 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5995 sec 2024-03-24T02:04:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:26 smithi078 ceph-mon[32429]: pgmap v3303: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:26 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:04:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:26 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:04:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:26 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:04:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:26 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:04:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:26 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:04:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:26 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:04:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:26 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:04:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:26 smithi107 ceph-mon[36485]: pgmap v3303: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:26.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:26 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:04:26.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:26 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:04:26.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:26 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:04:26.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:26 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:04:26.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:26 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:04:26.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:26 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:04:26.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:26 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:04:26.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:26 smithi106 ceph-mon[36505]: pgmap v3303: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:26.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:26 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:04:26.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:26 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:04:26.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:26 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:04:26.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:26 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:04:26.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:26 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:04:26.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:26 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:04:26.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:26 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:04:27.182 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5996 sec 2024-03-24T02:04:27.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:27 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:04:27.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:27 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:04:27.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:27 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:04:27.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:27 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:04:27.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:27 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:04:27.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:27 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:04:27.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:27 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:04:27.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:27 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:04:27.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:27 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:04:28.183 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5997 sec 2024-03-24T02:04:28.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:28 smithi078 ceph-mon[32429]: pgmap v3304: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T02:04:28.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:28 smithi078 ceph-mon[32429]: pgmap v3305: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:04:28.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5343.181427 secs 2024-03-24T02:04:28.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:28 smithi107 ceph-mon[36485]: pgmap v3304: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T02:04:28.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:28 smithi107 ceph-mon[36485]: pgmap v3305: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:04:28.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5343.181427 secs 2024-03-24T02:04:28.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:28 smithi106 ceph-mon[36505]: pgmap v3304: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T02:04:28.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:28 smithi106 ceph-mon[36505]: pgmap v3305: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:04:28.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5343.181427 secs 2024-03-24T02:04:29.183 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5998 sec 2024-03-24T02:04:30.183 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 5999 sec 2024-03-24T02:04:30.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:30 smithi078 ceph-mon[32429]: pgmap v3306: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:04:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:30 smithi107 ceph-mon[36485]: pgmap v3306: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:04:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:30 smithi106 ceph-mon[36505]: pgmap v3306: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:04:31.183 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6000 sec 2024-03-24T02:04:31.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:31 smithi078 ceph-mon[32429]: pgmap v3307: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:04:31.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:31 smithi107 ceph-mon[36485]: pgmap v3307: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:04:31.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:31 smithi106 ceph-mon[36505]: pgmap v3307: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:04:32.183 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6001 sec 2024-03-24T02:04:33.183 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6002 sec 2024-03-24T02:04:34.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:33 smithi078 ceph-mon[32429]: pgmap v3308: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:04:34.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5348.181515 secs 2024-03-24T02:04:34.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:33 smithi107 ceph-mon[36485]: pgmap v3308: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:04:34.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5348.181515 secs 2024-03-24T02:04:34.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:33 smithi106 ceph-mon[36505]: pgmap v3308: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:04:34.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5348.181515 secs 2024-03-24T02:04:34.183 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6003 sec 2024-03-24T02:04:35.183 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6004 sec 2024-03-24T02:04:36.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:35 smithi078 ceph-mon[32429]: pgmap v3309: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:04:36.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:35 smithi107 ceph-mon[36485]: pgmap v3309: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:04:36.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:35 smithi106 ceph-mon[36505]: pgmap v3309: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:04:36.183 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6005 sec 2024-03-24T02:04:37.183 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6006 sec 2024-03-24T02:04:38.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:37 smithi078 ceph-mon[32429]: pgmap v3310: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:04:38.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:37 smithi107 ceph-mon[36485]: pgmap v3310: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:04:38.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:37 smithi106 ceph-mon[36505]: pgmap v3310: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:04:38.183 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6007 sec 2024-03-24T02:04:39.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5353.181600 secs 2024-03-24T02:04:39.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5353.181600 secs 2024-03-24T02:04:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5353.181600 secs 2024-03-24T02:04:39.183 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6008 sec 2024-03-24T02:04:40.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:39 smithi078 ceph-mon[32429]: pgmap v3311: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:40.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:39 smithi107 ceph-mon[36485]: pgmap v3311: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:39 smithi106 ceph-mon[36505]: pgmap v3311: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:40.183 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6009 sec 2024-03-24T02:04:41.184 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6010 sec 2024-03-24T02:04:42.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:41 smithi078 ceph-mon[32429]: pgmap v3312: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:42.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:41 smithi107 ceph-mon[36485]: pgmap v3312: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:42.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:41 smithi106 ceph-mon[36505]: pgmap v3312: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:42.184 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6011 sec 2024-03-24T02:04:43.184 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6012 sec 2024-03-24T02:04:44.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:43 smithi107 ceph-mon[36485]: pgmap v3313: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:44.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5358.181688 secs 2024-03-24T02:04:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:43 smithi106 ceph-mon[36505]: pgmap v3313: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:44.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5358.181688 secs 2024-03-24T02:04:44.184 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6013 sec 2024-03-24T02:04:44.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:43 smithi078 ceph-mon[32429]: pgmap v3313: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:44.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5358.181688 secs 2024-03-24T02:04:45.184 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6014 sec 2024-03-24T02:04:46.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:45 smithi106 ceph-mon[36505]: pgmap v3314: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:46.184 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6015 sec 2024-03-24T02:04:46.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:45 smithi078 ceph-mon[32429]: pgmap v3314: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:46.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:45 smithi107 ceph-mon[36485]: pgmap v3314: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:47.184 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6016 sec 2024-03-24T02:04:48.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:47 smithi106 ceph-mon[36505]: pgmap v3315: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:48.184 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6017 sec 2024-03-24T02:04:48.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:47 smithi078 ceph-mon[32429]: pgmap v3315: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:48.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:47 smithi107 ceph-mon[36485]: pgmap v3315: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5363.181749 secs 2024-03-24T02:04:49.184 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6018 sec 2024-03-24T02:04:49.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5363.181749 secs 2024-03-24T02:04:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5363.181749 secs 2024-03-24T02:04:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:49 smithi106 ceph-mon[36505]: pgmap v3316: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:50.184 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6019 sec 2024-03-24T02:04:50.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:49 smithi078 ceph-mon[32429]: pgmap v3316: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:50.345 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:49 smithi107 ceph-mon[36485]: pgmap v3316: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:51.185 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6020 sec 2024-03-24T02:04:51.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:51 smithi078 ceph-mon[32429]: pgmap v3317: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:51.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:51 smithi107 ceph-mon[36485]: pgmap v3317: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:51 smithi106 ceph-mon[36505]: pgmap v3317: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:52.185 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6021 sec 2024-03-24T02:04:53.185 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6022 sec 2024-03-24T02:04:54.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:53 smithi078 ceph-mon[32429]: pgmap v3318: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:54.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5368.181807 secs 2024-03-24T02:04:54.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:53 smithi107 ceph-mon[36485]: pgmap v3318: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:54.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5368.181807 secs 2024-03-24T02:04:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:53 smithi106 ceph-mon[36505]: pgmap v3318: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5368.181807 secs 2024-03-24T02:04:54.185 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6023 sec 2024-03-24T02:04:55.185 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6024 sec 2024-03-24T02:04:56.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:55 smithi078 ceph-mon[32429]: pgmap v3319: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:56.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:55 smithi107 ceph-mon[36485]: pgmap v3319: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:56.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:55 smithi106 ceph-mon[36505]: pgmap v3319: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:04:56.185 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6025 sec 2024-03-24T02:04:57.185 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6026 sec 2024-03-24T02:04:58.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:57 smithi078 ceph-mon[32429]: pgmap v3320: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:58.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:57 smithi107 ceph-mon[36485]: pgmap v3320: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:58.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:57 smithi106 ceph-mon[36505]: pgmap v3320: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:58.185 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6027 sec 2024-03-24T02:04:59.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5373.181891 secs 2024-03-24T02:04:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5373.181891 secs 2024-03-24T02:04:59.185 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6028 sec 2024-03-24T02:04:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5373.181891 secs 2024-03-24T02:05:00.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:04:59 smithi107 ceph-mon[36485]: pgmap v3321: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:00.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:04:59 smithi106 ceph-mon[36505]: pgmap v3321: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:00.185 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6029 sec 2024-03-24T02:05:00.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:04:59 smithi078 ceph-mon[32429]: pgmap v3321: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:01.186 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6030 sec 2024-03-24T02:05:02.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:01 smithi106 ceph-mon[36505]: pgmap v3322: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:02.186 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6031 sec 2024-03-24T02:05:02.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:01 smithi078 ceph-mon[32429]: pgmap v3322: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:02.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:01 smithi107 ceph-mon[36485]: pgmap v3322: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:03.186 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6032 sec 2024-03-24T02:05:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:03 smithi106 ceph-mon[36505]: pgmap v3323: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:04.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5378.181951 secs 2024-03-24T02:05:04.186 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6033 sec 2024-03-24T02:05:04.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:03 smithi078 ceph-mon[32429]: pgmap v3323: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:04.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5378.181951 secs 2024-03-24T02:05:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:03 smithi107 ceph-mon[36485]: pgmap v3323: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:04.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5378.181951 secs 2024-03-24T02:05:05.186 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6034 sec 2024-03-24T02:05:06.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:05 smithi106 ceph-mon[36505]: pgmap v3324: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:06.186 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6035 sec 2024-03-24T02:05:06.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:05 smithi078 ceph-mon[32429]: pgmap v3324: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:06.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:05 smithi107 ceph-mon[36485]: pgmap v3324: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:07.186 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6036 sec 2024-03-24T02:05:08.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:07 smithi106 ceph-mon[36505]: pgmap v3325: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:08.186 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6037 sec 2024-03-24T02:05:08.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:07 smithi078 ceph-mon[32429]: pgmap v3325: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:08.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:07 smithi107 ceph-mon[36485]: pgmap v3325: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:09.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5383.182012 secs 2024-03-24T02:05:09.186 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6038 sec 2024-03-24T02:05:09.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5383.182012 secs 2024-03-24T02:05:09.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5383.182012 secs 2024-03-24T02:05:10.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:09 smithi106 ceph-mon[36505]: pgmap v3326: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:10.186 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6039 sec 2024-03-24T02:05:10.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:09 smithi078 ceph-mon[32429]: pgmap v3326: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:10.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:09 smithi107 ceph-mon[36485]: pgmap v3326: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:11.187 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6040 sec 2024-03-24T02:05:12.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:11 smithi106 ceph-mon[36505]: pgmap v3327: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:12.186 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6041 sec 2024-03-24T02:05:12.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:11 smithi078 ceph-mon[32429]: pgmap v3327: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:12.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:11 smithi107 ceph-mon[36485]: pgmap v3327: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:13.187 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6042 sec 2024-03-24T02:05:14.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:13 smithi106 ceph-mon[36505]: pgmap v3328: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:14.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5388.182105 secs 2024-03-24T02:05:14.187 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6043 sec 2024-03-24T02:05:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:13 smithi078 ceph-mon[32429]: pgmap v3328: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5388.182105 secs 2024-03-24T02:05:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:13 smithi107 ceph-mon[36485]: pgmap v3328: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:14.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5388.182105 secs 2024-03-24T02:05:15.187 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6044 sec 2024-03-24T02:05:16.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:15 smithi106 ceph-mon[36505]: pgmap v3329: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:16.187 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6045 sec 2024-03-24T02:05:16.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:15 smithi078 ceph-mon[32429]: pgmap v3329: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:16.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:15 smithi107 ceph-mon[36485]: pgmap v3329: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:17.187 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6046 sec 2024-03-24T02:05:18.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:17 smithi106 ceph-mon[36505]: pgmap v3330: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:18.187 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6047 sec 2024-03-24T02:05:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:17 smithi078 ceph-mon[32429]: pgmap v3330: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:17 smithi107 ceph-mon[36485]: pgmap v3330: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:19.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5393.182196 secs 2024-03-24T02:05:19.187 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6048 sec 2024-03-24T02:05:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5393.182196 secs 2024-03-24T02:05:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5393.182196 secs 2024-03-24T02:05:20.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:19 smithi106 ceph-mon[36505]: pgmap v3331: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:20.187 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6049 sec 2024-03-24T02:05:20.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:19 smithi078 ceph-mon[32429]: pgmap v3331: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:20.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:19 smithi107 ceph-mon[36485]: pgmap v3331: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:21.187 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6050 sec 2024-03-24T02:05:22.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:21 smithi106 ceph-mon[36505]: pgmap v3332: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:22.187 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6051 sec 2024-03-24T02:05:22.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:21 smithi078 ceph-mon[32429]: pgmap v3332: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:22.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:21 smithi107 ceph-mon[36485]: pgmap v3332: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:23.187 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6052 sec 2024-03-24T02:05:24.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:23 smithi107 ceph-mon[36485]: pgmap v3333: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:24.147 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5398.182292 secs 2024-03-24T02:05:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:23 smithi106 ceph-mon[36505]: pgmap v3333: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5398.182292 secs 2024-03-24T02:05:24.187 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6053 sec 2024-03-24T02:05:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:23 smithi078 ceph-mon[32429]: pgmap v3333: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5398.182292 secs 2024-03-24T02:05:25.188 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6054 sec 2024-03-24T02:05:26.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:25 smithi106 ceph-mon[36505]: pgmap v3334: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:26.187 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6055 sec 2024-03-24T02:05:26.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:25 smithi078 ceph-mon[32429]: pgmap v3334: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:26.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:25 smithi107 ceph-mon[36485]: pgmap v3334: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:27.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:26 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:05:27.188 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6056 sec 2024-03-24T02:05:27.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:26 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:05:27.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:26 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:05:28.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:27 smithi106 ceph-mon[36505]: pgmap v3335: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:28.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:27 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:05:28.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:27 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:05:28.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:27 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:05:28.188 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6057 sec 2024-03-24T02:05:28.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:27 smithi078 ceph-mon[32429]: pgmap v3335: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:28.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:27 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:05:28.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:27 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:05:28.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:27 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:05:28.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:27 smithi107 ceph-mon[36485]: pgmap v3335: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:28.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:27 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:05:28.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:27 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:05:28.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:27 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:05:29.188 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6058 sec 2024-03-24T02:05:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:28 smithi078 ceph-mon[32429]: pgmap v3336: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:05:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:28 smithi078 ceph-mon[32429]: pgmap v3337: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:05:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5403.182379 secs 2024-03-24T02:05:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:28 smithi107 ceph-mon[36485]: pgmap v3336: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:05:29.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:28 smithi107 ceph-mon[36485]: pgmap v3337: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:05:29.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5403.182379 secs 2024-03-24T02:05:29.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:28 smithi106 ceph-mon[36505]: pgmap v3336: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:05:29.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:28 smithi106 ceph-mon[36505]: pgmap v3337: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:05:29.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5403.182379 secs 2024-03-24T02:05:30.188 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6059 sec 2024-03-24T02:05:31.188 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6060 sec 2024-03-24T02:05:31.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:30 smithi078 ceph-mon[32429]: pgmap v3338: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:05:31.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:30 smithi107 ceph-mon[36485]: pgmap v3338: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:05:31.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:30 smithi106 ceph-mon[36505]: pgmap v3338: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:05:32.188 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6061 sec 2024-03-24T02:05:33.189 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6062 sec 2024-03-24T02:05:33.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:32 smithi078 ceph-mon[32429]: pgmap v3339: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:05:33.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:32 smithi107 ceph-mon[36485]: pgmap v3339: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:05:33.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:32 smithi106 ceph-mon[36505]: pgmap v3339: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:05:34.188 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6063 sec 2024-03-24T02:05:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5408.182441 secs 2024-03-24T02:05:34.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5408.182441 secs 2024-03-24T02:05:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5408.182441 secs 2024-03-24T02:05:35.188 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6064 sec 2024-03-24T02:05:35.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:34 smithi078 ceph-mon[32429]: pgmap v3340: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:05:35.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:34 smithi107 ceph-mon[36485]: pgmap v3340: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:05:35.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:34 smithi106 ceph-mon[36505]: pgmap v3340: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:05:36.189 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6065 sec 2024-03-24T02:05:37.189 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6066 sec 2024-03-24T02:05:37.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:36 smithi078 ceph-mon[32429]: pgmap v3341: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:05:37.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:36 smithi107 ceph-mon[36485]: pgmap v3341: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:05:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:36 smithi106 ceph-mon[36505]: pgmap v3341: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:05:38.189 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6067 sec 2024-03-24T02:05:39.189 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6068 sec 2024-03-24T02:05:39.204 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:38 smithi106 ceph-mon[36505]: pgmap v3342: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:05:39.204 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5413.182501 secs 2024-03-24T02:05:39.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:38 smithi078 ceph-mon[32429]: pgmap v3342: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:05:39.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5413.182501 secs 2024-03-24T02:05:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:38 smithi107 ceph-mon[36485]: pgmap v3342: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:05:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5413.182501 secs 2024-03-24T02:05:40.189 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6069 sec 2024-03-24T02:05:41.189 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6070 sec 2024-03-24T02:05:41.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:40 smithi078 ceph-mon[32429]: pgmap v3343: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:40 smithi107 ceph-mon[36485]: pgmap v3343: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:41.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:40 smithi106 ceph-mon[36505]: pgmap v3343: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:42.189 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6071 sec 2024-03-24T02:05:43.189 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6072 sec 2024-03-24T02:05:43.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:42 smithi078 ceph-mon[32429]: pgmap v3344: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:43.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:42 smithi107 ceph-mon[36485]: pgmap v3344: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:43.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:42 smithi106 ceph-mon[36505]: pgmap v3344: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:44.189 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6073 sec 2024-03-24T02:05:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5418.182588 secs 2024-03-24T02:05:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5418.182588 secs 2024-03-24T02:05:44.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5418.182588 secs 2024-03-24T02:05:45.190 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6074 sec 2024-03-24T02:05:45.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:44 smithi078 ceph-mon[32429]: pgmap v3345: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:45.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:44 smithi107 ceph-mon[36485]: pgmap v3345: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:45.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:44 smithi106 ceph-mon[36505]: pgmap v3345: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:46.190 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6075 sec 2024-03-24T02:05:47.190 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6076 sec 2024-03-24T02:05:47.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:46 smithi078 ceph-mon[32429]: pgmap v3346: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:47.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:46 smithi107 ceph-mon[36485]: pgmap v3346: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:47.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:46 smithi106 ceph-mon[36505]: pgmap v3346: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:48.190 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6077 sec 2024-03-24T02:05:49.190 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6078 sec 2024-03-24T02:05:49.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:48 smithi078 ceph-mon[32429]: pgmap v3347: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:49.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5423.182649 secs 2024-03-24T02:05:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:48 smithi107 ceph-mon[36485]: pgmap v3347: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5423.182649 secs 2024-03-24T02:05:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:48 smithi106 ceph-mon[36505]: pgmap v3347: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5423.182649 secs 2024-03-24T02:05:50.190 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6079 sec 2024-03-24T02:05:51.190 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6080 sec 2024-03-24T02:05:51.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:50 smithi078 ceph-mon[32429]: pgmap v3348: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:51.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:50 smithi107 ceph-mon[36485]: pgmap v3348: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:51.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:50 smithi106 ceph-mon[36505]: pgmap v3348: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:52.190 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6081 sec 2024-03-24T02:05:53.190 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6082 sec 2024-03-24T02:05:53.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:52 smithi078 ceph-mon[32429]: pgmap v3349: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:53.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:52 smithi107 ceph-mon[36485]: pgmap v3349: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:53.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:52 smithi106 ceph-mon[36505]: pgmap v3349: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:54.190 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6083 sec 2024-03-24T02:05:54.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5428.182710 secs 2024-03-24T02:05:54.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5428.182710 secs 2024-03-24T02:05:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5428.182710 secs 2024-03-24T02:05:55.190 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6084 sec 2024-03-24T02:05:55.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:54 smithi078 ceph-mon[32429]: pgmap v3350: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:55.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:54 smithi107 ceph-mon[36485]: pgmap v3350: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:55.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:54 smithi106 ceph-mon[36505]: pgmap v3350: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:56.190 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6085 sec 2024-03-24T02:05:57.191 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6086 sec 2024-03-24T02:05:57.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:56 smithi078 ceph-mon[32429]: pgmap v3351: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:57.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:56 smithi107 ceph-mon[36485]: pgmap v3351: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:57.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:56 smithi106 ceph-mon[36505]: pgmap v3351: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:58.191 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6087 sec 2024-03-24T02:05:59.191 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6088 sec 2024-03-24T02:05:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:58 smithi078 ceph-mon[32429]: pgmap v3352: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:05:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5433.182769 secs 2024-03-24T02:05:59.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:58 smithi107 ceph-mon[36485]: pgmap v3352: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:59.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:05:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5433.182769 secs 2024-03-24T02:05:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:58 smithi106 ceph-mon[36505]: pgmap v3352: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:05:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:05:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5433.182769 secs 2024-03-24T02:06:00.191 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6089 sec 2024-03-24T02:06:01.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:00 smithi078 ceph-mon[32429]: pgmap v3353: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:01.191 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6090 sec 2024-03-24T02:06:01.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:00 smithi107 ceph-mon[36485]: pgmap v3353: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:00 smithi106 ceph-mon[36505]: pgmap v3353: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:02.191 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6091 sec 2024-03-24T02:06:03.191 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6092 sec 2024-03-24T02:06:03.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:02 smithi078 ceph-mon[32429]: pgmap v3354: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:03.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:02 smithi107 ceph-mon[36485]: pgmap v3354: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:03.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:02 smithi106 ceph-mon[36505]: pgmap v3354: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:04.191 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6093 sec 2024-03-24T02:06:04.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5438.182805 secs 2024-03-24T02:06:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5438.182805 secs 2024-03-24T02:06:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5438.182805 secs 2024-03-24T02:06:05.191 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6094 sec 2024-03-24T02:06:05.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:04 smithi078 ceph-mon[32429]: pgmap v3355: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:05.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:04 smithi107 ceph-mon[36485]: pgmap v3355: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:05.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:04 smithi106 ceph-mon[36505]: pgmap v3355: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:06.191 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6095 sec 2024-03-24T02:06:07.192 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6096 sec 2024-03-24T02:06:07.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:06 smithi078 ceph-mon[32429]: pgmap v3356: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:07.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:06 smithi107 ceph-mon[36485]: pgmap v3356: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:07.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:06 smithi106 ceph-mon[36505]: pgmap v3356: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:08.192 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6097 sec 2024-03-24T02:06:09.191 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6098 sec 2024-03-24T02:06:09.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:08 smithi078 ceph-mon[32429]: pgmap v3357: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:09.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5443.182892 secs 2024-03-24T02:06:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:08 smithi107 ceph-mon[36485]: pgmap v3357: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:09.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5443.182892 secs 2024-03-24T02:06:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:08 smithi106 ceph-mon[36505]: pgmap v3357: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:09.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5443.182892 secs 2024-03-24T02:06:10.191 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6099 sec 2024-03-24T02:06:11.192 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6100 sec 2024-03-24T02:06:11.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:10 smithi078 ceph-mon[32429]: pgmap v3358: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:11.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:10 smithi107 ceph-mon[36485]: pgmap v3358: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:11.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:10 smithi106 ceph-mon[36505]: pgmap v3358: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:12.192 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6101 sec 2024-03-24T02:06:13.192 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6102 sec 2024-03-24T02:06:13.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:12 smithi078 ceph-mon[32429]: pgmap v3359: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:13.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:12 smithi107 ceph-mon[36485]: pgmap v3359: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:13.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:12 smithi106 ceph-mon[36505]: pgmap v3359: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:14.192 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6103 sec 2024-03-24T02:06:14.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5448.182977 secs 2024-03-24T02:06:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5448.182977 secs 2024-03-24T02:06:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5448.182977 secs 2024-03-24T02:06:15.192 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6104 sec 2024-03-24T02:06:15.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:14 smithi078 ceph-mon[32429]: pgmap v3360: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:15.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:14 smithi107 ceph-mon[36485]: pgmap v3360: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:15.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:14 smithi106 ceph-mon[36505]: pgmap v3360: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:16.192 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6105 sec 2024-03-24T02:06:17.193 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6106 sec 2024-03-24T02:06:17.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:17 smithi078 ceph-mon[32429]: pgmap v3361: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:17.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:17 smithi107 ceph-mon[36485]: pgmap v3361: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:17.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:17 smithi106 ceph-mon[36505]: pgmap v3361: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:18.192 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6107 sec 2024-03-24T02:06:19.192 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6108 sec 2024-03-24T02:06:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:19 smithi078 ceph-mon[32429]: pgmap v3362: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5453.183066 secs 2024-03-24T02:06:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:19 smithi107 ceph-mon[36485]: pgmap v3362: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:19.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5453.183066 secs 2024-03-24T02:06:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:19 smithi106 ceph-mon[36505]: pgmap v3362: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5453.183066 secs 2024-03-24T02:06:20.193 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6109 sec 2024-03-24T02:06:21.193 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6110 sec 2024-03-24T02:06:21.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:21 smithi078 ceph-mon[32429]: pgmap v3363: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:21.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:21 smithi107 ceph-mon[36485]: pgmap v3363: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:21.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:21 smithi106 ceph-mon[36505]: pgmap v3363: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:22.193 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6111 sec 2024-03-24T02:06:23.193 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6112 sec 2024-03-24T02:06:23.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:23 smithi078 ceph-mon[32429]: pgmap v3364: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:23.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:23 smithi107 ceph-mon[36485]: pgmap v3364: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:23.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:23 smithi106 ceph-mon[36505]: pgmap v3364: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:24.193 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6113 sec 2024-03-24T02:06:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5458.183125 secs 2024-03-24T02:06:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5458.183125 secs 2024-03-24T02:06:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5458.183125 secs 2024-03-24T02:06:25.193 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6114 sec 2024-03-24T02:06:25.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:25 smithi078 ceph-mon[32429]: pgmap v3365: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:25.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:25 smithi107 ceph-mon[36485]: pgmap v3365: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:25.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:25 smithi106 ceph-mon[36505]: pgmap v3365: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:26.193 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6115 sec 2024-03-24T02:06:27.193 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6116 sec 2024-03-24T02:06:27.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:27 smithi078 ceph-mon[32429]: pgmap v3366: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:27.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:27 smithi107 ceph-mon[36485]: pgmap v3366: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:27.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:27 smithi106 ceph-mon[36505]: pgmap v3366: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:28.193 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6117 sec 2024-03-24T02:06:28.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:28 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:06:28.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:28 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:06:28.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:28 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:06:29.193 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6118 sec 2024-03-24T02:06:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:29 smithi078 ceph-mon[32429]: pgmap v3367: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5463.183189 secs 2024-03-24T02:06:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:29 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:06:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:29 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:06:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:29 smithi078 ceph-mon[32429]: pgmap v3368: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:06:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:29 smithi078 ceph-mon[32429]: pgmap v3369: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:06:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:29 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:06:29.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:29 smithi107 ceph-mon[36485]: pgmap v3367: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:29.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5463.183189 secs 2024-03-24T02:06:29.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:29 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:06:29.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:29 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:06:29.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:29 smithi107 ceph-mon[36485]: pgmap v3368: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:06:29.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:29 smithi107 ceph-mon[36485]: pgmap v3369: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:06:29.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:29 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:06:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:29 smithi106 ceph-mon[36505]: pgmap v3367: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:29.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5463.183189 secs 2024-03-24T02:06:29.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:29 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:06:29.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:29 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:06:29.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:29 smithi106 ceph-mon[36505]: pgmap v3368: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:06:29.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:29 smithi106 ceph-mon[36505]: pgmap v3369: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:06:29.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:29 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:06:30.193 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6119 sec 2024-03-24T02:06:31.193 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6120 sec 2024-03-24T02:06:31.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:31 smithi106 ceph-mon[36505]: pgmap v3370: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:31.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:31 smithi078 ceph-mon[32429]: pgmap v3370: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:31.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:31 smithi107 ceph-mon[36485]: pgmap v3370: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:32.193 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6121 sec 2024-03-24T02:06:33.194 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6122 sec 2024-03-24T02:06:33.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:33 smithi106 ceph-mon[36505]: pgmap v3371: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:06:33.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:33 smithi078 ceph-mon[32429]: pgmap v3371: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:06:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:33 smithi107 ceph-mon[36485]: pgmap v3371: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:06:34.194 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6123 sec 2024-03-24T02:06:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5468.183275 secs 2024-03-24T02:06:34.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5468.183275 secs 2024-03-24T02:06:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5468.183275 secs 2024-03-24T02:06:35.194 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6124 sec 2024-03-24T02:06:35.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:35 smithi106 ceph-mon[36505]: pgmap v3372: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:06:35.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:35 smithi078 ceph-mon[32429]: pgmap v3372: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:06:35.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:35 smithi107 ceph-mon[36485]: pgmap v3372: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:06:36.194 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6125 sec 2024-03-24T02:06:37.194 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6126 sec 2024-03-24T02:06:37.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:37 smithi106 ceph-mon[36505]: pgmap v3373: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:06:37.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:37 smithi078 ceph-mon[32429]: pgmap v3373: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:06:37.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:37 smithi107 ceph-mon[36485]: pgmap v3373: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:06:38.194 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6127 sec 2024-03-24T02:06:39.194 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6128 sec 2024-03-24T02:06:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5473.183368 secs 2024-03-24T02:06:39.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:39 smithi106 ceph-mon[36505]: pgmap v3374: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:06:39.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5473.183368 secs 2024-03-24T02:06:39.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:39 smithi078 ceph-mon[32429]: pgmap v3374: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:06:39.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5473.183368 secs 2024-03-24T02:06:39.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:39 smithi107 ceph-mon[36485]: pgmap v3374: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:06:40.194 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6129 sec 2024-03-24T02:06:41.194 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6130 sec 2024-03-24T02:06:41.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:41 smithi106 ceph-mon[36505]: pgmap v3375: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:06:41.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:41 smithi078 ceph-mon[32429]: pgmap v3375: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:06:41.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:41 smithi107 ceph-mon[36485]: pgmap v3375: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:06:42.194 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6131 sec 2024-03-24T02:06:43.194 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6132 sec 2024-03-24T02:06:43.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:43 smithi078 ceph-mon[32429]: pgmap v3376: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:06:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:43 smithi107 ceph-mon[36485]: pgmap v3376: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:06:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:43 smithi106 ceph-mon[36505]: pgmap v3376: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:06:44.194 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6133 sec 2024-03-24T02:06:44.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5478.183427 secs 2024-03-24T02:06:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5478.183427 secs 2024-03-24T02:06:44.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5478.183427 secs 2024-03-24T02:06:45.194 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6134 sec 2024-03-24T02:06:45.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:45 smithi078 ceph-mon[32429]: pgmap v3377: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:45.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:45 smithi107 ceph-mon[36485]: pgmap v3377: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:45.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:45 smithi106 ceph-mon[36505]: pgmap v3377: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:46.194 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6135 sec 2024-03-24T02:06:47.195 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6136 sec 2024-03-24T02:06:47.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:47 smithi078 ceph-mon[32429]: pgmap v3378: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:47.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:47 smithi107 ceph-mon[36485]: pgmap v3378: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:47.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:47 smithi106 ceph-mon[36505]: pgmap v3378: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:48.195 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6137 sec 2024-03-24T02:06:49.195 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6138 sec 2024-03-24T02:06:49.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5483.183513 secs 2024-03-24T02:06:49.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:49 smithi078 ceph-mon[32429]: pgmap v3379: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:49.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5483.183513 secs 2024-03-24T02:06:49.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:49 smithi107 ceph-mon[36485]: pgmap v3379: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:49.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5483.183513 secs 2024-03-24T02:06:49.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:49 smithi106 ceph-mon[36505]: pgmap v3379: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:50.195 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6139 sec 2024-03-24T02:06:51.195 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6140 sec 2024-03-24T02:06:51.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:51 smithi078 ceph-mon[32429]: pgmap v3380: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:06:51.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:51 smithi107 ceph-mon[36485]: pgmap v3380: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:06:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:51 smithi106 ceph-mon[36505]: pgmap v3380: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:06:52.195 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6141 sec 2024-03-24T02:06:53.195 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6142 sec 2024-03-24T02:06:53.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:53 smithi078 ceph-mon[32429]: pgmap v3381: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:06:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:53 smithi107 ceph-mon[36485]: pgmap v3381: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:06:53.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:53 smithi106 ceph-mon[36505]: pgmap v3381: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:06:54.195 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6143 sec 2024-03-24T02:06:54.744 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5488.183625 secs 2024-03-24T02:06:54.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5488.183625 secs 2024-03-24T02:06:54.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5488.183625 secs 2024-03-24T02:06:55.195 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6144 sec 2024-03-24T02:06:55.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:55 smithi078 ceph-mon[32429]: pgmap v3382: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:55.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:55 smithi107 ceph-mon[36485]: pgmap v3382: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:55.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:55 smithi106 ceph-mon[36505]: pgmap v3382: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:56.195 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6145 sec 2024-03-24T02:06:57.196 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6146 sec 2024-03-24T02:06:57.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:57 smithi078 ceph-mon[32429]: pgmap v3383: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:57.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:57 smithi107 ceph-mon[36485]: pgmap v3383: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:57.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:57 smithi106 ceph-mon[36505]: pgmap v3383: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:58.196 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6147 sec 2024-03-24T02:06:59.196 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6148 sec 2024-03-24T02:06:59.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5493.183688 secs 2024-03-24T02:06:59.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:06:59 smithi078 ceph-mon[32429]: pgmap v3384: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5493.183688 secs 2024-03-24T02:06:59.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:06:59 smithi107 ceph-mon[36485]: pgmap v3384: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:06:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5493.183688 secs 2024-03-24T02:06:59.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:06:59 smithi106 ceph-mon[36505]: pgmap v3384: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:00.196 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6149 sec 2024-03-24T02:07:01.196 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6150 sec 2024-03-24T02:07:01.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:01 smithi078 ceph-mon[32429]: pgmap v3385: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:07:01.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:01 smithi107 ceph-mon[36485]: pgmap v3385: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:07:01.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:01 smithi106 ceph-mon[36505]: pgmap v3385: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:07:02.196 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6151 sec 2024-03-24T02:07:03.196 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6152 sec 2024-03-24T02:07:03.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:03 smithi078 ceph-mon[32429]: pgmap v3386: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:07:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:03 smithi107 ceph-mon[36485]: pgmap v3386: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:07:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:03 smithi106 ceph-mon[36505]: pgmap v3386: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:07:04.196 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6153 sec 2024-03-24T02:07:04.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5498.183781 secs 2024-03-24T02:07:04.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5498.183781 secs 2024-03-24T02:07:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5498.183781 secs 2024-03-24T02:07:05.196 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6154 sec 2024-03-24T02:07:05.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:05 smithi078 ceph-mon[32429]: pgmap v3387: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:05.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:05 smithi107 ceph-mon[36485]: pgmap v3387: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:05.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:05 smithi106 ceph-mon[36505]: pgmap v3387: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:06.197 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6155 sec 2024-03-24T02:07:07.197 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6156 sec 2024-03-24T02:07:07.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:07 smithi078 ceph-mon[32429]: pgmap v3388: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:07.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:07 smithi107 ceph-mon[36485]: pgmap v3388: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:07.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:07 smithi106 ceph-mon[36505]: pgmap v3388: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:08.197 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6157 sec 2024-03-24T02:07:09.197 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6158 sec 2024-03-24T02:07:09.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5503.183868 secs 2024-03-24T02:07:09.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:09 smithi078 ceph-mon[32429]: pgmap v3389: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:09.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5503.183868 secs 2024-03-24T02:07:09.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:09 smithi107 ceph-mon[36485]: pgmap v3389: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:09.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5503.183868 secs 2024-03-24T02:07:09.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:09 smithi106 ceph-mon[36505]: pgmap v3389: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:10.197 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6159 sec 2024-03-24T02:07:11.197 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6160 sec 2024-03-24T02:07:11.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:11 smithi078 ceph-mon[32429]: pgmap v3390: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:07:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:11 smithi107 ceph-mon[36485]: pgmap v3390: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:07:11.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:11 smithi106 ceph-mon[36505]: pgmap v3390: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:07:12.197 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6161 sec 2024-03-24T02:07:13.197 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6162 sec 2024-03-24T02:07:13.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:13 smithi078 ceph-mon[32429]: pgmap v3391: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:07:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:13 smithi107 ceph-mon[36485]: pgmap v3391: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:07:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:13 smithi106 ceph-mon[36505]: pgmap v3391: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:07:14.197 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6163 sec 2024-03-24T02:07:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5508.183956 secs 2024-03-24T02:07:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5508.183956 secs 2024-03-24T02:07:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5508.183956 secs 2024-03-24T02:07:15.197 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6164 sec 2024-03-24T02:07:15.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:15 smithi078 ceph-mon[32429]: pgmap v3392: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:15 smithi107 ceph-mon[36485]: pgmap v3392: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:15 smithi106 ceph-mon[36505]: pgmap v3392: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:16.197 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6165 sec 2024-03-24T02:07:17.197 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6166 sec 2024-03-24T02:07:17.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:17 smithi078 ceph-mon[32429]: pgmap v3393: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:17.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:17 smithi107 ceph-mon[36485]: pgmap v3393: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:17.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:17 smithi106 ceph-mon[36505]: pgmap v3393: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:18.198 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6167 sec 2024-03-24T02:07:18.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5513.184040 secs 2024-03-24T02:07:18.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5513.184040 secs 2024-03-24T02:07:18.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5513.184040 secs 2024-03-24T02:07:19.198 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6168 sec 2024-03-24T02:07:19.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:19 smithi078 ceph-mon[32429]: pgmap v3394: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:19 smithi107 ceph-mon[36485]: pgmap v3394: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:19.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:19 smithi106 ceph-mon[36505]: pgmap v3394: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:20.198 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6169 sec 2024-03-24T02:07:21.198 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6170 sec 2024-03-24T02:07:21.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:21 smithi078 ceph-mon[32429]: pgmap v3395: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:07:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:21 smithi107 ceph-mon[36485]: pgmap v3395: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:07:21.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:21 smithi106 ceph-mon[36505]: pgmap v3395: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:07:22.198 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6171 sec 2024-03-24T02:07:23.198 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6172 sec 2024-03-24T02:07:23.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:23 smithi078 ceph-mon[32429]: pgmap v3396: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:07:23.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5518.184131 secs 2024-03-24T02:07:23.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:23 smithi107 ceph-mon[36485]: pgmap v3396: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:07:23.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5518.184131 secs 2024-03-24T02:07:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:23 smithi106 ceph-mon[36505]: pgmap v3396: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:07:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5518.184131 secs 2024-03-24T02:07:24.198 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6173 sec 2024-03-24T02:07:24.745 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:24 smithi107 ceph-mon[36485]: pgmap v3397: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:24 smithi078 ceph-mon[32429]: pgmap v3397: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:24 smithi106 ceph-mon[36505]: pgmap v3397: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:25.198 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6174 sec 2024-03-24T02:07:26.199 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6175 sec 2024-03-24T02:07:27.198 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6176 sec 2024-03-24T02:07:27.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:27 smithi078 ceph-mon[32429]: pgmap v3398: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:27.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:27 smithi107 ceph-mon[36485]: pgmap v3398: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:27.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:27 smithi106 ceph-mon[36505]: pgmap v3398: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:28.199 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6177 sec 2024-03-24T02:07:29.199 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6178 sec 2024-03-24T02:07:29.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5523.184217 secs 2024-03-24T02:07:29.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:29 smithi078 ceph-mon[32429]: pgmap v3399: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:29.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:29 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:07:29.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:29 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:07:29.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:29 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:07:29.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:29 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:07:29.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5523.184217 secs 2024-03-24T02:07:29.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:29 smithi107 ceph-mon[36485]: pgmap v3399: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:29.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:29 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:07:29.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:29 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:07:29.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:29 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:07:29.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:29 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:07:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5523.184217 secs 2024-03-24T02:07:29.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:29 smithi106 ceph-mon[36505]: pgmap v3399: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:29.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:29 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:07:29.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:29 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:07:29.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:29 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:07:29.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:29 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:07:30.199 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6179 sec 2024-03-24T02:07:30.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:30 smithi078 ceph-mon[32429]: pgmap v3400: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:07:30.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:30 smithi078 ceph-mon[32429]: pgmap v3401: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:30.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:30 smithi107 ceph-mon[36485]: pgmap v3400: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:07:30.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:30 smithi107 ceph-mon[36485]: pgmap v3401: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:30 smithi106 ceph-mon[36505]: pgmap v3400: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:07:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:30 smithi106 ceph-mon[36505]: pgmap v3401: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:31.199 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6180 sec 2024-03-24T02:07:32.199 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6181 sec 2024-03-24T02:07:32.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:32 smithi078 ceph-mon[32429]: pgmap v3402: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:07:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:32 smithi107 ceph-mon[36485]: pgmap v3402: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:07:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:32 smithi106 ceph-mon[36505]: pgmap v3402: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:07:33.199 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6182 sec 2024-03-24T02:07:34.199 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6183 sec 2024-03-24T02:07:34.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5528.184304 secs 2024-03-24T02:07:34.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:34 smithi078 ceph-mon[32429]: pgmap v3403: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:07:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5528.184304 secs 2024-03-24T02:07:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:34 smithi107 ceph-mon[36485]: pgmap v3403: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:07:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5528.184304 secs 2024-03-24T02:07:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:34 smithi106 ceph-mon[36505]: pgmap v3403: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:07:35.199 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6184 sec 2024-03-24T02:07:36.199 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6185 sec 2024-03-24T02:07:36.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:36 smithi078 ceph-mon[32429]: pgmap v3404: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:07:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:36 smithi107 ceph-mon[36485]: pgmap v3404: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:07:36.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:36 smithi106 ceph-mon[36505]: pgmap v3404: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:07:37.199 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6186 sec 2024-03-24T02:07:38.199 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6187 sec 2024-03-24T02:07:38.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:38 smithi078 ceph-mon[32429]: pgmap v3405: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:07:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:38 smithi107 ceph-mon[36485]: pgmap v3405: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:07:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:38 smithi106 ceph-mon[36505]: pgmap v3405: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:07:39.200 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6188 sec 2024-03-24T02:07:39.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5533.184362 secs 2024-03-24T02:07:39.804 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5533.184362 secs 2024-03-24T02:07:39.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5533.184362 secs 2024-03-24T02:07:40.200 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6189 sec 2024-03-24T02:07:40.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:40 smithi078 ceph-mon[32429]: pgmap v3406: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:07:40.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:40 smithi107 ceph-mon[36485]: pgmap v3406: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:07:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:40 smithi106 ceph-mon[36505]: pgmap v3406: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:07:41.200 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6190 sec 2024-03-24T02:07:42.200 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6191 sec 2024-03-24T02:07:42.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:42 smithi078 ceph-mon[32429]: pgmap v3407: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:42.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:42 smithi107 ceph-mon[36485]: pgmap v3407: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:42.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:42 smithi106 ceph-mon[36505]: pgmap v3407: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:43.200 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6192 sec 2024-03-24T02:07:44.200 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6193 sec 2024-03-24T02:07:44.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5538.184449 secs 2024-03-24T02:07:44.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:44 smithi078 ceph-mon[32429]: pgmap v3408: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5538.184449 secs 2024-03-24T02:07:44.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:44 smithi107 ceph-mon[36485]: pgmap v3408: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:44.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5538.184449 secs 2024-03-24T02:07:44.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:44 smithi106 ceph-mon[36505]: pgmap v3408: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:45.200 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6194 sec 2024-03-24T02:07:46.200 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6195 sec 2024-03-24T02:07:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:46 smithi078 ceph-mon[32429]: pgmap v3409: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:46.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:46 smithi107 ceph-mon[36485]: pgmap v3409: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:46.906 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:46 smithi106 ceph-mon[36505]: pgmap v3409: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:47.200 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6196 sec 2024-03-24T02:07:48.200 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6197 sec 2024-03-24T02:07:48.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:48 smithi078 ceph-mon[32429]: pgmap v3410: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:48 smithi107 ceph-mon[36485]: pgmap v3410: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:48 smithi106 ceph-mon[36505]: pgmap v3410: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:49.201 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6198 sec 2024-03-24T02:07:49.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5543.184506 secs 2024-03-24T02:07:49.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5543.184506 secs 2024-03-24T02:07:49.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5543.184506 secs 2024-03-24T02:07:50.201 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6199 sec 2024-03-24T02:07:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:50 smithi078 ceph-mon[32429]: pgmap v3411: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:50.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:50 smithi107 ceph-mon[36485]: pgmap v3411: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:50.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:50 smithi106 ceph-mon[36505]: pgmap v3411: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:51.201 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6200 sec 2024-03-24T02:07:52.201 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6201 sec 2024-03-24T02:07:52.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:52 smithi078 ceph-mon[32429]: pgmap v3412: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:52.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:52 smithi107 ceph-mon[36485]: pgmap v3412: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:52.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:52 smithi106 ceph-mon[36505]: pgmap v3412: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:53.201 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6202 sec 2024-03-24T02:07:54.201 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6203 sec 2024-03-24T02:07:54.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5548.184589 secs 2024-03-24T02:07:54.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:54 smithi078 ceph-mon[32429]: pgmap v3413: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:54.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5548.184589 secs 2024-03-24T02:07:54.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:54 smithi107 ceph-mon[36485]: pgmap v3413: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5548.184589 secs 2024-03-24T02:07:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:54 smithi106 ceph-mon[36505]: pgmap v3413: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:55.201 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6204 sec 2024-03-24T02:07:56.201 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6205 sec 2024-03-24T02:07:56.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:56 smithi078 ceph-mon[32429]: pgmap v3414: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:56.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:56 smithi107 ceph-mon[36485]: pgmap v3414: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:56.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:56 smithi106 ceph-mon[36505]: pgmap v3414: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:07:57.201 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6206 sec 2024-03-24T02:07:58.201 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6207 sec 2024-03-24T02:07:58.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:58 smithi078 ceph-mon[32429]: pgmap v3415: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:58 smithi107 ceph-mon[36485]: pgmap v3415: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:58 smithi106 ceph-mon[36505]: pgmap v3415: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:59.201 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6208 sec 2024-03-24T02:07:59.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:07:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5553.184647 secs 2024-03-24T02:07:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:07:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5553.184647 secs 2024-03-24T02:07:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:07:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5553.184647 secs 2024-03-24T02:08:00.201 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6209 sec 2024-03-24T02:08:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:00 smithi078 ceph-mon[32429]: pgmap v3416: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:00 smithi107 ceph-mon[36485]: pgmap v3416: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:00.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:00 smithi106 ceph-mon[36505]: pgmap v3416: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:01.201 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6210 sec 2024-03-24T02:08:02.201 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6211 sec 2024-03-24T02:08:02.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:02 smithi078 ceph-mon[32429]: pgmap v3417: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:02 smithi107 ceph-mon[36485]: pgmap v3417: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:02.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:02 smithi106 ceph-mon[36505]: pgmap v3417: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:03.202 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6212 sec 2024-03-24T02:08:03.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5558.184706 secs 2024-03-24T02:08:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5558.184706 secs 2024-03-24T02:08:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5558.184706 secs 2024-03-24T02:08:04.202 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6213 sec 2024-03-24T02:08:04.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:04 smithi078 ceph-mon[32429]: pgmap v3418: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:04 smithi107 ceph-mon[36485]: pgmap v3418: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:04 smithi106 ceph-mon[36505]: pgmap v3418: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:05.202 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6214 sec 2024-03-24T02:08:06.202 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6215 sec 2024-03-24T02:08:06.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:06 smithi078 ceph-mon[32429]: pgmap v3419: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:06 smithi107 ceph-mon[36485]: pgmap v3419: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:06 smithi106 ceph-mon[36505]: pgmap v3419: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:07.202 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6216 sec 2024-03-24T02:08:08.202 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6217 sec 2024-03-24T02:08:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:08 smithi078 ceph-mon[32429]: pgmap v3420: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5563.184781 secs 2024-03-24T02:08:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:08 smithi107 ceph-mon[36485]: pgmap v3420: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5563.184781 secs 2024-03-24T02:08:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:08 smithi106 ceph-mon[36505]: pgmap v3420: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:08.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5563.184781 secs 2024-03-24T02:08:09.202 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6218 sec 2024-03-24T02:08:10.202 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6219 sec 2024-03-24T02:08:10.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:10 smithi078 ceph-mon[32429]: pgmap v3421: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:10 smithi107 ceph-mon[36485]: pgmap v3421: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:10 smithi106 ceph-mon[36505]: pgmap v3421: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:11.202 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6220 sec 2024-03-24T02:08:11.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:11 smithi078 ceph-mon[32429]: pgmap v3422: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:11 smithi107 ceph-mon[36485]: pgmap v3422: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:11.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:11 smithi106 ceph-mon[36505]: pgmap v3422: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:12.202 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6221 sec 2024-03-24T02:08:13.202 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6222 sec 2024-03-24T02:08:14.203 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6223 sec 2024-03-24T02:08:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5568.184818 secs 2024-03-24T02:08:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:14 smithi078 ceph-mon[32429]: pgmap v3423: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5568.184818 secs 2024-03-24T02:08:14.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:14 smithi107 ceph-mon[36485]: pgmap v3423: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5568.184818 secs 2024-03-24T02:08:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:14 smithi106 ceph-mon[36505]: pgmap v3423: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:15.202 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6224 sec 2024-03-24T02:08:16.203 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6225 sec 2024-03-24T02:08:16.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:16 smithi078 ceph-mon[32429]: pgmap v3424: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:16.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:16 smithi107 ceph-mon[36485]: pgmap v3424: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:16 smithi106 ceph-mon[36505]: pgmap v3424: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:17.203 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6226 sec 2024-03-24T02:08:18.203 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6227 sec 2024-03-24T02:08:18.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:18 smithi078 ceph-mon[32429]: pgmap v3425: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:18.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:18 smithi107 ceph-mon[36485]: pgmap v3425: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:18.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:18 smithi106 ceph-mon[36505]: pgmap v3425: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:19.203 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6228 sec 2024-03-24T02:08:19.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5573.184877 secs 2024-03-24T02:08:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5573.184877 secs 2024-03-24T02:08:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5573.184877 secs 2024-03-24T02:08:20.203 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6229 sec 2024-03-24T02:08:20.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:20 smithi078 ceph-mon[32429]: pgmap v3426: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:20.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:20 smithi107 ceph-mon[36485]: pgmap v3426: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:20.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:20 smithi106 ceph-mon[36505]: pgmap v3426: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:21.203 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6230 sec 2024-03-24T02:08:22.203 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6231 sec 2024-03-24T02:08:22.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:22 smithi078 ceph-mon[32429]: pgmap v3427: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:08:22.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:22 smithi107 ceph-mon[36485]: pgmap v3427: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:08:22.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:22 smithi106 ceph-mon[36505]: pgmap v3427: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:08:23.203 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6232 sec 2024-03-24T02:08:24.203 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6233 sec 2024-03-24T02:08:24.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5578.184962 secs 2024-03-24T02:08:24.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:24 smithi078 ceph-mon[32429]: pgmap v3428: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:08:24.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5578.184962 secs 2024-03-24T02:08:24.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:24 smithi107 ceph-mon[36485]: pgmap v3428: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:08:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5578.184962 secs 2024-03-24T02:08:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:24 smithi106 ceph-mon[36505]: pgmap v3428: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:08:25.203 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6234 sec 2024-03-24T02:08:26.203 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6235 sec 2024-03-24T02:08:26.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:26 smithi078 ceph-mon[32429]: pgmap v3429: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:08:26.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:26 smithi107 ceph-mon[36485]: pgmap v3429: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:08:26.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:26 smithi106 ceph-mon[36505]: pgmap v3429: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:08:27.203 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6236 sec 2024-03-24T02:08:28.203 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6237 sec 2024-03-24T02:08:28.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:28 smithi078 ceph-mon[32429]: pgmap v3430: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:08:28.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:28 smithi107 ceph-mon[36485]: pgmap v3430: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:08:28.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:28 smithi106 ceph-mon[36505]: pgmap v3430: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:08:29.203 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6238 sec 2024-03-24T02:08:29.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5583.185001 secs 2024-03-24T02:08:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5583.185001 secs 2024-03-24T02:08:29.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5583.185001 secs 2024-03-24T02:08:30.203 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6239 sec 2024-03-24T02:08:30.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:30 smithi078 ceph-mon[32429]: pgmap v3431: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:08:30.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:30 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:08:30.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:30 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:08:30.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:30 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:08:30.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:30 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:08:30.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:30 smithi107 ceph-mon[36485]: pgmap v3431: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:08:30.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:30 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:08:30.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:30 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:08:30.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:30 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:08:30.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:30 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:08:30.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:30 smithi106 ceph-mon[36505]: pgmap v3431: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:08:30.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:30 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:08:30.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:30 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:08:30.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:30 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:08:30.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:30 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:08:31.203 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6240 sec 2024-03-24T02:08:31.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:31 smithi078 ceph-mon[32429]: pgmap v3432: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T02:08:31.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:31 smithi078 ceph-mon[32429]: pgmap v3433: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:08:31.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:31 smithi107 ceph-mon[36485]: pgmap v3432: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T02:08:31.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:31 smithi107 ceph-mon[36485]: pgmap v3433: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:08:31.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:31 smithi106 ceph-mon[36505]: pgmap v3432: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T02:08:31.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:31 smithi106 ceph-mon[36505]: pgmap v3433: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:08:32.204 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6241 sec 2024-03-24T02:08:33.204 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6242 sec 2024-03-24T02:08:33.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:33 smithi078 ceph-mon[32429]: pgmap v3434: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:08:33.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:33 smithi107 ceph-mon[36485]: pgmap v3434: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:08:33.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:33 smithi106 ceph-mon[36505]: pgmap v3434: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:08:34.204 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6243 sec 2024-03-24T02:08:34.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5588.185086 secs 2024-03-24T02:08:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5588.185086 secs 2024-03-24T02:08:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5588.185086 secs 2024-03-24T02:08:35.204 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6244 sec 2024-03-24T02:08:35.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:35 smithi078 ceph-mon[32429]: pgmap v3435: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:08:35.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:35 smithi107 ceph-mon[36485]: pgmap v3435: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:08:35.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:35 smithi106 ceph-mon[36505]: pgmap v3435: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:08:36.204 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6245 sec 2024-03-24T02:08:37.204 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6246 sec 2024-03-24T02:08:37.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:37 smithi078 ceph-mon[32429]: pgmap v3436: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:08:37.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:37 smithi107 ceph-mon[36485]: pgmap v3436: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:08:37.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:37 smithi106 ceph-mon[36505]: pgmap v3436: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:08:38.204 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6247 sec 2024-03-24T02:08:39.204 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6248 sec 2024-03-24T02:08:39.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:39 smithi078 ceph-mon[32429]: pgmap v3437: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:08:39.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5593.185170 secs 2024-03-24T02:08:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:39 smithi107 ceph-mon[36485]: pgmap v3437: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:08:39.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5593.185170 secs 2024-03-24T02:08:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:39 smithi106 ceph-mon[36505]: pgmap v3437: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:08:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5593.185170 secs 2024-03-24T02:08:40.204 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6249 sec 2024-03-24T02:08:41.204 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6250 sec 2024-03-24T02:08:41.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:41 smithi078 ceph-mon[32429]: pgmap v3438: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:08:41.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:41 smithi107 ceph-mon[36485]: pgmap v3438: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:08:41.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:41 smithi106 ceph-mon[36505]: pgmap v3438: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:08:42.205 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6251 sec 2024-03-24T02:08:43.205 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6252 sec 2024-03-24T02:08:43.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:43 smithi078 ceph-mon[32429]: pgmap v3439: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:43.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:43 smithi107 ceph-mon[36485]: pgmap v3439: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:43.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:43 smithi106 ceph-mon[36505]: pgmap v3439: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:44.205 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6253 sec 2024-03-24T02:08:44.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5598.185256 secs 2024-03-24T02:08:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5598.185256 secs 2024-03-24T02:08:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5598.185256 secs 2024-03-24T02:08:45.205 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6254 sec 2024-03-24T02:08:45.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:45 smithi078 ceph-mon[32429]: pgmap v3440: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:45.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:45 smithi107 ceph-mon[36485]: pgmap v3440: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:45.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:45 smithi106 ceph-mon[36505]: pgmap v3440: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:46.205 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6255 sec 2024-03-24T02:08:47.205 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6256 sec 2024-03-24T02:08:47.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:47 smithi078 ceph-mon[32429]: pgmap v3441: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:47 smithi107 ceph-mon[36485]: pgmap v3441: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:47.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:47 smithi106 ceph-mon[36505]: pgmap v3441: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:48.206 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6257 sec 2024-03-24T02:08:49.205 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6258 sec 2024-03-24T02:08:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:49 smithi078 ceph-mon[32429]: pgmap v3442: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5603.185342 secs 2024-03-24T02:08:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:49 smithi107 ceph-mon[36485]: pgmap v3442: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5603.185342 secs 2024-03-24T02:08:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:49 smithi106 ceph-mon[36505]: pgmap v3442: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5603.185342 secs 2024-03-24T02:08:50.205 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6259 sec 2024-03-24T02:08:51.206 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6260 sec 2024-03-24T02:08:51.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:51 smithi078 ceph-mon[32429]: pgmap v3443: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:51.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:51 smithi107 ceph-mon[36485]: pgmap v3443: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:51.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:51 smithi106 ceph-mon[36505]: pgmap v3443: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:52.206 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6261 sec 2024-03-24T02:08:53.206 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6262 sec 2024-03-24T02:08:53.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:53 smithi078 ceph-mon[32429]: pgmap v3444: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:53.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:53 smithi107 ceph-mon[36485]: pgmap v3444: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:53.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:53 smithi106 ceph-mon[36505]: pgmap v3444: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:54.206 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6263 sec 2024-03-24T02:08:54.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5608.185429 secs 2024-03-24T02:08:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5608.185429 secs 2024-03-24T02:08:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5608.185429 secs 2024-03-24T02:08:55.206 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6264 sec 2024-03-24T02:08:55.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:55 smithi078 ceph-mon[32429]: pgmap v3445: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:55.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:55 smithi107 ceph-mon[36485]: pgmap v3445: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:55.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:55 smithi106 ceph-mon[36505]: pgmap v3445: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:56.206 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6265 sec 2024-03-24T02:08:57.206 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6266 sec 2024-03-24T02:08:57.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:57 smithi078 ceph-mon[32429]: pgmap v3446: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:57.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:57 smithi107 ceph-mon[36485]: pgmap v3446: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:57.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:57 smithi106 ceph-mon[36505]: pgmap v3446: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:58.206 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6267 sec 2024-03-24T02:08:59.206 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6268 sec 2024-03-24T02:08:59.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:59 smithi078 ceph-mon[32429]: pgmap v3447: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:59.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:08:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5613.185514 secs 2024-03-24T02:08:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:59 smithi107 ceph-mon[36485]: pgmap v3447: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:59.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:08:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5613.185514 secs 2024-03-24T02:08:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:59 smithi106 ceph-mon[36505]: pgmap v3447: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:08:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:08:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5613.185514 secs 2024-03-24T02:09:00.206 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6269 sec 2024-03-24T02:09:01.206 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6270 sec 2024-03-24T02:09:01.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:01 smithi078 ceph-mon[32429]: pgmap v3448: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:01.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:01 smithi107 ceph-mon[36485]: pgmap v3448: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:01.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:01 smithi106 ceph-mon[36505]: pgmap v3448: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:02.206 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6271 sec 2024-03-24T02:09:03.206 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6272 sec 2024-03-24T02:09:03.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:03 smithi078 ceph-mon[32429]: pgmap v3449: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:03.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:03 smithi107 ceph-mon[36485]: pgmap v3449: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:03.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:03 smithi106 ceph-mon[36505]: pgmap v3449: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:04.207 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6273 sec 2024-03-24T02:09:04.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5618.185623 secs 2024-03-24T02:09:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5618.185623 secs 2024-03-24T02:09:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5618.185623 secs 2024-03-24T02:09:05.207 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6274 sec 2024-03-24T02:09:05.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:05 smithi078 ceph-mon[32429]: pgmap v3450: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:05.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:05 smithi107 ceph-mon[36485]: pgmap v3450: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:05.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:05 smithi106 ceph-mon[36505]: pgmap v3450: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:06.207 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6275 sec 2024-03-24T02:09:07.207 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6276 sec 2024-03-24T02:09:07.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:07 smithi078 ceph-mon[32429]: pgmap v3451: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:07.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:07 smithi107 ceph-mon[36485]: pgmap v3451: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:07.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:07 smithi106 ceph-mon[36505]: pgmap v3451: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:08.207 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6277 sec 2024-03-24T02:09:09.207 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6278 sec 2024-03-24T02:09:09.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:09 smithi078 ceph-mon[32429]: pgmap v3452: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:09.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5623.185682 secs 2024-03-24T02:09:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:09 smithi107 ceph-mon[36485]: pgmap v3452: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5623.185682 secs 2024-03-24T02:09:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:09 smithi106 ceph-mon[36505]: pgmap v3452: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5623.185682 secs 2024-03-24T02:09:10.207 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6279 sec 2024-03-24T02:09:11.207 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6280 sec 2024-03-24T02:09:11.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:11 smithi078 ceph-mon[32429]: pgmap v3453: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:11.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:11 smithi107 ceph-mon[36485]: pgmap v3453: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:11.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:11 smithi106 ceph-mon[36505]: pgmap v3453: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:12.207 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6281 sec 2024-03-24T02:09:13.208 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6282 sec 2024-03-24T02:09:13.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:13 smithi078 ceph-mon[32429]: pgmap v3454: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:13.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:13 smithi107 ceph-mon[36485]: pgmap v3454: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:13.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:13 smithi106 ceph-mon[36505]: pgmap v3454: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:14.208 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6283 sec 2024-03-24T02:09:14.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5628.185769 secs 2024-03-24T02:09:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5628.185769 secs 2024-03-24T02:09:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5628.185769 secs 2024-03-24T02:09:15.208 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6284 sec 2024-03-24T02:09:15.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:15 smithi078 ceph-mon[32429]: pgmap v3455: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:15.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:15 smithi107 ceph-mon[36485]: pgmap v3455: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:15.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:15 smithi106 ceph-mon[36505]: pgmap v3455: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:16.208 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6285 sec 2024-03-24T02:09:17.208 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6286 sec 2024-03-24T02:09:17.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:17 smithi078 ceph-mon[32429]: pgmap v3456: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:17.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:17 smithi107 ceph-mon[36485]: pgmap v3456: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:17.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:17 smithi106 ceph-mon[36505]: pgmap v3456: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:18.208 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6287 sec 2024-03-24T02:09:19.208 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6288 sec 2024-03-24T02:09:19.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:19 smithi078 ceph-mon[32429]: pgmap v3457: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:19.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5633.185828 secs 2024-03-24T02:09:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:19 smithi107 ceph-mon[36485]: pgmap v3457: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5633.185828 secs 2024-03-24T02:09:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:19 smithi106 ceph-mon[36505]: pgmap v3457: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5633.185828 secs 2024-03-24T02:09:20.208 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6289 sec 2024-03-24T02:09:21.208 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6290 sec 2024-03-24T02:09:21.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:21 smithi078 ceph-mon[32429]: pgmap v3458: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:21.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:21 smithi107 ceph-mon[36485]: pgmap v3458: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:21.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:21 smithi106 ceph-mon[36505]: pgmap v3458: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:22.208 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6291 sec 2024-03-24T02:09:23.208 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6292 sec 2024-03-24T02:09:23.580 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:23 smithi078 ceph-mon[32429]: pgmap v3459: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:23.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:23 smithi107 ceph-mon[36485]: pgmap v3459: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:23.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:23 smithi106 ceph-mon[36505]: pgmap v3459: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:24.208 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6293 sec 2024-03-24T02:09:24.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5638.185886 secs 2024-03-24T02:09:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5638.185886 secs 2024-03-24T02:09:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5638.185886 secs 2024-03-24T02:09:25.208 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6294 sec 2024-03-24T02:09:25.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:25 smithi078 ceph-mon[32429]: pgmap v3460: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:25.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:25 smithi107 ceph-mon[36485]: pgmap v3460: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:25.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:25 smithi106 ceph-mon[36505]: pgmap v3460: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:26.209 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6295 sec 2024-03-24T02:09:27.209 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6296 sec 2024-03-24T02:09:27.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:27 smithi078 ceph-mon[32429]: pgmap v3461: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:27.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:27 smithi107 ceph-mon[36485]: pgmap v3461: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:27.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:27 smithi106 ceph-mon[36505]: pgmap v3461: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:28.209 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6297 sec 2024-03-24T02:09:29.209 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6298 sec 2024-03-24T02:09:29.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:29 smithi078 ceph-mon[32429]: pgmap v3462: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:29.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5643.185940 secs 2024-03-24T02:09:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:29 smithi107 ceph-mon[36485]: pgmap v3462: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5643.185940 secs 2024-03-24T02:09:29.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:29 smithi106 ceph-mon[36505]: pgmap v3462: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:29.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5643.185940 secs 2024-03-24T02:09:30.209 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6299 sec 2024-03-24T02:09:30.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:30 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:09:30.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:30 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:09:30.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:30 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:09:31.209 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6300 sec 2024-03-24T02:09:31.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:31 smithi106 ceph-mon[36505]: pgmap v3463: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:31.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:31 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:09:31.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:31 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:09:31.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:31 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:09:31.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:31 smithi078 ceph-mon[32429]: pgmap v3463: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:31.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:31 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:09:31.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:31 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:09:31.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:31 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:09:31.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:31 smithi107 ceph-mon[36485]: pgmap v3463: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:31.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:31 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:09:31.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:31 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:09:31.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:31 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:09:32.209 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6301 sec 2024-03-24T02:09:32.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:32 smithi106 ceph-mon[36505]: pgmap v3464: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:09:32.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:32 smithi106 ceph-mon[36505]: pgmap v3465: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:09:32.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:32 smithi078 ceph-mon[32429]: pgmap v3464: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:09:32.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:32 smithi078 ceph-mon[32429]: pgmap v3465: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:09:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:32 smithi107 ceph-mon[36485]: pgmap v3464: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:09:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:32 smithi107 ceph-mon[36485]: pgmap v3465: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:09:33.209 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6302 sec 2024-03-24T02:09:34.209 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6303 sec 2024-03-24T02:09:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:34 smithi106 ceph-mon[36505]: pgmap v3466: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:09:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5648.186003 secs 2024-03-24T02:09:34.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:34 smithi078 ceph-mon[32429]: pgmap v3466: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:09:34.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5648.186003 secs 2024-03-24T02:09:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:34 smithi107 ceph-mon[36485]: pgmap v3466: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:09:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5648.186003 secs 2024-03-24T02:09:35.209 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6304 sec 2024-03-24T02:09:36.209 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6305 sec 2024-03-24T02:09:36.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:36 smithi106 ceph-mon[36505]: pgmap v3467: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:09:36.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:36 smithi078 ceph-mon[32429]: pgmap v3467: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:09:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:36 smithi107 ceph-mon[36485]: pgmap v3467: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:09:37.210 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6306 sec 2024-03-24T02:09:38.210 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6307 sec 2024-03-24T02:09:38.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:38 smithi106 ceph-mon[36505]: pgmap v3468: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:09:38.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:38 smithi078 ceph-mon[32429]: pgmap v3468: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:09:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:38 smithi107 ceph-mon[36485]: pgmap v3468: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:09:39.210 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6308 sec 2024-03-24T02:09:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5653.186082 secs 2024-03-24T02:09:39.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5653.186082 secs 2024-03-24T02:09:39.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5653.186082 secs 2024-03-24T02:09:40.210 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6309 sec 2024-03-24T02:09:40.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:40 smithi106 ceph-mon[36505]: pgmap v3469: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:09:40.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:40 smithi078 ceph-mon[32429]: pgmap v3469: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:09:40.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:40 smithi107 ceph-mon[36485]: pgmap v3469: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:09:41.210 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6310 sec 2024-03-24T02:09:42.210 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6311 sec 2024-03-24T02:09:42.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:42 smithi078 ceph-mon[32429]: pgmap v3470: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:09:42.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:42 smithi106 ceph-mon[36505]: pgmap v3470: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:09:42.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:42 smithi107 ceph-mon[36485]: pgmap v3470: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:09:43.210 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6312 sec 2024-03-24T02:09:44.210 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6313 sec 2024-03-24T02:09:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:44 smithi106 ceph-mon[36505]: pgmap v3471: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:44.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5658.186167 secs 2024-03-24T02:09:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:44 smithi078 ceph-mon[32429]: pgmap v3471: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:44.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5658.186167 secs 2024-03-24T02:09:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:44 smithi107 ceph-mon[36485]: pgmap v3471: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5658.186167 secs 2024-03-24T02:09:45.210 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6314 sec 2024-03-24T02:09:46.210 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6315 sec 2024-03-24T02:09:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:46 smithi106 ceph-mon[36505]: pgmap v3472: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:46.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:46 smithi078 ceph-mon[32429]: pgmap v3472: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:46.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:46 smithi107 ceph-mon[36485]: pgmap v3472: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:47.210 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6316 sec 2024-03-24T02:09:48.211 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6317 sec 2024-03-24T02:09:48.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:48 smithi106 ceph-mon[36505]: pgmap v3473: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:48.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:48 smithi078 ceph-mon[32429]: pgmap v3473: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:48 smithi107 ceph-mon[36485]: pgmap v3473: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:49.211 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6318 sec 2024-03-24T02:09:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5663.186219 secs 2024-03-24T02:09:49.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5663.186219 secs 2024-03-24T02:09:49.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5663.186219 secs 2024-03-24T02:09:50.211 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6319 sec 2024-03-24T02:09:50.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:50 smithi106 ceph-mon[36505]: pgmap v3474: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:50.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:50 smithi078 ceph-mon[32429]: pgmap v3474: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:50 smithi107 ceph-mon[36485]: pgmap v3474: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:51.211 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6320 sec 2024-03-24T02:09:52.211 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6321 sec 2024-03-24T02:09:52.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:52 smithi106 ceph-mon[36505]: pgmap v3475: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:52.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:52 smithi078 ceph-mon[32429]: pgmap v3475: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:52.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:52 smithi107 ceph-mon[36485]: pgmap v3475: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:53.211 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6322 sec 2024-03-24T02:09:54.211 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6323 sec 2024-03-24T02:09:54.646 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:54 smithi107 ceph-mon[36485]: pgmap v3476: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:54.646 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5668.186279 secs 2024-03-24T02:09:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:54 smithi106 ceph-mon[36505]: pgmap v3476: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5668.186279 secs 2024-03-24T02:09:54.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:54 smithi078 ceph-mon[32429]: pgmap v3476: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:54.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5668.186279 secs 2024-03-24T02:09:55.211 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6324 sec 2024-03-24T02:09:56.211 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6325 sec 2024-03-24T02:09:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:56 smithi106 ceph-mon[36505]: pgmap v3477: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:56 smithi078 ceph-mon[32429]: pgmap v3477: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:56 smithi107 ceph-mon[36485]: pgmap v3477: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:09:57.211 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6326 sec 2024-03-24T02:09:58.211 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6327 sec 2024-03-24T02:09:58.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:58 smithi106 ceph-mon[36505]: pgmap v3478: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:58.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:58 smithi078 ceph-mon[32429]: pgmap v3478: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:58 smithi107 ceph-mon[36485]: pgmap v3478: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:59.211 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6328 sec 2024-03-24T02:09:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:09:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5673.186358 secs 2024-03-24T02:09:59.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:09:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5673.186358 secs 2024-03-24T02:09:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:09:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5673.186358 secs 2024-03-24T02:10:00.211 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6329 sec 2024-03-24T02:10:00.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:00 smithi106 ceph-mon[36505]: pgmap v3479: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:00.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:00 smithi106 ceph-mon[36505]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:10:00.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:00 smithi078 ceph-mon[32429]: pgmap v3479: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:00.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:00 smithi078 ceph-mon[32429]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:10:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:00 smithi107 ceph-mon[36485]: pgmap v3479: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:00 smithi107 ceph-mon[36485]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:10:01.212 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6330 sec 2024-03-24T02:10:02.212 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6331 sec 2024-03-24T02:10:02.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:02 smithi078 ceph-mon[32429]: pgmap v3480: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:02 smithi107 ceph-mon[36485]: pgmap v3480: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:02.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:02 smithi106 ceph-mon[36505]: pgmap v3480: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:03.212 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6332 sec 2024-03-24T02:10:04.212 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6333 sec 2024-03-24T02:10:04.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:04 smithi078 ceph-mon[32429]: pgmap v3481: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:04.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5678.186418 secs 2024-03-24T02:10:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:04 smithi107 ceph-mon[36485]: pgmap v3481: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5678.186418 secs 2024-03-24T02:10:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:04 smithi106 ceph-mon[36505]: pgmap v3481: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5678.186418 secs 2024-03-24T02:10:05.212 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6334 sec 2024-03-24T02:10:06.213 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6335 sec 2024-03-24T02:10:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:06 smithi078 ceph-mon[32429]: pgmap v3482: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:06 smithi107 ceph-mon[36485]: pgmap v3482: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:06 smithi106 ceph-mon[36505]: pgmap v3482: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:07.213 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6336 sec 2024-03-24T02:10:08.213 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6337 sec 2024-03-24T02:10:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:08 smithi078 ceph-mon[32429]: pgmap v3483: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:08 smithi107 ceph-mon[36485]: pgmap v3483: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:08 smithi106 ceph-mon[36505]: pgmap v3483: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:09.213 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6338 sec 2024-03-24T02:10:09.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5683.186502 secs 2024-03-24T02:10:09.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5683.186502 secs 2024-03-24T02:10:09.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5683.186502 secs 2024-03-24T02:10:10.213 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6339 sec 2024-03-24T02:10:10.684 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:10 smithi078 ceph-mon[32429]: pgmap v3484: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:10 smithi107 ceph-mon[36485]: pgmap v3484: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:10 smithi106 ceph-mon[36505]: pgmap v3484: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:11.213 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6340 sec 2024-03-24T02:10:12.213 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6341 sec 2024-03-24T02:10:12.704 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:12 smithi106 ceph-mon[36505]: pgmap v3485: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:12 smithi078 ceph-mon[32429]: pgmap v3485: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:12 smithi107 ceph-mon[36485]: pgmap v3485: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:13.213 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6342 sec 2024-03-24T02:10:14.214 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6343 sec 2024-03-24T02:10:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:14 smithi078 ceph-mon[32429]: pgmap v3486: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5688.186590 secs 2024-03-24T02:10:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:14 smithi107 ceph-mon[36485]: pgmap v3486: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5688.186590 secs 2024-03-24T02:10:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:14 smithi106 ceph-mon[36505]: pgmap v3486: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5688.186590 secs 2024-03-24T02:10:15.213 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6344 sec 2024-03-24T02:10:16.214 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6345 sec 2024-03-24T02:10:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:16 smithi078 ceph-mon[32429]: pgmap v3487: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:16 smithi107 ceph-mon[36485]: pgmap v3487: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:16.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:16 smithi106 ceph-mon[36505]: pgmap v3487: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:17.213 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6346 sec 2024-03-24T02:10:18.214 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6347 sec 2024-03-24T02:10:18.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:18 smithi078 ceph-mon[32429]: pgmap v3488: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:18 smithi107 ceph-mon[36485]: pgmap v3488: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:18 smithi106 ceph-mon[36505]: pgmap v3488: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:19.214 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6348 sec 2024-03-24T02:10:19.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5693.186652 secs 2024-03-24T02:10:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5693.186652 secs 2024-03-24T02:10:19.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5693.186652 secs 2024-03-24T02:10:20.214 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6349 sec 2024-03-24T02:10:20.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:20 smithi078 ceph-mon[32429]: pgmap v3489: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:20 smithi107 ceph-mon[36485]: pgmap v3489: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:20.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:20 smithi106 ceph-mon[36505]: pgmap v3489: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:21.214 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6350 sec 2024-03-24T02:10:22.214 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6351 sec 2024-03-24T02:10:22.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:22 smithi078 ceph-mon[32429]: pgmap v3490: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:22.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:22 smithi107 ceph-mon[36485]: pgmap v3490: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:22.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:22 smithi106 ceph-mon[36505]: pgmap v3490: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:23.214 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6352 sec 2024-03-24T02:10:24.214 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6353 sec 2024-03-24T02:10:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:24 smithi078 ceph-mon[32429]: pgmap v3491: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:24.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5698.186739 secs 2024-03-24T02:10:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:24 smithi107 ceph-mon[36485]: pgmap v3491: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5698.186739 secs 2024-03-24T02:10:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:24 smithi106 ceph-mon[36505]: pgmap v3491: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5698.186739 secs 2024-03-24T02:10:25.214 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6354 sec 2024-03-24T02:10:26.214 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6355 sec 2024-03-24T02:10:26.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:26 smithi078 ceph-mon[32429]: pgmap v3492: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:26 smithi107 ceph-mon[36485]: pgmap v3492: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:26.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:26 smithi106 ceph-mon[36505]: pgmap v3492: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:27.214 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6356 sec 2024-03-24T02:10:28.214 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6357 sec 2024-03-24T02:10:28.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:28 smithi078 ceph-mon[32429]: pgmap v3493: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:28 smithi107 ceph-mon[36485]: pgmap v3493: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:28 smithi106 ceph-mon[36505]: pgmap v3493: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:29.215 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6358 sec 2024-03-24T02:10:29.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5703.186826 secs 2024-03-24T02:10:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5703.186826 secs 2024-03-24T02:10:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5703.186826 secs 2024-03-24T02:10:30.215 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6359 sec 2024-03-24T02:10:30.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:30 smithi078 ceph-mon[32429]: pgmap v3494: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:30 smithi107 ceph-mon[36485]: pgmap v3494: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:30 smithi106 ceph-mon[36505]: pgmap v3494: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:31.215 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6360 sec 2024-03-24T02:10:31.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:31 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:10:31.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:31 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:10:31.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:31 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:10:31.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:31 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:10:31.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:31 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:10:31.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:31 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:10:31.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:31 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:10:31.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:31 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:10:31.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:31 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:10:31.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:31 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:10:31.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:31 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:10:31.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:31 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:10:32.215 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6361 sec 2024-03-24T02:10:32.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:32 smithi078 ceph-mon[32429]: pgmap v3495: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:32.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:32 smithi078 ceph-mon[32429]: pgmap v3496: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:10:32.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:32 smithi078 ceph-mon[32429]: pgmap v3497: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:10:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:32 smithi107 ceph-mon[36485]: pgmap v3495: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:32 smithi107 ceph-mon[36485]: pgmap v3496: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:10:32.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:32 smithi107 ceph-mon[36485]: pgmap v3497: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:10:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:32 smithi106 ceph-mon[36505]: pgmap v3495: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:32 smithi106 ceph-mon[36505]: pgmap v3496: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:10:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:32 smithi106 ceph-mon[36505]: pgmap v3497: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:10:33.215 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6362 sec 2024-03-24T02:10:34.215 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6363 sec 2024-03-24T02:10:34.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5708.186903 secs 2024-03-24T02:10:34.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:34 smithi078 ceph-mon[32429]: pgmap v3498: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:10:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5708.186903 secs 2024-03-24T02:10:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:34 smithi107 ceph-mon[36485]: pgmap v3498: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:10:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5708.186903 secs 2024-03-24T02:10:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:34 smithi106 ceph-mon[36505]: pgmap v3498: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:10:35.215 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6364 sec 2024-03-24T02:10:36.215 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6365 sec 2024-03-24T02:10:36.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:36 smithi078 ceph-mon[32429]: pgmap v3499: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:10:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:36 smithi107 ceph-mon[36485]: pgmap v3499: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:10:36.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:36 smithi106 ceph-mon[36505]: pgmap v3499: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:10:37.215 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6366 sec 2024-03-24T02:10:38.215 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6367 sec 2024-03-24T02:10:38.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:38 smithi078 ceph-mon[32429]: pgmap v3500: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:10:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:38 smithi107 ceph-mon[36485]: pgmap v3500: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:10:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:38 smithi106 ceph-mon[36505]: pgmap v3500: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:10:39.215 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6368 sec 2024-03-24T02:10:39.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5713.186940 secs 2024-03-24T02:10:39.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5713.186940 secs 2024-03-24T02:10:39.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5713.186940 secs 2024-03-24T02:10:40.215 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6369 sec 2024-03-24T02:10:40.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:40 smithi078 ceph-mon[32429]: pgmap v3501: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:10:40.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:40 smithi107 ceph-mon[36485]: pgmap v3501: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:10:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:40 smithi106 ceph-mon[36505]: pgmap v3501: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:10:41.215 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6370 sec 2024-03-24T02:10:42.216 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6371 sec 2024-03-24T02:10:42.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:42 smithi078 ceph-mon[32429]: pgmap v3502: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:10:42.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:42 smithi107 ceph-mon[36485]: pgmap v3502: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:10:42.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:42 smithi106 ceph-mon[36505]: pgmap v3502: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:10:43.216 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6372 sec 2024-03-24T02:10:43.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5718.187024 secs 2024-03-24T02:10:43.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:43 smithi078 ceph-mon[32429]: pgmap v3503: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5718.187024 secs 2024-03-24T02:10:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:43 smithi107 ceph-mon[36485]: pgmap v3503: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5718.187024 secs 2024-03-24T02:10:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:43 smithi106 ceph-mon[36505]: pgmap v3503: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:44.216 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6373 sec 2024-03-24T02:10:45.216 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6374 sec 2024-03-24T02:10:46.216 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6375 sec 2024-03-24T02:10:46.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:46 smithi078 ceph-mon[32429]: pgmap v3504: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:46.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:46 smithi107 ceph-mon[36485]: pgmap v3504: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:46 smithi106 ceph-mon[36505]: pgmap v3504: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:47.216 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6376 sec 2024-03-24T02:10:48.216 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6377 sec 2024-03-24T02:10:48.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:48 smithi078 ceph-mon[32429]: pgmap v3505: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:48.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5723.187112 secs 2024-03-24T02:10:48.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:48 smithi107 ceph-mon[36485]: pgmap v3505: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:48.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5723.187112 secs 2024-03-24T02:10:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:48 smithi106 ceph-mon[36505]: pgmap v3505: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:48.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5723.187112 secs 2024-03-24T02:10:49.216 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6378 sec 2024-03-24T02:10:50.216 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6379 sec 2024-03-24T02:10:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:50 smithi078 ceph-mon[32429]: pgmap v3506: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:50 smithi107 ceph-mon[36485]: pgmap v3506: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:50.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:50 smithi106 ceph-mon[36505]: pgmap v3506: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:51.216 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6380 sec 2024-03-24T02:10:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:51 smithi106 ceph-mon[36505]: pgmap v3507: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:52.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:51 smithi078 ceph-mon[32429]: pgmap v3507: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:52.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:51 smithi107 ceph-mon[36485]: pgmap v3507: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:52.216 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6381 sec 2024-03-24T02:10:53.217 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6382 sec 2024-03-24T02:10:53.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5728.187196 secs 2024-03-24T02:10:53.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:53 smithi078 ceph-mon[32429]: pgmap v3508: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5728.187196 secs 2024-03-24T02:10:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:53 smithi107 ceph-mon[36485]: pgmap v3508: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:53.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5728.187196 secs 2024-03-24T02:10:53.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:53 smithi106 ceph-mon[36505]: pgmap v3508: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:54.217 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6383 sec 2024-03-24T02:10:55.217 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6384 sec 2024-03-24T02:10:56.217 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6385 sec 2024-03-24T02:10:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:56 smithi078 ceph-mon[32429]: pgmap v3509: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:56 smithi107 ceph-mon[36485]: pgmap v3509: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:56.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:56 smithi106 ceph-mon[36505]: pgmap v3509: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:57.217 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6386 sec 2024-03-24T02:10:58.217 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6387 sec 2024-03-24T02:10:58.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:58 smithi078 ceph-mon[32429]: pgmap v3510: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:58.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:10:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5733.187257 secs 2024-03-24T02:10:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:58 smithi107 ceph-mon[36485]: pgmap v3510: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:10:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5733.187257 secs 2024-03-24T02:10:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:58 smithi106 ceph-mon[36505]: pgmap v3510: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:10:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:10:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5733.187257 secs 2024-03-24T02:10:59.217 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6388 sec 2024-03-24T02:11:00.217 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6389 sec 2024-03-24T02:11:00.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:00 smithi078 ceph-mon[32429]: pgmap v3511: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:00 smithi107 ceph-mon[36485]: pgmap v3511: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:00.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:00 smithi106 ceph-mon[36505]: pgmap v3511: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:01.217 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6390 sec 2024-03-24T02:11:02.217 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6391 sec 2024-03-24T02:11:02.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:02 smithi078 ceph-mon[32429]: pgmap v3512: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:02 smithi107 ceph-mon[36485]: pgmap v3512: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:02.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:02 smithi106 ceph-mon[36505]: pgmap v3512: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:03.217 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6392 sec 2024-03-24T02:11:03.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5738.187295 secs 2024-03-24T02:11:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5738.187295 secs 2024-03-24T02:11:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5738.187295 secs 2024-03-24T02:11:04.218 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6393 sec 2024-03-24T02:11:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:04 smithi078 ceph-mon[32429]: pgmap v3513: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:04 smithi107 ceph-mon[36485]: pgmap v3513: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:04 smithi106 ceph-mon[36505]: pgmap v3513: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:05.218 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6394 sec 2024-03-24T02:11:05.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:05 smithi078 ceph-mon[32429]: pgmap v3514: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:05.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:05 smithi107 ceph-mon[36485]: pgmap v3514: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:05.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:05 smithi106 ceph-mon[36505]: pgmap v3514: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:06.218 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6395 sec 2024-03-24T02:11:07.218 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6396 sec 2024-03-24T02:11:08.219 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6397 sec 2024-03-24T02:11:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:08 smithi078 ceph-mon[32429]: pgmap v3515: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5743.187381 secs 2024-03-24T02:11:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:08 smithi107 ceph-mon[36485]: pgmap v3515: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5743.187381 secs 2024-03-24T02:11:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:08 smithi106 ceph-mon[36505]: pgmap v3515: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5743.187381 secs 2024-03-24T02:11:09.218 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6398 sec 2024-03-24T02:11:10.218 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6399 sec 2024-03-24T02:11:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:10 smithi078 ceph-mon[32429]: pgmap v3516: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:10 smithi107 ceph-mon[36485]: pgmap v3516: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:10 smithi106 ceph-mon[36505]: pgmap v3516: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:11.218 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6400 sec 2024-03-24T02:11:12.218 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6401 sec 2024-03-24T02:11:12.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:12 smithi078 ceph-mon[32429]: pgmap v3517: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:12 smithi107 ceph-mon[36485]: pgmap v3517: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:12.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:12 smithi106 ceph-mon[36505]: pgmap v3517: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:13.218 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6402 sec 2024-03-24T02:11:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5748.187464 secs 2024-03-24T02:11:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5748.187464 secs 2024-03-24T02:11:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5748.187464 secs 2024-03-24T02:11:14.218 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6403 sec 2024-03-24T02:11:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:14 smithi078 ceph-mon[32429]: pgmap v3518: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:14 smithi107 ceph-mon[36485]: pgmap v3518: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:14 smithi106 ceph-mon[36505]: pgmap v3518: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:15.218 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6404 sec 2024-03-24T02:11:15.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:15 smithi078 ceph-mon[32429]: pgmap v3519: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:15 smithi107 ceph-mon[36485]: pgmap v3519: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:15 smithi106 ceph-mon[36505]: pgmap v3519: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:16.219 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6405 sec 2024-03-24T02:11:17.219 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6406 sec 2024-03-24T02:11:18.219 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6407 sec 2024-03-24T02:11:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:18 smithi078 ceph-mon[32429]: pgmap v3520: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:18.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5753.187524 secs 2024-03-24T02:11:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:18 smithi107 ceph-mon[36485]: pgmap v3520: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5753.187524 secs 2024-03-24T02:11:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:18 smithi106 ceph-mon[36505]: pgmap v3520: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5753.187524 secs 2024-03-24T02:11:19.219 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6408 sec 2024-03-24T02:11:20.219 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6409 sec 2024-03-24T02:11:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:20 smithi078 ceph-mon[32429]: pgmap v3521: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:20 smithi107 ceph-mon[36485]: pgmap v3521: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:20.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:20 smithi106 ceph-mon[36505]: pgmap v3521: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:21.219 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6410 sec 2024-03-24T02:11:22.221 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6411 sec 2024-03-24T02:11:22.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:22 smithi078 ceph-mon[32429]: pgmap v3522: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:22.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:22 smithi107 ceph-mon[36485]: pgmap v3522: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:22.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:22 smithi106 ceph-mon[36505]: pgmap v3522: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:23.219 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6412 sec 2024-03-24T02:11:23.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5758.187613 secs 2024-03-24T02:11:23.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5758.187613 secs 2024-03-24T02:11:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5758.187613 secs 2024-03-24T02:11:24.219 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6413 sec 2024-03-24T02:11:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:24 smithi078 ceph-mon[32429]: pgmap v3523: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:24 smithi107 ceph-mon[36485]: pgmap v3523: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:24 smithi106 ceph-mon[36505]: pgmap v3523: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:25.219 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6414 sec 2024-03-24T02:11:25.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:25 smithi078 ceph-mon[32429]: pgmap v3524: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:25.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:25 smithi107 ceph-mon[36485]: pgmap v3524: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:25.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:25 smithi106 ceph-mon[36505]: pgmap v3524: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:26.219 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6415 sec 2024-03-24T02:11:27.220 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6416 sec 2024-03-24T02:11:28.220 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6417 sec 2024-03-24T02:11:28.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:28 smithi078 ceph-mon[32429]: pgmap v3525: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:28.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:28 smithi078 ceph-mon[32429]: 2 slow requests, 1 included below; oldest blocked for > 5763.187672 secs 2024-03-24T02:11:28.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:28 smithi078 ceph-mon[32429]: slow request 5763.187671 seconds old, received at 2024-03-24T00:35:24.775244+0000: peer_request:client.15462:3775 currently acquired locks 2024-03-24T02:11:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:28 smithi107 ceph-mon[36485]: pgmap v3525: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:28 smithi107 ceph-mon[36485]: 2 slow requests, 1 included below; oldest blocked for > 5763.187672 secs 2024-03-24T02:11:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:28 smithi107 ceph-mon[36485]: slow request 5763.187671 seconds old, received at 2024-03-24T00:35:24.775244+0000: peer_request:client.15462:3775 currently acquired locks 2024-03-24T02:11:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:28 smithi106 ceph-mon[36505]: pgmap v3525: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:28 smithi106 ceph-mon[36505]: 2 slow requests, 1 included below; oldest blocked for > 5763.187672 secs 2024-03-24T02:11:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:28 smithi106 ceph-mon[36505]: slow request 5763.187671 seconds old, received at 2024-03-24T00:35:24.775244+0000: peer_request:client.15462:3775 currently acquired locks 2024-03-24T02:11:29.220 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6418 sec 2024-03-24T02:11:30.220 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6419 sec 2024-03-24T02:11:30.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:30 smithi078 ceph-mon[32429]: pgmap v3526: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:30 smithi107 ceph-mon[36485]: pgmap v3526: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:30 smithi106 ceph-mon[36505]: pgmap v3526: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:31.220 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6420 sec 2024-03-24T02:11:32.220 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6421 sec 2024-03-24T02:11:32.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:32 smithi078 ceph-mon[32429]: pgmap v3527: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:32.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:32 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:11:32.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:32 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:11:32.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:32 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:11:32.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:32 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:11:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:32 smithi107 ceph-mon[36485]: pgmap v3527: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:32 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:11:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:32 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:11:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:32 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:11:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:32 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:11:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:32 smithi106 ceph-mon[36505]: pgmap v3527: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:32 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:11:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:32 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:11:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:32 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:11:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:32 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:11:33.220 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6422 sec 2024-03-24T02:11:33.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:33 smithi078 ceph-mon[32429]: pgmap v3528: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:11:33.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:33 smithi078 ceph-mon[32429]: pgmap v3529: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:11:33.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5768.187761 secs 2024-03-24T02:11:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:33 smithi107 ceph-mon[36485]: pgmap v3528: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:11:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:33 smithi107 ceph-mon[36485]: pgmap v3529: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:11:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5768.187761 secs 2024-03-24T02:11:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:33 smithi106 ceph-mon[36505]: pgmap v3528: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:11:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:33 smithi106 ceph-mon[36505]: pgmap v3529: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:11:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5768.187761 secs 2024-03-24T02:11:34.220 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6423 sec 2024-03-24T02:11:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:34 smithi078 ceph-mon[32429]: pgmap v3530: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:11:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:34 smithi107 ceph-mon[36485]: pgmap v3530: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:11:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:34 smithi106 ceph-mon[36505]: pgmap v3530: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:11:35.220 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6424 sec 2024-03-24T02:11:36.220 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6425 sec 2024-03-24T02:11:37.220 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6426 sec 2024-03-24T02:11:37.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:37 smithi078 ceph-mon[32429]: pgmap v3531: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:11:37.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:37 smithi107 ceph-mon[36485]: pgmap v3531: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:11:37.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:37 smithi106 ceph-mon[36505]: pgmap v3531: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:11:38.220 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6427 sec 2024-03-24T02:11:39.220 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6428 sec 2024-03-24T02:11:39.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5773.187823 secs 2024-03-24T02:11:39.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:39 smithi078 ceph-mon[32429]: pgmap v3532: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:11:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5773.187823 secs 2024-03-24T02:11:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:39 smithi107 ceph-mon[36485]: pgmap v3532: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:11:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5773.187823 secs 2024-03-24T02:11:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:39 smithi106 ceph-mon[36505]: pgmap v3532: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:11:40.220 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6429 sec 2024-03-24T02:11:41.221 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6430 sec 2024-03-24T02:11:41.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:41 smithi078 ceph-mon[32429]: pgmap v3533: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:11:41.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:41 smithi107 ceph-mon[36485]: pgmap v3533: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:11:41.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:41 smithi106 ceph-mon[36505]: pgmap v3533: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:11:42.221 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6431 sec 2024-03-24T02:11:43.221 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6432 sec 2024-03-24T02:11:43.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:43 smithi078 ceph-mon[32429]: pgmap v3534: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:11:43.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:43 smithi107 ceph-mon[36485]: pgmap v3534: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:11:43.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:43 smithi106 ceph-mon[36505]: pgmap v3534: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:11:44.221 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6433 sec 2024-03-24T02:11:44.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5778.187911 secs 2024-03-24T02:11:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5778.187911 secs 2024-03-24T02:11:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5778.187911 secs 2024-03-24T02:11:45.221 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6434 sec 2024-03-24T02:11:45.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:45 smithi078 ceph-mon[32429]: pgmap v3535: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:45.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:45 smithi107 ceph-mon[36485]: pgmap v3535: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:45.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:45 smithi106 ceph-mon[36505]: pgmap v3535: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:46.221 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6435 sec 2024-03-24T02:11:47.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:46 smithi078 ceph-mon[32429]: pgmap v3536: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:47.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:46 smithi107 ceph-mon[36485]: pgmap v3536: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:47.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:46 smithi106 ceph-mon[36505]: pgmap v3536: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:47.221 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6436 sec 2024-03-24T02:11:48.221 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6437 sec 2024-03-24T02:11:49.221 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6438 sec 2024-03-24T02:11:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5783.187948 secs 2024-03-24T02:11:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:49 smithi078 ceph-mon[32429]: pgmap v3537: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5783.187948 secs 2024-03-24T02:11:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:49 smithi107 ceph-mon[36485]: pgmap v3537: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5783.187948 secs 2024-03-24T02:11:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:49 smithi106 ceph-mon[36505]: pgmap v3537: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:50.222 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6439 sec 2024-03-24T02:11:51.222 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6440 sec 2024-03-24T02:11:51.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:51 smithi078 ceph-mon[32429]: pgmap v3538: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:51.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:51 smithi107 ceph-mon[36485]: pgmap v3538: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:51.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:51 smithi106 ceph-mon[36505]: pgmap v3538: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:52.222 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6441 sec 2024-03-24T02:11:53.222 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6442 sec 2024-03-24T02:11:53.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:53 smithi078 ceph-mon[32429]: pgmap v3539: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:53.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:53 smithi107 ceph-mon[36485]: pgmap v3539: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:53.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:53 smithi106 ceph-mon[36505]: pgmap v3539: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:54.222 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6443 sec 2024-03-24T02:11:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5788.188007 secs 2024-03-24T02:11:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5788.188007 secs 2024-03-24T02:11:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5788.188007 secs 2024-03-24T02:11:55.222 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6444 sec 2024-03-24T02:11:55.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:55 smithi078 ceph-mon[32429]: pgmap v3540: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:55.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:55 smithi107 ceph-mon[36485]: pgmap v3540: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:55.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:55 smithi106 ceph-mon[36505]: pgmap v3540: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:56.223 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6445 sec 2024-03-24T02:11:57.222 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6446 sec 2024-03-24T02:11:57.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:57 smithi078 ceph-mon[32429]: pgmap v3541: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:57.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:57 smithi107 ceph-mon[36485]: pgmap v3541: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:57.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:57 smithi106 ceph-mon[36505]: pgmap v3541: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:58.222 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6447 sec 2024-03-24T02:11:59.223 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6448 sec 2024-03-24T02:11:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5793.188089 secs 2024-03-24T02:11:59.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:11:59 smithi078 ceph-mon[32429]: pgmap v3542: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5793.188089 secs 2024-03-24T02:11:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:11:59 smithi107 ceph-mon[36485]: pgmap v3542: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:11:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5793.188089 secs 2024-03-24T02:11:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:11:59 smithi106 ceph-mon[36505]: pgmap v3542: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:00.222 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6449 sec 2024-03-24T02:12:01.223 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6450 sec 2024-03-24T02:12:01.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:01 smithi078 ceph-mon[32429]: pgmap v3543: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:01.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:01 smithi107 ceph-mon[36485]: pgmap v3543: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:01.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:01 smithi106 ceph-mon[36505]: pgmap v3543: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:02.223 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6451 sec 2024-03-24T02:12:03.223 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6452 sec 2024-03-24T02:12:03.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:03 smithi078 ceph-mon[32429]: pgmap v3544: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:03.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:03 smithi107 ceph-mon[36485]: pgmap v3544: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:03.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:03 smithi106 ceph-mon[36505]: pgmap v3544: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:04.223 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6453 sec 2024-03-24T02:12:04.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:04 smithi078 ceph-mon[32429]: 2 slow requests, 1 included below; oldest blocked for > 5798.188173 secs 2024-03-24T02:12:04.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:04 smithi078 ceph-mon[32429]: slow request 5762.921102 seconds old, received at 2024-03-24T00:36:00.042315+0000: client_request(client.25385:166 mksnap #0x1//scheduled-2024-03-24-00_36_00_UTC 2024-03-24T00:36:00.042488+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-24T02:12:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:04 smithi107 ceph-mon[36485]: 2 slow requests, 1 included below; oldest blocked for > 5798.188173 secs 2024-03-24T02:12:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:04 smithi107 ceph-mon[36485]: slow request 5762.921102 seconds old, received at 2024-03-24T00:36:00.042315+0000: client_request(client.25385:166 mksnap #0x1//scheduled-2024-03-24-00_36_00_UTC 2024-03-24T00:36:00.042488+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-24T02:12:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:04 smithi106 ceph-mon[36505]: 2 slow requests, 1 included below; oldest blocked for > 5798.188173 secs 2024-03-24T02:12:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:04 smithi106 ceph-mon[36505]: slow request 5762.921102 seconds old, received at 2024-03-24T00:36:00.042315+0000: client_request(client.25385:166 mksnap #0x1//scheduled-2024-03-24-00_36_00_UTC 2024-03-24T00:36:00.042488+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-03-24T02:12:05.223 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6454 sec 2024-03-24T02:12:05.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:05 smithi078 ceph-mon[32429]: pgmap v3545: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:05.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:05 smithi107 ceph-mon[36485]: pgmap v3545: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:05.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:05 smithi106 ceph-mon[36505]: pgmap v3545: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:06.223 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6455 sec 2024-03-24T02:12:07.223 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6456 sec 2024-03-24T02:12:07.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:07 smithi078 ceph-mon[32429]: pgmap v3546: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:07.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:07 smithi107 ceph-mon[36485]: pgmap v3546: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:07.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:07 smithi106 ceph-mon[36505]: pgmap v3546: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:08.223 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6457 sec 2024-03-24T02:12:09.223 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6458 sec 2024-03-24T02:12:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5803.188214 secs 2024-03-24T02:12:09.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:09 smithi078 ceph-mon[32429]: pgmap v3547: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5803.188214 secs 2024-03-24T02:12:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:09 smithi107 ceph-mon[36485]: pgmap v3547: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5803.188214 secs 2024-03-24T02:12:09.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:09 smithi106 ceph-mon[36505]: pgmap v3547: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:10.223 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6459 sec 2024-03-24T02:12:11.224 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6460 sec 2024-03-24T02:12:11.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:11 smithi078 ceph-mon[32429]: pgmap v3548: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:11.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:11 smithi107 ceph-mon[36485]: pgmap v3548: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:11.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:11 smithi106 ceph-mon[36505]: pgmap v3548: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:12.224 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6461 sec 2024-03-24T02:12:13.224 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6462 sec 2024-03-24T02:12:13.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:13 smithi078 ceph-mon[32429]: pgmap v3549: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:13.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:13 smithi107 ceph-mon[36485]: pgmap v3549: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:13.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:13 smithi106 ceph-mon[36505]: pgmap v3549: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:14.224 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6463 sec 2024-03-24T02:12:14.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5808.188299 secs 2024-03-24T02:12:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5808.188299 secs 2024-03-24T02:12:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5808.188299 secs 2024-03-24T02:12:15.224 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6464 sec 2024-03-24T02:12:15.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:15 smithi078 ceph-mon[32429]: pgmap v3550: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:15.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:15 smithi107 ceph-mon[36485]: pgmap v3550: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:15.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:15 smithi106 ceph-mon[36505]: pgmap v3550: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:16.224 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6465 sec 2024-03-24T02:12:17.224 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6466 sec 2024-03-24T02:12:17.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:17 smithi078 ceph-mon[32429]: pgmap v3551: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:17.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:17 smithi107 ceph-mon[36485]: pgmap v3551: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:17.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:17 smithi106 ceph-mon[36505]: pgmap v3551: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:18.224 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6467 sec 2024-03-24T02:12:19.225 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6468 sec 2024-03-24T02:12:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5813.188381 secs 2024-03-24T02:12:19.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:19 smithi107 ceph-mon[36485]: pgmap v3552: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5813.188381 secs 2024-03-24T02:12:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:19 smithi106 ceph-mon[36505]: pgmap v3552: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:19.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5813.188381 secs 2024-03-24T02:12:19.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:19 smithi078 ceph-mon[32429]: pgmap v3552: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:20.224 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6469 sec 2024-03-24T02:12:21.224 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6470 sec 2024-03-24T02:12:21.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:21 smithi106 ceph-mon[36505]: pgmap v3553: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:21.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:21 smithi078 ceph-mon[32429]: pgmap v3553: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:21 smithi107 ceph-mon[36485]: pgmap v3553: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:22.225 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6471 sec 2024-03-24T02:12:23.225 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6472 sec 2024-03-24T02:12:23.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:23 smithi106 ceph-mon[36505]: pgmap v3554: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:23.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:23 smithi078 ceph-mon[32429]: pgmap v3554: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:23.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:23 smithi107 ceph-mon[36485]: pgmap v3554: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:24.225 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6473 sec 2024-03-24T02:12:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5818.188466 secs 2024-03-24T02:12:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5818.188466 secs 2024-03-24T02:12:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5818.188466 secs 2024-03-24T02:12:25.225 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6474 sec 2024-03-24T02:12:25.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:25 smithi106 ceph-mon[36505]: pgmap v3555: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:25.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:25 smithi078 ceph-mon[32429]: pgmap v3555: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:25.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:25 smithi107 ceph-mon[36485]: pgmap v3555: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:26.225 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6475 sec 2024-03-24T02:12:27.225 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6476 sec 2024-03-24T02:12:27.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:27 smithi106 ceph-mon[36505]: pgmap v3556: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:27.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:27 smithi078 ceph-mon[32429]: pgmap v3556: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:27.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:27 smithi107 ceph-mon[36485]: pgmap v3556: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:28.225 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6477 sec 2024-03-24T02:12:29.225 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6478 sec 2024-03-24T02:12:29.646 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5823.188528 secs 2024-03-24T02:12:29.646 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:29 smithi107 ceph-mon[36485]: pgmap v3557: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:29.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5823.188528 secs 2024-03-24T02:12:29.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:29 smithi106 ceph-mon[36505]: pgmap v3557: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:29.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5823.188528 secs 2024-03-24T02:12:29.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:29 smithi078 ceph-mon[32429]: pgmap v3557: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:30.225 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6479 sec 2024-03-24T02:12:31.225 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6480 sec 2024-03-24T02:12:31.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:31 smithi106 ceph-mon[36505]: pgmap v3558: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:31.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:31 smithi078 ceph-mon[32429]: pgmap v3558: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:31.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:31 smithi107 ceph-mon[36485]: pgmap v3558: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:32.225 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6481 sec 2024-03-24T02:12:32.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:32 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:12:32.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:32 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:12:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:32 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:12:33.226 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6482 sec 2024-03-24T02:12:33.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:33 smithi106 ceph-mon[36505]: pgmap v3559: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:33.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:33 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:12:33.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:33 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:12:33.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:33 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:12:33.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:33 smithi078 ceph-mon[32429]: pgmap v3559: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:33.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:33 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:12:33.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:33 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:12:33.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:33 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:12:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:33 smithi107 ceph-mon[36485]: pgmap v3559: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:33 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:12:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:33 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:12:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:33 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:12:34.226 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6483 sec 2024-03-24T02:12:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5828.188594 secs 2024-03-24T02:12:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:34 smithi106 ceph-mon[36505]: pgmap v3560: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:12:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:34 smithi106 ceph-mon[36505]: pgmap v3561: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:12:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5828.188594 secs 2024-03-24T02:12:34.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:34 smithi078 ceph-mon[32429]: pgmap v3560: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:12:34.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:34 smithi078 ceph-mon[32429]: pgmap v3561: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:12:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5828.188594 secs 2024-03-24T02:12:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:34 smithi107 ceph-mon[36485]: pgmap v3560: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:12:34.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:34 smithi107 ceph-mon[36485]: pgmap v3561: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:12:35.226 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6484 sec 2024-03-24T02:12:36.226 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6485 sec 2024-03-24T02:12:36.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:36 smithi106 ceph-mon[36505]: pgmap v3562: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:12:36.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:36 smithi078 ceph-mon[32429]: pgmap v3562: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:12:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:36 smithi107 ceph-mon[36485]: pgmap v3562: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:12:37.226 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6486 sec 2024-03-24T02:12:38.226 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6487 sec 2024-03-24T02:12:38.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:38 smithi106 ceph-mon[36505]: pgmap v3563: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:12:38.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:38 smithi078 ceph-mon[32429]: pgmap v3563: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:12:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:38 smithi107 ceph-mon[36485]: pgmap v3563: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:12:39.226 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6488 sec 2024-03-24T02:12:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5833.188683 secs 2024-03-24T02:12:39.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5833.188683 secs 2024-03-24T02:12:39.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5833.188683 secs 2024-03-24T02:12:40.226 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6489 sec 2024-03-24T02:12:40.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:40 smithi106 ceph-mon[36505]: pgmap v3564: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:12:40.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:40 smithi078 ceph-mon[32429]: pgmap v3564: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:12:40.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:40 smithi107 ceph-mon[36485]: pgmap v3564: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:12:41.226 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6490 sec 2024-03-24T02:12:42.226 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6491 sec 2024-03-24T02:12:42.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:42 smithi106 ceph-mon[36505]: pgmap v3565: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:12:42.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:42 smithi078 ceph-mon[32429]: pgmap v3565: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:12:42.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:42 smithi107 ceph-mon[36485]: pgmap v3565: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:12:43.226 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6492 sec 2024-03-24T02:12:44.227 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6493 sec 2024-03-24T02:12:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5838.188743 secs 2024-03-24T02:12:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:44 smithi106 ceph-mon[36505]: pgmap v3566: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:12:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5838.188743 secs 2024-03-24T02:12:44.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:44 smithi078 ceph-mon[32429]: pgmap v3566: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:12:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5838.188743 secs 2024-03-24T02:12:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:44 smithi107 ceph-mon[36485]: pgmap v3566: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:12:45.227 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6494 sec 2024-03-24T02:12:46.227 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6495 sec 2024-03-24T02:12:46.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:46 smithi078 ceph-mon[32429]: pgmap v3567: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:46.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:46 smithi107 ceph-mon[36485]: pgmap v3567: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:46 smithi106 ceph-mon[36505]: pgmap v3567: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:47.227 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6496 sec 2024-03-24T02:12:48.227 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6497 sec 2024-03-24T02:12:48.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:48 smithi078 ceph-mon[32429]: pgmap v3568: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:48 smithi107 ceph-mon[36485]: pgmap v3568: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:48 smithi106 ceph-mon[36505]: pgmap v3568: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:49.227 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6498 sec 2024-03-24T02:12:49.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5843.188828 secs 2024-03-24T02:12:49.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5843.188828 secs 2024-03-24T02:12:49.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5843.188828 secs 2024-03-24T02:12:50.227 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6499 sec 2024-03-24T02:12:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:50 smithi078 ceph-mon[32429]: pgmap v3569: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:50 smithi107 ceph-mon[36485]: pgmap v3569: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:50.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:50 smithi106 ceph-mon[36505]: pgmap v3569: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:51.227 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6500 sec 2024-03-24T02:12:52.227 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6501 sec 2024-03-24T02:12:52.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:52 smithi078 ceph-mon[32429]: pgmap v3570: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:52.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:52 smithi107 ceph-mon[36485]: pgmap v3570: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:52.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:52 smithi106 ceph-mon[36505]: pgmap v3570: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:53.227 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6502 sec 2024-03-24T02:12:54.227 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6503 sec 2024-03-24T02:12:54.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5848.188915 secs 2024-03-24T02:12:54.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:54 smithi078 ceph-mon[32429]: pgmap v3571: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5848.188915 secs 2024-03-24T02:12:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:54 smithi107 ceph-mon[36485]: pgmap v3571: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5848.188915 secs 2024-03-24T02:12:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:54 smithi106 ceph-mon[36505]: pgmap v3571: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:55.227 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6504 sec 2024-03-24T02:12:56.227 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6505 sec 2024-03-24T02:12:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:56 smithi078 ceph-mon[32429]: pgmap v3572: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:56 smithi107 ceph-mon[36485]: pgmap v3572: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:56.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:56 smithi106 ceph-mon[36505]: pgmap v3572: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:12:57.228 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6506 sec 2024-03-24T02:12:58.228 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6507 sec 2024-03-24T02:12:58.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:58 smithi078 ceph-mon[32429]: pgmap v3573: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:58 smithi107 ceph-mon[36485]: pgmap v3573: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:58 smithi106 ceph-mon[36505]: pgmap v3573: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:59.228 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6508 sec 2024-03-24T02:12:59.745 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:12:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5853.188958 secs 2024-03-24T02:12:59.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:12:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5853.188958 secs 2024-03-24T02:12:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:12:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5853.188958 secs 2024-03-24T02:13:00.228 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6509 sec 2024-03-24T02:13:00.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:00 smithi078 ceph-mon[32429]: pgmap v3574: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:00 smithi107 ceph-mon[36485]: pgmap v3574: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:00.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:00 smithi106 ceph-mon[36505]: pgmap v3574: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:01.227 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6510 sec 2024-03-24T02:13:02.228 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6511 sec 2024-03-24T02:13:02.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:02 smithi078 ceph-mon[32429]: pgmap v3575: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:02 smithi107 ceph-mon[36485]: pgmap v3575: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:02.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:02 smithi106 ceph-mon[36505]: pgmap v3575: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:03.228 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6512 sec 2024-03-24T02:13:04.228 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6513 sec 2024-03-24T02:13:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5858.189000 secs 2024-03-24T02:13:04.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:04 smithi078 ceph-mon[32429]: pgmap v3576: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5858.189000 secs 2024-03-24T02:13:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:04 smithi107 ceph-mon[36485]: pgmap v3576: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5858.189000 secs 2024-03-24T02:13:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:04 smithi106 ceph-mon[36505]: pgmap v3576: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:05.228 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6514 sec 2024-03-24T02:13:06.228 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6515 sec 2024-03-24T02:13:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:06 smithi078 ceph-mon[32429]: pgmap v3577: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:06 smithi107 ceph-mon[36485]: pgmap v3577: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:06 smithi106 ceph-mon[36505]: pgmap v3577: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:07.228 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6516 sec 2024-03-24T02:13:08.228 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6517 sec 2024-03-24T02:13:08.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:08 smithi078 ceph-mon[32429]: pgmap v3578: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:08 smithi107 ceph-mon[36485]: pgmap v3578: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:08 smithi106 ceph-mon[36505]: pgmap v3578: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:09.228 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6518 sec 2024-03-24T02:13:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5863.189059 secs 2024-03-24T02:13:09.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5863.189059 secs 2024-03-24T02:13:09.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5863.189059 secs 2024-03-24T02:13:10.228 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6519 sec 2024-03-24T02:13:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:10 smithi078 ceph-mon[32429]: pgmap v3579: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:10 smithi107 ceph-mon[36485]: pgmap v3579: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:10 smithi106 ceph-mon[36505]: pgmap v3579: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:11.229 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6520 sec 2024-03-24T02:13:12.229 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6521 sec 2024-03-24T02:13:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:12 smithi078 ceph-mon[32429]: pgmap v3580: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:12 smithi107 ceph-mon[36485]: pgmap v3580: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:12.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:12 smithi106 ceph-mon[36505]: pgmap v3580: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:13.229 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6522 sec 2024-03-24T02:13:14.229 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6523 sec 2024-03-24T02:13:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5868.189146 secs 2024-03-24T02:13:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:14 smithi078 ceph-mon[32429]: pgmap v3581: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5868.189146 secs 2024-03-24T02:13:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:14 smithi107 ceph-mon[36485]: pgmap v3581: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5868.189146 secs 2024-03-24T02:13:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:14 smithi106 ceph-mon[36505]: pgmap v3581: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:15.229 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6524 sec 2024-03-24T02:13:16.229 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6525 sec 2024-03-24T02:13:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:16 smithi078 ceph-mon[32429]: pgmap v3582: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:16 smithi107 ceph-mon[36485]: pgmap v3582: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:16.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:16 smithi106 ceph-mon[36505]: pgmap v3582: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:17.229 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6526 sec 2024-03-24T02:13:18.229 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6527 sec 2024-03-24T02:13:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:18 smithi078 ceph-mon[32429]: pgmap v3583: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:18 smithi107 ceph-mon[36485]: pgmap v3583: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:18 smithi106 ceph-mon[36505]: pgmap v3583: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:19.229 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6528 sec 2024-03-24T02:13:19.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5873.189236 secs 2024-03-24T02:13:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5873.189236 secs 2024-03-24T02:13:19.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5873.189236 secs 2024-03-24T02:13:20.229 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6529 sec 2024-03-24T02:13:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:20 smithi078 ceph-mon[32429]: pgmap v3584: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:20 smithi107 ceph-mon[36485]: pgmap v3584: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:20.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:20 smithi106 ceph-mon[36505]: pgmap v3584: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:21.229 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6530 sec 2024-03-24T02:13:22.230 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6531 sec 2024-03-24T02:13:22.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:22 smithi078 ceph-mon[32429]: pgmap v3585: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:22.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:22 smithi107 ceph-mon[36485]: pgmap v3585: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:22.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:22 smithi106 ceph-mon[36505]: pgmap v3585: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:23.230 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6532 sec 2024-03-24T02:13:24.230 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6533 sec 2024-03-24T02:13:24.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5878.189275 secs 2024-03-24T02:13:24.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:24 smithi078 ceph-mon[32429]: pgmap v3586: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:24.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5878.189275 secs 2024-03-24T02:13:24.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:24 smithi107 ceph-mon[36485]: pgmap v3586: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5878.189275 secs 2024-03-24T02:13:24.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:24 smithi106 ceph-mon[36505]: pgmap v3586: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:25.230 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6534 sec 2024-03-24T02:13:26.230 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6535 sec 2024-03-24T02:13:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:26 smithi078 ceph-mon[32429]: pgmap v3587: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:26 smithi107 ceph-mon[36485]: pgmap v3587: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:26.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:26 smithi106 ceph-mon[36505]: pgmap v3587: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:27.230 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6536 sec 2024-03-24T02:13:28.230 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6537 sec 2024-03-24T02:13:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:28 smithi078 ceph-mon[32429]: pgmap v3588: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:28 smithi107 ceph-mon[36485]: pgmap v3588: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:28 smithi106 ceph-mon[36505]: pgmap v3588: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:29.230 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6538 sec 2024-03-24T02:13:29.745 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5883.189358 secs 2024-03-24T02:13:29.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5883.189358 secs 2024-03-24T02:13:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5883.189358 secs 2024-03-24T02:13:30.230 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6539 sec 2024-03-24T02:13:30.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:30 smithi078 ceph-mon[32429]: pgmap v3589: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:30 smithi107 ceph-mon[36485]: pgmap v3589: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:30 smithi106 ceph-mon[36505]: pgmap v3589: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:31.230 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6540 sec 2024-03-24T02:13:32.231 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6541 sec 2024-03-24T02:13:32.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:32 smithi078 ceph-mon[32429]: pgmap v3590: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:32 smithi107 ceph-mon[36485]: pgmap v3590: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:32 smithi106 ceph-mon[36505]: pgmap v3590: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:33.230 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6542 sec 2024-03-24T02:13:33.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:33 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:13:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:33 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:13:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:33 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:13:34.231 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6543 sec 2024-03-24T02:13:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5888.189448 secs 2024-03-24T02:13:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:34 smithi078 ceph-mon[32429]: pgmap v3591: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5888.189448 secs 2024-03-24T02:13:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:34 smithi107 ceph-mon[36485]: pgmap v3591: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5888.189448 secs 2024-03-24T02:13:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:34 smithi106 ceph-mon[36505]: pgmap v3591: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:35.230 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6544 sec 2024-03-24T02:13:36.231 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6545 sec 2024-03-24T02:13:36.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:36 smithi078 ceph-mon[32429]: pgmap v3592: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:36.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:13:36.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:13:36.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:13:36.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:13:36.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:13:36.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:13:36.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:36.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:36.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:36.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:36.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:36.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:36.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:36.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:36.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:36 smithi107 ceph-mon[36485]: pgmap v3592: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:36.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:13:36.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:13:36.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:13:36.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:13:36.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:13:36.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:13:36.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:36.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:36.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:36.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:36.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:36.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:36.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:36.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:36.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:36 smithi106 ceph-mon[36505]: pgmap v3592: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:36.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:13:36.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:13:36.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:13:36.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:13:36.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:13:36.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:13:36.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:36.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:36.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:36.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:36.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:36.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:36.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:36.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:37.231 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6546 sec 2024-03-24T02:13:37.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:37 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T02:13:37.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:37 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T02:13:37.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:37 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T02:13:37.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:37 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T02:13:37.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:37.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:37.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:37.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:37.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:37 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T02:13:37.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:37 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T02:13:37.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:13:37.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:13:37.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:37 smithi078 ceph-mon[32429]: pgmap v3593: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T02:13:37.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:37 smithi078 ceph-mon[32429]: pgmap v3594: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:13:37.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:13:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:37 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T02:13:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:37 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T02:13:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:37 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T02:13:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:37 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T02:13:37.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:37.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:37.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:37.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:37.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:37 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T02:13:37.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:37 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T02:13:37.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:13:37.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:13:37.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:37 smithi107 ceph-mon[36485]: pgmap v3593: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T02:13:37.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:37 smithi107 ceph-mon[36485]: pgmap v3594: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:13:37.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:13:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:37 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T02:13:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:37 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T02:13:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:37 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T02:13:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:37 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T02:13:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:37.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:37.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:37.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:13:37.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:37 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T02:13:37.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:37 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T02:13:37.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:13:37.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:13:37.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:37 smithi106 ceph-mon[36505]: pgmap v3593: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T02:13:37.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:37 smithi106 ceph-mon[36505]: pgmap v3594: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:13:37.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:13:38.231 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6547 sec 2024-03-24T02:13:39.231 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6548 sec 2024-03-24T02:13:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5893.189508 secs 2024-03-24T02:13:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:39 smithi106 ceph-mon[36505]: pgmap v3595: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:13:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5893.189508 secs 2024-03-24T02:13:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:39 smithi078 ceph-mon[32429]: pgmap v3595: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:13:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5893.189508 secs 2024-03-24T02:13:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:39 smithi107 ceph-mon[36485]: pgmap v3595: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:13:40.231 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6549 sec 2024-03-24T02:13:41.231 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6550 sec 2024-03-24T02:13:41.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:41 smithi106 ceph-mon[36505]: pgmap v3596: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:13:41.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:41 smithi078 ceph-mon[32429]: pgmap v3596: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:13:41.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:41 smithi107 ceph-mon[36485]: pgmap v3596: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:13:42.231 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6551 sec 2024-03-24T02:13:43.231 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6552 sec 2024-03-24T02:13:43.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:43 smithi106 ceph-mon[36505]: pgmap v3597: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:13:43.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:43 smithi078 ceph-mon[32429]: pgmap v3597: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:13:43.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:43 smithi107 ceph-mon[36485]: pgmap v3597: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:13:44.231 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6553 sec 2024-03-24T02:13:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5898.189543 secs 2024-03-24T02:13:44.491 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5898.189543 secs 2024-03-24T02:13:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5898.189543 secs 2024-03-24T02:13:45.231 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6554 sec 2024-03-24T02:13:45.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:45 smithi106 ceph-mon[36505]: pgmap v3598: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:13:45.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:45 smithi078 ceph-mon[32429]: pgmap v3598: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:13:45.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:45 smithi107 ceph-mon[36485]: pgmap v3598: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:13:46.231 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6555 sec 2024-03-24T02:13:47.232 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6556 sec 2024-03-24T02:13:47.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:47 smithi106 ceph-mon[36505]: pgmap v3599: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:13:47.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:47 smithi078 ceph-mon[32429]: pgmap v3599: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:13:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:47 smithi107 ceph-mon[36485]: pgmap v3599: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:13:48.232 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6557 sec 2024-03-24T02:13:49.232 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6558 sec 2024-03-24T02:13:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5903.189579 secs 2024-03-24T02:13:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:49 smithi078 ceph-mon[32429]: pgmap v3600: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5903.189579 secs 2024-03-24T02:13:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:49 smithi107 ceph-mon[36485]: pgmap v3600: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5903.189579 secs 2024-03-24T02:13:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:49 smithi106 ceph-mon[36505]: pgmap v3600: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:50.232 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6559 sec 2024-03-24T02:13:51.232 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6560 sec 2024-03-24T02:13:51.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:51 smithi078 ceph-mon[32429]: pgmap v3601: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:51.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:51 smithi107 ceph-mon[36485]: pgmap v3601: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:51.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:51 smithi106 ceph-mon[36505]: pgmap v3601: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:52.232 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6561 sec 2024-03-24T02:13:53.232 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6562 sec 2024-03-24T02:13:53.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:53 smithi078 ceph-mon[32429]: pgmap v3602: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:53.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:53 smithi107 ceph-mon[36485]: pgmap v3602: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:53.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:53 smithi106 ceph-mon[36505]: pgmap v3602: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:54.232 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6563 sec 2024-03-24T02:13:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5908.189636 secs 2024-03-24T02:13:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5908.189636 secs 2024-03-24T02:13:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5908.189636 secs 2024-03-24T02:13:55.232 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6564 sec 2024-03-24T02:13:55.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:55 smithi106 ceph-mon[36505]: pgmap v3603: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:55.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:55 smithi078 ceph-mon[32429]: pgmap v3603: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:55.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:55 smithi107 ceph-mon[36485]: pgmap v3603: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:56.232 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6565 sec 2024-03-24T02:13:57.233 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6566 sec 2024-03-24T02:13:57.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:57 smithi078 ceph-mon[32429]: pgmap v3604: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:57.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:57 smithi107 ceph-mon[36485]: pgmap v3604: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:57.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:57 smithi106 ceph-mon[36505]: pgmap v3604: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:58.232 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6567 sec 2024-03-24T02:13:59.233 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6568 sec 2024-03-24T02:13:59.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5913.189696 secs 2024-03-24T02:13:59.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:13:59 smithi078 ceph-mon[32429]: pgmap v3605: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5913.189696 secs 2024-03-24T02:13:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:13:59 smithi107 ceph-mon[36485]: pgmap v3605: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:13:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5913.189696 secs 2024-03-24T02:13:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:13:59 smithi106 ceph-mon[36505]: pgmap v3605: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:00.233 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6569 sec 2024-03-24T02:14:01.233 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6570 sec 2024-03-24T02:14:01.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:01 smithi078 ceph-mon[32429]: pgmap v3606: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:01.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:01 smithi107 ceph-mon[36485]: pgmap v3606: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:01.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:01 smithi106 ceph-mon[36505]: pgmap v3606: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:02.233 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6571 sec 2024-03-24T02:14:03.233 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6572 sec 2024-03-24T02:14:03.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:03 smithi078 ceph-mon[32429]: pgmap v3607: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:03.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:03 smithi107 ceph-mon[36485]: pgmap v3607: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:03.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:03 smithi106 ceph-mon[36505]: pgmap v3607: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:04.233 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6573 sec 2024-03-24T02:14:04.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5918.189756 secs 2024-03-24T02:14:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5918.189756 secs 2024-03-24T02:14:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5918.189756 secs 2024-03-24T02:14:05.233 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6574 sec 2024-03-24T02:14:05.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:05 smithi078 ceph-mon[32429]: pgmap v3608: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:05.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:05 smithi107 ceph-mon[36485]: pgmap v3608: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:05.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:05 smithi106 ceph-mon[36505]: pgmap v3608: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:06.233 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6575 sec 2024-03-24T02:14:07.233 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6576 sec 2024-03-24T02:14:07.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:07 smithi078 ceph-mon[32429]: pgmap v3609: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:07.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:07 smithi107 ceph-mon[36485]: pgmap v3609: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:07.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:07 smithi106 ceph-mon[36505]: pgmap v3609: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:08.233 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6577 sec 2024-03-24T02:14:09.233 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6578 sec 2024-03-24T02:14:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5923.189837 secs 2024-03-24T02:14:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:09 smithi078 ceph-mon[32429]: pgmap v3610: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5923.189837 secs 2024-03-24T02:14:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:09 smithi107 ceph-mon[36485]: pgmap v3610: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5923.189837 secs 2024-03-24T02:14:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:09 smithi106 ceph-mon[36505]: pgmap v3610: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:10.234 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6579 sec 2024-03-24T02:14:11.234 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6580 sec 2024-03-24T02:14:11.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:11 smithi078 ceph-mon[32429]: pgmap v3611: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:11.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:11 smithi107 ceph-mon[36485]: pgmap v3611: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:11.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:11 smithi106 ceph-mon[36505]: pgmap v3611: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:12.234 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6581 sec 2024-03-24T02:14:13.234 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6582 sec 2024-03-24T02:14:13.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:13 smithi078 ceph-mon[32429]: pgmap v3612: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:13.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:13 smithi107 ceph-mon[36485]: pgmap v3612: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:13.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:13 smithi106 ceph-mon[36505]: pgmap v3612: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:14.234 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6583 sec 2024-03-24T02:14:14.491 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5928.189892 secs 2024-03-24T02:14:14.552 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5928.189892 secs 2024-03-24T02:14:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5928.189892 secs 2024-03-24T02:14:15.234 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6584 sec 2024-03-24T02:14:15.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:15 smithi078 ceph-mon[32429]: pgmap v3613: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:15.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:15 smithi107 ceph-mon[36485]: pgmap v3613: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:15.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:15 smithi106 ceph-mon[36505]: pgmap v3613: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:16.234 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6585 sec 2024-03-24T02:14:17.234 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6586 sec 2024-03-24T02:14:17.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:17 smithi078 ceph-mon[32429]: pgmap v3614: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:14:17.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:17 smithi107 ceph-mon[36485]: pgmap v3614: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:14:17.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:17 smithi106 ceph-mon[36505]: pgmap v3614: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:14:18.234 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6587 sec 2024-03-24T02:14:19.234 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6588 sec 2024-03-24T02:14:19.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5933.189949 secs 2024-03-24T02:14:19.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:19 smithi078 ceph-mon[32429]: pgmap v3615: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:14:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5933.189949 secs 2024-03-24T02:14:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:19 smithi107 ceph-mon[36485]: pgmap v3615: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:14:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5933.189949 secs 2024-03-24T02:14:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:19 smithi106 ceph-mon[36505]: pgmap v3615: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:14:20.234 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6589 sec 2024-03-24T02:14:21.235 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6590 sec 2024-03-24T02:14:21.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:21 smithi078 ceph-mon[32429]: pgmap v3616: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:14:21.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:21 smithi107 ceph-mon[36485]: pgmap v3616: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:14:21.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:21 smithi106 ceph-mon[36505]: pgmap v3616: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:14:22.234 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6591 sec 2024-03-24T02:14:23.235 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6592 sec 2024-03-24T02:14:23.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:23 smithi078 ceph-mon[32429]: pgmap v3617: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:14:23.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:23 smithi107 ceph-mon[36485]: pgmap v3617: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:14:23.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:23 smithi106 ceph-mon[36505]: pgmap v3617: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:14:24.235 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6593 sec 2024-03-24T02:14:24.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5938.190035 secs 2024-03-24T02:14:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5938.190035 secs 2024-03-24T02:14:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5938.190035 secs 2024-03-24T02:14:25.235 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6594 sec 2024-03-24T02:14:25.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:25 smithi078 ceph-mon[32429]: pgmap v3618: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:14:25.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:25 smithi107 ceph-mon[36485]: pgmap v3618: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:14:25.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:25 smithi106 ceph-mon[36505]: pgmap v3618: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:14:26.235 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6595 sec 2024-03-24T02:14:27.235 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6596 sec 2024-03-24T02:14:27.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:27 smithi078 ceph-mon[32429]: pgmap v3619: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:14:27.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:27 smithi107 ceph-mon[36485]: pgmap v3619: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:14:27.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:27 smithi106 ceph-mon[36505]: pgmap v3619: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:14:28.235 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6597 sec 2024-03-24T02:14:29.235 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6598 sec 2024-03-24T02:14:29.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5943.190075 secs 2024-03-24T02:14:29.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:29 smithi078 ceph-mon[32429]: pgmap v3620: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5943.190075 secs 2024-03-24T02:14:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:29 smithi107 ceph-mon[36485]: pgmap v3620: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:29.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5943.190075 secs 2024-03-24T02:14:29.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:29 smithi106 ceph-mon[36505]: pgmap v3620: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:30.235 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6599 sec 2024-03-24T02:14:31.235 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6600 sec 2024-03-24T02:14:31.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:31 smithi078 ceph-mon[32429]: pgmap v3621: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:31.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:31 smithi107 ceph-mon[36485]: pgmap v3621: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:31.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:31 smithi106 ceph-mon[36505]: pgmap v3621: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:32.235 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6601 sec 2024-03-24T02:14:33.235 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6602 sec 2024-03-24T02:14:33.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:33 smithi078 ceph-mon[32429]: pgmap v3622: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:33.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:33 smithi107 ceph-mon[36485]: pgmap v3622: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:33.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:33 smithi106 ceph-mon[36505]: pgmap v3622: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:34.235 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6603 sec 2024-03-24T02:14:34.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5948.190115 secs 2024-03-24T02:14:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5948.190115 secs 2024-03-24T02:14:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5948.190115 secs 2024-03-24T02:14:35.235 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6604 sec 2024-03-24T02:14:35.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:35 smithi078 ceph-mon[32429]: pgmap v3623: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:35.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:35 smithi107 ceph-mon[36485]: pgmap v3623: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:35.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:35 smithi106 ceph-mon[36505]: pgmap v3623: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:36.235 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6605 sec 2024-03-24T02:14:36.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:14:36.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:14:36.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:14:37.236 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6606 sec 2024-03-24T02:14:37.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:37 smithi078 ceph-mon[32429]: pgmap v3624: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:37.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:14:37.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:14:37.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:14:37.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:14:37.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:14:37.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:14:37.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:37 smithi107 ceph-mon[36485]: pgmap v3624: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:37.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:14:37.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:14:37.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:14:37.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:14:37.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:14:37.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:14:37.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:37 smithi106 ceph-mon[36505]: pgmap v3624: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:37.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:14:37.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:14:37.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:14:37.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:14:37.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:14:37.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:14:38.236 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6607 sec 2024-03-24T02:14:38.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:38 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:14:38.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:38 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:14:38.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:38 smithi078 ceph-mon[32429]: pgmap v3625: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T02:14:38.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:38 smithi078 ceph-mon[32429]: pgmap v3626: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:14:38.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:38 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:14:38.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:38 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:14:38.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:38 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:14:38.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:38 smithi107 ceph-mon[36485]: pgmap v3625: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T02:14:38.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:38 smithi107 ceph-mon[36485]: pgmap v3626: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:14:38.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:38 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:14:38.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:38 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:14:38.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:38 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:14:38.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:38 smithi106 ceph-mon[36505]: pgmap v3625: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T02:14:38.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:38 smithi106 ceph-mon[36505]: pgmap v3626: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:14:38.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:38 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:14:39.236 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6608 sec 2024-03-24T02:14:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5953.190200 secs 2024-03-24T02:14:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5953.190200 secs 2024-03-24T02:14:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5953.190200 secs 2024-03-24T02:14:40.236 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6609 sec 2024-03-24T02:14:40.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:40 smithi078 ceph-mon[32429]: pgmap v3627: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:14:40.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:40 smithi107 ceph-mon[36485]: pgmap v3627: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:14:40.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:40 smithi106 ceph-mon[36505]: pgmap v3627: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:14:41.236 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6610 sec 2024-03-24T02:14:42.236 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6611 sec 2024-03-24T02:14:42.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:42 smithi078 ceph-mon[32429]: pgmap v3628: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:14:42.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:42 smithi107 ceph-mon[36485]: pgmap v3628: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:14:42.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:42 smithi106 ceph-mon[36505]: pgmap v3628: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:14:43.236 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6612 sec 2024-03-24T02:14:44.236 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6613 sec 2024-03-24T02:14:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5958.190237 secs 2024-03-24T02:14:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:44 smithi078 ceph-mon[32429]: pgmap v3629: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:14:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5958.190237 secs 2024-03-24T02:14:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:44 smithi107 ceph-mon[36485]: pgmap v3629: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:14:44.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5958.190237 secs 2024-03-24T02:14:44.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:44 smithi106 ceph-mon[36505]: pgmap v3629: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:14:45.236 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6614 sec 2024-03-24T02:14:46.236 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6615 sec 2024-03-24T02:14:46.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:46 smithi078 ceph-mon[32429]: pgmap v3630: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-24T02:14:46.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:46 smithi107 ceph-mon[36485]: pgmap v3630: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-24T02:14:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:46 smithi106 ceph-mon[36505]: pgmap v3630: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-24T02:14:47.236 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6616 sec 2024-03-24T02:14:48.237 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6617 sec 2024-03-24T02:14:48.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:48 smithi078 ceph-mon[32429]: pgmap v3631: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:14:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:48 smithi107 ceph-mon[36485]: pgmap v3631: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:14:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:48 smithi106 ceph-mon[36505]: pgmap v3631: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:14:49.237 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6618 sec 2024-03-24T02:14:49.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5963.190314 secs 2024-03-24T02:14:49.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5963.190314 secs 2024-03-24T02:14:49.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5963.190314 secs 2024-03-24T02:14:50.237 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6619 sec 2024-03-24T02:14:50.748 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:50 smithi078 ceph-mon[32429]: pgmap v3632: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:50 smithi107 ceph-mon[36485]: pgmap v3632: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:50.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:50 smithi106 ceph-mon[36505]: pgmap v3632: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:51.237 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6620 sec 2024-03-24T02:14:52.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:51 smithi078 ceph-mon[32429]: pgmap v3633: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:52.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:51 smithi107 ceph-mon[36485]: pgmap v3633: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:52.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:51 smithi106 ceph-mon[36505]: pgmap v3633: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:52.237 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6621 sec 2024-03-24T02:14:53.237 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6622 sec 2024-03-24T02:14:54.237 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6623 sec 2024-03-24T02:14:54.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5968.190374 secs 2024-03-24T02:14:54.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:54 smithi078 ceph-mon[32429]: pgmap v3634: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5968.190374 secs 2024-03-24T02:14:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:54 smithi107 ceph-mon[36485]: pgmap v3634: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5968.190374 secs 2024-03-24T02:14:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:54 smithi106 ceph-mon[36505]: pgmap v3634: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:55.237 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6624 sec 2024-03-24T02:14:56.238 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6625 sec 2024-03-24T02:14:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:56 smithi078 ceph-mon[32429]: pgmap v3635: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:56 smithi107 ceph-mon[36485]: pgmap v3635: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:56.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:56 smithi106 ceph-mon[36505]: pgmap v3635: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:57.238 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6626 sec 2024-03-24T02:14:58.238 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6627 sec 2024-03-24T02:14:58.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:58 smithi078 ceph-mon[32429]: pgmap v3636: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:58 smithi107 ceph-mon[36485]: pgmap v3636: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:58 smithi106 ceph-mon[36505]: pgmap v3636: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:14:59.238 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6628 sec 2024-03-24T02:14:59.744 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:14:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5973.190434 secs 2024-03-24T02:14:59.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:14:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5973.190434 secs 2024-03-24T02:14:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:14:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5973.190434 secs 2024-03-24T02:15:00.238 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6629 sec 2024-03-24T02:15:00.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:00 smithi078 ceph-mon[32429]: pgmap v3637: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:00 smithi107 ceph-mon[36485]: pgmap v3637: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:00.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:00 smithi106 ceph-mon[36505]: pgmap v3637: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:01.238 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6630 sec 2024-03-24T02:15:02.238 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6631 sec 2024-03-24T02:15:02.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:02 smithi078 ceph-mon[32429]: pgmap v3638: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:02 smithi107 ceph-mon[36485]: pgmap v3638: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:02.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:02 smithi106 ceph-mon[36505]: pgmap v3638: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:03.238 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6632 sec 2024-03-24T02:15:04.238 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6633 sec 2024-03-24T02:15:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5978.190446 secs 2024-03-24T02:15:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:04 smithi078 ceph-mon[32429]: pgmap v3639: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5978.190446 secs 2024-03-24T02:15:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:04 smithi107 ceph-mon[36485]: pgmap v3639: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5978.190446 secs 2024-03-24T02:15:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:04 smithi106 ceph-mon[36505]: pgmap v3639: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:05.238 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6634 sec 2024-03-24T02:15:06.239 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6635 sec 2024-03-24T02:15:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:06 smithi078 ceph-mon[32429]: pgmap v3640: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:06 smithi107 ceph-mon[36485]: pgmap v3640: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:06 smithi106 ceph-mon[36505]: pgmap v3640: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:07.239 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6636 sec 2024-03-24T02:15:08.239 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6637 sec 2024-03-24T02:15:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:08 smithi078 ceph-mon[32429]: pgmap v3641: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:08 smithi107 ceph-mon[36485]: pgmap v3641: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:08 smithi106 ceph-mon[36505]: pgmap v3641: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:09.239 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6638 sec 2024-03-24T02:15:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5983.190532 secs 2024-03-24T02:15:09.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5983.190532 secs 2024-03-24T02:15:09.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5983.190532 secs 2024-03-24T02:15:10.239 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6639 sec 2024-03-24T02:15:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:10 smithi078 ceph-mon[32429]: pgmap v3642: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:10 smithi107 ceph-mon[36485]: pgmap v3642: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:10 smithi106 ceph-mon[36505]: pgmap v3642: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:11.239 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6640 sec 2024-03-24T02:15:12.239 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6641 sec 2024-03-24T02:15:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:12 smithi078 ceph-mon[32429]: pgmap v3643: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:12 smithi107 ceph-mon[36485]: pgmap v3643: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:12.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:12 smithi106 ceph-mon[36505]: pgmap v3643: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:13.239 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6642 sec 2024-03-24T02:15:14.239 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6643 sec 2024-03-24T02:15:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5988.190592 secs 2024-03-24T02:15:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:14 smithi078 ceph-mon[32429]: pgmap v3644: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5988.190592 secs 2024-03-24T02:15:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:14 smithi107 ceph-mon[36485]: pgmap v3644: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5988.190592 secs 2024-03-24T02:15:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:14 smithi106 ceph-mon[36505]: pgmap v3644: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:15.239 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6644 sec 2024-03-24T02:15:16.239 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6645 sec 2024-03-24T02:15:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:16 smithi078 ceph-mon[32429]: pgmap v3645: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:16 smithi107 ceph-mon[36485]: pgmap v3645: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:16.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:16 smithi106 ceph-mon[36505]: pgmap v3645: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:17.239 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6646 sec 2024-03-24T02:15:18.240 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6647 sec 2024-03-24T02:15:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:18 smithi078 ceph-mon[32429]: pgmap v3646: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:18 smithi107 ceph-mon[36485]: pgmap v3646: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:18 smithi106 ceph-mon[36505]: pgmap v3646: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:19.240 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6648 sec 2024-03-24T02:15:19.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5993.190678 secs 2024-03-24T02:15:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5993.190678 secs 2024-03-24T02:15:19.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5993.190678 secs 2024-03-24T02:15:20.240 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6649 sec 2024-03-24T02:15:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:20 smithi078 ceph-mon[32429]: pgmap v3647: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:20 smithi107 ceph-mon[36485]: pgmap v3647: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:20.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:20 smithi106 ceph-mon[36505]: pgmap v3647: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:21.240 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6650 sec 2024-03-24T02:15:22.240 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6651 sec 2024-03-24T02:15:22.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:22 smithi078 ceph-mon[32429]: pgmap v3648: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:22.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:22 smithi107 ceph-mon[36485]: pgmap v3648: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:22.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:22 smithi106 ceph-mon[36505]: pgmap v3648: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:23.240 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6652 sec 2024-03-24T02:15:23.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 5998.190740 secs 2024-03-24T02:15:23.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 5998.190740 secs 2024-03-24T02:15:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 5998.190740 secs 2024-03-24T02:15:24.240 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6653 sec 2024-03-24T02:15:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:24 smithi078 ceph-mon[32429]: pgmap v3649: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:24 smithi107 ceph-mon[36485]: pgmap v3649: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:24 smithi106 ceph-mon[36505]: pgmap v3649: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:25.240 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6654 sec 2024-03-24T02:15:26.240 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6655 sec 2024-03-24T02:15:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:26 smithi078 ceph-mon[32429]: pgmap v3650: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:26 smithi107 ceph-mon[36485]: pgmap v3650: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:26.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:26 smithi106 ceph-mon[36505]: pgmap v3650: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:27.240 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6656 sec 2024-03-24T02:15:28.240 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6657 sec 2024-03-24T02:15:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:28 smithi078 ceph-mon[32429]: pgmap v3651: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:28.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6003.190829 secs 2024-03-24T02:15:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:28 smithi107 ceph-mon[36485]: pgmap v3651: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6003.190829 secs 2024-03-24T02:15:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:28 smithi106 ceph-mon[36505]: pgmap v3651: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6003.190829 secs 2024-03-24T02:15:29.240 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6658 sec 2024-03-24T02:15:30.241 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6659 sec 2024-03-24T02:15:30.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:30 smithi078 ceph-mon[32429]: pgmap v3652: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:30 smithi107 ceph-mon[36485]: pgmap v3652: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:30 smithi106 ceph-mon[36505]: pgmap v3652: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:31.241 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6660 sec 2024-03-24T02:15:32.241 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6661 sec 2024-03-24T02:15:32.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:32 smithi078 ceph-mon[32429]: pgmap v3653: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:32 smithi107 ceph-mon[36485]: pgmap v3653: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:32 smithi106 ceph-mon[36505]: pgmap v3653: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:33.241 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6662 sec 2024-03-24T02:15:33.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6008.190919 secs 2024-03-24T02:15:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6008.190919 secs 2024-03-24T02:15:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6008.190919 secs 2024-03-24T02:15:34.241 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6663 sec 2024-03-24T02:15:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:34 smithi078 ceph-mon[32429]: pgmap v3654: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:34 smithi107 ceph-mon[36485]: pgmap v3654: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:34 smithi106 ceph-mon[36505]: pgmap v3654: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:35.241 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6664 sec 2024-03-24T02:15:36.241 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6665 sec 2024-03-24T02:15:36.828 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:36 smithi078 ceph-mon[32429]: pgmap v3655: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:36 smithi107 ceph-mon[36485]: pgmap v3655: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:36.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:36 smithi106 ceph-mon[36505]: pgmap v3655: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:37.241 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6666 sec 2024-03-24T02:15:37.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:15:37.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:15:37.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:15:38.241 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6667 sec 2024-03-24T02:15:38.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:38 smithi078 ceph-mon[32429]: pgmap v3656: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:38.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6013.191005 secs 2024-03-24T02:15:38.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:38 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:15:38.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:38 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:15:38.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:38 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:15:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:38 smithi107 ceph-mon[36485]: pgmap v3656: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6013.191005 secs 2024-03-24T02:15:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:38 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:15:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:38 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:15:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:38 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:15:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:38 smithi106 ceph-mon[36505]: pgmap v3656: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6013.191005 secs 2024-03-24T02:15:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:38 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:15:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:38 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:15:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:38 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:15:39.241 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6668 sec 2024-03-24T02:15:39.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:39 smithi078 ceph-mon[32429]: pgmap v3657: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:15:39.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:39 smithi078 ceph-mon[32429]: pgmap v3658: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:15:39.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:39 smithi107 ceph-mon[36485]: pgmap v3657: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:15:39.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:39 smithi107 ceph-mon[36485]: pgmap v3658: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:15:39.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:39 smithi106 ceph-mon[36505]: pgmap v3657: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:15:39.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:39 smithi106 ceph-mon[36505]: pgmap v3658: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:15:40.241 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6669 sec 2024-03-24T02:15:41.241 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6670 sec 2024-03-24T02:15:41.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:41 smithi078 ceph-mon[32429]: pgmap v3659: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:15:41.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:41 smithi107 ceph-mon[36485]: pgmap v3659: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:15:41.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:41 smithi106 ceph-mon[36505]: pgmap v3659: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:15:42.241 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6671 sec 2024-03-24T02:15:43.241 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6672 sec 2024-03-24T02:15:43.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:43 smithi078 ceph-mon[32429]: pgmap v3660: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:15:43.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6018.191090 secs 2024-03-24T02:15:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:43 smithi107 ceph-mon[36485]: pgmap v3660: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:15:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6018.191090 secs 2024-03-24T02:15:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:43 smithi106 ceph-mon[36505]: pgmap v3660: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:15:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6018.191090 secs 2024-03-24T02:15:44.241 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6673 sec 2024-03-24T02:15:44.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:44 smithi106 ceph-mon[36505]: pgmap v3661: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:15:45.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:44 smithi078 ceph-mon[32429]: pgmap v3661: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:15:45.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:44 smithi107 ceph-mon[36485]: pgmap v3661: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:15:45.242 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6674 sec 2024-03-24T02:15:46.242 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6675 sec 2024-03-24T02:15:47.242 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6676 sec 2024-03-24T02:15:47.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:47 smithi078 ceph-mon[32429]: pgmap v3662: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:15:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:47 smithi107 ceph-mon[36485]: pgmap v3662: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:15:47.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:47 smithi106 ceph-mon[36505]: pgmap v3662: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:15:48.242 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6677 sec 2024-03-24T02:15:49.242 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6678 sec 2024-03-24T02:15:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6023.191179 secs 2024-03-24T02:15:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:49 smithi078 ceph-mon[32429]: pgmap v3663: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:15:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6023.191179 secs 2024-03-24T02:15:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:49 smithi107 ceph-mon[36485]: pgmap v3663: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:15:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6023.191179 secs 2024-03-24T02:15:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:49 smithi106 ceph-mon[36505]: pgmap v3663: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:15:50.242 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6679 sec 2024-03-24T02:15:51.242 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6680 sec 2024-03-24T02:15:51.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:51 smithi078 ceph-mon[32429]: pgmap v3664: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:51.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:51 smithi107 ceph-mon[36485]: pgmap v3664: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:51.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:51 smithi106 ceph-mon[36505]: pgmap v3664: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:52.242 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6681 sec 2024-03-24T02:15:53.242 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6682 sec 2024-03-24T02:15:53.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:53 smithi078 ceph-mon[32429]: pgmap v3665: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:53.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:53 smithi107 ceph-mon[36485]: pgmap v3665: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:53.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:53 smithi106 ceph-mon[36505]: pgmap v3665: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:54.242 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6683 sec 2024-03-24T02:15:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6028.191238 secs 2024-03-24T02:15:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6028.191238 secs 2024-03-24T02:15:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6028.191238 secs 2024-03-24T02:15:55.242 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6684 sec 2024-03-24T02:15:55.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:55 smithi078 ceph-mon[32429]: pgmap v3666: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:55.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:55 smithi107 ceph-mon[36485]: pgmap v3666: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:55.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:55 smithi106 ceph-mon[36505]: pgmap v3666: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:56.243 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6685 sec 2024-03-24T02:15:57.243 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6686 sec 2024-03-24T02:15:57.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:57 smithi078 ceph-mon[32429]: pgmap v3667: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:57.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:57 smithi107 ceph-mon[36485]: pgmap v3667: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:57.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:57 smithi106 ceph-mon[36505]: pgmap v3667: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:58.243 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6687 sec 2024-03-24T02:15:59.243 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6688 sec 2024-03-24T02:15:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6033.191342 secs 2024-03-24T02:15:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:15:59 smithi078 ceph-mon[32429]: pgmap v3668: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6033.191342 secs 2024-03-24T02:15:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:15:59 smithi107 ceph-mon[36485]: pgmap v3668: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:15:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6033.191342 secs 2024-03-24T02:15:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:15:59 smithi106 ceph-mon[36505]: pgmap v3668: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:00.243 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6689 sec 2024-03-24T02:16:01.243 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6690 sec 2024-03-24T02:16:01.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:01 smithi078 ceph-mon[32429]: pgmap v3669: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:01.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:01 smithi107 ceph-mon[36485]: pgmap v3669: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:01.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:01 smithi106 ceph-mon[36505]: pgmap v3669: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:02.243 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6691 sec 2024-03-24T02:16:03.243 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6692 sec 2024-03-24T02:16:03.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:03 smithi078 ceph-mon[32429]: pgmap v3670: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:03.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:03 smithi107 ceph-mon[36485]: pgmap v3670: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:03.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:03 smithi106 ceph-mon[36505]: pgmap v3670: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:04.243 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6693 sec 2024-03-24T02:16:04.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6038.191401 secs 2024-03-24T02:16:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6038.191401 secs 2024-03-24T02:16:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6038.191401 secs 2024-03-24T02:16:05.243 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6694 sec 2024-03-24T02:16:05.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:05 smithi078 ceph-mon[32429]: pgmap v3671: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:05.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:05 smithi107 ceph-mon[36485]: pgmap v3671: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:05.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:05 smithi106 ceph-mon[36505]: pgmap v3671: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:06.244 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6695 sec 2024-03-24T02:16:07.243 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6696 sec 2024-03-24T02:16:07.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:07 smithi078 ceph-mon[32429]: pgmap v3672: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:07.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:07 smithi107 ceph-mon[36485]: pgmap v3672: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:07.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:07 smithi106 ceph-mon[36505]: pgmap v3672: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:08.244 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6697 sec 2024-03-24T02:16:09.244 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6698 sec 2024-03-24T02:16:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6043.191489 secs 2024-03-24T02:16:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:09 smithi078 ceph-mon[32429]: pgmap v3673: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6043.191489 secs 2024-03-24T02:16:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:09 smithi107 ceph-mon[36485]: pgmap v3673: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6043.191489 secs 2024-03-24T02:16:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:09 smithi106 ceph-mon[36505]: pgmap v3673: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:10.244 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6699 sec 2024-03-24T02:16:11.244 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6700 sec 2024-03-24T02:16:11.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:11 smithi078 ceph-mon[32429]: pgmap v3674: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:11.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:11 smithi107 ceph-mon[36485]: pgmap v3674: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:11.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:11 smithi106 ceph-mon[36505]: pgmap v3674: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:12.244 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6701 sec 2024-03-24T02:16:13.244 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6702 sec 2024-03-24T02:16:13.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:13 smithi078 ceph-mon[32429]: pgmap v3675: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:13.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:13 smithi107 ceph-mon[36485]: pgmap v3675: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:13.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:13 smithi106 ceph-mon[36505]: pgmap v3675: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:14.244 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6703 sec 2024-03-24T02:16:14.553 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6048.191554 secs 2024-03-24T02:16:14.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6048.191554 secs 2024-03-24T02:16:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6048.191554 secs 2024-03-24T02:16:15.244 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6704 sec 2024-03-24T02:16:15.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:15 smithi078 ceph-mon[32429]: pgmap v3676: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:15.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:15 smithi107 ceph-mon[36485]: pgmap v3676: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:15.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:15 smithi106 ceph-mon[36505]: pgmap v3676: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:16.244 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6705 sec 2024-03-24T02:16:17.244 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6706 sec 2024-03-24T02:16:17.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:17 smithi078 ceph-mon[32429]: pgmap v3677: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:17.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:17 smithi107 ceph-mon[36485]: pgmap v3677: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:17.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:17 smithi106 ceph-mon[36505]: pgmap v3677: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:18.244 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6707 sec 2024-03-24T02:16:19.245 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6708 sec 2024-03-24T02:16:19.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6053.191635 secs 2024-03-24T02:16:19.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:19 smithi078 ceph-mon[32429]: pgmap v3678: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6053.191635 secs 2024-03-24T02:16:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:19 smithi107 ceph-mon[36485]: pgmap v3678: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6053.191635 secs 2024-03-24T02:16:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:19 smithi106 ceph-mon[36505]: pgmap v3678: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:20.245 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6709 sec 2024-03-24T02:16:21.245 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6710 sec 2024-03-24T02:16:21.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:21 smithi078 ceph-mon[32429]: pgmap v3679: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:21.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:21 smithi107 ceph-mon[36485]: pgmap v3679: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:21.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:21 smithi106 ceph-mon[36505]: pgmap v3679: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:22.245 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6711 sec 2024-03-24T02:16:23.245 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6712 sec 2024-03-24T02:16:23.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:23 smithi078 ceph-mon[32429]: pgmap v3680: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:23.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:23 smithi107 ceph-mon[36485]: pgmap v3680: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:23.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:23 smithi106 ceph-mon[36505]: pgmap v3680: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:24.245 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6713 sec 2024-03-24T02:16:24.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6058.191709 secs 2024-03-24T02:16:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6058.191709 secs 2024-03-24T02:16:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6058.191709 secs 2024-03-24T02:16:25.245 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6714 sec 2024-03-24T02:16:25.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:25 smithi078 ceph-mon[32429]: pgmap v3681: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:25.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:25 smithi107 ceph-mon[36485]: pgmap v3681: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:25.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:25 smithi106 ceph-mon[36505]: pgmap v3681: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:26.245 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6715 sec 2024-03-24T02:16:27.246 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6716 sec 2024-03-24T02:16:27.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:27 smithi078 ceph-mon[32429]: pgmap v3682: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:27.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:27 smithi107 ceph-mon[36485]: pgmap v3682: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:27.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:27 smithi106 ceph-mon[36505]: pgmap v3682: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:28.245 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6717 sec 2024-03-24T02:16:29.245 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6718 sec 2024-03-24T02:16:29.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6063.191798 secs 2024-03-24T02:16:29.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:29 smithi078 ceph-mon[32429]: pgmap v3683: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6063.191798 secs 2024-03-24T02:16:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:29 smithi107 ceph-mon[36485]: pgmap v3683: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:29.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6063.191798 secs 2024-03-24T02:16:29.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:29 smithi106 ceph-mon[36505]: pgmap v3683: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:30.246 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6719 sec 2024-03-24T02:16:31.246 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6720 sec 2024-03-24T02:16:31.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:31 smithi078 ceph-mon[32429]: pgmap v3684: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:31.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:31 smithi107 ceph-mon[36485]: pgmap v3684: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:31.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:31 smithi106 ceph-mon[36505]: pgmap v3684: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:32.246 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6721 sec 2024-03-24T02:16:33.246 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6722 sec 2024-03-24T02:16:33.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:33 smithi078 ceph-mon[32429]: pgmap v3685: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:33.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:33 smithi107 ceph-mon[36485]: pgmap v3685: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:33.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:33 smithi106 ceph-mon[36505]: pgmap v3685: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:34.246 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6723 sec 2024-03-24T02:16:34.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6068.191883 secs 2024-03-24T02:16:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6068.191883 secs 2024-03-24T02:16:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6068.191883 secs 2024-03-24T02:16:35.246 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6724 sec 2024-03-24T02:16:35.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:35 smithi078 ceph-mon[32429]: pgmap v3686: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:35.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:35 smithi107 ceph-mon[36485]: pgmap v3686: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:35.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:35 smithi106 ceph-mon[36505]: pgmap v3686: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:36.246 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6725 sec 2024-03-24T02:16:37.246 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6726 sec 2024-03-24T02:16:37.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:37 smithi078 ceph-mon[32429]: pgmap v3687: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:37.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:37 smithi107 ceph-mon[36485]: pgmap v3687: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:37.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:37 smithi106 ceph-mon[36505]: pgmap v3687: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:38.246 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6727 sec 2024-03-24T02:16:38.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:38 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:16:38.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:38 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:16:38.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:38 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:16:39.246 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6728 sec 2024-03-24T02:16:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6073.191970 secs 2024-03-24T02:16:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:39 smithi106 ceph-mon[36505]: pgmap v3688: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:39 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:16:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:39 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:16:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:39 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:16:39.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6073.191970 secs 2024-03-24T02:16:39.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:39 smithi078 ceph-mon[32429]: pgmap v3688: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:39.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:39 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:16:39.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:39 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:16:39.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:39 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:16:39.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6073.191970 secs 2024-03-24T02:16:39.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:39 smithi107 ceph-mon[36485]: pgmap v3688: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:39.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:39 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:16:39.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:39 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:16:39.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:39 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:16:40.247 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6729 sec 2024-03-24T02:16:40.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:40 smithi106 ceph-mon[36505]: pgmap v3689: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:16:40.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:40 smithi078 ceph-mon[32429]: pgmap v3689: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:16:40.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:40 smithi107 ceph-mon[36485]: pgmap v3689: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:16:41.246 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6730 sec 2024-03-24T02:16:42.247 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6731 sec 2024-03-24T02:16:42.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:42 smithi106 ceph-mon[36505]: pgmap v3690: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:16:42.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:42 smithi078 ceph-mon[32429]: pgmap v3690: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:16:42.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:42 smithi107 ceph-mon[36485]: pgmap v3690: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:16:43.247 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6732 sec 2024-03-24T02:16:44.247 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6733 sec 2024-03-24T02:16:44.642 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:44 smithi078 ceph-mon[32429]: pgmap v3691: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:16:44.642 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6078.192056 secs 2024-03-24T02:16:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:44 smithi106 ceph-mon[36505]: pgmap v3691: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:16:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6078.192056 secs 2024-03-24T02:16:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:44 smithi107 ceph-mon[36485]: pgmap v3691: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:16:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6078.192056 secs 2024-03-24T02:16:45.247 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6734 sec 2024-03-24T02:16:46.247 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6735 sec 2024-03-24T02:16:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:46 smithi106 ceph-mon[36505]: pgmap v3692: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:16:46.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:46 smithi078 ceph-mon[32429]: pgmap v3692: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:16:46.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:46 smithi107 ceph-mon[36485]: pgmap v3692: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:16:47.247 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6736 sec 2024-03-24T02:16:48.247 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6737 sec 2024-03-24T02:16:48.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:48 smithi106 ceph-mon[36505]: pgmap v3693: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:16:48.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:48 smithi078 ceph-mon[32429]: pgmap v3693: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:16:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:48 smithi107 ceph-mon[36485]: pgmap v3693: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:16:49.247 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6738 sec 2024-03-24T02:16:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6083.192091 secs 2024-03-24T02:16:49.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6083.192091 secs 2024-03-24T02:16:49.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6083.192091 secs 2024-03-24T02:16:50.247 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6739 sec 2024-03-24T02:16:50.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:50 smithi106 ceph-mon[36505]: pgmap v3694: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:16:50.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:50 smithi078 ceph-mon[32429]: pgmap v3694: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:16:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:50 smithi107 ceph-mon[36485]: pgmap v3694: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:16:51.247 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6740 sec 2024-03-24T02:16:52.247 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6741 sec 2024-03-24T02:16:52.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:52 smithi106 ceph-mon[36505]: pgmap v3695: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:52.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:52 smithi078 ceph-mon[32429]: pgmap v3695: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:52.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:52 smithi107 ceph-mon[36485]: pgmap v3695: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:53.248 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6742 sec 2024-03-24T02:16:54.248 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6743 sec 2024-03-24T02:16:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:54 smithi106 ceph-mon[36505]: pgmap v3696: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6088.192178 secs 2024-03-24T02:16:54.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:54 smithi078 ceph-mon[32429]: pgmap v3696: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:54.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6088.192178 secs 2024-03-24T02:16:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:54 smithi107 ceph-mon[36485]: pgmap v3696: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6088.192178 secs 2024-03-24T02:16:55.248 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6744 sec 2024-03-24T02:16:56.248 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6745 sec 2024-03-24T02:16:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:56 smithi106 ceph-mon[36505]: pgmap v3697: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:56 smithi078 ceph-mon[32429]: pgmap v3697: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:56 smithi107 ceph-mon[36485]: pgmap v3697: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:16:57.248 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6746 sec 2024-03-24T02:16:58.248 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6747 sec 2024-03-24T02:16:58.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:58 smithi106 ceph-mon[36505]: pgmap v3698: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:58.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:58 smithi078 ceph-mon[32429]: pgmap v3698: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:58 smithi107 ceph-mon[36485]: pgmap v3698: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:59.248 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6748 sec 2024-03-24T02:16:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:16:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6093.192264 secs 2024-03-24T02:16:59.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:16:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6093.192264 secs 2024-03-24T02:16:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:16:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6093.192264 secs 2024-03-24T02:17:00.248 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6749 sec 2024-03-24T02:17:00.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:00 smithi106 ceph-mon[36505]: pgmap v3699: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:00.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:00 smithi078 ceph-mon[32429]: pgmap v3699: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:00 smithi107 ceph-mon[36485]: pgmap v3699: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:01.248 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6750 sec 2024-03-24T02:17:02.249 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6751 sec 2024-03-24T02:17:02.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:02 smithi106 ceph-mon[36505]: pgmap v3700: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:02.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:02 smithi078 ceph-mon[32429]: pgmap v3700: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:02 smithi107 ceph-mon[36485]: pgmap v3700: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:03.249 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6752 sec 2024-03-24T02:17:04.249 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6753 sec 2024-03-24T02:17:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6098.192325 secs 2024-03-24T02:17:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:04 smithi106 ceph-mon[36505]: pgmap v3701: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6098.192325 secs 2024-03-24T02:17:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:04 smithi078 ceph-mon[32429]: pgmap v3701: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6098.192325 secs 2024-03-24T02:17:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:04 smithi107 ceph-mon[36485]: pgmap v3701: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:05.249 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6754 sec 2024-03-24T02:17:06.249 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6755 sec 2024-03-24T02:17:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:06 smithi078 ceph-mon[32429]: pgmap v3702: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:06 smithi107 ceph-mon[36485]: pgmap v3702: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:06 smithi106 ceph-mon[36505]: pgmap v3702: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:07.249 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6756 sec 2024-03-24T02:17:08.249 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6757 sec 2024-03-24T02:17:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:08 smithi078 ceph-mon[32429]: pgmap v3703: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:08 smithi107 ceph-mon[36485]: pgmap v3703: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:08 smithi106 ceph-mon[36505]: pgmap v3703: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:09.249 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6758 sec 2024-03-24T02:17:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6103.192420 secs 2024-03-24T02:17:09.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6103.192420 secs 2024-03-24T02:17:09.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6103.192420 secs 2024-03-24T02:17:10.249 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6759 sec 2024-03-24T02:17:10.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:10 smithi078 ceph-mon[32429]: pgmap v3704: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:10 smithi107 ceph-mon[36485]: pgmap v3704: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:10 smithi106 ceph-mon[36505]: pgmap v3704: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:11.249 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6760 sec 2024-03-24T02:17:12.249 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6761 sec 2024-03-24T02:17:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:12 smithi078 ceph-mon[32429]: pgmap v3705: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:12 smithi107 ceph-mon[36485]: pgmap v3705: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:12.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:12 smithi106 ceph-mon[36505]: pgmap v3705: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:13.249 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6762 sec 2024-03-24T02:17:14.249 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6763 sec 2024-03-24T02:17:14.704 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6108.192497 secs 2024-03-24T02:17:14.704 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:14 smithi106 ceph-mon[36505]: pgmap v3706: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6108.192497 secs 2024-03-24T02:17:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:14 smithi078 ceph-mon[32429]: pgmap v3706: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6108.192497 secs 2024-03-24T02:17:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:14 smithi107 ceph-mon[36485]: pgmap v3706: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:15.250 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6764 sec 2024-03-24T02:17:16.250 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6765 sec 2024-03-24T02:17:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:16 smithi078 ceph-mon[32429]: pgmap v3707: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:16 smithi107 ceph-mon[36485]: pgmap v3707: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:16.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:16 smithi106 ceph-mon[36505]: pgmap v3707: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:17.250 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6766 sec 2024-03-24T02:17:18.250 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6767 sec 2024-03-24T02:17:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:18 smithi078 ceph-mon[32429]: pgmap v3708: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:18 smithi107 ceph-mon[36485]: pgmap v3708: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:18 smithi106 ceph-mon[36505]: pgmap v3708: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:19.250 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6768 sec 2024-03-24T02:17:19.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6113.192557 secs 2024-03-24T02:17:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6113.192557 secs 2024-03-24T02:17:19.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6113.192557 secs 2024-03-24T02:17:20.250 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6769 sec 2024-03-24T02:17:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:20 smithi078 ceph-mon[32429]: pgmap v3709: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:20 smithi107 ceph-mon[36485]: pgmap v3709: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:20.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:20 smithi106 ceph-mon[36505]: pgmap v3709: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:21.250 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6770 sec 2024-03-24T02:17:22.250 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6771 sec 2024-03-24T02:17:22.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:22 smithi078 ceph-mon[32429]: pgmap v3710: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:22.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:22 smithi107 ceph-mon[36485]: pgmap v3710: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:22.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:22 smithi106 ceph-mon[36505]: pgmap v3710: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:23.250 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6772 sec 2024-03-24T02:17:24.250 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6773 sec 2024-03-24T02:17:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6118.192619 secs 2024-03-24T02:17:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:24 smithi078 ceph-mon[32429]: pgmap v3711: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6118.192619 secs 2024-03-24T02:17:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:24 smithi107 ceph-mon[36485]: pgmap v3711: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6118.192619 secs 2024-03-24T02:17:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:24 smithi106 ceph-mon[36505]: pgmap v3711: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:25.251 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6774 sec 2024-03-24T02:17:26.251 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6775 sec 2024-03-24T02:17:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:26 smithi078 ceph-mon[32429]: pgmap v3712: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:26 smithi107 ceph-mon[36485]: pgmap v3712: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:26.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:26 smithi106 ceph-mon[36505]: pgmap v3712: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:27.251 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6776 sec 2024-03-24T02:17:28.251 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6777 sec 2024-03-24T02:17:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:28 smithi078 ceph-mon[32429]: pgmap v3713: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:28 smithi107 ceph-mon[36485]: pgmap v3713: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:28 smithi106 ceph-mon[36505]: pgmap v3713: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:29.251 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6778 sec 2024-03-24T02:17:29.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6123.192637 secs 2024-03-24T02:17:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6123.192637 secs 2024-03-24T02:17:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6123.192637 secs 2024-03-24T02:17:30.251 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6779 sec 2024-03-24T02:17:30.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:30 smithi078 ceph-mon[32429]: pgmap v3714: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:30 smithi107 ceph-mon[36485]: pgmap v3714: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:30 smithi106 ceph-mon[36505]: pgmap v3714: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:31.251 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6780 sec 2024-03-24T02:17:32.251 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6781 sec 2024-03-24T02:17:32.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:32 smithi078 ceph-mon[32429]: pgmap v3715: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:32.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:32 smithi107 ceph-mon[36485]: pgmap v3715: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:32 smithi106 ceph-mon[36505]: pgmap v3715: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:33.251 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6782 sec 2024-03-24T02:17:34.251 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6783 sec 2024-03-24T02:17:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6128.192726 secs 2024-03-24T02:17:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:34 smithi078 ceph-mon[32429]: pgmap v3716: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6128.192726 secs 2024-03-24T02:17:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:34 smithi107 ceph-mon[36485]: pgmap v3716: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6128.192726 secs 2024-03-24T02:17:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:34 smithi106 ceph-mon[36505]: pgmap v3716: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:35.252 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6784 sec 2024-03-24T02:17:36.252 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6785 sec 2024-03-24T02:17:36.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:36 smithi078 ceph-mon[32429]: pgmap v3717: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:36 smithi107 ceph-mon[36485]: pgmap v3717: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:36.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:36 smithi106 ceph-mon[36505]: pgmap v3717: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:37.252 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6786 sec 2024-03-24T02:17:38.252 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6787 sec 2024-03-24T02:17:38.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:38 smithi078 ceph-mon[32429]: pgmap v3718: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:38 smithi107 ceph-mon[36485]: pgmap v3718: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:38 smithi106 ceph-mon[36505]: pgmap v3718: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:39.252 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6788 sec 2024-03-24T02:17:39.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6133.192816 secs 2024-03-24T02:17:39.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:39 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:17:39.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6133.192816 secs 2024-03-24T02:17:39.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:39 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:17:39.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6133.192816 secs 2024-03-24T02:17:39.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:39 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:17:40.252 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6789 sec 2024-03-24T02:17:40.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:40 smithi078 ceph-mon[32429]: pgmap v3719: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:40.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:40 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:17:40.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:40 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:17:40.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:40 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:17:40.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:40 smithi107 ceph-mon[36485]: pgmap v3719: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:40.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:40 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:17:40.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:40 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:17:40.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:40 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:17:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:40 smithi106 ceph-mon[36505]: pgmap v3719: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:40 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:17:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:40 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:17:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:40 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:17:41.252 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6790 sec 2024-03-24T02:17:41.756 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:41 smithi078 ceph-mon[32429]: pgmap v3720: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:17:41.756 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:41 smithi078 ceph-mon[32429]: pgmap v3721: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:17:41.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:41 smithi107 ceph-mon[36485]: pgmap v3720: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:17:41.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:41 smithi107 ceph-mon[36485]: pgmap v3721: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:17:41.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:41 smithi106 ceph-mon[36505]: pgmap v3720: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:17:41.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:41 smithi106 ceph-mon[36505]: pgmap v3721: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:17:42.252 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6791 sec 2024-03-24T02:17:43.252 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6792 sec 2024-03-24T02:17:43.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:43 smithi078 ceph-mon[32429]: pgmap v3722: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:17:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:43 smithi107 ceph-mon[36485]: pgmap v3722: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:17:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:43 smithi106 ceph-mon[36505]: pgmap v3722: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:17:44.252 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6793 sec 2024-03-24T02:17:44.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6138.192895 secs 2024-03-24T02:17:44.802 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6138.192895 secs 2024-03-24T02:17:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6138.192895 secs 2024-03-24T02:17:45.252 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6794 sec 2024-03-24T02:17:45.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:45 smithi078 ceph-mon[32429]: pgmap v3723: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:17:45.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:45 smithi107 ceph-mon[36485]: pgmap v3723: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:17:45.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:45 smithi106 ceph-mon[36505]: pgmap v3723: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:17:46.252 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6795 sec 2024-03-24T02:17:47.253 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6796 sec 2024-03-24T02:17:47.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:47 smithi078 ceph-mon[32429]: pgmap v3724: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:17:47.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:47 smithi107 ceph-mon[36485]: pgmap v3724: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:17:47.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:47 smithi106 ceph-mon[36505]: pgmap v3724: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:17:48.253 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6797 sec 2024-03-24T02:17:49.253 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6798 sec 2024-03-24T02:17:49.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:49 smithi078 ceph-mon[32429]: pgmap v3725: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:17:49.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6143.192953 secs 2024-03-24T02:17:49.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:49 smithi107 ceph-mon[36485]: pgmap v3725: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:17:49.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6143.192953 secs 2024-03-24T02:17:49.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:49 smithi106 ceph-mon[36505]: pgmap v3725: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:17:49.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6143.192953 secs 2024-03-24T02:17:50.253 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6799 sec 2024-03-24T02:17:51.253 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6800 sec 2024-03-24T02:17:51.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:51 smithi078 ceph-mon[32429]: pgmap v3726: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:17:51.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:51 smithi107 ceph-mon[36485]: pgmap v3726: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:17:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:51 smithi106 ceph-mon[36505]: pgmap v3726: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:17:52.253 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6801 sec 2024-03-24T02:17:53.254 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6802 sec 2024-03-24T02:17:53.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:53 smithi078 ceph-mon[32429]: pgmap v3727: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:53 smithi107 ceph-mon[36485]: pgmap v3727: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:53.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:53 smithi106 ceph-mon[36505]: pgmap v3727: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:54.253 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6803 sec 2024-03-24T02:17:54.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6148.193063 secs 2024-03-24T02:17:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6148.193063 secs 2024-03-24T02:17:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6148.193063 secs 2024-03-24T02:17:55.253 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6804 sec 2024-03-24T02:17:55.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:55 smithi078 ceph-mon[32429]: pgmap v3728: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:55.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:55 smithi107 ceph-mon[36485]: pgmap v3728: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:55.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:55 smithi106 ceph-mon[36505]: pgmap v3728: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:56.253 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6805 sec 2024-03-24T02:17:57.253 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6806 sec 2024-03-24T02:17:57.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:57 smithi078 ceph-mon[32429]: pgmap v3729: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:57.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:57 smithi107 ceph-mon[36485]: pgmap v3729: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:57.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:57 smithi106 ceph-mon[36505]: pgmap v3729: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:58.253 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6807 sec 2024-03-24T02:17:59.254 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6808 sec 2024-03-24T02:17:59.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:59 smithi078 ceph-mon[32429]: pgmap v3730: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:59.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:17:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6153.193146 secs 2024-03-24T02:17:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:59 smithi107 ceph-mon[36485]: pgmap v3730: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:17:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6153.193146 secs 2024-03-24T02:17:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:59 smithi106 ceph-mon[36505]: pgmap v3730: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:17:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:17:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6153.193146 secs 2024-03-24T02:18:00.254 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6809 sec 2024-03-24T02:18:01.254 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6810 sec 2024-03-24T02:18:01.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:01 smithi078 ceph-mon[32429]: pgmap v3731: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:01.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:01 smithi107 ceph-mon[36485]: pgmap v3731: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:01.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:01 smithi106 ceph-mon[36505]: pgmap v3731: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:02.254 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6811 sec 2024-03-24T02:18:03.254 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6812 sec 2024-03-24T02:18:03.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:03 smithi078 ceph-mon[32429]: pgmap v3732: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:03 smithi107 ceph-mon[36485]: pgmap v3732: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:03 smithi106 ceph-mon[36505]: pgmap v3732: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:04.254 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6813 sec 2024-03-24T02:18:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6158.193230 secs 2024-03-24T02:18:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6158.193230 secs 2024-03-24T02:18:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6158.193230 secs 2024-03-24T02:18:05.254 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6814 sec 2024-03-24T02:18:05.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:05 smithi078 ceph-mon[32429]: pgmap v3733: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:05.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:05 smithi107 ceph-mon[36485]: pgmap v3733: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:05.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:05 smithi106 ceph-mon[36505]: pgmap v3733: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:06.254 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6815 sec 2024-03-24T02:18:07.254 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6816 sec 2024-03-24T02:18:07.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:07 smithi078 ceph-mon[32429]: pgmap v3734: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:07.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:07 smithi107 ceph-mon[36485]: pgmap v3734: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:07.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:07 smithi106 ceph-mon[36505]: pgmap v3734: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:08.254 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6817 sec 2024-03-24T02:18:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6163.193313 secs 2024-03-24T02:18:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6163.193313 secs 2024-03-24T02:18:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6163.193313 secs 2024-03-24T02:18:09.254 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6818 sec 2024-03-24T02:18:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:09 smithi078 ceph-mon[32429]: pgmap v3735: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:09.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:09 smithi107 ceph-mon[36485]: pgmap v3735: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:09.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:09 smithi106 ceph-mon[36505]: pgmap v3735: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:10.255 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6819 sec 2024-03-24T02:18:11.255 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6820 sec 2024-03-24T02:18:11.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:11 smithi078 ceph-mon[32429]: pgmap v3736: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:11 smithi107 ceph-mon[36485]: pgmap v3736: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:11.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:11 smithi106 ceph-mon[36505]: pgmap v3736: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:12.254 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6821 sec 2024-03-24T02:18:13.255 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6822 sec 2024-03-24T02:18:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:13 smithi078 ceph-mon[32429]: pgmap v3737: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6168.193397 secs 2024-03-24T02:18:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:13 smithi107 ceph-mon[36485]: pgmap v3737: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6168.193397 secs 2024-03-24T02:18:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:13 smithi106 ceph-mon[36505]: pgmap v3737: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6168.193397 secs 2024-03-24T02:18:14.255 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6823 sec 2024-03-24T02:18:15.255 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6824 sec 2024-03-24T02:18:15.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:15 smithi078 ceph-mon[32429]: pgmap v3738: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:15.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:15 smithi107 ceph-mon[36485]: pgmap v3738: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:15 smithi106 ceph-mon[36505]: pgmap v3738: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:16.255 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6825 sec 2024-03-24T02:18:17.255 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6826 sec 2024-03-24T02:18:17.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:17 smithi078 ceph-mon[32429]: pgmap v3739: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:17.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:17 smithi107 ceph-mon[36485]: pgmap v3739: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:17.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:17 smithi106 ceph-mon[36505]: pgmap v3739: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:18.255 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6827 sec 2024-03-24T02:18:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6173.193482 secs 2024-03-24T02:18:18.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6173.193482 secs 2024-03-24T02:18:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6173.193482 secs 2024-03-24T02:18:19.255 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6828 sec 2024-03-24T02:18:19.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:19 smithi078 ceph-mon[32429]: pgmap v3740: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:19 smithi107 ceph-mon[36485]: pgmap v3740: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:19.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:19 smithi106 ceph-mon[36505]: pgmap v3740: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:20.255 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6829 sec 2024-03-24T02:18:21.255 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6830 sec 2024-03-24T02:18:21.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:21 smithi078 ceph-mon[32429]: pgmap v3741: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:21 smithi107 ceph-mon[36485]: pgmap v3741: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:21.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:21 smithi106 ceph-mon[36505]: pgmap v3741: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:22.255 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6831 sec 2024-03-24T02:18:23.256 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6832 sec 2024-03-24T02:18:23.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:23 smithi078 ceph-mon[32429]: pgmap v3742: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:23.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6178.193567 secs 2024-03-24T02:18:23.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:23 smithi107 ceph-mon[36485]: pgmap v3742: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:23.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6178.193567 secs 2024-03-24T02:18:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:23 smithi106 ceph-mon[36505]: pgmap v3742: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6178.193567 secs 2024-03-24T02:18:24.256 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6833 sec 2024-03-24T02:18:25.256 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6834 sec 2024-03-24T02:18:25.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:25 smithi078 ceph-mon[32429]: pgmap v3743: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:25.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:25 smithi107 ceph-mon[36485]: pgmap v3743: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:25.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:25 smithi106 ceph-mon[36505]: pgmap v3743: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:26.256 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6835 sec 2024-03-24T02:18:27.256 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6836 sec 2024-03-24T02:18:27.835 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:27 smithi078 ceph-mon[32429]: pgmap v3744: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:27.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:27 smithi107 ceph-mon[36485]: pgmap v3744: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:27.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:27 smithi106 ceph-mon[36505]: pgmap v3744: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:28.256 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6837 sec 2024-03-24T02:18:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6183.193629 secs 2024-03-24T02:18:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6183.193629 secs 2024-03-24T02:18:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6183.193629 secs 2024-03-24T02:18:29.256 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6838 sec 2024-03-24T02:18:29.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:29 smithi078 ceph-mon[32429]: pgmap v3745: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:29 smithi107 ceph-mon[36485]: pgmap v3745: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:29 smithi106 ceph-mon[36505]: pgmap v3745: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:30.256 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6839 sec 2024-03-24T02:18:31.256 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6840 sec 2024-03-24T02:18:31.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:31 smithi078 ceph-mon[32429]: pgmap v3746: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:31.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:31 smithi107 ceph-mon[36485]: pgmap v3746: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:31.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:31 smithi106 ceph-mon[36505]: pgmap v3746: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:32.257 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6841 sec 2024-03-24T02:18:33.257 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6842 sec 2024-03-24T02:18:33.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:33 smithi078 ceph-mon[32429]: pgmap v3747: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:33.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6188.193712 secs 2024-03-24T02:18:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:33 smithi107 ceph-mon[36485]: pgmap v3747: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6188.193712 secs 2024-03-24T02:18:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:33 smithi106 ceph-mon[36505]: pgmap v3747: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6188.193712 secs 2024-03-24T02:18:34.257 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6843 sec 2024-03-24T02:18:35.257 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6844 sec 2024-03-24T02:18:35.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:35 smithi078 ceph-mon[32429]: pgmap v3748: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:35.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:35 smithi107 ceph-mon[36485]: pgmap v3748: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:35.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:35 smithi106 ceph-mon[36505]: pgmap v3748: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:36.257 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6845 sec 2024-03-24T02:18:37.257 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6846 sec 2024-03-24T02:18:37.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:37 smithi078 ceph-mon[32429]: pgmap v3749: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:37.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:37 smithi107 ceph-mon[36485]: pgmap v3749: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:37.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:37 smithi106 ceph-mon[36505]: pgmap v3749: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:38.257 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6847 sec 2024-03-24T02:18:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6193.193753 secs 2024-03-24T02:18:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6193.193753 secs 2024-03-24T02:18:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6193.193753 secs 2024-03-24T02:18:39.257 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6848 sec 2024-03-24T02:18:39.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:39 smithi106 ceph-mon[36505]: pgmap v3750: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:39 smithi078 ceph-mon[32429]: pgmap v3750: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:40.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:39 smithi107 ceph-mon[36485]: pgmap v3750: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:40.257 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6849 sec 2024-03-24T02:18:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:40 smithi106 ceph-mon[36505]: pgmap v3751: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:40 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:18:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:40 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:18:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:40 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:18:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:40 smithi106 ceph-mon[36505]: pgmap v3752: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:18:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:40 smithi106 ceph-mon[36505]: pgmap v3753: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:18:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:40 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:18:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:40 smithi078 ceph-mon[32429]: pgmap v3751: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:40 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:18:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:40 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:18:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:40 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:18:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:40 smithi078 ceph-mon[32429]: pgmap v3752: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:18:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:40 smithi078 ceph-mon[32429]: pgmap v3753: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:18:41.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:40 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:18:41.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:40 smithi107 ceph-mon[36485]: pgmap v3751: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:41.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:40 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:18:41.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:40 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:18:41.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:40 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:18:41.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:40 smithi107 ceph-mon[36485]: pgmap v3752: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:18:41.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:40 smithi107 ceph-mon[36485]: pgmap v3753: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:18:41.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:40 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:18:41.257 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6850 sec 2024-03-24T02:18:42.257 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6851 sec 2024-03-24T02:18:42.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:42 smithi106 ceph-mon[36505]: pgmap v3754: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:18:43.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:42 smithi078 ceph-mon[32429]: pgmap v3754: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:18:43.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:42 smithi107 ceph-mon[36485]: pgmap v3754: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:18:43.258 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6852 sec 2024-03-24T02:18:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6198.193842 secs 2024-03-24T02:18:43.991 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6198.193842 secs 2024-03-24T02:18:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6198.193842 secs 2024-03-24T02:18:44.258 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6853 sec 2024-03-24T02:18:45.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:44 smithi078 ceph-mon[32429]: pgmap v3755: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:18:45.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:44 smithi107 ceph-mon[36485]: pgmap v3755: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:18:45.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:44 smithi106 ceph-mon[36505]: pgmap v3755: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:18:45.258 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6854 sec 2024-03-24T02:18:46.258 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6855 sec 2024-03-24T02:18:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:46 smithi106 ceph-mon[36505]: pgmap v3756: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:18:47.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:46 smithi078 ceph-mon[32429]: pgmap v3756: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:18:47.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:46 smithi107 ceph-mon[36485]: pgmap v3756: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:18:47.258 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6856 sec 2024-03-24T02:18:48.258 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6857 sec 2024-03-24T02:18:48.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6203.193929 secs 2024-03-24T02:18:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6203.193929 secs 2024-03-24T02:18:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6203.193929 secs 2024-03-24T02:18:49.258 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6858 sec 2024-03-24T02:18:49.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:49 smithi078 ceph-mon[32429]: pgmap v3757: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:18:49.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:49 smithi107 ceph-mon[36485]: pgmap v3757: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:18:49.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:49 smithi106 ceph-mon[36505]: pgmap v3757: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:18:50.258 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6859 sec 2024-03-24T02:18:51.258 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6860 sec 2024-03-24T02:18:51.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:51 smithi078 ceph-mon[32429]: pgmap v3758: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:18:51.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:51 smithi107 ceph-mon[36485]: pgmap v3758: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:18:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:51 smithi106 ceph-mon[36505]: pgmap v3758: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:18:52.258 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6861 sec 2024-03-24T02:18:53.258 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6862 sec 2024-03-24T02:18:53.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:53 smithi078 ceph-mon[32429]: pgmap v3759: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:53.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6208.193990 secs 2024-03-24T02:18:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:53 smithi107 ceph-mon[36485]: pgmap v3759: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6208.193990 secs 2024-03-24T02:18:53.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:53 smithi106 ceph-mon[36505]: pgmap v3759: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:53.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6208.193990 secs 2024-03-24T02:18:54.258 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6863 sec 2024-03-24T02:18:55.259 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6864 sec 2024-03-24T02:18:55.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:55 smithi078 ceph-mon[32429]: pgmap v3760: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:55.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:55 smithi107 ceph-mon[36485]: pgmap v3760: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:55.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:55 smithi106 ceph-mon[36505]: pgmap v3760: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:56.259 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6865 sec 2024-03-24T02:18:57.259 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6866 sec 2024-03-24T02:18:57.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:57 smithi106 ceph-mon[36505]: pgmap v3761: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:58.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:57 smithi078 ceph-mon[32429]: pgmap v3761: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:58.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:57 smithi107 ceph-mon[36485]: pgmap v3761: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:58.259 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6867 sec 2024-03-24T02:18:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6213.194089 secs 2024-03-24T02:18:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6213.194089 secs 2024-03-24T02:18:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6213.194089 secs 2024-03-24T02:18:59.259 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6868 sec 2024-03-24T02:18:59.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:18:59 smithi078 ceph-mon[32429]: pgmap v3762: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:18:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:18:59 smithi106 ceph-mon[36505]: pgmap v3762: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:00.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:18:59 smithi107 ceph-mon[36485]: pgmap v3762: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:00.259 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6869 sec 2024-03-24T02:19:00.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:00 smithi106 ceph-mon[36505]: pgmap v3763: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:01.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:00 smithi078 ceph-mon[32429]: pgmap v3763: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:01.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:00 smithi107 ceph-mon[36485]: pgmap v3763: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:01.260 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6870 sec 2024-03-24T02:19:02.259 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6871 sec 2024-03-24T02:19:02.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:02 smithi106 ceph-mon[36505]: pgmap v3764: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:03.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:02 smithi078 ceph-mon[32429]: pgmap v3764: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:03.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:02 smithi107 ceph-mon[36485]: pgmap v3764: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:03.259 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6872 sec 2024-03-24T02:19:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6218.194160 secs 2024-03-24T02:19:03.994 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6218.194160 secs 2024-03-24T02:19:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6218.194160 secs 2024-03-24T02:19:04.260 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6873 sec 2024-03-24T02:19:05.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:04 smithi078 ceph-mon[32429]: pgmap v3765: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:05.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:04 smithi107 ceph-mon[36485]: pgmap v3765: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:05.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:04 smithi106 ceph-mon[36505]: pgmap v3765: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:05.260 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6874 sec 2024-03-24T02:19:06.260 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6875 sec 2024-03-24T02:19:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:06 smithi106 ceph-mon[36505]: pgmap v3766: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:06 smithi078 ceph-mon[32429]: pgmap v3766: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:06 smithi107 ceph-mon[36485]: pgmap v3766: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:07.260 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6876 sec 2024-03-24T02:19:08.260 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6877 sec 2024-03-24T02:19:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6223.194248 secs 2024-03-24T02:19:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6223.194248 secs 2024-03-24T02:19:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6223.194248 secs 2024-03-24T02:19:09.260 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6878 sec 2024-03-24T02:19:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:09 smithi078 ceph-mon[32429]: pgmap v3767: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:09.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:09 smithi107 ceph-mon[36485]: pgmap v3767: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:09.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:09 smithi106 ceph-mon[36505]: pgmap v3767: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:10.260 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6879 sec 2024-03-24T02:19:11.260 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6880 sec 2024-03-24T02:19:11.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:11 smithi078 ceph-mon[32429]: pgmap v3768: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:11 smithi107 ceph-mon[36485]: pgmap v3768: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:11.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:11 smithi106 ceph-mon[36505]: pgmap v3768: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:12.260 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6881 sec 2024-03-24T02:19:13.260 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6882 sec 2024-03-24T02:19:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:13 smithi078 ceph-mon[32429]: pgmap v3769: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6228.194337 secs 2024-03-24T02:19:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:13 smithi107 ceph-mon[36485]: pgmap v3769: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6228.194337 secs 2024-03-24T02:19:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:13 smithi106 ceph-mon[36505]: pgmap v3769: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6228.194337 secs 2024-03-24T02:19:14.261 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6883 sec 2024-03-24T02:19:15.261 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6884 sec 2024-03-24T02:19:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:15 smithi106 ceph-mon[36505]: pgmap v3770: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:16.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:15 smithi078 ceph-mon[32429]: pgmap v3770: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:16.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:15 smithi107 ceph-mon[36485]: pgmap v3770: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:16.261 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6885 sec 2024-03-24T02:19:17.261 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6886 sec 2024-03-24T02:19:17.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:17 smithi106 ceph-mon[36505]: pgmap v3771: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:18.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:17 smithi078 ceph-mon[32429]: pgmap v3771: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:18.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:17 smithi107 ceph-mon[36485]: pgmap v3771: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:18.261 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6887 sec 2024-03-24T02:19:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6233.194401 secs 2024-03-24T02:19:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6233.194401 secs 2024-03-24T02:19:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6233.194401 secs 2024-03-24T02:19:19.261 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6888 sec 2024-03-24T02:19:19.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:19 smithi106 ceph-mon[36505]: pgmap v3772: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:20.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:19 smithi078 ceph-mon[32429]: pgmap v3772: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:20.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:19 smithi107 ceph-mon[36485]: pgmap v3772: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:20.261 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6889 sec 2024-03-24T02:19:20.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:20 smithi106 ceph-mon[36505]: pgmap v3773: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:21.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:20 smithi078 ceph-mon[32429]: pgmap v3773: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:21.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:20 smithi107 ceph-mon[36485]: pgmap v3773: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:21.261 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6890 sec 2024-03-24T02:19:22.261 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6891 sec 2024-03-24T02:19:22.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:22 smithi106 ceph-mon[36505]: pgmap v3774: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:23.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:22 smithi078 ceph-mon[32429]: pgmap v3774: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:23.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:22 smithi107 ceph-mon[36485]: pgmap v3774: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:23.261 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6892 sec 2024-03-24T02:19:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6238.194461 secs 2024-03-24T02:19:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6238.194461 secs 2024-03-24T02:19:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6238.194461 secs 2024-03-24T02:19:24.261 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6893 sec 2024-03-24T02:19:25.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:24 smithi078 ceph-mon[32429]: pgmap v3775: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:24 smithi107 ceph-mon[36485]: pgmap v3775: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:25.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:24 smithi106 ceph-mon[36505]: pgmap v3775: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:25.261 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6894 sec 2024-03-24T02:19:26.261 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6895 sec 2024-03-24T02:19:27.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:26 smithi078 ceph-mon[32429]: pgmap v3776: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:27.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:26 smithi107 ceph-mon[36485]: pgmap v3776: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:27.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:26 smithi106 ceph-mon[36505]: pgmap v3776: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:27.261 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6896 sec 2024-03-24T02:19:28.262 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6897 sec 2024-03-24T02:19:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6243.194545 secs 2024-03-24T02:19:28.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6243.194545 secs 2024-03-24T02:19:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6243.194545 secs 2024-03-24T02:19:29.262 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6898 sec 2024-03-24T02:19:29.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:29 smithi078 ceph-mon[32429]: pgmap v3777: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:29 smithi107 ceph-mon[36485]: pgmap v3777: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:29.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:29 smithi106 ceph-mon[36505]: pgmap v3777: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:30.262 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6899 sec 2024-03-24T02:19:31.262 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6900 sec 2024-03-24T02:19:31.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:31 smithi078 ceph-mon[32429]: pgmap v3778: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:31.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:31 smithi107 ceph-mon[36485]: pgmap v3778: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:31.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:31 smithi106 ceph-mon[36505]: pgmap v3778: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:32.262 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6901 sec 2024-03-24T02:19:33.262 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6902 sec 2024-03-24T02:19:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:33 smithi106 ceph-mon[36505]: pgmap v3779: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6248.194634 secs 2024-03-24T02:19:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:33 smithi078 ceph-mon[32429]: pgmap v3779: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6248.194634 secs 2024-03-24T02:19:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:33 smithi107 ceph-mon[36485]: pgmap v3779: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6248.194634 secs 2024-03-24T02:19:34.262 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6903 sec 2024-03-24T02:19:35.262 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6904 sec 2024-03-24T02:19:35.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:35 smithi106 ceph-mon[36505]: pgmap v3780: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:36.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:35 smithi078 ceph-mon[32429]: pgmap v3780: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:36.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:35 smithi107 ceph-mon[36485]: pgmap v3780: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:36.262 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6905 sec 2024-03-24T02:19:37.262 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6906 sec 2024-03-24T02:19:37.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:37 smithi106 ceph-mon[36505]: pgmap v3781: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:37 smithi078 ceph-mon[32429]: pgmap v3781: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:38.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:37 smithi107 ceph-mon[36485]: pgmap v3781: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:38.262 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6907 sec 2024-03-24T02:19:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6253.194728 secs 2024-03-24T02:19:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:38 smithi106 ceph-mon[36505]: pgmap v3782: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6253.194728 secs 2024-03-24T02:19:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:38 smithi078 ceph-mon[32429]: pgmap v3782: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6253.194728 secs 2024-03-24T02:19:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:38 smithi107 ceph-mon[36485]: pgmap v3782: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:39.263 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6908 sec 2024-03-24T02:19:40.263 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6909 sec 2024-03-24T02:19:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:40 smithi078 ceph-mon[32429]: pgmap v3783: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:41.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:40 smithi107 ceph-mon[36485]: pgmap v3783: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:41.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:40 smithi106 ceph-mon[36505]: pgmap v3783: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:41.262 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6910 sec 2024-03-24T02:19:41.925 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:41 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:19:42.037 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:41 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:19:42.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:41 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:19:42.263 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6911 sec 2024-03-24T02:19:42.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:42 smithi078 ceph-mon[32429]: pgmap v3784: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:43.084 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:42 smithi106 ceph-mon[36505]: pgmap v3784: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:43.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:42 smithi107 ceph-mon[36485]: pgmap v3784: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:43.263 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6912 sec 2024-03-24T02:19:43.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6258.194782 secs 2024-03-24T02:19:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6258.194782 secs 2024-03-24T02:19:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6258.194782 secs 2024-03-24T02:19:44.263 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6913 sec 2024-03-24T02:19:44.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:44 smithi106 ceph-mon[36505]: pgmap v3785: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:44.829 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:44 smithi078 ceph-mon[32429]: pgmap v3785: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:45.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:44 smithi107 ceph-mon[36485]: pgmap v3785: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:19:45.263 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6914 sec 2024-03-24T02:19:46.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:19:46.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:19:46.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:19:46.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:19:46.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:19:46.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:19:46.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:45 smithi106 ceph-mon[36505]: pgmap v3786: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:19:46.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:45 smithi106 ceph-mon[36505]: pgmap v3787: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T02:19:46.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:19:46.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:19:46.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:19:46.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:19:46.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:19:46.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:19:46.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:19:46.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:45 smithi078 ceph-mon[32429]: pgmap v3786: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:19:46.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:45 smithi078 ceph-mon[32429]: pgmap v3787: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T02:19:46.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:19:46.263 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6915 sec 2024-03-24T02:19:46.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:19:46.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:19:46.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:19:46.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:19:46.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:19:46.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:19:46.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:45 smithi107 ceph-mon[36485]: pgmap v3786: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:19:46.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:45 smithi107 ceph-mon[36485]: pgmap v3787: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T02:19:46.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:19:47.263 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6916 sec 2024-03-24T02:19:48.263 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6917 sec 2024-03-24T02:19:48.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:48 smithi078 ceph-mon[32429]: pgmap v3788: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:19:48.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:48 smithi107 ceph-mon[36485]: pgmap v3788: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:19:48.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:48 smithi106 ceph-mon[36505]: pgmap v3788: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:19:49.263 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6918 sec 2024-03-24T02:19:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6263.194853 secs 2024-03-24T02:19:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6263.194853 secs 2024-03-24T02:19:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6263.194853 secs 2024-03-24T02:19:50.263 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6919 sec 2024-03-24T02:19:50.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:50 smithi078 ceph-mon[32429]: pgmap v3789: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T02:19:50.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:50 smithi107 ceph-mon[36485]: pgmap v3789: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T02:19:50.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:50 smithi106 ceph-mon[36505]: pgmap v3789: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T02:19:51.263 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6920 sec 2024-03-24T02:19:52.263 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6921 sec 2024-03-24T02:19:52.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:52 smithi078 ceph-mon[32429]: pgmap v3790: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:19:52.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:52 smithi107 ceph-mon[36485]: pgmap v3790: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:19:52.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:52 smithi106 ceph-mon[36505]: pgmap v3790: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:19:53.263 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6922 sec 2024-03-24T02:19:54.264 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6923 sec 2024-03-24T02:19:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6268.194892 secs 2024-03-24T02:19:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:54 smithi078 ceph-mon[32429]: pgmap v3791: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:19:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6268.194892 secs 2024-03-24T02:19:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:54 smithi107 ceph-mon[36485]: pgmap v3791: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:19:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6268.194892 secs 2024-03-24T02:19:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:54 smithi106 ceph-mon[36505]: pgmap v3791: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:19:55.263 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6924 sec 2024-03-24T02:19:56.264 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6925 sec 2024-03-24T02:19:56.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:56 smithi078 ceph-mon[32429]: pgmap v3792: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:19:56.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:56 smithi107 ceph-mon[36485]: pgmap v3792: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:19:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:56 smithi106 ceph-mon[36505]: pgmap v3792: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:19:57.264 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6926 sec 2024-03-24T02:19:58.264 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6927 sec 2024-03-24T02:19:58.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:58 smithi078 ceph-mon[32429]: pgmap v3793: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:58.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:58 smithi107 ceph-mon[36485]: pgmap v3793: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:58.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:58 smithi106 ceph-mon[36505]: pgmap v3793: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:59.264 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6928 sec 2024-03-24T02:19:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:19:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6273.194953 secs 2024-03-24T02:19:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:19:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6273.194953 secs 2024-03-24T02:19:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:19:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6273.194953 secs 2024-03-24T02:20:00.264 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6929 sec 2024-03-24T02:20:00.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:00 smithi078 ceph-mon[32429]: pgmap v3794: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:00.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:00 smithi078 ceph-mon[32429]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:20:00.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:00 smithi107 ceph-mon[36485]: pgmap v3794: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:00.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:00 smithi107 ceph-mon[36485]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:20:00.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:00 smithi106 ceph-mon[36505]: pgmap v3794: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:00.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:00 smithi106 ceph-mon[36505]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:20:01.264 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6930 sec 2024-03-24T02:20:02.264 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6931 sec 2024-03-24T02:20:02.555 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:02 smithi078 ceph-mon[32429]: pgmap v3795: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:02.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:02 smithi107 ceph-mon[36485]: pgmap v3795: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:02.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:02 smithi106 ceph-mon[36505]: pgmap v3795: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:03.264 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6932 sec 2024-03-24T02:20:04.264 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6933 sec 2024-03-24T02:20:04.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6278.195039 secs 2024-03-24T02:20:04.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:04 smithi078 ceph-mon[32429]: pgmap v3796: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6278.195039 secs 2024-03-24T02:20:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:04 smithi107 ceph-mon[36485]: pgmap v3796: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6278.195039 secs 2024-03-24T02:20:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:04 smithi106 ceph-mon[36505]: pgmap v3796: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:05.264 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6934 sec 2024-03-24T02:20:06.265 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6935 sec 2024-03-24T02:20:06.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:06 smithi078 ceph-mon[32429]: pgmap v3797: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:06.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:06 smithi107 ceph-mon[36485]: pgmap v3797: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:06.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:06 smithi106 ceph-mon[36505]: pgmap v3797: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:07.265 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6936 sec 2024-03-24T02:20:08.265 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6937 sec 2024-03-24T02:20:08.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:08 smithi078 ceph-mon[32429]: pgmap v3798: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:08.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:08 smithi107 ceph-mon[36485]: pgmap v3798: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:08.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:08 smithi106 ceph-mon[36505]: pgmap v3798: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:09.265 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6938 sec 2024-03-24T02:20:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6283.195099 secs 2024-03-24T02:20:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6283.195099 secs 2024-03-24T02:20:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6283.195099 secs 2024-03-24T02:20:10.265 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6939 sec 2024-03-24T02:20:10.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:10 smithi078 ceph-mon[32429]: pgmap v3799: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:10.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:10 smithi107 ceph-mon[36485]: pgmap v3799: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:10.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:10 smithi106 ceph-mon[36505]: pgmap v3799: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:11.265 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6940 sec 2024-03-24T02:20:12.265 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6941 sec 2024-03-24T02:20:12.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:12 smithi078 ceph-mon[32429]: pgmap v3800: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:12.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:12 smithi107 ceph-mon[36485]: pgmap v3800: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:12.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:12 smithi106 ceph-mon[36505]: pgmap v3800: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:13.266 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6942 sec 2024-03-24T02:20:14.265 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6943 sec 2024-03-24T02:20:14.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6288.195174 secs 2024-03-24T02:20:14.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:14 smithi078 ceph-mon[32429]: pgmap v3801: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6288.195174 secs 2024-03-24T02:20:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:14 smithi107 ceph-mon[36485]: pgmap v3801: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6288.195174 secs 2024-03-24T02:20:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:14 smithi106 ceph-mon[36505]: pgmap v3801: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:15.265 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6944 sec 2024-03-24T02:20:16.265 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6945 sec 2024-03-24T02:20:16.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:16 smithi078 ceph-mon[32429]: pgmap v3802: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:16.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:16 smithi107 ceph-mon[36485]: pgmap v3802: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:16 smithi106 ceph-mon[36505]: pgmap v3802: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:17.266 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6946 sec 2024-03-24T02:20:18.265 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6947 sec 2024-03-24T02:20:18.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:18 smithi078 ceph-mon[32429]: pgmap v3803: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:18.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:18 smithi107 ceph-mon[36485]: pgmap v3803: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:18.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:18 smithi106 ceph-mon[36505]: pgmap v3803: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:19.266 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6948 sec 2024-03-24T02:20:19.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6293.195234 secs 2024-03-24T02:20:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6293.195234 secs 2024-03-24T02:20:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6293.195234 secs 2024-03-24T02:20:20.266 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6949 sec 2024-03-24T02:20:20.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:20 smithi078 ceph-mon[32429]: pgmap v3804: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:20.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:20 smithi107 ceph-mon[36485]: pgmap v3804: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:20.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:20 smithi106 ceph-mon[36505]: pgmap v3804: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:21.266 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6950 sec 2024-03-24T02:20:22.266 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6951 sec 2024-03-24T02:20:22.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:22 smithi078 ceph-mon[32429]: pgmap v3805: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:22.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:22 smithi107 ceph-mon[36485]: pgmap v3805: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:22.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:22 smithi106 ceph-mon[36505]: pgmap v3805: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:23.266 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6952 sec 2024-03-24T02:20:24.266 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6953 sec 2024-03-24T02:20:24.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6298.195334 secs 2024-03-24T02:20:24.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:24 smithi078 ceph-mon[32429]: pgmap v3806: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6298.195334 secs 2024-03-24T02:20:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:24 smithi107 ceph-mon[36485]: pgmap v3806: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6298.195334 secs 2024-03-24T02:20:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:24 smithi106 ceph-mon[36505]: pgmap v3806: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:25.266 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6954 sec 2024-03-24T02:20:26.266 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6955 sec 2024-03-24T02:20:26.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:26 smithi078 ceph-mon[32429]: pgmap v3807: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:26.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:26 smithi107 ceph-mon[36485]: pgmap v3807: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:26.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:26 smithi106 ceph-mon[36505]: pgmap v3807: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:27.266 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6956 sec 2024-03-24T02:20:28.266 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6957 sec 2024-03-24T02:20:28.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:28 smithi078 ceph-mon[32429]: pgmap v3808: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:28.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:28 smithi107 ceph-mon[36485]: pgmap v3808: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:28.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:28 smithi106 ceph-mon[36505]: pgmap v3808: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:29.266 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6958 sec 2024-03-24T02:20:29.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6303.195375 secs 2024-03-24T02:20:29.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6303.195375 secs 2024-03-24T02:20:29.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6303.195375 secs 2024-03-24T02:20:30.266 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6959 sec 2024-03-24T02:20:30.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:30 smithi078 ceph-mon[32429]: pgmap v3809: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:30.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:30 smithi107 ceph-mon[36485]: pgmap v3809: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:30.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:30 smithi106 ceph-mon[36505]: pgmap v3809: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:31.267 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6960 sec 2024-03-24T02:20:32.267 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6961 sec 2024-03-24T02:20:32.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:32 smithi078 ceph-mon[32429]: pgmap v3810: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:32.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:32 smithi107 ceph-mon[36485]: pgmap v3810: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:32.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:32 smithi106 ceph-mon[36505]: pgmap v3810: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:33.267 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6962 sec 2024-03-24T02:20:34.267 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6963 sec 2024-03-24T02:20:34.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6308.195457 secs 2024-03-24T02:20:34.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:34 smithi078 ceph-mon[32429]: pgmap v3811: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:34.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6308.195457 secs 2024-03-24T02:20:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:34 smithi107 ceph-mon[36485]: pgmap v3811: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6308.195457 secs 2024-03-24T02:20:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:34 smithi106 ceph-mon[36505]: pgmap v3811: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:35.267 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6964 sec 2024-03-24T02:20:36.267 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6965 sec 2024-03-24T02:20:36.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:36 smithi078 ceph-mon[32429]: pgmap v3812: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:36.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:36 smithi107 ceph-mon[36485]: pgmap v3812: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:36.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:36 smithi106 ceph-mon[36505]: pgmap v3812: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:37.267 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6966 sec 2024-03-24T02:20:38.267 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6967 sec 2024-03-24T02:20:38.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:38 smithi078 ceph-mon[32429]: pgmap v3813: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:38.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:38 smithi107 ceph-mon[36485]: pgmap v3813: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:38.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:38 smithi106 ceph-mon[36505]: pgmap v3813: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:39.267 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6968 sec 2024-03-24T02:20:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6313.195547 secs 2024-03-24T02:20:39.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6313.195547 secs 2024-03-24T02:20:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6313.195547 secs 2024-03-24T02:20:40.268 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6969 sec 2024-03-24T02:20:40.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:40 smithi078 ceph-mon[32429]: pgmap v3814: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:40.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:40 smithi107 ceph-mon[36485]: pgmap v3814: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:40.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:40 smithi106 ceph-mon[36505]: pgmap v3814: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:41.268 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6970 sec 2024-03-24T02:20:42.270 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6971 sec 2024-03-24T02:20:42.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:42 smithi078 ceph-mon[32429]: pgmap v3815: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:42.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:42 smithi107 ceph-mon[36485]: pgmap v3815: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:42.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:42 smithi106 ceph-mon[36505]: pgmap v3815: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:43.268 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6972 sec 2024-03-24T02:20:44.268 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6973 sec 2024-03-24T02:20:44.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6318.195607 secs 2024-03-24T02:20:44.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:44 smithi078 ceph-mon[32429]: pgmap v3816: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:44.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6318.195607 secs 2024-03-24T02:20:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:44 smithi107 ceph-mon[36485]: pgmap v3816: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6318.195607 secs 2024-03-24T02:20:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:44 smithi106 ceph-mon[36505]: pgmap v3816: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:45.268 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6974 sec 2024-03-24T02:20:45.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:20:45.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:20:45.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:20:46.268 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6975 sec 2024-03-24T02:20:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:46 smithi106 ceph-mon[36505]: pgmap v3817: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:46.714 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:46 smithi107 ceph-mon[36485]: pgmap v3817: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:46.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:46 smithi078 ceph-mon[32429]: pgmap v3817: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:47.268 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6976 sec 2024-03-24T02:20:48.268 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6977 sec 2024-03-24T02:20:48.428 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:48 smithi107 ceph-mon[36485]: pgmap v3818: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:48.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:48 smithi106 ceph-mon[36505]: pgmap v3818: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:48.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:48 smithi078 ceph-mon[32429]: pgmap v3818: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:49.268 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6978 sec 2024-03-24T02:20:49.580 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6323.195696 secs 2024-03-24T02:20:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6323.195696 secs 2024-03-24T02:20:49.659 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6323.195696 secs 2024-03-24T02:20:50.268 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6979 sec 2024-03-24T02:20:50.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:50 smithi106 ceph-mon[36505]: pgmap v3819: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:50.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:50 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:20:50.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:50 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:20:50.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:50 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:20:50.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:50 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:20:50.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:50 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:20:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:50 smithi078 ceph-mon[32429]: pgmap v3819: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:50 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:20:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:50 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:20:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:50 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:20:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:50 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:20:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:50 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:20:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:50 smithi107 ceph-mon[36485]: pgmap v3819: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:20:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:50 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:20:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:50 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:20:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:50 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:20:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:50 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:20:50.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:50 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:20:51.269 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6980 sec 2024-03-24T02:20:51.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:51 smithi106 ceph-mon[36505]: pgmap v3820: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:20:51.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:51 smithi106 ceph-mon[36505]: pgmap v3821: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T02:20:51.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:51 smithi078 ceph-mon[32429]: pgmap v3820: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:20:51.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:51 smithi078 ceph-mon[32429]: pgmap v3821: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T02:20:51.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:51 smithi107 ceph-mon[36485]: pgmap v3820: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:20:51.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:51 smithi107 ceph-mon[36485]: pgmap v3821: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T02:20:52.269 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6981 sec 2024-03-24T02:20:53.269 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6982 sec 2024-03-24T02:20:53.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:53 smithi106 ceph-mon[36505]: pgmap v3822: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:20:53.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:53 smithi078 ceph-mon[32429]: pgmap v3822: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:20:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:53 smithi107 ceph-mon[36485]: pgmap v3822: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:20:54.269 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6983 sec 2024-03-24T02:20:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6328.195783 secs 2024-03-24T02:20:54.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6328.195783 secs 2024-03-24T02:20:54.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6328.195783 secs 2024-03-24T02:20:55.269 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6984 sec 2024-03-24T02:20:55.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:55 smithi106 ceph-mon[36505]: pgmap v3823: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:20:55.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:55 smithi078 ceph-mon[32429]: pgmap v3823: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:20:55.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:55 smithi107 ceph-mon[36485]: pgmap v3823: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:20:56.269 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6985 sec 2024-03-24T02:20:57.269 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6986 sec 2024-03-24T02:20:57.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:57 smithi078 ceph-mon[32429]: pgmap v3824: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:20:57.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:57 smithi107 ceph-mon[36485]: pgmap v3824: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:20:57.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:57 smithi106 ceph-mon[36505]: pgmap v3824: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:20:58.270 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6987 sec 2024-03-24T02:20:59.269 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6988 sec 2024-03-24T02:20:59.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:59 smithi078 ceph-mon[32429]: pgmap v3825: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:20:59.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:20:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6333.195874 secs 2024-03-24T02:20:59.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:59 smithi107 ceph-mon[36485]: pgmap v3825: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:20:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:20:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6333.195874 secs 2024-03-24T02:20:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:59 smithi106 ceph-mon[36505]: pgmap v3825: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:20:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:20:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6333.195874 secs 2024-03-24T02:21:00.269 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6989 sec 2024-03-24T02:21:01.269 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6990 sec 2024-03-24T02:21:01.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:01 smithi078 ceph-mon[32429]: pgmap v3826: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:21:01.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:01 smithi107 ceph-mon[36485]: pgmap v3826: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:21:01.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:01 smithi106 ceph-mon[36505]: pgmap v3826: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:21:02.270 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6991 sec 2024-03-24T02:21:03.270 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6992 sec 2024-03-24T02:21:03.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:03 smithi078 ceph-mon[32429]: pgmap v3827: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:03.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:03 smithi107 ceph-mon[36485]: pgmap v3827: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:03 smithi106 ceph-mon[36505]: pgmap v3827: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:04.270 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6993 sec 2024-03-24T02:21:04.744 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6338.195931 secs 2024-03-24T02:21:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6338.195931 secs 2024-03-24T02:21:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6338.195931 secs 2024-03-24T02:21:05.270 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6994 sec 2024-03-24T02:21:05.757 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:05 smithi107 ceph-mon[36485]: pgmap v3828: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:05.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:05 smithi078 ceph-mon[32429]: pgmap v3828: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:05.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:05 smithi106 ceph-mon[36505]: pgmap v3828: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:06.270 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6995 sec 2024-03-24T02:21:07.270 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6996 sec 2024-03-24T02:21:07.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:07 smithi078 ceph-mon[32429]: pgmap v3829: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:07.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:07 smithi107 ceph-mon[36485]: pgmap v3829: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:07.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:07 smithi106 ceph-mon[36505]: pgmap v3829: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:08.270 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6997 sec 2024-03-24T02:21:09.271 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6998 sec 2024-03-24T02:21:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:09 smithi078 ceph-mon[32429]: pgmap v3830: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6343.196013 secs 2024-03-24T02:21:09.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:09 smithi107 ceph-mon[36485]: pgmap v3830: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:09.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6343.196013 secs 2024-03-24T02:21:09.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:09 smithi106 ceph-mon[36505]: pgmap v3830: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:09.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6343.196013 secs 2024-03-24T02:21:10.270 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 6999 sec 2024-03-24T02:21:11.270 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7000 sec 2024-03-24T02:21:11.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:11 smithi078 ceph-mon[32429]: pgmap v3831: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:11 smithi107 ceph-mon[36485]: pgmap v3831: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:11.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:11 smithi106 ceph-mon[36505]: pgmap v3831: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:12.270 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7001 sec 2024-03-24T02:21:13.271 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7002 sec 2024-03-24T02:21:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:13 smithi078 ceph-mon[32429]: pgmap v3832: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:21:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:13 smithi107 ceph-mon[36485]: pgmap v3832: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:21:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:13 smithi106 ceph-mon[36505]: pgmap v3832: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:21:14.271 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7003 sec 2024-03-24T02:21:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6348.196089 secs 2024-03-24T02:21:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6348.196089 secs 2024-03-24T02:21:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6348.196089 secs 2024-03-24T02:21:15.271 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7004 sec 2024-03-24T02:21:15.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:15 smithi078 ceph-mon[32429]: pgmap v3833: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:21:15.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:15 smithi107 ceph-mon[36485]: pgmap v3833: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:21:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:15 smithi106 ceph-mon[36505]: pgmap v3833: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:21:16.271 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7005 sec 2024-03-24T02:21:17.271 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7006 sec 2024-03-24T02:21:17.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:17 smithi078 ceph-mon[32429]: pgmap v3834: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:21:17.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:17 smithi107 ceph-mon[36485]: pgmap v3834: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:21:17.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:17 smithi106 ceph-mon[36505]: pgmap v3834: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:21:18.271 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7007 sec 2024-03-24T02:21:19.272 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7008 sec 2024-03-24T02:21:19.703 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:19 smithi106 ceph-mon[36505]: pgmap v3835: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:21:19.703 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6353.196175 secs 2024-03-24T02:21:19.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:19 smithi078 ceph-mon[32429]: pgmap v3835: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:21:19.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6353.196175 secs 2024-03-24T02:21:19.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:19 smithi107 ceph-mon[36485]: pgmap v3835: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:21:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6353.196175 secs 2024-03-24T02:21:20.271 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7009 sec 2024-03-24T02:21:21.271 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7010 sec 2024-03-24T02:21:21.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:21 smithi078 ceph-mon[32429]: pgmap v3836: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:21:21.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:21 smithi107 ceph-mon[36485]: pgmap v3836: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:21:21.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:21 smithi106 ceph-mon[36505]: pgmap v3836: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:21:22.271 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7011 sec 2024-03-24T02:21:23.271 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7012 sec 2024-03-24T02:21:23.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:23 smithi078 ceph-mon[32429]: pgmap v3837: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:21:23.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:23 smithi107 ceph-mon[36485]: pgmap v3837: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:21:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:23 smithi106 ceph-mon[36505]: pgmap v3837: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:21:24.272 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7013 sec 2024-03-24T02:21:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6358.196264 secs 2024-03-24T02:21:24.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6358.196264 secs 2024-03-24T02:21:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6358.196264 secs 2024-03-24T02:21:25.272 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7014 sec 2024-03-24T02:21:25.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:25 smithi078 ceph-mon[32429]: pgmap v3838: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:25.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:25 smithi107 ceph-mon[36485]: pgmap v3838: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:25.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:25 smithi106 ceph-mon[36505]: pgmap v3838: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:26.272 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7015 sec 2024-03-24T02:21:27.272 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7016 sec 2024-03-24T02:21:27.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:27 smithi078 ceph-mon[32429]: pgmap v3839: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:27.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:27 smithi107 ceph-mon[36485]: pgmap v3839: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:27.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:27 smithi106 ceph-mon[36505]: pgmap v3839: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:28.272 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7017 sec 2024-03-24T02:21:29.272 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7018 sec 2024-03-24T02:21:29.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:29 smithi078 ceph-mon[32429]: pgmap v3840: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:29.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6363.196351 secs 2024-03-24T02:21:29.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:29 smithi107 ceph-mon[36485]: pgmap v3840: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6363.196351 secs 2024-03-24T02:21:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:29 smithi106 ceph-mon[36505]: pgmap v3840: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6363.196351 secs 2024-03-24T02:21:30.272 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7019 sec 2024-03-24T02:21:31.272 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7020 sec 2024-03-24T02:21:31.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:31 smithi078 ceph-mon[32429]: pgmap v3841: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:31.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:31 smithi107 ceph-mon[36485]: pgmap v3841: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:31.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:31 smithi106 ceph-mon[36505]: pgmap v3841: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:32.272 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7021 sec 2024-03-24T02:21:33.272 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7022 sec 2024-03-24T02:21:33.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:33 smithi078 ceph-mon[32429]: pgmap v3842: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:33.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:33 smithi107 ceph-mon[36485]: pgmap v3842: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:33.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:33 smithi106 ceph-mon[36505]: pgmap v3842: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:34.272 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7023 sec 2024-03-24T02:21:34.744 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6368.196431 secs 2024-03-24T02:21:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6368.196431 secs 2024-03-24T02:21:34.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6368.196431 secs 2024-03-24T02:21:35.273 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7024 sec 2024-03-24T02:21:35.739 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:35 smithi078 ceph-mon[32429]: pgmap v3843: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:35.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:35 smithi107 ceph-mon[36485]: pgmap v3843: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:35.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:35 smithi106 ceph-mon[36505]: pgmap v3843: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:36.273 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7025 sec 2024-03-24T02:21:37.273 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7026 sec 2024-03-24T02:21:37.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:37 smithi078 ceph-mon[32429]: pgmap v3844: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:37.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:37 smithi107 ceph-mon[36485]: pgmap v3844: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:37.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:37 smithi106 ceph-mon[36505]: pgmap v3844: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:38.273 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7027 sec 2024-03-24T02:21:39.273 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7028 sec 2024-03-24T02:21:39.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:39 smithi078 ceph-mon[32429]: pgmap v3845: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:39.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6373.196520 secs 2024-03-24T02:21:39.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:39 smithi107 ceph-mon[36485]: pgmap v3845: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:39.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6373.196520 secs 2024-03-24T02:21:39.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:39 smithi106 ceph-mon[36505]: pgmap v3845: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:39.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6373.196520 secs 2024-03-24T02:21:40.273 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7029 sec 2024-03-24T02:21:41.273 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7030 sec 2024-03-24T02:21:41.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:41 smithi078 ceph-mon[32429]: pgmap v3846: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:41.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:41 smithi107 ceph-mon[36485]: pgmap v3846: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:41.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:41 smithi106 ceph-mon[36505]: pgmap v3846: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:42.273 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7031 sec 2024-03-24T02:21:43.273 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7032 sec 2024-03-24T02:21:43.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:43 smithi078 ceph-mon[32429]: pgmap v3847: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:43 smithi107 ceph-mon[36485]: pgmap v3847: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:43 smithi106 ceph-mon[36505]: pgmap v3847: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:44.273 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7033 sec 2024-03-24T02:21:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6378.196609 secs 2024-03-24T02:21:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6378.196609 secs 2024-03-24T02:21:44.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6378.196609 secs 2024-03-24T02:21:45.274 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7034 sec 2024-03-24T02:21:45.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:45 smithi078 ceph-mon[32429]: pgmap v3848: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:45.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:45 smithi107 ceph-mon[36485]: pgmap v3848: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:45.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:45 smithi106 ceph-mon[36505]: pgmap v3848: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:46.274 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7035 sec 2024-03-24T02:21:47.274 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7036 sec 2024-03-24T02:21:47.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:47 smithi078 ceph-mon[32429]: pgmap v3849: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:47.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:47 smithi107 ceph-mon[36485]: pgmap v3849: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:47.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:47 smithi106 ceph-mon[36505]: pgmap v3849: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:48.274 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7037 sec 2024-03-24T02:21:49.274 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7038 sec 2024-03-24T02:21:49.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:49 smithi078 ceph-mon[32429]: pgmap v3850: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:49.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6383.196700 secs 2024-03-24T02:21:49.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:49 smithi107 ceph-mon[36485]: pgmap v3850: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:49.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6383.196700 secs 2024-03-24T02:21:49.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:49 smithi106 ceph-mon[36505]: pgmap v3850: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:49.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6383.196700 secs 2024-03-24T02:21:50.274 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7039 sec 2024-03-24T02:21:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:50 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:21:50.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:50 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:21:50.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:50 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:21:51.274 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7040 sec 2024-03-24T02:21:51.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:51 smithi078 ceph-mon[32429]: pgmap v3851: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:51.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:51 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:21:51.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:51 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:21:51.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:51 smithi078 ceph-mon[32429]: pgmap v3852: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:21:51.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:51 smithi078 ceph-mon[32429]: pgmap v3853: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:21:51.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:51 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:21:51.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:51 smithi107 ceph-mon[36485]: pgmap v3851: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:51.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:51 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:21:51.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:51 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:21:51.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:51 smithi107 ceph-mon[36485]: pgmap v3852: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:21:51.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:51 smithi107 ceph-mon[36485]: pgmap v3853: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:21:51.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:51 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:21:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:51 smithi106 ceph-mon[36505]: pgmap v3851: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:21:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:51 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:21:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:51 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:21:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:51 smithi106 ceph-mon[36505]: pgmap v3852: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:21:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:51 smithi106 ceph-mon[36505]: pgmap v3853: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:21:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:51 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:21:52.274 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7041 sec 2024-03-24T02:21:53.274 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7042 sec 2024-03-24T02:21:53.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:53 smithi078 ceph-mon[32429]: pgmap v3854: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:21:53.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6388.196758 secs 2024-03-24T02:21:53.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:53 smithi107 ceph-mon[36485]: pgmap v3854: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:21:53.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6388.196758 secs 2024-03-24T02:21:53.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:53 smithi106 ceph-mon[36505]: pgmap v3854: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:21:53.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6388.196758 secs 2024-03-24T02:21:54.274 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7043 sec 2024-03-24T02:21:55.274 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7044 sec 2024-03-24T02:21:55.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:55 smithi078 ceph-mon[32429]: pgmap v3855: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:21:55.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:55 smithi107 ceph-mon[36485]: pgmap v3855: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:21:55.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:55 smithi106 ceph-mon[36505]: pgmap v3855: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:21:56.274 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7045 sec 2024-03-24T02:21:57.275 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7046 sec 2024-03-24T02:21:57.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:57 smithi078 ceph-mon[32429]: pgmap v3856: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:21:57.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:57 smithi107 ceph-mon[36485]: pgmap v3856: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:21:57.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:57 smithi106 ceph-mon[36505]: pgmap v3856: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:21:58.275 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7047 sec 2024-03-24T02:21:58.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6393.196820 secs 2024-03-24T02:21:58.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6393.196820 secs 2024-03-24T02:21:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6393.196820 secs 2024-03-24T02:21:59.275 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7048 sec 2024-03-24T02:21:59.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:21:59 smithi078 ceph-mon[32429]: pgmap v3857: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:21:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:21:59 smithi107 ceph-mon[36485]: pgmap v3857: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:21:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:21:59 smithi106 ceph-mon[36505]: pgmap v3857: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:22:00.277 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7049 sec 2024-03-24T02:22:01.275 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7050 sec 2024-03-24T02:22:01.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:01 smithi078 ceph-mon[32429]: pgmap v3858: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:22:01.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:01 smithi107 ceph-mon[36485]: pgmap v3858: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:22:01.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:01 smithi106 ceph-mon[36505]: pgmap v3858: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:22:02.275 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7051 sec 2024-03-24T02:22:03.275 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7052 sec 2024-03-24T02:22:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:03 smithi107 ceph-mon[36485]: pgmap v3859: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:03.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6398.196880 secs 2024-03-24T02:22:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:03 smithi106 ceph-mon[36505]: pgmap v3859: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:03.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6398.196880 secs 2024-03-24T02:22:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:03 smithi078 ceph-mon[32429]: pgmap v3859: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6398.196880 secs 2024-03-24T02:22:04.275 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7053 sec 2024-03-24T02:22:05.275 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7054 sec 2024-03-24T02:22:05.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:05 smithi106 ceph-mon[36505]: pgmap v3860: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:06.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:05 smithi078 ceph-mon[32429]: pgmap v3860: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:06.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:05 smithi107 ceph-mon[36485]: pgmap v3860: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:06.275 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7055 sec 2024-03-24T02:22:07.275 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7056 sec 2024-03-24T02:22:07.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:07 smithi106 ceph-mon[36505]: pgmap v3861: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:08.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:07 smithi078 ceph-mon[32429]: pgmap v3861: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:08.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:07 smithi107 ceph-mon[36485]: pgmap v3861: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:08.276 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7057 sec 2024-03-24T02:22:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6403.196958 secs 2024-03-24T02:22:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6403.196958 secs 2024-03-24T02:22:09.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6403.196958 secs 2024-03-24T02:22:09.276 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7058 sec 2024-03-24T02:22:09.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:09 smithi106 ceph-mon[36505]: pgmap v3862: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:09 smithi078 ceph-mon[32429]: pgmap v3862: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:10.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:09 smithi107 ceph-mon[36485]: pgmap v3862: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:10.276 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7059 sec 2024-03-24T02:22:11.276 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7060 sec 2024-03-24T02:22:11.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:11 smithi106 ceph-mon[36505]: pgmap v3863: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:12.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:11 smithi078 ceph-mon[32429]: pgmap v3863: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:12.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:11 smithi107 ceph-mon[36485]: pgmap v3863: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:12.276 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7061 sec 2024-03-24T02:22:13.276 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7062 sec 2024-03-24T02:22:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:13 smithi106 ceph-mon[36505]: pgmap v3864: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6408.197016 secs 2024-03-24T02:22:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:13 smithi078 ceph-mon[32429]: pgmap v3864: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6408.197016 secs 2024-03-24T02:22:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:13 smithi107 ceph-mon[36485]: pgmap v3864: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6408.197016 secs 2024-03-24T02:22:14.276 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7063 sec 2024-03-24T02:22:15.276 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7064 sec 2024-03-24T02:22:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:15 smithi106 ceph-mon[36505]: pgmap v3865: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:16.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:15 smithi078 ceph-mon[32429]: pgmap v3865: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:16.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:15 smithi107 ceph-mon[36485]: pgmap v3865: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:16.276 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7065 sec 2024-03-24T02:22:17.277 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7066 sec 2024-03-24T02:22:17.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:17 smithi078 ceph-mon[32429]: pgmap v3866: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:17.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:17 smithi106 ceph-mon[36505]: pgmap v3866: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:18.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:17 smithi107 ceph-mon[36485]: pgmap v3866: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:18.276 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7067 sec 2024-03-24T02:22:18.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6413.197076 secs 2024-03-24T02:22:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6413.197076 secs 2024-03-24T02:22:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6413.197076 secs 2024-03-24T02:22:19.276 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7068 sec 2024-03-24T02:22:19.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:19 smithi106 ceph-mon[36505]: pgmap v3867: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:20.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:19 smithi078 ceph-mon[32429]: pgmap v3867: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:20.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:19 smithi107 ceph-mon[36485]: pgmap v3867: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:20.277 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7069 sec 2024-03-24T02:22:21.277 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7070 sec 2024-03-24T02:22:21.891 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:21 smithi078 ceph-mon[32429]: pgmap v3868: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:21.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:21 smithi106 ceph-mon[36505]: pgmap v3868: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:22.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:21 smithi107 ceph-mon[36485]: pgmap v3868: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:22.277 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7071 sec 2024-03-24T02:22:23.277 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7072 sec 2024-03-24T02:22:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:23 smithi106 ceph-mon[36505]: pgmap v3869: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6418.197155 secs 2024-03-24T02:22:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:23 smithi078 ceph-mon[32429]: pgmap v3869: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6418.197155 secs 2024-03-24T02:22:24.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:23 smithi107 ceph-mon[36485]: pgmap v3869: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6418.197155 secs 2024-03-24T02:22:24.277 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7073 sec 2024-03-24T02:22:25.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:24 smithi078 ceph-mon[32429]: pgmap v3870: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:25.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:24 smithi107 ceph-mon[36485]: pgmap v3870: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:25.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:24 smithi106 ceph-mon[36505]: pgmap v3870: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:25.277 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7074 sec 2024-03-24T02:22:26.277 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7075 sec 2024-03-24T02:22:27.277 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7076 sec 2024-03-24T02:22:27.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:27 smithi078 ceph-mon[32429]: pgmap v3871: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:27.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:27 smithi107 ceph-mon[36485]: pgmap v3871: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:27.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:27 smithi106 ceph-mon[36505]: pgmap v3871: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:28.277 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7077 sec 2024-03-24T02:22:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6423.197216 secs 2024-03-24T02:22:28.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6423.197216 secs 2024-03-24T02:22:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6423.197216 secs 2024-03-24T02:22:29.278 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7078 sec 2024-03-24T02:22:29.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:29 smithi078 ceph-mon[32429]: pgmap v3872: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:29.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:29 smithi107 ceph-mon[36485]: pgmap v3872: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:29 smithi106 ceph-mon[36505]: pgmap v3872: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:30.278 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7079 sec 2024-03-24T02:22:31.278 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7080 sec 2024-03-24T02:22:31.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:31 smithi106 ceph-mon[36505]: pgmap v3873: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:32.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:31 smithi078 ceph-mon[32429]: pgmap v3873: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:32.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:31 smithi107 ceph-mon[36485]: pgmap v3873: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:32.278 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7081 sec 2024-03-24T02:22:33.278 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7082 sec 2024-03-24T02:22:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:33 smithi106 ceph-mon[36505]: pgmap v3874: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6428.197303 secs 2024-03-24T02:22:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:33 smithi078 ceph-mon[32429]: pgmap v3874: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6428.197303 secs 2024-03-24T02:22:34.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:33 smithi107 ceph-mon[36485]: pgmap v3874: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:34.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6428.197303 secs 2024-03-24T02:22:34.278 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7083 sec 2024-03-24T02:22:35.278 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7084 sec 2024-03-24T02:22:35.894 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:35 smithi107 ceph-mon[36485]: pgmap v3875: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:35.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:35 smithi106 ceph-mon[36505]: pgmap v3875: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:36.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:35 smithi078 ceph-mon[32429]: pgmap v3875: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:36.278 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7085 sec 2024-03-24T02:22:37.278 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7086 sec 2024-03-24T02:22:37.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:37 smithi106 ceph-mon[36505]: pgmap v3876: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:37 smithi078 ceph-mon[32429]: pgmap v3876: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:38.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:37 smithi107 ceph-mon[36485]: pgmap v3876: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:38.278 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7087 sec 2024-03-24T02:22:38.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6433.197363 secs 2024-03-24T02:22:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6433.197363 secs 2024-03-24T02:22:39.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6433.197363 secs 2024-03-24T02:22:39.278 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7088 sec 2024-03-24T02:22:39.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:39 smithi106 ceph-mon[36505]: pgmap v3877: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:39 smithi078 ceph-mon[32429]: pgmap v3877: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:39 smithi107 ceph-mon[36485]: pgmap v3877: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:40.278 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7089 sec 2024-03-24T02:22:41.279 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7090 sec 2024-03-24T02:22:41.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:41 smithi106 ceph-mon[36505]: pgmap v3878: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:42.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:41 smithi078 ceph-mon[32429]: pgmap v3878: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:42.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:41 smithi107 ceph-mon[36485]: pgmap v3878: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:42.279 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7091 sec 2024-03-24T02:22:43.279 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7092 sec 2024-03-24T02:22:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:43 smithi106 ceph-mon[36505]: pgmap v3879: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6438.197445 secs 2024-03-24T02:22:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:43 smithi078 ceph-mon[32429]: pgmap v3879: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6438.197445 secs 2024-03-24T02:22:44.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:43 smithi107 ceph-mon[36485]: pgmap v3879: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6438.197445 secs 2024-03-24T02:22:44.279 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7093 sec 2024-03-24T02:22:45.279 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7094 sec 2024-03-24T02:22:45.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:45 smithi106 ceph-mon[36505]: pgmap v3880: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:46.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:45 smithi078 ceph-mon[32429]: pgmap v3880: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:45 smithi107 ceph-mon[36485]: pgmap v3880: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:46.279 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7095 sec 2024-03-24T02:22:47.279 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7096 sec 2024-03-24T02:22:47.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:47 smithi078 ceph-mon[32429]: pgmap v3881: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:47.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:47 smithi106 ceph-mon[36505]: pgmap v3881: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:48.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:47 smithi107 ceph-mon[36485]: pgmap v3881: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:48.279 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7097 sec 2024-03-24T02:22:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6443.197508 secs 2024-03-24T02:22:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6443.197508 secs 2024-03-24T02:22:49.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6443.197508 secs 2024-03-24T02:22:49.279 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7098 sec 2024-03-24T02:22:49.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:49 smithi106 ceph-mon[36505]: pgmap v3882: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:49 smithi078 ceph-mon[32429]: pgmap v3882: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:50.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:49 smithi107 ceph-mon[36485]: pgmap v3882: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:22:50.279 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7099 sec 2024-03-24T02:22:50.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:50 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:22:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:50 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:22:51.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:50 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:22:51.279 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7100 sec 2024-03-24T02:22:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:51 smithi106 ceph-mon[36505]: pgmap v3883: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:51 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:22:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:51 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:22:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:51 smithi106 ceph-mon[36505]: pgmap v3884: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:22:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:51 smithi106 ceph-mon[36505]: pgmap v3885: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:22:51.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:51 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:22:52.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:51 smithi078 ceph-mon[32429]: pgmap v3883: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:52.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:51 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:22:52.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:51 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:22:52.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:51 smithi078 ceph-mon[32429]: pgmap v3884: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:22:52.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:51 smithi078 ceph-mon[32429]: pgmap v3885: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:22:52.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:51 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:22:52.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:51 smithi107 ceph-mon[36485]: pgmap v3883: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:52.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:51 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:22:52.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:51 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:22:52.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:51 smithi107 ceph-mon[36485]: pgmap v3884: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:22:52.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:51 smithi107 ceph-mon[36485]: pgmap v3885: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:22:52.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:51 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:22:52.280 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7101 sec 2024-03-24T02:22:53.280 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7102 sec 2024-03-24T02:22:54.280 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7103 sec 2024-03-24T02:22:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6448.197545 secs 2024-03-24T02:22:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:54 smithi106 ceph-mon[36505]: pgmap v3886: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:22:54.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6448.197545 secs 2024-03-24T02:22:54.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:54 smithi078 ceph-mon[32429]: pgmap v3886: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:22:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6448.197545 secs 2024-03-24T02:22:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:54 smithi107 ceph-mon[36485]: pgmap v3886: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:22:55.280 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7104 sec 2024-03-24T02:22:56.280 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7105 sec 2024-03-24T02:22:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:56 smithi106 ceph-mon[36505]: pgmap v3887: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:22:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:56 smithi078 ceph-mon[32429]: pgmap v3887: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:22:56.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:56 smithi107 ceph-mon[36485]: pgmap v3887: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:22:57.280 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7106 sec 2024-03-24T02:22:58.280 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7107 sec 2024-03-24T02:22:58.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:58 smithi106 ceph-mon[36505]: pgmap v3888: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:22:58.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:58 smithi078 ceph-mon[32429]: pgmap v3888: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:22:58.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:58 smithi107 ceph-mon[36485]: pgmap v3888: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:22:59.280 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7108 sec 2024-03-24T02:22:59.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:22:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6453.197634 secs 2024-03-24T02:22:59.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:22:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6453.197634 secs 2024-03-24T02:22:59.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:22:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6453.197634 secs 2024-03-24T02:23:00.280 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7109 sec 2024-03-24T02:23:00.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:00 smithi106 ceph-mon[36505]: pgmap v3889: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:23:00.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:00 smithi078 ceph-mon[32429]: pgmap v3889: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:23:00.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:00 smithi107 ceph-mon[36485]: pgmap v3889: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:23:01.280 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7110 sec 2024-03-24T02:23:02.280 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7111 sec 2024-03-24T02:23:02.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:02 smithi106 ceph-mon[36505]: pgmap v3890: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:23:02.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:02 smithi078 ceph-mon[32429]: pgmap v3890: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:23:02.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:02 smithi107 ceph-mon[36485]: pgmap v3890: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:23:03.281 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7112 sec 2024-03-24T02:23:04.281 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7113 sec 2024-03-24T02:23:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6458.197692 secs 2024-03-24T02:23:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:04 smithi106 ceph-mon[36505]: pgmap v3891: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6458.197692 secs 2024-03-24T02:23:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:04 smithi078 ceph-mon[32429]: pgmap v3891: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6458.197692 secs 2024-03-24T02:23:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:04 smithi107 ceph-mon[36485]: pgmap v3891: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:05.281 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7114 sec 2024-03-24T02:23:06.281 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7115 sec 2024-03-24T02:23:06.645 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:06 smithi107 ceph-mon[36485]: pgmap v3892: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:06.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:06 smithi106 ceph-mon[36505]: pgmap v3892: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:06 smithi078 ceph-mon[32429]: pgmap v3892: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:07.281 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7116 sec 2024-03-24T02:23:08.281 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7117 sec 2024-03-24T02:23:08.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:08 smithi106 ceph-mon[36505]: pgmap v3893: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:08 smithi078 ceph-mon[32429]: pgmap v3893: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:08.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:08 smithi107 ceph-mon[36485]: pgmap v3893: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:09.281 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7118 sec 2024-03-24T02:23:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6463.197778 secs 2024-03-24T02:23:09.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6463.197778 secs 2024-03-24T02:23:09.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6463.197778 secs 2024-03-24T02:23:10.281 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7119 sec 2024-03-24T02:23:10.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:10 smithi106 ceph-mon[36505]: pgmap v3894: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:10 smithi078 ceph-mon[32429]: pgmap v3894: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:10.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:10 smithi107 ceph-mon[36485]: pgmap v3894: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:11.281 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7120 sec 2024-03-24T02:23:12.281 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7121 sec 2024-03-24T02:23:12.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:12 smithi106 ceph-mon[36505]: pgmap v3895: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:12 smithi078 ceph-mon[32429]: pgmap v3895: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:12.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:12 smithi107 ceph-mon[36485]: pgmap v3895: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:13.282 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7122 sec 2024-03-24T02:23:14.282 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7123 sec 2024-03-24T02:23:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6468.197839 secs 2024-03-24T02:23:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:14 smithi106 ceph-mon[36505]: pgmap v3896: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6468.197839 secs 2024-03-24T02:23:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:14 smithi078 ceph-mon[32429]: pgmap v3896: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:14.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6468.197839 secs 2024-03-24T02:23:14.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:14 smithi107 ceph-mon[36485]: pgmap v3896: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:15.282 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7124 sec 2024-03-24T02:23:16.282 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7125 sec 2024-03-24T02:23:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:16 smithi078 ceph-mon[32429]: pgmap v3897: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:16.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:16 smithi107 ceph-mon[36485]: pgmap v3897: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:16.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:16 smithi106 ceph-mon[36505]: pgmap v3897: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:17.282 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7126 sec 2024-03-24T02:23:18.282 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7127 sec 2024-03-24T02:23:18.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:18 smithi078 ceph-mon[32429]: pgmap v3898: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:18.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:18 smithi107 ceph-mon[36485]: pgmap v3898: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:18 smithi106 ceph-mon[36505]: pgmap v3898: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:19.282 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7128 sec 2024-03-24T02:23:19.803 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6473.197927 secs 2024-03-24T02:23:19.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6473.197927 secs 2024-03-24T02:23:19.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6473.197927 secs 2024-03-24T02:23:20.282 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7129 sec 2024-03-24T02:23:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:20 smithi078 ceph-mon[32429]: pgmap v3899: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:20 smithi107 ceph-mon[36485]: pgmap v3899: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:20.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:20 smithi106 ceph-mon[36505]: pgmap v3899: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:21.282 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7130 sec 2024-03-24T02:23:22.282 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7131 sec 2024-03-24T02:23:22.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:22 smithi078 ceph-mon[32429]: pgmap v3900: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:22.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:22 smithi107 ceph-mon[36485]: pgmap v3900: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:22.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:22 smithi106 ceph-mon[36505]: pgmap v3900: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:23.282 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7132 sec 2024-03-24T02:23:24.282 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7133 sec 2024-03-24T02:23:24.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6478.198002 secs 2024-03-24T02:23:24.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:24 smithi078 ceph-mon[32429]: pgmap v3901: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:24.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6478.198002 secs 2024-03-24T02:23:24.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:24 smithi107 ceph-mon[36485]: pgmap v3901: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:24.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6478.198002 secs 2024-03-24T02:23:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:24 smithi106 ceph-mon[36505]: pgmap v3901: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:25.282 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7134 sec 2024-03-24T02:23:26.283 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7135 sec 2024-03-24T02:23:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:26 smithi078 ceph-mon[32429]: pgmap v3902: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:26.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:26 smithi107 ceph-mon[36485]: pgmap v3902: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:26.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:26 smithi106 ceph-mon[36505]: pgmap v3902: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:27.283 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7136 sec 2024-03-24T02:23:28.283 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7137 sec 2024-03-24T02:23:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:28 smithi078 ceph-mon[32429]: pgmap v3903: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:28.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:28 smithi107 ceph-mon[36485]: pgmap v3903: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:28 smithi106 ceph-mon[36505]: pgmap v3903: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:29.283 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7138 sec 2024-03-24T02:23:29.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6483.198093 secs 2024-03-24T02:23:29.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6483.198093 secs 2024-03-24T02:23:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6483.198093 secs 2024-03-24T02:23:30.283 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7139 sec 2024-03-24T02:23:30.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:30 smithi078 ceph-mon[32429]: pgmap v3904: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:30.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:30 smithi107 ceph-mon[36485]: pgmap v3904: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:30.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:30 smithi106 ceph-mon[36505]: pgmap v3904: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:31.283 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7140 sec 2024-03-24T02:23:32.283 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7141 sec 2024-03-24T02:23:32.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:32 smithi078 ceph-mon[32429]: pgmap v3905: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:32.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:32 smithi107 ceph-mon[36485]: pgmap v3905: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:32 smithi106 ceph-mon[36505]: pgmap v3905: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:33.283 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7142 sec 2024-03-24T02:23:34.284 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7143 sec 2024-03-24T02:23:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6488.198132 secs 2024-03-24T02:23:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:34 smithi078 ceph-mon[32429]: pgmap v3906: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:34.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6488.198132 secs 2024-03-24T02:23:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:34 smithi107 ceph-mon[36485]: pgmap v3906: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:34.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6488.198132 secs 2024-03-24T02:23:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:34 smithi106 ceph-mon[36505]: pgmap v3906: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:35.284 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7144 sec 2024-03-24T02:23:36.283 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7145 sec 2024-03-24T02:23:36.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:36 smithi078 ceph-mon[32429]: pgmap v3907: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:36.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:36 smithi107 ceph-mon[36485]: pgmap v3907: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:36.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:36 smithi106 ceph-mon[36505]: pgmap v3907: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:37.283 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7146 sec 2024-03-24T02:23:38.284 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7147 sec 2024-03-24T02:23:38.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:38 smithi078 ceph-mon[32429]: pgmap v3908: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:38.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:38 smithi107 ceph-mon[36485]: pgmap v3908: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:38.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:38 smithi106 ceph-mon[36505]: pgmap v3908: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:39.284 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7148 sec 2024-03-24T02:23:39.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6493.198222 secs 2024-03-24T02:23:39.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6493.198222 secs 2024-03-24T02:23:39.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6493.198222 secs 2024-03-24T02:23:40.284 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7149 sec 2024-03-24T02:23:40.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:40 smithi078 ceph-mon[32429]: pgmap v3909: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:40.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:40 smithi107 ceph-mon[36485]: pgmap v3909: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:40 smithi106 ceph-mon[36505]: pgmap v3909: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:41.284 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7150 sec 2024-03-24T02:23:42.284 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7151 sec 2024-03-24T02:23:42.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:42 smithi078 ceph-mon[32429]: pgmap v3910: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:42.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:42 smithi107 ceph-mon[36485]: pgmap v3910: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:42.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:42 smithi106 ceph-mon[36505]: pgmap v3910: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:43.284 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7152 sec 2024-03-24T02:23:44.284 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7153 sec 2024-03-24T02:23:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6498.198305 secs 2024-03-24T02:23:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:44 smithi078 ceph-mon[32429]: pgmap v3911: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6498.198305 secs 2024-03-24T02:23:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:44 smithi107 ceph-mon[36485]: pgmap v3911: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:44.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6498.198305 secs 2024-03-24T02:23:44.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:44 smithi106 ceph-mon[36505]: pgmap v3911: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:45.284 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7154 sec 2024-03-24T02:23:46.284 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7155 sec 2024-03-24T02:23:46.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:46 smithi078 ceph-mon[32429]: pgmap v3912: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:46.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:46 smithi107 ceph-mon[36485]: pgmap v3912: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:46 smithi106 ceph-mon[36505]: pgmap v3912: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:47.285 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7156 sec 2024-03-24T02:23:48.285 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7157 sec 2024-03-24T02:23:48.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:48 smithi078 ceph-mon[32429]: pgmap v3913: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:48.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:48 smithi107 ceph-mon[36485]: pgmap v3913: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:48 smithi106 ceph-mon[36505]: pgmap v3913: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:49.285 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7158 sec 2024-03-24T02:23:49.803 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6503.198388 secs 2024-03-24T02:23:49.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6503.198388 secs 2024-03-24T02:23:49.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6503.198388 secs 2024-03-24T02:23:50.285 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7159 sec 2024-03-24T02:23:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:50 smithi078 ceph-mon[32429]: pgmap v3914: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:50.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:50 smithi107 ceph-mon[36485]: pgmap v3914: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:50.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:50 smithi106 ceph-mon[36505]: pgmap v3914: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:51.285 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7160 sec 2024-03-24T02:23:51.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:51 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:23:51.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:51 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:23:51.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:51 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:23:52.285 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7161 sec 2024-03-24T02:23:52.487 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:52 smithi107 ceph-mon[36485]: pgmap v3915: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:52.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:52 smithi078 ceph-mon[32429]: pgmap v3915: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:52.571 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:52 smithi106 ceph-mon[36505]: pgmap v3915: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:53.285 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7162 sec 2024-03-24T02:23:54.285 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7163 sec 2024-03-24T02:23:54.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6508.198455 secs 2024-03-24T02:23:54.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:54 smithi078 ceph-mon[32429]: pgmap v3916: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:54.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:54 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:54.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:54 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:54.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:54 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:54.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:54 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:54.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:54 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:54.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:54 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6508.198455 secs 2024-03-24T02:23:54.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:54 smithi107 ceph-mon[36485]: pgmap v3916: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:54.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:54 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:54.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:54 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:54.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:54 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:54.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:54 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:54.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:54 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:54.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:54 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6508.198455 secs 2024-03-24T02:23:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:54 smithi106 ceph-mon[36505]: pgmap v3916: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:54 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:54 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:54 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:54 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:54 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:54 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:55.285 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7164 sec 2024-03-24T02:23:55.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:55.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:55.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:55 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T02:23:55.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:55 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T02:23:55.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:55.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:55.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:55.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:55.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:23:55.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:23:55.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:55.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:55.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:55.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:55 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T02:23:55.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:55 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T02:23:55.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:55.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:55.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:55.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:55.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:23:55.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:23:55.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:55.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:55.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:55.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:55 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T02:23:55.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:55 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T02:23:55.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:55.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:55.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:55.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:55.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:23:55.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:23:55.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:23:55.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:23:56.285 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7165 sec 2024-03-24T02:23:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:56 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T02:23:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:56 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T02:23:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:56 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T02:23:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:56 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T02:23:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:56 smithi078 ceph-mon[32429]: pgmap v3917: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:56.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:56 smithi078 ceph-mon[32429]: pgmap v3918: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-24T02:23:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:56 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T02:23:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:56 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T02:23:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:56 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T02:23:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:56 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T02:23:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:56 smithi107 ceph-mon[36485]: pgmap v3917: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:56 smithi107 ceph-mon[36485]: pgmap v3918: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-24T02:23:56.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:56 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T02:23:56.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:56 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T02:23:56.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:56 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T02:23:56.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:56 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T02:23:56.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:56 smithi106 ceph-mon[36505]: pgmap v3917: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:23:56.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:56 smithi106 ceph-mon[36505]: pgmap v3918: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-24T02:23:57.286 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7166 sec 2024-03-24T02:23:58.286 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7167 sec 2024-03-24T02:23:58.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:58 smithi078 ceph-mon[32429]: pgmap v3919: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:23:58.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:58 smithi107 ceph-mon[36485]: pgmap v3919: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:23:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:58 smithi106 ceph-mon[36505]: pgmap v3919: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:23:59.286 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7168 sec 2024-03-24T02:23:59.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:23:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6513.198512 secs 2024-03-24T02:23:59.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:23:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6513.198512 secs 2024-03-24T02:23:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:23:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6513.198512 secs 2024-03-24T02:24:00.286 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7169 sec 2024-03-24T02:24:00.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:00 smithi078 ceph-mon[32429]: pgmap v3920: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:24:00.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:00 smithi107 ceph-mon[36485]: pgmap v3920: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:24:00.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:00 smithi106 ceph-mon[36505]: pgmap v3920: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:24:01.286 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7170 sec 2024-03-24T02:24:02.286 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7171 sec 2024-03-24T02:24:02.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:02 smithi078 ceph-mon[32429]: pgmap v3921: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:24:02.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:02 smithi107 ceph-mon[36485]: pgmap v3921: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:24:02.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:02 smithi106 ceph-mon[36505]: pgmap v3921: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:24:03.286 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7172 sec 2024-03-24T02:24:04.286 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7173 sec 2024-03-24T02:24:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6518.198528 secs 2024-03-24T02:24:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:04 smithi078 ceph-mon[32429]: pgmap v3922: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:24:04.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6518.198528 secs 2024-03-24T02:24:04.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:04 smithi107 ceph-mon[36485]: pgmap v3922: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:24:04.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6518.198528 secs 2024-03-24T02:24:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:04 smithi106 ceph-mon[36505]: pgmap v3922: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:24:05.286 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7174 sec 2024-03-24T02:24:06.286 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7175 sec 2024-03-24T02:24:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:06 smithi078 ceph-mon[32429]: pgmap v3923: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:24:06.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:06 smithi107 ceph-mon[36485]: pgmap v3923: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:24:06.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:06 smithi106 ceph-mon[36505]: pgmap v3923: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:24:07.287 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7176 sec 2024-03-24T02:24:08.286 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7177 sec 2024-03-24T02:24:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:08 smithi078 ceph-mon[32429]: pgmap v3924: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:08.840 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:08 smithi107 ceph-mon[36485]: pgmap v3924: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:08 smithi106 ceph-mon[36505]: pgmap v3924: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:09.287 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7178 sec 2024-03-24T02:24:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6523.198565 secs 2024-03-24T02:24:09.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6523.198565 secs 2024-03-24T02:24:09.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6523.198565 secs 2024-03-24T02:24:10.287 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7179 sec 2024-03-24T02:24:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:10 smithi078 ceph-mon[32429]: pgmap v3925: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:10.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:10 smithi107 ceph-mon[36485]: pgmap v3925: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:10.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:10 smithi106 ceph-mon[36505]: pgmap v3925: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:11.287 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7180 sec 2024-03-24T02:24:12.287 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7181 sec 2024-03-24T02:24:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:12 smithi078 ceph-mon[32429]: pgmap v3926: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:12 smithi107 ceph-mon[36485]: pgmap v3926: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:12.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:12 smithi106 ceph-mon[36505]: pgmap v3926: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:13.287 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7182 sec 2024-03-24T02:24:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6528.198620 secs 2024-03-24T02:24:13.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6528.198620 secs 2024-03-24T02:24:13.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6528.198620 secs 2024-03-24T02:24:14.287 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7183 sec 2024-03-24T02:24:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:14 smithi078 ceph-mon[32429]: pgmap v3927: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:14.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:14 smithi107 ceph-mon[36485]: pgmap v3927: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:14.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:14 smithi106 ceph-mon[36505]: pgmap v3927: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:15.287 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7184 sec 2024-03-24T02:24:16.287 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7185 sec 2024-03-24T02:24:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:16 smithi078 ceph-mon[32429]: pgmap v3928: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:16.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:16 smithi107 ceph-mon[36485]: pgmap v3928: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:16.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:16 smithi106 ceph-mon[36505]: pgmap v3928: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:17.287 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7186 sec 2024-03-24T02:24:18.287 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7187 sec 2024-03-24T02:24:18.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:18 smithi107 ceph-mon[36485]: pgmap v3929: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:18.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6533.198710 secs 2024-03-24T02:24:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:18 smithi078 ceph-mon[32429]: pgmap v3929: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6533.198710 secs 2024-03-24T02:24:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:18 smithi106 ceph-mon[36505]: pgmap v3929: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6533.198710 secs 2024-03-24T02:24:19.287 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7188 sec 2024-03-24T02:24:20.287 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7189 sec 2024-03-24T02:24:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:20 smithi078 ceph-mon[32429]: pgmap v3930: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:20.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:20 smithi107 ceph-mon[36485]: pgmap v3930: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:20.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:20 smithi106 ceph-mon[36505]: pgmap v3930: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:21.288 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7190 sec 2024-03-24T02:24:22.288 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7191 sec 2024-03-24T02:24:22.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:22 smithi078 ceph-mon[32429]: pgmap v3931: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:22.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:22 smithi107 ceph-mon[36485]: pgmap v3931: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:22.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:22 smithi106 ceph-mon[36505]: pgmap v3931: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:23.288 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7192 sec 2024-03-24T02:24:23.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6538.198793 secs 2024-03-24T02:24:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6538.198793 secs 2024-03-24T02:24:24.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6538.198793 secs 2024-03-24T02:24:24.288 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7193 sec 2024-03-24T02:24:24.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:24 smithi106 ceph-mon[36505]: pgmap v3932: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:25.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:24 smithi078 ceph-mon[32429]: pgmap v3932: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:24 smithi107 ceph-mon[36485]: pgmap v3932: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:25.288 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7194 sec 2024-03-24T02:24:26.288 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7195 sec 2024-03-24T02:24:26.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:26 smithi078 ceph-mon[32429]: pgmap v3933: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:26.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:26 smithi106 ceph-mon[36505]: pgmap v3933: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:27.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:26 smithi107 ceph-mon[36485]: pgmap v3933: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:27.288 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7196 sec 2024-03-24T02:24:28.288 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7197 sec 2024-03-24T02:24:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:28 smithi106 ceph-mon[36505]: pgmap v3934: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6543.198854 secs 2024-03-24T02:24:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:28 smithi078 ceph-mon[32429]: pgmap v3934: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6543.198854 secs 2024-03-24T02:24:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:28 smithi107 ceph-mon[36485]: pgmap v3934: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6543.198854 secs 2024-03-24T02:24:29.288 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7198 sec 2024-03-24T02:24:30.288 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7199 sec 2024-03-24T02:24:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:30 smithi106 ceph-mon[36505]: pgmap v3935: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:31.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:30 smithi078 ceph-mon[32429]: pgmap v3935: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:31.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:30 smithi107 ceph-mon[36485]: pgmap v3935: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:31.289 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7200 sec 2024-03-24T02:24:32.288 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7201 sec 2024-03-24T02:24:32.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:32 smithi106 ceph-mon[36505]: pgmap v3936: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:33.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:32 smithi078 ceph-mon[32429]: pgmap v3936: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:33.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:32 smithi107 ceph-mon[36485]: pgmap v3936: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:33.289 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7202 sec 2024-03-24T02:24:33.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6548.198914 secs 2024-03-24T02:24:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6548.198914 secs 2024-03-24T02:24:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6548.198914 secs 2024-03-24T02:24:34.289 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7203 sec 2024-03-24T02:24:34.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:34 smithi106 ceph-mon[36505]: pgmap v3937: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:35.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:34 smithi078 ceph-mon[32429]: pgmap v3937: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:35.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:34 smithi107 ceph-mon[36485]: pgmap v3937: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:35.289 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7204 sec 2024-03-24T02:24:36.289 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7205 sec 2024-03-24T02:24:36.895 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:36 smithi107 ceph-mon[36485]: pgmap v3938: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:36.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:36 smithi106 ceph-mon[36505]: pgmap v3938: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:37.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:36 smithi078 ceph-mon[32429]: pgmap v3938: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:37.289 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7206 sec 2024-03-24T02:24:38.289 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7207 sec 2024-03-24T02:24:38.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:38 smithi106 ceph-mon[36505]: pgmap v3939: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:38.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6553.199001 secs 2024-03-24T02:24:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:38 smithi078 ceph-mon[32429]: pgmap v3939: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6553.199001 secs 2024-03-24T02:24:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:38 smithi107 ceph-mon[36485]: pgmap v3939: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6553.199001 secs 2024-03-24T02:24:39.289 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7208 sec 2024-03-24T02:24:40.290 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7209 sec 2024-03-24T02:24:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:40 smithi106 ceph-mon[36505]: pgmap v3940: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:40 smithi078 ceph-mon[32429]: pgmap v3940: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:41.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:40 smithi107 ceph-mon[36485]: pgmap v3940: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:41.290 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7210 sec 2024-03-24T02:24:42.290 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7211 sec 2024-03-24T02:24:42.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:42 smithi106 ceph-mon[36505]: pgmap v3941: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:43.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:42 smithi078 ceph-mon[32429]: pgmap v3941: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:43.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:42 smithi107 ceph-mon[36485]: pgmap v3941: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:43.290 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7212 sec 2024-03-24T02:24:43.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6558.199087 secs 2024-03-24T02:24:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6558.199087 secs 2024-03-24T02:24:44.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6558.199087 secs 2024-03-24T02:24:44.290 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7213 sec 2024-03-24T02:24:44.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:44 smithi106 ceph-mon[36505]: pgmap v3942: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:45.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:44 smithi078 ceph-mon[32429]: pgmap v3942: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:45.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:44 smithi107 ceph-mon[36485]: pgmap v3942: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:45.290 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7214 sec 2024-03-24T02:24:46.290 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7215 sec 2024-03-24T02:24:46.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:46 smithi106 ceph-mon[36505]: pgmap v3943: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:47.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:46 smithi078 ceph-mon[32429]: pgmap v3943: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:47.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:46 smithi107 ceph-mon[36485]: pgmap v3943: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:47.290 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7216 sec 2024-03-24T02:24:48.290 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7217 sec 2024-03-24T02:24:48.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:48 smithi078 ceph-mon[32429]: pgmap v3944: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:48.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6563.199147 secs 2024-03-24T02:24:48.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:48 smithi106 ceph-mon[36505]: pgmap v3944: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:48.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6563.199147 secs 2024-03-24T02:24:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:48 smithi107 ceph-mon[36485]: pgmap v3944: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:49.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6563.199147 secs 2024-03-24T02:24:49.290 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7218 sec 2024-03-24T02:24:50.290 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7219 sec 2024-03-24T02:24:50.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:50 smithi106 ceph-mon[36505]: pgmap v3945: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:50 smithi078 ceph-mon[32429]: pgmap v3945: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:51.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:50 smithi107 ceph-mon[36485]: pgmap v3945: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:51.290 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7220 sec 2024-03-24T02:24:52.290 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7221 sec 2024-03-24T02:24:52.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:52 smithi106 ceph-mon[36505]: pgmap v3946: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:53.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:52 smithi078 ceph-mon[32429]: pgmap v3946: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:53.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:52 smithi107 ceph-mon[36485]: pgmap v3946: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:53.290 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7222 sec 2024-03-24T02:24:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6568.199229 secs 2024-03-24T02:24:54.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6568.199229 secs 2024-03-24T02:24:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6568.199229 secs 2024-03-24T02:24:54.290 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7223 sec 2024-03-24T02:24:55.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:54 smithi078 ceph-mon[32429]: pgmap v3947: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:55.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:54 smithi107 ceph-mon[36485]: pgmap v3947: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:55.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:54 smithi106 ceph-mon[36505]: pgmap v3947: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:55.290 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7224 sec 2024-03-24T02:24:56.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:24:56.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:24:56.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:24:56.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:24:56.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:55 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:24:56.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:24:56.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:24:56.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:24:56.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:24:56.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:55 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:24:56.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:24:56.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:24:56.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:24:56.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:24:56.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:55 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:24:56.290 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7225 sec 2024-03-24T02:24:57.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:56 smithi078 ceph-mon[32429]: pgmap v3948: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:57.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:24:57.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:24:57.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:24:57.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:24:57.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:56 smithi078 ceph-mon[32429]: pgmap v3949: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T02:24:57.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:56 smithi078 ceph-mon[32429]: pgmap v3950: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:24:57.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:24:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:56 smithi107 ceph-mon[36485]: pgmap v3948: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:24:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:24:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:24:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:24:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:56 smithi107 ceph-mon[36485]: pgmap v3949: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T02:24:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:56 smithi107 ceph-mon[36485]: pgmap v3950: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:24:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:24:57.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:56 smithi106 ceph-mon[36505]: pgmap v3948: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:24:57.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:24:57.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:24:57.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:24:57.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:24:57.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:56 smithi106 ceph-mon[36505]: pgmap v3949: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T02:24:57.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:56 smithi106 ceph-mon[36505]: pgmap v3950: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:24:57.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:24:57.291 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7226 sec 2024-03-24T02:24:58.291 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7227 sec 2024-03-24T02:24:59.291 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7228 sec 2024-03-24T02:24:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6573.199315 secs 2024-03-24T02:24:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:24:59 smithi106 ceph-mon[36505]: pgmap v3951: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:24:59.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6573.199315 secs 2024-03-24T02:24:59.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:24:59 smithi078 ceph-mon[32429]: pgmap v3951: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:24:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6573.199315 secs 2024-03-24T02:24:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:24:59 smithi107 ceph-mon[36485]: pgmap v3951: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:25:00.291 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7229 sec 2024-03-24T02:25:01.291 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7230 sec 2024-03-24T02:25:01.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:01 smithi106 ceph-mon[36505]: pgmap v3952: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:25:01.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:01 smithi078 ceph-mon[32429]: pgmap v3952: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:25:01.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:01 smithi107 ceph-mon[36485]: pgmap v3952: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:25:02.291 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7231 sec 2024-03-24T02:25:03.291 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7232 sec 2024-03-24T02:25:03.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:03 smithi106 ceph-mon[36505]: pgmap v3953: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:25:03.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:03 smithi078 ceph-mon[32429]: pgmap v3953: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:25:03.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:03 smithi107 ceph-mon[36485]: pgmap v3953: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:25:04.291 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7233 sec 2024-03-24T02:25:04.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6578.199332 secs 2024-03-24T02:25:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6578.199332 secs 2024-03-24T02:25:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6578.199332 secs 2024-03-24T02:25:05.291 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7234 sec 2024-03-24T02:25:05.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:05 smithi106 ceph-mon[36505]: pgmap v3954: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:25:05.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:05 smithi078 ceph-mon[32429]: pgmap v3954: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:25:05.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:05 smithi107 ceph-mon[36485]: pgmap v3954: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:25:06.291 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7235 sec 2024-03-24T02:25:07.291 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7236 sec 2024-03-24T02:25:07.645 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:07 smithi107 ceph-mon[36485]: pgmap v3955: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:25:07.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:07 smithi106 ceph-mon[36505]: pgmap v3955: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:25:07.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:07 smithi078 ceph-mon[32429]: pgmap v3955: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:25:08.292 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7237 sec 2024-03-24T02:25:09.292 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7238 sec 2024-03-24T02:25:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6583.199420 secs 2024-03-24T02:25:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:09 smithi106 ceph-mon[36505]: pgmap v3956: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6583.199420 secs 2024-03-24T02:25:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:09 smithi078 ceph-mon[32429]: pgmap v3956: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:09.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6583.199420 secs 2024-03-24T02:25:09.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:09 smithi107 ceph-mon[36485]: pgmap v3956: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:10.292 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7239 sec 2024-03-24T02:25:11.292 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7240 sec 2024-03-24T02:25:11.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:11 smithi106 ceph-mon[36505]: pgmap v3957: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:25:11.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:11 smithi078 ceph-mon[32429]: pgmap v3957: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:25:11.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:11 smithi107 ceph-mon[36485]: pgmap v3957: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:25:12.292 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7241 sec 2024-03-24T02:25:13.292 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7242 sec 2024-03-24T02:25:13.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:13 smithi106 ceph-mon[36505]: pgmap v3958: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:25:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:13 smithi078 ceph-mon[32429]: pgmap v3958: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:25:13.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:13 smithi107 ceph-mon[36485]: pgmap v3958: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:25:14.292 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7243 sec 2024-03-24T02:25:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6588.199481 secs 2024-03-24T02:25:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6588.199481 secs 2024-03-24T02:25:14.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6588.199481 secs 2024-03-24T02:25:15.292 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7244 sec 2024-03-24T02:25:15.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:15 smithi106 ceph-mon[36505]: pgmap v3959: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:25:15.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:15 smithi078 ceph-mon[32429]: pgmap v3959: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:25:15.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:15 smithi107 ceph-mon[36485]: pgmap v3959: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:25:16.292 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7245 sec 2024-03-24T02:25:17.292 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7246 sec 2024-03-24T02:25:17.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:17 smithi106 ceph-mon[36505]: pgmap v3960: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:25:17.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:17 smithi078 ceph-mon[32429]: pgmap v3960: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:25:17.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:17 smithi107 ceph-mon[36485]: pgmap v3960: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:25:18.293 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7247 sec 2024-03-24T02:25:19.292 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7248 sec 2024-03-24T02:25:19.642 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6593.199569 secs 2024-03-24T02:25:19.642 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:19 smithi078 ceph-mon[32429]: pgmap v3961: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:25:19.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6593.199569 secs 2024-03-24T02:25:19.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:19 smithi106 ceph-mon[36505]: pgmap v3961: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:25:19.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6593.199569 secs 2024-03-24T02:25:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:19 smithi107 ceph-mon[36485]: pgmap v3961: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:25:20.292 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7249 sec 2024-03-24T02:25:21.292 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7250 sec 2024-03-24T02:25:21.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:21 smithi106 ceph-mon[36505]: pgmap v3962: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:25:21.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:21 smithi078 ceph-mon[32429]: pgmap v3962: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:25:21.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:21 smithi107 ceph-mon[36485]: pgmap v3962: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:25:22.293 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7251 sec 2024-03-24T02:25:23.293 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7252 sec 2024-03-24T02:25:23.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:23 smithi106 ceph-mon[36505]: pgmap v3963: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:23.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:23 smithi078 ceph-mon[32429]: pgmap v3963: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:23.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:23 smithi107 ceph-mon[36485]: pgmap v3963: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:24.293 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7253 sec 2024-03-24T02:25:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6598.199649 secs 2024-03-24T02:25:24.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6598.199649 secs 2024-03-24T02:25:24.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6598.199649 secs 2024-03-24T02:25:25.293 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7254 sec 2024-03-24T02:25:25.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:25 smithi078 ceph-mon[32429]: pgmap v3964: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:25.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:25 smithi107 ceph-mon[36485]: pgmap v3964: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:25.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:25 smithi106 ceph-mon[36505]: pgmap v3964: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:26.293 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7255 sec 2024-03-24T02:25:27.293 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7256 sec 2024-03-24T02:25:27.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:27 smithi078 ceph-mon[32429]: pgmap v3965: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:27.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:27 smithi107 ceph-mon[36485]: pgmap v3965: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:27.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:27 smithi106 ceph-mon[36505]: pgmap v3965: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:28.293 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7257 sec 2024-03-24T02:25:29.293 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7258 sec 2024-03-24T02:25:29.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6603.199705 secs 2024-03-24T02:25:29.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:29 smithi078 ceph-mon[32429]: pgmap v3966: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:29.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6603.199705 secs 2024-03-24T02:25:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:29 smithi107 ceph-mon[36485]: pgmap v3966: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6603.199705 secs 2024-03-24T02:25:29.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:29 smithi106 ceph-mon[36505]: pgmap v3966: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:30.293 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7259 sec 2024-03-24T02:25:31.293 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7260 sec 2024-03-24T02:25:31.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:31 smithi078 ceph-mon[32429]: pgmap v3967: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:31.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:31 smithi107 ceph-mon[36485]: pgmap v3967: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:31.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:31 smithi106 ceph-mon[36505]: pgmap v3967: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:32.294 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7261 sec 2024-03-24T02:25:33.294 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7262 sec 2024-03-24T02:25:33.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:33 smithi078 ceph-mon[32429]: pgmap v3968: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:33 smithi107 ceph-mon[36485]: pgmap v3968: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:33 smithi106 ceph-mon[36505]: pgmap v3968: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:34.294 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7263 sec 2024-03-24T02:25:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6608.199746 secs 2024-03-24T02:25:34.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6608.199746 secs 2024-03-24T02:25:34.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6608.199746 secs 2024-03-24T02:25:35.294 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7264 sec 2024-03-24T02:25:35.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:35 smithi078 ceph-mon[32429]: pgmap v3969: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:35.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:35 smithi107 ceph-mon[36485]: pgmap v3969: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:35.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:35 smithi106 ceph-mon[36505]: pgmap v3969: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:36.294 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7265 sec 2024-03-24T02:25:37.294 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7266 sec 2024-03-24T02:25:37.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:37 smithi078 ceph-mon[32429]: pgmap v3970: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:37.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:37 smithi107 ceph-mon[36485]: pgmap v3970: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:37.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:37 smithi106 ceph-mon[36505]: pgmap v3970: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:38.294 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7267 sec 2024-03-24T02:25:39.294 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7268 sec 2024-03-24T02:25:39.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6613.199834 secs 2024-03-24T02:25:39.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:39 smithi078 ceph-mon[32429]: pgmap v3971: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:39.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6613.199834 secs 2024-03-24T02:25:39.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:39 smithi107 ceph-mon[36485]: pgmap v3971: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:39.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6613.199834 secs 2024-03-24T02:25:39.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:39 smithi106 ceph-mon[36505]: pgmap v3971: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:40.294 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7269 sec 2024-03-24T02:25:41.294 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7270 sec 2024-03-24T02:25:41.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:41 smithi078 ceph-mon[32429]: pgmap v3972: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:41.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:41 smithi107 ceph-mon[36485]: pgmap v3972: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:41.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:41 smithi106 ceph-mon[36505]: pgmap v3972: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:42.294 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7271 sec 2024-03-24T02:25:43.294 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7272 sec 2024-03-24T02:25:43.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:43 smithi078 ceph-mon[32429]: pgmap v3973: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:43.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:43 smithi107 ceph-mon[36485]: pgmap v3973: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:43.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:43 smithi106 ceph-mon[36505]: pgmap v3973: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:44.294 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7273 sec 2024-03-24T02:25:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6618.199919 secs 2024-03-24T02:25:44.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6618.199919 secs 2024-03-24T02:25:44.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6618.199919 secs 2024-03-24T02:25:45.295 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7274 sec 2024-03-24T02:25:45.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:45 smithi078 ceph-mon[32429]: pgmap v3974: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:45.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:45 smithi107 ceph-mon[36485]: pgmap v3974: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:45.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:45 smithi106 ceph-mon[36505]: pgmap v3974: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:46.295 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7275 sec 2024-03-24T02:25:47.295 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7276 sec 2024-03-24T02:25:47.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:47 smithi078 ceph-mon[32429]: pgmap v3975: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:47.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:47 smithi107 ceph-mon[36485]: pgmap v3975: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:47.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:47 smithi106 ceph-mon[36505]: pgmap v3975: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:48.295 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7277 sec 2024-03-24T02:25:49.295 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7278 sec 2024-03-24T02:25:49.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6623.200008 secs 2024-03-24T02:25:49.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:49 smithi078 ceph-mon[32429]: pgmap v3976: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:49.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6623.200008 secs 2024-03-24T02:25:49.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:49 smithi107 ceph-mon[36485]: pgmap v3976: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:49.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6623.200008 secs 2024-03-24T02:25:49.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:49 smithi106 ceph-mon[36505]: pgmap v3976: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:50.295 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7279 sec 2024-03-24T02:25:51.295 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7280 sec 2024-03-24T02:25:51.704 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:51 smithi106 ceph-mon[36505]: pgmap v3977: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:51.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:51 smithi078 ceph-mon[32429]: pgmap v3977: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:51.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:51 smithi107 ceph-mon[36485]: pgmap v3977: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:52.295 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7281 sec 2024-03-24T02:25:53.295 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7282 sec 2024-03-24T02:25:53.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:53 smithi078 ceph-mon[32429]: pgmap v3978: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:53.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:53 smithi107 ceph-mon[36485]: pgmap v3978: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:53.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:53 smithi106 ceph-mon[36505]: pgmap v3978: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:54.295 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7283 sec 2024-03-24T02:25:54.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6628.200095 secs 2024-03-24T02:25:54.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6628.200095 secs 2024-03-24T02:25:54.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6628.200095 secs 2024-03-24T02:25:55.296 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7284 sec 2024-03-24T02:25:55.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:55 smithi078 ceph-mon[32429]: pgmap v3979: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:55.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:55 smithi107 ceph-mon[36485]: pgmap v3979: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:55.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:55 smithi106 ceph-mon[36505]: pgmap v3979: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:25:56.296 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7285 sec 2024-03-24T02:25:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:25:56.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:25:56.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:25:57.296 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7286 sec 2024-03-24T02:25:57.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:57 smithi078 ceph-mon[32429]: pgmap v3980: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:57.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:57 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:25:57.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:57 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:25:57.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:57 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:25:57.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:57 smithi107 ceph-mon[36485]: pgmap v3980: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:57.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:57 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:25:57.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:57 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:25:57.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:57 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:25:57.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:57 smithi106 ceph-mon[36505]: pgmap v3980: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:57.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:57 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:25:57.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:57 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:25:57.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:57 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:25:58.296 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7287 sec 2024-03-24T02:25:58.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:58 smithi078 ceph-mon[32429]: pgmap v3981: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:25:58.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:58 smithi078 ceph-mon[32429]: pgmap v3982: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:25:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:58 smithi107 ceph-mon[36485]: pgmap v3981: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:25:58.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:58 smithi107 ceph-mon[36485]: pgmap v3982: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:25:58.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:58 smithi106 ceph-mon[36505]: pgmap v3981: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:25:58.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:58 smithi106 ceph-mon[36505]: pgmap v3982: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:25:59.296 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7288 sec 2024-03-24T02:25:59.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:25:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6633.200156 secs 2024-03-24T02:25:59.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:25:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6633.200156 secs 2024-03-24T02:25:59.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:25:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6633.200156 secs 2024-03-24T02:26:00.296 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7289 sec 2024-03-24T02:26:00.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:00 smithi078 ceph-mon[32429]: pgmap v3983: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:26:00.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:00 smithi107 ceph-mon[36485]: pgmap v3983: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:26:00.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:00 smithi106 ceph-mon[36505]: pgmap v3983: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:26:01.296 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7290 sec 2024-03-24T02:26:02.296 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7291 sec 2024-03-24T02:26:02.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:02 smithi078 ceph-mon[32429]: pgmap v3984: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:26:02.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:02 smithi107 ceph-mon[36485]: pgmap v3984: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:26:02.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:02 smithi106 ceph-mon[36505]: pgmap v3984: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:26:03.296 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7292 sec 2024-03-24T02:26:04.296 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7293 sec 2024-03-24T02:26:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6638.200216 secs 2024-03-24T02:26:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:04 smithi078 ceph-mon[32429]: pgmap v3985: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:26:04.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6638.200216 secs 2024-03-24T02:26:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:04 smithi107 ceph-mon[36485]: pgmap v3985: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:26:04.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6638.200216 secs 2024-03-24T02:26:04.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:04 smithi106 ceph-mon[36505]: pgmap v3985: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:26:05.296 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7294 sec 2024-03-24T02:26:06.296 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7295 sec 2024-03-24T02:26:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:06 smithi078 ceph-mon[32429]: pgmap v3986: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:26:06.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:06 smithi107 ceph-mon[36485]: pgmap v3986: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:26:06.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:06 smithi106 ceph-mon[36505]: pgmap v3986: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:26:07.296 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7296 sec 2024-03-24T02:26:08.297 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7297 sec 2024-03-24T02:26:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:08 smithi078 ceph-mon[32429]: pgmap v3987: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:26:08.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:08 smithi107 ceph-mon[36485]: pgmap v3987: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:26:08.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:08 smithi106 ceph-mon[36505]: pgmap v3987: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:26:09.297 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7298 sec 2024-03-24T02:26:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6643.200272 secs 2024-03-24T02:26:09.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6643.200272 secs 2024-03-24T02:26:09.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6643.200272 secs 2024-03-24T02:26:10.297 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7299 sec 2024-03-24T02:26:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:10 smithi078 ceph-mon[32429]: pgmap v3988: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:10 smithi107 ceph-mon[36485]: pgmap v3988: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:10 smithi106 ceph-mon[36505]: pgmap v3988: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:11.297 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7300 sec 2024-03-24T02:26:12.297 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7301 sec 2024-03-24T02:26:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:12 smithi078 ceph-mon[32429]: pgmap v3989: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:12.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:12 smithi107 ceph-mon[36485]: pgmap v3989: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:12.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:12 smithi106 ceph-mon[36505]: pgmap v3989: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:13.297 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7302 sec 2024-03-24T02:26:14.297 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7303 sec 2024-03-24T02:26:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6648.200332 secs 2024-03-24T02:26:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:14 smithi078 ceph-mon[32429]: pgmap v3990: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:14.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6648.200332 secs 2024-03-24T02:26:14.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:14 smithi107 ceph-mon[36485]: pgmap v3990: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:14.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6648.200332 secs 2024-03-24T02:26:14.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:14 smithi106 ceph-mon[36505]: pgmap v3990: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:15.297 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7304 sec 2024-03-24T02:26:16.297 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7305 sec 2024-03-24T02:26:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:16 smithi078 ceph-mon[32429]: pgmap v3991: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:16.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:16 smithi107 ceph-mon[36485]: pgmap v3991: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:16.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:16 smithi106 ceph-mon[36505]: pgmap v3991: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:17.297 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7306 sec 2024-03-24T02:26:18.298 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7307 sec 2024-03-24T02:26:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:18 smithi078 ceph-mon[32429]: pgmap v3992: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:18.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:18 smithi107 ceph-mon[36485]: pgmap v3992: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:18.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:18 smithi106 ceph-mon[36505]: pgmap v3992: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:19.297 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7308 sec 2024-03-24T02:26:19.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6653.200417 secs 2024-03-24T02:26:19.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6653.200417 secs 2024-03-24T02:26:19.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6653.200417 secs 2024-03-24T02:26:20.297 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7309 sec 2024-03-24T02:26:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:20 smithi078 ceph-mon[32429]: pgmap v3993: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:20 smithi107 ceph-mon[36485]: pgmap v3993: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:20.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:20 smithi106 ceph-mon[36505]: pgmap v3993: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:21.298 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7310 sec 2024-03-24T02:26:22.298 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7311 sec 2024-03-24T02:26:22.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:22 smithi078 ceph-mon[32429]: pgmap v3994: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:22.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:22 smithi107 ceph-mon[36485]: pgmap v3994: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:22.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:22 smithi106 ceph-mon[36505]: pgmap v3994: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:23.298 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7312 sec 2024-03-24T02:26:24.298 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7313 sec 2024-03-24T02:26:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6658.200454 secs 2024-03-24T02:26:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:24 smithi078 ceph-mon[32429]: pgmap v3995: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:24.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6658.200454 secs 2024-03-24T02:26:24.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:24 smithi107 ceph-mon[36485]: pgmap v3995: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6658.200454 secs 2024-03-24T02:26:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:24 smithi106 ceph-mon[36505]: pgmap v3995: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:25.298 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7314 sec 2024-03-24T02:26:26.298 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7315 sec 2024-03-24T02:26:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:26 smithi078 ceph-mon[32429]: pgmap v3996: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:26.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:26 smithi107 ceph-mon[36485]: pgmap v3996: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:26.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:26 smithi106 ceph-mon[36505]: pgmap v3996: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:27.298 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7316 sec 2024-03-24T02:26:28.298 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7317 sec 2024-03-24T02:26:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:28 smithi078 ceph-mon[32429]: pgmap v3997: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:28.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:28 smithi107 ceph-mon[36485]: pgmap v3997: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:28.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:28 smithi106 ceph-mon[36505]: pgmap v3997: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:29.298 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7318 sec 2024-03-24T02:26:29.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6663.200538 secs 2024-03-24T02:26:29.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6663.200538 secs 2024-03-24T02:26:29.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6663.200538 secs 2024-03-24T02:26:30.298 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7319 sec 2024-03-24T02:26:30.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:30 smithi078 ceph-mon[32429]: pgmap v3998: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:30.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:30 smithi107 ceph-mon[36485]: pgmap v3998: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:30.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:30 smithi106 ceph-mon[36505]: pgmap v3998: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:31.298 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7320 sec 2024-03-24T02:26:32.299 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7321 sec 2024-03-24T02:26:32.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:32 smithi078 ceph-mon[32429]: pgmap v3999: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:32.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:32 smithi107 ceph-mon[36485]: pgmap v3999: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:32.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:32 smithi106 ceph-mon[36505]: pgmap v3999: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:33.299 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7322 sec 2024-03-24T02:26:34.299 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7323 sec 2024-03-24T02:26:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6668.200629 secs 2024-03-24T02:26:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:34 smithi078 ceph-mon[32429]: pgmap v4000: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:34.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6668.200629 secs 2024-03-24T02:26:34.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:34 smithi107 ceph-mon[36485]: pgmap v4000: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:34.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6668.200629 secs 2024-03-24T02:26:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:34 smithi106 ceph-mon[36505]: pgmap v4000: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:35.299 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7324 sec 2024-03-24T02:26:36.299 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7325 sec 2024-03-24T02:26:36.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:36 smithi078 ceph-mon[32429]: pgmap v4001: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:36.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:36 smithi107 ceph-mon[36485]: pgmap v4001: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:36.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:36 smithi106 ceph-mon[36505]: pgmap v4001: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:37.299 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7326 sec 2024-03-24T02:26:38.299 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7327 sec 2024-03-24T02:26:38.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:38 smithi078 ceph-mon[32429]: pgmap v4002: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:38.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:38 smithi107 ceph-mon[36485]: pgmap v4002: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:38.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:38 smithi106 ceph-mon[36505]: pgmap v4002: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:39.299 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7328 sec 2024-03-24T02:26:39.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6673.200715 secs 2024-03-24T02:26:39.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6673.200715 secs 2024-03-24T02:26:39.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6673.200715 secs 2024-03-24T02:26:40.299 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7329 sec 2024-03-24T02:26:40.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:40 smithi078 ceph-mon[32429]: pgmap v4003: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:40.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:40 smithi107 ceph-mon[36485]: pgmap v4003: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:40.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:40 smithi106 ceph-mon[36505]: pgmap v4003: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:41.300 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7330 sec 2024-03-24T02:26:42.300 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7331 sec 2024-03-24T02:26:42.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:42 smithi078 ceph-mon[32429]: pgmap v4004: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:42.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:42 smithi107 ceph-mon[36485]: pgmap v4004: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:42.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:42 smithi106 ceph-mon[36505]: pgmap v4004: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:43.300 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7332 sec 2024-03-24T02:26:44.300 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7333 sec 2024-03-24T02:26:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6678.200803 secs 2024-03-24T02:26:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:44 smithi078 ceph-mon[32429]: pgmap v4005: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:44.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6678.200803 secs 2024-03-24T02:26:44.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:44 smithi107 ceph-mon[36485]: pgmap v4005: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:44.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6678.200803 secs 2024-03-24T02:26:44.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:44 smithi106 ceph-mon[36505]: pgmap v4005: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:45.300 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7334 sec 2024-03-24T02:26:46.300 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7335 sec 2024-03-24T02:26:46.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:46 smithi078 ceph-mon[32429]: pgmap v4006: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:46.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:46 smithi107 ceph-mon[36485]: pgmap v4006: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:46 smithi106 ceph-mon[36505]: pgmap v4006: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:47.300 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7336 sec 2024-03-24T02:26:48.300 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7337 sec 2024-03-24T02:26:48.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:48 smithi078 ceph-mon[32429]: pgmap v4007: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:48.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:48 smithi107 ceph-mon[36485]: pgmap v4007: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:48 smithi106 ceph-mon[36505]: pgmap v4007: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:49.300 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7338 sec 2024-03-24T02:26:49.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6683.200861 secs 2024-03-24T02:26:49.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6683.200861 secs 2024-03-24T02:26:49.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6683.200861 secs 2024-03-24T02:26:50.300 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7339 sec 2024-03-24T02:26:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:50 smithi078 ceph-mon[32429]: pgmap v4008: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:50.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:50 smithi107 ceph-mon[36485]: pgmap v4008: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:50.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:50 smithi106 ceph-mon[36505]: pgmap v4008: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:51.301 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7340 sec 2024-03-24T02:26:52.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:51 smithi106 ceph-mon[36505]: pgmap v4009: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:52.300 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7341 sec 2024-03-24T02:26:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:51 smithi078 ceph-mon[32429]: pgmap v4009: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:52.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:51 smithi107 ceph-mon[36485]: pgmap v4009: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:53.300 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7342 sec 2024-03-24T02:26:54.300 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7343 sec 2024-03-24T02:26:54.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6688.200921 secs 2024-03-24T02:26:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:54 smithi106 ceph-mon[36505]: pgmap v4010: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6688.200921 secs 2024-03-24T02:26:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:54 smithi078 ceph-mon[32429]: pgmap v4010: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6688.200921 secs 2024-03-24T02:26:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:54 smithi107 ceph-mon[36485]: pgmap v4010: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:55.300 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7344 sec 2024-03-24T02:26:56.301 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7345 sec 2024-03-24T02:26:56.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:56 smithi078 ceph-mon[32429]: pgmap v4011: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:56.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:56 smithi107 ceph-mon[36485]: pgmap v4011: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:56.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:56 smithi106 ceph-mon[36505]: pgmap v4011: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:26:57.300 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7346 sec 2024-03-24T02:26:58.301 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7347 sec 2024-03-24T02:26:58.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:58 smithi078 ceph-mon[32429]: pgmap v4012: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:58.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:58 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:26:58.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:58 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:26:58.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:58 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:26:58.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:58 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:26:58.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:58 smithi107 ceph-mon[36485]: pgmap v4012: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:58.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:58 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:26:58.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:58 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:26:58.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:58 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:26:58.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:58 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:26:58.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:58 smithi106 ceph-mon[36505]: pgmap v4012: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:58.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:58 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:26:58.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:58 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:26:58.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:58 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:26:58.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:58 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:26:59.301 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7348 sec 2024-03-24T02:26:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:59 smithi078 ceph-mon[32429]: pgmap v4013: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:26:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:59 smithi078 ceph-mon[32429]: pgmap v4014: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:26:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:26:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6693.201007 secs 2024-03-24T02:26:59.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:59 smithi107 ceph-mon[36485]: pgmap v4013: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:26:59.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:59 smithi107 ceph-mon[36485]: pgmap v4014: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:26:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:26:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6693.201007 secs 2024-03-24T02:26:59.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:59 smithi106 ceph-mon[36505]: pgmap v4013: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:26:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:59 smithi106 ceph-mon[36505]: pgmap v4014: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:26:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:26:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6693.201007 secs 2024-03-24T02:27:00.301 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7349 sec 2024-03-24T02:27:01.301 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7350 sec 2024-03-24T02:27:01.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:01 smithi078 ceph-mon[32429]: pgmap v4015: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:27:01.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:01 smithi107 ceph-mon[36485]: pgmap v4015: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:27:01.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:01 smithi106 ceph-mon[36505]: pgmap v4015: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:27:02.301 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7351 sec 2024-03-24T02:27:03.301 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7352 sec 2024-03-24T02:27:03.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:03 smithi078 ceph-mon[32429]: pgmap v4016: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:27:03.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:03 smithi107 ceph-mon[36485]: pgmap v4016: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:27:03.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:03 smithi106 ceph-mon[36505]: pgmap v4016: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:27:04.301 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7353 sec 2024-03-24T02:27:04.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6698.201064 secs 2024-03-24T02:27:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6698.201064 secs 2024-03-24T02:27:04.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6698.201064 secs 2024-03-24T02:27:05.301 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7354 sec 2024-03-24T02:27:05.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:05 smithi078 ceph-mon[32429]: pgmap v4017: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:27:05.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:05 smithi107 ceph-mon[36485]: pgmap v4017: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:27:05.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:05 smithi106 ceph-mon[36505]: pgmap v4017: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:27:06.301 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7355 sec 2024-03-24T02:27:07.302 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7356 sec 2024-03-24T02:27:07.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:07 smithi078 ceph-mon[32429]: pgmap v4018: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:27:07.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:07 smithi107 ceph-mon[36485]: pgmap v4018: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:27:07.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:07 smithi106 ceph-mon[36505]: pgmap v4018: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:27:08.302 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7357 sec 2024-03-24T02:27:09.302 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7358 sec 2024-03-24T02:27:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:09 smithi078 ceph-mon[32429]: pgmap v4019: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:27:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6703.201146 secs 2024-03-24T02:27:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:09 smithi107 ceph-mon[36485]: pgmap v4019: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:27:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6703.201146 secs 2024-03-24T02:27:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:09 smithi106 ceph-mon[36505]: pgmap v4019: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:27:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6703.201146 secs 2024-03-24T02:27:10.302 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7359 sec 2024-03-24T02:27:11.302 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7360 sec 2024-03-24T02:27:11.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:11 smithi078 ceph-mon[32429]: pgmap v4020: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:11.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:11 smithi107 ceph-mon[36485]: pgmap v4020: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:11.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:11 smithi106 ceph-mon[36505]: pgmap v4020: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:12.302 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7361 sec 2024-03-24T02:27:13.302 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7362 sec 2024-03-24T02:27:13.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:13 smithi078 ceph-mon[32429]: pgmap v4021: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:13.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:13 smithi107 ceph-mon[36485]: pgmap v4021: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:13.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:13 smithi106 ceph-mon[36505]: pgmap v4021: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:14.302 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7363 sec 2024-03-24T02:27:14.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6708.201232 secs 2024-03-24T02:27:14.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6708.201232 secs 2024-03-24T02:27:14.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6708.201232 secs 2024-03-24T02:27:15.302 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7364 sec 2024-03-24T02:27:15.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:15 smithi078 ceph-mon[32429]: pgmap v4022: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:15.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:15 smithi107 ceph-mon[36485]: pgmap v4022: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:15.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:15 smithi106 ceph-mon[36505]: pgmap v4022: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:16.302 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7365 sec 2024-03-24T02:27:17.302 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7366 sec 2024-03-24T02:27:17.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:17 smithi078 ceph-mon[32429]: pgmap v4023: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:17.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:17 smithi107 ceph-mon[36485]: pgmap v4023: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:17.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:17 smithi106 ceph-mon[36505]: pgmap v4023: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:18.302 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7367 sec 2024-03-24T02:27:19.303 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7368 sec 2024-03-24T02:27:19.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:19 smithi078 ceph-mon[32429]: pgmap v4024: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:19.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6713.201312 secs 2024-03-24T02:27:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:19 smithi107 ceph-mon[36485]: pgmap v4024: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6713.201312 secs 2024-03-24T02:27:19.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:19 smithi106 ceph-mon[36505]: pgmap v4024: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6713.201312 secs 2024-03-24T02:27:20.303 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7369 sec 2024-03-24T02:27:21.303 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7370 sec 2024-03-24T02:27:21.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:21 smithi078 ceph-mon[32429]: pgmap v4025: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:21.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:21 smithi107 ceph-mon[36485]: pgmap v4025: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:21.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:21 smithi106 ceph-mon[36505]: pgmap v4025: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:22.303 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7371 sec 2024-03-24T02:27:23.303 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7372 sec 2024-03-24T02:27:23.453 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:23 smithi106 ceph-mon[36505]: pgmap v4026: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:23.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:23 smithi078 ceph-mon[32429]: pgmap v4026: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:23.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:23 smithi107 ceph-mon[36485]: pgmap v4026: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:24.303 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7373 sec 2024-03-24T02:27:24.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6718.201369 secs 2024-03-24T02:27:24.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6718.201369 secs 2024-03-24T02:27:24.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6718.201369 secs 2024-03-24T02:27:25.303 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7374 sec 2024-03-24T02:27:25.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:25 smithi078 ceph-mon[32429]: pgmap v4027: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:25.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:25 smithi107 ceph-mon[36485]: pgmap v4027: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:25.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:25 smithi106 ceph-mon[36505]: pgmap v4027: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:26.303 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7375 sec 2024-03-24T02:27:27.303 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7376 sec 2024-03-24T02:27:27.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:27 smithi078 ceph-mon[32429]: pgmap v4028: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:27.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:27 smithi107 ceph-mon[36485]: pgmap v4028: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:27.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:27 smithi106 ceph-mon[36505]: pgmap v4028: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:28.303 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7377 sec 2024-03-24T02:27:29.303 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7378 sec 2024-03-24T02:27:29.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:29 smithi078 ceph-mon[32429]: pgmap v4029: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:29.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6723.201429 secs 2024-03-24T02:27:29.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:29 smithi107 ceph-mon[36485]: pgmap v4029: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6723.201429 secs 2024-03-24T02:27:29.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:29 smithi106 ceph-mon[36505]: pgmap v4029: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:29.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6723.201429 secs 2024-03-24T02:27:30.303 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7379 sec 2024-03-24T02:27:31.304 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7380 sec 2024-03-24T02:27:31.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:31 smithi078 ceph-mon[32429]: pgmap v4030: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:31.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:31 smithi107 ceph-mon[36485]: pgmap v4030: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:31.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:31 smithi106 ceph-mon[36505]: pgmap v4030: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:32.304 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7381 sec 2024-03-24T02:27:33.304 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7382 sec 2024-03-24T02:27:33.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:33 smithi078 ceph-mon[32429]: pgmap v4031: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:33.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:33 smithi107 ceph-mon[36485]: pgmap v4031: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:33.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:33 smithi106 ceph-mon[36505]: pgmap v4031: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:34.304 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7383 sec 2024-03-24T02:27:34.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6728.201484 secs 2024-03-24T02:27:34.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6728.201484 secs 2024-03-24T02:27:34.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6728.201484 secs 2024-03-24T02:27:35.304 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7384 sec 2024-03-24T02:27:35.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:35 smithi078 ceph-mon[32429]: pgmap v4032: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:35.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:35 smithi107 ceph-mon[36485]: pgmap v4032: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:35.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:35 smithi106 ceph-mon[36505]: pgmap v4032: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:36.304 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7385 sec 2024-03-24T02:27:37.304 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7386 sec 2024-03-24T02:27:37.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:37 smithi078 ceph-mon[32429]: pgmap v4033: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:37.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:37 smithi107 ceph-mon[36485]: pgmap v4033: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:37.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:37 smithi106 ceph-mon[36505]: pgmap v4033: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:38.304 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7387 sec 2024-03-24T02:27:39.304 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7388 sec 2024-03-24T02:27:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:39 smithi078 ceph-mon[32429]: pgmap v4034: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6733.201545 secs 2024-03-24T02:27:39.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:39 smithi107 ceph-mon[36485]: pgmap v4034: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:39.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6733.201545 secs 2024-03-24T02:27:39.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:39 smithi106 ceph-mon[36505]: pgmap v4034: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:39.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6733.201545 secs 2024-03-24T02:27:40.304 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7389 sec 2024-03-24T02:27:41.304 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7390 sec 2024-03-24T02:27:41.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:41 smithi078 ceph-mon[32429]: pgmap v4035: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:41.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:41 smithi107 ceph-mon[36485]: pgmap v4035: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:41.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:41 smithi106 ceph-mon[36505]: pgmap v4035: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:42.305 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7391 sec 2024-03-24T02:27:43.305 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7392 sec 2024-03-24T02:27:43.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:43 smithi078 ceph-mon[32429]: pgmap v4036: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:43.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:43 smithi107 ceph-mon[36485]: pgmap v4036: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:43.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:43 smithi106 ceph-mon[36505]: pgmap v4036: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:44.305 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7393 sec 2024-03-24T02:27:44.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6738.201602 secs 2024-03-24T02:27:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6738.201602 secs 2024-03-24T02:27:44.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6738.201602 secs 2024-03-24T02:27:45.305 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7394 sec 2024-03-24T02:27:45.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:45 smithi078 ceph-mon[32429]: pgmap v4037: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:45.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:45 smithi107 ceph-mon[36485]: pgmap v4037: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:45.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:45 smithi106 ceph-mon[36505]: pgmap v4037: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:46.305 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7395 sec 2024-03-24T02:27:47.305 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7396 sec 2024-03-24T02:27:47.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:47 smithi078 ceph-mon[32429]: pgmap v4038: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:47.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:47 smithi107 ceph-mon[36485]: pgmap v4038: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:47.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:47 smithi106 ceph-mon[36505]: pgmap v4038: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:48.305 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7397 sec 2024-03-24T02:27:49.305 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7398 sec 2024-03-24T02:27:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:49 smithi078 ceph-mon[32429]: pgmap v4039: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6743.201663 secs 2024-03-24T02:27:49.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:49 smithi107 ceph-mon[36485]: pgmap v4039: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6743.201663 secs 2024-03-24T02:27:49.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:49 smithi106 ceph-mon[36505]: pgmap v4039: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:49.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6743.201663 secs 2024-03-24T02:27:50.305 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7399 sec 2024-03-24T02:27:51.305 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7400 sec 2024-03-24T02:27:51.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:51 smithi078 ceph-mon[32429]: pgmap v4040: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:51.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:51 smithi107 ceph-mon[36485]: pgmap v4040: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:51.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:51 smithi106 ceph-mon[36505]: pgmap v4040: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:52.305 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7401 sec 2024-03-24T02:27:53.305 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7402 sec 2024-03-24T02:27:53.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:53 smithi078 ceph-mon[32429]: pgmap v4041: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:53.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:53 smithi107 ceph-mon[36485]: pgmap v4041: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:53.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:53 smithi106 ceph-mon[36505]: pgmap v4041: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:54.306 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7403 sec 2024-03-24T02:27:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6748.201752 secs 2024-03-24T02:27:54.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6748.201752 secs 2024-03-24T02:27:54.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6748.201752 secs 2024-03-24T02:27:55.306 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7404 sec 2024-03-24T02:27:55.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:55 smithi078 ceph-mon[32429]: pgmap v4042: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:55.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:55 smithi107 ceph-mon[36485]: pgmap v4042: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:55.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:55 smithi106 ceph-mon[36505]: pgmap v4042: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:56.306 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7405 sec 2024-03-24T02:27:57.306 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7406 sec 2024-03-24T02:27:57.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:57 smithi078 ceph-mon[32429]: pgmap v4043: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:57.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:57 smithi107 ceph-mon[36485]: pgmap v4043: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:57.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:57 smithi106 ceph-mon[36505]: pgmap v4043: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:58.306 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7407 sec 2024-03-24T02:27:58.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:58 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:27:58.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:58 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:27:58.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:58 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:27:59.306 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7408 sec 2024-03-24T02:27:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:59 smithi078 ceph-mon[32429]: pgmap v4044: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6753.201813 secs 2024-03-24T02:27:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:59 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:27:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:59 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:27:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:59 smithi078 ceph-mon[32429]: pgmap v4045: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:27:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:59 smithi078 ceph-mon[32429]: pgmap v4046: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:27:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:27:59 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:27:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:59 smithi107 ceph-mon[36485]: pgmap v4044: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6753.201813 secs 2024-03-24T02:27:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:59 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:27:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:59 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:27:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:59 smithi107 ceph-mon[36485]: pgmap v4045: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:27:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:59 smithi107 ceph-mon[36485]: pgmap v4046: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:27:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:27:59 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:27:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:59 smithi106 ceph-mon[36505]: pgmap v4044: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:27:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6753.201813 secs 2024-03-24T02:27:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:59 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:27:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:59 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:27:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:59 smithi106 ceph-mon[36505]: pgmap v4045: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:27:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:59 smithi106 ceph-mon[36505]: pgmap v4046: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:27:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:27:59 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:28:00.306 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7409 sec 2024-03-24T02:28:01.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:00 smithi078 ceph-mon[32429]: pgmap v4047: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:28:01.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:00 smithi107 ceph-mon[36485]: pgmap v4047: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:28:01.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:00 smithi106 ceph-mon[36505]: pgmap v4047: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:28:01.306 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7410 sec 2024-03-24T02:28:02.306 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7411 sec 2024-03-24T02:28:03.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:02 smithi078 ceph-mon[32429]: pgmap v4048: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:28:03.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:02 smithi107 ceph-mon[36485]: pgmap v4048: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:28:03.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:02 smithi106 ceph-mon[36505]: pgmap v4048: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:28:03.306 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7412 sec 2024-03-24T02:28:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6758.201895 secs 2024-03-24T02:28:04.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6758.201895 secs 2024-03-24T02:28:04.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6758.201895 secs 2024-03-24T02:28:04.306 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7413 sec 2024-03-24T02:28:05.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:04 smithi078 ceph-mon[32429]: pgmap v4049: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:28:05.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:04 smithi107 ceph-mon[36485]: pgmap v4049: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:28:05.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:04 smithi106 ceph-mon[36505]: pgmap v4049: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:28:05.306 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7414 sec 2024-03-24T02:28:06.307 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7415 sec 2024-03-24T02:28:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:06 smithi078 ceph-mon[32429]: pgmap v4050: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:28:07.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:06 smithi107 ceph-mon[36485]: pgmap v4050: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:28:07.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:06 smithi106 ceph-mon[36505]: pgmap v4050: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:28:07.306 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7416 sec 2024-03-24T02:28:08.307 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7417 sec 2024-03-24T02:28:08.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6763.201953 secs 2024-03-24T02:28:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6763.201953 secs 2024-03-24T02:28:09.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6763.201953 secs 2024-03-24T02:28:09.307 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7418 sec 2024-03-24T02:28:09.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:09 smithi106 ceph-mon[36505]: pgmap v4051: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:28:09.994 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:09 smithi107 ceph-mon[36485]: pgmap v4051: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:28:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:09 smithi078 ceph-mon[32429]: pgmap v4051: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:28:10.307 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7419 sec 2024-03-24T02:28:11.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:10 smithi078 ceph-mon[32429]: pgmap v4052: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:11.124 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:10 smithi107 ceph-mon[36485]: pgmap v4052: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:11.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:10 smithi106 ceph-mon[36505]: pgmap v4052: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:11.307 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7420 sec 2024-03-24T02:28:12.307 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7421 sec 2024-03-24T02:28:13.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:12 smithi078 ceph-mon[32429]: pgmap v4053: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:13.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:12 smithi107 ceph-mon[36485]: pgmap v4053: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:13.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:12 smithi106 ceph-mon[36505]: pgmap v4053: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:13.307 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7422 sec 2024-03-24T02:28:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6768.202043 secs 2024-03-24T02:28:14.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6768.202043 secs 2024-03-24T02:28:14.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6768.202043 secs 2024-03-24T02:28:14.307 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7423 sec 2024-03-24T02:28:15.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:14 smithi078 ceph-mon[32429]: pgmap v4054: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:15.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:14 smithi107 ceph-mon[36485]: pgmap v4054: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:15.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:14 smithi106 ceph-mon[36505]: pgmap v4054: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:15.307 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7424 sec 2024-03-24T02:28:16.308 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7425 sec 2024-03-24T02:28:17.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:16 smithi078 ceph-mon[32429]: pgmap v4055: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:17.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:16 smithi107 ceph-mon[36485]: pgmap v4055: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:17.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:16 smithi106 ceph-mon[36505]: pgmap v4055: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:17.307 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7426 sec 2024-03-24T02:28:18.307 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7427 sec 2024-03-24T02:28:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6773.202129 secs 2024-03-24T02:28:19.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6773.202129 secs 2024-03-24T02:28:19.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6773.202129 secs 2024-03-24T02:28:19.307 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7428 sec 2024-03-24T02:28:19.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:19 smithi106 ceph-mon[36505]: pgmap v4056: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:19.991 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:19 smithi078 ceph-mon[32429]: pgmap v4056: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:20.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:19 smithi107 ceph-mon[36485]: pgmap v4056: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:20.308 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7429 sec 2024-03-24T02:28:21.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:20 smithi078 ceph-mon[32429]: pgmap v4057: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:21.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:20 smithi107 ceph-mon[36485]: pgmap v4057: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:21.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:20 smithi106 ceph-mon[36505]: pgmap v4057: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:21.308 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7430 sec 2024-03-24T02:28:22.308 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7431 sec 2024-03-24T02:28:23.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:22 smithi078 ceph-mon[32429]: pgmap v4058: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:23.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:22 smithi107 ceph-mon[36485]: pgmap v4058: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:23.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:22 smithi106 ceph-mon[36505]: pgmap v4058: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:23.308 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7432 sec 2024-03-24T02:28:24.052 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6778.202217 secs 2024-03-24T02:28:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6778.202217 secs 2024-03-24T02:28:24.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6778.202217 secs 2024-03-24T02:28:24.308 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7433 sec 2024-03-24T02:28:25.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:24 smithi078 ceph-mon[32429]: pgmap v4059: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:25.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:24 smithi107 ceph-mon[36485]: pgmap v4059: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:25.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:24 smithi106 ceph-mon[36505]: pgmap v4059: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:25.308 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7434 sec 2024-03-24T02:28:26.308 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7435 sec 2024-03-24T02:28:27.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:26 smithi078 ceph-mon[32429]: pgmap v4060: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:27.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:26 smithi107 ceph-mon[36485]: pgmap v4060: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:27.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:26 smithi106 ceph-mon[36505]: pgmap v4060: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:27.308 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7436 sec 2024-03-24T02:28:28.308 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7437 sec 2024-03-24T02:28:28.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6783.202303 secs 2024-03-24T02:28:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6783.202303 secs 2024-03-24T02:28:29.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6783.202303 secs 2024-03-24T02:28:29.308 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7438 sec 2024-03-24T02:28:29.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:29 smithi106 ceph-mon[36505]: pgmap v4061: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:30.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:29 smithi078 ceph-mon[32429]: pgmap v4061: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:30.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:29 smithi107 ceph-mon[36485]: pgmap v4061: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:30.308 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7439 sec 2024-03-24T02:28:31.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:30 smithi078 ceph-mon[32429]: pgmap v4062: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:31.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:30 smithi107 ceph-mon[36485]: pgmap v4062: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:31.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:30 smithi106 ceph-mon[36505]: pgmap v4062: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:31.309 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7440 sec 2024-03-24T02:28:32.309 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7441 sec 2024-03-24T02:28:33.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:32 smithi078 ceph-mon[32429]: pgmap v4063: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:33.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:32 smithi107 ceph-mon[36485]: pgmap v4063: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:33.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:32 smithi106 ceph-mon[36505]: pgmap v4063: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:33.309 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7442 sec 2024-03-24T02:28:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6788.202366 secs 2024-03-24T02:28:34.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6788.202366 secs 2024-03-24T02:28:34.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6788.202366 secs 2024-03-24T02:28:34.309 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7443 sec 2024-03-24T02:28:35.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:34 smithi078 ceph-mon[32429]: pgmap v4064: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:35.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:34 smithi107 ceph-mon[36485]: pgmap v4064: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:35.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:34 smithi106 ceph-mon[36505]: pgmap v4064: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:35.309 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7444 sec 2024-03-24T02:28:36.309 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7445 sec 2024-03-24T02:28:37.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:36 smithi078 ceph-mon[32429]: pgmap v4065: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:37.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:36 smithi107 ceph-mon[36485]: pgmap v4065: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:37.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:36 smithi106 ceph-mon[36505]: pgmap v4065: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:37.309 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7446 sec 2024-03-24T02:28:38.309 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7447 sec 2024-03-24T02:28:38.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6793.202453 secs 2024-03-24T02:28:39.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6793.202453 secs 2024-03-24T02:28:39.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6793.202453 secs 2024-03-24T02:28:39.309 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7448 sec 2024-03-24T02:28:39.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:39 smithi106 ceph-mon[36505]: pgmap v4066: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:39.994 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:39 smithi107 ceph-mon[36485]: pgmap v4066: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:39 smithi078 ceph-mon[32429]: pgmap v4066: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:40.310 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7449 sec 2024-03-24T02:28:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:40 smithi078 ceph-mon[32429]: pgmap v4067: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:41.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:40 smithi107 ceph-mon[36485]: pgmap v4067: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:41.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:40 smithi106 ceph-mon[36505]: pgmap v4067: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:41.310 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7450 sec 2024-03-24T02:28:42.310 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7451 sec 2024-03-24T02:28:43.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:42 smithi078 ceph-mon[32429]: pgmap v4068: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:43.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:42 smithi107 ceph-mon[36485]: pgmap v4068: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:43.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:42 smithi106 ceph-mon[36505]: pgmap v4068: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:43.310 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7452 sec 2024-03-24T02:28:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6798.202512 secs 2024-03-24T02:28:44.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6798.202512 secs 2024-03-24T02:28:44.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6798.202512 secs 2024-03-24T02:28:44.310 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7453 sec 2024-03-24T02:28:45.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:44 smithi078 ceph-mon[32429]: pgmap v4069: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:45.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:44 smithi107 ceph-mon[36485]: pgmap v4069: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:45.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:44 smithi106 ceph-mon[36505]: pgmap v4069: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:45.310 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7454 sec 2024-03-24T02:28:46.310 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7455 sec 2024-03-24T02:28:47.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:46 smithi078 ceph-mon[32429]: pgmap v4070: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:47.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:46 smithi107 ceph-mon[36485]: pgmap v4070: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:47.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:46 smithi106 ceph-mon[36505]: pgmap v4070: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:47.310 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7456 sec 2024-03-24T02:28:48.310 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7457 sec 2024-03-24T02:28:48.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6803.202594 secs 2024-03-24T02:28:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6803.202594 secs 2024-03-24T02:28:49.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6803.202594 secs 2024-03-24T02:28:49.310 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7458 sec 2024-03-24T02:28:49.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:49 smithi106 ceph-mon[36505]: pgmap v4071: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:49.991 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:49 smithi078 ceph-mon[32429]: pgmap v4071: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:50.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:49 smithi107 ceph-mon[36485]: pgmap v4071: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:50.310 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7459 sec 2024-03-24T02:28:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:50 smithi078 ceph-mon[32429]: pgmap v4072: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:51.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:50 smithi107 ceph-mon[36485]: pgmap v4072: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:51.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:50 smithi106 ceph-mon[36505]: pgmap v4072: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:51.310 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7460 sec 2024-03-24T02:28:52.311 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7461 sec 2024-03-24T02:28:53.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:52 smithi078 ceph-mon[32429]: pgmap v4073: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:53.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:52 smithi107 ceph-mon[36485]: pgmap v4073: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:53.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:52 smithi106 ceph-mon[36505]: pgmap v4073: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:53.311 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7462 sec 2024-03-24T02:28:54.053 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6808.202681 secs 2024-03-24T02:28:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6808.202681 secs 2024-03-24T02:28:54.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6808.202681 secs 2024-03-24T02:28:54.311 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7463 sec 2024-03-24T02:28:55.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:54 smithi078 ceph-mon[32429]: pgmap v4074: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:55.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:54 smithi107 ceph-mon[36485]: pgmap v4074: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:55.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:54 smithi106 ceph-mon[36505]: pgmap v4074: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:55.311 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7464 sec 2024-03-24T02:28:56.311 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7465 sec 2024-03-24T02:28:57.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:56 smithi078 ceph-mon[32429]: pgmap v4075: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:57.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:56 smithi107 ceph-mon[36485]: pgmap v4075: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:57.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:56 smithi106 ceph-mon[36505]: pgmap v4075: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:57.311 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7466 sec 2024-03-24T02:28:58.311 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7467 sec 2024-03-24T02:28:58.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6813.202740 secs 2024-03-24T02:28:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6813.202740 secs 2024-03-24T02:28:59.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6813.202740 secs 2024-03-24T02:28:59.311 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7468 sec 2024-03-24T02:28:59.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:59 smithi106 ceph-mon[36505]: pgmap v4076: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:28:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:59 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:28:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:59 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:28:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:59 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:28:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:28:59 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:29:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:59 smithi078 ceph-mon[32429]: pgmap v4076: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:59 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:29:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:59 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:29:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:59 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:29:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:28:59 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:29:00.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:59 smithi107 ceph-mon[36485]: pgmap v4076: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:00.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:59 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:29:00.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:59 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:29:00.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:59 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:29:00.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:28:59 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:29:00.311 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7469 sec 2024-03-24T02:29:00.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:00 smithi106 ceph-mon[36505]: pgmap v4077: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:29:00.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:00 smithi106 ceph-mon[36505]: pgmap v4078: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:29:01.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:00 smithi078 ceph-mon[32429]: pgmap v4077: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:29:01.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:00 smithi078 ceph-mon[32429]: pgmap v4078: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:29:01.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:00 smithi107 ceph-mon[36485]: pgmap v4077: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:29:01.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:00 smithi107 ceph-mon[36485]: pgmap v4078: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:29:01.311 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7470 sec 2024-03-24T02:29:02.312 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7471 sec 2024-03-24T02:29:02.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:02 smithi106 ceph-mon[36505]: pgmap v4079: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:29:03.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:02 smithi078 ceph-mon[32429]: pgmap v4079: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:29:03.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:02 smithi107 ceph-mon[36485]: pgmap v4079: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:29:03.312 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7472 sec 2024-03-24T02:29:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6818.202800 secs 2024-03-24T02:29:04.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6818.202800 secs 2024-03-24T02:29:04.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6818.202800 secs 2024-03-24T02:29:04.312 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7473 sec 2024-03-24T02:29:05.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:04 smithi078 ceph-mon[32429]: pgmap v4080: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:29:05.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:04 smithi107 ceph-mon[36485]: pgmap v4080: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:29:05.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:04 smithi106 ceph-mon[36505]: pgmap v4080: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:29:05.312 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7474 sec 2024-03-24T02:29:06.312 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7475 sec 2024-03-24T02:29:07.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:06 smithi078 ceph-mon[32429]: pgmap v4081: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:29:07.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:06 smithi107 ceph-mon[36485]: pgmap v4081: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:29:07.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:06 smithi106 ceph-mon[36505]: pgmap v4081: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:29:07.312 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7476 sec 2024-03-24T02:29:08.312 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7477 sec 2024-03-24T02:29:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:08 smithi078 ceph-mon[32429]: pgmap v4082: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:29:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6823.202859 secs 2024-03-24T02:29:09.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:08 smithi107 ceph-mon[36485]: pgmap v4082: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:29:09.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6823.202859 secs 2024-03-24T02:29:09.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:08 smithi106 ceph-mon[36505]: pgmap v4082: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:29:09.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6823.202859 secs 2024-03-24T02:29:09.312 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7478 sec 2024-03-24T02:29:10.312 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7479 sec 2024-03-24T02:29:11.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:10 smithi078 ceph-mon[32429]: pgmap v4083: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:29:11.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:10 smithi107 ceph-mon[36485]: pgmap v4083: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:29:11.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:10 smithi106 ceph-mon[36505]: pgmap v4083: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:29:11.312 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7480 sec 2024-03-24T02:29:12.312 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7481 sec 2024-03-24T02:29:13.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:12 smithi078 ceph-mon[32429]: pgmap v4084: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:13.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:12 smithi107 ceph-mon[36485]: pgmap v4084: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:13.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:12 smithi106 ceph-mon[36505]: pgmap v4084: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:13.313 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7482 sec 2024-03-24T02:29:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6828.202941 secs 2024-03-24T02:29:14.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6828.202941 secs 2024-03-24T02:29:14.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6828.202941 secs 2024-03-24T02:29:14.313 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7483 sec 2024-03-24T02:29:15.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:14 smithi078 ceph-mon[32429]: pgmap v4085: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:15.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:14 smithi107 ceph-mon[36485]: pgmap v4085: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:15.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:14 smithi106 ceph-mon[36505]: pgmap v4085: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:15.313 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7484 sec 2024-03-24T02:29:16.313 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7485 sec 2024-03-24T02:29:17.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:16 smithi078 ceph-mon[32429]: pgmap v4086: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:17.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:16 smithi107 ceph-mon[36485]: pgmap v4086: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:17.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:16 smithi106 ceph-mon[36505]: pgmap v4086: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:17.313 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7486 sec 2024-03-24T02:29:18.313 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7487 sec 2024-03-24T02:29:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:18 smithi078 ceph-mon[32429]: pgmap v4087: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6833.203002 secs 2024-03-24T02:29:19.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:18 smithi107 ceph-mon[36485]: pgmap v4087: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6833.203002 secs 2024-03-24T02:29:19.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:18 smithi106 ceph-mon[36505]: pgmap v4087: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:19.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6833.203002 secs 2024-03-24T02:29:19.313 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7488 sec 2024-03-24T02:29:20.313 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7489 sec 2024-03-24T02:29:21.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:20 smithi078 ceph-mon[32429]: pgmap v4088: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:21.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:20 smithi107 ceph-mon[36485]: pgmap v4088: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:21.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:20 smithi106 ceph-mon[36505]: pgmap v4088: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:21.313 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7490 sec 2024-03-24T02:29:22.313 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7491 sec 2024-03-24T02:29:23.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:22 smithi078 ceph-mon[32429]: pgmap v4089: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:23.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:22 smithi107 ceph-mon[36485]: pgmap v4089: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:23.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:22 smithi106 ceph-mon[36505]: pgmap v4089: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:23.313 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7492 sec 2024-03-24T02:29:24.052 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6838.203060 secs 2024-03-24T02:29:24.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6838.203060 secs 2024-03-24T02:29:24.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6838.203060 secs 2024-03-24T02:29:24.313 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7493 sec 2024-03-24T02:29:25.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:24 smithi078 ceph-mon[32429]: pgmap v4090: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:25.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:24 smithi107 ceph-mon[36485]: pgmap v4090: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:25.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:24 smithi106 ceph-mon[36505]: pgmap v4090: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:25.313 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7494 sec 2024-03-24T02:29:26.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:25 smithi078 ceph-mon[32429]: pgmap v4091: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:26.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:25 smithi107 ceph-mon[36485]: pgmap v4091: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:26.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:25 smithi106 ceph-mon[36505]: pgmap v4091: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:26.314 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7495 sec 2024-03-24T02:29:27.314 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7496 sec 2024-03-24T02:29:28.314 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7497 sec 2024-03-24T02:29:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:28 smithi078 ceph-mon[32429]: pgmap v4092: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:28.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:28 smithi107 ceph-mon[36485]: pgmap v4092: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:28.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:28 smithi106 ceph-mon[36505]: pgmap v4092: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:29.314 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7498 sec 2024-03-24T02:29:29.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6843.203120 secs 2024-03-24T02:29:29.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6843.203120 secs 2024-03-24T02:29:29.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6843.203120 secs 2024-03-24T02:29:30.314 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7499 sec 2024-03-24T02:29:30.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:30 smithi078 ceph-mon[32429]: pgmap v4093: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:30.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:30 smithi107 ceph-mon[36485]: pgmap v4093: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:30.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:30 smithi106 ceph-mon[36505]: pgmap v4093: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:31.314 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7500 sec 2024-03-24T02:29:32.314 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7501 sec 2024-03-24T02:29:32.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:32 smithi078 ceph-mon[32429]: pgmap v4094: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:32.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:32 smithi107 ceph-mon[36485]: pgmap v4094: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:32.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:32 smithi106 ceph-mon[36505]: pgmap v4094: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:33.314 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7502 sec 2024-03-24T02:29:34.314 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7503 sec 2024-03-24T02:29:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6848.203174 secs 2024-03-24T02:29:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:34 smithi078 ceph-mon[32429]: pgmap v4095: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:34.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6848.203174 secs 2024-03-24T02:29:34.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:34 smithi107 ceph-mon[36485]: pgmap v4095: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:34.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6848.203174 secs 2024-03-24T02:29:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:34 smithi106 ceph-mon[36505]: pgmap v4095: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:35.314 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7504 sec 2024-03-24T02:29:36.314 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7505 sec 2024-03-24T02:29:36.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:36 smithi078 ceph-mon[32429]: pgmap v4096: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:36.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:36 smithi107 ceph-mon[36485]: pgmap v4096: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:36.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:36 smithi106 ceph-mon[36505]: pgmap v4096: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:37.314 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7506 sec 2024-03-24T02:29:38.315 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7507 sec 2024-03-24T02:29:38.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:38 smithi078 ceph-mon[32429]: pgmap v4097: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:38.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:38 smithi107 ceph-mon[36485]: pgmap v4097: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:38.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:38 smithi106 ceph-mon[36505]: pgmap v4097: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:39.315 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7508 sec 2024-03-24T02:29:39.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6853.203184 secs 2024-03-24T02:29:39.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6853.203184 secs 2024-03-24T02:29:39.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6853.203184 secs 2024-03-24T02:29:40.315 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7509 sec 2024-03-24T02:29:40.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:40 smithi078 ceph-mon[32429]: pgmap v4098: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:40.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:40 smithi107 ceph-mon[36485]: pgmap v4098: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:40.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:40 smithi106 ceph-mon[36505]: pgmap v4098: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:41.315 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7510 sec 2024-03-24T02:29:42.315 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7511 sec 2024-03-24T02:29:42.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:42 smithi078 ceph-mon[32429]: pgmap v4099: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:42.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:42 smithi107 ceph-mon[36485]: pgmap v4099: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:42.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:42 smithi106 ceph-mon[36505]: pgmap v4099: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:43.315 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7512 sec 2024-03-24T02:29:44.315 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7513 sec 2024-03-24T02:29:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6858.203243 secs 2024-03-24T02:29:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:44 smithi078 ceph-mon[32429]: pgmap v4100: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:44.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6858.203243 secs 2024-03-24T02:29:44.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:44 smithi107 ceph-mon[36485]: pgmap v4100: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:44.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6858.203243 secs 2024-03-24T02:29:44.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:44 smithi106 ceph-mon[36505]: pgmap v4100: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:45.315 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7514 sec 2024-03-24T02:29:46.315 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7515 sec 2024-03-24T02:29:46.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:46 smithi078 ceph-mon[32429]: pgmap v4101: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:46.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:46 smithi107 ceph-mon[36485]: pgmap v4101: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:46.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:46 smithi106 ceph-mon[36505]: pgmap v4101: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:47.315 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7516 sec 2024-03-24T02:29:48.315 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7517 sec 2024-03-24T02:29:48.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:48 smithi078 ceph-mon[32429]: pgmap v4102: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:48.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:48 smithi107 ceph-mon[36485]: pgmap v4102: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:48.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:48 smithi106 ceph-mon[36505]: pgmap v4102: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:49.316 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7518 sec 2024-03-24T02:29:49.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6863.203334 secs 2024-03-24T02:29:49.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6863.203334 secs 2024-03-24T02:29:49.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6863.203334 secs 2024-03-24T02:29:50.316 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7519 sec 2024-03-24T02:29:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:50 smithi078 ceph-mon[32429]: pgmap v4103: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:50.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:50 smithi107 ceph-mon[36485]: pgmap v4103: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:50.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:50 smithi106 ceph-mon[36505]: pgmap v4103: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:51.316 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7520 sec 2024-03-24T02:29:52.316 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7521 sec 2024-03-24T02:29:52.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:52 smithi078 ceph-mon[32429]: pgmap v4104: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:52.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:52 smithi107 ceph-mon[36485]: pgmap v4104: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:52.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:52 smithi106 ceph-mon[36505]: pgmap v4104: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:53.316 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7522 sec 2024-03-24T02:29:54.316 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7523 sec 2024-03-24T02:29:54.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6868.203394 secs 2024-03-24T02:29:54.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:54 smithi078 ceph-mon[32429]: pgmap v4105: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:54.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6868.203394 secs 2024-03-24T02:29:54.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:54 smithi107 ceph-mon[36485]: pgmap v4105: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:54.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6868.203394 secs 2024-03-24T02:29:54.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:54 smithi106 ceph-mon[36505]: pgmap v4105: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:55.316 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7524 sec 2024-03-24T02:29:56.316 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7525 sec 2024-03-24T02:29:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:56 smithi078 ceph-mon[32429]: pgmap v4106: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:56.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:56 smithi107 ceph-mon[36485]: pgmap v4106: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:56.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:56 smithi106 ceph-mon[36505]: pgmap v4106: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:57.316 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7526 sec 2024-03-24T02:29:58.316 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7527 sec 2024-03-24T02:29:58.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:58 smithi078 ceph-mon[32429]: pgmap v4107: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:58.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:58 smithi107 ceph-mon[36485]: pgmap v4107: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:58.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:58 smithi106 ceph-mon[36505]: pgmap v4107: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:29:59.316 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7528 sec 2024-03-24T02:29:59.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6873.203457 secs 2024-03-24T02:29:59.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:29:59 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:29:59.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6873.203457 secs 2024-03-24T02:29:59.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:29:59 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:29:59.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6873.203457 secs 2024-03-24T02:29:59.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:29:59 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:30:00.317 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7529 sec 2024-03-24T02:30:00.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:00 smithi078 ceph-mon[32429]: pgmap v4108: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:00.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:00 smithi078 ceph-mon[32429]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:30:00.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:00 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:30:00.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:00 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:30:00.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:00 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:30:00.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:00 smithi107 ceph-mon[36485]: pgmap v4108: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:00 smithi107 ceph-mon[36485]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:30:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:00 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:30:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:00 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:30:00.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:00 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:30:00.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:00 smithi106 ceph-mon[36505]: pgmap v4108: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:00.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:00 smithi106 ceph-mon[36505]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:30:00.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:00 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:30:00.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:00 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:30:00.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:00 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:30:01.317 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7530 sec 2024-03-24T02:30:01.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:01 smithi078 ceph-mon[32429]: pgmap v4109: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:30:01.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:01 smithi078 ceph-mon[32429]: pgmap v4110: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:30:01.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:01 smithi107 ceph-mon[36485]: pgmap v4109: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:30:01.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:01 smithi107 ceph-mon[36485]: pgmap v4110: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:30:01.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:01 smithi106 ceph-mon[36505]: pgmap v4109: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:30:01.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:01 smithi106 ceph-mon[36505]: pgmap v4110: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:30:02.317 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7531 sec 2024-03-24T02:30:03.317 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7532 sec 2024-03-24T02:30:03.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:03 smithi078 ceph-mon[32429]: pgmap v4111: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:30:03.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:03 smithi107 ceph-mon[36485]: pgmap v4111: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:30:03.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:03 smithi106 ceph-mon[36505]: pgmap v4111: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:30:04.317 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7533 sec 2024-03-24T02:30:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6878.203517 secs 2024-03-24T02:30:04.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6878.203517 secs 2024-03-24T02:30:04.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6878.203517 secs 2024-03-24T02:30:05.317 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7534 sec 2024-03-24T02:30:05.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:05 smithi078 ceph-mon[32429]: pgmap v4112: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:30:05.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:05 smithi107 ceph-mon[36485]: pgmap v4112: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:30:05.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:05 smithi106 ceph-mon[36505]: pgmap v4112: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:30:06.317 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7535 sec 2024-03-24T02:30:07.317 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7536 sec 2024-03-24T02:30:07.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:07 smithi107 ceph-mon[36485]: pgmap v4113: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:30:07.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:07 smithi106 ceph-mon[36505]: pgmap v4113: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:30:08.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:07 smithi078 ceph-mon[32429]: pgmap v4113: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:30:08.317 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7537 sec 2024-03-24T02:30:08.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6883.203577 secs 2024-03-24T02:30:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6883.203577 secs 2024-03-24T02:30:09.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6883.203577 secs 2024-03-24T02:30:09.317 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7538 sec 2024-03-24T02:30:09.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:09 smithi106 ceph-mon[36505]: pgmap v4114: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:30:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:09 smithi078 ceph-mon[32429]: pgmap v4114: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:30:10.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:09 smithi107 ceph-mon[36485]: pgmap v4114: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:30:10.317 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7539 sec 2024-03-24T02:30:11.318 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7540 sec 2024-03-24T02:30:11.895 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:11 smithi107 ceph-mon[36485]: pgmap v4115: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:30:11.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:11 smithi106 ceph-mon[36505]: pgmap v4115: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:30:12.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:11 smithi078 ceph-mon[32429]: pgmap v4115: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:30:12.317 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7541 sec 2024-03-24T02:30:13.318 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7542 sec 2024-03-24T02:30:13.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:13 smithi106 ceph-mon[36505]: pgmap v4116: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6888.203661 secs 2024-03-24T02:30:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:13 smithi078 ceph-mon[32429]: pgmap v4116: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6888.203661 secs 2024-03-24T02:30:14.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:13 smithi107 ceph-mon[36485]: pgmap v4116: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:14.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6888.203661 secs 2024-03-24T02:30:14.318 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7543 sec 2024-03-24T02:30:15.318 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7544 sec 2024-03-24T02:30:15.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:15 smithi106 ceph-mon[36505]: pgmap v4117: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:16.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:15 smithi078 ceph-mon[32429]: pgmap v4117: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:16.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:15 smithi107 ceph-mon[36485]: pgmap v4117: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:16.318 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7545 sec 2024-03-24T02:30:17.318 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7546 sec 2024-03-24T02:30:17.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:17 smithi106 ceph-mon[36505]: pgmap v4118: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:18.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:17 smithi078 ceph-mon[32429]: pgmap v4118: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:18.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:17 smithi107 ceph-mon[36485]: pgmap v4118: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:18.318 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7547 sec 2024-03-24T02:30:18.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6893.203739 secs 2024-03-24T02:30:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6893.203739 secs 2024-03-24T02:30:19.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6893.203739 secs 2024-03-24T02:30:19.318 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7548 sec 2024-03-24T02:30:19.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:19 smithi106 ceph-mon[36505]: pgmap v4119: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:20.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:19 smithi078 ceph-mon[32429]: pgmap v4119: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:20.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:19 smithi107 ceph-mon[36485]: pgmap v4119: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:20.318 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7549 sec 2024-03-24T02:30:21.318 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7550 sec 2024-03-24T02:30:21.891 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:21 smithi078 ceph-mon[32429]: pgmap v4120: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:21.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:21 smithi106 ceph-mon[36505]: pgmap v4120: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:22.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:21 smithi107 ceph-mon[36485]: pgmap v4120: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:22.318 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7551 sec 2024-03-24T02:30:23.318 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7552 sec 2024-03-24T02:30:23.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:23 smithi106 ceph-mon[36505]: pgmap v4121: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:23.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6898.203798 secs 2024-03-24T02:30:24.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:23 smithi078 ceph-mon[32429]: pgmap v4121: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:24.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6898.203798 secs 2024-03-24T02:30:24.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:23 smithi107 ceph-mon[36485]: pgmap v4121: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:24.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6898.203798 secs 2024-03-24T02:30:24.319 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7553 sec 2024-03-24T02:30:25.319 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7554 sec 2024-03-24T02:30:25.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:25 smithi106 ceph-mon[36505]: pgmap v4122: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:26.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:25 smithi078 ceph-mon[32429]: pgmap v4122: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:26.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:25 smithi107 ceph-mon[36485]: pgmap v4122: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:26.319 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7555 sec 2024-03-24T02:30:27.319 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7556 sec 2024-03-24T02:30:27.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:27 smithi106 ceph-mon[36505]: pgmap v4123: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:28.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:27 smithi078 ceph-mon[32429]: pgmap v4123: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:28.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:27 smithi107 ceph-mon[36485]: pgmap v4123: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:28.319 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7557 sec 2024-03-24T02:30:28.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6903.203835 secs 2024-03-24T02:30:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6903.203835 secs 2024-03-24T02:30:29.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6903.203835 secs 2024-03-24T02:30:29.319 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7558 sec 2024-03-24T02:30:29.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:29 smithi106 ceph-mon[36505]: pgmap v4124: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:29 smithi078 ceph-mon[32429]: pgmap v4124: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:30.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:29 smithi107 ceph-mon[36485]: pgmap v4124: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:30.319 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7559 sec 2024-03-24T02:30:31.319 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7560 sec 2024-03-24T02:30:31.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:31 smithi106 ceph-mon[36505]: pgmap v4125: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:32.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:31 smithi078 ceph-mon[32429]: pgmap v4125: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:32.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:31 smithi107 ceph-mon[36485]: pgmap v4125: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:32.319 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7561 sec 2024-03-24T02:30:33.319 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7562 sec 2024-03-24T02:30:33.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:33 smithi106 ceph-mon[36505]: pgmap v4126: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:33.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6908.203894 secs 2024-03-24T02:30:34.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:33 smithi078 ceph-mon[32429]: pgmap v4126: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6908.203894 secs 2024-03-24T02:30:34.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:33 smithi107 ceph-mon[36485]: pgmap v4126: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:34.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6908.203894 secs 2024-03-24T02:30:34.319 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7563 sec 2024-03-24T02:30:35.320 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7564 sec 2024-03-24T02:30:35.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:35 smithi106 ceph-mon[36505]: pgmap v4127: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:36.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:35 smithi078 ceph-mon[32429]: pgmap v4127: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:36.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:35 smithi107 ceph-mon[36485]: pgmap v4127: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:36.320 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7565 sec 2024-03-24T02:30:37.320 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7566 sec 2024-03-24T02:30:37.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:37 smithi106 ceph-mon[36505]: pgmap v4128: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:37 smithi078 ceph-mon[32429]: pgmap v4128: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:38.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:37 smithi107 ceph-mon[36485]: pgmap v4128: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:38.320 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7567 sec 2024-03-24T02:30:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6913.203933 secs 2024-03-24T02:30:39.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6913.203933 secs 2024-03-24T02:30:39.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6913.203933 secs 2024-03-24T02:30:39.320 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7568 sec 2024-03-24T02:30:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:39 smithi078 ceph-mon[32429]: pgmap v4129: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:40.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:39 smithi107 ceph-mon[36485]: pgmap v4129: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:40.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:39 smithi106 ceph-mon[36505]: pgmap v4129: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:40.320 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7569 sec 2024-03-24T02:30:41.320 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7570 sec 2024-03-24T02:30:42.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:41 smithi078 ceph-mon[32429]: pgmap v4130: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:42.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:41 smithi107 ceph-mon[36485]: pgmap v4130: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:42.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:41 smithi106 ceph-mon[36505]: pgmap v4130: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:42.320 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7571 sec 2024-03-24T02:30:43.320 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7572 sec 2024-03-24T02:30:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:43 smithi078 ceph-mon[32429]: pgmap v4131: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6918.204021 secs 2024-03-24T02:30:44.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:43 smithi107 ceph-mon[36485]: pgmap v4131: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:44.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6918.204021 secs 2024-03-24T02:30:44.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:43 smithi106 ceph-mon[36505]: pgmap v4131: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:44.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6918.204021 secs 2024-03-24T02:30:44.320 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7573 sec 2024-03-24T02:30:45.321 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7574 sec 2024-03-24T02:30:46.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:45 smithi078 ceph-mon[32429]: pgmap v4132: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:46.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:45 smithi107 ceph-mon[36485]: pgmap v4132: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:46.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:45 smithi106 ceph-mon[36505]: pgmap v4132: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:46.321 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7575 sec 2024-03-24T02:30:47.320 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7576 sec 2024-03-24T02:30:48.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:47 smithi078 ceph-mon[32429]: pgmap v4133: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:48.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:47 smithi107 ceph-mon[36485]: pgmap v4133: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:48.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:47 smithi106 ceph-mon[36505]: pgmap v4133: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:48.320 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7577 sec 2024-03-24T02:30:49.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6923.204129 secs 2024-03-24T02:30:49.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6923.204129 secs 2024-03-24T02:30:49.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6923.204129 secs 2024-03-24T02:30:49.321 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7578 sec 2024-03-24T02:30:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:49 smithi078 ceph-mon[32429]: pgmap v4134: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:50.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:49 smithi107 ceph-mon[36485]: pgmap v4134: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:50.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:49 smithi106 ceph-mon[36505]: pgmap v4134: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:50.321 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7579 sec 2024-03-24T02:30:51.321 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7580 sec 2024-03-24T02:30:52.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:51 smithi078 ceph-mon[32429]: pgmap v4135: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:52.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:51 smithi107 ceph-mon[36485]: pgmap v4135: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:52.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:51 smithi106 ceph-mon[36505]: pgmap v4135: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:52.321 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7581 sec 2024-03-24T02:30:53.321 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7582 sec 2024-03-24T02:30:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:53 smithi078 ceph-mon[32429]: pgmap v4136: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6928.204183 secs 2024-03-24T02:30:54.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:53 smithi107 ceph-mon[36485]: pgmap v4136: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:54.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6928.204183 secs 2024-03-24T02:30:54.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:53 smithi106 ceph-mon[36505]: pgmap v4136: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6928.204183 secs 2024-03-24T02:30:54.321 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7583 sec 2024-03-24T02:30:55.321 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7584 sec 2024-03-24T02:30:55.953 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:55 smithi106 ceph-mon[36505]: pgmap v4137: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:56.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:55 smithi078 ceph-mon[32429]: pgmap v4137: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:56.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:55 smithi107 ceph-mon[36485]: pgmap v4137: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:30:56.321 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7585 sec 2024-03-24T02:30:57.321 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7586 sec 2024-03-24T02:30:58.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:57 smithi078 ceph-mon[32429]: pgmap v4138: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:58.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:57 smithi107 ceph-mon[36485]: pgmap v4138: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:58.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:57 smithi106 ceph-mon[36505]: pgmap v4138: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:58.321 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7587 sec 2024-03-24T02:30:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6933.204266 secs 2024-03-24T02:30:59.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6933.204266 secs 2024-03-24T02:30:59.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6933.204266 secs 2024-03-24T02:30:59.321 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7588 sec 2024-03-24T02:31:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:30:59 smithi078 ceph-mon[32429]: pgmap v4139: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:00.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:30:59 smithi107 ceph-mon[36485]: pgmap v4139: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:00.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:30:59 smithi106 ceph-mon[36505]: pgmap v4139: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:00.322 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7589 sec 2024-03-24T02:31:01.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:00 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:31:01.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:00 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:31:01.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:00 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:31:01.322 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7590 sec 2024-03-24T02:31:02.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:01 smithi078 ceph-mon[32429]: pgmap v4140: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:02.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:01 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:31:02.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:01 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:31:02.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:01 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:31:02.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:01 smithi107 ceph-mon[36485]: pgmap v4140: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:02.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:01 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:31:02.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:01 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:31:02.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:01 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:31:02.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:01 smithi106 ceph-mon[36505]: pgmap v4140: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:02.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:01 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:31:02.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:01 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:31:02.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:01 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:31:02.322 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7591 sec 2024-03-24T02:31:03.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:02 smithi078 ceph-mon[32429]: pgmap v4141: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:31:03.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:02 smithi078 ceph-mon[32429]: pgmap v4142: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:31:03.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:02 smithi107 ceph-mon[36485]: pgmap v4141: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:31:03.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:02 smithi107 ceph-mon[36485]: pgmap v4142: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:31:03.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:02 smithi106 ceph-mon[36505]: pgmap v4141: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:31:03.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:02 smithi106 ceph-mon[36505]: pgmap v4142: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:31:03.322 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7592 sec 2024-03-24T02:31:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6938.204325 secs 2024-03-24T02:31:04.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6938.204325 secs 2024-03-24T02:31:04.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6938.204325 secs 2024-03-24T02:31:04.322 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7593 sec 2024-03-24T02:31:05.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:04 smithi078 ceph-mon[32429]: pgmap v4143: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:31:05.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:04 smithi107 ceph-mon[36485]: pgmap v4143: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:31:05.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:04 smithi106 ceph-mon[36505]: pgmap v4143: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:31:05.322 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7594 sec 2024-03-24T02:31:06.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:05 smithi078 ceph-mon[32429]: pgmap v4144: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:31:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:05 smithi107 ceph-mon[36485]: pgmap v4144: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:31:06.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:05 smithi106 ceph-mon[36505]: pgmap v4144: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:31:06.322 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7595 sec 2024-03-24T02:31:07.322 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7596 sec 2024-03-24T02:31:08.322 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7597 sec 2024-03-24T02:31:08.339 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:08 smithi078 ceph-mon[32429]: pgmap v4145: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:31:08.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:08 smithi107 ceph-mon[36485]: pgmap v4145: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:31:08.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:08 smithi106 ceph-mon[36505]: pgmap v4145: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:31:09.322 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7598 sec 2024-03-24T02:31:09.339 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6943.204386 secs 2024-03-24T02:31:09.358 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6943.204386 secs 2024-03-24T02:31:09.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6943.204386 secs 2024-03-24T02:31:10.322 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7599 sec 2024-03-24T02:31:10.339 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:10 smithi078 ceph-mon[32429]: pgmap v4146: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:31:10.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:10 smithi107 ceph-mon[36485]: pgmap v4146: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:31:10.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:10 smithi106 ceph-mon[36505]: pgmap v4146: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:31:11.323 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7600 sec 2024-03-24T02:31:12.323 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7601 sec 2024-03-24T02:31:12.339 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:12 smithi078 ceph-mon[32429]: pgmap v4147: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:31:12.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:12 smithi107 ceph-mon[36485]: pgmap v4147: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:31:12.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:12 smithi106 ceph-mon[36505]: pgmap v4147: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:31:13.323 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7602 sec 2024-03-24T02:31:14.323 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7603 sec 2024-03-24T02:31:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6948.204401 secs 2024-03-24T02:31:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:14 smithi078 ceph-mon[32429]: pgmap v4148: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:14.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6948.204401 secs 2024-03-24T02:31:14.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:14 smithi107 ceph-mon[36485]: pgmap v4148: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:14.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6948.204401 secs 2024-03-24T02:31:14.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:14 smithi106 ceph-mon[36505]: pgmap v4148: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:15.323 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7604 sec 2024-03-24T02:31:16.323 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7605 sec 2024-03-24T02:31:16.339 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:16 smithi078 ceph-mon[32429]: pgmap v4149: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:16.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:16 smithi107 ceph-mon[36485]: pgmap v4149: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:16.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:16 smithi106 ceph-mon[36505]: pgmap v4149: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:17.323 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7606 sec 2024-03-24T02:31:18.324 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7607 sec 2024-03-24T02:31:18.339 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:18 smithi078 ceph-mon[32429]: pgmap v4150: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:18.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:18 smithi107 ceph-mon[36485]: pgmap v4150: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:18.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:18 smithi106 ceph-mon[36505]: pgmap v4150: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:19.323 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7608 sec 2024-03-24T02:31:19.339 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6953.204436 secs 2024-03-24T02:31:19.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6953.204436 secs 2024-03-24T02:31:19.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6953.204436 secs 2024-03-24T02:31:20.323 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7609 sec 2024-03-24T02:31:20.339 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:20 smithi078 ceph-mon[32429]: pgmap v4151: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:20.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:20 smithi107 ceph-mon[36485]: pgmap v4151: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:20.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:20 smithi106 ceph-mon[36505]: pgmap v4151: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:21.323 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7610 sec 2024-03-24T02:31:22.324 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7611 sec 2024-03-24T02:31:22.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:22 smithi078 ceph-mon[32429]: pgmap v4152: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:22.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:22 smithi107 ceph-mon[36485]: pgmap v4152: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:22.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:22 smithi106 ceph-mon[36505]: pgmap v4152: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:23.324 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7612 sec 2024-03-24T02:31:24.324 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7613 sec 2024-03-24T02:31:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6958.204497 secs 2024-03-24T02:31:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:24 smithi078 ceph-mon[32429]: pgmap v4153: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:24.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6958.204497 secs 2024-03-24T02:31:24.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:24 smithi107 ceph-mon[36485]: pgmap v4153: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:24.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6958.204497 secs 2024-03-24T02:31:24.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:24 smithi106 ceph-mon[36505]: pgmap v4153: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:25.324 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7614 sec 2024-03-24T02:31:26.324 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7615 sec 2024-03-24T02:31:26.339 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:26 smithi078 ceph-mon[32429]: pgmap v4154: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:26.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:26 smithi107 ceph-mon[36485]: pgmap v4154: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:26.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:26 smithi106 ceph-mon[36505]: pgmap v4154: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:27.324 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7616 sec 2024-03-24T02:31:28.324 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7617 sec 2024-03-24T02:31:28.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:28 smithi078 ceph-mon[32429]: pgmap v4155: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:28.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:28 smithi107 ceph-mon[36485]: pgmap v4155: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:28.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:28 smithi106 ceph-mon[36505]: pgmap v4155: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:29.324 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7618 sec 2024-03-24T02:31:29.339 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6963.204558 secs 2024-03-24T02:31:29.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6963.204558 secs 2024-03-24T02:31:29.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6963.204558 secs 2024-03-24T02:31:30.324 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7619 sec 2024-03-24T02:31:30.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:30 smithi107 ceph-mon[36485]: pgmap v4156: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:30.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:30 smithi106 ceph-mon[36505]: pgmap v4156: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:30.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:30 smithi078 ceph-mon[32429]: pgmap v4156: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:31.324 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7620 sec 2024-03-24T02:31:32.324 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7621 sec 2024-03-24T02:31:32.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:32 smithi106 ceph-mon[36505]: pgmap v4157: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:32.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:32 smithi078 ceph-mon[32429]: pgmap v4157: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:32.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:32 smithi107 ceph-mon[36485]: pgmap v4157: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:33.325 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7622 sec 2024-03-24T02:31:34.325 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7623 sec 2024-03-24T02:31:34.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6968.204588 secs 2024-03-24T02:31:34.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:34 smithi106 ceph-mon[36505]: pgmap v4158: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:34.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6968.204588 secs 2024-03-24T02:31:34.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:34 smithi078 ceph-mon[32429]: pgmap v4158: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:34.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6968.204588 secs 2024-03-24T02:31:34.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:34 smithi107 ceph-mon[36485]: pgmap v4158: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:35.325 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7624 sec 2024-03-24T02:31:36.325 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7625 sec 2024-03-24T02:31:36.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:36 smithi106 ceph-mon[36505]: pgmap v4159: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:36.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:36 smithi078 ceph-mon[32429]: pgmap v4159: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:36.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:36 smithi107 ceph-mon[36485]: pgmap v4159: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:37.325 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7626 sec 2024-03-24T02:31:38.325 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7627 sec 2024-03-24T02:31:38.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:38 smithi106 ceph-mon[36505]: pgmap v4160: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:38.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:38 smithi078 ceph-mon[32429]: pgmap v4160: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:38.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:38 smithi107 ceph-mon[36485]: pgmap v4160: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:39.325 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7628 sec 2024-03-24T02:31:39.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6973.204647 secs 2024-03-24T02:31:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6973.204647 secs 2024-03-24T02:31:39.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6973.204647 secs 2024-03-24T02:31:40.325 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7629 sec 2024-03-24T02:31:40.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:40 smithi106 ceph-mon[36505]: pgmap v4161: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:40.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:40 smithi078 ceph-mon[32429]: pgmap v4161: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:40.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:40 smithi107 ceph-mon[36485]: pgmap v4161: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:41.325 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7630 sec 2024-03-24T02:31:42.325 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7631 sec 2024-03-24T02:31:42.395 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:42 smithi107 ceph-mon[36485]: pgmap v4162: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:42.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:42 smithi106 ceph-mon[36505]: pgmap v4162: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:42.459 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:42 smithi078 ceph-mon[32429]: pgmap v4162: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:43.325 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7632 sec 2024-03-24T02:31:44.326 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7633 sec 2024-03-24T02:31:44.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6978.204706 secs 2024-03-24T02:31:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:44 smithi106 ceph-mon[36505]: pgmap v4163: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:44.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6978.204706 secs 2024-03-24T02:31:44.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:44 smithi078 ceph-mon[32429]: pgmap v4163: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:44.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6978.204706 secs 2024-03-24T02:31:44.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:44 smithi107 ceph-mon[36485]: pgmap v4163: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:45.326 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7634 sec 2024-03-24T02:31:46.326 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7635 sec 2024-03-24T02:31:46.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:46 smithi106 ceph-mon[36505]: pgmap v4164: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:46.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:46 smithi078 ceph-mon[32429]: pgmap v4164: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:46.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:46 smithi107 ceph-mon[36485]: pgmap v4164: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:47.326 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7636 sec 2024-03-24T02:31:48.326 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7637 sec 2024-03-24T02:31:48.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:48 smithi106 ceph-mon[36505]: pgmap v4165: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:48.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:48 smithi078 ceph-mon[32429]: pgmap v4165: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:48.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:48 smithi107 ceph-mon[36485]: pgmap v4165: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:49.326 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7638 sec 2024-03-24T02:31:49.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6983.204766 secs 2024-03-24T02:31:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6983.204766 secs 2024-03-24T02:31:49.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6983.204766 secs 2024-03-24T02:31:50.326 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7639 sec 2024-03-24T02:31:50.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:50 smithi106 ceph-mon[36505]: pgmap v4166: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:50.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:50 smithi078 ceph-mon[32429]: pgmap v4166: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:50.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:50 smithi107 ceph-mon[36485]: pgmap v4166: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:51.326 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7640 sec 2024-03-24T02:31:52.326 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7641 sec 2024-03-24T02:31:52.391 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:52 smithi078 ceph-mon[32429]: pgmap v4167: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:52.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:52 smithi106 ceph-mon[36505]: pgmap v4167: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:52.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:52 smithi107 ceph-mon[36485]: pgmap v4167: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:53.326 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7642 sec 2024-03-24T02:31:54.327 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7643 sec 2024-03-24T02:31:54.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6988.204850 secs 2024-03-24T02:31:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:54 smithi106 ceph-mon[36505]: pgmap v4168: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6988.204850 secs 2024-03-24T02:31:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:54 smithi078 ceph-mon[32429]: pgmap v4168: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:54.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6988.204850 secs 2024-03-24T02:31:54.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:54 smithi107 ceph-mon[36485]: pgmap v4168: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:55.326 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7644 sec 2024-03-24T02:31:56.327 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7645 sec 2024-03-24T02:31:56.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:56 smithi106 ceph-mon[36505]: pgmap v4169: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:56.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:56 smithi078 ceph-mon[32429]: pgmap v4169: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:56.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:56 smithi107 ceph-mon[36485]: pgmap v4169: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:31:57.327 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7646 sec 2024-03-24T02:31:58.327 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7647 sec 2024-03-24T02:31:58.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:58 smithi106 ceph-mon[36505]: pgmap v4170: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:58.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:58 smithi078 ceph-mon[32429]: pgmap v4170: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:58.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:58 smithi107 ceph-mon[36485]: pgmap v4170: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:59.327 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7648 sec 2024-03-24T02:31:59.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:31:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6993.204937 secs 2024-03-24T02:31:59.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:31:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6993.204937 secs 2024-03-24T02:31:59.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:31:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6993.204937 secs 2024-03-24T02:32:00.327 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7649 sec 2024-03-24T02:32:00.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:00 smithi078 ceph-mon[32429]: pgmap v4171: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:00.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:00 smithi107 ceph-mon[36485]: pgmap v4171: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:00.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:00 smithi106 ceph-mon[36505]: pgmap v4171: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:01.327 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7650 sec 2024-03-24T02:32:01.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:01 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:32:01.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:01 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:32:01.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:01 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:32:02.327 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7651 sec 2024-03-24T02:32:02.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:02 smithi078 ceph-mon[32429]: pgmap v4172: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:02.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:02 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:32:02.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:02 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:32:02.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:02 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:32:02.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:02 smithi107 ceph-mon[36485]: pgmap v4172: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:02.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:02 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:32:02.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:02 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:32:02.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:02 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:32:02.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:02 smithi106 ceph-mon[36505]: pgmap v4172: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:02.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:02 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:32:02.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:02 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:32:02.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:02 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:32:03.327 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7652 sec 2024-03-24T02:32:03.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:03 smithi078 ceph-mon[32429]: pgmap v4173: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:32:03.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:03 smithi078 ceph-mon[32429]: pgmap v4174: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:32:03.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:03 smithi107 ceph-mon[36485]: pgmap v4173: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:32:03.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:03 smithi107 ceph-mon[36485]: pgmap v4174: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:32:03.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:03 smithi106 ceph-mon[36505]: pgmap v4173: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:32:03.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:03 smithi106 ceph-mon[36505]: pgmap v4174: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:32:04.327 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7653 sec 2024-03-24T02:32:04.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 6998.205025 secs 2024-03-24T02:32:04.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 6998.205025 secs 2024-03-24T02:32:04.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 6998.205025 secs 2024-03-24T02:32:05.327 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7654 sec 2024-03-24T02:32:05.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:05 smithi078 ceph-mon[32429]: pgmap v4175: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:32:05.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:05 smithi107 ceph-mon[36485]: pgmap v4175: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:32:05.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:05 smithi106 ceph-mon[36505]: pgmap v4175: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:32:06.327 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7655 sec 2024-03-24T02:32:07.328 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7656 sec 2024-03-24T02:32:07.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:07 smithi078 ceph-mon[32429]: pgmap v4176: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:32:07.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:07 smithi107 ceph-mon[36485]: pgmap v4176: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:32:07.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:07 smithi106 ceph-mon[36505]: pgmap v4176: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:32:08.328 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7657 sec 2024-03-24T02:32:09.328 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7658 sec 2024-03-24T02:32:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:09 smithi078 ceph-mon[32429]: pgmap v4177: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:32:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7003.205110 secs 2024-03-24T02:32:09.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:09 smithi107 ceph-mon[36485]: pgmap v4177: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:32:09.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7003.205110 secs 2024-03-24T02:32:09.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:09 smithi106 ceph-mon[36505]: pgmap v4177: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:32:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7003.205110 secs 2024-03-24T02:32:10.328 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7659 sec 2024-03-24T02:32:11.328 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7660 sec 2024-03-24T02:32:11.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:11 smithi078 ceph-mon[32429]: pgmap v4178: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:32:11.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:11 smithi107 ceph-mon[36485]: pgmap v4178: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:32:11.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:11 smithi106 ceph-mon[36505]: pgmap v4178: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:32:12.328 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7661 sec 2024-03-24T02:32:13.328 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7662 sec 2024-03-24T02:32:13.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:13 smithi078 ceph-mon[32429]: pgmap v4179: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:32:13.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:13 smithi107 ceph-mon[36485]: pgmap v4179: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:32:13.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:13 smithi106 ceph-mon[36505]: pgmap v4179: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:32:14.328 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7663 sec 2024-03-24T02:32:14.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7008.205199 secs 2024-03-24T02:32:14.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7008.205199 secs 2024-03-24T02:32:14.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7008.205199 secs 2024-03-24T02:32:15.328 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7664 sec 2024-03-24T02:32:15.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:15 smithi078 ceph-mon[32429]: pgmap v4180: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:15.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:15 smithi107 ceph-mon[36485]: pgmap v4180: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:15.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:15 smithi106 ceph-mon[36505]: pgmap v4180: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:16.328 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7665 sec 2024-03-24T02:32:17.329 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7666 sec 2024-03-24T02:32:17.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:17 smithi078 ceph-mon[32429]: pgmap v4181: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:17.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:17 smithi107 ceph-mon[36485]: pgmap v4181: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:17.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:17 smithi106 ceph-mon[36505]: pgmap v4181: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:18.329 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7667 sec 2024-03-24T02:32:19.329 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7668 sec 2024-03-24T02:32:19.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:19 smithi078 ceph-mon[32429]: pgmap v4182: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:19.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7013.205294 secs 2024-03-24T02:32:19.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:19 smithi107 ceph-mon[36485]: pgmap v4182: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:19.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7013.205294 secs 2024-03-24T02:32:19.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:19 smithi106 ceph-mon[36505]: pgmap v4182: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:19.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7013.205294 secs 2024-03-24T02:32:20.329 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7669 sec 2024-03-24T02:32:21.329 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7670 sec 2024-03-24T02:32:21.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:21 smithi078 ceph-mon[32429]: pgmap v4183: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:21.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:21 smithi107 ceph-mon[36485]: pgmap v4183: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:21.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:21 smithi106 ceph-mon[36505]: pgmap v4183: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:22.329 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7671 sec 2024-03-24T02:32:23.329 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7672 sec 2024-03-24T02:32:23.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:23 smithi078 ceph-mon[32429]: pgmap v4184: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:23.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:23 smithi107 ceph-mon[36485]: pgmap v4184: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:23.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:23 smithi106 ceph-mon[36505]: pgmap v4184: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:24.329 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7673 sec 2024-03-24T02:32:24.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7018.205340 secs 2024-03-24T02:32:24.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7018.205340 secs 2024-03-24T02:32:24.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7018.205340 secs 2024-03-24T02:32:25.329 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7674 sec 2024-03-24T02:32:25.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:25 smithi078 ceph-mon[32429]: pgmap v4185: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:25.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:25 smithi107 ceph-mon[36485]: pgmap v4185: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:25.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:25 smithi106 ceph-mon[36505]: pgmap v4185: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:26.329 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7675 sec 2024-03-24T02:32:27.329 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7676 sec 2024-03-24T02:32:27.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:27 smithi078 ceph-mon[32429]: pgmap v4186: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:27.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:27 smithi107 ceph-mon[36485]: pgmap v4186: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:27.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:27 smithi106 ceph-mon[36505]: pgmap v4186: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:28.330 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7677 sec 2024-03-24T02:32:29.330 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7678 sec 2024-03-24T02:32:29.564 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:29 smithi078 ceph-mon[32429]: pgmap v4187: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:29.564 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7023.205401 secs 2024-03-24T02:32:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:29 smithi107 ceph-mon[36485]: pgmap v4187: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7023.205401 secs 2024-03-24T02:32:29.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:29 smithi106 ceph-mon[36505]: pgmap v4187: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:29.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7023.205401 secs 2024-03-24T02:32:30.330 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7679 sec 2024-03-24T02:32:31.330 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7680 sec 2024-03-24T02:32:31.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:31 smithi078 ceph-mon[32429]: pgmap v4188: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:31.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:31 smithi107 ceph-mon[36485]: pgmap v4188: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:31.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:31 smithi106 ceph-mon[36505]: pgmap v4188: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:32.330 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7681 sec 2024-03-24T02:32:33.330 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7682 sec 2024-03-24T02:32:33.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:33 smithi078 ceph-mon[32429]: pgmap v4189: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:33.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:33 smithi107 ceph-mon[36485]: pgmap v4189: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:33.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:33 smithi106 ceph-mon[36505]: pgmap v4189: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:34.330 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7683 sec 2024-03-24T02:32:34.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7028.205416 secs 2024-03-24T02:32:34.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7028.205416 secs 2024-03-24T02:32:34.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7028.205416 secs 2024-03-24T02:32:35.330 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7684 sec 2024-03-24T02:32:35.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:35 smithi078 ceph-mon[32429]: pgmap v4190: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:35.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:35 smithi107 ceph-mon[36485]: pgmap v4190: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:35.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:35 smithi106 ceph-mon[36505]: pgmap v4190: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:36.330 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7685 sec 2024-03-24T02:32:37.331 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7686 sec 2024-03-24T02:32:37.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:37 smithi078 ceph-mon[32429]: pgmap v4191: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:37.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:37 smithi107 ceph-mon[36485]: pgmap v4191: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:37.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:37 smithi106 ceph-mon[36505]: pgmap v4191: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:38.330 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7687 sec 2024-03-24T02:32:39.331 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7688 sec 2024-03-24T02:32:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:39 smithi078 ceph-mon[32429]: pgmap v4192: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7033.205504 secs 2024-03-24T02:32:39.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:39 smithi107 ceph-mon[36485]: pgmap v4192: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:39.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7033.205504 secs 2024-03-24T02:32:39.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:39 smithi106 ceph-mon[36505]: pgmap v4192: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:39.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7033.205504 secs 2024-03-24T02:32:40.331 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7689 sec 2024-03-24T02:32:41.331 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7690 sec 2024-03-24T02:32:41.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:41 smithi078 ceph-mon[32429]: pgmap v4193: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:41.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:41 smithi107 ceph-mon[36485]: pgmap v4193: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:41.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:41 smithi106 ceph-mon[36505]: pgmap v4193: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:42.331 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7691 sec 2024-03-24T02:32:43.331 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7692 sec 2024-03-24T02:32:43.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:43 smithi078 ceph-mon[32429]: pgmap v4194: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:43.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:43 smithi107 ceph-mon[36485]: pgmap v4194: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:43.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:43 smithi106 ceph-mon[36505]: pgmap v4194: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:44.331 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7693 sec 2024-03-24T02:32:44.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7038.205592 secs 2024-03-24T02:32:44.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7038.205592 secs 2024-03-24T02:32:44.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7038.205592 secs 2024-03-24T02:32:45.332 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7694 sec 2024-03-24T02:32:45.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:45 smithi078 ceph-mon[32429]: pgmap v4195: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:45.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:45 smithi107 ceph-mon[36485]: pgmap v4195: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:45.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:45 smithi106 ceph-mon[36505]: pgmap v4195: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:46.331 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7695 sec 2024-03-24T02:32:47.331 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7696 sec 2024-03-24T02:32:47.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:47 smithi078 ceph-mon[32429]: pgmap v4196: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:47.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:47 smithi107 ceph-mon[36485]: pgmap v4196: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:47.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:47 smithi106 ceph-mon[36505]: pgmap v4196: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:48.331 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7697 sec 2024-03-24T02:32:49.331 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7698 sec 2024-03-24T02:32:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:49 smithi078 ceph-mon[32429]: pgmap v4197: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7043.205633 secs 2024-03-24T02:32:49.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:49 smithi107 ceph-mon[36485]: pgmap v4197: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:49.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7043.205633 secs 2024-03-24T02:32:49.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:49 smithi106 ceph-mon[36505]: pgmap v4197: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:49.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7043.205633 secs 2024-03-24T02:32:50.332 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7699 sec 2024-03-24T02:32:51.332 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7700 sec 2024-03-24T02:32:51.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:51 smithi078 ceph-mon[32429]: pgmap v4198: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:51.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:51 smithi107 ceph-mon[36485]: pgmap v4198: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:51.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:51 smithi106 ceph-mon[36505]: pgmap v4198: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:52.332 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7701 sec 2024-03-24T02:32:53.332 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7702 sec 2024-03-24T02:32:53.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:53 smithi078 ceph-mon[32429]: pgmap v4199: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:53.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:53 smithi107 ceph-mon[36485]: pgmap v4199: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:53.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:53 smithi106 ceph-mon[36505]: pgmap v4199: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:54.332 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7703 sec 2024-03-24T02:32:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7048.205674 secs 2024-03-24T02:32:54.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7048.205674 secs 2024-03-24T02:32:54.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7048.205674 secs 2024-03-24T02:32:55.332 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7704 sec 2024-03-24T02:32:55.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:55 smithi078 ceph-mon[32429]: pgmap v4200: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:55.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:55 smithi107 ceph-mon[36485]: pgmap v4200: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:55.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:55 smithi106 ceph-mon[36505]: pgmap v4200: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:56.332 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7705 sec 2024-03-24T02:32:57.332 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7706 sec 2024-03-24T02:32:57.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:57 smithi078 ceph-mon[32429]: pgmap v4201: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:57.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:57 smithi107 ceph-mon[36485]: pgmap v4201: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:57.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:57 smithi106 ceph-mon[36505]: pgmap v4201: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:58.332 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7707 sec 2024-03-24T02:32:59.332 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7708 sec 2024-03-24T02:32:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:59 smithi078 ceph-mon[32429]: pgmap v4202: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:32:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7053.205759 secs 2024-03-24T02:32:59.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:59 smithi107 ceph-mon[36485]: pgmap v4202: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:59.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:32:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7053.205759 secs 2024-03-24T02:32:59.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:59 smithi106 ceph-mon[36505]: pgmap v4202: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:32:59.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:32:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7053.205759 secs 2024-03-24T02:33:00.332 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7709 sec 2024-03-24T02:33:01.332 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7710 sec 2024-03-24T02:33:01.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:01 smithi078 ceph-mon[32429]: pgmap v4203: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:33:01.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:01 smithi107 ceph-mon[36485]: pgmap v4203: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:33:01.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:01 smithi106 ceph-mon[36505]: pgmap v4203: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:33:02.332 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7711 sec 2024-03-24T02:33:02.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:02 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:33:02.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:02 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:33:02.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:02 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:33:03.333 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7712 sec 2024-03-24T02:33:03.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:03 smithi078 ceph-mon[32429]: pgmap v4204: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:03.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:03 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:33:03.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:03 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:33:03.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:03 smithi078 ceph-mon[32429]: pgmap v4205: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:33:03.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:03 smithi078 ceph-mon[32429]: pgmap v4206: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:33:03.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:03 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:33:03.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:03 smithi107 ceph-mon[36485]: pgmap v4204: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:03.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:03 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:33:03.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:03 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:33:03.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:03 smithi107 ceph-mon[36485]: pgmap v4205: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:33:03.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:03 smithi107 ceph-mon[36485]: pgmap v4206: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:33:03.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:03 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:33:03.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:03 smithi106 ceph-mon[36505]: pgmap v4204: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:03.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:03 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:33:03.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:03 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:33:03.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:03 smithi106 ceph-mon[36505]: pgmap v4205: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:33:03.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:03 smithi106 ceph-mon[36505]: pgmap v4206: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:33:03.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:03 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:33:04.333 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7713 sec 2024-03-24T02:33:04.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7058.205819 secs 2024-03-24T02:33:04.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7058.205819 secs 2024-03-24T02:33:04.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7058.205819 secs 2024-03-24T02:33:05.333 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7714 sec 2024-03-24T02:33:05.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:05 smithi078 ceph-mon[32429]: pgmap v4207: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:33:05.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:05 smithi107 ceph-mon[36485]: pgmap v4207: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:33:05.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:05 smithi106 ceph-mon[36505]: pgmap v4207: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:33:06.333 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7715 sec 2024-03-24T02:33:07.333 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7716 sec 2024-03-24T02:33:07.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:07 smithi106 ceph-mon[36505]: pgmap v4208: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:33:08.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:07 smithi078 ceph-mon[32429]: pgmap v4208: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:33:08.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:07 smithi107 ceph-mon[36485]: pgmap v4208: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:33:08.333 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7717 sec 2024-03-24T02:33:09.333 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7718 sec 2024-03-24T02:33:09.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7063.205902 secs 2024-03-24T02:33:09.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:09 smithi106 ceph-mon[36505]: pgmap v4209: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:33:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7063.205902 secs 2024-03-24T02:33:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:09 smithi078 ceph-mon[32429]: pgmap v4209: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:33:10.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7063.205902 secs 2024-03-24T02:33:10.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:09 smithi107 ceph-mon[36485]: pgmap v4209: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:33:10.333 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7719 sec 2024-03-24T02:33:11.334 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7720 sec 2024-03-24T02:33:11.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:11 smithi106 ceph-mon[36505]: pgmap v4210: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:33:12.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:11 smithi078 ceph-mon[32429]: pgmap v4210: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:33:12.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:11 smithi107 ceph-mon[36485]: pgmap v4210: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:33:12.333 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7721 sec 2024-03-24T02:33:13.334 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7722 sec 2024-03-24T02:33:13.895 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:13 smithi107 ceph-mon[36485]: pgmap v4211: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:33:13.895 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7068.205987 secs 2024-03-24T02:33:13.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:13 smithi106 ceph-mon[36505]: pgmap v4211: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:33:13.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7068.205987 secs 2024-03-24T02:33:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:13 smithi078 ceph-mon[32429]: pgmap v4211: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:33:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7068.205987 secs 2024-03-24T02:33:14.334 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7723 sec 2024-03-24T02:33:15.334 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7724 sec 2024-03-24T02:33:15.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:15 smithi106 ceph-mon[36505]: pgmap v4212: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:33:16.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:15 smithi078 ceph-mon[32429]: pgmap v4212: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:33:16.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:15 smithi107 ceph-mon[36485]: pgmap v4212: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:33:16.334 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7725 sec 2024-03-24T02:33:17.334 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7726 sec 2024-03-24T02:33:17.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:17 smithi106 ceph-mon[36505]: pgmap v4213: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:18.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:17 smithi078 ceph-mon[32429]: pgmap v4213: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:18.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:17 smithi107 ceph-mon[36485]: pgmap v4213: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:18.334 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7727 sec 2024-03-24T02:33:18.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7073.206076 secs 2024-03-24T02:33:19.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7073.206076 secs 2024-03-24T02:33:19.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7073.206076 secs 2024-03-24T02:33:19.334 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7728 sec 2024-03-24T02:33:19.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:19 smithi106 ceph-mon[36505]: pgmap v4214: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:33:20.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:19 smithi078 ceph-mon[32429]: pgmap v4214: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:33:20.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:19 smithi107 ceph-mon[36485]: pgmap v4214: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:33:20.334 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7729 sec 2024-03-24T02:33:21.334 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7730 sec 2024-03-24T02:33:21.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:21 smithi106 ceph-mon[36505]: pgmap v4215: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:22.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:21 smithi078 ceph-mon[32429]: pgmap v4215: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:22.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:21 smithi107 ceph-mon[36485]: pgmap v4215: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:22.334 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7731 sec 2024-03-24T02:33:23.334 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7732 sec 2024-03-24T02:33:23.891 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:23 smithi078 ceph-mon[32429]: pgmap v4216: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:33:23.891 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7078.206159 secs 2024-03-24T02:33:23.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:23 smithi106 ceph-mon[36505]: pgmap v4216: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:33:23.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7078.206159 secs 2024-03-24T02:33:24.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:23 smithi107 ceph-mon[36485]: pgmap v4216: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:33:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7078.206159 secs 2024-03-24T02:33:24.334 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7733 sec 2024-03-24T02:33:25.334 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7734 sec 2024-03-24T02:33:25.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:25 smithi106 ceph-mon[36505]: pgmap v4217: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:33:26.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:25 smithi078 ceph-mon[32429]: pgmap v4217: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:33:26.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:25 smithi107 ceph-mon[36485]: pgmap v4217: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:33:26.334 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7735 sec 2024-03-24T02:33:27.335 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7736 sec 2024-03-24T02:33:27.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:27 smithi106 ceph-mon[36505]: pgmap v4218: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:28.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:27 smithi078 ceph-mon[32429]: pgmap v4218: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:28.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:27 smithi107 ceph-mon[36485]: pgmap v4218: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:28.335 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7737 sec 2024-03-24T02:33:28.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7083.206219 secs 2024-03-24T02:33:29.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7083.206219 secs 2024-03-24T02:33:29.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7083.206219 secs 2024-03-24T02:33:29.335 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7738 sec 2024-03-24T02:33:29.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:29 smithi106 ceph-mon[36505]: pgmap v4219: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:33:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:29 smithi078 ceph-mon[32429]: pgmap v4219: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:33:30.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:29 smithi107 ceph-mon[36485]: pgmap v4219: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:33:30.335 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7739 sec 2024-03-24T02:33:31.335 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7740 sec 2024-03-24T02:33:31.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:31 smithi078 ceph-mon[32429]: pgmap v4220: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:31.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:31 smithi106 ceph-mon[36505]: pgmap v4220: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:32.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:31 smithi107 ceph-mon[36485]: pgmap v4220: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:32.335 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7741 sec 2024-03-24T02:33:33.335 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7742 sec 2024-03-24T02:33:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:33 smithi078 ceph-mon[32429]: pgmap v4221: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:33:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7088.206278 secs 2024-03-24T02:33:34.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:33 smithi107 ceph-mon[36485]: pgmap v4221: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:33:34.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7088.206278 secs 2024-03-24T02:33:34.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:33 smithi106 ceph-mon[36505]: pgmap v4221: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:33:34.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7088.206278 secs 2024-03-24T02:33:34.335 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7743 sec 2024-03-24T02:33:35.335 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7744 sec 2024-03-24T02:33:36.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:35 smithi078 ceph-mon[32429]: pgmap v4222: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:33:36.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:35 smithi107 ceph-mon[36485]: pgmap v4222: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:33:36.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:35 smithi106 ceph-mon[36505]: pgmap v4222: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:33:36.335 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7745 sec 2024-03-24T02:33:37.336 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7746 sec 2024-03-24T02:33:38.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:37 smithi078 ceph-mon[32429]: pgmap v4223: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:33:38.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:37 smithi107 ceph-mon[36485]: pgmap v4223: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:33:38.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:37 smithi106 ceph-mon[36505]: pgmap v4223: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:33:38.336 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7747 sec 2024-03-24T02:33:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7093.206314 secs 2024-03-24T02:33:39.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7093.206314 secs 2024-03-24T02:33:39.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7093.206314 secs 2024-03-24T02:33:39.336 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7748 sec 2024-03-24T02:33:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:39 smithi078 ceph-mon[32429]: pgmap v4224: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:33:40.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:39 smithi107 ceph-mon[36485]: pgmap v4224: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:33:40.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:39 smithi106 ceph-mon[36505]: pgmap v4224: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:33:40.336 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7749 sec 2024-03-24T02:33:41.336 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7750 sec 2024-03-24T02:33:42.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:41 smithi078 ceph-mon[32429]: pgmap v4225: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:33:42.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:41 smithi107 ceph-mon[36485]: pgmap v4225: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:33:42.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:41 smithi106 ceph-mon[36505]: pgmap v4225: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:33:42.336 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7751 sec 2024-03-24T02:33:43.336 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7752 sec 2024-03-24T02:33:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:43 smithi078 ceph-mon[32429]: pgmap v4226: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:33:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7098.206397 secs 2024-03-24T02:33:44.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:43 smithi107 ceph-mon[36485]: pgmap v4226: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:33:44.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7098.206397 secs 2024-03-24T02:33:44.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:43 smithi106 ceph-mon[36505]: pgmap v4226: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:33:44.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7098.206397 secs 2024-03-24T02:33:44.336 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7753 sec 2024-03-24T02:33:45.336 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7754 sec 2024-03-24T02:33:46.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:45 smithi078 ceph-mon[32429]: pgmap v4227: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:33:46.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:45 smithi107 ceph-mon[36485]: pgmap v4227: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:33:46.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:45 smithi106 ceph-mon[36505]: pgmap v4227: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:33:46.336 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7755 sec 2024-03-24T02:33:47.337 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7756 sec 2024-03-24T02:33:48.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:47 smithi078 ceph-mon[32429]: pgmap v4228: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:33:48.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:47 smithi107 ceph-mon[36485]: pgmap v4228: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:33:48.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:47 smithi106 ceph-mon[36505]: pgmap v4228: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:33:48.337 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7757 sec 2024-03-24T02:33:49.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7103.206458 secs 2024-03-24T02:33:49.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7103.206458 secs 2024-03-24T02:33:49.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7103.206458 secs 2024-03-24T02:33:49.337 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7758 sec 2024-03-24T02:33:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:49 smithi078 ceph-mon[32429]: pgmap v4229: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:33:50.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:49 smithi107 ceph-mon[36485]: pgmap v4229: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:33:50.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:49 smithi106 ceph-mon[36505]: pgmap v4229: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:33:50.337 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7759 sec 2024-03-24T02:33:51.337 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7760 sec 2024-03-24T02:33:52.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:51 smithi078 ceph-mon[32429]: pgmap v4230: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:52.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:51 smithi107 ceph-mon[36485]: pgmap v4230: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:52.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:51 smithi106 ceph-mon[36505]: pgmap v4230: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:52.337 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7761 sec 2024-03-24T02:33:53.337 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7762 sec 2024-03-24T02:33:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:53 smithi078 ceph-mon[32429]: pgmap v4231: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:33:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7108.206516 secs 2024-03-24T02:33:54.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:53 smithi107 ceph-mon[36485]: pgmap v4231: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:33:54.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7108.206516 secs 2024-03-24T02:33:54.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:53 smithi106 ceph-mon[36505]: pgmap v4231: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:33:54.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7108.206516 secs 2024-03-24T02:33:54.337 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7763 sec 2024-03-24T02:33:55.337 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7764 sec 2024-03-24T02:33:56.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:55 smithi078 ceph-mon[32429]: pgmap v4232: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:33:56.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:55 smithi107 ceph-mon[36485]: pgmap v4232: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:33:56.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:55 smithi106 ceph-mon[36505]: pgmap v4232: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:33:56.337 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7765 sec 2024-03-24T02:33:57.337 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7766 sec 2024-03-24T02:33:57.953 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:57 smithi106 ceph-mon[36505]: pgmap v4233: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:58.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:57 smithi078 ceph-mon[32429]: pgmap v4233: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:58.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:57 smithi107 ceph-mon[36485]: pgmap v4233: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:58.337 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7767 sec 2024-03-24T02:33:59.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7113.206598 secs 2024-03-24T02:33:59.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7113.206598 secs 2024-03-24T02:33:59.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7113.206598 secs 2024-03-24T02:33:59.337 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7768 sec 2024-03-24T02:34:00.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:33:59 smithi078 ceph-mon[32429]: pgmap v4234: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:00.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:33:59 smithi107 ceph-mon[36485]: pgmap v4234: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:00.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:33:59 smithi106 ceph-mon[36505]: pgmap v4234: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:00.338 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7769 sec 2024-03-24T02:34:01.338 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7770 sec 2024-03-24T02:34:02.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:01 smithi078 ceph-mon[32429]: pgmap v4235: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:02.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:01 smithi107 ceph-mon[36485]: pgmap v4235: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:02.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:01 smithi106 ceph-mon[36505]: pgmap v4235: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:02.338 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7771 sec 2024-03-24T02:34:03.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:02 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:34:03.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:02 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:34:03.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:02 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:34:03.338 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7772 sec 2024-03-24T02:34:03.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:03 smithi078 ceph-mon[32429]: pgmap v4236: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:03.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7118.206663 secs 2024-03-24T02:34:03.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:03 smithi107 ceph-mon[36485]: pgmap v4236: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:03.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7118.206663 secs 2024-03-24T02:34:04.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:03 smithi106 ceph-mon[36505]: pgmap v4236: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7118.206663 secs 2024-03-24T02:34:04.338 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7773 sec 2024-03-24T02:34:05.338 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7774 sec 2024-03-24T02:34:05.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:05 smithi106 ceph-mon[36505]: pgmap v4237: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:05.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:34:05.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:34:05.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:34:05.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:34:05.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:34:05.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:34:05.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:05.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:05.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:05.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:05.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:05.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:05.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:05.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:06.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:05 smithi078 ceph-mon[32429]: pgmap v4237: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:06.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:34:06.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:34:06.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:34:06.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:34:06.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:34:06.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:34:06.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:06.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:06.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:06.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:06.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:06.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:06.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:06.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:06.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:05 smithi107 ceph-mon[36485]: pgmap v4237: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:34:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:34:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:34:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:34:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:34:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:34:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:06.338 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7775 sec 2024-03-24T02:34:06.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:06 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T02:34:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:06 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T02:34:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:06 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T02:34:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:06 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T02:34:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:06 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:06 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:06 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:06 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:06 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T02:34:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:06 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T02:34:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:06 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:34:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:06 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:34:06.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:06 smithi106 ceph-mon[36505]: pgmap v4238: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T02:34:06.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:06 smithi106 ceph-mon[36505]: pgmap v4239: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:34:06.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:06 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:34:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:06 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T02:34:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:06 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T02:34:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:06 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T02:34:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:06 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T02:34:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:06 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:06 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:06 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:06 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:07.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:06 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T02:34:07.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:06 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T02:34:07.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:06 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:34:07.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:06 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:34:07.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:06 smithi078 ceph-mon[32429]: pgmap v4238: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T02:34:07.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:06 smithi078 ceph-mon[32429]: pgmap v4239: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:34:07.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:06 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:34:07.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:06 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T02:34:07.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:06 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T02:34:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:06 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T02:34:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:06 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T02:34:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:06 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:06 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:06 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:06 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:34:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:06 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T02:34:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:06 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T02:34:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:06 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:34:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:06 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:34:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:06 smithi107 ceph-mon[36485]: pgmap v4238: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T02:34:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:06 smithi107 ceph-mon[36485]: pgmap v4239: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:34:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:06 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:34:07.338 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7776 sec 2024-03-24T02:34:08.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:07 smithi078 ceph-mon[32429]: pgmap v4240: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:34:08.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:07 smithi107 ceph-mon[36485]: pgmap v4240: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:34:08.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:07 smithi106 ceph-mon[36505]: pgmap v4240: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:34:08.338 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7777 sec 2024-03-24T02:34:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7123.206727 secs 2024-03-24T02:34:09.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7123.206727 secs 2024-03-24T02:34:09.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7123.206727 secs 2024-03-24T02:34:09.338 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7778 sec 2024-03-24T02:34:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:09 smithi078 ceph-mon[32429]: pgmap v4241: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:34:10.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:09 smithi107 ceph-mon[36485]: pgmap v4241: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:34:10.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:09 smithi106 ceph-mon[36505]: pgmap v4241: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:34:10.338 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7779 sec 2024-03-24T02:34:11.338 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7780 sec 2024-03-24T02:34:12.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:11 smithi078 ceph-mon[32429]: pgmap v4242: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:34:12.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:11 smithi107 ceph-mon[36485]: pgmap v4242: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:34:12.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:11 smithi106 ceph-mon[36505]: pgmap v4242: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:34:12.338 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7781 sec 2024-03-24T02:34:13.339 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7782 sec 2024-03-24T02:34:14.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7128.206784 secs 2024-03-24T02:34:14.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7128.206784 secs 2024-03-24T02:34:14.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7128.206784 secs 2024-03-24T02:34:14.339 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7783 sec 2024-03-24T02:34:15.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:14 smithi078 ceph-mon[32429]: pgmap v4243: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:34:15.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:14 smithi107 ceph-mon[36485]: pgmap v4243: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:34:15.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:14 smithi106 ceph-mon[36505]: pgmap v4243: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:34:15.339 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7784 sec 2024-03-24T02:34:16.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:15 smithi078 ceph-mon[32429]: pgmap v4244: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:34:16.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:15 smithi107 ceph-mon[36485]: pgmap v4244: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:34:16.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:15 smithi106 ceph-mon[36505]: pgmap v4244: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:34:16.339 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7785 sec 2024-03-24T02:34:17.339 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7786 sec 2024-03-24T02:34:18.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:17 smithi078 ceph-mon[32429]: pgmap v4245: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:18.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:17 smithi107 ceph-mon[36485]: pgmap v4245: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:18.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:17 smithi106 ceph-mon[36505]: pgmap v4245: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:18.339 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7787 sec 2024-03-24T02:34:18.991 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7133.206874 secs 2024-03-24T02:34:19.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7133.206874 secs 2024-03-24T02:34:19.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7133.206874 secs 2024-03-24T02:34:19.339 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7788 sec 2024-03-24T02:34:20.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:19 smithi078 ceph-mon[32429]: pgmap v4246: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:20.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:19 smithi107 ceph-mon[36485]: pgmap v4246: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:20.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:19 smithi106 ceph-mon[36505]: pgmap v4246: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:20.339 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7789 sec 2024-03-24T02:34:21.339 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7790 sec 2024-03-24T02:34:22.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:21 smithi078 ceph-mon[32429]: pgmap v4247: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:22.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:21 smithi107 ceph-mon[36485]: pgmap v4247: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:22.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:21 smithi106 ceph-mon[36505]: pgmap v4247: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:22.339 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7791 sec 2024-03-24T02:34:23.339 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7792 sec 2024-03-24T02:34:24.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7138.206959 secs 2024-03-24T02:34:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7138.206959 secs 2024-03-24T02:34:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7138.206959 secs 2024-03-24T02:34:24.339 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7793 sec 2024-03-24T02:34:25.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:24 smithi078 ceph-mon[32429]: pgmap v4248: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:25.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:24 smithi107 ceph-mon[36485]: pgmap v4248: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:25.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:24 smithi106 ceph-mon[36505]: pgmap v4248: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:25.339 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7794 sec 2024-03-24T02:34:26.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:25 smithi078 ceph-mon[32429]: pgmap v4249: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:26.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:25 smithi107 ceph-mon[36485]: pgmap v4249: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:26.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:25 smithi106 ceph-mon[36505]: pgmap v4249: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:26.339 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7795 sec 2024-03-24T02:34:27.340 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7796 sec 2024-03-24T02:34:28.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:27 smithi078 ceph-mon[32429]: pgmap v4250: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:28.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:27 smithi107 ceph-mon[36485]: pgmap v4250: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:28.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:27 smithi106 ceph-mon[36505]: pgmap v4250: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:28.340 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7797 sec 2024-03-24T02:34:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7143.207048 secs 2024-03-24T02:34:29.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7143.207048 secs 2024-03-24T02:34:29.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7143.207048 secs 2024-03-24T02:34:29.340 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7798 sec 2024-03-24T02:34:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:29 smithi078 ceph-mon[32429]: pgmap v4251: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:30.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:29 smithi107 ceph-mon[36485]: pgmap v4251: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:30.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:29 smithi106 ceph-mon[36505]: pgmap v4251: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:30.340 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7799 sec 2024-03-24T02:34:31.340 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7800 sec 2024-03-24T02:34:32.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:31 smithi078 ceph-mon[32429]: pgmap v4252: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:32.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:31 smithi107 ceph-mon[36485]: pgmap v4252: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:32.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:31 smithi106 ceph-mon[36505]: pgmap v4252: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:32.340 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7801 sec 2024-03-24T02:34:33.340 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7802 sec 2024-03-24T02:34:33.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7148.207135 secs 2024-03-24T02:34:34.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7148.207135 secs 2024-03-24T02:34:34.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7148.207135 secs 2024-03-24T02:34:34.340 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7803 sec 2024-03-24T02:34:35.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:34 smithi078 ceph-mon[32429]: pgmap v4253: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:35.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:34 smithi107 ceph-mon[36485]: pgmap v4253: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:35.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:34 smithi106 ceph-mon[36505]: pgmap v4253: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:35.340 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7804 sec 2024-03-24T02:34:36.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:35 smithi078 ceph-mon[32429]: pgmap v4254: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:36.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:35 smithi107 ceph-mon[36485]: pgmap v4254: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:36.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:35 smithi106 ceph-mon[36505]: pgmap v4254: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:36.340 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7805 sec 2024-03-24T02:34:37.340 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7806 sec 2024-03-24T02:34:38.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:37 smithi078 ceph-mon[32429]: pgmap v4255: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:38.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:37 smithi107 ceph-mon[36485]: pgmap v4255: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:38.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:37 smithi106 ceph-mon[36505]: pgmap v4255: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:38.341 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7807 sec 2024-03-24T02:34:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7153.207177 secs 2024-03-24T02:34:39.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7153.207177 secs 2024-03-24T02:34:39.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7153.207177 secs 2024-03-24T02:34:39.341 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7808 sec 2024-03-24T02:34:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:39 smithi078 ceph-mon[32429]: pgmap v4256: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:40.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:39 smithi107 ceph-mon[36485]: pgmap v4256: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:40.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:39 smithi106 ceph-mon[36505]: pgmap v4256: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:40.341 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7809 sec 2024-03-24T02:34:41.341 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7810 sec 2024-03-24T02:34:42.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:41 smithi078 ceph-mon[32429]: pgmap v4257: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:42.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:41 smithi107 ceph-mon[36485]: pgmap v4257: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:42.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:41 smithi106 ceph-mon[36505]: pgmap v4257: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:42.341 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7811 sec 2024-03-24T02:34:43.341 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7812 sec 2024-03-24T02:34:44.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7158.207263 secs 2024-03-24T02:34:44.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7158.207263 secs 2024-03-24T02:34:44.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7158.207263 secs 2024-03-24T02:34:44.341 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7813 sec 2024-03-24T02:34:45.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:44 smithi078 ceph-mon[32429]: pgmap v4258: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:45.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:44 smithi107 ceph-mon[36485]: pgmap v4258: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:45.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:44 smithi106 ceph-mon[36505]: pgmap v4258: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:45.341 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7814 sec 2024-03-24T02:34:46.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:45 smithi078 ceph-mon[32429]: pgmap v4259: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:46.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:45 smithi107 ceph-mon[36485]: pgmap v4259: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:46.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:45 smithi106 ceph-mon[36505]: pgmap v4259: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:46.341 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7815 sec 2024-03-24T02:34:47.341 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7816 sec 2024-03-24T02:34:48.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:47 smithi078 ceph-mon[32429]: pgmap v4260: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:48.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:47 smithi107 ceph-mon[36485]: pgmap v4260: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:48.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:47 smithi106 ceph-mon[36505]: pgmap v4260: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:48.341 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7817 sec 2024-03-24T02:34:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7163.207303 secs 2024-03-24T02:34:49.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7163.207303 secs 2024-03-24T02:34:49.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7163.207303 secs 2024-03-24T02:34:49.341 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7818 sec 2024-03-24T02:34:50.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:49 smithi078 ceph-mon[32429]: pgmap v4261: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:50.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:49 smithi107 ceph-mon[36485]: pgmap v4261: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:50.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:49 smithi106 ceph-mon[36505]: pgmap v4261: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:50.341 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7819 sec 2024-03-24T02:34:51.342 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7820 sec 2024-03-24T02:34:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:52 smithi078 ceph-mon[32429]: pgmap v4262: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:52.341 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7821 sec 2024-03-24T02:34:52.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:52 smithi107 ceph-mon[36485]: pgmap v4262: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:52.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:52 smithi106 ceph-mon[36505]: pgmap v4262: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:53.342 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7822 sec 2024-03-24T02:34:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7168.207361 secs 2024-03-24T02:34:54.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7168.207361 secs 2024-03-24T02:34:54.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7168.207361 secs 2024-03-24T02:34:54.342 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7823 sec 2024-03-24T02:34:55.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:54 smithi078 ceph-mon[32429]: pgmap v4263: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:55.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:54 smithi107 ceph-mon[36485]: pgmap v4263: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:55.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:54 smithi106 ceph-mon[36505]: pgmap v4263: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:55.342 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7824 sec 2024-03-24T02:34:56.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:55 smithi078 ceph-mon[32429]: pgmap v4264: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:56.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:55 smithi107 ceph-mon[36485]: pgmap v4264: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:56.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:55 smithi106 ceph-mon[36505]: pgmap v4264: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:56.342 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7825 sec 2024-03-24T02:34:57.342 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7826 sec 2024-03-24T02:34:58.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:57 smithi078 ceph-mon[32429]: pgmap v4265: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:58.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:57 smithi107 ceph-mon[36485]: pgmap v4265: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:58.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:57 smithi106 ceph-mon[36505]: pgmap v4265: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:34:58.342 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7827 sec 2024-03-24T02:34:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7173.207452 secs 2024-03-24T02:34:59.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7173.207452 secs 2024-03-24T02:34:59.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7173.207452 secs 2024-03-24T02:34:59.342 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7828 sec 2024-03-24T02:35:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:34:59 smithi078 ceph-mon[32429]: pgmap v4266: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:00.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:34:59 smithi107 ceph-mon[36485]: pgmap v4266: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:00.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:34:59 smithi106 ceph-mon[36505]: pgmap v4266: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:00.342 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7829 sec 2024-03-24T02:35:01.342 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7830 sec 2024-03-24T02:35:02.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:01 smithi078 ceph-mon[32429]: pgmap v4267: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:02.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:01 smithi107 ceph-mon[36485]: pgmap v4267: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:02.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:01 smithi106 ceph-mon[36505]: pgmap v4267: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:02.342 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7831 sec 2024-03-24T02:35:03.343 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7832 sec 2024-03-24T02:35:04.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7178.207536 secs 2024-03-24T02:35:04.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7178.207536 secs 2024-03-24T02:35:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7178.207536 secs 2024-03-24T02:35:04.343 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7833 sec 2024-03-24T02:35:05.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:04 smithi078 ceph-mon[32429]: pgmap v4268: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:05.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:04 smithi107 ceph-mon[36485]: pgmap v4268: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:05.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:04 smithi106 ceph-mon[36505]: pgmap v4268: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:05.343 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7834 sec 2024-03-24T02:35:06.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:05 smithi078 ceph-mon[32429]: pgmap v4269: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:06.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:05 smithi107 ceph-mon[36485]: pgmap v4269: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:06.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:05 smithi106 ceph-mon[36505]: pgmap v4269: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:06.343 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7835 sec 2024-03-24T02:35:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:06 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:35:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:06 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:35:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:06 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:35:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:06 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:35:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:06 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:35:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:06 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:35:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:06 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:35:07.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:06 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:35:07.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:06 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:35:07.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:06 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:35:07.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:06 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:35:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:06 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:35:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:06 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:35:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:06 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:35:07.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:06 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:35:07.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:06 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:35:07.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:06 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:35:07.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:06 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:35:07.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:06 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:35:07.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:06 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:35:07.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:06 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:35:07.343 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7836 sec 2024-03-24T02:35:08.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:07 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:35:08.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:07 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:35:08.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:07 smithi078 ceph-mon[32429]: pgmap v4270: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T02:35:08.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:07 smithi078 ceph-mon[32429]: pgmap v4271: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:35:08.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:07 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:35:08.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:07 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:35:08.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:07 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:35:08.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:07 smithi107 ceph-mon[36485]: pgmap v4270: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T02:35:08.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:07 smithi107 ceph-mon[36485]: pgmap v4271: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:35:08.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:07 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:35:08.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:07 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:35:08.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:07 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:35:08.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:07 smithi106 ceph-mon[36505]: pgmap v4270: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T02:35:08.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:07 smithi106 ceph-mon[36505]: pgmap v4271: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:35:08.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:07 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:35:08.343 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7837 sec 2024-03-24T02:35:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7183.207625 secs 2024-03-24T02:35:09.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7183.207625 secs 2024-03-24T02:35:09.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7183.207625 secs 2024-03-24T02:35:09.343 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7838 sec 2024-03-24T02:35:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:09 smithi078 ceph-mon[32429]: pgmap v4272: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:35:10.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:09 smithi107 ceph-mon[36485]: pgmap v4272: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:35:10.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:09 smithi106 ceph-mon[36505]: pgmap v4272: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:35:10.343 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7839 sec 2024-03-24T02:35:11.343 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7840 sec 2024-03-24T02:35:12.339 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:12 smithi078 ceph-mon[32429]: pgmap v4273: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:35:12.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:12 smithi107 ceph-mon[36485]: pgmap v4273: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:35:12.343 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7841 sec 2024-03-24T02:35:12.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:12 smithi106 ceph-mon[36505]: pgmap v4273: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:35:13.343 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7842 sec 2024-03-24T02:35:14.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7188.207713 secs 2024-03-24T02:35:14.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:14 smithi078 ceph-mon[32429]: pgmap v4274: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:35:14.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7188.207713 secs 2024-03-24T02:35:14.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:14 smithi107 ceph-mon[36485]: pgmap v4274: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:35:14.343 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7843 sec 2024-03-24T02:35:14.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7188.207713 secs 2024-03-24T02:35:14.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:14 smithi106 ceph-mon[36505]: pgmap v4274: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:35:15.343 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7844 sec 2024-03-24T02:35:16.339 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:16 smithi078 ceph-mon[32429]: pgmap v4275: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:35:16.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:16 smithi107 ceph-mon[36485]: pgmap v4275: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:35:16.343 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7845 sec 2024-03-24T02:35:16.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:16 smithi106 ceph-mon[36505]: pgmap v4275: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:35:17.344 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7846 sec 2024-03-24T02:35:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:18 smithi078 ceph-mon[32429]: pgmap v4276: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:35:18.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:18 smithi107 ceph-mon[36485]: pgmap v4276: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:35:18.344 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7847 sec 2024-03-24T02:35:18.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:18 smithi106 ceph-mon[36505]: pgmap v4276: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:35:19.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7193.207775 secs 2024-03-24T02:35:19.344 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7848 sec 2024-03-24T02:35:19.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7193.207775 secs 2024-03-24T02:35:19.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7193.207775 secs 2024-03-24T02:35:20.344 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7849 sec 2024-03-24T02:35:20.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:20 smithi106 ceph-mon[36505]: pgmap v4277: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:20.571 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:20 smithi078 ceph-mon[32429]: pgmap v4277: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:20.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:20 smithi107 ceph-mon[36485]: pgmap v4277: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:21.344 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7850 sec 2024-03-24T02:35:22.344 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7851 sec 2024-03-24T02:35:22.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:22 smithi106 ceph-mon[36505]: pgmap v4278: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:22.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:22 smithi078 ceph-mon[32429]: pgmap v4278: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:22.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:22 smithi107 ceph-mon[36485]: pgmap v4278: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:23.344 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7852 sec 2024-03-24T02:35:24.344 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7853 sec 2024-03-24T02:35:24.392 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7198.207859 secs 2024-03-24T02:35:24.392 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:24 smithi078 ceph-mon[32429]: pgmap v4279: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:24.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7198.207859 secs 2024-03-24T02:35:24.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:24 smithi106 ceph-mon[36505]: pgmap v4279: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:24.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7198.207859 secs 2024-03-24T02:35:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:24 smithi107 ceph-mon[36485]: pgmap v4279: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:25.344 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7854 sec 2024-03-24T02:35:26.346 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7855 sec 2024-03-24T02:35:26.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:26 smithi106 ceph-mon[36505]: pgmap v4280: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:26.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:26 smithi078 ceph-mon[32429]: pgmap v4280: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:26.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:26 smithi107 ceph-mon[36485]: pgmap v4280: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:27.345 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7856 sec 2024-03-24T02:35:28.345 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7857 sec 2024-03-24T02:35:28.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:28 smithi106 ceph-mon[36505]: pgmap v4281: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:28.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:28 smithi078 ceph-mon[32429]: pgmap v4281: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:28.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:28 smithi107 ceph-mon[36485]: pgmap v4281: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:29.345 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7858 sec 2024-03-24T02:35:29.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7203.207950 secs 2024-03-24T02:35:29.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7203.207950 secs 2024-03-24T02:35:29.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7203.207950 secs 2024-03-24T02:35:30.345 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7859 sec 2024-03-24T02:35:30.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:30 smithi106 ceph-mon[36505]: pgmap v4282: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:30.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:30 smithi078 ceph-mon[32429]: pgmap v4282: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:30.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:30 smithi107 ceph-mon[36485]: pgmap v4282: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:31.345 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7860 sec 2024-03-24T02:35:32.345 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7861 sec 2024-03-24T02:35:32.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:32 smithi106 ceph-mon[36505]: pgmap v4283: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:32.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:32 smithi078 ceph-mon[32429]: pgmap v4283: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:32.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:32 smithi107 ceph-mon[36485]: pgmap v4283: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:33.345 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7862 sec 2024-03-24T02:35:34.345 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7863 sec 2024-03-24T02:35:34.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7208.208036 secs 2024-03-24T02:35:34.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:34 smithi106 ceph-mon[36505]: pgmap v4284: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:34.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7208.208036 secs 2024-03-24T02:35:34.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:34 smithi078 ceph-mon[32429]: pgmap v4284: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:34.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7208.208036 secs 2024-03-24T02:35:34.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:34 smithi107 ceph-mon[36485]: pgmap v4284: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:35.345 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7864 sec 2024-03-24T02:35:36.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:36 smithi078 ceph-mon[32429]: pgmap v4285: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:36.345 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7865 sec 2024-03-24T02:35:36.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:36 smithi106 ceph-mon[36505]: pgmap v4285: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:36.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:36 smithi107 ceph-mon[36485]: pgmap v4285: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:37.345 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7866 sec 2024-03-24T02:35:38.345 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7867 sec 2024-03-24T02:35:38.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:38 smithi106 ceph-mon[36505]: pgmap v4286: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:38.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:38 smithi078 ceph-mon[32429]: pgmap v4286: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:38.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:38 smithi107 ceph-mon[36485]: pgmap v4286: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:39.346 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7868 sec 2024-03-24T02:35:39.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7213.208120 secs 2024-03-24T02:35:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7213.208120 secs 2024-03-24T02:35:39.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7213.208120 secs 2024-03-24T02:35:40.346 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7869 sec 2024-03-24T02:35:40.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:40 smithi106 ceph-mon[36505]: pgmap v4287: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:40.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:40 smithi078 ceph-mon[32429]: pgmap v4287: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:40.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:40 smithi107 ceph-mon[36485]: pgmap v4287: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:41.346 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7870 sec 2024-03-24T02:35:42.346 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7871 sec 2024-03-24T02:35:42.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:42 smithi106 ceph-mon[36505]: pgmap v4288: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:42.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:42 smithi078 ceph-mon[32429]: pgmap v4288: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:42.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:42 smithi107 ceph-mon[36485]: pgmap v4288: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:43.346 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7872 sec 2024-03-24T02:35:44.346 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7873 sec 2024-03-24T02:35:44.395 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7218.208205 secs 2024-03-24T02:35:44.395 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:44 smithi107 ceph-mon[36485]: pgmap v4289: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:44.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7218.208205 secs 2024-03-24T02:35:44.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:44 smithi106 ceph-mon[36505]: pgmap v4289: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:44.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7218.208205 secs 2024-03-24T02:35:44.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:44 smithi078 ceph-mon[32429]: pgmap v4289: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:45.346 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7874 sec 2024-03-24T02:35:46.346 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7875 sec 2024-03-24T02:35:46.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:46 smithi106 ceph-mon[36505]: pgmap v4290: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:46.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:46 smithi078 ceph-mon[32429]: pgmap v4290: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:46.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:46 smithi107 ceph-mon[36485]: pgmap v4290: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:47.346 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7876 sec 2024-03-24T02:35:48.346 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7877 sec 2024-03-24T02:35:48.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:48 smithi106 ceph-mon[36505]: pgmap v4291: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:48.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:48 smithi078 ceph-mon[32429]: pgmap v4291: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:48.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:48 smithi107 ceph-mon[36485]: pgmap v4291: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:49.346 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7878 sec 2024-03-24T02:35:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7223.208308 secs 2024-03-24T02:35:49.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7223.208308 secs 2024-03-24T02:35:49.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7223.208308 secs 2024-03-24T02:35:50.346 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7879 sec 2024-03-24T02:35:50.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:50 smithi078 ceph-mon[32429]: pgmap v4292: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:50.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:50 smithi107 ceph-mon[36485]: pgmap v4292: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:50.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:50 smithi106 ceph-mon[36505]: pgmap v4292: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:51.346 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7880 sec 2024-03-24T02:35:52.347 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7881 sec 2024-03-24T02:35:52.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:52 smithi078 ceph-mon[32429]: pgmap v4293: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:52.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:52 smithi107 ceph-mon[36485]: pgmap v4293: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:52.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:52 smithi106 ceph-mon[36505]: pgmap v4293: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:53.347 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7882 sec 2024-03-24T02:35:54.347 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7883 sec 2024-03-24T02:35:54.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7228.208390 secs 2024-03-24T02:35:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:54 smithi078 ceph-mon[32429]: pgmap v4294: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:54.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7228.208390 secs 2024-03-24T02:35:54.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:54 smithi107 ceph-mon[36485]: pgmap v4294: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:54.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7228.208390 secs 2024-03-24T02:35:54.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:54 smithi106 ceph-mon[36505]: pgmap v4294: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:55.347 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7884 sec 2024-03-24T02:35:56.347 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7885 sec 2024-03-24T02:35:56.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:56 smithi078 ceph-mon[32429]: pgmap v4295: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:56.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:56 smithi107 ceph-mon[36485]: pgmap v4295: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:56.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:56 smithi106 ceph-mon[36505]: pgmap v4295: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:35:57.347 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7886 sec 2024-03-24T02:35:58.347 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7887 sec 2024-03-24T02:35:58.453 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:58 smithi106 ceph-mon[36505]: pgmap v4296: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:58.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:58 smithi078 ceph-mon[32429]: pgmap v4296: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:58.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:58 smithi107 ceph-mon[36485]: pgmap v4296: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:59.347 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7888 sec 2024-03-24T02:35:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:35:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7233.208451 secs 2024-03-24T02:35:59.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:35:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7233.208451 secs 2024-03-24T02:35:59.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:35:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7233.208451 secs 2024-03-24T02:36:00.347 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7889 sec 2024-03-24T02:36:00.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:00 smithi078 ceph-mon[32429]: pgmap v4297: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:00.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:00 smithi107 ceph-mon[36485]: pgmap v4297: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:00.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:00 smithi106 ceph-mon[36505]: pgmap v4297: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:01.347 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7890 sec 2024-03-24T02:36:02.348 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7891 sec 2024-03-24T02:36:02.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:02 smithi078 ceph-mon[32429]: pgmap v4298: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:02.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:02 smithi107 ceph-mon[36485]: pgmap v4298: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:02.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:02 smithi106 ceph-mon[36505]: pgmap v4298: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:03.348 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7892 sec 2024-03-24T02:36:04.348 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7893 sec 2024-03-24T02:36:04.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7238.208533 secs 2024-03-24T02:36:04.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:04 smithi078 ceph-mon[32429]: pgmap v4299: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:04.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7238.208533 secs 2024-03-24T02:36:04.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:04 smithi107 ceph-mon[36485]: pgmap v4299: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:04.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7238.208533 secs 2024-03-24T02:36:04.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:04 smithi106 ceph-mon[36505]: pgmap v4299: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:05.348 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7894 sec 2024-03-24T02:36:06.348 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7895 sec 2024-03-24T02:36:06.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:06 smithi078 ceph-mon[32429]: pgmap v4300: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:06.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:06 smithi107 ceph-mon[36485]: pgmap v4300: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:06.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:06 smithi106 ceph-mon[36505]: pgmap v4300: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:07.348 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7896 sec 2024-03-24T02:36:07.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:07 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:36:07.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:07 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:36:07.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:07 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:36:08.348 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7897 sec 2024-03-24T02:36:08.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:08 smithi078 ceph-mon[32429]: pgmap v4301: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:08.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:08 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:36:08.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:08 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:36:08.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:08 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:36:08.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:08 smithi107 ceph-mon[36485]: pgmap v4301: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:08.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:08 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:36:08.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:08 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:36:08.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:08 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:36:08.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:08 smithi106 ceph-mon[36505]: pgmap v4301: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:08.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:08 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:36:08.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:08 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:36:08.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:08 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:36:09.348 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7898 sec 2024-03-24T02:36:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:09 smithi078 ceph-mon[32429]: pgmap v4302: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:36:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:09 smithi078 ceph-mon[32429]: pgmap v4303: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:36:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7243.208594 secs 2024-03-24T02:36:09.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:09 smithi107 ceph-mon[36485]: pgmap v4302: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:36:09.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:09 smithi107 ceph-mon[36485]: pgmap v4303: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:36:09.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7243.208594 secs 2024-03-24T02:36:09.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:09 smithi106 ceph-mon[36505]: pgmap v4302: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:36:09.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:09 smithi106 ceph-mon[36505]: pgmap v4303: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:36:09.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7243.208594 secs 2024-03-24T02:36:10.348 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7899 sec 2024-03-24T02:36:11.348 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7900 sec 2024-03-24T02:36:11.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:11 smithi078 ceph-mon[32429]: pgmap v4304: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:36:11.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:11 smithi107 ceph-mon[36485]: pgmap v4304: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:36:11.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:11 smithi106 ceph-mon[36505]: pgmap v4304: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:36:12.348 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7901 sec 2024-03-24T02:36:13.349 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7902 sec 2024-03-24T02:36:13.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:13 smithi078 ceph-mon[32429]: pgmap v4305: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:36:13.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:13 smithi107 ceph-mon[36485]: pgmap v4305: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:36:13.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:13 smithi106 ceph-mon[36505]: pgmap v4305: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:36:14.348 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7903 sec 2024-03-24T02:36:14.494 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7248.208652 secs 2024-03-24T02:36:14.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7248.208652 secs 2024-03-24T02:36:14.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7248.208652 secs 2024-03-24T02:36:15.348 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7904 sec 2024-03-24T02:36:15.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:15 smithi078 ceph-mon[32429]: pgmap v4306: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:36:15.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:15 smithi107 ceph-mon[36485]: pgmap v4306: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:36:15.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:15 smithi106 ceph-mon[36505]: pgmap v4306: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:36:16.349 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7905 sec 2024-03-24T02:36:17.349 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7906 sec 2024-03-24T02:36:17.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:17 smithi078 ceph-mon[32429]: pgmap v4307: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:36:17.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:17 smithi107 ceph-mon[36485]: pgmap v4307: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:36:17.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:17 smithi106 ceph-mon[36505]: pgmap v4307: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:36:18.349 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7907 sec 2024-03-24T02:36:19.349 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7908 sec 2024-03-24T02:36:19.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:19 smithi078 ceph-mon[32429]: pgmap v4308: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:36:19.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7253.208738 secs 2024-03-24T02:36:19.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:19 smithi107 ceph-mon[36485]: pgmap v4308: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:36:19.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7253.208738 secs 2024-03-24T02:36:19.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:19 smithi106 ceph-mon[36505]: pgmap v4308: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:36:19.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7253.208738 secs 2024-03-24T02:36:20.349 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7909 sec 2024-03-24T02:36:21.349 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7910 sec 2024-03-24T02:36:21.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:21 smithi078 ceph-mon[32429]: pgmap v4309: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:21.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:21 smithi107 ceph-mon[36485]: pgmap v4309: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:21.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:21 smithi106 ceph-mon[36505]: pgmap v4309: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:22.349 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7911 sec 2024-03-24T02:36:23.349 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7912 sec 2024-03-24T02:36:23.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:23 smithi078 ceph-mon[32429]: pgmap v4310: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:23.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:23 smithi107 ceph-mon[36485]: pgmap v4310: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:23.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:23 smithi106 ceph-mon[36505]: pgmap v4310: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:24.339 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7258.208826 secs 2024-03-24T02:36:24.349 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7913 sec 2024-03-24T02:36:24.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7258.208826 secs 2024-03-24T02:36:24.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7258.208826 secs 2024-03-24T02:36:25.349 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7914 sec 2024-03-24T02:36:25.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:25 smithi078 ceph-mon[32429]: pgmap v4311: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:25.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:25 smithi107 ceph-mon[36485]: pgmap v4311: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:25.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:25 smithi106 ceph-mon[36505]: pgmap v4311: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:26.349 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7915 sec 2024-03-24T02:36:27.349 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7916 sec 2024-03-24T02:36:27.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:27 smithi078 ceph-mon[32429]: pgmap v4312: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:27.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:27 smithi107 ceph-mon[36485]: pgmap v4312: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:27.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:27 smithi106 ceph-mon[36505]: pgmap v4312: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:28.349 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7917 sec 2024-03-24T02:36:29.350 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7918 sec 2024-03-24T02:36:29.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:29 smithi078 ceph-mon[32429]: pgmap v4313: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:29.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7263.208841 secs 2024-03-24T02:36:29.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:29 smithi107 ceph-mon[36485]: pgmap v4313: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:29.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7263.208841 secs 2024-03-24T02:36:29.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:29 smithi106 ceph-mon[36505]: pgmap v4313: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:29.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7263.208841 secs 2024-03-24T02:36:30.350 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7919 sec 2024-03-24T02:36:31.350 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7920 sec 2024-03-24T02:36:31.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:31 smithi078 ceph-mon[32429]: pgmap v4314: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:31.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:31 smithi107 ceph-mon[36485]: pgmap v4314: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:31.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:31 smithi106 ceph-mon[36505]: pgmap v4314: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:32.350 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7921 sec 2024-03-24T02:36:33.350 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7922 sec 2024-03-24T02:36:33.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:33 smithi078 ceph-mon[32429]: pgmap v4315: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:33.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:33 smithi107 ceph-mon[36485]: pgmap v4315: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:33.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:33 smithi106 ceph-mon[36505]: pgmap v4315: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:34.350 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7923 sec 2024-03-24T02:36:34.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7268.208897 secs 2024-03-24T02:36:34.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7268.208897 secs 2024-03-24T02:36:34.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7268.208897 secs 2024-03-24T02:36:35.350 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7924 sec 2024-03-24T02:36:35.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:35 smithi078 ceph-mon[32429]: pgmap v4316: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:35.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:35 smithi107 ceph-mon[36485]: pgmap v4316: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:35.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:35 smithi106 ceph-mon[36505]: pgmap v4316: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:36.350 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7925 sec 2024-03-24T02:36:37.350 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7926 sec 2024-03-24T02:36:37.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:37 smithi078 ceph-mon[32429]: pgmap v4317: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:37.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:37 smithi107 ceph-mon[36485]: pgmap v4317: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:37.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:37 smithi106 ceph-mon[36505]: pgmap v4317: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:38.350 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7927 sec 2024-03-24T02:36:39.350 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7928 sec 2024-03-24T02:36:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:39 smithi078 ceph-mon[32429]: pgmap v4318: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:39.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7273.208940 secs 2024-03-24T02:36:39.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:39 smithi107 ceph-mon[36485]: pgmap v4318: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:39.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7273.208940 secs 2024-03-24T02:36:39.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:39 smithi106 ceph-mon[36505]: pgmap v4318: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:39.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7273.208940 secs 2024-03-24T02:36:40.351 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7929 sec 2024-03-24T02:36:41.351 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7930 sec 2024-03-24T02:36:41.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:41 smithi078 ceph-mon[32429]: pgmap v4319: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:41.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:41 smithi107 ceph-mon[36485]: pgmap v4319: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:41.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:41 smithi106 ceph-mon[36505]: pgmap v4319: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:42.351 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7931 sec 2024-03-24T02:36:43.351 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7932 sec 2024-03-24T02:36:43.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:43 smithi078 ceph-mon[32429]: pgmap v4320: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:43.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:43 smithi107 ceph-mon[36485]: pgmap v4320: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:43.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:43 smithi106 ceph-mon[36505]: pgmap v4320: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:44.351 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7933 sec 2024-03-24T02:36:44.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7278.209024 secs 2024-03-24T02:36:44.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7278.209024 secs 2024-03-24T02:36:44.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7278.209024 secs 2024-03-24T02:36:45.351 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7934 sec 2024-03-24T02:36:45.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:45 smithi078 ceph-mon[32429]: pgmap v4321: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:45.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:45 smithi107 ceph-mon[36485]: pgmap v4321: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:45.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:45 smithi106 ceph-mon[36505]: pgmap v4321: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:46.351 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7935 sec 2024-03-24T02:36:47.351 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7936 sec 2024-03-24T02:36:47.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:47 smithi078 ceph-mon[32429]: pgmap v4322: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:47.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:47 smithi107 ceph-mon[36485]: pgmap v4322: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:47.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:47 smithi106 ceph-mon[36505]: pgmap v4322: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:48.351 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7937 sec 2024-03-24T02:36:49.352 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7938 sec 2024-03-24T02:36:49.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:49 smithi078 ceph-mon[32429]: pgmap v4323: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7283.209064 secs 2024-03-24T02:36:49.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:49 smithi107 ceph-mon[36485]: pgmap v4323: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:49.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7283.209064 secs 2024-03-24T02:36:49.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:49 smithi106 ceph-mon[36505]: pgmap v4323: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:49.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7283.209064 secs 2024-03-24T02:36:50.351 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7939 sec 2024-03-24T02:36:51.352 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7940 sec 2024-03-24T02:36:51.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:51 smithi078 ceph-mon[32429]: pgmap v4324: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:51.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:51 smithi107 ceph-mon[36485]: pgmap v4324: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:51.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:51 smithi106 ceph-mon[36505]: pgmap v4324: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:52.352 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7941 sec 2024-03-24T02:36:53.352 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7942 sec 2024-03-24T02:36:53.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:53 smithi078 ceph-mon[32429]: pgmap v4325: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:53.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:53 smithi107 ceph-mon[36485]: pgmap v4325: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:53.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:53 smithi106 ceph-mon[36505]: pgmap v4325: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:54.352 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7943 sec 2024-03-24T02:36:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7288.209118 secs 2024-03-24T02:36:54.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7288.209118 secs 2024-03-24T02:36:54.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7288.209118 secs 2024-03-24T02:36:55.352 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7944 sec 2024-03-24T02:36:55.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:55 smithi078 ceph-mon[32429]: pgmap v4326: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:55.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:55 smithi107 ceph-mon[36485]: pgmap v4326: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:55.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:55 smithi106 ceph-mon[36505]: pgmap v4326: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:56.352 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7945 sec 2024-03-24T02:36:57.354 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7946 sec 2024-03-24T02:36:57.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:57 smithi078 ceph-mon[32429]: pgmap v4327: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:57.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:57 smithi107 ceph-mon[36485]: pgmap v4327: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:57.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:57 smithi106 ceph-mon[36505]: pgmap v4327: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:58.352 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7947 sec 2024-03-24T02:36:59.352 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7948 sec 2024-03-24T02:36:59.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:59 smithi078 ceph-mon[32429]: pgmap v4328: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:59.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:36:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7293.209172 secs 2024-03-24T02:36:59.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:59 smithi107 ceph-mon[36485]: pgmap v4328: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:59.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:36:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7293.209172 secs 2024-03-24T02:36:59.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:59 smithi106 ceph-mon[36505]: pgmap v4328: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:36:59.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:36:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7293.209172 secs 2024-03-24T02:37:00.352 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7949 sec 2024-03-24T02:37:01.352 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7950 sec 2024-03-24T02:37:01.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:01 smithi078 ceph-mon[32429]: pgmap v4329: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:37:01.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:01 smithi107 ceph-mon[36485]: pgmap v4329: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:37:01.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:01 smithi106 ceph-mon[36505]: pgmap v4329: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:37:02.352 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7951 sec 2024-03-24T02:37:03.353 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7952 sec 2024-03-24T02:37:03.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:03 smithi078 ceph-mon[32429]: pgmap v4330: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:03.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:03 smithi107 ceph-mon[36485]: pgmap v4330: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:03.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:03 smithi106 ceph-mon[36505]: pgmap v4330: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:04.353 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7953 sec 2024-03-24T02:37:04.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7298.209230 secs 2024-03-24T02:37:04.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7298.209230 secs 2024-03-24T02:37:04.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7298.209230 secs 2024-03-24T02:37:05.355 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7954 sec 2024-03-24T02:37:05.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:05 smithi078 ceph-mon[32429]: pgmap v4331: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:37:05.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:05 smithi107 ceph-mon[36485]: pgmap v4331: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:37:05.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:05 smithi106 ceph-mon[36505]: pgmap v4331: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:37:06.353 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7955 sec 2024-03-24T02:37:07.353 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7956 sec 2024-03-24T02:37:07.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:07 smithi078 ceph-mon[32429]: pgmap v4332: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:07.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:07 smithi107 ceph-mon[36485]: pgmap v4332: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:07.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:07 smithi106 ceph-mon[36505]: pgmap v4332: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:08.353 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7957 sec 2024-03-24T02:37:08.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:08 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:37:08.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:08 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:37:08.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:08 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:37:09.353 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7958 sec 2024-03-24T02:37:09.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:09 smithi078 ceph-mon[32429]: pgmap v4333: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:37:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7303.209298 secs 2024-03-24T02:37:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:37:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:37:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:09 smithi078 ceph-mon[32429]: pgmap v4334: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:37:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:09 smithi078 ceph-mon[32429]: pgmap v4335: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:37:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:37:09.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:09 smithi107 ceph-mon[36485]: pgmap v4333: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:37:09.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7303.209298 secs 2024-03-24T02:37:09.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:37:09.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:37:09.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:09 smithi107 ceph-mon[36485]: pgmap v4334: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:37:09.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:09 smithi107 ceph-mon[36485]: pgmap v4335: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:37:09.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:37:09.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:09 smithi106 ceph-mon[36505]: pgmap v4333: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:37:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7303.209298 secs 2024-03-24T02:37:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:37:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:37:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:09 smithi106 ceph-mon[36505]: pgmap v4334: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:37:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:09 smithi106 ceph-mon[36505]: pgmap v4335: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:37:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:37:10.353 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7959 sec 2024-03-24T02:37:11.353 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7960 sec 2024-03-24T02:37:11.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:11 smithi106 ceph-mon[36505]: pgmap v4336: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:37:12.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:11 smithi078 ceph-mon[32429]: pgmap v4336: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:37:12.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:11 smithi107 ceph-mon[36485]: pgmap v4336: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:37:12.353 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7961 sec 2024-03-24T02:37:13.353 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7962 sec 2024-03-24T02:37:13.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:13 smithi106 ceph-mon[36505]: pgmap v4337: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:37:13.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7308.209357 secs 2024-03-24T02:37:14.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:13 smithi078 ceph-mon[32429]: pgmap v4337: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:37:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7308.209357 secs 2024-03-24T02:37:14.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:13 smithi107 ceph-mon[36485]: pgmap v4337: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:37:14.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7308.209357 secs 2024-03-24T02:37:14.354 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7963 sec 2024-03-24T02:37:15.354 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7964 sec 2024-03-24T02:37:15.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:15 smithi106 ceph-mon[36505]: pgmap v4338: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:37:16.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:15 smithi078 ceph-mon[32429]: pgmap v4338: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:37:16.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:15 smithi107 ceph-mon[36485]: pgmap v4338: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:37:16.354 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7965 sec 2024-03-24T02:37:17.354 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7966 sec 2024-03-24T02:37:17.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:17 smithi106 ceph-mon[36505]: pgmap v4339: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:37:18.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:17 smithi078 ceph-mon[32429]: pgmap v4339: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:37:18.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:17 smithi107 ceph-mon[36485]: pgmap v4339: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:37:18.354 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7967 sec 2024-03-24T02:37:19.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7313.209420 secs 2024-03-24T02:37:19.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7313.209420 secs 2024-03-24T02:37:19.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7313.209420 secs 2024-03-24T02:37:19.354 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7968 sec 2024-03-24T02:37:20.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:19 smithi078 ceph-mon[32429]: pgmap v4340: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:37:20.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:19 smithi107 ceph-mon[36485]: pgmap v4340: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:37:20.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:19 smithi106 ceph-mon[36505]: pgmap v4340: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:37:20.354 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7969 sec 2024-03-24T02:37:21.354 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7970 sec 2024-03-24T02:37:22.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:21 smithi078 ceph-mon[32429]: pgmap v4341: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:22.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:21 smithi107 ceph-mon[36485]: pgmap v4341: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:22.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:21 smithi106 ceph-mon[36505]: pgmap v4341: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:22.354 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7971 sec 2024-03-24T02:37:23.354 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7972 sec 2024-03-24T02:37:24.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:23 smithi078 ceph-mon[32429]: pgmap v4342: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:37:24.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7318.209483 secs 2024-03-24T02:37:24.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:23 smithi107 ceph-mon[36485]: pgmap v4342: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:37:24.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7318.209483 secs 2024-03-24T02:37:24.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:23 smithi106 ceph-mon[36505]: pgmap v4342: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:37:24.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7318.209483 secs 2024-03-24T02:37:24.354 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7973 sec 2024-03-24T02:37:25.354 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7974 sec 2024-03-24T02:37:26.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:25 smithi078 ceph-mon[32429]: pgmap v4343: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:37:26.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:25 smithi107 ceph-mon[36485]: pgmap v4343: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:37:26.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:25 smithi106 ceph-mon[36505]: pgmap v4343: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:37:26.354 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7975 sec 2024-03-24T02:37:27.355 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7976 sec 2024-03-24T02:37:28.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:27 smithi078 ceph-mon[32429]: pgmap v4344: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:28.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:27 smithi107 ceph-mon[36485]: pgmap v4344: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:28.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:27 smithi106 ceph-mon[36505]: pgmap v4344: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:28.355 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7977 sec 2024-03-24T02:37:29.052 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7323.209546 secs 2024-03-24T02:37:29.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7323.209546 secs 2024-03-24T02:37:29.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7323.209546 secs 2024-03-24T02:37:29.355 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7978 sec 2024-03-24T02:37:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:29 smithi078 ceph-mon[32429]: pgmap v4345: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:37:30.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:29 smithi107 ceph-mon[36485]: pgmap v4345: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:37:30.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:29 smithi106 ceph-mon[36505]: pgmap v4345: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:37:30.355 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7979 sec 2024-03-24T02:37:31.355 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7980 sec 2024-03-24T02:37:32.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:31 smithi078 ceph-mon[32429]: pgmap v4346: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:32.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:31 smithi107 ceph-mon[36485]: pgmap v4346: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:32.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:31 smithi106 ceph-mon[36505]: pgmap v4346: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:32.355 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7981 sec 2024-03-24T02:37:33.355 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7982 sec 2024-03-24T02:37:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:33 smithi078 ceph-mon[32429]: pgmap v4347: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:37:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7328.209625 secs 2024-03-24T02:37:34.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:33 smithi107 ceph-mon[36485]: pgmap v4347: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:37:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7328.209625 secs 2024-03-24T02:37:34.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:33 smithi106 ceph-mon[36505]: pgmap v4347: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:37:34.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7328.209625 secs 2024-03-24T02:37:34.355 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7983 sec 2024-03-24T02:37:35.355 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7984 sec 2024-03-24T02:37:36.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:35 smithi078 ceph-mon[32429]: pgmap v4348: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:37:36.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:35 smithi107 ceph-mon[36485]: pgmap v4348: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:37:36.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:35 smithi106 ceph-mon[36505]: pgmap v4348: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:37:36.355 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7985 sec 2024-03-24T02:37:37.355 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7986 sec 2024-03-24T02:37:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:37 smithi078 ceph-mon[32429]: pgmap v4349: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:37:38.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:37 smithi107 ceph-mon[36485]: pgmap v4349: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:37:38.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:37 smithi106 ceph-mon[36505]: pgmap v4349: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:37:38.355 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7987 sec 2024-03-24T02:37:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7333.209714 secs 2024-03-24T02:37:39.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7333.209714 secs 2024-03-24T02:37:39.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7333.209714 secs 2024-03-24T02:37:39.355 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7988 sec 2024-03-24T02:37:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:39 smithi078 ceph-mon[32429]: pgmap v4350: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:37:40.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:39 smithi107 ceph-mon[36485]: pgmap v4350: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:37:40.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:39 smithi106 ceph-mon[36505]: pgmap v4350: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:37:40.356 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7989 sec 2024-03-24T02:37:41.356 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7990 sec 2024-03-24T02:37:42.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:41 smithi078 ceph-mon[32429]: pgmap v4351: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:37:42.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:41 smithi107 ceph-mon[36485]: pgmap v4351: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:37:42.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:41 smithi106 ceph-mon[36505]: pgmap v4351: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:37:42.356 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7991 sec 2024-03-24T02:37:43.356 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7992 sec 2024-03-24T02:37:44.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:43 smithi078 ceph-mon[32429]: pgmap v4352: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:37:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7338.209773 secs 2024-03-24T02:37:44.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:43 smithi107 ceph-mon[36485]: pgmap v4352: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:37:44.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7338.209773 secs 2024-03-24T02:37:44.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:43 smithi106 ceph-mon[36505]: pgmap v4352: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:37:44.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7338.209773 secs 2024-03-24T02:37:44.356 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7993 sec 2024-03-24T02:37:45.356 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7994 sec 2024-03-24T02:37:46.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:45 smithi078 ceph-mon[32429]: pgmap v4353: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:37:46.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:45 smithi107 ceph-mon[36485]: pgmap v4353: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:37:46.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:45 smithi106 ceph-mon[36505]: pgmap v4353: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:37:46.356 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7995 sec 2024-03-24T02:37:47.356 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7996 sec 2024-03-24T02:37:48.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:47 smithi078 ceph-mon[32429]: pgmap v4354: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:37:48.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:47 smithi107 ceph-mon[36485]: pgmap v4354: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:37:48.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:47 smithi106 ceph-mon[36505]: pgmap v4354: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:37:48.356 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7997 sec 2024-03-24T02:37:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7343.209853 secs 2024-03-24T02:37:49.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7343.209853 secs 2024-03-24T02:37:49.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7343.209853 secs 2024-03-24T02:37:49.356 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7998 sec 2024-03-24T02:37:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:49 smithi078 ceph-mon[32429]: pgmap v4355: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:37:50.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:49 smithi107 ceph-mon[36485]: pgmap v4355: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:37:50.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:49 smithi106 ceph-mon[36505]: pgmap v4355: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:37:50.356 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 7999 sec 2024-03-24T02:37:51.357 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8000 sec 2024-03-24T02:37:52.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:51 smithi078 ceph-mon[32429]: pgmap v4356: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:52.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:51 smithi107 ceph-mon[36485]: pgmap v4356: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:52.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:51 smithi106 ceph-mon[36505]: pgmap v4356: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:52.357 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8001 sec 2024-03-24T02:37:53.357 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8002 sec 2024-03-24T02:37:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:53 smithi078 ceph-mon[32429]: pgmap v4357: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:37:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7348.209942 secs 2024-03-24T02:37:54.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:53 smithi107 ceph-mon[36485]: pgmap v4357: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:37:54.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7348.209942 secs 2024-03-24T02:37:54.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:53 smithi106 ceph-mon[36505]: pgmap v4357: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:37:54.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7348.209942 secs 2024-03-24T02:37:54.357 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8003 sec 2024-03-24T02:37:55.357 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8004 sec 2024-03-24T02:37:56.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:55 smithi078 ceph-mon[32429]: pgmap v4358: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:37:56.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:55 smithi107 ceph-mon[36485]: pgmap v4358: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:37:56.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:55 smithi106 ceph-mon[36505]: pgmap v4358: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:37:56.357 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8005 sec 2024-03-24T02:37:57.357 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8006 sec 2024-03-24T02:37:58.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:57 smithi078 ceph-mon[32429]: pgmap v4359: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:58.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:57 smithi107 ceph-mon[36485]: pgmap v4359: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:58.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:57 smithi106 ceph-mon[36505]: pgmap v4359: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:58.357 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8007 sec 2024-03-24T02:37:59.052 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7353.210001 secs 2024-03-24T02:37:59.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7353.210001 secs 2024-03-24T02:37:59.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7353.210001 secs 2024-03-24T02:37:59.357 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8008 sec 2024-03-24T02:38:00.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:37:59 smithi078 ceph-mon[32429]: pgmap v4360: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:00.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:37:59 smithi107 ceph-mon[36485]: pgmap v4360: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:00.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:37:59 smithi106 ceph-mon[36505]: pgmap v4360: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:00.357 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8009 sec 2024-03-24T02:38:01.358 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8010 sec 2024-03-24T02:38:02.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:01 smithi078 ceph-mon[32429]: pgmap v4361: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:02.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:01 smithi107 ceph-mon[36485]: pgmap v4361: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:02.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:01 smithi106 ceph-mon[36505]: pgmap v4361: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:02.358 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8011 sec 2024-03-24T02:38:03.358 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8012 sec 2024-03-24T02:38:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:03 smithi078 ceph-mon[32429]: pgmap v4362: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7358.210017 secs 2024-03-24T02:38:04.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:03 smithi107 ceph-mon[36485]: pgmap v4362: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:04.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7358.210017 secs 2024-03-24T02:38:04.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:03 smithi106 ceph-mon[36505]: pgmap v4362: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:04.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7358.210017 secs 2024-03-24T02:38:04.358 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8013 sec 2024-03-24T02:38:05.358 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8014 sec 2024-03-24T02:38:06.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:05 smithi078 ceph-mon[32429]: pgmap v4363: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:06.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:05 smithi107 ceph-mon[36485]: pgmap v4363: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:06.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:05 smithi106 ceph-mon[36505]: pgmap v4363: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:06.358 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8015 sec 2024-03-24T02:38:07.358 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8016 sec 2024-03-24T02:38:08.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:07 smithi078 ceph-mon[32429]: pgmap v4364: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:08.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:07 smithi107 ceph-mon[36485]: pgmap v4364: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:08.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:07 smithi106 ceph-mon[36505]: pgmap v4364: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:08.358 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8017 sec 2024-03-24T02:38:09.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7363.210076 secs 2024-03-24T02:38:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:08 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:38:09.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7363.210076 secs 2024-03-24T02:38:09.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:08 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:38:09.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7363.210076 secs 2024-03-24T02:38:09.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:08 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:38:09.358 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8018 sec 2024-03-24T02:38:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:09 smithi078 ceph-mon[32429]: pgmap v4365: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:38:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:38:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:09 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:38:10.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:09 smithi107 ceph-mon[36485]: pgmap v4365: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:10.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:38:10.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:38:10.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:09 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:38:10.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:09 smithi106 ceph-mon[36505]: pgmap v4365: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:10.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:38:10.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:38:10.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:09 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:38:10.358 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8019 sec 2024-03-24T02:38:11.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:10 smithi078 ceph-mon[32429]: pgmap v4366: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:38:11.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:10 smithi078 ceph-mon[32429]: pgmap v4367: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:38:11.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:10 smithi107 ceph-mon[36485]: pgmap v4366: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:38:11.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:10 smithi107 ceph-mon[36485]: pgmap v4367: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:38:11.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:10 smithi106 ceph-mon[36505]: pgmap v4366: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:38:11.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:10 smithi106 ceph-mon[36505]: pgmap v4367: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:38:11.358 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8020 sec 2024-03-24T02:38:12.359 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8021 sec 2024-03-24T02:38:13.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:12 smithi078 ceph-mon[32429]: pgmap v4368: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:38:13.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:12 smithi107 ceph-mon[36485]: pgmap v4368: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:38:13.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:12 smithi106 ceph-mon[36505]: pgmap v4368: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:38:13.359 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8022 sec 2024-03-24T02:38:14.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7368.210092 secs 2024-03-24T02:38:14.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7368.210092 secs 2024-03-24T02:38:14.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7368.210092 secs 2024-03-24T02:38:14.359 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8023 sec 2024-03-24T02:38:15.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:14 smithi078 ceph-mon[32429]: pgmap v4369: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:38:15.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:14 smithi107 ceph-mon[36485]: pgmap v4369: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:38:15.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:14 smithi106 ceph-mon[36505]: pgmap v4369: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:38:15.359 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8024 sec 2024-03-24T02:38:16.359 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8025 sec 2024-03-24T02:38:17.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:16 smithi078 ceph-mon[32429]: pgmap v4370: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:38:17.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:16 smithi107 ceph-mon[36485]: pgmap v4370: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:38:17.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:16 smithi106 ceph-mon[36505]: pgmap v4370: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:38:17.359 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8026 sec 2024-03-24T02:38:18.359 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8027 sec 2024-03-24T02:38:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:18 smithi078 ceph-mon[32429]: pgmap v4371: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:38:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7373.210150 secs 2024-03-24T02:38:19.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:18 smithi107 ceph-mon[36485]: pgmap v4371: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:38:19.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7373.210150 secs 2024-03-24T02:38:19.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:18 smithi106 ceph-mon[36505]: pgmap v4371: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:38:19.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7373.210150 secs 2024-03-24T02:38:19.359 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8028 sec 2024-03-24T02:38:20.359 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8029 sec 2024-03-24T02:38:21.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:20 smithi078 ceph-mon[32429]: pgmap v4372: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:38:21.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:20 smithi107 ceph-mon[36485]: pgmap v4372: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:38:21.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:20 smithi106 ceph-mon[36505]: pgmap v4372: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:38:21.360 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8030 sec 2024-03-24T02:38:22.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:22 smithi078 ceph-mon[32429]: pgmap v4373: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:22.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:22 smithi107 ceph-mon[36485]: pgmap v4373: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:22.359 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8031 sec 2024-03-24T02:38:22.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:22 smithi106 ceph-mon[36505]: pgmap v4373: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:23.360 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8032 sec 2024-03-24T02:38:24.360 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8033 sec 2024-03-24T02:38:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7378.210235 secs 2024-03-24T02:38:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:24 smithi078 ceph-mon[32429]: pgmap v4374: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:24.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7378.210235 secs 2024-03-24T02:38:24.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:24 smithi107 ceph-mon[36485]: pgmap v4374: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:24.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7378.210235 secs 2024-03-24T02:38:24.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:24 smithi106 ceph-mon[36505]: pgmap v4374: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:25.360 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8034 sec 2024-03-24T02:38:26.360 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8035 sec 2024-03-24T02:38:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:26 smithi078 ceph-mon[32429]: pgmap v4375: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:26.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:26 smithi107 ceph-mon[36485]: pgmap v4375: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:26.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:26 smithi106 ceph-mon[36505]: pgmap v4375: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:27.360 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8036 sec 2024-03-24T02:38:28.360 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8037 sec 2024-03-24T02:38:28.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:28 smithi078 ceph-mon[32429]: pgmap v4376: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:28.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:28 smithi107 ceph-mon[36485]: pgmap v4376: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:28.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:28 smithi106 ceph-mon[36505]: pgmap v4376: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:29.360 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8038 sec 2024-03-24T02:38:29.802 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7383.210320 secs 2024-03-24T02:38:29.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7383.210320 secs 2024-03-24T02:38:29.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7383.210320 secs 2024-03-24T02:38:30.360 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8039 sec 2024-03-24T02:38:30.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:30 smithi078 ceph-mon[32429]: pgmap v4377: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:30.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:30 smithi107 ceph-mon[36485]: pgmap v4377: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:30.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:30 smithi106 ceph-mon[36505]: pgmap v4377: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:31.360 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8040 sec 2024-03-24T02:38:32.361 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8041 sec 2024-03-24T02:38:32.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:32 smithi078 ceph-mon[32429]: pgmap v4378: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:32.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:32 smithi107 ceph-mon[36485]: pgmap v4378: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:32.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:32 smithi106 ceph-mon[36505]: pgmap v4378: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:33.361 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8042 sec 2024-03-24T02:38:34.361 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8043 sec 2024-03-24T02:38:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7388.210333 secs 2024-03-24T02:38:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:34 smithi078 ceph-mon[32429]: pgmap v4379: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:34.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7388.210333 secs 2024-03-24T02:38:34.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:34 smithi107 ceph-mon[36485]: pgmap v4379: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:34.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7388.210333 secs 2024-03-24T02:38:34.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:34 smithi106 ceph-mon[36505]: pgmap v4379: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:35.361 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8044 sec 2024-03-24T02:38:36.361 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8045 sec 2024-03-24T02:38:36.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:36 smithi078 ceph-mon[32429]: pgmap v4380: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:36.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:36 smithi107 ceph-mon[36485]: pgmap v4380: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:36.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:36 smithi106 ceph-mon[36505]: pgmap v4380: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:37.361 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8046 sec 2024-03-24T02:38:38.361 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8047 sec 2024-03-24T02:38:38.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:38 smithi078 ceph-mon[32429]: pgmap v4381: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:38.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:38 smithi107 ceph-mon[36485]: pgmap v4381: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:38.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:38 smithi106 ceph-mon[36505]: pgmap v4381: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:39.361 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8048 sec 2024-03-24T02:38:39.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7393.210415 secs 2024-03-24T02:38:39.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7393.210415 secs 2024-03-24T02:38:39.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7393.210415 secs 2024-03-24T02:38:40.361 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8049 sec 2024-03-24T02:38:40.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:40 smithi078 ceph-mon[32429]: pgmap v4382: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:40.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:40 smithi107 ceph-mon[36485]: pgmap v4382: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:40.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:40 smithi106 ceph-mon[36505]: pgmap v4382: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:41.361 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8050 sec 2024-03-24T02:38:42.361 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8051 sec 2024-03-24T02:38:42.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:42 smithi078 ceph-mon[32429]: pgmap v4383: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:42.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:42 smithi107 ceph-mon[36485]: pgmap v4383: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:42.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:42 smithi106 ceph-mon[36505]: pgmap v4383: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:43.361 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8052 sec 2024-03-24T02:38:44.361 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8053 sec 2024-03-24T02:38:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7398.210500 secs 2024-03-24T02:38:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:44 smithi078 ceph-mon[32429]: pgmap v4384: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:44.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7398.210500 secs 2024-03-24T02:38:44.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:44 smithi107 ceph-mon[36485]: pgmap v4384: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:44.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7398.210500 secs 2024-03-24T02:38:44.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:44 smithi106 ceph-mon[36505]: pgmap v4384: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:45.362 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8054 sec 2024-03-24T02:38:46.362 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8055 sec 2024-03-24T02:38:46.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:46 smithi078 ceph-mon[32429]: pgmap v4385: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:46.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:46 smithi107 ceph-mon[36485]: pgmap v4385: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:46.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:46 smithi106 ceph-mon[36505]: pgmap v4385: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:47.362 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8056 sec 2024-03-24T02:38:48.362 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8057 sec 2024-03-24T02:38:48.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:48 smithi078 ceph-mon[32429]: pgmap v4386: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:48.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:48 smithi107 ceph-mon[36485]: pgmap v4386: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:48.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:48 smithi106 ceph-mon[36505]: pgmap v4386: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:49.362 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8058 sec 2024-03-24T02:38:49.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7403.210560 secs 2024-03-24T02:38:49.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7403.210560 secs 2024-03-24T02:38:49.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7403.210560 secs 2024-03-24T02:38:50.362 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8059 sec 2024-03-24T02:38:50.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:50 smithi078 ceph-mon[32429]: pgmap v4387: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:50.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:50 smithi107 ceph-mon[36485]: pgmap v4387: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:50.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:50 smithi106 ceph-mon[36505]: pgmap v4387: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:51.362 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8060 sec 2024-03-24T02:38:52.339 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:52 smithi078 ceph-mon[32429]: pgmap v4388: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:52.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:52 smithi107 ceph-mon[36485]: pgmap v4388: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:52.362 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8061 sec 2024-03-24T02:38:52.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:52 smithi106 ceph-mon[36505]: pgmap v4388: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:53.362 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8062 sec 2024-03-24T02:38:54.362 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8063 sec 2024-03-24T02:38:54.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7408.210645 secs 2024-03-24T02:38:54.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:54 smithi078 ceph-mon[32429]: pgmap v4389: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:54.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7408.210645 secs 2024-03-24T02:38:54.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:54 smithi107 ceph-mon[36485]: pgmap v4389: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:54.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7408.210645 secs 2024-03-24T02:38:54.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:54 smithi106 ceph-mon[36505]: pgmap v4389: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:55.362 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8064 sec 2024-03-24T02:38:56.362 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8065 sec 2024-03-24T02:38:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:56 smithi078 ceph-mon[32429]: pgmap v4390: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:56.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:56 smithi107 ceph-mon[36485]: pgmap v4390: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:56.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:56 smithi106 ceph-mon[36505]: pgmap v4390: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:57.363 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8066 sec 2024-03-24T02:38:58.363 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8067 sec 2024-03-24T02:38:58.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:58 smithi078 ceph-mon[32429]: pgmap v4391: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:58.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:58 smithi107 ceph-mon[36485]: pgmap v4391: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:58.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:58 smithi106 ceph-mon[36505]: pgmap v4391: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:38:59.363 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8068 sec 2024-03-24T02:38:59.802 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:38:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7413.210730 secs 2024-03-24T02:38:59.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:38:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7413.210730 secs 2024-03-24T02:38:59.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:38:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7413.210730 secs 2024-03-24T02:39:00.363 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8069 sec 2024-03-24T02:39:00.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:00 smithi078 ceph-mon[32429]: pgmap v4392: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:00.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:00 smithi107 ceph-mon[36485]: pgmap v4392: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:00.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:00 smithi106 ceph-mon[36505]: pgmap v4392: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:01.363 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8070 sec 2024-03-24T02:39:02.363 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8071 sec 2024-03-24T02:39:02.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:02 smithi078 ceph-mon[32429]: pgmap v4393: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:02.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:02 smithi107 ceph-mon[36485]: pgmap v4393: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:02.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:02 smithi106 ceph-mon[36505]: pgmap v4393: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:03.363 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8072 sec 2024-03-24T02:39:04.363 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8073 sec 2024-03-24T02:39:04.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7418.210791 secs 2024-03-24T02:39:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:04 smithi078 ceph-mon[32429]: pgmap v4394: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:04.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7418.210791 secs 2024-03-24T02:39:04.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:04 smithi107 ceph-mon[36485]: pgmap v4394: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:04.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7418.210791 secs 2024-03-24T02:39:04.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:04 smithi106 ceph-mon[36505]: pgmap v4394: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:05.363 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8074 sec 2024-03-24T02:39:06.363 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8075 sec 2024-03-24T02:39:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:06 smithi078 ceph-mon[32429]: pgmap v4395: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:06.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:06 smithi107 ceph-mon[36485]: pgmap v4395: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:06.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:06 smithi106 ceph-mon[36505]: pgmap v4395: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:07.363 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8076 sec 2024-03-24T02:39:08.363 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8077 sec 2024-03-24T02:39:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:08 smithi078 ceph-mon[32429]: pgmap v4396: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:08.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:08 smithi107 ceph-mon[36485]: pgmap v4396: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:08.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:08 smithi106 ceph-mon[36505]: pgmap v4396: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:09.364 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8078 sec 2024-03-24T02:39:09.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7423.210804 secs 2024-03-24T02:39:09.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7423.210804 secs 2024-03-24T02:39:09.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7423.210804 secs 2024-03-24T02:39:10.363 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8079 sec 2024-03-24T02:39:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:10 smithi078 ceph-mon[32429]: pgmap v4397: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:39:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:39:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:39:10.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:39:10.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:10 smithi107 ceph-mon[36485]: pgmap v4397: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:10.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:39:10.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:39:10.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:39:10.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:39:10.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:10 smithi106 ceph-mon[36505]: pgmap v4397: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:39:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:39:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:39:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:39:11.364 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8080 sec 2024-03-24T02:39:11.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:11 smithi078 ceph-mon[32429]: pgmap v4398: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:39:11.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:11 smithi078 ceph-mon[32429]: pgmap v4399: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:39:11.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:11 smithi107 ceph-mon[36485]: pgmap v4398: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:39:11.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:11 smithi107 ceph-mon[36485]: pgmap v4399: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:39:11.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:11 smithi106 ceph-mon[36505]: pgmap v4398: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:39:11.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:11 smithi106 ceph-mon[36505]: pgmap v4399: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:39:12.364 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8081 sec 2024-03-24T02:39:13.364 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8082 sec 2024-03-24T02:39:13.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:13 smithi078 ceph-mon[32429]: pgmap v4400: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:39:13.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:13 smithi107 ceph-mon[36485]: pgmap v4400: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:39:13.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:13 smithi106 ceph-mon[36505]: pgmap v4400: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:39:14.364 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8083 sec 2024-03-24T02:39:14.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7428.210888 secs 2024-03-24T02:39:14.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7428.210888 secs 2024-03-24T02:39:14.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7428.210888 secs 2024-03-24T02:39:15.364 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8084 sec 2024-03-24T02:39:15.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:15 smithi078 ceph-mon[32429]: pgmap v4401: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:39:15.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:15 smithi107 ceph-mon[36485]: pgmap v4401: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:39:15.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:15 smithi106 ceph-mon[36505]: pgmap v4401: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:39:16.364 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8085 sec 2024-03-24T02:39:17.364 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8086 sec 2024-03-24T02:39:17.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:17 smithi078 ceph-mon[32429]: pgmap v4402: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:39:17.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:17 smithi107 ceph-mon[36485]: pgmap v4402: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:39:17.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:17 smithi106 ceph-mon[36505]: pgmap v4402: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:39:18.364 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8087 sec 2024-03-24T02:39:19.365 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8088 sec 2024-03-24T02:39:19.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7433.210966 secs 2024-03-24T02:39:19.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:19 smithi078 ceph-mon[32429]: pgmap v4403: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:39:19.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7433.210966 secs 2024-03-24T02:39:19.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:19 smithi107 ceph-mon[36485]: pgmap v4403: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:39:19.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7433.210966 secs 2024-03-24T02:39:19.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:19 smithi106 ceph-mon[36505]: pgmap v4403: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:39:20.364 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8089 sec 2024-03-24T02:39:21.364 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8090 sec 2024-03-24T02:39:21.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:21 smithi078 ceph-mon[32429]: pgmap v4404: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:39:21.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:21 smithi107 ceph-mon[36485]: pgmap v4404: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:39:21.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:21 smithi106 ceph-mon[36505]: pgmap v4404: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:39:22.364 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8091 sec 2024-03-24T02:39:23.365 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8092 sec 2024-03-24T02:39:23.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:23 smithi078 ceph-mon[32429]: pgmap v4405: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:23.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:23 smithi107 ceph-mon[36485]: pgmap v4405: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:23.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:23 smithi106 ceph-mon[36505]: pgmap v4405: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:24.365 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8093 sec 2024-03-24T02:39:24.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7438.211052 secs 2024-03-24T02:39:24.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7438.211052 secs 2024-03-24T02:39:24.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7438.211052 secs 2024-03-24T02:39:25.365 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8094 sec 2024-03-24T02:39:25.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:25 smithi078 ceph-mon[32429]: pgmap v4406: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:25.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:25 smithi107 ceph-mon[36485]: pgmap v4406: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:25.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:25 smithi106 ceph-mon[36505]: pgmap v4406: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:26.365 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8095 sec 2024-03-24T02:39:27.365 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8096 sec 2024-03-24T02:39:27.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:27 smithi078 ceph-mon[32429]: pgmap v4407: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:27.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:27 smithi107 ceph-mon[36485]: pgmap v4407: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:27.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:27 smithi106 ceph-mon[36505]: pgmap v4407: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:28.365 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8097 sec 2024-03-24T02:39:29.365 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8098 sec 2024-03-24T02:39:29.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7443.211142 secs 2024-03-24T02:39:29.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:29 smithi078 ceph-mon[32429]: pgmap v4408: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:29.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7443.211142 secs 2024-03-24T02:39:29.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:29 smithi107 ceph-mon[36485]: pgmap v4408: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:29.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7443.211142 secs 2024-03-24T02:39:29.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:29 smithi106 ceph-mon[36505]: pgmap v4408: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:30.365 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8099 sec 2024-03-24T02:39:31.365 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8100 sec 2024-03-24T02:39:31.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:31 smithi078 ceph-mon[32429]: pgmap v4409: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:31.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:31 smithi107 ceph-mon[36485]: pgmap v4409: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:31.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:31 smithi106 ceph-mon[36505]: pgmap v4409: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:32.365 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8101 sec 2024-03-24T02:39:33.366 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8102 sec 2024-03-24T02:39:33.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:33 smithi078 ceph-mon[32429]: pgmap v4410: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:33.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:33 smithi107 ceph-mon[36485]: pgmap v4410: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:33.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:33 smithi106 ceph-mon[36505]: pgmap v4410: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:34.366 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8103 sec 2024-03-24T02:39:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7448.211230 secs 2024-03-24T02:39:34.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7448.211230 secs 2024-03-24T02:39:34.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7448.211230 secs 2024-03-24T02:39:35.366 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8104 sec 2024-03-24T02:39:35.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:35 smithi078 ceph-mon[32429]: pgmap v4411: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:35.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:35 smithi107 ceph-mon[36485]: pgmap v4411: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:35.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:35 smithi106 ceph-mon[36505]: pgmap v4411: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:36.366 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8105 sec 2024-03-24T02:39:37.366 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8106 sec 2024-03-24T02:39:37.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:37 smithi078 ceph-mon[32429]: pgmap v4412: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:37.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:37 smithi107 ceph-mon[36485]: pgmap v4412: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:37.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:37 smithi106 ceph-mon[36505]: pgmap v4412: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:38.366 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8107 sec 2024-03-24T02:39:39.366 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8108 sec 2024-03-24T02:39:39.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7453.211322 secs 2024-03-24T02:39:39.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:39 smithi078 ceph-mon[32429]: pgmap v4413: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:39.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7453.211322 secs 2024-03-24T02:39:39.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:39 smithi107 ceph-mon[36485]: pgmap v4413: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:39.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7453.211322 secs 2024-03-24T02:39:39.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:39 smithi106 ceph-mon[36505]: pgmap v4413: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:40.367 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8109 sec 2024-03-24T02:39:41.366 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8110 sec 2024-03-24T02:39:41.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:41 smithi078 ceph-mon[32429]: pgmap v4414: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:41.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:41 smithi107 ceph-mon[36485]: pgmap v4414: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:41.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:41 smithi106 ceph-mon[36505]: pgmap v4414: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:42.366 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8111 sec 2024-03-24T02:39:43.366 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8112 sec 2024-03-24T02:39:43.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:43 smithi078 ceph-mon[32429]: pgmap v4415: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:43.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:43 smithi107 ceph-mon[36485]: pgmap v4415: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:43.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:43 smithi106 ceph-mon[36505]: pgmap v4415: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:44.367 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8113 sec 2024-03-24T02:39:44.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7458.211404 secs 2024-03-24T02:39:44.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7458.211404 secs 2024-03-24T02:39:44.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7458.211404 secs 2024-03-24T02:39:45.367 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8114 sec 2024-03-24T02:39:45.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:45 smithi078 ceph-mon[32429]: pgmap v4416: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:45.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:45 smithi107 ceph-mon[36485]: pgmap v4416: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:45.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:45 smithi106 ceph-mon[36505]: pgmap v4416: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:46.367 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8115 sec 2024-03-24T02:39:47.367 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8116 sec 2024-03-24T02:39:47.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:47 smithi078 ceph-mon[32429]: pgmap v4417: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:47.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:47 smithi107 ceph-mon[36485]: pgmap v4417: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:47.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:47 smithi106 ceph-mon[36505]: pgmap v4417: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:48.367 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8117 sec 2024-03-24T02:39:49.367 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8118 sec 2024-03-24T02:39:49.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7463.211467 secs 2024-03-24T02:39:49.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:49 smithi078 ceph-mon[32429]: pgmap v4418: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:49.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7463.211467 secs 2024-03-24T02:39:49.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:49 smithi107 ceph-mon[36485]: pgmap v4418: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:49.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7463.211467 secs 2024-03-24T02:39:49.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:49 smithi106 ceph-mon[36505]: pgmap v4418: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:50.367 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8119 sec 2024-03-24T02:39:51.367 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8120 sec 2024-03-24T02:39:51.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:51 smithi078 ceph-mon[32429]: pgmap v4419: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:51.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:51 smithi107 ceph-mon[36485]: pgmap v4419: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:51.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:51 smithi106 ceph-mon[36505]: pgmap v4419: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:52.367 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8121 sec 2024-03-24T02:39:53.367 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8122 sec 2024-03-24T02:39:53.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:53 smithi078 ceph-mon[32429]: pgmap v4420: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:53.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:53 smithi107 ceph-mon[36485]: pgmap v4420: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:53.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:53 smithi106 ceph-mon[36505]: pgmap v4420: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:54.368 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8123 sec 2024-03-24T02:39:54.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7468.211524 secs 2024-03-24T02:39:54.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7468.211524 secs 2024-03-24T02:39:54.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7468.211524 secs 2024-03-24T02:39:55.368 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8124 sec 2024-03-24T02:39:55.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:55 smithi078 ceph-mon[32429]: pgmap v4421: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:55.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:55 smithi107 ceph-mon[36485]: pgmap v4421: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:55.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:55 smithi106 ceph-mon[36505]: pgmap v4421: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:56.368 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8125 sec 2024-03-24T02:39:57.368 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8126 sec 2024-03-24T02:39:57.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:57 smithi078 ceph-mon[32429]: pgmap v4422: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:57.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:57 smithi107 ceph-mon[36485]: pgmap v4422: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:57.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:57 smithi106 ceph-mon[36505]: pgmap v4422: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:58.368 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8127 sec 2024-03-24T02:39:59.368 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8128 sec 2024-03-24T02:39:59.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7473.211583 secs 2024-03-24T02:39:59.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:39:59 smithi078 ceph-mon[32429]: pgmap v4423: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:59.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7473.211583 secs 2024-03-24T02:39:59.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:39:59 smithi107 ceph-mon[36485]: pgmap v4423: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:39:59.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7473.211583 secs 2024-03-24T02:39:59.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:39:59 smithi106 ceph-mon[36505]: pgmap v4423: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:00.368 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8129 sec 2024-03-24T02:40:00.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:00 smithi078 ceph-mon[32429]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:40:00.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:00 smithi107 ceph-mon[36485]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:40:00.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:00 smithi106 ceph-mon[36505]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:40:01.368 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8130 sec 2024-03-24T02:40:01.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:01 smithi078 ceph-mon[32429]: pgmap v4424: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:01.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:01 smithi107 ceph-mon[36485]: pgmap v4424: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:01.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:01 smithi106 ceph-mon[36505]: pgmap v4424: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:02.368 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8131 sec 2024-03-24T02:40:03.368 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8132 sec 2024-03-24T02:40:03.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:03 smithi078 ceph-mon[32429]: pgmap v4425: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:03.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:03 smithi107 ceph-mon[36485]: pgmap v4425: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:03.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:03 smithi106 ceph-mon[36505]: pgmap v4425: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:04.368 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8133 sec 2024-03-24T02:40:04.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7478.211667 secs 2024-03-24T02:40:04.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7478.211667 secs 2024-03-24T02:40:04.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7478.211667 secs 2024-03-24T02:40:05.368 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8134 sec 2024-03-24T02:40:05.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:05 smithi078 ceph-mon[32429]: pgmap v4426: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:05.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:05 smithi107 ceph-mon[36485]: pgmap v4426: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:05.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:05 smithi106 ceph-mon[36505]: pgmap v4426: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:06.368 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8135 sec 2024-03-24T02:40:07.368 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8136 sec 2024-03-24T02:40:07.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:07 smithi078 ceph-mon[32429]: pgmap v4427: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:07.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:07 smithi107 ceph-mon[36485]: pgmap v4427: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:07.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:07 smithi106 ceph-mon[36505]: pgmap v4427: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:08.369 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8137 sec 2024-03-24T02:40:09.369 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8138 sec 2024-03-24T02:40:09.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7483.211728 secs 2024-03-24T02:40:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:09 smithi078 ceph-mon[32429]: pgmap v4428: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:09.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7483.211728 secs 2024-03-24T02:40:09.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:09 smithi107 ceph-mon[36485]: pgmap v4428: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:09.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7483.211728 secs 2024-03-24T02:40:09.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:09 smithi106 ceph-mon[36505]: pgmap v4428: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:10.369 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8139 sec 2024-03-24T02:40:10.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:40:10.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:40:10.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:40:11.369 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8140 sec 2024-03-24T02:40:11.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:11 smithi106 ceph-mon[36505]: pgmap v4429: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:11.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:11 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:40:11.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:11 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:40:11.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:11 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:40:12.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:11 smithi078 ceph-mon[32429]: pgmap v4429: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:12.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:11 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:40:12.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:11 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:40:12.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:11 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:40:12.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:11 smithi107 ceph-mon[36485]: pgmap v4429: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:12.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:11 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:40:12.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:11 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:40:12.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:11 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:40:12.369 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8141 sec 2024-03-24T02:40:12.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:12 smithi106 ceph-mon[36505]: pgmap v4430: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:40:12.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:12 smithi106 ceph-mon[36505]: pgmap v4431: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:40:13.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:12 smithi078 ceph-mon[32429]: pgmap v4430: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:40:13.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:12 smithi078 ceph-mon[32429]: pgmap v4431: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:40:13.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:12 smithi107 ceph-mon[36485]: pgmap v4430: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:40:13.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:12 smithi107 ceph-mon[36485]: pgmap v4431: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:40:13.369 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8142 sec 2024-03-24T02:40:14.369 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8143 sec 2024-03-24T02:40:14.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:14 smithi106 ceph-mon[36505]: pgmap v4432: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:40:14.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7488.211817 secs 2024-03-24T02:40:15.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:14 smithi078 ceph-mon[32429]: pgmap v4432: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:40:15.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7488.211817 secs 2024-03-24T02:40:15.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:14 smithi107 ceph-mon[36485]: pgmap v4432: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:40:15.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7488.211817 secs 2024-03-24T02:40:15.369 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8144 sec 2024-03-24T02:40:16.369 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8145 sec 2024-03-24T02:40:16.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:16 smithi078 ceph-mon[32429]: pgmap v4433: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:40:16.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:16 smithi106 ceph-mon[36505]: pgmap v4433: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:40:17.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:16 smithi107 ceph-mon[36485]: pgmap v4433: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:40:17.369 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8146 sec 2024-03-24T02:40:18.369 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8147 sec 2024-03-24T02:40:18.895 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:18 smithi107 ceph-mon[36485]: pgmap v4434: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:40:18.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:18 smithi106 ceph-mon[36505]: pgmap v4434: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:40:19.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:18 smithi078 ceph-mon[32429]: pgmap v4434: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:40:19.369 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8148 sec 2024-03-24T02:40:19.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7493.211905 secs 2024-03-24T02:40:20.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7493.211905 secs 2024-03-24T02:40:20.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7493.211905 secs 2024-03-24T02:40:20.370 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8149 sec 2024-03-24T02:40:20.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:20 smithi106 ceph-mon[36505]: pgmap v4435: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:40:21.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:20 smithi078 ceph-mon[32429]: pgmap v4435: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:40:21.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:20 smithi107 ceph-mon[36485]: pgmap v4435: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:40:21.369 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8150 sec 2024-03-24T02:40:22.370 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8151 sec 2024-03-24T02:40:22.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:22 smithi106 ceph-mon[36505]: pgmap v4436: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:40:23.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:22 smithi078 ceph-mon[32429]: pgmap v4436: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:40:23.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:22 smithi107 ceph-mon[36485]: pgmap v4436: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:40:23.370 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8152 sec 2024-03-24T02:40:24.370 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8153 sec 2024-03-24T02:40:24.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:24 smithi106 ceph-mon[36505]: pgmap v4437: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:24.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7498.211993 secs 2024-03-24T02:40:24.958 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:24 smithi107 ceph-mon[36485]: pgmap v4437: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:24.958 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7498.211993 secs 2024-03-24T02:40:25.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:24 smithi078 ceph-mon[32429]: pgmap v4437: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:25.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7498.211993 secs 2024-03-24T02:40:25.370 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8154 sec 2024-03-24T02:40:26.370 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8155 sec 2024-03-24T02:40:26.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:26 smithi106 ceph-mon[36505]: pgmap v4438: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:27.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:26 smithi078 ceph-mon[32429]: pgmap v4438: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:27.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:26 smithi107 ceph-mon[36485]: pgmap v4438: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:27.370 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8156 sec 2024-03-24T02:40:28.370 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8157 sec 2024-03-24T02:40:28.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:28 smithi078 ceph-mon[32429]: pgmap v4439: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:28.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:28 smithi106 ceph-mon[36505]: pgmap v4439: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:29.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:28 smithi107 ceph-mon[36485]: pgmap v4439: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:29.370 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8158 sec 2024-03-24T02:40:29.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7503.212071 secs 2024-03-24T02:40:30.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7503.212071 secs 2024-03-24T02:40:30.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7503.212071 secs 2024-03-24T02:40:30.370 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8159 sec 2024-03-24T02:40:30.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:30 smithi106 ceph-mon[36505]: pgmap v4440: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:31.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:30 smithi078 ceph-mon[32429]: pgmap v4440: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:31.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:30 smithi107 ceph-mon[36485]: pgmap v4440: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:31.371 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8160 sec 2024-03-24T02:40:32.371 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8161 sec 2024-03-24T02:40:32.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:32 smithi106 ceph-mon[36505]: pgmap v4441: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:33.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:32 smithi078 ceph-mon[32429]: pgmap v4441: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:33.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:32 smithi107 ceph-mon[36485]: pgmap v4441: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:33.371 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8162 sec 2024-03-24T02:40:34.371 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8163 sec 2024-03-24T02:40:34.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:34 smithi106 ceph-mon[36505]: pgmap v4442: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:34.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7508.212147 secs 2024-03-24T02:40:35.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:34 smithi078 ceph-mon[32429]: pgmap v4442: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:35.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7508.212147 secs 2024-03-24T02:40:35.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:34 smithi107 ceph-mon[36485]: pgmap v4442: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:35.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7508.212147 secs 2024-03-24T02:40:35.371 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8164 sec 2024-03-24T02:40:36.371 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8165 sec 2024-03-24T02:40:36.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:36 smithi106 ceph-mon[36505]: pgmap v4443: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:37.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:36 smithi078 ceph-mon[32429]: pgmap v4443: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:37.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:36 smithi107 ceph-mon[36485]: pgmap v4443: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:37.371 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8166 sec 2024-03-24T02:40:38.371 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8167 sec 2024-03-24T02:40:38.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:38 smithi106 ceph-mon[36505]: pgmap v4444: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:39.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:38 smithi078 ceph-mon[32429]: pgmap v4444: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:39.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:38 smithi107 ceph-mon[36485]: pgmap v4444: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:39.371 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8168 sec 2024-03-24T02:40:40.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7513.212228 secs 2024-03-24T02:40:40.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7513.212228 secs 2024-03-24T02:40:40.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7513.212228 secs 2024-03-24T02:40:40.371 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8169 sec 2024-03-24T02:40:41.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:40 smithi078 ceph-mon[32429]: pgmap v4445: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:41.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:40 smithi107 ceph-mon[36485]: pgmap v4445: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:41.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:40 smithi106 ceph-mon[36505]: pgmap v4445: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:41.372 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8170 sec 2024-03-24T02:40:42.372 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8171 sec 2024-03-24T02:40:43.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:42 smithi078 ceph-mon[32429]: pgmap v4446: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:43.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:42 smithi107 ceph-mon[36485]: pgmap v4446: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:43.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:42 smithi106 ceph-mon[36505]: pgmap v4446: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:43.372 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8172 sec 2024-03-24T02:40:44.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7518.212313 secs 2024-03-24T02:40:44.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7518.212313 secs 2024-03-24T02:40:44.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7518.212313 secs 2024-03-24T02:40:44.372 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8173 sec 2024-03-24T02:40:45.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:44 smithi078 ceph-mon[32429]: pgmap v4447: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:45.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:44 smithi107 ceph-mon[36485]: pgmap v4447: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:45.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:44 smithi106 ceph-mon[36505]: pgmap v4447: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:45.373 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8174 sec 2024-03-24T02:40:46.372 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8175 sec 2024-03-24T02:40:47.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:46 smithi078 ceph-mon[32429]: pgmap v4448: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:47.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:46 smithi107 ceph-mon[36485]: pgmap v4448: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:47.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:46 smithi106 ceph-mon[36505]: pgmap v4448: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:47.372 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8176 sec 2024-03-24T02:40:48.372 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8177 sec 2024-03-24T02:40:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:48 smithi078 ceph-mon[32429]: pgmap v4449: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7523.212397 secs 2024-03-24T02:40:49.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:48 smithi107 ceph-mon[36485]: pgmap v4449: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:49.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7523.212397 secs 2024-03-24T02:40:49.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:48 smithi106 ceph-mon[36505]: pgmap v4449: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:49.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7523.212397 secs 2024-03-24T02:40:49.372 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8178 sec 2024-03-24T02:40:50.372 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8179 sec 2024-03-24T02:40:51.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:50 smithi078 ceph-mon[32429]: pgmap v4450: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:51.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:50 smithi107 ceph-mon[36485]: pgmap v4450: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:51.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:50 smithi106 ceph-mon[36505]: pgmap v4450: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:51.372 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8180 sec 2024-03-24T02:40:52.373 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8181 sec 2024-03-24T02:40:53.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:52 smithi078 ceph-mon[32429]: pgmap v4451: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:53.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:52 smithi107 ceph-mon[36485]: pgmap v4451: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:53.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:52 smithi106 ceph-mon[36505]: pgmap v4451: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:53.373 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8182 sec 2024-03-24T02:40:54.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7528.212483 secs 2024-03-24T02:40:54.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7528.212483 secs 2024-03-24T02:40:54.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7528.212483 secs 2024-03-24T02:40:54.373 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8183 sec 2024-03-24T02:40:55.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:54 smithi078 ceph-mon[32429]: pgmap v4452: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:55.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:54 smithi107 ceph-mon[36485]: pgmap v4452: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:55.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:54 smithi106 ceph-mon[36505]: pgmap v4452: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:55.373 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8184 sec 2024-03-24T02:40:56.373 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8185 sec 2024-03-24T02:40:57.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:56 smithi078 ceph-mon[32429]: pgmap v4453: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:57.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:56 smithi107 ceph-mon[36485]: pgmap v4453: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:57.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:56 smithi106 ceph-mon[36505]: pgmap v4453: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:40:57.373 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8186 sec 2024-03-24T02:40:58.373 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8187 sec 2024-03-24T02:40:59.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:58 smithi078 ceph-mon[32429]: pgmap v4454: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:59.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:40:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7533.212543 secs 2024-03-24T02:40:59.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:58 smithi107 ceph-mon[36485]: pgmap v4454: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:59.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:40:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7533.212543 secs 2024-03-24T02:40:59.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:58 smithi106 ceph-mon[36505]: pgmap v4454: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:59.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:40:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7533.212543 secs 2024-03-24T02:40:59.373 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8188 sec 2024-03-24T02:41:00.373 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8189 sec 2024-03-24T02:41:00.953 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:00 smithi106 ceph-mon[36505]: pgmap v4455: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:01.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:00 smithi078 ceph-mon[32429]: pgmap v4455: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:01.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:00 smithi107 ceph-mon[36485]: pgmap v4455: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:01.374 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8190 sec 2024-03-24T02:41:02.374 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8191 sec 2024-03-24T02:41:03.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:02 smithi078 ceph-mon[32429]: pgmap v4456: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:03.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:02 smithi107 ceph-mon[36485]: pgmap v4456: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:03.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:02 smithi106 ceph-mon[36505]: pgmap v4456: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:03.374 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8192 sec 2024-03-24T02:41:03.991 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7538.212602 secs 2024-03-24T02:41:04.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7538.212602 secs 2024-03-24T02:41:04.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7538.212602 secs 2024-03-24T02:41:04.374 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8193 sec 2024-03-24T02:41:05.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:04 smithi078 ceph-mon[32429]: pgmap v4457: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:05.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:04 smithi107 ceph-mon[36485]: pgmap v4457: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:05.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:04 smithi106 ceph-mon[36505]: pgmap v4457: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:05.374 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8194 sec 2024-03-24T02:41:06.374 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8195 sec 2024-03-24T02:41:07.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:06 smithi078 ceph-mon[32429]: pgmap v4458: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:07.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:06 smithi107 ceph-mon[36485]: pgmap v4458: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:07.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:06 smithi106 ceph-mon[36505]: pgmap v4458: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:07.374 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8196 sec 2024-03-24T02:41:08.374 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8197 sec 2024-03-24T02:41:09.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:08 smithi078 ceph-mon[32429]: pgmap v4459: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7543.212663 secs 2024-03-24T02:41:09.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:08 smithi107 ceph-mon[36485]: pgmap v4459: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:09.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7543.212663 secs 2024-03-24T02:41:09.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:08 smithi106 ceph-mon[36505]: pgmap v4459: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:09.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7543.212663 secs 2024-03-24T02:41:09.374 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8198 sec 2024-03-24T02:41:10.374 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8199 sec 2024-03-24T02:41:11.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:10 smithi078 ceph-mon[32429]: pgmap v4460: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:11.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:10 smithi107 ceph-mon[36485]: pgmap v4460: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:11.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:10 smithi106 ceph-mon[36505]: pgmap v4460: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:11.374 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8200 sec 2024-03-24T02:41:12.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:11 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:41:12.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:11 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:41:12.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:11 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:41:12.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:11 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:41:12.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:11 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:41:12.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:11 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:41:12.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:11 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:41:12.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:11 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:41:12.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:11 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:41:12.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:11 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:41:12.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:11 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:41:12.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:11 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:41:12.374 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8201 sec 2024-03-24T02:41:13.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:12 smithi078 ceph-mon[32429]: pgmap v4461: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:13.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:12 smithi078 ceph-mon[32429]: pgmap v4462: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:41:13.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:12 smithi078 ceph-mon[32429]: pgmap v4463: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:41:13.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:12 smithi107 ceph-mon[36485]: pgmap v4461: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:13.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:12 smithi107 ceph-mon[36485]: pgmap v4462: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:41:13.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:12 smithi107 ceph-mon[36485]: pgmap v4463: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:41:13.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:12 smithi106 ceph-mon[36505]: pgmap v4461: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:13.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:12 smithi106 ceph-mon[36505]: pgmap v4462: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:41:13.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:12 smithi106 ceph-mon[36505]: pgmap v4463: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:41:13.375 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8202 sec 2024-03-24T02:41:14.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7548.212723 secs 2024-03-24T02:41:14.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7548.212723 secs 2024-03-24T02:41:14.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7548.212723 secs 2024-03-24T02:41:14.374 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8203 sec 2024-03-24T02:41:15.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:14 smithi078 ceph-mon[32429]: pgmap v4464: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:41:15.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:14 smithi107 ceph-mon[36485]: pgmap v4464: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:41:15.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:14 smithi106 ceph-mon[36505]: pgmap v4464: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:41:15.375 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8204 sec 2024-03-24T02:41:16.375 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8205 sec 2024-03-24T02:41:17.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:16 smithi078 ceph-mon[32429]: pgmap v4465: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:41:17.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:16 smithi107 ceph-mon[36485]: pgmap v4465: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:41:17.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:16 smithi106 ceph-mon[36505]: pgmap v4465: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:41:17.375 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8206 sec 2024-03-24T02:41:18.375 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8207 sec 2024-03-24T02:41:19.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:18 smithi078 ceph-mon[32429]: pgmap v4466: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:41:19.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7553.212807 secs 2024-03-24T02:41:19.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:18 smithi107 ceph-mon[36485]: pgmap v4466: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:41:19.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7553.212807 secs 2024-03-24T02:41:19.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:18 smithi106 ceph-mon[36505]: pgmap v4466: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:41:19.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7553.212807 secs 2024-03-24T02:41:19.375 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8208 sec 2024-03-24T02:41:20.375 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8209 sec 2024-03-24T02:41:21.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:20 smithi078 ceph-mon[32429]: pgmap v4467: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:41:21.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:20 smithi107 ceph-mon[36485]: pgmap v4467: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:41:21.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:20 smithi106 ceph-mon[36505]: pgmap v4467: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:41:21.375 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8210 sec 2024-03-24T02:41:22.376 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8211 sec 2024-03-24T02:41:23.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:22 smithi078 ceph-mon[32429]: pgmap v4468: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:41:23.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:22 smithi107 ceph-mon[36485]: pgmap v4468: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:41:23.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:22 smithi106 ceph-mon[36505]: pgmap v4468: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:41:23.375 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8212 sec 2024-03-24T02:41:24.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7558.212895 secs 2024-03-24T02:41:24.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7558.212895 secs 2024-03-24T02:41:24.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7558.212895 secs 2024-03-24T02:41:24.375 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8213 sec 2024-03-24T02:41:25.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:24 smithi078 ceph-mon[32429]: pgmap v4469: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:25.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:24 smithi107 ceph-mon[36485]: pgmap v4469: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:25.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:24 smithi106 ceph-mon[36505]: pgmap v4469: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:25.376 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8214 sec 2024-03-24T02:41:26.376 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8215 sec 2024-03-24T02:41:27.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:26 smithi078 ceph-mon[32429]: pgmap v4470: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:27.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:26 smithi107 ceph-mon[36485]: pgmap v4470: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:27.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:26 smithi106 ceph-mon[36505]: pgmap v4470: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:27.376 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8216 sec 2024-03-24T02:41:28.376 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8217 sec 2024-03-24T02:41:29.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:28 smithi078 ceph-mon[32429]: pgmap v4471: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:29.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7563.212952 secs 2024-03-24T02:41:29.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:28 smithi107 ceph-mon[36485]: pgmap v4471: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:29.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7563.212952 secs 2024-03-24T02:41:29.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:28 smithi106 ceph-mon[36505]: pgmap v4471: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:29.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7563.212952 secs 2024-03-24T02:41:29.376 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8218 sec 2024-03-24T02:41:30.376 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8219 sec 2024-03-24T02:41:31.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:30 smithi078 ceph-mon[32429]: pgmap v4472: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:31.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:30 smithi107 ceph-mon[36485]: pgmap v4472: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:31.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:30 smithi106 ceph-mon[36505]: pgmap v4472: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:31.376 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8220 sec 2024-03-24T02:41:32.376 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8221 sec 2024-03-24T02:41:33.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:32 smithi078 ceph-mon[32429]: pgmap v4473: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:33.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:32 smithi107 ceph-mon[36485]: pgmap v4473: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:33.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:32 smithi106 ceph-mon[36505]: pgmap v4473: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:33.376 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8222 sec 2024-03-24T02:41:34.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7568.213044 secs 2024-03-24T02:41:34.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7568.213044 secs 2024-03-24T02:41:34.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7568.213044 secs 2024-03-24T02:41:34.376 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8223 sec 2024-03-24T02:41:35.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:34 smithi078 ceph-mon[32429]: pgmap v4474: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:35.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:34 smithi107 ceph-mon[36485]: pgmap v4474: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:35.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:34 smithi106 ceph-mon[36505]: pgmap v4474: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:35.376 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8224 sec 2024-03-24T02:41:36.377 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8225 sec 2024-03-24T02:41:37.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:36 smithi078 ceph-mon[32429]: pgmap v4475: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:37.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:36 smithi107 ceph-mon[36485]: pgmap v4475: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:37.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:36 smithi106 ceph-mon[36505]: pgmap v4475: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:37.377 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8226 sec 2024-03-24T02:41:38.377 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8227 sec 2024-03-24T02:41:39.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:38 smithi078 ceph-mon[32429]: pgmap v4476: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7573.213131 secs 2024-03-24T02:41:39.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:38 smithi107 ceph-mon[36485]: pgmap v4476: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:39.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7573.213131 secs 2024-03-24T02:41:39.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:38 smithi106 ceph-mon[36505]: pgmap v4476: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:39.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7573.213131 secs 2024-03-24T02:41:39.377 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8228 sec 2024-03-24T02:41:40.377 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8229 sec 2024-03-24T02:41:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:40 smithi078 ceph-mon[32429]: pgmap v4477: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:41.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:40 smithi107 ceph-mon[36485]: pgmap v4477: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:41.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:40 smithi106 ceph-mon[36505]: pgmap v4477: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:41.377 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8230 sec 2024-03-24T02:41:42.377 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8231 sec 2024-03-24T02:41:43.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:42 smithi078 ceph-mon[32429]: pgmap v4478: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:43.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:42 smithi107 ceph-mon[36485]: pgmap v4478: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:43.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:42 smithi106 ceph-mon[36505]: pgmap v4478: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:43.377 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8232 sec 2024-03-24T02:41:44.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7578.213216 secs 2024-03-24T02:41:44.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7578.213216 secs 2024-03-24T02:41:44.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7578.213216 secs 2024-03-24T02:41:44.377 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8233 sec 2024-03-24T02:41:45.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:44 smithi078 ceph-mon[32429]: pgmap v4479: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:45.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:44 smithi107 ceph-mon[36485]: pgmap v4479: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:45.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:44 smithi106 ceph-mon[36505]: pgmap v4479: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:45.377 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8234 sec 2024-03-24T02:41:46.378 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8235 sec 2024-03-24T02:41:47.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:46 smithi078 ceph-mon[32429]: pgmap v4480: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:47.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:46 smithi107 ceph-mon[36485]: pgmap v4480: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:47.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:46 smithi106 ceph-mon[36505]: pgmap v4480: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:47.378 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8236 sec 2024-03-24T02:41:48.378 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8237 sec 2024-03-24T02:41:49.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:48 smithi078 ceph-mon[32429]: pgmap v4481: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7583.213306 secs 2024-03-24T02:41:49.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:48 smithi107 ceph-mon[36485]: pgmap v4481: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:49.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7583.213306 secs 2024-03-24T02:41:49.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:48 smithi106 ceph-mon[36505]: pgmap v4481: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:49.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7583.213306 secs 2024-03-24T02:41:49.378 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8238 sec 2024-03-24T02:41:50.378 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8239 sec 2024-03-24T02:41:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:50 smithi078 ceph-mon[32429]: pgmap v4482: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:51.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:50 smithi107 ceph-mon[36485]: pgmap v4482: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:51.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:50 smithi106 ceph-mon[36505]: pgmap v4482: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:51.378 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8240 sec 2024-03-24T02:41:52.378 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8241 sec 2024-03-24T02:41:53.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:52 smithi078 ceph-mon[32429]: pgmap v4483: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:53.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:52 smithi107 ceph-mon[36485]: pgmap v4483: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:53.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:52 smithi106 ceph-mon[36505]: pgmap v4483: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:53.378 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8242 sec 2024-03-24T02:41:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7588.213365 secs 2024-03-24T02:41:54.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7588.213365 secs 2024-03-24T02:41:54.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7588.213365 secs 2024-03-24T02:41:54.378 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8243 sec 2024-03-24T02:41:55.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:54 smithi078 ceph-mon[32429]: pgmap v4484: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:55.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:54 smithi107 ceph-mon[36485]: pgmap v4484: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:55.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:54 smithi106 ceph-mon[36505]: pgmap v4484: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:55.378 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8244 sec 2024-03-24T02:41:56.378 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8245 sec 2024-03-24T02:41:57.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:56 smithi078 ceph-mon[32429]: pgmap v4485: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:57.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:56 smithi107 ceph-mon[36485]: pgmap v4485: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:57.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:56 smithi106 ceph-mon[36505]: pgmap v4485: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:57.378 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8246 sec 2024-03-24T02:41:58.378 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8247 sec 2024-03-24T02:41:59.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:58 smithi078 ceph-mon[32429]: pgmap v4486: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:41:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7593.213426 secs 2024-03-24T02:41:59.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:58 smithi107 ceph-mon[36485]: pgmap v4486: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:59.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:41:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7593.213426 secs 2024-03-24T02:41:59.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:58 smithi106 ceph-mon[36505]: pgmap v4486: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:41:59.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:41:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7593.213426 secs 2024-03-24T02:41:59.378 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8248 sec 2024-03-24T02:42:00.379 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8249 sec 2024-03-24T02:42:01.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:00 smithi078 ceph-mon[32429]: pgmap v4487: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:01.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:00 smithi107 ceph-mon[36485]: pgmap v4487: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:01.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:00 smithi106 ceph-mon[36505]: pgmap v4487: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:01.379 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8250 sec 2024-03-24T02:42:02.379 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8251 sec 2024-03-24T02:42:03.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:02 smithi078 ceph-mon[32429]: pgmap v4488: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:03.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:02 smithi107 ceph-mon[36485]: pgmap v4488: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:03.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:02 smithi106 ceph-mon[36505]: pgmap v4488: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:03.379 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8252 sec 2024-03-24T02:42:04.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7598.213510 secs 2024-03-24T02:42:04.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7598.213510 secs 2024-03-24T02:42:04.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7598.213510 secs 2024-03-24T02:42:04.379 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8253 sec 2024-03-24T02:42:05.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:04 smithi078 ceph-mon[32429]: pgmap v4489: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:05.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:04 smithi107 ceph-mon[36485]: pgmap v4489: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:05.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:04 smithi106 ceph-mon[36505]: pgmap v4489: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:05.379 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8254 sec 2024-03-24T02:42:06.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:05 smithi106 ceph-mon[36505]: pgmap v4490: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:06.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:05 smithi078 ceph-mon[32429]: pgmap v4490: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:06.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:05 smithi107 ceph-mon[36485]: pgmap v4490: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:06.379 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8255 sec 2024-03-24T02:42:07.379 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8256 sec 2024-03-24T02:42:08.379 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8257 sec 2024-03-24T02:42:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:08 smithi078 ceph-mon[32429]: pgmap v4491: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7603.213572 secs 2024-03-24T02:42:09.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:08 smithi107 ceph-mon[36485]: pgmap v4491: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:09.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7603.213572 secs 2024-03-24T02:42:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:08 smithi106 ceph-mon[36505]: pgmap v4491: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7603.213572 secs 2024-03-24T02:42:09.380 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8258 sec 2024-03-24T02:42:10.380 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8259 sec 2024-03-24T02:42:11.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:10 smithi078 ceph-mon[32429]: pgmap v4492: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:11.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:10 smithi107 ceph-mon[36485]: pgmap v4492: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:11.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:10 smithi106 ceph-mon[36505]: pgmap v4492: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:11.380 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8260 sec 2024-03-24T02:42:12.380 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8261 sec 2024-03-24T02:42:13.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:12 smithi078 ceph-mon[32429]: pgmap v4493: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:13.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:12 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:42:13.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:12 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:42:13.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:12 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:42:13.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:12 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:42:13.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:12 smithi107 ceph-mon[36485]: pgmap v4493: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:13.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:12 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:42:13.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:12 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:42:13.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:12 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:42:13.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:12 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:42:13.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:12 smithi106 ceph-mon[36505]: pgmap v4493: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:13.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:12 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:42:13.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:12 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:42:13.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:12 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:42:13.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:12 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:42:13.380 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8262 sec 2024-03-24T02:42:14.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:13 smithi078 ceph-mon[32429]: pgmap v4494: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:42:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:13 smithi078 ceph-mon[32429]: pgmap v4495: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:42:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7608.213658 secs 2024-03-24T02:42:14.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:13 smithi107 ceph-mon[36485]: pgmap v4494: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:42:14.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:13 smithi107 ceph-mon[36485]: pgmap v4495: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:42:14.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7608.213658 secs 2024-03-24T02:42:14.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:13 smithi106 ceph-mon[36505]: pgmap v4494: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:42:14.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:13 smithi106 ceph-mon[36505]: pgmap v4495: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:42:14.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7608.213658 secs 2024-03-24T02:42:14.380 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8263 sec 2024-03-24T02:42:15.380 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8264 sec 2024-03-24T02:42:16.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:15 smithi078 ceph-mon[32429]: pgmap v4496: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:42:16.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:15 smithi107 ceph-mon[36485]: pgmap v4496: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:42:16.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:15 smithi106 ceph-mon[36505]: pgmap v4496: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:42:16.380 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8265 sec 2024-03-24T02:42:17.380 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8266 sec 2024-03-24T02:42:18.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:17 smithi078 ceph-mon[32429]: pgmap v4497: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:42:18.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:17 smithi107 ceph-mon[36485]: pgmap v4497: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:42:18.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:17 smithi106 ceph-mon[36505]: pgmap v4497: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:42:18.380 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8267 sec 2024-03-24T02:42:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7613.213720 secs 2024-03-24T02:42:19.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7613.213720 secs 2024-03-24T02:42:19.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7613.213720 secs 2024-03-24T02:42:19.381 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8268 sec 2024-03-24T02:42:20.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:19 smithi078 ceph-mon[32429]: pgmap v4498: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:42:20.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:19 smithi107 ceph-mon[36485]: pgmap v4498: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:42:20.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:19 smithi106 ceph-mon[36505]: pgmap v4498: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:42:20.380 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8269 sec 2024-03-24T02:42:21.381 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8270 sec 2024-03-24T02:42:22.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:21 smithi078 ceph-mon[32429]: pgmap v4499: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:42:22.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:21 smithi107 ceph-mon[36485]: pgmap v4499: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:42:22.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:21 smithi106 ceph-mon[36505]: pgmap v4499: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:42:22.381 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8271 sec 2024-03-24T02:42:23.381 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8272 sec 2024-03-24T02:42:24.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:23 smithi078 ceph-mon[32429]: pgmap v4500: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:42:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7618.213781 secs 2024-03-24T02:42:24.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:23 smithi107 ceph-mon[36485]: pgmap v4500: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:42:24.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7618.213781 secs 2024-03-24T02:42:24.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:23 smithi106 ceph-mon[36505]: pgmap v4500: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:42:24.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7618.213781 secs 2024-03-24T02:42:24.381 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8273 sec 2024-03-24T02:42:25.381 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8274 sec 2024-03-24T02:42:26.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:25 smithi078 ceph-mon[32429]: pgmap v4501: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:26.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:25 smithi107 ceph-mon[36485]: pgmap v4501: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:26.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:25 smithi106 ceph-mon[36505]: pgmap v4501: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:26.381 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8275 sec 2024-03-24T02:42:27.381 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8276 sec 2024-03-24T02:42:28.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:27 smithi078 ceph-mon[32429]: pgmap v4502: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:28.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:27 smithi107 ceph-mon[36485]: pgmap v4502: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:28.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:27 smithi106 ceph-mon[36505]: pgmap v4502: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:28.381 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8277 sec 2024-03-24T02:42:29.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7623.213867 secs 2024-03-24T02:42:29.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7623.213867 secs 2024-03-24T02:42:29.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7623.213867 secs 2024-03-24T02:42:29.381 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8278 sec 2024-03-24T02:42:30.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:29 smithi078 ceph-mon[32429]: pgmap v4503: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:30.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:29 smithi107 ceph-mon[36485]: pgmap v4503: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:30.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:29 smithi106 ceph-mon[36505]: pgmap v4503: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:30.381 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8279 sec 2024-03-24T02:42:31.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:30 smithi106 ceph-mon[36505]: pgmap v4504: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:31.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:30 smithi078 ceph-mon[32429]: pgmap v4504: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:31.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:30 smithi107 ceph-mon[36485]: pgmap v4504: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:31.381 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8280 sec 2024-03-24T02:42:32.381 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8281 sec 2024-03-24T02:42:33.381 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8282 sec 2024-03-24T02:42:33.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:33 smithi078 ceph-mon[32429]: pgmap v4505: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:33.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:33 smithi107 ceph-mon[36485]: pgmap v4505: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:33.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:33 smithi106 ceph-mon[36505]: pgmap v4505: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:34.382 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8283 sec 2024-03-24T02:42:34.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7628.213904 secs 2024-03-24T02:42:34.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7628.213904 secs 2024-03-24T02:42:34.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7628.213904 secs 2024-03-24T02:42:35.382 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8284 sec 2024-03-24T02:42:35.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:35 smithi078 ceph-mon[32429]: pgmap v4506: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:35.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:35 smithi107 ceph-mon[36485]: pgmap v4506: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:35.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:35 smithi106 ceph-mon[36505]: pgmap v4506: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:36.382 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8285 sec 2024-03-24T02:42:37.382 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8286 sec 2024-03-24T02:42:37.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:37 smithi078 ceph-mon[32429]: pgmap v4507: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:37.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:37 smithi107 ceph-mon[36485]: pgmap v4507: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:37.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:37 smithi106 ceph-mon[36505]: pgmap v4507: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:38.382 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8287 sec 2024-03-24T02:42:39.382 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8288 sec 2024-03-24T02:42:39.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7633.213988 secs 2024-03-24T02:42:39.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:39 smithi078 ceph-mon[32429]: pgmap v4508: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:39.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7633.213988 secs 2024-03-24T02:42:39.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:39 smithi107 ceph-mon[36485]: pgmap v4508: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:39.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7633.213988 secs 2024-03-24T02:42:39.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:39 smithi106 ceph-mon[36505]: pgmap v4508: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:40.382 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8289 sec 2024-03-24T02:42:41.382 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8290 sec 2024-03-24T02:42:41.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:41 smithi078 ceph-mon[32429]: pgmap v4509: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:41.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:41 smithi107 ceph-mon[36485]: pgmap v4509: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:41.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:41 smithi106 ceph-mon[36505]: pgmap v4509: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:42.382 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8291 sec 2024-03-24T02:42:43.383 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8292 sec 2024-03-24T02:42:43.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:43 smithi078 ceph-mon[32429]: pgmap v4510: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:43.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:43 smithi107 ceph-mon[36485]: pgmap v4510: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:43.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:43 smithi106 ceph-mon[36505]: pgmap v4510: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:44.383 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8293 sec 2024-03-24T02:42:44.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7638.214070 secs 2024-03-24T02:42:44.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7638.214070 secs 2024-03-24T02:42:44.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7638.214070 secs 2024-03-24T02:42:45.383 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8294 sec 2024-03-24T02:42:45.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:45 smithi078 ceph-mon[32429]: pgmap v4511: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:45.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:45 smithi107 ceph-mon[36485]: pgmap v4511: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:45.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:45 smithi106 ceph-mon[36505]: pgmap v4511: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:46.383 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8295 sec 2024-03-24T02:42:47.383 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8296 sec 2024-03-24T02:42:47.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:47 smithi078 ceph-mon[32429]: pgmap v4512: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:47.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:47 smithi107 ceph-mon[36485]: pgmap v4512: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:47.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:47 smithi106 ceph-mon[36505]: pgmap v4512: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:48.383 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8297 sec 2024-03-24T02:42:49.383 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8298 sec 2024-03-24T02:42:49.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7643.214159 secs 2024-03-24T02:42:49.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:49 smithi107 ceph-mon[36485]: pgmap v4513: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:49.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7643.214159 secs 2024-03-24T02:42:49.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:49 smithi106 ceph-mon[36505]: pgmap v4513: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:50.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7643.214159 secs 2024-03-24T02:42:50.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:49 smithi078 ceph-mon[32429]: pgmap v4513: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:50.384 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8299 sec 2024-03-24T02:42:51.383 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8300 sec 2024-03-24T02:42:51.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:51 smithi106 ceph-mon[36505]: pgmap v4514: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:52.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:51 smithi078 ceph-mon[32429]: pgmap v4514: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:52.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:51 smithi107 ceph-mon[36485]: pgmap v4514: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:52.383 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8301 sec 2024-03-24T02:42:53.383 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8302 sec 2024-03-24T02:42:53.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:53 smithi106 ceph-mon[36505]: pgmap v4515: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:54.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:53 smithi078 ceph-mon[32429]: pgmap v4515: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:54.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:53 smithi107 ceph-mon[36485]: pgmap v4515: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:54.384 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8303 sec 2024-03-24T02:42:54.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7648.214216 secs 2024-03-24T02:42:55.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7648.214216 secs 2024-03-24T02:42:55.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7648.214216 secs 2024-03-24T02:42:55.384 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8304 sec 2024-03-24T02:42:55.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:55 smithi106 ceph-mon[36505]: pgmap v4516: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:56.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:55 smithi078 ceph-mon[32429]: pgmap v4516: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:56.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:55 smithi107 ceph-mon[36485]: pgmap v4516: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:56.385 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8305 sec 2024-03-24T02:42:57.384 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8306 sec 2024-03-24T02:42:57.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:57 smithi106 ceph-mon[36505]: pgmap v4517: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:58.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:57 smithi078 ceph-mon[32429]: pgmap v4517: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:58.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:57 smithi107 ceph-mon[36485]: pgmap v4517: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:58.384 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8307 sec 2024-03-24T02:42:59.384 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8308 sec 2024-03-24T02:42:59.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7653.214278 secs 2024-03-24T02:42:59.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:42:59 smithi078 ceph-mon[32429]: pgmap v4518: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:42:59.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7653.214278 secs 2024-03-24T02:42:59.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:42:59 smithi106 ceph-mon[36505]: pgmap v4518: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:00.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7653.214278 secs 2024-03-24T02:43:00.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:42:59 smithi107 ceph-mon[36485]: pgmap v4518: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:00.384 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8309 sec 2024-03-24T02:43:01.384 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8310 sec 2024-03-24T02:43:01.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:01 smithi106 ceph-mon[36505]: pgmap v4519: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:02.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:01 smithi078 ceph-mon[32429]: pgmap v4519: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:02.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:01 smithi107 ceph-mon[36485]: pgmap v4519: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:02.384 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8311 sec 2024-03-24T02:43:03.384 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8312 sec 2024-03-24T02:43:03.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:03 smithi106 ceph-mon[36505]: pgmap v4520: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:03 smithi078 ceph-mon[32429]: pgmap v4520: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:04.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:03 smithi107 ceph-mon[36485]: pgmap v4520: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:04.384 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8313 sec 2024-03-24T02:43:04.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7658.214316 secs 2024-03-24T02:43:05.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7658.214316 secs 2024-03-24T02:43:05.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7658.214316 secs 2024-03-24T02:43:05.385 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8314 sec 2024-03-24T02:43:05.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:05 smithi106 ceph-mon[36505]: pgmap v4521: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:06.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:05 smithi078 ceph-mon[32429]: pgmap v4521: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:06.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:05 smithi107 ceph-mon[36485]: pgmap v4521: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:06.385 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8315 sec 2024-03-24T02:43:07.385 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8316 sec 2024-03-24T02:43:07.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:07 smithi078 ceph-mon[32429]: pgmap v4522: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:07.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:07 smithi106 ceph-mon[36505]: pgmap v4522: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:08.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:07 smithi107 ceph-mon[36485]: pgmap v4522: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:08.385 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8317 sec 2024-03-24T02:43:09.385 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8318 sec 2024-03-24T02:43:09.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7663.214378 secs 2024-03-24T02:43:09.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:09 smithi106 ceph-mon[36505]: pgmap v4523: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7663.214378 secs 2024-03-24T02:43:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:09 smithi078 ceph-mon[32429]: pgmap v4523: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:10.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7663.214378 secs 2024-03-24T02:43:10.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:09 smithi107 ceph-mon[36485]: pgmap v4523: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:10.385 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8319 sec 2024-03-24T02:43:11.385 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8320 sec 2024-03-24T02:43:11.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:11 smithi106 ceph-mon[36505]: pgmap v4524: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:12.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:11 smithi078 ceph-mon[32429]: pgmap v4524: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:12.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:11 smithi107 ceph-mon[36485]: pgmap v4524: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:12.385 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8321 sec 2024-03-24T02:43:12.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:12 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:43:13.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:12 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:43:13.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:12 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:43:13.385 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8322 sec 2024-03-24T02:43:13.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:13 smithi106 ceph-mon[36505]: pgmap v4525: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:13.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:13 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:43:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:13 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:43:13.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:13 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:43:14.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:13 smithi078 ceph-mon[32429]: pgmap v4525: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:13 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:43:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:13 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:43:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:13 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:43:14.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:13 smithi107 ceph-mon[36485]: pgmap v4525: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:14.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:13 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:43:14.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:13 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:43:14.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:13 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:43:14.385 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8323 sec 2024-03-24T02:43:14.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7668.214468 secs 2024-03-24T02:43:14.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:14 smithi106 ceph-mon[36505]: pgmap v4526: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:43:14.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:14 smithi106 ceph-mon[36505]: pgmap v4527: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:43:15.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7668.214468 secs 2024-03-24T02:43:15.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:14 smithi078 ceph-mon[32429]: pgmap v4526: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:43:15.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:14 smithi078 ceph-mon[32429]: pgmap v4527: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:43:15.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7668.214468 secs 2024-03-24T02:43:15.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:14 smithi107 ceph-mon[36485]: pgmap v4526: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:43:15.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:14 smithi107 ceph-mon[36485]: pgmap v4527: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:43:15.386 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8324 sec 2024-03-24T02:43:16.386 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8325 sec 2024-03-24T02:43:17.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:16 smithi078 ceph-mon[32429]: pgmap v4528: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:43:17.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:16 smithi107 ceph-mon[36485]: pgmap v4528: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:43:17.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:16 smithi106 ceph-mon[36505]: pgmap v4528: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:43:17.386 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8326 sec 2024-03-24T02:43:18.386 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8327 sec 2024-03-24T02:43:19.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:18 smithi078 ceph-mon[32429]: pgmap v4529: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:43:19.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:18 smithi107 ceph-mon[36485]: pgmap v4529: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:43:19.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:18 smithi106 ceph-mon[36505]: pgmap v4529: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:43:19.386 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8328 sec 2024-03-24T02:43:19.994 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7673.214531 secs 2024-03-24T02:43:20.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7673.214531 secs 2024-03-24T02:43:20.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7673.214531 secs 2024-03-24T02:43:20.386 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8329 sec 2024-03-24T02:43:21.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:20 smithi078 ceph-mon[32429]: pgmap v4530: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:43:21.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:20 smithi107 ceph-mon[36485]: pgmap v4530: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:43:21.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:20 smithi106 ceph-mon[36505]: pgmap v4530: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:43:21.386 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8330 sec 2024-03-24T02:43:22.386 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8331 sec 2024-03-24T02:43:23.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:22 smithi078 ceph-mon[32429]: pgmap v4531: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:43:23.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:22 smithi107 ceph-mon[36485]: pgmap v4531: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:43:23.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:22 smithi106 ceph-mon[36505]: pgmap v4531: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:43:23.386 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8332 sec 2024-03-24T02:43:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7678.214610 secs 2024-03-24T02:43:24.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7678.214610 secs 2024-03-24T02:43:24.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7678.214610 secs 2024-03-24T02:43:24.386 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8333 sec 2024-03-24T02:43:25.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:24 smithi078 ceph-mon[32429]: pgmap v4532: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:43:25.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:24 smithi107 ceph-mon[36485]: pgmap v4532: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:43:25.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:24 smithi106 ceph-mon[36505]: pgmap v4532: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:43:25.386 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8334 sec 2024-03-24T02:43:26.386 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8335 sec 2024-03-24T02:43:27.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:26 smithi078 ceph-mon[32429]: pgmap v4533: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:27.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:26 smithi107 ceph-mon[36485]: pgmap v4533: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:27.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:26 smithi106 ceph-mon[36505]: pgmap v4533: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:27.386 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8336 sec 2024-03-24T02:43:28.387 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8337 sec 2024-03-24T02:43:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:28 smithi078 ceph-mon[32429]: pgmap v4534: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7683.214673 secs 2024-03-24T02:43:29.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:28 smithi107 ceph-mon[36485]: pgmap v4534: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:29.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7683.214673 secs 2024-03-24T02:43:29.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:28 smithi106 ceph-mon[36505]: pgmap v4534: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:29.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7683.214673 secs 2024-03-24T02:43:29.386 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8338 sec 2024-03-24T02:43:30.387 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8339 sec 2024-03-24T02:43:31.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:30 smithi078 ceph-mon[32429]: pgmap v4535: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:31.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:30 smithi107 ceph-mon[36485]: pgmap v4535: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:31.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:30 smithi106 ceph-mon[36505]: pgmap v4535: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:31.387 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8340 sec 2024-03-24T02:43:32.387 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8341 sec 2024-03-24T02:43:32.953 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:32 smithi106 ceph-mon[36505]: pgmap v4536: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:33.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:32 smithi078 ceph-mon[32429]: pgmap v4536: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:33.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:32 smithi107 ceph-mon[36485]: pgmap v4536: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:33.387 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8342 sec 2024-03-24T02:43:34.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7688.214744 secs 2024-03-24T02:43:34.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7688.214744 secs 2024-03-24T02:43:34.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7688.214744 secs 2024-03-24T02:43:34.387 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8343 sec 2024-03-24T02:43:35.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:34 smithi078 ceph-mon[32429]: pgmap v4537: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:35.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:34 smithi107 ceph-mon[36485]: pgmap v4537: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:35.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:34 smithi106 ceph-mon[36505]: pgmap v4537: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:35.387 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8344 sec 2024-03-24T02:43:36.388 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8345 sec 2024-03-24T02:43:37.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:36 smithi078 ceph-mon[32429]: pgmap v4538: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:37.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:36 smithi107 ceph-mon[36485]: pgmap v4538: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:37.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:36 smithi106 ceph-mon[36505]: pgmap v4538: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:37.387 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8346 sec 2024-03-24T02:43:38.387 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8347 sec 2024-03-24T02:43:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:38 smithi078 ceph-mon[32429]: pgmap v4539: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7693.214832 secs 2024-03-24T02:43:39.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:38 smithi107 ceph-mon[36485]: pgmap v4539: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:39.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7693.214832 secs 2024-03-24T02:43:39.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:38 smithi106 ceph-mon[36505]: pgmap v4539: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:39.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7693.214832 secs 2024-03-24T02:43:39.387 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8348 sec 2024-03-24T02:43:40.388 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8349 sec 2024-03-24T02:43:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:40 smithi078 ceph-mon[32429]: pgmap v4540: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:41.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:40 smithi107 ceph-mon[36485]: pgmap v4540: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:41.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:40 smithi106 ceph-mon[36505]: pgmap v4540: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:41.388 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8350 sec 2024-03-24T02:43:42.388 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8351 sec 2024-03-24T02:43:43.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:42 smithi078 ceph-mon[32429]: pgmap v4541: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:43.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:42 smithi107 ceph-mon[36485]: pgmap v4541: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:43.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:42 smithi106 ceph-mon[36505]: pgmap v4541: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:43.388 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8352 sec 2024-03-24T02:43:44.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7698.214845 secs 2024-03-24T02:43:44.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7698.214845 secs 2024-03-24T02:43:44.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7698.214845 secs 2024-03-24T02:43:44.388 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8353 sec 2024-03-24T02:43:45.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:44 smithi078 ceph-mon[32429]: pgmap v4542: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:45.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:44 smithi107 ceph-mon[36485]: pgmap v4542: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:45.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:44 smithi106 ceph-mon[36505]: pgmap v4542: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:45.388 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8354 sec 2024-03-24T02:43:46.388 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8355 sec 2024-03-24T02:43:47.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:46 smithi078 ceph-mon[32429]: pgmap v4543: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:47.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:46 smithi107 ceph-mon[36485]: pgmap v4543: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:47.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:46 smithi106 ceph-mon[36505]: pgmap v4543: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:47.388 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8356 sec 2024-03-24T02:43:48.388 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8357 sec 2024-03-24T02:43:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:48 smithi078 ceph-mon[32429]: pgmap v4544: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7703.214935 secs 2024-03-24T02:43:49.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:48 smithi107 ceph-mon[36485]: pgmap v4544: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:49.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7703.214935 secs 2024-03-24T02:43:49.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:48 smithi106 ceph-mon[36505]: pgmap v4544: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:49.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7703.214935 secs 2024-03-24T02:43:49.388 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8358 sec 2024-03-24T02:43:50.389 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8359 sec 2024-03-24T02:43:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:50 smithi078 ceph-mon[32429]: pgmap v4545: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:51.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:50 smithi107 ceph-mon[36485]: pgmap v4545: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:51.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:50 smithi106 ceph-mon[36505]: pgmap v4545: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:51.389 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8360 sec 2024-03-24T02:43:52.389 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8361 sec 2024-03-24T02:43:53.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:52 smithi078 ceph-mon[32429]: pgmap v4546: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:53.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:52 smithi107 ceph-mon[36485]: pgmap v4546: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:53.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:52 smithi106 ceph-mon[36505]: pgmap v4546: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:53.389 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8362 sec 2024-03-24T02:43:54.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7708.214996 secs 2024-03-24T02:43:54.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7708.214996 secs 2024-03-24T02:43:54.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7708.214996 secs 2024-03-24T02:43:54.389 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8363 sec 2024-03-24T02:43:55.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:54 smithi078 ceph-mon[32429]: pgmap v4547: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:55.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:54 smithi107 ceph-mon[36485]: pgmap v4547: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:55.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:54 smithi106 ceph-mon[36505]: pgmap v4547: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:55.389 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8364 sec 2024-03-24T02:43:56.389 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8365 sec 2024-03-24T02:43:57.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:56 smithi078 ceph-mon[32429]: pgmap v4548: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:57.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:56 smithi107 ceph-mon[36485]: pgmap v4548: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:57.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:56 smithi106 ceph-mon[36505]: pgmap v4548: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:43:57.389 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8366 sec 2024-03-24T02:43:58.389 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8367 sec 2024-03-24T02:43:59.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:58 smithi078 ceph-mon[32429]: pgmap v4549: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:59.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:43:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7713.215058 secs 2024-03-24T02:43:59.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:58 smithi107 ceph-mon[36485]: pgmap v4549: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:59.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:43:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7713.215058 secs 2024-03-24T02:43:59.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:58 smithi106 ceph-mon[36505]: pgmap v4549: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:59.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:43:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7713.215058 secs 2024-03-24T02:43:59.389 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8368 sec 2024-03-24T02:44:00.389 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8369 sec 2024-03-24T02:44:01.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:00 smithi078 ceph-mon[32429]: pgmap v4550: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:01.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:00 smithi107 ceph-mon[36485]: pgmap v4550: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:01.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:00 smithi106 ceph-mon[36505]: pgmap v4550: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:01.389 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8370 sec 2024-03-24T02:44:02.390 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8371 sec 2024-03-24T02:44:03.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:02 smithi078 ceph-mon[32429]: pgmap v4551: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:03.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:02 smithi107 ceph-mon[36485]: pgmap v4551: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:03.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:02 smithi106 ceph-mon[36505]: pgmap v4551: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:03.390 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8372 sec 2024-03-24T02:44:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7718.215144 secs 2024-03-24T02:44:04.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7718.215144 secs 2024-03-24T02:44:04.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7718.215144 secs 2024-03-24T02:44:04.390 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8373 sec 2024-03-24T02:44:05.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:04 smithi078 ceph-mon[32429]: pgmap v4552: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:05.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:04 smithi107 ceph-mon[36485]: pgmap v4552: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:05.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:04 smithi106 ceph-mon[36505]: pgmap v4552: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:05.390 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8374 sec 2024-03-24T02:44:06.391 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8375 sec 2024-03-24T02:44:07.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:06 smithi078 ceph-mon[32429]: pgmap v4553: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:07.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:06 smithi107 ceph-mon[36485]: pgmap v4553: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:07.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:06 smithi106 ceph-mon[36505]: pgmap v4553: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:07.390 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8376 sec 2024-03-24T02:44:08.390 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8377 sec 2024-03-24T02:44:09.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:08 smithi078 ceph-mon[32429]: pgmap v4554: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7723.215207 secs 2024-03-24T02:44:09.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:08 smithi107 ceph-mon[36485]: pgmap v4554: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:09.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7723.215207 secs 2024-03-24T02:44:09.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:08 smithi106 ceph-mon[36505]: pgmap v4554: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:09.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7723.215207 secs 2024-03-24T02:44:09.390 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8378 sec 2024-03-24T02:44:10.390 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8379 sec 2024-03-24T02:44:11.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:10 smithi078 ceph-mon[32429]: pgmap v4555: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:11.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:10 smithi107 ceph-mon[36485]: pgmap v4555: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:11.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:10 smithi106 ceph-mon[36505]: pgmap v4555: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:11.390 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8380 sec 2024-03-24T02:44:12.390 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8381 sec 2024-03-24T02:44:13.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:12 smithi078 ceph-mon[32429]: pgmap v4556: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:13.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:12 smithi107 ceph-mon[36485]: pgmap v4556: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:13.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:12 smithi106 ceph-mon[36505]: pgmap v4556: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:13.390 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8382 sec 2024-03-24T02:44:14.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7728.215267 secs 2024-03-24T02:44:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:13 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:44:14.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7728.215267 secs 2024-03-24T02:44:14.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:13 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:44:14.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7728.215267 secs 2024-03-24T02:44:14.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:13 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:44:14.391 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8383 sec 2024-03-24T02:44:15.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:14 smithi078 ceph-mon[32429]: pgmap v4557: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:15.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:14 smithi107 ceph-mon[36485]: pgmap v4557: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:15.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:14 smithi106 ceph-mon[36505]: pgmap v4557: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:15.391 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8384 sec 2024-03-24T02:44:16.391 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8385 sec 2024-03-24T02:44:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:16 smithi106 ceph-mon[36505]: pgmap v4558: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:44:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:44:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:44:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:44:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:44:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:44:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:44:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:44:16.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:44:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:16 smithi078 ceph-mon[32429]: pgmap v4558: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:44:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:44:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:44:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:44:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:44:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:44:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:44:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:44:16.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:44:16.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:16 smithi107 ceph-mon[36485]: pgmap v4558: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:44:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:44:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:44:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:44:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:44:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:44:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:44:16.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:44:16.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:44:16.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:44:17.391 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8386 sec 2024-03-24T02:44:18.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:17 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T02:44:18.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:17 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T02:44:18.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:17 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T02:44:18.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:17 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T02:44:18.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:17 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T02:44:18.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:17 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T02:44:18.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:17 smithi078 ceph-mon[32429]: pgmap v4559: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-03-24T02:44:18.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:17 smithi078 ceph-mon[32429]: pgmap v4560: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:44:18.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:17 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T02:44:18.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:17 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T02:44:18.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:17 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T02:44:18.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:17 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T02:44:18.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:17 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T02:44:18.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:17 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T02:44:18.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:17 smithi107 ceph-mon[36485]: pgmap v4559: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-03-24T02:44:18.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:17 smithi107 ceph-mon[36485]: pgmap v4560: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:44:18.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:17 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T02:44:18.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:17 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T02:44:18.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:17 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T02:44:18.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:17 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T02:44:18.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:17 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T02:44:18.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:17 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T02:44:18.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:17 smithi106 ceph-mon[36505]: pgmap v4559: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-03-24T02:44:18.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:17 smithi106 ceph-mon[36505]: pgmap v4560: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:44:18.391 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8387 sec 2024-03-24T02:44:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7733.215326 secs 2024-03-24T02:44:19.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7733.215326 secs 2024-03-24T02:44:19.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7733.215326 secs 2024-03-24T02:44:19.391 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8388 sec 2024-03-24T02:44:20.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:19 smithi078 ceph-mon[32429]: pgmap v4561: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:44:20.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:19 smithi107 ceph-mon[36485]: pgmap v4561: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:44:20.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:19 smithi106 ceph-mon[36505]: pgmap v4561: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:44:20.391 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8389 sec 2024-03-24T02:44:21.391 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8390 sec 2024-03-24T02:44:22.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:21 smithi078 ceph-mon[32429]: pgmap v4562: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:22.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:21 smithi107 ceph-mon[36485]: pgmap v4562: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:22.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:21 smithi106 ceph-mon[36505]: pgmap v4562: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:22.391 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8391 sec 2024-03-24T02:44:23.391 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8392 sec 2024-03-24T02:44:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:23 smithi078 ceph-mon[32429]: pgmap v4563: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:44:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7738.215416 secs 2024-03-24T02:44:24.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:23 smithi107 ceph-mon[36485]: pgmap v4563: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:44:24.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7738.215416 secs 2024-03-24T02:44:24.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:23 smithi106 ceph-mon[36505]: pgmap v4563: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:44:24.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7738.215416 secs 2024-03-24T02:44:24.391 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8393 sec 2024-03-24T02:44:25.391 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8394 sec 2024-03-24T02:44:26.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:25 smithi078 ceph-mon[32429]: pgmap v4564: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-03-24T02:44:26.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:25 smithi107 ceph-mon[36485]: pgmap v4564: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-03-24T02:44:26.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:25 smithi106 ceph-mon[36505]: pgmap v4564: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-03-24T02:44:26.392 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8395 sec 2024-03-24T02:44:27.392 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8396 sec 2024-03-24T02:44:28.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:27 smithi078 ceph-mon[32429]: pgmap v4565: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:44:28.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:27 smithi107 ceph-mon[36485]: pgmap v4565: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:44:28.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:27 smithi106 ceph-mon[36505]: pgmap v4565: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:44:28.392 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8397 sec 2024-03-24T02:44:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7743.215465 secs 2024-03-24T02:44:29.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7743.215465 secs 2024-03-24T02:44:29.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7743.215465 secs 2024-03-24T02:44:29.392 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8398 sec 2024-03-24T02:44:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:29 smithi078 ceph-mon[32429]: pgmap v4566: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:30.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:29 smithi107 ceph-mon[36485]: pgmap v4566: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:30.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:29 smithi106 ceph-mon[36505]: pgmap v4566: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:30.392 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8399 sec 2024-03-24T02:44:31.392 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8400 sec 2024-03-24T02:44:32.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:31 smithi078 ceph-mon[32429]: pgmap v4567: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:32.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:31 smithi107 ceph-mon[36485]: pgmap v4567: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:32.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:31 smithi106 ceph-mon[36505]: pgmap v4567: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:32.392 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8401 sec 2024-03-24T02:44:33.392 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8402 sec 2024-03-24T02:44:34.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:33 smithi078 ceph-mon[32429]: pgmap v4568: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-24T02:44:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7748.215525 secs 2024-03-24T02:44:34.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:33 smithi107 ceph-mon[36485]: pgmap v4568: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-24T02:44:34.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7748.215525 secs 2024-03-24T02:44:34.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:33 smithi106 ceph-mon[36505]: pgmap v4568: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-24T02:44:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7748.215525 secs 2024-03-24T02:44:34.392 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8403 sec 2024-03-24T02:44:35.392 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8404 sec 2024-03-24T02:44:36.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:35 smithi078 ceph-mon[32429]: pgmap v4569: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:36.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:35 smithi107 ceph-mon[36485]: pgmap v4569: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:36.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:35 smithi106 ceph-mon[36505]: pgmap v4569: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:36.392 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8405 sec 2024-03-24T02:44:37.392 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8406 sec 2024-03-24T02:44:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:37 smithi078 ceph-mon[32429]: pgmap v4570: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:38.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:37 smithi107 ceph-mon[36485]: pgmap v4570: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:38.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:37 smithi106 ceph-mon[36505]: pgmap v4570: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:38.393 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8407 sec 2024-03-24T02:44:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7753.215542 secs 2024-03-24T02:44:39.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7753.215542 secs 2024-03-24T02:44:39.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7753.215542 secs 2024-03-24T02:44:39.393 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8408 sec 2024-03-24T02:44:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:39 smithi078 ceph-mon[32429]: pgmap v4571: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:40.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:39 smithi107 ceph-mon[36485]: pgmap v4571: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:40.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:39 smithi106 ceph-mon[36505]: pgmap v4571: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:40.393 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8409 sec 2024-03-24T02:44:41.393 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8410 sec 2024-03-24T02:44:42.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:41 smithi078 ceph-mon[32429]: pgmap v4572: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:42.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:41 smithi107 ceph-mon[36485]: pgmap v4572: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:42.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:41 smithi106 ceph-mon[36505]: pgmap v4572: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:42.393 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8411 sec 2024-03-24T02:44:43.393 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8412 sec 2024-03-24T02:44:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:43 smithi078 ceph-mon[32429]: pgmap v4573: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7758.215607 secs 2024-03-24T02:44:44.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:43 smithi107 ceph-mon[36485]: pgmap v4573: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:44.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7758.215607 secs 2024-03-24T02:44:44.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:43 smithi106 ceph-mon[36505]: pgmap v4573: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:44.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7758.215607 secs 2024-03-24T02:44:44.393 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8413 sec 2024-03-24T02:44:45.393 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8414 sec 2024-03-24T02:44:46.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:45 smithi078 ceph-mon[32429]: pgmap v4574: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:46.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:45 smithi107 ceph-mon[36485]: pgmap v4574: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:46.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:45 smithi106 ceph-mon[36505]: pgmap v4574: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:46.393 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8415 sec 2024-03-24T02:44:47.393 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8416 sec 2024-03-24T02:44:48.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:47 smithi078 ceph-mon[32429]: pgmap v4575: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:48.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:47 smithi107 ceph-mon[36485]: pgmap v4575: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:48.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:47 smithi106 ceph-mon[36505]: pgmap v4575: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:48.393 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8417 sec 2024-03-24T02:44:49.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7763.215669 secs 2024-03-24T02:44:49.339 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7763.215669 secs 2024-03-24T02:44:49.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7763.215669 secs 2024-03-24T02:44:49.393 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8418 sec 2024-03-24T02:44:50.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:49 smithi106 ceph-mon[36505]: pgmap v4576: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:50.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:49 smithi078 ceph-mon[32429]: pgmap v4576: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:50.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:49 smithi107 ceph-mon[36485]: pgmap v4576: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:50.394 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8419 sec 2024-03-24T02:44:51.394 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8420 sec 2024-03-24T02:44:52.145 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:51 smithi107 ceph-mon[36485]: pgmap v4577: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:52.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:51 smithi106 ceph-mon[36505]: pgmap v4577: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:51 smithi078 ceph-mon[32429]: pgmap v4577: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:52.394 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8421 sec 2024-03-24T02:44:53.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:52 smithi106 ceph-mon[36505]: pgmap v4578: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:53.339 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:52 smithi078 ceph-mon[32429]: pgmap v4578: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:53.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:52 smithi107 ceph-mon[36485]: pgmap v4578: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:53.394 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8422 sec 2024-03-24T02:44:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7768.215756 secs 2024-03-24T02:44:54.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7768.215756 secs 2024-03-24T02:44:54.394 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8423 sec 2024-03-24T02:44:54.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7768.215756 secs 2024-03-24T02:44:55.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:54 smithi078 ceph-mon[32429]: pgmap v4579: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:55.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:54 smithi107 ceph-mon[36485]: pgmap v4579: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:55.394 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8424 sec 2024-03-24T02:44:55.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:54 smithi106 ceph-mon[36505]: pgmap v4579: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:56.394 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8425 sec 2024-03-24T02:44:57.394 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8426 sec 2024-03-24T02:44:57.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:57 smithi078 ceph-mon[32429]: pgmap v4580: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:57.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:57 smithi107 ceph-mon[36485]: pgmap v4580: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:57.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:57 smithi106 ceph-mon[36505]: pgmap v4580: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:58.394 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8427 sec 2024-03-24T02:44:59.394 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8428 sec 2024-03-24T02:44:59.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7773.215816 secs 2024-03-24T02:44:59.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:44:59 smithi078 ceph-mon[32429]: pgmap v4581: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:59.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7773.215816 secs 2024-03-24T02:44:59.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:44:59 smithi107 ceph-mon[36485]: pgmap v4581: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:44:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7773.215816 secs 2024-03-24T02:44:59.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:44:59 smithi106 ceph-mon[36505]: pgmap v4581: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:00.394 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8429 sec 2024-03-24T02:45:01.395 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8430 sec 2024-03-24T02:45:01.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:01 smithi078 ceph-mon[32429]: pgmap v4582: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:01.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:01 smithi107 ceph-mon[36485]: pgmap v4582: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:01.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:01 smithi106 ceph-mon[36505]: pgmap v4582: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:02.395 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8431 sec 2024-03-24T02:45:03.395 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8432 sec 2024-03-24T02:45:03.703 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:03 smithi106 ceph-mon[36505]: pgmap v4583: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:03.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:03 smithi078 ceph-mon[32429]: pgmap v4583: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:03.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:03 smithi107 ceph-mon[36485]: pgmap v4583: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:04.395 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8433 sec 2024-03-24T02:45:04.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7778.215904 secs 2024-03-24T02:45:04.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7778.215904 secs 2024-03-24T02:45:04.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7778.215904 secs 2024-03-24T02:45:05.395 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8434 sec 2024-03-24T02:45:05.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:05 smithi078 ceph-mon[32429]: pgmap v4584: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:05.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:05 smithi107 ceph-mon[36485]: pgmap v4584: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:05.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:05 smithi106 ceph-mon[36505]: pgmap v4584: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:06.395 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8435 sec 2024-03-24T02:45:07.395 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8436 sec 2024-03-24T02:45:07.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:07 smithi078 ceph-mon[32429]: pgmap v4585: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:07.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:07 smithi107 ceph-mon[36485]: pgmap v4585: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:07.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:07 smithi106 ceph-mon[36505]: pgmap v4585: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:08.395 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8437 sec 2024-03-24T02:45:09.395 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8438 sec 2024-03-24T02:45:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7783.215964 secs 2024-03-24T02:45:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:09 smithi078 ceph-mon[32429]: pgmap v4586: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:09.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7783.215964 secs 2024-03-24T02:45:09.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:09 smithi107 ceph-mon[36485]: pgmap v4586: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:09.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7783.215964 secs 2024-03-24T02:45:09.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:09 smithi106 ceph-mon[36505]: pgmap v4586: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:10.395 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8439 sec 2024-03-24T02:45:11.395 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8440 sec 2024-03-24T02:45:11.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:11 smithi078 ceph-mon[32429]: pgmap v4587: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:11.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:11 smithi107 ceph-mon[36485]: pgmap v4587: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:11.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:11 smithi106 ceph-mon[36505]: pgmap v4587: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:12.396 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8441 sec 2024-03-24T02:45:13.396 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8442 sec 2024-03-24T02:45:13.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:13 smithi078 ceph-mon[32429]: pgmap v4588: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:13.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:13 smithi107 ceph-mon[36485]: pgmap v4588: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:13.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:13 smithi106 ceph-mon[36505]: pgmap v4588: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:14.396 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8443 sec 2024-03-24T02:45:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7788.216003 secs 2024-03-24T02:45:14.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7788.216003 secs 2024-03-24T02:45:14.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7788.216003 secs 2024-03-24T02:45:15.396 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8444 sec 2024-03-24T02:45:15.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:15 smithi078 ceph-mon[32429]: pgmap v4589: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:15.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:15 smithi107 ceph-mon[36485]: pgmap v4589: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:15.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:15 smithi106 ceph-mon[36505]: pgmap v4589: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:16.396 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8445 sec 2024-03-24T02:45:17.396 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8446 sec 2024-03-24T02:45:17.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:17 smithi078 ceph-mon[32429]: pgmap v4590: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:17.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:17 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:45:17.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:17 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:45:17.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:17 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:45:17.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:17 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:45:17.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:17 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:45:17.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:17 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:45:17.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:17 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:45:17.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:17 smithi107 ceph-mon[36485]: pgmap v4590: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:17.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:17 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:45:17.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:17 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:45:17.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:17 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:45:17.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:17 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:45:17.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:17 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:45:17.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:17 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:45:17.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:17 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:45:17.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:17 smithi106 ceph-mon[36505]: pgmap v4590: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:17.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:17 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:45:17.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:17 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:45:17.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:17 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:45:17.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:17 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:45:17.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:17 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:45:17.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:17 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:45:17.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:17 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:45:18.396 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8447 sec 2024-03-24T02:45:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:18 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:45:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:18 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:45:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:18 smithi078 ceph-mon[32429]: pgmap v4591: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T02:45:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:18 smithi078 ceph-mon[32429]: pgmap v4592: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:45:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:18 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:45:18.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:18 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:45:18.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:18 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:45:18.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:18 smithi107 ceph-mon[36485]: pgmap v4591: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T02:45:18.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:18 smithi107 ceph-mon[36485]: pgmap v4592: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:45:18.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:18 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:45:18.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:18 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:45:18.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:18 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:45:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:18 smithi106 ceph-mon[36505]: pgmap v4591: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T02:45:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:18 smithi106 ceph-mon[36505]: pgmap v4592: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:45:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:18 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:45:19.396 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8448 sec 2024-03-24T02:45:19.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7793.216066 secs 2024-03-24T02:45:19.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7793.216066 secs 2024-03-24T02:45:19.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7793.216066 secs 2024-03-24T02:45:20.396 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8449 sec 2024-03-24T02:45:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:20 smithi078 ceph-mon[32429]: pgmap v4593: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:45:20.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:20 smithi107 ceph-mon[36485]: pgmap v4593: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:45:20.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:20 smithi106 ceph-mon[36505]: pgmap v4593: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:45:21.396 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8450 sec 2024-03-24T02:45:22.396 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8451 sec 2024-03-24T02:45:23.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:22 smithi078 ceph-mon[32429]: pgmap v4594: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:45:23.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:22 smithi107 ceph-mon[36485]: pgmap v4594: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:45:23.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:22 smithi106 ceph-mon[36505]: pgmap v4594: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:45:23.396 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8452 sec 2024-03-24T02:45:24.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7798.216127 secs 2024-03-24T02:45:24.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7798.216127 secs 2024-03-24T02:45:24.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7798.216127 secs 2024-03-24T02:45:24.397 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8453 sec 2024-03-24T02:45:25.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:24 smithi078 ceph-mon[32429]: pgmap v4595: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:45:25.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:24 smithi107 ceph-mon[36485]: pgmap v4595: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:45:25.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:24 smithi106 ceph-mon[36505]: pgmap v4595: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:45:25.397 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8454 sec 2024-03-24T02:45:26.397 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8455 sec 2024-03-24T02:45:27.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:26 smithi078 ceph-mon[32429]: pgmap v4596: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:45:27.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:26 smithi107 ceph-mon[36485]: pgmap v4596: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:45:27.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:26 smithi106 ceph-mon[36505]: pgmap v4596: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:45:27.397 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8456 sec 2024-03-24T02:45:28.397 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8457 sec 2024-03-24T02:45:29.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:28 smithi078 ceph-mon[32429]: pgmap v4597: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:45:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7803.216216 secs 2024-03-24T02:45:29.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:28 smithi107 ceph-mon[36485]: pgmap v4597: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:45:29.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7803.216216 secs 2024-03-24T02:45:29.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:28 smithi106 ceph-mon[36505]: pgmap v4597: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:45:29.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7803.216216 secs 2024-03-24T02:45:29.397 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8458 sec 2024-03-24T02:45:30.397 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8459 sec 2024-03-24T02:45:31.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:30 smithi078 ceph-mon[32429]: pgmap v4598: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:31.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:30 smithi107 ceph-mon[36485]: pgmap v4598: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:31.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:30 smithi106 ceph-mon[36505]: pgmap v4598: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:31.397 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8460 sec 2024-03-24T02:45:32.398 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8461 sec 2024-03-24T02:45:33.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:32 smithi078 ceph-mon[32429]: pgmap v4599: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:33.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:32 smithi107 ceph-mon[36485]: pgmap v4599: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:33.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:32 smithi106 ceph-mon[36505]: pgmap v4599: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:33.397 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8462 sec 2024-03-24T02:45:34.053 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7808.216259 secs 2024-03-24T02:45:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7808.216259 secs 2024-03-24T02:45:34.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7808.216259 secs 2024-03-24T02:45:34.397 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8463 sec 2024-03-24T02:45:35.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:34 smithi078 ceph-mon[32429]: pgmap v4600: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:35.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:34 smithi107 ceph-mon[36485]: pgmap v4600: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:35.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:34 smithi106 ceph-mon[36505]: pgmap v4600: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:35.398 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8464 sec 2024-03-24T02:45:36.398 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8465 sec 2024-03-24T02:45:37.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:36 smithi078 ceph-mon[32429]: pgmap v4601: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:37.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:36 smithi107 ceph-mon[36485]: pgmap v4601: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:37.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:36 smithi106 ceph-mon[36505]: pgmap v4601: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:37.398 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8466 sec 2024-03-24T02:45:38.398 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8467 sec 2024-03-24T02:45:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:38 smithi078 ceph-mon[32429]: pgmap v4602: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7813.216320 secs 2024-03-24T02:45:39.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:38 smithi107 ceph-mon[36485]: pgmap v4602: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:39.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7813.216320 secs 2024-03-24T02:45:39.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:38 smithi106 ceph-mon[36505]: pgmap v4602: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:39.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7813.216320 secs 2024-03-24T02:45:39.398 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8468 sec 2024-03-24T02:45:40.398 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8469 sec 2024-03-24T02:45:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:40 smithi078 ceph-mon[32429]: pgmap v4603: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:41.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:40 smithi107 ceph-mon[36485]: pgmap v4603: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:41.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:40 smithi106 ceph-mon[36505]: pgmap v4603: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:41.398 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8470 sec 2024-03-24T02:45:42.398 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8471 sec 2024-03-24T02:45:43.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:42 smithi078 ceph-mon[32429]: pgmap v4604: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:43.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:42 smithi107 ceph-mon[36485]: pgmap v4604: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:43.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:42 smithi106 ceph-mon[36505]: pgmap v4604: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:43.398 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8472 sec 2024-03-24T02:45:43.991 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7818.216379 secs 2024-03-24T02:45:44.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7818.216379 secs 2024-03-24T02:45:44.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7818.216379 secs 2024-03-24T02:45:44.398 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8473 sec 2024-03-24T02:45:45.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:44 smithi078 ceph-mon[32429]: pgmap v4605: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:45.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:44 smithi107 ceph-mon[36485]: pgmap v4605: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:45.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:44 smithi106 ceph-mon[36505]: pgmap v4605: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:45.399 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8474 sec 2024-03-24T02:45:46.399 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8475 sec 2024-03-24T02:45:47.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:46 smithi078 ceph-mon[32429]: pgmap v4606: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:47.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:46 smithi107 ceph-mon[36485]: pgmap v4606: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:47.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:46 smithi106 ceph-mon[36505]: pgmap v4606: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:47.399 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8476 sec 2024-03-24T02:45:48.399 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8477 sec 2024-03-24T02:45:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:48 smithi078 ceph-mon[32429]: pgmap v4607: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7823.216424 secs 2024-03-24T02:45:49.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:48 smithi107 ceph-mon[36485]: pgmap v4607: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:49.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7823.216424 secs 2024-03-24T02:45:49.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:48 smithi106 ceph-mon[36505]: pgmap v4607: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:49.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7823.216424 secs 2024-03-24T02:45:49.399 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8478 sec 2024-03-24T02:45:50.399 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8479 sec 2024-03-24T02:45:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:50 smithi078 ceph-mon[32429]: pgmap v4608: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:51.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:50 smithi107 ceph-mon[36485]: pgmap v4608: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:51.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:50 smithi106 ceph-mon[36505]: pgmap v4608: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:51.399 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8480 sec 2024-03-24T02:45:52.399 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8481 sec 2024-03-24T02:45:53.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:52 smithi078 ceph-mon[32429]: pgmap v4609: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:53.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:52 smithi107 ceph-mon[36485]: pgmap v4609: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:53.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:52 smithi106 ceph-mon[36505]: pgmap v4609: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:53.399 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8482 sec 2024-03-24T02:45:53.994 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7828.216485 secs 2024-03-24T02:45:54.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7828.216485 secs 2024-03-24T02:45:54.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7828.216485 secs 2024-03-24T02:45:54.399 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8483 sec 2024-03-24T02:45:55.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:54 smithi078 ceph-mon[32429]: pgmap v4610: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:55.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:54 smithi107 ceph-mon[36485]: pgmap v4610: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:55.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:54 smithi106 ceph-mon[36505]: pgmap v4610: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:55.400 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8484 sec 2024-03-24T02:45:56.400 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8485 sec 2024-03-24T02:45:57.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:56 smithi078 ceph-mon[32429]: pgmap v4611: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:57.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:56 smithi107 ceph-mon[36485]: pgmap v4611: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:57.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:56 smithi106 ceph-mon[36505]: pgmap v4611: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:57.400 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8486 sec 2024-03-24T02:45:58.400 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8487 sec 2024-03-24T02:45:58.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:58 smithi078 ceph-mon[32429]: pgmap v4612: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:58.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:45:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7833.216544 secs 2024-03-24T02:45:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:58 smithi107 ceph-mon[36485]: pgmap v4612: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:45:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7833.216544 secs 2024-03-24T02:45:59.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:58 smithi106 ceph-mon[36505]: pgmap v4612: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:45:59.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:45:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7833.216544 secs 2024-03-24T02:45:59.400 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8488 sec 2024-03-24T02:46:00.400 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8489 sec 2024-03-24T02:46:01.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:00 smithi078 ceph-mon[32429]: pgmap v4613: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:01.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:00 smithi107 ceph-mon[36485]: pgmap v4613: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:01.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:00 smithi106 ceph-mon[36505]: pgmap v4613: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:01.400 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8490 sec 2024-03-24T02:46:02.400 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8491 sec 2024-03-24T02:46:03.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:02 smithi078 ceph-mon[32429]: pgmap v4614: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:03.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:02 smithi107 ceph-mon[36485]: pgmap v4614: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:03.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:02 smithi106 ceph-mon[36505]: pgmap v4614: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:03.400 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8492 sec 2024-03-24T02:46:04.052 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7838.216605 secs 2024-03-24T02:46:04.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7838.216605 secs 2024-03-24T02:46:04.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7838.216605 secs 2024-03-24T02:46:04.400 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8493 sec 2024-03-24T02:46:05.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:04 smithi078 ceph-mon[32429]: pgmap v4615: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:05.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:04 smithi107 ceph-mon[36485]: pgmap v4615: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:05.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:04 smithi106 ceph-mon[36505]: pgmap v4615: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:05.400 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8494 sec 2024-03-24T02:46:06.401 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8495 sec 2024-03-24T02:46:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:06 smithi078 ceph-mon[32429]: pgmap v4616: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:07.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:06 smithi107 ceph-mon[36485]: pgmap v4616: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:07.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:06 smithi106 ceph-mon[36505]: pgmap v4616: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:07.401 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8496 sec 2024-03-24T02:46:08.401 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8497 sec 2024-03-24T02:46:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:08 smithi078 ceph-mon[32429]: pgmap v4617: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7843.216716 secs 2024-03-24T02:46:09.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:08 smithi107 ceph-mon[36485]: pgmap v4617: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:09.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7843.216716 secs 2024-03-24T02:46:09.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:08 smithi106 ceph-mon[36505]: pgmap v4617: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:09.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7843.216716 secs 2024-03-24T02:46:09.401 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8498 sec 2024-03-24T02:46:10.401 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8499 sec 2024-03-24T02:46:11.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:10 smithi078 ceph-mon[32429]: pgmap v4618: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:11.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:10 smithi107 ceph-mon[36485]: pgmap v4618: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:11.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:10 smithi106 ceph-mon[36505]: pgmap v4618: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:11.401 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8500 sec 2024-03-24T02:46:12.401 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8501 sec 2024-03-24T02:46:13.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:12 smithi078 ceph-mon[32429]: pgmap v4619: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:13.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:12 smithi107 ceph-mon[36485]: pgmap v4619: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:13.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:12 smithi106 ceph-mon[36505]: pgmap v4619: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:13.401 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8502 sec 2024-03-24T02:46:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7848.216757 secs 2024-03-24T02:46:14.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7848.216757 secs 2024-03-24T02:46:14.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7848.216757 secs 2024-03-24T02:46:14.401 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8503 sec 2024-03-24T02:46:15.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:14 smithi078 ceph-mon[32429]: pgmap v4620: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:15.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:14 smithi107 ceph-mon[36485]: pgmap v4620: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:15.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:14 smithi106 ceph-mon[36505]: pgmap v4620: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:15.401 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8504 sec 2024-03-24T02:46:16.401 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8505 sec 2024-03-24T02:46:17.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:16 smithi078 ceph-mon[32429]: pgmap v4621: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:17.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:16 smithi107 ceph-mon[36485]: pgmap v4621: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:17.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:16 smithi106 ceph-mon[36505]: pgmap v4621: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:17.401 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8506 sec 2024-03-24T02:46:18.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:17 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:46:18.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:17 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:46:18.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:17 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:46:18.401 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8507 sec 2024-03-24T02:46:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:18 smithi078 ceph-mon[32429]: pgmap v4622: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7853.216847 secs 2024-03-24T02:46:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:18 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:46:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:18 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:46:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:18 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:46:19.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:18 smithi107 ceph-mon[36485]: pgmap v4622: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:19.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7853.216847 secs 2024-03-24T02:46:19.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:18 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:46:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:18 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:46:19.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:18 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:46:19.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:18 smithi106 ceph-mon[36505]: pgmap v4622: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:19.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7853.216847 secs 2024-03-24T02:46:19.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:18 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:46:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:18 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:46:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:18 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:46:19.401 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8508 sec 2024-03-24T02:46:20.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:19 smithi078 ceph-mon[32429]: pgmap v4623: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:46:20.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:19 smithi078 ceph-mon[32429]: pgmap v4624: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:46:20.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:19 smithi107 ceph-mon[36485]: pgmap v4623: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:46:20.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:19 smithi107 ceph-mon[36485]: pgmap v4624: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:46:20.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:19 smithi106 ceph-mon[36505]: pgmap v4623: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:46:20.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:19 smithi106 ceph-mon[36505]: pgmap v4624: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:46:20.402 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8509 sec 2024-03-24T02:46:21.402 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8510 sec 2024-03-24T02:46:22.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:21 smithi078 ceph-mon[32429]: pgmap v4625: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:46:22.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:21 smithi107 ceph-mon[36485]: pgmap v4625: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:46:22.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:21 smithi106 ceph-mon[36505]: pgmap v4625: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:46:22.402 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8511 sec 2024-03-24T02:46:23.402 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8512 sec 2024-03-24T02:46:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:23 smithi078 ceph-mon[32429]: pgmap v4626: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:46:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7858.216937 secs 2024-03-24T02:46:24.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:23 smithi107 ceph-mon[36485]: pgmap v4626: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:46:24.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7858.216937 secs 2024-03-24T02:46:24.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:23 smithi106 ceph-mon[36505]: pgmap v4626: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:46:24.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7858.216937 secs 2024-03-24T02:46:24.402 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8513 sec 2024-03-24T02:46:25.402 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8514 sec 2024-03-24T02:46:26.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:25 smithi078 ceph-mon[32429]: pgmap v4627: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:46:26.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:25 smithi107 ceph-mon[36485]: pgmap v4627: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:46:26.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:25 smithi106 ceph-mon[36505]: pgmap v4627: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:46:26.402 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8515 sec 2024-03-24T02:46:27.402 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8516 sec 2024-03-24T02:46:28.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:27 smithi078 ceph-mon[32429]: pgmap v4628: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:46:28.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:27 smithi107 ceph-mon[36485]: pgmap v4628: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:46:28.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:27 smithi106 ceph-mon[36505]: pgmap v4628: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:46:28.402 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8517 sec 2024-03-24T02:46:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7863.217022 secs 2024-03-24T02:46:29.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7863.217022 secs 2024-03-24T02:46:29.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7863.217022 secs 2024-03-24T02:46:29.402 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8518 sec 2024-03-24T02:46:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:29 smithi078 ceph-mon[32429]: pgmap v4629: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:46:30.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:29 smithi107 ceph-mon[36485]: pgmap v4629: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:46:30.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:29 smithi106 ceph-mon[36505]: pgmap v4629: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:46:30.402 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8519 sec 2024-03-24T02:46:31.402 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8520 sec 2024-03-24T02:46:32.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:31 smithi078 ceph-mon[32429]: pgmap v4630: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:46:32.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:31 smithi107 ceph-mon[36485]: pgmap v4630: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:46:32.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:31 smithi106 ceph-mon[36505]: pgmap v4630: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:46:32.402 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8521 sec 2024-03-24T02:46:33.402 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8522 sec 2024-03-24T02:46:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:33 smithi078 ceph-mon[32429]: pgmap v4631: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:46:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7868.217084 secs 2024-03-24T02:46:34.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:33 smithi107 ceph-mon[36485]: pgmap v4631: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:46:34.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7868.217084 secs 2024-03-24T02:46:34.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:33 smithi106 ceph-mon[36505]: pgmap v4631: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:46:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7868.217084 secs 2024-03-24T02:46:34.403 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8523 sec 2024-03-24T02:46:35.403 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8524 sec 2024-03-24T02:46:36.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:35 smithi078 ceph-mon[32429]: pgmap v4632: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:46:36.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:35 smithi107 ceph-mon[36485]: pgmap v4632: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:46:36.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:35 smithi106 ceph-mon[36505]: pgmap v4632: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:46:36.403 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8525 sec 2024-03-24T02:46:37.403 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8526 sec 2024-03-24T02:46:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:37 smithi078 ceph-mon[32429]: pgmap v4633: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:46:38.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:37 smithi107 ceph-mon[36485]: pgmap v4633: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:46:38.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:37 smithi106 ceph-mon[36505]: pgmap v4633: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:46:38.403 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8527 sec 2024-03-24T02:46:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7873.217170 secs 2024-03-24T02:46:39.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7873.217170 secs 2024-03-24T02:46:39.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7873.217170 secs 2024-03-24T02:46:39.403 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8528 sec 2024-03-24T02:46:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:39 smithi078 ceph-mon[32429]: pgmap v4634: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:40.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:39 smithi107 ceph-mon[36485]: pgmap v4634: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:40.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:39 smithi106 ceph-mon[36505]: pgmap v4634: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:40.403 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8529 sec 2024-03-24T02:46:41.403 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8530 sec 2024-03-24T02:46:42.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:41 smithi078 ceph-mon[32429]: pgmap v4635: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:42.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:41 smithi107 ceph-mon[36485]: pgmap v4635: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:42.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:41 smithi106 ceph-mon[36505]: pgmap v4635: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:42.403 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8531 sec 2024-03-24T02:46:43.403 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8532 sec 2024-03-24T02:46:44.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:43 smithi107 ceph-mon[36485]: pgmap v4636: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:44.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7878.217230 secs 2024-03-24T02:46:44.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:43 smithi106 ceph-mon[36505]: pgmap v4636: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7878.217230 secs 2024-03-24T02:46:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:43 smithi078 ceph-mon[32429]: pgmap v4636: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7878.217230 secs 2024-03-24T02:46:44.403 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8533 sec 2024-03-24T02:46:45.404 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8534 sec 2024-03-24T02:46:46.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:45 smithi078 ceph-mon[32429]: pgmap v4637: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:46.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:45 smithi106 ceph-mon[36505]: pgmap v4637: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:46.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:45 smithi107 ceph-mon[36485]: pgmap v4637: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:46.404 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8535 sec 2024-03-24T02:46:47.404 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8536 sec 2024-03-24T02:46:48.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:47 smithi106 ceph-mon[36505]: pgmap v4638: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:48.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:47 smithi078 ceph-mon[32429]: pgmap v4638: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:48.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:47 smithi107 ceph-mon[36485]: pgmap v4638: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:48.404 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8537 sec 2024-03-24T02:46:49.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7883.217318 secs 2024-03-24T02:46:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7883.217318 secs 2024-03-24T02:46:49.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7883.217318 secs 2024-03-24T02:46:49.404 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8538 sec 2024-03-24T02:46:50.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:49 smithi106 ceph-mon[36505]: pgmap v4639: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:50.339 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:49 smithi078 ceph-mon[32429]: pgmap v4639: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:50.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:49 smithi107 ceph-mon[36485]: pgmap v4639: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:50.404 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8539 sec 2024-03-24T02:46:51.404 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8540 sec 2024-03-24T02:46:52.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:51 smithi106 ceph-mon[36505]: pgmap v4640: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:51 smithi078 ceph-mon[32429]: pgmap v4640: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:52.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:51 smithi107 ceph-mon[36485]: pgmap v4640: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:52.404 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8541 sec 2024-03-24T02:46:53.404 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8542 sec 2024-03-24T02:46:54.144 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:53 smithi107 ceph-mon[36485]: pgmap v4641: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:54.145 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7888.217408 secs 2024-03-24T02:46:54.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:53 smithi106 ceph-mon[36505]: pgmap v4641: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:54.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7888.217408 secs 2024-03-24T02:46:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:53 smithi078 ceph-mon[32429]: pgmap v4641: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7888.217408 secs 2024-03-24T02:46:54.404 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8543 sec 2024-03-24T02:46:55.405 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8544 sec 2024-03-24T02:46:56.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:55 smithi106 ceph-mon[36505]: pgmap v4642: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:56.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:55 smithi078 ceph-mon[32429]: pgmap v4642: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:56.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:55 smithi107 ceph-mon[36485]: pgmap v4642: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:46:56.405 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8545 sec 2024-03-24T02:46:57.405 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8546 sec 2024-03-24T02:46:58.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:57 smithi106 ceph-mon[36505]: pgmap v4643: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:58.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:57 smithi078 ceph-mon[32429]: pgmap v4643: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:58.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:57 smithi107 ceph-mon[36485]: pgmap v4643: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:58.405 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8547 sec 2024-03-24T02:46:59.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7893.217463 secs 2024-03-24T02:46:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7893.217463 secs 2024-03-24T02:46:59.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7893.217463 secs 2024-03-24T02:46:59.405 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8548 sec 2024-03-24T02:47:00.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:46:59 smithi106 ceph-mon[36505]: pgmap v4644: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:00.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:46:59 smithi078 ceph-mon[32429]: pgmap v4644: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:00.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:46:59 smithi107 ceph-mon[36485]: pgmap v4644: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:00.405 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8549 sec 2024-03-24T02:47:01.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:00 smithi106 ceph-mon[36505]: pgmap v4645: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:01.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:00 smithi078 ceph-mon[32429]: pgmap v4645: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:01.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:00 smithi107 ceph-mon[36485]: pgmap v4645: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:01.405 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8550 sec 2024-03-24T02:47:02.405 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8551 sec 2024-03-24T02:47:03.405 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8552 sec 2024-03-24T02:47:03.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:03 smithi078 ceph-mon[32429]: pgmap v4646: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:03.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:03 smithi107 ceph-mon[36485]: pgmap v4646: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:03.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:03 smithi106 ceph-mon[36505]: pgmap v4646: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:04.405 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8553 sec 2024-03-24T02:47:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7898.217475 secs 2024-03-24T02:47:04.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7898.217475 secs 2024-03-24T02:47:04.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7898.217475 secs 2024-03-24T02:47:05.406 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8554 sec 2024-03-24T02:47:05.802 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:05 smithi106 ceph-mon[36505]: pgmap v4647: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:05.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:05 smithi078 ceph-mon[32429]: pgmap v4647: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:05.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:05 smithi107 ceph-mon[36485]: pgmap v4647: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:06.406 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8555 sec 2024-03-24T02:47:07.406 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8556 sec 2024-03-24T02:47:07.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:07 smithi078 ceph-mon[32429]: pgmap v4648: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:07.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:07 smithi107 ceph-mon[36485]: pgmap v4648: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:07.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:07 smithi106 ceph-mon[36505]: pgmap v4648: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:08.406 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8557 sec 2024-03-24T02:47:09.406 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8558 sec 2024-03-24T02:47:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7903.217511 secs 2024-03-24T02:47:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:09 smithi078 ceph-mon[32429]: pgmap v4649: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:09.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7903.217511 secs 2024-03-24T02:47:09.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:09 smithi107 ceph-mon[36485]: pgmap v4649: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:09.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7903.217511 secs 2024-03-24T02:47:09.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:09 smithi106 ceph-mon[36505]: pgmap v4649: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:10.406 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8559 sec 2024-03-24T02:47:11.406 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8560 sec 2024-03-24T02:47:11.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:11 smithi078 ceph-mon[32429]: pgmap v4650: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:11.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:11 smithi107 ceph-mon[36485]: pgmap v4650: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:11.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:11 smithi106 ceph-mon[36505]: pgmap v4650: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:12.406 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8561 sec 2024-03-24T02:47:13.406 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8562 sec 2024-03-24T02:47:13.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:13 smithi078 ceph-mon[32429]: pgmap v4651: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:13.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:13 smithi107 ceph-mon[36485]: pgmap v4651: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:13.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:13 smithi106 ceph-mon[36505]: pgmap v4651: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:14.406 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8563 sec 2024-03-24T02:47:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7908.217523 secs 2024-03-24T02:47:14.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7908.217523 secs 2024-03-24T02:47:14.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7908.217523 secs 2024-03-24T02:47:15.407 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8564 sec 2024-03-24T02:47:15.836 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:15 smithi078 ceph-mon[32429]: pgmap v4652: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:15.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:15 smithi107 ceph-mon[36485]: pgmap v4652: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:15.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:15 smithi106 ceph-mon[36505]: pgmap v4652: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:16.407 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8565 sec 2024-03-24T02:47:17.407 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8566 sec 2024-03-24T02:47:17.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:17 smithi078 ceph-mon[32429]: pgmap v4653: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:17.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:17 smithi107 ceph-mon[36485]: pgmap v4653: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:17.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:17 smithi106 ceph-mon[36505]: pgmap v4653: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:18.407 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8567 sec 2024-03-24T02:47:19.407 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8568 sec 2024-03-24T02:47:19.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7913.217562 secs 2024-03-24T02:47:19.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:19 smithi078 ceph-mon[32429]: pgmap v4654: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:19.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:19 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:47:19.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:19 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:47:19.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:19 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:47:19.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:19 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:47:19.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7913.217562 secs 2024-03-24T02:47:19.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:19 smithi107 ceph-mon[36485]: pgmap v4654: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:19.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:19 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:47:19.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:19 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:47:19.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:19 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:47:19.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:19 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:47:19.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7913.217562 secs 2024-03-24T02:47:19.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:19 smithi106 ceph-mon[36505]: pgmap v4654: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:19.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:19 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:47:19.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:19 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:47:19.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:19 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:47:19.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:19 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:47:20.407 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8569 sec 2024-03-24T02:47:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:20 smithi078 ceph-mon[32429]: pgmap v4655: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:47:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:20 smithi078 ceph-mon[32429]: pgmap v4656: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:47:20.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:20 smithi107 ceph-mon[36485]: pgmap v4655: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:47:20.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:20 smithi107 ceph-mon[36485]: pgmap v4656: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:47:20.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:20 smithi106 ceph-mon[36505]: pgmap v4655: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:47:20.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:20 smithi106 ceph-mon[36505]: pgmap v4656: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:47:21.407 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8570 sec 2024-03-24T02:47:22.407 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8571 sec 2024-03-24T02:47:22.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:22 smithi078 ceph-mon[32429]: pgmap v4657: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:47:22.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:22 smithi107 ceph-mon[36485]: pgmap v4657: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:47:22.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:22 smithi106 ceph-mon[36505]: pgmap v4657: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:47:23.407 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8572 sec 2024-03-24T02:47:24.407 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8573 sec 2024-03-24T02:47:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7918.217620 secs 2024-03-24T02:47:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:24 smithi078 ceph-mon[32429]: pgmap v4658: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:47:24.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7918.217620 secs 2024-03-24T02:47:24.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:24 smithi107 ceph-mon[36485]: pgmap v4658: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:47:24.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7918.217620 secs 2024-03-24T02:47:24.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:24 smithi106 ceph-mon[36505]: pgmap v4658: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:47:25.407 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8574 sec 2024-03-24T02:47:26.407 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8575 sec 2024-03-24T02:47:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:26 smithi078 ceph-mon[32429]: pgmap v4659: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:47:26.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:26 smithi107 ceph-mon[36485]: pgmap v4659: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:47:26.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:26 smithi106 ceph-mon[36505]: pgmap v4659: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:47:27.407 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8576 sec 2024-03-24T02:47:28.407 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8577 sec 2024-03-24T02:47:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:28 smithi078 ceph-mon[32429]: pgmap v4660: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:47:28.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:28 smithi107 ceph-mon[36485]: pgmap v4660: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:47:28.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:28 smithi106 ceph-mon[36505]: pgmap v4660: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:47:29.408 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8578 sec 2024-03-24T02:47:29.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7923.217707 secs 2024-03-24T02:47:29.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7923.217707 secs 2024-03-24T02:47:29.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7923.217707 secs 2024-03-24T02:47:30.408 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8579 sec 2024-03-24T02:47:30.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:30 smithi078 ceph-mon[32429]: pgmap v4661: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:47:30.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:30 smithi107 ceph-mon[36485]: pgmap v4661: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:47:30.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:30 smithi106 ceph-mon[36505]: pgmap v4661: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:47:31.408 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8580 sec 2024-03-24T02:47:32.408 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8581 sec 2024-03-24T02:47:32.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:32 smithi078 ceph-mon[32429]: pgmap v4662: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:32.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:32 smithi107 ceph-mon[36485]: pgmap v4662: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:32.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:32 smithi106 ceph-mon[36505]: pgmap v4662: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:33.408 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8582 sec 2024-03-24T02:47:34.408 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8583 sec 2024-03-24T02:47:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7928.217765 secs 2024-03-24T02:47:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:34 smithi078 ceph-mon[32429]: pgmap v4663: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:34.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7928.217765 secs 2024-03-24T02:47:34.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:34 smithi107 ceph-mon[36485]: pgmap v4663: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:34.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7928.217765 secs 2024-03-24T02:47:34.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:34 smithi106 ceph-mon[36505]: pgmap v4663: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:35.408 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8584 sec 2024-03-24T02:47:36.408 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8585 sec 2024-03-24T02:47:36.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:36 smithi078 ceph-mon[32429]: pgmap v4664: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:36.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:36 smithi107 ceph-mon[36485]: pgmap v4664: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:36.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:36 smithi106 ceph-mon[36505]: pgmap v4664: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:37.408 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8586 sec 2024-03-24T02:47:38.408 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8587 sec 2024-03-24T02:47:38.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:38 smithi078 ceph-mon[32429]: pgmap v4665: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:38.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:38 smithi107 ceph-mon[36485]: pgmap v4665: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:38.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:38 smithi106 ceph-mon[36505]: pgmap v4665: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:39.408 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8588 sec 2024-03-24T02:47:39.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7933.217852 secs 2024-03-24T02:47:39.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7933.217852 secs 2024-03-24T02:47:39.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7933.217852 secs 2024-03-24T02:47:40.408 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8589 sec 2024-03-24T02:47:40.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:40 smithi078 ceph-mon[32429]: pgmap v4666: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:40.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:40 smithi107 ceph-mon[36485]: pgmap v4666: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:40.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:40 smithi106 ceph-mon[36505]: pgmap v4666: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:41.409 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8590 sec 2024-03-24T02:47:42.409 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8591 sec 2024-03-24T02:47:42.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:42 smithi078 ceph-mon[32429]: pgmap v4667: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:42.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:42 smithi107 ceph-mon[36485]: pgmap v4667: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:42.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:42 smithi106 ceph-mon[36505]: pgmap v4667: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:43.409 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8592 sec 2024-03-24T02:47:44.409 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8593 sec 2024-03-24T02:47:44.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7938.217894 secs 2024-03-24T02:47:44.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:44 smithi078 ceph-mon[32429]: pgmap v4668: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:44.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7938.217894 secs 2024-03-24T02:47:44.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:44 smithi107 ceph-mon[36485]: pgmap v4668: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:44.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7938.217894 secs 2024-03-24T02:47:44.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:44 smithi106 ceph-mon[36505]: pgmap v4668: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:45.409 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8594 sec 2024-03-24T02:47:46.409 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8595 sec 2024-03-24T02:47:46.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:46 smithi078 ceph-mon[32429]: pgmap v4669: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:46.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:46 smithi107 ceph-mon[36485]: pgmap v4669: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:46.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:46 smithi106 ceph-mon[36505]: pgmap v4669: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:47.409 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8596 sec 2024-03-24T02:47:48.409 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8597 sec 2024-03-24T02:47:48.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:48 smithi078 ceph-mon[32429]: pgmap v4670: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:48.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:48 smithi107 ceph-mon[36485]: pgmap v4670: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:48.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:48 smithi106 ceph-mon[36505]: pgmap v4670: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:49.409 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8598 sec 2024-03-24T02:47:49.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7943.217984 secs 2024-03-24T02:47:49.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7943.217984 secs 2024-03-24T02:47:49.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7943.217984 secs 2024-03-24T02:47:50.410 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8599 sec 2024-03-24T02:47:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:50 smithi078 ceph-mon[32429]: pgmap v4671: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:50.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:50 smithi107 ceph-mon[36485]: pgmap v4671: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:50.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:50 smithi106 ceph-mon[36505]: pgmap v4671: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:51.410 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8600 sec 2024-03-24T02:47:52.411 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8601 sec 2024-03-24T02:47:52.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:52 smithi078 ceph-mon[32429]: pgmap v4672: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:52.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:52 smithi107 ceph-mon[36485]: pgmap v4672: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:52.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:52 smithi106 ceph-mon[36505]: pgmap v4672: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:53.410 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8602 sec 2024-03-24T02:47:54.410 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8603 sec 2024-03-24T02:47:54.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7948.218074 secs 2024-03-24T02:47:54.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:54 smithi078 ceph-mon[32429]: pgmap v4673: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:54.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7948.218074 secs 2024-03-24T02:47:54.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:54 smithi107 ceph-mon[36485]: pgmap v4673: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7948.218074 secs 2024-03-24T02:47:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:54 smithi106 ceph-mon[36505]: pgmap v4673: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:55.410 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8604 sec 2024-03-24T02:47:56.410 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8605 sec 2024-03-24T02:47:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:56 smithi078 ceph-mon[32429]: pgmap v4674: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:56.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:56 smithi107 ceph-mon[36485]: pgmap v4674: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:56.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:56 smithi106 ceph-mon[36505]: pgmap v4674: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:47:57.410 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8606 sec 2024-03-24T02:47:58.410 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8607 sec 2024-03-24T02:47:58.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:58 smithi078 ceph-mon[32429]: pgmap v4675: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:58.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:58 smithi107 ceph-mon[36485]: pgmap v4675: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:58.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:58 smithi106 ceph-mon[36505]: pgmap v4675: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:59.410 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8608 sec 2024-03-24T02:47:59.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:47:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7953.218134 secs 2024-03-24T02:47:59.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:47:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7953.218134 secs 2024-03-24T02:48:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:47:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7953.218134 secs 2024-03-24T02:48:00.410 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8609 sec 2024-03-24T02:48:00.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:00 smithi106 ceph-mon[36505]: pgmap v4676: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:01.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:00 smithi078 ceph-mon[32429]: pgmap v4676: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:01.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:00 smithi107 ceph-mon[36485]: pgmap v4676: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:01.410 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8610 sec 2024-03-24T02:48:02.410 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8611 sec 2024-03-24T02:48:02.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:02 smithi106 ceph-mon[36505]: pgmap v4677: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:02.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:02 smithi078 ceph-mon[32429]: pgmap v4677: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:03.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:02 smithi107 ceph-mon[36485]: pgmap v4677: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:03.410 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8612 sec 2024-03-24T02:48:04.410 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8613 sec 2024-03-24T02:48:04.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7958.218151 secs 2024-03-24T02:48:04.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:04 smithi106 ceph-mon[36505]: pgmap v4678: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:05.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7958.218151 secs 2024-03-24T02:48:05.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:04 smithi078 ceph-mon[32429]: pgmap v4678: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:05.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7958.218151 secs 2024-03-24T02:48:05.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:04 smithi107 ceph-mon[36485]: pgmap v4678: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:05.411 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8614 sec 2024-03-24T02:48:06.411 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8615 sec 2024-03-24T02:48:06.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:06 smithi106 ceph-mon[36505]: pgmap v4679: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:06 smithi078 ceph-mon[32429]: pgmap v4679: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:07.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:06 smithi107 ceph-mon[36485]: pgmap v4679: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:07.411 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8616 sec 2024-03-24T02:48:08.411 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8617 sec 2024-03-24T02:48:08.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:08 smithi106 ceph-mon[36505]: pgmap v4680: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:08 smithi078 ceph-mon[32429]: pgmap v4680: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:09.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:08 smithi107 ceph-mon[36485]: pgmap v4680: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:09.411 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8618 sec 2024-03-24T02:48:09.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7963.218238 secs 2024-03-24T02:48:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7963.218238 secs 2024-03-24T02:48:10.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7963.218238 secs 2024-03-24T02:48:10.411 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8619 sec 2024-03-24T02:48:10.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:10 smithi106 ceph-mon[36505]: pgmap v4681: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:11.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:10 smithi078 ceph-mon[32429]: pgmap v4681: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:11.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:10 smithi107 ceph-mon[36485]: pgmap v4681: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:11.411 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8620 sec 2024-03-24T02:48:12.411 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8621 sec 2024-03-24T02:48:12.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:12 smithi106 ceph-mon[36505]: pgmap v4682: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:13.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:12 smithi078 ceph-mon[32429]: pgmap v4682: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:13.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:12 smithi107 ceph-mon[36485]: pgmap v4682: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:13.411 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8622 sec 2024-03-24T02:48:14.411 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8623 sec 2024-03-24T02:48:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7968.218338 secs 2024-03-24T02:48:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:14 smithi106 ceph-mon[36505]: pgmap v4683: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:15.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7968.218338 secs 2024-03-24T02:48:15.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:14 smithi078 ceph-mon[32429]: pgmap v4683: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:15.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7968.218338 secs 2024-03-24T02:48:15.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:14 smithi107 ceph-mon[36485]: pgmap v4683: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:15.411 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8624 sec 2024-03-24T02:48:16.412 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8625 sec 2024-03-24T02:48:16.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:16 smithi106 ceph-mon[36505]: pgmap v4684: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:17.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:16 smithi078 ceph-mon[32429]: pgmap v4684: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:17.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:16 smithi107 ceph-mon[36485]: pgmap v4684: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:17.412 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8626 sec 2024-03-24T02:48:18.412 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8627 sec 2024-03-24T02:48:18.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:18 smithi106 ceph-mon[36505]: pgmap v4685: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:18 smithi078 ceph-mon[32429]: pgmap v4685: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:19.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:18 smithi107 ceph-mon[36485]: pgmap v4685: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:19.412 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8628 sec 2024-03-24T02:48:19.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7973.218401 secs 2024-03-24T02:48:19.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:19 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:48:20.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7973.218401 secs 2024-03-24T02:48:20.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:19 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:48:20.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7973.218401 secs 2024-03-24T02:48:20.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:19 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:48:20.412 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8629 sec 2024-03-24T02:48:20.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:20 smithi106 ceph-mon[36505]: pgmap v4686: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:20.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:20 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:48:20.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:20 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:48:20.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:20 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:48:21.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:20 smithi078 ceph-mon[32429]: pgmap v4686: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:21.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:20 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:48:21.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:20 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:48:21.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:20 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:48:21.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:20 smithi107 ceph-mon[36485]: pgmap v4686: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:21.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:20 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:48:21.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:20 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:48:21.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:20 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:48:21.412 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8630 sec 2024-03-24T02:48:21.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:21 smithi106 ceph-mon[36505]: pgmap v4687: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:48:21.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:21 smithi106 ceph-mon[36505]: pgmap v4688: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:48:22.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:21 smithi078 ceph-mon[32429]: pgmap v4687: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:48:22.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:21 smithi078 ceph-mon[32429]: pgmap v4688: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:48:22.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:21 smithi107 ceph-mon[36485]: pgmap v4687: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:48:22.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:21 smithi107 ceph-mon[36485]: pgmap v4688: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:48:22.412 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8631 sec 2024-03-24T02:48:23.412 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8632 sec 2024-03-24T02:48:23.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:23 smithi106 ceph-mon[36505]: pgmap v4689: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:48:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:23 smithi078 ceph-mon[32429]: pgmap v4689: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:48:24.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:23 smithi107 ceph-mon[36485]: pgmap v4689: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:48:24.412 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8633 sec 2024-03-24T02:48:24.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7978.218480 secs 2024-03-24T02:48:24.993 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7978.218480 secs 2024-03-24T02:48:25.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7978.218480 secs 2024-03-24T02:48:25.412 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8634 sec 2024-03-24T02:48:25.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:25 smithi106 ceph-mon[36505]: pgmap v4690: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:48:26.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:25 smithi078 ceph-mon[32429]: pgmap v4690: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:48:26.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:25 smithi107 ceph-mon[36485]: pgmap v4690: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:48:26.412 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8635 sec 2024-03-24T02:48:27.412 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8636 sec 2024-03-24T02:48:27.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:27 smithi106 ceph-mon[36505]: pgmap v4691: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:48:28.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:27 smithi078 ceph-mon[32429]: pgmap v4691: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:48:28.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:27 smithi107 ceph-mon[36485]: pgmap v4691: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:48:28.413 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8637 sec 2024-03-24T02:48:29.413 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8638 sec 2024-03-24T02:48:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:29 smithi078 ceph-mon[32429]: pgmap v4692: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:48:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7983.218565 secs 2024-03-24T02:48:30.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:29 smithi107 ceph-mon[36485]: pgmap v4692: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:48:30.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7983.218565 secs 2024-03-24T02:48:30.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:29 smithi106 ceph-mon[36505]: pgmap v4692: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:48:30.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7983.218565 secs 2024-03-24T02:48:30.413 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8639 sec 2024-03-24T02:48:31.413 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8640 sec 2024-03-24T02:48:32.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:31 smithi078 ceph-mon[32429]: pgmap v4693: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:48:32.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:31 smithi107 ceph-mon[36485]: pgmap v4693: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:48:32.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:31 smithi106 ceph-mon[36505]: pgmap v4693: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:48:32.413 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8641 sec 2024-03-24T02:48:33.413 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8642 sec 2024-03-24T02:48:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:33 smithi078 ceph-mon[32429]: pgmap v4694: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:34.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:33 smithi107 ceph-mon[36485]: pgmap v4694: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:34.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:33 smithi106 ceph-mon[36505]: pgmap v4694: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:34.413 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8643 sec 2024-03-24T02:48:34.991 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7988.218651 secs 2024-03-24T02:48:35.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7988.218651 secs 2024-03-24T02:48:35.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7988.218651 secs 2024-03-24T02:48:35.413 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8644 sec 2024-03-24T02:48:36.052 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:35 smithi106 ceph-mon[36505]: pgmap v4695: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:36.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:35 smithi078 ceph-mon[32429]: pgmap v4695: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:36.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:35 smithi107 ceph-mon[36485]: pgmap v4695: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:36.413 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8645 sec 2024-03-24T02:48:37.413 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8646 sec 2024-03-24T02:48:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:37 smithi078 ceph-mon[32429]: pgmap v4696: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:38.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:37 smithi107 ceph-mon[36485]: pgmap v4696: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:38.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:37 smithi106 ceph-mon[36505]: pgmap v4696: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:38.413 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8647 sec 2024-03-24T02:48:39.413 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8648 sec 2024-03-24T02:48:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:39 smithi078 ceph-mon[32429]: pgmap v4697: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7993.218712 secs 2024-03-24T02:48:40.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:39 smithi107 ceph-mon[36485]: pgmap v4697: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:40.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7993.218712 secs 2024-03-24T02:48:40.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:39 smithi106 ceph-mon[36505]: pgmap v4697: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:40.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7993.218712 secs 2024-03-24T02:48:40.413 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8649 sec 2024-03-24T02:48:41.414 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8650 sec 2024-03-24T02:48:42.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:41 smithi078 ceph-mon[32429]: pgmap v4698: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:42.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:41 smithi107 ceph-mon[36485]: pgmap v4698: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:42.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:41 smithi106 ceph-mon[36505]: pgmap v4698: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:42.414 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8651 sec 2024-03-24T02:48:43.414 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8652 sec 2024-03-24T02:48:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:43 smithi078 ceph-mon[32429]: pgmap v4699: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:44.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:43 smithi107 ceph-mon[36485]: pgmap v4699: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:44.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:43 smithi106 ceph-mon[36505]: pgmap v4699: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:44.414 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8653 sec 2024-03-24T02:48:45.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 7998.218771 secs 2024-03-24T02:48:45.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 7998.218771 secs 2024-03-24T02:48:45.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 7998.218771 secs 2024-03-24T02:48:45.414 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8654 sec 2024-03-24T02:48:46.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:45 smithi078 ceph-mon[32429]: pgmap v4700: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:46.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:45 smithi107 ceph-mon[36485]: pgmap v4700: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:46.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:45 smithi106 ceph-mon[36505]: pgmap v4700: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:46.414 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8655 sec 2024-03-24T02:48:47.414 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8656 sec 2024-03-24T02:48:48.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:47 smithi078 ceph-mon[32429]: pgmap v4701: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:48.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:47 smithi107 ceph-mon[36485]: pgmap v4701: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:48.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:47 smithi106 ceph-mon[36505]: pgmap v4701: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:48.414 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8657 sec 2024-03-24T02:48:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8003.218831 secs 2024-03-24T02:48:49.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8003.218831 secs 2024-03-24T02:48:49.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8003.218831 secs 2024-03-24T02:48:49.415 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8658 sec 2024-03-24T02:48:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:49 smithi078 ceph-mon[32429]: pgmap v4702: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:50.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:49 smithi107 ceph-mon[36485]: pgmap v4702: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:50.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:49 smithi106 ceph-mon[36505]: pgmap v4702: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:50.414 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8659 sec 2024-03-24T02:48:51.414 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8660 sec 2024-03-24T02:48:52.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:51 smithi078 ceph-mon[32429]: pgmap v4703: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:52.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:51 smithi107 ceph-mon[36485]: pgmap v4703: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:52.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:51 smithi106 ceph-mon[36505]: pgmap v4703: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:52.415 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8661 sec 2024-03-24T02:48:53.415 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8662 sec 2024-03-24T02:48:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:53 smithi078 ceph-mon[32429]: pgmap v4704: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8008.218890 secs 2024-03-24T02:48:54.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:53 smithi107 ceph-mon[36485]: pgmap v4704: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:54.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8008.218890 secs 2024-03-24T02:48:54.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:53 smithi106 ceph-mon[36505]: pgmap v4704: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:54.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8008.218890 secs 2024-03-24T02:48:54.415 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8663 sec 2024-03-24T02:48:55.415 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8664 sec 2024-03-24T02:48:56.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:55 smithi078 ceph-mon[32429]: pgmap v4705: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:56.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:55 smithi107 ceph-mon[36485]: pgmap v4705: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:56.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:55 smithi106 ceph-mon[36505]: pgmap v4705: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:48:56.415 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8665 sec 2024-03-24T02:48:57.415 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8666 sec 2024-03-24T02:48:58.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:57 smithi078 ceph-mon[32429]: pgmap v4706: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:58.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:57 smithi107 ceph-mon[36485]: pgmap v4706: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:58.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:57 smithi106 ceph-mon[36505]: pgmap v4706: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:58.415 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8667 sec 2024-03-24T02:48:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8013.218975 secs 2024-03-24T02:48:59.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8013.218975 secs 2024-03-24T02:48:59.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8013.218975 secs 2024-03-24T02:48:59.415 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8668 sec 2024-03-24T02:49:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:48:59 smithi078 ceph-mon[32429]: pgmap v4707: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:00.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:48:59 smithi107 ceph-mon[36485]: pgmap v4707: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:00.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:48:59 smithi106 ceph-mon[36505]: pgmap v4707: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:00.415 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8669 sec 2024-03-24T02:49:01.416 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8670 sec 2024-03-24T02:49:02.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:01 smithi078 ceph-mon[32429]: pgmap v4708: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:02.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:01 smithi107 ceph-mon[36485]: pgmap v4708: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:02.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:01 smithi106 ceph-mon[36505]: pgmap v4708: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:02.416 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8671 sec 2024-03-24T02:49:03.416 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8672 sec 2024-03-24T02:49:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:03 smithi078 ceph-mon[32429]: pgmap v4709: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8018.219061 secs 2024-03-24T02:49:04.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:03 smithi107 ceph-mon[36485]: pgmap v4709: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:04.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8018.219061 secs 2024-03-24T02:49:04.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:03 smithi106 ceph-mon[36505]: pgmap v4709: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:04.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8018.219061 secs 2024-03-24T02:49:04.416 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8673 sec 2024-03-24T02:49:05.416 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8674 sec 2024-03-24T02:49:06.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:05 smithi078 ceph-mon[32429]: pgmap v4710: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:06.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:05 smithi107 ceph-mon[36485]: pgmap v4710: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:06.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:05 smithi106 ceph-mon[36505]: pgmap v4710: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:06.416 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8675 sec 2024-03-24T02:49:07.416 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8676 sec 2024-03-24T02:49:08.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:07 smithi078 ceph-mon[32429]: pgmap v4711: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:08.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:07 smithi107 ceph-mon[36485]: pgmap v4711: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:08.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:07 smithi106 ceph-mon[36505]: pgmap v4711: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:08.416 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8677 sec 2024-03-24T02:49:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8023.219124 secs 2024-03-24T02:49:09.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8023.219124 secs 2024-03-24T02:49:09.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8023.219124 secs 2024-03-24T02:49:09.416 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8678 sec 2024-03-24T02:49:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:09 smithi078 ceph-mon[32429]: pgmap v4712: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:10.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:09 smithi107 ceph-mon[36485]: pgmap v4712: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:10.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:09 smithi106 ceph-mon[36505]: pgmap v4712: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:10.416 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8679 sec 2024-03-24T02:49:11.417 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8680 sec 2024-03-24T02:49:12.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:11 smithi078 ceph-mon[32429]: pgmap v4713: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:12.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:11 smithi107 ceph-mon[36485]: pgmap v4713: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:12.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:11 smithi106 ceph-mon[36505]: pgmap v4713: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:12.416 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8681 sec 2024-03-24T02:49:13.417 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8682 sec 2024-03-24T02:49:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:13 smithi078 ceph-mon[32429]: pgmap v4714: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8028.219210 secs 2024-03-24T02:49:14.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:13 smithi107 ceph-mon[36485]: pgmap v4714: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:14.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8028.219210 secs 2024-03-24T02:49:14.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:13 smithi106 ceph-mon[36505]: pgmap v4714: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:14.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8028.219210 secs 2024-03-24T02:49:14.417 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8683 sec 2024-03-24T02:49:15.417 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8684 sec 2024-03-24T02:49:16.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:15 smithi078 ceph-mon[32429]: pgmap v4715: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:16.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:15 smithi107 ceph-mon[36485]: pgmap v4715: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:16.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:15 smithi106 ceph-mon[36505]: pgmap v4715: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:16.417 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8685 sec 2024-03-24T02:49:17.417 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8686 sec 2024-03-24T02:49:18.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:17 smithi078 ceph-mon[32429]: pgmap v4716: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:18.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:17 smithi107 ceph-mon[36485]: pgmap v4716: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:18.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:17 smithi106 ceph-mon[36505]: pgmap v4716: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:18.417 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8687 sec 2024-03-24T02:49:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8033.219301 secs 2024-03-24T02:49:19.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8033.219301 secs 2024-03-24T02:49:19.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8033.219301 secs 2024-03-24T02:49:19.417 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8688 sec 2024-03-24T02:49:20.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:19 smithi078 ceph-mon[32429]: pgmap v4717: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:20.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:19 smithi107 ceph-mon[36485]: pgmap v4717: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:20.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:19 smithi106 ceph-mon[36505]: pgmap v4717: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:20.417 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8689 sec 2024-03-24T02:49:21.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:20 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:49:21.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:20 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:49:21.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:20 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:49:21.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:20 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:49:21.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:20 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:49:21.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:20 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:49:21.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:20 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:49:21.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:20 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:49:21.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:20 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:49:21.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:20 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:49:21.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:20 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:49:21.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:20 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:49:21.417 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8690 sec 2024-03-24T02:49:22.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:21 smithi078 ceph-mon[32429]: pgmap v4718: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:22.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:21 smithi078 ceph-mon[32429]: pgmap v4719: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:49:22.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:21 smithi078 ceph-mon[32429]: pgmap v4720: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:49:22.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:21 smithi107 ceph-mon[36485]: pgmap v4718: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:22.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:21 smithi107 ceph-mon[36485]: pgmap v4719: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:49:22.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:21 smithi107 ceph-mon[36485]: pgmap v4720: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:49:22.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:21 smithi106 ceph-mon[36505]: pgmap v4718: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:22.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:21 smithi106 ceph-mon[36505]: pgmap v4719: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:49:22.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:21 smithi106 ceph-mon[36505]: pgmap v4720: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:49:22.418 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8691 sec 2024-03-24T02:49:23.418 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8692 sec 2024-03-24T02:49:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:23 smithi078 ceph-mon[32429]: pgmap v4721: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:49:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8038.219321 secs 2024-03-24T02:49:24.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:23 smithi107 ceph-mon[36485]: pgmap v4721: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:49:24.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8038.219321 secs 2024-03-24T02:49:24.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:23 smithi106 ceph-mon[36505]: pgmap v4721: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:49:24.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8038.219321 secs 2024-03-24T02:49:24.418 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8693 sec 2024-03-24T02:49:25.418 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8694 sec 2024-03-24T02:49:26.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:25 smithi078 ceph-mon[32429]: pgmap v4722: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:49:26.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:25 smithi107 ceph-mon[36485]: pgmap v4722: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:49:26.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:25 smithi106 ceph-mon[36505]: pgmap v4722: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:49:26.418 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8695 sec 2024-03-24T02:49:27.418 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8696 sec 2024-03-24T02:49:28.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:27 smithi078 ceph-mon[32429]: pgmap v4723: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:49:28.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:27 smithi107 ceph-mon[36485]: pgmap v4723: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:49:28.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:27 smithi106 ceph-mon[36505]: pgmap v4723: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:49:28.418 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8697 sec 2024-03-24T02:49:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8043.219407 secs 2024-03-24T02:49:29.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8043.219407 secs 2024-03-24T02:49:29.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8043.219407 secs 2024-03-24T02:49:29.418 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8698 sec 2024-03-24T02:49:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:29 smithi078 ceph-mon[32429]: pgmap v4724: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:49:30.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:29 smithi107 ceph-mon[36485]: pgmap v4724: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:49:30.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:29 smithi106 ceph-mon[36505]: pgmap v4724: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:49:30.419 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8699 sec 2024-03-24T02:49:31.418 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8700 sec 2024-03-24T02:49:32.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:31 smithi078 ceph-mon[32429]: pgmap v4725: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:49:32.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:31 smithi107 ceph-mon[36485]: pgmap v4725: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:49:32.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:31 smithi106 ceph-mon[36505]: pgmap v4725: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:49:32.418 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8701 sec 2024-03-24T02:49:33.418 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8702 sec 2024-03-24T02:49:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:33 smithi078 ceph-mon[32429]: pgmap v4726: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8048.219494 secs 2024-03-24T02:49:34.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:33 smithi107 ceph-mon[36485]: pgmap v4726: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:34.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8048.219494 secs 2024-03-24T02:49:34.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:33 smithi106 ceph-mon[36505]: pgmap v4726: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:34.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8048.219494 secs 2024-03-24T02:49:34.418 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8703 sec 2024-03-24T02:49:35.419 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8704 sec 2024-03-24T02:49:36.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:35 smithi078 ceph-mon[32429]: pgmap v4727: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:36.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:35 smithi107 ceph-mon[36485]: pgmap v4727: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:36.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:35 smithi106 ceph-mon[36505]: pgmap v4727: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:36.419 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8705 sec 2024-03-24T02:49:37.419 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8706 sec 2024-03-24T02:49:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:37 smithi078 ceph-mon[32429]: pgmap v4728: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:38.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:37 smithi107 ceph-mon[36485]: pgmap v4728: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:38.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:37 smithi106 ceph-mon[36505]: pgmap v4728: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:38.419 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8707 sec 2024-03-24T02:49:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8053.219581 secs 2024-03-24T02:49:39.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8053.219581 secs 2024-03-24T02:49:39.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8053.219581 secs 2024-03-24T02:49:39.419 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8708 sec 2024-03-24T02:49:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:39 smithi078 ceph-mon[32429]: pgmap v4729: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:40.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:39 smithi107 ceph-mon[36485]: pgmap v4729: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:40.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:39 smithi106 ceph-mon[36505]: pgmap v4729: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:40.419 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8709 sec 2024-03-24T02:49:41.419 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8710 sec 2024-03-24T02:49:42.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:41 smithi078 ceph-mon[32429]: pgmap v4730: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:42.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:41 smithi107 ceph-mon[36485]: pgmap v4730: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:42.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:41 smithi106 ceph-mon[36505]: pgmap v4730: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:42.419 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8711 sec 2024-03-24T02:49:43.419 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8712 sec 2024-03-24T02:49:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:43 smithi078 ceph-mon[32429]: pgmap v4731: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8058.219620 secs 2024-03-24T02:49:44.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:43 smithi107 ceph-mon[36485]: pgmap v4731: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:44.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8058.219620 secs 2024-03-24T02:49:44.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:43 smithi106 ceph-mon[36505]: pgmap v4731: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:44.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8058.219620 secs 2024-03-24T02:49:44.419 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8713 sec 2024-03-24T02:49:45.419 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8714 sec 2024-03-24T02:49:46.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:45 smithi078 ceph-mon[32429]: pgmap v4732: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:46.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:45 smithi107 ceph-mon[36485]: pgmap v4732: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:46.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:45 smithi106 ceph-mon[36505]: pgmap v4732: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:46.420 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8715 sec 2024-03-24T02:49:47.420 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8716 sec 2024-03-24T02:49:48.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:47 smithi078 ceph-mon[32429]: pgmap v4733: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:48.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:47 smithi107 ceph-mon[36485]: pgmap v4733: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:48.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:47 smithi106 ceph-mon[36505]: pgmap v4733: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:48.420 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8717 sec 2024-03-24T02:49:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8063.219704 secs 2024-03-24T02:49:49.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8063.219704 secs 2024-03-24T02:49:49.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8063.219704 secs 2024-03-24T02:49:49.420 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8718 sec 2024-03-24T02:49:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:49 smithi078 ceph-mon[32429]: pgmap v4734: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:50.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:49 smithi107 ceph-mon[36485]: pgmap v4734: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:50.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:49 smithi106 ceph-mon[36505]: pgmap v4734: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:50.420 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8719 sec 2024-03-24T02:49:51.420 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8720 sec 2024-03-24T02:49:52.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:51 smithi078 ceph-mon[32429]: pgmap v4735: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:52.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:51 smithi107 ceph-mon[36485]: pgmap v4735: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:52.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:51 smithi106 ceph-mon[36505]: pgmap v4735: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:52.420 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8721 sec 2024-03-24T02:49:53.420 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8722 sec 2024-03-24T02:49:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:53 smithi078 ceph-mon[32429]: pgmap v4736: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8068.219766 secs 2024-03-24T02:49:54.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:53 smithi107 ceph-mon[36485]: pgmap v4736: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:54.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8068.219766 secs 2024-03-24T02:49:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:53 smithi106 ceph-mon[36505]: pgmap v4736: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8068.219766 secs 2024-03-24T02:49:54.420 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8723 sec 2024-03-24T02:49:55.420 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8724 sec 2024-03-24T02:49:56.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:55 smithi078 ceph-mon[32429]: pgmap v4737: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:56.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:55 smithi107 ceph-mon[36485]: pgmap v4737: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:56.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:55 smithi106 ceph-mon[36505]: pgmap v4737: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:49:56.420 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8725 sec 2024-03-24T02:49:57.420 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8726 sec 2024-03-24T02:49:58.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:57 smithi107 ceph-mon[36485]: pgmap v4738: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:58.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:57 smithi106 ceph-mon[36505]: pgmap v4738: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:58.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:57 smithi078 ceph-mon[32429]: pgmap v4738: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:58.420 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8727 sec 2024-03-24T02:49:59.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8073.219854 secs 2024-03-24T02:49:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8073.219854 secs 2024-03-24T02:49:59.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8073.219854 secs 2024-03-24T02:49:59.420 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8728 sec 2024-03-24T02:50:00.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:49:59 smithi106 ceph-mon[36505]: pgmap v4739: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:00.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:49:59 smithi078 ceph-mon[32429]: pgmap v4739: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:00.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:49:59 smithi107 ceph-mon[36485]: pgmap v4739: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:00.421 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8729 sec 2024-03-24T02:50:01.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:00 smithi106 ceph-mon[36505]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:50:01.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:00 smithi078 ceph-mon[32429]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:50:01.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:00 smithi107 ceph-mon[36485]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:50:01.421 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8730 sec 2024-03-24T02:50:02.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:01 smithi106 ceph-mon[36505]: pgmap v4740: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:02.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:01 smithi078 ceph-mon[32429]: pgmap v4740: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:02.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:01 smithi107 ceph-mon[36485]: pgmap v4740: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:02.421 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8731 sec 2024-03-24T02:50:03.421 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8732 sec 2024-03-24T02:50:04.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:03 smithi106 ceph-mon[36505]: pgmap v4741: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:04.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8078.219942 secs 2024-03-24T02:50:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:03 smithi078 ceph-mon[32429]: pgmap v4741: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8078.219942 secs 2024-03-24T02:50:04.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:03 smithi107 ceph-mon[36485]: pgmap v4741: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:04.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8078.219942 secs 2024-03-24T02:50:04.421 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8733 sec 2024-03-24T02:50:05.421 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8734 sec 2024-03-24T02:50:06.142 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:05 smithi078 ceph-mon[32429]: pgmap v4742: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:06.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:05 smithi106 ceph-mon[36505]: pgmap v4742: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:06.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:05 smithi107 ceph-mon[36485]: pgmap v4742: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:06.421 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8735 sec 2024-03-24T02:50:07.421 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8736 sec 2024-03-24T02:50:08.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:07 smithi106 ceph-mon[36505]: pgmap v4743: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:50:08.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:07 smithi078 ceph-mon[32429]: pgmap v4743: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:50:08.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:07 smithi107 ceph-mon[36485]: pgmap v4743: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:50:08.421 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8737 sec 2024-03-24T02:50:09.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8083.220004 secs 2024-03-24T02:50:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8083.220004 secs 2024-03-24T02:50:09.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8083.220004 secs 2024-03-24T02:50:09.421 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8738 sec 2024-03-24T02:50:10.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:09 smithi106 ceph-mon[36505]: pgmap v4744: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:50:10.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:09 smithi078 ceph-mon[32429]: pgmap v4744: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:50:10.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:09 smithi107 ceph-mon[36485]: pgmap v4744: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:50:10.421 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8739 sec 2024-03-24T02:50:11.422 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8740 sec 2024-03-24T02:50:12.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:11 smithi106 ceph-mon[36505]: pgmap v4745: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:50:12.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:11 smithi078 ceph-mon[32429]: pgmap v4745: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:50:12.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:11 smithi107 ceph-mon[36485]: pgmap v4745: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:50:12.422 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8741 sec 2024-03-24T02:50:13.422 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8742 sec 2024-03-24T02:50:14.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:13 smithi106 ceph-mon[36505]: pgmap v4746: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:50:14.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8088.220065 secs 2024-03-24T02:50:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:13 smithi078 ceph-mon[32429]: pgmap v4746: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:50:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8088.220065 secs 2024-03-24T02:50:14.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:13 smithi107 ceph-mon[36485]: pgmap v4746: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:50:14.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8088.220065 secs 2024-03-24T02:50:14.422 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8743 sec 2024-03-24T02:50:15.422 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8744 sec 2024-03-24T02:50:16.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:15 smithi106 ceph-mon[36505]: pgmap v4747: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:50:16.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:15 smithi078 ceph-mon[32429]: pgmap v4747: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:50:16.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:15 smithi107 ceph-mon[36485]: pgmap v4747: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:50:16.422 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8745 sec 2024-03-24T02:50:17.422 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8746 sec 2024-03-24T02:50:18.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:17 smithi106 ceph-mon[36505]: pgmap v4748: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:50:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:17 smithi078 ceph-mon[32429]: pgmap v4748: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:50:18.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:17 smithi107 ceph-mon[36485]: pgmap v4748: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:50:18.422 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8747 sec 2024-03-24T02:50:19.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8093.220108 secs 2024-03-24T02:50:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8093.220108 secs 2024-03-24T02:50:19.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8093.220108 secs 2024-03-24T02:50:19.422 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8748 sec 2024-03-24T02:50:20.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:19 smithi106 ceph-mon[36505]: pgmap v4749: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:20.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:19 smithi078 ceph-mon[32429]: pgmap v4749: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:20.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:19 smithi107 ceph-mon[36485]: pgmap v4749: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:20.422 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8749 sec 2024-03-24T02:50:21.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:20 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:50:21.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:20 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:50:21.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:20 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:50:21.422 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8750 sec 2024-03-24T02:50:22.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:21 smithi106 ceph-mon[36505]: pgmap v4750: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:22.250 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:21 smithi078 ceph-mon[32429]: pgmap v4750: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:22.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:21 smithi107 ceph-mon[36485]: pgmap v4750: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:22.422 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8751 sec 2024-03-24T02:50:23.423 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8752 sec 2024-03-24T02:50:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:23 smithi106 ceph-mon[36505]: pgmap v4751: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8098.220308 secs 2024-03-24T02:50:24.308 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:23 smithi078 ceph-mon[32429]: pgmap v4751: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:24.308 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8098.220308 secs 2024-03-24T02:50:24.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:23 smithi107 ceph-mon[36485]: pgmap v4751: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:24.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8098.220308 secs 2024-03-24T02:50:24.423 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8753 sec 2024-03-24T02:50:25.423 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8754 sec 2024-03-24T02:50:26.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:25 smithi106 ceph-mon[36505]: pgmap v4752: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:26.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:50:26.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:50:26.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:50:26.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:50:26.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:50:26.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:50:26.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:50:26.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:25 smithi078 ceph-mon[32429]: pgmap v4752: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:26.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:50:26.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:50:26.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:50:26.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:50:26.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:50:26.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:50:26.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:50:26.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:25 smithi107 ceph-mon[36485]: pgmap v4752: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:26.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:50:26.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:50:26.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:50:26.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:50:26.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:50:26.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:50:26.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:50:26.423 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8755 sec 2024-03-24T02:50:27.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:26 smithi078 ceph-mon[32429]: pgmap v4753: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-03-24T02:50:27.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:26 smithi078 ceph-mon[32429]: pgmap v4754: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T02:50:27.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:26 smithi107 ceph-mon[36485]: pgmap v4753: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-03-24T02:50:27.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:26 smithi107 ceph-mon[36485]: pgmap v4754: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T02:50:27.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:26 smithi106 ceph-mon[36505]: pgmap v4753: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-03-24T02:50:27.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:26 smithi106 ceph-mon[36505]: pgmap v4754: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T02:50:27.423 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8756 sec 2024-03-24T02:50:28.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:27 smithi078 ceph-mon[32429]: pgmap v4755: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:50:28.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:27 smithi107 ceph-mon[36485]: pgmap v4755: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:50:28.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:27 smithi106 ceph-mon[36505]: pgmap v4755: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:50:28.423 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8757 sec 2024-03-24T02:50:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8103.220394 secs 2024-03-24T02:50:29.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8103.220394 secs 2024-03-24T02:50:29.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8103.220394 secs 2024-03-24T02:50:29.423 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8758 sec 2024-03-24T02:50:30.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:29 smithi078 ceph-mon[32429]: pgmap v4756: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T02:50:30.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:29 smithi107 ceph-mon[36485]: pgmap v4756: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T02:50:30.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:29 smithi106 ceph-mon[36505]: pgmap v4756: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T02:50:30.423 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8759 sec 2024-03-24T02:50:31.423 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8760 sec 2024-03-24T02:50:32.424 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8761 sec 2024-03-24T02:50:32.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:32 smithi106 ceph-mon[36505]: pgmap v4757: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:50:32.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:32 smithi078 ceph-mon[32429]: pgmap v4757: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:50:32.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:32 smithi107 ceph-mon[36485]: pgmap v4757: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:50:33.424 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8762 sec 2024-03-24T02:50:34.424 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8763 sec 2024-03-24T02:50:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8108.220482 secs 2024-03-24T02:50:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:34 smithi106 ceph-mon[36505]: pgmap v4758: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:50:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8108.220482 secs 2024-03-24T02:50:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:34 smithi078 ceph-mon[32429]: pgmap v4758: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:50:34.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8108.220482 secs 2024-03-24T02:50:34.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:34 smithi107 ceph-mon[36485]: pgmap v4758: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:50:35.424 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8764 sec 2024-03-24T02:50:36.424 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8765 sec 2024-03-24T02:50:36.642 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:36 smithi078 ceph-mon[32429]: pgmap v4759: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:50:36.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:36 smithi106 ceph-mon[36505]: pgmap v4759: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:50:36.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:36 smithi107 ceph-mon[36485]: pgmap v4759: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:50:37.424 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8766 sec 2024-03-24T02:50:38.424 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8767 sec 2024-03-24T02:50:38.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:38 smithi106 ceph-mon[36505]: pgmap v4760: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:38.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:38 smithi078 ceph-mon[32429]: pgmap v4760: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:38.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:38 smithi107 ceph-mon[36485]: pgmap v4760: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:39.424 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8768 sec 2024-03-24T02:50:39.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8113.220567 secs 2024-03-24T02:50:39.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8113.220567 secs 2024-03-24T02:50:39.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8113.220567 secs 2024-03-24T02:50:40.424 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8769 sec 2024-03-24T02:50:40.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:40 smithi106 ceph-mon[36505]: pgmap v4761: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:40.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:40 smithi078 ceph-mon[32429]: pgmap v4761: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:40.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:40 smithi107 ceph-mon[36485]: pgmap v4761: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:41.424 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8770 sec 2024-03-24T02:50:42.425 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8771 sec 2024-03-24T02:50:42.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:42 smithi106 ceph-mon[36505]: pgmap v4762: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:42.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:42 smithi078 ceph-mon[32429]: pgmap v4762: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:42.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:42 smithi107 ceph-mon[36485]: pgmap v4762: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:43.425 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8772 sec 2024-03-24T02:50:44.425 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8773 sec 2024-03-24T02:50:44.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8118.220655 secs 2024-03-24T02:50:44.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:44 smithi106 ceph-mon[36505]: pgmap v4763: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8118.220655 secs 2024-03-24T02:50:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:44 smithi078 ceph-mon[32429]: pgmap v4763: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:44.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8118.220655 secs 2024-03-24T02:50:44.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:44 smithi107 ceph-mon[36485]: pgmap v4763: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:45.425 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8774 sec 2024-03-24T02:50:46.425 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8775 sec 2024-03-24T02:50:46.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:46 smithi106 ceph-mon[36505]: pgmap v4764: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:46.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:46 smithi078 ceph-mon[32429]: pgmap v4764: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:46.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:46 smithi107 ceph-mon[36485]: pgmap v4764: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:47.425 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8776 sec 2024-03-24T02:50:48.425 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8777 sec 2024-03-24T02:50:48.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:48 smithi078 ceph-mon[32429]: pgmap v4765: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:48.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:48 smithi107 ceph-mon[36485]: pgmap v4765: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:48 smithi106 ceph-mon[36505]: pgmap v4765: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:49.425 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8778 sec 2024-03-24T02:50:49.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8123.220745 secs 2024-03-24T02:50:49.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8123.220745 secs 2024-03-24T02:50:49.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8123.220745 secs 2024-03-24T02:50:50.425 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8779 sec 2024-03-24T02:50:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:50 smithi078 ceph-mon[32429]: pgmap v4766: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:50.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:50 smithi107 ceph-mon[36485]: pgmap v4766: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:50.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:50 smithi106 ceph-mon[36505]: pgmap v4766: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:51.426 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8780 sec 2024-03-24T02:50:52.425 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8781 sec 2024-03-24T02:50:52.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:52 smithi078 ceph-mon[32429]: pgmap v4767: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:52.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:52 smithi107 ceph-mon[36485]: pgmap v4767: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:52.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:52 smithi106 ceph-mon[36505]: pgmap v4767: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:53.425 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8782 sec 2024-03-24T02:50:54.425 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8783 sec 2024-03-24T02:50:54.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8128.220805 secs 2024-03-24T02:50:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:54 smithi106 ceph-mon[36505]: pgmap v4768: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:54.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8128.220805 secs 2024-03-24T02:50:54.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:54 smithi078 ceph-mon[32429]: pgmap v4768: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:54.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8128.220805 secs 2024-03-24T02:50:54.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:54 smithi107 ceph-mon[36485]: pgmap v4768: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:55.425 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8784 sec 2024-03-24T02:50:56.426 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8785 sec 2024-03-24T02:50:56.645 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:56 smithi107 ceph-mon[36485]: pgmap v4769: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:56.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:56 smithi106 ceph-mon[36505]: pgmap v4769: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:56.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:56 smithi078 ceph-mon[32429]: pgmap v4769: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:50:57.426 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8786 sec 2024-03-24T02:50:58.426 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8787 sec 2024-03-24T02:50:58.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:58 smithi106 ceph-mon[36505]: pgmap v4770: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:58.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:58 smithi078 ceph-mon[32429]: pgmap v4770: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:58.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:58 smithi107 ceph-mon[36485]: pgmap v4770: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:59.426 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8788 sec 2024-03-24T02:50:59.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:50:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8133.220864 secs 2024-03-24T02:50:59.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:50:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8133.220864 secs 2024-03-24T02:50:59.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:50:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8133.220864 secs 2024-03-24T02:51:00.426 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8789 sec 2024-03-24T02:51:00.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:00 smithi106 ceph-mon[36505]: pgmap v4771: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:00.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:00 smithi078 ceph-mon[32429]: pgmap v4771: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:00.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:00 smithi107 ceph-mon[36485]: pgmap v4771: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:01.426 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8790 sec 2024-03-24T02:51:02.426 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8791 sec 2024-03-24T02:51:02.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:02 smithi106 ceph-mon[36505]: pgmap v4772: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:02.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:02 smithi078 ceph-mon[32429]: pgmap v4772: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:02.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:02 smithi107 ceph-mon[36485]: pgmap v4772: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:03.426 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8792 sec 2024-03-24T02:51:04.426 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8793 sec 2024-03-24T02:51:04.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8138.220949 secs 2024-03-24T02:51:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:04 smithi106 ceph-mon[36505]: pgmap v4773: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8138.220949 secs 2024-03-24T02:51:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:04 smithi078 ceph-mon[32429]: pgmap v4773: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:04.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8138.220949 secs 2024-03-24T02:51:04.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:04 smithi107 ceph-mon[36485]: pgmap v4773: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:05.426 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8794 sec 2024-03-24T02:51:06.426 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8795 sec 2024-03-24T02:51:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:06 smithi078 ceph-mon[32429]: pgmap v4774: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:06.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:06 smithi107 ceph-mon[36485]: pgmap v4774: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:06.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:06 smithi106 ceph-mon[36505]: pgmap v4774: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:07.426 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8796 sec 2024-03-24T02:51:08.427 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8797 sec 2024-03-24T02:51:08.704 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:08 smithi106 ceph-mon[36505]: pgmap v4775: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:08 smithi078 ceph-mon[32429]: pgmap v4775: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:08.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:08 smithi107 ceph-mon[36485]: pgmap v4775: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:09.427 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8798 sec 2024-03-24T02:51:09.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8143.221013 secs 2024-03-24T02:51:09.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8143.221013 secs 2024-03-24T02:51:09.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8143.221013 secs 2024-03-24T02:51:10.427 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8799 sec 2024-03-24T02:51:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:10 smithi078 ceph-mon[32429]: pgmap v4776: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:10.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:10 smithi107 ceph-mon[36485]: pgmap v4776: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:10.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:10 smithi106 ceph-mon[36505]: pgmap v4776: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:11.427 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8800 sec 2024-03-24T02:51:12.427 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8801 sec 2024-03-24T02:51:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:12 smithi078 ceph-mon[32429]: pgmap v4777: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:12.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:12 smithi107 ceph-mon[36485]: pgmap v4777: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:12.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:12 smithi106 ceph-mon[36505]: pgmap v4777: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:13.427 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8802 sec 2024-03-24T02:51:14.427 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8803 sec 2024-03-24T02:51:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8148.221053 secs 2024-03-24T02:51:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:14 smithi078 ceph-mon[32429]: pgmap v4778: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:14.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8148.221053 secs 2024-03-24T02:51:14.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:14 smithi107 ceph-mon[36485]: pgmap v4778: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:14.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8148.221053 secs 2024-03-24T02:51:14.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:14 smithi106 ceph-mon[36505]: pgmap v4778: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:15.427 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8804 sec 2024-03-24T02:51:16.427 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8805 sec 2024-03-24T02:51:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:16 smithi078 ceph-mon[32429]: pgmap v4779: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:16.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:16 smithi107 ceph-mon[36485]: pgmap v4779: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:16.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:16 smithi106 ceph-mon[36505]: pgmap v4779: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:17.427 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8806 sec 2024-03-24T02:51:18.427 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8807 sec 2024-03-24T02:51:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:18 smithi078 ceph-mon[32429]: pgmap v4780: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:18.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:18 smithi107 ceph-mon[36485]: pgmap v4780: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:18 smithi106 ceph-mon[36505]: pgmap v4780: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:19.427 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8808 sec 2024-03-24T02:51:19.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8153.221136 secs 2024-03-24T02:51:19.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8153.221136 secs 2024-03-24T02:51:19.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8153.221136 secs 2024-03-24T02:51:20.427 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8809 sec 2024-03-24T02:51:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:20 smithi078 ceph-mon[32429]: pgmap v4781: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:20.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:20 smithi107 ceph-mon[36485]: pgmap v4781: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:20.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:20 smithi106 ceph-mon[36505]: pgmap v4781: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:21.427 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8810 sec 2024-03-24T02:51:22.428 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8811 sec 2024-03-24T02:51:22.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:22 smithi078 ceph-mon[32429]: pgmap v4782: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:22.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:22 smithi107 ceph-mon[36485]: pgmap v4782: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:22.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:22 smithi106 ceph-mon[36505]: pgmap v4782: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:23.428 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8812 sec 2024-03-24T02:51:24.428 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8813 sec 2024-03-24T02:51:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8158.221223 secs 2024-03-24T02:51:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:24 smithi078 ceph-mon[32429]: pgmap v4783: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:24.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8158.221223 secs 2024-03-24T02:51:24.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:24 smithi107 ceph-mon[36485]: pgmap v4783: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:24.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8158.221223 secs 2024-03-24T02:51:24.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:24 smithi106 ceph-mon[36505]: pgmap v4783: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:25.428 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8814 sec 2024-03-24T02:51:26.428 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8815 sec 2024-03-24T02:51:26.748 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:26 smithi107 ceph-mon[36485]: pgmap v4784: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:26.748 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:26 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:51:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:26 smithi078 ceph-mon[32429]: pgmap v4784: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:26 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:51:26.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:26 smithi106 ceph-mon[36505]: pgmap v4784: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:26.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:26 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:51:27.428 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8816 sec 2024-03-24T02:51:28.428 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8817 sec 2024-03-24T02:51:28.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:28 smithi107 ceph-mon[36485]: pgmap v4785: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:28 smithi078 ceph-mon[32429]: pgmap v4785: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:28.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:28 smithi106 ceph-mon[36505]: pgmap v4785: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:29.428 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8818 sec 2024-03-24T02:51:29.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8163.221313 secs 2024-03-24T02:51:29.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8163.221313 secs 2024-03-24T02:51:29.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8163.221313 secs 2024-03-24T02:51:30.428 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8819 sec 2024-03-24T02:51:31.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:30 smithi078 ceph-mon[32429]: pgmap v4786: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:31.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:30 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:51:31.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:30 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:51:31.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:30 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:51:31.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:30 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:51:31.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:30 smithi078 ceph-mon[32429]: pgmap v4787: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:51:31.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:30 smithi078 ceph-mon[32429]: pgmap v4788: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T02:51:31.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:30 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:51:31.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:30 smithi107 ceph-mon[36485]: pgmap v4786: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:31.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:30 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:51:31.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:30 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:51:31.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:30 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:51:31.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:30 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:51:31.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:30 smithi107 ceph-mon[36485]: pgmap v4787: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:51:31.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:30 smithi107 ceph-mon[36485]: pgmap v4788: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T02:51:31.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:30 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:51:31.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:30 smithi106 ceph-mon[36505]: pgmap v4786: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:31.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:30 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:51:31.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:30 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:51:31.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:30 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:51:31.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:30 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:51:31.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:30 smithi106 ceph-mon[36505]: pgmap v4787: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:51:31.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:30 smithi106 ceph-mon[36505]: pgmap v4788: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T02:51:31.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:30 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:51:31.428 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8820 sec 2024-03-24T02:51:32.428 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8821 sec 2024-03-24T02:51:33.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:32 smithi078 ceph-mon[32429]: pgmap v4789: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:51:33.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:32 smithi107 ceph-mon[36485]: pgmap v4789: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:51:33.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:32 smithi106 ceph-mon[36505]: pgmap v4789: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:51:33.428 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8822 sec 2024-03-24T02:51:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8168.221372 secs 2024-03-24T02:51:34.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8168.221372 secs 2024-03-24T02:51:34.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8168.221372 secs 2024-03-24T02:51:34.429 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8823 sec 2024-03-24T02:51:35.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:34 smithi078 ceph-mon[32429]: pgmap v4790: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T02:51:35.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:34 smithi107 ceph-mon[36485]: pgmap v4790: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T02:51:35.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:34 smithi106 ceph-mon[36505]: pgmap v4790: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T02:51:35.429 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8824 sec 2024-03-24T02:51:36.429 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8825 sec 2024-03-24T02:51:37.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:36 smithi078 ceph-mon[32429]: pgmap v4791: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:51:37.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:36 smithi107 ceph-mon[36485]: pgmap v4791: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:51:37.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:36 smithi106 ceph-mon[36505]: pgmap v4791: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:51:37.429 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8826 sec 2024-03-24T02:51:38.429 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8827 sec 2024-03-24T02:51:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:38 smithi078 ceph-mon[32429]: pgmap v4792: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:51:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8173.221429 secs 2024-03-24T02:51:39.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:38 smithi107 ceph-mon[36485]: pgmap v4792: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:51:39.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8173.221429 secs 2024-03-24T02:51:39.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:38 smithi106 ceph-mon[36505]: pgmap v4792: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:51:39.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8173.221429 secs 2024-03-24T02:51:39.429 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8828 sec 2024-03-24T02:51:40.429 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8829 sec 2024-03-24T02:51:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:40 smithi078 ceph-mon[32429]: pgmap v4793: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:51:41.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:40 smithi107 ceph-mon[36485]: pgmap v4793: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:51:41.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:40 smithi106 ceph-mon[36505]: pgmap v4793: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:51:41.429 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8830 sec 2024-03-24T02:51:42.429 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8831 sec 2024-03-24T02:51:43.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:42 smithi078 ceph-mon[32429]: pgmap v4794: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:43.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:42 smithi107 ceph-mon[36485]: pgmap v4794: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:43.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:42 smithi106 ceph-mon[36505]: pgmap v4794: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:43.429 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8832 sec 2024-03-24T02:51:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8178.221487 secs 2024-03-24T02:51:44.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8178.221487 secs 2024-03-24T02:51:44.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8178.221487 secs 2024-03-24T02:51:44.429 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8833 sec 2024-03-24T02:51:45.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:44 smithi078 ceph-mon[32429]: pgmap v4795: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:45.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:44 smithi107 ceph-mon[36485]: pgmap v4795: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:45.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:44 smithi106 ceph-mon[36505]: pgmap v4795: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:45.430 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8834 sec 2024-03-24T02:51:46.430 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8835 sec 2024-03-24T02:51:47.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:46 smithi078 ceph-mon[32429]: pgmap v4796: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:47.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:46 smithi107 ceph-mon[36485]: pgmap v4796: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:47.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:46 smithi106 ceph-mon[36505]: pgmap v4796: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:47.430 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8836 sec 2024-03-24T02:51:48.430 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8837 sec 2024-03-24T02:51:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:48 smithi078 ceph-mon[32429]: pgmap v4797: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8183.221572 secs 2024-03-24T02:51:49.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:48 smithi107 ceph-mon[36485]: pgmap v4797: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:49.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8183.221572 secs 2024-03-24T02:51:49.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:48 smithi106 ceph-mon[36505]: pgmap v4797: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:49.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8183.221572 secs 2024-03-24T02:51:49.430 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8838 sec 2024-03-24T02:51:50.430 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8839 sec 2024-03-24T02:51:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:50 smithi078 ceph-mon[32429]: pgmap v4798: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:51.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:50 smithi107 ceph-mon[36485]: pgmap v4798: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:51.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:50 smithi106 ceph-mon[36505]: pgmap v4798: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:51.430 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8840 sec 2024-03-24T02:51:52.430 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8841 sec 2024-03-24T02:51:52.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:52 smithi078 ceph-mon[32429]: pgmap v4799: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:52.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:52 smithi107 ceph-mon[36485]: pgmap v4799: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:52.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:52 smithi106 ceph-mon[36505]: pgmap v4799: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:53.430 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8842 sec 2024-03-24T02:51:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8188.221658 secs 2024-03-24T02:51:54.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8188.221658 secs 2024-03-24T02:51:54.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8188.221658 secs 2024-03-24T02:51:54.430 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8843 sec 2024-03-24T02:51:55.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:54 smithi078 ceph-mon[32429]: pgmap v4800: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:55.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:54 smithi107 ceph-mon[36485]: pgmap v4800: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:55.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:54 smithi106 ceph-mon[36505]: pgmap v4800: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:55.430 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8844 sec 2024-03-24T02:51:56.431 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8845 sec 2024-03-24T02:51:57.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:56 smithi078 ceph-mon[32429]: pgmap v4801: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:57.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:56 smithi107 ceph-mon[36485]: pgmap v4801: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:57.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:56 smithi106 ceph-mon[36505]: pgmap v4801: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:57.431 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8846 sec 2024-03-24T02:51:58.431 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8847 sec 2024-03-24T02:51:59.085 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:58 smithi107 ceph-mon[36485]: pgmap v4802: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:59.085 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:51:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8193.221700 secs 2024-03-24T02:51:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:58 smithi078 ceph-mon[32429]: pgmap v4802: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:51:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8193.221700 secs 2024-03-24T02:51:59.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:58 smithi106 ceph-mon[36505]: pgmap v4802: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:51:59.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:51:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8193.221700 secs 2024-03-24T02:51:59.431 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8848 sec 2024-03-24T02:52:00.431 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8849 sec 2024-03-24T02:52:01.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:00 smithi078 ceph-mon[32429]: pgmap v4803: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:01.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:00 smithi107 ceph-mon[36485]: pgmap v4803: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:01.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:00 smithi106 ceph-mon[36505]: pgmap v4803: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:01.431 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8850 sec 2024-03-24T02:52:02.431 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8851 sec 2024-03-24T02:52:03.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:02 smithi078 ceph-mon[32429]: pgmap v4804: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:03.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:02 smithi107 ceph-mon[36485]: pgmap v4804: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:03.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:02 smithi106 ceph-mon[36505]: pgmap v4804: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:03.431 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8852 sec 2024-03-24T02:52:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8198.221741 secs 2024-03-24T02:52:04.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8198.221741 secs 2024-03-24T02:52:04.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8198.221741 secs 2024-03-24T02:52:04.431 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8853 sec 2024-03-24T02:52:05.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:04 smithi078 ceph-mon[32429]: pgmap v4805: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:05.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:04 smithi107 ceph-mon[36485]: pgmap v4805: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:05.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:04 smithi106 ceph-mon[36505]: pgmap v4805: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:05.431 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8854 sec 2024-03-24T02:52:06.432 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8855 sec 2024-03-24T02:52:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:06 smithi078 ceph-mon[32429]: pgmap v4806: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:07.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:06 smithi107 ceph-mon[36485]: pgmap v4806: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:07.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:06 smithi106 ceph-mon[36505]: pgmap v4806: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:07.432 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8856 sec 2024-03-24T02:52:08.432 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8857 sec 2024-03-24T02:52:08.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:08 smithi107 ceph-mon[36485]: pgmap v4807: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:08.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8203.221800 secs 2024-03-24T02:52:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:08 smithi078 ceph-mon[32429]: pgmap v4807: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8203.221800 secs 2024-03-24T02:52:09.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:08 smithi106 ceph-mon[36505]: pgmap v4807: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:09.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8203.221800 secs 2024-03-24T02:52:09.432 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8858 sec 2024-03-24T02:52:10.432 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8859 sec 2024-03-24T02:52:11.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:10 smithi078 ceph-mon[32429]: pgmap v4808: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:11.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:10 smithi107 ceph-mon[36485]: pgmap v4808: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:11.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:10 smithi106 ceph-mon[36505]: pgmap v4808: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:11.432 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8860 sec 2024-03-24T02:52:12.432 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8861 sec 2024-03-24T02:52:13.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:12 smithi078 ceph-mon[32429]: pgmap v4809: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:13.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:12 smithi107 ceph-mon[36485]: pgmap v4809: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:13.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:12 smithi106 ceph-mon[36505]: pgmap v4809: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:13.432 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8862 sec 2024-03-24T02:52:14.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8208.221887 secs 2024-03-24T02:52:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8208.221887 secs 2024-03-24T02:52:14.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8208.221887 secs 2024-03-24T02:52:14.432 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8863 sec 2024-03-24T02:52:15.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:14 smithi106 ceph-mon[36505]: pgmap v4810: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:15.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:14 smithi078 ceph-mon[32429]: pgmap v4810: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:15.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:14 smithi107 ceph-mon[36485]: pgmap v4810: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:15.432 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8864 sec 2024-03-24T02:52:16.432 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8865 sec 2024-03-24T02:52:17.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:16 smithi106 ceph-mon[36505]: pgmap v4811: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:17.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:16 smithi078 ceph-mon[32429]: pgmap v4811: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:17.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:16 smithi107 ceph-mon[36485]: pgmap v4811: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:17.432 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8866 sec 2024-03-24T02:52:18.433 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8867 sec 2024-03-24T02:52:19.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:18 smithi106 ceph-mon[36505]: pgmap v4812: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8213.221929 secs 2024-03-24T02:52:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:18 smithi078 ceph-mon[32429]: pgmap v4812: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8213.221929 secs 2024-03-24T02:52:19.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:18 smithi107 ceph-mon[36485]: pgmap v4812: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:19.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8213.221929 secs 2024-03-24T02:52:19.433 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8868 sec 2024-03-24T02:52:20.433 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8869 sec 2024-03-24T02:52:21.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:20 smithi106 ceph-mon[36505]: pgmap v4813: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:21.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:20 smithi078 ceph-mon[32429]: pgmap v4813: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:21.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:20 smithi107 ceph-mon[36485]: pgmap v4813: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:21.433 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8870 sec 2024-03-24T02:52:22.433 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8871 sec 2024-03-24T02:52:23.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:22 smithi106 ceph-mon[36505]: pgmap v4814: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:23.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:22 smithi078 ceph-mon[32429]: pgmap v4814: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:23.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:22 smithi107 ceph-mon[36485]: pgmap v4814: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:23.433 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8872 sec 2024-03-24T02:52:24.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8218.221991 secs 2024-03-24T02:52:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8218.221991 secs 2024-03-24T02:52:24.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8218.221991 secs 2024-03-24T02:52:24.433 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8873 sec 2024-03-24T02:52:25.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:24 smithi106 ceph-mon[36505]: pgmap v4815: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:24 smithi078 ceph-mon[32429]: pgmap v4815: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:25.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:24 smithi107 ceph-mon[36485]: pgmap v4815: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:25.433 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8874 sec 2024-03-24T02:52:26.433 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8875 sec 2024-03-24T02:52:27.145 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:26 smithi107 ceph-mon[36485]: pgmap v4816: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:27.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:26 smithi106 ceph-mon[36505]: pgmap v4816: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:27.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:26 smithi078 ceph-mon[32429]: pgmap v4816: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:27.433 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8876 sec 2024-03-24T02:52:28.434 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8877 sec 2024-03-24T02:52:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:28 smithi078 ceph-mon[32429]: pgmap v4817: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8223.222075 secs 2024-03-24T02:52:29.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:28 smithi106 ceph-mon[36505]: pgmap v4817: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:29.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8223.222075 secs 2024-03-24T02:52:29.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:28 smithi107 ceph-mon[36485]: pgmap v4817: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8223.222075 secs 2024-03-24T02:52:29.434 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8878 sec 2024-03-24T02:52:30.434 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8879 sec 2024-03-24T02:52:31.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:30 smithi106 ceph-mon[36505]: pgmap v4818: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:31.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:30 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:52:31.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:30 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:52:31.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:30 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:52:31.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:30 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:52:31.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:30 smithi078 ceph-mon[32429]: pgmap v4818: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:31.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:30 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:52:31.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:30 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:52:31.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:30 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:52:31.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:30 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:52:31.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:30 smithi107 ceph-mon[36485]: pgmap v4818: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:31.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:30 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:52:31.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:30 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:52:31.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:30 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:52:31.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:30 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:52:31.434 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8880 sec 2024-03-24T02:52:32.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:31 smithi106 ceph-mon[36505]: pgmap v4819: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:52:32.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:31 smithi106 ceph-mon[36505]: pgmap v4820: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:52:32.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:31 smithi078 ceph-mon[32429]: pgmap v4819: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:52:32.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:31 smithi078 ceph-mon[32429]: pgmap v4820: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:52:32.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:31 smithi107 ceph-mon[36485]: pgmap v4819: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:52:32.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:31 smithi107 ceph-mon[36485]: pgmap v4820: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:52:32.434 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8881 sec 2024-03-24T02:52:33.434 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8882 sec 2024-03-24T02:52:34.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:33 smithi106 ceph-mon[36505]: pgmap v4821: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:52:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8228.222157 secs 2024-03-24T02:52:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:33 smithi078 ceph-mon[32429]: pgmap v4821: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:52:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8228.222157 secs 2024-03-24T02:52:34.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:33 smithi107 ceph-mon[36485]: pgmap v4821: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:52:34.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8228.222157 secs 2024-03-24T02:52:34.434 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8883 sec 2024-03-24T02:52:35.434 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8884 sec 2024-03-24T02:52:36.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:35 smithi106 ceph-mon[36505]: pgmap v4822: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:52:36.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:35 smithi078 ceph-mon[32429]: pgmap v4822: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:52:36.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:35 smithi107 ceph-mon[36485]: pgmap v4822: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:52:36.434 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8885 sec 2024-03-24T02:52:37.434 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8886 sec 2024-03-24T02:52:38.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:37 smithi106 ceph-mon[36505]: pgmap v4823: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:52:38.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:37 smithi078 ceph-mon[32429]: pgmap v4823: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:52:38.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:37 smithi107 ceph-mon[36485]: pgmap v4823: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:52:38.434 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8887 sec 2024-03-24T02:52:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8233.222239 secs 2024-03-24T02:52:39.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8233.222239 secs 2024-03-24T02:52:39.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8233.222239 secs 2024-03-24T02:52:39.435 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8888 sec 2024-03-24T02:52:40.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:39 smithi078 ceph-mon[32429]: pgmap v4824: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:52:40.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:39 smithi107 ceph-mon[36485]: pgmap v4824: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:52:40.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:39 smithi106 ceph-mon[36505]: pgmap v4824: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:52:40.435 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8889 sec 2024-03-24T02:52:41.435 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8890 sec 2024-03-24T02:52:42.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:41 smithi078 ceph-mon[32429]: pgmap v4825: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:52:42.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:41 smithi107 ceph-mon[36485]: pgmap v4825: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:52:42.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:41 smithi106 ceph-mon[36505]: pgmap v4825: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:52:42.435 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8891 sec 2024-03-24T02:52:43.435 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8892 sec 2024-03-24T02:52:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:43 smithi078 ceph-mon[32429]: pgmap v4826: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8238.222327 secs 2024-03-24T02:52:44.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:43 smithi107 ceph-mon[36485]: pgmap v4826: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:44.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8238.222327 secs 2024-03-24T02:52:44.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:43 smithi106 ceph-mon[36505]: pgmap v4826: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:44.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8238.222327 secs 2024-03-24T02:52:44.435 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8893 sec 2024-03-24T02:52:45.435 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8894 sec 2024-03-24T02:52:46.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:45 smithi078 ceph-mon[32429]: pgmap v4827: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:46.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:45 smithi107 ceph-mon[36485]: pgmap v4827: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:46.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:45 smithi106 ceph-mon[36505]: pgmap v4827: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:46.435 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8895 sec 2024-03-24T02:52:47.435 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8896 sec 2024-03-24T02:52:48.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:47 smithi078 ceph-mon[32429]: pgmap v4828: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:48.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:47 smithi107 ceph-mon[36485]: pgmap v4828: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:48.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:47 smithi106 ceph-mon[36505]: pgmap v4828: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:48.435 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8897 sec 2024-03-24T02:52:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8243.222411 secs 2024-03-24T02:52:49.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8243.222411 secs 2024-03-24T02:52:49.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8243.222411 secs 2024-03-24T02:52:49.435 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8898 sec 2024-03-24T02:52:50.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:49 smithi078 ceph-mon[32429]: pgmap v4829: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:50.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:49 smithi107 ceph-mon[36485]: pgmap v4829: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:50.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:49 smithi106 ceph-mon[36505]: pgmap v4829: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:50.435 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8899 sec 2024-03-24T02:52:51.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:50 smithi078 ceph-mon[32429]: pgmap v4830: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:51.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:50 smithi107 ceph-mon[36485]: pgmap v4830: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:51.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:50 smithi106 ceph-mon[36505]: pgmap v4830: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:51.435 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8900 sec 2024-03-24T02:52:52.436 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8901 sec 2024-03-24T02:52:53.436 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8902 sec 2024-03-24T02:52:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:53 smithi078 ceph-mon[32429]: pgmap v4831: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:54.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:53 smithi107 ceph-mon[36485]: pgmap v4831: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:54.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:53 smithi106 ceph-mon[36505]: pgmap v4831: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:54.436 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8903 sec 2024-03-24T02:52:55.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8248.222469 secs 2024-03-24T02:52:55.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8248.222469 secs 2024-03-24T02:52:55.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8248.222469 secs 2024-03-24T02:52:55.436 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8904 sec 2024-03-24T02:52:56.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:55 smithi078 ceph-mon[32429]: pgmap v4832: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:56.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:55 smithi107 ceph-mon[36485]: pgmap v4832: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:56.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:55 smithi106 ceph-mon[36505]: pgmap v4832: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:52:56.436 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8905 sec 2024-03-24T02:52:57.436 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8906 sec 2024-03-24T02:52:58.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:57 smithi078 ceph-mon[32429]: pgmap v4833: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:58.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:57 smithi107 ceph-mon[36485]: pgmap v4833: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:58.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:57 smithi106 ceph-mon[36505]: pgmap v4833: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:58.437 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8907 sec 2024-03-24T02:52:59.436 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8908 sec 2024-03-24T02:53:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8253.222552 secs 2024-03-24T02:53:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:52:59 smithi078 ceph-mon[32429]: pgmap v4834: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:00.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8253.222552 secs 2024-03-24T02:53:00.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:52:59 smithi107 ceph-mon[36485]: pgmap v4834: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:00.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8253.222552 secs 2024-03-24T02:53:00.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:52:59 smithi106 ceph-mon[36505]: pgmap v4834: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:00.438 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8909 sec 2024-03-24T02:53:01.437 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8910 sec 2024-03-24T02:53:02.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:01 smithi078 ceph-mon[32429]: pgmap v4835: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:02.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:01 smithi107 ceph-mon[36485]: pgmap v4835: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:02.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:01 smithi106 ceph-mon[36505]: pgmap v4835: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:02.437 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8911 sec 2024-03-24T02:53:03.437 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8912 sec 2024-03-24T02:53:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:03 smithi078 ceph-mon[32429]: pgmap v4836: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:04.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:03 smithi107 ceph-mon[36485]: pgmap v4836: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:04.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:03 smithi106 ceph-mon[36505]: pgmap v4836: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:04.437 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8913 sec 2024-03-24T02:53:05.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8258.222634 secs 2024-03-24T02:53:05.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8258.222634 secs 2024-03-24T02:53:05.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8258.222634 secs 2024-03-24T02:53:05.437 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8914 sec 2024-03-24T02:53:06.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:05 smithi078 ceph-mon[32429]: pgmap v4837: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:06.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:05 smithi107 ceph-mon[36485]: pgmap v4837: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:06.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:05 smithi106 ceph-mon[36505]: pgmap v4837: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:06.437 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8915 sec 2024-03-24T02:53:07.437 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8916 sec 2024-03-24T02:53:08.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:07 smithi078 ceph-mon[32429]: pgmap v4838: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:08.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:07 smithi107 ceph-mon[36485]: pgmap v4838: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:08.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:07 smithi106 ceph-mon[36505]: pgmap v4838: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:08.437 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8917 sec 2024-03-24T02:53:09.437 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8918 sec 2024-03-24T02:53:09.953 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8263.222721 secs 2024-03-24T02:53:09.954 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:09 smithi106 ceph-mon[36505]: pgmap v4839: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8263.222721 secs 2024-03-24T02:53:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:09 smithi078 ceph-mon[32429]: pgmap v4839: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:10.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8263.222721 secs 2024-03-24T02:53:10.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:09 smithi107 ceph-mon[36485]: pgmap v4839: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:10.437 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8919 sec 2024-03-24T02:53:11.437 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8920 sec 2024-03-24T02:53:12.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:11 smithi078 ceph-mon[32429]: pgmap v4840: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:12.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:11 smithi107 ceph-mon[36485]: pgmap v4840: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:12.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:11 smithi106 ceph-mon[36505]: pgmap v4840: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:12.437 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8921 sec 2024-03-24T02:53:13.437 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8922 sec 2024-03-24T02:53:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:13 smithi078 ceph-mon[32429]: pgmap v4841: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:14.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:13 smithi107 ceph-mon[36485]: pgmap v4841: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:14.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:13 smithi106 ceph-mon[36505]: pgmap v4841: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:14.437 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8923 sec 2024-03-24T02:53:14.991 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8268.222809 secs 2024-03-24T02:53:15.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8268.222809 secs 2024-03-24T02:53:15.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8268.222809 secs 2024-03-24T02:53:15.438 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8924 sec 2024-03-24T02:53:16.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:15 smithi078 ceph-mon[32429]: pgmap v4842: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:16.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:15 smithi107 ceph-mon[36485]: pgmap v4842: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:16.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:15 smithi106 ceph-mon[36505]: pgmap v4842: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:16.438 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8925 sec 2024-03-24T02:53:17.438 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8926 sec 2024-03-24T02:53:18.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:17 smithi078 ceph-mon[32429]: pgmap v4843: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:18.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:17 smithi107 ceph-mon[36485]: pgmap v4843: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:18.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:17 smithi106 ceph-mon[36505]: pgmap v4843: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:18.438 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8927 sec 2024-03-24T02:53:19.438 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8928 sec 2024-03-24T02:53:20.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8273.222868 secs 2024-03-24T02:53:20.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:19 smithi078 ceph-mon[32429]: pgmap v4844: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:20.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8273.222868 secs 2024-03-24T02:53:20.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:19 smithi107 ceph-mon[36485]: pgmap v4844: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:20.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8273.222868 secs 2024-03-24T02:53:20.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:19 smithi106 ceph-mon[36505]: pgmap v4844: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:20.438 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8929 sec 2024-03-24T02:53:21.438 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8930 sec 2024-03-24T02:53:22.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:21 smithi078 ceph-mon[32429]: pgmap v4845: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:22.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:21 smithi107 ceph-mon[36485]: pgmap v4845: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:22.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:21 smithi106 ceph-mon[36505]: pgmap v4845: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:22.438 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8931 sec 2024-03-24T02:53:23.438 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8932 sec 2024-03-24T02:53:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:23 smithi078 ceph-mon[32429]: pgmap v4846: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:24.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:23 smithi107 ceph-mon[36485]: pgmap v4846: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:24.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:23 smithi106 ceph-mon[36505]: pgmap v4846: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:24.438 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8933 sec 2024-03-24T02:53:25.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8278.222926 secs 2024-03-24T02:53:25.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8278.222926 secs 2024-03-24T02:53:25.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8278.222926 secs 2024-03-24T02:53:25.439 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8934 sec 2024-03-24T02:53:26.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:25 smithi078 ceph-mon[32429]: pgmap v4847: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:26.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:25 smithi107 ceph-mon[36485]: pgmap v4847: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:26.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:25 smithi106 ceph-mon[36505]: pgmap v4847: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:26.439 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8935 sec 2024-03-24T02:53:27.439 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8936 sec 2024-03-24T02:53:28.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:27 smithi078 ceph-mon[32429]: pgmap v4848: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:28.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:27 smithi107 ceph-mon[36485]: pgmap v4848: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:28.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:27 smithi106 ceph-mon[36505]: pgmap v4848: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:28.439 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8937 sec 2024-03-24T02:53:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8283.223007 secs 2024-03-24T02:53:29.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8283.223007 secs 2024-03-24T02:53:29.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8283.223007 secs 2024-03-24T02:53:29.442 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8938 sec 2024-03-24T02:53:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:29 smithi078 ceph-mon[32429]: pgmap v4849: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:30.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:29 smithi107 ceph-mon[36485]: pgmap v4849: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:30.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:29 smithi106 ceph-mon[36505]: pgmap v4849: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:30.439 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8939 sec 2024-03-24T02:53:31.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:30 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:53:31.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:30 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:53:31.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:30 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:53:31.439 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8940 sec 2024-03-24T02:53:32.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:31 smithi078 ceph-mon[32429]: pgmap v4850: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:32.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:31 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:53:32.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:31 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:53:32.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:31 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:53:32.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:31 smithi107 ceph-mon[36485]: pgmap v4850: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:32.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:31 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:53:32.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:31 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:53:32.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:31 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:53:32.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:31 smithi106 ceph-mon[36505]: pgmap v4850: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:32.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:31 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:53:32.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:31 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:53:32.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:31 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:53:32.439 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8941 sec 2024-03-24T02:53:33.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:32 smithi078 ceph-mon[32429]: pgmap v4851: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:53:33.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:32 smithi078 ceph-mon[32429]: pgmap v4852: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:53:33.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:32 smithi107 ceph-mon[36485]: pgmap v4851: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:53:33.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:32 smithi107 ceph-mon[36485]: pgmap v4852: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:53:33.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:32 smithi106 ceph-mon[36505]: pgmap v4851: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:53:33.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:32 smithi106 ceph-mon[36505]: pgmap v4852: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:53:33.439 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8942 sec 2024-03-24T02:53:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8288.223093 secs 2024-03-24T02:53:34.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8288.223093 secs 2024-03-24T02:53:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8288.223093 secs 2024-03-24T02:53:34.439 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8943 sec 2024-03-24T02:53:35.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:34 smithi078 ceph-mon[32429]: pgmap v4853: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:53:35.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:34 smithi107 ceph-mon[36485]: pgmap v4853: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:53:35.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:34 smithi106 ceph-mon[36505]: pgmap v4853: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:53:35.439 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8944 sec 2024-03-24T02:53:36.440 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8945 sec 2024-03-24T02:53:37.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:36 smithi078 ceph-mon[32429]: pgmap v4854: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:53:37.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:36 smithi107 ceph-mon[36485]: pgmap v4854: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:53:37.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:36 smithi106 ceph-mon[36505]: pgmap v4854: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:53:37.440 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8946 sec 2024-03-24T02:53:38.440 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8947 sec 2024-03-24T02:53:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:38 smithi078 ceph-mon[32429]: pgmap v4855: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:53:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8293.223184 secs 2024-03-24T02:53:39.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:38 smithi107 ceph-mon[36485]: pgmap v4855: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:53:39.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8293.223184 secs 2024-03-24T02:53:39.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:38 smithi106 ceph-mon[36505]: pgmap v4855: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:53:39.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8293.223184 secs 2024-03-24T02:53:39.440 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8948 sec 2024-03-24T02:53:40.440 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8949 sec 2024-03-24T02:53:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:40 smithi078 ceph-mon[32429]: pgmap v4856: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:53:41.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:40 smithi107 ceph-mon[36485]: pgmap v4856: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:53:41.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:40 smithi106 ceph-mon[36505]: pgmap v4856: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:53:41.440 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8950 sec 2024-03-24T02:53:42.440 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8951 sec 2024-03-24T02:53:43.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:42 smithi078 ceph-mon[32429]: pgmap v4857: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:53:43.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:42 smithi107 ceph-mon[36485]: pgmap v4857: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:53:43.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:42 smithi106 ceph-mon[36505]: pgmap v4857: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:53:43.440 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8952 sec 2024-03-24T02:53:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8298.223246 secs 2024-03-24T02:53:44.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8298.223246 secs 2024-03-24T02:53:44.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8298.223246 secs 2024-03-24T02:53:44.441 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8953 sec 2024-03-24T02:53:45.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:44 smithi078 ceph-mon[32429]: pgmap v4858: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:45.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:44 smithi107 ceph-mon[36485]: pgmap v4858: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:45.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:44 smithi106 ceph-mon[36505]: pgmap v4858: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:45.440 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8954 sec 2024-03-24T02:53:46.441 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8955 sec 2024-03-24T02:53:47.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:46 smithi078 ceph-mon[32429]: pgmap v4859: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:47.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:46 smithi107 ceph-mon[36485]: pgmap v4859: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:47.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:46 smithi106 ceph-mon[36505]: pgmap v4859: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:47.441 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8956 sec 2024-03-24T02:53:48.441 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8957 sec 2024-03-24T02:53:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:48 smithi078 ceph-mon[32429]: pgmap v4860: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8303.223323 secs 2024-03-24T02:53:49.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:48 smithi107 ceph-mon[36485]: pgmap v4860: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:49.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8303.223323 secs 2024-03-24T02:53:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:48 smithi106 ceph-mon[36505]: pgmap v4860: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8303.223323 secs 2024-03-24T02:53:49.441 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8958 sec 2024-03-24T02:53:50.441 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8959 sec 2024-03-24T02:53:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:50 smithi078 ceph-mon[32429]: pgmap v4861: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:51.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:50 smithi107 ceph-mon[36485]: pgmap v4861: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:51.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:50 smithi106 ceph-mon[36505]: pgmap v4861: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:51.441 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8960 sec 2024-03-24T02:53:52.441 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8961 sec 2024-03-24T02:53:53.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:52 smithi078 ceph-mon[32429]: pgmap v4862: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:53.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:52 smithi107 ceph-mon[36485]: pgmap v4862: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:53.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:52 smithi106 ceph-mon[36505]: pgmap v4862: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:53.441 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8962 sec 2024-03-24T02:53:54.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8308.223384 secs 2024-03-24T02:53:54.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8308.223384 secs 2024-03-24T02:53:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8308.223384 secs 2024-03-24T02:53:54.441 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8963 sec 2024-03-24T02:53:55.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:54 smithi107 ceph-mon[36485]: pgmap v4863: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:55.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:54 smithi106 ceph-mon[36505]: pgmap v4863: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:55.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:54 smithi078 ceph-mon[32429]: pgmap v4863: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:55.441 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8964 sec 2024-03-24T02:53:56.442 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8965 sec 2024-03-24T02:53:57.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:56 smithi106 ceph-mon[36505]: pgmap v4864: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:57.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:56 smithi078 ceph-mon[32429]: pgmap v4864: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:57.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:56 smithi107 ceph-mon[36485]: pgmap v4864: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:57.442 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8966 sec 2024-03-24T02:53:58.442 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8967 sec 2024-03-24T02:53:59.145 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:58 smithi107 ceph-mon[36485]: pgmap v4865: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:59.145 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:53:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8313.223447 secs 2024-03-24T02:53:59.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:58 smithi106 ceph-mon[36505]: pgmap v4865: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:53:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8313.223447 secs 2024-03-24T02:53:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:58 smithi078 ceph-mon[32429]: pgmap v4865: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:53:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:53:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8313.223447 secs 2024-03-24T02:53:59.442 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8968 sec 2024-03-24T02:54:00.442 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8969 sec 2024-03-24T02:54:01.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:00 smithi078 ceph-mon[32429]: pgmap v4866: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:01.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:00 smithi106 ceph-mon[36505]: pgmap v4866: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:00 smithi107 ceph-mon[36485]: pgmap v4866: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:01.442 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8970 sec 2024-03-24T02:54:02.442 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8971 sec 2024-03-24T02:54:03.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:02 smithi106 ceph-mon[36505]: pgmap v4867: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:03.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:02 smithi078 ceph-mon[32429]: pgmap v4867: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:03.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:02 smithi107 ceph-mon[36485]: pgmap v4867: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:03.442 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8972 sec 2024-03-24T02:54:04.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8318.223462 secs 2024-03-24T02:54:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8318.223462 secs 2024-03-24T02:54:04.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8318.223462 secs 2024-03-24T02:54:04.442 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8973 sec 2024-03-24T02:54:05.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:04 smithi106 ceph-mon[36505]: pgmap v4868: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:05.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:04 smithi078 ceph-mon[32429]: pgmap v4868: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:05.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:04 smithi107 ceph-mon[36485]: pgmap v4868: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:05.442 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8974 sec 2024-03-24T02:54:06.442 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8975 sec 2024-03-24T02:54:07.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:06 smithi106 ceph-mon[36505]: pgmap v4869: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:07.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:06 smithi078 ceph-mon[32429]: pgmap v4869: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:07.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:06 smithi107 ceph-mon[36485]: pgmap v4869: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:07.443 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8976 sec 2024-03-24T02:54:08.443 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8977 sec 2024-03-24T02:54:09.142 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:08 smithi078 ceph-mon[32429]: pgmap v4870: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:09.143 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8323.223501 secs 2024-03-24T02:54:09.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:08 smithi106 ceph-mon[36505]: pgmap v4870: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:09.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8323.223501 secs 2024-03-24T02:54:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:08 smithi107 ceph-mon[36485]: pgmap v4870: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8323.223501 secs 2024-03-24T02:54:09.443 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8978 sec 2024-03-24T02:54:10.443 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8979 sec 2024-03-24T02:54:11.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:10 smithi106 ceph-mon[36505]: pgmap v4871: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:11.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:10 smithi078 ceph-mon[32429]: pgmap v4871: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:11.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:10 smithi107 ceph-mon[36485]: pgmap v4871: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:11.443 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8980 sec 2024-03-24T02:54:12.443 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8981 sec 2024-03-24T02:54:13.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:12 smithi106 ceph-mon[36505]: pgmap v4872: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:13.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:12 smithi078 ceph-mon[32429]: pgmap v4872: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:13.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:12 smithi107 ceph-mon[36485]: pgmap v4872: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:13.443 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8982 sec 2024-03-24T02:54:14.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8328.223564 secs 2024-03-24T02:54:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8328.223564 secs 2024-03-24T02:54:14.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8328.223564 secs 2024-03-24T02:54:14.443 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8983 sec 2024-03-24T02:54:15.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:14 smithi106 ceph-mon[36505]: pgmap v4873: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:15.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:14 smithi078 ceph-mon[32429]: pgmap v4873: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:15.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:14 smithi107 ceph-mon[36485]: pgmap v4873: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:15.443 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8984 sec 2024-03-24T02:54:16.443 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8985 sec 2024-03-24T02:54:17.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:16 smithi078 ceph-mon[32429]: pgmap v4874: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:17.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:16 smithi106 ceph-mon[36505]: pgmap v4874: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:17.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:16 smithi107 ceph-mon[36485]: pgmap v4874: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:17.443 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8986 sec 2024-03-24T02:54:18.443 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8987 sec 2024-03-24T02:54:19.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:18 smithi106 ceph-mon[36505]: pgmap v4875: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:19.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8333.223621 secs 2024-03-24T02:54:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:18 smithi078 ceph-mon[32429]: pgmap v4875: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8333.223621 secs 2024-03-24T02:54:19.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:18 smithi107 ceph-mon[36485]: pgmap v4875: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8333.223621 secs 2024-03-24T02:54:19.443 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8988 sec 2024-03-24T02:54:20.444 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8989 sec 2024-03-24T02:54:21.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:20 smithi106 ceph-mon[36505]: pgmap v4876: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:21.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:20 smithi078 ceph-mon[32429]: pgmap v4876: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:21.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:20 smithi107 ceph-mon[36485]: pgmap v4876: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:21.443 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8990 sec 2024-03-24T02:54:22.444 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8991 sec 2024-03-24T02:54:23.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:22 smithi106 ceph-mon[36505]: pgmap v4877: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:23.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:22 smithi078 ceph-mon[32429]: pgmap v4877: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:23.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:22 smithi107 ceph-mon[36485]: pgmap v4877: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:23.444 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8992 sec 2024-03-24T02:54:24.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8338.223709 secs 2024-03-24T02:54:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8338.223709 secs 2024-03-24T02:54:24.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8338.223709 secs 2024-03-24T02:54:24.444 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8993 sec 2024-03-24T02:54:25.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:24 smithi106 ceph-mon[36505]: pgmap v4878: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:24 smithi078 ceph-mon[32429]: pgmap v4878: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:25.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:24 smithi107 ceph-mon[36485]: pgmap v4878: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:25.444 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8994 sec 2024-03-24T02:54:26.444 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8995 sec 2024-03-24T02:54:27.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:26 smithi106 ceph-mon[36505]: pgmap v4879: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:27.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:26 smithi078 ceph-mon[32429]: pgmap v4879: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:27.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:26 smithi107 ceph-mon[36485]: pgmap v4879: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:27.444 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8996 sec 2024-03-24T02:54:28.444 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8997 sec 2024-03-24T02:54:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:28 smithi078 ceph-mon[32429]: pgmap v4880: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8343.223773 secs 2024-03-24T02:54:29.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:28 smithi107 ceph-mon[36485]: pgmap v4880: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:29.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8343.223773 secs 2024-03-24T02:54:29.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:28 smithi106 ceph-mon[36505]: pgmap v4880: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8343.223773 secs 2024-03-24T02:54:29.444 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8998 sec 2024-03-24T02:54:30.445 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 8999 sec 2024-03-24T02:54:31.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:30 smithi078 ceph-mon[32429]: pgmap v4881: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:31.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:30 smithi107 ceph-mon[36485]: pgmap v4881: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:31.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:30 smithi106 ceph-mon[36505]: pgmap v4881: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:31.444 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9000 sec 2024-03-24T02:54:32.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:31 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:54:32.338 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:31 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:54:32.341 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:31 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:54:32.445 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9001 sec 2024-03-24T02:54:33.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:32 smithi078 ceph-mon[32429]: pgmap v4882: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:33.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:32 smithi107 ceph-mon[36485]: pgmap v4882: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:33.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:32 smithi106 ceph-mon[36505]: pgmap v4882: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:33.445 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9002 sec 2024-03-24T02:54:34.445 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9003 sec 2024-03-24T02:54:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8348.223839 secs 2024-03-24T02:54:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:34 smithi078 ceph-mon[32429]: pgmap v4883: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:34 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:34 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:34 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:34 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:34 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:34.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:34 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:34.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8348.223839 secs 2024-03-24T02:54:34.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:34 smithi107 ceph-mon[36485]: pgmap v4883: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:34 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:34 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:34 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:34 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:34 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:34 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8348.223839 secs 2024-03-24T02:54:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:34 smithi106 ceph-mon[36505]: pgmap v4883: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:34 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:34 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:34 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:34 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:34 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:34.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:34 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:35.445 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9004 sec 2024-03-24T02:54:35.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:35.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:35.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:35 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T02:54:35.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:35 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T02:54:35.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:35.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:35.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:35 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T02:54:35.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:35 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T02:54:35.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:35.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:35.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:35 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T02:54:35.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:35 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T02:54:35.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:54:35.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:54:35.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:35 smithi078 ceph-mon[32429]: pgmap v4884: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:35.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:35 smithi078 ceph-mon[32429]: pgmap v4885: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:54:35.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:35.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:35.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:35.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:35 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T02:54:35.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:35 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T02:54:35.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:35.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:35.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:35 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T02:54:35.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:35 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T02:54:35.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:35.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:35.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:35 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T02:54:35.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:35 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T02:54:35.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:54:35.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:54:35.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:35 smithi107 ceph-mon[36485]: pgmap v4884: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:35.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:35 smithi107 ceph-mon[36485]: pgmap v4885: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:54:35.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:35.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:35.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:35.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:35 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T02:54:35.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:35 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T02:54:35.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:35.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:35.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:35 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T02:54:35.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:35 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T02:54:35.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:35.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:35.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:54:35.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:35 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T02:54:35.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:35 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T02:54:35.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:54:35.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:54:35.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:35 smithi106 ceph-mon[36505]: pgmap v4884: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:35.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:35 smithi106 ceph-mon[36505]: pgmap v4885: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:54:35.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:54:36.445 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9005 sec 2024-03-24T02:54:37.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:36 smithi078 ceph-mon[32429]: pgmap v4886: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:54:37.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:36 smithi107 ceph-mon[36485]: pgmap v4886: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:54:37.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:36 smithi106 ceph-mon[36505]: pgmap v4886: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:54:37.445 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9006 sec 2024-03-24T02:54:38.445 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9007 sec 2024-03-24T02:54:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8353.223924 secs 2024-03-24T02:54:39.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8353.223924 secs 2024-03-24T02:54:39.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8353.223924 secs 2024-03-24T02:54:39.445 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9008 sec 2024-03-24T02:54:40.052 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:39 smithi106 ceph-mon[36505]: pgmap v4887: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:54:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:39 smithi078 ceph-mon[32429]: pgmap v4887: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:54:40.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:39 smithi107 ceph-mon[36485]: pgmap v4887: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:54:40.445 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9009 sec 2024-03-24T02:54:41.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:40 smithi078 ceph-mon[32429]: pgmap v4888: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:54:41.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:41 smithi107 ceph-mon[36485]: pgmap v4888: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:54:41.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:40 smithi106 ceph-mon[36505]: pgmap v4888: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:54:41.445 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9010 sec 2024-03-24T02:54:42.445 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9011 sec 2024-03-24T02:54:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:42 smithi078 ceph-mon[32429]: pgmap v4889: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:54:43.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:43 smithi107 ceph-mon[36485]: pgmap v4889: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:54:43.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:43 smithi106 ceph-mon[36505]: pgmap v4889: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:54:43.445 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9012 sec 2024-03-24T02:54:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8358.224013 secs 2024-03-24T02:54:44.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8358.224013 secs 2024-03-24T02:54:44.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8358.224013 secs 2024-03-24T02:54:44.445 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9013 sec 2024-03-24T02:54:45.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:45 smithi078 ceph-mon[32429]: pgmap v4890: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:54:45.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:45 smithi107 ceph-mon[36485]: pgmap v4890: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:54:45.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:45 smithi106 ceph-mon[36505]: pgmap v4890: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:54:45.445 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9014 sec 2024-03-24T02:54:46.446 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9015 sec 2024-03-24T02:54:47.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:47 smithi078 ceph-mon[32429]: pgmap v4891: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:47.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:47 smithi107 ceph-mon[36485]: pgmap v4891: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:47.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:47 smithi106 ceph-mon[36505]: pgmap v4891: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:47.446 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9016 sec 2024-03-24T02:54:48.446 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9017 sec 2024-03-24T02:54:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8363.224100 secs 2024-03-24T02:54:49.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8363.224100 secs 2024-03-24T02:54:49.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8363.224100 secs 2024-03-24T02:54:49.446 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9018 sec 2024-03-24T02:54:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:49 smithi078 ceph-mon[32429]: pgmap v4892: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:50.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:49 smithi107 ceph-mon[36485]: pgmap v4892: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:50.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:49 smithi106 ceph-mon[36505]: pgmap v4892: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:50.446 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9019 sec 2024-03-24T02:54:51.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:51 smithi078 ceph-mon[32429]: pgmap v4893: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:51.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:51 smithi107 ceph-mon[36485]: pgmap v4893: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:51.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:51 smithi106 ceph-mon[36505]: pgmap v4893: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:51.446 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9020 sec 2024-03-24T02:54:52.446 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9021 sec 2024-03-24T02:54:53.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:53 smithi078 ceph-mon[32429]: pgmap v4894: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:53.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:53 smithi107 ceph-mon[36485]: pgmap v4894: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:53.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:53 smithi106 ceph-mon[36505]: pgmap v4894: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:53.446 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9022 sec 2024-03-24T02:54:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8368.224161 secs 2024-03-24T02:54:54.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8368.224161 secs 2024-03-24T02:54:54.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8368.224161 secs 2024-03-24T02:54:54.446 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9023 sec 2024-03-24T02:54:55.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:55 smithi078 ceph-mon[32429]: pgmap v4895: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:55.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:55 smithi107 ceph-mon[36485]: pgmap v4895: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:55.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:55 smithi106 ceph-mon[36505]: pgmap v4895: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:54:55.446 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9024 sec 2024-03-24T02:54:56.446 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9025 sec 2024-03-24T02:54:57.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:57 smithi078 ceph-mon[32429]: pgmap v4896: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:57.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:57 smithi107 ceph-mon[36485]: pgmap v4896: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:57.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:57 smithi106 ceph-mon[36505]: pgmap v4896: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:57.446 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9026 sec 2024-03-24T02:54:58.447 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9027 sec 2024-03-24T02:54:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8373.224249 secs 2024-03-24T02:54:59.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8373.224249 secs 2024-03-24T02:54:59.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8373.224249 secs 2024-03-24T02:54:59.447 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9028 sec 2024-03-24T02:55:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:54:59 smithi078 ceph-mon[32429]: pgmap v4897: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:00.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:54:59 smithi107 ceph-mon[36485]: pgmap v4897: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:00.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:54:59 smithi106 ceph-mon[36505]: pgmap v4897: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:00.447 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9029 sec 2024-03-24T02:55:01.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:01 smithi078 ceph-mon[32429]: pgmap v4898: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:01.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:01 smithi107 ceph-mon[36485]: pgmap v4898: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:01.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:01 smithi106 ceph-mon[36505]: pgmap v4898: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:01.447 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9030 sec 2024-03-24T02:55:02.447 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9031 sec 2024-03-24T02:55:03.447 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9032 sec 2024-03-24T02:55:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:04 smithi078 ceph-mon[32429]: pgmap v4899: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8378.224316 secs 2024-03-24T02:55:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:04 smithi107 ceph-mon[36485]: pgmap v4899: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8378.224316 secs 2024-03-24T02:55:04.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:04 smithi106 ceph-mon[36505]: pgmap v4899: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8378.224316 secs 2024-03-24T02:55:04.447 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9033 sec 2024-03-24T02:55:05.447 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9034 sec 2024-03-24T02:55:06.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:06 smithi078 ceph-mon[32429]: pgmap v4900: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:06.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:06 smithi107 ceph-mon[36485]: pgmap v4900: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:06.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:06 smithi106 ceph-mon[36505]: pgmap v4900: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:06.448 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9035 sec 2024-03-24T02:55:07.448 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9036 sec 2024-03-24T02:55:08.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:08 smithi078 ceph-mon[32429]: pgmap v4901: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:08.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:08 smithi107 ceph-mon[36485]: pgmap v4901: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:08.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:08 smithi106 ceph-mon[36505]: pgmap v4901: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:08.448 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9037 sec 2024-03-24T02:55:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8383.224372 secs 2024-03-24T02:55:09.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8383.224372 secs 2024-03-24T02:55:09.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8383.224372 secs 2024-03-24T02:55:09.448 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9038 sec 2024-03-24T02:55:10.303 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:10 smithi106 ceph-mon[36505]: pgmap v4902: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:10.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:10 smithi078 ceph-mon[32429]: pgmap v4902: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:10.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:10 smithi107 ceph-mon[36485]: pgmap v4902: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:10.448 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9039 sec 2024-03-24T02:55:11.448 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9040 sec 2024-03-24T02:55:12.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:12 smithi078 ceph-mon[32429]: pgmap v4903: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:12.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:12 smithi107 ceph-mon[36485]: pgmap v4903: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:12.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:12 smithi106 ceph-mon[36505]: pgmap v4903: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:12.449 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9041 sec 2024-03-24T02:55:13.448 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9042 sec 2024-03-24T02:55:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:14 smithi078 ceph-mon[32429]: pgmap v4904: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8388.224433 secs 2024-03-24T02:55:14.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:14 smithi107 ceph-mon[36485]: pgmap v4904: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:14.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8388.224433 secs 2024-03-24T02:55:14.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:14 smithi106 ceph-mon[36505]: pgmap v4904: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8388.224433 secs 2024-03-24T02:55:14.448 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9043 sec 2024-03-24T02:55:15.448 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9044 sec 2024-03-24T02:55:16.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:16 smithi078 ceph-mon[32429]: pgmap v4905: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:16.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:16 smithi107 ceph-mon[36485]: pgmap v4905: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:16.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:16 smithi106 ceph-mon[36505]: pgmap v4905: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:16.448 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9045 sec 2024-03-24T02:55:17.449 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9046 sec 2024-03-24T02:55:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:18 smithi078 ceph-mon[32429]: pgmap v4906: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:18.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:18 smithi107 ceph-mon[36485]: pgmap v4906: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:18.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:18 smithi106 ceph-mon[36505]: pgmap v4906: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:18.449 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9047 sec 2024-03-24T02:55:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8393.224490 secs 2024-03-24T02:55:19.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8393.224490 secs 2024-03-24T02:55:19.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8393.224490 secs 2024-03-24T02:55:19.449 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9048 sec 2024-03-24T02:55:20.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:20 smithi078 ceph-mon[32429]: pgmap v4907: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:20.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:20 smithi107 ceph-mon[36485]: pgmap v4907: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:20.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:20 smithi106 ceph-mon[36505]: pgmap v4907: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:20.449 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9049 sec 2024-03-24T02:55:21.449 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9050 sec 2024-03-24T02:55:22.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:22 smithi078 ceph-mon[32429]: pgmap v4908: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:22.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:22 smithi107 ceph-mon[36485]: pgmap v4908: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:22.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:22 smithi106 ceph-mon[36505]: pgmap v4908: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:22.449 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9051 sec 2024-03-24T02:55:23.449 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9052 sec 2024-03-24T02:55:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:24 smithi078 ceph-mon[32429]: pgmap v4909: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8398.224536 secs 2024-03-24T02:55:24.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:24 smithi107 ceph-mon[36485]: pgmap v4909: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:24.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8398.224536 secs 2024-03-24T02:55:24.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:24 smithi106 ceph-mon[36505]: pgmap v4909: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:24.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8398.224536 secs 2024-03-24T02:55:24.449 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9053 sec 2024-03-24T02:55:25.449 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9054 sec 2024-03-24T02:55:26.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:26 smithi078 ceph-mon[32429]: pgmap v4910: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:26.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:26 smithi107 ceph-mon[36485]: pgmap v4910: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:26.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:26 smithi106 ceph-mon[36505]: pgmap v4910: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:26.449 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9055 sec 2024-03-24T02:55:27.450 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9056 sec 2024-03-24T02:55:28.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:28 smithi078 ceph-mon[32429]: pgmap v4911: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:28.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:28 smithi107 ceph-mon[36485]: pgmap v4911: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:28.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:28 smithi106 ceph-mon[36505]: pgmap v4911: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:28.449 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9057 sec 2024-03-24T02:55:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8403.224595 secs 2024-03-24T02:55:29.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8403.224595 secs 2024-03-24T02:55:29.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8403.224595 secs 2024-03-24T02:55:29.450 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9058 sec 2024-03-24T02:55:30.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:30 smithi078 ceph-mon[32429]: pgmap v4912: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:30.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:30 smithi107 ceph-mon[36485]: pgmap v4912: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:30.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:30 smithi106 ceph-mon[36505]: pgmap v4912: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:30.450 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9059 sec 2024-03-24T02:55:31.450 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9060 sec 2024-03-24T02:55:32.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:32 smithi078 ceph-mon[32429]: pgmap v4913: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:32.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:32 smithi107 ceph-mon[36485]: pgmap v4913: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:32.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:32 smithi106 ceph-mon[36505]: pgmap v4913: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:32.450 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9061 sec 2024-03-24T02:55:33.450 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9062 sec 2024-03-24T02:55:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:34 smithi078 ceph-mon[32429]: pgmap v4914: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8408.224678 secs 2024-03-24T02:55:34.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:34 smithi107 ceph-mon[36485]: pgmap v4914: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8408.224678 secs 2024-03-24T02:55:34.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:34 smithi106 ceph-mon[36505]: pgmap v4914: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:34.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8408.224678 secs 2024-03-24T02:55:34.450 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9063 sec 2024-03-24T02:55:35.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:35 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:55:35.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:35 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:55:35.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:35 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:55:35.450 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9064 sec 2024-03-24T02:55:36.451 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9065 sec 2024-03-24T02:55:36.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:36 smithi078 ceph-mon[32429]: pgmap v4915: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:36.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:55:36.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:55:36.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:55:36.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:55:36.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:55:36.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:55:36.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:55:36.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:55:36.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:55:36.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:36 smithi107 ceph-mon[36485]: pgmap v4915: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:55:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:55:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:55:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:55:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:55:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:55:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:55:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:55:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:55:36.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:36 smithi106 ceph-mon[36505]: pgmap v4915: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:36.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:55:36.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:55:36.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:55:36.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:55:36.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:55:36.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:55:36.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:55:36.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:55:36.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:55:37.451 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9066 sec 2024-03-24T02:55:37.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:37 smithi107 ceph-mon[36485]: pgmap v4916: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T02:55:37.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:37 smithi107 ceph-mon[36485]: pgmap v4917: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:55:37.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:37 smithi106 ceph-mon[36505]: pgmap v4916: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T02:55:37.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:37 smithi106 ceph-mon[36505]: pgmap v4917: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:55:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:37 smithi078 ceph-mon[32429]: pgmap v4916: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T02:55:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:37 smithi078 ceph-mon[32429]: pgmap v4917: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:55:38.450 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9067 sec 2024-03-24T02:55:39.451 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9068 sec 2024-03-24T02:55:39.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8413.224739 secs 2024-03-24T02:55:39.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:39 smithi078 ceph-mon[32429]: pgmap v4918: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:55:39.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8413.224739 secs 2024-03-24T02:55:39.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:39 smithi106 ceph-mon[36505]: pgmap v4918: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:55:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8413.224739 secs 2024-03-24T02:55:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:39 smithi107 ceph-mon[36485]: pgmap v4918: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:55:40.450 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9069 sec 2024-03-24T02:55:41.451 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9070 sec 2024-03-24T02:55:41.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:41 smithi106 ceph-mon[36505]: pgmap v4919: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:55:42.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:41 smithi078 ceph-mon[32429]: pgmap v4919: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:55:42.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:41 smithi107 ceph-mon[36485]: pgmap v4919: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:55:42.451 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9071 sec 2024-03-24T02:55:43.451 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9072 sec 2024-03-24T02:55:43.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:43 smithi106 ceph-mon[36505]: pgmap v4920: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:55:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:43 smithi078 ceph-mon[32429]: pgmap v4920: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:55:44.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:43 smithi107 ceph-mon[36485]: pgmap v4920: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:55:44.451 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9073 sec 2024-03-24T02:55:44.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8418.224823 secs 2024-03-24T02:55:45.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8418.224823 secs 2024-03-24T02:55:45.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8418.224823 secs 2024-03-24T02:55:45.451 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9074 sec 2024-03-24T02:55:45.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:45 smithi106 ceph-mon[36505]: pgmap v4921: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:55:46.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:45 smithi078 ceph-mon[32429]: pgmap v4921: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:55:46.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:45 smithi107 ceph-mon[36485]: pgmap v4921: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:55:46.451 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9075 sec 2024-03-24T02:55:47.451 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9076 sec 2024-03-24T02:55:47.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:47 smithi106 ceph-mon[36505]: pgmap v4922: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:55:48.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:47 smithi078 ceph-mon[32429]: pgmap v4922: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:55:48.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:47 smithi107 ceph-mon[36485]: pgmap v4922: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:55:48.451 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9077 sec 2024-03-24T02:55:49.451 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9078 sec 2024-03-24T02:55:49.883 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8423.224863 secs 2024-03-24T02:55:49.884 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:49 smithi078 ceph-mon[32429]: pgmap v4923: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:49.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8423.224863 secs 2024-03-24T02:55:49.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:49 smithi106 ceph-mon[36505]: pgmap v4923: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:50.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8423.224863 secs 2024-03-24T02:55:50.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:49 smithi107 ceph-mon[36485]: pgmap v4923: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:50.451 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9079 sec 2024-03-24T02:55:51.452 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9080 sec 2024-03-24T02:55:51.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:51 smithi106 ceph-mon[36505]: pgmap v4924: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:52.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:51 smithi078 ceph-mon[32429]: pgmap v4924: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:52.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:51 smithi107 ceph-mon[36485]: pgmap v4924: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:52.452 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9081 sec 2024-03-24T02:55:53.452 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9082 sec 2024-03-24T02:55:53.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:53 smithi106 ceph-mon[36505]: pgmap v4925: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:53 smithi078 ceph-mon[32429]: pgmap v4925: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:54.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:53 smithi107 ceph-mon[36485]: pgmap v4925: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:54.452 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9083 sec 2024-03-24T02:55:54.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8428.224918 secs 2024-03-24T02:55:55.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8428.224918 secs 2024-03-24T02:55:55.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8428.224918 secs 2024-03-24T02:55:55.452 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9084 sec 2024-03-24T02:55:55.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:55 smithi106 ceph-mon[36505]: pgmap v4926: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:56.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:55 smithi078 ceph-mon[32429]: pgmap v4926: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:56.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:55 smithi107 ceph-mon[36485]: pgmap v4926: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:56.452 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9085 sec 2024-03-24T02:55:57.452 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9086 sec 2024-03-24T02:55:57.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:57 smithi106 ceph-mon[36505]: pgmap v4927: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:58.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:57 smithi078 ceph-mon[32429]: pgmap v4927: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:58.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:57 smithi107 ceph-mon[36485]: pgmap v4927: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:58.452 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9087 sec 2024-03-24T02:55:59.452 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9088 sec 2024-03-24T02:55:59.895 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8433.224977 secs 2024-03-24T02:55:59.895 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:55:59 smithi107 ceph-mon[36485]: pgmap v4928: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:55:59.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8433.224977 secs 2024-03-24T02:55:59.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:55:59 smithi106 ceph-mon[36505]: pgmap v4928: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8433.224977 secs 2024-03-24T02:56:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:55:59 smithi078 ceph-mon[32429]: pgmap v4928: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:00.452 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9089 sec 2024-03-24T02:56:01.452 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9090 sec 2024-03-24T02:56:01.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:01 smithi106 ceph-mon[36505]: pgmap v4929: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:02.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:01 smithi078 ceph-mon[32429]: pgmap v4929: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:02.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:01 smithi107 ceph-mon[36485]: pgmap v4929: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:02.452 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9091 sec 2024-03-24T02:56:03.453 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9092 sec 2024-03-24T02:56:03.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:03 smithi106 ceph-mon[36505]: pgmap v4930: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:03 smithi078 ceph-mon[32429]: pgmap v4930: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:04.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:03 smithi107 ceph-mon[36485]: pgmap v4930: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:04.453 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9093 sec 2024-03-24T02:56:04.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8438.225065 secs 2024-03-24T02:56:05.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8438.225065 secs 2024-03-24T02:56:05.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8438.225065 secs 2024-03-24T02:56:05.453 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9094 sec 2024-03-24T02:56:05.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:05 smithi106 ceph-mon[36505]: pgmap v4931: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:05.991 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:05 smithi078 ceph-mon[32429]: pgmap v4931: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:05 smithi107 ceph-mon[36485]: pgmap v4931: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:06.453 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9095 sec 2024-03-24T02:56:07.453 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9096 sec 2024-03-24T02:56:07.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:07 smithi106 ceph-mon[36505]: pgmap v4932: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:08.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:07 smithi078 ceph-mon[32429]: pgmap v4932: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:08.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:07 smithi107 ceph-mon[36485]: pgmap v4932: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:08.453 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9097 sec 2024-03-24T02:56:09.453 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9098 sec 2024-03-24T02:56:09.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8443.225151 secs 2024-03-24T02:56:09.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:09 smithi106 ceph-mon[36505]: pgmap v4933: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8443.225151 secs 2024-03-24T02:56:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:09 smithi078 ceph-mon[32429]: pgmap v4933: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:10.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8443.225151 secs 2024-03-24T02:56:10.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:09 smithi107 ceph-mon[36485]: pgmap v4933: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:10.453 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9099 sec 2024-03-24T02:56:11.453 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9100 sec 2024-03-24T02:56:12.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:11 smithi078 ceph-mon[32429]: pgmap v4934: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:12.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:11 smithi107 ceph-mon[36485]: pgmap v4934: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:12.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:11 smithi106 ceph-mon[36505]: pgmap v4934: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:12.453 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9101 sec 2024-03-24T02:56:13.453 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9102 sec 2024-03-24T02:56:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:13 smithi078 ceph-mon[32429]: pgmap v4935: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:14.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:13 smithi107 ceph-mon[36485]: pgmap v4935: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:14.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:13 smithi106 ceph-mon[36505]: pgmap v4935: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:14.454 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9103 sec 2024-03-24T02:56:15.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8448.225167 secs 2024-03-24T02:56:15.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8448.225167 secs 2024-03-24T02:56:15.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8448.225167 secs 2024-03-24T02:56:15.454 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9104 sec 2024-03-24T02:56:16.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:15 smithi078 ceph-mon[32429]: pgmap v4936: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:16.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:15 smithi107 ceph-mon[36485]: pgmap v4936: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:16.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:15 smithi106 ceph-mon[36505]: pgmap v4936: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:16.454 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9105 sec 2024-03-24T02:56:17.454 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9106 sec 2024-03-24T02:56:18.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:17 smithi078 ceph-mon[32429]: pgmap v4937: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:18.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:17 smithi107 ceph-mon[36485]: pgmap v4937: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:18.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:17 smithi106 ceph-mon[36505]: pgmap v4937: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:18.454 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9107 sec 2024-03-24T02:56:19.454 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9108 sec 2024-03-24T02:56:20.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8453.225228 secs 2024-03-24T02:56:20.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:19 smithi078 ceph-mon[32429]: pgmap v4938: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:20.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8453.225228 secs 2024-03-24T02:56:20.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:19 smithi107 ceph-mon[36485]: pgmap v4938: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:20.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8453.225228 secs 2024-03-24T02:56:20.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:19 smithi106 ceph-mon[36505]: pgmap v4938: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:20.454 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9109 sec 2024-03-24T02:56:21.454 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9110 sec 2024-03-24T02:56:22.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:21 smithi078 ceph-mon[32429]: pgmap v4939: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:22.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:21 smithi107 ceph-mon[36485]: pgmap v4939: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:22.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:21 smithi106 ceph-mon[36505]: pgmap v4939: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:22.454 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9111 sec 2024-03-24T02:56:23.454 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9112 sec 2024-03-24T02:56:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:23 smithi078 ceph-mon[32429]: pgmap v4940: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:24.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:23 smithi107 ceph-mon[36485]: pgmap v4940: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:24.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:23 smithi106 ceph-mon[36505]: pgmap v4940: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:24.454 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9113 sec 2024-03-24T02:56:25.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8458.225265 secs 2024-03-24T02:56:25.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8458.225265 secs 2024-03-24T02:56:25.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8458.225265 secs 2024-03-24T02:56:25.454 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9114 sec 2024-03-24T02:56:26.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:25 smithi078 ceph-mon[32429]: pgmap v4941: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:26.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:25 smithi107 ceph-mon[36485]: pgmap v4941: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:26.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:25 smithi106 ceph-mon[36505]: pgmap v4941: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:26.455 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9115 sec 2024-03-24T02:56:27.455 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9116 sec 2024-03-24T02:56:28.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:27 smithi078 ceph-mon[32429]: pgmap v4942: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:28.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:27 smithi107 ceph-mon[36485]: pgmap v4942: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:28.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:27 smithi106 ceph-mon[36505]: pgmap v4942: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:28.455 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9117 sec 2024-03-24T02:56:29.455 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9118 sec 2024-03-24T02:56:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8463.225326 secs 2024-03-24T02:56:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:29 smithi078 ceph-mon[32429]: pgmap v4943: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:30.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8463.225326 secs 2024-03-24T02:56:30.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:29 smithi107 ceph-mon[36485]: pgmap v4943: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:30.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8463.225326 secs 2024-03-24T02:56:30.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:29 smithi106 ceph-mon[36505]: pgmap v4943: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:30.455 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9119 sec 2024-03-24T02:56:31.455 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9120 sec 2024-03-24T02:56:32.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:31 smithi078 ceph-mon[32429]: pgmap v4944: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-24T02:56:32.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:31 smithi107 ceph-mon[36485]: pgmap v4944: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-24T02:56:32.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:31 smithi106 ceph-mon[36505]: pgmap v4944: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-24T02:56:32.455 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9121 sec 2024-03-24T02:56:33.455 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9122 sec 2024-03-24T02:56:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:33 smithi078 ceph-mon[32429]: pgmap v4945: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:34.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:33 smithi107 ceph-mon[36485]: pgmap v4945: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:34.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:33 smithi106 ceph-mon[36505]: pgmap v4945: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:34.455 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9123 sec 2024-03-24T02:56:35.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8468.225387 secs 2024-03-24T02:56:35.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8468.225387 secs 2024-03-24T02:56:35.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8468.225387 secs 2024-03-24T02:56:35.456 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9124 sec 2024-03-24T02:56:36.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:35 smithi078 ceph-mon[32429]: pgmap v4946: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:36.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:35 smithi107 ceph-mon[36485]: pgmap v4946: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:36.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:35 smithi106 ceph-mon[36505]: pgmap v4946: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:36.456 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9125 sec 2024-03-24T02:56:36.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:36 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:56:37.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:36 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:56:37.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:36 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:56:37.456 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9126 sec 2024-03-24T02:56:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:37 smithi078 ceph-mon[32429]: pgmap v4947: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:56:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:56:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:56:38.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:37 smithi107 ceph-mon[36485]: pgmap v4947: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:38.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:56:38.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:56:38.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:56:38.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:37 smithi106 ceph-mon[36505]: pgmap v4947: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:38.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:56:38.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:56:38.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:56:38.456 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9127 sec 2024-03-24T02:56:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:38 smithi078 ceph-mon[32429]: pgmap v4948: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:56:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:38 smithi078 ceph-mon[32429]: pgmap v4949: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:56:39.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:38 smithi107 ceph-mon[36485]: pgmap v4948: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:56:39.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:38 smithi107 ceph-mon[36485]: pgmap v4949: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:56:39.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:38 smithi106 ceph-mon[36505]: pgmap v4948: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:56:39.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:38 smithi106 ceph-mon[36505]: pgmap v4949: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:56:39.456 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9128 sec 2024-03-24T02:56:39.991 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8473.225451 secs 2024-03-24T02:56:40.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8473.225451 secs 2024-03-24T02:56:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8473.225451 secs 2024-03-24T02:56:40.456 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9129 sec 2024-03-24T02:56:41.052 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:40 smithi106 ceph-mon[36505]: pgmap v4950: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:40 smithi078 ceph-mon[32429]: pgmap v4950: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:41.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:40 smithi107 ceph-mon[36485]: pgmap v4950: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:41.456 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9130 sec 2024-03-24T02:56:42.456 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9131 sec 2024-03-24T02:56:43.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:42 smithi078 ceph-mon[32429]: pgmap v4951: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:56:43.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:42 smithi107 ceph-mon[36485]: pgmap v4951: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:56:43.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:42 smithi106 ceph-mon[36505]: pgmap v4951: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:56:43.456 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9132 sec 2024-03-24T02:56:44.456 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9133 sec 2024-03-24T02:56:45.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8478.225535 secs 2024-03-24T02:56:45.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:44 smithi078 ceph-mon[32429]: pgmap v4952: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:56:45.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8478.225535 secs 2024-03-24T02:56:45.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:44 smithi107 ceph-mon[36485]: pgmap v4952: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:56:45.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8478.225535 secs 2024-03-24T02:56:45.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:44 smithi106 ceph-mon[36505]: pgmap v4952: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:56:45.456 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9134 sec 2024-03-24T02:56:46.456 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9135 sec 2024-03-24T02:56:47.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:46 smithi078 ceph-mon[32429]: pgmap v4953: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:56:47.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:46 smithi107 ceph-mon[36485]: pgmap v4953: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:56:47.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:46 smithi106 ceph-mon[36505]: pgmap v4953: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:56:47.456 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9136 sec 2024-03-24T02:56:48.457 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9137 sec 2024-03-24T02:56:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:48 smithi078 ceph-mon[32429]: pgmap v4954: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:56:49.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:48 smithi107 ceph-mon[36485]: pgmap v4954: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:56:49.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:48 smithi106 ceph-mon[36505]: pgmap v4954: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:56:49.456 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9138 sec 2024-03-24T02:56:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8483.225590 secs 2024-03-24T02:56:50.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8483.225590 secs 2024-03-24T02:56:50.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8483.225590 secs 2024-03-24T02:56:50.457 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9139 sec 2024-03-24T02:56:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:50 smithi078 ceph-mon[32429]: pgmap v4955: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:51.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:50 smithi107 ceph-mon[36485]: pgmap v4955: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:51.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:50 smithi106 ceph-mon[36505]: pgmap v4955: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:51.457 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9140 sec 2024-03-24T02:56:52.457 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9141 sec 2024-03-24T02:56:53.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:52 smithi078 ceph-mon[32429]: pgmap v4956: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:53.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:52 smithi107 ceph-mon[36485]: pgmap v4956: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:53.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:52 smithi106 ceph-mon[36505]: pgmap v4956: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:53.457 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9142 sec 2024-03-24T02:56:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8488.225676 secs 2024-03-24T02:56:54.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8488.225676 secs 2024-03-24T02:56:54.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8488.225676 secs 2024-03-24T02:56:54.457 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9143 sec 2024-03-24T02:56:55.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:54 smithi078 ceph-mon[32429]: pgmap v4957: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:55.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:54 smithi107 ceph-mon[36485]: pgmap v4957: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:55.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:54 smithi106 ceph-mon[36505]: pgmap v4957: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:55.457 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9144 sec 2024-03-24T02:56:56.457 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9145 sec 2024-03-24T02:56:57.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:56 smithi078 ceph-mon[32429]: pgmap v4958: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:56 smithi107 ceph-mon[36485]: pgmap v4958: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:57.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:56 smithi106 ceph-mon[36505]: pgmap v4958: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:56:57.457 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9146 sec 2024-03-24T02:56:58.457 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9147 sec 2024-03-24T02:56:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:58 smithi078 ceph-mon[32429]: pgmap v4959: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:56:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8493.225737 secs 2024-03-24T02:56:59.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:58 smithi107 ceph-mon[36485]: pgmap v4959: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:59.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:56:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8493.225737 secs 2024-03-24T02:56:59.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:58 smithi106 ceph-mon[36505]: pgmap v4959: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:56:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8493.225737 secs 2024-03-24T02:56:59.457 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9148 sec 2024-03-24T02:57:00.457 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9149 sec 2024-03-24T02:57:01.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:00 smithi078 ceph-mon[32429]: pgmap v4960: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:01.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:00 smithi107 ceph-mon[36485]: pgmap v4960: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:01.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:00 smithi106 ceph-mon[36505]: pgmap v4960: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:01.457 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9150 sec 2024-03-24T02:57:02.458 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9151 sec 2024-03-24T02:57:03.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:02 smithi078 ceph-mon[32429]: pgmap v4961: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:03.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:02 smithi107 ceph-mon[36485]: pgmap v4961: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:03.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:02 smithi106 ceph-mon[36505]: pgmap v4961: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:03.458 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9152 sec 2024-03-24T02:57:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8498.225813 secs 2024-03-24T02:57:04.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8498.225813 secs 2024-03-24T02:57:04.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8498.225813 secs 2024-03-24T02:57:04.458 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9153 sec 2024-03-24T02:57:05.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:04 smithi078 ceph-mon[32429]: pgmap v4962: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:05.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:04 smithi107 ceph-mon[36485]: pgmap v4962: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:05.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:04 smithi106 ceph-mon[36505]: pgmap v4962: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:05.458 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9154 sec 2024-03-24T02:57:06.458 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9155 sec 2024-03-24T02:57:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:06 smithi078 ceph-mon[32429]: pgmap v4963: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:07.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:06 smithi107 ceph-mon[36485]: pgmap v4963: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:07.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:06 smithi106 ceph-mon[36505]: pgmap v4963: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:07.458 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9156 sec 2024-03-24T02:57:08.458 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9157 sec 2024-03-24T02:57:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:08 smithi078 ceph-mon[32429]: pgmap v4964: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8503.225874 secs 2024-03-24T02:57:09.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:08 smithi107 ceph-mon[36485]: pgmap v4964: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:09.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8503.225874 secs 2024-03-24T02:57:09.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:08 smithi106 ceph-mon[36505]: pgmap v4964: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8503.225874 secs 2024-03-24T02:57:09.458 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9158 sec 2024-03-24T02:57:10.458 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9159 sec 2024-03-24T02:57:11.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:10 smithi078 ceph-mon[32429]: pgmap v4965: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:11.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:10 smithi107 ceph-mon[36485]: pgmap v4965: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:11.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:10 smithi106 ceph-mon[36505]: pgmap v4965: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:11.458 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9160 sec 2024-03-24T02:57:12.458 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9161 sec 2024-03-24T02:57:13.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:12 smithi078 ceph-mon[32429]: pgmap v4966: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:13.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:12 smithi107 ceph-mon[36485]: pgmap v4966: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:13.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:12 smithi106 ceph-mon[36505]: pgmap v4966: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:13.459 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9162 sec 2024-03-24T02:57:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8508.225958 secs 2024-03-24T02:57:14.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8508.225958 secs 2024-03-24T02:57:14.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8508.225958 secs 2024-03-24T02:57:14.459 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9163 sec 2024-03-24T02:57:15.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:14 smithi078 ceph-mon[32429]: pgmap v4967: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:15.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:14 smithi107 ceph-mon[36485]: pgmap v4967: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:15.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:14 smithi106 ceph-mon[36505]: pgmap v4967: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:15.459 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9164 sec 2024-03-24T02:57:16.459 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9165 sec 2024-03-24T02:57:17.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:16 smithi078 ceph-mon[32429]: pgmap v4968: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:17.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:16 smithi107 ceph-mon[36485]: pgmap v4968: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:17.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:16 smithi106 ceph-mon[36505]: pgmap v4968: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:17.459 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9166 sec 2024-03-24T02:57:18.459 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9167 sec 2024-03-24T02:57:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:18 smithi078 ceph-mon[32429]: pgmap v4969: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8513.226047 secs 2024-03-24T02:57:19.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:18 smithi107 ceph-mon[36485]: pgmap v4969: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:19.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8513.226047 secs 2024-03-24T02:57:19.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:18 smithi106 ceph-mon[36505]: pgmap v4969: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:19.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8513.226047 secs 2024-03-24T02:57:19.459 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9168 sec 2024-03-24T02:57:20.459 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9169 sec 2024-03-24T02:57:21.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:20 smithi078 ceph-mon[32429]: pgmap v4970: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:21.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:20 smithi107 ceph-mon[36485]: pgmap v4970: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:21.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:20 smithi106 ceph-mon[36505]: pgmap v4970: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:21.459 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9170 sec 2024-03-24T02:57:22.459 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9171 sec 2024-03-24T02:57:23.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:22 smithi078 ceph-mon[32429]: pgmap v4971: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:23.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:22 smithi107 ceph-mon[36485]: pgmap v4971: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:23.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:22 smithi106 ceph-mon[36505]: pgmap v4971: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:23.459 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9172 sec 2024-03-24T02:57:23.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8518.226106 secs 2024-03-24T02:57:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8518.226106 secs 2024-03-24T02:57:24.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8518.226106 secs 2024-03-24T02:57:24.460 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9173 sec 2024-03-24T02:57:25.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:24 smithi078 ceph-mon[32429]: pgmap v4972: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:25.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:24 smithi107 ceph-mon[36485]: pgmap v4972: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:25.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:24 smithi106 ceph-mon[36505]: pgmap v4972: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:25.460 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9174 sec 2024-03-24T02:57:26.460 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9175 sec 2024-03-24T02:57:27.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:26 smithi078 ceph-mon[32429]: pgmap v4973: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:27.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:26 smithi107 ceph-mon[36485]: pgmap v4973: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:27.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:26 smithi106 ceph-mon[36505]: pgmap v4973: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:27.460 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9176 sec 2024-03-24T02:57:28.460 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9177 sec 2024-03-24T02:57:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:28 smithi078 ceph-mon[32429]: pgmap v4974: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8523.226167 secs 2024-03-24T02:57:29.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:28 smithi107 ceph-mon[36485]: pgmap v4974: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:29.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8523.226167 secs 2024-03-24T02:57:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:28 smithi106 ceph-mon[36505]: pgmap v4974: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8523.226167 secs 2024-03-24T02:57:29.460 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9178 sec 2024-03-24T02:57:30.460 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9179 sec 2024-03-24T02:57:31.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:30 smithi078 ceph-mon[32429]: pgmap v4975: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:31.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:30 smithi107 ceph-mon[36485]: pgmap v4975: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:31.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:30 smithi106 ceph-mon[36505]: pgmap v4975: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:31.460 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9180 sec 2024-03-24T02:57:32.461 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9181 sec 2024-03-24T02:57:33.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:32 smithi078 ceph-mon[32429]: pgmap v4976: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:33.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:32 smithi107 ceph-mon[36485]: pgmap v4976: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:33.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:32 smithi106 ceph-mon[36505]: pgmap v4976: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:33.462 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9182 sec 2024-03-24T02:57:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8528.226227 secs 2024-03-24T02:57:34.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8528.226227 secs 2024-03-24T02:57:34.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8528.226227 secs 2024-03-24T02:57:34.461 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9183 sec 2024-03-24T02:57:35.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:34 smithi078 ceph-mon[32429]: pgmap v4977: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:35.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:34 smithi107 ceph-mon[36485]: pgmap v4977: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:35.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:34 smithi106 ceph-mon[36505]: pgmap v4977: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:35.461 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9184 sec 2024-03-24T02:57:36.461 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9185 sec 2024-03-24T02:57:37.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:36 smithi078 ceph-mon[32429]: pgmap v4978: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:37.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:36 smithi107 ceph-mon[36485]: pgmap v4978: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:37.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:36 smithi106 ceph-mon[36505]: pgmap v4978: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:37.461 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9186 sec 2024-03-24T02:57:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:57:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:57:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:57:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:37 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:57:38.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:57:38.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:57:38.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:57:38.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:37 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:57:38.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:57:38.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:57:38.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:57:38.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:37 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:57:38.461 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9187 sec 2024-03-24T02:57:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:38 smithi106 ceph-mon[36505]: pgmap v4979: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:38 smithi106 ceph-mon[36505]: pgmap v4980: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:57:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:38 smithi106 ceph-mon[36505]: pgmap v4981: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:57:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8533.226296 secs 2024-03-24T02:57:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:38 smithi078 ceph-mon[32429]: pgmap v4979: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:38 smithi078 ceph-mon[32429]: pgmap v4980: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:57:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:38 smithi078 ceph-mon[32429]: pgmap v4981: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:57:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8533.226296 secs 2024-03-24T02:57:39.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:38 smithi107 ceph-mon[36485]: pgmap v4979: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:39.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:38 smithi107 ceph-mon[36485]: pgmap v4980: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:57:39.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:38 smithi107 ceph-mon[36485]: pgmap v4981: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:57:39.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8533.226296 secs 2024-03-24T02:57:39.461 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9188 sec 2024-03-24T02:57:40.461 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9189 sec 2024-03-24T02:57:41.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:40 smithi106 ceph-mon[36505]: pgmap v4982: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:57:41.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:40 smithi078 ceph-mon[32429]: pgmap v4982: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:57:41.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:40 smithi107 ceph-mon[36485]: pgmap v4982: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:57:41.461 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9190 sec 2024-03-24T02:57:42.462 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9191 sec 2024-03-24T02:57:43.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:42 smithi106 ceph-mon[36505]: pgmap v4983: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:57:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:42 smithi078 ceph-mon[32429]: pgmap v4983: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:57:43.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:42 smithi107 ceph-mon[36485]: pgmap v4983: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:57:43.461 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9192 sec 2024-03-24T02:57:44.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8538.226357 secs 2024-03-24T02:57:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8538.226357 secs 2024-03-24T02:57:44.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8538.226357 secs 2024-03-24T02:57:44.461 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9193 sec 2024-03-24T02:57:45.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:44 smithi106 ceph-mon[36505]: pgmap v4984: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:57:45.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:44 smithi078 ceph-mon[32429]: pgmap v4984: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:57:45.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:44 smithi107 ceph-mon[36485]: pgmap v4984: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:57:45.462 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9194 sec 2024-03-24T02:57:46.462 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9195 sec 2024-03-24T02:57:47.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:46 smithi106 ceph-mon[36505]: pgmap v4985: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:57:47.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:46 smithi078 ceph-mon[32429]: pgmap v4985: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:57:47.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:46 smithi107 ceph-mon[36485]: pgmap v4985: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:57:47.462 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9196 sec 2024-03-24T02:57:48.462 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9197 sec 2024-03-24T02:57:49.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:48 smithi106 ceph-mon[36505]: pgmap v4986: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:57:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8543.226446 secs 2024-03-24T02:57:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:48 smithi078 ceph-mon[32429]: pgmap v4986: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:57:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8543.226446 secs 2024-03-24T02:57:49.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:48 smithi107 ceph-mon[36485]: pgmap v4986: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:57:49.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8543.226446 secs 2024-03-24T02:57:49.462 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9198 sec 2024-03-24T02:57:50.462 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9199 sec 2024-03-24T02:57:51.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:50 smithi106 ceph-mon[36505]: pgmap v4987: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:51.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:50 smithi078 ceph-mon[32429]: pgmap v4987: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:51.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:50 smithi107 ceph-mon[36485]: pgmap v4987: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:51.462 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9200 sec 2024-03-24T02:57:52.462 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9201 sec 2024-03-24T02:57:53.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:52 smithi106 ceph-mon[36505]: pgmap v4988: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:53.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:52 smithi078 ceph-mon[32429]: pgmap v4988: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:53.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:52 smithi107 ceph-mon[36485]: pgmap v4988: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:53.462 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9202 sec 2024-03-24T02:57:54.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8548.226502 secs 2024-03-24T02:57:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8548.226502 secs 2024-03-24T02:57:54.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8548.226502 secs 2024-03-24T02:57:54.462 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9203 sec 2024-03-24T02:57:55.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:54 smithi078 ceph-mon[32429]: pgmap v4989: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:55.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:54 smithi106 ceph-mon[36505]: pgmap v4989: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:55.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:54 smithi107 ceph-mon[36485]: pgmap v4989: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:55.463 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9204 sec 2024-03-24T02:57:56.463 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9205 sec 2024-03-24T02:57:57.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:56 smithi106 ceph-mon[36505]: pgmap v4990: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:57.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:56 smithi078 ceph-mon[32429]: pgmap v4990: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:57.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:56 smithi107 ceph-mon[36485]: pgmap v4990: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:57.463 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9206 sec 2024-03-24T02:57:58.463 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9207 sec 2024-03-24T02:57:59.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:58 smithi106 ceph-mon[36505]: pgmap v4991: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:57:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8553.226544 secs 2024-03-24T02:57:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:58 smithi078 ceph-mon[32429]: pgmap v4991: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:57:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8553.226544 secs 2024-03-24T02:57:59.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:58 smithi107 ceph-mon[36485]: pgmap v4991: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:57:59.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:57:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8553.226544 secs 2024-03-24T02:57:59.463 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9208 sec 2024-03-24T02:58:00.463 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9209 sec 2024-03-24T02:58:01.145 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:00 smithi107 ceph-mon[36485]: pgmap v4992: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:01.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:00 smithi106 ceph-mon[36505]: pgmap v4992: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:01.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:00 smithi078 ceph-mon[32429]: pgmap v4992: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:01.463 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9210 sec 2024-03-24T02:58:02.463 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9211 sec 2024-03-24T02:58:03.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:02 smithi106 ceph-mon[36505]: pgmap v4993: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:03.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:02 smithi078 ceph-mon[32429]: pgmap v4993: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:03.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:02 smithi107 ceph-mon[36485]: pgmap v4993: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:03.463 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9212 sec 2024-03-24T02:58:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8558.226626 secs 2024-03-24T02:58:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8558.226626 secs 2024-03-24T02:58:04.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8558.226626 secs 2024-03-24T02:58:04.463 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9213 sec 2024-03-24T02:58:05.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:04 smithi106 ceph-mon[36505]: pgmap v4994: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:05.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:04 smithi078 ceph-mon[32429]: pgmap v4994: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:05.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:04 smithi107 ceph-mon[36485]: pgmap v4994: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:05.464 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9214 sec 2024-03-24T02:58:06.464 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9215 sec 2024-03-24T02:58:07.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:06 smithi078 ceph-mon[32429]: pgmap v4995: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:07.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:06 smithi107 ceph-mon[36485]: pgmap v4995: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:07.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:06 smithi106 ceph-mon[36505]: pgmap v4995: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:07.464 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9216 sec 2024-03-24T02:58:08.464 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9217 sec 2024-03-24T02:58:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:08 smithi078 ceph-mon[32429]: pgmap v4996: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8563.226712 secs 2024-03-24T02:58:09.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:08 smithi107 ceph-mon[36485]: pgmap v4996: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:09.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8563.226712 secs 2024-03-24T02:58:09.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:08 smithi106 ceph-mon[36505]: pgmap v4996: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:09.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8563.226712 secs 2024-03-24T02:58:09.464 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9218 sec 2024-03-24T02:58:10.464 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9219 sec 2024-03-24T02:58:11.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:10 smithi078 ceph-mon[32429]: pgmap v4997: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:11.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:10 smithi107 ceph-mon[36485]: pgmap v4997: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:11.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:10 smithi106 ceph-mon[36505]: pgmap v4997: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:11.464 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9220 sec 2024-03-24T02:58:12.464 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9221 sec 2024-03-24T02:58:13.204 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:12 smithi106 ceph-mon[36505]: pgmap v4998: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:13.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:12 smithi078 ceph-mon[32429]: pgmap v4998: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:13.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:12 smithi107 ceph-mon[36485]: pgmap v4998: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:13.464 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9222 sec 2024-03-24T02:58:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8568.226784 secs 2024-03-24T02:58:14.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8568.226784 secs 2024-03-24T02:58:14.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8568.226784 secs 2024-03-24T02:58:14.464 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9223 sec 2024-03-24T02:58:15.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:14 smithi078 ceph-mon[32429]: pgmap v4999: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:15.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:14 smithi107 ceph-mon[36485]: pgmap v4999: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:15.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:14 smithi106 ceph-mon[36505]: pgmap v4999: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:15.464 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9224 sec 2024-03-24T02:58:16.465 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9225 sec 2024-03-24T02:58:17.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:16 smithi078 ceph-mon[32429]: pgmap v5000: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:17.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:16 smithi107 ceph-mon[36485]: pgmap v5000: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:17.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:16 smithi106 ceph-mon[36505]: pgmap v5000: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:17.465 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9226 sec 2024-03-24T02:58:18.465 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9227 sec 2024-03-24T02:58:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:18 smithi078 ceph-mon[32429]: pgmap v5001: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8573.226869 secs 2024-03-24T02:58:19.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:18 smithi107 ceph-mon[36485]: pgmap v5001: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:19.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8573.226869 secs 2024-03-24T02:58:19.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:18 smithi106 ceph-mon[36505]: pgmap v5001: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8573.226869 secs 2024-03-24T02:58:19.465 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9228 sec 2024-03-24T02:58:20.465 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9229 sec 2024-03-24T02:58:21.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:20 smithi078 ceph-mon[32429]: pgmap v5002: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:21.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:20 smithi107 ceph-mon[36485]: pgmap v5002: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:21.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:20 smithi106 ceph-mon[36505]: pgmap v5002: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:21.465 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9230 sec 2024-03-24T02:58:22.465 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9231 sec 2024-03-24T02:58:23.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:22 smithi078 ceph-mon[32429]: pgmap v5003: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:23.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:22 smithi107 ceph-mon[36485]: pgmap v5003: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:23.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:22 smithi106 ceph-mon[36505]: pgmap v5003: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:23.465 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9232 sec 2024-03-24T02:58:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8578.226925 secs 2024-03-24T02:58:24.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8578.226925 secs 2024-03-24T02:58:24.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8578.226925 secs 2024-03-24T02:58:24.465 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9233 sec 2024-03-24T02:58:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:24 smithi078 ceph-mon[32429]: pgmap v5004: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:25.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:24 smithi107 ceph-mon[36485]: pgmap v5004: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:25.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:24 smithi106 ceph-mon[36505]: pgmap v5004: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:25.465 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9234 sec 2024-03-24T02:58:26.465 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9235 sec 2024-03-24T02:58:27.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:26 smithi078 ceph-mon[32429]: pgmap v5005: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:27.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:26 smithi107 ceph-mon[36485]: pgmap v5005: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:27.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:26 smithi106 ceph-mon[36505]: pgmap v5005: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:27.465 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9236 sec 2024-03-24T02:58:28.466 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9237 sec 2024-03-24T02:58:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:28 smithi078 ceph-mon[32429]: pgmap v5006: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8583.226984 secs 2024-03-24T02:58:29.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:28 smithi107 ceph-mon[36485]: pgmap v5006: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:29.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8583.226984 secs 2024-03-24T02:58:29.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:28 smithi106 ceph-mon[36505]: pgmap v5006: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:29.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8583.226984 secs 2024-03-24T02:58:29.466 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9238 sec 2024-03-24T02:58:30.466 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9239 sec 2024-03-24T02:58:31.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:30 smithi078 ceph-mon[32429]: pgmap v5007: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:31.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:30 smithi107 ceph-mon[36485]: pgmap v5007: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:31.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:30 smithi106 ceph-mon[36505]: pgmap v5007: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:31.466 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9240 sec 2024-03-24T02:58:32.466 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9241 sec 2024-03-24T02:58:32.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:32 smithi106 ceph-mon[36505]: pgmap v5008: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:32.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:32 smithi078 ceph-mon[32429]: pgmap v5008: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:32.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:32 smithi107 ceph-mon[36485]: pgmap v5008: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:33.466 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9242 sec 2024-03-24T02:58:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8588.227037 secs 2024-03-24T02:58:34.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8588.227037 secs 2024-03-24T02:58:34.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8588.227037 secs 2024-03-24T02:58:34.466 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9243 sec 2024-03-24T02:58:35.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:34 smithi078 ceph-mon[32429]: pgmap v5009: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:35.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:34 smithi107 ceph-mon[36485]: pgmap v5009: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:35.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:34 smithi106 ceph-mon[36505]: pgmap v5009: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:35.466 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9244 sec 2024-03-24T02:58:36.466 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9245 sec 2024-03-24T02:58:37.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:36 smithi106 ceph-mon[36505]: pgmap v5010: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:37.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:36 smithi078 ceph-mon[32429]: pgmap v5010: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:36 smithi107 ceph-mon[36485]: pgmap v5010: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:37.466 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9246 sec 2024-03-24T02:58:38.466 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9247 sec 2024-03-24T02:58:39.142 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:38 smithi078 ceph-mon[32429]: pgmap v5011: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:39.142 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:38 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:58:39.142 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8593.227102 secs 2024-03-24T02:58:39.142 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:38 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:58:39.142 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:38 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:58:39.142 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:38 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:58:39.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:38 smithi106 ceph-mon[36505]: pgmap v5011: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:38 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:58:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8593.227102 secs 2024-03-24T02:58:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:38 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:58:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:38 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:58:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:38 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:58:39.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:38 smithi107 ceph-mon[36485]: pgmap v5011: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:39.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:38 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:58:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8593.227102 secs 2024-03-24T02:58:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:38 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:58:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:38 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:58:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:38 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:58:39.466 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9248 sec 2024-03-24T02:58:40.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:39 smithi106 ceph-mon[36505]: pgmap v5012: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:58:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:39 smithi106 ceph-mon[36505]: pgmap v5013: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:58:40.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:39 smithi078 ceph-mon[32429]: pgmap v5012: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:58:40.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:39 smithi078 ceph-mon[32429]: pgmap v5013: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:58:40.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:39 smithi107 ceph-mon[36485]: pgmap v5012: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:58:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:39 smithi107 ceph-mon[36485]: pgmap v5013: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:58:40.467 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9249 sec 2024-03-24T02:58:41.467 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9250 sec 2024-03-24T02:58:42.142 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:41 smithi078 ceph-mon[32429]: pgmap v5014: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:58:42.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:41 smithi106 ceph-mon[36505]: pgmap v5014: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:58:42.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:41 smithi107 ceph-mon[36485]: pgmap v5014: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:58:42.467 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9251 sec 2024-03-24T02:58:43.467 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9252 sec 2024-03-24T02:58:44.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:43 smithi106 ceph-mon[36505]: pgmap v5015: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:58:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8598.227140 secs 2024-03-24T02:58:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:43 smithi078 ceph-mon[32429]: pgmap v5015: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:58:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8598.227140 secs 2024-03-24T02:58:44.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:43 smithi107 ceph-mon[36485]: pgmap v5015: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:58:44.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8598.227140 secs 2024-03-24T02:58:44.467 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9253 sec 2024-03-24T02:58:45.467 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9254 sec 2024-03-24T02:58:46.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:45 smithi106 ceph-mon[36505]: pgmap v5016: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:58:46.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:45 smithi078 ceph-mon[32429]: pgmap v5016: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:58:46.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:45 smithi107 ceph-mon[36485]: pgmap v5016: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:58:46.467 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9255 sec 2024-03-24T02:58:47.467 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9256 sec 2024-03-24T02:58:48.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:47 smithi106 ceph-mon[36505]: pgmap v5017: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:58:48.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:47 smithi078 ceph-mon[32429]: pgmap v5017: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:58:48.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:47 smithi107 ceph-mon[36485]: pgmap v5017: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:58:48.467 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9257 sec 2024-03-24T02:58:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8603.227203 secs 2024-03-24T02:58:49.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8603.227203 secs 2024-03-24T02:58:49.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8603.227203 secs 2024-03-24T02:58:49.467 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9258 sec 2024-03-24T02:58:50.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:49 smithi078 ceph-mon[32429]: pgmap v5018: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:58:50.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:49 smithi107 ceph-mon[36485]: pgmap v5018: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:58:50.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:49 smithi106 ceph-mon[36505]: pgmap v5018: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:58:50.467 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9259 sec 2024-03-24T02:58:51.468 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9260 sec 2024-03-24T02:58:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:51 smithi078 ceph-mon[32429]: pgmap v5019: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:52.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:51 smithi107 ceph-mon[36485]: pgmap v5019: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:52.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:51 smithi106 ceph-mon[36505]: pgmap v5019: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:52.468 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9261 sec 2024-03-24T02:58:53.468 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9262 sec 2024-03-24T02:58:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:53 smithi078 ceph-mon[32429]: pgmap v5020: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8608.227294 secs 2024-03-24T02:58:54.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:53 smithi107 ceph-mon[36485]: pgmap v5020: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:54.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8608.227294 secs 2024-03-24T02:58:54.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:53 smithi106 ceph-mon[36505]: pgmap v5020: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8608.227294 secs 2024-03-24T02:58:54.468 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9263 sec 2024-03-24T02:58:55.468 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9264 sec 2024-03-24T02:58:56.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:55 smithi078 ceph-mon[32429]: pgmap v5021: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:56.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:55 smithi107 ceph-mon[36485]: pgmap v5021: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:56.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:55 smithi106 ceph-mon[36505]: pgmap v5021: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:58:56.468 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9265 sec 2024-03-24T02:58:57.468 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9266 sec 2024-03-24T02:58:58.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:57 smithi078 ceph-mon[32429]: pgmap v5022: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:58.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:57 smithi107 ceph-mon[36485]: pgmap v5022: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:58.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:57 smithi106 ceph-mon[36505]: pgmap v5022: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:58.468 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9267 sec 2024-03-24T02:58:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8613.227379 secs 2024-03-24T02:58:59.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8613.227379 secs 2024-03-24T02:58:59.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8613.227379 secs 2024-03-24T02:58:59.468 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9268 sec 2024-03-24T02:59:00.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:58:59 smithi078 ceph-mon[32429]: pgmap v5023: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:00.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:58:59 smithi107 ceph-mon[36485]: pgmap v5023: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:00.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:58:59 smithi106 ceph-mon[36505]: pgmap v5023: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:00.468 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9269 sec 2024-03-24T02:59:01.469 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9270 sec 2024-03-24T02:59:02.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:01 smithi078 ceph-mon[32429]: pgmap v5024: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:02.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:01 smithi107 ceph-mon[36485]: pgmap v5024: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:02.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:01 smithi106 ceph-mon[36505]: pgmap v5024: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:02.468 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9271 sec 2024-03-24T02:59:03.469 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9272 sec 2024-03-24T02:59:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:03 smithi078 ceph-mon[32429]: pgmap v5025: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8618.227436 secs 2024-03-24T02:59:04.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:03 smithi107 ceph-mon[36485]: pgmap v5025: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8618.227436 secs 2024-03-24T02:59:04.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:03 smithi106 ceph-mon[36505]: pgmap v5025: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:04.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8618.227436 secs 2024-03-24T02:59:04.469 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9273 sec 2024-03-24T02:59:05.469 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9274 sec 2024-03-24T02:59:06.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:05 smithi078 ceph-mon[32429]: pgmap v5026: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:06.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:05 smithi107 ceph-mon[36485]: pgmap v5026: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:06.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:05 smithi106 ceph-mon[36505]: pgmap v5026: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:06.469 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9275 sec 2024-03-24T02:59:07.469 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9276 sec 2024-03-24T02:59:08.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:07 smithi078 ceph-mon[32429]: pgmap v5027: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:59:08.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:07 smithi107 ceph-mon[36485]: pgmap v5027: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:59:08.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:07 smithi106 ceph-mon[36505]: pgmap v5027: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:59:08.469 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9277 sec 2024-03-24T02:59:09.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8623.227495 secs 2024-03-24T02:59:09.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8623.227495 secs 2024-03-24T02:59:09.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8623.227495 secs 2024-03-24T02:59:09.469 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9278 sec 2024-03-24T02:59:10.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:09 smithi078 ceph-mon[32429]: pgmap v5028: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:59:10.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:09 smithi107 ceph-mon[36485]: pgmap v5028: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:59:10.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:09 smithi106 ceph-mon[36505]: pgmap v5028: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:59:10.469 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9279 sec 2024-03-24T02:59:11.469 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9280 sec 2024-03-24T02:59:12.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:11 smithi078 ceph-mon[32429]: pgmap v5029: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:59:12.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:11 smithi107 ceph-mon[36485]: pgmap v5029: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:59:12.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:11 smithi106 ceph-mon[36505]: pgmap v5029: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:59:12.469 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9281 sec 2024-03-24T02:59:13.469 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9282 sec 2024-03-24T02:59:14.204 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:13 smithi106 ceph-mon[36505]: pgmap v5030: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:59:14.204 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8628.227580 secs 2024-03-24T02:59:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:13 smithi078 ceph-mon[32429]: pgmap v5030: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:59:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8628.227580 secs 2024-03-24T02:59:14.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:13 smithi107 ceph-mon[36485]: pgmap v5030: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:59:14.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8628.227580 secs 2024-03-24T02:59:14.469 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9283 sec 2024-03-24T02:59:15.470 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9284 sec 2024-03-24T02:59:16.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:15 smithi078 ceph-mon[32429]: pgmap v5031: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:59:16.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:15 smithi107 ceph-mon[36485]: pgmap v5031: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:59:16.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:15 smithi106 ceph-mon[36505]: pgmap v5031: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T02:59:16.470 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9285 sec 2024-03-24T02:59:17.470 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9286 sec 2024-03-24T02:59:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:17 smithi078 ceph-mon[32429]: pgmap v5032: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:59:18.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:17 smithi107 ceph-mon[36485]: pgmap v5032: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:59:18.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:17 smithi106 ceph-mon[36505]: pgmap v5032: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:59:18.470 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9287 sec 2024-03-24T02:59:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8633.227667 secs 2024-03-24T02:59:19.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8633.227667 secs 2024-03-24T02:59:19.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8633.227667 secs 2024-03-24T02:59:19.470 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9288 sec 2024-03-24T02:59:20.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:19 smithi078 ceph-mon[32429]: pgmap v5033: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:20.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:19 smithi107 ceph-mon[36485]: pgmap v5033: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:20.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:19 smithi106 ceph-mon[36505]: pgmap v5033: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:20.470 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9289 sec 2024-03-24T02:59:21.470 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9290 sec 2024-03-24T02:59:22.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:21 smithi078 ceph-mon[32429]: pgmap v5034: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:22.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:21 smithi107 ceph-mon[36485]: pgmap v5034: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:22.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:21 smithi106 ceph-mon[36505]: pgmap v5034: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:22.470 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9291 sec 2024-03-24T02:59:23.470 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9292 sec 2024-03-24T02:59:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:23 smithi078 ceph-mon[32429]: pgmap v5035: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8638.227727 secs 2024-03-24T02:59:24.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:23 smithi107 ceph-mon[36485]: pgmap v5035: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:24.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8638.227727 secs 2024-03-24T02:59:24.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:23 smithi106 ceph-mon[36505]: pgmap v5035: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:24.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8638.227727 secs 2024-03-24T02:59:24.470 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9293 sec 2024-03-24T02:59:25.471 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9294 sec 2024-03-24T02:59:26.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:25 smithi078 ceph-mon[32429]: pgmap v5036: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:26.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:25 smithi107 ceph-mon[36485]: pgmap v5036: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:26.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:25 smithi106 ceph-mon[36505]: pgmap v5036: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:26.470 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9295 sec 2024-03-24T02:59:27.471 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9296 sec 2024-03-24T02:59:28.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:27 smithi078 ceph-mon[32429]: pgmap v5037: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:28.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:27 smithi107 ceph-mon[36485]: pgmap v5037: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:28.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:27 smithi106 ceph-mon[36505]: pgmap v5037: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:28.471 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9297 sec 2024-03-24T02:59:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8643.227789 secs 2024-03-24T02:59:29.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8643.227789 secs 2024-03-24T02:59:29.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8643.227789 secs 2024-03-24T02:59:29.471 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9298 sec 2024-03-24T02:59:30.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:29 smithi078 ceph-mon[32429]: pgmap v5038: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:30.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:29 smithi107 ceph-mon[36485]: pgmap v5038: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:30.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:29 smithi106 ceph-mon[36505]: pgmap v5038: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:30.471 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9299 sec 2024-03-24T02:59:31.471 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9300 sec 2024-03-24T02:59:32.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:31 smithi078 ceph-mon[32429]: pgmap v5039: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:32.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:31 smithi107 ceph-mon[36485]: pgmap v5039: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:32.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:31 smithi106 ceph-mon[36505]: pgmap v5039: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:32.471 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9301 sec 2024-03-24T02:59:33.471 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9302 sec 2024-03-24T02:59:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:33 smithi078 ceph-mon[32429]: pgmap v5040: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8648.227849 secs 2024-03-24T02:59:34.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:33 smithi107 ceph-mon[36485]: pgmap v5040: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:34.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8648.227849 secs 2024-03-24T02:59:34.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:33 smithi106 ceph-mon[36505]: pgmap v5040: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:34.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8648.227849 secs 2024-03-24T02:59:34.471 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9303 sec 2024-03-24T02:59:35.471 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9304 sec 2024-03-24T02:59:36.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:35 smithi078 ceph-mon[32429]: pgmap v5041: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:36.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:35 smithi107 ceph-mon[36485]: pgmap v5041: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:36.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:35 smithi106 ceph-mon[36505]: pgmap v5041: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:36.471 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9305 sec 2024-03-24T02:59:37.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:37 smithi078 ceph-mon[32429]: pgmap v5042: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:37.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:37 smithi107 ceph-mon[36485]: pgmap v5042: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:37.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:37 smithi106 ceph-mon[36505]: pgmap v5042: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:37.471 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9306 sec 2024-03-24T02:59:38.472 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9307 sec 2024-03-24T02:59:39.472 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9308 sec 2024-03-24T02:59:40.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8653.227936 secs 2024-03-24T02:59:40.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:39 smithi078 ceph-mon[32429]: pgmap v5043: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:40.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:39 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:59:40.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:39 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:59:40.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:39 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:59:40.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:39 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:59:40.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8653.227936 secs 2024-03-24T02:59:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:39 smithi107 ceph-mon[36485]: pgmap v5043: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:39 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:59:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:39 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:59:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:39 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:59:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:39 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:59:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8653.227936 secs 2024-03-24T02:59:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:39 smithi106 ceph-mon[36505]: pgmap v5043: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:39 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:59:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:39 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:59:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:39 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:59:40.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:39 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T02:59:40.472 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9309 sec 2024-03-24T02:59:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:40 smithi078 ceph-mon[32429]: pgmap v5044: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:59:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:40 smithi078 ceph-mon[32429]: pgmap v5045: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:59:41.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:40 smithi107 ceph-mon[36485]: pgmap v5044: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:59:41.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:40 smithi107 ceph-mon[36485]: pgmap v5045: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:59:41.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:40 smithi106 ceph-mon[36505]: pgmap v5044: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T02:59:41.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:40 smithi106 ceph-mon[36505]: pgmap v5045: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T02:59:41.472 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9310 sec 2024-03-24T02:59:42.472 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9311 sec 2024-03-24T02:59:43.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:42 smithi078 ceph-mon[32429]: pgmap v5046: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:59:43.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:42 smithi107 ceph-mon[36485]: pgmap v5046: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:59:43.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:42 smithi106 ceph-mon[36505]: pgmap v5046: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:59:43.472 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9312 sec 2024-03-24T02:59:44.472 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9313 sec 2024-03-24T02:59:44.954 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8658.227995 secs 2024-03-24T02:59:44.954 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:44 smithi106 ceph-mon[36505]: pgmap v5047: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:59:45.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8658.227995 secs 2024-03-24T02:59:45.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:44 smithi078 ceph-mon[32429]: pgmap v5047: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:59:45.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8658.227995 secs 2024-03-24T02:59:45.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:44 smithi107 ceph-mon[36485]: pgmap v5047: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:59:45.472 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9314 sec 2024-03-24T02:59:46.472 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9315 sec 2024-03-24T02:59:47.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:46 smithi078 ceph-mon[32429]: pgmap v5048: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:59:47.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:46 smithi107 ceph-mon[36485]: pgmap v5048: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:59:47.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:46 smithi106 ceph-mon[36505]: pgmap v5048: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:59:47.472 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9316 sec 2024-03-24T02:59:48.472 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9317 sec 2024-03-24T02:59:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:48 smithi078 ceph-mon[32429]: pgmap v5049: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:59:49.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:48 smithi107 ceph-mon[36485]: pgmap v5049: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:59:49.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:48 smithi106 ceph-mon[36505]: pgmap v5049: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T02:59:49.473 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9318 sec 2024-03-24T02:59:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8663.228080 secs 2024-03-24T02:59:50.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8663.228080 secs 2024-03-24T02:59:50.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8663.228080 secs 2024-03-24T02:59:50.473 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9319 sec 2024-03-24T02:59:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:50 smithi078 ceph-mon[32429]: pgmap v5050: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:59:51.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:50 smithi107 ceph-mon[36485]: pgmap v5050: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:59:51.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:50 smithi106 ceph-mon[36505]: pgmap v5050: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T02:59:51.473 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9320 sec 2024-03-24T02:59:52.473 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9321 sec 2024-03-24T02:59:53.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:52 smithi078 ceph-mon[32429]: pgmap v5051: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:53.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:52 smithi107 ceph-mon[36485]: pgmap v5051: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:53.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:52 smithi106 ceph-mon[36505]: pgmap v5051: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:53.473 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9322 sec 2024-03-24T02:59:54.473 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9323 sec 2024-03-24T02:59:55.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8668.228139 secs 2024-03-24T02:59:55.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:54 smithi078 ceph-mon[32429]: pgmap v5052: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:55.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8668.228139 secs 2024-03-24T02:59:55.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:54 smithi107 ceph-mon[36485]: pgmap v5052: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:55.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8668.228139 secs 2024-03-24T02:59:55.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:54 smithi106 ceph-mon[36505]: pgmap v5052: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:55.473 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9324 sec 2024-03-24T02:59:56.473 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9325 sec 2024-03-24T02:59:57.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:56 smithi078 ceph-mon[32429]: pgmap v5053: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:57.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:56 smithi107 ceph-mon[36485]: pgmap v5053: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:57.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:56 smithi106 ceph-mon[36505]: pgmap v5053: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:57.474 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9326 sec 2024-03-24T02:59:58.474 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9327 sec 2024-03-24T02:59:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:58 smithi078 ceph-mon[32429]: pgmap v5054: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:59.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:58 smithi107 ceph-mon[36485]: pgmap v5054: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:59.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:58 smithi106 ceph-mon[36505]: pgmap v5054: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T02:59:59.474 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9328 sec 2024-03-24T03:00:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 02:59:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8673.228228 secs 2024-03-24T03:00:00.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 02:59:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8673.228228 secs 2024-03-24T03:00:00.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 02:59:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8673.228228 secs 2024-03-24T03:00:00.474 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9329 sec 2024-03-24T03:00:01.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:00 smithi078 ceph-mon[32429]: pgmap v5055: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:01.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:00 smithi078 ceph-mon[32429]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T03:00:01.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:00 smithi107 ceph-mon[36485]: pgmap v5055: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:01.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:00 smithi107 ceph-mon[36485]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T03:00:01.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:00 smithi106 ceph-mon[36505]: pgmap v5055: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:01.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:00 smithi106 ceph-mon[36505]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T03:00:01.474 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9330 sec 2024-03-24T03:00:02.474 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9331 sec 2024-03-24T03:00:03.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:02 smithi078 ceph-mon[32429]: pgmap v5056: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:03.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:02 smithi107 ceph-mon[36485]: pgmap v5056: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:03.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:02 smithi106 ceph-mon[36505]: pgmap v5056: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:03.474 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9332 sec 2024-03-24T03:00:04.474 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9333 sec 2024-03-24T03:00:05.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8678.228315 secs 2024-03-24T03:00:05.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:04 smithi078 ceph-mon[32429]: pgmap v5057: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:05.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8678.228315 secs 2024-03-24T03:00:05.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:04 smithi107 ceph-mon[36485]: pgmap v5057: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:05.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8678.228315 secs 2024-03-24T03:00:05.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:04 smithi106 ceph-mon[36505]: pgmap v5057: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:05.474 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9334 sec 2024-03-24T03:00:06.474 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9335 sec 2024-03-24T03:00:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:06 smithi078 ceph-mon[32429]: pgmap v5058: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:07.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:06 smithi107 ceph-mon[36485]: pgmap v5058: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:07.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:06 smithi106 ceph-mon[36505]: pgmap v5058: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:07.474 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9336 sec 2024-03-24T03:00:08.474 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9337 sec 2024-03-24T03:00:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:08 smithi078 ceph-mon[32429]: pgmap v5059: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:09.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:08 smithi107 ceph-mon[36485]: pgmap v5059: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:09.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:08 smithi106 ceph-mon[36505]: pgmap v5059: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:09.474 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9338 sec 2024-03-24T03:00:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8683.228375 secs 2024-03-24T03:00:10.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8683.228375 secs 2024-03-24T03:00:10.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8683.228375 secs 2024-03-24T03:00:10.475 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9339 sec 2024-03-24T03:00:10.991 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:10 smithi078 ceph-mon[32429]: pgmap v5060: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:11.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:10 smithi107 ceph-mon[36485]: pgmap v5060: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:11.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:10 smithi106 ceph-mon[36505]: pgmap v5060: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:11.475 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9340 sec 2024-03-24T03:00:12.475 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9341 sec 2024-03-24T03:00:13.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:12 smithi078 ceph-mon[32429]: pgmap v5061: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:13.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:12 smithi107 ceph-mon[36485]: pgmap v5061: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:13.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:12 smithi106 ceph-mon[36505]: pgmap v5061: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:13.475 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9342 sec 2024-03-24T03:00:14.475 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9343 sec 2024-03-24T03:00:15.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8688.228434 secs 2024-03-24T03:00:15.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:14 smithi078 ceph-mon[32429]: pgmap v5062: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:15.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8688.228434 secs 2024-03-24T03:00:15.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:14 smithi107 ceph-mon[36485]: pgmap v5062: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:15.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8688.228434 secs 2024-03-24T03:00:15.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:14 smithi106 ceph-mon[36505]: pgmap v5062: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:15.475 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9344 sec 2024-03-24T03:00:16.475 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9345 sec 2024-03-24T03:00:17.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:16 smithi078 ceph-mon[32429]: pgmap v5063: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:17.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:16 smithi107 ceph-mon[36485]: pgmap v5063: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:17.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:16 smithi106 ceph-mon[36505]: pgmap v5063: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:17.475 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9346 sec 2024-03-24T03:00:18.475 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9347 sec 2024-03-24T03:00:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:18 smithi078 ceph-mon[32429]: pgmap v5064: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:19.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:18 smithi107 ceph-mon[36485]: pgmap v5064: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:19.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:18 smithi106 ceph-mon[36505]: pgmap v5064: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:19.475 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9348 sec 2024-03-24T03:00:20.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8693.228495 secs 2024-03-24T03:00:20.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8693.228495 secs 2024-03-24T03:00:20.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8693.228495 secs 2024-03-24T03:00:20.475 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9349 sec 2024-03-24T03:00:21.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:20 smithi078 ceph-mon[32429]: pgmap v5065: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:21.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:20 smithi107 ceph-mon[36485]: pgmap v5065: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:21.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:20 smithi106 ceph-mon[36505]: pgmap v5065: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:21.476 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9350 sec 2024-03-24T03:00:22.476 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9351 sec 2024-03-24T03:00:23.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:22 smithi078 ceph-mon[32429]: pgmap v5066: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:23.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:22 smithi107 ceph-mon[36485]: pgmap v5066: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:23.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:22 smithi106 ceph-mon[36505]: pgmap v5066: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:23.476 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9352 sec 2024-03-24T03:00:24.476 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9353 sec 2024-03-24T03:00:25.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8698.228556 secs 2024-03-24T03:00:25.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:24 smithi078 ceph-mon[32429]: pgmap v5067: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:25.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8698.228556 secs 2024-03-24T03:00:25.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:24 smithi107 ceph-mon[36485]: pgmap v5067: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:25.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8698.228556 secs 2024-03-24T03:00:25.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:24 smithi106 ceph-mon[36505]: pgmap v5067: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:25.476 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9354 sec 2024-03-24T03:00:26.476 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9355 sec 2024-03-24T03:00:27.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:26 smithi078 ceph-mon[32429]: pgmap v5068: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:27.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:26 smithi107 ceph-mon[36485]: pgmap v5068: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:27.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:26 smithi106 ceph-mon[36505]: pgmap v5068: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:27.476 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9356 sec 2024-03-24T03:00:28.476 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9357 sec 2024-03-24T03:00:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:28 smithi078 ceph-mon[32429]: pgmap v5069: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8703.228639 secs 2024-03-24T03:00:29.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:28 smithi107 ceph-mon[36485]: pgmap v5069: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:29.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8703.228639 secs 2024-03-24T03:00:29.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:28 smithi106 ceph-mon[36505]: pgmap v5069: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8703.228639 secs 2024-03-24T03:00:29.476 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9358 sec 2024-03-24T03:00:30.476 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9359 sec 2024-03-24T03:00:31.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:30 smithi078 ceph-mon[32429]: pgmap v5070: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:31.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:30 smithi107 ceph-mon[36485]: pgmap v5070: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:31.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:30 smithi106 ceph-mon[36505]: pgmap v5070: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:31.477 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9360 sec 2024-03-24T03:00:32.477 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9361 sec 2024-03-24T03:00:33.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:32 smithi078 ceph-mon[32429]: pgmap v5071: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:33.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:32 smithi107 ceph-mon[36485]: pgmap v5071: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:33.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:32 smithi106 ceph-mon[36505]: pgmap v5071: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:33.477 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9362 sec 2024-03-24T03:00:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8708.228701 secs 2024-03-24T03:00:34.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8708.228701 secs 2024-03-24T03:00:34.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8708.228701 secs 2024-03-24T03:00:34.477 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9363 sec 2024-03-24T03:00:35.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:34 smithi078 ceph-mon[32429]: pgmap v5072: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:35.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:34 smithi107 ceph-mon[36485]: pgmap v5072: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:35.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:34 smithi106 ceph-mon[36505]: pgmap v5072: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:35.477 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9364 sec 2024-03-24T03:00:36.477 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9365 sec 2024-03-24T03:00:37.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:36 smithi078 ceph-mon[32429]: pgmap v5073: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:37.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:36 smithi107 ceph-mon[36485]: pgmap v5073: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:37.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:36 smithi106 ceph-mon[36505]: pgmap v5073: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:37.477 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9366 sec 2024-03-24T03:00:38.477 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9367 sec 2024-03-24T03:00:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:38 smithi078 ceph-mon[32429]: pgmap v5074: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8713.228787 secs 2024-03-24T03:00:39.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:38 smithi107 ceph-mon[36485]: pgmap v5074: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:39.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8713.228787 secs 2024-03-24T03:00:39.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:38 smithi106 ceph-mon[36505]: pgmap v5074: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:39.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8713.228787 secs 2024-03-24T03:00:39.477 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9368 sec 2024-03-24T03:00:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:39 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:00:40.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:39 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:00:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:39 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:00:40.477 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9369 sec 2024-03-24T03:00:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:40 smithi078 ceph-mon[32429]: pgmap v5075: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:40 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:00:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:40 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:00:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:40 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:00:41.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:40 smithi107 ceph-mon[36485]: pgmap v5075: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:41.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:40 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:00:41.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:40 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:00:41.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:40 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:00:41.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:40 smithi106 ceph-mon[36505]: pgmap v5075: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:00:41.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:40 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:00:41.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:40 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:00:41.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:40 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:00:41.477 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9370 sec 2024-03-24T03:00:42.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:41 smithi078 ceph-mon[32429]: pgmap v5076: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:00:42.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:41 smithi078 ceph-mon[32429]: pgmap v5077: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:00:42.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:41 smithi107 ceph-mon[36485]: pgmap v5076: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:00:42.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:41 smithi107 ceph-mon[36485]: pgmap v5077: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:00:42.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:41 smithi106 ceph-mon[36505]: pgmap v5076: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:00:42.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:41 smithi106 ceph-mon[36505]: pgmap v5077: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:00:42.477 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9371 sec 2024-03-24T03:00:43.478 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9372 sec 2024-03-24T03:00:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:43 smithi078 ceph-mon[32429]: pgmap v5078: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:00:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8718.228849 secs 2024-03-24T03:00:44.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:43 smithi107 ceph-mon[36485]: pgmap v5078: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:00:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8718.228849 secs 2024-03-24T03:00:44.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:43 smithi106 ceph-mon[36505]: pgmap v5078: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:00:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8718.228849 secs 2024-03-24T03:00:44.478 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9373 sec 2024-03-24T03:00:45.478 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9374 sec 2024-03-24T03:00:46.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:45 smithi078 ceph-mon[32429]: pgmap v5079: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:00:46.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:45 smithi107 ceph-mon[36485]: pgmap v5079: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:00:46.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:45 smithi106 ceph-mon[36505]: pgmap v5079: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:00:46.478 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9375 sec 2024-03-24T03:00:47.478 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9376 sec 2024-03-24T03:00:48.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:47 smithi078 ceph-mon[32429]: pgmap v5080: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:00:48.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:47 smithi107 ceph-mon[36485]: pgmap v5080: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:00:48.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:47 smithi106 ceph-mon[36505]: pgmap v5080: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:00:48.479 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9377 sec 2024-03-24T03:00:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8723.228939 secs 2024-03-24T03:00:49.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8723.228939 secs 2024-03-24T03:00:49.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8723.228939 secs 2024-03-24T03:00:49.478 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9378 sec 2024-03-24T03:00:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:49 smithi078 ceph-mon[32429]: pgmap v5081: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:00:50.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:49 smithi107 ceph-mon[36485]: pgmap v5081: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:00:50.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:49 smithi106 ceph-mon[36505]: pgmap v5081: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:00:50.479 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9379 sec 2024-03-24T03:00:51.478 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9380 sec 2024-03-24T03:00:52.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:51 smithi078 ceph-mon[32429]: pgmap v5082: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:00:52.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:51 smithi107 ceph-mon[36485]: pgmap v5082: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:00:52.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:51 smithi106 ceph-mon[36505]: pgmap v5082: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:00:52.479 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9381 sec 2024-03-24T03:00:53.478 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9382 sec 2024-03-24T03:00:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:53 smithi078 ceph-mon[32429]: pgmap v5083: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:00:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8728.228990 secs 2024-03-24T03:00:54.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:53 smithi107 ceph-mon[36485]: pgmap v5083: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:00:54.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8728.228990 secs 2024-03-24T03:00:54.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:53 smithi106 ceph-mon[36505]: pgmap v5083: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:00:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8728.228990 secs 2024-03-24T03:00:54.479 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9383 sec 2024-03-24T03:00:55.479 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9384 sec 2024-03-24T03:00:56.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:55 smithi078 ceph-mon[32429]: pgmap v5084: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T03:00:56.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:55 smithi107 ceph-mon[36485]: pgmap v5084: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T03:00:56.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:55 smithi106 ceph-mon[36505]: pgmap v5084: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T03:00:56.479 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9385 sec 2024-03-24T03:00:57.479 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9386 sec 2024-03-24T03:00:58.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:57 smithi078 ceph-mon[32429]: pgmap v5085: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:00:58.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:57 smithi107 ceph-mon[36485]: pgmap v5085: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:00:58.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:57 smithi106 ceph-mon[36505]: pgmap v5085: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:00:58.479 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9387 sec 2024-03-24T03:00:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8733.229051 secs 2024-03-24T03:00:59.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8733.229051 secs 2024-03-24T03:00:59.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8733.229051 secs 2024-03-24T03:00:59.479 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9388 sec 2024-03-24T03:01:00.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:00:59 smithi107 ceph-mon[36485]: pgmap v5086: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:00.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:00:59 smithi106 ceph-mon[36505]: pgmap v5086: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:00.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:00:59 smithi078 ceph-mon[32429]: pgmap v5086: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:00.481 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9389 sec 2024-03-24T03:01:01.479 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9390 sec 2024-03-24T03:01:01.971 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:01 smithi078 ceph-mon[32429]: pgmap v5087: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:02.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:01 smithi106 ceph-mon[36505]: pgmap v5087: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:02.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:01 smithi107 ceph-mon[36485]: pgmap v5087: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:02.479 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9391 sec 2024-03-24T03:01:03.479 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9392 sec 2024-03-24T03:01:04.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:03 smithi106 ceph-mon[36505]: pgmap v5088: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:04.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8738.229134 secs 2024-03-24T03:01:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:03 smithi078 ceph-mon[32429]: pgmap v5088: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8738.229134 secs 2024-03-24T03:01:04.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:03 smithi107 ceph-mon[36485]: pgmap v5088: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:04.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8738.229134 secs 2024-03-24T03:01:04.479 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9393 sec 2024-03-24T03:01:05.480 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9394 sec 2024-03-24T03:01:06.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:05 smithi106 ceph-mon[36505]: pgmap v5089: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:06.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:05 smithi078 ceph-mon[32429]: pgmap v5089: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:06.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:05 smithi107 ceph-mon[36485]: pgmap v5089: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:06.480 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9395 sec 2024-03-24T03:01:07.480 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9396 sec 2024-03-24T03:01:08.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:07 smithi106 ceph-mon[36505]: pgmap v5090: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:08.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:07 smithi078 ceph-mon[32429]: pgmap v5090: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:08.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:07 smithi107 ceph-mon[36485]: pgmap v5090: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:08.480 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9397 sec 2024-03-24T03:01:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8743.229225 secs 2024-03-24T03:01:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8743.229225 secs 2024-03-24T03:01:09.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8743.229225 secs 2024-03-24T03:01:09.480 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9398 sec 2024-03-24T03:01:10.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:09 smithi106 ceph-mon[36505]: pgmap v5091: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:10.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:09 smithi078 ceph-mon[32429]: pgmap v5091: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:10.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:09 smithi107 ceph-mon[36485]: pgmap v5091: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:10.480 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9399 sec 2024-03-24T03:01:11.480 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9400 sec 2024-03-24T03:01:12.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:11 smithi106 ceph-mon[36505]: pgmap v5092: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:12.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:11 smithi078 ceph-mon[32429]: pgmap v5092: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:12.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:11 smithi107 ceph-mon[36485]: pgmap v5092: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:12.480 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9401 sec 2024-03-24T03:01:13.481 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9402 sec 2024-03-24T03:01:14.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:13 smithi106 ceph-mon[36505]: pgmap v5093: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8748.229313 secs 2024-03-24T03:01:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:13 smithi078 ceph-mon[32429]: pgmap v5093: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8748.229313 secs 2024-03-24T03:01:14.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:13 smithi107 ceph-mon[36485]: pgmap v5093: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8748.229313 secs 2024-03-24T03:01:14.481 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9403 sec 2024-03-24T03:01:15.481 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9404 sec 2024-03-24T03:01:16.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:15 smithi106 ceph-mon[36505]: pgmap v5094: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:16.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:15 smithi078 ceph-mon[32429]: pgmap v5094: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:16.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:15 smithi107 ceph-mon[36485]: pgmap v5094: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:16.481 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9405 sec 2024-03-24T03:01:17.481 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9406 sec 2024-03-24T03:01:18.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:17 smithi106 ceph-mon[36505]: pgmap v5095: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:17 smithi078 ceph-mon[32429]: pgmap v5095: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:18.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:17 smithi107 ceph-mon[36485]: pgmap v5095: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:18.481 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9407 sec 2024-03-24T03:01:19.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8753.229352 secs 2024-03-24T03:01:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8753.229352 secs 2024-03-24T03:01:19.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8753.229352 secs 2024-03-24T03:01:19.481 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9408 sec 2024-03-24T03:01:20.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:19 smithi106 ceph-mon[36505]: pgmap v5096: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:20.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:19 smithi078 ceph-mon[32429]: pgmap v5096: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:20.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:19 smithi107 ceph-mon[36485]: pgmap v5096: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:20.481 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9409 sec 2024-03-24T03:01:21.481 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9410 sec 2024-03-24T03:01:22.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:21 smithi106 ceph-mon[36505]: pgmap v5097: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:22.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:21 smithi078 ceph-mon[32429]: pgmap v5097: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:22.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:21 smithi107 ceph-mon[36485]: pgmap v5097: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:22.481 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9411 sec 2024-03-24T03:01:23.481 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9412 sec 2024-03-24T03:01:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:23 smithi106 ceph-mon[36505]: pgmap v5098: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8758.229411 secs 2024-03-24T03:01:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:23 smithi078 ceph-mon[32429]: pgmap v5098: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8758.229411 secs 2024-03-24T03:01:24.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:23 smithi107 ceph-mon[36485]: pgmap v5098: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:24.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8758.229411 secs 2024-03-24T03:01:24.481 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9413 sec 2024-03-24T03:01:25.481 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9414 sec 2024-03-24T03:01:26.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:25 smithi106 ceph-mon[36505]: pgmap v5099: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:26.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:25 smithi078 ceph-mon[32429]: pgmap v5099: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:26.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:25 smithi107 ceph-mon[36485]: pgmap v5099: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:26.481 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9415 sec 2024-03-24T03:01:27.481 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9416 sec 2024-03-24T03:01:27.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:27 smithi106 ceph-mon[36505]: pgmap v5100: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:28.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:27 smithi078 ceph-mon[32429]: pgmap v5100: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:28.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:27 smithi107 ceph-mon[36485]: pgmap v5100: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:28.482 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9417 sec 2024-03-24T03:01:29.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8763.229498 secs 2024-03-24T03:01:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8763.229498 secs 2024-03-24T03:01:29.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8763.229498 secs 2024-03-24T03:01:29.482 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9418 sec 2024-03-24T03:01:30.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:29 smithi106 ceph-mon[36505]: pgmap v5101: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:30.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:29 smithi078 ceph-mon[32429]: pgmap v5101: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:30.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:29 smithi107 ceph-mon[36485]: pgmap v5101: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:30.482 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9419 sec 2024-03-24T03:01:31.482 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9420 sec 2024-03-24T03:01:32.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:31 smithi078 ceph-mon[32429]: pgmap v5102: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:32.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:31 smithi107 ceph-mon[36485]: pgmap v5102: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:32.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:31 smithi106 ceph-mon[36505]: pgmap v5102: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:32.482 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9421 sec 2024-03-24T03:01:33.482 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9422 sec 2024-03-24T03:01:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:33 smithi078 ceph-mon[32429]: pgmap v5103: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8768.229560 secs 2024-03-24T03:01:34.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:33 smithi107 ceph-mon[36485]: pgmap v5103: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8768.229560 secs 2024-03-24T03:01:34.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:33 smithi106 ceph-mon[36505]: pgmap v5103: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:34.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8768.229560 secs 2024-03-24T03:01:34.482 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9423 sec 2024-03-24T03:01:35.483 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9424 sec 2024-03-24T03:01:36.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:35 smithi078 ceph-mon[32429]: pgmap v5104: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:36.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:35 smithi107 ceph-mon[36485]: pgmap v5104: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:36.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:35 smithi106 ceph-mon[36505]: pgmap v5104: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:36.482 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9425 sec 2024-03-24T03:01:37.482 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9426 sec 2024-03-24T03:01:38.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:37 smithi078 ceph-mon[32429]: pgmap v5105: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:38.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:37 smithi107 ceph-mon[36485]: pgmap v5105: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:38.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:37 smithi106 ceph-mon[36505]: pgmap v5105: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:38.483 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9427 sec 2024-03-24T03:01:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8773.229621 secs 2024-03-24T03:01:39.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8773.229621 secs 2024-03-24T03:01:39.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8773.229621 secs 2024-03-24T03:01:39.483 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9428 sec 2024-03-24T03:01:40.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:39 smithi078 ceph-mon[32429]: pgmap v5106: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:40.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:39 smithi107 ceph-mon[36485]: pgmap v5106: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:40.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:39 smithi106 ceph-mon[36505]: pgmap v5106: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:40.483 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9429 sec 2024-03-24T03:01:41.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:40 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:01:41.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:40 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:01:41.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:40 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:01:41.483 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9430 sec 2024-03-24T03:01:42.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:41 smithi078 ceph-mon[32429]: pgmap v5107: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:42.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:41 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:01:42.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:41 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:01:42.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:41 smithi078 ceph-mon[32429]: pgmap v5108: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:01:42.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:41 smithi078 ceph-mon[32429]: pgmap v5109: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:01:42.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:41 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:01:42.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:41 smithi107 ceph-mon[36485]: pgmap v5107: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:42.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:41 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:01:42.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:41 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:01:42.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:41 smithi107 ceph-mon[36485]: pgmap v5108: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:01:42.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:41 smithi107 ceph-mon[36485]: pgmap v5109: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:01:42.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:41 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:01:42.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:41 smithi106 ceph-mon[36505]: pgmap v5107: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:42.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:41 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:01:42.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:41 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:01:42.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:41 smithi106 ceph-mon[36505]: pgmap v5108: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:01:42.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:41 smithi106 ceph-mon[36505]: pgmap v5109: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:01:42.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:41 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:01:42.483 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9431 sec 2024-03-24T03:01:43.483 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9432 sec 2024-03-24T03:01:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:43 smithi078 ceph-mon[32429]: pgmap v5110: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:01:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8778.229680 secs 2024-03-24T03:01:44.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:43 smithi107 ceph-mon[36485]: pgmap v5110: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:01:44.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8778.229680 secs 2024-03-24T03:01:44.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:43 smithi106 ceph-mon[36505]: pgmap v5110: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:01:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8778.229680 secs 2024-03-24T03:01:44.483 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9433 sec 2024-03-24T03:01:45.483 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9434 sec 2024-03-24T03:01:46.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:45 smithi078 ceph-mon[32429]: pgmap v5111: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:01:46.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:45 smithi107 ceph-mon[36485]: pgmap v5111: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:01:46.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:45 smithi106 ceph-mon[36505]: pgmap v5111: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:01:46.483 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9435 sec 2024-03-24T03:01:47.483 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9436 sec 2024-03-24T03:01:48.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:47 smithi078 ceph-mon[32429]: pgmap v5112: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:01:48.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:47 smithi107 ceph-mon[36485]: pgmap v5112: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:01:48.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:47 smithi106 ceph-mon[36505]: pgmap v5112: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:01:48.483 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9437 sec 2024-03-24T03:01:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8783.229740 secs 2024-03-24T03:01:49.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8783.229740 secs 2024-03-24T03:01:49.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8783.229740 secs 2024-03-24T03:01:49.483 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9438 sec 2024-03-24T03:01:50.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:50 smithi078 ceph-mon[32429]: pgmap v5113: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:01:50.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:50 smithi107 ceph-mon[36485]: pgmap v5113: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:01:50.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:50 smithi106 ceph-mon[36505]: pgmap v5113: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:01:50.484 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9439 sec 2024-03-24T03:01:51.484 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9440 sec 2024-03-24T03:01:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:52 smithi078 ceph-mon[32429]: pgmap v5114: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:01:52.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:52 smithi107 ceph-mon[36485]: pgmap v5114: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:01:52.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:52 smithi106 ceph-mon[36505]: pgmap v5114: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:01:52.484 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9441 sec 2024-03-24T03:01:53.484 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9442 sec 2024-03-24T03:01:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:54 smithi078 ceph-mon[32429]: pgmap v5115: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8788.229800 secs 2024-03-24T03:01:54.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:54 smithi107 ceph-mon[36485]: pgmap v5115: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:54.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8788.229800 secs 2024-03-24T03:01:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:54 smithi106 ceph-mon[36505]: pgmap v5115: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8788.229800 secs 2024-03-24T03:01:54.484 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9443 sec 2024-03-24T03:01:55.484 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9444 sec 2024-03-24T03:01:56.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:56 smithi078 ceph-mon[32429]: pgmap v5116: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:56.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:56 smithi107 ceph-mon[36485]: pgmap v5116: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:56.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:56 smithi106 ceph-mon[36505]: pgmap v5116: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:56.484 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9445 sec 2024-03-24T03:01:57.484 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9446 sec 2024-03-24T03:01:58.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:58 smithi078 ceph-mon[32429]: pgmap v5117: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:58.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:58 smithi107 ceph-mon[36485]: pgmap v5117: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:58.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:58 smithi106 ceph-mon[36505]: pgmap v5117: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:58.484 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9447 sec 2024-03-24T03:01:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8793.229860 secs 2024-03-24T03:01:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:01:59 smithi078 ceph-mon[32429]: pgmap v5118: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:59.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8793.229860 secs 2024-03-24T03:01:59.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:01:59 smithi107 ceph-mon[36485]: pgmap v5118: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:59.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8793.229860 secs 2024-03-24T03:01:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:01:59 smithi106 ceph-mon[36505]: pgmap v5118: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:01:59.484 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9448 sec 2024-03-24T03:02:00.485 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9449 sec 2024-03-24T03:02:01.485 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9450 sec 2024-03-24T03:02:02.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:01 smithi078 ceph-mon[32429]: pgmap v5119: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:02.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:01 smithi107 ceph-mon[36485]: pgmap v5119: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:02.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:01 smithi106 ceph-mon[36505]: pgmap v5119: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:02.485 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9451 sec 2024-03-24T03:02:03.485 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9452 sec 2024-03-24T03:02:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:04 smithi078 ceph-mon[32429]: pgmap v5120: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8798.229919 secs 2024-03-24T03:02:04.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:04 smithi107 ceph-mon[36485]: pgmap v5120: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:04.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8798.229919 secs 2024-03-24T03:02:04.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:04 smithi106 ceph-mon[36505]: pgmap v5120: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8798.229919 secs 2024-03-24T03:02:04.485 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9453 sec 2024-03-24T03:02:05.485 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9454 sec 2024-03-24T03:02:06.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:06 smithi078 ceph-mon[32429]: pgmap v5121: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:06.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:06 smithi107 ceph-mon[36485]: pgmap v5121: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:06.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:06 smithi106 ceph-mon[36505]: pgmap v5121: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:06.485 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9455 sec 2024-03-24T03:02:07.485 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9456 sec 2024-03-24T03:02:08.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:08 smithi078 ceph-mon[32429]: pgmap v5122: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:08.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:08 smithi107 ceph-mon[36485]: pgmap v5122: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:08.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:08 smithi106 ceph-mon[36505]: pgmap v5122: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:08.485 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9457 sec 2024-03-24T03:02:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8803.230006 secs 2024-03-24T03:02:09.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8803.230006 secs 2024-03-24T03:02:09.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8803.230006 secs 2024-03-24T03:02:09.485 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9458 sec 2024-03-24T03:02:10.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:10 smithi078 ceph-mon[32429]: pgmap v5123: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:10.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:10 smithi107 ceph-mon[36485]: pgmap v5123: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:10.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:10 smithi106 ceph-mon[36505]: pgmap v5123: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:10.485 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9459 sec 2024-03-24T03:02:11.485 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9460 sec 2024-03-24T03:02:12.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:12 smithi078 ceph-mon[32429]: pgmap v5124: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:12.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:12 smithi107 ceph-mon[36485]: pgmap v5124: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:12.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:12 smithi106 ceph-mon[36505]: pgmap v5124: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:12.485 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9461 sec 2024-03-24T03:02:13.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:13 smithi078 ceph-mon[32429]: pgmap v5125: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:13.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:13 smithi107 ceph-mon[36485]: pgmap v5125: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:13.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:13 smithi106 ceph-mon[36505]: pgmap v5125: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:13.486 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9462 sec 2024-03-24T03:02:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8808.230095 secs 2024-03-24T03:02:14.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8808.230095 secs 2024-03-24T03:02:14.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8808.230095 secs 2024-03-24T03:02:14.485 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9463 sec 2024-03-24T03:02:15.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:15 smithi078 ceph-mon[32429]: pgmap v5126: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:15.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:15 smithi107 ceph-mon[36485]: pgmap v5126: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:15.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:15 smithi106 ceph-mon[36505]: pgmap v5126: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:15.486 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9464 sec 2024-03-24T03:02:16.486 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9465 sec 2024-03-24T03:02:17.486 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9466 sec 2024-03-24T03:02:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:18 smithi078 ceph-mon[32429]: pgmap v5127: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:18.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:18 smithi107 ceph-mon[36485]: pgmap v5127: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:18.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:18 smithi106 ceph-mon[36505]: pgmap v5127: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:18.486 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9467 sec 2024-03-24T03:02:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8813.230158 secs 2024-03-24T03:02:19.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8813.230158 secs 2024-03-24T03:02:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8813.230158 secs 2024-03-24T03:02:19.486 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9468 sec 2024-03-24T03:02:20.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:20 smithi078 ceph-mon[32429]: pgmap v5128: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:20.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:20 smithi107 ceph-mon[36485]: pgmap v5128: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:20.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:20 smithi106 ceph-mon[36505]: pgmap v5128: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:20.486 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9469 sec 2024-03-24T03:02:21.486 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9470 sec 2024-03-24T03:02:22.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:22 smithi078 ceph-mon[32429]: pgmap v5129: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:22.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:22 smithi107 ceph-mon[36485]: pgmap v5129: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:22.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:22 smithi106 ceph-mon[36505]: pgmap v5129: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:22.486 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9471 sec 2024-03-24T03:02:23.486 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9472 sec 2024-03-24T03:02:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:24 smithi078 ceph-mon[32429]: pgmap v5130: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8818.230199 secs 2024-03-24T03:02:24.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:24 smithi107 ceph-mon[36485]: pgmap v5130: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:24.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8818.230199 secs 2024-03-24T03:02:24.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:24 smithi106 ceph-mon[36505]: pgmap v5130: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:24.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8818.230199 secs 2024-03-24T03:02:24.486 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9473 sec 2024-03-24T03:02:25.486 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9474 sec 2024-03-24T03:02:26.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:26 smithi078 ceph-mon[32429]: pgmap v5131: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:26.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:26 smithi107 ceph-mon[36485]: pgmap v5131: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:26.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:26 smithi106 ceph-mon[36505]: pgmap v5131: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:26.487 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9475 sec 2024-03-24T03:02:27.486 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9476 sec 2024-03-24T03:02:28.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:28 smithi078 ceph-mon[32429]: pgmap v5132: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:28.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:28 smithi107 ceph-mon[36485]: pgmap v5132: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:28.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:28 smithi106 ceph-mon[36505]: pgmap v5132: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:28.486 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9477 sec 2024-03-24T03:02:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8823.230238 secs 2024-03-24T03:02:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:29 smithi078 ceph-mon[32429]: pgmap v5133: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:29.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8823.230238 secs 2024-03-24T03:02:29.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:29 smithi107 ceph-mon[36485]: pgmap v5133: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:29.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8823.230238 secs 2024-03-24T03:02:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:29 smithi106 ceph-mon[36505]: pgmap v5133: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:29.487 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9478 sec 2024-03-24T03:02:30.487 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9479 sec 2024-03-24T03:02:31.487 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9480 sec 2024-03-24T03:02:32.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:32 smithi078 ceph-mon[32429]: pgmap v5134: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:32.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:32 smithi107 ceph-mon[36485]: pgmap v5134: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:32.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:32 smithi106 ceph-mon[36505]: pgmap v5134: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:32.487 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9481 sec 2024-03-24T03:02:33.487 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9482 sec 2024-03-24T03:02:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:34 smithi078 ceph-mon[32429]: pgmap v5135: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8828.230295 secs 2024-03-24T03:02:34.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:34 smithi107 ceph-mon[36485]: pgmap v5135: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:34.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8828.230295 secs 2024-03-24T03:02:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:34 smithi106 ceph-mon[36505]: pgmap v5135: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8828.230295 secs 2024-03-24T03:02:34.487 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9483 sec 2024-03-24T03:02:35.487 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9484 sec 2024-03-24T03:02:36.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:36 smithi078 ceph-mon[32429]: pgmap v5136: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:36.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:36 smithi107 ceph-mon[36485]: pgmap v5136: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:36.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:36 smithi106 ceph-mon[36505]: pgmap v5136: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:36.487 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9485 sec 2024-03-24T03:02:37.487 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9486 sec 2024-03-24T03:02:38.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:38 smithi078 ceph-mon[32429]: pgmap v5137: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:38.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:38 smithi107 ceph-mon[36485]: pgmap v5137: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:38.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:38 smithi106 ceph-mon[36505]: pgmap v5137: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:38.487 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9487 sec 2024-03-24T03:02:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8833.230381 secs 2024-03-24T03:02:39.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8833.230381 secs 2024-03-24T03:02:39.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8833.230381 secs 2024-03-24T03:02:39.487 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9488 sec 2024-03-24T03:02:40.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:40 smithi078 ceph-mon[32429]: pgmap v5138: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:40.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:40 smithi107 ceph-mon[36485]: pgmap v5138: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:40.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:40 smithi106 ceph-mon[36505]: pgmap v5138: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:40.487 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9489 sec 2024-03-24T03:02:41.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:41 smithi078 ceph-mon[32429]: pgmap v5139: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:41.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:41 smithi107 ceph-mon[36485]: pgmap v5139: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:41.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:41 smithi106 ceph-mon[36505]: pgmap v5139: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:41.487 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9490 sec 2024-03-24T03:02:42.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:42 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:02:42.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:42 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:02:42.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:42 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:02:42.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:42 smithi078 ceph-mon[32429]: pgmap v5140: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:02:42.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:42 smithi078 ceph-mon[32429]: pgmap v5141: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:02:42.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:42 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:02:42.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:42 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:02:42.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:42 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:02:42.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:42 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:02:42.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:42 smithi107 ceph-mon[36485]: pgmap v5140: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:02:42.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:42 smithi107 ceph-mon[36485]: pgmap v5141: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:02:42.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:42 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:02:42.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:42 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:02:42.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:42 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:02:42.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:42 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:02:42.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:42 smithi106 ceph-mon[36505]: pgmap v5140: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:02:42.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:42 smithi106 ceph-mon[36505]: pgmap v5141: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:02:42.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:42 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:02:42.487 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9491 sec 2024-03-24T03:02:43.489 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9492 sec 2024-03-24T03:02:44.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8838.230465 secs 2024-03-24T03:02:44.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8838.230465 secs 2024-03-24T03:02:44.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8838.230465 secs 2024-03-24T03:02:44.488 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9493 sec 2024-03-24T03:02:45.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:44 smithi078 ceph-mon[32429]: pgmap v5142: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:02:45.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:44 smithi107 ceph-mon[36485]: pgmap v5142: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:02:45.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:44 smithi106 ceph-mon[36505]: pgmap v5142: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:02:45.488 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9494 sec 2024-03-24T03:02:46.488 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9495 sec 2024-03-24T03:02:47.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:46 smithi078 ceph-mon[32429]: pgmap v5143: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:02:47.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:46 smithi107 ceph-mon[36485]: pgmap v5143: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:02:47.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:46 smithi106 ceph-mon[36505]: pgmap v5143: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:02:47.488 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9496 sec 2024-03-24T03:02:48.488 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9497 sec 2024-03-24T03:02:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:48 smithi078 ceph-mon[32429]: pgmap v5144: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:02:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8843.230554 secs 2024-03-24T03:02:49.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:48 smithi107 ceph-mon[36485]: pgmap v5144: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:02:49.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8843.230554 secs 2024-03-24T03:02:49.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:48 smithi106 ceph-mon[36505]: pgmap v5144: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:02:49.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8843.230554 secs 2024-03-24T03:02:49.488 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9498 sec 2024-03-24T03:02:50.488 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9499 sec 2024-03-24T03:02:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:50 smithi078 ceph-mon[32429]: pgmap v5145: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:02:51.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:50 smithi107 ceph-mon[36485]: pgmap v5145: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:02:51.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:50 smithi106 ceph-mon[36505]: pgmap v5145: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:02:51.488 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9500 sec 2024-03-24T03:02:52.489 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9501 sec 2024-03-24T03:02:52.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:52 smithi078 ceph-mon[32429]: pgmap v5146: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:02:52.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:52 smithi107 ceph-mon[36485]: pgmap v5146: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:02:52.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:52 smithi106 ceph-mon[36505]: pgmap v5146: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:02:53.489 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9502 sec 2024-03-24T03:02:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8848.230640 secs 2024-03-24T03:02:54.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8848.230640 secs 2024-03-24T03:02:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8848.230640 secs 2024-03-24T03:02:54.489 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9503 sec 2024-03-24T03:02:55.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:54 smithi078 ceph-mon[32429]: pgmap v5147: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:55.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:54 smithi107 ceph-mon[36485]: pgmap v5147: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:55.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:54 smithi106 ceph-mon[36505]: pgmap v5147: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:55.489 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9504 sec 2024-03-24T03:02:56.489 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9505 sec 2024-03-24T03:02:57.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:56 smithi078 ceph-mon[32429]: pgmap v5148: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:57.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:56 smithi107 ceph-mon[36485]: pgmap v5148: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:57.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:56 smithi106 ceph-mon[36505]: pgmap v5148: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:57.489 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9506 sec 2024-03-24T03:02:58.489 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9507 sec 2024-03-24T03:02:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:58 smithi078 ceph-mon[32429]: pgmap v5149: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:02:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8853.230700 secs 2024-03-24T03:02:59.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:58 smithi107 ceph-mon[36485]: pgmap v5149: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:59.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:02:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8853.230700 secs 2024-03-24T03:02:59.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:58 smithi106 ceph-mon[36505]: pgmap v5149: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:02:59.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:02:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8853.230700 secs 2024-03-24T03:02:59.489 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9508 sec 2024-03-24T03:03:00.489 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9509 sec 2024-03-24T03:03:01.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:00 smithi078 ceph-mon[32429]: pgmap v5150: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:01.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:00 smithi107 ceph-mon[36485]: pgmap v5150: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:01.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:00 smithi106 ceph-mon[36505]: pgmap v5150: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:01.489 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9510 sec 2024-03-24T03:03:02.489 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9511 sec 2024-03-24T03:03:03.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:02 smithi078 ceph-mon[32429]: pgmap v5151: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:03.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:02 smithi107 ceph-mon[36485]: pgmap v5151: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:03.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:02 smithi106 ceph-mon[36505]: pgmap v5151: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:03.489 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9512 sec 2024-03-24T03:03:04.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8858.230789 secs 2024-03-24T03:03:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8858.230789 secs 2024-03-24T03:03:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8858.230789 secs 2024-03-24T03:03:04.489 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9513 sec 2024-03-24T03:03:05.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:04 smithi106 ceph-mon[36505]: pgmap v5152: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:05.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:04 smithi078 ceph-mon[32429]: pgmap v5152: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:05.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:04 smithi107 ceph-mon[36485]: pgmap v5152: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:05.489 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9514 sec 2024-03-24T03:03:06.490 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9515 sec 2024-03-24T03:03:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:06 smithi078 ceph-mon[32429]: pgmap v5153: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:07.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:06 smithi106 ceph-mon[36505]: pgmap v5153: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:07.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:06 smithi107 ceph-mon[36485]: pgmap v5153: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:07.490 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9516 sec 2024-03-24T03:03:08.490 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9517 sec 2024-03-24T03:03:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:08 smithi106 ceph-mon[36505]: pgmap v5154: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8863.230834 secs 2024-03-24T03:03:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:08 smithi078 ceph-mon[32429]: pgmap v5154: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8863.230834 secs 2024-03-24T03:03:09.354 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:08 smithi107 ceph-mon[36485]: pgmap v5154: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:09.354 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8863.230834 secs 2024-03-24T03:03:09.490 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9518 sec 2024-03-24T03:03:10.490 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9519 sec 2024-03-24T03:03:11.141 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:10 smithi078 ceph-mon[32429]: pgmap v5155: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:11.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:10 smithi106 ceph-mon[36505]: pgmap v5155: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:11.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:10 smithi107 ceph-mon[36485]: pgmap v5155: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:11.490 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9520 sec 2024-03-24T03:03:12.490 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9521 sec 2024-03-24T03:03:13.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:12 smithi106 ceph-mon[36505]: pgmap v5156: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:13.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:12 smithi078 ceph-mon[32429]: pgmap v5156: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:13.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:12 smithi107 ceph-mon[36485]: pgmap v5156: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:13.490 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9522 sec 2024-03-24T03:03:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8868.230873 secs 2024-03-24T03:03:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8868.230873 secs 2024-03-24T03:03:14.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8868.230873 secs 2024-03-24T03:03:14.490 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9523 sec 2024-03-24T03:03:15.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:14 smithi106 ceph-mon[36505]: pgmap v5157: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:15.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:14 smithi078 ceph-mon[32429]: pgmap v5157: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:15.345 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:14 smithi107 ceph-mon[36485]: pgmap v5157: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:15.490 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9524 sec 2024-03-24T03:03:16.490 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9525 sec 2024-03-24T03:03:17.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:16 smithi106 ceph-mon[36505]: pgmap v5158: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:17.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:16 smithi078 ceph-mon[32429]: pgmap v5158: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:17.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:16 smithi107 ceph-mon[36485]: pgmap v5158: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:17.490 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9526 sec 2024-03-24T03:03:18.491 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9527 sec 2024-03-24T03:03:19.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:18 smithi106 ceph-mon[36505]: pgmap v5159: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8873.230962 secs 2024-03-24T03:03:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:18 smithi078 ceph-mon[32429]: pgmap v5159: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8873.230962 secs 2024-03-24T03:03:19.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:18 smithi107 ceph-mon[36485]: pgmap v5159: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:19.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8873.230962 secs 2024-03-24T03:03:19.491 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9528 sec 2024-03-24T03:03:20.491 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9529 sec 2024-03-24T03:03:21.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:20 smithi106 ceph-mon[36505]: pgmap v5160: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:21.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:20 smithi078 ceph-mon[32429]: pgmap v5160: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:21.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:20 smithi107 ceph-mon[36485]: pgmap v5160: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:21.491 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9530 sec 2024-03-24T03:03:22.491 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9531 sec 2024-03-24T03:03:23.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:22 smithi106 ceph-mon[36505]: pgmap v5161: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:23.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:22 smithi078 ceph-mon[32429]: pgmap v5161: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:23.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:22 smithi107 ceph-mon[36485]: pgmap v5161: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:23.491 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9532 sec 2024-03-24T03:03:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8878.231054 secs 2024-03-24T03:03:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8878.231054 secs 2024-03-24T03:03:24.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8878.231054 secs 2024-03-24T03:03:24.491 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9533 sec 2024-03-24T03:03:25.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:24 smithi106 ceph-mon[36505]: pgmap v5162: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:24 smithi078 ceph-mon[32429]: pgmap v5162: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:25.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:24 smithi107 ceph-mon[36485]: pgmap v5162: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:25.491 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9534 sec 2024-03-24T03:03:26.491 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9535 sec 2024-03-24T03:03:27.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:26 smithi106 ceph-mon[36505]: pgmap v5163: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:27.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:26 smithi078 ceph-mon[32429]: pgmap v5163: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:27.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:26 smithi107 ceph-mon[36485]: pgmap v5163: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:27.491 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9536 sec 2024-03-24T03:03:28.491 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9537 sec 2024-03-24T03:03:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:28 smithi106 ceph-mon[36505]: pgmap v5164: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8883.231096 secs 2024-03-24T03:03:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:28 smithi078 ceph-mon[32429]: pgmap v5164: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8883.231096 secs 2024-03-24T03:03:29.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:28 smithi107 ceph-mon[36485]: pgmap v5164: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:29.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8883.231096 secs 2024-03-24T03:03:29.491 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9538 sec 2024-03-24T03:03:30.492 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9539 sec 2024-03-24T03:03:31.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:30 smithi106 ceph-mon[36505]: pgmap v5165: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:31.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:30 smithi078 ceph-mon[32429]: pgmap v5165: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:31.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:30 smithi107 ceph-mon[36485]: pgmap v5165: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:31.492 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9540 sec 2024-03-24T03:03:32.492 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9541 sec 2024-03-24T03:03:33.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:32 smithi106 ceph-mon[36505]: pgmap v5166: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:33.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:32 smithi078 ceph-mon[32429]: pgmap v5166: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:33.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:32 smithi107 ceph-mon[36485]: pgmap v5166: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:33.492 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9542 sec 2024-03-24T03:03:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8888.231156 secs 2024-03-24T03:03:34.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8888.231156 secs 2024-03-24T03:03:34.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8888.231156 secs 2024-03-24T03:03:34.492 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9543 sec 2024-03-24T03:03:35.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:34 smithi078 ceph-mon[32429]: pgmap v5167: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:35.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:34 smithi107 ceph-mon[36485]: pgmap v5167: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:35.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:34 smithi106 ceph-mon[36505]: pgmap v5167: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:35.492 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9544 sec 2024-03-24T03:03:36.492 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9545 sec 2024-03-24T03:03:37.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:36 smithi078 ceph-mon[32429]: pgmap v5168: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:37.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:36 smithi107 ceph-mon[36485]: pgmap v5168: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:37.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:36 smithi106 ceph-mon[36505]: pgmap v5168: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:37.492 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9546 sec 2024-03-24T03:03:38.492 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9547 sec 2024-03-24T03:03:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:38 smithi078 ceph-mon[32429]: pgmap v5169: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8893.231243 secs 2024-03-24T03:03:39.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:38 smithi107 ceph-mon[36485]: pgmap v5169: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:39.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8893.231243 secs 2024-03-24T03:03:39.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:38 smithi106 ceph-mon[36505]: pgmap v5169: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8893.231243 secs 2024-03-24T03:03:39.492 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9548 sec 2024-03-24T03:03:40.493 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9549 sec 2024-03-24T03:03:41.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:40 smithi078 ceph-mon[32429]: pgmap v5170: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:41.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:40 smithi107 ceph-mon[36485]: pgmap v5170: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:41.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:40 smithi106 ceph-mon[36505]: pgmap v5170: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:41.492 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9550 sec 2024-03-24T03:03:42.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:41 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:03:42.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:41 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:03:42.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:41 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:03:42.493 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9551 sec 2024-03-24T03:03:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:42 smithi078 ceph-mon[32429]: pgmap v5171: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:42 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:03:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:42 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:03:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:42 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:03:43.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:42 smithi107 ceph-mon[36485]: pgmap v5171: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:43.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:42 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:03:43.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:42 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:03:43.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:42 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:03:43.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:42 smithi106 ceph-mon[36505]: pgmap v5171: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:43.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:42 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:03:43.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:42 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:03:43.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:42 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:03:43.493 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9552 sec 2024-03-24T03:03:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:43 smithi078 ceph-mon[32429]: pgmap v5172: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:03:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:43 smithi078 ceph-mon[32429]: pgmap v5173: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:03:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8898.231339 secs 2024-03-24T03:03:44.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:43 smithi107 ceph-mon[36485]: pgmap v5172: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:03:44.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:43 smithi107 ceph-mon[36485]: pgmap v5173: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:03:44.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8898.231339 secs 2024-03-24T03:03:44.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:43 smithi106 ceph-mon[36505]: pgmap v5172: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:03:44.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:43 smithi106 ceph-mon[36505]: pgmap v5173: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:03:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8898.231339 secs 2024-03-24T03:03:44.493 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9553 sec 2024-03-24T03:03:45.493 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9554 sec 2024-03-24T03:03:46.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:45 smithi078 ceph-mon[32429]: pgmap v5174: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:03:46.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:45 smithi107 ceph-mon[36485]: pgmap v5174: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:03:46.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:45 smithi106 ceph-mon[36505]: pgmap v5174: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:03:46.493 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9555 sec 2024-03-24T03:03:47.493 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9556 sec 2024-03-24T03:03:48.203 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:47 smithi106 ceph-mon[36505]: pgmap v5175: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:03:48.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:47 smithi078 ceph-mon[32429]: pgmap v5175: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:03:48.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:47 smithi107 ceph-mon[36485]: pgmap v5175: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:03:48.493 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9557 sec 2024-03-24T03:03:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8903.231402 secs 2024-03-24T03:03:49.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8903.231402 secs 2024-03-24T03:03:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8903.231402 secs 2024-03-24T03:03:49.493 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9558 sec 2024-03-24T03:03:50.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:49 smithi078 ceph-mon[32429]: pgmap v5176: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:03:50.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:49 smithi107 ceph-mon[36485]: pgmap v5176: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:03:50.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:49 smithi106 ceph-mon[36505]: pgmap v5176: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:03:50.493 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9559 sec 2024-03-24T03:03:51.493 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9560 sec 2024-03-24T03:03:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:51 smithi078 ceph-mon[32429]: pgmap v5177: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:03:52.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:51 smithi107 ceph-mon[36485]: pgmap v5177: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:03:52.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:51 smithi106 ceph-mon[36505]: pgmap v5177: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:03:52.493 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9561 sec 2024-03-24T03:03:53.494 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9562 sec 2024-03-24T03:03:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:53 smithi078 ceph-mon[32429]: pgmap v5178: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:03:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8908.231462 secs 2024-03-24T03:03:54.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:53 smithi107 ceph-mon[36485]: pgmap v5178: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:03:54.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8908.231462 secs 2024-03-24T03:03:54.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:53 smithi106 ceph-mon[36505]: pgmap v5178: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:03:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8908.231462 secs 2024-03-24T03:03:54.494 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9563 sec 2024-03-24T03:03:55.494 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9564 sec 2024-03-24T03:03:56.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:55 smithi078 ceph-mon[32429]: pgmap v5179: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:56.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:55 smithi107 ceph-mon[36485]: pgmap v5179: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:56.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:55 smithi106 ceph-mon[36505]: pgmap v5179: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:03:56.494 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9565 sec 2024-03-24T03:03:57.494 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9566 sec 2024-03-24T03:03:58.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:57 smithi078 ceph-mon[32429]: pgmap v5180: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:58.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:57 smithi107 ceph-mon[36485]: pgmap v5180: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:58.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:57 smithi106 ceph-mon[36505]: pgmap v5180: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:58.494 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9567 sec 2024-03-24T03:03:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8913.231523 secs 2024-03-24T03:03:59.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8913.231523 secs 2024-03-24T03:03:59.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8913.231523 secs 2024-03-24T03:03:59.494 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9568 sec 2024-03-24T03:04:00.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:03:59 smithi078 ceph-mon[32429]: pgmap v5181: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:00.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:03:59 smithi107 ceph-mon[36485]: pgmap v5181: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:00.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:03:59 smithi106 ceph-mon[36505]: pgmap v5181: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:00.494 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9569 sec 2024-03-24T03:04:01.494 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9570 sec 2024-03-24T03:04:02.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:01 smithi078 ceph-mon[32429]: pgmap v5182: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:02.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:01 smithi107 ceph-mon[36485]: pgmap v5182: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:02.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:01 smithi106 ceph-mon[36505]: pgmap v5182: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:02.494 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9571 sec 2024-03-24T03:04:03.494 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9572 sec 2024-03-24T03:04:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:03 smithi078 ceph-mon[32429]: pgmap v5183: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8918.231611 secs 2024-03-24T03:04:04.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:03 smithi107 ceph-mon[36485]: pgmap v5183: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:04.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8918.231611 secs 2024-03-24T03:04:04.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:03 smithi106 ceph-mon[36505]: pgmap v5183: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8918.231611 secs 2024-03-24T03:04:04.495 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9573 sec 2024-03-24T03:04:05.495 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9574 sec 2024-03-24T03:04:06.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:05 smithi078 ceph-mon[32429]: pgmap v5184: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:06.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:05 smithi107 ceph-mon[36485]: pgmap v5184: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:06.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:05 smithi106 ceph-mon[36505]: pgmap v5184: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:06.495 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9575 sec 2024-03-24T03:04:07.495 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9576 sec 2024-03-24T03:04:08.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:07 smithi078 ceph-mon[32429]: pgmap v5185: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:08.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:07 smithi107 ceph-mon[36485]: pgmap v5185: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:08.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:07 smithi106 ceph-mon[36505]: pgmap v5185: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:08.495 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9577 sec 2024-03-24T03:04:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8923.231625 secs 2024-03-24T03:04:09.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8923.231625 secs 2024-03-24T03:04:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8923.231625 secs 2024-03-24T03:04:09.495 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9578 sec 2024-03-24T03:04:10.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:09 smithi078 ceph-mon[32429]: pgmap v5186: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:10.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:09 smithi107 ceph-mon[36485]: pgmap v5186: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:10.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:09 smithi106 ceph-mon[36505]: pgmap v5186: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:10.495 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9579 sec 2024-03-24T03:04:11.495 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9580 sec 2024-03-24T03:04:12.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:11 smithi078 ceph-mon[32429]: pgmap v5187: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:12.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:11 smithi107 ceph-mon[36485]: pgmap v5187: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:12.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:11 smithi106 ceph-mon[36505]: pgmap v5187: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:12.495 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9581 sec 2024-03-24T03:04:13.495 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9582 sec 2024-03-24T03:04:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:13 smithi078 ceph-mon[32429]: pgmap v5188: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8928.231686 secs 2024-03-24T03:04:14.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:13 smithi107 ceph-mon[36485]: pgmap v5188: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:14.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8928.231686 secs 2024-03-24T03:04:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:13 smithi106 ceph-mon[36505]: pgmap v5188: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8928.231686 secs 2024-03-24T03:04:14.495 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9583 sec 2024-03-24T03:04:15.495 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9584 sec 2024-03-24T03:04:16.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:15 smithi078 ceph-mon[32429]: pgmap v5189: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:16.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:15 smithi107 ceph-mon[36485]: pgmap v5189: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:16.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:15 smithi106 ceph-mon[36505]: pgmap v5189: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:16.495 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9585 sec 2024-03-24T03:04:17.496 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9586 sec 2024-03-24T03:04:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:17 smithi078 ceph-mon[32429]: pgmap v5190: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:18.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:17 smithi107 ceph-mon[36485]: pgmap v5190: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:18.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:17 smithi106 ceph-mon[36505]: pgmap v5190: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:18.495 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9587 sec 2024-03-24T03:04:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8933.231746 secs 2024-03-24T03:04:19.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8933.231746 secs 2024-03-24T03:04:19.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8933.231746 secs 2024-03-24T03:04:19.496 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9588 sec 2024-03-24T03:04:20.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:20 smithi078 ceph-mon[32429]: pgmap v5191: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:20.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:20 smithi107 ceph-mon[36485]: pgmap v5191: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:20.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:20 smithi106 ceph-mon[36505]: pgmap v5191: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:20.495 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9589 sec 2024-03-24T03:04:21.496 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9590 sec 2024-03-24T03:04:22.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:22 smithi078 ceph-mon[32429]: pgmap v5192: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:22.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:22 smithi107 ceph-mon[36485]: pgmap v5192: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:22.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:22 smithi106 ceph-mon[36505]: pgmap v5192: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:22.496 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9591 sec 2024-03-24T03:04:23.496 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9592 sec 2024-03-24T03:04:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:24 smithi078 ceph-mon[32429]: pgmap v5193: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8938.231806 secs 2024-03-24T03:04:24.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:24 smithi107 ceph-mon[36485]: pgmap v5193: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:24.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8938.231806 secs 2024-03-24T03:04:24.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:24 smithi106 ceph-mon[36505]: pgmap v5193: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8938.231806 secs 2024-03-24T03:04:24.496 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9593 sec 2024-03-24T03:04:25.496 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9594 sec 2024-03-24T03:04:26.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:26 smithi078 ceph-mon[32429]: pgmap v5194: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:26.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:26 smithi107 ceph-mon[36485]: pgmap v5194: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:26.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:26 smithi106 ceph-mon[36505]: pgmap v5194: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:26.496 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9595 sec 2024-03-24T03:04:27.496 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9596 sec 2024-03-24T03:04:28.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:28 smithi078 ceph-mon[32429]: pgmap v5195: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:28.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:28 smithi107 ceph-mon[36485]: pgmap v5195: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:28.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:28 smithi106 ceph-mon[36505]: pgmap v5195: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:28.496 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9597 sec 2024-03-24T03:04:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8943.231865 secs 2024-03-24T03:04:29.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8943.231865 secs 2024-03-24T03:04:29.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8943.231865 secs 2024-03-24T03:04:29.496 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9598 sec 2024-03-24T03:04:30.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:30 smithi078 ceph-mon[32429]: pgmap v5196: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:30.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:30 smithi107 ceph-mon[36485]: pgmap v5196: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:30.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:30 smithi106 ceph-mon[36505]: pgmap v5196: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:30.496 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9599 sec 2024-03-24T03:04:31.496 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9600 sec 2024-03-24T03:04:32.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:32 smithi078 ceph-mon[32429]: pgmap v5197: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:32.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:32 smithi107 ceph-mon[36485]: pgmap v5197: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:32.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:32 smithi106 ceph-mon[36505]: pgmap v5197: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:32.497 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9601 sec 2024-03-24T03:04:33.497 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9602 sec 2024-03-24T03:04:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:34 smithi078 ceph-mon[32429]: pgmap v5198: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8948.231924 secs 2024-03-24T03:04:34.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:34 smithi107 ceph-mon[36485]: pgmap v5198: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:34.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8948.231924 secs 2024-03-24T03:04:34.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:34 smithi106 ceph-mon[36505]: pgmap v5198: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:34.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8948.231924 secs 2024-03-24T03:04:34.497 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9603 sec 2024-03-24T03:04:35.497 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9604 sec 2024-03-24T03:04:36.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:36 smithi078 ceph-mon[32429]: pgmap v5199: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:36.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:36 smithi107 ceph-mon[36485]: pgmap v5199: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:36.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:36 smithi106 ceph-mon[36505]: pgmap v5199: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:36.497 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9605 sec 2024-03-24T03:04:37.497 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9606 sec 2024-03-24T03:04:38.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:38 smithi078 ceph-mon[32429]: pgmap v5200: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:38.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:38 smithi107 ceph-mon[36485]: pgmap v5200: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:38.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:38 smithi106 ceph-mon[36505]: pgmap v5200: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:38.497 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9607 sec 2024-03-24T03:04:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8953.231984 secs 2024-03-24T03:04:39.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8953.231984 secs 2024-03-24T03:04:39.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8953.231984 secs 2024-03-24T03:04:39.497 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9608 sec 2024-03-24T03:04:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:40 smithi078 ceph-mon[32429]: pgmap v5201: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:40.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:40 smithi107 ceph-mon[36485]: pgmap v5201: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:40.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:40 smithi106 ceph-mon[36505]: pgmap v5201: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:40.497 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9609 sec 2024-03-24T03:04:41.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:41 smithi078 ceph-mon[32429]: pgmap v5202: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:41.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:41 smithi107 ceph-mon[36485]: pgmap v5202: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:41.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:41 smithi106 ceph-mon[36505]: pgmap v5202: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:41.497 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9610 sec 2024-03-24T03:04:42.497 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9611 sec 2024-03-24T03:04:43.498 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9612 sec 2024-03-24T03:04:43.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:43 smithi107 ceph-mon[36485]: pgmap v5203: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:43.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:43 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:04:43.688 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:43 smithi078 ceph-mon[32429]: pgmap v5203: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:43.688 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:43 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:04:43.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:43 smithi106 ceph-mon[36505]: pgmap v5203: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:43.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:43 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:04:44.498 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9613 sec 2024-03-24T03:04:44.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8958.232028 secs 2024-03-24T03:04:45.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8958.232028 secs 2024-03-24T03:04:45.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8958.232028 secs 2024-03-24T03:04:45.498 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9614 sec 2024-03-24T03:04:45.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:45 smithi106 ceph-mon[36505]: pgmap v5204: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:45.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:04:45.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:04:45.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:04:45.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:04:45.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:04:45.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:04:45.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:45.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:45.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:45.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:45.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:45.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:45.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:45.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:46.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:45 smithi078 ceph-mon[32429]: pgmap v5204: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:46.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:04:46.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:04:46.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:04:46.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:04:46.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:04:46.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:04:46.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:46.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:46.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:46.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:46.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:46.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:46.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:46.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:46.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:45 smithi107 ceph-mon[36485]: pgmap v5204: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:46.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:04:46.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:04:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:04:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:04:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:04:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:04:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:46.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:46.498 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9615 sec 2024-03-24T03:04:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:46 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T03:04:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:46 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T03:04:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:46 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T03:04:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:46 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T03:04:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:46 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:46 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:46 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:46 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:46 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T03:04:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:46 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T03:04:46.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:46 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:04:46.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:46 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:04:46.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:46 smithi106 ceph-mon[36505]: pgmap v5205: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T03:04:46.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:46 smithi106 ceph-mon[36505]: pgmap v5206: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:04:46.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:46 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:04:47.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:46 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T03:04:47.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:46 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T03:04:47.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:46 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T03:04:47.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:46 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T03:04:47.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:46 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:47.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:46 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:47.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:46 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:47.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:46 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:47.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:46 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T03:04:47.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:46 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T03:04:47.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:46 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:04:47.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:46 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:04:47.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:46 smithi078 ceph-mon[32429]: pgmap v5205: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T03:04:47.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:46 smithi078 ceph-mon[32429]: pgmap v5206: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:04:47.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:46 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:04:47.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:46 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T03:04:47.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:46 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T03:04:47.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:46 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T03:04:47.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:46 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T03:04:47.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:46 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:47.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:46 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:47.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:46 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:47.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:46 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:04:47.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:46 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T03:04:47.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:46 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T03:04:47.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:46 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:04:47.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:46 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:04:47.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:46 smithi107 ceph-mon[36485]: pgmap v5205: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T03:04:47.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:46 smithi107 ceph-mon[36485]: pgmap v5206: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:04:47.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:46 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:04:47.498 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9616 sec 2024-03-24T03:04:48.498 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9617 sec 2024-03-24T03:04:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:48 smithi078 ceph-mon[32429]: pgmap v5207: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:04:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8963.232116 secs 2024-03-24T03:04:49.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:48 smithi107 ceph-mon[36485]: pgmap v5207: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:04:49.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8963.232116 secs 2024-03-24T03:04:49.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:48 smithi106 ceph-mon[36505]: pgmap v5207: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:04:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8963.232116 secs 2024-03-24T03:04:49.498 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9618 sec 2024-03-24T03:04:50.498 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9619 sec 2024-03-24T03:04:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:50 smithi078 ceph-mon[32429]: pgmap v5208: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:04:51.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:50 smithi107 ceph-mon[36485]: pgmap v5208: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:04:51.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:50 smithi106 ceph-mon[36505]: pgmap v5208: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:04:51.498 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9620 sec 2024-03-24T03:04:52.499 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9621 sec 2024-03-24T03:04:53.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:52 smithi078 ceph-mon[32429]: pgmap v5209: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:04:53.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:52 smithi107 ceph-mon[36485]: pgmap v5209: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:04:53.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:52 smithi106 ceph-mon[36505]: pgmap v5209: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:04:53.499 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9622 sec 2024-03-24T03:04:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8968.232200 secs 2024-03-24T03:04:54.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8968.232200 secs 2024-03-24T03:04:54.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8968.232200 secs 2024-03-24T03:04:54.499 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9623 sec 2024-03-24T03:04:55.084 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:54 smithi078 ceph-mon[32429]: pgmap v5210: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:04:55.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:54 smithi107 ceph-mon[36485]: pgmap v5210: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:04:55.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:54 smithi106 ceph-mon[36505]: pgmap v5210: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:04:55.499 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9624 sec 2024-03-24T03:04:56.499 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9625 sec 2024-03-24T03:04:57.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:56 smithi078 ceph-mon[32429]: pgmap v5211: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:04:57.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:56 smithi107 ceph-mon[36485]: pgmap v5211: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:04:57.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:56 smithi106 ceph-mon[36505]: pgmap v5211: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:04:57.499 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9626 sec 2024-03-24T03:04:58.499 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9627 sec 2024-03-24T03:04:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:58 smithi078 ceph-mon[32429]: pgmap v5212: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:04:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8973.232296 secs 2024-03-24T03:04:59.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:58 smithi107 ceph-mon[36485]: pgmap v5212: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:59.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:04:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8973.232296 secs 2024-03-24T03:04:59.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:58 smithi106 ceph-mon[36505]: pgmap v5212: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:04:59.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:04:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8973.232296 secs 2024-03-24T03:04:59.499 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9628 sec 2024-03-24T03:05:00.499 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9629 sec 2024-03-24T03:05:01.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:00 smithi078 ceph-mon[32429]: pgmap v5213: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:01.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:00 smithi107 ceph-mon[36485]: pgmap v5213: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:01.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:00 smithi106 ceph-mon[36505]: pgmap v5213: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:01.499 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9630 sec 2024-03-24T03:05:02.500 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9631 sec 2024-03-24T03:05:03.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:02 smithi078 ceph-mon[32429]: pgmap v5214: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:03.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:02 smithi107 ceph-mon[36485]: pgmap v5214: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:03.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:02 smithi106 ceph-mon[36505]: pgmap v5214: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:03.499 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9632 sec 2024-03-24T03:05:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8978.232355 secs 2024-03-24T03:05:04.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8978.232355 secs 2024-03-24T03:05:04.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8978.232355 secs 2024-03-24T03:05:04.499 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9633 sec 2024-03-24T03:05:05.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:04 smithi078 ceph-mon[32429]: pgmap v5215: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:05.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:04 smithi107 ceph-mon[36485]: pgmap v5215: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:05.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:04 smithi106 ceph-mon[36505]: pgmap v5215: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:05.499 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9634 sec 2024-03-24T03:05:06.500 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9635 sec 2024-03-24T03:05:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:06 smithi078 ceph-mon[32429]: pgmap v5216: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:07.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:06 smithi107 ceph-mon[36485]: pgmap v5216: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:07.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:06 smithi106 ceph-mon[36505]: pgmap v5216: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:07.500 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9636 sec 2024-03-24T03:05:08.500 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9637 sec 2024-03-24T03:05:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:08 smithi078 ceph-mon[32429]: pgmap v5217: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8983.232439 secs 2024-03-24T03:05:09.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:08 smithi107 ceph-mon[36485]: pgmap v5217: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:09.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8983.232439 secs 2024-03-24T03:05:09.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:08 smithi106 ceph-mon[36505]: pgmap v5217: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8983.232439 secs 2024-03-24T03:05:09.500 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9638 sec 2024-03-24T03:05:10.500 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9639 sec 2024-03-24T03:05:11.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:10 smithi078 ceph-mon[32429]: pgmap v5218: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:11.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:10 smithi107 ceph-mon[36485]: pgmap v5218: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:11.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:10 smithi106 ceph-mon[36505]: pgmap v5218: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:11.500 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9640 sec 2024-03-24T03:05:12.500 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9641 sec 2024-03-24T03:05:13.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:12 smithi107 ceph-mon[36485]: pgmap v5219: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:13.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:12 smithi106 ceph-mon[36505]: pgmap v5219: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:13.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:12 smithi078 ceph-mon[32429]: pgmap v5219: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:13.500 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9642 sec 2024-03-24T03:05:14.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8988.232518 secs 2024-03-24T03:05:14.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8988.232518 secs 2024-03-24T03:05:14.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8988.232518 secs 2024-03-24T03:05:14.500 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9643 sec 2024-03-24T03:05:15.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:14 smithi106 ceph-mon[36505]: pgmap v5220: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:15.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:14 smithi078 ceph-mon[32429]: pgmap v5220: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:15.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:14 smithi107 ceph-mon[36485]: pgmap v5220: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:15.500 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9644 sec 2024-03-24T03:05:16.501 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9645 sec 2024-03-24T03:05:17.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:16 smithi106 ceph-mon[36505]: pgmap v5221: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:17.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:16 smithi078 ceph-mon[32429]: pgmap v5221: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:17.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:16 smithi107 ceph-mon[36485]: pgmap v5221: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:17.501 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9646 sec 2024-03-24T03:05:18.501 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9647 sec 2024-03-24T03:05:19.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:18 smithi106 ceph-mon[36505]: pgmap v5222: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8993.232624 secs 2024-03-24T03:05:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:18 smithi078 ceph-mon[32429]: pgmap v5222: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8993.232624 secs 2024-03-24T03:05:19.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:18 smithi107 ceph-mon[36485]: pgmap v5222: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:19.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8993.232624 secs 2024-03-24T03:05:19.501 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9648 sec 2024-03-24T03:05:20.501 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9649 sec 2024-03-24T03:05:21.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:20 smithi106 ceph-mon[36505]: pgmap v5223: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:21.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:20 smithi078 ceph-mon[32429]: pgmap v5223: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:21.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:20 smithi107 ceph-mon[36485]: pgmap v5223: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:21.501 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9650 sec 2024-03-24T03:05:22.501 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9651 sec 2024-03-24T03:05:23.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:22 smithi106 ceph-mon[36505]: pgmap v5224: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:23.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:22 smithi078 ceph-mon[32429]: pgmap v5224: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:23.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:22 smithi107 ceph-mon[36485]: pgmap v5224: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:23.501 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9652 sec 2024-03-24T03:05:24.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 8998.232683 secs 2024-03-24T03:05:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 8998.232683 secs 2024-03-24T03:05:24.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 8998.232683 secs 2024-03-24T03:05:24.501 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9653 sec 2024-03-24T03:05:25.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:24 smithi106 ceph-mon[36505]: pgmap v5225: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:24 smithi078 ceph-mon[32429]: pgmap v5225: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:25.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:24 smithi107 ceph-mon[36485]: pgmap v5225: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:25.501 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9654 sec 2024-03-24T03:05:26.501 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9655 sec 2024-03-24T03:05:27.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:26 smithi106 ceph-mon[36505]: pgmap v5226: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:27.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:26 smithi078 ceph-mon[32429]: pgmap v5226: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:27.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:26 smithi107 ceph-mon[36485]: pgmap v5226: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:27.501 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9656 sec 2024-03-24T03:05:28.502 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9657 sec 2024-03-24T03:05:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:28 smithi106 ceph-mon[36505]: pgmap v5227: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9003.232754 secs 2024-03-24T03:05:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:28 smithi078 ceph-mon[32429]: pgmap v5227: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9003.232754 secs 2024-03-24T03:05:29.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:28 smithi107 ceph-mon[36485]: pgmap v5227: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:29.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9003.232754 secs 2024-03-24T03:05:29.502 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9658 sec 2024-03-24T03:05:30.502 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9659 sec 2024-03-24T03:05:31.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:30 smithi106 ceph-mon[36505]: pgmap v5228: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:31.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:30 smithi078 ceph-mon[32429]: pgmap v5228: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:31.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:30 smithi107 ceph-mon[36485]: pgmap v5228: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:31.502 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9660 sec 2024-03-24T03:05:32.502 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9661 sec 2024-03-24T03:05:32.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:32 smithi078 ceph-mon[32429]: pgmap v5229: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:32.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:32 smithi107 ceph-mon[36485]: pgmap v5229: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:32.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:32 smithi106 ceph-mon[36505]: pgmap v5229: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:33.502 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9662 sec 2024-03-24T03:05:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9008.232843 secs 2024-03-24T03:05:34.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9008.232843 secs 2024-03-24T03:05:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9008.232843 secs 2024-03-24T03:05:34.502 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9663 sec 2024-03-24T03:05:35.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:34 smithi078 ceph-mon[32429]: pgmap v5230: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:35.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:34 smithi107 ceph-mon[36485]: pgmap v5230: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:35.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:34 smithi106 ceph-mon[36505]: pgmap v5230: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:35.502 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9664 sec 2024-03-24T03:05:36.502 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9665 sec 2024-03-24T03:05:37.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:36 smithi107 ceph-mon[36485]: pgmap v5231: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:37.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:36 smithi106 ceph-mon[36505]: pgmap v5231: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:37.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:36 smithi078 ceph-mon[32429]: pgmap v5231: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:37.502 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9666 sec 2024-03-24T03:05:38.502 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9667 sec 2024-03-24T03:05:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:38 smithi106 ceph-mon[36505]: pgmap v5232: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9013.232868 secs 2024-03-24T03:05:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:38 smithi078 ceph-mon[32429]: pgmap v5232: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9013.232868 secs 2024-03-24T03:05:39.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:38 smithi107 ceph-mon[36485]: pgmap v5232: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:39.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9013.232868 secs 2024-03-24T03:05:39.502 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9668 sec 2024-03-24T03:05:40.502 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9669 sec 2024-03-24T03:05:41.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:40 smithi106 ceph-mon[36505]: pgmap v5233: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:41.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:40 smithi078 ceph-mon[32429]: pgmap v5233: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:41.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:40 smithi107 ceph-mon[36485]: pgmap v5233: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:41.502 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9670 sec 2024-03-24T03:05:42.502 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9671 sec 2024-03-24T03:05:43.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:42 smithi106 ceph-mon[36505]: pgmap v5234: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:42 smithi078 ceph-mon[32429]: pgmap v5234: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:43.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:42 smithi107 ceph-mon[36485]: pgmap v5234: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:43.502 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9672 sec 2024-03-24T03:05:44.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9018.232929 secs 2024-03-24T03:05:44.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9018.232929 secs 2024-03-24T03:05:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9018.232929 secs 2024-03-24T03:05:44.503 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9673 sec 2024-03-24T03:05:45.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:44 smithi106 ceph-mon[36505]: pgmap v5235: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:45.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:44 smithi078 ceph-mon[32429]: pgmap v5235: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:45.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:44 smithi107 ceph-mon[36485]: pgmap v5235: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:05:45.503 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9674 sec 2024-03-24T03:05:46.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:45 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:05:46.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:45 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:05:46.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:45 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:05:46.503 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9675 sec 2024-03-24T03:05:47.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:46 smithi106 ceph-mon[36505]: pgmap v5236: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:47.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:46 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:05:47.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:46 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:05:47.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:46 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:05:47.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:46 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:05:47.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:46 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:05:47.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:46 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:05:47.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:46 smithi078 ceph-mon[32429]: pgmap v5236: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:47.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:46 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:05:47.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:46 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:05:47.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:46 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:05:47.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:46 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:05:47.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:46 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:05:47.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:46 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:05:47.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:46 smithi107 ceph-mon[36485]: pgmap v5236: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:47.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:46 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:05:47.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:46 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:05:47.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:46 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:05:47.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:46 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:05:47.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:46 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:05:47.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:46 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:05:47.503 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9676 sec 2024-03-24T03:05:48.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:47 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:05:48.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:47 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:05:48.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:47 smithi106 ceph-mon[36505]: pgmap v5237: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T03:05:48.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:47 smithi106 ceph-mon[36505]: pgmap v5238: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:05:48.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:47 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:05:48.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:47 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:05:48.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:47 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:05:48.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:47 smithi078 ceph-mon[32429]: pgmap v5237: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T03:05:48.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:47 smithi078 ceph-mon[32429]: pgmap v5238: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:05:48.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:47 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:05:48.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:47 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:05:48.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:47 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:05:48.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:47 smithi107 ceph-mon[36485]: pgmap v5237: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T03:05:48.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:47 smithi107 ceph-mon[36485]: pgmap v5238: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:05:48.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:47 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:05:48.503 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9677 sec 2024-03-24T03:05:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9023.232968 secs 2024-03-24T03:05:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9023.232968 secs 2024-03-24T03:05:49.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9023.232968 secs 2024-03-24T03:05:49.503 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9678 sec 2024-03-24T03:05:50.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:49 smithi106 ceph-mon[36505]: pgmap v5239: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:05:50.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:49 smithi078 ceph-mon[32429]: pgmap v5239: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:05:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:49 smithi107 ceph-mon[36485]: pgmap v5239: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:05:50.503 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9679 sec 2024-03-24T03:05:51.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:51 smithi106 ceph-mon[36505]: pgmap v5240: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:05:51.503 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9680 sec 2024-03-24T03:05:51.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:51 smithi078 ceph-mon[32429]: pgmap v5240: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:05:51.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:51 smithi107 ceph-mon[36485]: pgmap v5240: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:05:52.503 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9681 sec 2024-03-24T03:05:53.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:53 smithi106 ceph-mon[36505]: pgmap v5241: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:05:53.503 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9682 sec 2024-03-24T03:05:53.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:53 smithi078 ceph-mon[32429]: pgmap v5241: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:05:53.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:53 smithi107 ceph-mon[36485]: pgmap v5241: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:05:54.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9028.233026 secs 2024-03-24T03:05:54.503 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9683 sec 2024-03-24T03:05:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9028.233026 secs 2024-03-24T03:05:54.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9028.233026 secs 2024-03-24T03:05:55.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:55 smithi106 ceph-mon[36505]: pgmap v5242: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:05:55.503 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9684 sec 2024-03-24T03:05:55.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:55 smithi078 ceph-mon[32429]: pgmap v5242: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:05:55.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:55 smithi107 ceph-mon[36485]: pgmap v5242: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:05:56.503 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9685 sec 2024-03-24T03:05:57.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:57 smithi106 ceph-mon[36505]: pgmap v5243: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:05:57.503 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9686 sec 2024-03-24T03:05:57.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:57 smithi078 ceph-mon[32429]: pgmap v5243: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:05:57.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:57 smithi107 ceph-mon[36485]: pgmap v5243: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:05:58.504 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9687 sec 2024-03-24T03:05:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9033.233080 secs 2024-03-24T03:05:59.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9033.233080 secs 2024-03-24T03:05:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9033.233080 secs 2024-03-24T03:05:59.504 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9688 sec 2024-03-24T03:06:00.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:05:59 smithi106 ceph-mon[36505]: pgmap v5244: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:00.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:05:59 smithi078 ceph-mon[32429]: pgmap v5244: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:00.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:05:59 smithi107 ceph-mon[36485]: pgmap v5244: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:00.504 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9689 sec 2024-03-24T03:06:01.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:01 smithi106 ceph-mon[36505]: pgmap v5245: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:01.504 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9690 sec 2024-03-24T03:06:01.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:01 smithi078 ceph-mon[32429]: pgmap v5245: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:01.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:01 smithi107 ceph-mon[36485]: pgmap v5245: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:02.504 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9691 sec 2024-03-24T03:06:03.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:03 smithi106 ceph-mon[36505]: pgmap v5246: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:03.504 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9692 sec 2024-03-24T03:06:03.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:03 smithi078 ceph-mon[32429]: pgmap v5246: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:03.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:03 smithi107 ceph-mon[36485]: pgmap v5246: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:04.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9038.233141 secs 2024-03-24T03:06:04.494 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9038.233141 secs 2024-03-24T03:06:04.504 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9693 sec 2024-03-24T03:06:04.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9038.233141 secs 2024-03-24T03:06:05.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:05 smithi106 ceph-mon[36505]: pgmap v5247: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:05.504 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9694 sec 2024-03-24T03:06:05.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:05 smithi078 ceph-mon[32429]: pgmap v5247: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:05.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:05 smithi107 ceph-mon[36485]: pgmap v5247: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:06.504 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9695 sec 2024-03-24T03:06:07.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:07 smithi106 ceph-mon[36505]: pgmap v5248: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:07.505 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9696 sec 2024-03-24T03:06:07.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:07 smithi078 ceph-mon[32429]: pgmap v5248: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:07.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:07 smithi107 ceph-mon[36485]: pgmap v5248: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:08.505 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9697 sec 2024-03-24T03:06:09.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9043.233200 secs 2024-03-24T03:06:09.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9043.233200 secs 2024-03-24T03:06:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9043.233200 secs 2024-03-24T03:06:09.505 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9698 sec 2024-03-24T03:06:10.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:09 smithi106 ceph-mon[36505]: pgmap v5249: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:10.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:09 smithi078 ceph-mon[32429]: pgmap v5249: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:10.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:09 smithi107 ceph-mon[36485]: pgmap v5249: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:10.505 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9699 sec 2024-03-24T03:06:11.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:11 smithi106 ceph-mon[36505]: pgmap v5250: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:11.505 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9700 sec 2024-03-24T03:06:11.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:11 smithi078 ceph-mon[32429]: pgmap v5250: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:11.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:11 smithi107 ceph-mon[36485]: pgmap v5250: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:12.505 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9701 sec 2024-03-24T03:06:13.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:13 smithi106 ceph-mon[36505]: pgmap v5251: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:13.505 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9702 sec 2024-03-24T03:06:13.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:13 smithi078 ceph-mon[32429]: pgmap v5251: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:13.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:13 smithi107 ceph-mon[36485]: pgmap v5251: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:14.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9048.233294 secs 2024-03-24T03:06:14.491 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9048.233294 secs 2024-03-24T03:06:14.505 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9703 sec 2024-03-24T03:06:14.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9048.233294 secs 2024-03-24T03:06:15.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:15 smithi106 ceph-mon[36505]: pgmap v5252: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:15.505 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9704 sec 2024-03-24T03:06:15.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:15 smithi078 ceph-mon[32429]: pgmap v5252: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:15.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:15 smithi107 ceph-mon[36485]: pgmap v5252: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:16.505 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9705 sec 2024-03-24T03:06:17.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:17 smithi106 ceph-mon[36505]: pgmap v5253: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:17.505 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9706 sec 2024-03-24T03:06:17.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:17 smithi078 ceph-mon[32429]: pgmap v5253: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:17.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:17 smithi107 ceph-mon[36485]: pgmap v5253: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:18.506 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9707 sec 2024-03-24T03:06:19.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9053.233306 secs 2024-03-24T03:06:19.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9053.233306 secs 2024-03-24T03:06:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9053.233306 secs 2024-03-24T03:06:19.506 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9708 sec 2024-03-24T03:06:20.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:19 smithi106 ceph-mon[36505]: pgmap v5254: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:20.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:19 smithi078 ceph-mon[32429]: pgmap v5254: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:20.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:19 smithi107 ceph-mon[36485]: pgmap v5254: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:20.506 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9709 sec 2024-03-24T03:06:21.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:21 smithi106 ceph-mon[36505]: pgmap v5255: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:21.506 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9710 sec 2024-03-24T03:06:21.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:21 smithi078 ceph-mon[32429]: pgmap v5255: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:21.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:21 smithi107 ceph-mon[36485]: pgmap v5255: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:22.506 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9711 sec 2024-03-24T03:06:23.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:23 smithi106 ceph-mon[36505]: pgmap v5256: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:23.506 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9712 sec 2024-03-24T03:06:23.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:23 smithi078 ceph-mon[32429]: pgmap v5256: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:23.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:23 smithi107 ceph-mon[36485]: pgmap v5256: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:24.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9058.233389 secs 2024-03-24T03:06:24.506 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9713 sec 2024-03-24T03:06:24.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9058.233389 secs 2024-03-24T03:06:24.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9058.233389 secs 2024-03-24T03:06:25.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:25 smithi106 ceph-mon[36505]: pgmap v5257: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:25.506 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9714 sec 2024-03-24T03:06:25.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:25 smithi078 ceph-mon[32429]: pgmap v5257: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:25.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:25 smithi107 ceph-mon[36485]: pgmap v5257: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:26.506 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9715 sec 2024-03-24T03:06:27.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:27 smithi106 ceph-mon[36505]: pgmap v5258: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:27.506 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9716 sec 2024-03-24T03:06:27.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:27 smithi078 ceph-mon[32429]: pgmap v5258: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:27.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:27 smithi107 ceph-mon[36485]: pgmap v5258: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:28.506 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9717 sec 2024-03-24T03:06:29.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9063.233449 secs 2024-03-24T03:06:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9063.233449 secs 2024-03-24T03:06:29.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9063.233449 secs 2024-03-24T03:06:29.507 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9718 sec 2024-03-24T03:06:30.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:29 smithi106 ceph-mon[36505]: pgmap v5259: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:30.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:29 smithi078 ceph-mon[32429]: pgmap v5259: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:30.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:29 smithi107 ceph-mon[36485]: pgmap v5259: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:30.507 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9719 sec 2024-03-24T03:06:31.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:31 smithi106 ceph-mon[36505]: pgmap v5260: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:31.507 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9720 sec 2024-03-24T03:06:31.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:31 smithi078 ceph-mon[32429]: pgmap v5260: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:31.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:31 smithi107 ceph-mon[36485]: pgmap v5260: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:32.507 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9721 sec 2024-03-24T03:06:33.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:33 smithi106 ceph-mon[36505]: pgmap v5261: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:33.507 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9722 sec 2024-03-24T03:06:33.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:33 smithi078 ceph-mon[32429]: pgmap v5261: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:33.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:33 smithi107 ceph-mon[36485]: pgmap v5261: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:34.406 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9068.233463 secs 2024-03-24T03:06:34.495 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9068.233463 secs 2024-03-24T03:06:34.510 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9723 sec 2024-03-24T03:06:34.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9068.233463 secs 2024-03-24T03:06:35.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:35 smithi106 ceph-mon[36505]: pgmap v5262: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:35.507 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9724 sec 2024-03-24T03:06:35.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:35 smithi078 ceph-mon[32429]: pgmap v5262: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:35.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:35 smithi107 ceph-mon[36485]: pgmap v5262: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:36.507 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9725 sec 2024-03-24T03:06:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:37 smithi106 ceph-mon[36505]: pgmap v5263: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:37.507 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9726 sec 2024-03-24T03:06:37.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:37 smithi078 ceph-mon[32429]: pgmap v5263: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:37.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:37 smithi107 ceph-mon[36485]: pgmap v5263: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:38.507 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9727 sec 2024-03-24T03:06:39.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9073.233524 secs 2024-03-24T03:06:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9073.233524 secs 2024-03-24T03:06:39.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9073.233524 secs 2024-03-24T03:06:39.507 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9728 sec 2024-03-24T03:06:40.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:39 smithi106 ceph-mon[36505]: pgmap v5264: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:40.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:39 smithi078 ceph-mon[32429]: pgmap v5264: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:40.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:39 smithi107 ceph-mon[36485]: pgmap v5264: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:40.507 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9729 sec 2024-03-24T03:06:41.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:41 smithi106 ceph-mon[36505]: pgmap v5265: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:41.508 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9730 sec 2024-03-24T03:06:41.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:41 smithi078 ceph-mon[32429]: pgmap v5265: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:41.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:41 smithi107 ceph-mon[36485]: pgmap v5265: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:42.508 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9731 sec 2024-03-24T03:06:43.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:43 smithi106 ceph-mon[36505]: pgmap v5266: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:43.508 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9732 sec 2024-03-24T03:06:43.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:43 smithi078 ceph-mon[32429]: pgmap v5266: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:43.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:43 smithi107 ceph-mon[36485]: pgmap v5266: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9078.233583 secs 2024-03-24T03:06:44.508 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9733 sec 2024-03-24T03:06:44.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9078.233583 secs 2024-03-24T03:06:44.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9078.233583 secs 2024-03-24T03:06:45.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:45 smithi106 ceph-mon[36505]: pgmap v5267: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:45.508 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9734 sec 2024-03-24T03:06:45.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:45 smithi078 ceph-mon[32429]: pgmap v5267: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:45.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:45 smithi107 ceph-mon[36485]: pgmap v5267: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:06:46.508 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9735 sec 2024-03-24T03:06:47.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:47 smithi106 ceph-mon[36505]: pgmap v5268: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:47.508 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9736 sec 2024-03-24T03:06:47.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:47 smithi078 ceph-mon[32429]: pgmap v5268: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:47.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:47 smithi107 ceph-mon[36485]: pgmap v5268: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:48.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:48 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:06:48.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:48 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:06:48.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:48 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:06:48.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:48 smithi106 ceph-mon[36505]: pgmap v5269: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:06:48.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:48 smithi106 ceph-mon[36505]: pgmap v5270: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:06:48.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:48 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:06:48.508 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9737 sec 2024-03-24T03:06:48.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:48 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:06:48.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:48 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:06:48.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:48 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:06:48.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:48 smithi078 ceph-mon[32429]: pgmap v5269: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:06:48.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:48 smithi078 ceph-mon[32429]: pgmap v5270: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:06:48.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:48 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:06:48.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:48 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:06:48.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:48 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:06:48.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:48 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:06:48.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:48 smithi107 ceph-mon[36485]: pgmap v5269: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:06:48.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:48 smithi107 ceph-mon[36485]: pgmap v5270: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:06:48.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:48 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:06:49.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9083.233672 secs 2024-03-24T03:06:49.508 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9738 sec 2024-03-24T03:06:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9083.233672 secs 2024-03-24T03:06:49.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9083.233672 secs 2024-03-24T03:06:50.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:50 smithi106 ceph-mon[36505]: pgmap v5271: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:06:50.508 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9739 sec 2024-03-24T03:06:50.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:50 smithi078 ceph-mon[32429]: pgmap v5271: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:06:50.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:50 smithi107 ceph-mon[36485]: pgmap v5271: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:06:51.508 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9740 sec 2024-03-24T03:06:52.508 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9741 sec 2024-03-24T03:06:52.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:52 smithi078 ceph-mon[32429]: pgmap v5272: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:06:52.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:52 smithi107 ceph-mon[36485]: pgmap v5272: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:06:52.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:52 smithi106 ceph-mon[36505]: pgmap v5272: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:06:53.509 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9742 sec 2024-03-24T03:06:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9088.233759 secs 2024-03-24T03:06:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9088.233759 secs 2024-03-24T03:06:54.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9088.233759 secs 2024-03-24T03:06:54.509 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9743 sec 2024-03-24T03:06:55.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:54 smithi106 ceph-mon[36505]: pgmap v5273: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:06:55.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:54 smithi078 ceph-mon[32429]: pgmap v5273: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:06:55.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:54 smithi107 ceph-mon[36485]: pgmap v5273: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:06:55.509 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9744 sec 2024-03-24T03:06:56.509 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9745 sec 2024-03-24T03:06:57.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:56 smithi106 ceph-mon[36505]: pgmap v5274: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:06:57.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:56 smithi078 ceph-mon[32429]: pgmap v5274: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:06:57.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:56 smithi107 ceph-mon[36485]: pgmap v5274: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:06:57.509 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9746 sec 2024-03-24T03:06:58.509 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9747 sec 2024-03-24T03:06:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:58 smithi106 ceph-mon[36505]: pgmap v5275: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:06:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:06:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9093.233821 secs 2024-03-24T03:06:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:58 smithi078 ceph-mon[32429]: pgmap v5275: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:06:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:06:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9093.233821 secs 2024-03-24T03:06:59.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:58 smithi107 ceph-mon[36485]: pgmap v5275: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:06:59.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:06:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9093.233821 secs 2024-03-24T03:06:59.509 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9748 sec 2024-03-24T03:07:00.509 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9749 sec 2024-03-24T03:07:01.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:00 smithi106 ceph-mon[36505]: pgmap v5276: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:01.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:00 smithi078 ceph-mon[32429]: pgmap v5276: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:01.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:00 smithi107 ceph-mon[36485]: pgmap v5276: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:01.509 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9750 sec 2024-03-24T03:07:02.510 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9751 sec 2024-03-24T03:07:03.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:02 smithi106 ceph-mon[36505]: pgmap v5277: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:03.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:02 smithi078 ceph-mon[32429]: pgmap v5277: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:03.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:02 smithi107 ceph-mon[36485]: pgmap v5277: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:03.509 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9752 sec 2024-03-24T03:07:04.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9098.233883 secs 2024-03-24T03:07:04.243 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9098.233883 secs 2024-03-24T03:07:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9098.233883 secs 2024-03-24T03:07:04.510 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9753 sec 2024-03-24T03:07:05.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:04 smithi106 ceph-mon[36505]: pgmap v5278: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:05.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:04 smithi078 ceph-mon[32429]: pgmap v5278: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:05.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:04 smithi107 ceph-mon[36485]: pgmap v5278: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:05.510 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9754 sec 2024-03-24T03:07:06.510 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9755 sec 2024-03-24T03:07:07.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:06 smithi106 ceph-mon[36505]: pgmap v5279: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:07.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:06 smithi078 ceph-mon[32429]: pgmap v5279: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:07.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:06 smithi107 ceph-mon[36485]: pgmap v5279: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:07.510 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9756 sec 2024-03-24T03:07:08.510 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9757 sec 2024-03-24T03:07:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:08 smithi106 ceph-mon[36505]: pgmap v5280: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9103.233940 secs 2024-03-24T03:07:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:08 smithi078 ceph-mon[32429]: pgmap v5280: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9103.233940 secs 2024-03-24T03:07:09.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:08 smithi107 ceph-mon[36485]: pgmap v5280: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:09.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9103.233940 secs 2024-03-24T03:07:09.510 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9758 sec 2024-03-24T03:07:10.510 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9759 sec 2024-03-24T03:07:11.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:10 smithi106 ceph-mon[36505]: pgmap v5281: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:11.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:10 smithi078 ceph-mon[32429]: pgmap v5281: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:11.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:10 smithi107 ceph-mon[36485]: pgmap v5281: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:11.510 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9760 sec 2024-03-24T03:07:12.510 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9761 sec 2024-03-24T03:07:13.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:12 smithi106 ceph-mon[36505]: pgmap v5282: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:13.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:12 smithi078 ceph-mon[32429]: pgmap v5282: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:13.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:12 smithi107 ceph-mon[36485]: pgmap v5282: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:13.511 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9762 sec 2024-03-24T03:07:14.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9108.234026 secs 2024-03-24T03:07:14.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9108.234026 secs 2024-03-24T03:07:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9108.234026 secs 2024-03-24T03:07:14.510 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9763 sec 2024-03-24T03:07:15.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:14 smithi106 ceph-mon[36505]: pgmap v5283: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:15.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:14 smithi078 ceph-mon[32429]: pgmap v5283: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:15.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:14 smithi107 ceph-mon[36485]: pgmap v5283: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:15.510 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9764 sec 2024-03-24T03:07:16.510 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9765 sec 2024-03-24T03:07:17.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:16 smithi078 ceph-mon[32429]: pgmap v5284: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:17.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:16 smithi107 ceph-mon[36485]: pgmap v5284: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:17.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:16 smithi106 ceph-mon[36505]: pgmap v5284: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:17.510 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9766 sec 2024-03-24T03:07:18.510 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9767 sec 2024-03-24T03:07:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:18 smithi078 ceph-mon[32429]: pgmap v5285: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9113.234116 secs 2024-03-24T03:07:19.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:18 smithi107 ceph-mon[36485]: pgmap v5285: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:19.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9113.234116 secs 2024-03-24T03:07:19.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:18 smithi106 ceph-mon[36505]: pgmap v5285: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9113.234116 secs 2024-03-24T03:07:19.511 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9768 sec 2024-03-24T03:07:20.511 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9769 sec 2024-03-24T03:07:21.202 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:20 smithi106 ceph-mon[36505]: pgmap v5286: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:21.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:20 smithi078 ceph-mon[32429]: pgmap v5286: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:21.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:20 smithi107 ceph-mon[36485]: pgmap v5286: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:21.511 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9770 sec 2024-03-24T03:07:22.511 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9771 sec 2024-03-24T03:07:23.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:22 smithi078 ceph-mon[32429]: pgmap v5287: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:23.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:22 smithi107 ceph-mon[36485]: pgmap v5287: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:23.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:22 smithi106 ceph-mon[36505]: pgmap v5287: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:23.511 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9772 sec 2024-03-24T03:07:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9118.234204 secs 2024-03-24T03:07:24.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9118.234204 secs 2024-03-24T03:07:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9118.234204 secs 2024-03-24T03:07:24.511 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9773 sec 2024-03-24T03:07:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:24 smithi078 ceph-mon[32429]: pgmap v5288: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:25.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:24 smithi107 ceph-mon[36485]: pgmap v5288: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:25.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:24 smithi106 ceph-mon[36505]: pgmap v5288: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:25.511 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9774 sec 2024-03-24T03:07:26.511 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9775 sec 2024-03-24T03:07:27.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:26 smithi078 ceph-mon[32429]: pgmap v5289: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:27.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:26 smithi107 ceph-mon[36485]: pgmap v5289: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:27.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:26 smithi106 ceph-mon[36505]: pgmap v5289: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:27.511 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9776 sec 2024-03-24T03:07:28.511 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9777 sec 2024-03-24T03:07:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:28 smithi078 ceph-mon[32429]: pgmap v5290: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9123.234242 secs 2024-03-24T03:07:29.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:28 smithi107 ceph-mon[36485]: pgmap v5290: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9123.234242 secs 2024-03-24T03:07:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:28 smithi106 ceph-mon[36505]: pgmap v5290: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9123.234242 secs 2024-03-24T03:07:29.512 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9778 sec 2024-03-24T03:07:30.512 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9779 sec 2024-03-24T03:07:31.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:30 smithi078 ceph-mon[32429]: pgmap v5291: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:31.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:30 smithi107 ceph-mon[36485]: pgmap v5291: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:31.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:30 smithi106 ceph-mon[36505]: pgmap v5291: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:31.512 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9780 sec 2024-03-24T03:07:32.512 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9781 sec 2024-03-24T03:07:33.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:32 smithi078 ceph-mon[32429]: pgmap v5292: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:33.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:32 smithi107 ceph-mon[36485]: pgmap v5292: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:33.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:32 smithi106 ceph-mon[36505]: pgmap v5292: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:33.512 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9782 sec 2024-03-24T03:07:34.244 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9128.234338 secs 2024-03-24T03:07:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9128.234338 secs 2024-03-24T03:07:34.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9128.234338 secs 2024-03-24T03:07:34.512 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9783 sec 2024-03-24T03:07:35.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:34 smithi078 ceph-mon[32429]: pgmap v5293: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:35.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:34 smithi107 ceph-mon[36485]: pgmap v5293: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:35.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:34 smithi106 ceph-mon[36505]: pgmap v5293: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:35.512 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9784 sec 2024-03-24T03:07:36.512 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9785 sec 2024-03-24T03:07:37.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:36 smithi078 ceph-mon[32429]: pgmap v5294: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:37.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:36 smithi107 ceph-mon[36485]: pgmap v5294: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:36 smithi106 ceph-mon[36505]: pgmap v5294: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:37.512 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9786 sec 2024-03-24T03:07:38.512 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9787 sec 2024-03-24T03:07:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:38 smithi078 ceph-mon[32429]: pgmap v5295: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9133.234402 secs 2024-03-24T03:07:39.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:38 smithi107 ceph-mon[36485]: pgmap v5295: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:39.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9133.234402 secs 2024-03-24T03:07:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:38 smithi106 ceph-mon[36505]: pgmap v5295: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9133.234402 secs 2024-03-24T03:07:39.512 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9788 sec 2024-03-24T03:07:40.512 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9789 sec 2024-03-24T03:07:41.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:40 smithi078 ceph-mon[32429]: pgmap v5296: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:41.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:40 smithi107 ceph-mon[36485]: pgmap v5296: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:41.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:40 smithi106 ceph-mon[36505]: pgmap v5296: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:41.513 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9790 sec 2024-03-24T03:07:42.513 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9791 sec 2024-03-24T03:07:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:42 smithi078 ceph-mon[32429]: pgmap v5297: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:43.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:42 smithi107 ceph-mon[36485]: pgmap v5297: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:43.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:42 smithi106 ceph-mon[36505]: pgmap v5297: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:43.513 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9792 sec 2024-03-24T03:07:44.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9138.234463 secs 2024-03-24T03:07:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9138.234463 secs 2024-03-24T03:07:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9138.234463 secs 2024-03-24T03:07:44.513 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9793 sec 2024-03-24T03:07:45.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:44 smithi078 ceph-mon[32429]: pgmap v5298: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:45.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:44 smithi107 ceph-mon[36485]: pgmap v5298: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:45.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:44 smithi106 ceph-mon[36505]: pgmap v5298: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:45.513 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9794 sec 2024-03-24T03:07:46.513 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9795 sec 2024-03-24T03:07:47.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:46 smithi078 ceph-mon[32429]: pgmap v5299: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:47.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:46 smithi107 ceph-mon[36485]: pgmap v5299: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:47.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:46 smithi106 ceph-mon[36505]: pgmap v5299: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:47.513 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9796 sec 2024-03-24T03:07:48.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:47 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:07:48.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:47 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:07:48.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:47 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:07:48.513 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9797 sec 2024-03-24T03:07:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:48 smithi078 ceph-mon[32429]: pgmap v5300: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9143.234529 secs 2024-03-24T03:07:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:48 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:07:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:48 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:07:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:48 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:07:49.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:48 smithi107 ceph-mon[36485]: pgmap v5300: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:49.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9143.234529 secs 2024-03-24T03:07:49.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:48 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:07:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:48 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:07:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:48 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:07:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:48 smithi106 ceph-mon[36505]: pgmap v5300: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:07:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9143.234529 secs 2024-03-24T03:07:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:48 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:07:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:48 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:07:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:48 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:07:49.513 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9798 sec 2024-03-24T03:07:50.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:49 smithi078 ceph-mon[32429]: pgmap v5301: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:07:50.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:49 smithi078 ceph-mon[32429]: pgmap v5302: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:07:50.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:49 smithi107 ceph-mon[36485]: pgmap v5301: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:07:50.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:49 smithi107 ceph-mon[36485]: pgmap v5302: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:07:50.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:49 smithi106 ceph-mon[36505]: pgmap v5301: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:07:50.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:49 smithi106 ceph-mon[36505]: pgmap v5302: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:07:50.513 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9799 sec 2024-03-24T03:07:51.513 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9800 sec 2024-03-24T03:07:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:51 smithi078 ceph-mon[32429]: pgmap v5303: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:07:52.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:51 smithi107 ceph-mon[36485]: pgmap v5303: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:07:52.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:51 smithi106 ceph-mon[36505]: pgmap v5303: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:07:52.513 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9801 sec 2024-03-24T03:07:53.514 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9802 sec 2024-03-24T03:07:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:53 smithi078 ceph-mon[32429]: pgmap v5304: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:07:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9148.234616 secs 2024-03-24T03:07:54.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:53 smithi107 ceph-mon[36485]: pgmap v5304: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:07:54.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9148.234616 secs 2024-03-24T03:07:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:53 smithi106 ceph-mon[36505]: pgmap v5304: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:07:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9148.234616 secs 2024-03-24T03:07:54.514 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9803 sec 2024-03-24T03:07:55.514 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9804 sec 2024-03-24T03:07:56.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:55 smithi078 ceph-mon[32429]: pgmap v5305: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:07:56.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:55 smithi107 ceph-mon[36485]: pgmap v5305: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:07:56.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:55 smithi106 ceph-mon[36505]: pgmap v5305: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:07:56.514 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9805 sec 2024-03-24T03:07:57.514 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9806 sec 2024-03-24T03:07:58.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:57 smithi078 ceph-mon[32429]: pgmap v5306: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:07:58.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:57 smithi107 ceph-mon[36485]: pgmap v5306: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:07:58.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:57 smithi106 ceph-mon[36505]: pgmap v5306: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:07:58.514 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9807 sec 2024-03-24T03:07:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:07:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9153.234705 secs 2024-03-24T03:07:59.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:07:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9153.234705 secs 2024-03-24T03:07:59.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:07:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9153.234705 secs 2024-03-24T03:07:59.514 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9808 sec 2024-03-24T03:08:00.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:00 smithi078 ceph-mon[32429]: pgmap v5307: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:08:00.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:00 smithi107 ceph-mon[36485]: pgmap v5307: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:08:00.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:00 smithi106 ceph-mon[36505]: pgmap v5307: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:08:00.514 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9809 sec 2024-03-24T03:08:01.514 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9810 sec 2024-03-24T03:08:02.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:02 smithi078 ceph-mon[32429]: pgmap v5308: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:02.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:02 smithi107 ceph-mon[36485]: pgmap v5308: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:02.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:02 smithi106 ceph-mon[36505]: pgmap v5308: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:02.514 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9811 sec 2024-03-24T03:08:03.514 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9812 sec 2024-03-24T03:08:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:04 smithi078 ceph-mon[32429]: pgmap v5309: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9158.234771 secs 2024-03-24T03:08:04.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:04 smithi107 ceph-mon[36485]: pgmap v5309: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:04.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9158.234771 secs 2024-03-24T03:08:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:04 smithi106 ceph-mon[36505]: pgmap v5309: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9158.234771 secs 2024-03-24T03:08:04.514 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9813 sec 2024-03-24T03:08:05.515 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9814 sec 2024-03-24T03:08:06.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:06 smithi078 ceph-mon[32429]: pgmap v5310: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:06.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:06 smithi107 ceph-mon[36485]: pgmap v5310: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:06.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:06 smithi106 ceph-mon[36505]: pgmap v5310: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:06.514 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9815 sec 2024-03-24T03:08:07.515 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9816 sec 2024-03-24T03:08:08.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:08 smithi078 ceph-mon[32429]: pgmap v5311: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:08.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:08 smithi107 ceph-mon[36485]: pgmap v5311: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:08.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:08 smithi106 ceph-mon[36505]: pgmap v5311: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:08.515 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9817 sec 2024-03-24T03:08:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9163.234858 secs 2024-03-24T03:08:09.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9163.234858 secs 2024-03-24T03:08:09.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9163.234858 secs 2024-03-24T03:08:09.515 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9818 sec 2024-03-24T03:08:10.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:10 smithi078 ceph-mon[32429]: pgmap v5312: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:10.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:10 smithi107 ceph-mon[36485]: pgmap v5312: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:10.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:10 smithi106 ceph-mon[36505]: pgmap v5312: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:10.515 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9819 sec 2024-03-24T03:08:11.515 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9820 sec 2024-03-24T03:08:12.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:12 smithi078 ceph-mon[32429]: pgmap v5313: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:12.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:12 smithi107 ceph-mon[36485]: pgmap v5313: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:12.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:12 smithi106 ceph-mon[36505]: pgmap v5313: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:12.515 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9821 sec 2024-03-24T03:08:13.515 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9822 sec 2024-03-24T03:08:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:14 smithi078 ceph-mon[32429]: pgmap v5314: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9168.234891 secs 2024-03-24T03:08:14.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:14 smithi107 ceph-mon[36485]: pgmap v5314: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9168.234891 secs 2024-03-24T03:08:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:14 smithi106 ceph-mon[36505]: pgmap v5314: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9168.234891 secs 2024-03-24T03:08:14.515 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9823 sec 2024-03-24T03:08:15.515 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9824 sec 2024-03-24T03:08:16.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:16 smithi078 ceph-mon[32429]: pgmap v5315: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:16.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:16 smithi107 ceph-mon[36485]: pgmap v5315: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:16.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:16 smithi106 ceph-mon[36505]: pgmap v5315: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:16.515 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9825 sec 2024-03-24T03:08:17.516 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9826 sec 2024-03-24T03:08:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:18 smithi078 ceph-mon[32429]: pgmap v5316: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:18.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:18 smithi107 ceph-mon[36485]: pgmap v5316: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:18.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:18 smithi106 ceph-mon[36505]: pgmap v5316: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:18.515 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9827 sec 2024-03-24T03:08:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9173.234930 secs 2024-03-24T03:08:19.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9173.234930 secs 2024-03-24T03:08:19.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9173.234930 secs 2024-03-24T03:08:19.516 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9828 sec 2024-03-24T03:08:20.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:20 smithi078 ceph-mon[32429]: pgmap v5317: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:20.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:20 smithi107 ceph-mon[36485]: pgmap v5317: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:20.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:20 smithi106 ceph-mon[36505]: pgmap v5317: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:20.516 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9829 sec 2024-03-24T03:08:21.516 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9830 sec 2024-03-24T03:08:22.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:22 smithi078 ceph-mon[32429]: pgmap v5318: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:22.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:22 smithi107 ceph-mon[36485]: pgmap v5318: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:22.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:22 smithi106 ceph-mon[36505]: pgmap v5318: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:22.516 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9831 sec 2024-03-24T03:08:23.516 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9832 sec 2024-03-24T03:08:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:24 smithi078 ceph-mon[32429]: pgmap v5319: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9178.234972 secs 2024-03-24T03:08:24.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:24 smithi107 ceph-mon[36485]: pgmap v5319: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:24.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9178.234972 secs 2024-03-24T03:08:24.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:24 smithi106 ceph-mon[36505]: pgmap v5319: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9178.234972 secs 2024-03-24T03:08:24.516 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9833 sec 2024-03-24T03:08:25.516 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9834 sec 2024-03-24T03:08:26.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:26 smithi078 ceph-mon[32429]: pgmap v5320: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:26.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:26 smithi107 ceph-mon[36485]: pgmap v5320: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:26.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:26 smithi106 ceph-mon[36505]: pgmap v5320: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:26.516 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9835 sec 2024-03-24T03:08:27.516 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9836 sec 2024-03-24T03:08:28.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:28 smithi078 ceph-mon[32429]: pgmap v5321: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:28.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:28 smithi107 ceph-mon[36485]: pgmap v5321: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:28.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:28 smithi106 ceph-mon[36505]: pgmap v5321: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:28.516 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9837 sec 2024-03-24T03:08:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9183.235061 secs 2024-03-24T03:08:29.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9183.235061 secs 2024-03-24T03:08:29.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9183.235061 secs 2024-03-24T03:08:29.517 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9838 sec 2024-03-24T03:08:30.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:30 smithi078 ceph-mon[32429]: pgmap v5322: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:30.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:30 smithi107 ceph-mon[36485]: pgmap v5322: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:30.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:30 smithi106 ceph-mon[36505]: pgmap v5322: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:30.516 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9839 sec 2024-03-24T03:08:31.517 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9840 sec 2024-03-24T03:08:32.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:32 smithi078 ceph-mon[32429]: pgmap v5323: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:32.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:32 smithi107 ceph-mon[36485]: pgmap v5323: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:32.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:32 smithi106 ceph-mon[36505]: pgmap v5323: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:32.517 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9841 sec 2024-03-24T03:08:33.517 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9842 sec 2024-03-24T03:08:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:34 smithi078 ceph-mon[32429]: pgmap v5324: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9188.235147 secs 2024-03-24T03:08:34.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:34 smithi107 ceph-mon[36485]: pgmap v5324: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:34.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9188.235147 secs 2024-03-24T03:08:34.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:34 smithi106 ceph-mon[36505]: pgmap v5324: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:34.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9188.235147 secs 2024-03-24T03:08:34.517 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9843 sec 2024-03-24T03:08:35.517 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9844 sec 2024-03-24T03:08:36.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:36 smithi078 ceph-mon[32429]: pgmap v5325: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:36.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:36 smithi107 ceph-mon[36485]: pgmap v5325: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:36.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:36 smithi106 ceph-mon[36505]: pgmap v5325: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:36.517 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9845 sec 2024-03-24T03:08:37.517 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9846 sec 2024-03-24T03:08:38.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:38 smithi078 ceph-mon[32429]: pgmap v5326: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:38.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:38 smithi107 ceph-mon[36485]: pgmap v5326: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:38.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:38 smithi106 ceph-mon[36505]: pgmap v5326: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:38.517 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9847 sec 2024-03-24T03:08:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9193.235232 secs 2024-03-24T03:08:39.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9193.235232 secs 2024-03-24T03:08:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9193.235232 secs 2024-03-24T03:08:39.517 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9848 sec 2024-03-24T03:08:40.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:40 smithi078 ceph-mon[32429]: pgmap v5327: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:40.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:40 smithi107 ceph-mon[36485]: pgmap v5327: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:40.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:40 smithi106 ceph-mon[36505]: pgmap v5327: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:40.518 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9849 sec 2024-03-24T03:08:41.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:41 smithi078 ceph-mon[32429]: pgmap v5328: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:41.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:41 smithi107 ceph-mon[36485]: pgmap v5328: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:41.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:41 smithi106 ceph-mon[36505]: pgmap v5328: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:41.518 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9850 sec 2024-03-24T03:08:42.518 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9851 sec 2024-03-24T03:08:43.518 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9852 sec 2024-03-24T03:08:44.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:43 smithi078 ceph-mon[32429]: pgmap v5329: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:44.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:43 smithi107 ceph-mon[36485]: pgmap v5329: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:44.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:43 smithi106 ceph-mon[36505]: pgmap v5329: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:44.518 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9853 sec 2024-03-24T03:08:45.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9198.235322 secs 2024-03-24T03:08:45.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9198.235322 secs 2024-03-24T03:08:45.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9198.235322 secs 2024-03-24T03:08:45.518 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9854 sec 2024-03-24T03:08:46.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:45 smithi078 ceph-mon[32429]: pgmap v5330: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:46.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:45 smithi107 ceph-mon[36485]: pgmap v5330: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:46.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:45 smithi106 ceph-mon[36505]: pgmap v5330: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:46.518 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9855 sec 2024-03-24T03:08:47.518 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9856 sec 2024-03-24T03:08:48.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:47 smithi078 ceph-mon[32429]: pgmap v5331: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:48.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:47 smithi107 ceph-mon[36485]: pgmap v5331: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:48.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:47 smithi106 ceph-mon[36505]: pgmap v5331: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:48.518 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9857 sec 2024-03-24T03:08:49.518 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9858 sec 2024-03-24T03:08:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9203.235410 secs 2024-03-24T03:08:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:49 smithi078 ceph-mon[32429]: pgmap v5332: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:49 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:08:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:49 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:08:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:49 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:08:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:49 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:08:50.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9203.235410 secs 2024-03-24T03:08:50.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:49 smithi107 ceph-mon[36485]: pgmap v5332: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:50.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:49 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:08:50.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:49 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:08:50.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:49 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:08:50.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:49 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:08:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9203.235410 secs 2024-03-24T03:08:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:49 smithi106 ceph-mon[36505]: pgmap v5332: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:49 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:08:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:49 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:08:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:49 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:08:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:49 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:08:50.518 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9859 sec 2024-03-24T03:08:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:50 smithi078 ceph-mon[32429]: pgmap v5333: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:08:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:50 smithi078 ceph-mon[32429]: pgmap v5334: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:08:51.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:50 smithi107 ceph-mon[36485]: pgmap v5333: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:08:51.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:50 smithi107 ceph-mon[36485]: pgmap v5334: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:08:51.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:50 smithi106 ceph-mon[36505]: pgmap v5333: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:08:51.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:50 smithi106 ceph-mon[36505]: pgmap v5334: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:08:51.519 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9860 sec 2024-03-24T03:08:52.519 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9861 sec 2024-03-24T03:08:52.954 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:52 smithi106 ceph-mon[36505]: pgmap v5335: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:08:53.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:52 smithi078 ceph-mon[32429]: pgmap v5335: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:08:53.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:52 smithi107 ceph-mon[36485]: pgmap v5335: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:08:53.519 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9862 sec 2024-03-24T03:08:54.519 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9863 sec 2024-03-24T03:08:55.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9208.235497 secs 2024-03-24T03:08:55.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:54 smithi078 ceph-mon[32429]: pgmap v5336: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:08:55.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9208.235497 secs 2024-03-24T03:08:55.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:54 smithi107 ceph-mon[36485]: pgmap v5336: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:08:55.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9208.235497 secs 2024-03-24T03:08:55.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:54 smithi106 ceph-mon[36505]: pgmap v5336: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:08:55.519 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9864 sec 2024-03-24T03:08:56.520 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9865 sec 2024-03-24T03:08:57.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:56 smithi078 ceph-mon[32429]: pgmap v5337: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:57.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:56 smithi107 ceph-mon[36485]: pgmap v5337: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:57.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:56 smithi106 ceph-mon[36505]: pgmap v5337: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:08:57.519 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9866 sec 2024-03-24T03:08:58.519 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9867 sec 2024-03-24T03:08:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:58 smithi078 ceph-mon[32429]: pgmap v5338: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:08:59.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:58 smithi107 ceph-mon[36485]: pgmap v5338: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:08:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:58 smithi106 ceph-mon[36505]: pgmap v5338: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:08:59.519 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9868 sec 2024-03-24T03:09:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:08:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9213.235580 secs 2024-03-24T03:09:00.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:08:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9213.235580 secs 2024-03-24T03:09:00.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:08:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9213.235580 secs 2024-03-24T03:09:00.519 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9869 sec 2024-03-24T03:09:01.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:00 smithi078 ceph-mon[32429]: pgmap v5339: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:09:01.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:00 smithi107 ceph-mon[36485]: pgmap v5339: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:09:01.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:00 smithi106 ceph-mon[36505]: pgmap v5339: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:09:01.519 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9870 sec 2024-03-24T03:09:02.519 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9871 sec 2024-03-24T03:09:03.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:02 smithi078 ceph-mon[32429]: pgmap v5340: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:03.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:02 smithi107 ceph-mon[36485]: pgmap v5340: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:03.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:02 smithi106 ceph-mon[36505]: pgmap v5340: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:03.520 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9872 sec 2024-03-24T03:09:04.520 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9873 sec 2024-03-24T03:09:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9218.235595 secs 2024-03-24T03:09:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:04 smithi078 ceph-mon[32429]: pgmap v5341: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:05.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9218.235595 secs 2024-03-24T03:09:05.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:04 smithi107 ceph-mon[36485]: pgmap v5341: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:05.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9218.235595 secs 2024-03-24T03:09:05.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:04 smithi106 ceph-mon[36505]: pgmap v5341: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:05.520 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9874 sec 2024-03-24T03:09:06.520 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9875 sec 2024-03-24T03:09:07.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:06 smithi078 ceph-mon[32429]: pgmap v5342: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:09:07.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:06 smithi107 ceph-mon[36485]: pgmap v5342: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:09:07.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:06 smithi106 ceph-mon[36505]: pgmap v5342: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:09:07.520 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9876 sec 2024-03-24T03:09:08.520 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9877 sec 2024-03-24T03:09:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:08 smithi078 ceph-mon[32429]: pgmap v5343: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:09:09.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:08 smithi107 ceph-mon[36485]: pgmap v5343: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:09:09.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:08 smithi106 ceph-mon[36505]: pgmap v5343: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:09:09.520 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9878 sec 2024-03-24T03:09:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9223.235657 secs 2024-03-24T03:09:10.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9223.235657 secs 2024-03-24T03:09:10.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9223.235657 secs 2024-03-24T03:09:10.520 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9879 sec 2024-03-24T03:09:11.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:10 smithi078 ceph-mon[32429]: pgmap v5344: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:11.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:10 smithi107 ceph-mon[36485]: pgmap v5344: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:11.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:10 smithi106 ceph-mon[36505]: pgmap v5344: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:11.521 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9880 sec 2024-03-24T03:09:12.520 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9881 sec 2024-03-24T03:09:13.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:12 smithi078 ceph-mon[32429]: pgmap v5345: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:13.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:12 smithi107 ceph-mon[36485]: pgmap v5345: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:13.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:12 smithi106 ceph-mon[36505]: pgmap v5345: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:13.521 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9882 sec 2024-03-24T03:09:14.521 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9883 sec 2024-03-24T03:09:15.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9228.235716 secs 2024-03-24T03:09:15.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:14 smithi078 ceph-mon[32429]: pgmap v5346: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:15.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9228.235716 secs 2024-03-24T03:09:15.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:14 smithi107 ceph-mon[36485]: pgmap v5346: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:15.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9228.235716 secs 2024-03-24T03:09:15.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:14 smithi106 ceph-mon[36505]: pgmap v5346: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:15.521 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9884 sec 2024-03-24T03:09:16.521 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9885 sec 2024-03-24T03:09:17.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:16 smithi078 ceph-mon[32429]: pgmap v5347: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:17.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:16 smithi107 ceph-mon[36485]: pgmap v5347: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:17.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:16 smithi106 ceph-mon[36505]: pgmap v5347: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:17.521 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9886 sec 2024-03-24T03:09:18.521 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9887 sec 2024-03-24T03:09:19.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:18 smithi078 ceph-mon[32429]: pgmap v5348: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:09:19.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:18 smithi107 ceph-mon[36485]: pgmap v5348: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:09:19.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:18 smithi106 ceph-mon[36505]: pgmap v5348: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:09:19.521 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9888 sec 2024-03-24T03:09:20.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9233.235780 secs 2024-03-24T03:09:20.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9233.235780 secs 2024-03-24T03:09:20.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9233.235780 secs 2024-03-24T03:09:20.521 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9889 sec 2024-03-24T03:09:21.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:20 smithi078 ceph-mon[32429]: pgmap v5349: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:21.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:20 smithi107 ceph-mon[36485]: pgmap v5349: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:21.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:20 smithi106 ceph-mon[36505]: pgmap v5349: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:21.521 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9890 sec 2024-03-24T03:09:22.521 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9891 sec 2024-03-24T03:09:23.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:22 smithi078 ceph-mon[32429]: pgmap v5350: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:23.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:22 smithi107 ceph-mon[36485]: pgmap v5350: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:23.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:22 smithi106 ceph-mon[36505]: pgmap v5350: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:23.521 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9892 sec 2024-03-24T03:09:24.522 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9893 sec 2024-03-24T03:09:25.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9238.235866 secs 2024-03-24T03:09:25.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:24 smithi078 ceph-mon[32429]: pgmap v5351: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9238.235866 secs 2024-03-24T03:09:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:24 smithi107 ceph-mon[36485]: pgmap v5351: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:25.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9238.235866 secs 2024-03-24T03:09:25.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:24 smithi106 ceph-mon[36505]: pgmap v5351: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:25.522 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9894 sec 2024-03-24T03:09:26.522 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9895 sec 2024-03-24T03:09:27.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:26 smithi078 ceph-mon[32429]: pgmap v5352: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:27.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:26 smithi107 ceph-mon[36485]: pgmap v5352: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:27.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:26 smithi106 ceph-mon[36505]: pgmap v5352: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:27.522 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9896 sec 2024-03-24T03:09:28.522 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9897 sec 2024-03-24T03:09:29.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:28 smithi078 ceph-mon[32429]: pgmap v5353: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:29.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:28 smithi107 ceph-mon[36485]: pgmap v5353: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:29.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:28 smithi106 ceph-mon[36505]: pgmap v5353: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:29.522 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9898 sec 2024-03-24T03:09:30.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9243.235954 secs 2024-03-24T03:09:30.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9243.235954 secs 2024-03-24T03:09:30.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9243.235954 secs 2024-03-24T03:09:30.522 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9899 sec 2024-03-24T03:09:31.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:30 smithi078 ceph-mon[32429]: pgmap v5354: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:31.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:30 smithi107 ceph-mon[36485]: pgmap v5354: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:31.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:30 smithi106 ceph-mon[36505]: pgmap v5354: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:31.522 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9900 sec 2024-03-24T03:09:32.522 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9901 sec 2024-03-24T03:09:33.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:32 smithi078 ceph-mon[32429]: pgmap v5355: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:33.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:32 smithi107 ceph-mon[36485]: pgmap v5355: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:33.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:32 smithi106 ceph-mon[36505]: pgmap v5355: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:33.522 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9902 sec 2024-03-24T03:09:34.523 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9903 sec 2024-03-24T03:09:35.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9248.235991 secs 2024-03-24T03:09:35.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:34 smithi078 ceph-mon[32429]: pgmap v5356: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:35.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9248.235991 secs 2024-03-24T03:09:35.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:34 smithi107 ceph-mon[36485]: pgmap v5356: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:35.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9248.235991 secs 2024-03-24T03:09:35.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:34 smithi106 ceph-mon[36505]: pgmap v5356: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:35.522 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9904 sec 2024-03-24T03:09:36.522 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9905 sec 2024-03-24T03:09:37.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:36 smithi078 ceph-mon[32429]: pgmap v5357: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:37.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:36 smithi107 ceph-mon[36485]: pgmap v5357: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:37.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:36 smithi106 ceph-mon[36505]: pgmap v5357: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:37.522 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9906 sec 2024-03-24T03:09:38.523 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9907 sec 2024-03-24T03:09:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:38 smithi078 ceph-mon[32429]: pgmap v5358: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:39.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:38 smithi107 ceph-mon[36485]: pgmap v5358: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:38 smithi106 ceph-mon[36505]: pgmap v5358: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:39.523 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9908 sec 2024-03-24T03:09:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9253.236052 secs 2024-03-24T03:09:40.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9253.236052 secs 2024-03-24T03:09:40.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9253.236052 secs 2024-03-24T03:09:40.523 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9909 sec 2024-03-24T03:09:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:40 smithi078 ceph-mon[32429]: pgmap v5359: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:41.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:40 smithi107 ceph-mon[36485]: pgmap v5359: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:41.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:40 smithi106 ceph-mon[36505]: pgmap v5359: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:41.523 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9910 sec 2024-03-24T03:09:42.523 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9911 sec 2024-03-24T03:09:43.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:42 smithi078 ceph-mon[32429]: pgmap v5360: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:43.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:42 smithi107 ceph-mon[36485]: pgmap v5360: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:43.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:42 smithi106 ceph-mon[36505]: pgmap v5360: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:43.523 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9912 sec 2024-03-24T03:09:44.523 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9913 sec 2024-03-24T03:09:45.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9258.236138 secs 2024-03-24T03:09:45.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:44 smithi078 ceph-mon[32429]: pgmap v5361: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:45.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9258.236138 secs 2024-03-24T03:09:45.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:44 smithi107 ceph-mon[36485]: pgmap v5361: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:45.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9258.236138 secs 2024-03-24T03:09:45.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:44 smithi106 ceph-mon[36505]: pgmap v5361: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:45.523 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9914 sec 2024-03-24T03:09:46.524 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9915 sec 2024-03-24T03:09:47.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:46 smithi078 ceph-mon[32429]: pgmap v5362: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:47.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:46 smithi107 ceph-mon[36485]: pgmap v5362: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:47.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:46 smithi106 ceph-mon[36505]: pgmap v5362: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:47.524 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9916 sec 2024-03-24T03:09:48.524 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9917 sec 2024-03-24T03:09:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:48 smithi078 ceph-mon[32429]: pgmap v5363: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:49.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:48 smithi107 ceph-mon[36485]: pgmap v5363: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:48 smithi106 ceph-mon[36505]: pgmap v5363: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:49.524 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9918 sec 2024-03-24T03:09:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9263.236227 secs 2024-03-24T03:09:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:49 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:09:50.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9263.236227 secs 2024-03-24T03:09:50.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:49 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:09:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9263.236227 secs 2024-03-24T03:09:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:49 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:09:50.524 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9919 sec 2024-03-24T03:09:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:50 smithi078 ceph-mon[32429]: pgmap v5364: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:50 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:09:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:50 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:09:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:50 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:09:51.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:50 smithi107 ceph-mon[36485]: pgmap v5364: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:51.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:50 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:09:51.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:50 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:09:51.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:50 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:09:51.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:50 smithi106 ceph-mon[36505]: pgmap v5364: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:09:51.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:50 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:09:51.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:50 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:09:51.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:50 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:09:51.524 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9920 sec 2024-03-24T03:09:52.053 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:51 smithi106 ceph-mon[36505]: pgmap v5365: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:09:52.053 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:51 smithi106 ceph-mon[36505]: pgmap v5366: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:09:52.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:51 smithi078 ceph-mon[32429]: pgmap v5365: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:09:52.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:51 smithi078 ceph-mon[32429]: pgmap v5366: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:09:52.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:51 smithi107 ceph-mon[36485]: pgmap v5365: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:09:52.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:51 smithi107 ceph-mon[36485]: pgmap v5366: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:09:52.524 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9921 sec 2024-03-24T03:09:53.524 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9922 sec 2024-03-24T03:09:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:53 smithi078 ceph-mon[32429]: pgmap v5367: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:09:54.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:53 smithi107 ceph-mon[36485]: pgmap v5367: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:09:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:53 smithi106 ceph-mon[36505]: pgmap v5367: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:09:54.525 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9923 sec 2024-03-24T03:09:55.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9268.236318 secs 2024-03-24T03:09:55.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9268.236318 secs 2024-03-24T03:09:55.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9268.236318 secs 2024-03-24T03:09:55.524 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9924 sec 2024-03-24T03:09:56.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:55 smithi078 ceph-mon[32429]: pgmap v5368: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:09:56.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:55 smithi107 ceph-mon[36485]: pgmap v5368: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:09:56.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:55 smithi106 ceph-mon[36505]: pgmap v5368: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:09:56.524 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9925 sec 2024-03-24T03:09:57.525 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9926 sec 2024-03-24T03:09:58.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:57 smithi078 ceph-mon[32429]: pgmap v5369: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:09:58.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:57 smithi107 ceph-mon[36485]: pgmap v5369: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:09:58.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:57 smithi106 ceph-mon[36505]: pgmap v5369: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:09:58.525 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9927 sec 2024-03-24T03:09:59.525 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9928 sec 2024-03-24T03:10:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9273.236380 secs 2024-03-24T03:10:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:09:59 smithi078 ceph-mon[32429]: pgmap v5370: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:10:00.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9273.236380 secs 2024-03-24T03:10:00.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:09:59 smithi107 ceph-mon[36485]: pgmap v5370: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:10:00.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9273.236380 secs 2024-03-24T03:10:00.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:09:59 smithi106 ceph-mon[36505]: pgmap v5370: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:10:00.525 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9929 sec 2024-03-24T03:10:01.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:00 smithi078 ceph-mon[32429]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T03:10:01.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:00 smithi107 ceph-mon[36485]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T03:10:01.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:00 smithi106 ceph-mon[36505]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T03:10:01.525 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9930 sec 2024-03-24T03:10:02.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:01 smithi078 ceph-mon[32429]: pgmap v5371: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:10:02.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:01 smithi107 ceph-mon[36485]: pgmap v5371: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:10:02.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:01 smithi106 ceph-mon[36505]: pgmap v5371: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:10:02.525 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9931 sec 2024-03-24T03:10:03.525 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9932 sec 2024-03-24T03:10:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:03 smithi078 ceph-mon[32429]: pgmap v5372: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9278.236442 secs 2024-03-24T03:10:04.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:03 smithi107 ceph-mon[36485]: pgmap v5372: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:04.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9278.236442 secs 2024-03-24T03:10:04.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:03 smithi106 ceph-mon[36505]: pgmap v5372: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9278.236442 secs 2024-03-24T03:10:04.525 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9933 sec 2024-03-24T03:10:05.525 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9934 sec 2024-03-24T03:10:06.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:05 smithi078 ceph-mon[32429]: pgmap v5373: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:06.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:05 smithi107 ceph-mon[36485]: pgmap v5373: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:06.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:05 smithi106 ceph-mon[36505]: pgmap v5373: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:06.525 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9935 sec 2024-03-24T03:10:07.526 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9936 sec 2024-03-24T03:10:08.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:07 smithi078 ceph-mon[32429]: pgmap v5374: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:08.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:07 smithi107 ceph-mon[36485]: pgmap v5374: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:08.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:07 smithi106 ceph-mon[36505]: pgmap v5374: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:08.526 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9937 sec 2024-03-24T03:10:09.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9283.236528 secs 2024-03-24T03:10:09.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9283.236528 secs 2024-03-24T03:10:09.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9283.236528 secs 2024-03-24T03:10:09.525 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9938 sec 2024-03-24T03:10:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:09 smithi078 ceph-mon[32429]: pgmap v5375: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:10.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:09 smithi107 ceph-mon[36485]: pgmap v5375: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:10.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:09 smithi106 ceph-mon[36505]: pgmap v5375: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:10.526 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9939 sec 2024-03-24T03:10:11.526 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9940 sec 2024-03-24T03:10:12.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:11 smithi078 ceph-mon[32429]: pgmap v5376: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:12.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:11 smithi107 ceph-mon[36485]: pgmap v5376: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:12.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:11 smithi106 ceph-mon[36505]: pgmap v5376: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:12.526 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9941 sec 2024-03-24T03:10:13.526 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9942 sec 2024-03-24T03:10:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:13 smithi078 ceph-mon[32429]: pgmap v5377: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9288.236588 secs 2024-03-24T03:10:14.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:13 smithi107 ceph-mon[36485]: pgmap v5377: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:14.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9288.236588 secs 2024-03-24T03:10:14.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:13 smithi106 ceph-mon[36505]: pgmap v5377: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:14.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9288.236588 secs 2024-03-24T03:10:14.526 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9943 sec 2024-03-24T03:10:15.527 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9944 sec 2024-03-24T03:10:16.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:15 smithi078 ceph-mon[32429]: pgmap v5378: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:16.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:15 smithi107 ceph-mon[36485]: pgmap v5378: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:16.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:15 smithi106 ceph-mon[36505]: pgmap v5378: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:16.526 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9945 sec 2024-03-24T03:10:17.527 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9946 sec 2024-03-24T03:10:18.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:17 smithi107 ceph-mon[36485]: pgmap v5379: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:18.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:17 smithi106 ceph-mon[36505]: pgmap v5379: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:17 smithi078 ceph-mon[32429]: pgmap v5379: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:18.526 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9947 sec 2024-03-24T03:10:19.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9293.236650 secs 2024-03-24T03:10:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9293.236650 secs 2024-03-24T03:10:19.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9293.236650 secs 2024-03-24T03:10:19.526 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9948 sec 2024-03-24T03:10:20.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:19 smithi106 ceph-mon[36505]: pgmap v5380: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:20.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:19 smithi078 ceph-mon[32429]: pgmap v5380: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:20.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:19 smithi107 ceph-mon[36485]: pgmap v5380: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:20.527 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9949 sec 2024-03-24T03:10:21.527 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9950 sec 2024-03-24T03:10:22.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:21 smithi078 ceph-mon[32429]: pgmap v5381: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:22.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:21 smithi106 ceph-mon[36505]: pgmap v5381: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:22.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:21 smithi107 ceph-mon[36485]: pgmap v5381: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:22.527 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9951 sec 2024-03-24T03:10:23.527 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9952 sec 2024-03-24T03:10:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:23 smithi106 ceph-mon[36505]: pgmap v5382: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9298.236740 secs 2024-03-24T03:10:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:23 smithi078 ceph-mon[32429]: pgmap v5382: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9298.236740 secs 2024-03-24T03:10:24.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:23 smithi107 ceph-mon[36485]: pgmap v5382: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9298.236740 secs 2024-03-24T03:10:24.527 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9953 sec 2024-03-24T03:10:25.527 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9954 sec 2024-03-24T03:10:26.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:25 smithi106 ceph-mon[36505]: pgmap v5383: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:26.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:25 smithi078 ceph-mon[32429]: pgmap v5383: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:26.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:25 smithi107 ceph-mon[36485]: pgmap v5383: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:26.528 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9955 sec 2024-03-24T03:10:27.527 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9956 sec 2024-03-24T03:10:28.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:27 smithi106 ceph-mon[36505]: pgmap v5384: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:28.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:27 smithi078 ceph-mon[32429]: pgmap v5384: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:28.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:27 smithi107 ceph-mon[36485]: pgmap v5384: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:28.527 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9957 sec 2024-03-24T03:10:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9303.236783 secs 2024-03-24T03:10:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9303.236783 secs 2024-03-24T03:10:29.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9303.236783 secs 2024-03-24T03:10:29.527 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9958 sec 2024-03-24T03:10:30.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:29 smithi106 ceph-mon[36505]: pgmap v5385: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:30.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:29 smithi078 ceph-mon[32429]: pgmap v5385: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:30.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:29 smithi107 ceph-mon[36485]: pgmap v5385: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:30.527 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9959 sec 2024-03-24T03:10:31.528 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9960 sec 2024-03-24T03:10:32.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:31 smithi106 ceph-mon[36505]: pgmap v5386: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:32.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:31 smithi078 ceph-mon[32429]: pgmap v5386: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:32.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:31 smithi107 ceph-mon[36485]: pgmap v5386: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:32.527 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9961 sec 2024-03-24T03:10:33.528 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9962 sec 2024-03-24T03:10:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:33 smithi106 ceph-mon[36505]: pgmap v5387: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9308.236870 secs 2024-03-24T03:10:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:33 smithi078 ceph-mon[32429]: pgmap v5387: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9308.236870 secs 2024-03-24T03:10:34.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:33 smithi107 ceph-mon[36485]: pgmap v5387: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9308.236870 secs 2024-03-24T03:10:34.528 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9963 sec 2024-03-24T03:10:35.528 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9964 sec 2024-03-24T03:10:36.145 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:35 smithi107 ceph-mon[36485]: pgmap v5388: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:36.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:35 smithi106 ceph-mon[36505]: pgmap v5388: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:36.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:35 smithi078 ceph-mon[32429]: pgmap v5388: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:36.528 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9965 sec 2024-03-24T03:10:37.529 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9966 sec 2024-03-24T03:10:38.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:37 smithi078 ceph-mon[32429]: pgmap v5389: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:38.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:37 smithi106 ceph-mon[36505]: pgmap v5389: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:38.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:37 smithi107 ceph-mon[36485]: pgmap v5389: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:38.528 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9967 sec 2024-03-24T03:10:39.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9313.236886 secs 2024-03-24T03:10:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9313.236886 secs 2024-03-24T03:10:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9313.236886 secs 2024-03-24T03:10:39.528 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9968 sec 2024-03-24T03:10:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:39 smithi106 ceph-mon[36505]: pgmap v5390: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:40.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:39 smithi078 ceph-mon[32429]: pgmap v5390: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:40.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:39 smithi107 ceph-mon[36485]: pgmap v5390: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:40.528 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9969 sec 2024-03-24T03:10:41.529 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9970 sec 2024-03-24T03:10:42.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:41 smithi106 ceph-mon[36505]: pgmap v5391: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:42.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:41 smithi078 ceph-mon[32429]: pgmap v5391: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:42.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:41 smithi107 ceph-mon[36485]: pgmap v5391: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:42.529 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9971 sec 2024-03-24T03:10:43.529 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9972 sec 2024-03-24T03:10:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:43 smithi106 ceph-mon[36505]: pgmap v5392: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9318.236976 secs 2024-03-24T03:10:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:43 smithi078 ceph-mon[32429]: pgmap v5392: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9318.236976 secs 2024-03-24T03:10:44.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:43 smithi107 ceph-mon[36485]: pgmap v5392: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9318.236976 secs 2024-03-24T03:10:44.529 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9973 sec 2024-03-24T03:10:45.529 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9974 sec 2024-03-24T03:10:46.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:45 smithi106 ceph-mon[36505]: pgmap v5393: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:46.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:45 smithi078 ceph-mon[32429]: pgmap v5393: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:46.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:45 smithi107 ceph-mon[36485]: pgmap v5393: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:46.529 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9975 sec 2024-03-24T03:10:47.529 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9976 sec 2024-03-24T03:10:48.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:47 smithi106 ceph-mon[36505]: pgmap v5394: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:48.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:47 smithi078 ceph-mon[32429]: pgmap v5394: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:48.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:47 smithi107 ceph-mon[36485]: pgmap v5394: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:48.529 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9977 sec 2024-03-24T03:10:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9323.237011 secs 2024-03-24T03:10:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9323.237011 secs 2024-03-24T03:10:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9323.237011 secs 2024-03-24T03:10:49.529 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9978 sec 2024-03-24T03:10:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:49 smithi078 ceph-mon[32429]: pgmap v5395: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:50.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:49 smithi107 ceph-mon[36485]: pgmap v5395: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:50.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:49 smithi106 ceph-mon[36505]: pgmap v5395: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:50.529 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9979 sec 2024-03-24T03:10:51.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:50 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:10:51.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:50 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:10:51.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:50 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:10:51.529 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9980 sec 2024-03-24T03:10:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:51 smithi078 ceph-mon[32429]: pgmap v5396: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:51 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:10:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:51 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:10:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:51 smithi078 ceph-mon[32429]: pgmap v5397: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:10:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:51 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:10:52.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:51 smithi107 ceph-mon[36485]: pgmap v5396: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:52.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:51 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:10:52.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:51 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:10:52.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:51 smithi107 ceph-mon[36485]: pgmap v5397: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:10:52.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:51 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:10:52.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:51 smithi106 ceph-mon[36505]: pgmap v5396: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:10:52.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:51 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:10:52.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:51 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:10:52.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:51 smithi106 ceph-mon[36505]: pgmap v5397: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:10:52.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:51 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:10:52.529 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9981 sec 2024-03-24T03:10:53.529 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9982 sec 2024-03-24T03:10:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:54 smithi078 ceph-mon[32429]: pgmap v5398: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T03:10:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9328.237097 secs 2024-03-24T03:10:54.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:54 smithi107 ceph-mon[36485]: pgmap v5398: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T03:10:54.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9328.237097 secs 2024-03-24T03:10:54.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:54 smithi106 ceph-mon[36505]: pgmap v5398: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T03:10:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9328.237097 secs 2024-03-24T03:10:54.529 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9983 sec 2024-03-24T03:10:55.530 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9984 sec 2024-03-24T03:10:56.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:56 smithi078 ceph-mon[32429]: pgmap v5399: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T03:10:56.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:56 smithi107 ceph-mon[36485]: pgmap v5399: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T03:10:56.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:56 smithi106 ceph-mon[36505]: pgmap v5399: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T03:10:56.530 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9985 sec 2024-03-24T03:10:57.530 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9986 sec 2024-03-24T03:10:58.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:58 smithi078 ceph-mon[32429]: pgmap v5400: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T03:10:58.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:58 smithi107 ceph-mon[36485]: pgmap v5400: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T03:10:58.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:58 smithi106 ceph-mon[36505]: pgmap v5400: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T03:10:58.530 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9987 sec 2024-03-24T03:10:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:10:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9333.237186 secs 2024-03-24T03:10:59.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:10:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9333.237186 secs 2024-03-24T03:10:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:10:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9333.237186 secs 2024-03-24T03:10:59.530 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9988 sec 2024-03-24T03:11:00.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:00 smithi078 ceph-mon[32429]: pgmap v5401: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T03:11:00.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:00 smithi107 ceph-mon[36485]: pgmap v5401: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T03:11:00.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:00 smithi106 ceph-mon[36505]: pgmap v5401: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T03:11:00.530 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9989 sec 2024-03-24T03:11:01.530 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9990 sec 2024-03-24T03:11:02.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:02 smithi078 ceph-mon[32429]: pgmap v5402: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:11:02.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:02 smithi107 ceph-mon[36485]: pgmap v5402: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:11:02.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:02 smithi106 ceph-mon[36505]: pgmap v5402: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:11:02.530 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9991 sec 2024-03-24T03:11:03.530 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9992 sec 2024-03-24T03:11:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:04 smithi078 ceph-mon[32429]: pgmap v5403: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9338.237269 secs 2024-03-24T03:11:04.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:04 smithi107 ceph-mon[36485]: pgmap v5403: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:04.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9338.237269 secs 2024-03-24T03:11:04.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:04 smithi106 ceph-mon[36505]: pgmap v5403: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:04.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9338.237269 secs 2024-03-24T03:11:04.530 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9993 sec 2024-03-24T03:11:05.531 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9994 sec 2024-03-24T03:11:06.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:06 smithi078 ceph-mon[32429]: pgmap v5404: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:06.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:06 smithi107 ceph-mon[36485]: pgmap v5404: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:06.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:06 smithi106 ceph-mon[36505]: pgmap v5404: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:06.531 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9995 sec 2024-03-24T03:11:07.531 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9996 sec 2024-03-24T03:11:08.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:08 smithi078 ceph-mon[32429]: pgmap v5405: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:08.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:08 smithi107 ceph-mon[36485]: pgmap v5405: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:08.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:08 smithi106 ceph-mon[36505]: pgmap v5405: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:08.531 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9997 sec 2024-03-24T03:11:09.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9343.237290 secs 2024-03-24T03:11:09.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9343.237290 secs 2024-03-24T03:11:09.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9343.237290 secs 2024-03-24T03:11:09.531 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9998 sec 2024-03-24T03:11:10.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:10 smithi078 ceph-mon[32429]: pgmap v5406: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:10.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:10 smithi107 ceph-mon[36485]: pgmap v5406: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:10.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:10 smithi106 ceph-mon[36505]: pgmap v5406: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:10.531 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 9999 sec 2024-03-24T03:11:11.531 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10000 sec 2024-03-24T03:11:12.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:12 smithi078 ceph-mon[32429]: pgmap v5407: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:12.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:12 smithi107 ceph-mon[36485]: pgmap v5407: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:12.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:12 smithi106 ceph-mon[36505]: pgmap v5407: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:12.531 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10001 sec 2024-03-24T03:11:13.532 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10002 sec 2024-03-24T03:11:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:14 smithi078 ceph-mon[32429]: pgmap v5408: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9348.237351 secs 2024-03-24T03:11:14.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:14 smithi107 ceph-mon[36485]: pgmap v5408: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:14.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9348.237351 secs 2024-03-24T03:11:14.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:14 smithi106 ceph-mon[36505]: pgmap v5408: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9348.237351 secs 2024-03-24T03:11:14.532 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10003 sec 2024-03-24T03:11:15.532 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10004 sec 2024-03-24T03:11:16.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:16 smithi078 ceph-mon[32429]: pgmap v5409: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:16.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:16 smithi107 ceph-mon[36485]: pgmap v5409: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:16.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:16 smithi106 ceph-mon[36505]: pgmap v5409: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:16.532 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10005 sec 2024-03-24T03:11:17.532 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10006 sec 2024-03-24T03:11:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:18 smithi078 ceph-mon[32429]: pgmap v5410: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:18.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:18 smithi107 ceph-mon[36485]: pgmap v5410: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:18.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:18 smithi106 ceph-mon[36505]: pgmap v5410: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:18.532 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10007 sec 2024-03-24T03:11:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9353.237412 secs 2024-03-24T03:11:19.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9353.237412 secs 2024-03-24T03:11:19.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9353.237412 secs 2024-03-24T03:11:19.532 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10008 sec 2024-03-24T03:11:20.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:20 smithi078 ceph-mon[32429]: pgmap v5411: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:20.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:20 smithi107 ceph-mon[36485]: pgmap v5411: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:20.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:20 smithi106 ceph-mon[36505]: pgmap v5411: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:20.532 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10009 sec 2024-03-24T03:11:21.532 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10010 sec 2024-03-24T03:11:22.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:22 smithi078 ceph-mon[32429]: pgmap v5412: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:22.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:22 smithi107 ceph-mon[36485]: pgmap v5412: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:22.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:22 smithi106 ceph-mon[36505]: pgmap v5412: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:22.532 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10011 sec 2024-03-24T03:11:23.533 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10012 sec 2024-03-24T03:11:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:24 smithi078 ceph-mon[32429]: pgmap v5413: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9358.237470 secs 2024-03-24T03:11:24.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:24 smithi107 ceph-mon[36485]: pgmap v5413: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:24.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9358.237470 secs 2024-03-24T03:11:24.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:24 smithi106 ceph-mon[36505]: pgmap v5413: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9358.237470 secs 2024-03-24T03:11:24.533 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10013 sec 2024-03-24T03:11:25.533 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10014 sec 2024-03-24T03:11:26.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:26 smithi078 ceph-mon[32429]: pgmap v5414: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:26.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:26 smithi107 ceph-mon[36485]: pgmap v5414: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:26.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:26 smithi106 ceph-mon[36505]: pgmap v5414: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:26.533 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10015 sec 2024-03-24T03:11:27.533 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10016 sec 2024-03-24T03:11:28.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:28 smithi078 ceph-mon[32429]: pgmap v5415: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:28.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:28 smithi107 ceph-mon[36485]: pgmap v5415: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:28.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:28 smithi106 ceph-mon[36505]: pgmap v5415: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:28.533 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10017 sec 2024-03-24T03:11:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9363.237533 secs 2024-03-24T03:11:29.533 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10018 sec 2024-03-24T03:11:29.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9363.237533 secs 2024-03-24T03:11:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9363.237533 secs 2024-03-24T03:11:30.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:30 smithi106 ceph-mon[36505]: pgmap v5416: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:30.533 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10019 sec 2024-03-24T03:11:30.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:30 smithi078 ceph-mon[32429]: pgmap v5416: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:30.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:30 smithi107 ceph-mon[36485]: pgmap v5416: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:31.533 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10020 sec 2024-03-24T03:11:32.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:32 smithi106 ceph-mon[36505]: pgmap v5417: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:32.533 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10021 sec 2024-03-24T03:11:32.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:32 smithi078 ceph-mon[32429]: pgmap v5417: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:32.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:32 smithi107 ceph-mon[36485]: pgmap v5417: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:33.533 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10022 sec 2024-03-24T03:11:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:34 smithi106 ceph-mon[36505]: pgmap v5418: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9368.237593 secs 2024-03-24T03:11:34.533 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10023 sec 2024-03-24T03:11:34.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:34 smithi078 ceph-mon[32429]: pgmap v5418: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:34.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9368.237593 secs 2024-03-24T03:11:34.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:34 smithi107 ceph-mon[36485]: pgmap v5418: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:34.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9368.237593 secs 2024-03-24T03:11:35.534 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10024 sec 2024-03-24T03:11:36.395 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:36 smithi107 ceph-mon[36485]: pgmap v5419: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:36.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:36 smithi106 ceph-mon[36505]: pgmap v5419: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:36.534 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10025 sec 2024-03-24T03:11:36.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:36 smithi078 ceph-mon[32429]: pgmap v5419: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:37 smithi106 ceph-mon[36505]: pgmap v5420: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:37.534 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10026 sec 2024-03-24T03:11:37.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:37 smithi078 ceph-mon[32429]: pgmap v5420: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:37.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:37 smithi107 ceph-mon[36485]: pgmap v5420: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:38.534 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10027 sec 2024-03-24T03:11:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9373.237654 secs 2024-03-24T03:11:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9373.237654 secs 2024-03-24T03:11:39.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9373.237654 secs 2024-03-24T03:11:39.534 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10028 sec 2024-03-24T03:11:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:39 smithi106 ceph-mon[36505]: pgmap v5421: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:40.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:39 smithi078 ceph-mon[32429]: pgmap v5421: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:40.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:39 smithi107 ceph-mon[36485]: pgmap v5421: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:40.534 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10029 sec 2024-03-24T03:11:41.534 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10030 sec 2024-03-24T03:11:42.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:42 smithi078 ceph-mon[32429]: pgmap v5422: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:42.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:42 smithi107 ceph-mon[36485]: pgmap v5422: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:42.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:42 smithi106 ceph-mon[36505]: pgmap v5422: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:42.534 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10031 sec 2024-03-24T03:11:43.535 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10032 sec 2024-03-24T03:11:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:44 smithi078 ceph-mon[32429]: pgmap v5423: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9378.237739 secs 2024-03-24T03:11:44.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:44 smithi107 ceph-mon[36485]: pgmap v5423: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:44.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9378.237739 secs 2024-03-24T03:11:44.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:44 smithi106 ceph-mon[36505]: pgmap v5423: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9378.237739 secs 2024-03-24T03:11:44.535 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10033 sec 2024-03-24T03:11:45.535 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10034 sec 2024-03-24T03:11:46.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:46 smithi078 ceph-mon[32429]: pgmap v5424: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:46.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:46 smithi107 ceph-mon[36485]: pgmap v5424: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:46.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:46 smithi106 ceph-mon[36505]: pgmap v5424: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:46.535 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10035 sec 2024-03-24T03:11:47.535 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10036 sec 2024-03-24T03:11:48.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:48 smithi078 ceph-mon[32429]: pgmap v5425: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:48.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:48 smithi107 ceph-mon[36485]: pgmap v5425: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:48.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:48 smithi106 ceph-mon[36505]: pgmap v5425: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:48.535 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10037 sec 2024-03-24T03:11:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9383.237800 secs 2024-03-24T03:11:49.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9383.237800 secs 2024-03-24T03:11:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9383.237800 secs 2024-03-24T03:11:49.535 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10038 sec 2024-03-24T03:11:50.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:50 smithi078 ceph-mon[32429]: pgmap v5426: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:50.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:50 smithi107 ceph-mon[36485]: pgmap v5426: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:50.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:50 smithi106 ceph-mon[36505]: pgmap v5426: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:11:50.535 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10039 sec 2024-03-24T03:11:51.535 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10040 sec 2024-03-24T03:11:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:52 smithi078 ceph-mon[32429]: pgmap v5427: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:52 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:11:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:52 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:11:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:52 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:11:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:52 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:11:52.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:52 smithi107 ceph-mon[36485]: pgmap v5427: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:52.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:52 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:11:52.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:52 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:11:52.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:52 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:11:52.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:52 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:11:52.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:52 smithi106 ceph-mon[36505]: pgmap v5427: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:52.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:52 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:11:52.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:52 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:11:52.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:52 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:11:52.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:52 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:11:52.535 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10041 sec 2024-03-24T03:11:53.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:53 smithi078 ceph-mon[32429]: pgmap v5428: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T03:11:53.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:53 smithi078 ceph-mon[32429]: pgmap v5429: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:11:53.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:53 smithi107 ceph-mon[36485]: pgmap v5428: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T03:11:53.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:53 smithi107 ceph-mon[36485]: pgmap v5429: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:11:53.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:53 smithi106 ceph-mon[36505]: pgmap v5428: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T03:11:53.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:53 smithi106 ceph-mon[36505]: pgmap v5429: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:11:53.535 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10042 sec 2024-03-24T03:11:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9388.237888 secs 2024-03-24T03:11:54.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9388.237888 secs 2024-03-24T03:11:54.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9388.237888 secs 2024-03-24T03:11:54.535 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10043 sec 2024-03-24T03:11:55.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:55 smithi078 ceph-mon[32429]: pgmap v5430: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:11:55.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:55 smithi107 ceph-mon[36485]: pgmap v5430: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:11:55.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:55 smithi106 ceph-mon[36505]: pgmap v5430: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:11:55.535 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10044 sec 2024-03-24T03:11:56.535 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10045 sec 2024-03-24T03:11:57.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:57 smithi078 ceph-mon[32429]: pgmap v5431: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:11:57.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:57 smithi107 ceph-mon[36485]: pgmap v5431: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:11:57.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:57 smithi106 ceph-mon[36505]: pgmap v5431: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:11:57.536 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10046 sec 2024-03-24T03:11:58.536 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10047 sec 2024-03-24T03:11:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:59 smithi078 ceph-mon[32429]: pgmap v5432: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:11:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:11:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9393.237938 secs 2024-03-24T03:11:59.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:59 smithi107 ceph-mon[36485]: pgmap v5432: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:11:59.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:11:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9393.237938 secs 2024-03-24T03:11:59.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:59 smithi106 ceph-mon[36505]: pgmap v5432: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:11:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:11:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9393.237938 secs 2024-03-24T03:11:59.536 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10048 sec 2024-03-24T03:12:00.536 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10049 sec 2024-03-24T03:12:01.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:01 smithi078 ceph-mon[32429]: pgmap v5433: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:12:01.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:01 smithi107 ceph-mon[36485]: pgmap v5433: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:12:01.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:01 smithi106 ceph-mon[36505]: pgmap v5433: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:12:01.536 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10050 sec 2024-03-24T03:12:02.536 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10051 sec 2024-03-24T03:12:02.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:02 smithi106 ceph-mon[36505]: pgmap v5434: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:12:03.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:02 smithi078 ceph-mon[32429]: pgmap v5434: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:12:03.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:02 smithi107 ceph-mon[36485]: pgmap v5434: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:12:03.536 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10052 sec 2024-03-24T03:12:04.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9398.237984 secs 2024-03-24T03:12:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9398.237984 secs 2024-03-24T03:12:04.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9398.237984 secs 2024-03-24T03:12:04.536 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10053 sec 2024-03-24T03:12:05.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:04 smithi106 ceph-mon[36505]: pgmap v5435: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:05.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:04 smithi078 ceph-mon[32429]: pgmap v5435: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:05.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:04 smithi107 ceph-mon[36485]: pgmap v5435: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:05.536 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10054 sec 2024-03-24T03:12:06.536 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10055 sec 2024-03-24T03:12:07.145 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:06 smithi107 ceph-mon[36485]: pgmap v5436: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:07.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:06 smithi106 ceph-mon[36505]: pgmap v5436: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:07.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:06 smithi078 ceph-mon[32429]: pgmap v5436: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:07.536 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10056 sec 2024-03-24T03:12:08.537 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10057 sec 2024-03-24T03:12:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:08 smithi106 ceph-mon[36505]: pgmap v5437: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:09.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9403.238046 secs 2024-03-24T03:12:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:08 smithi078 ceph-mon[32429]: pgmap v5437: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9403.238046 secs 2024-03-24T03:12:09.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:08 smithi107 ceph-mon[36485]: pgmap v5437: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:09.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9403.238046 secs 2024-03-24T03:12:09.537 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10058 sec 2024-03-24T03:12:10.537 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10059 sec 2024-03-24T03:12:11.142 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:10 smithi078 ceph-mon[32429]: pgmap v5438: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:11.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:10 smithi106 ceph-mon[36505]: pgmap v5438: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:11.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:10 smithi107 ceph-mon[36485]: pgmap v5438: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:11.537 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10060 sec 2024-03-24T03:12:12.537 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10061 sec 2024-03-24T03:12:13.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:12 smithi106 ceph-mon[36505]: pgmap v5439: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:13.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:12 smithi078 ceph-mon[32429]: pgmap v5439: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:13.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:12 smithi107 ceph-mon[36485]: pgmap v5439: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:13.537 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10062 sec 2024-03-24T03:12:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9408.238068 secs 2024-03-24T03:12:14.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9408.238068 secs 2024-03-24T03:12:14.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9408.238068 secs 2024-03-24T03:12:14.537 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10063 sec 2024-03-24T03:12:15.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:14 smithi078 ceph-mon[32429]: pgmap v5440: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:15.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:14 smithi107 ceph-mon[36485]: pgmap v5440: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:15.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:14 smithi106 ceph-mon[36505]: pgmap v5440: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:15.538 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10064 sec 2024-03-24T03:12:16.538 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10065 sec 2024-03-24T03:12:17.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:16 smithi078 ceph-mon[32429]: pgmap v5441: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:17.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:16 smithi107 ceph-mon[36485]: pgmap v5441: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:17.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:16 smithi106 ceph-mon[36505]: pgmap v5441: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:17.537 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10066 sec 2024-03-24T03:12:18.538 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10067 sec 2024-03-24T03:12:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:18 smithi078 ceph-mon[32429]: pgmap v5442: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9413.238132 secs 2024-03-24T03:12:19.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:18 smithi107 ceph-mon[36485]: pgmap v5442: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:19.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9413.238132 secs 2024-03-24T03:12:19.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:18 smithi106 ceph-mon[36505]: pgmap v5442: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:19.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9413.238132 secs 2024-03-24T03:12:19.537 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10068 sec 2024-03-24T03:12:20.538 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10069 sec 2024-03-24T03:12:21.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:20 smithi078 ceph-mon[32429]: pgmap v5443: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:21.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:20 smithi107 ceph-mon[36485]: pgmap v5443: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:21.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:20 smithi106 ceph-mon[36505]: pgmap v5443: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:21.538 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10070 sec 2024-03-24T03:12:22.538 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10071 sec 2024-03-24T03:12:23.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:22 smithi078 ceph-mon[32429]: pgmap v5444: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:23.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:22 smithi107 ceph-mon[36485]: pgmap v5444: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:23.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:22 smithi106 ceph-mon[36505]: pgmap v5444: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:23.538 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10072 sec 2024-03-24T03:12:24.303 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9418.238220 secs 2024-03-24T03:12:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9418.238220 secs 2024-03-24T03:12:24.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9418.238220 secs 2024-03-24T03:12:24.538 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10073 sec 2024-03-24T03:12:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:24 smithi078 ceph-mon[32429]: pgmap v5445: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:25.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:24 smithi107 ceph-mon[36485]: pgmap v5445: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:25.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:24 smithi106 ceph-mon[36505]: pgmap v5445: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:25.538 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10074 sec 2024-03-24T03:12:26.538 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10075 sec 2024-03-24T03:12:27.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:26 smithi078 ceph-mon[32429]: pgmap v5446: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:27.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:26 smithi107 ceph-mon[36485]: pgmap v5446: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:27.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:26 smithi106 ceph-mon[36505]: pgmap v5446: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:27.538 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10076 sec 2024-03-24T03:12:28.538 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10077 sec 2024-03-24T03:12:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:28 smithi078 ceph-mon[32429]: pgmap v5447: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9423.238324 secs 2024-03-24T03:12:29.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:28 smithi107 ceph-mon[36485]: pgmap v5447: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:29.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9423.238324 secs 2024-03-24T03:12:29.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:28 smithi106 ceph-mon[36505]: pgmap v5447: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:29.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9423.238324 secs 2024-03-24T03:12:29.538 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10078 sec 2024-03-24T03:12:30.539 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10079 sec 2024-03-24T03:12:31.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:30 smithi078 ceph-mon[32429]: pgmap v5448: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:31.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:30 smithi107 ceph-mon[36485]: pgmap v5448: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:31.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:30 smithi106 ceph-mon[36505]: pgmap v5448: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:31.539 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10080 sec 2024-03-24T03:12:32.539 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10081 sec 2024-03-24T03:12:33.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:32 smithi078 ceph-mon[32429]: pgmap v5449: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:33.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:32 smithi107 ceph-mon[36485]: pgmap v5449: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:33.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:32 smithi106 ceph-mon[36505]: pgmap v5449: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:33.539 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10082 sec 2024-03-24T03:12:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9428.238408 secs 2024-03-24T03:12:34.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9428.238408 secs 2024-03-24T03:12:34.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9428.238408 secs 2024-03-24T03:12:34.539 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10083 sec 2024-03-24T03:12:35.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:34 smithi078 ceph-mon[32429]: pgmap v5450: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:35.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:34 smithi107 ceph-mon[36485]: pgmap v5450: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:35.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:34 smithi106 ceph-mon[36505]: pgmap v5450: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:35.539 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10084 sec 2024-03-24T03:12:36.539 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10085 sec 2024-03-24T03:12:37.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:36 smithi078 ceph-mon[32429]: pgmap v5451: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:37.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:36 smithi107 ceph-mon[36485]: pgmap v5451: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:37.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:36 smithi106 ceph-mon[36505]: pgmap v5451: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:37.539 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10086 sec 2024-03-24T03:12:38.539 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10087 sec 2024-03-24T03:12:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:38 smithi078 ceph-mon[32429]: pgmap v5452: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9433.238468 secs 2024-03-24T03:12:39.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:38 smithi107 ceph-mon[36485]: pgmap v5452: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:39.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9433.238468 secs 2024-03-24T03:12:39.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:38 smithi106 ceph-mon[36505]: pgmap v5452: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:39.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9433.238468 secs 2024-03-24T03:12:39.539 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10088 sec 2024-03-24T03:12:40.539 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10089 sec 2024-03-24T03:12:41.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:40 smithi078 ceph-mon[32429]: pgmap v5453: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:41.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:40 smithi107 ceph-mon[36485]: pgmap v5453: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:41.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:40 smithi106 ceph-mon[36505]: pgmap v5453: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:41.540 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10090 sec 2024-03-24T03:12:42.540 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10091 sec 2024-03-24T03:12:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:42 smithi078 ceph-mon[32429]: pgmap v5454: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:43.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:42 smithi107 ceph-mon[36485]: pgmap v5454: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:43.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:42 smithi106 ceph-mon[36505]: pgmap v5454: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:43.540 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10092 sec 2024-03-24T03:12:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9438.238528 secs 2024-03-24T03:12:44.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9438.238528 secs 2024-03-24T03:12:44.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9438.238528 secs 2024-03-24T03:12:44.540 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10093 sec 2024-03-24T03:12:45.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:44 smithi078 ceph-mon[32429]: pgmap v5455: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:45.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:44 smithi107 ceph-mon[36485]: pgmap v5455: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:45.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:44 smithi106 ceph-mon[36505]: pgmap v5455: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:45.540 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10094 sec 2024-03-24T03:12:46.540 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10095 sec 2024-03-24T03:12:47.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:46 smithi078 ceph-mon[32429]: pgmap v5456: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:47.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:46 smithi107 ceph-mon[36485]: pgmap v5456: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:47.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:46 smithi106 ceph-mon[36505]: pgmap v5456: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:47.540 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10096 sec 2024-03-24T03:12:48.540 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10097 sec 2024-03-24T03:12:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:48 smithi078 ceph-mon[32429]: pgmap v5457: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9443.238593 secs 2024-03-24T03:12:49.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:48 smithi107 ceph-mon[36485]: pgmap v5457: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:49.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9443.238593 secs 2024-03-24T03:12:49.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:48 smithi106 ceph-mon[36505]: pgmap v5457: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:49.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9443.238593 secs 2024-03-24T03:12:49.540 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10098 sec 2024-03-24T03:12:50.541 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10099 sec 2024-03-24T03:12:51.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:50 smithi078 ceph-mon[32429]: pgmap v5458: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:51.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:50 smithi107 ceph-mon[36485]: pgmap v5458: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:51.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:50 smithi106 ceph-mon[36505]: pgmap v5458: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:12:51.540 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10100 sec 2024-03-24T03:12:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:51 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:12:52.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:51 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:12:52.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:51 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:12:52.540 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10101 sec 2024-03-24T03:12:53.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:52 smithi078 ceph-mon[32429]: pgmap v5459: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:53.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:52 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:12:53.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:52 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:12:53.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:52 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:12:53.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:53 smithi107 ceph-mon[36485]: pgmap v5459: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:53.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:53 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:12:53.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:53 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:12:53.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:53 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:12:53.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:52 smithi106 ceph-mon[36505]: pgmap v5459: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:53.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:52 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:12:53.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:52 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:12:53.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:52 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:12:53.540 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10102 sec 2024-03-24T03:12:54.303 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:53 smithi106 ceph-mon[36505]: pgmap v5460: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:12:54.303 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:53 smithi106 ceph-mon[36505]: pgmap v5461: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:12:54.303 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9448.238681 secs 2024-03-24T03:12:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:53 smithi078 ceph-mon[32429]: pgmap v5460: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:12:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:53 smithi078 ceph-mon[32429]: pgmap v5461: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:12:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9448.238681 secs 2024-03-24T03:12:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:53 smithi107 ceph-mon[36485]: pgmap v5460: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:12:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:53 smithi107 ceph-mon[36485]: pgmap v5461: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:12:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9448.238681 secs 2024-03-24T03:12:54.541 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10103 sec 2024-03-24T03:12:55.543 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10104 sec 2024-03-24T03:12:56.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:56 smithi078 ceph-mon[32429]: pgmap v5462: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:12:56.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:56 smithi107 ceph-mon[36485]: pgmap v5462: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:12:56.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:56 smithi106 ceph-mon[36505]: pgmap v5462: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:12:56.541 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10105 sec 2024-03-24T03:12:57.541 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10106 sec 2024-03-24T03:12:58.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:58 smithi078 ceph-mon[32429]: pgmap v5463: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:12:58.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:58 smithi107 ceph-mon[36485]: pgmap v5463: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:12:58.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:58 smithi106 ceph-mon[36505]: pgmap v5463: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:12:58.541 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10107 sec 2024-03-24T03:12:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:12:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9453.238743 secs 2024-03-24T03:12:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:12:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9453.238743 secs 2024-03-24T03:12:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:12:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9453.238743 secs 2024-03-24T03:12:59.541 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10108 sec 2024-03-24T03:13:00.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:00 smithi078 ceph-mon[32429]: pgmap v5464: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:13:00.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:00 smithi107 ceph-mon[36485]: pgmap v5464: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:13:00.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:00 smithi106 ceph-mon[36505]: pgmap v5464: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:13:00.541 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10109 sec 2024-03-24T03:13:01.541 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10110 sec 2024-03-24T03:13:02.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:02 smithi078 ceph-mon[32429]: pgmap v5465: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:13:02.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:02 smithi107 ceph-mon[36485]: pgmap v5465: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:13:02.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:02 smithi106 ceph-mon[36505]: pgmap v5465: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:13:02.542 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10111 sec 2024-03-24T03:13:03.542 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10112 sec 2024-03-24T03:13:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:04 smithi078 ceph-mon[32429]: pgmap v5466: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:13:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9458.238803 secs 2024-03-24T03:13:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:04 smithi107 ceph-mon[36485]: pgmap v5466: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:13:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9458.238803 secs 2024-03-24T03:13:04.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:04 smithi106 ceph-mon[36505]: pgmap v5466: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:13:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9458.238803 secs 2024-03-24T03:13:04.542 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10113 sec 2024-03-24T03:13:05.542 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10114 sec 2024-03-24T03:13:06.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:06 smithi078 ceph-mon[32429]: pgmap v5467: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:06.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:06 smithi107 ceph-mon[36485]: pgmap v5467: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:06.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:06 smithi106 ceph-mon[36505]: pgmap v5467: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:06.542 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10115 sec 2024-03-24T03:13:07.542 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10116 sec 2024-03-24T03:13:08.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:08 smithi078 ceph-mon[32429]: pgmap v5468: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:08.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:08 smithi107 ceph-mon[36485]: pgmap v5468: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:08.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:08 smithi106 ceph-mon[36505]: pgmap v5468: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:08.542 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10117 sec 2024-03-24T03:13:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9463.238864 secs 2024-03-24T03:13:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9463.238864 secs 2024-03-24T03:13:09.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9463.238864 secs 2024-03-24T03:13:09.542 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10118 sec 2024-03-24T03:13:10.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:10 smithi078 ceph-mon[32429]: pgmap v5469: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:10.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:10 smithi107 ceph-mon[36485]: pgmap v5469: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:10.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:10 smithi106 ceph-mon[36505]: pgmap v5469: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:10.542 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10119 sec 2024-03-24T03:13:11.543 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10120 sec 2024-03-24T03:13:12.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:12 smithi078 ceph-mon[32429]: pgmap v5470: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:12.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:12 smithi107 ceph-mon[36485]: pgmap v5470: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:12.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:12 smithi106 ceph-mon[36505]: pgmap v5470: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:12.542 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10121 sec 2024-03-24T03:13:13.543 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10122 sec 2024-03-24T03:13:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:14 smithi078 ceph-mon[32429]: pgmap v5471: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9468.238953 secs 2024-03-24T03:13:14.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:14 smithi107 ceph-mon[36485]: pgmap v5471: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:14.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9468.238953 secs 2024-03-24T03:13:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:14 smithi106 ceph-mon[36505]: pgmap v5471: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9468.238953 secs 2024-03-24T03:13:14.543 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10123 sec 2024-03-24T03:13:15.543 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10124 sec 2024-03-24T03:13:16.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:16 smithi078 ceph-mon[32429]: pgmap v5472: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:16.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:16 smithi107 ceph-mon[36485]: pgmap v5472: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:16.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:16 smithi106 ceph-mon[36505]: pgmap v5472: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:16.543 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10125 sec 2024-03-24T03:13:17.543 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10126 sec 2024-03-24T03:13:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:18 smithi078 ceph-mon[32429]: pgmap v5473: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:13:18.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:18 smithi107 ceph-mon[36485]: pgmap v5473: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:13:18.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:18 smithi106 ceph-mon[36505]: pgmap v5473: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:13:18.543 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10127 sec 2024-03-24T03:13:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9473.239040 secs 2024-03-24T03:13:19.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9473.239040 secs 2024-03-24T03:13:19.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9473.239040 secs 2024-03-24T03:13:19.543 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10128 sec 2024-03-24T03:13:20.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:20 smithi078 ceph-mon[32429]: pgmap v5474: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T03:13:20.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:20 smithi107 ceph-mon[36485]: pgmap v5474: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T03:13:20.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:20 smithi106 ceph-mon[36505]: pgmap v5474: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T03:13:20.543 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10129 sec 2024-03-24T03:13:21.543 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10130 sec 2024-03-24T03:13:22.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:22 smithi078 ceph-mon[32429]: pgmap v5475: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:13:22.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:22 smithi107 ceph-mon[36485]: pgmap v5475: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:13:22.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:22 smithi106 ceph-mon[36505]: pgmap v5475: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:13:22.543 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10131 sec 2024-03-24T03:13:23.543 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10132 sec 2024-03-24T03:13:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:24 smithi078 ceph-mon[32429]: pgmap v5476: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-03-24T03:13:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9478.239093 secs 2024-03-24T03:13:24.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:24 smithi107 ceph-mon[36485]: pgmap v5476: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-03-24T03:13:24.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9478.239093 secs 2024-03-24T03:13:24.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:24 smithi106 ceph-mon[36505]: pgmap v5476: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-03-24T03:13:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9478.239093 secs 2024-03-24T03:13:24.544 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10133 sec 2024-03-24T03:13:25.544 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10134 sec 2024-03-24T03:13:26.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:26 smithi078 ceph-mon[32429]: pgmap v5477: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-03-24T03:13:26.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:26 smithi107 ceph-mon[36485]: pgmap v5477: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-03-24T03:13:26.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:26 smithi106 ceph-mon[36505]: pgmap v5477: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-03-24T03:13:26.544 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10135 sec 2024-03-24T03:13:27.544 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10136 sec 2024-03-24T03:13:28.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:28 smithi078 ceph-mon[32429]: pgmap v5478: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:13:28.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:28 smithi107 ceph-mon[36485]: pgmap v5478: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:13:28.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:28 smithi106 ceph-mon[36505]: pgmap v5478: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:13:28.544 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10137 sec 2024-03-24T03:13:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9483.239152 secs 2024-03-24T03:13:29.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9483.239152 secs 2024-03-24T03:13:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9483.239152 secs 2024-03-24T03:13:29.544 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10138 sec 2024-03-24T03:13:30.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:30 smithi078 ceph-mon[32429]: pgmap v5479: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T03:13:30.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:30 smithi107 ceph-mon[36485]: pgmap v5479: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T03:13:30.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:30 smithi106 ceph-mon[36505]: pgmap v5479: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T03:13:30.544 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10139 sec 2024-03-24T03:13:31.544 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10140 sec 2024-03-24T03:13:32.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:32 smithi078 ceph-mon[32429]: pgmap v5480: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:13:32.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:32 smithi107 ceph-mon[36485]: pgmap v5480: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:13:32.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:32 smithi106 ceph-mon[36505]: pgmap v5480: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:13:32.544 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10141 sec 2024-03-24T03:13:33.544 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10142 sec 2024-03-24T03:13:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:34 smithi078 ceph-mon[32429]: pgmap v5481: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9488.239233 secs 2024-03-24T03:13:34.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:34 smithi107 ceph-mon[36485]: pgmap v5481: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:34.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9488.239233 secs 2024-03-24T03:13:34.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:34 smithi106 ceph-mon[36505]: pgmap v5481: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:34.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9488.239233 secs 2024-03-24T03:13:34.544 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10143 sec 2024-03-24T03:13:35.544 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10144 sec 2024-03-24T03:13:36.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:36 smithi078 ceph-mon[32429]: pgmap v5482: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:36.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:36 smithi107 ceph-mon[36485]: pgmap v5482: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:36.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:36 smithi106 ceph-mon[36505]: pgmap v5482: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:36.545 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10145 sec 2024-03-24T03:13:37.545 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10146 sec 2024-03-24T03:13:38.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:38 smithi078 ceph-mon[32429]: pgmap v5483: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:38.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:38 smithi107 ceph-mon[36485]: pgmap v5483: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:38.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:38 smithi106 ceph-mon[36505]: pgmap v5483: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:38.545 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10147 sec 2024-03-24T03:13:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9493.239323 secs 2024-03-24T03:13:39.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9493.239323 secs 2024-03-24T03:13:39.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9493.239323 secs 2024-03-24T03:13:39.545 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10148 sec 2024-03-24T03:13:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:40 smithi107 ceph-mon[36485]: pgmap v5484: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:40.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:40 smithi106 ceph-mon[36505]: pgmap v5484: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:40.545 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10149 sec 2024-03-24T03:13:40.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:40 smithi078 ceph-mon[32429]: pgmap v5484: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:41.545 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10150 sec 2024-03-24T03:13:42.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:42 smithi106 ceph-mon[36505]: pgmap v5485: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:42.545 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10151 sec 2024-03-24T03:13:42.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:42 smithi078 ceph-mon[32429]: pgmap v5485: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:42.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:42 smithi107 ceph-mon[36485]: pgmap v5485: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:43.545 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10152 sec 2024-03-24T03:13:44.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:44 smithi106 ceph-mon[36505]: pgmap v5486: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9498.239384 secs 2024-03-24T03:13:44.545 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10153 sec 2024-03-24T03:13:44.573 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:44 smithi078 ceph-mon[32429]: pgmap v5486: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:44.574 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9498.239384 secs 2024-03-24T03:13:44.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:44 smithi107 ceph-mon[36485]: pgmap v5486: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:44.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9498.239384 secs 2024-03-24T03:13:45.546 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10154 sec 2024-03-24T03:13:46.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:46 smithi106 ceph-mon[36505]: pgmap v5487: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:46.546 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10155 sec 2024-03-24T03:13:46.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:46 smithi078 ceph-mon[32429]: pgmap v5487: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:46.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:46 smithi107 ceph-mon[36485]: pgmap v5487: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:47.546 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10156 sec 2024-03-24T03:13:48.392 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:48 smithi078 ceph-mon[32429]: pgmap v5488: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:48.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:48 smithi106 ceph-mon[36505]: pgmap v5488: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:48.546 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10157 sec 2024-03-24T03:13:48.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:48 smithi107 ceph-mon[36485]: pgmap v5488: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9503.239446 secs 2024-03-24T03:13:49.546 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10158 sec 2024-03-24T03:13:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9503.239446 secs 2024-03-24T03:13:49.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9503.239446 secs 2024-03-24T03:13:50.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:50 smithi106 ceph-mon[36505]: pgmap v5489: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:50.546 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10159 sec 2024-03-24T03:13:50.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:50 smithi078 ceph-mon[32429]: pgmap v5489: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:50.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:50 smithi107 ceph-mon[36485]: pgmap v5489: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:51.546 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10160 sec 2024-03-24T03:13:52.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:52 smithi106 ceph-mon[36505]: pgmap v5490: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:52.546 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10161 sec 2024-03-24T03:13:52.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:52 smithi078 ceph-mon[32429]: pgmap v5490: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:52.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:52 smithi107 ceph-mon[36485]: pgmap v5490: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:53.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:53 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:13:53.546 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10162 sec 2024-03-24T03:13:53.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:53 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:13:53.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:53 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:13:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:54 smithi106 ceph-mon[36505]: pgmap v5491: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9508.239510 secs 2024-03-24T03:13:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:54 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:13:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:54 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:13:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:54 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:13:54.546 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10163 sec 2024-03-24T03:13:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:54 smithi078 ceph-mon[32429]: pgmap v5491: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9508.239510 secs 2024-03-24T03:13:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:54 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:13:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:54 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:13:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:54 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:13:54.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:54 smithi107 ceph-mon[36485]: pgmap v5491: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:54.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9508.239510 secs 2024-03-24T03:13:54.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:54 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:13:54.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:54 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:13:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:54 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:13:55.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:55 smithi106 ceph-mon[36505]: pgmap v5492: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:13:55.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:55 smithi106 ceph-mon[36505]: pgmap v5493: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:13:55.546 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10164 sec 2024-03-24T03:13:55.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:55 smithi078 ceph-mon[32429]: pgmap v5492: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:13:55.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:55 smithi078 ceph-mon[32429]: pgmap v5493: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:13:55.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:55 smithi107 ceph-mon[36485]: pgmap v5492: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:13:55.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:55 smithi107 ceph-mon[36485]: pgmap v5493: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:13:56.546 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10165 sec 2024-03-24T03:13:56.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:56 smithi078 ceph-mon[32429]: pgmap v5494: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:56.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:56 smithi107 ceph-mon[36485]: pgmap v5494: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:56.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:56 smithi106 ceph-mon[36505]: pgmap v5494: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:13:57.547 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10166 sec 2024-03-24T03:13:58.547 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10167 sec 2024-03-24T03:13:58.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:58 smithi078 ceph-mon[32429]: pgmap v5495: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:13:58.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:58 smithi107 ceph-mon[36485]: pgmap v5495: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:13:58.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:58 smithi106 ceph-mon[36505]: pgmap v5495: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:13:59.547 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10168 sec 2024-03-24T03:13:59.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:13:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9513.239590 secs 2024-03-24T03:13:59.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:13:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9513.239590 secs 2024-03-24T03:13:59.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:13:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9513.239590 secs 2024-03-24T03:14:00.547 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10169 sec 2024-03-24T03:14:00.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:00 smithi078 ceph-mon[32429]: pgmap v5496: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:14:00.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:00 smithi107 ceph-mon[36485]: pgmap v5496: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:14:00.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:00 smithi106 ceph-mon[36505]: pgmap v5496: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:14:01.547 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10170 sec 2024-03-24T03:14:02.547 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10171 sec 2024-03-24T03:14:02.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:02 smithi078 ceph-mon[32429]: pgmap v5497: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:14:02.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:02 smithi107 ceph-mon[36485]: pgmap v5497: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:14:02.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:02 smithi106 ceph-mon[36505]: pgmap v5497: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:14:03.547 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10172 sec 2024-03-24T03:14:04.547 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10173 sec 2024-03-24T03:14:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9518.239626 secs 2024-03-24T03:14:04.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:04 smithi078 ceph-mon[32429]: pgmap v5498: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:14:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9518.239626 secs 2024-03-24T03:14:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:04 smithi107 ceph-mon[36485]: pgmap v5498: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:14:04.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9518.239626 secs 2024-03-24T03:14:04.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:04 smithi106 ceph-mon[36505]: pgmap v5498: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:14:05.547 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10174 sec 2024-03-24T03:14:06.547 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10175 sec 2024-03-24T03:14:06.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:06 smithi078 ceph-mon[32429]: pgmap v5499: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:14:06.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:06 smithi107 ceph-mon[36485]: pgmap v5499: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:14:06.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:06 smithi106 ceph-mon[36505]: pgmap v5499: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:14:07.547 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10176 sec 2024-03-24T03:14:08.548 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10177 sec 2024-03-24T03:14:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:08 smithi078 ceph-mon[32429]: pgmap v5500: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:14:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:08 smithi107 ceph-mon[36485]: pgmap v5500: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:14:08.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:08 smithi106 ceph-mon[36505]: pgmap v5500: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:14:09.548 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10178 sec 2024-03-24T03:14:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9523.239663 secs 2024-03-24T03:14:09.882 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9523.239663 secs 2024-03-24T03:14:09.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9523.239663 secs 2024-03-24T03:14:10.548 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10179 sec 2024-03-24T03:14:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:10 smithi078 ceph-mon[32429]: pgmap v5501: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:14:10.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:10 smithi107 ceph-mon[36485]: pgmap v5501: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:14:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:10 smithi106 ceph-mon[36505]: pgmap v5501: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:14:11.548 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10180 sec 2024-03-24T03:14:12.548 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10181 sec 2024-03-24T03:14:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:12 smithi078 ceph-mon[32429]: pgmap v5502: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:14:12.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:12 smithi107 ceph-mon[36485]: pgmap v5502: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:14:12.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:12 smithi106 ceph-mon[36505]: pgmap v5502: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:14:13.548 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10182 sec 2024-03-24T03:14:14.548 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10183 sec 2024-03-24T03:14:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9528.239721 secs 2024-03-24T03:14:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:14 smithi078 ceph-mon[32429]: pgmap v5503: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:14:14.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9528.239721 secs 2024-03-24T03:14:14.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:14 smithi107 ceph-mon[36485]: pgmap v5503: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:14:14.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9528.239721 secs 2024-03-24T03:14:14.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:14 smithi106 ceph-mon[36505]: pgmap v5503: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:14:15.548 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10184 sec 2024-03-24T03:14:16.548 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10185 sec 2024-03-24T03:14:16.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:16 smithi078 ceph-mon[32429]: pgmap v5504: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:14:16.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:16 smithi107 ceph-mon[36485]: pgmap v5504: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:14:16.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:16 smithi106 ceph-mon[36505]: pgmap v5504: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:14:17.548 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10186 sec 2024-03-24T03:14:18.548 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10187 sec 2024-03-24T03:14:18.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:18 smithi078 ceph-mon[32429]: pgmap v5505: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:14:18.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:18 smithi107 ceph-mon[36485]: pgmap v5505: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:14:18.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:18 smithi106 ceph-mon[36505]: pgmap v5505: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:14:19.548 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10188 sec 2024-03-24T03:14:19.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9533.239763 secs 2024-03-24T03:14:19.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9533.239763 secs 2024-03-24T03:14:19.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9533.239763 secs 2024-03-24T03:14:20.549 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10189 sec 2024-03-24T03:14:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:20 smithi078 ceph-mon[32429]: pgmap v5506: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:14:20.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:20 smithi107 ceph-mon[36485]: pgmap v5506: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:14:20.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:20 smithi106 ceph-mon[36505]: pgmap v5506: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:14:21.549 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10190 sec 2024-03-24T03:14:22.549 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10191 sec 2024-03-24T03:14:22.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:22 smithi078 ceph-mon[32429]: pgmap v5507: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:14:22.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:22 smithi107 ceph-mon[36485]: pgmap v5507: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:14:22.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:22 smithi106 ceph-mon[36505]: pgmap v5507: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:14:23.549 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10192 sec 2024-03-24T03:14:24.549 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10193 sec 2024-03-24T03:14:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9538.239817 secs 2024-03-24T03:14:24.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:24 smithi078 ceph-mon[32429]: pgmap v5508: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:14:24.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9538.239817 secs 2024-03-24T03:14:24.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:24 smithi107 ceph-mon[36485]: pgmap v5508: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:14:24.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9538.239817 secs 2024-03-24T03:14:24.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:24 smithi106 ceph-mon[36505]: pgmap v5508: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:14:25.549 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10194 sec 2024-03-24T03:14:26.549 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10195 sec 2024-03-24T03:14:26.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:26 smithi078 ceph-mon[32429]: pgmap v5509: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:14:26.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:26 smithi107 ceph-mon[36485]: pgmap v5509: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:14:26.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:26 smithi106 ceph-mon[36505]: pgmap v5509: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:14:27.549 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10196 sec 2024-03-24T03:14:28.549 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10197 sec 2024-03-24T03:14:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:28 smithi078 ceph-mon[32429]: pgmap v5510: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:14:28.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:28 smithi107 ceph-mon[36485]: pgmap v5510: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:14:28.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:28 smithi106 ceph-mon[36505]: pgmap v5510: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:14:29.549 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10198 sec 2024-03-24T03:14:29.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9543.239877 secs 2024-03-24T03:14:29.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9543.239877 secs 2024-03-24T03:14:29.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9543.239877 secs 2024-03-24T03:14:30.549 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10199 sec 2024-03-24T03:14:30.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:30 smithi078 ceph-mon[32429]: pgmap v5511: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:14:30.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:30 smithi107 ceph-mon[36485]: pgmap v5511: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:14:30.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:30 smithi106 ceph-mon[36505]: pgmap v5511: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:14:31.550 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10200 sec 2024-03-24T03:14:32.550 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10201 sec 2024-03-24T03:14:32.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:32 smithi078 ceph-mon[32429]: pgmap v5512: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:14:32.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:32 smithi107 ceph-mon[36485]: pgmap v5512: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:14:32.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:32 smithi106 ceph-mon[36505]: pgmap v5512: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:14:33.550 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10202 sec 2024-03-24T03:14:34.550 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10203 sec 2024-03-24T03:14:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9548.239963 secs 2024-03-24T03:14:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:34 smithi078 ceph-mon[32429]: pgmap v5513: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:14:34.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9548.239963 secs 2024-03-24T03:14:34.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:34 smithi107 ceph-mon[36485]: pgmap v5513: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:14:34.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9548.239963 secs 2024-03-24T03:14:34.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:34 smithi106 ceph-mon[36505]: pgmap v5513: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:14:35.550 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10204 sec 2024-03-24T03:14:36.550 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10205 sec 2024-03-24T03:14:36.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:36 smithi078 ceph-mon[32429]: pgmap v5514: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:14:36.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:36 smithi107 ceph-mon[36485]: pgmap v5514: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:14:36.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:36 smithi106 ceph-mon[36505]: pgmap v5514: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:14:37.550 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10206 sec 2024-03-24T03:14:38.550 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10207 sec 2024-03-24T03:14:38.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:38 smithi078 ceph-mon[32429]: pgmap v5515: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:14:38.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:38 smithi107 ceph-mon[36485]: pgmap v5515: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:14:38.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:38 smithi106 ceph-mon[36505]: pgmap v5515: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:14:39.550 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10208 sec 2024-03-24T03:14:39.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9553.240053 secs 2024-03-24T03:14:39.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9553.240053 secs 2024-03-24T03:14:39.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9553.240053 secs 2024-03-24T03:14:40.550 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10209 sec 2024-03-24T03:14:40.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:40 smithi078 ceph-mon[32429]: pgmap v5516: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:14:40.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:40 smithi107 ceph-mon[36485]: pgmap v5516: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:14:40.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:40 smithi106 ceph-mon[36505]: pgmap v5516: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:14:41.550 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10210 sec 2024-03-24T03:14:42.550 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10211 sec 2024-03-24T03:14:42.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:42 smithi078 ceph-mon[32429]: pgmap v5517: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:14:42.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:42 smithi107 ceph-mon[36485]: pgmap v5517: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:14:42.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:42 smithi106 ceph-mon[36505]: pgmap v5517: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:14:43.551 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10212 sec 2024-03-24T03:14:44.551 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10213 sec 2024-03-24T03:14:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9558.240142 secs 2024-03-24T03:14:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:44 smithi078 ceph-mon[32429]: pgmap v5518: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:14:44.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9558.240142 secs 2024-03-24T03:14:44.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:44 smithi107 ceph-mon[36485]: pgmap v5518: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:14:44.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9558.240142 secs 2024-03-24T03:14:44.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:44 smithi106 ceph-mon[36505]: pgmap v5518: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:14:45.551 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10214 sec 2024-03-24T03:14:46.551 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10215 sec 2024-03-24T03:14:46.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:46 smithi078 ceph-mon[32429]: pgmap v5519: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:14:46.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:46 smithi107 ceph-mon[36485]: pgmap v5519: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:14:46.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:46 smithi106 ceph-mon[36505]: pgmap v5519: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:14:47.551 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10216 sec 2024-03-24T03:14:48.551 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10217 sec 2024-03-24T03:14:48.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:48 smithi078 ceph-mon[32429]: pgmap v5520: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:14:48.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:48 smithi107 ceph-mon[36485]: pgmap v5520: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:14:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:48 smithi106 ceph-mon[36505]: pgmap v5520: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:14:49.551 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10218 sec 2024-03-24T03:14:49.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9563.240182 secs 2024-03-24T03:14:49.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9563.240182 secs 2024-03-24T03:14:49.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9563.240182 secs 2024-03-24T03:14:50.551 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10219 sec 2024-03-24T03:14:50.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:50 smithi078 ceph-mon[32429]: pgmap v5521: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:14:50.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:50 smithi107 ceph-mon[36485]: pgmap v5521: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:14:50.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:50 smithi106 ceph-mon[36505]: pgmap v5521: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:14:51.551 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10220 sec 2024-03-24T03:14:52.551 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10221 sec 2024-03-24T03:14:52.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:52 smithi106 ceph-mon[36505]: pgmap v5522: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:14:53.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:52 smithi078 ceph-mon[32429]: pgmap v5522: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:14:53.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:52 smithi107 ceph-mon[36485]: pgmap v5522: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:14:53.551 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10222 sec 2024-03-24T03:14:53.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:53 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:14:54.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:53 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:14:54.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:53 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:14:54.552 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10223 sec 2024-03-24T03:14:54.901 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9568.240274 secs 2024-03-24T03:14:54.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:54 smithi106 ceph-mon[36505]: pgmap v5523: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:14:55.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9568.240274 secs 2024-03-24T03:14:55.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:54 smithi078 ceph-mon[32429]: pgmap v5523: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:14:55.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9568.240274 secs 2024-03-24T03:14:55.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:54 smithi107 ceph-mon[36485]: pgmap v5523: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:14:55.551 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10224 sec 2024-03-24T03:14:56.552 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10225 sec 2024-03-24T03:14:57.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:56 smithi078 ceph-mon[32429]: pgmap v5524: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:14:57.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:14:57.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:14:57.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:14:57.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:14:57.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:14:57.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:14:57.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:14:57.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:14:57.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:14:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:56 smithi107 ceph-mon[36485]: pgmap v5524: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:14:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:14:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:14:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:14:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:14:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:14:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:14:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:14:57.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:14:57.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:14:57.150 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:56 smithi106 ceph-mon[36505]: pgmap v5524: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:14:57.150 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:14:57.150 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:14:57.150 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:14:57.150 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:14:57.150 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:14:57.150 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:14:57.150 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.150 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:14:57.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:14:57.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:14:57.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:14:57.552 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10226 sec 2024-03-24T03:14:58.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:57 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T03:14:58.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:57 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T03:14:58.093 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:57 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T03:14:58.094 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:57 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T03:14:58.094 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:57 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T03:14:58.094 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:57 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T03:14:58.094 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:57 smithi078 ceph-mon[32429]: pgmap v5525: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:14:58.094 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:57 smithi078 ceph-mon[32429]: pgmap v5526: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:14:58.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:57 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T03:14:58.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:57 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T03:14:58.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:57 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T03:14:58.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:57 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T03:14:58.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:57 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T03:14:58.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:57 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T03:14:58.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:57 smithi107 ceph-mon[36485]: pgmap v5525: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:14:58.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:57 smithi107 ceph-mon[36485]: pgmap v5526: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:14:58.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:57 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T03:14:58.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:57 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T03:14:58.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:57 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T03:14:58.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:57 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T03:14:58.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:57 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T03:14:58.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:57 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T03:14:58.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:57 smithi106 ceph-mon[36505]: pgmap v5525: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:14:58.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:57 smithi106 ceph-mon[36505]: pgmap v5526: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:14:58.552 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10227 sec 2024-03-24T03:14:59.552 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10228 sec 2024-03-24T03:15:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9573.240336 secs 2024-03-24T03:15:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:14:59 smithi078 ceph-mon[32429]: pgmap v5527: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:15:00.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9573.240336 secs 2024-03-24T03:15:00.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:14:59 smithi107 ceph-mon[36485]: pgmap v5527: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:15:00.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9573.240336 secs 2024-03-24T03:15:00.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:14:59 smithi106 ceph-mon[36505]: pgmap v5527: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:15:00.552 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10229 sec 2024-03-24T03:15:01.552 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10230 sec 2024-03-24T03:15:02.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:01 smithi078 ceph-mon[32429]: pgmap v5528: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:15:02.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:01 smithi107 ceph-mon[36485]: pgmap v5528: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:15:02.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:01 smithi106 ceph-mon[36505]: pgmap v5528: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:15:02.552 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10231 sec 2024-03-24T03:15:03.552 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10232 sec 2024-03-24T03:15:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:03 smithi078 ceph-mon[32429]: pgmap v5529: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:15:04.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:03 smithi107 ceph-mon[36485]: pgmap v5529: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:15:04.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:03 smithi106 ceph-mon[36505]: pgmap v5529: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:15:04.552 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10233 sec 2024-03-24T03:15:05.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9578.240395 secs 2024-03-24T03:15:05.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9578.240395 secs 2024-03-24T03:15:05.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9578.240395 secs 2024-03-24T03:15:05.552 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10234 sec 2024-03-24T03:15:06.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:05 smithi078 ceph-mon[32429]: pgmap v5530: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:06.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:05 smithi107 ceph-mon[36485]: pgmap v5530: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:06.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:05 smithi106 ceph-mon[36505]: pgmap v5530: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:06.553 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10235 sec 2024-03-24T03:15:07.553 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10236 sec 2024-03-24T03:15:08.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:07 smithi078 ceph-mon[32429]: pgmap v5531: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:15:08.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:07 smithi107 ceph-mon[36485]: pgmap v5531: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:15:08.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:07 smithi106 ceph-mon[36505]: pgmap v5531: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:15:08.553 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10237 sec 2024-03-24T03:15:09.553 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10238 sec 2024-03-24T03:15:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9583.240455 secs 2024-03-24T03:15:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:09 smithi078 ceph-mon[32429]: pgmap v5532: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:10.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9583.240455 secs 2024-03-24T03:15:10.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:09 smithi107 ceph-mon[36485]: pgmap v5532: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:10.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9583.240455 secs 2024-03-24T03:15:10.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:09 smithi106 ceph-mon[36505]: pgmap v5532: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:10.553 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10239 sec 2024-03-24T03:15:11.553 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10240 sec 2024-03-24T03:15:12.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:11 smithi078 ceph-mon[32429]: pgmap v5533: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:12.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:11 smithi107 ceph-mon[36485]: pgmap v5533: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:12.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:11 smithi106 ceph-mon[36505]: pgmap v5533: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:12.553 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10241 sec 2024-03-24T03:15:13.553 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10242 sec 2024-03-24T03:15:14.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:13 smithi078 ceph-mon[32429]: pgmap v5534: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:14.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:13 smithi107 ceph-mon[36485]: pgmap v5534: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:14.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:13 smithi106 ceph-mon[36505]: pgmap v5534: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:14.553 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10243 sec 2024-03-24T03:15:15.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9588.240513 secs 2024-03-24T03:15:15.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9588.240513 secs 2024-03-24T03:15:15.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9588.240513 secs 2024-03-24T03:15:15.553 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10244 sec 2024-03-24T03:15:16.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:15 smithi078 ceph-mon[32429]: pgmap v5535: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:16.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:15 smithi107 ceph-mon[36485]: pgmap v5535: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:16.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:15 smithi106 ceph-mon[36505]: pgmap v5535: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:16.553 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10245 sec 2024-03-24T03:15:17.553 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10246 sec 2024-03-24T03:15:18.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:17 smithi078 ceph-mon[32429]: pgmap v5536: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:18.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:17 smithi107 ceph-mon[36485]: pgmap v5536: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:18.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:17 smithi106 ceph-mon[36505]: pgmap v5536: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:18.553 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10247 sec 2024-03-24T03:15:19.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9593.240600 secs 2024-03-24T03:15:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9593.240600 secs 2024-03-24T03:15:19.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9593.240600 secs 2024-03-24T03:15:19.553 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10248 sec 2024-03-24T03:15:20.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:19 smithi106 ceph-mon[36505]: pgmap v5537: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:20.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:19 smithi078 ceph-mon[32429]: pgmap v5537: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:20.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:19 smithi107 ceph-mon[36485]: pgmap v5537: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:20.554 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10249 sec 2024-03-24T03:15:21.554 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10250 sec 2024-03-24T03:15:22.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:21 smithi106 ceph-mon[36505]: pgmap v5538: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:22.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:21 smithi078 ceph-mon[32429]: pgmap v5538: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:22.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:21 smithi107 ceph-mon[36485]: pgmap v5538: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:22.554 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10251 sec 2024-03-24T03:15:23.554 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10252 sec 2024-03-24T03:15:24.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:23 smithi106 ceph-mon[36505]: pgmap v5539: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9598.240688 secs 2024-03-24T03:15:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:23 smithi078 ceph-mon[32429]: pgmap v5539: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9598.240688 secs 2024-03-24T03:15:24.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:23 smithi107 ceph-mon[36485]: pgmap v5539: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9598.240688 secs 2024-03-24T03:15:24.554 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10253 sec 2024-03-24T03:15:25.554 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10254 sec 2024-03-24T03:15:26.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:25 smithi106 ceph-mon[36505]: pgmap v5540: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:26.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:25 smithi078 ceph-mon[32429]: pgmap v5540: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:26.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:25 smithi107 ceph-mon[36485]: pgmap v5540: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:26.554 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10255 sec 2024-03-24T03:15:27.554 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10256 sec 2024-03-24T03:15:28.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:27 smithi106 ceph-mon[36505]: pgmap v5541: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:28.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:27 smithi078 ceph-mon[32429]: pgmap v5541: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:28.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:27 smithi107 ceph-mon[36485]: pgmap v5541: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:28.554 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10257 sec 2024-03-24T03:15:29.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9603.240779 secs 2024-03-24T03:15:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9603.240779 secs 2024-03-24T03:15:29.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9603.240779 secs 2024-03-24T03:15:29.554 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10258 sec 2024-03-24T03:15:30.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:29 smithi106 ceph-mon[36505]: pgmap v5542: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:30.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:29 smithi078 ceph-mon[32429]: pgmap v5542: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:30.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:29 smithi107 ceph-mon[36485]: pgmap v5542: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:30.554 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10259 sec 2024-03-24T03:15:31.555 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10260 sec 2024-03-24T03:15:32.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:31 smithi106 ceph-mon[36505]: pgmap v5543: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:32.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:31 smithi078 ceph-mon[32429]: pgmap v5543: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:32.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:31 smithi107 ceph-mon[36485]: pgmap v5543: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:32.555 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10261 sec 2024-03-24T03:15:33.555 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10262 sec 2024-03-24T03:15:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:33 smithi078 ceph-mon[32429]: pgmap v5544: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:34.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9608.240869 secs 2024-03-24T03:15:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:33 smithi106 ceph-mon[36505]: pgmap v5544: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9608.240869 secs 2024-03-24T03:15:34.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:33 smithi107 ceph-mon[36485]: pgmap v5544: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9608.240869 secs 2024-03-24T03:15:34.555 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10263 sec 2024-03-24T03:15:35.555 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10264 sec 2024-03-24T03:15:36.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:35 smithi106 ceph-mon[36505]: pgmap v5545: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:36.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:35 smithi078 ceph-mon[32429]: pgmap v5545: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:36.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:35 smithi107 ceph-mon[36485]: pgmap v5545: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:36.555 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10265 sec 2024-03-24T03:15:37.555 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10266 sec 2024-03-24T03:15:38.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:37 smithi106 ceph-mon[36505]: pgmap v5546: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:38.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:37 smithi078 ceph-mon[32429]: pgmap v5546: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:38.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:37 smithi107 ceph-mon[36485]: pgmap v5546: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:38.555 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10267 sec 2024-03-24T03:15:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9613.240956 secs 2024-03-24T03:15:39.243 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9613.240956 secs 2024-03-24T03:15:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9613.240956 secs 2024-03-24T03:15:39.558 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10268 sec 2024-03-24T03:15:40.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:39 smithi106 ceph-mon[36505]: pgmap v5547: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:40.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:39 smithi078 ceph-mon[32429]: pgmap v5547: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:40.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:39 smithi107 ceph-mon[36485]: pgmap v5547: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:40.555 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10269 sec 2024-03-24T03:15:41.556 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10270 sec 2024-03-24T03:15:42.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:41 smithi106 ceph-mon[36505]: pgmap v5548: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:42.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:41 smithi078 ceph-mon[32429]: pgmap v5548: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:42.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:41 smithi107 ceph-mon[36485]: pgmap v5548: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:42.556 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10271 sec 2024-03-24T03:15:43.556 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10272 sec 2024-03-24T03:15:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:43 smithi106 ceph-mon[36505]: pgmap v5549: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9618.241040 secs 2024-03-24T03:15:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:43 smithi078 ceph-mon[32429]: pgmap v5549: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9618.241040 secs 2024-03-24T03:15:44.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:43 smithi107 ceph-mon[36485]: pgmap v5549: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:44.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9618.241040 secs 2024-03-24T03:15:44.556 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10273 sec 2024-03-24T03:15:45.556 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10274 sec 2024-03-24T03:15:46.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:45 smithi106 ceph-mon[36505]: pgmap v5550: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:46.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:45 smithi078 ceph-mon[32429]: pgmap v5550: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:46.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:45 smithi107 ceph-mon[36485]: pgmap v5550: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:46.556 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10275 sec 2024-03-24T03:15:47.556 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10276 sec 2024-03-24T03:15:48.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:47 smithi106 ceph-mon[36505]: pgmap v5551: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:48.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:47 smithi078 ceph-mon[32429]: pgmap v5551: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:48.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:47 smithi107 ceph-mon[36485]: pgmap v5551: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:48.556 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10277 sec 2024-03-24T03:15:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9623.241102 secs 2024-03-24T03:15:49.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9623.241102 secs 2024-03-24T03:15:49.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9623.241102 secs 2024-03-24T03:15:49.556 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10278 sec 2024-03-24T03:15:50.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:49 smithi078 ceph-mon[32429]: pgmap v5552: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:50.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:49 smithi107 ceph-mon[36485]: pgmap v5552: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:50.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:49 smithi106 ceph-mon[36505]: pgmap v5552: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:50.556 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10279 sec 2024-03-24T03:15:51.556 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10280 sec 2024-03-24T03:15:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:51 smithi078 ceph-mon[32429]: pgmap v5553: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:52.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:51 smithi107 ceph-mon[36485]: pgmap v5553: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:52.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:51 smithi106 ceph-mon[36505]: pgmap v5553: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:52.556 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10281 sec 2024-03-24T03:15:53.556 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10282 sec 2024-03-24T03:15:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:53 smithi078 ceph-mon[32429]: pgmap v5554: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9628.241187 secs 2024-03-24T03:15:54.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:53 smithi107 ceph-mon[36485]: pgmap v5554: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:54.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9628.241187 secs 2024-03-24T03:15:54.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:53 smithi106 ceph-mon[36505]: pgmap v5554: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:54.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9628.241187 secs 2024-03-24T03:15:54.556 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10283 sec 2024-03-24T03:15:55.556 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10284 sec 2024-03-24T03:15:56.203 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:55 smithi106 ceph-mon[36505]: pgmap v5555: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:56.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:55 smithi078 ceph-mon[32429]: pgmap v5555: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:56.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:55 smithi107 ceph-mon[36485]: pgmap v5555: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:15:56.556 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10285 sec 2024-03-24T03:15:57.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:56 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:15:57.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:56 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:15:57.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:56 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:15:57.556 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10286 sec 2024-03-24T03:15:58.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:58 smithi078 ceph-mon[32429]: pgmap v5556: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:58.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:58 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:15:58.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:58 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:15:58.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:58 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:15:58.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:58 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:15:58.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:58 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:15:58.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:58 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:15:58.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:58 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:15:58.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:58 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:15:58.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:58 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:15:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:58 smithi107 ceph-mon[36485]: pgmap v5556: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:58 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:15:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:58 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:15:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:58 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:15:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:58 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:15:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:58 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:15:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:58 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:15:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:58 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:15:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:58 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:15:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:58 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:15:58.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:58 smithi106 ceph-mon[36505]: pgmap v5556: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:58.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:58 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:15:58.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:58 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:15:58.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:58 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:15:58.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:58 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:15:58.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:58 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:15:58.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:58 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:15:58.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:58 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:15:58.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:58 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:15:58.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:58 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:15:58.557 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10287 sec 2024-03-24T03:15:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:59 smithi078 ceph-mon[32429]: pgmap v5557: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T03:15:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:59 smithi078 ceph-mon[32429]: pgmap v5558: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:15:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:15:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9633.241247 secs 2024-03-24T03:15:59.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:59 smithi107 ceph-mon[36485]: pgmap v5557: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T03:15:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:59 smithi107 ceph-mon[36485]: pgmap v5558: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:15:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:15:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9633.241247 secs 2024-03-24T03:15:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:59 smithi106 ceph-mon[36505]: pgmap v5557: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T03:15:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:59 smithi106 ceph-mon[36505]: pgmap v5558: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:15:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:15:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9633.241247 secs 2024-03-24T03:15:59.557 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10288 sec 2024-03-24T03:16:00.557 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10289 sec 2024-03-24T03:16:01.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:01 smithi078 ceph-mon[32429]: pgmap v5559: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:16:01.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:01 smithi107 ceph-mon[36485]: pgmap v5559: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:16:01.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:01 smithi106 ceph-mon[36505]: pgmap v5559: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:16:01.557 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10290 sec 2024-03-24T03:16:02.557 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10291 sec 2024-03-24T03:16:03.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:03 smithi078 ceph-mon[32429]: pgmap v5560: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:16:03.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:03 smithi107 ceph-mon[36485]: pgmap v5560: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:16:03.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:03 smithi106 ceph-mon[36505]: pgmap v5560: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:16:03.557 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10292 sec 2024-03-24T03:16:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9638.241319 secs 2024-03-24T03:16:04.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9638.241319 secs 2024-03-24T03:16:04.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9638.241319 secs 2024-03-24T03:16:04.557 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10293 sec 2024-03-24T03:16:05.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:05 smithi078 ceph-mon[32429]: pgmap v5561: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:16:05.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:05 smithi107 ceph-mon[36485]: pgmap v5561: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:16:05.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:05 smithi106 ceph-mon[36505]: pgmap v5561: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:16:05.557 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10294 sec 2024-03-24T03:16:06.557 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10295 sec 2024-03-24T03:16:07.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:07 smithi107 ceph-mon[36485]: pgmap v5562: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:16:07.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:07 smithi106 ceph-mon[36505]: pgmap v5562: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:16:07.557 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10296 sec 2024-03-24T03:16:07.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:07 smithi078 ceph-mon[32429]: pgmap v5562: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:16:08.557 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10297 sec 2024-03-24T03:16:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:09 smithi106 ceph-mon[36505]: pgmap v5563: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:16:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9643.241378 secs 2024-03-24T03:16:09.558 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10298 sec 2024-03-24T03:16:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:09 smithi078 ceph-mon[32429]: pgmap v5563: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:16:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9643.241378 secs 2024-03-24T03:16:09.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:09 smithi107 ceph-mon[36485]: pgmap v5563: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:16:09.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9643.241378 secs 2024-03-24T03:16:10.558 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10299 sec 2024-03-24T03:16:11.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:11 smithi106 ceph-mon[36505]: pgmap v5564: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:11.558 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10300 sec 2024-03-24T03:16:11.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:11 smithi078 ceph-mon[32429]: pgmap v5564: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:11.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:11 smithi107 ceph-mon[36485]: pgmap v5564: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:12.558 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10301 sec 2024-03-24T03:16:13.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:13 smithi106 ceph-mon[36505]: pgmap v5565: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:13.558 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10302 sec 2024-03-24T03:16:13.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:13 smithi078 ceph-mon[32429]: pgmap v5565: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:13.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:13 smithi107 ceph-mon[36485]: pgmap v5565: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9648.241456 secs 2024-03-24T03:16:14.558 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10303 sec 2024-03-24T03:16:14.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9648.241456 secs 2024-03-24T03:16:14.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9648.241456 secs 2024-03-24T03:16:15.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:15 smithi106 ceph-mon[36505]: pgmap v5566: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:15.558 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10304 sec 2024-03-24T03:16:15.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:15 smithi078 ceph-mon[32429]: pgmap v5566: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:15.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:15 smithi107 ceph-mon[36485]: pgmap v5566: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:16.558 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10305 sec 2024-03-24T03:16:17.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:17 smithi106 ceph-mon[36505]: pgmap v5567: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:17.558 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10306 sec 2024-03-24T03:16:17.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:17 smithi078 ceph-mon[32429]: pgmap v5567: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:17.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:17 smithi107 ceph-mon[36485]: pgmap v5567: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:18.558 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10307 sec 2024-03-24T03:16:19.392 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:19 smithi078 ceph-mon[32429]: pgmap v5568: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:19.392 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9653.241540 secs 2024-03-24T03:16:19.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:19 smithi106 ceph-mon[36505]: pgmap v5568: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9653.241540 secs 2024-03-24T03:16:19.558 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10308 sec 2024-03-24T03:16:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:19 smithi107 ceph-mon[36485]: pgmap v5568: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9653.241540 secs 2024-03-24T03:16:20.558 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10309 sec 2024-03-24T03:16:21.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:21 smithi106 ceph-mon[36505]: pgmap v5569: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:21.559 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10310 sec 2024-03-24T03:16:21.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:21 smithi078 ceph-mon[32429]: pgmap v5569: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:21.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:21 smithi107 ceph-mon[36485]: pgmap v5569: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:22.559 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10311 sec 2024-03-24T03:16:23.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:23 smithi106 ceph-mon[36505]: pgmap v5570: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:23.559 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10312 sec 2024-03-24T03:16:23.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:23 smithi078 ceph-mon[32429]: pgmap v5570: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:23.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:23 smithi107 ceph-mon[36485]: pgmap v5570: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:24.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9658.241597 secs 2024-03-24T03:16:24.559 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10313 sec 2024-03-24T03:16:24.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9658.241597 secs 2024-03-24T03:16:24.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9658.241597 secs 2024-03-24T03:16:25.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:25 smithi106 ceph-mon[36505]: pgmap v5571: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:25.559 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10314 sec 2024-03-24T03:16:25.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:25 smithi078 ceph-mon[32429]: pgmap v5571: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:25.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:25 smithi107 ceph-mon[36485]: pgmap v5571: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:26.561 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10315 sec 2024-03-24T03:16:27.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:27 smithi106 ceph-mon[36505]: pgmap v5572: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:27.559 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10316 sec 2024-03-24T03:16:27.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:27 smithi078 ceph-mon[32429]: pgmap v5572: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:27.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:27 smithi107 ceph-mon[36485]: pgmap v5572: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:28.559 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10317 sec 2024-03-24T03:16:29.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:29 smithi106 ceph-mon[36505]: pgmap v5573: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9663.241650 secs 2024-03-24T03:16:29.559 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10318 sec 2024-03-24T03:16:29.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:29 smithi078 ceph-mon[32429]: pgmap v5573: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:29.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9663.241650 secs 2024-03-24T03:16:29.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:29 smithi107 ceph-mon[36485]: pgmap v5573: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9663.241650 secs 2024-03-24T03:16:30.559 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10319 sec 2024-03-24T03:16:31.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:31 smithi106 ceph-mon[36505]: pgmap v5574: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:31.559 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10320 sec 2024-03-24T03:16:31.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:31 smithi078 ceph-mon[32429]: pgmap v5574: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:31.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:31 smithi107 ceph-mon[36485]: pgmap v5574: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:32.560 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10321 sec 2024-03-24T03:16:33.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:33 smithi106 ceph-mon[36505]: pgmap v5575: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:33.560 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10322 sec 2024-03-24T03:16:33.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:33 smithi078 ceph-mon[32429]: pgmap v5575: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:33.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:33 smithi107 ceph-mon[36485]: pgmap v5575: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9668.241736 secs 2024-03-24T03:16:34.560 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10323 sec 2024-03-24T03:16:34.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9668.241736 secs 2024-03-24T03:16:34.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9668.241736 secs 2024-03-24T03:16:35.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:35 smithi106 ceph-mon[36505]: pgmap v5576: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:35.560 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10324 sec 2024-03-24T03:16:35.579 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:35 smithi078 ceph-mon[32429]: pgmap v5576: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:35.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:35 smithi107 ceph-mon[36485]: pgmap v5576: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:36.560 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10325 sec 2024-03-24T03:16:36.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:36 smithi078 ceph-mon[32429]: pgmap v5577: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:36.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:36 smithi107 ceph-mon[36485]: pgmap v5577: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:36.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:36 smithi106 ceph-mon[36505]: pgmap v5577: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:37.560 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10326 sec 2024-03-24T03:16:38.560 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10327 sec 2024-03-24T03:16:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:38 smithi078 ceph-mon[32429]: pgmap v5578: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:39.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:38 smithi107 ceph-mon[36485]: pgmap v5578: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:39.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:38 smithi106 ceph-mon[36505]: pgmap v5578: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:39.560 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10328 sec 2024-03-24T03:16:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9673.241824 secs 2024-03-24T03:16:40.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9673.241824 secs 2024-03-24T03:16:40.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9673.241824 secs 2024-03-24T03:16:40.560 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10329 sec 2024-03-24T03:16:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:40 smithi078 ceph-mon[32429]: pgmap v5579: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:41.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:40 smithi107 ceph-mon[36485]: pgmap v5579: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:41.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:40 smithi106 ceph-mon[36505]: pgmap v5579: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:41.560 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10330 sec 2024-03-24T03:16:42.560 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10331 sec 2024-03-24T03:16:43.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:42 smithi078 ceph-mon[32429]: pgmap v5580: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:43.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:42 smithi107 ceph-mon[36485]: pgmap v5580: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:43.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:42 smithi106 ceph-mon[36505]: pgmap v5580: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:43.560 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10332 sec 2024-03-24T03:16:44.561 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10333 sec 2024-03-24T03:16:45.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9678.241883 secs 2024-03-24T03:16:45.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:44 smithi078 ceph-mon[32429]: pgmap v5581: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:45.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9678.241883 secs 2024-03-24T03:16:45.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:44 smithi107 ceph-mon[36485]: pgmap v5581: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:45.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9678.241883 secs 2024-03-24T03:16:45.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:44 smithi106 ceph-mon[36505]: pgmap v5581: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:45.561 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10334 sec 2024-03-24T03:16:46.561 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10335 sec 2024-03-24T03:16:47.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:46 smithi078 ceph-mon[32429]: pgmap v5582: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:47.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:46 smithi107 ceph-mon[36485]: pgmap v5582: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:47.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:46 smithi106 ceph-mon[36505]: pgmap v5582: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:47.561 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10336 sec 2024-03-24T03:16:48.561 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10337 sec 2024-03-24T03:16:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:48 smithi078 ceph-mon[32429]: pgmap v5583: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:49.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:48 smithi107 ceph-mon[36485]: pgmap v5583: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:49.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:48 smithi106 ceph-mon[36505]: pgmap v5583: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:49.561 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10338 sec 2024-03-24T03:16:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9683.241967 secs 2024-03-24T03:16:50.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9683.241967 secs 2024-03-24T03:16:50.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9683.241967 secs 2024-03-24T03:16:50.561 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10339 sec 2024-03-24T03:16:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:50 smithi078 ceph-mon[32429]: pgmap v5584: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:51.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:50 smithi107 ceph-mon[36485]: pgmap v5584: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:51.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:50 smithi106 ceph-mon[36505]: pgmap v5584: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:51.562 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10340 sec 2024-03-24T03:16:52.561 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10341 sec 2024-03-24T03:16:53.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:52 smithi078 ceph-mon[32429]: pgmap v5585: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:53.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:52 smithi107 ceph-mon[36485]: pgmap v5585: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:53.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:52 smithi106 ceph-mon[36505]: pgmap v5585: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:53.561 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10342 sec 2024-03-24T03:16:54.562 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10343 sec 2024-03-24T03:16:55.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9688.242028 secs 2024-03-24T03:16:55.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:54 smithi078 ceph-mon[32429]: pgmap v5586: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:55.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9688.242028 secs 2024-03-24T03:16:55.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:54 smithi107 ceph-mon[36485]: pgmap v5586: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:55.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9688.242028 secs 2024-03-24T03:16:55.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:54 smithi106 ceph-mon[36505]: pgmap v5586: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:55.562 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10344 sec 2024-03-24T03:16:56.562 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10345 sec 2024-03-24T03:16:57.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:56 smithi078 ceph-mon[32429]: pgmap v5587: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:57.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:56 smithi107 ceph-mon[36485]: pgmap v5587: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:57.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:56 smithi106 ceph-mon[36505]: pgmap v5587: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:57.562 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10346 sec 2024-03-24T03:16:58.562 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10347 sec 2024-03-24T03:16:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:58 smithi078 ceph-mon[32429]: pgmap v5588: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:58 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:16:59.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:58 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:16:59.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:58 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:16:59.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:58 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:16:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:58 smithi107 ceph-mon[36485]: pgmap v5588: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:58 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:16:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:58 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:16:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:58 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:16:59.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:58 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:16:59.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:58 smithi106 ceph-mon[36505]: pgmap v5588: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:16:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:58 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:16:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:58 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:16:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:58 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:16:59.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:58 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:16:59.562 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10348 sec 2024-03-24T03:17:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9693.242094 secs 2024-03-24T03:17:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:59 smithi078 ceph-mon[32429]: pgmap v5589: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:17:00.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:16:59 smithi078 ceph-mon[32429]: pgmap v5590: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:17:00.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9693.242094 secs 2024-03-24T03:17:00.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:59 smithi107 ceph-mon[36485]: pgmap v5589: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:17:00.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:16:59 smithi107 ceph-mon[36485]: pgmap v5590: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:17:00.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9693.242094 secs 2024-03-24T03:17:00.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:59 smithi106 ceph-mon[36505]: pgmap v5589: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:17:00.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:16:59 smithi106 ceph-mon[36505]: pgmap v5590: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:17:00.562 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10349 sec 2024-03-24T03:17:01.562 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10350 sec 2024-03-24T03:17:02.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:01 smithi078 ceph-mon[32429]: pgmap v5591: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:17:02.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:01 smithi107 ceph-mon[36485]: pgmap v5591: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:17:02.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:01 smithi106 ceph-mon[36505]: pgmap v5591: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:17:02.562 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10351 sec 2024-03-24T03:17:03.562 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10352 sec 2024-03-24T03:17:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:03 smithi078 ceph-mon[32429]: pgmap v5592: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:17:04.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:03 smithi107 ceph-mon[36485]: pgmap v5592: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:17:04.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:03 smithi106 ceph-mon[36505]: pgmap v5592: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:17:04.562 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10353 sec 2024-03-24T03:17:05.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9698.242153 secs 2024-03-24T03:17:05.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9698.242153 secs 2024-03-24T03:17:05.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9698.242153 secs 2024-03-24T03:17:05.562 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10354 sec 2024-03-24T03:17:06.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:05 smithi078 ceph-mon[32429]: pgmap v5593: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:17:06.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:05 smithi107 ceph-mon[36485]: pgmap v5593: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:17:06.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:05 smithi106 ceph-mon[36505]: pgmap v5593: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:17:06.563 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10355 sec 2024-03-24T03:17:07.563 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10356 sec 2024-03-24T03:17:08.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:07 smithi078 ceph-mon[32429]: pgmap v5594: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:17:08.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:07 smithi107 ceph-mon[36485]: pgmap v5594: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:17:08.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:07 smithi106 ceph-mon[36505]: pgmap v5594: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:17:08.563 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10357 sec 2024-03-24T03:17:09.563 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10358 sec 2024-03-24T03:17:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9703.242194 secs 2024-03-24T03:17:10.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:09 smithi078 ceph-mon[32429]: pgmap v5595: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:17:10.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9703.242194 secs 2024-03-24T03:17:10.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:09 smithi107 ceph-mon[36485]: pgmap v5595: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:17:10.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9703.242194 secs 2024-03-24T03:17:10.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:09 smithi106 ceph-mon[36505]: pgmap v5595: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:17:10.563 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10359 sec 2024-03-24T03:17:11.563 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10360 sec 2024-03-24T03:17:12.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:11 smithi107 ceph-mon[36485]: pgmap v5596: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:12.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:11 smithi106 ceph-mon[36505]: pgmap v5596: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:12.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:11 smithi078 ceph-mon[32429]: pgmap v5596: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:12.564 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10361 sec 2024-03-24T03:17:13.563 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10362 sec 2024-03-24T03:17:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:13 smithi106 ceph-mon[36505]: pgmap v5597: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:13 smithi078 ceph-mon[32429]: pgmap v5597: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:14.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:13 smithi107 ceph-mon[36485]: pgmap v5597: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:14.563 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10363 sec 2024-03-24T03:17:15.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9708.242289 secs 2024-03-24T03:17:15.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9708.242289 secs 2024-03-24T03:17:15.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9708.242289 secs 2024-03-24T03:17:15.563 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10364 sec 2024-03-24T03:17:16.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:15 smithi106 ceph-mon[36505]: pgmap v5598: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:16.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:15 smithi078 ceph-mon[32429]: pgmap v5598: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:16.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:15 smithi107 ceph-mon[36485]: pgmap v5598: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:16.563 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10365 sec 2024-03-24T03:17:17.563 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10366 sec 2024-03-24T03:17:18.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:17 smithi106 ceph-mon[36505]: pgmap v5599: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:17 smithi078 ceph-mon[32429]: pgmap v5599: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:18.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:17 smithi107 ceph-mon[36485]: pgmap v5599: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:18.563 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10367 sec 2024-03-24T03:17:19.151 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9713.242350 secs 2024-03-24T03:17:19.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9713.242350 secs 2024-03-24T03:17:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9713.242350 secs 2024-03-24T03:17:19.563 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10368 sec 2024-03-24T03:17:20.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:19 smithi106 ceph-mon[36505]: pgmap v5600: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:20.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:19 smithi078 ceph-mon[32429]: pgmap v5600: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:20.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:19 smithi107 ceph-mon[36485]: pgmap v5600: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:20.563 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10369 sec 2024-03-24T03:17:21.564 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10370 sec 2024-03-24T03:17:22.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:21 smithi106 ceph-mon[36505]: pgmap v5601: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:22.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:21 smithi078 ceph-mon[32429]: pgmap v5601: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:22.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:21 smithi107 ceph-mon[36485]: pgmap v5601: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:22.564 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10371 sec 2024-03-24T03:17:23.564 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10372 sec 2024-03-24T03:17:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:23 smithi106 ceph-mon[36505]: pgmap v5602: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9718.242430 secs 2024-03-24T03:17:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:23 smithi078 ceph-mon[32429]: pgmap v5602: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9718.242430 secs 2024-03-24T03:17:24.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:23 smithi107 ceph-mon[36485]: pgmap v5602: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9718.242430 secs 2024-03-24T03:17:24.564 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10373 sec 2024-03-24T03:17:25.564 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10374 sec 2024-03-24T03:17:26.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:25 smithi106 ceph-mon[36505]: pgmap v5603: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:26.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:25 smithi078 ceph-mon[32429]: pgmap v5603: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:26.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:25 smithi107 ceph-mon[36485]: pgmap v5603: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:26.564 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10375 sec 2024-03-24T03:17:27.564 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10376 sec 2024-03-24T03:17:28.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:27 smithi106 ceph-mon[36505]: pgmap v5604: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:28.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:27 smithi078 ceph-mon[32429]: pgmap v5604: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:28.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:27 smithi107 ceph-mon[36485]: pgmap v5604: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:28.564 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10377 sec 2024-03-24T03:17:29.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9723.242489 secs 2024-03-24T03:17:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9723.242489 secs 2024-03-24T03:17:29.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9723.242489 secs 2024-03-24T03:17:29.564 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10378 sec 2024-03-24T03:17:30.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:29 smithi106 ceph-mon[36505]: pgmap v5605: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:30.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:29 smithi078 ceph-mon[32429]: pgmap v5605: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:30.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:29 smithi107 ceph-mon[36485]: pgmap v5605: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:30.564 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10379 sec 2024-03-24T03:17:31.565 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10380 sec 2024-03-24T03:17:32.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:31 smithi106 ceph-mon[36505]: pgmap v5606: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:32.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:31 smithi078 ceph-mon[32429]: pgmap v5606: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:32.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:31 smithi107 ceph-mon[36485]: pgmap v5606: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:32.565 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10381 sec 2024-03-24T03:17:33.565 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10382 sec 2024-03-24T03:17:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:33 smithi106 ceph-mon[36505]: pgmap v5607: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9728.242548 secs 2024-03-24T03:17:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:33 smithi078 ceph-mon[32429]: pgmap v5607: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9728.242548 secs 2024-03-24T03:17:34.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:33 smithi107 ceph-mon[36485]: pgmap v5607: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:34.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9728.242548 secs 2024-03-24T03:17:34.565 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10383 sec 2024-03-24T03:17:35.565 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10384 sec 2024-03-24T03:17:36.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:35 smithi106 ceph-mon[36505]: pgmap v5608: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:36.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:35 smithi078 ceph-mon[32429]: pgmap v5608: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:36.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:35 smithi107 ceph-mon[36485]: pgmap v5608: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:36.565 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10385 sec 2024-03-24T03:17:37.565 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10386 sec 2024-03-24T03:17:38.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:37 smithi106 ceph-mon[36505]: pgmap v5609: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:38.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:37 smithi078 ceph-mon[32429]: pgmap v5609: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:38.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:37 smithi107 ceph-mon[36485]: pgmap v5609: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:38.565 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10387 sec 2024-03-24T03:17:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9733.242633 secs 2024-03-24T03:17:39.244 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9733.242633 secs 2024-03-24T03:17:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9733.242633 secs 2024-03-24T03:17:39.565 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10388 sec 2024-03-24T03:17:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:39 smithi106 ceph-mon[36505]: pgmap v5610: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:40.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:39 smithi078 ceph-mon[32429]: pgmap v5610: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:40.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:39 smithi107 ceph-mon[36485]: pgmap v5610: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:40.565 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10389 sec 2024-03-24T03:17:41.565 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10390 sec 2024-03-24T03:17:42.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:41 smithi078 ceph-mon[32429]: pgmap v5611: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:42.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:41 smithi107 ceph-mon[36485]: pgmap v5611: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:42.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:41 smithi106 ceph-mon[36505]: pgmap v5611: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:42.565 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10391 sec 2024-03-24T03:17:43.565 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10392 sec 2024-03-24T03:17:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:43 smithi078 ceph-mon[32429]: pgmap v5612: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9738.242715 secs 2024-03-24T03:17:44.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:43 smithi107 ceph-mon[36485]: pgmap v5612: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9738.242715 secs 2024-03-24T03:17:44.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:43 smithi106 ceph-mon[36505]: pgmap v5612: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9738.242715 secs 2024-03-24T03:17:44.565 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10393 sec 2024-03-24T03:17:45.566 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10394 sec 2024-03-24T03:17:46.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:45 smithi078 ceph-mon[32429]: pgmap v5613: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:46.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:45 smithi107 ceph-mon[36485]: pgmap v5613: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:46.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:45 smithi106 ceph-mon[36505]: pgmap v5613: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:46.566 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10395 sec 2024-03-24T03:17:47.566 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10396 sec 2024-03-24T03:17:48.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:47 smithi078 ceph-mon[32429]: pgmap v5614: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:48.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:47 smithi107 ceph-mon[36485]: pgmap v5614: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:48.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:47 smithi106 ceph-mon[36505]: pgmap v5614: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:48.566 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10397 sec 2024-03-24T03:17:49.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9743.242805 secs 2024-03-24T03:17:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9743.242805 secs 2024-03-24T03:17:49.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9743.242805 secs 2024-03-24T03:17:49.566 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10398 sec 2024-03-24T03:17:50.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:49 smithi078 ceph-mon[32429]: pgmap v5615: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:50.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:49 smithi107 ceph-mon[36485]: pgmap v5615: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:50.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:49 smithi106 ceph-mon[36505]: pgmap v5615: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:50.566 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10399 sec 2024-03-24T03:17:51.566 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10400 sec 2024-03-24T03:17:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:51 smithi078 ceph-mon[32429]: pgmap v5616: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:52.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:51 smithi107 ceph-mon[36485]: pgmap v5616: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:52.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:51 smithi106 ceph-mon[36505]: pgmap v5616: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:52.566 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10401 sec 2024-03-24T03:17:53.566 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10402 sec 2024-03-24T03:17:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:53 smithi078 ceph-mon[32429]: pgmap v5617: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9748.242892 secs 2024-03-24T03:17:54.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:53 smithi107 ceph-mon[36485]: pgmap v5617: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9748.242892 secs 2024-03-24T03:17:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:53 smithi106 ceph-mon[36505]: pgmap v5617: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9748.242892 secs 2024-03-24T03:17:54.566 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10403 sec 2024-03-24T03:17:55.566 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10404 sec 2024-03-24T03:17:56.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:55 smithi078 ceph-mon[32429]: pgmap v5618: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:56.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:55 smithi107 ceph-mon[36485]: pgmap v5618: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:56.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:55 smithi106 ceph-mon[36505]: pgmap v5618: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:17:56.566 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10405 sec 2024-03-24T03:17:57.566 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10406 sec 2024-03-24T03:17:58.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:57 smithi078 ceph-mon[32429]: pgmap v5619: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:58.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:57 smithi107 ceph-mon[36485]: pgmap v5619: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:58.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:57 smithi106 ceph-mon[36505]: pgmap v5619: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:58.566 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10407 sec 2024-03-24T03:17:59.302 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9753.242952 secs 2024-03-24T03:17:59.302 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:58 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:17:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9753.242952 secs 2024-03-24T03:17:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:58 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:17:59.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9753.242952 secs 2024-03-24T03:17:59.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:58 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:17:59.567 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10408 sec 2024-03-24T03:18:00.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:59 smithi078 ceph-mon[32429]: pgmap v5620: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:00.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:59 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:18:00.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:59 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:18:00.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:17:59 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:18:00.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:59 smithi107 ceph-mon[36485]: pgmap v5620: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:00.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:59 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:18:00.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:59 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:18:00.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:17:59 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:18:00.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:59 smithi106 ceph-mon[36505]: pgmap v5620: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:00.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:59 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:18:00.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:59 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:18:00.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:17:59 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:18:00.567 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10409 sec 2024-03-24T03:18:01.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:00 smithi078 ceph-mon[32429]: pgmap v5621: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:18:01.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:00 smithi078 ceph-mon[32429]: pgmap v5622: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:18:01.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:00 smithi107 ceph-mon[36485]: pgmap v5621: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:18:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:00 smithi107 ceph-mon[36485]: pgmap v5622: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:18:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:00 smithi106 ceph-mon[36505]: pgmap v5621: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:18:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:00 smithi106 ceph-mon[36505]: pgmap v5622: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:18:01.567 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10410 sec 2024-03-24T03:18:02.567 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10411 sec 2024-03-24T03:18:03.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:02 smithi078 ceph-mon[32429]: pgmap v5623: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:18:03.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:02 smithi107 ceph-mon[36485]: pgmap v5623: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:18:03.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:02 smithi106 ceph-mon[36505]: pgmap v5623: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:18:03.567 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10412 sec 2024-03-24T03:18:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9758.243010 secs 2024-03-24T03:18:04.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9758.243010 secs 2024-03-24T03:18:04.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9758.243010 secs 2024-03-24T03:18:04.567 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10413 sec 2024-03-24T03:18:05.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:04 smithi078 ceph-mon[32429]: pgmap v5624: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:18:05.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:04 smithi107 ceph-mon[36485]: pgmap v5624: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:18:05.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:04 smithi106 ceph-mon[36505]: pgmap v5624: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:18:05.567 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10414 sec 2024-03-24T03:18:06.567 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10415 sec 2024-03-24T03:18:07.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:06 smithi078 ceph-mon[32429]: pgmap v5625: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:18:07.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:06 smithi107 ceph-mon[36485]: pgmap v5625: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:18:07.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:06 smithi106 ceph-mon[36505]: pgmap v5625: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:18:07.567 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10416 sec 2024-03-24T03:18:08.568 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10417 sec 2024-03-24T03:18:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:08 smithi078 ceph-mon[32429]: pgmap v5626: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:18:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9763.243096 secs 2024-03-24T03:18:09.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:08 smithi107 ceph-mon[36485]: pgmap v5626: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:18:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9763.243096 secs 2024-03-24T03:18:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:08 smithi106 ceph-mon[36505]: pgmap v5626: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:18:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9763.243096 secs 2024-03-24T03:18:09.567 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10418 sec 2024-03-24T03:18:10.568 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10419 sec 2024-03-24T03:18:11.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:10 smithi078 ceph-mon[32429]: pgmap v5627: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:18:11.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:10 smithi107 ceph-mon[36485]: pgmap v5627: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:18:11.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:10 smithi106 ceph-mon[36505]: pgmap v5627: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:18:11.568 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10420 sec 2024-03-24T03:18:12.568 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10421 sec 2024-03-24T03:18:13.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:12 smithi078 ceph-mon[32429]: pgmap v5628: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:13.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:12 smithi107 ceph-mon[36485]: pgmap v5628: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:13.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:12 smithi106 ceph-mon[36505]: pgmap v5628: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:13.568 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10422 sec 2024-03-24T03:18:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9768.243151 secs 2024-03-24T03:18:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9768.243151 secs 2024-03-24T03:18:14.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9768.243151 secs 2024-03-24T03:18:14.568 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10423 sec 2024-03-24T03:18:15.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:14 smithi078 ceph-mon[32429]: pgmap v5629: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:15.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:14 smithi107 ceph-mon[36485]: pgmap v5629: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:15.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:14 smithi106 ceph-mon[36505]: pgmap v5629: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:15.568 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10424 sec 2024-03-24T03:18:16.568 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10425 sec 2024-03-24T03:18:17.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:16 smithi078 ceph-mon[32429]: pgmap v5630: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:17.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:16 smithi107 ceph-mon[36485]: pgmap v5630: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:17.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:16 smithi106 ceph-mon[36505]: pgmap v5630: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:17.568 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10426 sec 2024-03-24T03:18:18.568 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10427 sec 2024-03-24T03:18:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:18 smithi078 ceph-mon[32429]: pgmap v5631: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9773.243193 secs 2024-03-24T03:18:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:18 smithi107 ceph-mon[36485]: pgmap v5631: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9773.243193 secs 2024-03-24T03:18:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:18 smithi106 ceph-mon[36505]: pgmap v5631: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9773.243193 secs 2024-03-24T03:18:19.569 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10428 sec 2024-03-24T03:18:20.569 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10429 sec 2024-03-24T03:18:21.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:20 smithi078 ceph-mon[32429]: pgmap v5632: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:21.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:20 smithi107 ceph-mon[36485]: pgmap v5632: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:21.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:20 smithi106 ceph-mon[36505]: pgmap v5632: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:21.569 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10430 sec 2024-03-24T03:18:22.569 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10431 sec 2024-03-24T03:18:23.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:22 smithi078 ceph-mon[32429]: pgmap v5633: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:23.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:22 smithi107 ceph-mon[36485]: pgmap v5633: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:23.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:22 smithi106 ceph-mon[36505]: pgmap v5633: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:23.569 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10432 sec 2024-03-24T03:18:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9778.243280 secs 2024-03-24T03:18:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9778.243280 secs 2024-03-24T03:18:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9778.243280 secs 2024-03-24T03:18:24.569 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10433 sec 2024-03-24T03:18:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:25 smithi078 ceph-mon[32429]: pgmap v5634: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:25.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:25 smithi107 ceph-mon[36485]: pgmap v5634: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:25.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:25 smithi106 ceph-mon[36505]: pgmap v5634: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:25.569 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10434 sec 2024-03-24T03:18:26.569 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10435 sec 2024-03-24T03:18:27.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:27 smithi078 ceph-mon[32429]: pgmap v5635: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:27.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:27 smithi107 ceph-mon[36485]: pgmap v5635: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:27.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:27 smithi106 ceph-mon[36505]: pgmap v5635: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:27.569 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10436 sec 2024-03-24T03:18:28.569 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10437 sec 2024-03-24T03:18:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:29 smithi078 ceph-mon[32429]: pgmap v5636: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9783.243341 secs 2024-03-24T03:18:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:29 smithi107 ceph-mon[36485]: pgmap v5636: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9783.243341 secs 2024-03-24T03:18:29.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:29 smithi106 ceph-mon[36505]: pgmap v5636: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9783.243341 secs 2024-03-24T03:18:29.569 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10438 sec 2024-03-24T03:18:30.569 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10439 sec 2024-03-24T03:18:31.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:31 smithi078 ceph-mon[32429]: pgmap v5637: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:31.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:31 smithi107 ceph-mon[36485]: pgmap v5637: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:31.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:31 smithi106 ceph-mon[36505]: pgmap v5637: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:31.570 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10440 sec 2024-03-24T03:18:32.570 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10441 sec 2024-03-24T03:18:33.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:33 smithi078 ceph-mon[32429]: pgmap v5638: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:33.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:33 smithi107 ceph-mon[36485]: pgmap v5638: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:33.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:33 smithi106 ceph-mon[36505]: pgmap v5638: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:33.570 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10442 sec 2024-03-24T03:18:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9788.243398 secs 2024-03-24T03:18:34.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9788.243398 secs 2024-03-24T03:18:34.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9788.243398 secs 2024-03-24T03:18:34.570 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10443 sec 2024-03-24T03:18:35.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:35 smithi078 ceph-mon[32429]: pgmap v5639: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:35.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:35 smithi107 ceph-mon[36485]: pgmap v5639: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:35.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:35 smithi106 ceph-mon[36505]: pgmap v5639: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:35.570 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10444 sec 2024-03-24T03:18:36.570 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10445 sec 2024-03-24T03:18:37.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:37 smithi078 ceph-mon[32429]: pgmap v5640: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:37 smithi107 ceph-mon[36485]: pgmap v5640: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:37 smithi106 ceph-mon[36505]: pgmap v5640: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:37.570 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10446 sec 2024-03-24T03:18:38.570 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10447 sec 2024-03-24T03:18:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:39 smithi078 ceph-mon[32429]: pgmap v5641: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9793.243488 secs 2024-03-24T03:18:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:39 smithi107 ceph-mon[36485]: pgmap v5641: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9793.243488 secs 2024-03-24T03:18:39.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:39 smithi106 ceph-mon[36505]: pgmap v5641: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9793.243488 secs 2024-03-24T03:18:39.570 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10448 sec 2024-03-24T03:18:40.570 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10449 sec 2024-03-24T03:18:41.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:41 smithi078 ceph-mon[32429]: pgmap v5642: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:41 smithi107 ceph-mon[36485]: pgmap v5642: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:41.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:41 smithi106 ceph-mon[36505]: pgmap v5642: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:41.571 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10450 sec 2024-03-24T03:18:42.571 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10451 sec 2024-03-24T03:18:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:43 smithi078 ceph-mon[32429]: pgmap v5643: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:43.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:43 smithi107 ceph-mon[36485]: pgmap v5643: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:43.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:43 smithi106 ceph-mon[36505]: pgmap v5643: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:43.571 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10452 sec 2024-03-24T03:18:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9798.243545 secs 2024-03-24T03:18:44.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9798.243545 secs 2024-03-24T03:18:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9798.243545 secs 2024-03-24T03:18:44.571 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10453 sec 2024-03-24T03:18:45.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:45 smithi078 ceph-mon[32429]: pgmap v5644: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:45.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:45 smithi107 ceph-mon[36485]: pgmap v5644: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:45.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:45 smithi106 ceph-mon[36505]: pgmap v5644: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:45.571 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10454 sec 2024-03-24T03:18:46.571 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10455 sec 2024-03-24T03:18:47.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:47 smithi078 ceph-mon[32429]: pgmap v5645: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:47.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:47 smithi107 ceph-mon[36485]: pgmap v5645: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:47.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:47 smithi106 ceph-mon[36505]: pgmap v5645: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:47.571 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10456 sec 2024-03-24T03:18:48.571 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10457 sec 2024-03-24T03:18:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:49 smithi078 ceph-mon[32429]: pgmap v5646: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9803.243635 secs 2024-03-24T03:18:49.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:49 smithi107 ceph-mon[36485]: pgmap v5646: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:49.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9803.243635 secs 2024-03-24T03:18:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:49 smithi106 ceph-mon[36505]: pgmap v5646: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9803.243635 secs 2024-03-24T03:18:49.571 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10458 sec 2024-03-24T03:18:50.571 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10459 sec 2024-03-24T03:18:51.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:51 smithi078 ceph-mon[32429]: pgmap v5647: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:51.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:51 smithi107 ceph-mon[36485]: pgmap v5647: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:51.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:51 smithi106 ceph-mon[36505]: pgmap v5647: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:51.571 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10460 sec 2024-03-24T03:18:52.572 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10461 sec 2024-03-24T03:18:53.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:53 smithi078 ceph-mon[32429]: pgmap v5648: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:53.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:53 smithi107 ceph-mon[36485]: pgmap v5648: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:53.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:53 smithi106 ceph-mon[36505]: pgmap v5648: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:53.572 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10462 sec 2024-03-24T03:18:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9808.243709 secs 2024-03-24T03:18:54.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9808.243709 secs 2024-03-24T03:18:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9808.243709 secs 2024-03-24T03:18:54.572 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10463 sec 2024-03-24T03:18:55.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:55 smithi078 ceph-mon[32429]: pgmap v5649: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:55.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:55 smithi107 ceph-mon[36485]: pgmap v5649: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:55.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:55 smithi106 ceph-mon[36505]: pgmap v5649: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:55.572 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10464 sec 2024-03-24T03:18:56.572 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10465 sec 2024-03-24T03:18:57.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:57 smithi078 ceph-mon[32429]: pgmap v5650: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:57.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:57 smithi107 ceph-mon[36485]: pgmap v5650: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:57.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:57 smithi106 ceph-mon[36505]: pgmap v5650: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:18:57.572 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10466 sec 2024-03-24T03:18:58.572 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10467 sec 2024-03-24T03:18:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:59 smithi078 ceph-mon[32429]: pgmap v5651: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:18:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9813.243726 secs 2024-03-24T03:18:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:59 smithi107 ceph-mon[36485]: pgmap v5651: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:18:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9813.243726 secs 2024-03-24T03:18:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:59 smithi106 ceph-mon[36505]: pgmap v5651: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:18:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9813.243726 secs 2024-03-24T03:18:59.572 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10468 sec 2024-03-24T03:19:00.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:00 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:19:00.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:00 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:19:00.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:00 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:19:00.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:00 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:19:00.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:00 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:19:00.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:00 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:19:00.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:00 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:19:00.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:00 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:19:00.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:00 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:19:00.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:00 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:19:00.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:00 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:19:00.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:00 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:19:00.572 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10469 sec 2024-03-24T03:19:01.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:01 smithi078 ceph-mon[32429]: pgmap v5652: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:01.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:01 smithi078 ceph-mon[32429]: pgmap v5653: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:19:01.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:01 smithi078 ceph-mon[32429]: pgmap v5654: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:19:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:01 smithi107 ceph-mon[36485]: pgmap v5652: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:01 smithi107 ceph-mon[36485]: pgmap v5653: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:19:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:01 smithi107 ceph-mon[36485]: pgmap v5654: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:19:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:01 smithi106 ceph-mon[36505]: pgmap v5652: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:01 smithi106 ceph-mon[36505]: pgmap v5653: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:19:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:01 smithi106 ceph-mon[36505]: pgmap v5654: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:19:01.572 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10470 sec 2024-03-24T03:19:02.573 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10471 sec 2024-03-24T03:19:03.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:03 smithi078 ceph-mon[32429]: pgmap v5655: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:19:03.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:03 smithi107 ceph-mon[36485]: pgmap v5655: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:19:03.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:03 smithi106 ceph-mon[36505]: pgmap v5655: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:19:03.573 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10472 sec 2024-03-24T03:19:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9818.243784 secs 2024-03-24T03:19:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9818.243784 secs 2024-03-24T03:19:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9818.243784 secs 2024-03-24T03:19:04.573 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10473 sec 2024-03-24T03:19:05.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:05 smithi078 ceph-mon[32429]: pgmap v5656: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:19:05.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:05 smithi107 ceph-mon[36485]: pgmap v5656: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:19:05.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:05 smithi106 ceph-mon[36505]: pgmap v5656: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:19:05.573 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10474 sec 2024-03-24T03:19:06.573 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10475 sec 2024-03-24T03:19:07.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:07 smithi107 ceph-mon[36485]: pgmap v5657: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:19:07.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:07 smithi106 ceph-mon[36505]: pgmap v5657: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:19:07.573 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10476 sec 2024-03-24T03:19:07.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:07 smithi078 ceph-mon[32429]: pgmap v5657: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:19:08.573 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10477 sec 2024-03-24T03:19:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9823.243870 secs 2024-03-24T03:19:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:09 smithi106 ceph-mon[36505]: pgmap v5658: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:19:09.573 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10478 sec 2024-03-24T03:19:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9823.243870 secs 2024-03-24T03:19:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:09 smithi078 ceph-mon[32429]: pgmap v5658: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:19:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9823.243870 secs 2024-03-24T03:19:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:09 smithi107 ceph-mon[36485]: pgmap v5658: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:19:10.573 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10479 sec 2024-03-24T03:19:11.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:11 smithi106 ceph-mon[36505]: pgmap v5659: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:19:11.573 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10480 sec 2024-03-24T03:19:11.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:11 smithi078 ceph-mon[32429]: pgmap v5659: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:19:11.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:11 smithi107 ceph-mon[36485]: pgmap v5659: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:19:12.573 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10481 sec 2024-03-24T03:19:13.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:13 smithi106 ceph-mon[36505]: pgmap v5660: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:13.574 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10482 sec 2024-03-24T03:19:13.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:13 smithi078 ceph-mon[32429]: pgmap v5660: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:13.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:13 smithi107 ceph-mon[36485]: pgmap v5660: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9828.243928 secs 2024-03-24T03:19:14.573 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10483 sec 2024-03-24T03:19:14.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9828.243928 secs 2024-03-24T03:19:14.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9828.243928 secs 2024-03-24T03:19:15.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:15 smithi106 ceph-mon[36505]: pgmap v5661: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:15.574 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10484 sec 2024-03-24T03:19:15.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:15 smithi078 ceph-mon[32429]: pgmap v5661: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:15.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:15 smithi107 ceph-mon[36485]: pgmap v5661: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:16.574 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10485 sec 2024-03-24T03:19:17.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:17 smithi106 ceph-mon[36505]: pgmap v5662: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:17.574 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10486 sec 2024-03-24T03:19:17.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:17 smithi078 ceph-mon[32429]: pgmap v5662: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:17.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:17 smithi107 ceph-mon[36485]: pgmap v5662: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:18.574 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10487 sec 2024-03-24T03:19:19.392 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9833.243987 secs 2024-03-24T03:19:19.393 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:19 smithi078 ceph-mon[32429]: pgmap v5663: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9833.243987 secs 2024-03-24T03:19:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:19 smithi106 ceph-mon[36505]: pgmap v5663: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:19.574 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10488 sec 2024-03-24T03:19:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9833.243987 secs 2024-03-24T03:19:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:19 smithi107 ceph-mon[36485]: pgmap v5663: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:20.574 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10489 sec 2024-03-24T03:19:21.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:21 smithi106 ceph-mon[36505]: pgmap v5664: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:21.574 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10490 sec 2024-03-24T03:19:21.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:21 smithi078 ceph-mon[32429]: pgmap v5664: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:21.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:21 smithi107 ceph-mon[36485]: pgmap v5664: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:22.574 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10491 sec 2024-03-24T03:19:23.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:23 smithi106 ceph-mon[36505]: pgmap v5665: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:23.574 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10492 sec 2024-03-24T03:19:23.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:23 smithi078 ceph-mon[32429]: pgmap v5665: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:23.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:23 smithi107 ceph-mon[36485]: pgmap v5665: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9838.244042 secs 2024-03-24T03:19:24.575 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10493 sec 2024-03-24T03:19:24.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9838.244042 secs 2024-03-24T03:19:24.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9838.244042 secs 2024-03-24T03:19:25.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:25 smithi106 ceph-mon[36505]: pgmap v5666: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:25.575 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10494 sec 2024-03-24T03:19:25.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:25 smithi078 ceph-mon[32429]: pgmap v5666: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:25.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:25 smithi107 ceph-mon[36485]: pgmap v5666: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:26.575 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10495 sec 2024-03-24T03:19:27.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:27 smithi106 ceph-mon[36505]: pgmap v5667: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:27.575 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10496 sec 2024-03-24T03:19:27.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:27 smithi078 ceph-mon[32429]: pgmap v5667: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:27.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:27 smithi107 ceph-mon[36485]: pgmap v5667: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:28.575 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10497 sec 2024-03-24T03:19:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9843.244129 secs 2024-03-24T03:19:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:29 smithi106 ceph-mon[36505]: pgmap v5668: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:29.575 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10498 sec 2024-03-24T03:19:29.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9843.244129 secs 2024-03-24T03:19:29.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:29 smithi078 ceph-mon[32429]: pgmap v5668: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:29.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9843.244129 secs 2024-03-24T03:19:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:29 smithi107 ceph-mon[36485]: pgmap v5668: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:30.575 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10499 sec 2024-03-24T03:19:31.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:31 smithi106 ceph-mon[36505]: pgmap v5669: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:31.575 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10500 sec 2024-03-24T03:19:31.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:31 smithi078 ceph-mon[32429]: pgmap v5669: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:31.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:31 smithi107 ceph-mon[36485]: pgmap v5669: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:32.575 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10501 sec 2024-03-24T03:19:33.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:33 smithi106 ceph-mon[36505]: pgmap v5670: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:33.575 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10502 sec 2024-03-24T03:19:33.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:33 smithi078 ceph-mon[32429]: pgmap v5670: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:33.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:33 smithi107 ceph-mon[36485]: pgmap v5670: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9848.244214 secs 2024-03-24T03:19:34.575 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10503 sec 2024-03-24T03:19:34.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9848.244214 secs 2024-03-24T03:19:34.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9848.244214 secs 2024-03-24T03:19:35.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:35 smithi106 ceph-mon[36505]: pgmap v5671: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:35.575 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10504 sec 2024-03-24T03:19:35.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:35 smithi078 ceph-mon[32429]: pgmap v5671: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:35.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:35 smithi107 ceph-mon[36485]: pgmap v5671: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:36.575 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10505 sec 2024-03-24T03:19:37.576 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10506 sec 2024-03-24T03:19:37.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:37 smithi078 ceph-mon[32429]: pgmap v5672: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:37.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:37 smithi107 ceph-mon[36485]: pgmap v5672: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:37.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:37 smithi106 ceph-mon[36505]: pgmap v5672: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:38.576 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10507 sec 2024-03-24T03:19:39.576 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10508 sec 2024-03-24T03:19:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9853.244301 secs 2024-03-24T03:19:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:39 smithi078 ceph-mon[32429]: pgmap v5673: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:39.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9853.244301 secs 2024-03-24T03:19:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:39 smithi107 ceph-mon[36485]: pgmap v5673: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9853.244301 secs 2024-03-24T03:19:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:39 smithi106 ceph-mon[36505]: pgmap v5673: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:40.576 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10509 sec 2024-03-24T03:19:41.576 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10510 sec 2024-03-24T03:19:41.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:41 smithi078 ceph-mon[32429]: pgmap v5674: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:41.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:41 smithi107 ceph-mon[36485]: pgmap v5674: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:41.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:41 smithi106 ceph-mon[36505]: pgmap v5674: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:42.576 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10511 sec 2024-03-24T03:19:43.576 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10512 sec 2024-03-24T03:19:43.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:43 smithi078 ceph-mon[32429]: pgmap v5675: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:43.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:43 smithi107 ceph-mon[36485]: pgmap v5675: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:43.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:43 smithi106 ceph-mon[36505]: pgmap v5675: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:44.576 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10513 sec 2024-03-24T03:19:44.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9858.244385 secs 2024-03-24T03:19:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9858.244385 secs 2024-03-24T03:19:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9858.244385 secs 2024-03-24T03:19:45.576 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10514 sec 2024-03-24T03:19:45.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:45 smithi078 ceph-mon[32429]: pgmap v5676: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:45.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:45 smithi107 ceph-mon[36485]: pgmap v5676: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:45.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:45 smithi106 ceph-mon[36505]: pgmap v5676: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:46.577 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10515 sec 2024-03-24T03:19:47.577 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10516 sec 2024-03-24T03:19:47.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:47 smithi078 ceph-mon[32429]: pgmap v5677: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:47.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:47 smithi107 ceph-mon[36485]: pgmap v5677: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:47.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:47 smithi106 ceph-mon[36505]: pgmap v5677: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:48.577 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10517 sec 2024-03-24T03:19:49.577 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10518 sec 2024-03-24T03:19:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9863.244470 secs 2024-03-24T03:19:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:49 smithi078 ceph-mon[32429]: pgmap v5678: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9863.244470 secs 2024-03-24T03:19:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:49 smithi107 ceph-mon[36485]: pgmap v5678: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:49.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9863.244470 secs 2024-03-24T03:19:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:49 smithi106 ceph-mon[36505]: pgmap v5678: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:50.577 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10519 sec 2024-03-24T03:19:50.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:50 smithi078 ceph-mon[32429]: pgmap v5679: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:50.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:50 smithi107 ceph-mon[36485]: pgmap v5679: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:50.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:50 smithi106 ceph-mon[36505]: pgmap v5679: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:51.577 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10520 sec 2024-03-24T03:19:52.577 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10521 sec 2024-03-24T03:19:53.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:53 smithi106 ceph-mon[36505]: pgmap v5680: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:53.577 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10522 sec 2024-03-24T03:19:53.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:53 smithi078 ceph-mon[32429]: pgmap v5680: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:53.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:53 smithi107 ceph-mon[36485]: pgmap v5680: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9868.244530 secs 2024-03-24T03:19:54.577 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10523 sec 2024-03-24T03:19:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9868.244530 secs 2024-03-24T03:19:54.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9868.244530 secs 2024-03-24T03:19:55.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:55 smithi106 ceph-mon[36505]: pgmap v5681: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:55.577 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10524 sec 2024-03-24T03:19:55.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:55 smithi078 ceph-mon[32429]: pgmap v5681: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:55.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:55 smithi107 ceph-mon[36485]: pgmap v5681: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:56.578 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10525 sec 2024-03-24T03:19:57.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:57 smithi106 ceph-mon[36505]: pgmap v5682: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:57.578 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10526 sec 2024-03-24T03:19:57.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:57 smithi078 ceph-mon[32429]: pgmap v5682: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:57.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:57 smithi107 ceph-mon[36485]: pgmap v5682: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:58.578 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10527 sec 2024-03-24T03:19:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9873.244607 secs 2024-03-24T03:19:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:19:59 smithi106 ceph-mon[36505]: pgmap v5683: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:59.578 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10528 sec 2024-03-24T03:19:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9873.244607 secs 2024-03-24T03:19:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:19:59 smithi078 ceph-mon[32429]: pgmap v5683: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:19:59.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9873.244607 secs 2024-03-24T03:19:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:19:59 smithi107 ceph-mon[36485]: pgmap v5683: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:00.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:00 smithi106 ceph-mon[36505]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T03:20:00.578 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10529 sec 2024-03-24T03:20:00.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:00 smithi078 ceph-mon[32429]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T03:20:00.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:00 smithi107 ceph-mon[36485]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T03:20:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:01 smithi106 ceph-mon[36505]: pgmap v5684: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:01 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:20:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:01 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:20:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:01 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:20:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:01 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:20:01.578 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10530 sec 2024-03-24T03:20:01.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:01 smithi078 ceph-mon[32429]: pgmap v5684: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:01.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:01 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:20:01.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:01 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:20:01.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:01 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:20:01.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:01 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:20:01.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:01 smithi107 ceph-mon[36485]: pgmap v5684: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:01.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:01 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:20:01.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:01 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:20:01.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:01 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:20:01.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:01 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:20:02.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:02 smithi106 ceph-mon[36505]: pgmap v5685: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:20:02.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:02 smithi106 ceph-mon[36505]: pgmap v5686: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:20:02.579 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10531 sec 2024-03-24T03:20:02.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:02 smithi078 ceph-mon[32429]: pgmap v5685: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:20:02.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:02 smithi078 ceph-mon[32429]: pgmap v5686: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:20:02.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:02 smithi107 ceph-mon[36485]: pgmap v5685: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:20:02.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:02 smithi107 ceph-mon[36485]: pgmap v5686: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:20:03.578 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10532 sec 2024-03-24T03:20:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:04 smithi106 ceph-mon[36505]: pgmap v5687: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:20:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9878.244707 secs 2024-03-24T03:20:04.578 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10533 sec 2024-03-24T03:20:04.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:04 smithi078 ceph-mon[32429]: pgmap v5687: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:20:04.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9878.244707 secs 2024-03-24T03:20:04.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:04 smithi107 ceph-mon[36485]: pgmap v5687: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:20:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9878.244707 secs 2024-03-24T03:20:05.578 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10534 sec 2024-03-24T03:20:06.578 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10535 sec 2024-03-24T03:20:06.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:06 smithi078 ceph-mon[32429]: pgmap v5688: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:20:06.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:06 smithi107 ceph-mon[36485]: pgmap v5688: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:20:06.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:06 smithi106 ceph-mon[36505]: pgmap v5688: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:20:07.579 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10536 sec 2024-03-24T03:20:08.579 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10537 sec 2024-03-24T03:20:08.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:08 smithi078 ceph-mon[32429]: pgmap v5689: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:20:08.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:08 smithi107 ceph-mon[36485]: pgmap v5689: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:20:08.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:08 smithi106 ceph-mon[36505]: pgmap v5689: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:20:09.494 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9883.244795 secs 2024-03-24T03:20:09.579 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10538 sec 2024-03-24T03:20:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9883.244795 secs 2024-03-24T03:20:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9883.244795 secs 2024-03-24T03:20:10.579 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10539 sec 2024-03-24T03:20:10.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:10 smithi078 ceph-mon[32429]: pgmap v5690: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:20:10.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:10 smithi107 ceph-mon[36485]: pgmap v5690: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:20:10.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:10 smithi106 ceph-mon[36505]: pgmap v5690: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:20:11.579 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10540 sec 2024-03-24T03:20:12.579 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10541 sec 2024-03-24T03:20:12.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:12 smithi078 ceph-mon[32429]: pgmap v5691: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:20:12.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:12 smithi107 ceph-mon[36485]: pgmap v5691: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:20:12.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:12 smithi106 ceph-mon[36505]: pgmap v5691: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:20:13.579 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10542 sec 2024-03-24T03:20:14.579 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10543 sec 2024-03-24T03:20:14.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:14 smithi078 ceph-mon[32429]: pgmap v5692: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:14.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9888.244881 secs 2024-03-24T03:20:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:14 smithi107 ceph-mon[36485]: pgmap v5692: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9888.244881 secs 2024-03-24T03:20:14.651 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:14 smithi106 ceph-mon[36505]: pgmap v5692: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9888.244881 secs 2024-03-24T03:20:15.579 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10544 sec 2024-03-24T03:20:16.579 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10545 sec 2024-03-24T03:20:16.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:16 smithi078 ceph-mon[32429]: pgmap v5693: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:16.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:16 smithi107 ceph-mon[36485]: pgmap v5693: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:16 smithi106 ceph-mon[36505]: pgmap v5693: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:17.579 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10546 sec 2024-03-24T03:20:18.579 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10547 sec 2024-03-24T03:20:18.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:18 smithi078 ceph-mon[32429]: pgmap v5694: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:20:18.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:18 smithi107 ceph-mon[36485]: pgmap v5694: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:20:18.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:18 smithi106 ceph-mon[36505]: pgmap v5694: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:20:19.491 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9893.244957 secs 2024-03-24T03:20:19.580 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10548 sec 2024-03-24T03:20:19.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9893.244957 secs 2024-03-24T03:20:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9893.244957 secs 2024-03-24T03:20:20.580 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10549 sec 2024-03-24T03:20:20.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:20 smithi078 ceph-mon[32429]: pgmap v5695: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:20.592 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:20 smithi107 ceph-mon[36485]: pgmap v5695: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:20.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:20 smithi106 ceph-mon[36505]: pgmap v5695: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:21.580 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10550 sec 2024-03-24T03:20:21.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:21 smithi078 ceph-mon[32429]: pgmap v5696: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:20:21.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:21 smithi107 ceph-mon[36485]: pgmap v5696: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:20:21.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:21 smithi106 ceph-mon[36505]: pgmap v5696: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:20:22.580 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10551 sec 2024-03-24T03:20:23.580 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10552 sec 2024-03-24T03:20:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:23 smithi106 ceph-mon[36505]: pgmap v5697: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9898.245026 secs 2024-03-24T03:20:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:23 smithi078 ceph-mon[32429]: pgmap v5697: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9898.245026 secs 2024-03-24T03:20:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:23 smithi107 ceph-mon[36485]: pgmap v5697: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9898.245026 secs 2024-03-24T03:20:24.580 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10553 sec 2024-03-24T03:20:25.580 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10554 sec 2024-03-24T03:20:26.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:25 smithi106 ceph-mon[36505]: pgmap v5698: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:26.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:25 smithi078 ceph-mon[32429]: pgmap v5698: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:26.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:25 smithi107 ceph-mon[36485]: pgmap v5698: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:26.580 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10555 sec 2024-03-24T03:20:27.580 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10556 sec 2024-03-24T03:20:28.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:27 smithi106 ceph-mon[36505]: pgmap v5699: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:20:28.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:27 smithi078 ceph-mon[32429]: pgmap v5699: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:20:28.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:27 smithi107 ceph-mon[36485]: pgmap v5699: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:20:28.580 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10557 sec 2024-03-24T03:20:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9903.245115 secs 2024-03-24T03:20:29.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9903.245115 secs 2024-03-24T03:20:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9903.245115 secs 2024-03-24T03:20:29.580 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10558 sec 2024-03-24T03:20:30.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:29 smithi078 ceph-mon[32429]: pgmap v5700: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:30.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:29 smithi107 ceph-mon[36485]: pgmap v5700: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:30.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:29 smithi106 ceph-mon[36505]: pgmap v5700: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:30.580 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10559 sec 2024-03-24T03:20:31.581 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10560 sec 2024-03-24T03:20:32.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:31 smithi078 ceph-mon[32429]: pgmap v5701: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:20:32.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:31 smithi107 ceph-mon[36485]: pgmap v5701: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:20:32.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:31 smithi106 ceph-mon[36505]: pgmap v5701: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:20:32.581 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10561 sec 2024-03-24T03:20:33.581 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10562 sec 2024-03-24T03:20:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:33 smithi078 ceph-mon[32429]: pgmap v5702: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9908.245221 secs 2024-03-24T03:20:34.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:33 smithi107 ceph-mon[36485]: pgmap v5702: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9908.245221 secs 2024-03-24T03:20:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:33 smithi106 ceph-mon[36505]: pgmap v5702: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9908.245221 secs 2024-03-24T03:20:34.581 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10563 sec 2024-03-24T03:20:35.581 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10564 sec 2024-03-24T03:20:36.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:35 smithi078 ceph-mon[32429]: pgmap v5703: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:36.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:35 smithi107 ceph-mon[36485]: pgmap v5703: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:36.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:35 smithi106 ceph-mon[36505]: pgmap v5703: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:36.581 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10565 sec 2024-03-24T03:20:37.581 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10566 sec 2024-03-24T03:20:38.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:37 smithi078 ceph-mon[32429]: pgmap v5704: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:20:38.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:37 smithi107 ceph-mon[36485]: pgmap v5704: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:20:38.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:37 smithi106 ceph-mon[36505]: pgmap v5704: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:20:38.581 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10567 sec 2024-03-24T03:20:39.244 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9913.245319 secs 2024-03-24T03:20:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9913.245319 secs 2024-03-24T03:20:39.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9913.245319 secs 2024-03-24T03:20:39.581 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10568 sec 2024-03-24T03:20:40.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:39 smithi078 ceph-mon[32429]: pgmap v5705: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:39 smithi107 ceph-mon[36485]: pgmap v5705: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:40.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:39 smithi106 ceph-mon[36505]: pgmap v5705: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:40.581 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10569 sec 2024-03-24T03:20:41.582 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10570 sec 2024-03-24T03:20:42.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:41 smithi078 ceph-mon[32429]: pgmap v5706: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:20:42.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:41 smithi107 ceph-mon[36485]: pgmap v5706: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:20:42.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:41 smithi106 ceph-mon[36505]: pgmap v5706: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:20:42.582 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10571 sec 2024-03-24T03:20:43.582 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10572 sec 2024-03-24T03:20:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:43 smithi078 ceph-mon[32429]: pgmap v5707: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9918.245381 secs 2024-03-24T03:20:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:43 smithi107 ceph-mon[36485]: pgmap v5707: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9918.245381 secs 2024-03-24T03:20:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:43 smithi106 ceph-mon[36505]: pgmap v5707: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9918.245381 secs 2024-03-24T03:20:44.582 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10573 sec 2024-03-24T03:20:45.582 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10574 sec 2024-03-24T03:20:46.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:45 smithi078 ceph-mon[32429]: pgmap v5708: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:46.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:45 smithi107 ceph-mon[36485]: pgmap v5708: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:46.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:45 smithi106 ceph-mon[36505]: pgmap v5708: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:46.582 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10575 sec 2024-03-24T03:20:47.582 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10576 sec 2024-03-24T03:20:48.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:47 smithi078 ceph-mon[32429]: pgmap v5709: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:20:48.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:47 smithi107 ceph-mon[36485]: pgmap v5709: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:20:48.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:47 smithi106 ceph-mon[36505]: pgmap v5709: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:20:48.582 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10577 sec 2024-03-24T03:20:49.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9923.245442 secs 2024-03-24T03:20:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9923.245442 secs 2024-03-24T03:20:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9923.245442 secs 2024-03-24T03:20:49.582 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10578 sec 2024-03-24T03:20:50.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:49 smithi078 ceph-mon[32429]: pgmap v5710: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:50.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:49 smithi107 ceph-mon[36485]: pgmap v5710: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:50.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:49 smithi106 ceph-mon[36505]: pgmap v5710: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:50.582 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10579 sec 2024-03-24T03:20:51.583 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10580 sec 2024-03-24T03:20:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:51 smithi078 ceph-mon[32429]: pgmap v5711: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:20:52.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:51 smithi107 ceph-mon[36485]: pgmap v5711: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:20:52.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:51 smithi106 ceph-mon[36505]: pgmap v5711: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:20:52.582 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10581 sec 2024-03-24T03:20:53.583 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10582 sec 2024-03-24T03:20:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:53 smithi078 ceph-mon[32429]: pgmap v5712: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9928.245503 secs 2024-03-24T03:20:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:53 smithi107 ceph-mon[36485]: pgmap v5712: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9928.245503 secs 2024-03-24T03:20:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:53 smithi106 ceph-mon[36505]: pgmap v5712: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9928.245503 secs 2024-03-24T03:20:54.583 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10583 sec 2024-03-24T03:20:55.583 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10584 sec 2024-03-24T03:20:56.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:55 smithi078 ceph-mon[32429]: pgmap v5713: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:56.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:55 smithi107 ceph-mon[36485]: pgmap v5713: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:56.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:55 smithi106 ceph-mon[36505]: pgmap v5713: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:20:56.583 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10585 sec 2024-03-24T03:20:57.583 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10586 sec 2024-03-24T03:20:58.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:57 smithi078 ceph-mon[32429]: pgmap v5714: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:20:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:57 smithi107 ceph-mon[36485]: pgmap v5714: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:20:58.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:57 smithi106 ceph-mon[36505]: pgmap v5714: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:20:58.583 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10587 sec 2024-03-24T03:20:59.303 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9933.245542 secs 2024-03-24T03:20:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9933.245542 secs 2024-03-24T03:20:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9933.245542 secs 2024-03-24T03:20:59.583 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10588 sec 2024-03-24T03:21:00.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:20:59 smithi078 ceph-mon[32429]: pgmap v5715: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:00.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:20:59 smithi107 ceph-mon[36485]: pgmap v5715: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:00.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:20:59 smithi106 ceph-mon[36505]: pgmap v5715: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:00.583 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10589 sec 2024-03-24T03:21:01.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:00 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:21:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:00 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:21:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:00 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:21:01.583 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10590 sec 2024-03-24T03:21:02.254 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:01 smithi078 ceph-mon[32429]: pgmap v5716: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:21:02.301 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:01 smithi106 ceph-mon[36505]: pgmap v5716: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:21:02.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:01 smithi107 ceph-mon[36485]: pgmap v5716: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:21:02.584 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10591 sec 2024-03-24T03:21:03.583 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10592 sec 2024-03-24T03:21:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:03 smithi078 ceph-mon[32429]: pgmap v5717: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:04.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9938.245605 secs 2024-03-24T03:21:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:03 smithi107 ceph-mon[36485]: pgmap v5717: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9938.245605 secs 2024-03-24T03:21:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:03 smithi106 ceph-mon[36505]: pgmap v5717: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9938.245605 secs 2024-03-24T03:21:04.583 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10593 sec 2024-03-24T03:21:05.584 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10594 sec 2024-03-24T03:21:06.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:05 smithi106 ceph-mon[36505]: pgmap v5718: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:06.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:21:06.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:21:06.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:21:06.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:21:06.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:21:06.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:21:06.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:05 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:21:06.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:05 smithi078 ceph-mon[32429]: pgmap v5718: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:06.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:21:06.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:21:06.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:21:06.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:21:06.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:21:06.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:21:06.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:05 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:21:06.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:05 smithi107 ceph-mon[36485]: pgmap v5718: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:06.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:21:06.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:21:06.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:21:06.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:21:06.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:21:06.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:21:06.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:05 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:21:06.584 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10595 sec 2024-03-24T03:21:07.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:06 smithi078 ceph-mon[32429]: pgmap v5719: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:21:07.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:06 smithi078 ceph-mon[32429]: pgmap v5720: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T03:21:07.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:06 smithi107 ceph-mon[36485]: pgmap v5719: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:21:07.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:06 smithi107 ceph-mon[36485]: pgmap v5720: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T03:21:07.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:06 smithi106 ceph-mon[36505]: pgmap v5719: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:21:07.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:06 smithi106 ceph-mon[36505]: pgmap v5720: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T03:21:07.583 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10596 sec 2024-03-24T03:21:08.584 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10597 sec 2024-03-24T03:21:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:08 smithi078 ceph-mon[32429]: pgmap v5721: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:21:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9943.245688 secs 2024-03-24T03:21:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:08 smithi107 ceph-mon[36485]: pgmap v5721: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:21:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9943.245688 secs 2024-03-24T03:21:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:08 smithi106 ceph-mon[36505]: pgmap v5721: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:21:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9943.245688 secs 2024-03-24T03:21:09.584 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10598 sec 2024-03-24T03:21:10.584 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10599 sec 2024-03-24T03:21:11.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:10 smithi078 ceph-mon[32429]: pgmap v5722: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T03:21:11.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:10 smithi107 ceph-mon[36485]: pgmap v5722: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T03:21:11.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:10 smithi106 ceph-mon[36505]: pgmap v5722: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T03:21:11.584 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10600 sec 2024-03-24T03:21:12.584 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10601 sec 2024-03-24T03:21:13.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:12 smithi078 ceph-mon[32429]: pgmap v5723: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:21:13.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:12 smithi107 ceph-mon[36485]: pgmap v5723: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:21:13.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:12 smithi106 ceph-mon[36505]: pgmap v5723: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:21:13.584 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10602 sec 2024-03-24T03:21:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9948.245747 secs 2024-03-24T03:21:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9948.245747 secs 2024-03-24T03:21:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9948.245747 secs 2024-03-24T03:21:14.584 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10603 sec 2024-03-24T03:21:15.335 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:14 smithi106 ceph-mon[36505]: pgmap v5724: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:21:15.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:14 smithi078 ceph-mon[32429]: pgmap v5724: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:21:15.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:14 smithi107 ceph-mon[36485]: pgmap v5724: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:21:15.584 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10604 sec 2024-03-24T03:21:16.585 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10605 sec 2024-03-24T03:21:17.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:16 smithi078 ceph-mon[32429]: pgmap v5725: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:21:17.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:16 smithi107 ceph-mon[36485]: pgmap v5725: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:21:17.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:16 smithi106 ceph-mon[36505]: pgmap v5725: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:21:17.584 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10606 sec 2024-03-24T03:21:18.585 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10607 sec 2024-03-24T03:21:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:18 smithi078 ceph-mon[32429]: pgmap v5726: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:21:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9953.245806 secs 2024-03-24T03:21:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:18 smithi107 ceph-mon[36485]: pgmap v5726: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:21:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9953.245806 secs 2024-03-24T03:21:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:18 smithi106 ceph-mon[36505]: pgmap v5726: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:21:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9953.245806 secs 2024-03-24T03:21:19.585 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10608 sec 2024-03-24T03:21:20.585 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10609 sec 2024-03-24T03:21:21.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:20 smithi078 ceph-mon[32429]: pgmap v5727: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:21.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:20 smithi107 ceph-mon[36485]: pgmap v5727: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:21.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:20 smithi106 ceph-mon[36505]: pgmap v5727: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:21.585 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10610 sec 2024-03-24T03:21:22.585 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10611 sec 2024-03-24T03:21:23.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:23 smithi078 ceph-mon[32429]: pgmap v5728: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:21:23.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:23 smithi107 ceph-mon[36485]: pgmap v5728: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:21:23.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:23 smithi106 ceph-mon[36505]: pgmap v5728: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:21:23.585 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10612 sec 2024-03-24T03:21:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9958.245894 secs 2024-03-24T03:21:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9958.245894 secs 2024-03-24T03:21:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9958.245894 secs 2024-03-24T03:21:24.585 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10613 sec 2024-03-24T03:21:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:25 smithi078 ceph-mon[32429]: pgmap v5729: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:25.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:25 smithi107 ceph-mon[36485]: pgmap v5729: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:25.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:25 smithi106 ceph-mon[36505]: pgmap v5729: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:25.585 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10614 sec 2024-03-24T03:21:26.585 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10615 sec 2024-03-24T03:21:27.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:27 smithi078 ceph-mon[32429]: pgmap v5730: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:21:27.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:27 smithi107 ceph-mon[36485]: pgmap v5730: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:21:27.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:27 smithi106 ceph-mon[36505]: pgmap v5730: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:21:27.585 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10616 sec 2024-03-24T03:21:28.586 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10617 sec 2024-03-24T03:21:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:29 smithi078 ceph-mon[32429]: pgmap v5731: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9963.245934 secs 2024-03-24T03:21:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:29 smithi107 ceph-mon[36485]: pgmap v5731: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9963.245934 secs 2024-03-24T03:21:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:29 smithi106 ceph-mon[36505]: pgmap v5731: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:29.412 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9963.245934 secs 2024-03-24T03:21:29.586 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10618 sec 2024-03-24T03:21:30.586 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10619 sec 2024-03-24T03:21:31.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:31 smithi078 ceph-mon[32429]: pgmap v5732: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:31.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:31 smithi107 ceph-mon[36485]: pgmap v5732: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:31.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:31 smithi106 ceph-mon[36505]: pgmap v5732: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:31.586 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10620 sec 2024-03-24T03:21:32.586 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10621 sec 2024-03-24T03:21:33.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:33 smithi078 ceph-mon[32429]: pgmap v5733: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:21:33.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:33 smithi107 ceph-mon[36485]: pgmap v5733: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:21:33.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:33 smithi106 ceph-mon[36505]: pgmap v5733: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:21:33.586 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10622 sec 2024-03-24T03:21:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9968.245993 secs 2024-03-24T03:21:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9968.245993 secs 2024-03-24T03:21:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9968.245993 secs 2024-03-24T03:21:34.586 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10623 sec 2024-03-24T03:21:35.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:35 smithi078 ceph-mon[32429]: pgmap v5734: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:35.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:35 smithi107 ceph-mon[36485]: pgmap v5734: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:35.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:35 smithi106 ceph-mon[36505]: pgmap v5734: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:35.586 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10624 sec 2024-03-24T03:21:36.586 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10625 sec 2024-03-24T03:21:37.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:37 smithi078 ceph-mon[32429]: pgmap v5735: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:21:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:37 smithi107 ceph-mon[36485]: pgmap v5735: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:21:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:37 smithi106 ceph-mon[36505]: pgmap v5735: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:21:37.586 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10626 sec 2024-03-24T03:21:38.586 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10627 sec 2024-03-24T03:21:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:39 smithi078 ceph-mon[32429]: pgmap v5736: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9973.246080 secs 2024-03-24T03:21:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:39 smithi107 ceph-mon[36485]: pgmap v5736: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9973.246080 secs 2024-03-24T03:21:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:39 smithi106 ceph-mon[36505]: pgmap v5736: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9973.246080 secs 2024-03-24T03:21:39.586 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10628 sec 2024-03-24T03:21:40.586 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10629 sec 2024-03-24T03:21:41.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:41 smithi078 ceph-mon[32429]: pgmap v5737: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:41 smithi107 ceph-mon[36485]: pgmap v5737: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:41.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:41 smithi106 ceph-mon[36505]: pgmap v5737: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:41.586 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10630 sec 2024-03-24T03:21:42.587 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10631 sec 2024-03-24T03:21:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:43 smithi078 ceph-mon[32429]: pgmap v5738: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:21:43.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:43 smithi107 ceph-mon[36485]: pgmap v5738: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:21:43.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:43 smithi106 ceph-mon[36505]: pgmap v5738: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:21:43.587 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10632 sec 2024-03-24T03:21:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9978.246138 secs 2024-03-24T03:21:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9978.246138 secs 2024-03-24T03:21:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9978.246138 secs 2024-03-24T03:21:44.587 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10633 sec 2024-03-24T03:21:45.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:45 smithi078 ceph-mon[32429]: pgmap v5739: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:45.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:45 smithi107 ceph-mon[36485]: pgmap v5739: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:45.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:45 smithi106 ceph-mon[36505]: pgmap v5739: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:45.587 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10634 sec 2024-03-24T03:21:46.587 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10635 sec 2024-03-24T03:21:47.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:47 smithi078 ceph-mon[32429]: pgmap v5740: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:21:47.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:47 smithi107 ceph-mon[36485]: pgmap v5740: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:21:47.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:47 smithi106 ceph-mon[36505]: pgmap v5740: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:21:47.587 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10636 sec 2024-03-24T03:21:48.587 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10637 sec 2024-03-24T03:21:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:49 smithi078 ceph-mon[32429]: pgmap v5741: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9983.246199 secs 2024-03-24T03:21:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:49 smithi107 ceph-mon[36485]: pgmap v5741: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9983.246199 secs 2024-03-24T03:21:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:49 smithi106 ceph-mon[36505]: pgmap v5741: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9983.246199 secs 2024-03-24T03:21:49.587 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10638 sec 2024-03-24T03:21:50.589 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10639 sec 2024-03-24T03:21:51.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:51 smithi078 ceph-mon[32429]: pgmap v5742: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:51.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:51 smithi107 ceph-mon[36485]: pgmap v5742: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:51.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:51 smithi106 ceph-mon[36505]: pgmap v5742: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:51.587 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10640 sec 2024-03-24T03:21:52.588 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10641 sec 2024-03-24T03:21:53.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:53 smithi078 ceph-mon[32429]: pgmap v5743: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:21:53.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:53 smithi107 ceph-mon[36485]: pgmap v5743: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:21:53.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:53 smithi106 ceph-mon[36505]: pgmap v5743: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:21:53.588 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10642 sec 2024-03-24T03:21:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9988.246259 secs 2024-03-24T03:21:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9988.246259 secs 2024-03-24T03:21:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9988.246259 secs 2024-03-24T03:21:54.588 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10643 sec 2024-03-24T03:21:55.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:55 smithi078 ceph-mon[32429]: pgmap v5744: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:55.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:55 smithi107 ceph-mon[36485]: pgmap v5744: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:55.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:55 smithi106 ceph-mon[36505]: pgmap v5744: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:55.588 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10644 sec 2024-03-24T03:21:56.588 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10645 sec 2024-03-24T03:21:57.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:57 smithi078 ceph-mon[32429]: pgmap v5745: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:21:57.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:57 smithi107 ceph-mon[36485]: pgmap v5745: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:21:57.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:57 smithi106 ceph-mon[36505]: pgmap v5745: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:21:57.588 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10646 sec 2024-03-24T03:21:58.588 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10647 sec 2024-03-24T03:21:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:59 smithi078 ceph-mon[32429]: pgmap v5746: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:21:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9993.246319 secs 2024-03-24T03:21:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:59 smithi107 ceph-mon[36485]: pgmap v5746: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:21:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9993.246319 secs 2024-03-24T03:21:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:59 smithi106 ceph-mon[36505]: pgmap v5746: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:21:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:21:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9993.246319 secs 2024-03-24T03:21:59.588 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10648 sec 2024-03-24T03:22:00.588 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10649 sec 2024-03-24T03:22:01.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:01 smithi078 ceph-mon[32429]: pgmap v5747: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:01 smithi107 ceph-mon[36485]: pgmap v5747: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:01.401 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:01 smithi106 ceph-mon[36505]: pgmap v5747: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:01.588 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10650 sec 2024-03-24T03:22:02.589 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10651 sec 2024-03-24T03:22:03.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:03 smithi078 ceph-mon[32429]: pgmap v5748: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:22:03.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:03 smithi107 ceph-mon[36485]: pgmap v5748: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:22:03.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:03 smithi106 ceph-mon[36505]: pgmap v5748: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:22:03.589 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10652 sec 2024-03-24T03:22:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 9998.246336 secs 2024-03-24T03:22:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 9998.246336 secs 2024-03-24T03:22:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 9998.246336 secs 2024-03-24T03:22:04.589 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10653 sec 2024-03-24T03:22:05.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:05 smithi078 ceph-mon[32429]: pgmap v5749: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:05.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:05 smithi107 ceph-mon[36485]: pgmap v5749: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:05.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:05 smithi106 ceph-mon[36505]: pgmap v5749: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:05.589 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10654 sec 2024-03-24T03:22:06.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:06 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:22:06.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:06 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:22:06.589 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10655 sec 2024-03-24T03:22:06.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:06 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:22:07.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:07 smithi106 ceph-mon[36505]: pgmap v5750: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:22:07.589 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10656 sec 2024-03-24T03:22:07.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:07 smithi078 ceph-mon[32429]: pgmap v5750: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:22:07.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:07 smithi107 ceph-mon[36485]: pgmap v5750: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:22:08.589 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10657 sec 2024-03-24T03:22:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:09 smithi107 ceph-mon[36485]: pgmap v5751: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10003.246419 secs 2024-03-24T03:22:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:09 smithi106 ceph-mon[36505]: pgmap v5751: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10003.246419 secs 2024-03-24T03:22:09.590 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10658 sec 2024-03-24T03:22:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:09 smithi078 ceph-mon[32429]: pgmap v5751: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10003.246419 secs 2024-03-24T03:22:10.589 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10659 sec 2024-03-24T03:22:11.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:10 smithi078 ceph-mon[32429]: pgmap v5752: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:11.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:22:11.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:22:11.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:22:11.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:22:11.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:10 smithi078 ceph-mon[32429]: pgmap v5753: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:22:11.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:10 smithi078 ceph-mon[32429]: pgmap v5754: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T03:22:11.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:22:11.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:10 smithi107 ceph-mon[36485]: pgmap v5752: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:11.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:22:11.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:22:11.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:22:11.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:22:11.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:10 smithi107 ceph-mon[36485]: pgmap v5753: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:22:11.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:10 smithi107 ceph-mon[36485]: pgmap v5754: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T03:22:11.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:22:11.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:10 smithi106 ceph-mon[36505]: pgmap v5752: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:11.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:22:11.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:22:11.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:22:11.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:22:11.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:10 smithi106 ceph-mon[36505]: pgmap v5753: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:22:11.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:10 smithi106 ceph-mon[36505]: pgmap v5754: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T03:22:11.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:22:11.589 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10660 sec 2024-03-24T03:22:12.589 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10661 sec 2024-03-24T03:22:13.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:12 smithi106 ceph-mon[36505]: pgmap v5755: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:22:13.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:12 smithi078 ceph-mon[32429]: pgmap v5755: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:22:13.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:12 smithi107 ceph-mon[36485]: pgmap v5755: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:22:13.589 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10662 sec 2024-03-24T03:22:14.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10008.246504 secs 2024-03-24T03:22:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10008.246504 secs 2024-03-24T03:22:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10008.246504 secs 2024-03-24T03:22:14.590 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10663 sec 2024-03-24T03:22:15.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:14 smithi106 ceph-mon[36505]: pgmap v5756: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T03:22:15.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:14 smithi078 ceph-mon[32429]: pgmap v5756: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T03:22:15.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:14 smithi107 ceph-mon[36485]: pgmap v5756: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T03:22:15.590 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10664 sec 2024-03-24T03:22:16.590 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10665 sec 2024-03-24T03:22:17.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:16 smithi106 ceph-mon[36505]: pgmap v5757: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:22:17.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:16 smithi078 ceph-mon[32429]: pgmap v5757: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:22:17.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:16 smithi107 ceph-mon[36485]: pgmap v5757: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:22:17.590 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10666 sec 2024-03-24T03:22:18.590 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10667 sec 2024-03-24T03:22:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:18 smithi106 ceph-mon[36505]: pgmap v5758: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:22:19.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10013.246592 secs 2024-03-24T03:22:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:18 smithi078 ceph-mon[32429]: pgmap v5758: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:22:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10013.246592 secs 2024-03-24T03:22:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:18 smithi107 ceph-mon[36485]: pgmap v5758: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:22:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10013.246592 secs 2024-03-24T03:22:19.590 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10668 sec 2024-03-24T03:22:20.590 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10669 sec 2024-03-24T03:22:21.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:20 smithi106 ceph-mon[36505]: pgmap v5759: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:22:21.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:20 smithi078 ceph-mon[32429]: pgmap v5759: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:22:21.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:20 smithi107 ceph-mon[36485]: pgmap v5759: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:22:21.590 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10670 sec 2024-03-24T03:22:22.590 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10671 sec 2024-03-24T03:22:23.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:22 smithi106 ceph-mon[36505]: pgmap v5760: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:22:23.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:22 smithi078 ceph-mon[32429]: pgmap v5760: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:22:23.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:22 smithi107 ceph-mon[36485]: pgmap v5760: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:22:23.590 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10672 sec 2024-03-24T03:22:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10018.246631 secs 2024-03-24T03:22:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10018.246631 secs 2024-03-24T03:22:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10018.246631 secs 2024-03-24T03:22:24.591 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10673 sec 2024-03-24T03:22:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:24 smithi078 ceph-mon[32429]: pgmap v5761: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:25.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:24 smithi107 ceph-mon[36485]: pgmap v5761: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:25.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:24 smithi106 ceph-mon[36505]: pgmap v5761: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:25.591 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10674 sec 2024-03-24T03:22:26.591 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10675 sec 2024-03-24T03:22:27.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:26 smithi078 ceph-mon[32429]: pgmap v5762: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:22:27.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:26 smithi107 ceph-mon[36485]: pgmap v5762: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:22:27.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:26 smithi106 ceph-mon[36505]: pgmap v5762: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:22:27.591 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10676 sec 2024-03-24T03:22:28.591 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10677 sec 2024-03-24T03:22:29.204 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:28 smithi106 ceph-mon[36505]: pgmap v5763: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:29.204 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10023.246691 secs 2024-03-24T03:22:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:28 smithi078 ceph-mon[32429]: pgmap v5763: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10023.246691 secs 2024-03-24T03:22:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:28 smithi107 ceph-mon[36485]: pgmap v5763: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10023.246691 secs 2024-03-24T03:22:29.591 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10678 sec 2024-03-24T03:22:30.591 INFO:tasks.workunit.client.0.smithi078.stdout: 1 cleanup 10679 sec 2024-03-24T03:22:31.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:30 smithi078 ceph-mon[32429]: pgmap v5764: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:31.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:30 smithi107 ceph-mon[36485]: pgmap v5764: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:31.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:30 smithi106 ceph-mon[36505]: pgmap v5764: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:31.535 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-24T03:22:31.535 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-03-24T03:22:31.536 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-24T03:22:31.818 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi078 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-03-24T03:22:32.037 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=11c22a897d5742f3b23baba2cdec83a6 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi078 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-03-24T03:22:32.040 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-03-24T03:22:32.076 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-03-24T03:22:32.076 INFO:teuthology.task.exec:Executing custom commands... 2024-03-24T03:22:32.076 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2024-03-24T03:22:32.076 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-03-24T03:22:32.109 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-03-24T03:22:32.178 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-24T03:22:32.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.322+0000 7fabb9e4b640 1 -- 172.21.15.78:0/237222364 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fabb4107660 msgr2=0x7fabb4107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:22:32.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.322+0000 7fabb9e4b640 1 --2- 172.21.15.78:0/237222364 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fabb4107660 0x7fabb4107a40 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7faba4009930 tx=0x7faba402bc30 comp rx=0 tx=0).stop 2024-03-24T03:22:32.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.322+0000 7fabb9e4b640 1 -- 172.21.15.78:0/237222364 shutdown_connections 2024-03-24T03:22:32.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.322+0000 7fabb9e4b640 1 --2- 172.21.15.78:0/237222364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fabb4101650 0x7fabb4101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:22:32.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.322+0000 7fabb9e4b640 1 --2- 172.21.15.78:0/237222364 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fabb4102010 0x7fabb410e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:22:32.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.322+0000 7fabb9e4b640 1 --2- 172.21.15.78:0/237222364 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fabb4107660 0x7fabb4107a40 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:22:32.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.322+0000 7fabb9e4b640 1 -- 172.21.15.78:0/237222364 >> 172.21.15.78:0/237222364 conn(0x7fabb40fd160 msgr2=0x7fabb40ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:22:32.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.322+0000 7fabb9e4b640 1 -- 172.21.15.78:0/237222364 shutdown_connections 2024-03-24T03:22:32.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.322+0000 7fabb9e4b640 1 -- 172.21.15.78:0/237222364 wait complete. 2024-03-24T03:22:32.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.323+0000 7fabb9e4b640 1 Processor -- start 2024-03-24T03:22:32.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.323+0000 7fabb9e4b640 1 -- start start 2024-03-24T03:22:32.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.323+0000 7fabb9e4b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fabb4101650 0x7fabb419b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:22:32.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.323+0000 7fabb9e4b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fabb4102010 0x7fabb419be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:22:32.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.323+0000 7fabb9e4b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fabb4107660 0x7fabb4195ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:22:32.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.323+0000 7fabb9e4b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb410f860 con 0x7fabb4101650 2024-03-24T03:22:32.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.323+0000 7fabb9e4b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb410f6e0 con 0x7fabb4102010 2024-03-24T03:22:32.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.323+0000 7fabb9e4b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb410f9e0 con 0x7fabb4107660 2024-03-24T03:22:32.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.323+0000 7fabb37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fabb4101650 0x7fabb419b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:22:32.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.323+0000 7fabb37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fabb4101650 0x7fabb419b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35848/0 (socket says 172.21.15.78:35848) 2024-03-24T03:22:32.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.323+0000 7fabb37fe640 1 -- 172.21.15.78:0/1058387750 learned_addr learned my addr 172.21.15.78:0/1058387750 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T03:22:32.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.323+0000 7fabb2ffd640 1 --2- 172.21.15.78:0/1058387750 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fabb4102010 0x7fabb419be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:22:32.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.324+0000 7fabb37fe640 1 -- 172.21.15.78:0/1058387750 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fabb4107660 msgr2=0x7fabb4195ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:22:32.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.324+0000 7fabb37fe640 1 --2- 172.21.15.78:0/1058387750 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fabb4107660 0x7fabb4195ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:22:32.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.324+0000 7fabb37fe640 1 -- 172.21.15.78:0/1058387750 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fabb4102010 msgr2=0x7fabb419be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:22:32.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.324+0000 7fabb37fe640 1 --2- 172.21.15.78:0/1058387750 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fabb4102010 0x7fabb419be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:22:32.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.324+0000 7fabb37fe640 1 -- 172.21.15.78:0/1058387750 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb4196240 con 0x7fabb4101650 2024-03-24T03:22:32.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.324+0000 7fabb37fe640 1 --2- 172.21.15.78:0/1058387750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fabb4101650 0x7fabb419b910 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7faba4009230 tx=0x7faba402cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:22:32.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.324+0000 7fabb0ff9640 1 -- 172.21.15.78:0/1058387750 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba4041070 con 0x7fabb4101650 2024-03-24T03:22:32.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.324+0000 7fabb0ff9640 1 -- 172.21.15.78:0/1058387750 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faba4004450 con 0x7fabb4101650 2024-03-24T03:22:32.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.324+0000 7fabb9e4b640 1 -- 172.21.15.78:0/1058387750 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabb41964d0 con 0x7fabb4101650 2024-03-24T03:22:32.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.324+0000 7fabb0ff9640 1 -- 172.21.15.78:0/1058387750 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba4034440 con 0x7fabb4101650 2024-03-24T03:22:32.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.324+0000 7fabb9e4b640 1 -- 172.21.15.78:0/1058387750 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabb41a27d0 con 0x7fabb4101650 2024-03-24T03:22:32.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.325+0000 7fabb9e4b640 1 -- 172.21.15.78:0/1058387750 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabb404f400 con 0x7fabb4101650 2024-03-24T03:22:32.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.326+0000 7fabb0ff9640 1 -- 172.21.15.78:0/1058387750 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7faba4044070 con 0x7fabb4101650 2024-03-24T03:22:32.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.326+0000 7fabb0ff9640 1 --2- 172.21.15.78:0/1058387750 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fab9c0a2a10 0x7fab9c0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:22:32.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.327+0000 7fabb2ffd640 1 --2- 172.21.15.78:0/1058387750 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fab9c0a2a10 0x7fab9c0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:22:32.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.327+0000 7fabb0ff9640 1 -- 172.21.15.78:0/1058387750 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 13278+0+0 (secure 0 0 0) 0x7faba40f1a20 con 0x7fabb4101650 2024-03-24T03:22:32.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.327+0000 7fabb2ffd640 1 --2- 172.21.15.78:0/1058387750 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fab9c0a2a10 0x7fab9c0a4ed0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fabb40ff700 tx=0x7faba800b650 comp rx=0 tx=0).ready entity=mgr.25243 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:22:32.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.331+0000 7fabb0ff9640 1 -- 172.21.15.78:0/1058387750 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7faba40bd600 con 0x7fabb4101650 2024-03-24T03:22:32.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:22:32.526+0000 7fabb9e4b640 1 -- 172.21.15.78:0/1058387750 --> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7fabb4061ae0 con 0x7fab9c0a2a10 2024-03-24T03:22:33.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:32 smithi078 ceph-mon[32429]: pgmap v5765: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:22:33.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:32 smithi107 ceph-mon[36485]: pgmap v5765: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:22:33.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:32 smithi106 ceph-mon[36505]: pgmap v5765: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:22:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:33 smithi078 ceph-mon[32429]: from='client.15753 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T03:22:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10028.246776 secs 2024-03-24T03:22:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:33 smithi107 ceph-mon[36485]: from='client.15753 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T03:22:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10028.246776 secs 2024-03-24T03:22:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:33 smithi106 ceph-mon[36505]: from='client.15753 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T03:22:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10028.246776 secs 2024-03-24T03:22:35.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:34 smithi078 ceph-mon[32429]: pgmap v5766: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:35.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:34 smithi107 ceph-mon[36485]: pgmap v5766: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:35.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:34 smithi106 ceph-mon[36505]: pgmap v5766: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:37.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:36 smithi078 ceph-mon[32429]: pgmap v5767: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-24T03:22:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:36 smithi107 ceph-mon[36485]: pgmap v5767: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-24T03:22:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:36 smithi106 ceph-mon[36505]: pgmap v5767: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-24T03:22:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:38 smithi078 ceph-mon[32429]: pgmap v5768: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T03:22:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10033.246865 secs 2024-03-24T03:22:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:38 smithi107 ceph-mon[36485]: pgmap v5768: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T03:22:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10033.246865 secs 2024-03-24T03:22:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:38 smithi106 ceph-mon[36505]: pgmap v5768: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T03:22:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10033.246865 secs 2024-03-24T03:22:41.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:40 smithi078 ceph-mon[32429]: pgmap v5769: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T03:22:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:40 smithi107 ceph-mon[36485]: pgmap v5769: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T03:22:41.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:40 smithi106 ceph-mon[36505]: pgmap v5769: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T03:22:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:42 smithi078 ceph-mon[32429]: pgmap v5770: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-24T03:22:43.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:42 smithi107 ceph-mon[36485]: pgmap v5770: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-24T03:22:43.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:42 smithi106 ceph-mon[36505]: pgmap v5770: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-24T03:22:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10038.246925 secs 2024-03-24T03:22:44.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10038.246925 secs 2024-03-24T03:22:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10038.246925 secs 2024-03-24T03:22:45.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:44 smithi078 ceph-mon[32429]: pgmap v5771: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T03:22:45.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:44 smithi107 ceph-mon[36485]: pgmap v5771: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T03:22:45.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:44 smithi106 ceph-mon[36505]: pgmap v5771: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T03:22:47.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:46 smithi078 ceph-mon[32429]: pgmap v5772: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-24T03:22:47.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:46 smithi107 ceph-mon[36485]: pgmap v5772: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-24T03:22:47.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:46 smithi106 ceph-mon[36505]: pgmap v5772: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-24T03:22:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:48 smithi078 ceph-mon[32429]: pgmap v5773: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:49.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10043.247013 secs 2024-03-24T03:22:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:48 smithi107 ceph-mon[36485]: pgmap v5773: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10043.247013 secs 2024-03-24T03:22:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:48 smithi106 ceph-mon[36505]: pgmap v5773: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10043.247013 secs 2024-03-24T03:22:51.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:50 smithi078 ceph-mon[32429]: pgmap v5774: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:51.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:50 smithi107 ceph-mon[36485]: pgmap v5774: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:51.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:50 smithi106 ceph-mon[36505]: pgmap v5774: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:53.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:52 smithi078 ceph-mon[32429]: pgmap v5775: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:22:53.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:52 smithi107 ceph-mon[36485]: pgmap v5775: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:22:53.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:52 smithi106 ceph-mon[36505]: pgmap v5775: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:22:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10048.247072 secs 2024-03-24T03:22:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10048.247072 secs 2024-03-24T03:22:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10048.247072 secs 2024-03-24T03:22:55.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:54 smithi078 ceph-mon[32429]: pgmap v5776: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:55.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:54 smithi107 ceph-mon[36485]: pgmap v5776: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:55.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:54 smithi106 ceph-mon[36505]: pgmap v5776: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:57.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:56 smithi078 ceph-mon[32429]: pgmap v5777: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:22:57.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:56 smithi107 ceph-mon[36485]: pgmap v5777: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:22:57.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:56 smithi106 ceph-mon[36505]: pgmap v5777: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:22:59.203 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:58 smithi106 ceph-mon[36505]: pgmap v5778: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:59.204 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:22:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10053.247157 secs 2024-03-24T03:22:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:58 smithi078 ceph-mon[32429]: pgmap v5778: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:22:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10053.247157 secs 2024-03-24T03:22:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:58 smithi107 ceph-mon[36485]: pgmap v5778: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:22:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:22:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10053.247157 secs 2024-03-24T03:23:01.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:00 smithi078 ceph-mon[32429]: pgmap v5779: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:00 smithi107 ceph-mon[36485]: pgmap v5779: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:00 smithi106 ceph-mon[36505]: pgmap v5779: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:03.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:02 smithi078 ceph-mon[32429]: pgmap v5780: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:23:03.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:02 smithi107 ceph-mon[36485]: pgmap v5780: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:23:03.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:02 smithi106 ceph-mon[36505]: pgmap v5780: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:23:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10058.247218 secs 2024-03-24T03:23:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10058.247218 secs 2024-03-24T03:23:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10058.247218 secs 2024-03-24T03:23:05.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:04 smithi078 ceph-mon[32429]: pgmap v5781: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:05.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:04 smithi107 ceph-mon[36485]: pgmap v5781: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:05.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:04 smithi106 ceph-mon[36505]: pgmap v5781: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:07.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:06 smithi078 ceph-mon[32429]: pgmap v5782: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:23:07.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:06 smithi107 ceph-mon[36485]: pgmap v5782: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:23:07.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:06 smithi106 ceph-mon[36505]: pgmap v5782: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:23:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:08 smithi078 ceph-mon[32429]: pgmap v5783: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:09.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10063.247306 secs 2024-03-24T03:23:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:08 smithi107 ceph-mon[36485]: pgmap v5783: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10063.247306 secs 2024-03-24T03:23:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:08 smithi106 ceph-mon[36505]: pgmap v5783: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10063.247306 secs 2024-03-24T03:23:11.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:10 smithi078 ceph-mon[32429]: pgmap v5784: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:11.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:23:11.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:23:11.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:23:11.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:10 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:23:11.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:10 smithi107 ceph-mon[36485]: pgmap v5784: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:11.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:23:11.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:23:11.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:23:11.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:10 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:23:11.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:10 smithi106 ceph-mon[36505]: pgmap v5784: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:11.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:23:11.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:23:11.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:23:11.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:10 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:23:12.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:11 smithi078 ceph-mon[32429]: pgmap v5785: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:23:12.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:11 smithi078 ceph-mon[32429]: pgmap v5786: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:23:12.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:11 smithi107 ceph-mon[36485]: pgmap v5785: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:23:12.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:11 smithi107 ceph-mon[36485]: pgmap v5786: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:23:12.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:11 smithi106 ceph-mon[36505]: pgmap v5785: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:23:12.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:11 smithi106 ceph-mon[36505]: pgmap v5786: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:23:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:13 smithi078 ceph-mon[32429]: pgmap v5787: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:23:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10068.247390 secs 2024-03-24T03:23:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:13 smithi107 ceph-mon[36485]: pgmap v5787: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:23:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10068.247390 secs 2024-03-24T03:23:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:13 smithi106 ceph-mon[36505]: pgmap v5787: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:23:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10068.247390 secs 2024-03-24T03:23:16.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:15 smithi078 ceph-mon[32429]: pgmap v5788: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:23:16.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:15 smithi107 ceph-mon[36485]: pgmap v5788: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:23:16.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:15 smithi106 ceph-mon[36505]: pgmap v5788: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:23:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:17 smithi078 ceph-mon[32429]: pgmap v5789: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:23:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:17 smithi107 ceph-mon[36485]: pgmap v5789: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:23:18.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:17 smithi106 ceph-mon[36505]: pgmap v5789: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:23:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10073.247475 secs 2024-03-24T03:23:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10073.247475 secs 2024-03-24T03:23:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10073.247475 secs 2024-03-24T03:23:20.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:20 smithi078 ceph-mon[32429]: pgmap v5790: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:23:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:20 smithi107 ceph-mon[36485]: pgmap v5790: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:23:20.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:20 smithi106 ceph-mon[36505]: pgmap v5790: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:23:22.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:22 smithi078 ceph-mon[32429]: pgmap v5791: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:23:22.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:22 smithi107 ceph-mon[36485]: pgmap v5791: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:23:22.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:22 smithi106 ceph-mon[36505]: pgmap v5791: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:23:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:24 smithi078 ceph-mon[32429]: pgmap v5792: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10078.247535 secs 2024-03-24T03:23:24.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:24 smithi107 ceph-mon[36485]: pgmap v5792: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10078.247535 secs 2024-03-24T03:23:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:24 smithi106 ceph-mon[36505]: pgmap v5792: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10078.247535 secs 2024-03-24T03:23:26.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:26 smithi078 ceph-mon[32429]: pgmap v5793: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:26.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:26 smithi107 ceph-mon[36485]: pgmap v5793: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:26.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:26 smithi106 ceph-mon[36505]: pgmap v5793: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:28.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:28 smithi078 ceph-mon[32429]: pgmap v5794: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:23:28.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:28 smithi107 ceph-mon[36485]: pgmap v5794: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:23:28.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:28 smithi106 ceph-mon[36505]: pgmap v5794: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:23:29.302 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10083.247550 secs 2024-03-24T03:23:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10083.247550 secs 2024-03-24T03:23:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10083.247550 secs 2024-03-24T03:23:30.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:30 smithi078 ceph-mon[32429]: pgmap v5795: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:30.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:30 smithi107 ceph-mon[36485]: pgmap v5795: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:30.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:30 smithi106 ceph-mon[36505]: pgmap v5795: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:32.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:32 smithi078 ceph-mon[32429]: pgmap v5796: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:23:32.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:32 smithi107 ceph-mon[36485]: pgmap v5796: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:23:32.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:32 smithi106 ceph-mon[36505]: pgmap v5796: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:23:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:34 smithi078 ceph-mon[32429]: pgmap v5797: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10088.247633 secs 2024-03-24T03:23:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:34 smithi107 ceph-mon[36485]: pgmap v5797: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10088.247633 secs 2024-03-24T03:23:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:34 smithi106 ceph-mon[36505]: pgmap v5797: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10088.247633 secs 2024-03-24T03:23:36.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:36 smithi078 ceph-mon[32429]: pgmap v5798: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:36.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:36 smithi107 ceph-mon[36485]: pgmap v5798: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:36.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:36 smithi106 ceph-mon[36505]: pgmap v5798: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:38.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:38 smithi078 ceph-mon[32429]: pgmap v5799: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:23:38.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:38 smithi107 ceph-mon[36485]: pgmap v5799: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:23:38.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:38 smithi106 ceph-mon[36505]: pgmap v5799: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:23:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10093.247671 secs 2024-03-24T03:23:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10093.247671 secs 2024-03-24T03:23:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10093.247671 secs 2024-03-24T03:23:40.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:40 smithi078 ceph-mon[32429]: pgmap v5800: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:40.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:40 smithi107 ceph-mon[36485]: pgmap v5800: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:40.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:40 smithi106 ceph-mon[36505]: pgmap v5800: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:42.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:42 smithi078 ceph-mon[32429]: pgmap v5801: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:23:42.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:42 smithi107 ceph-mon[36485]: pgmap v5801: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:23:42.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:42 smithi106 ceph-mon[36505]: pgmap v5801: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:23:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:44 smithi078 ceph-mon[32429]: pgmap v5802: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10098.247730 secs 2024-03-24T03:23:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:44 smithi107 ceph-mon[36485]: pgmap v5802: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10098.247730 secs 2024-03-24T03:23:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:44 smithi106 ceph-mon[36505]: pgmap v5802: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10098.247730 secs 2024-03-24T03:23:46.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:46 smithi078 ceph-mon[32429]: pgmap v5803: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:46.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:46 smithi107 ceph-mon[36485]: pgmap v5803: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:46.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:46 smithi106 ceph-mon[36505]: pgmap v5803: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:48.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:48 smithi078 ceph-mon[32429]: pgmap v5804: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:23:48.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:48 smithi107 ceph-mon[36485]: pgmap v5804: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:23:48.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:48 smithi106 ceph-mon[36505]: pgmap v5804: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:23:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10103.247817 secs 2024-03-24T03:23:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10103.247817 secs 2024-03-24T03:23:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10103.247817 secs 2024-03-24T03:23:50.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:50 smithi078 ceph-mon[32429]: pgmap v5805: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:50 smithi107 ceph-mon[36485]: pgmap v5805: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:50.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:50 smithi106 ceph-mon[36505]: pgmap v5805: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:52 smithi078 ceph-mon[32429]: pgmap v5806: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:23:52.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:52 smithi107 ceph-mon[36485]: pgmap v5806: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:23:52.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:52 smithi106 ceph-mon[36505]: pgmap v5806: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:23:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:54 smithi078 ceph-mon[32429]: pgmap v5807: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10108.247904 secs 2024-03-24T03:23:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:54 smithi107 ceph-mon[36485]: pgmap v5807: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10108.247904 secs 2024-03-24T03:23:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:54 smithi106 ceph-mon[36505]: pgmap v5807: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10108.247904 secs 2024-03-24T03:23:56.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:56 smithi078 ceph-mon[32429]: pgmap v5808: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:56.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:56 smithi107 ceph-mon[36485]: pgmap v5808: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:56.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:56 smithi106 ceph-mon[36505]: pgmap v5808: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:23:58.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:58 smithi078 ceph-mon[32429]: pgmap v5809: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:23:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:58 smithi107 ceph-mon[36485]: pgmap v5809: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:23:58.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:58 smithi106 ceph-mon[36505]: pgmap v5809: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:23:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:23:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10113.247966 secs 2024-03-24T03:23:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:23:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10113.247966 secs 2024-03-24T03:23:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:23:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10113.247966 secs 2024-03-24T03:24:00.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:00 smithi078 ceph-mon[32429]: pgmap v5810: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:24:00.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:00 smithi107 ceph-mon[36485]: pgmap v5810: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:24:00.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:00 smithi106 ceph-mon[36505]: pgmap v5810: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:24:02.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:02 smithi078 ceph-mon[32429]: pgmap v5811: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:24:02.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:02 smithi107 ceph-mon[36485]: pgmap v5811: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:24:02.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:02 smithi106 ceph-mon[36505]: pgmap v5811: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:24:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:04 smithi078 ceph-mon[32429]: pgmap v5812: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:24:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10118.248053 secs 2024-03-24T03:24:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:04 smithi107 ceph-mon[36485]: pgmap v5812: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:24:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10118.248053 secs 2024-03-24T03:24:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:04 smithi106 ceph-mon[36505]: pgmap v5812: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:24:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10118.248053 secs 2024-03-24T03:24:06.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:06 smithi107 ceph-mon[36485]: pgmap v5813: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:24:06.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:06 smithi106 ceph-mon[36505]: pgmap v5813: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:24:06.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:06 smithi078 ceph-mon[32429]: pgmap v5813: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:24:08.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:08 smithi106 ceph-mon[36505]: pgmap v5814: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:24:08.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:08 smithi078 ceph-mon[32429]: pgmap v5814: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:24:08.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:08 smithi107 ceph-mon[36485]: pgmap v5814: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:24:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10123.248141 secs 2024-03-24T03:24:09.494 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10123.248141 secs 2024-03-24T03:24:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10123.248141 secs 2024-03-24T03:24:10.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:10 smithi106 ceph-mon[36505]: pgmap v5815: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:24:10.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:10 smithi078 ceph-mon[32429]: pgmap v5815: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:24:10.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:10 smithi107 ceph-mon[36485]: pgmap v5815: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:24:11.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:11 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:24:11.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:11 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:24:11.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:11 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:24:12.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:12 smithi106 ceph-mon[36505]: pgmap v5816: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:24:12.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:12 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:24:12.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:12 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:24:12.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:12 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:24:12.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:12 smithi078 ceph-mon[32429]: pgmap v5816: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:24:12.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:12 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:24:12.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:12 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:24:12.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:12 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:24:12.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:12 smithi107 ceph-mon[36485]: pgmap v5816: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:24:12.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:12 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:24:12.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:12 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:24:12.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:12 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:24:13.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:13 smithi106 ceph-mon[36505]: pgmap v5817: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:24:13.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:13 smithi106 ceph-mon[36505]: pgmap v5818: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:24:13.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:13 smithi078 ceph-mon[32429]: pgmap v5817: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:24:13.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:13 smithi078 ceph-mon[32429]: pgmap v5818: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:24:13.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:13 smithi107 ceph-mon[36485]: pgmap v5817: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:24:13.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:13 smithi107 ceph-mon[36485]: pgmap v5818: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:24:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10128.248226 secs 2024-03-24T03:24:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10128.248226 secs 2024-03-24T03:24:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10128.248226 secs 2024-03-24T03:24:15.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:15 smithi106 ceph-mon[36505]: pgmap v5819: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:24:15.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:15 smithi078 ceph-mon[32429]: pgmap v5819: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:24:15.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:15 smithi107 ceph-mon[36485]: pgmap v5819: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:24:17.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:17 smithi106 ceph-mon[36505]: pgmap v5820: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:24:17.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:17 smithi078 ceph-mon[32429]: pgmap v5820: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:24:17.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:17 smithi107 ceph-mon[36485]: pgmap v5820: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:24:19.393 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:19 smithi078 ceph-mon[32429]: pgmap v5821: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:24:19.393 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10133.248279 secs 2024-03-24T03:24:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:19 smithi106 ceph-mon[36505]: pgmap v5821: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:24:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10133.248279 secs 2024-03-24T03:24:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:19 smithi107 ceph-mon[36485]: pgmap v5821: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:24:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10133.248279 secs 2024-03-24T03:24:21.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:21 smithi106 ceph-mon[36505]: pgmap v5822: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-24T03:24:21.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:21 smithi078 ceph-mon[32429]: pgmap v5822: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-24T03:24:21.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:21 smithi107 ceph-mon[36485]: pgmap v5822: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-24T03:24:23.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:23 smithi106 ceph-mon[36505]: pgmap v5823: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:24:23.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:23 smithi078 ceph-mon[32429]: pgmap v5823: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:24:23.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:23 smithi107 ceph-mon[36485]: pgmap v5823: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:24:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10138.248316 secs 2024-03-24T03:24:24.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10138.248316 secs 2024-03-24T03:24:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10138.248316 secs 2024-03-24T03:24:25.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:25 smithi106 ceph-mon[36505]: pgmap v5824: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T03:24:25.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:25 smithi078 ceph-mon[32429]: pgmap v5824: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T03:24:25.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:25 smithi107 ceph-mon[36485]: pgmap v5824: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T03:24:27.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:27 smithi106 ceph-mon[36505]: pgmap v5825: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:24:27.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:27 smithi078 ceph-mon[32429]: pgmap v5825: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:24:27.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:27 smithi107 ceph-mon[36485]: pgmap v5825: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:24:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:29 smithi106 ceph-mon[36505]: pgmap v5826: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:24:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10143.248380 secs 2024-03-24T03:24:29.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:29 smithi078 ceph-mon[32429]: pgmap v5826: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:24:29.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10143.248380 secs 2024-03-24T03:24:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:29 smithi107 ceph-mon[36485]: pgmap v5826: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:24:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10143.248380 secs 2024-03-24T03:24:31.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:31 smithi106 ceph-mon[36505]: pgmap v5827: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:24:31.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:31 smithi078 ceph-mon[32429]: pgmap v5827: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:24:31.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:31 smithi107 ceph-mon[36485]: pgmap v5827: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:24:33.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:33 smithi078 ceph-mon[32429]: pgmap v5828: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:24:33.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:33 smithi107 ceph-mon[36485]: pgmap v5828: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:24:33.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:33 smithi106 ceph-mon[36505]: pgmap v5828: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:24:34.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10148.248438 secs 2024-03-24T03:24:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10148.248438 secs 2024-03-24T03:24:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10148.248438 secs 2024-03-24T03:24:35.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:35 smithi078 ceph-mon[32429]: pgmap v5829: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:24:35.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:35 smithi107 ceph-mon[36485]: pgmap v5829: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:24:35.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:35 smithi106 ceph-mon[36505]: pgmap v5829: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:24:37.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:37 smithi078 ceph-mon[32429]: pgmap v5830: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:24:37.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:37 smithi107 ceph-mon[36485]: pgmap v5830: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:24:37.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:37 smithi106 ceph-mon[36505]: pgmap v5830: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:24:39.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:39 smithi078 ceph-mon[32429]: pgmap v5831: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T03:24:39.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10153.248499 secs 2024-03-24T03:24:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:39 smithi107 ceph-mon[36485]: pgmap v5831: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T03:24:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10153.248499 secs 2024-03-24T03:24:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:39 smithi106 ceph-mon[36505]: pgmap v5831: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T03:24:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10153.248499 secs 2024-03-24T03:24:41.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:41 smithi078 ceph-mon[32429]: pgmap v5832: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T03:24:41.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:41 smithi107 ceph-mon[36485]: pgmap v5832: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T03:24:41.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:41 smithi106 ceph-mon[36505]: pgmap v5832: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T03:24:43.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:43 smithi078 ceph-mon[32429]: pgmap v5833: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:24:43.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:43 smithi107 ceph-mon[36485]: pgmap v5833: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:24:43.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:43 smithi106 ceph-mon[36505]: pgmap v5833: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:24:44.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10158.248602 secs 2024-03-24T03:24:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10158.248602 secs 2024-03-24T03:24:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10158.248602 secs 2024-03-24T03:24:45.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:45 smithi078 ceph-mon[32429]: pgmap v5834: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T03:24:45.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:45 smithi107 ceph-mon[36485]: pgmap v5834: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T03:24:45.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:45 smithi106 ceph-mon[36505]: pgmap v5834: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T03:24:47.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:47 smithi078 ceph-mon[32429]: pgmap v5835: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:24:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:47 smithi107 ceph-mon[36485]: pgmap v5835: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:24:47.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:47 smithi106 ceph-mon[36505]: pgmap v5835: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:24:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:49 smithi078 ceph-mon[32429]: pgmap v5836: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:24:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10163.248691 secs 2024-03-24T03:24:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:49 smithi107 ceph-mon[36485]: pgmap v5836: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:24:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10163.248691 secs 2024-03-24T03:24:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:49 smithi106 ceph-mon[36505]: pgmap v5836: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:24:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10163.248691 secs 2024-03-24T03:24:51.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:51 smithi078 ceph-mon[32429]: pgmap v5837: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:24:51.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:51 smithi107 ceph-mon[36485]: pgmap v5837: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:24:51.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:51 smithi106 ceph-mon[36505]: pgmap v5837: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:24:53.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:53 smithi078 ceph-mon[32429]: pgmap v5838: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:24:53.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:53 smithi107 ceph-mon[36485]: pgmap v5838: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:24:53.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:53 smithi106 ceph-mon[36505]: pgmap v5838: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:24:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10168.248750 secs 2024-03-24T03:24:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10168.248750 secs 2024-03-24T03:24:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10168.248750 secs 2024-03-24T03:24:55.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:55 smithi078 ceph-mon[32429]: pgmap v5839: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:24:55.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:55 smithi107 ceph-mon[36485]: pgmap v5839: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:24:55.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:55 smithi106 ceph-mon[36505]: pgmap v5839: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:24:57.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:57 smithi078 ceph-mon[32429]: pgmap v5840: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:24:57.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:57 smithi107 ceph-mon[36485]: pgmap v5840: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:24:57.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:57 smithi106 ceph-mon[36505]: pgmap v5840: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:24:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:59 smithi078 ceph-mon[32429]: pgmap v5841: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:24:59.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:24:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10173.248798 secs 2024-03-24T03:24:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:59 smithi107 ceph-mon[36485]: pgmap v5841: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:24:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:24:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10173.248798 secs 2024-03-24T03:24:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:59 smithi106 ceph-mon[36505]: pgmap v5841: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:24:59.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:24:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10173.248798 secs 2024-03-24T03:25:01.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:01 smithi078 ceph-mon[32429]: pgmap v5842: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:01.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:01 smithi107 ceph-mon[36485]: pgmap v5842: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:01.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:01 smithi106 ceph-mon[36505]: pgmap v5842: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:03.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:03 smithi078 ceph-mon[32429]: pgmap v5843: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:25:03.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:03 smithi107 ceph-mon[36485]: pgmap v5843: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:25:03.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:03 smithi106 ceph-mon[36505]: pgmap v5843: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:25:04.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10178.248853 secs 2024-03-24T03:25:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10178.248853 secs 2024-03-24T03:25:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10178.248853 secs 2024-03-24T03:25:05.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:05 smithi078 ceph-mon[32429]: pgmap v5844: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:05.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:05 smithi107 ceph-mon[36485]: pgmap v5844: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:05.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:05 smithi106 ceph-mon[36505]: pgmap v5844: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:06.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:06 smithi078 ceph-mon[32429]: pgmap v5845: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:25:06.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:06 smithi107 ceph-mon[36485]: pgmap v5845: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:25:06.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:06 smithi106 ceph-mon[36505]: pgmap v5845: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:25:08.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:08 smithi078 ceph-mon[32429]: pgmap v5846: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:08.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:08 smithi107 ceph-mon[36485]: pgmap v5846: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:08.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:08 smithi106 ceph-mon[36505]: pgmap v5846: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10183.248940 secs 2024-03-24T03:25:09.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10183.248940 secs 2024-03-24T03:25:09.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10183.248940 secs 2024-03-24T03:25:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:10 smithi078 ceph-mon[32429]: pgmap v5847: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:10.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:10 smithi107 ceph-mon[36485]: pgmap v5847: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:10.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:10 smithi106 ceph-mon[36505]: pgmap v5847: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:12.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:12 smithi106 ceph-mon[36505]: pgmap v5848: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:25:12.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:12 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:25:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:12 smithi078 ceph-mon[32429]: pgmap v5848: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:25:12.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:12 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:25:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:12 smithi107 ceph-mon[36485]: pgmap v5848: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:25:12.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:12 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:25:14.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10188.248983 secs 2024-03-24T03:25:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:14 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:14 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:14 smithi078 ceph-mon[32429]: pgmap v5849: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:14 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:14 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:14.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:14 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:14.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:14 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10188.248983 secs 2024-03-24T03:25:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:14 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:14 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:14 smithi107 ceph-mon[36485]: pgmap v5849: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:14 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:14 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:14 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:14.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:14 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10188.248983 secs 2024-03-24T03:25:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:14 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:14.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:14 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:14.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:14 smithi106 ceph-mon[36505]: pgmap v5849: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:14.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:14 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:14.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:14 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:14.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:14 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:14.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:14 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:15.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:15.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:15.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:15 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T03:25:15.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:15 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T03:25:15.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:15.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:15.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:15 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T03:25:15.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:15 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T03:25:15.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:15.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:15.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:15 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T03:25:15.844 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:15 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T03:25:15.844 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:25:15.844 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:25:15.844 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:15 smithi078 ceph-mon[32429]: pgmap v5850: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:15.844 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:15 smithi078 ceph-mon[32429]: pgmap v5851: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-24T03:25:15.844 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:15.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:15.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:15.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:15 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T03:25:15.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:15 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T03:25:15.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:15.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:15.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:15 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T03:25:15.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:15 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T03:25:15.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:15.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:15.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:15 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T03:25:15.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:15 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T03:25:15.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:25:15.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:25:15.848 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:15 smithi107 ceph-mon[36485]: pgmap v5850: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:15.848 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:15 smithi107 ceph-mon[36485]: pgmap v5851: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-24T03:25:15.848 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:15.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:15 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T03:25:15.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:15 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T03:25:15.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:15.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:15.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:15 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T03:25:15.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:15 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T03:25:15.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:15.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:15.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.905 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:25:15.905 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:15 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T03:25:15.905 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:15 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T03:25:15.905 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:25:15.905 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:25:15.905 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:15 smithi106 ceph-mon[36505]: pgmap v5850: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:15.905 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:15 smithi106 ceph-mon[36505]: pgmap v5851: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-24T03:25:15.905 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:25:18.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:18 smithi078 ceph-mon[32429]: pgmap v5852: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:25:18.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:18 smithi107 ceph-mon[36485]: pgmap v5852: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:25:18.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:18 smithi106 ceph-mon[36505]: pgmap v5852: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:25:19.492 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10193.249070 secs 2024-03-24T03:25:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10193.249070 secs 2024-03-24T03:25:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10193.249070 secs 2024-03-24T03:25:20.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:20 smithi078 ceph-mon[32429]: pgmap v5853: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:25:20.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:20 smithi107 ceph-mon[36485]: pgmap v5853: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:25:20.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:20 smithi106 ceph-mon[36505]: pgmap v5853: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:25:22.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:22 smithi078 ceph-mon[32429]: pgmap v5854: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:25:22.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:22 smithi107 ceph-mon[36485]: pgmap v5854: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:25:22.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:22 smithi106 ceph-mon[36505]: pgmap v5854: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:25:24.476 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10198.249127 secs 2024-03-24T03:25:24.476 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:24 smithi078 ceph-mon[32429]: pgmap v5855: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:25:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10198.249127 secs 2024-03-24T03:25:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:24 smithi107 ceph-mon[36485]: pgmap v5855: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:25:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10198.249127 secs 2024-03-24T03:25:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:24 smithi106 ceph-mon[36505]: pgmap v5855: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:25:26.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:26 smithi078 ceph-mon[32429]: pgmap v5856: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:25:26.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:26 smithi107 ceph-mon[36485]: pgmap v5856: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:25:26.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:26 smithi106 ceph-mon[36505]: pgmap v5856: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:25:28.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:28 smithi078 ceph-mon[32429]: pgmap v5857: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:25:28.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:28 smithi107 ceph-mon[36485]: pgmap v5857: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:25:28.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:28 smithi106 ceph-mon[36505]: pgmap v5857: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:25:29.552 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10203.249170 secs 2024-03-24T03:25:29.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10203.249170 secs 2024-03-24T03:25:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10203.249170 secs 2024-03-24T03:25:30.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:30 smithi078 ceph-mon[32429]: pgmap v5858: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:30.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:30 smithi107 ceph-mon[36485]: pgmap v5858: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:30.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:30 smithi106 ceph-mon[36505]: pgmap v5858: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:32.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:25:32.348+0000 7fab9b7fe640 1 -- 172.21.15.78:0/1058387750 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab940034c0 con 0x7fabb4101650 2024-03-24T03:25:32.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:25:32.348+0000 7fabb0ff9640 1 -- 172.21.15.78:0/1058387750 <== mon.0 v2:172.21.15.78:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7fab940034c0 con 0x7fabb4101650 2024-03-24T03:25:32.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:32 smithi078 ceph-mon[32429]: pgmap v5859: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:25:32.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:32 smithi107 ceph-mon[36485]: pgmap v5859: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:25:32.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:32 smithi106 ceph-mon[36505]: pgmap v5859: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:25:34.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10208.249246 secs 2024-03-24T03:25:34.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:34 smithi078 ceph-mon[32429]: pgmap v5860: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10208.249246 secs 2024-03-24T03:25:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:34 smithi107 ceph-mon[36485]: pgmap v5860: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10208.249246 secs 2024-03-24T03:25:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:34 smithi106 ceph-mon[36505]: pgmap v5860: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:36.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:36 smithi078 ceph-mon[32429]: pgmap v5861: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:36.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:36 smithi107 ceph-mon[36485]: pgmap v5861: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:36.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:36 smithi106 ceph-mon[36505]: pgmap v5861: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:38.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:38 smithi078 ceph-mon[32429]: pgmap v5862: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:25:38.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:38 smithi107 ceph-mon[36485]: pgmap v5862: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:25:38.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:38 smithi106 ceph-mon[36505]: pgmap v5862: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:25:39.494 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10213.249338 secs 2024-03-24T03:25:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10213.249338 secs 2024-03-24T03:25:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10213.249338 secs 2024-03-24T03:25:40.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:40 smithi078 ceph-mon[32429]: pgmap v5863: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:40.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:40 smithi107 ceph-mon[36485]: pgmap v5863: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:40.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:40 smithi106 ceph-mon[36505]: pgmap v5863: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:42.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:42 smithi078 ceph-mon[32429]: pgmap v5864: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:25:42.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:42 smithi107 ceph-mon[36485]: pgmap v5864: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:25:42.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:42 smithi106 ceph-mon[36505]: pgmap v5864: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:25:44.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10218.249395 secs 2024-03-24T03:25:44.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:44 smithi078 ceph-mon[32429]: pgmap v5865: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10218.249395 secs 2024-03-24T03:25:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:44 smithi107 ceph-mon[36485]: pgmap v5865: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10218.249395 secs 2024-03-24T03:25:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:44 smithi106 ceph-mon[36505]: pgmap v5865: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:46.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:46 smithi078 ceph-mon[32429]: pgmap v5866: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:46.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:46 smithi107 ceph-mon[36485]: pgmap v5866: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:46 smithi106 ceph-mon[36505]: pgmap v5866: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:48.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:48 smithi078 ceph-mon[32429]: pgmap v5867: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:25:48.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:48 smithi107 ceph-mon[36485]: pgmap v5867: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:25:48.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:48 smithi106 ceph-mon[36505]: pgmap v5867: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:25:49.491 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10223.249479 secs 2024-03-24T03:25:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10223.249479 secs 2024-03-24T03:25:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10223.249479 secs 2024-03-24T03:25:50.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:50 smithi078 ceph-mon[32429]: pgmap v5868: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:50.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:50 smithi107 ceph-mon[36485]: pgmap v5868: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:50.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:50 smithi106 ceph-mon[36505]: pgmap v5868: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:51.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:51 smithi078 ceph-mon[32429]: pgmap v5869: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:25:51.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:51 smithi107 ceph-mon[36485]: pgmap v5869: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:25:51.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:51 smithi106 ceph-mon[36505]: pgmap v5869: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:25:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10228.249534 secs 2024-03-24T03:25:54.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:54 smithi078 ceph-mon[32429]: pgmap v5870: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10228.249534 secs 2024-03-24T03:25:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:54 smithi107 ceph-mon[36485]: pgmap v5870: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10228.249534 secs 2024-03-24T03:25:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:54 smithi106 ceph-mon[36505]: pgmap v5870: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:56.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:56 smithi078 ceph-mon[32429]: pgmap v5871: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:56.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:56 smithi107 ceph-mon[36485]: pgmap v5871: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:56 smithi106 ceph-mon[36505]: pgmap v5871: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:25:58.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:58 smithi078 ceph-mon[32429]: pgmap v5872: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:25:58.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:58 smithi107 ceph-mon[36485]: pgmap v5872: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:25:58.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:58 smithi106 ceph-mon[36505]: pgmap v5872: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:25:59.552 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:25:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10233.249596 secs 2024-03-24T03:25:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:25:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10233.249596 secs 2024-03-24T03:25:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:25:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10233.249596 secs 2024-03-24T03:26:00.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:00 smithi078 ceph-mon[32429]: pgmap v5873: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:00.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:00 smithi107 ceph-mon[36485]: pgmap v5873: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:00.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:00 smithi106 ceph-mon[36505]: pgmap v5873: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:02.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:02 smithi078 ceph-mon[32429]: pgmap v5874: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:26:02.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:02 smithi107 ceph-mon[36485]: pgmap v5874: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:26:02.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:02 smithi106 ceph-mon[36505]: pgmap v5874: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:26:04.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10238.249635 secs 2024-03-24T03:26:04.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:04 smithi078 ceph-mon[32429]: pgmap v5875: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10238.249635 secs 2024-03-24T03:26:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:04 smithi107 ceph-mon[36485]: pgmap v5875: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10238.249635 secs 2024-03-24T03:26:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:04 smithi106 ceph-mon[36505]: pgmap v5875: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:06.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:06 smithi078 ceph-mon[32429]: pgmap v5876: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:06.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:06 smithi107 ceph-mon[36485]: pgmap v5876: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:06.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:06 smithi106 ceph-mon[36505]: pgmap v5876: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:08.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:08 smithi078 ceph-mon[32429]: pgmap v5877: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:26:08.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:08 smithi107 ceph-mon[36485]: pgmap v5877: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:26:08.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:08 smithi106 ceph-mon[36505]: pgmap v5877: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:26:09.494 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10243.249647 secs 2024-03-24T03:26:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10243.249647 secs 2024-03-24T03:26:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10243.249647 secs 2024-03-24T03:26:10.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:10 smithi078 ceph-mon[32429]: pgmap v5878: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:10.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:10 smithi107 ceph-mon[36485]: pgmap v5878: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:10.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:10 smithi106 ceph-mon[36505]: pgmap v5878: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:12.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:12 smithi078 ceph-mon[32429]: pgmap v5879: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:26:12.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:12 smithi107 ceph-mon[36485]: pgmap v5879: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:26:12.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:12 smithi106 ceph-mon[36505]: pgmap v5879: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:26:14.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10248.249681 secs 2024-03-24T03:26:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:14 smithi078 ceph-mon[32429]: pgmap v5880: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10248.249681 secs 2024-03-24T03:26:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:14 smithi107 ceph-mon[36485]: pgmap v5880: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10248.249681 secs 2024-03-24T03:26:14.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:14 smithi106 ceph-mon[36505]: pgmap v5880: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:15.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:15 smithi078 ceph-mon[32429]: pgmap v5881: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:15.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:15 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:26:15.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:15 smithi107 ceph-mon[36485]: pgmap v5881: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:15.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:15 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:26:15.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:15 smithi106 ceph-mon[36505]: pgmap v5881: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:15.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:15 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:26:17.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:26:17.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:26:17.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:26:17.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:26:17.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:26:17.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:26:17.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:26:17.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:26:17.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:26:17.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:26:17.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:26:17.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:26:17.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:26:17.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:26:17.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:26:17.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:26:17.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:26:17.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:26:17.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:26:17.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:26:17.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:26:17.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:26:17.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:26:17.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:26:17.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:26:17.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:26:17.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:26:18.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:17 smithi078 ceph-mon[32429]: pgmap v5882: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:26:18.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:17 smithi078 ceph-mon[32429]: pgmap v5883: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:26:18.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:17 smithi107 ceph-mon[36485]: pgmap v5882: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:26:18.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:17 smithi107 ceph-mon[36485]: pgmap v5883: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:26:18.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:17 smithi106 ceph-mon[36505]: pgmap v5882: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:26:18.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:17 smithi106 ceph-mon[36505]: pgmap v5883: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:26:20.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10253.249766 secs 2024-03-24T03:26:20.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:19 smithi078 ceph-mon[32429]: pgmap v5884: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:26:20.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10253.249766 secs 2024-03-24T03:26:20.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:19 smithi107 ceph-mon[36485]: pgmap v5884: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:26:20.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10253.249766 secs 2024-03-24T03:26:20.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:19 smithi106 ceph-mon[36505]: pgmap v5884: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:26:22.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:21 smithi078 ceph-mon[32429]: pgmap v5885: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:26:22.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:21 smithi107 ceph-mon[36485]: pgmap v5885: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:26:22.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:21 smithi106 ceph-mon[36505]: pgmap v5885: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:26:24.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:23 smithi078 ceph-mon[32429]: pgmap v5886: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:26:24.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:23 smithi107 ceph-mon[36485]: pgmap v5886: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:26:24.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:23 smithi106 ceph-mon[36505]: pgmap v5886: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:26:25.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10258.249825 secs 2024-03-24T03:26:25.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10258.249825 secs 2024-03-24T03:26:25.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10258.249825 secs 2024-03-24T03:26:26.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:25 smithi107 ceph-mon[36485]: pgmap v5887: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:26:26.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:25 smithi106 ceph-mon[36505]: pgmap v5887: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:26:26.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:25 smithi078 ceph-mon[32429]: pgmap v5887: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:26:28.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:27 smithi106 ceph-mon[36505]: pgmap v5888: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:26:28.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:27 smithi078 ceph-mon[32429]: pgmap v5888: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:26:28.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:27 smithi107 ceph-mon[36485]: pgmap v5888: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:26:30.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10263.249913 secs 2024-03-24T03:26:30.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:29 smithi106 ceph-mon[36505]: pgmap v5889: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:30.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10263.249913 secs 2024-03-24T03:26:30.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:29 smithi078 ceph-mon[32429]: pgmap v5889: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:30.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10263.249913 secs 2024-03-24T03:26:30.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:29 smithi107 ceph-mon[36485]: pgmap v5889: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:32.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:31 smithi106 ceph-mon[36505]: pgmap v5890: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:26:32.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:31 smithi078 ceph-mon[32429]: pgmap v5890: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:26:32.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:31 smithi107 ceph-mon[36485]: pgmap v5890: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:26:34.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:33 smithi106 ceph-mon[36505]: pgmap v5891: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:33 smithi078 ceph-mon[32429]: pgmap v5891: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:33 smithi107 ceph-mon[36485]: pgmap v5891: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:35.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10268.249993 secs 2024-03-24T03:26:35.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10268.249993 secs 2024-03-24T03:26:35.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10268.249993 secs 2024-03-24T03:26:36.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:35 smithi106 ceph-mon[36505]: pgmap v5892: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:36.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:35 smithi078 ceph-mon[32429]: pgmap v5892: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:36.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:35 smithi107 ceph-mon[36485]: pgmap v5892: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:38.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:37 smithi106 ceph-mon[36505]: pgmap v5893: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:26:38.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:37 smithi078 ceph-mon[32429]: pgmap v5893: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:26:38.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:37 smithi107 ceph-mon[36485]: pgmap v5893: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:26:40.145 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10273.250079 secs 2024-03-24T03:26:40.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:39 smithi107 ceph-mon[36485]: pgmap v5894: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10273.250079 secs 2024-03-24T03:26:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:39 smithi106 ceph-mon[36505]: pgmap v5894: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:40.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10273.250079 secs 2024-03-24T03:26:40.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:39 smithi078 ceph-mon[32429]: pgmap v5894: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:42.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:41 smithi106 ceph-mon[36505]: pgmap v5895: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:26:42.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:41 smithi078 ceph-mon[32429]: pgmap v5895: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:26:42.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:41 smithi107 ceph-mon[36485]: pgmap v5895: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:26:44.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:43 smithi106 ceph-mon[36505]: pgmap v5896: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:43 smithi078 ceph-mon[32429]: pgmap v5896: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:43 smithi107 ceph-mon[36485]: pgmap v5896: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:45.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10278.250137 secs 2024-03-24T03:26:45.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10278.250137 secs 2024-03-24T03:26:45.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10278.250137 secs 2024-03-24T03:26:46.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:45 smithi106 ceph-mon[36505]: pgmap v5897: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:46.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:45 smithi078 ceph-mon[32429]: pgmap v5897: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:46.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:45 smithi107 ceph-mon[36485]: pgmap v5897: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:48.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:47 smithi106 ceph-mon[36505]: pgmap v5898: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:26:48.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:47 smithi078 ceph-mon[32429]: pgmap v5898: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:26:48.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:47 smithi107 ceph-mon[36485]: pgmap v5898: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:26:50.143 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10283.250220 secs 2024-03-24T03:26:50.143 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:49 smithi078 ceph-mon[32429]: pgmap v5899: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10283.250220 secs 2024-03-24T03:26:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:49 smithi106 ceph-mon[36505]: pgmap v5899: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10283.250220 secs 2024-03-24T03:26:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:49 smithi107 ceph-mon[36485]: pgmap v5899: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:52.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:51 smithi106 ceph-mon[36505]: pgmap v5900: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:26:52.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:51 smithi078 ceph-mon[32429]: pgmap v5900: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:26:52.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:51 smithi107 ceph-mon[36485]: pgmap v5900: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:26:54.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:53 smithi106 ceph-mon[36505]: pgmap v5901: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:53 smithi078 ceph-mon[32429]: pgmap v5901: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:53 smithi107 ceph-mon[36485]: pgmap v5901: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:55.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10288.250303 secs 2024-03-24T03:26:55.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10288.250303 secs 2024-03-24T03:26:55.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10288.250303 secs 2024-03-24T03:26:56.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:55 smithi106 ceph-mon[36505]: pgmap v5902: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:56.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:55 smithi078 ceph-mon[32429]: pgmap v5902: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:56.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:55 smithi107 ceph-mon[36485]: pgmap v5902: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:26:58.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:57 smithi106 ceph-mon[36505]: pgmap v5903: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:26:58.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:57 smithi078 ceph-mon[32429]: pgmap v5903: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:26:58.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:57 smithi107 ceph-mon[36485]: pgmap v5903: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:27:00.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10293.250357 secs 2024-03-24T03:27:00.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:26:59 smithi106 ceph-mon[36505]: pgmap v5904: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:00.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10293.250357 secs 2024-03-24T03:27:00.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:26:59 smithi078 ceph-mon[32429]: pgmap v5904: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:00.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10293.250357 secs 2024-03-24T03:27:00.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:26:59 smithi107 ceph-mon[36485]: pgmap v5904: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:02.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:01 smithi078 ceph-mon[32429]: pgmap v5905: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:27:02.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:01 smithi107 ceph-mon[36485]: pgmap v5905: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:27:02.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:01 smithi106 ceph-mon[36505]: pgmap v5905: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:27:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:03 smithi078 ceph-mon[32429]: pgmap v5906: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10298.250441 secs 2024-03-24T03:27:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:03 smithi107 ceph-mon[36485]: pgmap v5906: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10298.250441 secs 2024-03-24T03:27:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:03 smithi106 ceph-mon[36505]: pgmap v5906: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10298.250441 secs 2024-03-24T03:27:06.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:05 smithi078 ceph-mon[32429]: pgmap v5907: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:06.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:05 smithi107 ceph-mon[36485]: pgmap v5907: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:06.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:05 smithi106 ceph-mon[36505]: pgmap v5907: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:08.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:07 smithi078 ceph-mon[32429]: pgmap v5908: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:27:08.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:07 smithi107 ceph-mon[36485]: pgmap v5908: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:27:08.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:07 smithi106 ceph-mon[36505]: pgmap v5908: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:27:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:08 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10303.250494 secs 2024-03-24T03:27:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:08 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10303.250494 secs 2024-03-24T03:27:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:08 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10303.250494 secs 2024-03-24T03:27:10.244 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:09 smithi107 ceph-mon[36485]: pgmap v5909: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:10.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:09 smithi078 ceph-mon[32429]: pgmap v5909: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:10.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:09 smithi106 ceph-mon[36505]: pgmap v5909: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:12.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:11 smithi078 ceph-mon[32429]: pgmap v5910: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:27:12.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:11 smithi107 ceph-mon[36485]: pgmap v5910: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:27:12.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:11 smithi106 ceph-mon[36505]: pgmap v5910: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:27:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:13 smithi078 ceph-mon[32429]: pgmap v5911: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:13 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10308.250580 secs 2024-03-24T03:27:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:13 smithi107 ceph-mon[36485]: pgmap v5911: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:13 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10308.250580 secs 2024-03-24T03:27:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:13 smithi106 ceph-mon[36505]: pgmap v5911: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:13 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10308.250580 secs 2024-03-24T03:27:16.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:15 smithi078 ceph-mon[32429]: pgmap v5912: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:16.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:15 smithi107 ceph-mon[36485]: pgmap v5912: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:16.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:15 smithi106 ceph-mon[36505]: pgmap v5912: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:17.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:16 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:27:17.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:16 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:27:17.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:16 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:27:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:17 smithi078 ceph-mon[32429]: pgmap v5913: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:27:18.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:17 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:27:18.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:17 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:27:18.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:17 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:27:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:17 smithi107 ceph-mon[36485]: pgmap v5913: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:27:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:17 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:27:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:17 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:27:18.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:17 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:27:18.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:17 smithi106 ceph-mon[36505]: pgmap v5913: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:27:18.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:17 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:27:18.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:17 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:27:18.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:17 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:27:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:18 smithi078 ceph-mon[32429]: pgmap v5914: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:27:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:18 smithi078 ceph-mon[32429]: pgmap v5915: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:27:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:18 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10313.250639 secs 2024-03-24T03:27:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:18 smithi107 ceph-mon[36485]: pgmap v5914: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:27:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:18 smithi107 ceph-mon[36485]: pgmap v5915: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:27:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:18 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10313.250639 secs 2024-03-24T03:27:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:18 smithi106 ceph-mon[36505]: pgmap v5914: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:27:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:18 smithi106 ceph-mon[36505]: pgmap v5915: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:27:19.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:18 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10313.250639 secs 2024-03-24T03:27:21.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:20 smithi078 ceph-mon[32429]: pgmap v5916: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:27:21.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:20 smithi107 ceph-mon[36485]: pgmap v5916: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:27:21.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:20 smithi106 ceph-mon[36505]: pgmap v5916: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:27:23.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:22 smithi078 ceph-mon[32429]: pgmap v5917: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:27:23.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:22 smithi107 ceph-mon[36485]: pgmap v5917: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:27:23.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:22 smithi106 ceph-mon[36505]: pgmap v5917: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:27:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:23 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10318.250726 secs 2024-03-24T03:27:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:23 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10318.250726 secs 2024-03-24T03:27:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:23 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10318.250726 secs 2024-03-24T03:27:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:24 smithi078 ceph-mon[32429]: pgmap v5918: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:27:25.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:24 smithi107 ceph-mon[36485]: pgmap v5918: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:27:25.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:24 smithi106 ceph-mon[36505]: pgmap v5918: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:27:27.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:26 smithi078 ceph-mon[32429]: pgmap v5919: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:27:27.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:26 smithi107 ceph-mon[36485]: pgmap v5919: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:27:27.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:26 smithi106 ceph-mon[36505]: pgmap v5919: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:27:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:28 smithi078 ceph-mon[32429]: pgmap v5920: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:27:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10323.250788 secs 2024-03-24T03:27:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:28 smithi107 ceph-mon[36485]: pgmap v5920: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:27:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10323.250788 secs 2024-03-24T03:27:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:28 smithi106 ceph-mon[36505]: pgmap v5920: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:27:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10323.250788 secs 2024-03-24T03:27:31.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:30 smithi078 ceph-mon[32429]: pgmap v5921: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:31.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:30 smithi107 ceph-mon[36485]: pgmap v5921: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:31.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:30 smithi106 ceph-mon[36505]: pgmap v5921: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:33.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:32 smithi078 ceph-mon[32429]: pgmap v5922: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:27:33.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:32 smithi107 ceph-mon[36485]: pgmap v5922: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:27:33.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:32 smithi106 ceph-mon[36505]: pgmap v5922: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:27:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10328.250866 secs 2024-03-24T03:27:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10328.250866 secs 2024-03-24T03:27:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10328.250866 secs 2024-03-24T03:27:35.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:34 smithi078 ceph-mon[32429]: pgmap v5923: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:35.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:34 smithi107 ceph-mon[36485]: pgmap v5923: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:35.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:34 smithi106 ceph-mon[36505]: pgmap v5923: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:37.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:36 smithi078 ceph-mon[32429]: pgmap v5924: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:36 smithi107 ceph-mon[36485]: pgmap v5924: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:36 smithi106 ceph-mon[36505]: pgmap v5924: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:38 smithi078 ceph-mon[32429]: pgmap v5925: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:27:39.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10333.250880 secs 2024-03-24T03:27:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:38 smithi107 ceph-mon[36485]: pgmap v5925: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:27:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10333.250880 secs 2024-03-24T03:27:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:38 smithi106 ceph-mon[36505]: pgmap v5925: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:27:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10333.250880 secs 2024-03-24T03:27:41.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:40 smithi078 ceph-mon[32429]: pgmap v5926: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:40 smithi107 ceph-mon[36485]: pgmap v5926: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:41.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:40 smithi106 ceph-mon[36505]: pgmap v5926: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:42 smithi078 ceph-mon[32429]: pgmap v5927: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:27:43.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:42 smithi107 ceph-mon[36485]: pgmap v5927: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:27:43.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:42 smithi106 ceph-mon[36505]: pgmap v5927: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:27:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10338.250918 secs 2024-03-24T03:27:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10338.250918 secs 2024-03-24T03:27:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10338.250918 secs 2024-03-24T03:27:45.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:44 smithi078 ceph-mon[32429]: pgmap v5928: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:45.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:44 smithi107 ceph-mon[36485]: pgmap v5928: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:45.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:44 smithi106 ceph-mon[36505]: pgmap v5928: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:47.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:46 smithi078 ceph-mon[32429]: pgmap v5929: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:47.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:46 smithi107 ceph-mon[36485]: pgmap v5929: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:47.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:46 smithi106 ceph-mon[36505]: pgmap v5929: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:49 smithi078 ceph-mon[32429]: pgmap v5930: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:27:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10343.251003 secs 2024-03-24T03:27:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:49 smithi107 ceph-mon[36485]: pgmap v5930: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:27:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10343.251003 secs 2024-03-24T03:27:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:49 smithi106 ceph-mon[36505]: pgmap v5930: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:27:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10343.251003 secs 2024-03-24T03:27:51.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:51 smithi078 ceph-mon[32429]: pgmap v5931: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:51.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:51 smithi107 ceph-mon[36485]: pgmap v5931: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:51.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:51 smithi106 ceph-mon[36505]: pgmap v5931: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:53.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:53 smithi078 ceph-mon[32429]: pgmap v5932: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:27:53.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:53 smithi107 ceph-mon[36485]: pgmap v5932: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:27:53.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:53 smithi106 ceph-mon[36505]: pgmap v5932: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:27:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10348.251085 secs 2024-03-24T03:27:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10348.251085 secs 2024-03-24T03:27:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10348.251085 secs 2024-03-24T03:27:55.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:55 smithi078 ceph-mon[32429]: pgmap v5933: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:55.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:55 smithi107 ceph-mon[36485]: pgmap v5933: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:55.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:55 smithi106 ceph-mon[36505]: pgmap v5933: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:57.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:57 smithi078 ceph-mon[32429]: pgmap v5934: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:57.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:57 smithi107 ceph-mon[36485]: pgmap v5934: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:57.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:57 smithi106 ceph-mon[36505]: pgmap v5934: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:27:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:59 smithi078 ceph-mon[32429]: pgmap v5935: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:27:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:27:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10353.251170 secs 2024-03-24T03:27:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:59 smithi107 ceph-mon[36485]: pgmap v5935: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:27:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:27:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10353.251170 secs 2024-03-24T03:27:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:59 smithi106 ceph-mon[36505]: pgmap v5935: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:27:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:27:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10353.251170 secs 2024-03-24T03:28:01.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:01 smithi078 ceph-mon[32429]: pgmap v5936: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:01 smithi107 ceph-mon[36485]: pgmap v5936: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:01 smithi106 ceph-mon[36505]: pgmap v5936: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:03.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:03 smithi078 ceph-mon[32429]: pgmap v5937: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:28:03.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:03 smithi107 ceph-mon[36485]: pgmap v5937: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:28:03.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:03 smithi106 ceph-mon[36505]: pgmap v5937: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:28:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10358.251229 secs 2024-03-24T03:28:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10358.251229 secs 2024-03-24T03:28:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10358.251229 secs 2024-03-24T03:28:05.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:05 smithi078 ceph-mon[32429]: pgmap v5938: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:05.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:05 smithi107 ceph-mon[36485]: pgmap v5938: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:05.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:05 smithi106 ceph-mon[36505]: pgmap v5938: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:07.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:07 smithi078 ceph-mon[32429]: pgmap v5939: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:07.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:07 smithi107 ceph-mon[36485]: pgmap v5939: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:07.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:07 smithi106 ceph-mon[36505]: pgmap v5939: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:09 smithi078 ceph-mon[32429]: pgmap v5940: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:28:09.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10363.251309 secs 2024-03-24T03:28:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:09 smithi107 ceph-mon[36485]: pgmap v5940: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:28:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10363.251309 secs 2024-03-24T03:28:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:09 smithi106 ceph-mon[36505]: pgmap v5940: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:28:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10363.251309 secs 2024-03-24T03:28:11.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:11 smithi078 ceph-mon[32429]: pgmap v5941: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:11.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:11 smithi107 ceph-mon[36485]: pgmap v5941: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:11.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:11 smithi106 ceph-mon[36505]: pgmap v5941: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:13.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:13 smithi078 ceph-mon[32429]: pgmap v5942: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:28:13.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:13 smithi107 ceph-mon[36485]: pgmap v5942: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:28:13.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:13 smithi106 ceph-mon[36505]: pgmap v5942: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:28:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10368.251369 secs 2024-03-24T03:28:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10368.251369 secs 2024-03-24T03:28:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10368.251369 secs 2024-03-24T03:28:15.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:15 smithi078 ceph-mon[32429]: pgmap v5943: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:15.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:15 smithi107 ceph-mon[36485]: pgmap v5943: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:15.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:15 smithi106 ceph-mon[36505]: pgmap v5943: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:17.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:17 smithi078 ceph-mon[32429]: pgmap v5944: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:17.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:17 smithi107 ceph-mon[36485]: pgmap v5944: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:17.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:17 smithi106 ceph-mon[36505]: pgmap v5944: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:18 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:28:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:18 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:28:18.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:18 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:28:18.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:18 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:28:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:18 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:28:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:18 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:28:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:18 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:28:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:18 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:28:18.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:18 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:28:18.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:18 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:28:18.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:18 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:28:18.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:18 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:28:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:19 smithi078 ceph-mon[32429]: pgmap v5945: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:28:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10373.251427 secs 2024-03-24T03:28:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:19 smithi078 ceph-mon[32429]: pgmap v5946: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:28:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:19 smithi078 ceph-mon[32429]: pgmap v5947: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:28:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:19 smithi107 ceph-mon[36485]: pgmap v5945: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:28:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10373.251427 secs 2024-03-24T03:28:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:19 smithi107 ceph-mon[36485]: pgmap v5946: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:28:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:19 smithi107 ceph-mon[36485]: pgmap v5947: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:28:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:19 smithi106 ceph-mon[36505]: pgmap v5945: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:28:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10373.251427 secs 2024-03-24T03:28:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:19 smithi106 ceph-mon[36505]: pgmap v5946: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:28:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:19 smithi106 ceph-mon[36505]: pgmap v5947: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:28:21.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:21 smithi078 ceph-mon[32429]: pgmap v5948: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:28:21.427 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:21 smithi106 ceph-mon[36505]: pgmap v5948: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:28:21.427 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:21 smithi107 ceph-mon[36485]: pgmap v5948: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:28:23.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:23 smithi078 ceph-mon[32429]: pgmap v5949: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:28:23.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:23 smithi107 ceph-mon[36485]: pgmap v5949: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:28:23.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:23 smithi106 ceph-mon[36505]: pgmap v5949: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:28:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10378.251485 secs 2024-03-24T03:28:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10378.251485 secs 2024-03-24T03:28:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10378.251485 secs 2024-03-24T03:28:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:25 smithi078 ceph-mon[32429]: pgmap v5950: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:28:25.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:25 smithi107 ceph-mon[36485]: pgmap v5950: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:28:25.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:25 smithi106 ceph-mon[36505]: pgmap v5950: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:28:27.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:27 smithi078 ceph-mon[32429]: pgmap v5951: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:28:27.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:27 smithi107 ceph-mon[36485]: pgmap v5951: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:28:27.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:27 smithi106 ceph-mon[36505]: pgmap v5951: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:28:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10383.251569 secs 2024-03-24T03:28:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:29 smithi078 ceph-mon[32429]: pgmap v5952: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:28:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10383.251569 secs 2024-03-24T03:28:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:29 smithi107 ceph-mon[36485]: pgmap v5952: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:28:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10383.251569 secs 2024-03-24T03:28:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:29 smithi106 ceph-mon[36505]: pgmap v5952: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:28:31.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:31 smithi106 ceph-mon[36505]: pgmap v5953: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:31.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:31 smithi078 ceph-mon[32429]: pgmap v5953: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:31.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:31 smithi107 ceph-mon[36485]: pgmap v5953: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:32.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:28:32.373+0000 7fab9b7fe640 1 -- 172.21.15.78:0/1058387750 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab94005850 con 0x7fabb4101650 2024-03-24T03:28:32.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:28:32.373+0000 7fabb0ff9640 1 -- 172.21.15.78:0/1058387750 <== mon.0 v2:172.21.15.78:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7fab94005850 con 0x7fabb4101650 2024-03-24T03:28:33.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:33 smithi106 ceph-mon[36505]: pgmap v5954: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:28:33.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:33 smithi078 ceph-mon[32429]: pgmap v5954: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:28:33.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:33 smithi107 ceph-mon[36485]: pgmap v5954: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:28:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10388.251656 secs 2024-03-24T03:28:34.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10388.251656 secs 2024-03-24T03:28:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10388.251656 secs 2024-03-24T03:28:35.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:35 smithi106 ceph-mon[36505]: pgmap v5955: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:35.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:35 smithi078 ceph-mon[32429]: pgmap v5955: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:35.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:35 smithi107 ceph-mon[36485]: pgmap v5955: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:37 smithi106 ceph-mon[36505]: pgmap v5956: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:28:37.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:37 smithi078 ceph-mon[32429]: pgmap v5956: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:28:37.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:37 smithi107 ceph-mon[36485]: pgmap v5956: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:28:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10393.251705 secs 2024-03-24T03:28:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:39 smithi106 ceph-mon[36505]: pgmap v5957: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10393.251705 secs 2024-03-24T03:28:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:39 smithi078 ceph-mon[32429]: pgmap v5957: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10393.251705 secs 2024-03-24T03:28:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:39 smithi107 ceph-mon[36485]: pgmap v5957: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:41.395 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:41 smithi107 ceph-mon[36485]: pgmap v5958: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:41.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:41 smithi106 ceph-mon[36505]: pgmap v5958: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:41.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:41 smithi078 ceph-mon[32429]: pgmap v5958: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:43.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:43 smithi106 ceph-mon[36505]: pgmap v5959: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:28:43.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:43 smithi078 ceph-mon[32429]: pgmap v5959: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:28:43.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:43 smithi107 ceph-mon[36485]: pgmap v5959: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:28:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10398.251792 secs 2024-03-24T03:28:44.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10398.251792 secs 2024-03-24T03:28:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10398.251792 secs 2024-03-24T03:28:45.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:45 smithi106 ceph-mon[36505]: pgmap v5960: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:45.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:45 smithi078 ceph-mon[32429]: pgmap v5960: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:45.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:45 smithi107 ceph-mon[36485]: pgmap v5960: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:47.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:47 smithi106 ceph-mon[36505]: pgmap v5961: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:28:47.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:47 smithi078 ceph-mon[32429]: pgmap v5961: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:28:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:47 smithi107 ceph-mon[36485]: pgmap v5961: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:28:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10403.251873 secs 2024-03-24T03:28:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:49 smithi106 ceph-mon[36505]: pgmap v5962: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10403.251873 secs 2024-03-24T03:28:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:49 smithi078 ceph-mon[32429]: pgmap v5962: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10403.251873 secs 2024-03-24T03:28:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:49 smithi107 ceph-mon[36485]: pgmap v5962: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:51.392 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:51 smithi078 ceph-mon[32429]: pgmap v5963: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:51.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:51 smithi106 ceph-mon[36505]: pgmap v5963: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:51.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:51 smithi107 ceph-mon[36485]: pgmap v5963: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:53.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:52 smithi106 ceph-mon[36505]: pgmap v5964: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:28:53.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:52 smithi078 ceph-mon[32429]: pgmap v5964: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:28:53.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:52 smithi107 ceph-mon[36485]: pgmap v5964: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:28:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10408.251933 secs 2024-03-24T03:28:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10408.251933 secs 2024-03-24T03:28:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10408.251933 secs 2024-03-24T03:28:55.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:54 smithi078 ceph-mon[32429]: pgmap v5965: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:55.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:54 smithi107 ceph-mon[36485]: pgmap v5965: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:55.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:54 smithi106 ceph-mon[36505]: pgmap v5965: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:57.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:57 smithi106 ceph-mon[36505]: pgmap v5966: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:28:57.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:57 smithi078 ceph-mon[32429]: pgmap v5966: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:28:57.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:57 smithi107 ceph-mon[36485]: pgmap v5966: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:28:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10413.252021 secs 2024-03-24T03:28:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:28:59 smithi106 ceph-mon[36505]: pgmap v5967: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10413.252021 secs 2024-03-24T03:28:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:28:59 smithi078 ceph-mon[32429]: pgmap v5967: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:28:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10413.252021 secs 2024-03-24T03:28:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:28:59 smithi107 ceph-mon[36485]: pgmap v5967: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:01 smithi106 ceph-mon[36505]: pgmap v5968: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:01.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:01 smithi078 ceph-mon[32429]: pgmap v5968: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:01.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:01 smithi107 ceph-mon[36485]: pgmap v5968: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:03.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:03 smithi106 ceph-mon[36505]: pgmap v5969: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:29:03.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:03 smithi078 ceph-mon[32429]: pgmap v5969: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:29:03.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:03 smithi107 ceph-mon[36485]: pgmap v5969: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:29:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10418.252104 secs 2024-03-24T03:29:04.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10418.252104 secs 2024-03-24T03:29:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10418.252104 secs 2024-03-24T03:29:05.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:05 smithi106 ceph-mon[36505]: pgmap v5970: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:05.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:05 smithi078 ceph-mon[32429]: pgmap v5970: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:05.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:05 smithi107 ceph-mon[36485]: pgmap v5970: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:07.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:07 smithi106 ceph-mon[36505]: pgmap v5971: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:29:07.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:07 smithi078 ceph-mon[32429]: pgmap v5971: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:29:07.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:07 smithi107 ceph-mon[36485]: pgmap v5971: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:29:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10423.252121 secs 2024-03-24T03:29:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:09 smithi106 ceph-mon[36505]: pgmap v5972: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10423.252121 secs 2024-03-24T03:29:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:09 smithi078 ceph-mon[32429]: pgmap v5972: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10423.252121 secs 2024-03-24T03:29:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:09 smithi107 ceph-mon[36485]: pgmap v5972: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:11.395 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:11 smithi107 ceph-mon[36485]: pgmap v5973: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:11.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:11 smithi106 ceph-mon[36505]: pgmap v5973: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:11.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:11 smithi078 ceph-mon[32429]: pgmap v5973: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:13.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:13 smithi106 ceph-mon[36505]: pgmap v5974: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:29:13.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:13 smithi078 ceph-mon[32429]: pgmap v5974: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:29:13.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:13 smithi107 ceph-mon[36485]: pgmap v5974: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:29:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10428.252208 secs 2024-03-24T03:29:14.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10428.252208 secs 2024-03-24T03:29:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10428.252208 secs 2024-03-24T03:29:15.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:15 smithi106 ceph-mon[36505]: pgmap v5975: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:15.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:15 smithi078 ceph-mon[32429]: pgmap v5975: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:15.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:15 smithi107 ceph-mon[36485]: pgmap v5975: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:17.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:17 smithi106 ceph-mon[36505]: pgmap v5976: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:29:17.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:17 smithi078 ceph-mon[32429]: pgmap v5976: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:29:17.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:17 smithi107 ceph-mon[36485]: pgmap v5976: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:29:18.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:18 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:29:18.459 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:18 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:29:18.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:18 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:29:19.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10433.252268 secs 2024-03-24T03:29:19.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:19 smithi078 ceph-mon[32429]: pgmap v5977: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:19.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:19 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:29:19.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:19 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:29:19.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:19 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:29:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10433.252268 secs 2024-03-24T03:29:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:19 smithi107 ceph-mon[36485]: pgmap v5977: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:19 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:29:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:19 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:29:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:19 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:29:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10433.252268 secs 2024-03-24T03:29:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:19 smithi106 ceph-mon[36505]: pgmap v5977: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:19 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:29:19.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:19 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:29:19.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:19 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:29:20.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:20 smithi078 ceph-mon[32429]: pgmap v5978: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:29:20.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:20 smithi107 ceph-mon[36485]: pgmap v5978: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:29:20.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:20 smithi106 ceph-mon[36505]: pgmap v5978: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:29:22.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:22 smithi078 ceph-mon[32429]: pgmap v5979: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:29:22.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:22 smithi107 ceph-mon[36485]: pgmap v5979: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:29:22.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:22 smithi106 ceph-mon[36505]: pgmap v5979: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:29:24.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:24 smithi078 ceph-mon[32429]: pgmap v5980: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:29:24.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10438.252337 secs 2024-03-24T03:29:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:24 smithi107 ceph-mon[36485]: pgmap v5980: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:29:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10438.252337 secs 2024-03-24T03:29:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:24 smithi106 ceph-mon[36505]: pgmap v5980: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:29:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10438.252337 secs 2024-03-24T03:29:26.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:26 smithi078 ceph-mon[32429]: pgmap v5981: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:29:26.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:26 smithi107 ceph-mon[36485]: pgmap v5981: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:29:26.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:26 smithi106 ceph-mon[36505]: pgmap v5981: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:29:28.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:28 smithi078 ceph-mon[32429]: pgmap v5982: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:29:28.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:28 smithi107 ceph-mon[36485]: pgmap v5982: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:29:28.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:28 smithi106 ceph-mon[36505]: pgmap v5982: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:29:29.552 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10443.252421 secs 2024-03-24T03:29:29.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10443.252421 secs 2024-03-24T03:29:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10443.252421 secs 2024-03-24T03:29:30.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:30 smithi078 ceph-mon[32429]: pgmap v5983: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:29:30.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:30 smithi107 ceph-mon[36485]: pgmap v5983: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:29:30.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:30 smithi106 ceph-mon[36505]: pgmap v5983: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:29:32.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:32 smithi078 ceph-mon[32429]: pgmap v5984: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:29:32.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:32 smithi107 ceph-mon[36485]: pgmap v5984: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:29:32.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:32 smithi106 ceph-mon[36505]: pgmap v5984: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:29:34.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:34 smithi078 ceph-mon[32429]: pgmap v5985: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:34.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10448.252506 secs 2024-03-24T03:29:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:34 smithi107 ceph-mon[36485]: pgmap v5985: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10448.252506 secs 2024-03-24T03:29:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:34 smithi106 ceph-mon[36505]: pgmap v5985: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10448.252506 secs 2024-03-24T03:29:36.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:36 smithi078 ceph-mon[32429]: pgmap v5986: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:36.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:36 smithi107 ceph-mon[36485]: pgmap v5986: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:36.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:36 smithi106 ceph-mon[36505]: pgmap v5986: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:38.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:38 smithi078 ceph-mon[32429]: pgmap v5987: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:29:38.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:38 smithi107 ceph-mon[36485]: pgmap v5987: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:29:38.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:38 smithi106 ceph-mon[36505]: pgmap v5987: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:29:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10453.252564 secs 2024-03-24T03:29:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10453.252564 secs 2024-03-24T03:29:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10453.252564 secs 2024-03-24T03:29:40.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:40 smithi078 ceph-mon[32429]: pgmap v5988: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:40.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:40 smithi107 ceph-mon[36485]: pgmap v5988: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:40.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:40 smithi106 ceph-mon[36505]: pgmap v5988: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:42.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:42 smithi078 ceph-mon[32429]: pgmap v5989: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:29:42.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:42 smithi107 ceph-mon[36485]: pgmap v5989: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:29:42.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:42 smithi106 ceph-mon[36505]: pgmap v5989: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:29:44.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:44 smithi078 ceph-mon[32429]: pgmap v5990: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:44.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10458.252650 secs 2024-03-24T03:29:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:44 smithi107 ceph-mon[36485]: pgmap v5990: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10458.252650 secs 2024-03-24T03:29:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:44 smithi106 ceph-mon[36505]: pgmap v5990: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10458.252650 secs 2024-03-24T03:29:46.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:46 smithi078 ceph-mon[32429]: pgmap v5991: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:46.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:46 smithi107 ceph-mon[36485]: pgmap v5991: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:46 smithi106 ceph-mon[36505]: pgmap v5991: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:48.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:48 smithi078 ceph-mon[32429]: pgmap v5992: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:29:48.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:48 smithi107 ceph-mon[36485]: pgmap v5992: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:29:48.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:48 smithi106 ceph-mon[36505]: pgmap v5992: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:29:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10463.252711 secs 2024-03-24T03:29:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10463.252711 secs 2024-03-24T03:29:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10463.252711 secs 2024-03-24T03:29:50.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:50 smithi078 ceph-mon[32429]: pgmap v5993: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:50.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:50 smithi107 ceph-mon[36485]: pgmap v5993: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:50.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:50 smithi106 ceph-mon[36505]: pgmap v5993: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:52.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:52 smithi078 ceph-mon[32429]: pgmap v5994: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:29:52.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:52 smithi107 ceph-mon[36485]: pgmap v5994: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:29:52.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:52 smithi106 ceph-mon[36505]: pgmap v5994: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:29:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:54 smithi078 ceph-mon[32429]: pgmap v5995: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:54.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10468.252770 secs 2024-03-24T03:29:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:54 smithi107 ceph-mon[36485]: pgmap v5995: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10468.252770 secs 2024-03-24T03:29:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:54 smithi106 ceph-mon[36505]: pgmap v5995: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10468.252770 secs 2024-03-24T03:29:56.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:56 smithi078 ceph-mon[32429]: pgmap v5996: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:56.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:56 smithi107 ceph-mon[36485]: pgmap v5996: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:56 smithi106 ceph-mon[36505]: pgmap v5996: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:29:58.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:58 smithi078 ceph-mon[32429]: pgmap v5997: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:29:58.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:58 smithi107 ceph-mon[36485]: pgmap v5997: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:29:58.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:58 smithi106 ceph-mon[36505]: pgmap v5997: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:29:59.552 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:29:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10473.252832 secs 2024-03-24T03:29:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:29:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10473.252832 secs 2024-03-24T03:29:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:29:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10473.252832 secs 2024-03-24T03:30:00.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:00 smithi078 ceph-mon[32429]: pgmap v5998: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:00.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:00 smithi078 ceph-mon[32429]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T03:30:00.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:00 smithi107 ceph-mon[36485]: pgmap v5998: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:00.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:00 smithi107 ceph-mon[36485]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T03:30:00.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:00 smithi106 ceph-mon[36505]: pgmap v5998: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:00.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:00 smithi106 ceph-mon[36505]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T03:30:02.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:02 smithi078 ceph-mon[32429]: pgmap v5999: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:30:02.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:02 smithi107 ceph-mon[36485]: pgmap v5999: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:30:02.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:02 smithi106 ceph-mon[36505]: pgmap v5999: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:30:04.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:04 smithi078 ceph-mon[32429]: pgmap v6000: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:04.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10478.252848 secs 2024-03-24T03:30:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:04 smithi107 ceph-mon[36485]: pgmap v6000: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10478.252848 secs 2024-03-24T03:30:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:04 smithi106 ceph-mon[36505]: pgmap v6000: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10478.252848 secs 2024-03-24T03:30:06.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:06 smithi078 ceph-mon[32429]: pgmap v6001: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:06.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:06 smithi107 ceph-mon[36485]: pgmap v6001: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:06.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:06 smithi106 ceph-mon[36505]: pgmap v6001: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:08.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:08 smithi078 ceph-mon[32429]: pgmap v6002: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:30:08.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:08 smithi107 ceph-mon[36485]: pgmap v6002: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:30:08.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:08 smithi106 ceph-mon[36505]: pgmap v6002: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:30:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10483.252907 secs 2024-03-24T03:30:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10483.252907 secs 2024-03-24T03:30:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10483.252907 secs 2024-03-24T03:30:10.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:10 smithi078 ceph-mon[32429]: pgmap v6003: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:10.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:10 smithi107 ceph-mon[36485]: pgmap v6003: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:10.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:10 smithi106 ceph-mon[36505]: pgmap v6003: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:12.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:12 smithi078 ceph-mon[32429]: pgmap v6004: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:30:12.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:12 smithi107 ceph-mon[36485]: pgmap v6004: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:30:12.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:12 smithi106 ceph-mon[36505]: pgmap v6004: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:30:14.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:14 smithi078 ceph-mon[32429]: pgmap v6005: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:14.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10488.252965 secs 2024-03-24T03:30:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:14 smithi107 ceph-mon[36485]: pgmap v6005: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10488.252965 secs 2024-03-24T03:30:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:14 smithi106 ceph-mon[36505]: pgmap v6005: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10488.252965 secs 2024-03-24T03:30:15.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:15 smithi078 ceph-mon[32429]: pgmap v6006: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:15.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:15 smithi107 ceph-mon[36485]: pgmap v6006: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:15.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:15 smithi106 ceph-mon[36505]: pgmap v6006: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:18.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:17 smithi106 ceph-mon[36505]: pgmap v6007: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:30:18.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:17 smithi078 ceph-mon[32429]: pgmap v6007: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:30:18.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:17 smithi107 ceph-mon[36485]: pgmap v6007: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:30:20.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10493.253054 secs 2024-03-24T03:30:20.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:19 smithi106 ceph-mon[36505]: pgmap v6008: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:20.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:19 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:30:20.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:19 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:30:20.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:19 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:30:20.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:19 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:30:20.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10493.253054 secs 2024-03-24T03:30:20.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:19 smithi078 ceph-mon[32429]: pgmap v6008: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:20.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:19 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:30:20.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:19 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:30:20.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:19 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:30:20.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:19 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:30:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10493.253054 secs 2024-03-24T03:30:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:19 smithi107 ceph-mon[36485]: pgmap v6008: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:19 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:30:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:19 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:30:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:19 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:30:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:19 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:30:21.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:20 smithi106 ceph-mon[36505]: pgmap v6009: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:30:21.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:20 smithi078 ceph-mon[32429]: pgmap v6009: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:30:21.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:20 smithi107 ceph-mon[36485]: pgmap v6009: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:30:23.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:22 smithi078 ceph-mon[32429]: pgmap v6010: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:30:23.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:22 smithi107 ceph-mon[36485]: pgmap v6010: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:30:23.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:22 smithi106 ceph-mon[36505]: pgmap v6010: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:30:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10498.253114 secs 2024-03-24T03:30:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:24 smithi078 ceph-mon[32429]: pgmap v6011: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:30:25.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10498.253114 secs 2024-03-24T03:30:25.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:24 smithi107 ceph-mon[36485]: pgmap v6011: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:30:25.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10498.253114 secs 2024-03-24T03:30:25.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:24 smithi106 ceph-mon[36505]: pgmap v6011: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:30:27.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:26 smithi078 ceph-mon[32429]: pgmap v6012: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:30:27.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:26 smithi107 ceph-mon[36485]: pgmap v6012: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:30:27.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:26 smithi106 ceph-mon[36505]: pgmap v6012: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:30:29.204 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:28 smithi106 ceph-mon[36505]: pgmap v6013: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:30:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:28 smithi078 ceph-mon[32429]: pgmap v6013: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:30:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:28 smithi107 ceph-mon[36485]: pgmap v6013: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:30:30.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10503.253196 secs 2024-03-24T03:30:30.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10503.253196 secs 2024-03-24T03:30:30.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10503.253196 secs 2024-03-24T03:30:31.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:30 smithi078 ceph-mon[32429]: pgmap v6014: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:30:31.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:30 smithi107 ceph-mon[36485]: pgmap v6014: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:30:31.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:30 smithi106 ceph-mon[36505]: pgmap v6014: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:30:33.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:32 smithi078 ceph-mon[32429]: pgmap v6015: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:30:33.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:32 smithi107 ceph-mon[36485]: pgmap v6015: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:30:33.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:32 smithi106 ceph-mon[36505]: pgmap v6015: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:30:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10508.253279 secs 2024-03-24T03:30:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10508.253279 secs 2024-03-24T03:30:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10508.253279 secs 2024-03-24T03:30:35.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:34 smithi078 ceph-mon[32429]: pgmap v6016: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:35.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:34 smithi107 ceph-mon[36485]: pgmap v6016: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:35.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:34 smithi106 ceph-mon[36505]: pgmap v6016: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:37.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:36 smithi078 ceph-mon[32429]: pgmap v6017: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:30:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:36 smithi107 ceph-mon[36485]: pgmap v6017: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:30:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:36 smithi106 ceph-mon[36505]: pgmap v6017: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:30:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:38 smithi078 ceph-mon[32429]: pgmap v6018: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:39.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10513.253365 secs 2024-03-24T03:30:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:38 smithi107 ceph-mon[36485]: pgmap v6018: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10513.253365 secs 2024-03-24T03:30:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:38 smithi106 ceph-mon[36505]: pgmap v6018: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10513.253365 secs 2024-03-24T03:30:41.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:40 smithi078 ceph-mon[32429]: pgmap v6019: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:40 smithi107 ceph-mon[36485]: pgmap v6019: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:41.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:40 smithi106 ceph-mon[36505]: pgmap v6019: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:42 smithi078 ceph-mon[32429]: pgmap v6020: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:30:43.345 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:42 smithi107 ceph-mon[36485]: pgmap v6020: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:30:43.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:42 smithi106 ceph-mon[36505]: pgmap v6020: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:30:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10518.253383 secs 2024-03-24T03:30:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10518.253383 secs 2024-03-24T03:30:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10518.253383 secs 2024-03-24T03:30:45.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:44 smithi078 ceph-mon[32429]: pgmap v6021: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:45.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:44 smithi107 ceph-mon[36485]: pgmap v6021: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:45.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:44 smithi106 ceph-mon[36505]: pgmap v6021: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:47.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:46 smithi078 ceph-mon[32429]: pgmap v6022: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:30:47.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:46 smithi107 ceph-mon[36485]: pgmap v6022: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:30:47.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:30:47.391+0000 7fab9b7fe640 1 -- 172.21.15.78:0/1058387750 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fab94005b90 con 0x7fabb4101650 2024-03-24T03:30:47.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:30:47.391+0000 7fabb0ff9640 1 -- 172.21.15.78:0/1058387750 <== mon.0 v2:172.21.15.78:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (secure 0 0 0) 0x7fab94005b90 con 0x7fabb4101650 2024-03-24T03:30:47.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:46 smithi106 ceph-mon[36505]: pgmap v6022: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:30:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:48 smithi078 ceph-mon[32429]: pgmap v6023: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10523.253444 secs 2024-03-24T03:30:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:48 smithi107 ceph-mon[36485]: pgmap v6023: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10523.253444 secs 2024-03-24T03:30:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:48 smithi106 ceph-mon[36505]: pgmap v6023: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10523.253444 secs 2024-03-24T03:30:51.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:50 smithi078 ceph-mon[32429]: pgmap v6024: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:51.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:50 smithi107 ceph-mon[36485]: pgmap v6024: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:51.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:50 smithi106 ceph-mon[36505]: pgmap v6024: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:53.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:52 smithi078 ceph-mon[32429]: pgmap v6025: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:30:53.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:52 smithi107 ceph-mon[36485]: pgmap v6025: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:30:53.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:52 smithi106 ceph-mon[36505]: pgmap v6025: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:30:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:53 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10528.253505 secs 2024-03-24T03:30:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:53 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10528.253505 secs 2024-03-24T03:30:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:53 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10528.253505 secs 2024-03-24T03:30:55.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:54 smithi078 ceph-mon[32429]: pgmap v6026: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:55.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:54 smithi107 ceph-mon[36485]: pgmap v6026: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:55.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:54 smithi106 ceph-mon[36505]: pgmap v6026: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:57.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:56 smithi078 ceph-mon[32429]: pgmap v6027: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:30:57.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:56 smithi107 ceph-mon[36485]: pgmap v6027: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:30:57.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:56 smithi106 ceph-mon[36505]: pgmap v6027: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:30:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:58 smithi078 ceph-mon[32429]: pgmap v6028: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:30:58 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10533.253594 secs 2024-03-24T03:30:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:58 smithi107 ceph-mon[36485]: pgmap v6028: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:30:58 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10533.253594 secs 2024-03-24T03:30:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:58 smithi106 ceph-mon[36505]: pgmap v6028: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:30:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:30:58 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10533.253594 secs 2024-03-24T03:31:01.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:00 smithi078 ceph-mon[32429]: pgmap v6029: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:00 smithi107 ceph-mon[36485]: pgmap v6029: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:00 smithi106 ceph-mon[36505]: pgmap v6029: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:03.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:02 smithi078 ceph-mon[32429]: pgmap v6030: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:31:03.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:02 smithi107 ceph-mon[36485]: pgmap v6030: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:31:03.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:02 smithi106 ceph-mon[36505]: pgmap v6030: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:31:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:03 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10538.253652 secs 2024-03-24T03:31:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:03 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10538.253652 secs 2024-03-24T03:31:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:03 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10538.253652 secs 2024-03-24T03:31:05.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:04 smithi078 ceph-mon[32429]: pgmap v6031: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:05.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:04 smithi107 ceph-mon[36485]: pgmap v6031: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:05.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:04 smithi106 ceph-mon[36505]: pgmap v6031: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:07.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:06 smithi078 ceph-mon[32429]: pgmap v6032: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:31:07.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:06 smithi107 ceph-mon[36485]: pgmap v6032: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:31:07.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:06 smithi106 ceph-mon[36505]: pgmap v6032: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:31:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:09 smithi078 ceph-mon[32429]: pgmap v6033: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:09.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10543.253738 secs 2024-03-24T03:31:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:09 smithi107 ceph-mon[36485]: pgmap v6033: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10543.253738 secs 2024-03-24T03:31:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:09 smithi106 ceph-mon[36505]: pgmap v6033: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10543.253738 secs 2024-03-24T03:31:11.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:11 smithi078 ceph-mon[32429]: pgmap v6034: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:11.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:11 smithi107 ceph-mon[36485]: pgmap v6034: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:11.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:11 smithi106 ceph-mon[36505]: pgmap v6034: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:13.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:13 smithi078 ceph-mon[32429]: pgmap v6035: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:31:13.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:13 smithi107 ceph-mon[36485]: pgmap v6035: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:31:13.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:13 smithi106 ceph-mon[36505]: pgmap v6035: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:31:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10548.253800 secs 2024-03-24T03:31:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10548.253800 secs 2024-03-24T03:31:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10548.253800 secs 2024-03-24T03:31:15.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:15 smithi078 ceph-mon[32429]: pgmap v6036: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:15.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:15 smithi107 ceph-mon[36485]: pgmap v6036: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:15.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:15 smithi106 ceph-mon[36505]: pgmap v6036: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:17.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:17 smithi078 ceph-mon[32429]: pgmap v6037: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:31:17.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:17 smithi107 ceph-mon[36485]: pgmap v6037: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:31:17.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:17 smithi106 ceph-mon[36505]: pgmap v6037: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:31:19.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:19 smithi078 ceph-mon[32429]: pgmap v6038: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:19.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10553.253861 secs 2024-03-24T03:31:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:19 smithi107 ceph-mon[36485]: pgmap v6038: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10553.253861 secs 2024-03-24T03:31:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:19 smithi106 ceph-mon[36505]: pgmap v6038: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10553.253861 secs 2024-03-24T03:31:20.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:20 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:31:20.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:20 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:31:20.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:20 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:31:21.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:21 smithi078 ceph-mon[32429]: pgmap v6039: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:21.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:21 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:31:21.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:21 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:31:21.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:21 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:31:21.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:21 smithi107 ceph-mon[36485]: pgmap v6039: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:21.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:21 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:31:21.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:21 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:31:21.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:21 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:31:21.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:21 smithi106 ceph-mon[36505]: pgmap v6039: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:21.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:21 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:31:21.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:21 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:31:21.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:21 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:31:22.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:22 smithi078 ceph-mon[32429]: pgmap v6040: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:31:22.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:22 smithi078 ceph-mon[32429]: pgmap v6041: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:31:22.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:22 smithi107 ceph-mon[36485]: pgmap v6040: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:31:22.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:22 smithi107 ceph-mon[36485]: pgmap v6041: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:31:22.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:22 smithi106 ceph-mon[36505]: pgmap v6040: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:31:22.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:22 smithi106 ceph-mon[36505]: pgmap v6041: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:31:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:24 smithi078 ceph-mon[32429]: pgmap v6042: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:31:24.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10558.253917 secs 2024-03-24T03:31:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:24 smithi107 ceph-mon[36485]: pgmap v6042: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:31:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10558.253917 secs 2024-03-24T03:31:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:24 smithi106 ceph-mon[36505]: pgmap v6042: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:31:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10558.253917 secs 2024-03-24T03:31:26.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:26 smithi078 ceph-mon[32429]: pgmap v6043: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:31:26.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:26 smithi107 ceph-mon[36485]: pgmap v6043: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:31:26.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:26 smithi106 ceph-mon[36505]: pgmap v6043: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:31:28.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:28 smithi078 ceph-mon[32429]: pgmap v6044: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:31:28.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:28 smithi107 ceph-mon[36485]: pgmap v6044: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:31:28.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:28 smithi106 ceph-mon[36505]: pgmap v6044: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:31:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10563.253958 secs 2024-03-24T03:31:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10563.253958 secs 2024-03-24T03:31:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10563.253958 secs 2024-03-24T03:31:30.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:30 smithi078 ceph-mon[32429]: pgmap v6045: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:31:30.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:30 smithi107 ceph-mon[36485]: pgmap v6045: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:31:30.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:30 smithi106 ceph-mon[36505]: pgmap v6045: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:31:32.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:32 smithi078 ceph-mon[32429]: pgmap v6046: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:31:32.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:32 smithi107 ceph-mon[36485]: pgmap v6046: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:31:32.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:31:32.397+0000 7fab9b7fe640 1 -- 172.21.15.78:0/1058387750 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab94005e90 con 0x7fabb4101650 2024-03-24T03:31:32.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:31:32.397+0000 7fabb0ff9640 1 -- 172.21.15.78:0/1058387750 <== mon.0 v2:172.21.15.78:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7fab94005e90 con 0x7fabb4101650 2024-03-24T03:31:32.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:32 smithi106 ceph-mon[36505]: pgmap v6046: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:31:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:34 smithi078 ceph-mon[32429]: pgmap v6047: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:34.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10568.254017 secs 2024-03-24T03:31:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:34 smithi107 ceph-mon[36485]: pgmap v6047: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10568.254017 secs 2024-03-24T03:31:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:34 smithi106 ceph-mon[36505]: pgmap v6047: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10568.254017 secs 2024-03-24T03:31:36.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:36 smithi078 ceph-mon[32429]: pgmap v6048: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:36.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:36 smithi107 ceph-mon[36485]: pgmap v6048: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:36.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:36 smithi106 ceph-mon[36505]: pgmap v6048: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:38.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:38 smithi078 ceph-mon[32429]: pgmap v6049: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:31:38.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:38 smithi107 ceph-mon[36485]: pgmap v6049: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:31:38.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:38 smithi106 ceph-mon[36505]: pgmap v6049: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:31:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10573.254088 secs 2024-03-24T03:31:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10573.254088 secs 2024-03-24T03:31:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10573.254088 secs 2024-03-24T03:31:40.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:40 smithi078 ceph-mon[32429]: pgmap v6050: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:40.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:40 smithi107 ceph-mon[36485]: pgmap v6050: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:40.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:40 smithi106 ceph-mon[36505]: pgmap v6050: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:42.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:42 smithi078 ceph-mon[32429]: pgmap v6051: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:31:42.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:42 smithi107 ceph-mon[36485]: pgmap v6051: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:31:42.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:42 smithi106 ceph-mon[36505]: pgmap v6051: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:31:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:44 smithi078 ceph-mon[32429]: pgmap v6052: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:44.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10578.254145 secs 2024-03-24T03:31:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:44 smithi107 ceph-mon[36485]: pgmap v6052: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10578.254145 secs 2024-03-24T03:31:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:44 smithi106 ceph-mon[36505]: pgmap v6052: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10578.254145 secs 2024-03-24T03:31:46.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:46 smithi078 ceph-mon[32429]: pgmap v6053: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:46.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:46 smithi107 ceph-mon[36485]: pgmap v6053: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:46.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:46 smithi106 ceph-mon[36505]: pgmap v6053: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:48.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:48 smithi078 ceph-mon[32429]: pgmap v6054: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:31:48.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:48 smithi107 ceph-mon[36485]: pgmap v6054: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:31:48.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:48 smithi106 ceph-mon[36505]: pgmap v6054: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:31:49.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10583.254228 secs 2024-03-24T03:31:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10583.254228 secs 2024-03-24T03:31:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10583.254228 secs 2024-03-24T03:31:50.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:50 smithi106 ceph-mon[36505]: pgmap v6055: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:50.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:50 smithi078 ceph-mon[32429]: pgmap v6055: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:50.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:50 smithi107 ceph-mon[36485]: pgmap v6055: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:52.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:52 smithi106 ceph-mon[36505]: pgmap v6056: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:31:52.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:52 smithi078 ceph-mon[32429]: pgmap v6056: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:31:52.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:52 smithi107 ceph-mon[36485]: pgmap v6056: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:31:54.392 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:54 smithi078 ceph-mon[32429]: pgmap v6057: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:54.392 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10588.254326 secs 2024-03-24T03:31:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:54 smithi106 ceph-mon[36505]: pgmap v6057: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10588.254326 secs 2024-03-24T03:31:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:54 smithi107 ceph-mon[36485]: pgmap v6057: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10588.254326 secs 2024-03-24T03:31:56.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:56 smithi106 ceph-mon[36505]: pgmap v6058: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:56.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:56 smithi078 ceph-mon[32429]: pgmap v6058: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:56.598 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:56 smithi107 ceph-mon[36485]: pgmap v6058: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:31:58.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:58 smithi106 ceph-mon[36505]: pgmap v6059: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:31:58.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:58 smithi078 ceph-mon[32429]: pgmap v6059: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:31:58.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:58 smithi107 ceph-mon[36485]: pgmap v6059: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:31:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:31:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10593.254413 secs 2024-03-24T03:31:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:31:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10593.254413 secs 2024-03-24T03:31:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:31:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10593.254413 secs 2024-03-24T03:32:00.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:00 smithi106 ceph-mon[36505]: pgmap v6060: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:00.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:00 smithi078 ceph-mon[32429]: pgmap v6060: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:00.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:00 smithi107 ceph-mon[36485]: pgmap v6060: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:02.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:02 smithi106 ceph-mon[36505]: pgmap v6061: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:32:02.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:02 smithi078 ceph-mon[32429]: pgmap v6061: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:32:02.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:02 smithi107 ceph-mon[36485]: pgmap v6061: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:32:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:04 smithi106 ceph-mon[36505]: pgmap v6062: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10598.254495 secs 2024-03-24T03:32:04.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:04 smithi078 ceph-mon[32429]: pgmap v6062: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:04.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10598.254495 secs 2024-03-24T03:32:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:04 smithi107 ceph-mon[36485]: pgmap v6062: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10598.254495 secs 2024-03-24T03:32:06.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:06 smithi106 ceph-mon[36505]: pgmap v6063: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:06.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:06 smithi078 ceph-mon[32429]: pgmap v6063: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:06.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:06 smithi107 ceph-mon[36485]: pgmap v6063: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:08.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:08 smithi106 ceph-mon[36505]: pgmap v6064: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:32:08.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:08 smithi078 ceph-mon[32429]: pgmap v6064: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:32:08.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:08 smithi107 ceph-mon[36485]: pgmap v6064: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:32:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10603.254556 secs 2024-03-24T03:32:09.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10603.254556 secs 2024-03-24T03:32:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10603.254556 secs 2024-03-24T03:32:10.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:10 smithi106 ceph-mon[36505]: pgmap v6065: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:10.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:10 smithi078 ceph-mon[32429]: pgmap v6065: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:10.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:10 smithi107 ceph-mon[36485]: pgmap v6065: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:12.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:12 smithi106 ceph-mon[36505]: pgmap v6066: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:32:12.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:12 smithi078 ceph-mon[32429]: pgmap v6066: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:32:12.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:12 smithi107 ceph-mon[36485]: pgmap v6066: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:32:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:14 smithi106 ceph-mon[36505]: pgmap v6067: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10608.254582 secs 2024-03-24T03:32:14.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:14 smithi078 ceph-mon[32429]: pgmap v6067: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:14.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10608.254582 secs 2024-03-24T03:32:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:14 smithi107 ceph-mon[36485]: pgmap v6067: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10608.254582 secs 2024-03-24T03:32:16.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:16 smithi106 ceph-mon[36505]: pgmap v6068: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:16.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:16 smithi078 ceph-mon[32429]: pgmap v6068: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:16.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:16 smithi107 ceph-mon[36485]: pgmap v6068: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:18.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:18 smithi106 ceph-mon[36505]: pgmap v6069: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:32:18.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:18 smithi078 ceph-mon[32429]: pgmap v6069: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:32:18.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:18 smithi107 ceph-mon[36485]: pgmap v6069: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:32:19.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10613.254627 secs 2024-03-24T03:32:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10613.254627 secs 2024-03-24T03:32:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10613.254627 secs 2024-03-24T03:32:20.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:20 smithi078 ceph-mon[32429]: pgmap v6070: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:20.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:20 smithi107 ceph-mon[36485]: pgmap v6070: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:20.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:20 smithi106 ceph-mon[36505]: pgmap v6070: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:21.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:21 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:32:21.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:21 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:32:21.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:21 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:32:21.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:21 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:32:21.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:21 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:32:21.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:21 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:32:21.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:21 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:32:21.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:21 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:32:21.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:21 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:32:22.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:22 smithi078 ceph-mon[32429]: pgmap v6071: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:32:22.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:22 smithi078 ceph-mon[32429]: pgmap v6072: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:32:22.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:22 smithi078 ceph-mon[32429]: pgmap v6073: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:32:22.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:22 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:32:22.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:22 smithi107 ceph-mon[36485]: pgmap v6071: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:32:22.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:22 smithi107 ceph-mon[36485]: pgmap v6072: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:32:22.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:22 smithi107 ceph-mon[36485]: pgmap v6073: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:32:22.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:22 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:32:22.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:22 smithi106 ceph-mon[36505]: pgmap v6071: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:32:22.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:22 smithi106 ceph-mon[36505]: pgmap v6072: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:32:22.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:22 smithi106 ceph-mon[36505]: pgmap v6073: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:32:22.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:22 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:32:24.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10618.254686 secs 2024-03-24T03:32:24.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:24 smithi078 ceph-mon[32429]: pgmap v6074: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:32:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10618.254686 secs 2024-03-24T03:32:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:24 smithi107 ceph-mon[36485]: pgmap v6074: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:32:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10618.254686 secs 2024-03-24T03:32:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:24 smithi106 ceph-mon[36505]: pgmap v6074: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:32:26.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:26 smithi078 ceph-mon[32429]: pgmap v6075: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:32:26.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:26 smithi107 ceph-mon[36485]: pgmap v6075: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:32:26.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:26 smithi106 ceph-mon[36505]: pgmap v6075: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:32:28.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:28 smithi078 ceph-mon[32429]: pgmap v6076: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:32:28.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:28 smithi107 ceph-mon[36485]: pgmap v6076: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:32:28.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:28 smithi106 ceph-mon[36505]: pgmap v6076: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:32:29.553 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10623.254769 secs 2024-03-24T03:32:29.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10623.254769 secs 2024-03-24T03:32:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10623.254769 secs 2024-03-24T03:32:30.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:30 smithi078 ceph-mon[32429]: pgmap v6077: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:32:30.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:30 smithi107 ceph-mon[36485]: pgmap v6077: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:32:30.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:30 smithi106 ceph-mon[36505]: pgmap v6077: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:32:32.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:32 smithi078 ceph-mon[32429]: pgmap v6078: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:32:32.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:32 smithi107 ceph-mon[36485]: pgmap v6078: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:32:32.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:32 smithi106 ceph-mon[36505]: pgmap v6078: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:32:34.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10628.254851 secs 2024-03-24T03:32:34.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:34 smithi078 ceph-mon[32429]: pgmap v6079: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10628.254851 secs 2024-03-24T03:32:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:34 smithi107 ceph-mon[36485]: pgmap v6079: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10628.254851 secs 2024-03-24T03:32:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:34 smithi106 ceph-mon[36505]: pgmap v6079: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:36.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:36 smithi078 ceph-mon[32429]: pgmap v6080: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:36.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:36 smithi107 ceph-mon[36485]: pgmap v6080: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:36.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:36 smithi106 ceph-mon[36505]: pgmap v6080: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:38.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:38 smithi078 ceph-mon[32429]: pgmap v6081: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:32:38.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:38 smithi107 ceph-mon[36485]: pgmap v6081: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:32:38.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:38 smithi106 ceph-mon[36505]: pgmap v6081: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:32:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10633.254910 secs 2024-03-24T03:32:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10633.254910 secs 2024-03-24T03:32:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10633.254910 secs 2024-03-24T03:32:40.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:40 smithi078 ceph-mon[32429]: pgmap v6082: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:40.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:40 smithi107 ceph-mon[36485]: pgmap v6082: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:40.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:40 smithi106 ceph-mon[36505]: pgmap v6082: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:42.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:42 smithi078 ceph-mon[32429]: pgmap v6083: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:32:42.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:42 smithi107 ceph-mon[36485]: pgmap v6083: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:32:42.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:42 smithi106 ceph-mon[36505]: pgmap v6083: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:32:44.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10638.254997 secs 2024-03-24T03:32:44.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:44 smithi078 ceph-mon[32429]: pgmap v6084: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10638.254997 secs 2024-03-24T03:32:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:44 smithi107 ceph-mon[36485]: pgmap v6084: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10638.254997 secs 2024-03-24T03:32:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:44 smithi106 ceph-mon[36505]: pgmap v6084: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:46.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:46 smithi078 ceph-mon[32429]: pgmap v6085: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:46.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:46 smithi107 ceph-mon[36485]: pgmap v6085: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:46 smithi106 ceph-mon[36505]: pgmap v6085: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:48.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:48 smithi078 ceph-mon[32429]: pgmap v6086: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:32:48.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:48 smithi107 ceph-mon[36485]: pgmap v6086: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:32:48.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:48 smithi106 ceph-mon[36505]: pgmap v6086: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:32:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10643.255057 secs 2024-03-24T03:32:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10643.255057 secs 2024-03-24T03:32:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10643.255057 secs 2024-03-24T03:32:50.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:50 smithi078 ceph-mon[32429]: pgmap v6087: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:50.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:50 smithi107 ceph-mon[36485]: pgmap v6087: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:50.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:50 smithi106 ceph-mon[36505]: pgmap v6087: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:52.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:52 smithi078 ceph-mon[32429]: pgmap v6088: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:32:52.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:52 smithi107 ceph-mon[36485]: pgmap v6088: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:32:52.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:52 smithi106 ceph-mon[36505]: pgmap v6088: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:32:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10648.255116 secs 2024-03-24T03:32:54.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:54 smithi078 ceph-mon[32429]: pgmap v6089: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10648.255116 secs 2024-03-24T03:32:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:54 smithi107 ceph-mon[36485]: pgmap v6089: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10648.255116 secs 2024-03-24T03:32:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:54 smithi106 ceph-mon[36505]: pgmap v6089: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:56.572 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:56 smithi078 ceph-mon[32429]: pgmap v6090: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:56.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:56 smithi107 ceph-mon[36485]: pgmap v6090: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:56 smithi106 ceph-mon[36505]: pgmap v6090: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:32:58.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:58 smithi078 ceph-mon[32429]: pgmap v6091: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:32:58.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:58 smithi107 ceph-mon[36485]: pgmap v6091: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:32:58.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:58 smithi106 ceph-mon[36505]: pgmap v6091: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:32:59.553 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:32:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10653.255204 secs 2024-03-24T03:32:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:32:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10653.255204 secs 2024-03-24T03:32:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:32:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10653.255204 secs 2024-03-24T03:33:00.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:00 smithi078 ceph-mon[32429]: pgmap v6092: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:00.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:00 smithi107 ceph-mon[36485]: pgmap v6092: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:00.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:00 smithi106 ceph-mon[36505]: pgmap v6092: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:02.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:02 smithi078 ceph-mon[32429]: pgmap v6093: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:33:02.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:02 smithi107 ceph-mon[36485]: pgmap v6093: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:33:02.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:02 smithi106 ceph-mon[36505]: pgmap v6093: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:33:04.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10658.255293 secs 2024-03-24T03:33:04.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:04 smithi078 ceph-mon[32429]: pgmap v6094: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10658.255293 secs 2024-03-24T03:33:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:04 smithi107 ceph-mon[36485]: pgmap v6094: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10658.255293 secs 2024-03-24T03:33:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:04 smithi106 ceph-mon[36505]: pgmap v6094: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:06.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:06 smithi078 ceph-mon[32429]: pgmap v6095: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:06.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:06 smithi107 ceph-mon[36485]: pgmap v6095: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:06.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:06 smithi106 ceph-mon[36505]: pgmap v6095: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:08.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:08 smithi078 ceph-mon[32429]: pgmap v6096: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:33:08.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:08 smithi107 ceph-mon[36485]: pgmap v6096: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:33:08.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:08 smithi106 ceph-mon[36505]: pgmap v6096: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:33:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10663.255350 secs 2024-03-24T03:33:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10663.255350 secs 2024-03-24T03:33:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10663.255350 secs 2024-03-24T03:33:10.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:10 smithi078 ceph-mon[32429]: pgmap v6097: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:10.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:10 smithi107 ceph-mon[36485]: pgmap v6097: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:10.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:10 smithi106 ceph-mon[36505]: pgmap v6097: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:12.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:12 smithi078 ceph-mon[32429]: pgmap v6098: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:33:12.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:12 smithi107 ceph-mon[36485]: pgmap v6098: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:33:12.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:12 smithi106 ceph-mon[36505]: pgmap v6098: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:33:14.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10668.255409 secs 2024-03-24T03:33:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:14 smithi078 ceph-mon[32429]: pgmap v6099: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10668.255409 secs 2024-03-24T03:33:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:14 smithi107 ceph-mon[36485]: pgmap v6099: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10668.255409 secs 2024-03-24T03:33:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:14 smithi106 ceph-mon[36505]: pgmap v6099: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:16.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:16 smithi078 ceph-mon[32429]: pgmap v6100: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:16.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:16 smithi107 ceph-mon[36485]: pgmap v6100: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:16 smithi106 ceph-mon[36505]: pgmap v6100: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:18.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:18 smithi078 ceph-mon[32429]: pgmap v6101: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:33:18.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:18 smithi107 ceph-mon[36485]: pgmap v6101: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:33:18.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:18 smithi106 ceph-mon[36505]: pgmap v6101: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:33:19.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10673.255470 secs 2024-03-24T03:33:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10673.255470 secs 2024-03-24T03:33:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10673.255470 secs 2024-03-24T03:33:20.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:20 smithi078 ceph-mon[32429]: pgmap v6102: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:20.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:20 smithi107 ceph-mon[36485]: pgmap v6102: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:20.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:20 smithi106 ceph-mon[36505]: pgmap v6102: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:21.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:21 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:33:21.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:21 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:33:21.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:21 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:33:22.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:22 smithi078 ceph-mon[32429]: pgmap v6103: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:33:22.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:22 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:33:22.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:22 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:33:22.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:22 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:33:22.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:22 smithi107 ceph-mon[36485]: pgmap v6103: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:33:22.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:22 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:33:22.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:22 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:33:22.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:22 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:33:22.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:22 smithi106 ceph-mon[36505]: pgmap v6103: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:33:22.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:22 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:33:22.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:22 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:33:22.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:22 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:33:23.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:23 smithi078 ceph-mon[32429]: pgmap v6104: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:33:23.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:23 smithi078 ceph-mon[32429]: pgmap v6105: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:33:23.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:23 smithi107 ceph-mon[36485]: pgmap v6104: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:33:23.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:23 smithi107 ceph-mon[36485]: pgmap v6105: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:33:23.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:23 smithi106 ceph-mon[36505]: pgmap v6104: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:33:23.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:23 smithi106 ceph-mon[36505]: pgmap v6105: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:33:24.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10678.255529 secs 2024-03-24T03:33:24.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:24 smithi078 ceph-mon[32429]: pgmap v6106: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:33:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10678.255529 secs 2024-03-24T03:33:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:24 smithi107 ceph-mon[36485]: pgmap v6106: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:33:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10678.255529 secs 2024-03-24T03:33:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:24 smithi106 ceph-mon[36505]: pgmap v6106: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:33:27.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:27 smithi078 ceph-mon[32429]: pgmap v6107: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:33:27.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:27 smithi107 ceph-mon[36485]: pgmap v6107: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:33:27.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:27 smithi106 ceph-mon[36505]: pgmap v6107: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:33:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:29 smithi078 ceph-mon[32429]: pgmap v6108: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:33:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10683.255591 secs 2024-03-24T03:33:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:29 smithi107 ceph-mon[36485]: pgmap v6108: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:33:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10683.255591 secs 2024-03-24T03:33:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:29 smithi106 ceph-mon[36505]: pgmap v6108: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:33:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10683.255591 secs 2024-03-24T03:33:31.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:31 smithi078 ceph-mon[32429]: pgmap v6109: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:33:31.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:31 smithi107 ceph-mon[36485]: pgmap v6109: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:33:31.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:31 smithi106 ceph-mon[36505]: pgmap v6109: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:33:33.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:33 smithi078 ceph-mon[32429]: pgmap v6110: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:33:33.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:33 smithi107 ceph-mon[36485]: pgmap v6110: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:33:33.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:33 smithi106 ceph-mon[36505]: pgmap v6110: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:33:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10688.255651 secs 2024-03-24T03:33:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10688.255651 secs 2024-03-24T03:33:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10688.255651 secs 2024-03-24T03:33:35.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:35 smithi078 ceph-mon[32429]: pgmap v6111: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:35.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:35 smithi107 ceph-mon[36485]: pgmap v6111: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:35.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:35 smithi106 ceph-mon[36505]: pgmap v6111: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:37.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:37 smithi078 ceph-mon[32429]: pgmap v6112: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:33:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:37 smithi107 ceph-mon[36485]: pgmap v6112: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:33:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:37 smithi106 ceph-mon[36505]: pgmap v6112: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:33:39.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:39 smithi078 ceph-mon[32429]: pgmap v6113: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:39.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10693.255732 secs 2024-03-24T03:33:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:39 smithi107 ceph-mon[36485]: pgmap v6113: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10693.255732 secs 2024-03-24T03:33:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:39 smithi106 ceph-mon[36505]: pgmap v6113: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10693.255732 secs 2024-03-24T03:33:41.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:41 smithi078 ceph-mon[32429]: pgmap v6114: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:41 smithi107 ceph-mon[36485]: pgmap v6114: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:41.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:41 smithi106 ceph-mon[36505]: pgmap v6114: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:43 smithi078 ceph-mon[32429]: pgmap v6115: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:33:43.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:43 smithi107 ceph-mon[36485]: pgmap v6115: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:33:43.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:43 smithi106 ceph-mon[36505]: pgmap v6115: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:33:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10698.255744 secs 2024-03-24T03:33:44.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10698.255744 secs 2024-03-24T03:33:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10698.255744 secs 2024-03-24T03:33:45.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:45 smithi078 ceph-mon[32429]: pgmap v6116: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:45.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:45 smithi107 ceph-mon[36485]: pgmap v6116: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:45.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:45 smithi106 ceph-mon[36505]: pgmap v6116: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:47.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:47 smithi078 ceph-mon[32429]: pgmap v6117: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:33:47.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:47 smithi107 ceph-mon[36485]: pgmap v6117: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:33:47.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:47 smithi106 ceph-mon[36505]: pgmap v6117: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:33:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:49 smithi078 ceph-mon[32429]: pgmap v6118: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:49.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10703.255804 secs 2024-03-24T03:33:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:49 smithi107 ceph-mon[36485]: pgmap v6118: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10703.255804 secs 2024-03-24T03:33:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:49 smithi106 ceph-mon[36505]: pgmap v6118: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10703.255804 secs 2024-03-24T03:33:51.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:51 smithi078 ceph-mon[32429]: pgmap v6119: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:51.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:51 smithi107 ceph-mon[36485]: pgmap v6119: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:51.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:51 smithi106 ceph-mon[36505]: pgmap v6119: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:53.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:53 smithi078 ceph-mon[32429]: pgmap v6120: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:33:53.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:53 smithi107 ceph-mon[36485]: pgmap v6120: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:33:53.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:53 smithi106 ceph-mon[36505]: pgmap v6120: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:33:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10708.255858 secs 2024-03-24T03:33:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10708.255858 secs 2024-03-24T03:33:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10708.255858 secs 2024-03-24T03:33:55.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:55 smithi078 ceph-mon[32429]: pgmap v6121: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:55.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:55 smithi107 ceph-mon[36485]: pgmap v6121: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:55.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:55 smithi106 ceph-mon[36505]: pgmap v6121: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:57.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:57 smithi078 ceph-mon[32429]: pgmap v6122: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:33:57.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:57 smithi107 ceph-mon[36485]: pgmap v6122: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:33:57.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:57 smithi106 ceph-mon[36505]: pgmap v6122: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:33:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:59 smithi078 ceph-mon[32429]: pgmap v6123: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:33:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10713.255891 secs 2024-03-24T03:33:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:59 smithi107 ceph-mon[36485]: pgmap v6123: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:33:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10713.255891 secs 2024-03-24T03:33:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:59 smithi106 ceph-mon[36505]: pgmap v6123: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:33:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:33:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10713.255891 secs 2024-03-24T03:34:01.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:01 smithi078 ceph-mon[32429]: pgmap v6124: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:01 smithi107 ceph-mon[36485]: pgmap v6124: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:01 smithi106 ceph-mon[36505]: pgmap v6124: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:03.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:03 smithi078 ceph-mon[32429]: pgmap v6125: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:34:03.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:03 smithi107 ceph-mon[36485]: pgmap v6125: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:34:03.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:03 smithi106 ceph-mon[36505]: pgmap v6125: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:34:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10718.255950 secs 2024-03-24T03:34:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10718.255950 secs 2024-03-24T03:34:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10718.255950 secs 2024-03-24T03:34:05.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:05 smithi078 ceph-mon[32429]: pgmap v6126: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:05.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:05 smithi107 ceph-mon[36485]: pgmap v6126: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:05.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:05 smithi106 ceph-mon[36505]: pgmap v6126: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:07.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:07 smithi078 ceph-mon[32429]: pgmap v6127: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:34:07.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:07 smithi107 ceph-mon[36485]: pgmap v6127: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:34:07.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:07 smithi106 ceph-mon[36505]: pgmap v6127: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:34:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:09 smithi106 ceph-mon[36505]: pgmap v6128: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10723.255990 secs 2024-03-24T03:34:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:09 smithi078 ceph-mon[32429]: pgmap v6128: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10723.255990 secs 2024-03-24T03:34:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:09 smithi107 ceph-mon[36485]: pgmap v6128: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10723.255990 secs 2024-03-24T03:34:11.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:11 smithi106 ceph-mon[36505]: pgmap v6129: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:11.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:11 smithi078 ceph-mon[32429]: pgmap v6129: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:11.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:11 smithi107 ceph-mon[36485]: pgmap v6129: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:13.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:13 smithi106 ceph-mon[36505]: pgmap v6130: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:34:13.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:13 smithi078 ceph-mon[32429]: pgmap v6130: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:34:13.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:13 smithi107 ceph-mon[36485]: pgmap v6130: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:34:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10728.256077 secs 2024-03-24T03:34:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10728.256077 secs 2024-03-24T03:34:14.494 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10728.256077 secs 2024-03-24T03:34:15.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:15 smithi106 ceph-mon[36505]: pgmap v6131: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:15.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:15 smithi078 ceph-mon[32429]: pgmap v6131: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:15.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:15 smithi107 ceph-mon[36485]: pgmap v6131: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:17.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:17 smithi106 ceph-mon[36505]: pgmap v6132: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:34:17.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:17 smithi078 ceph-mon[32429]: pgmap v6132: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:34:17.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:17 smithi107 ceph-mon[36485]: pgmap v6132: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:34:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:19 smithi106 ceph-mon[36505]: pgmap v6133: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10733.256165 secs 2024-03-24T03:34:19.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:19 smithi078 ceph-mon[32429]: pgmap v6133: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:19.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10733.256165 secs 2024-03-24T03:34:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:19 smithi107 ceph-mon[36485]: pgmap v6133: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10733.256165 secs 2024-03-24T03:34:21.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:21 smithi106 ceph-mon[36505]: pgmap v6134: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:21.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:21 smithi078 ceph-mon[32429]: pgmap v6134: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:21.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:21 smithi107 ceph-mon[36485]: pgmap v6134: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:22.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:22 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:34:22.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:22 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:34:22.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:22 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:34:23.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:23 smithi106 ceph-mon[36505]: pgmap v6135: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:34:23.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:23 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:34:23.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:23 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:34:23.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:23 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:34:23.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:23 smithi078 ceph-mon[32429]: pgmap v6135: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:34:23.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:23 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:34:23.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:23 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:34:23.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:23 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:34:23.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:23 smithi107 ceph-mon[36485]: pgmap v6135: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:34:23.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:23 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:34:23.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:23 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:34:23.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:23 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:34:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:24 smithi106 ceph-mon[36505]: pgmap v6136: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:34:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:24 smithi106 ceph-mon[36505]: pgmap v6137: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:34:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10738.256226 secs 2024-03-24T03:34:24.492 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:24 smithi078 ceph-mon[32429]: pgmap v6136: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:34:24.492 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:24 smithi078 ceph-mon[32429]: pgmap v6137: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:34:24.492 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10738.256226 secs 2024-03-24T03:34:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:24 smithi107 ceph-mon[36485]: pgmap v6136: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:34:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:24 smithi107 ceph-mon[36485]: pgmap v6137: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:34:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10738.256226 secs 2024-03-24T03:34:26.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:26 smithi106 ceph-mon[36505]: pgmap v6138: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:34:26.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:26 smithi078 ceph-mon[32429]: pgmap v6138: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:34:26.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:26 smithi107 ceph-mon[36485]: pgmap v6138: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:34:28.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:28 smithi106 ceph-mon[36505]: pgmap v6139: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:34:28.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:28 smithi078 ceph-mon[32429]: pgmap v6139: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:34:28.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:28 smithi107 ceph-mon[36485]: pgmap v6139: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:34:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10743.256309 secs 2024-03-24T03:34:29.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10743.256309 secs 2024-03-24T03:34:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10743.256309 secs 2024-03-24T03:34:30.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:30 smithi106 ceph-mon[36505]: pgmap v6140: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:34:30.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:30 smithi078 ceph-mon[32429]: pgmap v6140: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:34:30.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:30 smithi107 ceph-mon[36485]: pgmap v6140: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:34:32.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:32 smithi106 ceph-mon[36505]: pgmap v6141: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:34:32.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:34:32.422+0000 7fab9b7fe640 1 -- 172.21.15.78:0/1058387750 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab940060b0 con 0x7fabb4101650 2024-03-24T03:34:32.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:34:32.422+0000 7fabb0ff9640 1 -- 172.21.15.78:0/1058387750 <== mon.0 v2:172.21.15.78:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7fab940060b0 con 0x7fabb4101650 2024-03-24T03:34:32.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:32 smithi078 ceph-mon[32429]: pgmap v6141: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:34:32.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:32 smithi107 ceph-mon[36485]: pgmap v6141: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:34:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:34 smithi106 ceph-mon[36505]: pgmap v6142: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:34:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10748.256367 secs 2024-03-24T03:34:34.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:34 smithi078 ceph-mon[32429]: pgmap v6142: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:34:34.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10748.256367 secs 2024-03-24T03:34:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:34 smithi107 ceph-mon[36485]: pgmap v6142: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:34:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10748.256367 secs 2024-03-24T03:34:36.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:36 smithi106 ceph-mon[36505]: pgmap v6143: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:36.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:36 smithi078 ceph-mon[32429]: pgmap v6143: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:36.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:36 smithi107 ceph-mon[36485]: pgmap v6143: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:38.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:38 smithi078 ceph-mon[32429]: pgmap v6144: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:34:38.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:38 smithi107 ceph-mon[36485]: pgmap v6144: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:34:38.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:38 smithi106 ceph-mon[36505]: pgmap v6144: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:34:39.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10753.256427 secs 2024-03-24T03:34:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10753.256427 secs 2024-03-24T03:34:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10753.256427 secs 2024-03-24T03:34:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:40 smithi078 ceph-mon[32429]: pgmap v6145: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:40.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:40 smithi107 ceph-mon[36485]: pgmap v6145: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:40.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:40 smithi106 ceph-mon[36505]: pgmap v6145: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:42.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:42 smithi078 ceph-mon[32429]: pgmap v6146: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:34:42.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:42 smithi107 ceph-mon[36485]: pgmap v6146: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:34:42.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:42 smithi106 ceph-mon[36505]: pgmap v6146: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:34:44.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:44 smithi078 ceph-mon[32429]: pgmap v6147: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:44.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10758.256482 secs 2024-03-24T03:34:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:44 smithi107 ceph-mon[36485]: pgmap v6147: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10758.256482 secs 2024-03-24T03:34:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:44 smithi106 ceph-mon[36505]: pgmap v6147: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10758.256482 secs 2024-03-24T03:34:46.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:46 smithi078 ceph-mon[32429]: pgmap v6148: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:46.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:46 smithi107 ceph-mon[36485]: pgmap v6148: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:46 smithi106 ceph-mon[36505]: pgmap v6148: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:48.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:48 smithi078 ceph-mon[32429]: pgmap v6149: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:34:48.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:48 smithi107 ceph-mon[36485]: pgmap v6149: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:34:48.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:48 smithi106 ceph-mon[36505]: pgmap v6149: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:34:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10763.256496 secs 2024-03-24T03:34:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10763.256496 secs 2024-03-24T03:34:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10763.256496 secs 2024-03-24T03:34:50.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:50 smithi078 ceph-mon[32429]: pgmap v6150: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:50.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:50 smithi107 ceph-mon[36485]: pgmap v6150: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:50.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:50 smithi106 ceph-mon[36505]: pgmap v6150: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:52.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:52 smithi078 ceph-mon[32429]: pgmap v6151: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:34:52.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:52 smithi107 ceph-mon[36485]: pgmap v6151: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:34:52.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:52 smithi106 ceph-mon[36505]: pgmap v6151: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:34:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:54 smithi078 ceph-mon[32429]: pgmap v6152: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:54.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10768.256575 secs 2024-03-24T03:34:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:54 smithi107 ceph-mon[36485]: pgmap v6152: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10768.256575 secs 2024-03-24T03:34:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:54 smithi106 ceph-mon[36505]: pgmap v6152: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10768.256575 secs 2024-03-24T03:34:56.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:56 smithi078 ceph-mon[32429]: pgmap v6153: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:56.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:56 smithi107 ceph-mon[36485]: pgmap v6153: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:56 smithi106 ceph-mon[36505]: pgmap v6153: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:34:58.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:58 smithi078 ceph-mon[32429]: pgmap v6154: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:34:58.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:58 smithi107 ceph-mon[36485]: pgmap v6154: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:34:58.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:58 smithi106 ceph-mon[36505]: pgmap v6154: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:34:59.553 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:34:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10773.256619 secs 2024-03-24T03:34:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:34:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10773.256619 secs 2024-03-24T03:34:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:34:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10773.256619 secs 2024-03-24T03:35:00.476 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:00 smithi078 ceph-mon[32429]: pgmap v6155: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:00.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:00 smithi107 ceph-mon[36485]: pgmap v6155: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:00.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:00 smithi106 ceph-mon[36505]: pgmap v6155: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:02.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:02 smithi078 ceph-mon[32429]: pgmap v6156: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:35:02.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:02 smithi107 ceph-mon[36485]: pgmap v6156: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:35:02.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:02 smithi106 ceph-mon[36505]: pgmap v6156: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:35:04.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:04 smithi078 ceph-mon[32429]: pgmap v6157: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:04.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10778.256702 secs 2024-03-24T03:35:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:04 smithi107 ceph-mon[36485]: pgmap v6157: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10778.256702 secs 2024-03-24T03:35:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:04 smithi106 ceph-mon[36505]: pgmap v6157: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10778.256702 secs 2024-03-24T03:35:06.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:06 smithi078 ceph-mon[32429]: pgmap v6158: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:06.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:06 smithi107 ceph-mon[36485]: pgmap v6158: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:06.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:06 smithi106 ceph-mon[36505]: pgmap v6158: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:08.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:08 smithi078 ceph-mon[32429]: pgmap v6159: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:35:08.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:08 smithi107 ceph-mon[36485]: pgmap v6159: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:35:08.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:08 smithi106 ceph-mon[36505]: pgmap v6159: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:35:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10783.256791 secs 2024-03-24T03:35:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10783.256791 secs 2024-03-24T03:35:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10783.256791 secs 2024-03-24T03:35:10.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:10 smithi078 ceph-mon[32429]: pgmap v6160: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:10.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:10 smithi107 ceph-mon[36485]: pgmap v6160: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:10.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:10 smithi106 ceph-mon[36505]: pgmap v6160: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:12.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:12 smithi078 ceph-mon[32429]: pgmap v6161: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:35:12.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:12 smithi107 ceph-mon[36485]: pgmap v6161: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:35:12.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:12 smithi106 ceph-mon[36505]: pgmap v6161: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:35:14.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:14 smithi078 ceph-mon[32429]: pgmap v6162: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10788.256874 secs 2024-03-24T03:35:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:14 smithi107 ceph-mon[36485]: pgmap v6162: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10788.256874 secs 2024-03-24T03:35:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:14 smithi106 ceph-mon[36505]: pgmap v6162: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10788.256874 secs 2024-03-24T03:35:16.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:16 smithi078 ceph-mon[32429]: pgmap v6163: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:16.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:16 smithi107 ceph-mon[36485]: pgmap v6163: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:16 smithi106 ceph-mon[36505]: pgmap v6163: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:18.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:18 smithi078 ceph-mon[32429]: pgmap v6164: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:35:18.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:18 smithi107 ceph-mon[36485]: pgmap v6164: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:35:18.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:18 smithi106 ceph-mon[36505]: pgmap v6164: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:35:19.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10793.256934 secs 2024-03-24T03:35:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10793.256934 secs 2024-03-24T03:35:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10793.256934 secs 2024-03-24T03:35:20.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:20 smithi078 ceph-mon[32429]: pgmap v6165: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:20.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:20 smithi107 ceph-mon[36485]: pgmap v6165: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:20.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:20 smithi106 ceph-mon[36505]: pgmap v6165: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:22.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:22 smithi078 ceph-mon[32429]: pgmap v6166: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:35:22.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:22 smithi107 ceph-mon[36485]: pgmap v6166: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:35:22.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:22 smithi106 ceph-mon[36505]: pgmap v6166: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:35:23.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:23 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:35:23.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:23 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:35:23.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:23 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:35:24.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:24 smithi078 ceph-mon[32429]: pgmap v6167: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:24.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10798.257028 secs 2024-03-24T03:35:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:24 smithi107 ceph-mon[36485]: pgmap v6167: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10798.257028 secs 2024-03-24T03:35:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:24 smithi106 ceph-mon[36505]: pgmap v6167: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10798.257028 secs 2024-03-24T03:35:26.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:25 smithi078 ceph-mon[32429]: pgmap v6168: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:26.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:35:26.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:35:26.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:35:26.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:35:26.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:35:26.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:35:26.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:26.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:26.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:26.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:26.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:26.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:26.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:26.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:25 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:26.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:25 smithi107 ceph-mon[36485]: pgmap v6168: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:26.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:35:26.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:35:26.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:35:26.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:35:26.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:35:26.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:35:26.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:26.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:26.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:26.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:26.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:26.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:26.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:26.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:25 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:26.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:25 smithi106 ceph-mon[36505]: pgmap v6168: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:26.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:35:26.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:35:26.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:35:26.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:35:26.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:35:26.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:35:26.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:26.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:26.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:26.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:26.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:26.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:26.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:26.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:25 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:27.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:26 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T03:35:27.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:26 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T03:35:27.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:26 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T03:35:27.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:26 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T03:35:27.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:26 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:27.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:26 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:27.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:26 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:27.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:26 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:27.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:26 smithi078 ceph-mon[32429]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T03:35:27.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:26 smithi078 ceph-mon[32429]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T03:35:27.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:26 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:35:27.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:26 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:35:27.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:26 smithi078 ceph-mon[32429]: pgmap v6169: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T03:35:27.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:26 smithi078 ceph-mon[32429]: pgmap v6170: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:35:27.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:26 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:35:27.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:26 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T03:35:27.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:26 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T03:35:27.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:26 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T03:35:27.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:26 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T03:35:27.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:26 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:27.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:26 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:27.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:26 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:27.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:26 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:27.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:26 smithi107 ceph-mon[36485]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T03:35:27.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:26 smithi107 ceph-mon[36485]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T03:35:27.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:26 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:35:27.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:26 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:35:27.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:26 smithi107 ceph-mon[36485]: pgmap v6169: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T03:35:27.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:26 smithi107 ceph-mon[36485]: pgmap v6170: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:35:27.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:26 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:35:27.098 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:26 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-24T03:35:27.098 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:26 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi106 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T03:35:27.098 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:26 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-24T03:35:27.098 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:26 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi107 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-24T03:35:27.098 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:26 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:27.098 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:26 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:27.099 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:26 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:27.099 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:26 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:35:27.099 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:26 smithi106 ceph-mon[36505]: Adjusting osd_memory_target on smithi078 to 176.2M 2024-03-24T03:35:27.099 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:26 smithi106 ceph-mon[36505]: Unable to set osd_memory_target on smithi078 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-24T03:35:27.099 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:26 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:35:27.099 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:26 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:35:27.099 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:26 smithi106 ceph-mon[36505]: pgmap v6169: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T03:35:27.099 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:26 smithi106 ceph-mon[36505]: pgmap v6170: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:35:27.099 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:26 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:35:29.204 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:28 smithi106 ceph-mon[36505]: pgmap v6171: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:35:29.204 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:28 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10803.257087 secs 2024-03-24T03:35:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:28 smithi078 ceph-mon[32429]: pgmap v6171: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:35:29.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:28 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10803.257087 secs 2024-03-24T03:35:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:28 smithi107 ceph-mon[36485]: pgmap v6171: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:35:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:28 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10803.257087 secs 2024-03-24T03:35:31.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:30 smithi078 ceph-mon[32429]: pgmap v6172: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:35:31.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:30 smithi107 ceph-mon[36485]: pgmap v6172: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:35:31.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:30 smithi106 ceph-mon[36505]: pgmap v6172: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:35:33.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:32 smithi078 ceph-mon[32429]: pgmap v6173: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:35:33.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:32 smithi107 ceph-mon[36485]: pgmap v6173: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:35:33.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:32 smithi106 ceph-mon[36505]: pgmap v6173: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:35:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:33 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10808.257147 secs 2024-03-24T03:35:34.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:33 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10808.257147 secs 2024-03-24T03:35:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:33 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10808.257147 secs 2024-03-24T03:35:35.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:34 smithi078 ceph-mon[32429]: pgmap v6174: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:35:35.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:34 smithi107 ceph-mon[36485]: pgmap v6174: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:35:35.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:34 smithi106 ceph-mon[36505]: pgmap v6174: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T03:35:37.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:36 smithi078 ceph-mon[32429]: pgmap v6175: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:35:37.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:36 smithi107 ceph-mon[36485]: pgmap v6175: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:35:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:36 smithi106 ceph-mon[36505]: pgmap v6175: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:35:39.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:38 smithi078 ceph-mon[32429]: pgmap v6176: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:39.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:38 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10813.257208 secs 2024-03-24T03:35:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:38 smithi107 ceph-mon[36485]: pgmap v6176: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:39.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:38 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10813.257208 secs 2024-03-24T03:35:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:38 smithi106 ceph-mon[36505]: pgmap v6176: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:38 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10813.257208 secs 2024-03-24T03:35:41.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:40 smithi078 ceph-mon[32429]: pgmap v6177: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:41.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:40 smithi107 ceph-mon[36485]: pgmap v6177: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:41.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:40 smithi106 ceph-mon[36505]: pgmap v6177: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:43.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:42 smithi078 ceph-mon[32429]: pgmap v6178: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:35:43.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:42 smithi107 ceph-mon[36485]: pgmap v6178: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:35:43.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:42 smithi106 ceph-mon[36505]: pgmap v6178: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:35:44.244 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:43 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10818.257297 secs 2024-03-24T03:35:44.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:43 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10818.257297 secs 2024-03-24T03:35:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:43 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10818.257297 secs 2024-03-24T03:35:45.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:44 smithi078 ceph-mon[32429]: pgmap v6179: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:45.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:44 smithi107 ceph-mon[36485]: pgmap v6179: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:45.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:44 smithi106 ceph-mon[36505]: pgmap v6179: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:47.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:46 smithi078 ceph-mon[32429]: pgmap v6180: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:35:47.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:46 smithi107 ceph-mon[36485]: pgmap v6180: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:35:47.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:46 smithi106 ceph-mon[36505]: pgmap v6180: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:35:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:48 smithi078 ceph-mon[32429]: pgmap v6181: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:49.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:48 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10823.257389 secs 2024-03-24T03:35:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:48 smithi107 ceph-mon[36485]: pgmap v6181: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:49.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:48 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10823.257389 secs 2024-03-24T03:35:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:48 smithi106 ceph-mon[36505]: pgmap v6181: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:48 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10823.257389 secs 2024-03-24T03:35:51.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:50 smithi078 ceph-mon[32429]: pgmap v6182: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:51.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:50 smithi107 ceph-mon[36485]: pgmap v6182: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:51.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:50 smithi106 ceph-mon[36505]: pgmap v6182: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:53.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:52 smithi078 ceph-mon[32429]: pgmap v6183: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:35:53.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:53 smithi107 ceph-mon[36485]: pgmap v6183: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:35:53.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:52 smithi106 ceph-mon[36505]: pgmap v6183: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:35:54.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10828.257448 secs 2024-03-24T03:35:54.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10828.257448 secs 2024-03-24T03:35:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10828.257448 secs 2024-03-24T03:35:55.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:55 smithi078 ceph-mon[32429]: pgmap v6184: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:55.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:55 smithi107 ceph-mon[36485]: pgmap v6184: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:55.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:55 smithi106 ceph-mon[36505]: pgmap v6184: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:57.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:57 smithi078 ceph-mon[32429]: pgmap v6185: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:35:57.357 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:57 smithi107 ceph-mon[36485]: pgmap v6185: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:35:57.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:57 smithi106 ceph-mon[36505]: pgmap v6185: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:35:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:59 smithi078 ceph-mon[32429]: pgmap v6186: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:59.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:35:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10833.257507 secs 2024-03-24T03:35:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:59 smithi107 ceph-mon[36485]: pgmap v6186: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:35:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10833.257507 secs 2024-03-24T03:35:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:59 smithi106 ceph-mon[36505]: pgmap v6186: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:35:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:35:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10833.257507 secs 2024-03-24T03:36:01.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:01 smithi078 ceph-mon[32429]: pgmap v6187: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:01.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:01 smithi107 ceph-mon[36485]: pgmap v6187: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:01.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:01 smithi106 ceph-mon[36505]: pgmap v6187: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:03.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:03 smithi078 ceph-mon[32429]: pgmap v6188: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:36:03.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:03 smithi107 ceph-mon[36485]: pgmap v6188: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:36:03.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:03 smithi106 ceph-mon[36505]: pgmap v6188: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:36:04.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10838.257589 secs 2024-03-24T03:36:04.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10838.257589 secs 2024-03-24T03:36:04.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10838.257589 secs 2024-03-24T03:36:05.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:05 smithi078 ceph-mon[32429]: pgmap v6189: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:05.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:05 smithi107 ceph-mon[36485]: pgmap v6189: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:05.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:05 smithi106 ceph-mon[36505]: pgmap v6189: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:07.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:07 smithi078 ceph-mon[32429]: pgmap v6190: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:36:07.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:07 smithi107 ceph-mon[36485]: pgmap v6190: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:36:07.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:07 smithi106 ceph-mon[36505]: pgmap v6190: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:36:09.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:09 smithi078 ceph-mon[32429]: pgmap v6191: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:09.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10843.257650 secs 2024-03-24T03:36:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:09 smithi107 ceph-mon[36485]: pgmap v6191: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:09.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10843.257650 secs 2024-03-24T03:36:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:09 smithi106 ceph-mon[36505]: pgmap v6191: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:09.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10843.257650 secs 2024-03-24T03:36:11.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:11 smithi078 ceph-mon[32429]: pgmap v6192: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:11.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:11 smithi107 ceph-mon[36485]: pgmap v6192: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:11.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:11 smithi106 ceph-mon[36505]: pgmap v6192: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:13.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:13 smithi078 ceph-mon[32429]: pgmap v6193: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:36:13.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:13 smithi107 ceph-mon[36485]: pgmap v6193: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:36:13.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:13 smithi106 ceph-mon[36505]: pgmap v6193: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:36:14.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10848.257663 secs 2024-03-24T03:36:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10848.257663 secs 2024-03-24T03:36:14.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10848.257663 secs 2024-03-24T03:36:15.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:15 smithi078 ceph-mon[32429]: pgmap v6194: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:15.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:15 smithi107 ceph-mon[36485]: pgmap v6194: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:15.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:15 smithi106 ceph-mon[36505]: pgmap v6194: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:17.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:17 smithi078 ceph-mon[32429]: pgmap v6195: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:36:17.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:17 smithi107 ceph-mon[36485]: pgmap v6195: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:36:17.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:17 smithi106 ceph-mon[36505]: pgmap v6195: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:36:19.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:19 smithi078 ceph-mon[32429]: pgmap v6196: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:19.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10853.257743 secs 2024-03-24T03:36:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:19 smithi107 ceph-mon[36485]: pgmap v6196: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:19.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10853.257743 secs 2024-03-24T03:36:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:19 smithi106 ceph-mon[36505]: pgmap v6196: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:19.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10853.257743 secs 2024-03-24T03:36:21.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:21 smithi078 ceph-mon[32429]: pgmap v6197: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:21.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:21 smithi107 ceph-mon[36485]: pgmap v6197: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:21.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:21 smithi106 ceph-mon[36505]: pgmap v6197: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:23.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:23 smithi078 ceph-mon[32429]: pgmap v6198: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:36:23.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:23 smithi107 ceph-mon[36485]: pgmap v6198: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:36:23.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:23 smithi106 ceph-mon[36505]: pgmap v6198: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:36:24.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10858.257825 secs 2024-03-24T03:36:24.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10858.257825 secs 2024-03-24T03:36:24.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10858.257825 secs 2024-03-24T03:36:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:25 smithi078 ceph-mon[32429]: pgmap v6199: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:25.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:25 smithi107 ceph-mon[36485]: pgmap v6199: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:25.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:25 smithi106 ceph-mon[36505]: pgmap v6199: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:26.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:26 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:36:26.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:26 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:36:26.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:26 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:36:27.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:27 smithi078 ceph-mon[32429]: pgmap v6200: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:36:27.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:27 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:36:27.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:27 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:36:27.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:27 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:36:27.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:27 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:36:27.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:27 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:36:27.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:27 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:36:27.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:27 smithi107 ceph-mon[36485]: pgmap v6200: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:36:27.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:27 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:36:27.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:27 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:36:27.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:27 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:36:27.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:27 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:36:27.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:27 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:36:27.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:27 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:36:27.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:27 smithi106 ceph-mon[36505]: pgmap v6200: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:36:27.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:27 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:36:27.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:27 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:36:27.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:27 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:36:27.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:27 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:36:27.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:27 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:36:27.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:27 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:36:28.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:28 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:36:28.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:28 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:36:28.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:28 smithi078 ceph-mon[32429]: pgmap v6201: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T03:36:28.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:28 smithi078 ceph-mon[32429]: pgmap v6202: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:36:28.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:28 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:36:28.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:28 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:36:28.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:28 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:36:28.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:28 smithi107 ceph-mon[36485]: pgmap v6201: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T03:36:28.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:28 smithi107 ceph-mon[36485]: pgmap v6202: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:36:28.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:28 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:36:28.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:28 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:36:28.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:28 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:36:28.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:28 smithi106 ceph-mon[36505]: pgmap v6201: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T03:36:28.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:28 smithi106 ceph-mon[36505]: pgmap v6202: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:36:28.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:28 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:36:29.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10863.257914 secs 2024-03-24T03:36:29.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10863.257914 secs 2024-03-24T03:36:29.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10863.257914 secs 2024-03-24T03:36:30.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:30 smithi078 ceph-mon[32429]: pgmap v6203: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:36:30.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:30 smithi107 ceph-mon[36485]: pgmap v6203: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:36:30.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:30 smithi106 ceph-mon[36505]: pgmap v6203: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:36:32.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:32 smithi078 ceph-mon[32429]: pgmap v6204: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:36:32.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:32 smithi107 ceph-mon[36485]: pgmap v6204: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:36:32.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:32 smithi106 ceph-mon[36505]: pgmap v6204: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:36:34.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10868.257966 secs 2024-03-24T03:36:34.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:34 smithi078 ceph-mon[32429]: pgmap v6205: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:36:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10868.257966 secs 2024-03-24T03:36:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:34 smithi107 ceph-mon[36485]: pgmap v6205: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:36:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10868.257966 secs 2024-03-24T03:36:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:34 smithi106 ceph-mon[36505]: pgmap v6205: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T03:36:36.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:36 smithi078 ceph-mon[32429]: pgmap v6206: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-03-24T03:36:36.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:36 smithi107 ceph-mon[36485]: pgmap v6206: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-03-24T03:36:36.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:36 smithi106 ceph-mon[36505]: pgmap v6206: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-03-24T03:36:38.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:38 smithi078 ceph-mon[32429]: pgmap v6207: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:36:38.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:38 smithi107 ceph-mon[36485]: pgmap v6207: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:36:38.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:38 smithi106 ceph-mon[36505]: pgmap v6207: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:36:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10873.258053 secs 2024-03-24T03:36:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10873.258053 secs 2024-03-24T03:36:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10873.258053 secs 2024-03-24T03:36:40.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:40 smithi078 ceph-mon[32429]: pgmap v6208: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:40.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:40 smithi107 ceph-mon[36485]: pgmap v6208: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:40.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:40 smithi106 ceph-mon[36505]: pgmap v6208: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:42.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:42 smithi078 ceph-mon[32429]: pgmap v6209: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:36:42.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:42 smithi107 ceph-mon[36485]: pgmap v6209: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:36:42.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:42 smithi106 ceph-mon[36505]: pgmap v6209: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:36:43.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:43 smithi106 ceph-mon[36505]: pgmap v6210: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:43.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:43 smithi078 ceph-mon[32429]: pgmap v6210: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:43.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:43 smithi107 ceph-mon[36485]: pgmap v6210: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10878.258134 secs 2024-03-24T03:36:44.744 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10878.258134 secs 2024-03-24T03:36:44.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10878.258134 secs 2024-03-24T03:36:45.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:45 smithi106 ceph-mon[36505]: pgmap v6211: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-24T03:36:45.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:45 smithi078 ceph-mon[32429]: pgmap v6211: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-24T03:36:45.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:45 smithi107 ceph-mon[36485]: pgmap v6211: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-24T03:36:48.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:48 smithi078 ceph-mon[32429]: pgmap v6212: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:36:48.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:48 smithi107 ceph-mon[36485]: pgmap v6212: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:36:48.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:48 smithi106 ceph-mon[36505]: pgmap v6212: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:36:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10883.258216 secs 2024-03-24T03:36:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10883.258216 secs 2024-03-24T03:36:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10883.258216 secs 2024-03-24T03:36:50.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:50 smithi078 ceph-mon[32429]: pgmap v6213: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:50.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:50 smithi107 ceph-mon[36485]: pgmap v6213: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:50.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:50 smithi106 ceph-mon[36505]: pgmap v6213: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:52.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:52 smithi078 ceph-mon[32429]: pgmap v6214: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:36:52.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:52 smithi107 ceph-mon[36485]: pgmap v6214: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:36:52.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:52 smithi106 ceph-mon[36505]: pgmap v6214: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:36:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10888.258305 secs 2024-03-24T03:36:54.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:54 smithi078 ceph-mon[32429]: pgmap v6215: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10888.258305 secs 2024-03-24T03:36:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:54 smithi107 ceph-mon[36485]: pgmap v6215: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10888.258305 secs 2024-03-24T03:36:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:54 smithi106 ceph-mon[36505]: pgmap v6215: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:36:56.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:56 smithi078 ceph-mon[32429]: pgmap v6216: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-24T03:36:56.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:56 smithi107 ceph-mon[36485]: pgmap v6216: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-24T03:36:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:56 smithi106 ceph-mon[36505]: pgmap v6216: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-24T03:36:57.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:57 smithi106 ceph-mon[36505]: pgmap v6217: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:36:57.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:57 smithi078 ceph-mon[32429]: pgmap v6217: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:36:57.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:57 smithi107 ceph-mon[36485]: pgmap v6217: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:36:59.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:36:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10893.258366 secs 2024-03-24T03:36:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:36:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10893.258366 secs 2024-03-24T03:36:59.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:36:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10893.258366 secs 2024-03-24T03:37:00.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:00 smithi078 ceph-mon[32429]: pgmap v6218: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:37:00.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:00 smithi107 ceph-mon[36485]: pgmap v6218: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:37:00.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:00 smithi106 ceph-mon[36505]: pgmap v6218: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:37:02.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:02 smithi078 ceph-mon[32429]: pgmap v6219: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:37:02.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:02 smithi107 ceph-mon[36485]: pgmap v6219: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:37:02.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:02 smithi106 ceph-mon[36505]: pgmap v6219: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:37:04.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10898.258449 secs 2024-03-24T03:37:04.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:04 smithi078 ceph-mon[32429]: pgmap v6220: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:37:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10898.258449 secs 2024-03-24T03:37:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:04 smithi107 ceph-mon[36485]: pgmap v6220: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:37:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10898.258449 secs 2024-03-24T03:37:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:04 smithi106 ceph-mon[36505]: pgmap v6220: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:37:06.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:06 smithi078 ceph-mon[32429]: pgmap v6221: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-24T03:37:06.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:06 smithi107 ceph-mon[36485]: pgmap v6221: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-24T03:37:06.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:06 smithi106 ceph-mon[36505]: pgmap v6221: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-24T03:37:08.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:08 smithi078 ceph-mon[32429]: pgmap v6222: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:37:08.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:08 smithi107 ceph-mon[36485]: pgmap v6222: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:37:08.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:08 smithi106 ceph-mon[36505]: pgmap v6222: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:37:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10903.258507 secs 2024-03-24T03:37:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10903.258507 secs 2024-03-24T03:37:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10903.258507 secs 2024-03-24T03:37:10.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:10 smithi107 ceph-mon[36485]: pgmap v6223: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:37:10.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:10 smithi106 ceph-mon[36505]: pgmap v6223: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:37:10.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:10 smithi078 ceph-mon[32429]: pgmap v6223: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:37:11.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:11 smithi106 ceph-mon[36505]: pgmap v6224: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:37:11.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:11 smithi078 ceph-mon[32429]: pgmap v6224: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:37:11.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:11 smithi107 ceph-mon[36485]: pgmap v6224: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:37:14.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10908.258518 secs 2024-03-24T03:37:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:14 smithi078 ceph-mon[32429]: pgmap v6225: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T03:37:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10908.258518 secs 2024-03-24T03:37:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:14 smithi107 ceph-mon[36485]: pgmap v6225: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T03:37:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10908.258518 secs 2024-03-24T03:37:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:14 smithi106 ceph-mon[36505]: pgmap v6225: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T03:37:16.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:16 smithi078 ceph-mon[32429]: pgmap v6226: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T03:37:16.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:16 smithi107 ceph-mon[36485]: pgmap v6226: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T03:37:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:16 smithi106 ceph-mon[36505]: pgmap v6226: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T03:37:18.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:18 smithi078 ceph-mon[32429]: pgmap v6227: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 3 op/s 2024-03-24T03:37:18.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:18 smithi107 ceph-mon[36485]: pgmap v6227: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 3 op/s 2024-03-24T03:37:18.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:18 smithi106 ceph-mon[36505]: pgmap v6227: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 3 op/s 2024-03-24T03:37:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10913.258590 secs 2024-03-24T03:37:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10913.258590 secs 2024-03-24T03:37:19.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10913.258590 secs 2024-03-24T03:37:20.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:20 smithi106 ceph-mon[36505]: pgmap v6228: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T03:37:20.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:20 smithi078 ceph-mon[32429]: pgmap v6228: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T03:37:20.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:20 smithi107 ceph-mon[36485]: pgmap v6228: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-24T03:37:21.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:21 smithi106 ceph-mon[36505]: pgmap v6229: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:37:21.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:21 smithi078 ceph-mon[32429]: pgmap v6229: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:37:21.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:21 smithi107 ceph-mon[36485]: pgmap v6229: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:37:24.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10918.258651 secs 2024-03-24T03:37:24.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:24 smithi078 ceph-mon[32429]: pgmap v6230: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:37:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10918.258651 secs 2024-03-24T03:37:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:24 smithi107 ceph-mon[36485]: pgmap v6230: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:37:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10918.258651 secs 2024-03-24T03:37:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:24 smithi106 ceph-mon[36505]: pgmap v6230: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:37:26.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:26 smithi078 ceph-mon[32429]: pgmap v6231: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-24T03:37:26.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:26 smithi107 ceph-mon[36485]: pgmap v6231: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-24T03:37:26.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:26 smithi106 ceph-mon[36505]: pgmap v6231: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-24T03:37:28.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:28 smithi106 ceph-mon[36505]: pgmap v6232: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:37:28.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:28 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:37:28.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:28 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:37:28.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:28 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:37:28.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:28 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:37:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:28 smithi078 ceph-mon[32429]: pgmap v6232: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:37:28.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:28 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:37:28.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:28 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:37:28.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:28 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:37:28.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:28 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:37:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:28 smithi107 ceph-mon[36485]: pgmap v6232: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:37:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:28 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:37:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:28 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:37:28.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:28 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:37:28.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:28 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:37:29.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10923.258715 secs 2024-03-24T03:37:29.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:29 smithi106 ceph-mon[36505]: pgmap v6233: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:37:29.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:29 smithi106 ceph-mon[36505]: pgmap v6234: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:37:29.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10923.258715 secs 2024-03-24T03:37:29.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:29 smithi078 ceph-mon[32429]: pgmap v6233: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:37:29.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:29 smithi078 ceph-mon[32429]: pgmap v6234: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:37:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10923.258715 secs 2024-03-24T03:37:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:29 smithi107 ceph-mon[36485]: pgmap v6233: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:37:29.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:29 smithi107 ceph-mon[36485]: pgmap v6234: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:37:30.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:30 smithi106 ceph-mon[36505]: pgmap v6235: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:37:30.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:30 smithi078 ceph-mon[32429]: pgmap v6235: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:37:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:30 smithi107 ceph-mon[36485]: pgmap v6235: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:37:32.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.357+0000 7fabb2ffd640 1 -- 172.21.15.78:0/1058387750 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fab9c0a2a10 msgr2=0x7fab9c0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900029461) for more than 900000000 us, fault. 2024-03-24T03:37:32.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.357+0000 7fabb2ffd640 1 --2- 172.21.15.78:0/1058387750 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fab9c0a2a10 0x7fab9c0a4ed0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fabb40ff700 tx=0x7faba800b650 comp rx=0 tx=0).stop 2024-03-24T03:37:32.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.357+0000 7fabb0ff9640 1 -- 172.21.15.78:0/1058387750 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fab9c0a2a10 msgr2=0x7fab9c0a4ed0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-24T03:37:32.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.357+0000 7fabb0ff9640 1 --2- 172.21.15.78:0/1058387750 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fab9c0a2a10 0x7fab9c0a4ed0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:32.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.357+0000 7fabb0ff9640 1 --2- 172.21.15.78:0/1058387750 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fab9c0b0520 0x7fab9c0b2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:32.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.357+0000 7fabb0ff9640 1 -- 172.21.15.78:0/1058387750 --> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7faba4049e50 con 0x7fab9c0b0520 2024-03-24T03:37:32.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.357+0000 7fabb2ffd640 1 --2- 172.21.15.78:0/1058387750 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fab9c0b0520 0x7fab9c0b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:32.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.357+0000 7fabb2ffd640 1 --2- 172.21.15.78:0/1058387750 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fab9c0b0520 0x7fab9c0b2910 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7faba800b690 tx=0x7faba801a000 comp rx=0 tx=0).ready entity=mgr.25243 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:37:32.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.446+0000 7fab9b7fe640 1 -- 172.21.15.78:0/1058387750 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab940062d0 con 0x7fabb4101650 2024-03-24T03:37:32.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.446+0000 7fabb0ff9640 1 -- 172.21.15.78:0/1058387750 <== mon.0 v2:172.21.15.78:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7fab940062d0 con 0x7fabb4101650 2024-03-24T03:37:32.527 INFO:teuthology.orchestra.run.smithi078.stderr:Error ETIMEDOUT: Connection timed out. Please check the client's permission and connection. 2024-03-24T03:37:32.534 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-03-24T03:37:32.535 ERROR:teuthology.run_tasks:Manager failed: full_sequential_finally Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/full_sequential_finally.py", line 45, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi078 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/'" 2024-03-24T03:37:32.541 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-24T03:37:32.552 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T03:37:32.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.668+0000 7fe061fc4640 1 -- 172.21.15.78:0/4028881789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe05c107660 msgr2=0x7fe05c107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:32.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.668+0000 7fe061fc4640 1 --2- 172.21.15.78:0/4028881789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe05c107660 0x7fe05c107a40 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe04c009930 tx=0x7fe04c02bc30 comp rx=0 tx=0).stop 2024-03-24T03:37:32.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.668+0000 7fe061fc4640 1 -- 172.21.15.78:0/4028881789 shutdown_connections 2024-03-24T03:37:32.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.668+0000 7fe061fc4640 1 --2- 172.21.15.78:0/4028881789 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe05c0ffba0 0x7fe05c0fffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:32.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.668+0000 7fe061fc4640 1 --2- 172.21.15.78:0/4028881789 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe05c100520 0x7fe05c10e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:32.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.668+0000 7fe061fc4640 1 --2- 172.21.15.78:0/4028881789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe05c107660 0x7fe05c107a40 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:32.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.668+0000 7fe061fc4640 1 -- 172.21.15.78:0/4028881789 >> 172.21.15.78:0/4028881789 conn(0x7fe05c076b30 msgr2=0x7fe05c0fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:37:32.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.668+0000 7fe061fc4640 1 -- 172.21.15.78:0/4028881789 shutdown_connections 2024-03-24T03:37:32.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.668+0000 7fe061fc4640 1 -- 172.21.15.78:0/4028881789 wait complete. 2024-03-24T03:37:32.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.669+0000 7fe061fc4640 1 Processor -- start 2024-03-24T03:37:32.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.669+0000 7fe061fc4640 1 -- start start 2024-03-24T03:37:32.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.669+0000 7fe061fc4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe05c0ffba0 0x7fe05c1980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:32.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.669+0000 7fe061fc4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe05c100520 0x7fe05c198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:32.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.669+0000 7fe061fc4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe05c107660 0x7fe05c19c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:32.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.669+0000 7fe061fc4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05c111ab0 con 0x7fe05c100520 2024-03-24T03:37:32.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.669+0000 7fe061fc4640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05c111930 con 0x7fe05c107660 2024-03-24T03:37:32.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.669+0000 7fe061fc4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05c111c30 con 0x7fe05c0ffba0 2024-03-24T03:37:32.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.669+0000 7fe05affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe05c100520 0x7fe05c198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:32.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.670+0000 7fe05affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe05c100520 0x7fe05c198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43716/0 (socket says 172.21.15.78:43716) 2024-03-24T03:37:32.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.670+0000 7fe05affd640 1 -- 172.21.15.78:0/468590175 learned_addr learned my addr 172.21.15.78:0/468590175 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T03:37:32.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.670+0000 7fe05bfff640 1 --2- 172.21.15.78:0/468590175 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe05c107660 0x7fe05c19c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:32.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.670+0000 7fe05b7fe640 1 --2- 172.21.15.78:0/468590175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe05c0ffba0 0x7fe05c1980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:32.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.670+0000 7fe05affd640 1 -- 172.21.15.78:0/468590175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe05c0ffba0 msgr2=0x7fe05c1980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:32.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.670+0000 7fe05affd640 1 --2- 172.21.15.78:0/468590175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe05c0ffba0 0x7fe05c1980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:32.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.670+0000 7fe05affd640 1 -- 172.21.15.78:0/468590175 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe05c107660 msgr2=0x7fe05c19c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:32.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.670+0000 7fe05affd640 1 --2- 172.21.15.78:0/468590175 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe05c107660 0x7fe05c19c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:32.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.670+0000 7fe05affd640 1 -- 172.21.15.78:0/468590175 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe05c19d0b0 con 0x7fe05c100520 2024-03-24T03:37:32.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.670+0000 7fe05bfff640 1 --2- 172.21.15.78:0/468590175 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe05c107660 0x7fe05c19c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T03:37:32.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.670+0000 7fe05affd640 1 --2- 172.21.15.78:0/468590175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe05c100520 0x7fe05c198600 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe05000d950 tx=0x7fe05000dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:37:32.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.670+0000 7fe058ff9640 1 -- 172.21.15.78:0/468590175 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe050017070 con 0x7fe05c100520 2024-03-24T03:37:32.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.670+0000 7fe058ff9640 1 -- 172.21.15.78:0/468590175 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe05000de80 con 0x7fe05c100520 2024-03-24T03:37:32.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.670+0000 7fe061fc4640 1 -- 172.21.15.78:0/468590175 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe05c19d3a0 con 0x7fe05c100520 2024-03-24T03:37:32.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.670+0000 7fe058ff9640 1 -- 172.21.15.78:0/468590175 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05000f620 con 0x7fe05c100520 2024-03-24T03:37:32.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.670+0000 7fe061fc4640 1 -- 172.21.15.78:0/468590175 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe05c1a4c30 con 0x7fe05c100520 2024-03-24T03:37:32.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.671+0000 7fe061fc4640 1 -- 172.21.15.78:0/468590175 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe05c0688e0 con 0x7fe05c100520 2024-03-24T03:37:32.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.672+0000 7fe058ff9640 1 -- 172.21.15.78:0/468590175 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fe050009c30 con 0x7fe05c100520 2024-03-24T03:37:32.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.673+0000 7fe058ff9640 1 --2- 172.21.15.78:0/468590175 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fe0440a2970 0x7fe0440a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:32.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.673+0000 7fe05b7fe640 1 --2- 172.21.15.78:0/468590175 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fe0440a2970 0x7fe0440a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:32.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.673+0000 7fe058ff9640 1 -- 172.21.15.78:0/468590175 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 13278+0+0 (secure 0 0 0) 0x7fe0500cfc90 con 0x7fe05c100520 2024-03-24T03:37:32.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.673+0000 7fe05b7fe640 1 --2- 172.21.15.78:0/468590175 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fe0440a2970 0x7fe0440a4e30 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe04c009900 tx=0x7fe04c02df90 comp rx=0 tx=0).ready entity=mgr.25243 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:37:32.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.677+0000 7fe058ff9640 1 -- 172.21.15.78:0/468590175 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe05009b760 con 0x7fe05c100520 2024-03-24T03:37:32.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.856+0000 7fe061fc4640 1 -- 172.21.15.78:0/468590175 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe05c109b70 con 0x7fe05c100520 2024-03-24T03:37:32.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.858+0000 7fe058ff9640 1 -- 172.21.15.78:0/468590175 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v157) v1 ==== 74+0+30992 (secure 0 0 0) 0x7fe05009b060 con 0x7fe05c100520 2024-03-24T03:37:32.858 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T03:37:32.859 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":157,"fsid":"c7e2806a-e972-11ee-95cd-87774f69a715","created":"2024-03-24T00:10:44.134563+0000","modified":"2024-03-24T01:24:14.909719+0000","last_up_change":"2024-03-24T00:17:50.237686+0000","last_in_change":"2024-03-24T00:17:29.788699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:14:04.534391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:18:42.178418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:18:42.800424+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":23,"snap_epoch":155,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:18:46.112800+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":23,"snap_epoch":155,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b950388a-c437-426b-b703-3f8ae821dd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6803","nonce":1816879397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6805","nonce":1816879397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6809","nonce":1816879397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6807","nonce":1816879397}]},"public_addr":"172.21.15.78:6803/1816879397","cluster_addr":"172.21.15.78:6805/1816879397","heartbeat_back_addr":"172.21.15.78:6809/1816879397","heartbeat_front_addr":"172.21.15.78:6807/1816879397","state":["exists","up"]},{"osd":1,"uuid":"b8d64714-dd65-400f-b3f4-673ab14e1365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6801","nonce":1647915612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6803","nonce":1647915612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6807","nonce":1647915612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6805","nonce":1647915612}]},"public_addr":"172.21.15.106:6801/1647915612","cluster_addr":"172.21.15.106:6803/1647915612","heartbeat_back_addr":"172.21.15.106:6807/1647915612","heartbeat_front_addr":"172.21.15.106:6805/1647915612","state":["exists","up"]},{"osd":2,"uuid":"fcfc4dd1-aaf8-4131-a023-5e832212158d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6801","nonce":2010770435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6803","nonce":2010770435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6807","nonce":2010770435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6805","nonce":2010770435}]},"public_addr":"172.21.15.107:6801/2010770435","cluster_addr":"172.21.15.107:6803/2010770435","heartbeat_back_addr":"172.21.15.107:6807/2010770435","heartbeat_front_addr":"172.21.15.107:6805/2010770435","state":["exists","up"]},{"osd":3,"uuid":"47c718ab-a0f4-4e52-83c5-b89386b262d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6811","nonce":1912251346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6813","nonce":1912251346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6817","nonce":1912251346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6815","nonce":1912251346}]},"public_addr":"172.21.15.78:6811/1912251346","cluster_addr":"172.21.15.78:6813/1912251346","heartbeat_back_addr":"172.21.15.78:6817/1912251346","heartbeat_front_addr":"172.21.15.78:6815/1912251346","state":["exists","up"]},{"osd":4,"uuid":"7dad64dd-677e-4c1c-9638-e4f4f2195983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6809","nonce":374641205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6811","nonce":374641205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6815","nonce":374641205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6813","nonce":374641205}]},"public_addr":"172.21.15.106:6809/374641205","cluster_addr":"172.21.15.106:6811/374641205","heartbeat_back_addr":"172.21.15.106:6815/374641205","heartbeat_front_addr":"172.21.15.106:6813/374641205","state":["exists","up"]},{"osd":5,"uuid":"bbd524e5-328d-4309-8877-1149cbe110dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6809","nonce":3272806073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6811","nonce":3272806073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6815","nonce":3272806073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6813","nonce":3272806073}]},"public_addr":"172.21.15.107:6809/3272806073","cluster_addr":"172.21.15.107:6811/3272806073","heartbeat_back_addr":"172.21.15.107:6815/3272806073","heartbeat_front_addr":"172.21.15.107:6813/3272806073","state":["exists","up"]},{"osd":6,"uuid":"5696389a-918f-4ea8-b20d-4e2a0d4cb3bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6819","nonce":3091148508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6821","nonce":3091148508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6825","nonce":3091148508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6823","nonce":3091148508}]},"public_addr":"172.21.15.78:6819/3091148508","cluster_addr":"172.21.15.78:6821/3091148508","heartbeat_back_addr":"172.21.15.78:6825/3091148508","heartbeat_front_addr":"172.21.15.78:6823/3091148508","state":["exists","up"]},{"osd":7,"uuid":"f6b764e8-ed4e-4c9a-a85b-94af6dc6b924","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6817","nonce":4262103080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6819","nonce":4262103080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6823","nonce":4262103080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6821","nonce":4262103080}]},"public_addr":"172.21.15.106:6817/4262103080","cluster_addr":"172.21.15.106:6819/4262103080","heartbeat_back_addr":"172.21.15.106:6823/4262103080","heartbeat_front_addr":"172.21.15.106:6821/4262103080","state":["exists","up"]},{"osd":8,"uuid":"ba16039f-8093-4fa0-87ad-e28b7d142b23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6817","nonce":4032799356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6819","nonce":4032799356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6823","nonce":4032799356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6821","nonce":4032799356}]},"public_addr":"172.21.15.107:6817/4032799356","cluster_addr":"172.21.15.107:6819/4032799356","heartbeat_back_addr":"172.21.15.107:6823/4032799356","heartbeat_front_addr":"172.21.15.107:6821/4032799356","state":["exists","up"]},{"osd":9,"uuid":"651357f2-0669-42e6-aa91-1d5c02d00c5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6827","nonce":214023431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6829","nonce":214023431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6833","nonce":214023431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6831","nonce":214023431}]},"public_addr":"172.21.15.78:6827/214023431","cluster_addr":"172.21.15.78:6829/214023431","heartbeat_back_addr":"172.21.15.78:6833/214023431","heartbeat_front_addr":"172.21.15.78:6831/214023431","state":["exists","up"]},{"osd":10,"uuid":"2375119d-346e-426c-8a65-b64af71ad63a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6825","nonce":400029804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6827","nonce":400029804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6831","nonce":400029804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6829","nonce":400029804}]},"public_addr":"172.21.15.106:6825/400029804","cluster_addr":"172.21.15.106:6827/400029804","heartbeat_back_addr":"172.21.15.106:6831/400029804","heartbeat_front_addr":"172.21.15.106:6829/400029804","state":["exists","up"]},{"osd":11,"uuid":"b6b609d9-dd78-4c01-ae66-ba2a662ab22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6825","nonce":4191916230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6827","nonce":4191916230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6831","nonce":4191916230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6829","nonce":4191916230}]},"public_addr":"172.21.15.107:6825/4191916230","cluster_addr":"172.21.15.107:6827/4191916230","heartbeat_back_addr":"172.21.15.107:6831/4191916230","heartbeat_front_addr":"172.21.15.107:6829/4191916230","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:13.728602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:37.320809+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:01.269452+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:25.758281+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:50.298863+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:14.345219+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:38.980489+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:04.749214+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:29.992673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:56.269202+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:22.270316+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:48.074907+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6834/1985874955":"2024-03-25T00:35:20.737219+0000","172.21.15.78:6835/3414536872":"2024-03-25T00:35:04.921316+0000","172.21.15.78:6834/3414536872":"2024-03-25T00:35:04.921316+0000","172.21.15.78:6837/2612396245":"2024-03-25T00:34:49.129252+0000","172.21.15.78:6834/93340378":"2024-03-25T00:34:30.894376+0000","172.21.15.78:6837/878237454":"2024-03-25T00:34:15.858120+0000","172.21.15.78:6836/878237454":"2024-03-25T00:34:15.858120+0000","172.21.15.78:6835/2163967090":"2024-03-25T00:34:00.891452+0000","172.21.15.78:6834/2163967090":"2024-03-25T00:34:00.891452+0000","172.21.15.78:6836/1186751954":"2024-03-25T00:33:45.900090+0000","172.21.15.78:6835/1416685123":"2024-03-25T00:33:30.602155+0000","172.21.15.78:6834/1416685123":"2024-03-25T00:33:30.602155+0000","172.21.15.78:6836/1077781429":"2024-03-25T00:33:14.880383+0000","172.21.15.78:6839/2480119933":"2024-03-25T00:32:59.099199+0000","172.21.15.78:6838/2480119933":"2024-03-25T00:32:59.099199+0000","172.21.15.78:6837/2239323994":"2024-03-25T00:32:40.627803+0000","172.21.15.78:6839/3377749691":"2024-03-25T00:32:25.708147+0000","172.21.15.78:6836/1071274623":"2024-03-25T00:32:10.632135+0000","172.21.15.78:6839/4176145760":"2024-03-25T00:31:55.146631+0000","172.21.15.78:6838/4176145760":"2024-03-25T00:31:55.146631+0000","172.21.15.78:6840/1658072731":"2024-03-25T00:31:39.075579+0000","172.21.15.78:0/442826396":"2024-03-25T00:21:09.220548+0000","172.21.15.107:6832/4067467714":"2024-03-25T00:29:09.638984+0000","172.21.15.78:6837/1077781429":"2024-03-25T00:33:14.880383+0000","172.21.15.107:6839/3131891440":"2024-03-25T00:24:40.372092+0000","172.21.15.107:6838/3949153161":"2024-03-25T00:25:43.971852+0000","172.21.15.107:6833/794068267":"2024-03-25T00:28:05.611074+0000","172.21.15.107:6838/3131891440":"2024-03-25T00:24:40.372092+0000","172.21.15.107:6833/1751539133":"2024-03-25T00:27:35.655923+0000","172.21.15.107:6834/4046882186":"2024-03-25T00:24:25.648083+0000","172.21.15.107:6839/3949153161":"2024-03-25T00:25:43.971852+0000","172.21.15.78:6835/1985874955":"2024-03-25T00:35:20.737219+0000","172.21.15.78:6801/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6841/3424505910":"2024-03-25T00:29:25.370378+0000","172.21.15.78:0/1982202541":"2024-03-25T00:11:40.199569+0000","172.21.15.107:6834/108936439":"2024-03-25T00:23:06.644918+0000","172.21.15.107:6832/794068267":"2024-03-25T00:28:05.611074+0000","172.21.15.78:6838/3377749691":"2024-03-25T00:32:25.708147+0000","172.21.15.107:6839/3531100722":"2024-03-25T00:23:22.383674+0000","172.21.15.107:6838/117119462":"2024-03-25T00:22:50.169074+0000","172.21.15.78:0/2236879401":"2024-03-25T00:11:40.199569+0000","172.21.15.107:6833/3086318443":"2024-03-25T00:27:05.852992+0000","172.21.15.78:6837/1186751954":"2024-03-25T00:33:45.900090+0000","172.21.15.107:6834/23104891":"2024-03-25T00:24:55.644935+0000","172.21.15.78:0/1502752625":"2024-03-25T00:11:14.637881+0000","172.21.15.107:6834/2447512361":"2024-03-25T00:26:33.985150+0000","172.21.15.78:0/359613096":"2024-03-25T00:21:09.220548+0000","172.21.15.78:6800/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6801/1983101026":"2024-03-25T00:21:09.220548+0000","172.21.15.78:0/465691596":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/3962523609":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/1510740971":"2024-03-25T00:21:09.220548+0000","172.21.15.78:6840/1960340276":"2024-03-25T00:31:02.121119+0000","172.21.15.78:6836/2612396245":"2024-03-25T00:34:49.129252+0000","172.21.15.78:6801/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6800/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6836/2239323994":"2024-03-25T00:32:40.627803+0000","172.21.15.107:6835/2950120609":"2024-03-25T00:25:25.629098+0000","172.21.15.78:0/1933054339":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6838/4117918862":"2024-03-25T00:31:18.139126+0000","172.21.15.107:6838/607202225":"2024-03-25T00:25:10.629186+0000","172.21.15.78:0/1685151467":"2024-03-25T00:21:09.220548+0000","172.21.15.107:6838/3531100722":"2024-03-25T00:23:22.383674+0000","172.21.15.78:6838/2758357551":"2024-03-25T00:30:15.603375+0000","172.21.15.78:6800/1983101026":"2024-03-25T00:21:09.220548+0000","172.21.15.107:6835/758053763":"2024-03-25T00:25:59.620946+0000","172.21.15.107:6835/4046882186":"2024-03-25T00:24:25.648083+0000","172.21.15.107:6835/108936439":"2024-03-25T00:23:06.644918+0000","172.21.15.106:6833/2367018868":"2024-03-25T00:26:15.443626+0000","172.21.15.78:6835/93340378":"2024-03-25T00:34:30.894376+0000","172.21.15.107:6835/2447512361":"2024-03-25T00:26:33.985150+0000","172.21.15.107:6835/23104891":"2024-03-25T00:24:55.644935+0000","172.21.15.106:6832/2611554095":"2024-03-25T00:26:49.797488+0000","172.21.15.106:6833/2611554095":"2024-03-25T00:26:49.797488+0000","172.21.15.107:6832/3086318443":"2024-03-25T00:27:05.852992+0000","172.21.15.106:6832/2299054161":"2024-03-25T00:27:20.754602+0000","172.21.15.107:6832/1006503957":"2024-03-25T00:29:44.040588+0000","172.21.15.106:6832/2367018868":"2024-03-25T00:26:15.443626+0000","172.21.15.106:6833/2299054161":"2024-03-25T00:27:20.754602+0000","172.21.15.107:6832/1751539133":"2024-03-25T00:27:35.655923+0000","172.21.15.106:6832/459568830":"2024-03-25T00:27:50.719456+0000","172.21.15.106:6833/459568830":"2024-03-25T00:27:50.719456+0000","172.21.15.106:6832/93515314":"2024-03-25T00:28:20.713453+0000","172.21.15.107:6833/3928685127":"2024-03-25T00:28:35.891033+0000","172.21.15.107:6839/117119462":"2024-03-25T00:22:50.169074+0000","172.21.15.106:6832/2104934388":"2024-03-25T00:28:54.025064+0000","172.21.15.78:6837/1071274623":"2024-03-25T00:32:10.632135+0000","172.21.15.106:6833/2104934388":"2024-03-25T00:28:54.025064+0000","172.21.15.78:6841/1658072731":"2024-03-25T00:31:39.075579+0000","172.21.15.107:6833/4067467714":"2024-03-25T00:29:09.638984+0000","172.21.15.78:6840/3424505910":"2024-03-25T00:29:25.370378+0000","172.21.15.107:6834/758053763":"2024-03-25T00:25:59.620946+0000","172.21.15.78:6838/2645579087":"2024-03-25T00:30:46.105435+0000","172.21.15.107:6834/2950120609":"2024-03-25T00:25:25.629098+0000","172.21.15.78:6840/1196331852":"2024-03-25T00:29:59.753104+0000","172.21.15.107:6833/1006503957":"2024-03-25T00:29:44.040588+0000","172.21.15.78:6841/1196331852":"2024-03-25T00:29:59.753104+0000","172.21.15.107:6839/607202225":"2024-03-25T00:25:10.629186+0000","172.21.15.78:6841/1759489401":"2024-03-25T00:30:30.711566+0000","172.21.15.78:6839/2758357551":"2024-03-25T00:30:15.603375+0000","172.21.15.78:6840/1759489401":"2024-03-25T00:30:30.711566+0000","172.21.15.78:6839/2645579087":"2024-03-25T00:30:46.105435+0000","172.21.15.107:6832/3928685127":"2024-03-25T00:28:35.891033+0000","172.21.15.78:6841/1960340276":"2024-03-25T00:31:02.121119+0000","172.21.15.106:6833/93515314":"2024-03-25T00:28:20.713453+0000","172.21.15.78:6839/4117918862":"2024-03-25T00:31:18.139126+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T03:37:32.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.860+0000 7fe061fc4640 1 -- 172.21.15.78:0/468590175 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fe0440a2970 msgr2=0x7fe0440a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:32.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.860+0000 7fe061fc4640 1 --2- 172.21.15.78:0/468590175 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fe0440a2970 0x7fe0440a4e30 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe04c009900 tx=0x7fe04c02df90 comp rx=0 tx=0).stop 2024-03-24T03:37:32.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.860+0000 7fe061fc4640 1 -- 172.21.15.78:0/468590175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe05c100520 msgr2=0x7fe05c198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:32.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.860+0000 7fe061fc4640 1 --2- 172.21.15.78:0/468590175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe05c100520 0x7fe05c198600 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe05000d950 tx=0x7fe05000dd50 comp rx=0 tx=0).stop 2024-03-24T03:37:32.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.860+0000 7fe061fc4640 1 -- 172.21.15.78:0/468590175 shutdown_connections 2024-03-24T03:37:32.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.860+0000 7fe061fc4640 1 --2- 172.21.15.78:0/468590175 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fe0440a2970 0x7fe0440a4e30 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:32.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.860+0000 7fe061fc4640 1 --2- 172.21.15.78:0/468590175 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe05c107660 0x7fe05c19c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:32.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.860+0000 7fe061fc4640 1 --2- 172.21.15.78:0/468590175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe05c100520 0x7fe05c198600 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:32.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.860+0000 7fe061fc4640 1 --2- 172.21.15.78:0/468590175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe05c0ffba0 0x7fe05c1980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:32.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.860+0000 7fe061fc4640 1 -- 172.21.15.78:0/468590175 >> 172.21.15.78:0/468590175 conn(0x7fe05c076b30 msgr2=0x7fe05c0fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:37:32.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.860+0000 7fe061fc4640 1 -- 172.21.15.78:0/468590175 shutdown_connections 2024-03-24T03:37:32.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.860+0000 7fe061fc4640 1 -- 172.21.15.78:0/468590175 wait complete. 2024-03-24T03:37:32.872 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:14:04.534391+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:18:42.178418+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:18:42.800424+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '155', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 23, 'snap_epoch': 155, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T00:18:46.112800+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '155', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 23, 'snap_epoch': 155, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T03:37:32.872 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T03:37:32.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.977+0000 7f1ad15b9640 1 -- 172.21.15.78:0/3397725917 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1acc106610 msgr2=0x7f1acc1069f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:32.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.977+0000 7f1ad15b9640 1 --2- 172.21.15.78:0/3397725917 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1acc106610 0x7f1acc1069f0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f1abc009900 tx=0x7f1abc02bc30 comp rx=0 tx=0).stop 2024-03-24T03:37:32.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.977+0000 7f1ad15b9640 1 -- 172.21.15.78:0/3397725917 shutdown_connections 2024-03-24T03:37:32.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.977+0000 7f1ad15b9640 1 --2- 172.21.15.78:0/3397725917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1acc100550 0x7f1acc1009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:32.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.977+0000 7f1ad15b9640 1 --2- 172.21.15.78:0/3397725917 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1acc100f10 0x7f1acc10d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:32.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.977+0000 7f1ad15b9640 1 --2- 172.21.15.78:0/3397725917 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1acc106610 0x7f1acc1069f0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:32.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.977+0000 7f1ad15b9640 1 -- 172.21.15.78:0/3397725917 >> 172.21.15.78:0/3397725917 conn(0x7f1acc077670 msgr2=0x7f1acc077a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:37:32.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.977+0000 7f1ad15b9640 1 -- 172.21.15.78:0/3397725917 shutdown_connections 2024-03-24T03:37:32.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.977+0000 7f1ad15b9640 1 -- 172.21.15.78:0/3397725917 wait complete. 2024-03-24T03:37:32.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.977+0000 7f1ad15b9640 1 Processor -- start 2024-03-24T03:37:32.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.978+0000 7f1ad15b9640 1 -- start start 2024-03-24T03:37:32.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.978+0000 7f1ad15b9640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1acc100550 0x7f1acc197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:32.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.978+0000 7f1ad15b9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1acc100f10 0x7f1acc1984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:32.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.978+0000 7f1ad15b9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1acc106610 0x7f1acc19c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:32.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.978+0000 7f1ad15b9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc111aa0 con 0x7f1acc106610 2024-03-24T03:37:32.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.978+0000 7f1ad15b9640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc111920 con 0x7f1acc100550 2024-03-24T03:37:32.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.978+0000 7f1ad15b9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc111c20 con 0x7f1acc100f10 2024-03-24T03:37:32.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.978+0000 7f1acb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1acc106610 0x7f1acc19c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:32.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.978+0000 7f1acb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1acc106610 0x7f1acc19c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43724/0 (socket says 172.21.15.78:43724) 2024-03-24T03:37:32.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.978+0000 7f1acb7fe640 1 -- 172.21.15.78:0/2052913777 learned_addr learned my addr 172.21.15.78:0/2052913777 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T03:37:32.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.978+0000 7f1acaffd640 1 --2- 172.21.15.78:0/2052913777 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1acc100550 0x7f1acc197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:32.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.978+0000 7f1aca7fc640 1 --2- 172.21.15.78:0/2052913777 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1acc100f10 0x7f1acc1984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:32.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.978+0000 7f1acb7fe640 1 -- 172.21.15.78:0/2052913777 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1acc100f10 msgr2=0x7f1acc1984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:32.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.978+0000 7f1acb7fe640 1 --2- 172.21.15.78:0/2052913777 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1acc100f10 0x7f1acc1984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:32.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.978+0000 7f1acb7fe640 1 -- 172.21.15.78:0/2052913777 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1acc100550 msgr2=0x7f1acc197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:32.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.978+0000 7f1acb7fe640 1 --2- 172.21.15.78:0/2052913777 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1acc100550 0x7f1acc197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:32.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.978+0000 7f1acb7fe640 1 -- 172.21.15.78:0/2052913777 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1acc19cf50 con 0x7f1acc106610 2024-03-24T03:37:32.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.978+0000 7f1aca7fc640 1 --2- 172.21.15.78:0/2052913777 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1acc100f10 0x7f1acc1984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T03:37:32.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.978+0000 7f1acb7fe640 1 --2- 172.21.15.78:0/2052913777 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1acc106610 0x7f1acc19c820 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f1ac400eb10 tx=0x7f1ac400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:37:32.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.979+0000 7f1ab3fff640 1 -- 172.21.15.78:0/2052913777 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac4004b20 con 0x7f1acc106610 2024-03-24T03:37:32.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.979+0000 7f1ab3fff640 1 -- 172.21.15.78:0/2052913777 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ac4004cc0 con 0x7f1acc106610 2024-03-24T03:37:32.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.979+0000 7f1ad15b9640 1 -- 172.21.15.78:0/2052913777 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1acc19d240 con 0x7f1acc106610 2024-03-24T03:37:32.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.979+0000 7f1ab3fff640 1 -- 172.21.15.78:0/2052913777 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac4010760 con 0x7f1acc106610 2024-03-24T03:37:32.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.979+0000 7f1ad15b9640 1 -- 172.21.15.78:0/2052913777 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1acc1a4ad0 con 0x7f1acc106610 2024-03-24T03:37:32.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.980+0000 7f1ad15b9640 1 -- 172.21.15.78:0/2052913777 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1acc101c70 con 0x7f1acc106610 2024-03-24T03:37:32.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.981+0000 7f1ab3fff640 1 -- 172.21.15.78:0/2052913777 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f1ac4005020 con 0x7f1acc106610 2024-03-24T03:37:32.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.981+0000 7f1ab3fff640 1 --2- 172.21.15.78:0/2052913777 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f1ab40a2970 0x7f1ab40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:32.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.981+0000 7f1acaffd640 1 --2- 172.21.15.78:0/2052913777 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f1ab40a2970 0x7f1ab40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:32.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.981+0000 7f1ab3fff640 1 -- 172.21.15.78:0/2052913777 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 13278+0+0 (secure 0 0 0) 0x7f1ac40d06b0 con 0x7f1acc106610 2024-03-24T03:37:32.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.982+0000 7f1acaffd640 1 --2- 172.21.15.78:0/2052913777 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f1ab40a2970 0x7f1ab40a4e30 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1abc02efd0 tx=0x7f1abc000bc0 comp rx=0 tx=0).ready entity=mgr.25243 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:37:32.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:32.986+0000 7f1ab3fff640 1 -- 172.21.15.78:0/2052913777 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1ac409c180 con 0x7f1acc106610 2024-03-24T03:37:33.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.160+0000 7f1ad15b9640 1 -- 172.21.15.78:0/2052913777 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1acc1992b0 con 0x7f1acc106610 2024-03-24T03:37:33.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.160+0000 7f1ab3fff640 1 -- 172.21.15.78:0/2052913777 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v157) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1ac409ba80 con 0x7f1acc106610 2024-03-24T03:37:33.160 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-24T03:37:33.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.162+0000 7f1ad15b9640 1 -- 172.21.15.78:0/2052913777 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f1ab40a2970 msgr2=0x7f1ab40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:33.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.162+0000 7f1ad15b9640 1 --2- 172.21.15.78:0/2052913777 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f1ab40a2970 0x7f1ab40a4e30 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1abc02efd0 tx=0x7f1abc000bc0 comp rx=0 tx=0).stop 2024-03-24T03:37:33.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.162+0000 7f1ad15b9640 1 -- 172.21.15.78:0/2052913777 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1acc106610 msgr2=0x7f1acc19c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:33.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.162+0000 7f1ad15b9640 1 --2- 172.21.15.78:0/2052913777 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1acc106610 0x7f1acc19c820 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f1ac400eb10 tx=0x7f1ac400c2d0 comp rx=0 tx=0).stop 2024-03-24T03:37:33.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.162+0000 7f1ad15b9640 1 -- 172.21.15.78:0/2052913777 shutdown_connections 2024-03-24T03:37:33.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.162+0000 7f1ad15b9640 1 --2- 172.21.15.78:0/2052913777 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f1ab40a2970 0x7f1ab40a4e30 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:33.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.162+0000 7f1ad15b9640 1 --2- 172.21.15.78:0/2052913777 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1acc100f10 0x7f1acc1984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:33.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.162+0000 7f1ad15b9640 1 --2- 172.21.15.78:0/2052913777 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1acc106610 0x7f1acc19c820 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:33.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.162+0000 7f1ad15b9640 1 --2- 172.21.15.78:0/2052913777 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1acc100550 0x7f1acc197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:33.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.162+0000 7f1ad15b9640 1 -- 172.21.15.78:0/2052913777 >> 172.21.15.78:0/2052913777 conn(0x7f1acc077670 msgr2=0x7f1acc0ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:37:33.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.162+0000 7f1ad15b9640 1 -- 172.21.15.78:0/2052913777 shutdown_connections 2024-03-24T03:37:33.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.162+0000 7f1ad15b9640 1 -- 172.21.15.78:0/2052913777 wait complete. 2024-03-24T03:37:33.173 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:33 smithi078 ceph-mon[32429]: pgmap v6236: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:37:33.173 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:33 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/468590175' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T03:37:33.174 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T03:37:33.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.279+0000 7fcf945c4640 1 -- 172.21.15.78:0/2339079153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf8c1065f0 msgr2=0x7fcf8c1069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:33.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.279+0000 7fcf945c4640 1 --2- 172.21.15.78:0/2339079153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf8c1065f0 0x7fcf8c1069d0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c009900 tx=0x7fcf7c02bc30 comp rx=0 tx=0).stop 2024-03-24T03:37:33.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.279+0000 7fcf945c4640 1 -- 172.21.15.78:0/2339079153 shutdown_connections 2024-03-24T03:37:33.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.279+0000 7fcf945c4640 1 --2- 172.21.15.78:0/2339079153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf8c100ef0 0x7fcf8c10d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:33.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.279+0000 7fcf945c4640 1 --2- 172.21.15.78:0/2339079153 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcf8c100530 0x7fcf8c1009b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:33.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.279+0000 7fcf945c4640 1 --2- 172.21.15.78:0/2339079153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf8c1065f0 0x7fcf8c1069d0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:33.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.279+0000 7fcf945c4640 1 -- 172.21.15.78:0/2339079153 >> 172.21.15.78:0/2339079153 conn(0x7fcf8c0776a0 msgr2=0x7fcf8c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:37:33.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.279+0000 7fcf945c4640 1 -- 172.21.15.78:0/2339079153 shutdown_connections 2024-03-24T03:37:33.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.279+0000 7fcf945c4640 1 -- 172.21.15.78:0/2339079153 wait complete. 2024-03-24T03:37:33.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.279+0000 7fcf945c4640 1 Processor -- start 2024-03-24T03:37:33.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.280+0000 7fcf945c4640 1 -- start start 2024-03-24T03:37:33.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.280+0000 7fcf945c4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf8c100530 0x7fcf8c197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:33.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.280+0000 7fcf945c4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf8c100ef0 0x7fcf8c198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:33.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.280+0000 7fcf945c4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcf8c1065f0 0x7fcf8c19c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:33.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.280+0000 7fcf945c4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8c1108d0 con 0x7fcf8c100ef0 2024-03-24T03:37:33.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.280+0000 7fcf945c4640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8c110750 con 0x7fcf8c1065f0 2024-03-24T03:37:33.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.280+0000 7fcf945c4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8c110a50 con 0x7fcf8c100530 2024-03-24T03:37:33.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.280+0000 7fcf91b5f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf8c100ef0 0x7fcf8c198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:33.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.280+0000 7fcf91b5f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf8c100ef0 0x7fcf8c198400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43740/0 (socket says 172.21.15.78:43740) 2024-03-24T03:37:33.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.280+0000 7fcf91b5f640 1 -- 172.21.15.78:0/3812716474 learned_addr learned my addr 172.21.15.78:0/3812716474 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T03:37:33.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.280+0000 7fcf92b61640 1 --2- 172.21.15.78:0/3812716474 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcf8c1065f0 0x7fcf8c19c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:33.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.280+0000 7fcf92360640 1 --2- 172.21.15.78:0/3812716474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf8c100530 0x7fcf8c197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:33.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.280+0000 7fcf91b5f640 1 -- 172.21.15.78:0/3812716474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf8c100530 msgr2=0x7fcf8c197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:33.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.280+0000 7fcf91b5f640 1 --2- 172.21.15.78:0/3812716474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf8c100530 0x7fcf8c197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:33.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.280+0000 7fcf91b5f640 1 -- 172.21.15.78:0/3812716474 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcf8c1065f0 msgr2=0x7fcf8c19c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:33.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.280+0000 7fcf91b5f640 1 --2- 172.21.15.78:0/3812716474 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcf8c1065f0 0x7fcf8c19c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:33.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.280+0000 7fcf91b5f640 1 -- 172.21.15.78:0/3812716474 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf8c19ceb0 con 0x7fcf8c100ef0 2024-03-24T03:37:33.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.280+0000 7fcf91b5f640 1 --2- 172.21.15.78:0/3812716474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf8c100ef0 0x7fcf8c198400 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fcf8800e9f0 tx=0x7fcf8800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:37:33.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.280+0000 7fcf837fe640 1 -- 172.21.15.78:0/3812716474 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8800cd80 con 0x7fcf8c100ef0 2024-03-24T03:37:33.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.280+0000 7fcf945c4640 1 -- 172.21.15.78:0/3812716474 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf8c19d1a0 con 0x7fcf8c100ef0 2024-03-24T03:37:33.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.281+0000 7fcf837fe640 1 -- 172.21.15.78:0/3812716474 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf88010910 con 0x7fcf8c100ef0 2024-03-24T03:37:33.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.281+0000 7fcf837fe640 1 -- 172.21.15.78:0/3812716474 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf880188f0 con 0x7fcf8c100ef0 2024-03-24T03:37:33.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.281+0000 7fcf945c4640 1 -- 172.21.15.78:0/3812716474 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf8c1a4a30 con 0x7fcf8c100ef0 2024-03-24T03:37:33.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.282+0000 7fcf945c4640 1 -- 172.21.15.78:0/3812716474 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf8c068490 con 0x7fcf8c100ef0 2024-03-24T03:37:33.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.283+0000 7fcf837fe640 1 -- 172.21.15.78:0/3812716474 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fcf88010ac0 con 0x7fcf8c100ef0 2024-03-24T03:37:33.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.283+0000 7fcf837fe640 1 --2- 172.21.15.78:0/3812716474 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fcf780a2970 0x7fcf780a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:33.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.283+0000 7fcf92360640 1 --2- 172.21.15.78:0/3812716474 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fcf780a2970 0x7fcf780a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:33.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.283+0000 7fcf837fe640 1 -- 172.21.15.78:0/3812716474 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 13278+0+0 (secure 0 0 0) 0x7fcf880d04a0 con 0x7fcf8c100ef0 2024-03-24T03:37:33.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.284+0000 7fcf92360640 1 --2- 172.21.15.78:0/3812716474 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fcf780a2970 0x7fcf780a4e30 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c009230 tx=0x7fcf7c005fd0 comp rx=0 tx=0).ready entity=mgr.25243 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:37:33.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.288+0000 7fcf837fe640 1 -- 172.21.15.78:0/3812716474 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcf8809b920 con 0x7fcf8c100ef0 2024-03-24T03:37:33.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:33 smithi106 ceph-mon[36505]: pgmap v6236: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:37:33.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:33 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/468590175' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T03:37:33.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.462+0000 7fcf945c4640 1 -- 172.21.15.78:0/3812716474 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcf8c1a4cf0 con 0x7fcf8c100ef0 2024-03-24T03:37:33.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.462+0000 7fcf837fe640 1 -- 172.21.15.78:0/3812716474 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v157) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcf8809f9c0 con 0x7fcf8c100ef0 2024-03-24T03:37:33.462 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-24T03:37:33.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.464+0000 7fcf945c4640 1 -- 172.21.15.78:0/3812716474 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fcf780a2970 msgr2=0x7fcf780a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:33.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.464+0000 7fcf945c4640 1 --2- 172.21.15.78:0/3812716474 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fcf780a2970 0x7fcf780a4e30 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c009230 tx=0x7fcf7c005fd0 comp rx=0 tx=0).stop 2024-03-24T03:37:33.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.464+0000 7fcf945c4640 1 -- 172.21.15.78:0/3812716474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf8c100ef0 msgr2=0x7fcf8c198400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:33.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.464+0000 7fcf945c4640 1 --2- 172.21.15.78:0/3812716474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf8c100ef0 0x7fcf8c198400 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fcf8800e9f0 tx=0x7fcf8800eeb0 comp rx=0 tx=0).stop 2024-03-24T03:37:33.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.464+0000 7fcf945c4640 1 -- 172.21.15.78:0/3812716474 shutdown_connections 2024-03-24T03:37:33.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.464+0000 7fcf945c4640 1 --2- 172.21.15.78:0/3812716474 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fcf780a2970 0x7fcf780a4e30 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:33.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.464+0000 7fcf945c4640 1 --2- 172.21.15.78:0/3812716474 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcf8c1065f0 0x7fcf8c19c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:33.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.464+0000 7fcf945c4640 1 --2- 172.21.15.78:0/3812716474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf8c100ef0 0x7fcf8c198400 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:33.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.464+0000 7fcf945c4640 1 --2- 172.21.15.78:0/3812716474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf8c100530 0x7fcf8c197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:33.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.464+0000 7fcf945c4640 1 -- 172.21.15.78:0/3812716474 >> 172.21.15.78:0/3812716474 conn(0x7fcf8c0776a0 msgr2=0x7fcf8c0ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:37:33.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.464+0000 7fcf945c4640 1 -- 172.21.15.78:0/3812716474 shutdown_connections 2024-03-24T03:37:33.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.464+0000 7fcf945c4640 1 -- 172.21.15.78:0/3812716474 wait complete. 2024-03-24T03:37:33.475 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T03:37:33.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.580+0000 7fea8d052640 1 -- 172.21.15.78:0/164897253 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea88100f10 msgr2=0x7fea8810d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:33.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.580+0000 7fea8d052640 1 --2- 172.21.15.78:0/164897253 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea88100f10 0x7fea8810d3d0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fea80009900 tx=0x7fea8002bc30 comp rx=0 tx=0).stop 2024-03-24T03:37:33.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.580+0000 7fea8d052640 1 -- 172.21.15.78:0/164897253 shutdown_connections 2024-03-24T03:37:33.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.580+0000 7fea8d052640 1 --2- 172.21.15.78:0/164897253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea88100550 0x7fea881009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:33.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.580+0000 7fea8d052640 1 --2- 172.21.15.78:0/164897253 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea88100f10 0x7fea8810d3d0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:33.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.580+0000 7fea8d052640 1 --2- 172.21.15.78:0/164897253 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea88106610 0x7fea881069f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:33.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.580+0000 7fea8d052640 1 -- 172.21.15.78:0/164897253 >> 172.21.15.78:0/164897253 conn(0x7fea88077670 msgr2=0x7fea88077a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:37:33.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.580+0000 7fea8d052640 1 -- 172.21.15.78:0/164897253 shutdown_connections 2024-03-24T03:37:33.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.580+0000 7fea8d052640 1 -- 172.21.15.78:0/164897253 wait complete. 2024-03-24T03:37:33.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.581+0000 7fea8d052640 1 Processor -- start 2024-03-24T03:37:33.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.581+0000 7fea8d052640 1 -- start start 2024-03-24T03:37:33.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.581+0000 7fea8d052640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea88100550 0x7fea88197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:33.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.581+0000 7fea8d052640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea88100f10 0x7fea88198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:33.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.581+0000 7fea8d052640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea88106610 0x7fea8819c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:33.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.581+0000 7fea8d052640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea881108f0 con 0x7fea88100550 2024-03-24T03:37:33.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.581+0000 7fea8d052640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea88110770 con 0x7fea88100f10 2024-03-24T03:37:33.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.581+0000 7fea8d052640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea88110a70 con 0x7fea88106610 2024-03-24T03:37:33.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.581+0000 7fea86d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea88100550 0x7fea88197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:33.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.581+0000 7fea86d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea88100550 0x7fea88197e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43758/0 (socket says 172.21.15.78:43758) 2024-03-24T03:37:33.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.581+0000 7fea86d9d640 1 -- 172.21.15.78:0/390182772 learned_addr learned my addr 172.21.15.78:0/390182772 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T03:37:33.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.581+0000 7fea8659c640 1 --2- 172.21.15.78:0/390182772 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea88100f10 0x7fea88198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:33.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.581+0000 7fea8759e640 1 --2- 172.21.15.78:0/390182772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea88106610 0x7fea8819c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:33.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.582+0000 7fea86d9d640 1 -- 172.21.15.78:0/390182772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea88106610 msgr2=0x7fea8819c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:33.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.582+0000 7fea86d9d640 1 --2- 172.21.15.78:0/390182772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea88106610 0x7fea8819c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:33.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.582+0000 7fea86d9d640 1 -- 172.21.15.78:0/390182772 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea88100f10 msgr2=0x7fea88198360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:33.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.582+0000 7fea86d9d640 1 --2- 172.21.15.78:0/390182772 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea88100f10 0x7fea88198360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:33.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.582+0000 7fea86d9d640 1 -- 172.21.15.78:0/390182772 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea8819cdc0 con 0x7fea88100550 2024-03-24T03:37:33.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.582+0000 7fea8759e640 1 --2- 172.21.15.78:0/390182772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea88106610 0x7fea8819c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T03:37:33.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.582+0000 7fea8659c640 1 --2- 172.21.15.78:0/390182772 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea88100f10 0x7fea88198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T03:37:33.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.582+0000 7fea86d9d640 1 --2- 172.21.15.78:0/390182772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea88100550 0x7fea88197e20 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fea7800ebc0 tx=0x7fea7800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:37:33.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.582+0000 7fea6ffff640 1 -- 172.21.15.78:0/390182772 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea78010630 con 0x7fea88100550 2024-03-24T03:37:33.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.582+0000 7fea6ffff640 1 -- 172.21.15.78:0/390182772 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fea78010cb0 con 0x7fea88100550 2024-03-24T03:37:33.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.582+0000 7fea8d052640 1 -- 172.21.15.78:0/390182772 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea8819d0b0 con 0x7fea88100550 2024-03-24T03:37:33.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.582+0000 7fea6ffff640 1 -- 172.21.15.78:0/390182772 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea78018bd0 con 0x7fea88100550 2024-03-24T03:37:33.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.582+0000 7fea8d052640 1 -- 172.21.15.78:0/390182772 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea881a4940 con 0x7fea88100550 2024-03-24T03:37:33.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.583+0000 7fea8d052640 1 -- 172.21.15.78:0/390182772 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea88101c70 con 0x7fea88100550 2024-03-24T03:37:33.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.584+0000 7fea6ffff640 1 -- 172.21.15.78:0/390182772 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fea780107d0 con 0x7fea88100550 2024-03-24T03:37:33.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.585+0000 7fea6ffff640 1 --2- 172.21.15.78:0/390182772 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fea700a2970 0x7fea700a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:33.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.585+0000 7fea8659c640 1 --2- 172.21.15.78:0/390182772 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fea700a2970 0x7fea700a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:33.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.585+0000 7fea6ffff640 1 -- 172.21.15.78:0/390182772 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 13278+0+0 (secure 0 0 0) 0x7fea780d0790 con 0x7fea88100550 2024-03-24T03:37:33.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.585+0000 7fea8659c640 1 --2- 172.21.15.78:0/390182772 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fea700a2970 0x7fea700a4e30 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fea88105c00 tx=0x7fea7c00b390 comp rx=0 tx=0).ready entity=mgr.25243 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:37:33.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.589+0000 7fea6ffff640 1 -- 172.21.15.78:0/390182772 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fea7809c260 con 0x7fea88100550 2024-03-24T03:37:33.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:33 smithi107 ceph-mon[36485]: pgmap v6236: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:37:33.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:33 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/468590175' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T03:37:33.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.763+0000 7fea8d052640 1 -- 172.21.15.78:0/390182772 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fea881a4ba0 con 0x7fea88100550 2024-03-24T03:37:33.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.763+0000 7fea6ffff640 1 -- 172.21.15.78:0/390182772 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v157) v1 ==== 100+0+11 (secure 0 0 0) 0x7fea7809bb60 con 0x7fea88100550 2024-03-24T03:37:33.763 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-03-24T03:37:33.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.765+0000 7fea8d052640 1 -- 172.21.15.78:0/390182772 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fea700a2970 msgr2=0x7fea700a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:33.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.765+0000 7fea8d052640 1 --2- 172.21.15.78:0/390182772 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fea700a2970 0x7fea700a4e30 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fea88105c00 tx=0x7fea7c00b390 comp rx=0 tx=0).stop 2024-03-24T03:37:33.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.765+0000 7fea8d052640 1 -- 172.21.15.78:0/390182772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea88100550 msgr2=0x7fea88197e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:33.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.765+0000 7fea8d052640 1 --2- 172.21.15.78:0/390182772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea88100550 0x7fea88197e20 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fea7800ebc0 tx=0x7fea7800c6a0 comp rx=0 tx=0).stop 2024-03-24T03:37:33.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.765+0000 7fea8d052640 1 -- 172.21.15.78:0/390182772 shutdown_connections 2024-03-24T03:37:33.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.765+0000 7fea8d052640 1 --2- 172.21.15.78:0/390182772 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fea700a2970 0x7fea700a4e30 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:33.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.765+0000 7fea8d052640 1 --2- 172.21.15.78:0/390182772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea88106610 0x7fea8819c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:33.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.765+0000 7fea8d052640 1 --2- 172.21.15.78:0/390182772 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea88100f10 0x7fea88198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:33.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.765+0000 7fea8d052640 1 --2- 172.21.15.78:0/390182772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea88100550 0x7fea88197e20 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:33.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.765+0000 7fea8d052640 1 -- 172.21.15.78:0/390182772 >> 172.21.15.78:0/390182772 conn(0x7fea88077670 msgr2=0x7fea880ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:37:33.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.765+0000 7fea8d052640 1 -- 172.21.15.78:0/390182772 shutdown_connections 2024-03-24T03:37:33.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.765+0000 7fea8d052640 1 -- 172.21.15.78:0/390182772 wait complete. 2024-03-24T03:37:33.776 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T03:37:33.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.880+0000 7f04add9e640 1 -- 172.21.15.78:0/666556540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04a81076b0 msgr2=0x7f04a8107a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:33.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.880+0000 7f04add9e640 1 --2- 172.21.15.78:0/666556540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04a81076b0 0x7f04a8107a90 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f0498009930 tx=0x7f049802bc30 comp rx=0 tx=0).stop 2024-03-24T03:37:33.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.880+0000 7f04add9e640 1 -- 172.21.15.78:0/666556540 shutdown_connections 2024-03-24T03:37:33.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.880+0000 7f04add9e640 1 --2- 172.21.15.78:0/666556540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a8069240 0x7f04a810e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:33.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.880+0000 7f04add9e640 1 --2- 172.21.15.78:0/666556540 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04a8068880 0x7f04a8068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:33.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.880+0000 7f04add9e640 1 --2- 172.21.15.78:0/666556540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04a81076b0 0x7f04a8107a90 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:33.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.880+0000 7f04add9e640 1 -- 172.21.15.78:0/666556540 >> 172.21.15.78:0/666556540 conn(0x7f04a80776a0 msgr2=0x7f04a8077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:37:33.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.880+0000 7f04add9e640 1 -- 172.21.15.78:0/666556540 shutdown_connections 2024-03-24T03:37:33.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.880+0000 7f04add9e640 1 -- 172.21.15.78:0/666556540 wait complete. 2024-03-24T03:37:33.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.881+0000 7f04add9e640 1 Processor -- start 2024-03-24T03:37:33.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.881+0000 7f04add9e640 1 -- start start 2024-03-24T03:37:33.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.881+0000 7f04add9e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a8068880 0x7f04a8197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:33.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.881+0000 7f04add9e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04a8069240 0x7f04a81983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:33.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.881+0000 7f04add9e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04a81076b0 0x7f04a819c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:33.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.881+0000 7f04add9e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a81119f0 con 0x7f04a8069240 2024-03-24T03:37:33.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.881+0000 7f04add9e640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a8111870 con 0x7f04a81076b0 2024-03-24T03:37:33.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.881+0000 7f04add9e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a8111b70 con 0x7f04a8068880 2024-03-24T03:37:33.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.881+0000 7f04a6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04a8069240 0x7f04a81983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:33.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.882+0000 7f04a6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04a8069240 0x7f04a81983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43786/0 (socket says 172.21.15.78:43786) 2024-03-24T03:37:33.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.882+0000 7f04a7fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04a81076b0 0x7f04a819c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:33.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.882+0000 7f04a6ffd640 1 -- 172.21.15.78:0/789111460 learned_addr learned my addr 172.21.15.78:0/789111460 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T03:37:33.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.882+0000 7f04a77fe640 1 --2- 172.21.15.78:0/789111460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a8068880 0x7f04a8197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:33.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.882+0000 7f04a6ffd640 1 -- 172.21.15.78:0/789111460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a8068880 msgr2=0x7f04a8197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:33.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.882+0000 7f04a6ffd640 1 --2- 172.21.15.78:0/789111460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a8068880 0x7f04a8197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:33.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.882+0000 7f04a6ffd640 1 -- 172.21.15.78:0/789111460 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04a81076b0 msgr2=0x7f04a819c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:33.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.882+0000 7f04a6ffd640 1 --2- 172.21.15.78:0/789111460 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04a81076b0 0x7f04a819c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:33.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.882+0000 7f04a6ffd640 1 -- 172.21.15.78:0/789111460 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a819ce50 con 0x7f04a8069240 2024-03-24T03:37:33.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.882+0000 7f04a7fff640 1 --2- 172.21.15.78:0/789111460 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04a81076b0 0x7f04a819c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T03:37:33.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.882+0000 7f04a6ffd640 1 --2- 172.21.15.78:0/789111460 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04a8069240 0x7f04a81983f0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f049c00e9f0 tx=0x7f049c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:37:33.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.882+0000 7f04a4ff9640 1 -- 172.21.15.78:0/789111460 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f049c00cd80 con 0x7f04a8069240 2024-03-24T03:37:33.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.882+0000 7f04a4ff9640 1 -- 172.21.15.78:0/789111460 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f049c010910 con 0x7f04a8069240 2024-03-24T03:37:33.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.882+0000 7f04add9e640 1 -- 172.21.15.78:0/789111460 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04a819d140 con 0x7f04a8069240 2024-03-24T03:37:33.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.882+0000 7f04a4ff9640 1 -- 172.21.15.78:0/789111460 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f049c018990 con 0x7f04a8069240 2024-03-24T03:37:33.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.882+0000 7f04add9e640 1 -- 172.21.15.78:0/789111460 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04a81a49d0 con 0x7f04a8069240 2024-03-24T03:37:33.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.883+0000 7f04add9e640 1 -- 172.21.15.78:0/789111460 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04a8068490 con 0x7f04a8069240 2024-03-24T03:37:33.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.884+0000 7f04a4ff9640 1 -- 172.21.15.78:0/789111460 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f049c010ac0 con 0x7f04a8069240 2024-03-24T03:37:33.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.885+0000 7f04a4ff9640 1 --2- 172.21.15.78:0/789111460 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f04900a2970 0x7f04900a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:33.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.885+0000 7f04a4ff9640 1 -- 172.21.15.78:0/789111460 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 13278+0+0 (secure 0 0 0) 0x7f049c0d0810 con 0x7f04a8069240 2024-03-24T03:37:33.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.885+0000 7f04a77fe640 1 --2- 172.21.15.78:0/789111460 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f04900a2970 0x7f04900a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:33.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.885+0000 7f04a77fe640 1 --2- 172.21.15.78:0/789111460 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f04900a2970 0x7f04900a4e30 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0498000c00 tx=0x7f0498005fd0 comp rx=0 tx=0).ready entity=mgr.25243 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:37:33.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:33.888+0000 7f04a4ff9640 1 -- 172.21.15.78:0/789111460 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f049c09bc90 con 0x7f04a8069240 2024-03-24T03:37:34.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.065+0000 7f04add9e640 1 -- 172.21.15.78:0/789111460 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f04a802d070 con 0x7f04a8069240 2024-03-24T03:37:34.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.065+0000 7f04a4ff9640 1 -- 172.21.15.78:0/789111460 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v157) v1 ==== 103+0+11 (secure 0 0 0) 0x7f049c09fd30 con 0x7f04a8069240 2024-03-24T03:37:34.065 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-24T03:37:34.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.067+0000 7f04add9e640 1 -- 172.21.15.78:0/789111460 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f04900a2970 msgr2=0x7f04900a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:34.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.067+0000 7f04add9e640 1 --2- 172.21.15.78:0/789111460 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f04900a2970 0x7f04900a4e30 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0498000c00 tx=0x7f0498005fd0 comp rx=0 tx=0).stop 2024-03-24T03:37:34.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.067+0000 7f04add9e640 1 -- 172.21.15.78:0/789111460 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04a8069240 msgr2=0x7f04a81983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:34.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.067+0000 7f04add9e640 1 --2- 172.21.15.78:0/789111460 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04a8069240 0x7f04a81983f0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f049c00e9f0 tx=0x7f049c00eeb0 comp rx=0 tx=0).stop 2024-03-24T03:37:34.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.067+0000 7f04add9e640 1 -- 172.21.15.78:0/789111460 shutdown_connections 2024-03-24T03:37:34.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.067+0000 7f04add9e640 1 --2- 172.21.15.78:0/789111460 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f04900a2970 0x7f04900a4e30 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:34.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.067+0000 7f04add9e640 1 --2- 172.21.15.78:0/789111460 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04a81076b0 0x7f04a819c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:34.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.067+0000 7f04add9e640 1 --2- 172.21.15.78:0/789111460 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04a8069240 0x7f04a81983f0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:34.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.067+0000 7f04add9e640 1 --2- 172.21.15.78:0/789111460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a8068880 0x7f04a8197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:34.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.067+0000 7f04add9e640 1 -- 172.21.15.78:0/789111460 >> 172.21.15.78:0/789111460 conn(0x7f04a80776a0 msgr2=0x7f04a80ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:37:34.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.067+0000 7f04add9e640 1 -- 172.21.15.78:0/789111460 shutdown_connections 2024-03-24T03:37:34.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.067+0000 7f04add9e640 1 -- 172.21.15.78:0/789111460 wait complete. 2024-03-24T03:37:34.078 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-24T03:37:34.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.181+0000 7f97cb80c640 1 -- 172.21.15.78:0/3443685302 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97c4100560 msgr2=0x7f97c41009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:34.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.181+0000 7f97cb80c640 1 --2- 172.21.15.78:0/3443685302 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97c4100560 0x7f97c41009e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f97c0009890 tx=0x7f97c002bb70 comp rx=0 tx=0).stop 2024-03-24T03:37:34.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.181+0000 7f97cb80c640 1 -- 172.21.15.78:0/3443685302 shutdown_connections 2024-03-24T03:37:34.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.181+0000 7f97cb80c640 1 --2- 172.21.15.78:0/3443685302 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97c4100f20 0x7f97c410d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:34.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.181+0000 7f97cb80c640 1 --2- 172.21.15.78:0/3443685302 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97c4100560 0x7f97c41009e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:34.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.181+0000 7f97cb80c640 1 --2- 172.21.15.78:0/3443685302 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97c4106620 0x7f97c4106a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:34.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.181+0000 7f97cb80c640 1 -- 172.21.15.78:0/3443685302 >> 172.21.15.78:0/3443685302 conn(0x7f97c4076920 msgr2=0x7f97c4076d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:37:34.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.181+0000 7f97cb80c640 1 -- 172.21.15.78:0/3443685302 shutdown_connections 2024-03-24T03:37:34.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.181+0000 7f97cb80c640 1 -- 172.21.15.78:0/3443685302 wait complete. 2024-03-24T03:37:34.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.181+0000 7f97cb80c640 1 Processor -- start 2024-03-24T03:37:34.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.181+0000 7f97cb80c640 1 -- start start 2024-03-24T03:37:34.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.182+0000 7f97cb80c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97c4100560 0x7f97c419a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:34.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.182+0000 7f97cb80c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97c4100f20 0x7f97c419a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:34.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.182+0000 7f97cb80c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97c4106620 0x7f97c419ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:34.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.182+0000 7f97cb80c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c4110970 con 0x7f97c4106620 2024-03-24T03:37:34.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.182+0000 7f97cb80c640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c41107f0 con 0x7f97c4100f20 2024-03-24T03:37:34.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.182+0000 7f97cb80c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c4110af0 con 0x7f97c4100560 2024-03-24T03:37:34.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.182+0000 7f97c9da9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97c4106620 0x7f97c419ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:34.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.182+0000 7f97c9da9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97c4106620 0x7f97c419ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43792/0 (socket says 172.21.15.78:43792) 2024-03-24T03:37:34.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.182+0000 7f97c9da9640 1 -- 172.21.15.78:0/2338065178 learned_addr learned my addr 172.21.15.78:0/2338065178 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T03:37:34.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.182+0000 7f97c8da7640 1 --2- 172.21.15.78:0/2338065178 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97c4100f20 0x7f97c419a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:34.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.182+0000 7f97c95a8640 1 --2- 172.21.15.78:0/2338065178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97c4100560 0x7f97c419a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:34.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.182+0000 7f97c9da9640 1 -- 172.21.15.78:0/2338065178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97c4100560 msgr2=0x7f97c419a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:34.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.182+0000 7f97c9da9640 1 --2- 172.21.15.78:0/2338065178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97c4100560 0x7f97c419a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:34.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.182+0000 7f97c9da9640 1 -- 172.21.15.78:0/2338065178 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97c4100f20 msgr2=0x7f97c419a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:34.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.182+0000 7f97c9da9640 1 --2- 172.21.15.78:0/2338065178 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97c4100f20 0x7f97c419a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:34.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.182+0000 7f97c9da9640 1 -- 172.21.15.78:0/2338065178 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c419f150 con 0x7f97c4106620 2024-03-24T03:37:34.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.182+0000 7f97c8da7640 1 --2- 172.21.15.78:0/2338065178 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97c4100f20 0x7f97c419a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T03:37:34.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.182+0000 7f97c9da9640 1 --2- 172.21.15.78:0/2338065178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97c4106620 0x7f97c419ea20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f97bc009fd0 tx=0x7f97bc00c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:37:34.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.182+0000 7f97ba7fc640 1 -- 172.21.15.78:0/2338065178 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc004b20 con 0x7f97c4106620 2024-03-24T03:37:34.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.182+0000 7f97cb80c640 1 -- 172.21.15.78:0/2338065178 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97c419f440 con 0x7f97c4106620 2024-03-24T03:37:34.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.182+0000 7f97ba7fc640 1 -- 172.21.15.78:0/2338065178 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f97bc003e50 con 0x7f97c4106620 2024-03-24T03:37:34.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.182+0000 7f97cb80c640 1 -- 172.21.15.78:0/2338065178 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97c41a6cd0 con 0x7f97c4106620 2024-03-24T03:37:34.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.182+0000 7f97ba7fc640 1 -- 172.21.15.78:0/2338065178 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc01a020 con 0x7f97c4106620 2024-03-24T03:37:34.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.183+0000 7f97cb80c640 1 -- 172.21.15.78:0/2338065178 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97c4101c80 con 0x7f97c4106620 2024-03-24T03:37:34.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.184+0000 7f97ba7fc640 1 -- 172.21.15.78:0/2338065178 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f97bc01d020 con 0x7f97c4106620 2024-03-24T03:37:34.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.185+0000 7f97ba7fc640 1 --2- 172.21.15.78:0/2338065178 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f97b00a2970 0x7f97b00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:34.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.185+0000 7f97c95a8640 1 --2- 172.21.15.78:0/2338065178 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f97b00a2970 0x7f97b00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:34.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.185+0000 7f97ba7fc640 1 -- 172.21.15.78:0/2338065178 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 13278+0+0 (secure 0 0 0) 0x7f97bc0d0060 con 0x7f97c4106620 2024-03-24T03:37:34.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.186+0000 7f97c95a8640 1 --2- 172.21.15.78:0/2338065178 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f97b00a2970 0x7f97b00a4e30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f97b4009990 tx=0x7f97b4009110 comp rx=0 tx=0).ready entity=mgr.25243 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:37:34.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.190+0000 7f97ba7fc640 1 -- 172.21.15.78:0/2338065178 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f97bc09bb30 con 0x7f97c4106620 2024-03-24T03:37:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:34 smithi078 ceph-mon[32429]: from='client.15753 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T03:37:34.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10928.258801 secs 2024-03-24T03:37:34.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:34 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2052913777' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T03:37:34.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:34 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3812716474' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T03:37:34.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:34 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/390182772' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T03:37:34.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:34 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/789111460' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T03:37:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:34 smithi106 ceph-mon[36505]: from='client.15753 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T03:37:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10928.258801 secs 2024-03-24T03:37:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:34 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2052913777' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T03:37:34.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:34 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3812716474' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T03:37:34.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:34 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/390182772' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T03:37:34.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:34 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/789111460' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T03:37:34.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.413+0000 7f97cb80c640 1 -- 172.21.15.78:0/2338065178 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7f97c419b4b0 con 0x7f97c4106620 2024-03-24T03:37:34.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.421+0000 7f97ba7fc640 1 -- 172.21.15.78:0/2338065178 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v25) v1 ==== 81+0+740361 (secure 0 0 0) 0x7f97bc09b430 con 0x7f97c4106620 2024-03-24T03:37:34.431 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T03:37:34.431 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-24T03:37:34.432 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 25, 2024-03-24T03:37:34.432 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.432 INFO:teuthology.orchestra.run.smithi078.stdout: "active_gid": 25243, 2024-03-24T03:37:34.432 INFO:teuthology.orchestra.run.smithi078.stdout: "active_name": "x", 2024-03-24T03:37:34.432 INFO:teuthology.orchestra.run.smithi078.stdout: "active_addrs": { 2024-03-24T03:37:34.432 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-03-24T03:37:34.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:34.432 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-03-24T03:37:34.433 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6800", 2024-03-24T03:37:34.433 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3337965055 2024-03-24T03:37:34.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:34.433 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-03-24T03:37:34.433 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6801", 2024-03-24T03:37:34.433 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3337965055 2024-03-24T03:37:34.433 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.434 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-24T03:37:34.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.434 INFO:teuthology.orchestra.run.smithi078.stdout: "active_addr": "172.21.15.78:6801/3337965055", 2024-03-24T03:37:34.434 INFO:teuthology.orchestra.run.smithi078.stdout: "active_change": "2024-03-24T00:21:09.220816+0000", 2024-03-24T03:37:34.434 INFO:teuthology.orchestra.run.smithi078.stdout: "active_mgr_features": 4540701547738038271, 2024-03-24T03:37:34.434 INFO:teuthology.orchestra.run.smithi078.stdout: "available": true, 2024-03-24T03:37:34.434 INFO:teuthology.orchestra.run.smithi078.stdout: "standbys": [ 2024-03-24T03:37:34.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:34.435 INFO:teuthology.orchestra.run.smithi078.stdout: "gid": 25240, 2024-03-24T03:37:34.435 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "z", 2024-03-24T03:37:34.435 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_features": 4540701547738038271, 2024-03-24T03:37:34.435 INFO:teuthology.orchestra.run.smithi078.stdout: "available_modules": [ 2024-03-24T03:37:34.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:34.435 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "alerts", 2024-03-24T03:37:34.435 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:34.435 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:34.436 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:34.436 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-24T03:37:34.436 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-24T03:37:34.436 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:34.436 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.436 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.436 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-24T03:37:34.436 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.436 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.437 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.437 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How frequently to reexamine health status", 2024-03-24T03:37:34.437 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.437 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.437 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.437 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:34.437 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:34.438 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.438 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.438 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.438 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.438 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.438 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.438 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.438 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.438 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.439 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.439 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.439 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.439 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.439 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.439 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.439 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.439 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.439 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.440 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:34.440 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:34.440 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.440 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.440 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.440 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.440 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.440 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.440 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.441 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.441 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.441 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.441 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.441 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:34.441 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:34.441 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.441 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.442 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.442 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:34.442 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.442 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.442 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.442 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.442 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.442 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.443 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.443 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.443 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.443 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.443 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.443 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.443 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.443 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.444 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:34.444 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:34.444 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.444 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.444 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.444 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.444 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.444 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.445 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.445 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.445 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.445 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.445 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.445 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_destination": { 2024-03-24T03:37:34.445 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_destination", 2024-03-24T03:37:34.445 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.446 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.446 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.446 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.446 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.446 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.446 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.446 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Email address to send alerts to", 2024-03-24T03:37:34.446 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.447 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.447 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.447 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_from_name": { 2024-03-24T03:37:34.447 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_from_name", 2024-03-24T03:37:34.447 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.447 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.447 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.448 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "Ceph", 2024-03-24T03:37:34.448 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.448 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.448 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.448 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Email From: name", 2024-03-24T03:37:34.448 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.448 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.448 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.449 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_host": { 2024-03-24T03:37:34.449 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_host", 2024-03-24T03:37:34.449 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.449 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.449 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.449 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.449 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.450 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.450 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.450 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP server", 2024-03-24T03:37:34.450 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.450 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.450 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.450 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_password": { 2024-03-24T03:37:34.450 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_password", 2024-03-24T03:37:34.451 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.451 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.451 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.451 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.451 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.451 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.451 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.451 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Password to authenticate with", 2024-03-24T03:37:34.452 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.452 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.452 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.452 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_port": { 2024-03-24T03:37:34.452 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_port", 2024-03-24T03:37:34.452 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:34.452 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.452 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.453 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "465", 2024-03-24T03:37:34.453 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.453 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.453 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.453 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP port", 2024-03-24T03:37:34.453 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.453 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.453 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.454 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_sender": { 2024-03-24T03:37:34.454 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_sender", 2024-03-24T03:37:34.454 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.454 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.454 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.454 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.454 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.454 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.454 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.455 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP envelope sender", 2024-03-24T03:37:34.455 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.455 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.455 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.455 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_ssl": { 2024-03-24T03:37:34.455 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_ssl", 2024-03-24T03:37:34.456 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.456 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.456 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.456 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.456 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.456 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.456 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.456 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-24T03:37:34.456 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.457 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.457 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.457 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_user": { 2024-03-24T03:37:34.457 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_user", 2024-03-24T03:37:34.457 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.457 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.457 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.457 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.457 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.458 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.458 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.458 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "User to authenticate as", 2024-03-24T03:37:34.458 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.458 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.458 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.458 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.458 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:34.459 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "balancer", 2024-03-24T03:37:34.459 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:34.459 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:34.459 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:34.459 INFO:teuthology.orchestra.run.smithi078.stdout: "active": { 2024-03-24T03:37:34.459 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "active", 2024-03-24T03:37:34.459 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.460 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.460 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.460 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.460 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.460 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.460 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.460 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "automatically balance PGs across cluster", 2024-03-24T03:37:34.460 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.461 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.461 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.461 INFO:teuthology.orchestra.run.smithi078.stdout: "begin_time": { 2024-03-24T03:37:34.461 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "begin_time", 2024-03-24T03:37:34.461 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.461 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.461 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.461 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0000", 2024-03-24T03:37:34.462 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.462 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.462 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.462 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "beginning time of day to automatically balance", 2024-03-24T03:37:34.462 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T03:37:34.462 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.462 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.463 INFO:teuthology.orchestra.run.smithi078.stdout: "begin_weekday": { 2024-03-24T03:37:34.463 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "begin_weekday", 2024-03-24T03:37:34.463 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-24T03:37:34.463 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.463 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.463 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-24T03:37:34.463 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0", 2024-03-24T03:37:34.463 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "6", 2024-03-24T03:37:34.463 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.463 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-24T03:37:34.464 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T03:37:34.464 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.464 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.464 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_max_iterations": { 2024-03-24T03:37:34.464 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_max_iterations", 2024-03-24T03:37:34.464 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-24T03:37:34.464 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.465 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.465 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "25", 2024-03-24T03:37:34.465 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-24T03:37:34.465 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "250", 2024-03-24T03:37:34.465 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.465 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-24T03:37:34.465 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.465 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.465 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.466 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_metrics": { 2024-03-24T03:37:34.466 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_metrics", 2024-03-24T03:37:34.466 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.466 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.466 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.466 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "pgs,objects,bytes", 2024-03-24T03:37:34.466 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.466 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.466 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.467 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-24T03:37:34.467 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-24T03:37:34.467 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.467 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.467 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_step": { 2024-03-24T03:37:34.467 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_step", 2024-03-24T03:37:34.467 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-24T03:37:34.467 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.468 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.468 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0.5", 2024-03-24T03:37:34.468 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0.001", 2024-03-24T03:37:34.468 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "0.999", 2024-03-24T03:37:34.468 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.468 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "aggressiveness of optimization", 2024-03-24T03:37:34.468 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-24T03:37:34.468 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.468 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.469 INFO:teuthology.orchestra.run.smithi078.stdout: "end_time": { 2024-03-24T03:37:34.469 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "end_time", 2024-03-24T03:37:34.469 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.469 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.469 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.469 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "2359", 2024-03-24T03:37:34.469 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.469 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.470 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.470 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "ending time of day to automatically balance", 2024-03-24T03:37:34.470 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T03:37:34.470 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.470 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.470 INFO:teuthology.orchestra.run.smithi078.stdout: "end_weekday": { 2024-03-24T03:37:34.470 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "end_weekday", 2024-03-24T03:37:34.471 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-24T03:37:34.471 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.471 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.471 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-24T03:37:34.471 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0", 2024-03-24T03:37:34.471 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "6", 2024-03-24T03:37:34.471 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.471 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-24T03:37:34.471 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T03:37:34.471 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.472 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.472 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:34.472 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:34.472 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.472 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.472 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.472 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.472 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.473 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.473 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.473 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.473 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.473 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.473 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.473 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.473 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.473 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.473 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.474 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.474 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.474 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.474 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:34.474 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:34.474 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.474 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.475 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.475 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.475 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.475 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.475 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.475 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.475 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.475 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.476 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.476 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:34.476 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:34.476 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.476 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.476 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.476 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:34.476 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.476 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.477 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.477 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.477 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.477 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.477 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.477 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.477 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.477 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.477 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.478 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.478 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.478 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.478 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:34.478 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:34.478 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.478 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.478 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.479 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.479 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.479 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.479 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.479 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.479 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.479 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.479 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.479 INFO:teuthology.orchestra.run.smithi078.stdout: "min_score": { 2024-03-24T03:37:34.480 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "min_score", 2024-03-24T03:37:34.480 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-24T03:37:34.480 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.480 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.480 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-24T03:37:34.480 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.480 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.480 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.480 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-24T03:37:34.481 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.481 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.481 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.481 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": { 2024-03-24T03:37:34.481 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mode", 2024-03-24T03:37:34.481 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.481 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.481 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.481 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "upmap", 2024-03-24T03:37:34.482 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.482 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.482 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.482 INFO:teuthology.orchestra.run.smithi078.stdout: "crush-compat", 2024-03-24T03:37:34.482 INFO:teuthology.orchestra.run.smithi078.stdout: "none", 2024-03-24T03:37:34.482 INFO:teuthology.orchestra.run.smithi078.stdout: "read", 2024-03-24T03:37:34.482 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap", 2024-03-24T03:37:34.482 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap-read" 2024-03-24T03:37:34.482 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.483 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Balancer mode", 2024-03-24T03:37:34.483 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.483 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.483 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.483 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_ids": { 2024-03-24T03:37:34.483 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pool_ids", 2024-03-24T03:37:34.483 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.484 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.484 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.484 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.484 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.484 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.484 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.484 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-24T03:37:34.484 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.484 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.485 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.485 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-24T03:37:34.485 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-24T03:37:34.485 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:34.485 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.485 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.485 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-24T03:37:34.485 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.486 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.486 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.486 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-24T03:37:34.486 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.486 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.486 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.486 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap_max_deviation": { 2024-03-24T03:37:34.486 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "upmap_max_deviation", 2024-03-24T03:37:34.487 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:34.487 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.487 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.487 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-24T03:37:34.487 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-24T03:37:34.487 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.487 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.487 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-24T03:37:34.488 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-24T03:37:34.488 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.488 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.488 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap_max_optimizations": { 2024-03-24T03:37:34.488 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "upmap_max_optimizations", 2024-03-24T03:37:34.488 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-24T03:37:34.488 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.488 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.489 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-24T03:37:34.489 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.489 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.489 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.489 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-24T03:37:34.489 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.489 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.489 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.490 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.490 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:34.490 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cephadm", 2024-03-24T03:37:34.490 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:34.490 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:34.490 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:34.490 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_down_multiplier": { 2024-03-24T03:37:34.491 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_down_multiplier", 2024-03-24T03:37:34.491 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-24T03:37:34.491 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.491 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.491 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3.0", 2024-03-24T03:37:34.491 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.491 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.491 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.491 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-24T03:37:34.492 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.492 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.492 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.492 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_refresh_rate": { 2024-03-24T03:37:34.492 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_refresh_rate", 2024-03-24T03:37:34.492 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:34.492 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.492 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.492 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "20", 2024-03-24T03:37:34.493 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.493 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.493 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.493 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-24T03:37:34.493 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.493 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.493 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.493 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_starting_port": { 2024-03-24T03:37:34.494 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_starting_port", 2024-03-24T03:37:34.494 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:34.494 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.494 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.494 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "4721", 2024-03-24T03:37:34.494 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.494 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.506 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.506 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-24T03:37:34.506 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.506 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.507 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.507 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_ptrace": { 2024-03-24T03:37:34.507 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_ptrace", 2024-03-24T03:37:34.507 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.509 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.509 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.509 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.509 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.509 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.510 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.510 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-24T03:37:34.510 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-24T03:37:34.510 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.510 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.510 INFO:teuthology.orchestra.run.smithi078.stdout: "autotune_interval": { 2024-03-24T03:37:34.510 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "autotune_interval", 2024-03-24T03:37:34.510 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:34.511 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.511 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.511 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-24T03:37:34.511 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.511 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.511 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.511 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-24T03:37:34.511 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.512 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.512 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.512 INFO:teuthology.orchestra.run.smithi078.stdout: "autotune_memory_target_ratio": { 2024-03-24T03:37:34.512 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "autotune_memory_target_ratio", 2024-03-24T03:37:34.512 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-24T03:37:34.512 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.512 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.512 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0.7", 2024-03-24T03:37:34.513 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.513 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.513 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.513 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-24T03:37:34.513 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.513 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.513 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.514 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm_log_destination": { 2024-03-24T03:37:34.514 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cephadm_log_destination", 2024-03-24T03:37:34.514 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.514 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.514 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.514 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.514 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.514 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.515 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.515 INFO:teuthology.orchestra.run.smithi078.stdout: "file", 2024-03-24T03:37:34.515 INFO:teuthology.orchestra.run.smithi078.stdout: "file,syslog", 2024-03-24T03:37:34.515 INFO:teuthology.orchestra.run.smithi078.stdout: "syslog" 2024-03-24T03:37:34.515 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.515 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-24T03:37:34.515 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.515 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.515 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.516 INFO:teuthology.orchestra.run.smithi078.stdout: "cgroups_split": { 2024-03-24T03:37:34.516 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cgroups_split", 2024-03-24T03:37:34.516 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.516 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.516 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.516 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.516 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.516 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.517 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.517 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-24T03:37:34.517 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.517 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.517 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.517 INFO:teuthology.orchestra.run.smithi078.stdout: "config_checks_enabled": { 2024-03-24T03:37:34.517 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_checks_enabled", 2024-03-24T03:37:34.518 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.518 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.518 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.518 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.518 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.518 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.518 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.518 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-24T03:37:34.519 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.519 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.519 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.519 INFO:teuthology.orchestra.run.smithi078.stdout: "config_dashboard": { 2024-03-24T03:37:34.519 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_dashboard", 2024-03-24T03:37:34.519 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.519 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.520 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.520 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.520 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.520 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.520 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.520 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-24T03:37:34.520 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.520 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.520 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.521 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_alertmanager": { 2024-03-24T03:37:34.521 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_alertmanager", 2024-03-24T03:37:34.521 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.521 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.521 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.521 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-24T03:37:34.521 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.521 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.522 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.522 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-24T03:37:34.522 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.522 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.522 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.522 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_base": { 2024-03-24T03:37:34.522 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_base", 2024-03-24T03:37:34.523 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.523 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.523 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.523 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-24T03:37:34.523 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.523 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.523 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.523 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Container image name, without the tag", 2024-03-24T03:37:34.523 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.524 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.524 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.524 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_elasticsearch": { 2024-03-24T03:37:34.524 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_elasticsearch", 2024-03-24T03:37:34.524 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.524 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.524 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.524 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-24T03:37:34.525 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.525 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.525 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.525 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "elasticsearch container image", 2024-03-24T03:37:34.525 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.525 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.525 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.525 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_grafana": { 2024-03-24T03:37:34.526 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_grafana", 2024-03-24T03:37:34.526 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.526 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.526 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.526 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-24T03:37:34.526 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.526 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.526 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.527 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-24T03:37:34.527 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.527 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.527 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.527 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_haproxy": { 2024-03-24T03:37:34.527 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_haproxy", 2024-03-24T03:37:34.527 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.527 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.527 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.528 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-24T03:37:34.528 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.528 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.528 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.528 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "HAproxy container image", 2024-03-24T03:37:34.528 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.528 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.528 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.529 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_agent": { 2024-03-24T03:37:34.529 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_agent", 2024-03-24T03:37:34.529 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.529 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.529 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.529 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-24T03:37:34.529 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.529 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.530 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.530 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger agent container image", 2024-03-24T03:37:34.530 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.530 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.530 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.530 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_collector": { 2024-03-24T03:37:34.530 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_collector", 2024-03-24T03:37:34.531 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.531 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.531 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.531 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-24T03:37:34.531 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.531 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.531 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.531 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger collector container image", 2024-03-24T03:37:34.532 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.532 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.532 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.532 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_query": { 2024-03-24T03:37:34.532 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_query", 2024-03-24T03:37:34.532 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.532 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.532 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.533 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-24T03:37:34.533 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.533 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.533 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.533 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger query container image", 2024-03-24T03:37:34.533 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.533 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.533 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.534 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_keepalived": { 2024-03-24T03:37:34.534 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_keepalived", 2024-03-24T03:37:34.534 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.534 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.534 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.534 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-24T03:37:34.534 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.534 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.535 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.535 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Keepalived container image", 2024-03-24T03:37:34.535 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.535 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.535 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.535 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_loki": { 2024-03-24T03:37:34.535 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_loki", 2024-03-24T03:37:34.536 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.536 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.536 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.536 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-24T03:37:34.536 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.536 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.536 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.536 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Loki container image", 2024-03-24T03:37:34.537 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.537 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.537 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.537 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_node_exporter": { 2024-03-24T03:37:34.537 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_node_exporter", 2024-03-24T03:37:34.537 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.537 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.537 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.538 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-24T03:37:34.538 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.538 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.538 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.538 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-24T03:37:34.538 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.538 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.538 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.539 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_nvmeof": { 2024-03-24T03:37:34.539 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_nvmeof", 2024-03-24T03:37:34.539 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.539 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.539 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.539 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-24T03:37:34.539 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.539 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.539 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.540 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Nvme-of container image", 2024-03-24T03:37:34.540 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.540 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.540 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.540 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_prometheus": { 2024-03-24T03:37:34.540 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_prometheus", 2024-03-24T03:37:34.540 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.540 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.541 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.541 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-24T03:37:34.541 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.541 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.541 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.541 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-24T03:37:34.541 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.541 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.542 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.542 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_promtail": { 2024-03-24T03:37:34.542 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_promtail", 2024-03-24T03:37:34.542 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.542 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.542 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.542 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-24T03:37:34.542 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.543 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.543 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.543 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Promtail container image", 2024-03-24T03:37:34.543 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.543 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.543 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.543 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_snmp_gateway": { 2024-03-24T03:37:34.544 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_snmp_gateway", 2024-03-24T03:37:34.544 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.544 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.544 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.544 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-24T03:37:34.544 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.544 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.544 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.544 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SNMP Gateway container image", 2024-03-24T03:37:34.544 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.545 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.545 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.545 INFO:teuthology.orchestra.run.smithi078.stdout: "container_init": { 2024-03-24T03:37:34.545 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_init", 2024-03-24T03:37:34.545 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.545 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.546 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.546 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.546 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.546 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.546 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.546 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Run podman/docker with `--init`", 2024-03-24T03:37:34.546 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.546 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.546 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.547 INFO:teuthology.orchestra.run.smithi078.stdout: "daemon_cache_timeout": { 2024-03-24T03:37:34.547 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "daemon_cache_timeout", 2024-03-24T03:37:34.547 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:34.547 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.547 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.547 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-24T03:37:34.547 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.548 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.548 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.548 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-24T03:37:34.548 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.548 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.548 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.548 INFO:teuthology.orchestra.run.smithi078.stdout: "default_cephadm_command_timeout": { 2024-03-24T03:37:34.548 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default_cephadm_command_timeout", 2024-03-24T03:37:34.549 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:34.549 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.549 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.549 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "900", 2024-03-24T03:37:34.549 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.549 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.549 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.549 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-24T03:37:34.549 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.550 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.550 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.550 INFO:teuthology.orchestra.run.smithi078.stdout: "default_registry": { 2024-03-24T03:37:34.550 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default_registry", 2024-03-24T03:37:34.550 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.550 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.550 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.551 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io", 2024-03-24T03:37:34.551 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.551 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.551 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.551 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-24T03:37:34.551 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.551 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.551 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.552 INFO:teuthology.orchestra.run.smithi078.stdout: "device_cache_timeout": { 2024-03-24T03:37:34.552 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_cache_timeout", 2024-03-24T03:37:34.552 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:34.552 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.552 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.552 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1800", 2024-03-24T03:37:34.552 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.552 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.553 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.553 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache device inventory", 2024-03-24T03:37:34.553 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.553 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.553 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.553 INFO:teuthology.orchestra.run.smithi078.stdout: "device_enhanced_scan": { 2024-03-24T03:37:34.553 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_enhanced_scan", 2024-03-24T03:37:34.554 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.554 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.554 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.554 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.554 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.554 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.554 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.554 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-24T03:37:34.555 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.555 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.555 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.555 INFO:teuthology.orchestra.run.smithi078.stdout: "facts_cache_timeout": { 2024-03-24T03:37:34.555 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "facts_cache_timeout", 2024-03-24T03:37:34.555 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:34.555 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.555 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.556 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-24T03:37:34.556 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.556 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.556 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.556 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache host facts data", 2024-03-24T03:37:34.556 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.556 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.556 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.557 INFO:teuthology.orchestra.run.smithi078.stdout: "host_check_interval": { 2024-03-24T03:37:34.557 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "host_check_interval", 2024-03-24T03:37:34.557 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:34.557 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.557 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.557 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-24T03:37:34.557 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.558 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.558 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.558 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to perform a host check", 2024-03-24T03:37:34.558 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.558 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.558 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.558 INFO:teuthology.orchestra.run.smithi078.stdout: "hw_monitoring": { 2024-03-24T03:37:34.559 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "hw_monitoring", 2024-03-24T03:37:34.559 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.559 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.559 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.559 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.559 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.559 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.559 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.560 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-24T03:37:34.560 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.560 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.560 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.560 INFO:teuthology.orchestra.run.smithi078.stdout: "inventory_list_all": { 2024-03-24T03:37:34.560 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "inventory_list_all", 2024-03-24T03:37:34.560 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.561 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.561 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.561 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.561 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.561 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.561 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.561 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-24T03:37:34.561 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.562 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.562 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.562 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:34.562 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:34.562 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.562 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.562 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.562 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.563 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.563 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.563 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.563 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.563 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.563 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.563 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.563 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.563 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.563 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.564 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.564 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.564 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.564 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.564 INFO:teuthology.orchestra.run.smithi078.stdout: "log_refresh_metadata": { 2024-03-24T03:37:34.564 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_refresh_metadata", 2024-03-24T03:37:34.564 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.564 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.565 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.565 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.565 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.565 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.565 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.565 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-24T03:37:34.565 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.565 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.566 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.566 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:34.566 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:34.566 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.566 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.566 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.566 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.566 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.566 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.567 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.567 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-24T03:37:34.567 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.567 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.567 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.567 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:34.567 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:34.567 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.568 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.568 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.568 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:34.568 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.568 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.568 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.568 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.568 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.569 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.569 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.569 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.569 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.569 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.569 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.569 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.569 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.569 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.570 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:34.570 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:34.570 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.570 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.570 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.570 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.570 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.571 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.571 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.571 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.571 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.571 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.571 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.571 INFO:teuthology.orchestra.run.smithi078.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-24T03:37:34.571 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-24T03:37:34.572 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.572 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.572 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.572 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.572 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.572 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.572 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.572 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-24T03:37:34.572 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.572 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.573 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.573 INFO:teuthology.orchestra.run.smithi078.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-24T03:37:34.573 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-24T03:37:34.573 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.573 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.573 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.573 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "*", 2024-03-24T03:37:34.573 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.574 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.574 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.574 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-24T03:37:34.574 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.574 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.574 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.574 INFO:teuthology.orchestra.run.smithi078.stdout: "max_count_per_host": { 2024-03-24T03:37:34.574 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_count_per_host", 2024-03-24T03:37:34.575 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:34.575 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.575 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.575 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-24T03:37:34.575 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.575 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.575 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.575 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "max number of daemons per service per host", 2024-03-24T03:37:34.575 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.576 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.576 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.576 INFO:teuthology.orchestra.run.smithi078.stdout: "max_osd_draining_count": { 2024-03-24T03:37:34.576 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_osd_draining_count", 2024-03-24T03:37:34.576 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:34.576 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.576 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.576 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-24T03:37:34.576 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.577 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.577 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.577 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-24T03:37:34.577 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.577 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.577 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.577 INFO:teuthology.orchestra.run.smithi078.stdout: "migration_current": { 2024-03-24T03:37:34.577 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "migration_current", 2024-03-24T03:37:34.578 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:34.578 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.578 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.578 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.578 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.578 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.578 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.578 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "internal - do not modify", 2024-03-24T03:37:34.578 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.579 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.579 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.579 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": { 2024-03-24T03:37:34.579 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mode", 2024-03-24T03:37:34.579 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.579 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.579 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.579 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "root", 2024-03-24T03:37:34.579 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.580 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.580 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.580 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm-package", 2024-03-24T03:37:34.580 INFO:teuthology.orchestra.run.smithi078.stdout: "root" 2024-03-24T03:37:34.580 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.580 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "mode for remote execution of cephadm", 2024-03-24T03:37:34.580 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.580 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.580 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.581 INFO:teuthology.orchestra.run.smithi078.stdout: "oob_default_addr": { 2024-03-24T03:37:34.581 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "oob_default_addr", 2024-03-24T03:37:34.581 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.581 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.581 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.581 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "169.254.1.1", 2024-03-24T03:37:34.581 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.582 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.582 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.582 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-24T03:37:34.582 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.582 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.582 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.582 INFO:teuthology.orchestra.run.smithi078.stdout: "prometheus_alerts_path": { 2024-03-24T03:37:34.582 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus_alerts_path", 2024-03-24T03:37:34.583 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.583 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.583 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.583 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-24T03:37:34.583 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.583 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.583 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.583 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-24T03:37:34.583 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.584 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.584 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.584 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_insecure": { 2024-03-24T03:37:34.584 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_insecure", 2024-03-24T03:37:34.584 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.584 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.584 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.584 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.585 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.585 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.585 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.585 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-24T03:37:34.585 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.585 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.585 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.585 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_password": { 2024-03-24T03:37:34.585 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_password", 2024-03-24T03:37:34.586 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.586 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.586 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.586 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.586 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.586 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.586 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.586 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-24T03:37:34.586 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.587 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.587 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.587 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_url": { 2024-03-24T03:37:34.587 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_url", 2024-03-24T03:37:34.587 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.587 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.587 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.587 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.588 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.588 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.588 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.588 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-24T03:37:34.588 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.588 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.588 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.588 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_username": { 2024-03-24T03:37:34.589 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_username", 2024-03-24T03:37:34.589 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.589 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.589 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.589 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.589 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.589 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.589 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.589 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-24T03:37:34.590 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.590 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.590 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.590 INFO:teuthology.orchestra.run.smithi078.stdout: "secure_monitoring_stack": { 2024-03-24T03:37:34.590 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "secure_monitoring_stack", 2024-03-24T03:37:34.590 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.590 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.591 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.591 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.591 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.591 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.591 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.591 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-24T03:37:34.591 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.591 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.591 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.592 INFO:teuthology.orchestra.run.smithi078.stdout: "service_discovery_port": { 2024-03-24T03:37:34.592 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "service_discovery_port", 2024-03-24T03:37:34.592 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:34.592 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.592 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.592 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8765", 2024-03-24T03:37:34.592 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.593 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.593 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.593 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "cephadm service discovery port", 2024-03-24T03:37:34.593 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.593 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.593 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.593 INFO:teuthology.orchestra.run.smithi078.stdout: "ssh_config_file": { 2024-03-24T03:37:34.593 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssh_config_file", 2024-03-24T03:37:34.594 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.594 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.594 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.594 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.594 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.594 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.594 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.594 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-24T03:37:34.595 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.595 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.595 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.595 INFO:teuthology.orchestra.run.smithi078.stdout: "use_agent": { 2024-03-24T03:37:34.595 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "use_agent", 2024-03-24T03:37:34.595 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.595 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.595 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.596 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.596 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.596 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.596 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.596 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-24T03:37:34.596 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.596 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.596 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.597 INFO:teuthology.orchestra.run.smithi078.stdout: "use_repo_digest": { 2024-03-24T03:37:34.597 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "use_repo_digest", 2024-03-24T03:37:34.597 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.597 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.597 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.597 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.597 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.597 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.597 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.598 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-24T03:37:34.598 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.598 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.598 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.598 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_failed_host_check": { 2024-03-24T03:37:34.598 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_failed_host_check", 2024-03-24T03:37:34.598 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.599 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.599 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.599 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.599 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.599 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.599 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.599 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if the host check fails", 2024-03-24T03:37:34.599 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.599 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.599 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.600 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_stray_daemons": { 2024-03-24T03:37:34.600 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_stray_daemons", 2024-03-24T03:37:34.600 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.600 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.600 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.600 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.600 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.600 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.601 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.601 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-24T03:37:34.601 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.601 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.601 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.601 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_stray_hosts": { 2024-03-24T03:37:34.601 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_stray_hosts", 2024-03-24T03:37:34.601 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.602 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.602 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.602 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.602 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.602 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.602 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.602 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-24T03:37:34.602 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.603 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.603 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.603 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.603 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:34.603 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crash", 2024-03-24T03:37:34.603 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:34.603 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:34.604 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:34.604 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:34.604 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:34.604 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.604 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.604 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.604 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.604 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.604 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.604 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.605 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.605 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.605 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.605 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.605 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.605 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.605 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.605 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.605 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.606 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.606 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.606 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:34.606 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:34.606 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.606 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.606 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.606 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.607 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.607 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.607 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.607 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.607 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.607 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.607 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.607 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:34.608 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:34.608 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.608 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.608 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.608 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:34.608 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.608 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.608 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.608 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.609 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.609 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.609 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.609 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.609 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.609 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.609 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.609 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.609 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.609 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.610 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:34.610 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:34.610 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.610 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.610 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.610 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.610 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.611 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.611 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.611 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.611 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.611 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.611 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.611 INFO:teuthology.orchestra.run.smithi078.stdout: "retain_interval": { 2024-03-24T03:37:34.611 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "retain_interval", 2024-03-24T03:37:34.612 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:34.612 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.612 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.612 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "31536000", 2024-03-24T03:37:34.612 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.612 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.612 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.612 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-24T03:37:34.612 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.613 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.613 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.613 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_recent_interval": { 2024-03-24T03:37:34.613 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_recent_interval", 2024-03-24T03:37:34.613 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:34.613 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.613 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.613 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1209600", 2024-03-24T03:37:34.614 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.614 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.614 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.614 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-24T03:37:34.614 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.614 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.614 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.614 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.614 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:34.615 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "dashboard", 2024-03-24T03:37:34.615 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:34.615 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:34.615 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:34.615 INFO:teuthology.orchestra.run.smithi078.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-24T03:37:34.615 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-24T03:37:34.616 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:34.616 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.616 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.616 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-24T03:37:34.616 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.616 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.616 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.616 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.616 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.616 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.617 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.617 INFO:teuthology.orchestra.run.smithi078.stdout: "ALERTMANAGER_API_HOST": { 2024-03-24T03:37:34.617 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-24T03:37:34.617 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.617 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.617 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.617 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.617 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.618 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.618 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.618 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.618 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.618 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.618 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.618 INFO:teuthology.orchestra.run.smithi078.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-24T03:37:34.618 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-24T03:37:34.619 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.619 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.619 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.619 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.619 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.619 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.619 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.619 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.619 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.620 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.620 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.620 INFO:teuthology.orchestra.run.smithi078.stdout: "AUDIT_API_ENABLED": { 2024-03-24T03:37:34.620 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "AUDIT_API_ENABLED", 2024-03-24T03:37:34.620 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.620 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.620 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.620 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.620 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.621 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.621 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.621 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.621 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.621 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.621 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.621 INFO:teuthology.orchestra.run.smithi078.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-24T03:37:34.622 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-24T03:37:34.622 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.622 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.622 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.622 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.622 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.622 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.622 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.623 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.623 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.623 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.625 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.625 INFO:teuthology.orchestra.run.smithi078.stdout: "ENABLE_BROWSABLE_API": { 2024-03-24T03:37:34.625 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-24T03:37:34.625 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.625 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.625 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.626 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.626 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.626 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.626 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.626 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.626 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.626 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.626 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.627 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-24T03:37:34.627 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-24T03:37:34.627 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.627 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.627 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.627 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.627 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.628 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.628 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.628 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.628 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.628 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.628 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.628 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-24T03:37:34.628 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-24T03:37:34.629 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.629 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.629 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.629 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.629 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.629 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.629 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.629 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.630 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.630 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.630 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.630 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-24T03:37:34.630 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-24T03:37:34.630 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.630 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.631 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.631 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.631 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.631 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.631 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.631 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.631 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.631 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.632 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.632 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-24T03:37:34.632 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-24T03:37:34.632 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.632 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.632 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.632 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.632 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.633 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.633 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.633 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.633 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.633 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.633 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.633 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-24T03:37:34.633 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-24T03:37:34.634 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.634 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.634 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.634 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.634 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.634 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.634 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.634 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.635 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.635 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.635 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.635 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-24T03:37:34.635 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-24T03:37:34.635 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.635 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.636 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.636 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.636 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.636 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.636 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.636 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.636 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.636 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.636 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.637 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-24T03:37:34.637 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-24T03:37:34.637 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.637 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.637 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.637 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.637 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.637 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.638 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.638 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.638 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.638 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.638 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.638 INFO:teuthology.orchestra.run.smithi078.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-24T03:37:34.639 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-24T03:37:34.639 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.639 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.639 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.639 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.639 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.639 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.639 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.639 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.640 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.640 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.640 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.640 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_PASSWORD": { 2024-03-24T03:37:34.640 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-24T03:37:34.640 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.640 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.640 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.641 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-03-24T03:37:34.641 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.641 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.641 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.641 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.641 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.641 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.641 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.642 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-24T03:37:34.642 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-24T03:37:34.642 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.642 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.642 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.642 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.642 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.642 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.643 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.643 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.643 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.643 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.643 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.643 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_URL": { 2024-03-24T03:37:34.643 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_URL", 2024-03-24T03:37:34.644 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.644 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.644 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.644 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.644 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.644 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.644 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.644 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.645 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.645 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.645 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.645 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_USERNAME": { 2024-03-24T03:37:34.645 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-24T03:37:34.645 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.645 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.645 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.646 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-03-24T03:37:34.646 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.646 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.646 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.646 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.646 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.646 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.646 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.647 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-24T03:37:34.647 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-24T03:37:34.647 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.647 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.647 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.647 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.647 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.647 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.648 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.648 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.648 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.648 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.648 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.648 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-24T03:37:34.648 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-24T03:37:34.648 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.649 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.649 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.649 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.649 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.649 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.649 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.649 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.649 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.649 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.650 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.650 INFO:teuthology.orchestra.run.smithi078.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-24T03:37:34.650 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-24T03:37:34.650 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.650 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.650 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.650 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.651 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.651 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.651 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.651 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.651 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.651 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.651 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.651 INFO:teuthology.orchestra.run.smithi078.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-24T03:37:34.651 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-24T03:37:34.652 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.652 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.652 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.652 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.652 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.652 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.652 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.652 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.653 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.653 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.653 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.653 INFO:teuthology.orchestra.run.smithi078.stdout: "MULTICLUSTER_CONFIG": { 2024-03-24T03:37:34.653 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-24T03:37:34.653 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.653 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.653 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.654 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "{}", 2024-03-24T03:37:34.654 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.654 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.654 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.654 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.654 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.654 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.654 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.655 INFO:teuthology.orchestra.run.smithi078.stdout: "PROMETHEUS_API_HOST": { 2024-03-24T03:37:34.655 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-24T03:37:34.655 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.655 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.655 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.655 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.655 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.655 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.656 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.656 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.656 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.656 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.656 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.656 INFO:teuthology.orchestra.run.smithi078.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-24T03:37:34.656 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-24T03:37:34.656 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.656 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.657 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.657 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.657 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.657 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.657 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.657 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.657 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.657 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.657 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.658 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-24T03:37:34.658 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-24T03:37:34.658 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.658 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.658 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.658 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.658 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.659 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.659 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.659 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.659 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.659 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.659 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.659 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-24T03:37:34.659 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-24T03:37:34.660 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.660 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.660 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.660 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.660 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.660 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.660 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.660 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.660 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.661 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.661 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.661 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-24T03:37:34.661 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-24T03:37:34.661 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.661 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.661 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.661 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.661 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.662 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.662 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.662 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.662 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.662 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.662 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.662 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-24T03:37:34.663 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-24T03:37:34.663 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.663 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.663 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.663 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.663 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.663 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.663 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.663 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.663 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.664 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.664 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.664 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-24T03:37:34.664 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-24T03:37:34.664 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.664 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.664 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.665 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.665 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.665 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.665 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.665 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.665 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.665 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.665 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.666 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-24T03:37:34.666 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-24T03:37:34.666 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.666 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.666 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.666 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.666 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.666 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.666 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.666 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.667 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.667 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.667 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.667 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-24T03:37:34.667 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-24T03:37:34.667 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.667 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.667 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.668 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.668 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.668 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.668 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.668 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.668 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.668 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.668 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.669 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_ENABLED": { 2024-03-24T03:37:34.669 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-24T03:37:34.669 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.669 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.669 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.669 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.669 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.669 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.669 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.670 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.670 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.670 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.670 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.670 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-24T03:37:34.670 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-24T03:37:34.670 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.671 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.671 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.671 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-24T03:37:34.671 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.671 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.671 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.671 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.671 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.671 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.671 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.672 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-24T03:37:34.672 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-24T03:37:34.672 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:34.672 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.672 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.672 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-24T03:37:34.672 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.672 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.673 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.673 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.673 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.673 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.673 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.673 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-24T03:37:34.673 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-24T03:37:34.673 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:34.674 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.674 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.674 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8", 2024-03-24T03:37:34.674 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.674 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.674 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.674 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.674 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.674 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.675 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.675 INFO:teuthology.orchestra.run.smithi078.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-24T03:37:34.675 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-24T03:37:34.675 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:34.675 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.675 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.675 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "45", 2024-03-24T03:37:34.675 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.676 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.676 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.676 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.676 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.676 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.676 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.676 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_ACCESS_KEY": { 2024-03-24T03:37:34.676 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-24T03:37:34.677 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.677 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.677 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.677 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.677 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.677 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.677 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.677 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.677 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.677 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.678 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.678 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-24T03:37:34.678 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-24T03:37:34.678 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.678 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.678 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.678 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-03-24T03:37:34.679 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.679 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.679 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.679 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.679 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.679 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.679 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.679 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_SECRET_KEY": { 2024-03-24T03:37:34.679 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-24T03:37:34.680 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.680 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.680 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.680 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.680 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.680 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.680 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.680 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.680 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.680 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.681 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.681 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_SSL_VERIFY": { 2024-03-24T03:37:34.681 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-24T03:37:34.681 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.681 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.681 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.681 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.681 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.682 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.682 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.682 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.682 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.682 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.682 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.682 INFO:teuthology.orchestra.run.smithi078.stdout: "UNSAFE_TLS_v1_2": { 2024-03-24T03:37:34.682 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-24T03:37:34.683 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.683 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.683 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.683 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.683 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.683 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.683 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.683 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.683 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.684 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.684 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.684 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-24T03:37:34.684 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-24T03:37:34.684 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:34.684 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.684 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.684 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-24T03:37:34.685 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.685 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.685 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.685 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.685 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.685 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.685 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.685 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-24T03:37:34.686 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-24T03:37:34.686 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:34.686 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.686 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.686 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-24T03:37:34.686 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.686 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.686 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.686 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.687 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.687 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.687 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.687 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-24T03:37:34.687 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-24T03:37:34.687 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:34.687 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.687 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.688 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-24T03:37:34.688 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.688 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.688 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.688 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.688 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.688 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.688 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.688 INFO:teuthology.orchestra.run.smithi078.stdout: "cross_origin_url": { 2024-03-24T03:37:34.689 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cross_origin_url", 2024-03-24T03:37:34.689 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.689 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.689 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.689 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.689 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.689 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.689 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.689 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.690 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.690 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.690 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.690 INFO:teuthology.orchestra.run.smithi078.stdout: "crt_file": { 2024-03-24T03:37:34.690 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crt_file", 2024-03-24T03:37:34.690 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.690 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.690 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.691 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.691 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.691 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.691 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.691 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.691 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.691 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.691 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.692 INFO:teuthology.orchestra.run.smithi078.stdout: "debug": { 2024-03-24T03:37:34.692 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "debug", 2024-03-24T03:37:34.692 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.692 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.692 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.692 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.692 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.692 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.693 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.693 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable/disable debug options", 2024-03-24T03:37:34.693 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.693 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.693 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.693 INFO:teuthology.orchestra.run.smithi078.stdout: "jwt_token_ttl": { 2024-03-24T03:37:34.693 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "jwt_token_ttl", 2024-03-24T03:37:34.693 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:34.694 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.694 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.694 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "28800", 2024-03-24T03:37:34.694 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.694 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.694 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.694 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.694 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.694 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.695 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.695 INFO:teuthology.orchestra.run.smithi078.stdout: "key_file": { 2024-03-24T03:37:34.695 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "key_file", 2024-03-24T03:37:34.695 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.695 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.695 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.695 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.695 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.696 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.696 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.696 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.696 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.696 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.696 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.696 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:34.696 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:34.697 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.697 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.697 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.697 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.697 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.697 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.697 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.697 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.697 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.698 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.698 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.698 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.698 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.698 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.698 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.698 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.698 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.698 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.699 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:34.699 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:34.699 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.699 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.699 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.699 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.699 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.700 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.700 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.700 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.700 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.700 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.700 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.700 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:34.700 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:34.701 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.701 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.701 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.701 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:34.701 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.701 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.701 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.701 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.701 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.702 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.702 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.702 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.702 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.702 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.702 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.702 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.702 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.702 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.703 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:34.703 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:34.703 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.703 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.703 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.703 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.703 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.703 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.704 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.704 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.704 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.704 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.704 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.704 INFO:teuthology.orchestra.run.smithi078.stdout: "motd": { 2024-03-24T03:37:34.704 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "motd", 2024-03-24T03:37:34.705 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.705 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.705 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.705 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.705 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.705 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.705 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.705 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "The message of the day", 2024-03-24T03:37:34.705 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.705 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.706 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.706 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_resolve_ip_addr": { 2024-03-24T03:37:34.706 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "redirect_resolve_ip_addr", 2024-03-24T03:37:34.706 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.706 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.706 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.706 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.707 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.707 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.707 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.707 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.707 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.707 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.707 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.707 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-03-24T03:37:34.708 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-03-24T03:37:34.708 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.708 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.708 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.708 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "::", 2024-03-24T03:37:34.708 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.708 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.708 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.708 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.709 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.709 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.709 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.709 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-03-24T03:37:34.709 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-03-24T03:37:34.709 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:34.709 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.709 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.710 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8080", 2024-03-24T03:37:34.710 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.710 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.710 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.710 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.710 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.710 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.710 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.711 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl": { 2024-03-24T03:37:34.711 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl", 2024-03-24T03:37:34.711 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.711 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.711 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.711 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.711 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.712 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.712 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.712 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.712 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.712 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.712 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.712 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl_server_port": { 2024-03-24T03:37:34.712 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl_server_port", 2024-03-24T03:37:34.712 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:34.713 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.713 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.713 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8443", 2024-03-24T03:37:34.713 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.713 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.713 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.713 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.713 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.713 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.714 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.714 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_behaviour": { 2024-03-24T03:37:34.714 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_behaviour", 2024-03-24T03:37:34.714 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.714 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.714 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.715 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "redirect", 2024-03-24T03:37:34.715 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.715 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.715 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.715 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.715 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect" 2024-03-24T03:37:34.715 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.715 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.716 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.716 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.716 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.716 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_error_status_code": { 2024-03-24T03:37:34.716 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_error_status_code", 2024-03-24T03:37:34.716 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:34.716 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.716 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.717 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "500", 2024-03-24T03:37:34.717 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "400", 2024-03-24T03:37:34.717 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "599", 2024-03-24T03:37:34.717 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.717 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.717 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.717 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.717 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.718 INFO:teuthology.orchestra.run.smithi078.stdout: "url_prefix": { 2024-03-24T03:37:34.718 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "url_prefix", 2024-03-24T03:37:34.718 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.718 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.718 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.718 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.718 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.718 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.719 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.719 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.719 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.719 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.719 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.719 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.719 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:34.720 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "devicehealth", 2024-03-24T03:37:34.720 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:34.720 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:34.720 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:34.720 INFO:teuthology.orchestra.run.smithi078.stdout: "enable_monitoring": { 2024-03-24T03:37:34.720 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enable_monitoring", 2024-03-24T03:37:34.720 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.720 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.720 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.721 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.721 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.721 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.721 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.721 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "monitor device health metrics", 2024-03-24T03:37:34.721 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.721 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.721 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.722 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:34.722 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:34.722 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.722 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.722 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.722 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.722 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.722 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.723 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.723 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.723 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.723 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.723 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.723 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.723 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.723 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.723 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.724 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.724 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.724 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.724 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:34.724 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:34.724 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.724 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.724 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.725 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.725 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.725 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.725 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.725 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.725 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.725 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.725 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.726 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:34.726 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:34.726 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.726 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.726 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.726 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:34.726 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.726 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.727 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.727 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.727 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.727 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.727 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.727 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.727 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.727 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.727 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.728 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.728 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.728 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.728 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:34.728 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:34.728 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.728 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.728 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.728 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.729 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.729 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.729 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.729 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.729 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.729 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.729 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.729 INFO:teuthology.orchestra.run.smithi078.stdout: "mark_out_threshold": { 2024-03-24T03:37:34.729 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mark_out_threshold", 2024-03-24T03:37:34.730 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:34.730 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.730 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.730 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "2419200", 2024-03-24T03:37:34.730 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.730 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.730 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.730 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-24T03:37:34.731 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.731 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.731 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.731 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": { 2024-03-24T03:37:34.731 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pool_name", 2024-03-24T03:37:34.731 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.731 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.731 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.732 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "device_health_metrics", 2024-03-24T03:37:34.732 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.732 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.732 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.732 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-24T03:37:34.732 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.732 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.732 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.732 INFO:teuthology.orchestra.run.smithi078.stdout: "retention_period": { 2024-03-24T03:37:34.733 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "retention_period", 2024-03-24T03:37:34.733 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:34.733 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.733 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.733 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15552000", 2024-03-24T03:37:34.733 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.733 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.733 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.733 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long to retain device health metrics", 2024-03-24T03:37:34.734 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.734 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.734 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.734 INFO:teuthology.orchestra.run.smithi078.stdout: "scrape_frequency": { 2024-03-24T03:37:34.734 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "scrape_frequency", 2024-03-24T03:37:34.734 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:34.734 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.735 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.735 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "86400", 2024-03-24T03:37:34.735 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.735 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.735 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.735 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-24T03:37:34.735 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.735 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.736 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.736 INFO:teuthology.orchestra.run.smithi078.stdout: "self_heal": { 2024-03-24T03:37:34.736 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "self_heal", 2024-03-24T03:37:34.736 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.736 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.736 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.736 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.736 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.736 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.737 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.737 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-24T03:37:34.737 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.737 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.737 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.737 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-24T03:37:34.737 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-24T03:37:34.737 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:34.738 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.738 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.738 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-24T03:37:34.738 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.738 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.738 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.738 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to wake up and check device health", 2024-03-24T03:37:34.738 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.738 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.739 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.739 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_threshold": { 2024-03-24T03:37:34.739 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_threshold", 2024-03-24T03:37:34.739 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:34.739 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.739 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.739 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "7257600", 2024-03-24T03:37:34.739 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.740 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.740 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.740 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-24T03:37:34.740 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.740 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.740 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.740 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.740 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:34.741 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "diskprediction_local", 2024-03-24T03:37:34.741 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:34.741 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:34.741 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:34.741 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:34.741 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:34.741 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.741 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.741 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.742 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.742 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.742 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.742 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.742 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.742 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.742 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.742 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.742 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.743 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.743 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.743 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.743 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.743 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.743 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.743 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:34.743 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:34.744 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.744 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.744 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.744 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.744 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.744 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.744 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.744 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.744 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.745 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.745 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.745 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:34.745 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:34.745 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.745 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.745 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.745 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:34.746 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.746 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.746 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.746 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.746 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.746 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.746 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.746 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.746 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.747 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.747 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.747 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.747 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.747 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.747 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:34.747 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:34.747 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.748 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.748 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.748 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.748 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.748 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.748 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.748 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.748 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.748 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.748 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.749 INFO:teuthology.orchestra.run.smithi078.stdout: "predict_interval": { 2024-03-24T03:37:34.749 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "predict_interval", 2024-03-24T03:37:34.749 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.749 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.749 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.749 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "86400", 2024-03-24T03:37:34.749 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.749 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.750 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.750 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.750 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.750 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.750 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.750 INFO:teuthology.orchestra.run.smithi078.stdout: "predictor_model": { 2024-03-24T03:37:34.750 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "predictor_model", 2024-03-24T03:37:34.750 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.751 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.751 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.751 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "prophetstor", 2024-03-24T03:37:34.751 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.751 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.751 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.751 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.751 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.752 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.752 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.752 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-24T03:37:34.752 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-24T03:37:34.752 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.752 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.752 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.752 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-24T03:37:34.753 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.753 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.753 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.753 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.753 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.753 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.753 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.753 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.753 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.754 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:34.754 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "influx", 2024-03-24T03:37:34.754 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": false, 2024-03-24T03:37:34.754 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "influxdb python module not found", 2024-03-24T03:37:34.754 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:34.754 INFO:teuthology.orchestra.run.smithi078.stdout: "batch_size": { 2024-03-24T03:37:34.755 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "batch_size", 2024-03-24T03:37:34.755 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:34.755 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.755 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.755 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5000", 2024-03-24T03:37:34.755 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.755 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.755 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.755 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-24T03:37:34.756 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.756 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.756 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.756 INFO:teuthology.orchestra.run.smithi078.stdout: "database": { 2024-03-24T03:37:34.756 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "database", 2024-03-24T03:37:34.756 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.756 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.757 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.757 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "ceph", 2024-03-24T03:37:34.757 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.757 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.757 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.757 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-24T03:37:34.757 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.757 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.757 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.758 INFO:teuthology.orchestra.run.smithi078.stdout: "hostname": { 2024-03-24T03:37:34.758 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "hostname", 2024-03-24T03:37:34.758 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.758 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.758 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.758 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.758 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.758 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.759 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.759 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB server hostname", 2024-03-24T03:37:34.759 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.759 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.759 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.759 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-24T03:37:34.759 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-24T03:37:34.759 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:34.760 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.760 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.760 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "30", 2024-03-24T03:37:34.760 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "5", 2024-03-24T03:37:34.760 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.760 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.760 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-24T03:37:34.760 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.760 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.761 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.761 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:34.761 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:34.761 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.761 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.761 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.761 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.761 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.762 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.762 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.762 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.762 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.762 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.762 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.762 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.762 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.762 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.763 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.763 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.763 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.763 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.763 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:34.763 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:34.763 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.763 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.764 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.764 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.764 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.764 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.764 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.764 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.764 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.764 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.764 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.765 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:34.765 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:34.765 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.765 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.765 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.765 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:34.765 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.765 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.766 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.766 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.766 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.766 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.766 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.766 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.766 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.766 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.767 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.767 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.767 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.767 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.767 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:34.767 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:34.767 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.767 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.768 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.768 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.768 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.768 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.768 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.768 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.768 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.768 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.768 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.769 INFO:teuthology.orchestra.run.smithi078.stdout: "password": { 2024-03-24T03:37:34.769 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "password", 2024-03-24T03:37:34.769 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.769 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.769 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.769 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.769 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.769 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.769 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.770 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "password of InfluxDB server user", 2024-03-24T03:37:34.770 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.770 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.770 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.770 INFO:teuthology.orchestra.run.smithi078.stdout: "port": { 2024-03-24T03:37:34.770 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "port", 2024-03-24T03:37:34.770 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:34.770 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.771 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.771 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8086", 2024-03-24T03:37:34.771 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.771 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.771 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.771 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB server port", 2024-03-24T03:37:34.771 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.771 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.771 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.772 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl": { 2024-03-24T03:37:34.772 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl", 2024-03-24T03:37:34.772 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.772 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.772 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.772 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "false", 2024-03-24T03:37:34.772 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.772 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.772 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.773 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T03:37:34.773 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.773 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.773 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.773 INFO:teuthology.orchestra.run.smithi078.stdout: "threads": { 2024-03-24T03:37:34.773 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "threads", 2024-03-24T03:37:34.773 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:34.773 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.774 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.774 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-24T03:37:34.774 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-24T03:37:34.774 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "32", 2024-03-24T03:37:34.774 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.774 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-24T03:37:34.774 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.774 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.774 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.775 INFO:teuthology.orchestra.run.smithi078.stdout: "username": { 2024-03-24T03:37:34.775 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "username", 2024-03-24T03:37:34.775 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.775 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.775 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.775 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.775 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.776 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.776 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.776 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "username of InfluxDB server user", 2024-03-24T03:37:34.776 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.776 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.776 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.776 INFO:teuthology.orchestra.run.smithi078.stdout: "verify_ssl": { 2024-03-24T03:37:34.776 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "verify_ssl", 2024-03-24T03:37:34.776 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.777 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.777 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.777 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "true", 2024-03-24T03:37:34.777 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.777 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.777 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.777 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T03:37:34.777 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.777 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.778 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.778 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.778 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:34.778 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "insights", 2024-03-24T03:37:34.778 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:34.778 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:34.778 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:34.779 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:34.779 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:34.779 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.779 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.779 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.779 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.779 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.779 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.780 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.780 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.780 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.780 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.780 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.780 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.780 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.780 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.780 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.781 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.781 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.781 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.781 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:34.781 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:34.781 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.781 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.781 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.782 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.782 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.782 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.782 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.782 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.782 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.782 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.782 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.783 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:34.783 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:34.783 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.783 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.783 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.783 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:34.783 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.783 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.784 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.784 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.784 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.784 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.784 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.784 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.784 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.784 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.785 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.785 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.785 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.785 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.785 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:34.785 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:34.785 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.785 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.786 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.786 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.786 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.786 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.786 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.786 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.786 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.786 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.786 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.787 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.787 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:34.787 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "iostat", 2024-03-24T03:37:34.787 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:34.787 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:34.787 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:34.788 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:34.788 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:34.788 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.788 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.788 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.788 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.788 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.788 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.788 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.789 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.789 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.789 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.789 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.789 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.789 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.789 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.789 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.790 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.790 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.790 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.790 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:34.790 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:34.790 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.791 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.791 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.791 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.791 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.791 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.791 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.791 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.791 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.791 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.792 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.792 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:34.792 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:34.792 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.792 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.792 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.792 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:34.793 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.793 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.793 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.793 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.793 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.793 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.793 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.793 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.793 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.794 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.794 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.794 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.794 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.794 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.794 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:34.794 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:34.795 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.795 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.795 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.795 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.795 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.795 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.795 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.795 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.796 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.796 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.796 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.796 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.796 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:34.796 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "k8sevents", 2024-03-24T03:37:34.797 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:34.797 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:34.797 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:34.797 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_event_retention_days": { 2024-03-24T03:37:34.797 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ceph_event_retention_days", 2024-03-24T03:37:34.797 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:34.797 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.797 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.797 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "7", 2024-03-24T03:37:34.798 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.798 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.798 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.798 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-24T03:37:34.798 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.798 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.798 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.799 INFO:teuthology.orchestra.run.smithi078.stdout: "config_check_secs": { 2024-03-24T03:37:34.799 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_check_secs", 2024-03-24T03:37:34.799 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:34.799 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.799 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.799 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-24T03:37:34.799 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "10", 2024-03-24T03:37:34.799 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.800 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.800 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-24T03:37:34.800 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.800 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.800 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.800 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:34.800 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:34.801 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.801 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.801 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.801 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.801 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.801 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.801 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.801 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.802 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.802 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.802 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.802 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.802 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.802 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.802 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.802 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.803 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.803 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.803 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:34.803 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:34.803 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.803 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.803 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.804 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.804 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.804 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.804 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.804 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.804 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.804 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.805 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.805 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:34.805 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:34.805 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.805 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.805 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.805 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:34.805 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.806 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.806 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.806 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.806 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.806 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.806 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.806 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.806 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.807 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.807 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.807 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.807 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.807 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.807 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:34.807 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:34.807 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.808 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.808 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.808 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.808 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.808 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.808 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.808 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.808 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.808 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.809 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.809 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.809 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:34.809 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "localpool", 2024-03-24T03:37:34.809 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:34.809 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:34.810 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:34.810 INFO:teuthology.orchestra.run.smithi078.stdout: "failure_domain": { 2024-03-24T03:37:34.810 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "failure_domain", 2024-03-24T03:37:34.810 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.810 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.810 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.810 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "host", 2024-03-24T03:37:34.810 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.811 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.811 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.811 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "failure domain for any created local pool", 2024-03-24T03:37:34.811 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-24T03:37:34.811 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.811 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.811 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:34.811 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:34.812 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.812 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.812 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.812 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.812 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.812 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.812 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.812 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.812 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.813 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.813 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.813 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.813 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.813 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.813 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.813 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.813 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.814 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.814 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:34.814 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:34.814 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.814 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.814 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.814 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.814 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.815 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.815 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.815 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.815 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.815 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.815 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.815 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:34.816 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:34.816 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.816 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.816 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.816 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:34.816 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.816 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.816 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.816 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.817 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.817 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.817 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.817 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.817 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.817 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.817 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.817 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.817 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.818 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.818 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:34.818 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:34.818 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.818 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.818 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.818 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.819 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.819 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.819 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.819 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.819 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.819 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.819 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.819 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": { 2024-03-24T03:37:34.820 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "min_size", 2024-03-24T03:37:34.820 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:34.820 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.820 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.820 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.820 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.820 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.820 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.820 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default min_size for any created local pool", 2024-03-24T03:37:34.821 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-24T03:37:34.821 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.821 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.821 INFO:teuthology.orchestra.run.smithi078.stdout: "num_rep": { 2024-03-24T03:37:34.821 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "num_rep", 2024-03-24T03:37:34.821 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:34.821 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.821 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.822 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3", 2024-03-24T03:37:34.822 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.822 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.822 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.822 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default replica count for any created local pool", 2024-03-24T03:37:34.822 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.822 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.822 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.823 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": { 2024-03-24T03:37:34.823 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pg_num", 2024-03-24T03:37:34.823 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:34.823 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.823 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.823 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "128", 2024-03-24T03:37:34.823 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.823 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.824 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.824 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default pg_num for any created local pool", 2024-03-24T03:37:34.824 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.824 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.824 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.824 INFO:teuthology.orchestra.run.smithi078.stdout: "prefix": { 2024-03-24T03:37:34.824 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prefix", 2024-03-24T03:37:34.824 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.825 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.825 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.825 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.825 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.825 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.825 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.825 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "name prefix for any created local pool", 2024-03-24T03:37:34.825 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.826 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.826 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.826 INFO:teuthology.orchestra.run.smithi078.stdout: "subtree": { 2024-03-24T03:37:34.826 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "subtree", 2024-03-24T03:37:34.826 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.826 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.826 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.826 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "rack", 2024-03-24T03:37:34.827 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.827 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.827 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.827 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-24T03:37:34.827 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-24T03:37:34.827 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.827 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.827 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.828 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:34.828 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mds_autoscaler", 2024-03-24T03:37:34.828 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:34.828 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:34.828 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:34.828 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:34.828 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:34.829 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.829 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.829 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.829 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.829 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.829 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.829 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.829 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.829 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.830 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.830 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.830 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.830 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.830 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.830 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.830 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.830 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.831 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.831 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:34.831 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:34.831 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.831 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.831 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.831 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.831 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.832 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.832 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.832 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.832 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.832 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.832 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.832 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:34.832 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:34.833 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.833 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.833 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.833 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:34.833 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.833 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.833 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.833 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.834 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.834 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.834 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.834 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.834 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.834 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.834 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.834 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.835 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.835 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.835 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:34.835 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:34.835 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.835 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.835 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.835 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.836 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.836 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.836 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.836 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.836 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.836 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.836 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.836 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.836 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:34.837 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mirroring", 2024-03-24T03:37:34.837 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:34.837 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:34.837 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:34.837 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:34.837 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:34.837 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.838 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.838 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.838 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.838 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.838 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.838 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.838 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.838 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.839 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.839 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.839 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.839 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.839 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.839 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.839 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.839 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.839 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.840 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:34.840 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:34.840 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.840 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.840 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.840 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.840 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.840 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.840 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.841 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.841 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.841 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.841 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.841 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:34.841 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:34.841 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.841 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.842 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.842 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:34.842 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.842 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.842 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.842 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.842 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.842 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.843 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.843 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.843 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.843 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.843 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.843 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.843 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.843 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.844 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:34.844 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:34.844 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.844 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.844 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.844 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.844 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.844 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.844 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.845 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.845 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.845 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.845 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.845 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.845 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:34.846 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "nfs", 2024-03-24T03:37:34.846 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:34.846 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:34.846 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:34.846 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:34.846 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:34.846 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.846 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.847 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.847 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.847 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.847 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.847 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.847 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.847 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.847 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.847 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.848 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.848 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.848 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.848 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.848 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.848 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.848 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.848 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:34.848 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:34.849 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.849 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.849 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.849 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.849 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.849 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.849 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.849 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.850 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.850 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.850 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.850 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:34.850 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:34.850 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.850 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.851 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.851 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:34.851 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.851 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.851 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.851 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.851 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.851 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.852 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.852 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.852 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.852 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.852 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.852 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.852 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.852 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.853 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:34.853 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:34.853 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.853 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.853 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.853 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.853 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.854 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.854 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.854 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.854 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.854 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.854 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.854 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.854 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:34.855 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "orchestrator", 2024-03-24T03:37:34.855 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:34.855 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:34.855 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:34.855 INFO:teuthology.orchestra.run.smithi078.stdout: "fail_fs": { 2024-03-24T03:37:34.855 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "fail_fs", 2024-03-24T03:37:34.856 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.856 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.856 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.856 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.856 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.856 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.856 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.856 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-24T03:37:34.857 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.857 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.857 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.857 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:34.857 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:34.857 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.857 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.857 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.858 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.858 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.858 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.858 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.858 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.858 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.858 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.859 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.859 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.859 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.859 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.859 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.859 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.859 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.859 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.860 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:34.860 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:34.860 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.860 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.860 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.860 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.860 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.860 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.861 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.861 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.861 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.861 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.861 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.861 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:34.862 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:34.862 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.862 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.862 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.862 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:34.862 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.862 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.862 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.863 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.863 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.863 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.863 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.863 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.863 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.863 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.863 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.863 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.864 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.864 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.864 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:34.864 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:34.864 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.864 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.864 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.865 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.865 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.865 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.865 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.865 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.865 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.865 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.865 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.866 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator": { 2024-03-24T03:37:34.866 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "orchestrator", 2024-03-24T03:37:34.866 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.866 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.866 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.866 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.866 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.867 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.867 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.867 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm", 2024-03-24T03:37:34.867 INFO:teuthology.orchestra.run.smithi078.stdout: "rook", 2024-03-24T03:37:34.867 INFO:teuthology.orchestra.run.smithi078.stdout: "test_orchestrator" 2024-03-24T03:37:34.867 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.867 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Orchestrator backend", 2024-03-24T03:37:34.867 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.868 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.868 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.868 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.868 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:34.868 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "osd_perf_query", 2024-03-24T03:37:34.868 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:34.869 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:34.869 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:34.869 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:34.869 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:34.869 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.869 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.869 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.870 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.870 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.870 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.870 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.870 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.870 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.870 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.870 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.871 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.871 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.871 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.871 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.871 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.871 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.871 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.872 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:34.872 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:34.872 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.872 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.872 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.872 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.872 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.873 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.873 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.873 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.873 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.873 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.873 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.873 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:34.874 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:34.874 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.874 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.874 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.874 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:34.874 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.874 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.874 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.875 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.875 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.875 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.875 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.875 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.875 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.875 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.875 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.876 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.876 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.876 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.876 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:34.876 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:34.876 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.876 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.877 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.877 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.877 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.877 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.877 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.877 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.877 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.877 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.878 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.878 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.878 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:34.878 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "osd_support", 2024-03-24T03:37:34.878 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:34.878 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:34.879 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:34.879 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:34.879 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:34.879 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.879 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.879 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.879 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.879 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.880 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.880 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.880 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.880 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.880 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.880 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.880 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.881 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.881 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.881 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.881 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.881 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.881 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.881 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:34.882 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:34.882 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.882 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.882 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.882 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.882 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.882 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.882 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.883 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.883 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.883 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.883 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.883 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:34.883 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:34.883 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.884 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.884 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.884 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:34.884 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.884 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.884 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.884 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.884 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.885 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.885 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.885 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.885 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.885 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.885 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.885 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.885 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.886 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.886 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:34.886 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:34.886 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.886 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.886 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.886 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.887 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.887 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.887 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.887 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.887 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.887 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.887 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.887 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.887 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:34.888 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pg_autoscaler", 2024-03-24T03:37:34.888 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:34.888 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:34.888 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:34.888 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:34.888 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:34.889 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.889 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.889 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.889 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.889 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.889 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.889 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.889 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.890 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.890 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.890 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.890 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.890 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.890 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.890 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.891 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.891 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.891 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.891 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:34.891 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:34.891 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.891 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.892 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.892 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.892 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.892 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.892 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.892 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.892 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.892 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.892 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.893 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:34.893 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:34.893 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.893 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.893 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.893 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:34.893 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.894 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.894 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.894 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.894 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.894 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.894 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.894 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.894 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.894 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.895 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.895 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.895 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.895 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.895 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:34.895 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:34.895 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.895 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.896 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.896 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.896 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.896 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.896 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.896 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.896 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.896 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.896 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.897 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-24T03:37:34.897 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-24T03:37:34.897 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:34.897 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.897 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.897 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-24T03:37:34.897 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.897 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.898 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.898 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.898 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.898 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.898 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.898 INFO:teuthology.orchestra.run.smithi078.stdout: "threshold": { 2024-03-24T03:37:34.898 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "threshold", 2024-03-24T03:37:34.898 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-24T03:37:34.899 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.899 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.899 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3.0", 2024-03-24T03:37:34.899 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1.0", 2024-03-24T03:37:34.899 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.899 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.899 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "scaling threshold", 2024-03-24T03:37:34.899 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-24T03:37:34.900 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.900 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.900 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.900 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:34.900 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "progress", 2024-03-24T03:37:34.900 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:34.900 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:34.901 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:34.901 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_pg_recovery_event": { 2024-03-24T03:37:34.901 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_pg_recovery_event", 2024-03-24T03:37:34.901 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.901 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.901 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.901 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.901 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.901 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.902 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.902 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-24T03:37:34.902 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.902 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.902 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.902 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": { 2024-03-24T03:37:34.902 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enabled", 2024-03-24T03:37:34.902 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.903 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.903 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.903 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.903 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.903 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.903 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.903 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.903 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.903 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.904 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.904 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:34.904 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:34.904 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.904 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.904 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.904 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.904 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.905 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.905 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.905 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.905 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.905 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.905 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.905 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.905 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.906 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.906 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.906 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.906 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.906 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.906 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:34.906 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:34.906 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.906 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.907 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.907 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.907 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.907 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.907 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.907 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.907 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.907 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.907 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.908 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:34.908 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:34.908 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.908 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.908 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.908 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:34.908 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.908 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.909 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.909 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.909 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.909 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.909 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.909 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.909 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.909 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.909 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.909 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.910 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.910 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.910 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:34.910 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:34.910 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.910 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.910 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.910 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.911 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.911 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.911 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.911 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.911 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.911 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.911 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.912 INFO:teuthology.orchestra.run.smithi078.stdout: "max_completed_events": { 2024-03-24T03:37:34.912 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_completed_events", 2024-03-24T03:37:34.912 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:34.912 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.912 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.912 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "50", 2024-03-24T03:37:34.912 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.912 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.912 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.913 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "number of past completed events to remember", 2024-03-24T03:37:34.913 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.913 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.913 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.913 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-24T03:37:34.913 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-24T03:37:34.913 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:34.913 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.914 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.914 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-24T03:37:34.914 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.914 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.914 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.914 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long the module is going to sleep", 2024-03-24T03:37:34.914 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.914 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.915 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.915 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.917 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:34.918 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus", 2024-03-24T03:37:34.918 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:34.918 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:34.918 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:34.918 INFO:teuthology.orchestra.run.smithi078.stdout: "cache": { 2024-03-24T03:37:34.918 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cache", 2024-03-24T03:37:34.919 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.919 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.919 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.919 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.919 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.919 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.919 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.919 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.920 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.920 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.920 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.920 INFO:teuthology.orchestra.run.smithi078.stdout: "exclude_perf_counters": { 2024-03-24T03:37:34.920 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "exclude_perf_counters", 2024-03-24T03:37:34.920 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.920 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.921 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.921 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.921 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.921 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.921 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.921 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-24T03:37:34.921 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-24T03:37:34.921 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.921 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.922 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:34.922 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:34.922 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.922 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.922 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.922 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.923 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.923 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.923 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.923 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.923 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.923 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.923 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.923 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.923 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.924 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.924 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.924 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.924 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.924 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.924 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:34.924 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:34.924 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.925 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.925 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.925 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.925 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.925 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.925 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.925 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.926 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.926 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.926 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.926 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:34.926 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:34.926 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.926 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.927 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.927 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:34.927 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.927 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.927 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.927 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.927 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.927 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.927 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.928 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.928 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.928 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.928 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.928 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.928 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.928 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.928 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:34.929 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:34.929 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.929 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.929 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.929 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.929 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.929 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.929 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.929 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.930 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.930 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.930 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.930 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_stats_pools": { 2024-03-24T03:37:34.930 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_stats_pools", 2024-03-24T03:37:34.930 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.930 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.931 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.931 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.931 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.931 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.931 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.931 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.931 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.931 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.931 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.932 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-24T03:37:34.932 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-24T03:37:34.932 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:34.932 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.932 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.932 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "300", 2024-03-24T03:37:34.932 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.932 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.933 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.933 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.933 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.933 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.933 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.933 INFO:teuthology.orchestra.run.smithi078.stdout: "scrape_interval": { 2024-03-24T03:37:34.933 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "scrape_interval", 2024-03-24T03:37:34.934 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-24T03:37:34.934 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.934 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.934 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15.0", 2024-03-24T03:37:34.934 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.934 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.934 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.934 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.935 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.935 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.935 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.935 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-03-24T03:37:34.935 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-03-24T03:37:34.935 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.935 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.935 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.936 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "::", 2024-03-24T03:37:34.936 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.936 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.936 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.936 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-24T03:37:34.936 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.936 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.936 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.937 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-03-24T03:37:34.937 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-03-24T03:37:34.937 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:34.937 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.937 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.937 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "9283", 2024-03-24T03:37:34.937 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.938 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.938 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.938 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-24T03:37:34.938 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.938 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.938 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.938 INFO:teuthology.orchestra.run.smithi078.stdout: "stale_cache_strategy": { 2024-03-24T03:37:34.938 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "stale_cache_strategy", 2024-03-24T03:37:34.939 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.939 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.939 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.939 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "log", 2024-03-24T03:37:34.939 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.939 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.939 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.939 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.940 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.940 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.940 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.940 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_behaviour": { 2024-03-24T03:37:34.940 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_behaviour", 2024-03-24T03:37:34.940 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.940 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.941 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.941 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "default", 2024-03-24T03:37:34.941 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.941 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.941 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.941 INFO:teuthology.orchestra.run.smithi078.stdout: "default", 2024-03-24T03:37:34.941 INFO:teuthology.orchestra.run.smithi078.stdout: "error" 2024-03-24T03:37:34.941 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.941 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.942 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.942 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.942 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.942 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_error_status_code": { 2024-03-24T03:37:34.942 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_error_status_code", 2024-03-24T03:37:34.942 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:34.942 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.943 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.943 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "500", 2024-03-24T03:37:34.943 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "400", 2024-03-24T03:37:34.943 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "599", 2024-03-24T03:37:34.943 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.943 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.943 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.943 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.943 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.944 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.944 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:34.944 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_support", 2024-03-24T03:37:34.944 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:34.944 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:34.944 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:34.945 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:34.945 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:34.945 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.945 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.945 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.945 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.945 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.945 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.945 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.946 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.946 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.946 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.946 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.946 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.946 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.946 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.946 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.947 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.947 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.947 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.947 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:34.947 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:34.947 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.947 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.947 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.948 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.948 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.948 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.948 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.948 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.948 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.948 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.948 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.949 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:34.949 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:34.949 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.949 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.949 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.949 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:34.949 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.949 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.950 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.950 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.950 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.950 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.950 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.950 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.950 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.950 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.951 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.951 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.951 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.951 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.951 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:34.951 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:34.951 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.952 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.952 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.952 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.952 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.952 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.952 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.952 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.952 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.952 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.953 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.953 INFO:teuthology.orchestra.run.smithi078.stdout: "max_concurrent_snap_create": { 2024-03-24T03:37:34.953 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_concurrent_snap_create", 2024-03-24T03:37:34.953 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:34.953 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.953 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.953 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-24T03:37:34.953 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.954 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.954 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.954 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.954 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.954 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.954 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.954 INFO:teuthology.orchestra.run.smithi078.stdout: "mirror_snapshot_schedule": { 2024-03-24T03:37:34.955 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mirror_snapshot_schedule", 2024-03-24T03:37:34.955 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.955 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.955 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.955 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.955 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.955 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.955 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.955 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.956 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.956 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.956 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.956 INFO:teuthology.orchestra.run.smithi078.stdout: "trash_purge_schedule": { 2024-03-24T03:37:34.956 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "trash_purge_schedule", 2024-03-24T03:37:34.956 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.956 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.956 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.957 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.957 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.957 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.957 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.957 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.957 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.957 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.957 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.958 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.958 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:34.958 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "restful", 2024-03-24T03:37:34.958 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:34.958 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:34.958 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:34.959 INFO:teuthology.orchestra.run.smithi078.stdout: "enable_auth": { 2024-03-24T03:37:34.959 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enable_auth", 2024-03-24T03:37:34.959 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.959 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.959 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.959 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:34.959 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.959 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.959 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.960 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.960 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.960 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.960 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.960 INFO:teuthology.orchestra.run.smithi078.stdout: "key_file": { 2024-03-24T03:37:34.960 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "key_file", 2024-03-24T03:37:34.960 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.960 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.961 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.961 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.961 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.961 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.961 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.961 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.961 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.961 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.961 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.962 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:34.962 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:34.963 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.963 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.963 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.963 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.963 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.963 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.964 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.964 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.964 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.964 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.964 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.964 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.964 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.965 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.965 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.965 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.965 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.965 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.965 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:34.965 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:34.966 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.966 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.966 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.966 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.966 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.966 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.966 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.966 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.967 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.967 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.967 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.967 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:34.967 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:34.967 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.967 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.968 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.968 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:34.968 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.968 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.968 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.968 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.968 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.968 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.968 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.968 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.969 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.969 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.969 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.969 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.969 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.969 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.969 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:34.969 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:34.970 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.970 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.970 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.970 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.970 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.970 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.970 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.970 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.971 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.971 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.971 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.971 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-03-24T03:37:34.971 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-03-24T03:37:34.971 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.971 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.971 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.972 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.972 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.972 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.972 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.972 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.972 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.972 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.972 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.973 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-03-24T03:37:34.973 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-03-24T03:37:34.973 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.973 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.973 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.973 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.973 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.973 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.973 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.973 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.974 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.974 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.974 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.974 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.974 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:34.974 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rgw", 2024-03-24T03:37:34.975 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:34.975 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:34.975 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:34.975 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:34.975 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:34.975 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.975 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.975 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.975 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.976 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.976 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.976 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.976 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.976 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.976 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.976 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.976 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.976 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.977 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.977 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.977 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.977 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.977 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.977 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:34.977 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:34.977 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.978 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.978 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.978 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.978 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.978 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.978 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.978 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.978 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.978 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.979 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.979 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:34.979 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:34.979 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.979 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.979 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.979 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:34.979 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.980 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.980 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.980 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.980 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.980 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.980 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.980 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.980 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.980 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.980 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.981 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.981 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.981 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.981 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:34.981 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:34.981 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.981 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.981 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.982 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.982 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.982 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.982 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.982 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.982 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.982 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.982 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.982 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.983 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:34.983 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rook", 2024-03-24T03:37:34.983 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:34.983 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:34.983 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:34.983 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:34.983 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:34.984 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.984 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.984 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.984 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.984 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.984 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.984 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.984 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.984 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.985 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.985 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.985 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.985 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.985 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.985 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.985 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.985 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.986 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.986 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:34.986 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:34.986 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.986 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.986 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.986 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.986 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.987 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.987 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.987 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.987 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.987 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.987 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.987 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:34.987 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:34.988 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.988 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.988 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.988 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:34.988 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.988 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.988 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.988 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.988 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.989 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.989 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.989 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.989 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.989 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.989 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.989 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.989 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.989 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.990 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:34.990 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:34.990 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.990 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.990 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.990 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.990 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.990 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.990 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.991 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.991 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.991 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.991 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.991 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_class": { 2024-03-24T03:37:34.991 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "storage_class", 2024-03-24T03:37:34.991 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.991 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.992 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:34.992 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "local", 2024-03-24T03:37:34.992 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.992 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.992 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.992 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-24T03:37:34.992 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.992 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.992 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.993 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.993 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:34.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:34.993 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "selftest", 2024-03-24T03:37:34.993 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:34.993 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:34.993 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:34.993 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:34.994 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:34.994 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.994 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.994 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.994 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:34.994 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.994 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.994 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.994 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.995 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.995 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.995 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.995 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.995 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.995 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.995 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.995 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.995 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.996 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.996 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:34.996 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:34.996 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:34.996 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.996 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.996 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:34.996 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.996 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.997 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:34.997 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.997 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.997 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.997 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:34.997 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:34.997 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:34.997 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:34.998 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:34.998 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:34.998 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:34.998 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:34.998 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:34.998 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:34.998 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:34.998 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:34.998 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:34.999 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:34.999 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:34.999 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:34.999 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:34.999 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:34.999 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:34.999 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:34.999 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:34.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.000 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.000 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.000 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.000 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.000 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.000 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.000 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.000 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.000 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.000 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.001 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.001 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.001 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.001 INFO:teuthology.orchestra.run.smithi078.stdout: "roption1": { 2024-03-24T03:37:35.001 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "roption1", 2024-03-24T03:37:35.001 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.001 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.001 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.002 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.002 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.002 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.002 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.002 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.002 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.002 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.002 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.003 INFO:teuthology.orchestra.run.smithi078.stdout: "roption2": { 2024-03-24T03:37:35.003 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "roption2", 2024-03-24T03:37:35.003 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.003 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.003 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.003 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "xyz", 2024-03-24T03:37:35.003 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.003 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.003 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.003 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.004 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.004 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.004 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.004 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption1": { 2024-03-24T03:37:35.004 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption1", 2024-03-24T03:37:35.004 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.004 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.004 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.005 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.005 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.005 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.005 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.005 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.005 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.005 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.005 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.006 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption2": { 2024-03-24T03:37:35.006 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption2", 2024-03-24T03:37:35.006 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.006 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.007 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.007 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.007 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.007 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.007 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.007 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.008 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.008 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.008 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.008 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption3": { 2024-03-24T03:37:35.009 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption3", 2024-03-24T03:37:35.009 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-24T03:37:35.009 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.009 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.009 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.009 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.010 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.010 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.010 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.010 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.010 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.011 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.011 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption4": { 2024-03-24T03:37:35.011 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption4", 2024-03-24T03:37:35.011 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.011 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.012 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.012 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.012 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.012 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.012 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.013 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.013 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.013 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.013 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.013 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption5": { 2024-03-24T03:37:35.014 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption5", 2024-03-24T03:37:35.014 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.014 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.014 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.014 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.015 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.015 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.015 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.015 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.015 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.015 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.016 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.016 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption6": { 2024-03-24T03:37:35.016 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption6", 2024-03-24T03:37:35.016 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.017 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.017 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.017 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.017 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.017 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.017 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.018 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.018 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.018 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.018 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.018 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption7": { 2024-03-24T03:37:35.018 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption7", 2024-03-24T03:37:35.019 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.019 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.019 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.019 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.019 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-24T03:37:35.019 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "42", 2024-03-24T03:37:35.019 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.019 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.019 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.019 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.020 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.020 INFO:teuthology.orchestra.run.smithi078.stdout: "testkey": { 2024-03-24T03:37:35.020 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testkey", 2024-03-24T03:37:35.020 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.020 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.020 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.020 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.020 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.021 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.021 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.021 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.021 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.021 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.021 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.021 INFO:teuthology.orchestra.run.smithi078.stdout: "testlkey": { 2024-03-24T03:37:35.021 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testlkey", 2024-03-24T03:37:35.022 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.022 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.022 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.022 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.022 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.022 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.022 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.022 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.022 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.023 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.023 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.023 INFO:teuthology.orchestra.run.smithi078.stdout: "testnewline": { 2024-03-24T03:37:35.023 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testnewline", 2024-03-24T03:37:35.023 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.023 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.023 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.023 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.023 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.024 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.024 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.024 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.024 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.024 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.024 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.024 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.024 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.025 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snap_schedule", 2024-03-24T03:37:35.025 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.025 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.025 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.025 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_m_granularity": { 2024-03-24T03:37:35.025 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_m_granularity", 2024-03-24T03:37:35.025 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.026 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.026 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.026 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.026 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.026 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.026 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.026 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow minute scheduled snapshots", 2024-03-24T03:37:35.026 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.026 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.027 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.027 INFO:teuthology.orchestra.run.smithi078.stdout: "dump_on_update": { 2024-03-24T03:37:35.027 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "dump_on_update", 2024-03-24T03:37:35.027 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.027 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.027 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.027 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.027 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.027 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.028 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.028 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "dump database to debug log on update", 2024-03-24T03:37:35.028 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.028 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.028 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.028 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.028 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.028 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.029 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.029 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.029 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.029 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.029 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.029 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.029 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.029 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.029 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.029 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.030 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.030 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.030 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.030 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.030 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.030 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.030 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.030 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.031 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.031 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.031 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.031 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.031 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.031 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.031 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.031 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.031 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.032 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.032 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.032 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.032 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.032 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.032 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.032 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.032 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.033 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.033 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.033 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.033 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.033 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.033 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.033 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.033 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.033 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.034 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.034 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.034 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.034 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.034 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.034 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.034 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.034 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.035 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.035 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.035 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.035 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.035 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.035 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.035 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.035 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.035 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.036 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.036 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.036 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.036 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.036 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "stats", 2024-03-24T03:37:35.036 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.036 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.037 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.037 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.037 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.037 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.037 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.037 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.037 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.037 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.037 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.038 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.038 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.038 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.038 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.038 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.038 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.038 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.038 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.038 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.039 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.039 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.039 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.039 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.039 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.039 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.039 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.039 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.039 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.040 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.040 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.040 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.040 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.040 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.040 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.040 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.040 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.041 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.041 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.041 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.041 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.041 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.041 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.041 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.041 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.041 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.042 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.042 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.042 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.042 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.042 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.042 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.042 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.042 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.042 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.043 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.043 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.043 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.043 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.043 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.043 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.043 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.043 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.044 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.044 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.044 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.044 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.044 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.044 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.044 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.044 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.045 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "status", 2024-03-24T03:37:35.045 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.045 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.045 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.045 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.045 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.045 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.045 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.045 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.046 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.046 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.046 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.046 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.046 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.046 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.046 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.046 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.046 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.047 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.047 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.047 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.047 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.047 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.047 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.047 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.047 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.048 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.048 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.048 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.048 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.048 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.048 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.048 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.048 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.048 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.049 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.049 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.049 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.049 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.049 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.049 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.049 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.049 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.049 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.050 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.050 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.050 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.050 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.050 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.050 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.050 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.050 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.050 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.051 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.051 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.051 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.051 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.051 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.051 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.051 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.051 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.052 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.052 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.052 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.052 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.052 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.052 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.052 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.052 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.052 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.052 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.053 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.053 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "telegraf", 2024-03-24T03:37:35.053 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.053 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.053 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.053 INFO:teuthology.orchestra.run.smithi078.stdout: "address": { 2024-03-24T03:37:35.053 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "address", 2024-03-24T03:37:35.054 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.054 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.054 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.054 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-24T03:37:35.054 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.054 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.054 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.054 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.054 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.055 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.055 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.055 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-24T03:37:35.055 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-24T03:37:35.055 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:35.055 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.055 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.056 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15", 2024-03-24T03:37:35.056 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.056 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.056 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.056 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.056 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.056 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.056 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.057 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.057 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.057 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.057 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.057 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.057 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.057 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.057 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.057 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.058 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.058 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.058 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.058 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.058 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.058 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.058 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.058 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.058 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.059 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.059 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.059 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.059 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.059 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.059 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.059 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.059 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.060 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.060 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.060 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.060 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.060 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.060 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.060 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.061 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.061 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.061 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.061 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.061 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.061 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.061 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.061 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.061 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.062 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.062 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.062 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.062 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.062 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.062 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.062 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.062 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.062 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.063 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.063 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.063 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.063 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.063 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.063 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.063 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.063 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.064 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.064 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.064 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.064 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.064 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.064 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.064 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.064 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.065 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.065 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "telemetry", 2024-03-24T03:37:35.065 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.065 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.065 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.065 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_basic": { 2024-03-24T03:37:35.066 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_basic", 2024-03-24T03:37:35.066 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.066 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.066 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.066 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.066 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.066 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.066 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.066 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-24T03:37:35.067 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.067 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.067 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.067 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_crash": { 2024-03-24T03:37:35.067 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_crash", 2024-03-24T03:37:35.067 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.067 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.068 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.068 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.068 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.068 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.068 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.068 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-24T03:37:35.068 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.068 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.068 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.069 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_device": { 2024-03-24T03:37:35.069 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_device", 2024-03-24T03:37:35.069 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.069 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.069 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.069 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.069 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.069 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.069 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.070 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-24T03:37:35.070 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.070 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.070 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.070 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_ident": { 2024-03-24T03:37:35.070 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_ident", 2024-03-24T03:37:35.070 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.070 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.071 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.071 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.071 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.071 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.071 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.071 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-24T03:37:35.071 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.071 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.071 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.072 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_perf": { 2024-03-24T03:37:35.072 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_perf", 2024-03-24T03:37:35.072 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.072 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.072 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.072 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.072 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.072 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.072 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.073 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-24T03:37:35.073 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.073 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.073 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.073 INFO:teuthology.orchestra.run.smithi078.stdout: "contact": { 2024-03-24T03:37:35.073 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "contact", 2024-03-24T03:37:35.073 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.073 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.074 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.074 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.074 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.074 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.074 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.074 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.074 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.074 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.074 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.075 INFO:teuthology.orchestra.run.smithi078.stdout: "description": { 2024-03-24T03:37:35.075 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "description", 2024-03-24T03:37:35.075 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.075 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.075 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.075 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.075 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.075 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.076 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.076 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.076 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.076 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.076 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.076 INFO:teuthology.orchestra.run.smithi078.stdout: "device_url": { 2024-03-24T03:37:35.076 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_url", 2024-03-24T03:37:35.076 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.076 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.077 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.077 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-24T03:37:35.077 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.077 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.077 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.077 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.077 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.077 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.077 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.078 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": { 2024-03-24T03:37:35.078 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enabled", 2024-03-24T03:37:35.078 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.078 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.078 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.078 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.078 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.078 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.079 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.079 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.079 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.079 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.079 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.079 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-24T03:37:35.079 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-24T03:37:35.079 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.080 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.080 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.080 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "24", 2024-03-24T03:37:35.080 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "8", 2024-03-24T03:37:35.080 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.080 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.080 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.080 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.080 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.080 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.081 INFO:teuthology.orchestra.run.smithi078.stdout: "last_opt_revision": { 2024-03-24T03:37:35.081 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "last_opt_revision", 2024-03-24T03:37:35.081 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.081 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.081 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.081 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1", 2024-03-24T03:37:35.081 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.081 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.082 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.082 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.082 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.082 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.082 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.082 INFO:teuthology.orchestra.run.smithi078.stdout: "leaderboard": { 2024-03-24T03:37:35.082 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "leaderboard", 2024-03-24T03:37:35.082 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.083 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.083 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.083 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.083 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.083 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.083 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.083 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.083 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.083 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.084 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.084 INFO:teuthology.orchestra.run.smithi078.stdout: "leaderboard_description": { 2024-03-24T03:37:35.084 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "leaderboard_description", 2024-03-24T03:37:35.084 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.084 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.084 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.084 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.084 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.085 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.085 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.085 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.085 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.085 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.085 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.085 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.085 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.085 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.086 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.086 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.086 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.086 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.086 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.086 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.086 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.086 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.086 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.087 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.087 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.087 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.087 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.087 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.087 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.087 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.087 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.088 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.088 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.088 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.088 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.088 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.088 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.088 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.088 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.088 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.088 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.089 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.089 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.089 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.089 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.089 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.089 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.089 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.089 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.090 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.090 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.090 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.090 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.090 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.090 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.090 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.090 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.090 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.091 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.091 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.091 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.091 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.091 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.091 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.091 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.091 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.092 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.092 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.092 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.092 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.092 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.092 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.092 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.092 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.092 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.092 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.093 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.093 INFO:teuthology.orchestra.run.smithi078.stdout: "organization": { 2024-03-24T03:37:35.093 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "organization", 2024-03-24T03:37:35.093 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.093 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.093 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.093 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.093 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.094 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.094 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.094 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.094 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.094 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.094 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.094 INFO:teuthology.orchestra.run.smithi078.stdout: "proxy": { 2024-03-24T03:37:35.094 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "proxy", 2024-03-24T03:37:35.095 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.095 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.095 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.095 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.095 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.095 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.095 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.095 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.095 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.096 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.096 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.096 INFO:teuthology.orchestra.run.smithi078.stdout: "url": { 2024-03-24T03:37:35.096 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "url", 2024-03-24T03:37:35.096 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.096 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.096 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.096 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-24T03:37:35.097 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.097 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.097 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.097 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.097 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.097 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.097 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.097 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.097 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.098 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test_orchestrator", 2024-03-24T03:37:35.098 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.098 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.098 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.098 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.098 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.098 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.099 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.099 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.099 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.099 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.099 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.099 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.099 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.099 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.099 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.100 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.100 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.100 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.100 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.100 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.100 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.100 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.100 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.100 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.101 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.101 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.101 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.101 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.101 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.101 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.101 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.101 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.101 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.102 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.102 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.102 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.102 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.102 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.102 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.102 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.102 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.103 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.103 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.103 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.103 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.103 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.103 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.103 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.103 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.103 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.104 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.104 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.104 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.104 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.104 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.104 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.104 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.104 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.104 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.105 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.105 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.105 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.105 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.105 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.105 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.105 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.105 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.105 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.106 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.106 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.106 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.106 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "volumes", 2024-03-24T03:37:35.106 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.106 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.106 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.107 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.107 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.107 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.107 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.107 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.107 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.107 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.107 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.107 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.108 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.108 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.108 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.108 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.108 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.108 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.108 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.108 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.108 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.108 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.109 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.109 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.109 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.109 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.109 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.109 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.109 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.109 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.110 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.110 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.110 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.110 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.110 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.110 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.110 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.110 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.111 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.111 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.111 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.111 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.111 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.111 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.111 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.111 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.111 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.111 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.112 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.112 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.112 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.112 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.112 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.112 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.112 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.112 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.113 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.113 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.113 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.113 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.113 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.113 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.113 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.113 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.113 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.114 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.114 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.114 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.114 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.114 INFO:teuthology.orchestra.run.smithi078.stdout: "max_concurrent_clones": { 2024-03-24T03:37:35.114 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_concurrent_clones", 2024-03-24T03:37:35.114 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.114 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.115 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.115 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "4", 2024-03-24T03:37:35.115 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.115 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.115 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.115 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-24T03:37:35.115 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.115 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.115 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.116 INFO:teuthology.orchestra.run.smithi078.stdout: "periodic_async_work": { 2024-03-24T03:37:35.116 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "periodic_async_work", 2024-03-24T03:37:35.116 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.116 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.116 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.116 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.116 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.116 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.117 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.117 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Periodically check for async work", 2024-03-24T03:37:35.117 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.117 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.117 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.117 INFO:teuthology.orchestra.run.smithi078.stdout: "snapshot_clone_delay": { 2024-03-24T03:37:35.117 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snapshot_clone_delay", 2024-03-24T03:37:35.117 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.118 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.118 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.118 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-24T03:37:35.118 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.118 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.118 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.118 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-24T03:37:35.118 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.118 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.119 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.119 INFO:teuthology.orchestra.run.smithi078.stdout: "snapshot_clone_no_wait": { 2024-03-24T03:37:35.119 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snapshot_clone_no_wait", 2024-03-24T03:37:35.119 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.119 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.121 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.122 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.122 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.122 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.122 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.122 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-24T03:37:35.122 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.122 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.122 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.123 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.123 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.123 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix", 2024-03-24T03:37:35.123 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.123 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.123 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.123 INFO:teuthology.orchestra.run.smithi078.stdout: "discovery_interval": { 2024-03-24T03:37:35.124 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "discovery_interval", 2024-03-24T03:37:35.124 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-24T03:37:35.124 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.124 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.124 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "100", 2024-03-24T03:37:35.124 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.124 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.124 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.124 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.125 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.125 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.125 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.125 INFO:teuthology.orchestra.run.smithi078.stdout: "identifier": { 2024-03-24T03:37:35.125 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "identifier", 2024-03-24T03:37:35.125 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.125 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.125 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.126 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.126 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.126 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.126 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.126 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.126 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.126 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.126 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.127 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-24T03:37:35.127 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-24T03:37:35.127 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:35.127 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.127 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.127 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-24T03:37:35.127 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.127 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.127 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.127 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.128 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.128 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.128 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.128 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.128 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.128 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.128 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.129 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.129 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.129 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.129 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.129 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.129 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.129 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.129 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.129 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.130 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.130 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.130 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.130 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.130 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.130 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.130 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.130 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.131 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.131 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.131 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.131 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.131 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.131 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.131 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.131 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.132 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.132 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.132 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.132 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.132 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.132 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.132 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.132 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.133 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.133 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.133 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.133 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.133 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.133 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.133 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.133 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.134 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.134 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.134 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.134 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.134 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.134 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.134 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.134 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.135 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.135 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.135 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.135 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.135 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.135 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.135 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.136 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.136 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.136 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.136 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.136 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.136 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.136 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_host": { 2024-03-24T03:37:35.137 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_host", 2024-03-24T03:37:35.137 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.137 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.137 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.137 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.137 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.137 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.137 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.137 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.138 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.138 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.138 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.138 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_port": { 2024-03-24T03:37:35.138 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_port", 2024-03-24T03:37:35.138 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.139 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.139 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.139 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10051", 2024-03-24T03:37:35.139 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.139 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.139 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.139 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.139 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.139 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.139 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.140 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_sender": { 2024-03-24T03:37:35.140 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_sender", 2024-03-24T03:37:35.140 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.140 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.140 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.140 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-24T03:37:35.140 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.140 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.141 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.141 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.141 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.141 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.141 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.141 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.141 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.141 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.141 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-24T03:37:35.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.142 INFO:teuthology.orchestra.run.smithi078.stdout: "gid": 25298, 2024-03-24T03:37:35.142 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "y", 2024-03-24T03:37:35.142 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_features": 4540701547738038271, 2024-03-24T03:37:35.142 INFO:teuthology.orchestra.run.smithi078.stdout: "available_modules": [ 2024-03-24T03:37:35.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.142 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "alerts", 2024-03-24T03:37:35.143 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.143 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.143 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.143 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-24T03:37:35.143 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-24T03:37:35.143 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:35.143 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.143 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.144 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-24T03:37:35.144 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.144 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.144 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.144 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How frequently to reexamine health status", 2024-03-24T03:37:35.144 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.144 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.144 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.145 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.145 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.145 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.145 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.145 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.145 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.145 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.145 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.145 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.145 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.146 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.146 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.146 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.146 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.146 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.146 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.146 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.146 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.147 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.147 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.147 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.147 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.147 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.147 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.147 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.147 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.147 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.148 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.148 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.148 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.148 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.148 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.148 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.148 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.148 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.149 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.149 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.149 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.149 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.149 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.149 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.149 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.149 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.149 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.150 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.150 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.150 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.150 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.150 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.150 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.150 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.150 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.150 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.151 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.151 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.151 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.151 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.151 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.151 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.151 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.151 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.152 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.152 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.152 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.152 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.152 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.152 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_destination": { 2024-03-24T03:37:35.152 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_destination", 2024-03-24T03:37:35.152 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.152 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.153 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.153 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.153 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.153 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.153 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.153 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Email address to send alerts to", 2024-03-24T03:37:35.153 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.153 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.153 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.154 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_from_name": { 2024-03-24T03:37:35.154 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_from_name", 2024-03-24T03:37:35.154 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.154 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.154 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.154 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "Ceph", 2024-03-24T03:37:35.154 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.154 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.155 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.155 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Email From: name", 2024-03-24T03:37:35.155 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.155 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.155 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.155 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_host": { 2024-03-24T03:37:35.155 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_host", 2024-03-24T03:37:35.155 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.156 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.156 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.156 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.156 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.156 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.156 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.156 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP server", 2024-03-24T03:37:35.156 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.156 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.157 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.157 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_password": { 2024-03-24T03:37:35.157 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_password", 2024-03-24T03:37:35.157 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.157 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.157 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.157 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.157 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.157 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.158 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.158 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Password to authenticate with", 2024-03-24T03:37:35.158 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.158 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.158 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.158 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_port": { 2024-03-24T03:37:35.158 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_port", 2024-03-24T03:37:35.158 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.159 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.159 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.159 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "465", 2024-03-24T03:37:35.159 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.159 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.159 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.159 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP port", 2024-03-24T03:37:35.159 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.159 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.160 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.160 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_sender": { 2024-03-24T03:37:35.160 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_sender", 2024-03-24T03:37:35.160 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.160 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.160 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.160 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.160 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.160 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.161 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.161 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP envelope sender", 2024-03-24T03:37:35.161 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.161 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.161 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.161 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_ssl": { 2024-03-24T03:37:35.161 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_ssl", 2024-03-24T03:37:35.162 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.162 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.162 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.162 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.162 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.162 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.162 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.162 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-24T03:37:35.163 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.163 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.163 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.163 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_user": { 2024-03-24T03:37:35.163 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_user", 2024-03-24T03:37:35.163 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.163 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.164 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.164 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.164 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.164 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.164 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.164 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "User to authenticate as", 2024-03-24T03:37:35.164 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.164 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.164 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.165 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.165 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.165 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "balancer", 2024-03-24T03:37:35.165 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.165 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.165 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.166 INFO:teuthology.orchestra.run.smithi078.stdout: "active": { 2024-03-24T03:37:35.166 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "active", 2024-03-24T03:37:35.166 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.166 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.166 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.166 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.166 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.166 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.167 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.167 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "automatically balance PGs across cluster", 2024-03-24T03:37:35.167 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.167 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.167 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.167 INFO:teuthology.orchestra.run.smithi078.stdout: "begin_time": { 2024-03-24T03:37:35.167 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "begin_time", 2024-03-24T03:37:35.167 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.168 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.168 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.168 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0000", 2024-03-24T03:37:35.168 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.168 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.168 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.168 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "beginning time of day to automatically balance", 2024-03-24T03:37:35.168 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T03:37:35.168 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.169 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.169 INFO:teuthology.orchestra.run.smithi078.stdout: "begin_weekday": { 2024-03-24T03:37:35.169 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "begin_weekday", 2024-03-24T03:37:35.169 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-24T03:37:35.169 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.169 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.169 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-24T03:37:35.169 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0", 2024-03-24T03:37:35.170 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "6", 2024-03-24T03:37:35.170 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.170 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-24T03:37:35.170 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T03:37:35.170 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.170 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.170 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_max_iterations": { 2024-03-24T03:37:35.170 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_max_iterations", 2024-03-24T03:37:35.171 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-24T03:37:35.171 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.171 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.171 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "25", 2024-03-24T03:37:35.171 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-24T03:37:35.171 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "250", 2024-03-24T03:37:35.171 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.171 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-24T03:37:35.171 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.172 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.172 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.172 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_metrics": { 2024-03-24T03:37:35.172 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_metrics", 2024-03-24T03:37:35.172 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.172 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.172 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.172 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "pgs,objects,bytes", 2024-03-24T03:37:35.173 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.173 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.173 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.173 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-24T03:37:35.173 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-24T03:37:35.173 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.173 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.173 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_step": { 2024-03-24T03:37:35.174 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_step", 2024-03-24T03:37:35.174 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-24T03:37:35.174 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.174 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.174 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0.5", 2024-03-24T03:37:35.174 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0.001", 2024-03-24T03:37:35.174 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "0.999", 2024-03-24T03:37:35.174 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.175 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "aggressiveness of optimization", 2024-03-24T03:37:35.175 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-24T03:37:35.175 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.175 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.175 INFO:teuthology.orchestra.run.smithi078.stdout: "end_time": { 2024-03-24T03:37:35.175 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "end_time", 2024-03-24T03:37:35.175 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.175 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.176 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.176 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "2359", 2024-03-24T03:37:35.176 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.176 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.176 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.176 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "ending time of day to automatically balance", 2024-03-24T03:37:35.176 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T03:37:35.176 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.176 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.177 INFO:teuthology.orchestra.run.smithi078.stdout: "end_weekday": { 2024-03-24T03:37:35.177 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "end_weekday", 2024-03-24T03:37:35.177 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-24T03:37:35.177 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.177 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.177 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-24T03:37:35.177 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0", 2024-03-24T03:37:35.177 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "6", 2024-03-24T03:37:35.177 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.178 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-24T03:37:35.178 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T03:37:35.178 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.178 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.178 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.178 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.178 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.178 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.179 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.179 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.179 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.179 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.179 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.179 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.179 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.179 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.179 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.180 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.180 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.180 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.180 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.180 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.180 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.180 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.180 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.180 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.181 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.181 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.181 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.181 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.181 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.181 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.181 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.181 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.181 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.182 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.182 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.182 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.182 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.182 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.182 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.182 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.182 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.183 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.183 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.183 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.183 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.183 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.183 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.183 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.183 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.183 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.184 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.184 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.184 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.184 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.184 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.184 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.184 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.184 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.185 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.185 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.185 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.185 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.185 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.185 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.185 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.185 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.185 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.186 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.186 INFO:teuthology.orchestra.run.smithi078.stdout: "min_score": { 2024-03-24T03:37:35.186 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "min_score", 2024-03-24T03:37:35.186 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-24T03:37:35.186 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.186 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.186 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-24T03:37:35.187 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.187 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.187 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.187 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-24T03:37:35.187 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.187 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.187 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.187 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": { 2024-03-24T03:37:35.187 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mode", 2024-03-24T03:37:35.188 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.188 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.188 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.188 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "upmap", 2024-03-24T03:37:35.188 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.188 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.188 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.188 INFO:teuthology.orchestra.run.smithi078.stdout: "crush-compat", 2024-03-24T03:37:35.188 INFO:teuthology.orchestra.run.smithi078.stdout: "none", 2024-03-24T03:37:35.189 INFO:teuthology.orchestra.run.smithi078.stdout: "read", 2024-03-24T03:37:35.189 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap", 2024-03-24T03:37:35.189 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap-read" 2024-03-24T03:37:35.189 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.189 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Balancer mode", 2024-03-24T03:37:35.189 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.189 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.189 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.189 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_ids": { 2024-03-24T03:37:35.190 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pool_ids", 2024-03-24T03:37:35.190 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.190 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.190 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.190 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.190 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.190 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.190 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.190 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-24T03:37:35.191 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.191 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.191 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.191 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-24T03:37:35.191 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-24T03:37:35.191 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:35.191 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.191 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.192 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-24T03:37:35.192 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.192 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.192 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.192 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-24T03:37:35.192 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.192 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.192 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.193 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap_max_deviation": { 2024-03-24T03:37:35.193 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "upmap_max_deviation", 2024-03-24T03:37:35.193 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.193 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.193 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.193 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-24T03:37:35.193 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-24T03:37:35.193 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.193 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.194 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-24T03:37:35.194 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-24T03:37:35.194 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.194 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.194 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap_max_optimizations": { 2024-03-24T03:37:35.194 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "upmap_max_optimizations", 2024-03-24T03:37:35.194 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-24T03:37:35.194 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.195 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.195 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-24T03:37:35.195 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.195 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.195 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.195 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-24T03:37:35.195 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.195 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.195 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.196 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.196 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.196 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cephadm", 2024-03-24T03:37:35.196 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.196 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.196 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.196 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_down_multiplier": { 2024-03-24T03:37:35.197 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_down_multiplier", 2024-03-24T03:37:35.197 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-24T03:37:35.197 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.197 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.197 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3.0", 2024-03-24T03:37:35.197 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.197 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.197 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.197 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-24T03:37:35.198 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.198 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.198 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.198 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_refresh_rate": { 2024-03-24T03:37:35.198 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_refresh_rate", 2024-03-24T03:37:35.198 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:35.198 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.199 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.199 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "20", 2024-03-24T03:37:35.199 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.199 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.199 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.199 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-24T03:37:35.199 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.199 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.200 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.200 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_starting_port": { 2024-03-24T03:37:35.200 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_starting_port", 2024-03-24T03:37:35.200 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.200 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.200 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.200 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "4721", 2024-03-24T03:37:35.200 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.201 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.201 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.201 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-24T03:37:35.201 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.201 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.201 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.201 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_ptrace": { 2024-03-24T03:37:35.201 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_ptrace", 2024-03-24T03:37:35.202 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.202 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.202 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.202 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.202 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.202 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.202 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.202 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-24T03:37:35.203 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-24T03:37:35.203 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.203 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.203 INFO:teuthology.orchestra.run.smithi078.stdout: "autotune_interval": { 2024-03-24T03:37:35.203 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "autotune_interval", 2024-03-24T03:37:35.203 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:35.203 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.204 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.204 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-24T03:37:35.204 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.204 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.204 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.204 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-24T03:37:35.204 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.204 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.204 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.205 INFO:teuthology.orchestra.run.smithi078.stdout: "autotune_memory_target_ratio": { 2024-03-24T03:37:35.205 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "autotune_memory_target_ratio", 2024-03-24T03:37:35.205 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-24T03:37:35.205 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.205 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.205 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0.7", 2024-03-24T03:37:35.205 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.206 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.206 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.206 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-24T03:37:35.206 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.206 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.206 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.206 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm_log_destination": { 2024-03-24T03:37:35.207 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cephadm_log_destination", 2024-03-24T03:37:35.207 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.207 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.207 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.207 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.207 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.207 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.208 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.208 INFO:teuthology.orchestra.run.smithi078.stdout: "file", 2024-03-24T03:37:35.208 INFO:teuthology.orchestra.run.smithi078.stdout: "file,syslog", 2024-03-24T03:37:35.208 INFO:teuthology.orchestra.run.smithi078.stdout: "syslog" 2024-03-24T03:37:35.208 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.208 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-24T03:37:35.208 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.208 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.208 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.209 INFO:teuthology.orchestra.run.smithi078.stdout: "cgroups_split": { 2024-03-24T03:37:35.209 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cgroups_split", 2024-03-24T03:37:35.209 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.209 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.209 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.209 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.209 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.210 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.210 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.210 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-24T03:37:35.210 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.210 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.210 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.210 INFO:teuthology.orchestra.run.smithi078.stdout: "config_checks_enabled": { 2024-03-24T03:37:35.210 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_checks_enabled", 2024-03-24T03:37:35.211 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.211 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.211 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.211 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.211 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.211 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.211 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.211 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-24T03:37:35.211 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.212 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.212 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.212 INFO:teuthology.orchestra.run.smithi078.stdout: "config_dashboard": { 2024-03-24T03:37:35.212 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_dashboard", 2024-03-24T03:37:35.212 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.212 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.212 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.212 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.213 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.213 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.213 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.213 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-24T03:37:35.213 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.213 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.213 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.213 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_alertmanager": { 2024-03-24T03:37:35.214 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_alertmanager", 2024-03-24T03:37:35.214 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.214 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.214 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.214 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-24T03:37:35.214 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.214 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.214 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.215 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-24T03:37:35.215 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.215 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.215 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.215 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_base": { 2024-03-24T03:37:35.215 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_base", 2024-03-24T03:37:35.215 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.215 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.215 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.216 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-24T03:37:35.216 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.216 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.216 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.216 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Container image name, without the tag", 2024-03-24T03:37:35.216 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.216 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.216 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.217 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_elasticsearch": { 2024-03-24T03:37:35.217 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_elasticsearch", 2024-03-24T03:37:35.217 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.217 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.217 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.217 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-24T03:37:35.217 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.217 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.217 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.218 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "elasticsearch container image", 2024-03-24T03:37:35.218 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.218 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.218 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.218 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_grafana": { 2024-03-24T03:37:35.218 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_grafana", 2024-03-24T03:37:35.218 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.218 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.219 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.219 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-24T03:37:35.219 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.219 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.219 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.219 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-24T03:37:35.219 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.219 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.219 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.220 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_haproxy": { 2024-03-24T03:37:35.220 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_haproxy", 2024-03-24T03:37:35.220 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.220 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.220 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.220 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-24T03:37:35.220 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.220 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.220 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.221 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "HAproxy container image", 2024-03-24T03:37:35.221 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.221 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.221 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.221 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_agent": { 2024-03-24T03:37:35.221 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_agent", 2024-03-24T03:37:35.221 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.221 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.222 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.222 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-24T03:37:35.222 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.222 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.222 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.222 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger agent container image", 2024-03-24T03:37:35.222 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.222 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.222 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.223 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_collector": { 2024-03-24T03:37:35.223 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_collector", 2024-03-24T03:37:35.223 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.223 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.223 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.223 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-24T03:37:35.223 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.223 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.224 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.224 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger collector container image", 2024-03-24T03:37:35.224 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.224 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.224 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.224 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_query": { 2024-03-24T03:37:35.224 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_query", 2024-03-24T03:37:35.224 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.225 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.225 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.225 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-24T03:37:35.225 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.225 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.225 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.225 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger query container image", 2024-03-24T03:37:35.225 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.225 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.226 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.226 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_keepalived": { 2024-03-24T03:37:35.226 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_keepalived", 2024-03-24T03:37:35.226 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.226 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.226 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.226 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-24T03:37:35.227 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.227 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.227 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.227 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Keepalived container image", 2024-03-24T03:37:35.227 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.227 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.227 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.227 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_loki": { 2024-03-24T03:37:35.227 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_loki", 2024-03-24T03:37:35.228 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.228 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.228 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.228 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-24T03:37:35.228 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.228 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.228 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.228 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Loki container image", 2024-03-24T03:37:35.228 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.229 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.229 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.229 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_node_exporter": { 2024-03-24T03:37:35.229 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_node_exporter", 2024-03-24T03:37:35.229 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.229 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.229 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.229 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-24T03:37:35.230 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.230 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.230 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.230 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-24T03:37:35.230 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.230 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.230 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.230 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_nvmeof": { 2024-03-24T03:37:35.230 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_nvmeof", 2024-03-24T03:37:35.231 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.231 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.231 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.231 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-24T03:37:35.231 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.231 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.231 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.231 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Nvme-of container image", 2024-03-24T03:37:35.231 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.232 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.232 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.232 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_prometheus": { 2024-03-24T03:37:35.232 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_prometheus", 2024-03-24T03:37:35.232 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.232 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.232 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.232 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-24T03:37:35.233 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.233 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.233 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.233 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-24T03:37:35.233 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.233 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.233 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.234 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_promtail": { 2024-03-24T03:37:35.234 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_promtail", 2024-03-24T03:37:35.234 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.234 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.234 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.234 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-24T03:37:35.234 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.234 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.235 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.235 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Promtail container image", 2024-03-24T03:37:35.235 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.235 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.235 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.235 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_snmp_gateway": { 2024-03-24T03:37:35.235 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_snmp_gateway", 2024-03-24T03:37:35.236 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.236 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.236 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.236 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-24T03:37:35.236 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.236 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.236 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.236 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SNMP Gateway container image", 2024-03-24T03:37:35.237 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.237 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.237 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.237 INFO:teuthology.orchestra.run.smithi078.stdout: "container_init": { 2024-03-24T03:37:35.237 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_init", 2024-03-24T03:37:35.237 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.237 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.237 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Run podman/docker with `--init`", 2024-03-24T03:37:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.239 INFO:teuthology.orchestra.run.smithi078.stdout: "daemon_cache_timeout": { 2024-03-24T03:37:35.239 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "daemon_cache_timeout", 2024-03-24T03:37:35.239 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:35.239 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.239 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.239 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-24T03:37:35.239 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.239 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.239 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.240 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-24T03:37:35.240 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.240 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.240 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.240 INFO:teuthology.orchestra.run.smithi078.stdout: "default_cephadm_command_timeout": { 2024-03-24T03:37:35.240 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default_cephadm_command_timeout", 2024-03-24T03:37:35.240 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.241 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.241 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.241 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "900", 2024-03-24T03:37:35.241 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.241 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.241 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.241 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-24T03:37:35.241 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.241 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.242 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.242 INFO:teuthology.orchestra.run.smithi078.stdout: "default_registry": { 2024-03-24T03:37:35.242 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default_registry", 2024-03-24T03:37:35.242 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.242 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.242 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.242 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io", 2024-03-24T03:37:35.242 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.243 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.243 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.243 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-24T03:37:35.243 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.243 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.243 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.243 INFO:teuthology.orchestra.run.smithi078.stdout: "device_cache_timeout": { 2024-03-24T03:37:35.243 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_cache_timeout", 2024-03-24T03:37:35.244 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:35.244 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.244 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.244 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1800", 2024-03-24T03:37:35.244 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.244 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.244 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.244 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache device inventory", 2024-03-24T03:37:35.244 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.245 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.245 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.245 INFO:teuthology.orchestra.run.smithi078.stdout: "device_enhanced_scan": { 2024-03-24T03:37:35.245 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_enhanced_scan", 2024-03-24T03:37:35.245 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.245 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.245 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.245 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.246 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.246 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.246 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.246 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-24T03:37:35.246 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.246 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.246 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.247 INFO:teuthology.orchestra.run.smithi078.stdout: "facts_cache_timeout": { 2024-03-24T03:37:35.247 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "facts_cache_timeout", 2024-03-24T03:37:35.247 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:35.247 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.247 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.247 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-24T03:37:35.247 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.247 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.247 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.248 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache host facts data", 2024-03-24T03:37:35.248 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.248 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.248 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.248 INFO:teuthology.orchestra.run.smithi078.stdout: "host_check_interval": { 2024-03-24T03:37:35.248 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "host_check_interval", 2024-03-24T03:37:35.248 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:35.249 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.249 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.249 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-24T03:37:35.249 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.249 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.249 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.249 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to perform a host check", 2024-03-24T03:37:35.249 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.249 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.249 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.250 INFO:teuthology.orchestra.run.smithi078.stdout: "hw_monitoring": { 2024-03-24T03:37:35.250 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "hw_monitoring", 2024-03-24T03:37:35.250 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.250 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.250 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.250 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.250 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.250 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.251 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.251 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-24T03:37:35.251 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.251 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.251 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.251 INFO:teuthology.orchestra.run.smithi078.stdout: "inventory_list_all": { 2024-03-24T03:37:35.251 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "inventory_list_all", 2024-03-24T03:37:35.251 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.252 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.252 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.252 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.252 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.252 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.252 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.252 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-24T03:37:35.252 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.252 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.252 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.253 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.253 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.253 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.253 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.253 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.253 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.253 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.253 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.254 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.254 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.254 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.254 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.254 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.254 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.254 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.254 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.254 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.255 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.255 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.255 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.255 INFO:teuthology.orchestra.run.smithi078.stdout: "log_refresh_metadata": { 2024-03-24T03:37:35.255 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_refresh_metadata", 2024-03-24T03:37:35.255 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.255 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.255 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.256 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.256 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.256 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.256 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.256 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-24T03:37:35.256 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.256 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.256 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.257 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.257 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.257 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.257 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.257 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.257 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.257 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.257 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.257 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.258 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-24T03:37:35.258 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.258 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.258 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.258 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.258 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.258 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.258 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.259 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.259 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.259 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.259 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.259 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.259 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.259 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.259 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.259 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.259 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.260 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.260 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.260 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.260 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.260 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.260 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.260 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.260 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.261 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.261 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.261 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.261 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.261 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.261 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.261 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.261 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.262 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.262 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.262 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.262 INFO:teuthology.orchestra.run.smithi078.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-24T03:37:35.262 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-24T03:37:35.262 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.262 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.262 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.263 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.263 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.263 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.263 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.263 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-24T03:37:35.263 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.263 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.263 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.264 INFO:teuthology.orchestra.run.smithi078.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-24T03:37:35.264 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-24T03:37:35.264 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.264 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.264 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.264 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "*", 2024-03-24T03:37:35.264 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.264 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.264 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.265 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-24T03:37:35.265 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.265 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.265 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.265 INFO:teuthology.orchestra.run.smithi078.stdout: "max_count_per_host": { 2024-03-24T03:37:35.265 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_count_per_host", 2024-03-24T03:37:35.265 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.265 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.266 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.266 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-24T03:37:35.266 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.266 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.266 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.266 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "max number of daemons per service per host", 2024-03-24T03:37:35.266 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.266 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.267 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.267 INFO:teuthology.orchestra.run.smithi078.stdout: "max_osd_draining_count": { 2024-03-24T03:37:35.267 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_osd_draining_count", 2024-03-24T03:37:35.267 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.267 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.267 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.267 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-24T03:37:35.268 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.268 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.268 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.268 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-24T03:37:35.268 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.268 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.268 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.268 INFO:teuthology.orchestra.run.smithi078.stdout: "migration_current": { 2024-03-24T03:37:35.269 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "migration_current", 2024-03-24T03:37:35.269 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.269 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.269 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.269 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.269 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.269 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.269 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.270 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "internal - do not modify", 2024-03-24T03:37:35.270 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.270 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.270 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.270 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": { 2024-03-24T03:37:35.270 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mode", 2024-03-24T03:37:35.270 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.271 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.271 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.271 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "root", 2024-03-24T03:37:35.271 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.271 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.271 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.271 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm-package", 2024-03-24T03:37:35.271 INFO:teuthology.orchestra.run.smithi078.stdout: "root" 2024-03-24T03:37:35.272 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.272 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "mode for remote execution of cephadm", 2024-03-24T03:37:35.272 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.272 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.272 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.272 INFO:teuthology.orchestra.run.smithi078.stdout: "oob_default_addr": { 2024-03-24T03:37:35.272 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "oob_default_addr", 2024-03-24T03:37:35.272 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.273 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.273 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.273 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "169.254.1.1", 2024-03-24T03:37:35.273 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.273 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.273 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.273 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-24T03:37:35.274 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.274 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.274 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.274 INFO:teuthology.orchestra.run.smithi078.stdout: "prometheus_alerts_path": { 2024-03-24T03:37:35.274 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus_alerts_path", 2024-03-24T03:37:35.274 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.274 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.274 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.275 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-24T03:37:35.275 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.275 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.275 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.275 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-24T03:37:35.275 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.275 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.275 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.276 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_insecure": { 2024-03-24T03:37:35.276 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_insecure", 2024-03-24T03:37:35.276 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.276 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.276 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.276 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.276 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.277 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.277 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.277 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-24T03:37:35.277 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.277 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.277 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.277 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_password": { 2024-03-24T03:37:35.278 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_password", 2024-03-24T03:37:35.278 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.278 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.278 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.278 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.278 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.278 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.278 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.279 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-24T03:37:35.279 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.279 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.279 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.279 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_url": { 2024-03-24T03:37:35.279 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_url", 2024-03-24T03:37:35.279 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.280 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.280 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.280 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.280 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.280 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.280 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.280 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-24T03:37:35.280 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.280 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.281 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.281 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_username": { 2024-03-24T03:37:35.281 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_username", 2024-03-24T03:37:35.281 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.281 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.281 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.281 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.281 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.282 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.282 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.282 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-24T03:37:35.282 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.282 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.282 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.282 INFO:teuthology.orchestra.run.smithi078.stdout: "secure_monitoring_stack": { 2024-03-24T03:37:35.283 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "secure_monitoring_stack", 2024-03-24T03:37:35.283 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.283 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.283 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.283 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.283 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.283 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.283 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.283 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-24T03:37:35.284 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.284 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.284 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.284 INFO:teuthology.orchestra.run.smithi078.stdout: "service_discovery_port": { 2024-03-24T03:37:35.284 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "service_discovery_port", 2024-03-24T03:37:35.284 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.284 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.285 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.285 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8765", 2024-03-24T03:37:35.285 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.285 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.285 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.285 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "cephadm service discovery port", 2024-03-24T03:37:35.285 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.285 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.285 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.286 INFO:teuthology.orchestra.run.smithi078.stdout: "ssh_config_file": { 2024-03-24T03:37:35.286 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssh_config_file", 2024-03-24T03:37:35.286 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.286 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.286 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.286 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.286 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.286 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.287 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.287 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-24T03:37:35.287 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.287 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.287 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.287 INFO:teuthology.orchestra.run.smithi078.stdout: "use_agent": { 2024-03-24T03:37:35.287 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "use_agent", 2024-03-24T03:37:35.287 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.288 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.288 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.288 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.288 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.288 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.288 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.288 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-24T03:37:35.288 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.288 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.289 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.289 INFO:teuthology.orchestra.run.smithi078.stdout: "use_repo_digest": { 2024-03-24T03:37:35.289 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "use_repo_digest", 2024-03-24T03:37:35.289 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.289 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.289 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.289 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.289 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.289 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.290 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.290 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-24T03:37:35.290 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.290 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.290 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.290 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_failed_host_check": { 2024-03-24T03:37:35.290 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_failed_host_check", 2024-03-24T03:37:35.290 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.291 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.291 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.291 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.291 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.291 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.291 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.291 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if the host check fails", 2024-03-24T03:37:35.291 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.291 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.292 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.292 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_stray_daemons": { 2024-03-24T03:37:35.292 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_stray_daemons", 2024-03-24T03:37:35.292 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.292 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.292 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.292 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.293 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.293 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.293 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.293 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-24T03:37:35.293 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.293 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.293 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.293 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_stray_hosts": { 2024-03-24T03:37:35.294 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_stray_hosts", 2024-03-24T03:37:35.294 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.294 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.294 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.294 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.294 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.294 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.294 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.294 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-24T03:37:35.295 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.295 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.295 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.295 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.295 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.295 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crash", 2024-03-24T03:37:35.295 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.296 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.296 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.296 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.296 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.296 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.296 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.296 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.296 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.296 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.297 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.297 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.297 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.297 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.297 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.297 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.297 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.297 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.298 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.298 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.298 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.298 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.298 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.298 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.298 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.299 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.299 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.299 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.299 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.299 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.299 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.299 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.299 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.299 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.300 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.300 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.300 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.300 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.300 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.300 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.300 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.300 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.301 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.301 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.301 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.301 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.301 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.301 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.301 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.301 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.302 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.302 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.302 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.302 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.302 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.302 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.302 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.303 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.303 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.303 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.303 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.303 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.303 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.303 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.303 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.303 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: "retain_interval": { 2024-03-24T03:37:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "retain_interval", 2024-03-24T03:37:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "31536000", 2024-03-24T03:37:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-24T03:37:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_recent_interval": { 2024-03-24T03:37:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_recent_interval", 2024-03-24T03:37:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1209600", 2024-03-24T03:37:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-24T03:37:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "dashboard", 2024-03-24T03:37:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-24T03:37:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-24T03:37:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-24T03:37:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: "ALERTMANAGER_API_HOST": { 2024-03-24T03:37:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-24T03:37:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.311 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.311 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.311 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.311 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.311 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.311 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.311 INFO:teuthology.orchestra.run.smithi078.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-24T03:37:35.311 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-24T03:37:35.312 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.312 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.312 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.312 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.312 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.312 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.312 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.312 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.312 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.313 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.313 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.313 INFO:teuthology.orchestra.run.smithi078.stdout: "AUDIT_API_ENABLED": { 2024-03-24T03:37:35.313 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "AUDIT_API_ENABLED", 2024-03-24T03:37:35.313 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.313 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.313 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.313 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.313 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-24T03:37:35.315 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-24T03:37:35.315 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.315 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.315 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.315 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.315 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.315 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.315 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.315 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.315 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.316 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.316 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.316 INFO:teuthology.orchestra.run.smithi078.stdout: "ENABLE_BROWSABLE_API": { 2024-03-24T03:37:35.316 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-24T03:37:35.316 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.316 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.316 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.316 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.317 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.317 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.317 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.317 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.317 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.317 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.317 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.317 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-24T03:37:35.317 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-24T03:37:35.318 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.318 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.318 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.318 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.318 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.318 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.318 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.318 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.318 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.319 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.319 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.319 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-24T03:37:35.319 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-24T03:37:35.319 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.319 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.319 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.319 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.320 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.320 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.320 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.320 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.320 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.320 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.320 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.320 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-24T03:37:35.321 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-24T03:37:35.321 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.321 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.321 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.321 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.321 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.321 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.321 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.321 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.322 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.322 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.322 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.322 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-24T03:37:35.322 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-24T03:37:35.322 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.322 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.325 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.325 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.325 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.325 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.325 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.326 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.326 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.326 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.326 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.326 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-24T03:37:35.326 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-24T03:37:35.326 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.327 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.327 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.327 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.327 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.327 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.327 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.327 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.327 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.327 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.328 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.328 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-24T03:37:35.328 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-24T03:37:35.328 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.328 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.328 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.328 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.328 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.328 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.329 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.329 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.329 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.329 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.329 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.329 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-24T03:37:35.329 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-24T03:37:35.329 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.330 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.330 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.330 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.330 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.330 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.330 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.330 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.330 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.331 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.331 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.331 INFO:teuthology.orchestra.run.smithi078.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-24T03:37:35.331 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-24T03:37:35.331 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.331 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.331 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.331 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.332 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.332 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.332 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.332 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.332 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.332 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.332 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.332 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_PASSWORD": { 2024-03-24T03:37:35.332 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-24T03:37:35.333 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.333 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.333 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.333 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-03-24T03:37:35.333 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.333 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.333 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.333 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.333 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.334 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.334 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.334 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-24T03:37:35.334 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-24T03:37:35.334 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.334 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.334 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.335 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.335 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.335 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.335 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.335 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.335 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.335 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.335 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.336 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_URL": { 2024-03-24T03:37:35.336 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_URL", 2024-03-24T03:37:35.336 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.336 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.336 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.336 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.336 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.336 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.336 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.336 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.337 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.337 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.337 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.337 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_USERNAME": { 2024-03-24T03:37:35.337 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-24T03:37:35.337 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.337 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.337 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.338 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-03-24T03:37:35.338 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.338 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.338 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.338 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.338 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.338 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.338 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.339 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-24T03:37:35.339 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-24T03:37:35.339 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.339 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.339 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.339 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.339 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.339 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.339 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.340 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.340 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.340 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.340 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.340 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-24T03:37:35.340 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-24T03:37:35.340 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.340 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.341 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.341 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.341 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.341 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.341 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.341 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.341 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.341 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.341 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.342 INFO:teuthology.orchestra.run.smithi078.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-24T03:37:35.342 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-24T03:37:35.342 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.342 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.342 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.342 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.342 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.342 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.343 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.343 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.343 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.343 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.343 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.343 INFO:teuthology.orchestra.run.smithi078.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-24T03:37:35.343 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-24T03:37:35.343 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.344 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.344 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.344 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.344 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.344 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.344 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.344 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.344 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.344 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.345 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.345 INFO:teuthology.orchestra.run.smithi078.stdout: "MULTICLUSTER_CONFIG": { 2024-03-24T03:37:35.345 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-24T03:37:35.345 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.345 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.345 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.345 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "{}", 2024-03-24T03:37:35.346 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.346 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.346 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.346 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.346 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.346 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.346 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.347 INFO:teuthology.orchestra.run.smithi078.stdout: "PROMETHEUS_API_HOST": { 2024-03-24T03:37:35.347 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-24T03:37:35.347 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.347 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.347 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.347 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.347 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.347 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.347 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.348 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.348 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.348 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.348 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.348 INFO:teuthology.orchestra.run.smithi078.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-24T03:37:35.348 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-24T03:37:35.348 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.349 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.349 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.349 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.349 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.349 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.349 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.349 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.349 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.349 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.349 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.350 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-24T03:37:35.350 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-24T03:37:35.350 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.350 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.350 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.350 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.350 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.351 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.351 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.351 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.351 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.351 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.351 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.351 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-24T03:37:35.351 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-24T03:37:35.352 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.352 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.352 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.352 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.352 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.352 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.352 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.352 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.352 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.353 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.353 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.353 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-24T03:37:35.353 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-24T03:37:35.353 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.353 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.353 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.353 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.354 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.354 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.354 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.354 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.354 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.354 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.354 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.355 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-24T03:37:35.355 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-24T03:37:35.355 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.355 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.355 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.355 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.355 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.355 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.355 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.356 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.356 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.356 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.356 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.356 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-24T03:37:35.356 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-24T03:37:35.356 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.356 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.357 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.357 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.357 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.357 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.357 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.357 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.357 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.357 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.357 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.358 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-24T03:37:35.358 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-24T03:37:35.358 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.358 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.358 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.358 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.358 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.359 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.359 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.359 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.359 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.359 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.359 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.359 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-24T03:37:35.359 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-24T03:37:35.360 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.360 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.360 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.360 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.360 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.360 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.360 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.360 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.360 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.361 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.361 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.361 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_ENABLED": { 2024-03-24T03:37:35.361 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-24T03:37:35.361 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.361 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.361 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.361 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.361 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.362 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.362 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.362 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.362 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.362 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.362 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.362 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-24T03:37:35.363 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-24T03:37:35.363 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.363 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.363 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.363 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-24T03:37:35.363 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.363 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.363 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.363 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.364 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.364 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.364 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.364 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-24T03:37:35.364 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-24T03:37:35.364 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.364 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.364 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.365 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-24T03:37:35.365 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.365 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.365 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.365 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.365 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.365 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.365 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.365 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-24T03:37:35.366 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-24T03:37:35.366 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.366 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.366 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.366 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8", 2024-03-24T03:37:35.366 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.366 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.366 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.367 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.367 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.367 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.367 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.367 INFO:teuthology.orchestra.run.smithi078.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-24T03:37:35.367 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-24T03:37:35.367 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.367 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.368 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.368 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "45", 2024-03-24T03:37:35.368 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.368 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.368 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.368 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.368 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.368 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.368 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.369 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_ACCESS_KEY": { 2024-03-24T03:37:35.369 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-24T03:37:35.369 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.369 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.369 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.369 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.369 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.369 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.369 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.370 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.370 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.370 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.370 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.370 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-24T03:37:35.370 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-24T03:37:35.370 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.371 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.371 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.371 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-03-24T03:37:35.371 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.371 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.371 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.371 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.371 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.371 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.371 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.372 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_SECRET_KEY": { 2024-03-24T03:37:35.372 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-24T03:37:35.372 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.372 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.372 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.372 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.372 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.372 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.373 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.373 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.373 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.373 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.373 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.373 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_SSL_VERIFY": { 2024-03-24T03:37:35.373 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-24T03:37:35.373 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.374 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.374 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.374 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.374 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.374 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.374 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.374 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.374 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.374 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.375 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.375 INFO:teuthology.orchestra.run.smithi078.stdout: "UNSAFE_TLS_v1_2": { 2024-03-24T03:37:35.375 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-24T03:37:35.375 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.375 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.375 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.375 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.375 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.376 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.376 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.376 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.376 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.376 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.376 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.376 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-24T03:37:35.377 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-24T03:37:35.377 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.377 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.377 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.377 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-24T03:37:35.377 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.377 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.377 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.377 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.378 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.378 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.378 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.378 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-24T03:37:35.378 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-24T03:37:35.378 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.378 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.379 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.379 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-24T03:37:35.379 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.379 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.379 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.379 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.379 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.379 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.380 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.380 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-24T03:37:35.380 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-24T03:37:35.380 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.380 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.380 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.380 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-24T03:37:35.380 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.381 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.381 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.381 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.381 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.381 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.381 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.381 INFO:teuthology.orchestra.run.smithi078.stdout: "cross_origin_url": { 2024-03-24T03:37:35.382 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cross_origin_url", 2024-03-24T03:37:35.382 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.382 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.382 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.382 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.382 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.382 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.382 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.382 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.383 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.383 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.383 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.383 INFO:teuthology.orchestra.run.smithi078.stdout: "crt_file": { 2024-03-24T03:37:35.383 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crt_file", 2024-03-24T03:37:35.383 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.383 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.384 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.384 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.384 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.384 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.384 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.384 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.384 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.384 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.385 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.385 INFO:teuthology.orchestra.run.smithi078.stdout: "debug": { 2024-03-24T03:37:35.385 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "debug", 2024-03-24T03:37:35.385 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.385 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.385 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.385 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.385 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.386 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.386 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.386 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable/disable debug options", 2024-03-24T03:37:35.386 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.386 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.386 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.386 INFO:teuthology.orchestra.run.smithi078.stdout: "jwt_token_ttl": { 2024-03-24T03:37:35.386 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "jwt_token_ttl", 2024-03-24T03:37:35.387 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.387 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.387 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.387 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "28800", 2024-03-24T03:37:35.387 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.387 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.387 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.387 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.388 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.388 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.388 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.388 INFO:teuthology.orchestra.run.smithi078.stdout: "key_file": { 2024-03-24T03:37:35.388 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "key_file", 2024-03-24T03:37:35.388 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.388 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.388 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.389 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.389 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.389 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.389 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.389 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.389 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.389 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.389 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.390 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.390 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.390 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.390 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.390 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.390 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.390 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.390 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.391 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.391 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.391 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.391 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.391 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.391 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.391 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.391 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.391 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.392 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.392 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.392 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.392 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.392 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.392 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.392 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.393 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.393 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.393 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.393 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.393 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.393 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.393 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.393 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.394 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.394 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.394 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.394 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.394 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.394 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.394 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.395 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.395 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.395 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.395 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.395 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.395 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.395 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.395 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.395 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.396 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.396 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.396 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.396 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.396 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.396 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.396 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.397 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.397 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.397 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.397 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.397 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.397 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.397 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.397 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.397 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.398 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.398 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.398 INFO:teuthology.orchestra.run.smithi078.stdout: "motd": { 2024-03-24T03:37:35.398 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "motd", 2024-03-24T03:37:35.398 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.398 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.398 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.398 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.399 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.399 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.399 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.399 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "The message of the day", 2024-03-24T03:37:35.399 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.399 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.399 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.399 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_resolve_ip_addr": { 2024-03-24T03:37:35.400 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "redirect_resolve_ip_addr", 2024-03-24T03:37:35.400 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.400 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.400 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.400 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.400 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.400 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.400 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.400 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.401 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.401 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.401 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.401 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-03-24T03:37:35.401 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-03-24T03:37:35.401 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.401 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.401 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.402 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "::", 2024-03-24T03:37:35.402 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.402 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.402 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.402 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.402 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.402 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.402 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.403 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-03-24T03:37:35.403 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-03-24T03:37:35.403 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.403 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.403 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.403 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8080", 2024-03-24T03:37:35.403 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.403 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.403 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.404 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.404 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.404 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.404 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.404 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl": { 2024-03-24T03:37:35.404 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl", 2024-03-24T03:37:35.404 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.404 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.405 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.405 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.405 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.405 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.405 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.405 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.405 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.405 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.405 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.406 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl_server_port": { 2024-03-24T03:37:35.406 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl_server_port", 2024-03-24T03:37:35.406 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.406 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.406 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.406 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8443", 2024-03-24T03:37:35.406 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.406 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.406 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.407 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.407 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.407 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.407 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.407 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_behaviour": { 2024-03-24T03:37:35.407 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_behaviour", 2024-03-24T03:37:35.407 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.407 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.408 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.408 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "redirect", 2024-03-24T03:37:35.408 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.408 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.408 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.408 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.408 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect" 2024-03-24T03:37:35.408 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.408 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.409 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.409 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.409 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.409 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_error_status_code": { 2024-03-24T03:37:35.409 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_error_status_code", 2024-03-24T03:37:35.409 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.409 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.409 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.410 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "500", 2024-03-24T03:37:35.410 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "400", 2024-03-24T03:37:35.410 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "599", 2024-03-24T03:37:35.410 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.410 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.410 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.410 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.410 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.411 INFO:teuthology.orchestra.run.smithi078.stdout: "url_prefix": { 2024-03-24T03:37:35.411 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "url_prefix", 2024-03-24T03:37:35.411 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.411 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.411 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.411 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.411 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.411 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.411 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.411 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.412 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.412 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.412 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.412 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.412 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.412 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "devicehealth", 2024-03-24T03:37:35.412 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.413 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.413 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.413 INFO:teuthology.orchestra.run.smithi078.stdout: "enable_monitoring": { 2024-03-24T03:37:35.413 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enable_monitoring", 2024-03-24T03:37:35.413 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.413 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.413 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.413 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.413 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.413 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.414 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.414 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "monitor device health metrics", 2024-03-24T03:37:35.414 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.414 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.414 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.414 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.414 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.414 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.415 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.415 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.415 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.415 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.415 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.415 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.415 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.415 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.415 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.416 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.416 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.416 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.416 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.416 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.416 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.416 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.416 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.417 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.417 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.417 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.417 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.417 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.417 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.417 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.417 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.417 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.418 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.418 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.418 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.418 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.418 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.418 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.418 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.418 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.419 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.419 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.419 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.419 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.419 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.419 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.419 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.419 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.420 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.420 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.420 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.420 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.420 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.420 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.420 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.420 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.421 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.421 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.421 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.421 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.421 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.421 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.421 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.421 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.421 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.421 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.422 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.422 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.422 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.422 INFO:teuthology.orchestra.run.smithi078.stdout: "mark_out_threshold": { 2024-03-24T03:37:35.422 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mark_out_threshold", 2024-03-24T03:37:35.422 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:35.422 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.423 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.423 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "2419200", 2024-03-24T03:37:35.423 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.423 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.423 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.423 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-24T03:37:35.423 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.423 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.423 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.424 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": { 2024-03-24T03:37:35.424 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pool_name", 2024-03-24T03:37:35.424 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.424 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.424 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.424 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "device_health_metrics", 2024-03-24T03:37:35.424 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.424 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.425 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.425 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-24T03:37:35.425 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.425 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.425 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.425 INFO:teuthology.orchestra.run.smithi078.stdout: "retention_period": { 2024-03-24T03:37:35.425 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "retention_period", 2024-03-24T03:37:35.425 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:35.426 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.426 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.426 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15552000", 2024-03-24T03:37:35.426 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.426 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.426 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.426 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long to retain device health metrics", 2024-03-24T03:37:35.426 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.426 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.427 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.427 INFO:teuthology.orchestra.run.smithi078.stdout: "scrape_frequency": { 2024-03-24T03:37:35.427 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "scrape_frequency", 2024-03-24T03:37:35.427 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:35.427 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.427 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.427 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "86400", 2024-03-24T03:37:35.427 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.427 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.428 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.428 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-24T03:37:35.428 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.428 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.428 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.428 INFO:teuthology.orchestra.run.smithi078.stdout: "self_heal": { 2024-03-24T03:37:35.428 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "self_heal", 2024-03-24T03:37:35.428 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.429 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.429 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.429 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.429 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.429 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.429 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.429 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-24T03:37:35.429 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.429 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.429 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.430 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-24T03:37:35.430 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-24T03:37:35.430 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:35.430 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.430 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.430 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-24T03:37:35.430 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.430 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.431 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.431 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to wake up and check device health", 2024-03-24T03:37:35.431 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.431 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.431 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.431 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_threshold": { 2024-03-24T03:37:35.431 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_threshold", 2024-03-24T03:37:35.431 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:35.432 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.432 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.432 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "7257600", 2024-03-24T03:37:35.432 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.432 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.432 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.432 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-24T03:37:35.432 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.432 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.433 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.433 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.433 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.433 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "diskprediction_local", 2024-03-24T03:37:35.433 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.433 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.433 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.433 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.434 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.434 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.434 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.434 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.434 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.434 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.434 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.434 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.435 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.435 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.435 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.435 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.435 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.435 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.435 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.435 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.435 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.435 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.436 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.436 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.436 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.436 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.436 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.436 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.436 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.436 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.437 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.437 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.437 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.437 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.437 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.437 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.437 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.437 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.438 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.438 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.438 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.438 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.438 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.438 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.438 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.438 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.438 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.439 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.439 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.439 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.439 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.439 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.439 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.439 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.439 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.439 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.440 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.440 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.440 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.440 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.440 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.440 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.440 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.440 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.440 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.441 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.441 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.441 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.441 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.441 INFO:teuthology.orchestra.run.smithi078.stdout: "predict_interval": { 2024-03-24T03:37:35.441 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "predict_interval", 2024-03-24T03:37:35.441 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.441 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.442 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.442 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "86400", 2024-03-24T03:37:35.442 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.442 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.442 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.442 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.442 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.442 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.442 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.443 INFO:teuthology.orchestra.run.smithi078.stdout: "predictor_model": { 2024-03-24T03:37:35.443 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "predictor_model", 2024-03-24T03:37:35.443 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.443 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.443 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.443 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "prophetstor", 2024-03-24T03:37:35.443 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.443 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.444 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.444 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.444 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.444 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.444 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.444 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-24T03:37:35.444 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-24T03:37:35.444 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.445 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.445 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.445 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-24T03:37:35.445 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.445 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.445 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.445 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.445 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.445 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.446 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.446 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.446 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.446 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "influx", 2024-03-24T03:37:35.446 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": false, 2024-03-24T03:37:35.446 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "influxdb python module not found", 2024-03-24T03:37:35.446 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.447 INFO:teuthology.orchestra.run.smithi078.stdout: "batch_size": { 2024-03-24T03:37:35.447 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "batch_size", 2024-03-24T03:37:35.447 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.447 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.447 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.447 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5000", 2024-03-24T03:37:35.447 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.447 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.447 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.448 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-24T03:37:35.448 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.448 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.448 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.448 INFO:teuthology.orchestra.run.smithi078.stdout: "database": { 2024-03-24T03:37:35.448 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "database", 2024-03-24T03:37:35.448 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.448 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.449 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.449 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "ceph", 2024-03-24T03:37:35.449 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.449 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.449 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.449 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-24T03:37:35.449 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.449 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.449 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.450 INFO:teuthology.orchestra.run.smithi078.stdout: "hostname": { 2024-03-24T03:37:35.450 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "hostname", 2024-03-24T03:37:35.450 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.450 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.450 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.450 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.450 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.450 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.451 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.451 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB server hostname", 2024-03-24T03:37:35.451 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.451 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.451 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.451 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-24T03:37:35.451 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-24T03:37:35.451 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:35.451 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.452 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.452 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "30", 2024-03-24T03:37:35.452 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "5", 2024-03-24T03:37:35.452 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.452 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.452 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-24T03:37:35.452 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.452 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.452 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.453 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.453 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.453 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.453 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.453 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.453 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.453 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.453 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.454 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.454 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.454 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.454 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.454 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.454 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.454 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.454 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.455 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.455 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.455 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.455 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.455 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.455 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.455 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.455 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.455 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.456 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.456 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.456 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.456 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.456 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.456 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.456 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.456 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.457 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.457 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.457 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.457 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.457 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.457 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.457 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.457 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.457 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.457 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.458 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.458 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.458 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.458 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.458 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.458 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.458 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.458 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.458 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.459 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.459 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.459 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.459 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.459 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.459 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.459 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.459 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.460 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.460 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.460 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.460 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.460 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.460 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.460 INFO:teuthology.orchestra.run.smithi078.stdout: "password": { 2024-03-24T03:37:35.461 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "password", 2024-03-24T03:37:35.461 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.461 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.461 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.461 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.461 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.461 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.461 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.461 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "password of InfluxDB server user", 2024-03-24T03:37:35.461 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.462 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.462 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.462 INFO:teuthology.orchestra.run.smithi078.stdout: "port": { 2024-03-24T03:37:35.462 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "port", 2024-03-24T03:37:35.462 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.462 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.462 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.463 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8086", 2024-03-24T03:37:35.463 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.463 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.463 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.463 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB server port", 2024-03-24T03:37:35.463 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.463 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.463 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.464 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl": { 2024-03-24T03:37:35.464 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl", 2024-03-24T03:37:35.464 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.464 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.464 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.464 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "false", 2024-03-24T03:37:35.464 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.464 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.464 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.465 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T03:37:35.465 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.465 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.465 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.465 INFO:teuthology.orchestra.run.smithi078.stdout: "threads": { 2024-03-24T03:37:35.465 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "threads", 2024-03-24T03:37:35.466 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.466 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.466 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.466 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-24T03:37:35.466 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-24T03:37:35.466 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "32", 2024-03-24T03:37:35.466 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.466 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-24T03:37:35.466 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.467 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.467 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.467 INFO:teuthology.orchestra.run.smithi078.stdout: "username": { 2024-03-24T03:37:35.467 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "username", 2024-03-24T03:37:35.467 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.467 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.467 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.467 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.468 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.468 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.468 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.468 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "username of InfluxDB server user", 2024-03-24T03:37:35.468 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.468 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.468 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.468 INFO:teuthology.orchestra.run.smithi078.stdout: "verify_ssl": { 2024-03-24T03:37:35.469 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "verify_ssl", 2024-03-24T03:37:35.469 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.469 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.469 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.469 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "true", 2024-03-24T03:37:35.469 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.469 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.469 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.469 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T03:37:35.470 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.470 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.470 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.470 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.470 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.470 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "insights", 2024-03-24T03:37:35.471 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.471 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.471 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.471 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.471 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.471 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.471 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.471 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.471 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.472 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.472 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.472 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.472 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.472 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.472 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.472 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.472 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.472 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.473 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.473 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.473 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.473 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.473 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.473 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.473 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.473 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.474 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.474 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.474 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.474 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.474 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.474 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.474 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.474 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.474 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.475 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.475 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.475 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.475 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.475 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.475 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.475 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.475 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.476 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.476 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.476 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.476 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.476 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.476 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.476 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.476 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.476 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.477 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.477 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.477 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.477 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.477 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.477 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.477 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.477 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.477 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.478 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.478 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.478 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.478 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.478 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.478 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.478 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.478 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.478 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.479 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.479 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "iostat", 2024-03-24T03:37:35.479 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.479 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.479 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.479 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.479 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.480 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.480 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.480 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.480 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.480 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.480 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.480 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.480 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.480 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.480 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.481 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.481 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.481 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.481 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.481 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.481 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.481 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.481 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.482 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.482 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.482 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.482 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.482 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.482 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.482 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.482 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.482 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.483 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.483 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.483 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.483 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.483 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.483 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.483 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.483 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.484 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.484 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.484 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.484 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.484 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.484 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.484 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.484 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.485 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.485 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.485 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.485 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.485 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.485 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.485 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.485 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.486 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.486 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.486 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.486 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.486 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.486 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.486 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.487 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.487 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.487 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.487 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.487 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.487 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.487 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.487 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.488 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "k8sevents", 2024-03-24T03:37:35.488 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.488 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.488 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.488 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_event_retention_days": { 2024-03-24T03:37:35.488 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ceph_event_retention_days", 2024-03-24T03:37:35.488 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.488 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.489 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.489 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "7", 2024-03-24T03:37:35.489 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.489 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.489 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.489 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-24T03:37:35.489 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.489 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.490 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.490 INFO:teuthology.orchestra.run.smithi078.stdout: "config_check_secs": { 2024-03-24T03:37:35.490 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_check_secs", 2024-03-24T03:37:35.490 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.490 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.490 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.490 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-24T03:37:35.491 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "10", 2024-03-24T03:37:35.491 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.491 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.491 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-24T03:37:35.491 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.491 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.491 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.492 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.492 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.492 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.492 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.492 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.492 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.492 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.492 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.493 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.493 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.493 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.493 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.493 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.493 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.493 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.493 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.494 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.494 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.494 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.494 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.494 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.494 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.494 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.495 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.495 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.495 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.495 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.495 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.495 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.495 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.496 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.496 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.496 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.496 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.496 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.496 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.496 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.496 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.497 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.497 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.497 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.497 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.497 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.497 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.497 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.497 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.498 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.498 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.498 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.498 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.498 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.498 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.498 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.499 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.499 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.499 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.499 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.499 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.499 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.499 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.499 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.499 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.500 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.500 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.500 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.500 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.500 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.500 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.501 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "localpool", 2024-03-24T03:37:35.501 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.501 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.501 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.501 INFO:teuthology.orchestra.run.smithi078.stdout: "failure_domain": { 2024-03-24T03:37:35.501 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "failure_domain", 2024-03-24T03:37:35.501 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.501 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.501 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.502 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "host", 2024-03-24T03:37:35.502 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.502 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.502 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.502 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "failure domain for any created local pool", 2024-03-24T03:37:35.502 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-24T03:37:35.502 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.502 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.503 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.503 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.503 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.503 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.503 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.503 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.503 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.504 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.504 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.504 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.504 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.504 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.504 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.504 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.504 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.504 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.505 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.505 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.505 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.505 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.505 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.505 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.505 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.506 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.506 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.506 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.506 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.506 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.506 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.506 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.506 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.507 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.507 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.507 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.507 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.507 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.507 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.507 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.508 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.508 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.508 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.508 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.508 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.508 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.508 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.508 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.508 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.509 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.509 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.509 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.509 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.509 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.509 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.509 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.509 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.509 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.510 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.510 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.510 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.510 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.510 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.510 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.510 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.510 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.511 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.511 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.511 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": { 2024-03-24T03:37:35.511 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "min_size", 2024-03-24T03:37:35.511 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.511 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.511 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.511 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.512 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.512 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.512 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.512 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default min_size for any created local pool", 2024-03-24T03:37:35.512 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-24T03:37:35.512 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.512 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.512 INFO:teuthology.orchestra.run.smithi078.stdout: "num_rep": { 2024-03-24T03:37:35.513 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "num_rep", 2024-03-24T03:37:35.513 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.513 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.513 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.513 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3", 2024-03-24T03:37:35.513 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.513 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.513 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.513 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default replica count for any created local pool", 2024-03-24T03:37:35.514 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.514 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.514 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.514 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": { 2024-03-24T03:37:35.514 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pg_num", 2024-03-24T03:37:35.514 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.514 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.515 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.515 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "128", 2024-03-24T03:37:35.515 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.515 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.515 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.515 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default pg_num for any created local pool", 2024-03-24T03:37:35.515 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.515 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.516 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.516 INFO:teuthology.orchestra.run.smithi078.stdout: "prefix": { 2024-03-24T03:37:35.516 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prefix", 2024-03-24T03:37:35.516 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.516 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.516 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.516 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.517 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.517 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.517 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.517 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "name prefix for any created local pool", 2024-03-24T03:37:35.517 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.517 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.517 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.517 INFO:teuthology.orchestra.run.smithi078.stdout: "subtree": { 2024-03-24T03:37:35.518 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "subtree", 2024-03-24T03:37:35.518 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.518 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.518 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.518 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "rack", 2024-03-24T03:37:35.518 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.518 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.518 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.518 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-24T03:37:35.519 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-24T03:37:35.519 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.519 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.519 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.519 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.519 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mds_autoscaler", 2024-03-24T03:37:35.520 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.520 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.520 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.520 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.520 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.520 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.520 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.520 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.520 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.520 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.521 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.521 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.521 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.521 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.521 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.521 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.521 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.521 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.521 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.522 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.522 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.522 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.522 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.522 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.522 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.522 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.522 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.523 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.523 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.523 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.523 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.523 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.523 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.523 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.523 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.523 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.524 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.524 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.524 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.524 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.524 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.524 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.524 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.524 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.524 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.525 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.525 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.525 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.525 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.525 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.525 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.525 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.525 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.525 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.526 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.526 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.526 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.526 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.526 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.526 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.526 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.526 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.527 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.527 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.527 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.527 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.527 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.527 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.527 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.527 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.527 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.529 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mirroring", 2024-03-24T03:37:35.529 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.529 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.529 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.529 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.530 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.530 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.530 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.530 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.530 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.530 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.530 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.530 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.530 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.531 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.531 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.531 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.531 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.531 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.531 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.531 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.531 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.531 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.531 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.532 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.532 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.532 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.532 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.532 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.532 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.532 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.532 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.533 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.533 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.533 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.533 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.533 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.533 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.533 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.533 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.534 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.534 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.534 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.534 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.534 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.534 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.534 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.534 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.534 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.535 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.535 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.535 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.535 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.535 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.535 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.535 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.535 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.536 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.536 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.536 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.536 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.536 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.536 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.536 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.536 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.537 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.537 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.537 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.537 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.537 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.537 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.537 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.537 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "nfs", 2024-03-24T03:37:35.538 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.538 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.538 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.538 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.538 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.538 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.538 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.538 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.538 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.539 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.539 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.539 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.539 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.539 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.539 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.539 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.539 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.539 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.540 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.540 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.540 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.540 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.540 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.540 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.540 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.540 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.541 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.541 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.541 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.541 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.541 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.541 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.541 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.541 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.541 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.542 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.542 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.542 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.542 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.542 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.542 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.542 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.542 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.543 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.543 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.543 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.543 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.543 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.543 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.543 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.543 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.543 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.543 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.544 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.544 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.544 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.544 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.544 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.544 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.544 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.544 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.545 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.545 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.545 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.545 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.545 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.545 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.545 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.545 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.545 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.546 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.546 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "orchestrator", 2024-03-24T03:37:35.546 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.546 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.546 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.546 INFO:teuthology.orchestra.run.smithi078.stdout: "fail_fs": { 2024-03-24T03:37:35.546 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "fail_fs", 2024-03-24T03:37:35.547 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.547 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.547 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.547 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.547 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.547 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.547 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.547 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-24T03:37:35.547 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.548 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.548 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.548 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.548 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.548 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.548 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.548 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.548 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.549 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.549 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.549 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.549 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.549 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.549 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.549 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.549 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.549 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.549 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.550 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.550 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.550 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.550 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.550 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.550 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.550 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.550 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.551 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.551 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.551 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.551 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.551 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.551 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.551 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.551 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.551 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.552 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.552 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.552 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.552 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.552 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.552 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.552 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.552 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.552 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.553 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.553 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.553 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.553 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.553 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.553 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.553 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.553 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.553 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.554 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.554 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.554 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.554 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.554 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.554 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.554 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.554 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.555 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.555 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.555 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.555 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.555 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.555 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.555 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.555 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator": { 2024-03-24T03:37:35.556 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "orchestrator", 2024-03-24T03:37:35.556 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.556 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.556 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.556 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.556 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.556 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.556 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.556 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm", 2024-03-24T03:37:35.557 INFO:teuthology.orchestra.run.smithi078.stdout: "rook", 2024-03-24T03:37:35.557 INFO:teuthology.orchestra.run.smithi078.stdout: "test_orchestrator" 2024-03-24T03:37:35.557 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.557 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Orchestrator backend", 2024-03-24T03:37:35.557 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.557 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.557 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.557 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.557 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.558 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "osd_perf_query", 2024-03-24T03:37:35.558 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.558 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.558 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.558 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.558 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.558 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.558 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.559 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.559 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.559 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.559 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.559 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.559 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.559 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.559 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.559 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.560 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.560 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.560 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.560 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.560 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.560 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.560 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.561 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.561 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.561 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.561 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.561 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.561 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.561 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.561 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.561 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.562 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.562 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.562 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.562 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.562 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.562 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.562 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.562 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.563 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.563 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.563 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.563 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.563 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.563 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.563 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.563 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.563 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.564 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.564 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.564 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.564 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.564 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.564 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.564 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.564 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.565 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.565 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.565 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.565 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.565 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.565 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.565 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.565 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.566 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.566 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.566 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.566 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.566 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.566 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.566 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "osd_support", 2024-03-24T03:37:35.567 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.567 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.567 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.567 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.567 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.567 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.567 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.567 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.567 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.568 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.568 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.568 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.568 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.568 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.568 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.568 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.568 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.568 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.569 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.569 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.569 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.569 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.569 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.569 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.569 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.569 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.570 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.570 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.570 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.570 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.570 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.570 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.570 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.570 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.571 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.571 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.571 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.571 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.571 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.571 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.571 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.571 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.572 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.572 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.572 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.572 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.572 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.572 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.572 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.572 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.572 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.572 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.573 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.573 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.573 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.573 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.573 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.573 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.573 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.573 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.574 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.574 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.574 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.574 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.574 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.574 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.574 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.574 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.575 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.575 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.575 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.575 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pg_autoscaler", 2024-03-24T03:37:35.575 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.575 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.575 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.576 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.576 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.576 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.576 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.576 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.576 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.576 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.576 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.576 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.577 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.577 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.577 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.577 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.577 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.577 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.577 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.577 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.577 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.578 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.578 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.578 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.578 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.578 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.578 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.578 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.579 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.579 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.579 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.579 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.579 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.579 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.579 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.579 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.580 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.580 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.580 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.580 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.580 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.580 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.580 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.580 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.581 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.581 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.581 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.581 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.581 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.581 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.581 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.581 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.581 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.582 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.582 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.582 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.582 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.582 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.582 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.582 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.583 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.583 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.583 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.583 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.583 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.583 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.583 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.583 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.584 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.584 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-24T03:37:35.584 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-24T03:37:35.584 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:35.584 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.584 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.584 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-24T03:37:35.584 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.585 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.585 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.585 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.585 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.585 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.585 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.585 INFO:teuthology.orchestra.run.smithi078.stdout: "threshold": { 2024-03-24T03:37:35.585 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "threshold", 2024-03-24T03:37:35.586 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-24T03:37:35.586 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.586 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.586 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3.0", 2024-03-24T03:37:35.586 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1.0", 2024-03-24T03:37:35.586 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.586 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.586 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "scaling threshold", 2024-03-24T03:37:35.586 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-24T03:37:35.587 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.587 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.587 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.587 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.587 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "progress", 2024-03-24T03:37:35.587 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.587 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.588 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.588 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_pg_recovery_event": { 2024-03-24T03:37:35.588 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_pg_recovery_event", 2024-03-24T03:37:35.588 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.588 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.588 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.588 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.588 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.588 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.589 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.589 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-24T03:37:35.589 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.589 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.589 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.589 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": { 2024-03-24T03:37:35.589 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enabled", 2024-03-24T03:37:35.589 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.590 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.590 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.590 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.590 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.590 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.590 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.590 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.590 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.590 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.591 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.591 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.591 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.591 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.591 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.591 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.591 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.591 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.591 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.592 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.592 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.592 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.592 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.592 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.592 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.592 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.592 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.592 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.593 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.593 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.593 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.593 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.593 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.593 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.593 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.593 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.593 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.594 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.594 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.594 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.594 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.594 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.594 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.594 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.594 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.595 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.595 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.595 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.595 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.595 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.595 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.595 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.595 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.595 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.596 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.596 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.596 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.596 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.596 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.596 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.596 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.596 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.596 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.597 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.597 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.597 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.597 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.597 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.597 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.597 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.597 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.597 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.598 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.598 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.598 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.598 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.598 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.598 INFO:teuthology.orchestra.run.smithi078.stdout: "max_completed_events": { 2024-03-24T03:37:35.598 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_completed_events", 2024-03-24T03:37:35.598 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.599 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.599 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.599 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "50", 2024-03-24T03:37:35.599 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.599 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.599 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.599 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "number of past completed events to remember", 2024-03-24T03:37:35.599 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.599 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.600 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.600 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-24T03:37:35.600 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-24T03:37:35.600 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:35.600 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.600 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.600 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-24T03:37:35.600 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.601 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.601 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.601 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long the module is going to sleep", 2024-03-24T03:37:35.601 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.601 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.601 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.601 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.601 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.602 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus", 2024-03-24T03:37:35.602 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.602 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.602 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.602 INFO:teuthology.orchestra.run.smithi078.stdout: "cache": { 2024-03-24T03:37:35.602 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cache", 2024-03-24T03:37:35.602 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.602 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.602 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.603 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.603 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.603 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.603 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.603 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.603 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.603 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.603 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.604 INFO:teuthology.orchestra.run.smithi078.stdout: "exclude_perf_counters": { 2024-03-24T03:37:35.604 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "exclude_perf_counters", 2024-03-24T03:37:35.604 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.604 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.604 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.604 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.604 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.604 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.604 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.605 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-24T03:37:35.605 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-24T03:37:35.605 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.605 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.605 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.605 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.605 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.606 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.606 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.606 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.606 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.606 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.606 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.606 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.606 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.606 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.607 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.607 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.607 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.607 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.607 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.607 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.607 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.607 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.608 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.608 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.608 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.608 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.608 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.608 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.608 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.608 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.608 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.609 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.609 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.609 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.609 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.609 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.609 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.609 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.610 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.610 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.610 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.610 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.610 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.610 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.610 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.610 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.610 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.611 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.611 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.611 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.611 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.611 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.611 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.611 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.611 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.612 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.612 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.612 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.612 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.612 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.612 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.612 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.612 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.613 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.613 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.613 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.613 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.613 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.613 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_stats_pools": { 2024-03-24T03:37:35.613 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_stats_pools", 2024-03-24T03:37:35.613 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.614 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.614 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.614 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.614 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.614 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.614 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.614 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.614 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.615 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.615 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.615 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-24T03:37:35.615 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-24T03:37:35.615 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.615 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.615 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.616 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "300", 2024-03-24T03:37:35.616 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.616 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.616 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.616 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.616 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.616 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.616 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.617 INFO:teuthology.orchestra.run.smithi078.stdout: "scrape_interval": { 2024-03-24T03:37:35.617 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "scrape_interval", 2024-03-24T03:37:35.617 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-24T03:37:35.617 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.617 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.617 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15.0", 2024-03-24T03:37:35.617 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.617 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.617 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.618 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.618 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.618 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.618 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.618 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-03-24T03:37:35.618 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-03-24T03:37:35.619 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.619 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.619 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.619 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "::", 2024-03-24T03:37:35.619 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.619 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.619 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.619 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-24T03:37:35.619 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.620 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.620 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.620 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-03-24T03:37:35.620 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-03-24T03:37:35.620 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.620 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.620 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.620 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "9283", 2024-03-24T03:37:35.621 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.621 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.621 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.621 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-24T03:37:35.621 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.621 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.621 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.621 INFO:teuthology.orchestra.run.smithi078.stdout: "stale_cache_strategy": { 2024-03-24T03:37:35.621 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "stale_cache_strategy", 2024-03-24T03:37:35.622 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.622 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.622 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.622 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "log", 2024-03-24T03:37:35.622 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.622 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.622 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.622 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.622 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.623 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.623 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.623 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_behaviour": { 2024-03-24T03:37:35.623 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_behaviour", 2024-03-24T03:37:35.623 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.623 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.623 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.623 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "default", 2024-03-24T03:37:35.624 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.624 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.624 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.624 INFO:teuthology.orchestra.run.smithi078.stdout: "default", 2024-03-24T03:37:35.624 INFO:teuthology.orchestra.run.smithi078.stdout: "error" 2024-03-24T03:37:35.624 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.624 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.624 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.624 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.625 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.625 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_error_status_code": { 2024-03-24T03:37:35.625 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_error_status_code", 2024-03-24T03:37:35.625 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.625 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.625 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.625 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "500", 2024-03-24T03:37:35.625 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "400", 2024-03-24T03:37:35.626 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "599", 2024-03-24T03:37:35.626 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.626 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.626 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.626 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.626 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.626 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.626 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.627 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_support", 2024-03-24T03:37:35.627 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.627 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.627 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.627 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.627 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.627 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.628 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.628 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.628 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.628 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.628 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.628 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.628 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.628 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.628 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.629 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.629 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.629 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.629 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.629 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.629 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.629 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.629 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.630 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.630 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.630 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.630 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.630 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.630 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.630 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.630 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.631 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.631 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.631 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.631 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.631 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.631 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.631 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.632 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.632 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.632 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.632 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.632 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.632 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.632 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.632 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.633 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.633 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.633 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.633 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.633 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.633 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.633 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.633 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.634 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.634 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.634 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.634 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.634 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.634 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.634 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.635 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.635 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.635 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.635 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.635 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.635 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.635 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.635 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.636 INFO:teuthology.orchestra.run.smithi078.stdout: "max_concurrent_snap_create": { 2024-03-24T03:37:35.636 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_concurrent_snap_create", 2024-03-24T03:37:35.636 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.636 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.636 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.636 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-24T03:37:35.636 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.636 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.637 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.637 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.637 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.637 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.637 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.637 INFO:teuthology.orchestra.run.smithi078.stdout: "mirror_snapshot_schedule": { 2024-03-24T03:37:35.637 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mirror_snapshot_schedule", 2024-03-24T03:37:35.637 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.638 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.638 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.638 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.638 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.638 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.638 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.638 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.638 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.638 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.639 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.639 INFO:teuthology.orchestra.run.smithi078.stdout: "trash_purge_schedule": { 2024-03-24T03:37:35.639 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "trash_purge_schedule", 2024-03-24T03:37:35.639 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.639 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.639 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.639 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.639 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.640 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.640 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.640 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.640 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.640 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.640 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.640 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.640 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.641 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "restful", 2024-03-24T03:37:35.641 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.641 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.641 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.641 INFO:teuthology.orchestra.run.smithi078.stdout: "enable_auth": { 2024-03-24T03:37:35.641 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enable_auth", 2024-03-24T03:37:35.641 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.641 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.641 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.642 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.642 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.642 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.642 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.642 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.642 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.642 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.642 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.643 INFO:teuthology.orchestra.run.smithi078.stdout: "key_file": { 2024-03-24T03:37:35.643 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "key_file", 2024-03-24T03:37:35.643 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.643 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.643 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.643 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.643 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.643 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.643 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.644 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.644 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.644 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.644 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.644 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.644 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.644 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.644 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.645 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.645 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.645 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.645 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.645 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.645 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.645 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.645 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.646 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.646 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.646 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.646 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.646 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.646 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.646 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.646 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.647 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.647 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.647 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.647 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.647 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.647 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.647 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.647 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.647 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.648 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.648 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.648 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.648 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.648 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.648 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.648 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.648 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.649 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.649 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.649 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.649 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.649 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.649 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.649 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.649 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.649 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.650 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.650 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.650 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.650 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.650 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.650 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.650 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.651 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.651 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.651 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.651 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.651 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.651 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.651 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.651 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.652 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.652 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.652 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.652 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.652 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.652 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-03-24T03:37:35.652 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-03-24T03:37:35.652 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.653 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.653 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.653 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.653 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.653 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.653 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.653 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.653 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.653 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.654 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.654 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-03-24T03:37:35.654 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-03-24T03:37:35.654 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.654 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.654 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.654 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.654 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.655 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.655 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.655 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.655 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.655 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.655 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.655 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.655 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.656 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rgw", 2024-03-24T03:37:35.656 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.656 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.656 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.656 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.656 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.656 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.657 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.657 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.657 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.657 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.657 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.657 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.657 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.657 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.657 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.658 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.658 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.658 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.658 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.658 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.658 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.658 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.658 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.659 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.659 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.659 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.659 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.659 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.659 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.659 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.659 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.659 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.660 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.660 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.660 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.660 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.660 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.660 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.660 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.660 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.661 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.661 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.661 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.661 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.661 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.661 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.661 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.661 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.662 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.662 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.662 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.662 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.662 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.662 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.662 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.662 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.663 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.663 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.663 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.663 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.663 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.663 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.663 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.663 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.664 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.664 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.664 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.664 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.664 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.664 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.664 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.665 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rook", 2024-03-24T03:37:35.665 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.665 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.665 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.665 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.665 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.665 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.665 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.666 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.666 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.666 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.666 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.666 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.666 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.666 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.666 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.666 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.667 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.667 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.667 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.667 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.667 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.667 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.667 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.667 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.668 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.668 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.668 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.668 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.668 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.668 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.668 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.668 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.668 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.669 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.669 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.669 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.669 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.669 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.669 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.669 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.669 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.670 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.670 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.670 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.670 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.670 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.670 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.670 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.670 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.670 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.670 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.671 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.671 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.671 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.671 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.671 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.671 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.671 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.671 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.672 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.672 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.672 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.672 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.672 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.672 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.672 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.672 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.672 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.673 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.673 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_class": { 2024-03-24T03:37:35.673 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "storage_class", 2024-03-24T03:37:35.673 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.673 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.673 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.673 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "local", 2024-03-24T03:37:35.673 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.673 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.674 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.674 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-24T03:37:35.674 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.674 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.674 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.674 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.674 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.675 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "selftest", 2024-03-24T03:37:35.675 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.675 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.675 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.675 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.675 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.675 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.675 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.676 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.676 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.676 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.676 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.676 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.676 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.676 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.676 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.676 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.677 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.677 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.677 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.677 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.677 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.677 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.677 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.677 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.678 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.678 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.678 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.678 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.678 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.678 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.678 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.678 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.678 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.679 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.679 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.679 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.679 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.679 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.679 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.679 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.679 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.680 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.680 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.680 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.680 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.680 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.680 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.680 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.680 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.680 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.681 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.681 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.681 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.681 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.681 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.681 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.681 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.681 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.682 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.682 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.682 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.682 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.682 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.682 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.682 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.682 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.683 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.683 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.683 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.683 INFO:teuthology.orchestra.run.smithi078.stdout: "roption1": { 2024-03-24T03:37:35.683 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "roption1", 2024-03-24T03:37:35.683 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.683 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.683 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.684 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.684 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.684 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.684 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.684 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.684 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.684 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.684 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.685 INFO:teuthology.orchestra.run.smithi078.stdout: "roption2": { 2024-03-24T03:37:35.685 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "roption2", 2024-03-24T03:37:35.685 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.685 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.685 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.685 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "xyz", 2024-03-24T03:37:35.685 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.686 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.686 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.686 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.686 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.686 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.686 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.686 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption1": { 2024-03-24T03:37:35.687 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption1", 2024-03-24T03:37:35.687 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.687 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.687 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.687 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.687 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.687 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.687 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.687 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.687 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.688 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.688 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.688 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption2": { 2024-03-24T03:37:35.688 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption2", 2024-03-24T03:37:35.688 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.688 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.688 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.688 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.689 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.689 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.689 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.689 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.689 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.689 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.689 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.689 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption3": { 2024-03-24T03:37:35.690 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption3", 2024-03-24T03:37:35.690 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-24T03:37:35.690 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.690 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.690 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.690 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.690 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.690 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.690 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.691 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.691 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.691 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.691 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption4": { 2024-03-24T03:37:35.691 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption4", 2024-03-24T03:37:35.691 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.691 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.691 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.692 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.692 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.692 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.692 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.692 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.692 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.692 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.692 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.693 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption5": { 2024-03-24T03:37:35.693 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption5", 2024-03-24T03:37:35.693 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.693 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.693 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.693 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.693 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.693 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.693 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.694 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.694 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.694 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.694 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.694 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption6": { 2024-03-24T03:37:35.694 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption6", 2024-03-24T03:37:35.694 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.694 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.695 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.695 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.695 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.695 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.695 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.695 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.695 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.695 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.696 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.696 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption7": { 2024-03-24T03:37:35.696 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption7", 2024-03-24T03:37:35.696 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.696 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.696 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.696 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.696 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-24T03:37:35.697 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "42", 2024-03-24T03:37:35.697 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.697 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.697 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.697 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.697 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.697 INFO:teuthology.orchestra.run.smithi078.stdout: "testkey": { 2024-03-24T03:37:35.697 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testkey", 2024-03-24T03:37:35.697 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.698 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.698 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.698 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.698 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.698 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.698 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.698 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.698 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.698 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.699 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.699 INFO:teuthology.orchestra.run.smithi078.stdout: "testlkey": { 2024-03-24T03:37:35.699 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testlkey", 2024-03-24T03:37:35.699 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.699 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.699 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.699 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.700 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.700 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.700 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.700 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.700 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.700 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.700 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.700 INFO:teuthology.orchestra.run.smithi078.stdout: "testnewline": { 2024-03-24T03:37:35.701 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testnewline", 2024-03-24T03:37:35.701 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.701 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.701 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.701 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.701 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.701 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.701 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.701 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.701 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.702 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.702 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.702 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.702 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.702 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snap_schedule", 2024-03-24T03:37:35.702 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.702 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.703 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.703 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_m_granularity": { 2024-03-24T03:37:35.703 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_m_granularity", 2024-03-24T03:37:35.703 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.703 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.703 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.703 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.703 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.703 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.704 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.704 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow minute scheduled snapshots", 2024-03-24T03:37:35.704 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.704 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.704 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.704 INFO:teuthology.orchestra.run.smithi078.stdout: "dump_on_update": { 2024-03-24T03:37:35.704 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "dump_on_update", 2024-03-24T03:37:35.704 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.705 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.705 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.705 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.705 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.705 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.705 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.705 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "dump database to debug log on update", 2024-03-24T03:37:35.705 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.705 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.706 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.706 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.706 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.706 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.706 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.706 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.706 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.707 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.707 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.707 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.707 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.707 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.707 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.707 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.707 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.707 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.707 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.708 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.708 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.708 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.708 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.708 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.708 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.708 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.709 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.709 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.709 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.709 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.709 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.709 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.709 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.709 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.709 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.710 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.710 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.710 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.710 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.710 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.710 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.710 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.710 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.711 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.711 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.711 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.711 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.711 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.711 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.711 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.711 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.711 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.711 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.712 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.712 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.712 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.712 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.712 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.712 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.712 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.713 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.713 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.713 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.713 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.713 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.713 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.713 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.713 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.713 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.714 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.714 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.714 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "stats", 2024-03-24T03:37:35.714 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.714 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.714 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.715 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.715 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.715 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.715 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.715 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.715 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.715 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.715 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.716 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.716 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.716 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.716 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.716 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.716 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.716 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.716 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.716 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.717 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.717 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.717 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.717 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.717 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.717 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.717 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.717 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.718 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.718 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.718 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.718 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.718 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.718 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.718 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.718 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.719 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.719 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.719 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.719 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.719 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.719 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.719 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.719 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.720 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.720 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.720 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.720 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.720 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.720 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.720 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.720 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.720 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.721 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.721 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.721 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.721 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.721 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.721 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.721 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.722 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.722 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.722 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.722 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.722 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.722 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.722 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.722 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.723 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.723 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.723 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.723 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "status", 2024-03-24T03:37:35.723 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.723 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.724 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.724 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.724 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.724 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.724 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.724 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.724 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.724 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.724 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.725 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.725 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.725 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.725 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.725 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.725 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.725 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.725 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.726 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.726 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.726 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.726 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.726 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.726 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.726 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.727 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.727 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.727 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.727 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.727 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.727 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.727 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.727 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.727 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.728 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.728 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.728 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.728 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.728 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.728 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.728 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.728 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.729 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.729 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.729 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.729 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.729 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.729 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.729 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.729 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.730 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.730 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.730 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.730 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.730 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.733 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.733 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.733 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.734 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.734 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.734 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.734 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.734 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.734 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.734 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.734 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.734 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.735 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.735 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.735 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.735 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "telegraf", 2024-03-24T03:37:35.735 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.735 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.735 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.736 INFO:teuthology.orchestra.run.smithi078.stdout: "address": { 2024-03-24T03:37:35.736 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "address", 2024-03-24T03:37:35.736 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.736 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.736 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.736 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-24T03:37:35.736 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.736 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.737 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.737 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.737 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.737 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.737 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.737 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-24T03:37:35.737 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-24T03:37:35.737 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:35.738 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.738 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.738 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15", 2024-03-24T03:37:35.738 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.738 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.738 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.738 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.738 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.738 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.739 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.739 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.739 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.739 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.739 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.739 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.739 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.740 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.740 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.740 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.740 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.740 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.740 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.740 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.740 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.740 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.740 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.741 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.741 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.741 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.741 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.741 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.741 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.741 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.741 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.742 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.742 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.742 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.742 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.742 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.742 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.742 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.742 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.742 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.743 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.743 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.743 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.743 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.743 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.743 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.743 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.743 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.744 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.744 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.744 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.744 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.744 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.744 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.744 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.744 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.744 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.745 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.745 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.745 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.745 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.745 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.745 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.745 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.745 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.746 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.746 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.746 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.746 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.746 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.746 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.746 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.746 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.746 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.747 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.747 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "telemetry", 2024-03-24T03:37:35.747 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.747 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.747 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.747 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_basic": { 2024-03-24T03:37:35.747 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_basic", 2024-03-24T03:37:35.748 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.748 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.748 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.748 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.748 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.748 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.748 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.748 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-24T03:37:35.748 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.749 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.749 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.749 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_crash": { 2024-03-24T03:37:35.749 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_crash", 2024-03-24T03:37:35.749 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.749 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.749 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.749 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.749 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.750 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.750 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.750 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-24T03:37:35.750 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.750 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.750 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.750 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_device": { 2024-03-24T03:37:35.750 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_device", 2024-03-24T03:37:35.751 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.751 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.751 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.751 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.751 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.751 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.751 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.751 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-24T03:37:35.751 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.752 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.752 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.752 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_ident": { 2024-03-24T03:37:35.752 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_ident", 2024-03-24T03:37:35.752 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.752 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.752 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.752 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.752 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.753 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.753 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.753 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-24T03:37:35.753 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.753 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.753 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.753 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_perf": { 2024-03-24T03:37:35.754 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_perf", 2024-03-24T03:37:35.754 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.754 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.754 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.754 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.754 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.754 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.754 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.755 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-24T03:37:35.755 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.755 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.755 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.755 INFO:teuthology.orchestra.run.smithi078.stdout: "contact": { 2024-03-24T03:37:35.755 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "contact", 2024-03-24T03:37:35.755 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.755 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.756 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.756 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.756 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.756 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.756 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.756 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.756 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.756 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.756 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.757 INFO:teuthology.orchestra.run.smithi078.stdout: "description": { 2024-03-24T03:37:35.757 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "description", 2024-03-24T03:37:35.757 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.757 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.757 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.757 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.757 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.757 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.758 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.758 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.758 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.758 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.758 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.758 INFO:teuthology.orchestra.run.smithi078.stdout: "device_url": { 2024-03-24T03:37:35.758 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_url", 2024-03-24T03:37:35.758 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.759 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.759 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.759 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-24T03:37:35.759 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.759 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.759 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.759 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.759 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.759 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.760 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.760 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": { 2024-03-24T03:37:35.760 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enabled", 2024-03-24T03:37:35.760 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.760 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.760 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.760 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.760 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.761 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.761 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.761 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.761 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.761 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.761 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.761 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-24T03:37:35.761 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-24T03:37:35.762 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.762 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.762 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.762 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "24", 2024-03-24T03:37:35.762 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "8", 2024-03-24T03:37:35.762 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.762 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.762 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.762 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.763 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.763 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.763 INFO:teuthology.orchestra.run.smithi078.stdout: "last_opt_revision": { 2024-03-24T03:37:35.763 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "last_opt_revision", 2024-03-24T03:37:35.763 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.763 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.763 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.763 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1", 2024-03-24T03:37:35.764 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.764 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.764 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.764 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.764 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.764 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.764 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.764 INFO:teuthology.orchestra.run.smithi078.stdout: "leaderboard": { 2024-03-24T03:37:35.765 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "leaderboard", 2024-03-24T03:37:35.765 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.765 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.765 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.765 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.765 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.765 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.765 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.765 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.766 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.766 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.766 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.766 INFO:teuthology.orchestra.run.smithi078.stdout: "leaderboard_description": { 2024-03-24T03:37:35.766 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "leaderboard_description", 2024-03-24T03:37:35.766 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.766 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.766 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.767 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.767 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.767 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.767 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.767 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.767 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.767 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.767 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.768 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.768 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.768 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.768 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.768 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.768 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.768 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.768 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.768 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.769 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.769 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.769 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.769 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.769 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.769 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.769 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.769 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.769 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.770 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.770 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.770 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.770 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.770 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.770 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.770 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.770 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.770 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.771 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.771 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.771 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.771 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.771 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.771 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.771 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.771 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.772 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.772 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.772 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.772 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.772 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.772 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.772 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.772 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.772 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.773 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.773 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.773 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.773 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.773 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.773 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.773 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.773 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.773 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.774 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.774 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.774 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.774 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.774 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.774 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.774 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.774 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.775 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.775 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.775 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.775 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.775 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.775 INFO:teuthology.orchestra.run.smithi078.stdout: "organization": { 2024-03-24T03:37:35.775 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "organization", 2024-03-24T03:37:35.775 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.775 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.776 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.776 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.776 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.776 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.776 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.776 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.776 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.776 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.776 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.777 INFO:teuthology.orchestra.run.smithi078.stdout: "proxy": { 2024-03-24T03:37:35.777 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "proxy", 2024-03-24T03:37:35.777 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.777 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.777 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.777 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.777 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.777 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.777 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.778 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.778 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.778 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.778 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.778 INFO:teuthology.orchestra.run.smithi078.stdout: "url": { 2024-03-24T03:37:35.778 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "url", 2024-03-24T03:37:35.778 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.778 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.779 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.779 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-24T03:37:35.779 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.779 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.779 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.779 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.779 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.779 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.779 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.780 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.780 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.780 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test_orchestrator", 2024-03-24T03:37:35.780 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.780 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.780 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.780 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.781 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.781 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.781 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.781 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.781 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.781 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.781 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.781 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.781 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.782 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.782 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.782 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.782 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.782 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.782 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.782 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.782 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.782 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.783 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.783 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.783 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.783 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.783 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.783 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.783 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.783 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.784 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.784 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.784 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.784 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.784 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.784 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.784 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.784 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.785 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.785 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.785 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.785 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.785 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.785 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.785 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.785 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.786 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.786 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.786 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.786 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.786 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.786 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.786 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.786 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.786 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.787 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.787 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.787 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.787 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.787 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.787 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.787 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.787 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.787 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.788 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.788 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.788 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.788 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.788 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.788 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.788 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.789 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "volumes", 2024-03-24T03:37:35.789 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.789 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.789 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.789 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.789 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.789 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.789 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.789 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.790 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.790 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.790 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.790 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.790 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.790 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.790 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.790 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.790 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.791 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.791 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.791 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.791 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.791 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.791 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.791 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.791 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.791 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.792 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.792 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.792 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.792 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.792 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.792 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.792 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.792 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.792 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.793 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.793 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.793 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.793 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.793 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.793 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.793 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.793 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.794 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.794 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.794 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.794 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.794 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.794 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.794 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.794 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.794 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.795 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.795 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.795 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.795 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.795 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.795 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.795 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.795 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.796 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.796 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.796 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.796 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.796 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.796 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.796 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.796 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.796 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.797 INFO:teuthology.orchestra.run.smithi078.stdout: "max_concurrent_clones": { 2024-03-24T03:37:35.797 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_concurrent_clones", 2024-03-24T03:37:35.797 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.797 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.797 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.797 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "4", 2024-03-24T03:37:35.797 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.797 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.797 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.798 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-24T03:37:35.798 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.798 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.798 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.798 INFO:teuthology.orchestra.run.smithi078.stdout: "periodic_async_work": { 2024-03-24T03:37:35.798 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "periodic_async_work", 2024-03-24T03:37:35.798 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.798 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.798 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.799 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.799 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.799 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.799 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.799 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Periodically check for async work", 2024-03-24T03:37:35.799 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.799 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.799 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.800 INFO:teuthology.orchestra.run.smithi078.stdout: "snapshot_clone_delay": { 2024-03-24T03:37:35.800 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snapshot_clone_delay", 2024-03-24T03:37:35.800 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.800 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.800 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.800 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-24T03:37:35.800 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.800 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.800 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.801 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-24T03:37:35.801 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.801 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.801 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.801 INFO:teuthology.orchestra.run.smithi078.stdout: "snapshot_clone_no_wait": { 2024-03-24T03:37:35.801 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snapshot_clone_no_wait", 2024-03-24T03:37:35.801 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.801 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.802 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.802 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.802 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.802 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.802 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.802 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-24T03:37:35.802 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.802 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.802 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.803 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.803 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.803 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix", 2024-03-24T03:37:35.803 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.803 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.803 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.803 INFO:teuthology.orchestra.run.smithi078.stdout: "discovery_interval": { 2024-03-24T03:37:35.803 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "discovery_interval", 2024-03-24T03:37:35.804 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-24T03:37:35.804 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.804 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.804 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "100", 2024-03-24T03:37:35.804 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.804 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.804 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.804 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.804 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.805 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.805 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.805 INFO:teuthology.orchestra.run.smithi078.stdout: "identifier": { 2024-03-24T03:37:35.805 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "identifier", 2024-03-24T03:37:35.805 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.805 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.805 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.805 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.806 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.806 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.806 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.806 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.806 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.806 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.806 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.806 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-24T03:37:35.807 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-24T03:37:35.807 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:35.807 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.807 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.807 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-24T03:37:35.807 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.807 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.807 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.807 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.808 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.808 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.808 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.808 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.808 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.808 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.808 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.808 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.809 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.809 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.809 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.809 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.809 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.809 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.809 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.809 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.809 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.810 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.810 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.810 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.810 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.810 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.810 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.810 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.810 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.811 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.811 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.811 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.811 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.811 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.811 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.811 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.811 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.811 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.812 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.812 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.812 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.812 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.812 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.812 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.812 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.812 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.812 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.813 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.813 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.813 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.813 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.813 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.813 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.813 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.813 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.813 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.814 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.814 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.814 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.814 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.814 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.814 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.814 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.814 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.815 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.815 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.815 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.815 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.815 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.815 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.815 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.815 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.815 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.816 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_host": { 2024-03-24T03:37:35.816 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_host", 2024-03-24T03:37:35.816 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.816 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.816 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.816 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.816 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.816 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.816 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.817 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.817 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.817 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.817 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.817 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_port": { 2024-03-24T03:37:35.817 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_port", 2024-03-24T03:37:35.817 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.817 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.818 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.818 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10051", 2024-03-24T03:37:35.818 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.818 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.818 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.818 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.818 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.818 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.818 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.819 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_sender": { 2024-03-24T03:37:35.819 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_sender", 2024-03-24T03:37:35.819 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.819 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.819 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.819 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-24T03:37:35.819 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.819 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.820 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.820 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.820 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.820 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.820 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.820 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.820 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.820 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.820 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-24T03:37:35.820 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.821 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.821 INFO:teuthology.orchestra.run.smithi078.stdout: "modules": [ 2024-03-24T03:37:35.821 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm", 2024-03-24T03:37:35.821 INFO:teuthology.orchestra.run.smithi078.stdout: "dashboard", 2024-03-24T03:37:35.821 INFO:teuthology.orchestra.run.smithi078.stdout: "iostat", 2024-03-24T03:37:35.821 INFO:teuthology.orchestra.run.smithi078.stdout: "nfs", 2024-03-24T03:37:35.821 INFO:teuthology.orchestra.run.smithi078.stdout: "restful", 2024-03-24T03:37:35.821 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_schedule" 2024-03-24T03:37:35.821 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.822 INFO:teuthology.orchestra.run.smithi078.stdout: "available_modules": [ 2024-03-24T03:37:35.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.822 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "alerts", 2024-03-24T03:37:35.822 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.822 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.822 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.822 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-24T03:37:35.822 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-24T03:37:35.822 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:35.823 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.823 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.823 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-24T03:37:35.823 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.823 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.823 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.823 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How frequently to reexamine health status", 2024-03-24T03:37:35.823 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.823 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.824 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.824 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.824 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.824 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.824 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.824 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.824 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.824 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.825 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.825 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.825 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.825 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.825 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.825 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.825 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.825 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.826 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.826 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.826 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.826 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.826 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.826 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.826 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.826 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.827 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.827 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.827 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.827 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.827 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.827 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.827 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.827 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.827 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.827 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.828 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.828 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.828 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.828 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.828 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.828 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.828 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.828 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.829 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.829 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.829 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.829 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.829 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.829 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.829 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.829 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.829 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.830 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.830 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.830 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.830 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.830 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.830 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.830 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.830 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.831 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.831 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.831 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.831 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.831 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.831 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.831 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.831 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.831 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_destination": { 2024-03-24T03:37:35.832 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_destination", 2024-03-24T03:37:35.832 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.832 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.832 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.832 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.832 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.832 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.832 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.832 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Email address to send alerts to", 2024-03-24T03:37:35.833 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.833 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.833 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.833 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_from_name": { 2024-03-24T03:37:35.833 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_from_name", 2024-03-24T03:37:35.833 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.833 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.833 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.834 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "Ceph", 2024-03-24T03:37:35.834 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.834 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.834 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.834 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Email From: name", 2024-03-24T03:37:35.834 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.834 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.834 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.835 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_host": { 2024-03-24T03:37:35.835 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_host", 2024-03-24T03:37:35.835 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.835 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.835 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.835 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.835 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.835 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.835 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.836 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP server", 2024-03-24T03:37:35.836 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.836 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.836 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.836 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_password": { 2024-03-24T03:37:35.836 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_password", 2024-03-24T03:37:35.836 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.836 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.836 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.837 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.837 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.837 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.837 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.837 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Password to authenticate with", 2024-03-24T03:37:35.837 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.837 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.837 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.838 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_port": { 2024-03-24T03:37:35.838 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_port", 2024-03-24T03:37:35.838 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.838 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.838 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.838 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "465", 2024-03-24T03:37:35.838 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.838 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.838 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.839 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP port", 2024-03-24T03:37:35.839 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.839 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.839 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.839 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_sender": { 2024-03-24T03:37:35.839 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_sender", 2024-03-24T03:37:35.839 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.839 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.840 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.840 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.840 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.840 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.840 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.840 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP envelope sender", 2024-03-24T03:37:35.840 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.840 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.840 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.841 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_ssl": { 2024-03-24T03:37:35.841 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_ssl", 2024-03-24T03:37:35.841 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.841 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.841 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.841 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.841 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.841 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.842 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.842 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-24T03:37:35.842 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.842 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.842 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.842 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_user": { 2024-03-24T03:37:35.842 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_user", 2024-03-24T03:37:35.842 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.843 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.843 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.843 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.843 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.843 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.843 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.843 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "User to authenticate as", 2024-03-24T03:37:35.843 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.843 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.844 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.844 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.844 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.844 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "balancer", 2024-03-24T03:37:35.844 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.844 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.844 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.845 INFO:teuthology.orchestra.run.smithi078.stdout: "active": { 2024-03-24T03:37:35.845 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "active", 2024-03-24T03:37:35.845 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.845 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.845 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.845 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.845 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.845 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.845 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.846 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "automatically balance PGs across cluster", 2024-03-24T03:37:35.846 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.846 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.846 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.846 INFO:teuthology.orchestra.run.smithi078.stdout: "begin_time": { 2024-03-24T03:37:35.846 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "begin_time", 2024-03-24T03:37:35.846 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.846 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.847 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.847 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0000", 2024-03-24T03:37:35.847 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.847 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.847 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.847 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "beginning time of day to automatically balance", 2024-03-24T03:37:35.847 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T03:37:35.847 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.847 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.848 INFO:teuthology.orchestra.run.smithi078.stdout: "begin_weekday": { 2024-03-24T03:37:35.848 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "begin_weekday", 2024-03-24T03:37:35.848 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-24T03:37:35.848 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.848 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.848 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-24T03:37:35.848 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0", 2024-03-24T03:37:35.848 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "6", 2024-03-24T03:37:35.848 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.849 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-24T03:37:35.849 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T03:37:35.849 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.849 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.849 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_max_iterations": { 2024-03-24T03:37:35.849 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_max_iterations", 2024-03-24T03:37:35.849 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-24T03:37:35.849 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.850 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.850 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "25", 2024-03-24T03:37:35.850 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-24T03:37:35.850 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "250", 2024-03-24T03:37:35.850 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.850 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-24T03:37:35.850 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.850 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.850 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.851 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_metrics": { 2024-03-24T03:37:35.851 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_metrics", 2024-03-24T03:37:35.851 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.851 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.851 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.851 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "pgs,objects,bytes", 2024-03-24T03:37:35.851 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.852 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.852 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.852 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-24T03:37:35.852 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-24T03:37:35.852 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.852 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.852 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_step": { 2024-03-24T03:37:35.852 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_step", 2024-03-24T03:37:35.853 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-24T03:37:35.853 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.853 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.853 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0.5", 2024-03-24T03:37:35.853 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0.001", 2024-03-24T03:37:35.853 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "0.999", 2024-03-24T03:37:35.853 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.853 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "aggressiveness of optimization", 2024-03-24T03:37:35.853 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-24T03:37:35.853 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.854 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.854 INFO:teuthology.orchestra.run.smithi078.stdout: "end_time": { 2024-03-24T03:37:35.854 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "end_time", 2024-03-24T03:37:35.854 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.854 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.854 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.854 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "2359", 2024-03-24T03:37:35.855 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.855 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.855 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.855 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "ending time of day to automatically balance", 2024-03-24T03:37:35.855 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T03:37:35.855 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.855 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.855 INFO:teuthology.orchestra.run.smithi078.stdout: "end_weekday": { 2024-03-24T03:37:35.856 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "end_weekday", 2024-03-24T03:37:35.856 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-24T03:37:35.856 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.856 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.856 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-24T03:37:35.856 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0", 2024-03-24T03:37:35.856 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "6", 2024-03-24T03:37:35.856 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.856 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-24T03:37:35.857 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T03:37:35.857 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.857 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.857 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.857 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.857 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.857 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.858 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.858 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.858 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.858 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.858 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.858 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.858 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.858 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.858 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.859 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.859 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.859 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.859 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.859 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.859 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.859 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.859 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.860 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.860 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.860 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.860 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.860 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.860 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.860 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.860 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.860 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.861 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.861 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.861 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.861 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.861 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.861 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.861 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.862 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.862 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.862 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.862 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.862 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.862 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.862 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.862 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.862 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.863 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.863 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.863 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.863 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.863 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.863 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.863 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.864 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.864 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.864 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.864 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.864 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.864 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.864 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.864 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.864 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.865 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.865 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.865 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.865 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.865 INFO:teuthology.orchestra.run.smithi078.stdout: "min_score": { 2024-03-24T03:37:35.865 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "min_score", 2024-03-24T03:37:35.865 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-24T03:37:35.865 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.866 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.866 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-24T03:37:35.866 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.866 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.866 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.866 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-24T03:37:35.866 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.866 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.866 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.867 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": { 2024-03-24T03:37:35.867 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mode", 2024-03-24T03:37:35.867 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.867 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.867 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.867 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "upmap", 2024-03-24T03:37:35.867 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.867 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.867 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.868 INFO:teuthology.orchestra.run.smithi078.stdout: "crush-compat", 2024-03-24T03:37:35.868 INFO:teuthology.orchestra.run.smithi078.stdout: "none", 2024-03-24T03:37:35.868 INFO:teuthology.orchestra.run.smithi078.stdout: "read", 2024-03-24T03:37:35.868 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap", 2024-03-24T03:37:35.868 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap-read" 2024-03-24T03:37:35.868 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.868 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Balancer mode", 2024-03-24T03:37:35.868 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.868 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.869 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.869 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_ids": { 2024-03-24T03:37:35.869 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pool_ids", 2024-03-24T03:37:35.869 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.869 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.869 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.869 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.869 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.870 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.870 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.870 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-24T03:37:35.870 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.870 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.870 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.870 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-24T03:37:35.870 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-24T03:37:35.871 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:35.871 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.871 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.871 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-24T03:37:35.871 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.871 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.871 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.871 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-24T03:37:35.871 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.872 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.872 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.872 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap_max_deviation": { 2024-03-24T03:37:35.872 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "upmap_max_deviation", 2024-03-24T03:37:35.872 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.872 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.872 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.872 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-24T03:37:35.873 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-24T03:37:35.873 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.873 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.873 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-24T03:37:35.873 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-24T03:37:35.873 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.873 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.873 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap_max_optimizations": { 2024-03-24T03:37:35.874 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "upmap_max_optimizations", 2024-03-24T03:37:35.874 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-24T03:37:35.874 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.874 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.874 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-24T03:37:35.874 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.874 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.874 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.875 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-24T03:37:35.875 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.875 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.875 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.875 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.875 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.875 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cephadm", 2024-03-24T03:37:35.876 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.876 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.876 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.876 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_down_multiplier": { 2024-03-24T03:37:35.876 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_down_multiplier", 2024-03-24T03:37:35.876 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-24T03:37:35.876 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.876 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.876 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3.0", 2024-03-24T03:37:35.876 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.877 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.877 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.877 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-24T03:37:35.877 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.877 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.877 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.877 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_refresh_rate": { 2024-03-24T03:37:35.878 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_refresh_rate", 2024-03-24T03:37:35.878 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:35.878 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.878 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.878 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "20", 2024-03-24T03:37:35.878 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.878 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.878 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.879 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-24T03:37:35.879 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.879 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.879 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.879 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_starting_port": { 2024-03-24T03:37:35.879 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_starting_port", 2024-03-24T03:37:35.879 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.879 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.880 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.880 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "4721", 2024-03-24T03:37:35.880 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.880 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.880 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.880 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-24T03:37:35.880 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.880 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.880 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.881 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_ptrace": { 2024-03-24T03:37:35.881 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_ptrace", 2024-03-24T03:37:35.881 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.881 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.881 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.881 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.881 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.881 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.881 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.882 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-24T03:37:35.882 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-24T03:37:35.882 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.882 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.882 INFO:teuthology.orchestra.run.smithi078.stdout: "autotune_interval": { 2024-03-24T03:37:35.882 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "autotune_interval", 2024-03-24T03:37:35.882 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:35.882 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.883 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.883 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-24T03:37:35.883 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.883 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.883 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.883 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-24T03:37:35.883 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.883 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.883 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.884 INFO:teuthology.orchestra.run.smithi078.stdout: "autotune_memory_target_ratio": { 2024-03-24T03:37:35.884 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "autotune_memory_target_ratio", 2024-03-24T03:37:35.884 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-24T03:37:35.884 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.884 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.884 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0.7", 2024-03-24T03:37:35.884 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.884 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.885 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.885 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-24T03:37:35.885 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.885 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.885 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.885 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm_log_destination": { 2024-03-24T03:37:35.885 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cephadm_log_destination", 2024-03-24T03:37:35.885 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.885 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.886 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.886 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.886 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.886 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.886 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.886 INFO:teuthology.orchestra.run.smithi078.stdout: "file", 2024-03-24T03:37:35.886 INFO:teuthology.orchestra.run.smithi078.stdout: "file,syslog", 2024-03-24T03:37:35.886 INFO:teuthology.orchestra.run.smithi078.stdout: "syslog" 2024-03-24T03:37:35.887 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.887 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-24T03:37:35.887 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.887 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.887 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.887 INFO:teuthology.orchestra.run.smithi078.stdout: "cgroups_split": { 2024-03-24T03:37:35.887 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cgroups_split", 2024-03-24T03:37:35.887 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.887 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.888 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.888 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.888 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.888 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.888 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.888 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-24T03:37:35.888 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.888 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.888 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.889 INFO:teuthology.orchestra.run.smithi078.stdout: "config_checks_enabled": { 2024-03-24T03:37:35.889 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_checks_enabled", 2024-03-24T03:37:35.889 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.889 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.889 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.889 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.889 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.889 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.890 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.890 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-24T03:37:35.890 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.890 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.890 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.890 INFO:teuthology.orchestra.run.smithi078.stdout: "config_dashboard": { 2024-03-24T03:37:35.890 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_dashboard", 2024-03-24T03:37:35.890 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.891 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.891 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.891 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.891 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.891 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.891 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.891 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-24T03:37:35.891 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.891 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.891 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.892 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_alertmanager": { 2024-03-24T03:37:35.892 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_alertmanager", 2024-03-24T03:37:35.892 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.892 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.892 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.892 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-24T03:37:35.892 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.892 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.893 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.893 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-24T03:37:35.893 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.893 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.893 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.893 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_base": { 2024-03-24T03:37:35.893 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_base", 2024-03-24T03:37:35.893 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.893 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.894 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.894 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-24T03:37:35.894 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.894 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.894 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.894 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Container image name, without the tag", 2024-03-24T03:37:35.894 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.894 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.895 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.895 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_elasticsearch": { 2024-03-24T03:37:35.895 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_elasticsearch", 2024-03-24T03:37:35.895 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.895 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.895 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.895 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-24T03:37:35.895 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.896 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.896 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.896 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "elasticsearch container image", 2024-03-24T03:37:35.896 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.896 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.896 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.896 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_grafana": { 2024-03-24T03:37:35.896 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_grafana", 2024-03-24T03:37:35.897 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.897 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.897 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.897 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-24T03:37:35.897 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.897 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.897 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.897 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-24T03:37:35.897 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.897 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.898 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.898 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_haproxy": { 2024-03-24T03:37:35.898 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_haproxy", 2024-03-24T03:37:35.898 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.898 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.898 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.898 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-24T03:37:35.899 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.899 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.899 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.899 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "HAproxy container image", 2024-03-24T03:37:35.899 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.899 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.899 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.900 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_agent": { 2024-03-24T03:37:35.900 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_agent", 2024-03-24T03:37:35.900 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.900 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.900 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.900 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-24T03:37:35.900 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.901 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.901 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.901 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger agent container image", 2024-03-24T03:37:35.901 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.901 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.902 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.902 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_collector": { 2024-03-24T03:37:35.902 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_collector", 2024-03-24T03:37:35.902 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.902 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.903 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.903 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-24T03:37:35.903 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.903 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.903 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.903 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger collector container image", 2024-03-24T03:37:35.903 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.903 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.903 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.904 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_query": { 2024-03-24T03:37:35.904 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_query", 2024-03-24T03:37:35.904 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.904 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.904 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.904 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-24T03:37:35.904 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.905 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.905 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.905 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger query container image", 2024-03-24T03:37:35.905 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.905 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.905 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.905 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_keepalived": { 2024-03-24T03:37:35.905 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_keepalived", 2024-03-24T03:37:35.906 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.906 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.906 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.906 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-24T03:37:35.906 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.906 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.906 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.906 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Keepalived container image", 2024-03-24T03:37:35.907 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.907 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.907 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.907 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_loki": { 2024-03-24T03:37:35.907 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_loki", 2024-03-24T03:37:35.907 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.907 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.908 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.908 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-24T03:37:35.908 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.908 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.908 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.908 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Loki container image", 2024-03-24T03:37:35.908 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.908 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.909 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.909 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_node_exporter": { 2024-03-24T03:37:35.909 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_node_exporter", 2024-03-24T03:37:35.909 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.909 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.909 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.909 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-24T03:37:35.909 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.910 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.910 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.910 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-24T03:37:35.910 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.910 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.910 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.910 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_nvmeof": { 2024-03-24T03:37:35.910 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_nvmeof", 2024-03-24T03:37:35.911 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.911 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.911 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.911 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-24T03:37:35.911 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.911 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.911 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.911 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Nvme-of container image", 2024-03-24T03:37:35.911 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.912 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.912 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.912 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_prometheus": { 2024-03-24T03:37:35.912 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_prometheus", 2024-03-24T03:37:35.912 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.912 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.912 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.912 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-24T03:37:35.912 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.913 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.913 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.913 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-24T03:37:35.913 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.913 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.913 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.913 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_promtail": { 2024-03-24T03:37:35.914 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_promtail", 2024-03-24T03:37:35.914 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.914 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.914 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.914 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-24T03:37:35.914 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.914 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.914 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.915 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Promtail container image", 2024-03-24T03:37:35.915 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.915 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.915 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.915 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_snmp_gateway": { 2024-03-24T03:37:35.915 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_snmp_gateway", 2024-03-24T03:37:35.915 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.916 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.916 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.916 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-24T03:37:35.916 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.916 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.916 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.916 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SNMP Gateway container image", 2024-03-24T03:37:35.916 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.916 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.917 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.917 INFO:teuthology.orchestra.run.smithi078.stdout: "container_init": { 2024-03-24T03:37:35.917 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_init", 2024-03-24T03:37:35.917 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.917 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.917 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.917 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.917 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.918 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.918 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.918 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Run podman/docker with `--init`", 2024-03-24T03:37:35.918 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.918 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.918 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.918 INFO:teuthology.orchestra.run.smithi078.stdout: "daemon_cache_timeout": { 2024-03-24T03:37:35.919 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "daemon_cache_timeout", 2024-03-24T03:37:35.919 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:35.919 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.919 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.919 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-24T03:37:35.919 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.919 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.919 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.919 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-24T03:37:35.920 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.920 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.920 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.920 INFO:teuthology.orchestra.run.smithi078.stdout: "default_cephadm_command_timeout": { 2024-03-24T03:37:35.920 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default_cephadm_command_timeout", 2024-03-24T03:37:35.920 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.920 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.920 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.921 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "900", 2024-03-24T03:37:35.921 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.921 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.921 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.921 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-24T03:37:35.921 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.921 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.921 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.922 INFO:teuthology.orchestra.run.smithi078.stdout: "default_registry": { 2024-03-24T03:37:35.922 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default_registry", 2024-03-24T03:37:35.922 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.922 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.922 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.922 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io", 2024-03-24T03:37:35.922 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.922 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.922 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.923 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-24T03:37:35.923 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.923 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.923 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.923 INFO:teuthology.orchestra.run.smithi078.stdout: "device_cache_timeout": { 2024-03-24T03:37:35.923 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_cache_timeout", 2024-03-24T03:37:35.923 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:35.923 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.923 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.924 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1800", 2024-03-24T03:37:35.924 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.924 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.924 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.924 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache device inventory", 2024-03-24T03:37:35.924 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.924 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.924 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.925 INFO:teuthology.orchestra.run.smithi078.stdout: "device_enhanced_scan": { 2024-03-24T03:37:35.925 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_enhanced_scan", 2024-03-24T03:37:35.925 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.925 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.925 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.925 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.925 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.925 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.925 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.926 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-24T03:37:35.926 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.926 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.926 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.926 INFO:teuthology.orchestra.run.smithi078.stdout: "facts_cache_timeout": { 2024-03-24T03:37:35.926 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "facts_cache_timeout", 2024-03-24T03:37:35.926 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:35.926 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.926 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.927 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-24T03:37:35.927 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.927 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.927 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.927 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache host facts data", 2024-03-24T03:37:35.927 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.927 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.927 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.928 INFO:teuthology.orchestra.run.smithi078.stdout: "host_check_interval": { 2024-03-24T03:37:35.928 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "host_check_interval", 2024-03-24T03:37:35.928 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:35.928 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.928 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.928 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-24T03:37:35.928 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.928 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.928 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.928 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to perform a host check", 2024-03-24T03:37:35.929 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.929 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.929 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.929 INFO:teuthology.orchestra.run.smithi078.stdout: "hw_monitoring": { 2024-03-24T03:37:35.929 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "hw_monitoring", 2024-03-24T03:37:35.929 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.929 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.929 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.930 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.930 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.930 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.930 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.930 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-24T03:37:35.930 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.930 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.930 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.931 INFO:teuthology.orchestra.run.smithi078.stdout: "inventory_list_all": { 2024-03-24T03:37:35.931 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "inventory_list_all", 2024-03-24T03:37:35.931 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.931 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.931 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.931 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.931 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.931 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.931 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.932 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-24T03:37:35.932 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.932 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.932 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.932 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.932 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.932 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.932 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.932 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.933 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.933 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.933 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.933 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.933 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.933 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.933 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.933 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.933 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.934 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.934 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.934 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.934 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.934 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.934 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.934 INFO:teuthology.orchestra.run.smithi078.stdout: "log_refresh_metadata": { 2024-03-24T03:37:35.934 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_refresh_metadata", 2024-03-24T03:37:35.935 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.935 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.938 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.938 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.938 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.938 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.939 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.939 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-24T03:37:35.939 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.939 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.939 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.939 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.939 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.939 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.939 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.940 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.940 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.940 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.940 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.940 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.940 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-24T03:37:35.940 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.940 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.940 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.941 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.941 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.941 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.941 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.941 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.941 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.941 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.941 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.942 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.942 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.942 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.942 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.942 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.942 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.942 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.942 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.943 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.943 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.943 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.943 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.943 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.943 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.943 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.943 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.943 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.944 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.944 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.944 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.944 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.944 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.944 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.944 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.944 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.945 INFO:teuthology.orchestra.run.smithi078.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-24T03:37:35.945 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-24T03:37:35.945 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.945 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.945 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.945 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.945 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.945 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.945 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.946 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-24T03:37:35.946 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.946 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.946 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.946 INFO:teuthology.orchestra.run.smithi078.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-24T03:37:35.946 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-24T03:37:35.946 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.946 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.947 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.947 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "*", 2024-03-24T03:37:35.947 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.947 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.947 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.947 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-24T03:37:35.947 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.947 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.947 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.948 INFO:teuthology.orchestra.run.smithi078.stdout: "max_count_per_host": { 2024-03-24T03:37:35.948 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_count_per_host", 2024-03-24T03:37:35.948 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.948 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.948 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.948 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-24T03:37:35.948 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.948 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.948 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.949 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "max number of daemons per service per host", 2024-03-24T03:37:35.949 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.949 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.949 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.949 INFO:teuthology.orchestra.run.smithi078.stdout: "max_osd_draining_count": { 2024-03-24T03:37:35.949 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_osd_draining_count", 2024-03-24T03:37:35.949 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.949 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.950 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.950 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-24T03:37:35.950 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.950 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.950 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.950 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-24T03:37:35.950 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.950 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.951 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.951 INFO:teuthology.orchestra.run.smithi078.stdout: "migration_current": { 2024-03-24T03:37:35.951 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "migration_current", 2024-03-24T03:37:35.951 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.951 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.951 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.951 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.951 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.951 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.952 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.952 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "internal - do not modify", 2024-03-24T03:37:35.952 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.952 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.952 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.952 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": { 2024-03-24T03:37:35.952 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mode", 2024-03-24T03:37:35.953 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.953 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.953 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.953 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "root", 2024-03-24T03:37:35.953 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.953 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.953 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.953 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm-package", 2024-03-24T03:37:35.953 INFO:teuthology.orchestra.run.smithi078.stdout: "root" 2024-03-24T03:37:35.954 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.954 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "mode for remote execution of cephadm", 2024-03-24T03:37:35.954 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.954 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.954 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.954 INFO:teuthology.orchestra.run.smithi078.stdout: "oob_default_addr": { 2024-03-24T03:37:35.954 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "oob_default_addr", 2024-03-24T03:37:35.955 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.955 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.955 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.955 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "169.254.1.1", 2024-03-24T03:37:35.955 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.955 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.955 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.955 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-24T03:37:35.956 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.956 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.956 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.956 INFO:teuthology.orchestra.run.smithi078.stdout: "prometheus_alerts_path": { 2024-03-24T03:37:35.956 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus_alerts_path", 2024-03-24T03:37:35.956 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.956 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.956 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.957 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-24T03:37:35.957 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.957 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.957 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.957 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-24T03:37:35.957 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.957 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.957 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.958 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_insecure": { 2024-03-24T03:37:35.958 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_insecure", 2024-03-24T03:37:35.958 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.958 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.958 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.958 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.958 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.958 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.959 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.959 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-24T03:37:35.959 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.959 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.959 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.959 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_password": { 2024-03-24T03:37:35.959 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_password", 2024-03-24T03:37:35.959 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.960 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.960 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.960 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.960 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.960 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.960 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.960 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-24T03:37:35.960 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.960 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.961 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.961 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_url": { 2024-03-24T03:37:35.961 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_url", 2024-03-24T03:37:35.961 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.961 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.961 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.961 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.961 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.962 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.962 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.962 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-24T03:37:35.962 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.962 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.962 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.962 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_username": { 2024-03-24T03:37:35.962 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_username", 2024-03-24T03:37:35.963 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.963 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.963 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.963 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.963 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.963 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.963 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.963 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-24T03:37:35.963 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.964 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.964 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.964 INFO:teuthology.orchestra.run.smithi078.stdout: "secure_monitoring_stack": { 2024-03-24T03:37:35.964 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "secure_monitoring_stack", 2024-03-24T03:37:35.964 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.964 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.964 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.964 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.965 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.965 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.965 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.965 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-24T03:37:35.965 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.965 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.965 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.965 INFO:teuthology.orchestra.run.smithi078.stdout: "service_discovery_port": { 2024-03-24T03:37:35.966 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "service_discovery_port", 2024-03-24T03:37:35.966 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.966 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.966 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.966 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8765", 2024-03-24T03:37:35.966 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.966 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.966 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.966 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "cephadm service discovery port", 2024-03-24T03:37:35.967 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.967 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.967 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.967 INFO:teuthology.orchestra.run.smithi078.stdout: "ssh_config_file": { 2024-03-24T03:37:35.967 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssh_config_file", 2024-03-24T03:37:35.967 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.967 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.967 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.967 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.968 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.968 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.968 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.968 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-24T03:37:35.968 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.968 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.968 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.968 INFO:teuthology.orchestra.run.smithi078.stdout: "use_agent": { 2024-03-24T03:37:35.969 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "use_agent", 2024-03-24T03:37:35.969 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.969 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.969 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.969 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.969 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.969 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.969 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.969 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-24T03:37:35.970 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.970 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.970 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.970 INFO:teuthology.orchestra.run.smithi078.stdout: "use_repo_digest": { 2024-03-24T03:37:35.970 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "use_repo_digest", 2024-03-24T03:37:35.970 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.970 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.970 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.971 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.971 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.971 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.971 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.971 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-24T03:37:35.971 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.971 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.971 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.972 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_failed_host_check": { 2024-03-24T03:37:35.972 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_failed_host_check", 2024-03-24T03:37:35.972 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.972 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.972 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.972 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.972 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.972 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.972 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.972 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if the host check fails", 2024-03-24T03:37:35.973 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.973 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.973 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.973 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_stray_daemons": { 2024-03-24T03:37:35.973 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_stray_daemons", 2024-03-24T03:37:35.973 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.973 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.973 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.974 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.974 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.974 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.974 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.974 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-24T03:37:35.974 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.974 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.974 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.975 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_stray_hosts": { 2024-03-24T03:37:35.975 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_stray_hosts", 2024-03-24T03:37:35.975 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.975 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.975 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.975 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.975 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.975 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.975 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.976 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-24T03:37:35.976 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.976 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.976 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.976 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.976 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.976 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crash", 2024-03-24T03:37:35.976 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.977 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.977 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.977 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:35.977 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:35.977 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.977 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.977 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.977 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.977 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.978 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.978 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.978 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.978 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.978 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.978 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.978 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.978 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.978 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.979 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.979 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.979 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.979 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.979 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:35.979 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:35.979 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.979 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.979 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.980 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.980 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.980 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.980 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.980 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.980 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.980 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.980 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.981 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:35.981 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:35.981 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.981 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.981 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.981 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:35.981 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.981 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.982 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:35.982 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:35.982 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:35.982 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:35.982 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:35.982 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:35.982 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:35.982 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:35.983 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.983 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.983 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.983 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.983 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:35.983 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:35.983 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.983 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.984 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.984 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.984 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.984 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.984 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.984 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.984 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.984 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: "retain_interval": { 2024-03-24T03:37:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "retain_interval", 2024-03-24T03:37:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "31536000", 2024-03-24T03:37:35.986 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.986 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.986 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.986 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-24T03:37:35.986 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.986 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.986 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.986 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_recent_interval": { 2024-03-24T03:37:35.987 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_recent_interval", 2024-03-24T03:37:35.987 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:35.987 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.987 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:35.987 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1209600", 2024-03-24T03:37:35.987 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.987 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.987 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.988 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-24T03:37:35.988 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.988 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.988 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.988 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.988 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:35.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:35.989 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "dashboard", 2024-03-24T03:37:35.989 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:35.989 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:35.989 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:35.989 INFO:teuthology.orchestra.run.smithi078.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-24T03:37:35.989 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-24T03:37:35.989 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:35.989 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.989 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.990 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-24T03:37:35.990 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.990 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.990 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.990 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.990 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.990 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.990 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.991 INFO:teuthology.orchestra.run.smithi078.stdout: "ALERTMANAGER_API_HOST": { 2024-03-24T03:37:35.991 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-24T03:37:35.991 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:35.991 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.991 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.991 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:35.991 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.991 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.992 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.992 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.992 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.992 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.992 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.992 INFO:teuthology.orchestra.run.smithi078.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-24T03:37:35.992 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-24T03:37:35.992 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.992 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.993 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.993 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.993 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.993 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.993 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.993 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.993 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.993 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.993 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.994 INFO:teuthology.orchestra.run.smithi078.stdout: "AUDIT_API_ENABLED": { 2024-03-24T03:37:35.994 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "AUDIT_API_ENABLED", 2024-03-24T03:37:35.994 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.994 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.994 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.994 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:35.994 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.995 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.995 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.995 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.995 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.995 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.995 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.995 INFO:teuthology.orchestra.run.smithi078.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-24T03:37:35.995 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-24T03:37:35.995 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.996 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.996 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.996 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.996 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.996 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.996 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.996 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.996 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.996 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.997 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.997 INFO:teuthology.orchestra.run.smithi078.stdout: "ENABLE_BROWSABLE_API": { 2024-03-24T03:37:35.997 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-24T03:37:35.997 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.997 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.997 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.997 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.997 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.998 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.998 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.998 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.998 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.998 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:35.998 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:35.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:35.998 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-24T03:37:35.998 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-24T03:37:35.999 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:35.999 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:35.999 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:35.999 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:35.999 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:35.999 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:35.999 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:35.999 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:35.999 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:35.999 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.000 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.000 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-24T03:37:36.000 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-24T03:37:36.000 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.000 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.000 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.000 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:36.000 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-24T03:37:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-24T03:37:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.002 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.002 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.002 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:36.002 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.002 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.002 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.002 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.002 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.002 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.003 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.003 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-24T03:37:36.003 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-24T03:37:36.003 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.003 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.003 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.003 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:36.003 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.003 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.004 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.004 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.004 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.004 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.004 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.004 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-24T03:37:36.004 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-24T03:37:36.004 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.005 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.005 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.005 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:36.005 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.005 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.005 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.005 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.005 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.005 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.006 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.006 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-24T03:37:36.006 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-24T03:37:36.006 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.006 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.006 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.006 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:36.006 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.007 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.007 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.007 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.007 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.007 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.007 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.007 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-24T03:37:36.007 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-24T03:37:36.008 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.008 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.008 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.008 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:36.008 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.008 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.008 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.008 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.008 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.009 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.009 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.009 INFO:teuthology.orchestra.run.smithi078.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-24T03:37:36.009 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-24T03:37:36.009 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.009 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.009 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.009 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.010 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.010 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.010 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.010 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.010 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.010 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.010 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.010 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_PASSWORD": { 2024-03-24T03:37:36.011 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-24T03:37:36.011 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.011 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.011 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.011 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-03-24T03:37:36.011 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.011 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.011 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.011 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.012 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.012 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.012 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.012 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-24T03:37:36.012 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-24T03:37:36.012 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.012 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.012 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_URL": { 2024-03-24T03:37:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_URL", 2024-03-24T03:37:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.015 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.015 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.015 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.015 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.015 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_USERNAME": { 2024-03-24T03:37:36.015 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-24T03:37:36.015 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.015 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.016 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.016 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-03-24T03:37:36.016 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.016 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.016 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.016 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.016 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.016 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.016 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.017 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-24T03:37:36.017 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-24T03:37:36.017 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.017 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.017 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.017 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.017 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.017 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.017 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.018 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.018 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.018 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.018 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.018 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-24T03:37:36.018 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-24T03:37:36.018 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.018 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.019 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.019 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.019 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.019 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.019 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.019 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.019 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.019 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.019 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.020 INFO:teuthology.orchestra.run.smithi078.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-24T03:37:36.020 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-24T03:37:36.020 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.020 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.020 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.020 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:36.020 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.020 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.021 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.021 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.021 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.021 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.021 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.021 INFO:teuthology.orchestra.run.smithi078.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-24T03:37:36.021 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-24T03:37:36.021 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.021 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.022 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.022 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.022 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.022 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.022 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.022 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.022 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.022 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.022 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.023 INFO:teuthology.orchestra.run.smithi078.stdout: "MULTICLUSTER_CONFIG": { 2024-03-24T03:37:36.023 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-24T03:37:36.023 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.023 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.023 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.023 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "{}", 2024-03-24T03:37:36.023 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.023 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.024 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.024 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.024 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.024 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.024 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.024 INFO:teuthology.orchestra.run.smithi078.stdout: "PROMETHEUS_API_HOST": { 2024-03-24T03:37:36.024 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-24T03:37:36.024 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.024 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.025 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.025 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.025 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.025 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.025 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.025 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.025 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.025 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.026 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.026 INFO:teuthology.orchestra.run.smithi078.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-24T03:37:36.026 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-24T03:37:36.026 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.026 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.026 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.026 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:36.026 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.027 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.027 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.027 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.027 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.027 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.027 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.027 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-24T03:37:36.027 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-24T03:37:36.027 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.028 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.028 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.028 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.028 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.028 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.028 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.028 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.028 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.028 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.029 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.029 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-24T03:37:36.029 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-24T03:37:36.029 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.029 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.029 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.029 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.029 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.029 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.030 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.030 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.030 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.030 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.030 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.030 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-24T03:37:36.030 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-24T03:37:36.030 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.031 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.031 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.031 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:36.031 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.031 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.031 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.031 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.031 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.031 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.032 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.032 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-24T03:37:36.032 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-24T03:37:36.032 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.032 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.032 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.032 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:36.032 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.033 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.033 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.033 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.033 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.033 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.033 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.033 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-24T03:37:36.033 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-24T03:37:36.034 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.034 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.034 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.034 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.034 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.034 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.034 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.034 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.034 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.034 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.035 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.035 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-24T03:37:36.035 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-24T03:37:36.035 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.035 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.035 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.035 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.036 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.036 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.036 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.036 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.036 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.036 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.036 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.036 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-24T03:37:36.036 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-24T03:37:36.037 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.037 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.037 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.037 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.037 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.037 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.037 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.037 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.037 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.038 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.038 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.038 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_ENABLED": { 2024-03-24T03:37:36.038 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-24T03:37:36.038 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.038 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.038 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.038 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:36.039 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.039 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.039 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.039 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.039 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.039 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.039 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.039 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-24T03:37:36.039 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-24T03:37:36.040 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.040 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.040 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.040 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-24T03:37:36.040 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.040 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.040 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.040 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.040 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.041 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.041 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.041 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-24T03:37:36.041 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-24T03:37:36.041 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:36.041 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.041 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.041 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-24T03:37:36.042 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.042 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.042 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.042 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.042 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.042 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.042 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.042 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-24T03:37:36.043 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-24T03:37:36.043 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:36.043 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.043 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.043 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8", 2024-03-24T03:37:36.043 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.043 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.043 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.043 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.044 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.044 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.044 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.044 INFO:teuthology.orchestra.run.smithi078.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-24T03:37:36.044 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-24T03:37:36.044 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:36.044 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.044 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.045 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "45", 2024-03-24T03:37:36.045 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.045 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.045 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.045 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.045 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.045 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.045 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.045 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_ACCESS_KEY": { 2024-03-24T03:37:36.046 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-24T03:37:36.046 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.046 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.046 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.046 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.046 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.046 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.046 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.046 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.047 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.047 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.047 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.047 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-24T03:37:36.047 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-24T03:37:36.047 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.047 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.047 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.047 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-03-24T03:37:36.048 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.048 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.048 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.048 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.048 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.048 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.048 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.048 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_SECRET_KEY": { 2024-03-24T03:37:36.049 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-24T03:37:36.049 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.049 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.049 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.049 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.049 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.049 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.049 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.049 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.050 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.050 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.050 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.050 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_SSL_VERIFY": { 2024-03-24T03:37:36.050 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-24T03:37:36.050 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.050 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.050 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: "UNSAFE_TLS_v1_2": { 2024-03-24T03:37:36.052 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-24T03:37:36.052 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.052 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.052 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.052 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.052 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.052 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.052 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.052 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.053 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.053 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.053 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.053 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-24T03:37:36.053 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-24T03:37:36.053 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:36.053 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.053 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.053 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-24T03:37:36.054 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.054 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.054 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.054 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.054 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.054 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.054 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.055 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-24T03:37:36.055 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-24T03:37:36.055 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:36.055 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.055 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.055 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-24T03:37:36.055 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.055 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.055 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.056 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.056 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.056 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.056 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.056 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-24T03:37:36.056 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-24T03:37:36.056 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:36.056 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.057 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.057 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-24T03:37:36.057 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.057 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.057 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.057 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.057 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.057 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.057 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.058 INFO:teuthology.orchestra.run.smithi078.stdout: "cross_origin_url": { 2024-03-24T03:37:36.058 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cross_origin_url", 2024-03-24T03:37:36.058 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.058 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.058 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.058 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.058 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.058 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.059 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.059 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.059 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.059 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.059 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.059 INFO:teuthology.orchestra.run.smithi078.stdout: "crt_file": { 2024-03-24T03:37:36.059 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crt_file", 2024-03-24T03:37:36.059 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.060 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.060 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.060 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.060 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.060 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.060 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.060 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.060 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.060 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.061 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.061 INFO:teuthology.orchestra.run.smithi078.stdout: "debug": { 2024-03-24T03:37:36.061 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "debug", 2024-03-24T03:37:36.061 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.061 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.061 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.061 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.061 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.062 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.062 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.062 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable/disable debug options", 2024-03-24T03:37:36.062 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.062 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.062 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.062 INFO:teuthology.orchestra.run.smithi078.stdout: "jwt_token_ttl": { 2024-03-24T03:37:36.062 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "jwt_token_ttl", 2024-03-24T03:37:36.063 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:36.063 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.063 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.063 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "28800", 2024-03-24T03:37:36.063 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.063 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.063 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.063 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.063 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.064 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.064 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.064 INFO:teuthology.orchestra.run.smithi078.stdout: "key_file": { 2024-03-24T03:37:36.064 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "key_file", 2024-03-24T03:37:36.064 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.064 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.064 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.064 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.064 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.065 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.065 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.065 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.065 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.065 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.065 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.065 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:36.066 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:36.066 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.066 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.066 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.066 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.066 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.066 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.066 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.066 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.067 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.067 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.067 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.067 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.067 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.067 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.067 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.067 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.067 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.067 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.068 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:36.068 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:36.068 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.068 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.068 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.068 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.068 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.068 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.069 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.069 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.069 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.069 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.069 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.069 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:36.069 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:36.069 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.070 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.070 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.070 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:36.070 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.070 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.070 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.070 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.070 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.070 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.071 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.071 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.071 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.071 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.071 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.071 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.071 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.071 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.072 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:36.072 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:36.072 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.072 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.072 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.072 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.072 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.072 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.072 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.072 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.073 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.073 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.073 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.073 INFO:teuthology.orchestra.run.smithi078.stdout: "motd": { 2024-03-24T03:37:36.073 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "motd", 2024-03-24T03:37:36.073 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.073 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.073 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.074 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.074 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.074 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.074 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.074 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "The message of the day", 2024-03-24T03:37:36.074 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.074 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.074 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.075 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_resolve_ip_addr": { 2024-03-24T03:37:36.075 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "redirect_resolve_ip_addr", 2024-03-24T03:37:36.075 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.075 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.075 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.075 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.075 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.075 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.075 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.076 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.076 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.076 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.076 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.076 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-03-24T03:37:36.076 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-03-24T03:37:36.076 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.076 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.077 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.077 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "::", 2024-03-24T03:37:36.077 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.077 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.077 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.077 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.077 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.077 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.077 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.078 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-03-24T03:37:36.078 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-03-24T03:37:36.078 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:36.078 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.078 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.078 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8080", 2024-03-24T03:37:36.078 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.078 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.079 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.079 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.079 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.079 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.079 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.079 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl": { 2024-03-24T03:37:36.079 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl", 2024-03-24T03:37:36.079 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.080 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.080 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.080 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:36.080 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.080 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.080 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.080 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.080 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.080 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.081 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.081 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl_server_port": { 2024-03-24T03:37:36.081 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl_server_port", 2024-03-24T03:37:36.081 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:36.081 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.081 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.081 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8443", 2024-03-24T03:37:36.081 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.081 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.082 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.082 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.082 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.082 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.082 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.082 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_behaviour": { 2024-03-24T03:37:36.082 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_behaviour", 2024-03-24T03:37:36.082 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.083 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.083 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.083 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "redirect", 2024-03-24T03:37:36.083 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.083 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.083 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.083 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.083 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect" 2024-03-24T03:37:36.083 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.084 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.084 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.084 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.084 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.084 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_error_status_code": { 2024-03-24T03:37:36.084 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_error_status_code", 2024-03-24T03:37:36.084 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:36.084 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.085 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.085 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "500", 2024-03-24T03:37:36.085 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "400", 2024-03-24T03:37:36.085 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "599", 2024-03-24T03:37:36.085 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.085 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.085 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.085 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.085 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.086 INFO:teuthology.orchestra.run.smithi078.stdout: "url_prefix": { 2024-03-24T03:37:36.086 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "url_prefix", 2024-03-24T03:37:36.086 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.086 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.086 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.086 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.086 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.086 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.086 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.087 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.087 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.087 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.087 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.087 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.087 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.087 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "devicehealth", 2024-03-24T03:37:36.088 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:36.088 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:36.088 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:36.088 INFO:teuthology.orchestra.run.smithi078.stdout: "enable_monitoring": { 2024-03-24T03:37:36.088 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enable_monitoring", 2024-03-24T03:37:36.088 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.088 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.088 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.088 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:36.089 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.089 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.089 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.089 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "monitor device health metrics", 2024-03-24T03:37:36.089 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.089 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.089 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.089 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:36.089 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:36.090 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.090 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.090 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.090 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.090 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.090 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.090 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.090 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.090 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.091 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.091 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.091 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.091 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.091 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.091 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.091 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.091 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.091 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.092 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:36.092 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:36.092 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.092 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.092 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.092 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.092 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.092 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.092 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.093 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.093 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.093 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.093 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.093 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:36.093 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:36.093 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.093 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.094 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.094 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:36.094 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.094 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.094 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.094 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.094 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.094 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.094 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.095 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.095 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.095 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.095 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.095 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.095 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.095 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.095 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:36.095 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:36.096 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.096 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.096 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.096 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.096 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.096 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.096 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.096 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.096 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.097 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.097 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.097 INFO:teuthology.orchestra.run.smithi078.stdout: "mark_out_threshold": { 2024-03-24T03:37:36.097 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mark_out_threshold", 2024-03-24T03:37:36.097 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:36.097 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.097 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.097 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "2419200", 2024-03-24T03:37:36.098 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.098 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.098 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.098 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-24T03:37:36.098 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.098 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.098 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.098 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": { 2024-03-24T03:37:36.099 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pool_name", 2024-03-24T03:37:36.099 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.099 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.099 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.099 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "device_health_metrics", 2024-03-24T03:37:36.099 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.099 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.099 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.099 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-24T03:37:36.099 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.100 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.100 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.100 INFO:teuthology.orchestra.run.smithi078.stdout: "retention_period": { 2024-03-24T03:37:36.100 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "retention_period", 2024-03-24T03:37:36.100 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:36.100 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.100 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.100 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15552000", 2024-03-24T03:37:36.101 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.101 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.101 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.101 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long to retain device health metrics", 2024-03-24T03:37:36.101 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.101 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.101 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.101 INFO:teuthology.orchestra.run.smithi078.stdout: "scrape_frequency": { 2024-03-24T03:37:36.102 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "scrape_frequency", 2024-03-24T03:37:36.102 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:36.102 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.102 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.102 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "86400", 2024-03-24T03:37:36.102 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.102 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.102 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.102 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-24T03:37:36.102 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.103 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.103 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.103 INFO:teuthology.orchestra.run.smithi078.stdout: "self_heal": { 2024-03-24T03:37:36.103 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "self_heal", 2024-03-24T03:37:36.103 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.103 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.103 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.103 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:36.104 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.104 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.104 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.104 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-24T03:37:36.104 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.104 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.104 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.104 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-24T03:37:36.105 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-24T03:37:36.105 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:36.105 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.105 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.105 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-24T03:37:36.105 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.105 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.105 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.105 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to wake up and check device health", 2024-03-24T03:37:36.106 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.106 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.106 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.106 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_threshold": { 2024-03-24T03:37:36.106 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_threshold", 2024-03-24T03:37:36.106 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:36.106 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.106 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.107 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "7257600", 2024-03-24T03:37:36.107 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.107 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.107 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.107 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-24T03:37:36.107 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.107 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.107 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.107 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.107 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.108 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "diskprediction_local", 2024-03-24T03:37:36.108 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:36.108 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:36.108 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:36.108 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:36.108 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:36.108 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.109 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.109 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.109 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.109 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.109 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.109 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.109 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.109 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.109 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.109 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.110 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.110 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.110 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.110 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.110 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.110 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.110 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.110 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:36.111 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:36.111 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.111 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.111 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.111 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.111 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.111 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.111 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.112 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.112 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.112 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.112 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.112 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:36.112 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:36.112 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.112 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.113 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.113 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:36.113 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.113 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.113 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.113 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.113 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.113 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.113 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.113 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.114 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.114 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.114 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.114 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.114 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.114 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.114 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:36.114 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:36.115 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.115 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.115 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.115 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.115 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.115 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.115 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.115 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.115 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.116 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.116 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.116 INFO:teuthology.orchestra.run.smithi078.stdout: "predict_interval": { 2024-03-24T03:37:36.116 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "predict_interval", 2024-03-24T03:37:36.116 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.116 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.116 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.116 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "86400", 2024-03-24T03:37:36.117 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.117 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.117 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.117 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.117 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.117 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.117 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.117 INFO:teuthology.orchestra.run.smithi078.stdout: "predictor_model": { 2024-03-24T03:37:36.118 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "predictor_model", 2024-03-24T03:37:36.118 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.118 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.118 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.118 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "prophetstor", 2024-03-24T03:37:36.118 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.118 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.119 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.119 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.119 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.119 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.119 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.119 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-24T03:37:36.119 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-24T03:37:36.119 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.120 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.120 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.120 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-24T03:37:36.120 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.120 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.120 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.120 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.120 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.120 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.120 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.121 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.121 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.121 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "influx", 2024-03-24T03:37:36.121 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": false, 2024-03-24T03:37:36.121 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "influxdb python module not found", 2024-03-24T03:37:36.121 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:36.121 INFO:teuthology.orchestra.run.smithi078.stdout: "batch_size": { 2024-03-24T03:37:36.122 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "batch_size", 2024-03-24T03:37:36.122 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:36.122 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.122 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.122 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5000", 2024-03-24T03:37:36.122 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.122 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.122 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.122 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-24T03:37:36.123 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.123 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.123 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.123 INFO:teuthology.orchestra.run.smithi078.stdout: "database": { 2024-03-24T03:37:36.123 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "database", 2024-03-24T03:37:36.123 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.123 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.123 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.124 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "ceph", 2024-03-24T03:37:36.124 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.124 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.124 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.124 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-24T03:37:36.124 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.124 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.124 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.124 INFO:teuthology.orchestra.run.smithi078.stdout: "hostname": { 2024-03-24T03:37:36.125 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "hostname", 2024-03-24T03:37:36.125 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.125 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.125 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.125 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.125 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.125 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.125 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.125 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB server hostname", 2024-03-24T03:37:36.126 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.126 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.126 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.126 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-24T03:37:36.126 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-24T03:37:36.126 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:36.126 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.127 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.127 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "30", 2024-03-24T03:37:36.127 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "5", 2024-03-24T03:37:36.127 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.127 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.127 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-24T03:37:36.127 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.127 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.127 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.128 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:36.128 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:36.128 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.128 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.128 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.128 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.128 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.128 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.128 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.129 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.129 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.129 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.129 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.129 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.129 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.129 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.129 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.129 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.130 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.130 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.130 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:36.130 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:36.130 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.130 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.130 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.130 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.131 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.131 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.131 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.131 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.131 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.131 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.131 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.131 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:36.131 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:36.132 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.132 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.132 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.132 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:36.132 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.132 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.132 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.132 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.132 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.133 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.133 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.133 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.133 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.133 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.133 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.133 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.133 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.133 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.134 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:36.134 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:36.134 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.134 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.134 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.134 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.134 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.134 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.135 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.135 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.135 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.135 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.135 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.135 INFO:teuthology.orchestra.run.smithi078.stdout: "password": { 2024-03-24T03:37:36.135 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "password", 2024-03-24T03:37:36.135 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.136 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.136 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.136 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.136 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.136 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.136 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.136 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "password of InfluxDB server user", 2024-03-24T03:37:36.136 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.136 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.137 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.137 INFO:teuthology.orchestra.run.smithi078.stdout: "port": { 2024-03-24T03:37:36.137 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "port", 2024-03-24T03:37:36.137 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:36.137 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.137 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.137 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8086", 2024-03-24T03:37:36.137 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.137 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.138 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.138 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB server port", 2024-03-24T03:37:36.138 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.138 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.138 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.138 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl": { 2024-03-24T03:37:36.138 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl", 2024-03-24T03:37:36.138 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.139 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.139 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.139 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "false", 2024-03-24T03:37:36.139 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.139 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.139 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.139 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T03:37:36.139 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.139 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.140 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.140 INFO:teuthology.orchestra.run.smithi078.stdout: "threads": { 2024-03-24T03:37:36.140 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "threads", 2024-03-24T03:37:36.140 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:36.140 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.140 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.140 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-24T03:37:36.140 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-24T03:37:36.140 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "32", 2024-03-24T03:37:36.141 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.141 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-24T03:37:36.141 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.141 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.141 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.141 INFO:teuthology.orchestra.run.smithi078.stdout: "username": { 2024-03-24T03:37:36.141 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "username", 2024-03-24T03:37:36.141 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.142 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.142 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.142 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.142 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.142 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.142 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.142 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "username of InfluxDB server user", 2024-03-24T03:37:36.142 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.142 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.143 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.143 INFO:teuthology.orchestra.run.smithi078.stdout: "verify_ssl": { 2024-03-24T03:37:36.143 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "verify_ssl", 2024-03-24T03:37:36.143 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.143 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.143 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.143 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "true", 2024-03-24T03:37:36.143 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.144 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.144 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.144 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T03:37:36.144 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.144 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.144 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.144 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.144 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.145 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "insights", 2024-03-24T03:37:36.145 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:36.145 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:36.145 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:36.145 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:36.145 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:36.145 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.145 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.146 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.146 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.146 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.146 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.146 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.146 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.146 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.146 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.146 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.147 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.147 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.147 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.147 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.147 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.147 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.147 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.147 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:36.147 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:36.148 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.148 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.148 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.148 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.148 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.148 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.148 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.148 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.148 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.149 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.149 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.149 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:36.149 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:36.149 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.149 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.149 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.149 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:36.149 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.150 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.150 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.150 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.150 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.150 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.150 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.150 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.150 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.150 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.151 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.151 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.151 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.151 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.151 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:36.151 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:36.151 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.151 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.152 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.152 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.152 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.152 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.152 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.152 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.152 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.152 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.152 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.152 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.153 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.153 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "iostat", 2024-03-24T03:37:36.153 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:36.153 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:36.153 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:36.153 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:36.153 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:36.154 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.154 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.154 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.154 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.154 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.154 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.154 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.154 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.155 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.155 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.155 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.155 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.155 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.155 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.155 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.155 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.155 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.156 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.156 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:36.156 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:36.156 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.156 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.156 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.156 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.156 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.157 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.157 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.157 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.157 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.157 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.157 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.157 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:36.157 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:36.158 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.158 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.158 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.158 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:36.158 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.158 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.158 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.158 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.159 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.159 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.159 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.159 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.159 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.159 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.159 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.159 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.159 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.160 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.160 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:36.160 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:36.160 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.160 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.160 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.160 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.160 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.161 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.161 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.161 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.161 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.161 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.161 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.161 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.161 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.162 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "k8sevents", 2024-03-24T03:37:36.162 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:36.162 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:36.162 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:36.162 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_event_retention_days": { 2024-03-24T03:37:36.162 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ceph_event_retention_days", 2024-03-24T03:37:36.162 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:36.163 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.163 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.163 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "7", 2024-03-24T03:37:36.163 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.163 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.163 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.163 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-24T03:37:36.163 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.164 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.164 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.164 INFO:teuthology.orchestra.run.smithi078.stdout: "config_check_secs": { 2024-03-24T03:37:36.164 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_check_secs", 2024-03-24T03:37:36.164 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:36.164 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.164 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.164 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-24T03:37:36.165 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "10", 2024-03-24T03:37:36.165 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.165 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.165 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-24T03:37:36.165 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.165 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.165 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.166 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:36.166 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:36.166 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.166 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.166 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.166 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.166 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.166 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.167 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.167 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.167 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.167 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.167 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.167 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.167 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.167 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.167 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.167 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.168 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.168 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.168 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:36.168 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:36.168 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.168 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.168 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.169 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.169 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.169 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.169 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.169 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.169 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.169 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.169 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.170 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:36.170 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:36.170 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.170 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.170 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.170 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:36.170 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.170 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.171 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.171 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.171 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.171 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.171 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.171 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.171 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.171 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.171 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.172 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.172 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.172 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.172 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:36.172 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:36.172 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.172 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.173 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.173 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.173 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.173 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.173 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.173 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.173 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.173 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.173 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.174 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.174 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.174 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "localpool", 2024-03-24T03:37:36.174 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:36.174 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:36.174 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:36.175 INFO:teuthology.orchestra.run.smithi078.stdout: "failure_domain": { 2024-03-24T03:37:36.175 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "failure_domain", 2024-03-24T03:37:36.175 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.175 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.175 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.175 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "host", 2024-03-24T03:37:36.175 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.175 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.175 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.176 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "failure domain for any created local pool", 2024-03-24T03:37:36.176 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-24T03:37:36.176 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.176 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.176 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:36.176 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:36.176 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.176 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.176 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.177 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.177 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.177 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.177 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.177 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.177 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.177 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.177 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.177 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.178 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.178 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.178 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.178 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.178 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.178 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.178 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:36.178 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:36.179 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.179 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.179 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.179 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.179 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.179 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.179 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.179 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.179 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.180 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.180 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.180 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:36.180 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:36.180 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.180 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.180 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.180 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:36.181 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.181 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.181 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.181 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.181 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.181 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.181 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.181 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.181 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.182 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.182 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.182 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.182 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.182 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.182 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:36.182 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:36.182 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.183 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.183 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.183 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.183 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.183 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.183 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.183 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.183 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.183 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.183 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.184 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": { 2024-03-24T03:37:36.184 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "min_size", 2024-03-24T03:37:36.184 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:36.184 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.184 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.184 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.184 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.184 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.185 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.185 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default min_size for any created local pool", 2024-03-24T03:37:36.185 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-24T03:37:36.185 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.185 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.185 INFO:teuthology.orchestra.run.smithi078.stdout: "num_rep": { 2024-03-24T03:37:36.185 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "num_rep", 2024-03-24T03:37:36.186 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:36.186 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.186 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.186 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3", 2024-03-24T03:37:36.186 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.186 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.186 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.186 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default replica count for any created local pool", 2024-03-24T03:37:36.186 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.187 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.187 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.187 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": { 2024-03-24T03:37:36.187 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pg_num", 2024-03-24T03:37:36.187 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:36.187 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.187 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.187 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "128", 2024-03-24T03:37:36.188 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.188 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.188 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.188 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default pg_num for any created local pool", 2024-03-24T03:37:36.188 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.188 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.188 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.188 INFO:teuthology.orchestra.run.smithi078.stdout: "prefix": { 2024-03-24T03:37:36.188 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prefix", 2024-03-24T03:37:36.189 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.189 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.189 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.189 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.189 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.189 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.189 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.189 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "name prefix for any created local pool", 2024-03-24T03:37:36.189 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.190 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.190 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.190 INFO:teuthology.orchestra.run.smithi078.stdout: "subtree": { 2024-03-24T03:37:36.190 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "subtree", 2024-03-24T03:37:36.190 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.190 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.190 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.191 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "rack", 2024-03-24T03:37:36.191 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.191 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.191 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.191 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-24T03:37:36.191 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-24T03:37:36.191 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.191 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.191 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.191 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.192 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mds_autoscaler", 2024-03-24T03:37:36.192 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:36.192 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:36.192 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:36.192 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:36.192 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:36.192 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.193 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.193 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.193 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.193 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.193 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.193 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.193 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.193 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.193 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.194 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.194 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.194 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.194 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.194 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.194 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.194 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.194 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.195 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:36.195 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:36.195 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.195 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.195 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.195 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.195 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.195 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.195 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.196 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.196 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.196 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.196 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.196 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:36.196 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:36.196 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.196 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.197 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.197 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:36.197 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.197 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.197 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.197 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.197 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.197 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.197 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.197 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.198 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.198 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.198 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.198 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.198 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.198 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.198 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.200 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.200 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.200 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.200 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.200 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mirroring", 2024-03-24T03:37:36.200 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:36.201 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:36.201 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:36.201 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:36.201 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:36.201 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.201 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.201 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.201 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.201 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.202 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.202 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.202 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.202 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.202 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.202 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.202 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.203 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.203 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.203 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.203 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.203 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.203 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.203 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:36.204 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:36.204 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.204 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.204 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.204 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.204 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.204 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.204 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.204 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.205 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.205 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.205 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.205 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:36.205 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:36.205 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.205 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.206 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.206 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:36.206 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.206 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.206 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.206 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.206 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.206 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.207 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.207 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.207 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.207 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.207 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.207 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.207 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.207 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.208 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:36.208 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:36.208 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.208 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.208 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.208 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.208 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.208 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.209 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.209 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.209 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.209 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.209 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.209 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.209 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.210 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "nfs", 2024-03-24T03:37:36.210 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:36.210 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:36.210 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:36.210 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:36.210 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:36.210 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.210 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.211 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.211 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.211 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.211 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.211 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.211 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.211 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.211 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.211 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.211 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.212 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.212 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.212 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.212 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.212 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.212 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.212 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:36.212 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:36.213 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.213 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.213 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.213 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.213 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.213 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.213 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.213 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.214 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.214 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.214 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.214 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:36.214 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:36.214 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.214 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.214 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.215 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:36.215 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.215 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.215 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.215 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.215 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.215 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.215 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.215 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.216 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.216 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.216 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.216 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.216 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.216 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.216 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:36.216 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:36.217 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.217 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.217 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.217 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.217 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.217 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.217 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.217 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.217 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.218 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.218 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.218 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.218 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.218 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "orchestrator", 2024-03-24T03:37:36.218 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:36.219 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:36.219 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:36.219 INFO:teuthology.orchestra.run.smithi078.stdout: "fail_fs": { 2024-03-24T03:37:36.219 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "fail_fs", 2024-03-24T03:37:36.219 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.219 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.219 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.219 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.219 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.220 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.220 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.220 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-24T03:37:36.220 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.220 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.220 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.220 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:36.220 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:36.221 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.221 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.221 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.221 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.221 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.221 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.221 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.221 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.222 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.222 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.222 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.222 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.222 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.222 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.222 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.222 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.223 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.223 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.223 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:36.223 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:36.223 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.223 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.223 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.223 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.223 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.224 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.224 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.224 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.224 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.224 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.224 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.224 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:36.224 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:36.225 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.225 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.225 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.225 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:36.225 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.225 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.225 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.225 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.226 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.226 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.226 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.226 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.226 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.226 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.226 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.226 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.227 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.227 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.227 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:36.227 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:36.227 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.227 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.227 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.227 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.227 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.228 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.228 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.228 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.228 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.228 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.228 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.228 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator": { 2024-03-24T03:37:36.228 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "orchestrator", 2024-03-24T03:37:36.229 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.229 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.229 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.229 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.229 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.229 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.229 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.229 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm", 2024-03-24T03:37:36.229 INFO:teuthology.orchestra.run.smithi078.stdout: "rook", 2024-03-24T03:37:36.230 INFO:teuthology.orchestra.run.smithi078.stdout: "test_orchestrator" 2024-03-24T03:37:36.230 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.230 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Orchestrator backend", 2024-03-24T03:37:36.230 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.230 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.230 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.230 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.230 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.231 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "osd_perf_query", 2024-03-24T03:37:36.231 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:36.231 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:36.231 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:36.231 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:36.231 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:36.231 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.231 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.232 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.232 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.232 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.232 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.232 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.232 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.232 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.232 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.232 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.232 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.233 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.233 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.233 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.233 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.233 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.233 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.233 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:36.233 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:36.234 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.234 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.234 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.234 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.234 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.234 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.234 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.234 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.235 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.235 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.235 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.235 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:36.235 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:36.235 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.235 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.235 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.236 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:36.236 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.236 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.236 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.236 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.236 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.236 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.236 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.236 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.236 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.237 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.237 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.237 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.237 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.237 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.237 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:36.237 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:36.237 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.238 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.238 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.238 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.238 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.238 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.238 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.238 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.238 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.238 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.239 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.239 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.239 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.239 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "osd_support", 2024-03-24T03:37:36.239 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:36.239 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:36.239 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:36.240 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:36.240 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:36.240 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.240 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.240 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.240 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.240 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.240 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.240 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.240 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.241 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.241 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.241 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.241 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.241 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.241 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.241 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.241 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.241 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.242 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.242 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:36.242 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:36.242 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.242 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.242 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.242 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.243 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.243 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.243 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.243 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.243 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.243 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.243 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.243 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:36.244 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:36.244 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.244 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.244 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.244 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:36.244 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.244 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.244 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.244 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.245 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.245 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.245 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.245 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.245 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.245 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.245 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.245 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.246 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.246 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.246 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:36.246 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:36.246 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.246 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.246 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.246 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.247 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.247 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.247 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.247 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.247 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.247 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.247 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.247 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.247 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.248 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pg_autoscaler", 2024-03-24T03:37:36.248 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:36.248 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:36.248 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:36.248 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:36.248 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:36.248 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.248 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.249 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.249 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.249 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.249 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.249 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.249 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.249 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.249 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.249 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.250 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.250 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.250 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.250 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.250 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.250 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.250 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.250 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:36.251 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:36.251 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.251 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.251 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.251 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.251 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.251 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.251 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.251 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.251 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.252 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.252 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.252 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:36.252 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:36.252 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.252 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.252 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.252 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:36.253 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.253 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.253 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.253 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.253 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.253 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.253 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.253 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.253 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.254 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.254 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.254 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.254 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.254 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.254 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:36.254 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:36.254 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.255 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.255 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.255 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.255 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.255 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.255 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.255 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.255 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.255 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.256 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.256 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-24T03:37:36.256 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-24T03:37:36.256 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:36.256 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.256 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.256 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-24T03:37:36.256 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.257 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.257 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.257 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.257 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.257 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.257 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.257 INFO:teuthology.orchestra.run.smithi078.stdout: "threshold": { 2024-03-24T03:37:36.257 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "threshold", 2024-03-24T03:37:36.258 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-24T03:37:36.258 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.258 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.258 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3.0", 2024-03-24T03:37:36.258 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1.0", 2024-03-24T03:37:36.258 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.258 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.258 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "scaling threshold", 2024-03-24T03:37:36.258 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-24T03:37:36.259 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.259 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.259 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.259 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.259 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "progress", 2024-03-24T03:37:36.259 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:36.259 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:36.260 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:36.260 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_pg_recovery_event": { 2024-03-24T03:37:36.260 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_pg_recovery_event", 2024-03-24T03:37:36.260 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.260 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.260 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.260 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.260 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.260 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.261 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.261 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-24T03:37:36.261 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.261 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.261 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.261 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": { 2024-03-24T03:37:36.261 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enabled", 2024-03-24T03:37:36.261 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.262 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.262 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.262 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:36.262 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.262 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.262 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.262 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.262 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.262 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.262 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.263 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:36.263 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:36.263 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.263 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.263 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.263 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.263 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.264 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.264 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.264 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.264 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.264 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.264 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.264 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.264 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.264 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.265 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.265 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.265 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.265 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.265 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:36.265 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:36.265 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.265 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.266 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.266 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.266 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.266 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.266 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.266 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.266 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.266 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.267 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.267 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:36.267 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:36.267 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.267 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.267 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.267 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:36.267 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.268 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.268 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.268 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.268 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.268 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.268 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.268 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.268 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.268 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.269 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.269 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.269 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.269 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.269 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:36.269 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:36.269 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.269 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.270 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.270 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.270 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.270 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.270 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.270 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.270 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.270 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.271 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.271 INFO:teuthology.orchestra.run.smithi078.stdout: "max_completed_events": { 2024-03-24T03:37:36.271 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_completed_events", 2024-03-24T03:37:36.271 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:36.271 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.271 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.271 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "50", 2024-03-24T03:37:36.271 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.272 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.272 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.272 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "number of past completed events to remember", 2024-03-24T03:37:36.272 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.272 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.272 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.272 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-24T03:37:36.272 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-24T03:37:36.273 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:36.273 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.273 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.273 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-24T03:37:36.273 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.273 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.273 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.273 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long the module is going to sleep", 2024-03-24T03:37:36.273 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.274 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.274 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.274 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.274 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.274 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus", 2024-03-24T03:37:36.274 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:36.274 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:36.275 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:36.275 INFO:teuthology.orchestra.run.smithi078.stdout: "cache": { 2024-03-24T03:37:36.275 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cache", 2024-03-24T03:37:36.275 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.275 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.275 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.275 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:36.275 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.275 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.276 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.276 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.276 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.276 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.276 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.276 INFO:teuthology.orchestra.run.smithi078.stdout: "exclude_perf_counters": { 2024-03-24T03:37:36.276 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "exclude_perf_counters", 2024-03-24T03:37:36.276 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.277 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.277 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.277 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:36.277 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.277 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.277 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.277 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-24T03:37:36.277 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-24T03:37:36.277 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.278 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.278 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:36.278 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:36.278 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.278 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.278 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.278 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.278 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.279 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.279 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.279 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.279 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.279 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.279 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.279 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.279 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.279 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.280 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.280 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.280 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.280 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.280 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:36.280 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:36.280 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.280 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.280 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.281 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.281 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.281 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.281 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.281 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.281 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.281 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.281 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.282 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:36.282 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:36.282 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.282 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.282 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.282 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:36.282 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.282 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.282 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.283 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.283 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.283 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.283 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.283 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.283 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.283 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.283 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.283 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.284 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.284 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.284 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:36.284 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:36.284 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.284 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.284 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.284 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.285 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.285 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.285 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.285 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.285 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.285 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.285 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.286 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_stats_pools": { 2024-03-24T03:37:36.286 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_stats_pools", 2024-03-24T03:37:36.286 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.286 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.286 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.286 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.286 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.286 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.286 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.287 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.287 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.287 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.287 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.287 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-24T03:37:36.287 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-24T03:37:36.287 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:36.287 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.288 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.288 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "300", 2024-03-24T03:37:36.288 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.288 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.288 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.288 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.288 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.288 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.288 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.289 INFO:teuthology.orchestra.run.smithi078.stdout: "scrape_interval": { 2024-03-24T03:37:36.289 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "scrape_interval", 2024-03-24T03:37:36.289 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-24T03:37:36.289 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.289 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.289 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15.0", 2024-03-24T03:37:36.289 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.290 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.290 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.290 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.290 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.290 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.290 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.290 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-03-24T03:37:36.290 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-03-24T03:37:36.291 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.291 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.291 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.291 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "::", 2024-03-24T03:37:36.291 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.291 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.291 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.291 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-24T03:37:36.292 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.292 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.292 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.292 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-03-24T03:37:36.292 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-03-24T03:37:36.292 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:36.292 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.293 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.293 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "9283", 2024-03-24T03:37:36.293 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.293 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.293 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.293 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-24T03:37:36.293 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.293 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.294 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.294 INFO:teuthology.orchestra.run.smithi078.stdout: "stale_cache_strategy": { 2024-03-24T03:37:36.294 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "stale_cache_strategy", 2024-03-24T03:37:36.294 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.294 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.294 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.294 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "log", 2024-03-24T03:37:36.294 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.295 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.295 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.295 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.295 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.295 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.295 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.295 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_behaviour": { 2024-03-24T03:37:36.296 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_behaviour", 2024-03-24T03:37:36.296 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.296 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.296 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.296 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "default", 2024-03-24T03:37:36.296 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.296 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.296 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.297 INFO:teuthology.orchestra.run.smithi078.stdout: "default", 2024-03-24T03:37:36.297 INFO:teuthology.orchestra.run.smithi078.stdout: "error" 2024-03-24T03:37:36.297 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.297 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.297 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.297 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.297 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.297 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_error_status_code": { 2024-03-24T03:37:36.298 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_error_status_code", 2024-03-24T03:37:36.298 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:36.298 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.298 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.298 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "500", 2024-03-24T03:37:36.298 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "400", 2024-03-24T03:37:36.298 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "599", 2024-03-24T03:37:36.298 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.298 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.299 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.299 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.299 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.299 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.299 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.299 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_support", 2024-03-24T03:37:36.299 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:36.300 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:36.300 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:36.300 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:36.300 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:36.300 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.300 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.300 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.300 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.300 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.301 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.301 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.301 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.301 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.301 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.301 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.301 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.301 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.301 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.302 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.302 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.302 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.302 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.302 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:36.302 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:36.302 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.302 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.303 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.303 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.303 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.303 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.303 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.303 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.303 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.303 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.303 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.304 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:36.304 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:36.304 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.304 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.304 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.304 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:36.304 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.304 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.305 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.305 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.305 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.305 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.305 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.305 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.305 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.305 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.305 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.306 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.306 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.306 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.306 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:36.306 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:36.306 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.306 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.306 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.307 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.307 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.307 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.307 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.307 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.307 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.307 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.307 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.308 INFO:teuthology.orchestra.run.smithi078.stdout: "max_concurrent_snap_create": { 2024-03-24T03:37:36.308 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_concurrent_snap_create", 2024-03-24T03:37:36.308 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:36.308 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.308 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.308 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-24T03:37:36.308 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.308 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.308 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.308 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.309 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.309 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.309 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.309 INFO:teuthology.orchestra.run.smithi078.stdout: "mirror_snapshot_schedule": { 2024-03-24T03:37:36.309 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mirror_snapshot_schedule", 2024-03-24T03:37:36.309 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.309 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.309 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.310 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.310 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.310 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.310 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.310 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.310 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.310 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.310 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.311 INFO:teuthology.orchestra.run.smithi078.stdout: "trash_purge_schedule": { 2024-03-24T03:37:36.311 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "trash_purge_schedule", 2024-03-24T03:37:36.311 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.311 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.311 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.311 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.311 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.311 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.311 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.312 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.312 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.312 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.312 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.312 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.312 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.312 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "restful", 2024-03-24T03:37:36.313 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:36.313 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:36.313 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:36.313 INFO:teuthology.orchestra.run.smithi078.stdout: "enable_auth": { 2024-03-24T03:37:36.313 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enable_auth", 2024-03-24T03:37:36.313 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.313 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.313 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.313 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:36.314 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.314 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.314 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.314 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.314 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.314 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.314 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.314 INFO:teuthology.orchestra.run.smithi078.stdout: "key_file": { 2024-03-24T03:37:36.315 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "key_file", 2024-03-24T03:37:36.315 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.315 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.315 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.315 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.315 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.315 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.315 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.315 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.316 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.316 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.316 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.316 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:36.316 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:36.316 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.316 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.316 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.317 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.317 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.317 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.317 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.317 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.317 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.317 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.317 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.317 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.317 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.318 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.318 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.318 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.318 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.318 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.318 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:36.318 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:36.319 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.319 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.319 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.319 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.319 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.319 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.319 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.319 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.319 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.319 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.320 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.320 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:36.320 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:36.320 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.320 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.320 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.320 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:36.320 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.321 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.321 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.321 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.321 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.321 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.321 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.321 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.321 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.321 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.322 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.322 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.322 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.322 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.322 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:36.322 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:36.322 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.322 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.323 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.323 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.323 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.323 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.323 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.323 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.323 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.323 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.323 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.324 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-03-24T03:37:36.324 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-03-24T03:37:36.324 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.324 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.324 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.324 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.324 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.324 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.324 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.325 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.325 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.325 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.325 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.325 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-03-24T03:37:36.325 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-03-24T03:37:36.325 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.325 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.326 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.326 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.326 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.326 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.326 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.326 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.326 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.326 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.326 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.327 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.327 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.327 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rgw", 2024-03-24T03:37:36.327 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:36.327 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:36.327 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:36.327 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:36.327 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:36.328 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.328 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.328 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.328 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.328 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.328 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.328 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.328 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.328 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.329 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.329 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.329 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.329 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.329 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.329 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.329 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.329 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.329 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.330 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:36.330 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:36.330 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.330 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.330 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.330 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.330 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.330 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.331 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.331 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.331 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.331 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.331 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.331 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:36.331 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:36.331 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.331 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.332 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.332 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:36.332 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.332 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.332 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.332 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.332 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.332 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.332 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.333 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.333 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.333 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.333 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.333 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.333 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.333 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.333 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:36.334 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:36.334 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.334 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.334 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.334 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.334 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.334 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.334 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.335 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.335 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.335 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.335 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.335 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.335 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.335 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rook", 2024-03-24T03:37:36.336 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:36.336 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:36.336 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:36.336 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:36.336 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:36.336 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.336 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.336 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.336 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.337 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.337 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.337 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.337 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.337 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.337 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.337 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.337 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.338 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.338 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.338 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.338 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.338 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.338 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.338 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:36.338 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:36.339 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.339 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.339 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.339 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.339 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.339 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.339 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.339 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.339 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.340 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.340 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.340 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:36.340 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:36.340 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.340 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.340 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.340 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:36.341 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.341 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.341 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.341 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.341 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.341 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.341 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.341 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.341 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.342 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.342 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.342 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.342 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.342 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.342 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:36.342 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:36.343 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.343 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.343 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.343 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.343 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.343 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.343 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.343 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.344 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.344 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.344 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.344 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_class": { 2024-03-24T03:37:36.344 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "storage_class", 2024-03-24T03:37:36.344 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.344 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.344 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.344 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "local", 2024-03-24T03:37:36.345 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.345 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.345 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.345 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-24T03:37:36.345 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.345 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.345 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.345 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.346 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.346 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "selftest", 2024-03-24T03:37:36.346 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:36.346 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:36.346 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:36.346 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:36.346 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:36.347 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.347 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.347 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.347 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.347 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.347 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.347 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.347 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.347 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.348 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.348 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.348 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.348 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.348 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.348 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.348 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.348 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.348 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.349 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:36.349 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:36.349 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.349 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.349 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.349 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.349 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.349 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.349 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.350 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.350 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.350 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.350 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.350 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:36.350 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:36.350 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.350 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.351 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.351 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:36.351 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.351 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.351 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.351 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.351 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.351 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.351 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.352 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.352 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.352 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.352 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.352 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.352 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.352 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.352 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:36.353 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:36.353 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.353 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.353 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.353 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.353 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.353 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.353 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.353 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.353 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.354 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.354 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.354 INFO:teuthology.orchestra.run.smithi078.stdout: "roption1": { 2024-03-24T03:37:36.354 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "roption1", 2024-03-24T03:37:36.354 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.354 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.354 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.355 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.355 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.355 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.355 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.355 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.355 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.355 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.355 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.356 INFO:teuthology.orchestra.run.smithi078.stdout: "roption2": { 2024-03-24T03:37:36.356 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "roption2", 2024-03-24T03:37:36.356 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.356 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.356 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.356 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "xyz", 2024-03-24T03:37:36.356 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.356 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.356 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.357 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.357 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.357 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.357 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.357 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption1": { 2024-03-24T03:37:36.357 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption1", 2024-03-24T03:37:36.357 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.357 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.358 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.358 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.358 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.358 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.358 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.358 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.358 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.358 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.358 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.359 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption2": { 2024-03-24T03:37:36.359 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption2", 2024-03-24T03:37:36.359 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:36.359 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.359 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.359 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.359 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.359 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.360 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.360 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.360 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.360 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.360 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.360 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption3": { 2024-03-24T03:37:36.360 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption3", 2024-03-24T03:37:36.360 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-24T03:37:36.361 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.361 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.361 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.361 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.361 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.361 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.361 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.361 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.361 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.361 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.362 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption4": { 2024-03-24T03:37:36.362 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption4", 2024-03-24T03:37:36.362 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.362 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.362 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.362 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.362 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.363 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.363 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.363 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.363 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.363 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.363 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.363 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption5": { 2024-03-24T03:37:36.363 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption5", 2024-03-24T03:37:36.364 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.364 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.364 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.364 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.364 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.364 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.364 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.364 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.364 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.365 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.365 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.365 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption6": { 2024-03-24T03:37:36.365 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption6", 2024-03-24T03:37:36.365 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.365 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.365 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.365 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:36.365 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.366 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.366 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.366 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.366 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.366 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.366 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.366 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption7": { 2024-03-24T03:37:36.366 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption7", 2024-03-24T03:37:36.367 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:36.367 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.367 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.367 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.367 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-24T03:37:36.367 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "42", 2024-03-24T03:37:36.367 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.367 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.367 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.368 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.368 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.368 INFO:teuthology.orchestra.run.smithi078.stdout: "testkey": { 2024-03-24T03:37:36.368 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testkey", 2024-03-24T03:37:36.368 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.368 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.368 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.368 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.369 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.369 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.369 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.369 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.369 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.369 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.369 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.369 INFO:teuthology.orchestra.run.smithi078.stdout: "testlkey": { 2024-03-24T03:37:36.369 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testlkey", 2024-03-24T03:37:36.370 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.370 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.370 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.370 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.370 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.370 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.370 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.370 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.370 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.371 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.371 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.371 INFO:teuthology.orchestra.run.smithi078.stdout: "testnewline": { 2024-03-24T03:37:36.371 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testnewline", 2024-03-24T03:37:36.371 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.371 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.371 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.371 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.372 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.372 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.372 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.372 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.372 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.372 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.372 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.372 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.372 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.373 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snap_schedule", 2024-03-24T03:37:36.373 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:36.373 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:36.373 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:36.373 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_m_granularity": { 2024-03-24T03:37:36.373 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_m_granularity", 2024-03-24T03:37:36.373 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.373 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.374 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.374 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.374 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.374 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.374 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.374 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow minute scheduled snapshots", 2024-03-24T03:37:36.374 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.374 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.374 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.375 INFO:teuthology.orchestra.run.smithi078.stdout: "dump_on_update": { 2024-03-24T03:37:36.375 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "dump_on_update", 2024-03-24T03:37:36.375 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.375 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.375 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.375 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.375 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.375 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.376 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.376 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "dump database to debug log on update", 2024-03-24T03:37:36.376 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.376 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.376 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.376 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:36.376 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:36.376 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.377 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.377 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.377 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.377 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.377 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.377 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.377 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.377 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.377 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.377 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.378 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.378 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.378 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.378 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.378 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.378 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.378 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.378 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:36.379 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:36.379 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.379 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.379 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.379 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.379 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.379 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.379 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.379 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.380 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.380 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.380 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.380 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:36.380 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:36.380 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.380 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.381 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.381 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:36.381 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.381 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.381 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.381 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.381 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.381 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.381 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.381 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.382 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.382 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.382 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.382 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.382 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.382 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.382 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:36.382 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:36.383 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.383 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.383 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.383 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.383 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.383 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.383 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.383 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.383 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.384 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.384 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.384 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.384 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.384 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "stats", 2024-03-24T03:37:36.384 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:36.384 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:36.385 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:36.385 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:36.385 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:36.385 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.385 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.385 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.385 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.385 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.386 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.386 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.386 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.386 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.386 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.386 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.386 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.386 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.386 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.387 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.387 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.387 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.387 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.387 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:36.387 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:36.387 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.387 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.388 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.388 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.388 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.388 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.388 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.388 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.388 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.388 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.389 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.389 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:36.389 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:36.389 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.389 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.389 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.389 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:36.389 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.390 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.390 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.390 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.390 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.390 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.390 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.390 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.390 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.391 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.391 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.391 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.391 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.391 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.391 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:36.391 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:36.391 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.392 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.392 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.392 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.392 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.392 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.392 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.392 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.392 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.392 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.393 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.393 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.393 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.393 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.393 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "status", 2024-03-24T03:37:36.393 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:36.393 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:36.393 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:36.393 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:36.394 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:36.394 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.394 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.394 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.394 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.394 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.394 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.395 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.395 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.395 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.395 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.395 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.395 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.395 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.395 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.395 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.396 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.396 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.396 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.396 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:36.396 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:36.396 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.396 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.396 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.397 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.397 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.397 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.397 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.397 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.397 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.397 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.397 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.397 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:36.398 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:36.398 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.398 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.398 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.398 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:36.398 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.398 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.398 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.398 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.399 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.399 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.399 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.399 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.399 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.399 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.399 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.399 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.399 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.400 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.400 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:36.400 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:36.400 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.400 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.400 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.400 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.400 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.400 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.401 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.401 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.401 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.401 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.401 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.401 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.401 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.402 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "telegraf", 2024-03-24T03:37:36.402 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:36.402 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:36.402 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:36.402 INFO:teuthology.orchestra.run.smithi078.stdout: "address": { 2024-03-24T03:37:36.402 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "address", 2024-03-24T03:37:36.402 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.402 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.402 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.403 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-24T03:37:36.403 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.403 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.403 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.403 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.403 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.403 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.403 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.403 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-24T03:37:36.404 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-24T03:37:36.404 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:36.404 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.404 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.404 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15", 2024-03-24T03:37:36.404 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.404 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.404 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.404 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.405 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.405 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.405 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.405 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:36.405 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:36.405 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.405 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.405 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.406 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.406 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.406 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.406 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.406 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.406 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.406 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.406 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.406 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.407 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.407 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.407 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.407 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.407 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.407 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.407 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:36.407 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:36.407 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.408 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.408 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.408 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.408 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.408 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.408 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.408 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.408 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.408 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.409 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.409 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:36.409 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:36.409 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.409 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.409 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.409 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:36.409 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.410 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.410 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.410 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.410 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.410 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.410 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.410 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.410 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.410 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.411 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.411 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.411 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.411 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.411 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:36.411 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:36.411 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.411 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.412 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.412 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.412 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.412 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.412 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.412 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.412 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.412 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.412 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.412 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.413 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.413 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "telemetry", 2024-03-24T03:37:36.413 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:36.413 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:36.413 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:36.413 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_basic": { 2024-03-24T03:37:36.414 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_basic", 2024-03-24T03:37:36.414 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.414 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.414 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.414 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:36.414 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.414 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.414 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.414 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-24T03:37:36.415 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.415 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.415 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.415 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_crash": { 2024-03-24T03:37:36.415 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_crash", 2024-03-24T03:37:36.415 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.415 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.415 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.415 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:36.416 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.416 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.416 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.416 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-24T03:37:36.416 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.416 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.416 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.416 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_device": { 2024-03-24T03:37:36.417 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_device", 2024-03-24T03:37:36.417 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.417 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.417 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.417 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:36.417 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.417 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.417 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.417 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-24T03:37:36.418 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.418 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.418 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.418 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_ident": { 2024-03-24T03:37:36.418 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_ident", 2024-03-24T03:37:36.418 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.418 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.418 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.419 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.419 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.419 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.419 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.419 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-24T03:37:36.419 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.419 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.419 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.420 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_perf": { 2024-03-24T03:37:36.420 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_perf", 2024-03-24T03:37:36.420 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.420 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.420 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.420 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.420 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.420 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.420 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.421 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-24T03:37:36.421 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.421 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.421 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.421 INFO:teuthology.orchestra.run.smithi078.stdout: "contact": { 2024-03-24T03:37:36.421 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "contact", 2024-03-24T03:37:36.421 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.422 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.422 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.422 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.422 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.422 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.422 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.422 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.422 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.422 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.423 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.423 INFO:teuthology.orchestra.run.smithi078.stdout: "description": { 2024-03-24T03:37:36.423 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "description", 2024-03-24T03:37:36.423 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.423 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.423 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.423 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.424 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.424 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.424 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.424 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.424 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.424 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.424 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.424 INFO:teuthology.orchestra.run.smithi078.stdout: "device_url": { 2024-03-24T03:37:36.425 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_url", 2024-03-24T03:37:36.425 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.425 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.425 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.425 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-24T03:37:36.425 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.425 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.425 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.426 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.426 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.426 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.426 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.426 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": { 2024-03-24T03:37:36.426 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enabled", 2024-03-24T03:37:36.426 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.427 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.427 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.427 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.427 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.427 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.427 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.427 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.427 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.427 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.428 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.428 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-24T03:37:36.428 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-24T03:37:36.428 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:36.428 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.428 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.428 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "24", 2024-03-24T03:37:36.428 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "8", 2024-03-24T03:37:36.429 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.429 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.429 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.429 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.429 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.429 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.429 INFO:teuthology.orchestra.run.smithi078.stdout: "last_opt_revision": { 2024-03-24T03:37:36.429 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "last_opt_revision", 2024-03-24T03:37:36.430 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:36.430 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.430 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.430 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1", 2024-03-24T03:37:36.430 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.430 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.430 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.430 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.430 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.431 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.431 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.431 INFO:teuthology.orchestra.run.smithi078.stdout: "leaderboard": { 2024-03-24T03:37:36.431 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "leaderboard", 2024-03-24T03:37:36.431 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.431 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.431 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.431 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.432 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.432 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.432 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.432 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.432 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.432 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.432 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.432 INFO:teuthology.orchestra.run.smithi078.stdout: "leaderboard_description": { 2024-03-24T03:37:36.433 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "leaderboard_description", 2024-03-24T03:37:36.433 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.433 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.433 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.433 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.433 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.433 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.433 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.433 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.434 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.434 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.434 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.434 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:36.434 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:36.434 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.434 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.434 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.435 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.435 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.435 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.435 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.435 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.435 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.435 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.435 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.435 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.436 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.436 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.436 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.436 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.436 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.436 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.436 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:36.436 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:36.437 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.437 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.437 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.437 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.437 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.437 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.437 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.437 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.437 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.438 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.438 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.438 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:36.438 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:36.438 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.438 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.438 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.438 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:36.439 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.439 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.439 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.439 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.439 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.439 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.439 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.439 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.439 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.440 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.440 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.440 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.440 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.440 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.440 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:36.440 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:36.440 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.440 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.441 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.441 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.441 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.441 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.441 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.441 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.441 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.441 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.441 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.442 INFO:teuthology.orchestra.run.smithi078.stdout: "organization": { 2024-03-24T03:37:36.442 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "organization", 2024-03-24T03:37:36.442 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.442 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.442 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.442 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.442 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.442 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.443 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.443 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.443 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.443 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.443 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.443 INFO:teuthology.orchestra.run.smithi078.stdout: "proxy": { 2024-03-24T03:37:36.443 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "proxy", 2024-03-24T03:37:36.443 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.443 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.444 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.444 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.444 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.444 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.444 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.444 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.444 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.444 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.444 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.445 INFO:teuthology.orchestra.run.smithi078.stdout: "url": { 2024-03-24T03:37:36.445 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "url", 2024-03-24T03:37:36.445 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.445 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.445 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.445 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-24T03:37:36.445 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.446 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.446 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.446 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.446 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.446 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.446 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.446 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.446 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.447 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test_orchestrator", 2024-03-24T03:37:36.447 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:36.447 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:36.447 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:36.447 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:36.447 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:36.447 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.447 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.447 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.448 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.448 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.448 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.448 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.448 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.448 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.448 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.448 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.448 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.449 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.449 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.449 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.449 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.449 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.449 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.449 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:36.449 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:36.450 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.450 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.450 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.450 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.450 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.450 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.450 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.450 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.450 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.451 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.451 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.451 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:36.451 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:36.451 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.451 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.451 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.451 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:36.452 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.452 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.452 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.452 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.452 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.452 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.452 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.452 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.452 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.452 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.453 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.453 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.453 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.453 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.453 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:36.453 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:36.453 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.453 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.454 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.454 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.454 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.454 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.454 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.454 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.454 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.454 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.455 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.455 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.455 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.455 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "volumes", 2024-03-24T03:37:36.455 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:36.455 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:36.455 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:36.456 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:36.456 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:36.456 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.456 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.456 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.456 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.456 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.456 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.457 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.457 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.457 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.457 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.457 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.457 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.457 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.457 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.458 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.458 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.458 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.458 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.458 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:36.458 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:36.458 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.458 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.459 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.459 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.459 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.459 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.459 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.459 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.459 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.459 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.460 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.460 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:36.460 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:36.460 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.460 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.460 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.460 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:36.460 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.460 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.461 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.461 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.461 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.461 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.461 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.461 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.461 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.461 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.461 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.462 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.462 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.462 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.462 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:36.462 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:36.462 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.463 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.463 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.463 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.463 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.463 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.463 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.463 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.463 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.463 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.464 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.464 INFO:teuthology.orchestra.run.smithi078.stdout: "max_concurrent_clones": { 2024-03-24T03:37:36.464 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_concurrent_clones", 2024-03-24T03:37:36.464 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:36.464 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.464 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.464 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "4", 2024-03-24T03:37:36.464 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.465 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.465 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.465 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-24T03:37:36.465 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.465 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.465 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.465 INFO:teuthology.orchestra.run.smithi078.stdout: "periodic_async_work": { 2024-03-24T03:37:36.466 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "periodic_async_work", 2024-03-24T03:37:36.466 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.466 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.466 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.466 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.466 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.466 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.466 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.466 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Periodically check for async work", 2024-03-24T03:37:36.467 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.467 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.467 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.467 INFO:teuthology.orchestra.run.smithi078.stdout: "snapshot_clone_delay": { 2024-03-24T03:37:36.467 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snapshot_clone_delay", 2024-03-24T03:37:36.467 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:36.467 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.467 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.468 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-24T03:37:36.468 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.468 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.468 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.468 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-24T03:37:36.468 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.468 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.468 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.469 INFO:teuthology.orchestra.run.smithi078.stdout: "snapshot_clone_no_wait": { 2024-03-24T03:37:36.469 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snapshot_clone_no_wait", 2024-03-24T03:37:36.469 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.469 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.469 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.469 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-24T03:37:36.469 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.469 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.470 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.470 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-24T03:37:36.470 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.470 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.470 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.470 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.470 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.471 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix", 2024-03-24T03:37:36.471 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-24T03:37:36.471 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-24T03:37:36.471 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-24T03:37:36.471 INFO:teuthology.orchestra.run.smithi078.stdout: "discovery_interval": { 2024-03-24T03:37:36.471 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "discovery_interval", 2024-03-24T03:37:36.471 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-24T03:37:36.471 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.471 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.472 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "100", 2024-03-24T03:37:36.472 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.472 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.472 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.472 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.472 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.472 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.472 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.473 INFO:teuthology.orchestra.run.smithi078.stdout: "identifier": { 2024-03-24T03:37:36.473 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "identifier", 2024-03-24T03:37:36.473 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.473 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.473 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.473 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.473 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.473 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.473 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.474 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.474 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.474 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.474 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.474 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-24T03:37:36.474 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-24T03:37:36.474 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-24T03:37:36.474 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.475 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.475 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-24T03:37:36.475 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.475 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.475 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.475 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.475 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.475 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.476 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.476 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-24T03:37:36.476 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-24T03:37:36.476 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.476 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.476 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.476 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.476 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.477 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.477 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.477 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.477 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.477 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.477 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.477 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.477 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.477 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.477 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.478 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.478 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.478 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.478 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-24T03:37:36.478 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-24T03:37:36.478 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.478 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.479 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.479 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.479 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.479 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.479 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.479 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.479 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.479 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.479 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.480 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-24T03:37:36.480 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-24T03:37:36.480 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.480 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.480 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.480 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-24T03:37:36.480 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.480 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.481 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-24T03:37:36.481 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-24T03:37:36.481 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-24T03:37:36.481 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-24T03:37:36.481 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-24T03:37:36.481 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-24T03:37:36.481 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-24T03:37:36.481 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.481 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.482 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.482 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.482 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.482 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-24T03:37:36.482 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-24T03:37:36.482 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-24T03:37:36.482 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.482 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-24T03:37:36.483 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-24T03:37:36.483 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.483 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.483 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.483 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.483 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.483 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.483 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.484 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_host": { 2024-03-24T03:37:36.484 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_host", 2024-03-24T03:37:36.484 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.484 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.484 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.484 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-24T03:37:36.484 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.484 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.484 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.485 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.485 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.485 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.485 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.485 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_port": { 2024-03-24T03:37:36.485 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_port", 2024-03-24T03:37:36.485 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-24T03:37:36.485 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.486 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.486 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10051", 2024-03-24T03:37:36.486 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.486 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.486 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.486 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.486 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.486 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.487 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.487 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_sender": { 2024-03-24T03:37:36.487 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_sender", 2024-03-24T03:37:36.487 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-24T03:37:36.487 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-24T03:37:36.487 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-24T03:37:36.487 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-24T03:37:36.487 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-24T03:37:36.488 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-24T03:37:36.488 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-24T03:37:36.488 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-24T03:37:36.488 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-24T03:37:36.488 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-24T03:37:36.488 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-24T03:37:36.488 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.488 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.488 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.489 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.489 INFO:teuthology.orchestra.run.smithi078.stdout: "services": { 2024-03-24T03:37:36.489 INFO:teuthology.orchestra.run.smithi078.stdout: "dashboard": "https://172.21.15.78:8443/" 2024-03-24T03:37:36.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.489 INFO:teuthology.orchestra.run.smithi078.stdout: "always_on_modules": { 2024-03-24T03:37:36.489 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus": [ 2024-03-24T03:37:36.489 INFO:teuthology.orchestra.run.smithi078.stdout: "balancer", 2024-03-24T03:37:36.489 INFO:teuthology.orchestra.run.smithi078.stdout: "crash", 2024-03-24T03:37:36.489 INFO:teuthology.orchestra.run.smithi078.stdout: "devicehealth", 2024-03-24T03:37:36.490 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator", 2024-03-24T03:37:36.490 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscaler", 2024-03-24T03:37:36.490 INFO:teuthology.orchestra.run.smithi078.stdout: "progress", 2024-03-24T03:37:36.490 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_support", 2024-03-24T03:37:36.490 INFO:teuthology.orchestra.run.smithi078.stdout: "status", 2024-03-24T03:37:36.490 INFO:teuthology.orchestra.run.smithi078.stdout: "telemetry", 2024-03-24T03:37:36.490 INFO:teuthology.orchestra.run.smithi078.stdout: "volumes" 2024-03-24T03:37:36.490 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.490 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific": [ 2024-03-24T03:37:36.491 INFO:teuthology.orchestra.run.smithi078.stdout: "balancer", 2024-03-24T03:37:36.491 INFO:teuthology.orchestra.run.smithi078.stdout: "crash", 2024-03-24T03:37:36.491 INFO:teuthology.orchestra.run.smithi078.stdout: "devicehealth", 2024-03-24T03:37:36.491 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator", 2024-03-24T03:37:36.491 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscaler", 2024-03-24T03:37:36.491 INFO:teuthology.orchestra.run.smithi078.stdout: "progress", 2024-03-24T03:37:36.491 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_support", 2024-03-24T03:37:36.491 INFO:teuthology.orchestra.run.smithi078.stdout: "status", 2024-03-24T03:37:36.491 INFO:teuthology.orchestra.run.smithi078.stdout: "telemetry", 2024-03-24T03:37:36.491 INFO:teuthology.orchestra.run.smithi078.stdout: "volumes" 2024-03-24T03:37:36.492 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.492 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy": [ 2024-03-24T03:37:36.492 INFO:teuthology.orchestra.run.smithi078.stdout: "balancer", 2024-03-24T03:37:36.492 INFO:teuthology.orchestra.run.smithi078.stdout: "crash", 2024-03-24T03:37:36.492 INFO:teuthology.orchestra.run.smithi078.stdout: "devicehealth", 2024-03-24T03:37:36.492 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator", 2024-03-24T03:37:36.492 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscaler", 2024-03-24T03:37:36.492 INFO:teuthology.orchestra.run.smithi078.stdout: "progress", 2024-03-24T03:37:36.492 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_support", 2024-03-24T03:37:36.493 INFO:teuthology.orchestra.run.smithi078.stdout: "status", 2024-03-24T03:37:36.493 INFO:teuthology.orchestra.run.smithi078.stdout: "telemetry", 2024-03-24T03:37:36.493 INFO:teuthology.orchestra.run.smithi078.stdout: "volumes" 2024-03-24T03:37:36.493 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.493 INFO:teuthology.orchestra.run.smithi078.stdout: "reef": [ 2024-03-24T03:37:36.493 INFO:teuthology.orchestra.run.smithi078.stdout: "balancer", 2024-03-24T03:37:36.493 INFO:teuthology.orchestra.run.smithi078.stdout: "crash", 2024-03-24T03:37:36.493 INFO:teuthology.orchestra.run.smithi078.stdout: "devicehealth", 2024-03-24T03:37:36.493 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator", 2024-03-24T03:37:36.494 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscaler", 2024-03-24T03:37:36.494 INFO:teuthology.orchestra.run.smithi078.stdout: "progress", 2024-03-24T03:37:36.494 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_support", 2024-03-24T03:37:36.494 INFO:teuthology.orchestra.run.smithi078.stdout: "status", 2024-03-24T03:37:36.494 INFO:teuthology.orchestra.run.smithi078.stdout: "telemetry", 2024-03-24T03:37:36.494 INFO:teuthology.orchestra.run.smithi078.stdout: "volumes" 2024-03-24T03:37:36.494 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-24T03:37:36.494 INFO:teuthology.orchestra.run.smithi078.stdout: "squid": [ 2024-03-24T03:37:36.495 INFO:teuthology.orchestra.run.smithi078.stdout: "balancer", 2024-03-24T03:37:36.495 INFO:teuthology.orchestra.run.smithi078.stdout: "crash", 2024-03-24T03:37:36.495 INFO:teuthology.orchestra.run.smithi078.stdout: "devicehealth", 2024-03-24T03:37:36.495 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator", 2024-03-24T03:37:36.495 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscaler", 2024-03-24T03:37:36.495 INFO:teuthology.orchestra.run.smithi078.stdout: "progress", 2024-03-24T03:37:36.495 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_support", 2024-03-24T03:37:36.495 INFO:teuthology.orchestra.run.smithi078.stdout: "status", 2024-03-24T03:37:36.495 INFO:teuthology.orchestra.run.smithi078.stdout: "telemetry", 2024-03-24T03:37:36.496 INFO:teuthology.orchestra.run.smithi078.stdout: "volumes" 2024-03-24T03:37:36.496 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-24T03:37:36.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.496 INFO:teuthology.orchestra.run.smithi078.stdout: "last_failure_osd_epoch": 94, 2024-03-24T03:37:36.496 INFO:teuthology.orchestra.run.smithi078.stdout: "active_clients": [ 2024-03-24T03:37:36.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.496 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "devicehealth", 2024-03-24T03:37:36.496 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-03-24T03:37:36.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.497 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-03-24T03:37:36.497 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:0", 2024-03-24T03:37:36.497 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1438206358 2024-03-24T03:37:36.497 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.497 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-24T03:37:36.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.497 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "libcephsqlite", 2024-03-24T03:37:36.497 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-03-24T03:37:36.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.498 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-03-24T03:37:36.498 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:0", 2024-03-24T03:37:36.498 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 565773526 2024-03-24T03:37:36.498 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.498 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-24T03:37:36.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.498 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_support", 2024-03-24T03:37:36.499 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-03-24T03:37:36.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.499 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-03-24T03:37:36.499 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:0", 2024-03-24T03:37:36.499 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3607719889 2024-03-24T03:37:36.499 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.499 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-24T03:37:36.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.500 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snap_schedule", 2024-03-24T03:37:36.500 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-03-24T03:37:36.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.500 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-03-24T03:37:36.500 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:0", 2024-03-24T03:37:36.500 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1803481403 2024-03-24T03:37:36.500 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.500 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-24T03:37:36.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.501 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snap_schedule", 2024-03-24T03:37:36.501 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-03-24T03:37:36.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.501 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-03-24T03:37:36.501 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:0", 2024-03-24T03:37:36.501 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2545293742 2024-03-24T03:37:36.501 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.501 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-24T03:37:36.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-24T03:37:36.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.502 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "volumes", 2024-03-24T03:37:36.502 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-03-24T03:37:36.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-24T03:37:36.502 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-03-24T03:37:36.502 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:0", 2024-03-24T03:37:36.502 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 729636813 2024-03-24T03:37:36.502 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.503 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-24T03:37:36.503 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-24T03:37:36.503 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-24T03:37:36.503 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-24T03:37:36.505 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:35 smithi106 ceph-mon[36505]: pgmap v6237: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-03-24T03:37:36.505 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:35 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2338065178' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-24T03:37:36.505 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:34 smithi107 ceph-mon[36485]: from='client.15753 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T03:37:36.505 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10928.258801 secs 2024-03-24T03:37:36.505 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:34 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2052913777' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T03:37:36.505 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:34 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3812716474' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T03:37:36.506 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:34 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/390182772' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T03:37:36.506 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:34 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/789111460' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T03:37:36.506 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:35 smithi107 ceph-mon[36485]: pgmap v6237: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-03-24T03:37:36.506 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:35 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2338065178' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-24T03:37:36.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.426+0000 7f97cb80c640 1 -- 172.21.15.78:0/2338065178 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f97b00a2970 msgr2=0x7f97b00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:36.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.426+0000 7f97cb80c640 1 --2- 172.21.15.78:0/2338065178 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f97b00a2970 0x7f97b00a4e30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f97b4009990 tx=0x7f97b4009110 comp rx=0 tx=0).stop 2024-03-24T03:37:36.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.426+0000 7f97cb80c640 1 -- 172.21.15.78:0/2338065178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97c4106620 msgr2=0x7f97c419ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:36.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.426+0000 7f97cb80c640 1 --2- 172.21.15.78:0/2338065178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97c4106620 0x7f97c419ea20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f97bc009fd0 tx=0x7f97bc00c740 comp rx=0 tx=0).stop 2024-03-24T03:37:36.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.426+0000 7f97cb80c640 1 -- 172.21.15.78:0/2338065178 shutdown_connections 2024-03-24T03:37:36.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.426+0000 7f97cb80c640 1 --2- 172.21.15.78:0/2338065178 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f97b00a2970 0x7f97b00a4e30 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:36.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.426+0000 7f97cb80c640 1 --2- 172.21.15.78:0/2338065178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97c4106620 0x7f97c419ea20 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:36.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.426+0000 7f97cb80c640 1 --2- 172.21.15.78:0/2338065178 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97c4100f20 0x7f97c419a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:36.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.426+0000 7f97cb80c640 1 --2- 172.21.15.78:0/2338065178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97c4100560 0x7f97c419a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:36.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.426+0000 7f97cb80c640 1 -- 172.21.15.78:0/2338065178 >> 172.21.15.78:0/2338065178 conn(0x7f97c4076920 msgr2=0x7f97c40feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:37:36.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.426+0000 7f97cb80c640 1 -- 172.21.15.78:0/2338065178 shutdown_connections 2024-03-24T03:37:36.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:34.426+0000 7f97cb80c640 1 -- 172.21.15.78:0/2338065178 wait complete. 2024-03-24T03:37:36.513 DEBUG:tasks.check_counter:Getting stats from a 2024-03-24T03:37:36.514 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-24T03:37:36.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:35 smithi078 ceph-mon[32429]: pgmap v6237: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-03-24T03:37:36.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:35 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2338065178' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-24T03:37:36.760 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T03:37:37.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:37 smithi078 ceph-mon[32429]: pgmap v6238: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:37:37.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:37 smithi106 ceph-mon[36505]: pgmap v6238: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:37:37.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:37 smithi107 ceph-mon[36485]: pgmap v6238: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:37:38.563 INFO:teuthology.orchestra.run.smithi078.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-24T03:37:39.175 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10933.258841 secs 2024-03-24T03:37:39.176 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:39 smithi078 ceph-mon[32429]: pgmap v6239: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:37:39.203 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T03:37:39.203 ERROR:teuthology.run_tasks:Manager failed: check-counter Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/check_counter.py", line 84, in end proc = manager.admin_socket(daemon_type, daemon_id, ["perf", "dump"]) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1875, in admin_socket return shell( File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 43, in shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi078 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid c7e2806a-e972-11ee-95cd-87774f69a715 -- ceph daemon mds.a perf dump' 2024-03-24T03:37:39.211 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-24T03:37:39.222 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-24T03:37:39.222 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-24T03:37:39.222 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T03:37:39.241 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-24T03:37:39.241 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T03:37:39.297 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T03:37:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10933.258841 secs 2024-03-24T03:37:39.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:39 smithi106 ceph-mon[36505]: pgmap v6239: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:37:39.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.441+0000 7f4c8c2b2640 1 -- 172.21.15.78:0/2098079461 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c84101ff0 msgr2=0x7f4c8410e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:39.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.441+0000 7f4c8c2b2640 1 --2- 172.21.15.78:0/2098079461 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c84101ff0 0x7f4c8410e500 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f4c74009cd0 tx=0x7f4c7402c0f0 comp rx=0 tx=0).stop 2024-03-24T03:37:39.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.441+0000 7f4c8c2b2640 1 -- 172.21.15.78:0/2098079461 shutdown_connections 2024-03-24T03:37:39.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.441+0000 7f4c8c2b2640 1 --2- 172.21.15.78:0/2098079461 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c84101ff0 0x7f4c8410e500 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:39.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.441+0000 7f4c8c2b2640 1 --2- 172.21.15.78:0/2098079461 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4c84101630 0x7f4c84101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:39.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.441+0000 7f4c8c2b2640 1 --2- 172.21.15.78:0/2098079461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c84107640 0x7f4c84107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:39.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.441+0000 7f4c8c2b2640 1 -- 172.21.15.78:0/2098079461 >> 172.21.15.78:0/2098079461 conn(0x7f4c840fd140 msgr2=0x7f4c840ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:37:39.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.441+0000 7f4c8c2b2640 1 -- 172.21.15.78:0/2098079461 shutdown_connections 2024-03-24T03:37:39.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.441+0000 7f4c8c2b2640 1 -- 172.21.15.78:0/2098079461 wait complete. 2024-03-24T03:37:39.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.442+0000 7f4c8c2b2640 1 Processor -- start 2024-03-24T03:37:39.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.442+0000 7f4c8c2b2640 1 -- start start 2024-03-24T03:37:39.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.442+0000 7f4c8c2b2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c84101630 0x7f4c84197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:39.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.442+0000 7f4c8c2b2640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4c84101ff0 0x7f4c84198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:39.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.442+0000 7f4c8c2b2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c84107640 0x7f4c8419c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:39.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.442+0000 7f4c8c2b2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c84111a50 con 0x7f4c84107640 2024-03-24T03:37:39.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.442+0000 7f4c8c2b2640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c841118d0 con 0x7f4c84101ff0 2024-03-24T03:37:39.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.442+0000 7f4c8c2b2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c84111bd0 con 0x7f4c84101630 2024-03-24T03:37:39.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.442+0000 7f4c8a84f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c84107640 0x7f4c8419c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:39.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.442+0000 7f4c8a84f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c84107640 0x7f4c8419c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56330/0 (socket says 172.21.15.78:56330) 2024-03-24T03:37:39.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.442+0000 7f4c8a84f640 1 -- 172.21.15.78:0/1687077806 learned_addr learned my addr 172.21.15.78:0/1687077806 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T03:37:39.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.442+0000 7f4c8984d640 1 --2- 172.21.15.78:0/1687077806 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4c84101ff0 0x7f4c84198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:39.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.443+0000 7f4c8a84f640 1 -- 172.21.15.78:0/1687077806 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c84101630 msgr2=0x7f4c84197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:39.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.443+0000 7f4c8a04e640 1 --2- 172.21.15.78:0/1687077806 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c84101630 0x7f4c84197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:39.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.443+0000 7f4c8a84f640 1 --2- 172.21.15.78:0/1687077806 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c84101630 0x7f4c84197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:39.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.443+0000 7f4c8a84f640 1 -- 172.21.15.78:0/1687077806 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4c84101ff0 msgr2=0x7f4c84198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:39.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.443+0000 7f4c8a84f640 1 --2- 172.21.15.78:0/1687077806 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4c84101ff0 0x7f4c84198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:39.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.443+0000 7f4c8a84f640 1 -- 172.21.15.78:0/1687077806 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c8419cf20 con 0x7f4c84107640 2024-03-24T03:37:39.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.443+0000 7f4c8a04e640 1 --2- 172.21.15.78:0/1687077806 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c84101630 0x7f4c84197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T03:37:39.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.443+0000 7f4c8a84f640 1 --2- 172.21.15.78:0/1687077806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c84107640 0x7f4c8419c7f0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f4c74004af0 tx=0x7f4c74032ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:37:39.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.443+0000 7f4c837fe640 1 -- 172.21.15.78:0/1687077806 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c74041020 con 0x7f4c84107640 2024-03-24T03:37:39.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.443+0000 7f4c837fe640 1 -- 172.21.15.78:0/1687077806 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4c74034cd0 con 0x7f4c84107640 2024-03-24T03:37:39.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.443+0000 7f4c8c2b2640 1 -- 172.21.15.78:0/1687077806 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c8419d1b0 con 0x7f4c84107640 2024-03-24T03:37:39.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.443+0000 7f4c837fe640 1 -- 172.21.15.78:0/1687077806 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c74039040 con 0x7f4c84107640 2024-03-24T03:37:39.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.443+0000 7f4c8c2b2640 1 -- 172.21.15.78:0/1687077806 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c841a4aa0 con 0x7f4c84107640 2024-03-24T03:37:39.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.444+0000 7f4c8c2b2640 1 -- 172.21.15.78:0/1687077806 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c840688e0 con 0x7f4c84107640 2024-03-24T03:37:39.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.445+0000 7f4c837fe640 1 -- 172.21.15.78:0/1687077806 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f4c74044020 con 0x7f4c84107640 2024-03-24T03:37:39.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.446+0000 7f4c837fe640 1 --2- 172.21.15.78:0/1687077806 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f4c6c0a2970 0x7f4c6c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:39.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.446+0000 7f4c8a04e640 1 --2- 172.21.15.78:0/1687077806 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f4c6c0a2970 0x7f4c6c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:39.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.446+0000 7f4c837fe640 1 -- 172.21.15.78:0/1687077806 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 13278+0+0 (secure 0 0 0) 0x7f4c740f21a0 con 0x7f4c84107640 2024-03-24T03:37:39.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.446+0000 7f4c8a04e640 1 --2- 172.21.15.78:0/1687077806 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f4c6c0a2970 0x7f4c6c0a4e30 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c0058c0 tx=0x7f4c7c005cb0 comp rx=0 tx=0).ready entity=mgr.25243 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:37:39.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.449+0000 7f4c837fe640 1 -- 172.21.15.78:0/1687077806 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4c7404b020 con 0x7f4c84107640 2024-03-24T03:37:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10933.258841 secs 2024-03-24T03:37:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:39 smithi107 ceph-mon[36485]: pgmap v6239: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:37:39.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.627+0000 7f4c8c2b2640 1 -- 172.21.15.78:0/1687077806 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c84109bd0 con 0x7f4c84107640 2024-03-24T03:37:39.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.629+0000 7f4c837fe640 1 -- 172.21.15.78:0/1687077806 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v157) v1 ==== 74+0+30992 (secure 0 0 0) 0x7f4c740bdd80 con 0x7f4c84107640 2024-03-24T03:37:39.629 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T03:37:39.629 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":157,"fsid":"c7e2806a-e972-11ee-95cd-87774f69a715","created":"2024-03-24T00:10:44.134563+0000","modified":"2024-03-24T01:24:14.909719+0000","last_up_change":"2024-03-24T00:17:50.237686+0000","last_in_change":"2024-03-24T00:17:29.788699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:14:04.534391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:18:42.178418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:18:42.800424+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":23,"snap_epoch":155,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:18:46.112800+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":23,"snap_epoch":155,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b950388a-c437-426b-b703-3f8ae821dd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6803","nonce":1816879397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6805","nonce":1816879397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6809","nonce":1816879397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6807","nonce":1816879397}]},"public_addr":"172.21.15.78:6803/1816879397","cluster_addr":"172.21.15.78:6805/1816879397","heartbeat_back_addr":"172.21.15.78:6809/1816879397","heartbeat_front_addr":"172.21.15.78:6807/1816879397","state":["exists","up"]},{"osd":1,"uuid":"b8d64714-dd65-400f-b3f4-673ab14e1365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6801","nonce":1647915612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6803","nonce":1647915612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6807","nonce":1647915612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6805","nonce":1647915612}]},"public_addr":"172.21.15.106:6801/1647915612","cluster_addr":"172.21.15.106:6803/1647915612","heartbeat_back_addr":"172.21.15.106:6807/1647915612","heartbeat_front_addr":"172.21.15.106:6805/1647915612","state":["exists","up"]},{"osd":2,"uuid":"fcfc4dd1-aaf8-4131-a023-5e832212158d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6801","nonce":2010770435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6803","nonce":2010770435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6807","nonce":2010770435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6805","nonce":2010770435}]},"public_addr":"172.21.15.107:6801/2010770435","cluster_addr":"172.21.15.107:6803/2010770435","heartbeat_back_addr":"172.21.15.107:6807/2010770435","heartbeat_front_addr":"172.21.15.107:6805/2010770435","state":["exists","up"]},{"osd":3,"uuid":"47c718ab-a0f4-4e52-83c5-b89386b262d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6811","nonce":1912251346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6813","nonce":1912251346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6817","nonce":1912251346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6815","nonce":1912251346}]},"public_addr":"172.21.15.78:6811/1912251346","cluster_addr":"172.21.15.78:6813/1912251346","heartbeat_back_addr":"172.21.15.78:6817/1912251346","heartbeat_front_addr":"172.21.15.78:6815/1912251346","state":["exists","up"]},{"osd":4,"uuid":"7dad64dd-677e-4c1c-9638-e4f4f2195983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6809","nonce":374641205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6811","nonce":374641205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6815","nonce":374641205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6813","nonce":374641205}]},"public_addr":"172.21.15.106:6809/374641205","cluster_addr":"172.21.15.106:6811/374641205","heartbeat_back_addr":"172.21.15.106:6815/374641205","heartbeat_front_addr":"172.21.15.106:6813/374641205","state":["exists","up"]},{"osd":5,"uuid":"bbd524e5-328d-4309-8877-1149cbe110dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6809","nonce":3272806073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6811","nonce":3272806073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6815","nonce":3272806073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6813","nonce":3272806073}]},"public_addr":"172.21.15.107:6809/3272806073","cluster_addr":"172.21.15.107:6811/3272806073","heartbeat_back_addr":"172.21.15.107:6815/3272806073","heartbeat_front_addr":"172.21.15.107:6813/3272806073","state":["exists","up"]},{"osd":6,"uuid":"5696389a-918f-4ea8-b20d-4e2a0d4cb3bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6819","nonce":3091148508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6821","nonce":3091148508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6825","nonce":3091148508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6823","nonce":3091148508}]},"public_addr":"172.21.15.78:6819/3091148508","cluster_addr":"172.21.15.78:6821/3091148508","heartbeat_back_addr":"172.21.15.78:6825/3091148508","heartbeat_front_addr":"172.21.15.78:6823/3091148508","state":["exists","up"]},{"osd":7,"uuid":"f6b764e8-ed4e-4c9a-a85b-94af6dc6b924","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6817","nonce":4262103080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6819","nonce":4262103080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6823","nonce":4262103080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6821","nonce":4262103080}]},"public_addr":"172.21.15.106:6817/4262103080","cluster_addr":"172.21.15.106:6819/4262103080","heartbeat_back_addr":"172.21.15.106:6823/4262103080","heartbeat_front_addr":"172.21.15.106:6821/4262103080","state":["exists","up"]},{"osd":8,"uuid":"ba16039f-8093-4fa0-87ad-e28b7d142b23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6817","nonce":4032799356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6819","nonce":4032799356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6823","nonce":4032799356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6821","nonce":4032799356}]},"public_addr":"172.21.15.107:6817/4032799356","cluster_addr":"172.21.15.107:6819/4032799356","heartbeat_back_addr":"172.21.15.107:6823/4032799356","heartbeat_front_addr":"172.21.15.107:6821/4032799356","state":["exists","up"]},{"osd":9,"uuid":"651357f2-0669-42e6-aa91-1d5c02d00c5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6827","nonce":214023431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6829","nonce":214023431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6833","nonce":214023431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6831","nonce":214023431}]},"public_addr":"172.21.15.78:6827/214023431","cluster_addr":"172.21.15.78:6829/214023431","heartbeat_back_addr":"172.21.15.78:6833/214023431","heartbeat_front_addr":"172.21.15.78:6831/214023431","state":["exists","up"]},{"osd":10,"uuid":"2375119d-346e-426c-8a65-b64af71ad63a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6825","nonce":400029804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6827","nonce":400029804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6831","nonce":400029804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6829","nonce":400029804}]},"public_addr":"172.21.15.106:6825/400029804","cluster_addr":"172.21.15.106:6827/400029804","heartbeat_back_addr":"172.21.15.106:6831/400029804","heartbeat_front_addr":"172.21.15.106:6829/400029804","state":["exists","up"]},{"osd":11,"uuid":"b6b609d9-dd78-4c01-ae66-ba2a662ab22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6825","nonce":4191916230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6827","nonce":4191916230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6831","nonce":4191916230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6829","nonce":4191916230}]},"public_addr":"172.21.15.107:6825/4191916230","cluster_addr":"172.21.15.107:6827/4191916230","heartbeat_back_addr":"172.21.15.107:6831/4191916230","heartbeat_front_addr":"172.21.15.107:6829/4191916230","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:13.728602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:37.320809+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:01.269452+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:25.758281+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:50.298863+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:14.345219+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:38.980489+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:04.749214+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:29.992673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:56.269202+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:22.270316+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:48.074907+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6834/1985874955":"2024-03-25T00:35:20.737219+0000","172.21.15.78:6835/3414536872":"2024-03-25T00:35:04.921316+0000","172.21.15.78:6834/3414536872":"2024-03-25T00:35:04.921316+0000","172.21.15.78:6837/2612396245":"2024-03-25T00:34:49.129252+0000","172.21.15.78:6834/93340378":"2024-03-25T00:34:30.894376+0000","172.21.15.78:6837/878237454":"2024-03-25T00:34:15.858120+0000","172.21.15.78:6836/878237454":"2024-03-25T00:34:15.858120+0000","172.21.15.78:6835/2163967090":"2024-03-25T00:34:00.891452+0000","172.21.15.78:6834/2163967090":"2024-03-25T00:34:00.891452+0000","172.21.15.78:6836/1186751954":"2024-03-25T00:33:45.900090+0000","172.21.15.78:6835/1416685123":"2024-03-25T00:33:30.602155+0000","172.21.15.78:6834/1416685123":"2024-03-25T00:33:30.602155+0000","172.21.15.78:6836/1077781429":"2024-03-25T00:33:14.880383+0000","172.21.15.78:6839/2480119933":"2024-03-25T00:32:59.099199+0000","172.21.15.78:6838/2480119933":"2024-03-25T00:32:59.099199+0000","172.21.15.78:6837/2239323994":"2024-03-25T00:32:40.627803+0000","172.21.15.78:6839/3377749691":"2024-03-25T00:32:25.708147+0000","172.21.15.78:6836/1071274623":"2024-03-25T00:32:10.632135+0000","172.21.15.78:6839/4176145760":"2024-03-25T00:31:55.146631+0000","172.21.15.78:6838/4176145760":"2024-03-25T00:31:55.146631+0000","172.21.15.78:6840/1658072731":"2024-03-25T00:31:39.075579+0000","172.21.15.78:0/442826396":"2024-03-25T00:21:09.220548+0000","172.21.15.107:6832/4067467714":"2024-03-25T00:29:09.638984+0000","172.21.15.78:6837/1077781429":"2024-03-25T00:33:14.880383+0000","172.21.15.107:6839/3131891440":"2024-03-25T00:24:40.372092+0000","172.21.15.107:6838/3949153161":"2024-03-25T00:25:43.971852+0000","172.21.15.107:6833/794068267":"2024-03-25T00:28:05.611074+0000","172.21.15.107:6838/3131891440":"2024-03-25T00:24:40.372092+0000","172.21.15.107:6833/1751539133":"2024-03-25T00:27:35.655923+0000","172.21.15.107:6834/4046882186":"2024-03-25T00:24:25.648083+0000","172.21.15.107:6839/3949153161":"2024-03-25T00:25:43.971852+0000","172.21.15.78:6835/1985874955":"2024-03-25T00:35:20.737219+0000","172.21.15.78:6801/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6841/3424505910":"2024-03-25T00:29:25.370378+0000","172.21.15.78:0/1982202541":"2024-03-25T00:11:40.199569+0000","172.21.15.107:6834/108936439":"2024-03-25T00:23:06.644918+0000","172.21.15.107:6832/794068267":"2024-03-25T00:28:05.611074+0000","172.21.15.78:6838/3377749691":"2024-03-25T00:32:25.708147+0000","172.21.15.107:6839/3531100722":"2024-03-25T00:23:22.383674+0000","172.21.15.107:6838/117119462":"2024-03-25T00:22:50.169074+0000","172.21.15.78:0/2236879401":"2024-03-25T00:11:40.199569+0000","172.21.15.107:6833/3086318443":"2024-03-25T00:27:05.852992+0000","172.21.15.78:6837/1186751954":"2024-03-25T00:33:45.900090+0000","172.21.15.107:6834/23104891":"2024-03-25T00:24:55.644935+0000","172.21.15.78:0/1502752625":"2024-03-25T00:11:14.637881+0000","172.21.15.107:6834/2447512361":"2024-03-25T00:26:33.985150+0000","172.21.15.78:0/359613096":"2024-03-25T00:21:09.220548+0000","172.21.15.78:6800/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6801/1983101026":"2024-03-25T00:21:09.220548+0000","172.21.15.78:0/465691596":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/3962523609":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/1510740971":"2024-03-25T00:21:09.220548+0000","172.21.15.78:6840/1960340276":"2024-03-25T00:31:02.121119+0000","172.21.15.78:6836/2612396245":"2024-03-25T00:34:49.129252+0000","172.21.15.78:6801/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6800/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6836/2239323994":"2024-03-25T00:32:40.627803+0000","172.21.15.107:6835/2950120609":"2024-03-25T00:25:25.629098+0000","172.21.15.78:0/1933054339":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6838/4117918862":"2024-03-25T00:31:18.139126+0000","172.21.15.107:6838/607202225":"2024-03-25T00:25:10.629186+0000","172.21.15.78:0/1685151467":"2024-03-25T00:21:09.220548+0000","172.21.15.107:6838/3531100722":"2024-03-25T00:23:22.383674+0000","172.21.15.78:6838/2758357551":"2024-03-25T00:30:15.603375+0000","172.21.15.78:6800/1983101026":"2024-03-25T00:21:09.220548+0000","172.21.15.107:6835/758053763":"2024-03-25T00:25:59.620946+0000","172.21.15.107:6835/4046882186":"2024-03-25T00:24:25.648083+0000","172.21.15.107:6835/108936439":"2024-03-25T00:23:06.644918+0000","172.21.15.106:6833/2367018868":"2024-03-25T00:26:15.443626+0000","172.21.15.78:6835/93340378":"2024-03-25T00:34:30.894376+0000","172.21.15.107:6835/2447512361":"2024-03-25T00:26:33.985150+0000","172.21.15.107:6835/23104891":"2024-03-25T00:24:55.644935+0000","172.21.15.106:6832/2611554095":"2024-03-25T00:26:49.797488+0000","172.21.15.106:6833/2611554095":"2024-03-25T00:26:49.797488+0000","172.21.15.107:6832/3086318443":"2024-03-25T00:27:05.852992+0000","172.21.15.106:6832/2299054161":"2024-03-25T00:27:20.754602+0000","172.21.15.107:6832/1006503957":"2024-03-25T00:29:44.040588+0000","172.21.15.106:6832/2367018868":"2024-03-25T00:26:15.443626+0000","172.21.15.106:6833/2299054161":"2024-03-25T00:27:20.754602+0000","172.21.15.107:6832/1751539133":"2024-03-25T00:27:35.655923+0000","172.21.15.106:6832/459568830":"2024-03-25T00:27:50.719456+0000","172.21.15.106:6833/459568830":"2024-03-25T00:27:50.719456+0000","172.21.15.106:6832/93515314":"2024-03-25T00:28:20.713453+0000","172.21.15.107:6833/3928685127":"2024-03-25T00:28:35.891033+0000","172.21.15.107:6839/117119462":"2024-03-25T00:22:50.169074+0000","172.21.15.106:6832/2104934388":"2024-03-25T00:28:54.025064+0000","172.21.15.78:6837/1071274623":"2024-03-25T00:32:10.632135+0000","172.21.15.106:6833/2104934388":"2024-03-25T00:28:54.025064+0000","172.21.15.78:6841/1658072731":"2024-03-25T00:31:39.075579+0000","172.21.15.107:6833/4067467714":"2024-03-25T00:29:09.638984+0000","172.21.15.78:6840/3424505910":"2024-03-25T00:29:25.370378+0000","172.21.15.107:6834/758053763":"2024-03-25T00:25:59.620946+0000","172.21.15.78:6838/2645579087":"2024-03-25T00:30:46.105435+0000","172.21.15.107:6834/2950120609":"2024-03-25T00:25:25.629098+0000","172.21.15.78:6840/1196331852":"2024-03-25T00:29:59.753104+0000","172.21.15.107:6833/1006503957":"2024-03-25T00:29:44.040588+0000","172.21.15.78:6841/1196331852":"2024-03-25T00:29:59.753104+0000","172.21.15.107:6839/607202225":"2024-03-25T00:25:10.629186+0000","172.21.15.78:6841/1759489401":"2024-03-25T00:30:30.711566+0000","172.21.15.78:6839/2758357551":"2024-03-25T00:30:15.603375+0000","172.21.15.78:6840/1759489401":"2024-03-25T00:30:30.711566+0000","172.21.15.78:6839/2645579087":"2024-03-25T00:30:46.105435+0000","172.21.15.107:6832/3928685127":"2024-03-25T00:28:35.891033+0000","172.21.15.78:6841/1960340276":"2024-03-25T00:31:02.121119+0000","172.21.15.106:6833/93515314":"2024-03-25T00:28:20.713453+0000","172.21.15.78:6839/4117918862":"2024-03-25T00:31:18.139126+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T03:37:39.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.631+0000 7f4c8c2b2640 1 -- 172.21.15.78:0/1687077806 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f4c6c0a2970 msgr2=0x7f4c6c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:39.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.631+0000 7f4c8c2b2640 1 --2- 172.21.15.78:0/1687077806 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f4c6c0a2970 0x7f4c6c0a4e30 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c0058c0 tx=0x7f4c7c005cb0 comp rx=0 tx=0).stop 2024-03-24T03:37:39.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.631+0000 7f4c8c2b2640 1 -- 172.21.15.78:0/1687077806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c84107640 msgr2=0x7f4c8419c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:39.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.631+0000 7f4c8c2b2640 1 --2- 172.21.15.78:0/1687077806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c84107640 0x7f4c8419c7f0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f4c74004af0 tx=0x7f4c74032ec0 comp rx=0 tx=0).stop 2024-03-24T03:37:39.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.631+0000 7f4c8c2b2640 1 -- 172.21.15.78:0/1687077806 shutdown_connections 2024-03-24T03:37:39.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.631+0000 7f4c8c2b2640 1 --2- 172.21.15.78:0/1687077806 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f4c6c0a2970 0x7f4c6c0a4e30 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:39.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.631+0000 7f4c8c2b2640 1 --2- 172.21.15.78:0/1687077806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c84107640 0x7f4c8419c7f0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:39.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.631+0000 7f4c8c2b2640 1 --2- 172.21.15.78:0/1687077806 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4c84101ff0 0x7f4c84198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:39.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.631+0000 7f4c8c2b2640 1 --2- 172.21.15.78:0/1687077806 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c84101630 0x7f4c84197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:39.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.631+0000 7f4c8c2b2640 1 -- 172.21.15.78:0/1687077806 >> 172.21.15.78:0/1687077806 conn(0x7f4c840fd140 msgr2=0x7f4c840fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:37:39.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.631+0000 7f4c8c2b2640 1 -- 172.21.15.78:0/1687077806 shutdown_connections 2024-03-24T03:37:39.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.631+0000 7f4c8c2b2640 1 -- 172.21.15.78:0/1687077806 wait complete. 2024-03-24T03:37:39.642 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:14:04.534391+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:18:42.178418+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:18:42.800424+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '155', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 23, 'snap_epoch': 155, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T00:18:46.112800+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '155', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 23, 'snap_epoch': 155, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T03:37:39.643 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T03:37:39.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.749+0000 7f5dcb73b640 1 -- 172.21.15.78:0/893309698 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dc41065f0 msgr2=0x7f5dc41069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:39.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.749+0000 7f5dcb73b640 1 --2- 172.21.15.78:0/893309698 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dc41065f0 0x7f5dc41069d0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0009ca0 tx=0x7f5dc002c0f0 comp rx=0 tx=0).stop 2024-03-24T03:37:39.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.749+0000 7f5dcb73b640 1 -- 172.21.15.78:0/893309698 shutdown_connections 2024-03-24T03:37:39.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.749+0000 7f5dcb73b640 1 --2- 172.21.15.78:0/893309698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5dc4100ef0 0x7f5dc410d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:39.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.749+0000 7f5dcb73b640 1 --2- 172.21.15.78:0/893309698 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5dc4100530 0x7f5dc41009b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:39.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.749+0000 7f5dcb73b640 1 --2- 172.21.15.78:0/893309698 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dc41065f0 0x7f5dc41069d0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:39.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.749+0000 7f5dcb73b640 1 -- 172.21.15.78:0/893309698 >> 172.21.15.78:0/893309698 conn(0x7f5dc40776a0 msgr2=0x7f5dc4077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:37:39.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.749+0000 7f5dcb73b640 1 -- 172.21.15.78:0/893309698 shutdown_connections 2024-03-24T03:37:39.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.749+0000 7f5dcb73b640 1 -- 172.21.15.78:0/893309698 wait complete. 2024-03-24T03:37:39.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.749+0000 7f5dcb73b640 1 Processor -- start 2024-03-24T03:37:39.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.749+0000 7f5dcb73b640 1 -- start start 2024-03-24T03:37:39.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.749+0000 7f5dcb73b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dc4100530 0x7f5dc4196dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:39.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.749+0000 7f5dcb73b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5dc4100ef0 0x7f5dc4197310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:39.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.749+0000 7f5dcb73b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5dc41065f0 0x7f5dc419b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:39.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.749+0000 7f5dcb73b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc41108b0 con 0x7f5dc4100530 2024-03-24T03:37:39.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.749+0000 7f5dcb73b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc4110730 con 0x7f5dc4100ef0 2024-03-24T03:37:39.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.749+0000 7f5dcb73b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc4110a30 con 0x7f5dc41065f0 2024-03-24T03:37:39.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.750+0000 7f5dc94d7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dc4100530 0x7f5dc4196dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:39.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.750+0000 7f5dc94d7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dc4100530 0x7f5dc4196dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56346/0 (socket says 172.21.15.78:56346) 2024-03-24T03:37:39.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.750+0000 7f5dc94d7640 1 -- 172.21.15.78:0/3481467441 learned_addr learned my addr 172.21.15.78:0/3481467441 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T03:37:39.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.750+0000 7f5dc8cd6640 1 --2- 172.21.15.78:0/3481467441 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5dc4100ef0 0x7f5dc4197310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:39.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.750+0000 7f5dc9cd8640 1 --2- 172.21.15.78:0/3481467441 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5dc41065f0 0x7f5dc419b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:39.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.750+0000 7f5dc94d7640 1 -- 172.21.15.78:0/3481467441 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5dc41065f0 msgr2=0x7f5dc419b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:39.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.750+0000 7f5dc94d7640 1 --2- 172.21.15.78:0/3481467441 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5dc41065f0 0x7f5dc419b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:39.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.750+0000 7f5dc94d7640 1 -- 172.21.15.78:0/3481467441 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5dc4100ef0 msgr2=0x7f5dc4197310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:39.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.750+0000 7f5dc94d7640 1 --2- 172.21.15.78:0/3481467441 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5dc4100ef0 0x7f5dc4197310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:39.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.750+0000 7f5dc94d7640 1 -- 172.21.15.78:0/3481467441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc419bdc0 con 0x7f5dc4100530 2024-03-24T03:37:39.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.750+0000 7f5dc9cd8640 1 --2- 172.21.15.78:0/3481467441 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5dc41065f0 0x7f5dc419b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T03:37:39.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.750+0000 7f5dc94d7640 1 --2- 172.21.15.78:0/3481467441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dc4100530 0x7f5dc4196dd0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0005e30 tx=0x7f5dc0032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:37:39.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.750+0000 7f5dba7fc640 1 -- 172.21.15.78:0/3481467441 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0042020 con 0x7f5dc4100530 2024-03-24T03:37:39.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.750+0000 7f5dcb73b640 1 -- 172.21.15.78:0/3481467441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dc419c050 con 0x7f5dc4100530 2024-03-24T03:37:39.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.750+0000 7f5dba7fc640 1 -- 172.21.15.78:0/3481467441 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5dc0034700 con 0x7f5dc4100530 2024-03-24T03:37:39.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.750+0000 7f5dba7fc640 1 -- 172.21.15.78:0/3481467441 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0039040 con 0x7f5dc4100530 2024-03-24T03:37:39.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.750+0000 7f5dcb73b640 1 -- 172.21.15.78:0/3481467441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dc41a3940 con 0x7f5dc4100530 2024-03-24T03:37:39.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.751+0000 7f5dcb73b640 1 -- 172.21.15.78:0/3481467441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dc4101c50 con 0x7f5dc4100530 2024-03-24T03:37:39.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.752+0000 7f5dba7fc640 1 -- 172.21.15.78:0/3481467441 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f5dc0045020 con 0x7f5dc4100530 2024-03-24T03:37:39.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.753+0000 7f5dba7fc640 1 --2- 172.21.15.78:0/3481467441 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f5db00a2970 0x7f5db00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:39.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.753+0000 7f5dc8cd6640 1 --2- 172.21.15.78:0/3481467441 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f5db00a2970 0x7f5db00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:39.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.753+0000 7f5dba7fc640 1 -- 172.21.15.78:0/3481467441 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 13278+0+0 (secure 0 0 0) 0x7f5dc00f1d10 con 0x7f5dc4100530 2024-03-24T03:37:39.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.753+0000 7f5dc8cd6640 1 --2- 172.21.15.78:0/3481467441 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f5db00a2970 0x7f5db00a4e30 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4105c20 tx=0x7f5db4005f90 comp rx=0 tx=0).ready entity=mgr.25243 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:37:39.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.757+0000 7f5dba7fc640 1 -- 172.21.15.78:0/3481467441 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5dc004c020 con 0x7f5dc4100530 2024-03-24T03:37:39.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.935+0000 7f5dcb73b640 1 -- 172.21.15.78:0/3481467441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5dc4108a30 con 0x7f5dc4100530 2024-03-24T03:37:39.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.935+0000 7f5dba7fc640 1 -- 172.21.15.78:0/3481467441 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v157) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5dc00bd8f0 con 0x7f5dc4100530 2024-03-24T03:37:39.935 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-24T03:37:39.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.937+0000 7f5dcb73b640 1 -- 172.21.15.78:0/3481467441 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f5db00a2970 msgr2=0x7f5db00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:39.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.937+0000 7f5dcb73b640 1 --2- 172.21.15.78:0/3481467441 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f5db00a2970 0x7f5db00a4e30 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4105c20 tx=0x7f5db4005f90 comp rx=0 tx=0).stop 2024-03-24T03:37:39.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.937+0000 7f5dcb73b640 1 -- 172.21.15.78:0/3481467441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dc4100530 msgr2=0x7f5dc4196dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:39.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.937+0000 7f5dcb73b640 1 --2- 172.21.15.78:0/3481467441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dc4100530 0x7f5dc4196dd0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0005e30 tx=0x7f5dc0032f40 comp rx=0 tx=0).stop 2024-03-24T03:37:39.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.937+0000 7f5dcb73b640 1 -- 172.21.15.78:0/3481467441 shutdown_connections 2024-03-24T03:37:39.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.937+0000 7f5dcb73b640 1 --2- 172.21.15.78:0/3481467441 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f5db00a2970 0x7f5db00a4e30 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:39.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.937+0000 7f5dcb73b640 1 --2- 172.21.15.78:0/3481467441 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5dc41065f0 0x7f5dc419b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:39.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.937+0000 7f5dcb73b640 1 --2- 172.21.15.78:0/3481467441 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5dc4100ef0 0x7f5dc4197310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:39.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.937+0000 7f5dcb73b640 1 --2- 172.21.15.78:0/3481467441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dc4100530 0x7f5dc4196dd0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:39.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.937+0000 7f5dcb73b640 1 -- 172.21.15.78:0/3481467441 >> 172.21.15.78:0/3481467441 conn(0x7f5dc40776a0 msgr2=0x7f5dc40ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:37:39.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.937+0000 7f5dcb73b640 1 -- 172.21.15.78:0/3481467441 shutdown_connections 2024-03-24T03:37:39.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:39.937+0000 7f5dcb73b640 1 -- 172.21.15.78:0/3481467441 wait complete. 2024-03-24T03:37:39.948 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T03:37:40.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.053+0000 7f1f7a145640 1 -- 172.21.15.78:0/3183550162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f74100460 msgr2=0x7f1f74100840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:40.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.053+0000 7f1f7a145640 1 --2- 172.21.15.78:0/3183550162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f74100460 0x7f1f74100840 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f1f64009900 tx=0x7f1f6402bc30 comp rx=0 tx=0).stop 2024-03-24T03:37:40.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.053+0000 7f1f7a145640 1 -- 172.21.15.78:0/3183550162 shutdown_connections 2024-03-24T03:37:40.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.053+0000 7f1f7a145640 1 --2- 172.21.15.78:0/3183550162 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f74100d80 0x7f1f7410dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.053+0000 7f1f7a145640 1 --2- 172.21.15.78:0/3183550162 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1f7410e010 0x7f1f74110400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.053+0000 7f1f7a145640 1 --2- 172.21.15.78:0/3183550162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f74100460 0x7f1f74100840 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.053+0000 7f1f7a145640 1 -- 172.21.15.78:0/3183550162 >> 172.21.15.78:0/3183550162 conn(0x7f1f74076920 msgr2=0x7f1f74076d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:37:40.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.053+0000 7f1f7a145640 1 -- 172.21.15.78:0/3183550162 shutdown_connections 2024-03-24T03:37:40.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.053+0000 7f1f7a145640 1 -- 172.21.15.78:0/3183550162 wait complete. 2024-03-24T03:37:40.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.054+0000 7f1f7a145640 1 Processor -- start 2024-03-24T03:37:40.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.054+0000 7f1f7a145640 1 -- start start 2024-03-24T03:37:40.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.054+0000 7f1f7a145640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1f74100460 0x7f1f7419a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:40.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.054+0000 7f1f7a145640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f74100d80 0x7f1f7419a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:40.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.054+0000 7f1f7a145640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f7410e010 0x7f1f7419e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:40.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.054+0000 7f1f7a145640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f74113c70 con 0x7f1f74100d80 2024-03-24T03:37:40.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.054+0000 7f1f7a145640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f74113af0 con 0x7f1f74100460 2024-03-24T03:37:40.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.054+0000 7f1f7a145640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f74113df0 con 0x7f1f7410e010 2024-03-24T03:37:40.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.054+0000 7f1f72ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f74100d80 0x7f1f7419a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:40.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.054+0000 7f1f72ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f74100d80 0x7f1f7419a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56366/0 (socket says 172.21.15.78:56366) 2024-03-24T03:37:40.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.054+0000 7f1f72ffd640 1 -- 172.21.15.78:0/1523995889 learned_addr learned my addr 172.21.15.78:0/1523995889 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T03:37:40.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.054+0000 7f1f737fe640 1 --2- 172.21.15.78:0/1523995889 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1f74100460 0x7f1f7419a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:40.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.054+0000 7f1f73fff640 1 --2- 172.21.15.78:0/1523995889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f7410e010 0x7f1f7419e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:40.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.054+0000 7f1f72ffd640 1 -- 172.21.15.78:0/1523995889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f7410e010 msgr2=0x7f1f7419e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:40.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.054+0000 7f1f72ffd640 1 --2- 172.21.15.78:0/1523995889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f7410e010 0x7f1f7419e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.054+0000 7f1f72ffd640 1 -- 172.21.15.78:0/1523995889 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1f74100460 msgr2=0x7f1f7419a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:40.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.054+0000 7f1f72ffd640 1 --2- 172.21.15.78:0/1523995889 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1f74100460 0x7f1f7419a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.054+0000 7f1f72ffd640 1 -- 172.21.15.78:0/1523995889 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f7419f0e0 con 0x7f1f74100d80 2024-03-24T03:37:40.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.054+0000 7f1f737fe640 1 --2- 172.21.15.78:0/1523995889 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1f74100460 0x7f1f7419a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T03:37:40.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.055+0000 7f1f72ffd640 1 --2- 172.21.15.78:0/1523995889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f74100d80 0x7f1f7419a630 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f1f6800c920 tx=0x7f1f6800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:37:40.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.055+0000 7f1f70ff9640 1 -- 172.21.15.78:0/1523995889 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f68007c70 con 0x7f1f74100d80 2024-03-24T03:37:40.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.055+0000 7f1f70ff9640 1 -- 172.21.15.78:0/1523995889 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f68007e10 con 0x7f1f74100d80 2024-03-24T03:37:40.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.055+0000 7f1f70ff9640 1 -- 172.21.15.78:0/1523995889 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f68018890 con 0x7f1f74100d80 2024-03-24T03:37:40.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.055+0000 7f1f7a145640 1 -- 172.21.15.78:0/1523995889 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f7419f3d0 con 0x7f1f74100d80 2024-03-24T03:37:40.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.055+0000 7f1f7a145640 1 -- 172.21.15.78:0/1523995889 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f741a6c60 con 0x7f1f74100d80 2024-03-24T03:37:40.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.056+0000 7f1f7a145640 1 -- 172.21.15.78:0/1523995889 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f7406a490 con 0x7f1f74100d80 2024-03-24T03:37:40.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.057+0000 7f1f70ff9640 1 -- 172.21.15.78:0/1523995889 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f1f68018a70 con 0x7f1f74100d80 2024-03-24T03:37:40.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.057+0000 7f1f70ff9640 1 --2- 172.21.15.78:0/1523995889 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f1f5c0a2970 0x7f1f5c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:40.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.057+0000 7f1f737fe640 1 --2- 172.21.15.78:0/1523995889 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f1f5c0a2970 0x7f1f5c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:40.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.057+0000 7f1f70ff9640 1 -- 172.21.15.78:0/1523995889 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 13278+0+0 (secure 0 0 0) 0x7f1f680d01a0 con 0x7f1f74100d80 2024-03-24T03:37:40.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.058+0000 7f1f737fe640 1 --2- 172.21.15.78:0/1523995889 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f1f5c0a2970 0x7f1f5c0a4e30 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1f64006010 tx=0x7f1f64005b50 comp rx=0 tx=0).ready entity=mgr.25243 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:37:40.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.062+0000 7f1f70ff9640 1 -- 172.21.15.78:0/1523995889 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1f68014030 con 0x7f1f74100d80 2024-03-24T03:37:40.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.241+0000 7f1f7a145640 1 -- 172.21.15.78:0/1523995889 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1f7419b440 con 0x7f1f74100d80 2024-03-24T03:37:40.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.241+0000 7f1f70ff9640 1 -- 172.21.15.78:0/1523995889 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v157) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1f6809bc70 con 0x7f1f74100d80 2024-03-24T03:37:40.242 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-24T03:37:40.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.243+0000 7f1f7a145640 1 -- 172.21.15.78:0/1523995889 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f1f5c0a2970 msgr2=0x7f1f5c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:40.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.243+0000 7f1f7a145640 1 --2- 172.21.15.78:0/1523995889 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f1f5c0a2970 0x7f1f5c0a4e30 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1f64006010 tx=0x7f1f64005b50 comp rx=0 tx=0).stop 2024-03-24T03:37:40.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.243+0000 7f1f7a145640 1 -- 172.21.15.78:0/1523995889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f74100d80 msgr2=0x7f1f7419a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:40.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.243+0000 7f1f7a145640 1 --2- 172.21.15.78:0/1523995889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f74100d80 0x7f1f7419a630 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f1f6800c920 tx=0x7f1f6800cde0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.244+0000 7f1f7a145640 1 -- 172.21.15.78:0/1523995889 shutdown_connections 2024-03-24T03:37:40.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.244+0000 7f1f7a145640 1 --2- 172.21.15.78:0/1523995889 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f1f5c0a2970 0x7f1f5c0a4e30 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.244+0000 7f1f7a145640 1 --2- 172.21.15.78:0/1523995889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f7410e010 0x7f1f7419e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.244+0000 7f1f7a145640 1 --2- 172.21.15.78:0/1523995889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f74100d80 0x7f1f7419a630 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.244+0000 7f1f7a145640 1 --2- 172.21.15.78:0/1523995889 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1f74100460 0x7f1f7419a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.244+0000 7f1f7a145640 1 -- 172.21.15.78:0/1523995889 >> 172.21.15.78:0/1523995889 conn(0x7f1f74076920 msgr2=0x7f1f7410f480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:37:40.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.244+0000 7f1f7a145640 1 -- 172.21.15.78:0/1523995889 shutdown_connections 2024-03-24T03:37:40.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.244+0000 7f1f7a145640 1 -- 172.21.15.78:0/1523995889 wait complete. 2024-03-24T03:37:40.254 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:40 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1687077806' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T03:37:40.254 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:40 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3481467441' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T03:37:40.255 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T03:37:40.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.363+0000 7f397d5b9640 1 -- 172.21.15.78:0/3661992173 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3978100540 msgr2=0x7f397810e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:40.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.363+0000 7f397d5b9640 1 --2- 172.21.15.78:0/3661992173 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3978100540 0x7f397810e520 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f3970009cd0 tx=0x7f397002c0f0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.363+0000 7f397d5b9640 1 -- 172.21.15.78:0/3661992173 shutdown_connections 2024-03-24T03:37:40.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.363+0000 7f397d5b9640 1 --2- 172.21.15.78:0/3661992173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39780ffba0 0x7f3978100000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.363+0000 7f397d5b9640 1 --2- 172.21.15.78:0/3661992173 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3978100540 0x7f397810e520 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.363+0000 7f397d5b9640 1 --2- 172.21.15.78:0/3661992173 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3978107660 0x7f3978107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.363+0000 7f397d5b9640 1 -- 172.21.15.78:0/3661992173 >> 172.21.15.78:0/3661992173 conn(0x7f3978076b60 msgr2=0x7f39780fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:37:40.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.363+0000 7f397d5b9640 1 -- 172.21.15.78:0/3661992173 shutdown_connections 2024-03-24T03:37:40.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.363+0000 7f397d5b9640 1 -- 172.21.15.78:0/3661992173 wait complete. 2024-03-24T03:37:40.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.363+0000 7f397d5b9640 1 Processor -- start 2024-03-24T03:37:40.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.364+0000 7f397d5b9640 1 -- start start 2024-03-24T03:37:40.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.364+0000 7f397d5b9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39780ffba0 0x7f3978197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:40.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.364+0000 7f397d5b9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3978100540 0x7f3978198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:40.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.364+0000 7f397d5b9640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3978107660 0x7f397819c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:40.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.364+0000 7f397d5b9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3978111aa0 con 0x7f3978100540 2024-03-24T03:37:40.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.364+0000 7f397d5b9640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3978111920 con 0x7f3978107660 2024-03-24T03:37:40.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.364+0000 7f397d5b9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3978111c20 con 0x7f39780ffba0 2024-03-24T03:37:40.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.364+0000 7f39767fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3978100540 0x7f3978198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:40.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.364+0000 7f39767fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3978100540 0x7f3978198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56374/0 (socket says 172.21.15.78:56374) 2024-03-24T03:37:40.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.364+0000 7f39767fc640 1 -- 172.21.15.78:0/3196612941 learned_addr learned my addr 172.21.15.78:0/3196612941 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T03:37:40.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.364+0000 7f39777fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3978107660 0x7f397819c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:40.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.364+0000 7f3976ffd640 1 --2- 172.21.15.78:0/3196612941 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39780ffba0 0x7f3978197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:40.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.364+0000 7f39767fc640 1 -- 172.21.15.78:0/3196612941 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39780ffba0 msgr2=0x7f3978197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:40.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.364+0000 7f39767fc640 1 --2- 172.21.15.78:0/3196612941 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39780ffba0 0x7f3978197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.364+0000 7f39767fc640 1 -- 172.21.15.78:0/3196612941 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3978107660 msgr2=0x7f397819c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:40.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.364+0000 7f39767fc640 1 --2- 172.21.15.78:0/3196612941 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3978107660 0x7f397819c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.364+0000 7f39767fc640 1 -- 172.21.15.78:0/3196612941 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f397819cf10 con 0x7f3978100540 2024-03-24T03:37:40.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.364+0000 7f39777fe640 1 --2- 172.21.15.78:0/3196612941 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3978107660 0x7f397819c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T03:37:40.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.364+0000 7f39767fc640 1 --2- 172.21.15.78:0/3196612941 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3978100540 0x7f3978198460 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f396c00d770 tx=0x7f396c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:37:40.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.364+0000 7f395ffff640 1 -- 172.21.15.78:0/3196612941 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396c017070 con 0x7f3978100540 2024-03-24T03:37:40.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.364+0000 7f395ffff640 1 -- 172.21.15.78:0/3196612941 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f396c005500 con 0x7f3978100540 2024-03-24T03:37:40.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.364+0000 7f397d5b9640 1 -- 172.21.15.78:0/3196612941 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f397819d200 con 0x7f3978100540 2024-03-24T03:37:40.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.364+0000 7f395ffff640 1 -- 172.21.15.78:0/3196612941 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396c004e00 con 0x7f3978100540 2024-03-24T03:37:40.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.365+0000 7f397d5b9640 1 -- 172.21.15.78:0/3196612941 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39781a4a90 con 0x7f3978100540 2024-03-24T03:37:40.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.365+0000 7f397d5b9640 1 -- 172.21.15.78:0/3196612941 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39780688e0 con 0x7f3978100540 2024-03-24T03:37:40.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.367+0000 7f395ffff640 1 -- 172.21.15.78:0/3196612941 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f396c004fa0 con 0x7f3978100540 2024-03-24T03:37:40.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.367+0000 7f395ffff640 1 --2- 172.21.15.78:0/3196612941 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f39600a2970 0x7f39600a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:40.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.367+0000 7f3976ffd640 1 --2- 172.21.15.78:0/3196612941 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f39600a2970 0x7f39600a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:40.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.367+0000 7f395ffff640 1 -- 172.21.15.78:0/3196612941 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 13278+0+0 (secure 0 0 0) 0x7f396c0cfe90 con 0x7f3978100540 2024-03-24T03:37:40.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.367+0000 7f3976ffd640 1 --2- 172.21.15.78:0/3196612941 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f39600a2970 0x7f39600a4e30 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3968009b50 tx=0x7f3968005f90 comp rx=0 tx=0).ready entity=mgr.25243 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:37:40.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.371+0000 7f395ffff640 1 -- 172.21.15.78:0/3196612941 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f396c09b310 con 0x7f3978100540 2024-03-24T03:37:40.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:40 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1687077806' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T03:37:40.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:40 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3481467441' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T03:37:40.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.544+0000 7f397d5b9640 1 -- 172.21.15.78:0/3196612941 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3978109630 con 0x7f3978100540 2024-03-24T03:37:40.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.544+0000 7f395ffff640 1 -- 172.21.15.78:0/3196612941 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v157) v1 ==== 100+0+11 (secure 0 0 0) 0x7f396c09f3b0 con 0x7f3978100540 2024-03-24T03:37:40.544 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-03-24T03:37:40.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.546+0000 7f397d5b9640 1 -- 172.21.15.78:0/3196612941 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f39600a2970 msgr2=0x7f39600a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:40.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.546+0000 7f397d5b9640 1 --2- 172.21.15.78:0/3196612941 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f39600a2970 0x7f39600a4e30 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3968009b50 tx=0x7f3968005f90 comp rx=0 tx=0).stop 2024-03-24T03:37:40.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.546+0000 7f397d5b9640 1 -- 172.21.15.78:0/3196612941 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3978100540 msgr2=0x7f3978198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:40.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.546+0000 7f397d5b9640 1 --2- 172.21.15.78:0/3196612941 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3978100540 0x7f3978198460 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f396c00d770 tx=0x7f396c00dc30 comp rx=0 tx=0).stop 2024-03-24T03:37:40.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.546+0000 7f397d5b9640 1 -- 172.21.15.78:0/3196612941 shutdown_connections 2024-03-24T03:37:40.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.546+0000 7f397d5b9640 1 --2- 172.21.15.78:0/3196612941 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f39600a2970 0x7f39600a4e30 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.546+0000 7f397d5b9640 1 --2- 172.21.15.78:0/3196612941 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3978107660 0x7f397819c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.546+0000 7f397d5b9640 1 --2- 172.21.15.78:0/3196612941 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3978100540 0x7f3978198460 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.546+0000 7f397d5b9640 1 --2- 172.21.15.78:0/3196612941 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39780ffba0 0x7f3978197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.546+0000 7f397d5b9640 1 -- 172.21.15.78:0/3196612941 >> 172.21.15.78:0/3196612941 conn(0x7f3978076b60 msgr2=0x7f39780fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:37:40.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.546+0000 7f397d5b9640 1 -- 172.21.15.78:0/3196612941 shutdown_connections 2024-03-24T03:37:40.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.546+0000 7f397d5b9640 1 -- 172.21.15.78:0/3196612941 wait complete. 2024-03-24T03:37:40.557 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T03:37:40.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:40 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1687077806' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T03:37:40.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:40 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3481467441' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T03:37:40.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.663+0000 7f948fe4b640 1 -- 172.21.15.78:0/51898649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9488100f00 msgr2=0x7f948810d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:40.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.663+0000 7f948fe4b640 1 --2- 172.21.15.78:0/51898649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9488100f00 0x7f948810d3c0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f9480009ca0 tx=0x7f948002c0f0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.663+0000 7f948fe4b640 1 -- 172.21.15.78:0/51898649 shutdown_connections 2024-03-24T03:37:40.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.663+0000 7f948fe4b640 1 --2- 172.21.15.78:0/51898649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9488100f00 0x7f948810d3c0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.663+0000 7f948fe4b640 1 --2- 172.21.15.78:0/51898649 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9488100540 0x7f94881009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.663+0000 7f948fe4b640 1 --2- 172.21.15.78:0/51898649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9488106600 0x7f94881069e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.663+0000 7f948fe4b640 1 -- 172.21.15.78:0/51898649 >> 172.21.15.78:0/51898649 conn(0x7f9488076920 msgr2=0x7f9488076d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:37:40.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.663+0000 7f948fe4b640 1 -- 172.21.15.78:0/51898649 shutdown_connections 2024-03-24T03:37:40.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.663+0000 7f948fe4b640 1 -- 172.21.15.78:0/51898649 wait complete. 2024-03-24T03:37:40.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.663+0000 7f948fe4b640 1 Processor -- start 2024-03-24T03:37:40.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.663+0000 7f948fe4b640 1 -- start start 2024-03-24T03:37:40.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.663+0000 7f948fe4b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9488100540 0x7f9488197db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:40.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.663+0000 7f948fe4b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9488100f00 0x7f94881982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:40.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.663+0000 7f948fe4b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9488106600 0x7f948819c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:40.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.663+0000 7f948fe4b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94881119d0 con 0x7f9488106600 2024-03-24T03:37:40.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.664+0000 7f948fe4b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9488111850 con 0x7f9488100f00 2024-03-24T03:37:40.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.664+0000 7f948fe4b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9488111b50 con 0x7f9488100540 2024-03-24T03:37:40.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.664+0000 7f948e3e8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9488106600 0x7f948819c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:40.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.664+0000 7f948e3e8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9488106600 0x7f948819c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56392/0 (socket says 172.21.15.78:56392) 2024-03-24T03:37:40.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.664+0000 7f948e3e8640 1 -- 172.21.15.78:0/2431630037 learned_addr learned my addr 172.21.15.78:0/2431630037 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T03:37:40.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.664+0000 7f948d3e6640 1 --2- 172.21.15.78:0/2431630037 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9488100f00 0x7f94881982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:40.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.664+0000 7f948dbe7640 1 --2- 172.21.15.78:0/2431630037 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9488100540 0x7f9488197db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:40.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.664+0000 7f948e3e8640 1 -- 172.21.15.78:0/2431630037 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9488100540 msgr2=0x7f9488197db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:40.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.664+0000 7f948e3e8640 1 --2- 172.21.15.78:0/2431630037 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9488100540 0x7f9488197db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.664+0000 7f948e3e8640 1 -- 172.21.15.78:0/2431630037 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9488100f00 msgr2=0x7f94881982f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:40.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.664+0000 7f948e3e8640 1 --2- 172.21.15.78:0/2431630037 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9488100f00 0x7f94881982f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.664+0000 7f948e3e8640 1 -- 172.21.15.78:0/2431630037 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f948819cda0 con 0x7f9488106600 2024-03-24T03:37:40.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.664+0000 7f948e3e8640 1 --2- 172.21.15.78:0/2431630037 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9488106600 0x7f948819c670 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f9480005e30 tx=0x7f9480032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:37:40.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.664+0000 7f947affd640 1 -- 172.21.15.78:0/2431630037 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9480042020 con 0x7f9488106600 2024-03-24T03:37:40.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.664+0000 7f947affd640 1 -- 172.21.15.78:0/2431630037 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9480034640 con 0x7f9488106600 2024-03-24T03:37:40.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.664+0000 7f948fe4b640 1 -- 172.21.15.78:0/2431630037 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f948819d030 con 0x7f9488106600 2024-03-24T03:37:40.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.664+0000 7f947affd640 1 -- 172.21.15.78:0/2431630037 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9480039040 con 0x7f9488106600 2024-03-24T03:37:40.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.664+0000 7f948fe4b640 1 -- 172.21.15.78:0/2431630037 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94881a4920 con 0x7f9488106600 2024-03-24T03:37:40.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.665+0000 7f948fe4b640 1 -- 172.21.15.78:0/2431630037 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9488101c60 con 0x7f9488106600 2024-03-24T03:37:40.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.666+0000 7f947affd640 1 -- 172.21.15.78:0/2431630037 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f9480045020 con 0x7f9488106600 2024-03-24T03:37:40.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.667+0000 7f947affd640 1 --2- 172.21.15.78:0/2431630037 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f94740a2970 0x7f94740a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:40.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.667+0000 7f948dbe7640 1 --2- 172.21.15.78:0/2431630037 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f94740a2970 0x7f94740a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:40.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.667+0000 7f947affd640 1 -- 172.21.15.78:0/2431630037 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 13278+0+0 (secure 0 0 0) 0x7f94800f1f50 con 0x7f9488106600 2024-03-24T03:37:40.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.667+0000 7f948dbe7640 1 --2- 172.21.15.78:0/2431630037 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f94740a2970 0x7f94740a4e30 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f947c00abe0 tx=0x7f947c00a2d0 comp rx=0 tx=0).ready entity=mgr.25243 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:37:40.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.671+0000 7f947affd640 1 -- 172.21.15.78:0/2431630037 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f948004c020 con 0x7f9488106600 2024-03-24T03:37:40.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.845+0000 7f948fe4b640 1 -- 172.21.15.78:0/2431630037 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f9488108950 con 0x7f9488106600 2024-03-24T03:37:40.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.845+0000 7f947affd640 1 -- 172.21.15.78:0/2431630037 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v157) v1 ==== 103+0+11 (secure 0 0 0) 0x7f94800bdb30 con 0x7f9488106600 2024-03-24T03:37:40.846 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-24T03:37:40.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.847+0000 7f948fe4b640 1 -- 172.21.15.78:0/2431630037 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f94740a2970 msgr2=0x7f94740a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:40.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.847+0000 7f948fe4b640 1 --2- 172.21.15.78:0/2431630037 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f94740a2970 0x7f94740a4e30 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f947c00abe0 tx=0x7f947c00a2d0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.847+0000 7f948fe4b640 1 -- 172.21.15.78:0/2431630037 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9488106600 msgr2=0x7f948819c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:40.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.847+0000 7f948fe4b640 1 --2- 172.21.15.78:0/2431630037 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9488106600 0x7f948819c670 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f9480005e30 tx=0x7f9480032f40 comp rx=0 tx=0).stop 2024-03-24T03:37:40.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.847+0000 7f948fe4b640 1 -- 172.21.15.78:0/2431630037 shutdown_connections 2024-03-24T03:37:40.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.847+0000 7f948fe4b640 1 --2- 172.21.15.78:0/2431630037 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f94740a2970 0x7f94740a4e30 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.847+0000 7f948fe4b640 1 --2- 172.21.15.78:0/2431630037 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9488106600 0x7f948819c670 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.847+0000 7f948fe4b640 1 --2- 172.21.15.78:0/2431630037 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9488100f00 0x7f94881982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.847+0000 7f948fe4b640 1 --2- 172.21.15.78:0/2431630037 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9488100540 0x7f9488197db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.847+0000 7f948fe4b640 1 -- 172.21.15.78:0/2431630037 >> 172.21.15.78:0/2431630037 conn(0x7f9488076920 msgr2=0x7f94880fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:37:40.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.847+0000 7f948fe4b640 1 -- 172.21.15.78:0/2431630037 shutdown_connections 2024-03-24T03:37:40.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.847+0000 7f948fe4b640 1 -- 172.21.15.78:0/2431630037 wait complete. 2024-03-24T03:37:40.859 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T03:37:40.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.962+0000 7fba3c1b2640 1 -- 172.21.15.78:0/2322198528 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba341078a0 msgr2=0x7fba34107c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:40.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.962+0000 7fba3c1b2640 1 --2- 172.21.15.78:0/2322198528 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba341078a0 0x7fba34107c80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fba30009890 tx=0x7fba3002bb70 comp rx=0 tx=0).stop 2024-03-24T03:37:40.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.962+0000 7fba3c1b2640 1 -- 172.21.15.78:0/2322198528 shutdown_connections 2024-03-24T03:37:40.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.962+0000 7fba3c1b2640 1 --2- 172.21.15.78:0/2322198528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba340fff60 0x7fba341003e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.962+0000 7fba3c1b2640 1 --2- 172.21.15.78:0/2322198528 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba34100920 0x7fba3410e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.962+0000 7fba3c1b2640 1 --2- 172.21.15.78:0/2322198528 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba341078a0 0x7fba34107c80 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.962+0000 7fba3c1b2640 1 -- 172.21.15.78:0/2322198528 >> 172.21.15.78:0/2322198528 conn(0x7fba340758e0 msgr2=0x7fba34075cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:37:40.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.962+0000 7fba3c1b2640 1 -- 172.21.15.78:0/2322198528 shutdown_connections 2024-03-24T03:37:40.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.962+0000 7fba3c1b2640 1 -- 172.21.15.78:0/2322198528 wait complete. 2024-03-24T03:37:40.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.963+0000 7fba3c1b2640 1 Processor -- start 2024-03-24T03:37:40.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.963+0000 7fba3c1b2640 1 -- start start 2024-03-24T03:37:40.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.963+0000 7fba3c1b2640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba340fff60 0x7fba34198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:40.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.963+0000 7fba3c1b2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba34100920 0x7fba34198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:40.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.963+0000 7fba3c1b2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba341078a0 0x7fba3419c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:40.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.963+0000 7fba3c1b2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba34111ac0 con 0x7fba34100920 2024-03-24T03:37:40.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.963+0000 7fba3c1b2640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba34111940 con 0x7fba340fff60 2024-03-24T03:37:40.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.963+0000 7fba3c1b2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba34111c40 con 0x7fba341078a0 2024-03-24T03:37:40.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.963+0000 7fba3974d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba34100920 0x7fba34198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:40.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.963+0000 7fba3974d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba34100920 0x7fba34198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56420/0 (socket says 172.21.15.78:56420) 2024-03-24T03:37:40.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.963+0000 7fba3974d640 1 -- 172.21.15.78:0/798428796 learned_addr learned my addr 172.21.15.78:0/798428796 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T03:37:40.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.963+0000 7fba39f4e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba340fff60 0x7fba34198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:40.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.963+0000 7fba3974d640 1 -- 172.21.15.78:0/798428796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba341078a0 msgr2=0x7fba3419c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:40.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.963+0000 7fba3974d640 1 --2- 172.21.15.78:0/798428796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba341078a0 0x7fba3419c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.963+0000 7fba3974d640 1 -- 172.21.15.78:0/798428796 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba340fff60 msgr2=0x7fba34198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:40.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.963+0000 7fba3974d640 1 --2- 172.21.15.78:0/798428796 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba340fff60 0x7fba34198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:40.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.963+0000 7fba3974d640 1 -- 172.21.15.78:0/798428796 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba3419cfc0 con 0x7fba34100920 2024-03-24T03:37:40.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.964+0000 7fba3974d640 1 --2- 172.21.15.78:0/798428796 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba34100920 0x7fba34198560 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fba2400d950 tx=0x7fba2400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:37:40.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.964+0000 7fba2affd640 1 -- 172.21.15.78:0/798428796 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba24017070 con 0x7fba34100920 2024-03-24T03:37:40.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.964+0000 7fba3c1b2640 1 -- 172.21.15.78:0/798428796 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba3419d2b0 con 0x7fba34100920 2024-03-24T03:37:40.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.964+0000 7fba2affd640 1 -- 172.21.15.78:0/798428796 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba24009c30 con 0x7fba34100920 2024-03-24T03:37:40.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.964+0000 7fba2affd640 1 -- 172.21.15.78:0/798428796 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2400f6b0 con 0x7fba34100920 2024-03-24T03:37:40.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.964+0000 7fba3c1b2640 1 -- 172.21.15.78:0/798428796 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba341a4b40 con 0x7fba34100920 2024-03-24T03:37:40.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.965+0000 7fba3c1b2640 1 -- 172.21.15.78:0/798428796 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba340688e0 con 0x7fba34100920 2024-03-24T03:37:40.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.966+0000 7fba2affd640 1 -- 172.21.15.78:0/798428796 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fba240109d0 con 0x7fba34100920 2024-03-24T03:37:40.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.966+0000 7fba2affd640 1 --2- 172.21.15.78:0/798428796 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fba200a2970 0x7fba200a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:37:40.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.966+0000 7fba39f4e640 1 --2- 172.21.15.78:0/798428796 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fba200a2970 0x7fba200a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:37:40.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.966+0000 7fba2affd640 1 -- 172.21.15.78:0/798428796 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 13278+0+0 (secure 0 0 0) 0x7fba240cfd10 con 0x7fba34100920 2024-03-24T03:37:40.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.967+0000 7fba39f4e640 1 --2- 172.21.15.78:0/798428796 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fba200a2970 0x7fba200a4e30 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fba300097b0 tx=0x7fba300058d0 comp rx=0 tx=0).ready entity=mgr.25243 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:37:40.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:40.971+0000 7fba2affd640 1 -- 172.21.15.78:0/798428796 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fba24009de0 con 0x7fba34100920 2024-03-24T03:37:41.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:41.158+0000 7fba3c1b2640 1 -- 172.21.15.78:0/798428796 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fba3402d070 con 0x7fba34100920 2024-03-24T03:37:41.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:41.158+0000 7fba2affd640 1 -- 172.21.15.78:0/798428796 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 106 entries v157) v1 ==== 82+0+6453 (secure 0 0 0) 0x7fba2409b7e0 con 0x7fba34100920 2024-03-24T03:37:41.159 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6834/1985874955 2024-03-25T00:35:20.737219+0000 2024-03-24T03:37:41.159 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6835/3414536872 2024-03-25T00:35:04.921316+0000 2024-03-24T03:37:41.159 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6834/3414536872 2024-03-25T00:35:04.921316+0000 2024-03-24T03:37:41.159 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6837/2612396245 2024-03-25T00:34:49.129252+0000 2024-03-24T03:37:41.159 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6834/93340378 2024-03-25T00:34:30.894376+0000 2024-03-24T03:37:41.159 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6837/878237454 2024-03-25T00:34:15.858120+0000 2024-03-24T03:37:41.159 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6836/878237454 2024-03-25T00:34:15.858120+0000 2024-03-24T03:37:41.160 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6835/2163967090 2024-03-25T00:34:00.891452+0000 2024-03-24T03:37:41.160 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6834/2163967090 2024-03-25T00:34:00.891452+0000 2024-03-24T03:37:41.160 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6836/1186751954 2024-03-25T00:33:45.900090+0000 2024-03-24T03:37:41.160 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6835/1416685123 2024-03-25T00:33:30.602155+0000 2024-03-24T03:37:41.160 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6834/1416685123 2024-03-25T00:33:30.602155+0000 2024-03-24T03:37:41.160 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6836/1077781429 2024-03-25T00:33:14.880383+0000 2024-03-24T03:37:41.160 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6839/2480119933 2024-03-25T00:32:59.099199+0000 2024-03-24T03:37:41.160 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6838/2480119933 2024-03-25T00:32:59.099199+0000 2024-03-24T03:37:41.161 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6837/2239323994 2024-03-25T00:32:40.627803+0000 2024-03-24T03:37:41.161 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6839/3377749691 2024-03-25T00:32:25.708147+0000 2024-03-24T03:37:41.161 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6836/1071274623 2024-03-25T00:32:10.632135+0000 2024-03-24T03:37:41.161 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6839/4176145760 2024-03-25T00:31:55.146631+0000 2024-03-24T03:37:41.161 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6838/4176145760 2024-03-25T00:31:55.146631+0000 2024-03-24T03:37:41.161 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6840/1658072731 2024-03-25T00:31:39.075579+0000 2024-03-24T03:37:41.161 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/442826396 2024-03-25T00:21:09.220548+0000 2024-03-24T03:37:41.161 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6832/4067467714 2024-03-25T00:29:09.638984+0000 2024-03-24T03:37:41.161 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6837/1077781429 2024-03-25T00:33:14.880383+0000 2024-03-24T03:37:41.161 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6839/3131891440 2024-03-25T00:24:40.372092+0000 2024-03-24T03:37:41.162 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6838/3949153161 2024-03-25T00:25:43.971852+0000 2024-03-24T03:37:41.162 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6833/794068267 2024-03-25T00:28:05.611074+0000 2024-03-24T03:37:41.162 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6838/3131891440 2024-03-25T00:24:40.372092+0000 2024-03-24T03:37:41.162 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6833/1751539133 2024-03-25T00:27:35.655923+0000 2024-03-24T03:37:41.162 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6834/4046882186 2024-03-25T00:24:25.648083+0000 2024-03-24T03:37:41.162 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6839/3949153161 2024-03-25T00:25:43.971852+0000 2024-03-24T03:37:41.162 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6835/1985874955 2024-03-25T00:35:20.737219+0000 2024-03-24T03:37:41.162 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/2748920178 2024-03-25T00:11:40.199569+0000 2024-03-24T03:37:41.162 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6841/3424505910 2024-03-25T00:29:25.370378+0000 2024-03-24T03:37:41.163 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1982202541 2024-03-25T00:11:40.199569+0000 2024-03-24T03:37:41.163 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6834/108936439 2024-03-25T00:23:06.644918+0000 2024-03-24T03:37:41.163 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6832/794068267 2024-03-25T00:28:05.611074+0000 2024-03-24T03:37:41.163 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6838/3377749691 2024-03-25T00:32:25.708147+0000 2024-03-24T03:37:41.163 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6839/3531100722 2024-03-25T00:23:22.383674+0000 2024-03-24T03:37:41.163 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6838/117119462 2024-03-25T00:22:50.169074+0000 2024-03-24T03:37:41.163 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/2236879401 2024-03-25T00:11:40.199569+0000 2024-03-24T03:37:41.163 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6833/3086318443 2024-03-25T00:27:05.852992+0000 2024-03-24T03:37:41.163 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6837/1186751954 2024-03-25T00:33:45.900090+0000 2024-03-24T03:37:41.163 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6834/23104891 2024-03-25T00:24:55.644935+0000 2024-03-24T03:37:41.164 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1502752625 2024-03-25T00:11:14.637881+0000 2024-03-24T03:37:41.164 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6834/2447512361 2024-03-25T00:26:33.985150+0000 2024-03-24T03:37:41.164 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/359613096 2024-03-25T00:21:09.220548+0000 2024-03-24T03:37:41.164 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/2913847518 2024-03-25T00:11:14.637881+0000 2024-03-24T03:37:41.164 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/1983101026 2024-03-25T00:21:09.220548+0000 2024-03-24T03:37:41.164 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/465691596 2024-03-25T00:11:14.637881+0000 2024-03-24T03:37:41.164 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/3962523609 2024-03-25T00:11:14.637881+0000 2024-03-24T03:37:41.164 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1510740971 2024-03-25T00:21:09.220548+0000 2024-03-24T03:37:41.164 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6840/1960340276 2024-03-25T00:31:02.121119+0000 2024-03-24T03:37:41.165 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6836/2612396245 2024-03-25T00:34:49.129252+0000 2024-03-24T03:37:41.165 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/2913847518 2024-03-25T00:11:14.637881+0000 2024-03-24T03:37:41.165 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/2748920178 2024-03-25T00:11:40.199569+0000 2024-03-24T03:37:41.165 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6836/2239323994 2024-03-25T00:32:40.627803+0000 2024-03-24T03:37:41.165 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6835/2950120609 2024-03-25T00:25:25.629098+0000 2024-03-24T03:37:41.165 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1933054339 2024-03-25T00:11:40.199569+0000 2024-03-24T03:37:41.165 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6838/4117918862 2024-03-25T00:31:18.139126+0000 2024-03-24T03:37:41.165 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6838/607202225 2024-03-25T00:25:10.629186+0000 2024-03-24T03:37:41.165 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1685151467 2024-03-25T00:21:09.220548+0000 2024-03-24T03:37:41.166 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6838/3531100722 2024-03-25T00:23:22.383674+0000 2024-03-24T03:37:41.166 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6838/2758357551 2024-03-25T00:30:15.603375+0000 2024-03-24T03:37:41.166 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/1983101026 2024-03-25T00:21:09.220548+0000 2024-03-24T03:37:41.166 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6835/758053763 2024-03-25T00:25:59.620946+0000 2024-03-24T03:37:41.166 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6835/4046882186 2024-03-25T00:24:25.648083+0000 2024-03-24T03:37:41.166 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6835/108936439 2024-03-25T00:23:06.644918+0000 2024-03-24T03:37:41.166 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.106:6833/2367018868 2024-03-25T00:26:15.443626+0000 2024-03-24T03:37:41.166 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6835/93340378 2024-03-25T00:34:30.894376+0000 2024-03-24T03:37:41.167 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6835/2447512361 2024-03-25T00:26:33.985150+0000 2024-03-24T03:37:41.167 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6835/23104891 2024-03-25T00:24:55.644935+0000 2024-03-24T03:37:41.167 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.106:6832/2611554095 2024-03-25T00:26:49.797488+0000 2024-03-24T03:37:41.167 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.106:6833/2611554095 2024-03-25T00:26:49.797488+0000 2024-03-24T03:37:41.167 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6832/3086318443 2024-03-25T00:27:05.852992+0000 2024-03-24T03:37:41.167 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.106:6832/2299054161 2024-03-25T00:27:20.754602+0000 2024-03-24T03:37:41.167 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6832/1006503957 2024-03-25T00:29:44.040588+0000 2024-03-24T03:37:41.167 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.106:6832/2367018868 2024-03-25T00:26:15.443626+0000 2024-03-24T03:37:41.167 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.106:6833/2299054161 2024-03-25T00:27:20.754602+0000 2024-03-24T03:37:41.168 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6832/1751539133 2024-03-25T00:27:35.655923+0000 2024-03-24T03:37:41.168 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.106:6832/459568830 2024-03-25T00:27:50.719456+0000 2024-03-24T03:37:41.168 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.106:6833/459568830 2024-03-25T00:27:50.719456+0000 2024-03-24T03:37:41.168 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.106:6832/93515314 2024-03-25T00:28:20.713453+0000 2024-03-24T03:37:41.168 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6833/3928685127 2024-03-25T00:28:35.891033+0000 2024-03-24T03:37:41.168 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6839/117119462 2024-03-25T00:22:50.169074+0000 2024-03-24T03:37:41.168 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.106:6832/2104934388 2024-03-25T00:28:54.025064+0000 2024-03-24T03:37:41.168 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6837/1071274623 2024-03-25T00:32:10.632135+0000 2024-03-24T03:37:41.168 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.106:6833/2104934388 2024-03-25T00:28:54.025064+0000 2024-03-24T03:37:41.169 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6841/1658072731 2024-03-25T00:31:39.075579+0000 2024-03-24T03:37:41.169 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6833/4067467714 2024-03-25T00:29:09.638984+0000 2024-03-24T03:37:41.169 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6840/3424505910 2024-03-25T00:29:25.370378+0000 2024-03-24T03:37:41.169 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6834/758053763 2024-03-25T00:25:59.620946+0000 2024-03-24T03:37:41.169 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6838/2645579087 2024-03-25T00:30:46.105435+0000 2024-03-24T03:37:41.169 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6834/2950120609 2024-03-25T00:25:25.629098+0000 2024-03-24T03:37:41.169 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6840/1196331852 2024-03-25T00:29:59.753104+0000 2024-03-24T03:37:41.169 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6833/1006503957 2024-03-25T00:29:44.040588+0000 2024-03-24T03:37:41.169 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6841/1196331852 2024-03-25T00:29:59.753104+0000 2024-03-24T03:37:41.170 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6839/607202225 2024-03-25T00:25:10.629186+0000 2024-03-24T03:37:41.170 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6841/1759489401 2024-03-25T00:30:30.711566+0000 2024-03-24T03:37:41.170 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6839/2758357551 2024-03-25T00:30:15.603375+0000 2024-03-24T03:37:41.170 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6840/1759489401 2024-03-25T00:30:30.711566+0000 2024-03-24T03:37:41.170 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6839/2645579087 2024-03-25T00:30:46.105435+0000 2024-03-24T03:37:41.170 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.107:6832/3928685127 2024-03-25T00:28:35.891033+0000 2024-03-24T03:37:41.170 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6841/1960340276 2024-03-25T00:31:02.121119+0000 2024-03-24T03:37:41.170 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.106:6833/93515314 2024-03-25T00:28:20.713453+0000 2024-03-24T03:37:41.170 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6839/4117918862 2024-03-25T00:31:18.139126+0000 2024-03-24T03:37:41.172 INFO:teuthology.orchestra.run.smithi078.stderr:listed 106 entries 2024-03-24T03:37:41.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:41.160+0000 7fba3c1b2640 1 -- 172.21.15.78:0/798428796 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fba200a2970 msgr2=0x7fba200a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:41.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:41.160+0000 7fba3c1b2640 1 --2- 172.21.15.78:0/798428796 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fba200a2970 0x7fba200a4e30 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fba300097b0 tx=0x7fba300058d0 comp rx=0 tx=0).stop 2024-03-24T03:37:41.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:41.160+0000 7fba3c1b2640 1 -- 172.21.15.78:0/798428796 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba34100920 msgr2=0x7fba34198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:37:41.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:41.160+0000 7fba3c1b2640 1 --2- 172.21.15.78:0/798428796 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba34100920 0x7fba34198560 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fba2400d950 tx=0x7fba2400de10 comp rx=0 tx=0).stop 2024-03-24T03:37:41.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:41.160+0000 7fba3c1b2640 1 -- 172.21.15.78:0/798428796 shutdown_connections 2024-03-24T03:37:41.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:41.160+0000 7fba3c1b2640 1 --2- 172.21.15.78:0/798428796 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fba200a2970 0x7fba200a4e30 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:41.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:41.160+0000 7fba3c1b2640 1 --2- 172.21.15.78:0/798428796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba341078a0 0x7fba3419c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:41.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:41.160+0000 7fba3c1b2640 1 --2- 172.21.15.78:0/798428796 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba34100920 0x7fba34198560 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:41.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:41.160+0000 7fba3c1b2640 1 --2- 172.21.15.78:0/798428796 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba340fff60 0x7fba34198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:37:41.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:41.161+0000 7fba3c1b2640 1 -- 172.21.15.78:0/798428796 >> 172.21.15.78:0/798428796 conn(0x7fba340758e0 msgr2=0x7fba340fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:37:41.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:41.161+0000 7fba3c1b2640 1 -- 172.21.15.78:0/798428796 shutdown_connections 2024-03-24T03:37:41.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:37:41.161+0000 7fba3c1b2640 1 -- 172.21.15.78:0/798428796 wait complete. 2024-03-24T03:37:41.173 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-24T03:37:41.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T03:37:41.174 DEBUG:teuthology.orchestra.run.smithi078:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-24T03:37:41.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:41 smithi106 ceph-mon[36505]: pgmap v6240: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:37:41.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:41 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1523995889' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T03:37:41.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:41 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/3196612941' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T03:37:41.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:41 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2431630037' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T03:37:41.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:41 smithi078 ceph-mon[32429]: pgmap v6240: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:37:41.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:41 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1523995889' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T03:37:41.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:41 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/3196612941' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T03:37:41.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:41 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2431630037' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T03:37:41.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:41 smithi107 ceph-mon[36485]: pgmap v6240: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:37:41.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:41 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1523995889' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T03:37:41.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:41 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/3196612941' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T03:37:41.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:41 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2431630037' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T03:37:42.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:42 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/798428796' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T03:37:42.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:42 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/798428796' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T03:37:42.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:42 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/798428796' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T03:37:43.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:43 smithi106 ceph-mon[36505]: pgmap v6241: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:37:43.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:43 smithi078 ceph-mon[32429]: pgmap v6241: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:37:43.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:43 smithi107 ceph-mon[36485]: pgmap v6241: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:37:44.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10938.258926 secs 2024-03-24T03:37:44.494 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10938.258926 secs 2024-03-24T03:37:44.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10938.258926 secs 2024-03-24T03:37:45.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:45 smithi106 ceph-mon[36505]: pgmap v6242: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:37:45.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:45 smithi078 ceph-mon[32429]: pgmap v6242: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:37:45.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:45 smithi107 ceph-mon[36485]: pgmap v6242: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:37:47.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:47 smithi106 ceph-mon[36505]: pgmap v6243: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:37:47.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:47 smithi078 ceph-mon[32429]: pgmap v6243: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:37:47.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:47 smithi107 ceph-mon[36485]: pgmap v6243: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:37:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10943.258986 secs 2024-03-24T03:37:49.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:49 smithi106 ceph-mon[36505]: pgmap v6244: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:37:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10943.258986 secs 2024-03-24T03:37:49.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:49 smithi078 ceph-mon[32429]: pgmap v6244: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:37:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10943.258986 secs 2024-03-24T03:37:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:49 smithi107 ceph-mon[36485]: pgmap v6244: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:37:51.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:51 smithi106 ceph-mon[36505]: pgmap v6245: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:37:51.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:51 smithi078 ceph-mon[32429]: pgmap v6245: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:37:51.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:51 smithi107 ceph-mon[36485]: pgmap v6245: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:37:53.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:53 smithi106 ceph-mon[36505]: pgmap v6246: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:37:53.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:53 smithi078 ceph-mon[32429]: pgmap v6246: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:37:53.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:53 smithi107 ceph-mon[36485]: pgmap v6246: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:37:54.402 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10948.259026 secs 2024-03-24T03:37:54.492 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10948.259026 secs 2024-03-24T03:37:54.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10948.259026 secs 2024-03-24T03:37:55.508 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:55 smithi078 ceph-mon[32429]: pgmap v6247: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:37:55.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:55 smithi107 ceph-mon[36485]: pgmap v6247: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:37:55.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:55 smithi106 ceph-mon[36505]: pgmap v6247: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:37:57.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:57 smithi078 ceph-mon[32429]: pgmap v6248: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:37:57.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:57 smithi107 ceph-mon[36485]: pgmap v6248: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:37:57.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:57 smithi106 ceph-mon[36505]: pgmap v6248: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:37:59.454 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10953.259116 secs 2024-03-24T03:37:59.455 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:37:59 smithi106 ceph-mon[36505]: pgmap v6249: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:37:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10953.259116 secs 2024-03-24T03:37:59.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:37:59 smithi078 ceph-mon[32429]: pgmap v6249: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:37:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10953.259116 secs 2024-03-24T03:37:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:37:59 smithi107 ceph-mon[36485]: pgmap v6249: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:01.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:01 smithi078 ceph-mon[32429]: pgmap v6250: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:01.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:01 smithi107 ceph-mon[36485]: pgmap v6250: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:01.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:01 smithi106 ceph-mon[36505]: pgmap v6250: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:03.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:03 smithi078 ceph-mon[32429]: pgmap v6251: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:38:03.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:03 smithi107 ceph-mon[36485]: pgmap v6251: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:38:03.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:03 smithi106 ceph-mon[36505]: pgmap v6251: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:38:04.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10958.259204 secs 2024-03-24T03:38:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10958.259204 secs 2024-03-24T03:38:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10958.259204 secs 2024-03-24T03:38:05.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:05 smithi078 ceph-mon[32429]: pgmap v6252: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:05.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:05 smithi107 ceph-mon[36485]: pgmap v6252: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:05.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:05 smithi106 ceph-mon[36505]: pgmap v6252: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:07.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:07 smithi078 ceph-mon[32429]: pgmap v6253: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:38:07.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:07 smithi107 ceph-mon[36485]: pgmap v6253: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:38:07.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:07 smithi106 ceph-mon[36505]: pgmap v6253: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:38:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10963.259265 secs 2024-03-24T03:38:09.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:09 smithi078 ceph-mon[32429]: pgmap v6254: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10963.259265 secs 2024-03-24T03:38:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:09 smithi107 ceph-mon[36485]: pgmap v6254: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10963.259265 secs 2024-03-24T03:38:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:09 smithi106 ceph-mon[36505]: pgmap v6254: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:11.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:11 smithi078 ceph-mon[32429]: pgmap v6255: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:11.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:11 smithi107 ceph-mon[36485]: pgmap v6255: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:11.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:11 smithi106 ceph-mon[36505]: pgmap v6255: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:13.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:13 smithi078 ceph-mon[32429]: pgmap v6256: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:38:13.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:13 smithi107 ceph-mon[36485]: pgmap v6256: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:38:13.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:13 smithi106 ceph-mon[36505]: pgmap v6256: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:38:14.494 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10968.259348 secs 2024-03-24T03:38:14.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10968.259348 secs 2024-03-24T03:38:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10968.259348 secs 2024-03-24T03:38:15.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:15 smithi078 ceph-mon[32429]: pgmap v6257: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:15.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:15 smithi107 ceph-mon[36485]: pgmap v6257: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:15.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:15 smithi106 ceph-mon[36505]: pgmap v6257: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:17.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:17 smithi078 ceph-mon[32429]: pgmap v6258: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:38:17.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:17 smithi107 ceph-mon[36485]: pgmap v6258: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:38:17.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:17 smithi106 ceph-mon[36505]: pgmap v6258: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:38:19.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10973.259435 secs 2024-03-24T03:38:19.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:19 smithi078 ceph-mon[32429]: pgmap v6259: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10973.259435 secs 2024-03-24T03:38:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:19 smithi107 ceph-mon[36485]: pgmap v6259: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10973.259435 secs 2024-03-24T03:38:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:19 smithi106 ceph-mon[36505]: pgmap v6259: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:21.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:21 smithi078 ceph-mon[32429]: pgmap v6260: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:21.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:21 smithi107 ceph-mon[36485]: pgmap v6260: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:21.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:21 smithi106 ceph-mon[36505]: pgmap v6260: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:23.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:23 smithi078 ceph-mon[32429]: pgmap v6261: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:38:23.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:23 smithi107 ceph-mon[36485]: pgmap v6261: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:38:23.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:23 smithi106 ceph-mon[36505]: pgmap v6261: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:38:24.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10978.259493 secs 2024-03-24T03:38:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10978.259493 secs 2024-03-24T03:38:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10978.259493 secs 2024-03-24T03:38:25.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:25 smithi078 ceph-mon[32429]: pgmap v6262: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:25.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:25 smithi107 ceph-mon[36485]: pgmap v6262: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:25.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:25 smithi106 ceph-mon[36505]: pgmap v6262: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:27.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:27 smithi078 ceph-mon[32429]: pgmap v6263: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:38:27.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:27 smithi107 ceph-mon[36485]: pgmap v6263: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:38:27.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:27 smithi106 ceph-mon[36505]: pgmap v6263: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:38:28.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:28 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:38:28.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:28 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:38:28.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:28 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:38:29.553 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10983.259552 secs 2024-03-24T03:38:29.554 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:29 smithi106 ceph-mon[36505]: pgmap v6264: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:29.554 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:29 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:38:29.554 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:29 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:38:29.554 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:29 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:38:29.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10983.259552 secs 2024-03-24T03:38:29.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:29 smithi078 ceph-mon[32429]: pgmap v6264: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:29.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:29 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:38:29.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:29 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:38:29.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:29 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:38:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10983.259552 secs 2024-03-24T03:38:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:29 smithi107 ceph-mon[36485]: pgmap v6264: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:29 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:38:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:29 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:38:29.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:29 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:38:30.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:30 smithi078 ceph-mon[32429]: pgmap v6265: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:38:30.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:30 smithi078 ceph-mon[32429]: pgmap v6266: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:38:30.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:30 smithi107 ceph-mon[36485]: pgmap v6265: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:38:30.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:30 smithi107 ceph-mon[36485]: pgmap v6266: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:38:30.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:30 smithi106 ceph-mon[36505]: pgmap v6265: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:38:30.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:30 smithi106 ceph-mon[36505]: pgmap v6266: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:38:32.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:32 smithi078 ceph-mon[32429]: pgmap v6267: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:38:32.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:32 smithi107 ceph-mon[36485]: pgmap v6267: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:38:32.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:32 smithi106 ceph-mon[36505]: pgmap v6267: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:38:34.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:34 smithi078 ceph-mon[32429]: pgmap v6268: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:38:34.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10988.259635 secs 2024-03-24T03:38:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:34 smithi107 ceph-mon[36485]: pgmap v6268: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:38:34.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10988.259635 secs 2024-03-24T03:38:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:34 smithi106 ceph-mon[36505]: pgmap v6268: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:38:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10988.259635 secs 2024-03-24T03:38:36.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:36 smithi078 ceph-mon[32429]: pgmap v6269: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:38:36.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:36 smithi107 ceph-mon[36485]: pgmap v6269: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:38:36.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:36 smithi106 ceph-mon[36505]: pgmap v6269: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:38:38.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:38 smithi078 ceph-mon[32429]: pgmap v6270: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:38:38.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:38 smithi107 ceph-mon[36485]: pgmap v6270: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:38:38.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:38 smithi106 ceph-mon[36505]: pgmap v6270: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:38:39.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10993.259676 secs 2024-03-24T03:38:39.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10993.259676 secs 2024-03-24T03:38:39.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10993.259676 secs 2024-03-24T03:38:40.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:40 smithi078 ceph-mon[32429]: pgmap v6271: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:38:40.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:40 smithi107 ceph-mon[36485]: pgmap v6271: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:38:40.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:40 smithi106 ceph-mon[36505]: pgmap v6271: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:38:42.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:42 smithi078 ceph-mon[32429]: pgmap v6272: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:38:42.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:42 smithi107 ceph-mon[36485]: pgmap v6272: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:38:42.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:42 smithi106 ceph-mon[36505]: pgmap v6272: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:38:44.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:44 smithi078 ceph-mon[32429]: pgmap v6273: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:44.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 10998.259736 secs 2024-03-24T03:38:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:44 smithi107 ceph-mon[36485]: pgmap v6273: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:44.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 10998.259736 secs 2024-03-24T03:38:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:44 smithi106 ceph-mon[36505]: pgmap v6273: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:44.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 10998.259736 secs 2024-03-24T03:38:46.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:46 smithi078 ceph-mon[32429]: pgmap v6274: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:46.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:46 smithi107 ceph-mon[36485]: pgmap v6274: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:46.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:46 smithi106 ceph-mon[36505]: pgmap v6274: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:38:48.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:48 smithi078 ceph-mon[32429]: pgmap v6275: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-24T03:38:48.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:48 smithi107 ceph-mon[36485]: pgmap v6275: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-24T03:38:48.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:48 smithi106 ceph-mon[36505]: pgmap v6275: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-24T03:38:49.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:49 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 11003.259795 secs 2024-03-24T03:38:49.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:49 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 11003.259795 secs 2024-03-24T03:38:49.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:49 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 11003.259795 secs 2024-03-24T03:38:50.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:50 smithi078 ceph-mon[32429]: pgmap v6276: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-24T03:38:50.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:50 smithi107 ceph-mon[36485]: pgmap v6276: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-24T03:38:50.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:50 smithi106 ceph-mon[36505]: pgmap v6276: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-24T03:38:52.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:52 smithi078 ceph-mon[32429]: pgmap v6277: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-24T03:38:52.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:52 smithi107 ceph-mon[36485]: pgmap v6277: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-24T03:38:52.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:52 smithi106 ceph-mon[36505]: pgmap v6277: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-24T03:38:54.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:54 smithi078 ceph-mon[32429]: pgmap v6278: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 426 B/s wr, 4 op/s 2024-03-24T03:38:54.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:54 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 11008.259882 secs 2024-03-24T03:38:54.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:54 smithi107 ceph-mon[36485]: pgmap v6278: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 426 B/s wr, 4 op/s 2024-03-24T03:38:54.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:54 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 11008.259882 secs 2024-03-24T03:38:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:54 smithi106 ceph-mon[36505]: pgmap v6278: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 426 B/s wr, 4 op/s 2024-03-24T03:38:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:54 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 11008.259882 secs 2024-03-24T03:38:56.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:56 smithi078 ceph-mon[32429]: pgmap v6279: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 426 B/s wr, 4 op/s 2024-03-24T03:38:56.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:56 smithi107 ceph-mon[36485]: pgmap v6279: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 426 B/s wr, 4 op/s 2024-03-24T03:38:56.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:56 smithi106 ceph-mon[36505]: pgmap v6279: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 426 B/s wr, 4 op/s 2024-03-24T03:38:58.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:58 smithi078 ceph-mon[32429]: pgmap v6280: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-24T03:38:58.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:58 smithi107 ceph-mon[36485]: pgmap v6280: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-24T03:38:58.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:58 smithi106 ceph-mon[36505]: pgmap v6280: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-24T03:38:59.553 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:38:59 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 11013.259976 secs 2024-03-24T03:38:59.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:38:59 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 11013.259976 secs 2024-03-24T03:38:59.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:38:59 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 11013.259976 secs 2024-03-24T03:39:00.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:00 smithi078 ceph-mon[32429]: pgmap v6281: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 85 B/s wr, 3 op/s 2024-03-24T03:39:00.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:00 smithi107 ceph-mon[36485]: pgmap v6281: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 85 B/s wr, 3 op/s 2024-03-24T03:39:00.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:00 smithi106 ceph-mon[36505]: pgmap v6281: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 85 B/s wr, 3 op/s 2024-03-24T03:39:02.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:02 smithi078 ceph-mon[32429]: pgmap v6282: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-24T03:39:02.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:02 smithi107 ceph-mon[36485]: pgmap v6282: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-24T03:39:02.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:02 smithi106 ceph-mon[36505]: pgmap v6282: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-24T03:39:04.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:04 smithi078 ceph-mon[32429]: pgmap v6283: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:39:04.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:04 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 11018.260054 secs 2024-03-24T03:39:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:04 smithi107 ceph-mon[36485]: pgmap v6283: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:39:04.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:04 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 11018.260054 secs 2024-03-24T03:39:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:04 smithi106 ceph-mon[36505]: pgmap v6283: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:39:04.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:04 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 11018.260054 secs 2024-03-24T03:39:06.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:06 smithi078 ceph-mon[32429]: pgmap v6284: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:39:06.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:06 smithi107 ceph-mon[36485]: pgmap v6284: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:39:06.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:06 smithi106 ceph-mon[36505]: pgmap v6284: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:39:08.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:08 smithi078 ceph-mon[32429]: pgmap v6285: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:39:08.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:08 smithi107 ceph-mon[36485]: pgmap v6285: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:39:08.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:08 smithi106 ceph-mon[36505]: pgmap v6285: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:39:09.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:09 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 11023.260142 secs 2024-03-24T03:39:09.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:09 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 11023.260142 secs 2024-03-24T03:39:09.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:09 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 11023.260142 secs 2024-03-24T03:39:10.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:10 smithi078 ceph-mon[32429]: pgmap v6286: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:39:10.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:10 smithi107 ceph-mon[36485]: pgmap v6286: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:39:10.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:10 smithi106 ceph-mon[36505]: pgmap v6286: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:39:12.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:12 smithi078 ceph-mon[32429]: pgmap v6287: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:39:12.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:12 smithi107 ceph-mon[36485]: pgmap v6287: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:39:12.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:12 smithi106 ceph-mon[36505]: pgmap v6287: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:39:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:14 smithi078 ceph-mon[32429]: pgmap v6288: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:39:14.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:14 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 11028.260223 secs 2024-03-24T03:39:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:14 smithi107 ceph-mon[36485]: pgmap v6288: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:39:14.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:14 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 11028.260223 secs 2024-03-24T03:39:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:14 smithi106 ceph-mon[36505]: pgmap v6288: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:39:14.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:14 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 11028.260223 secs 2024-03-24T03:39:16.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:16 smithi078 ceph-mon[32429]: pgmap v6289: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:39:16.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:16 smithi107 ceph-mon[36485]: pgmap v6289: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:39:16.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:16 smithi106 ceph-mon[36505]: pgmap v6289: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:39:18.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:18 smithi078 ceph-mon[32429]: pgmap v6290: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:39:18.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:18 smithi107 ceph-mon[36485]: pgmap v6290: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:39:18.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:18 smithi106 ceph-mon[36505]: pgmap v6290: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:39:19.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:19 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 11033.260310 secs 2024-03-24T03:39:19.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:19 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 11033.260310 secs 2024-03-24T03:39:19.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:19 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 11033.260310 secs 2024-03-24T03:39:20.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:20 smithi078 ceph-mon[32429]: pgmap v6291: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:39:20.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:20 smithi107 ceph-mon[36485]: pgmap v6291: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:39:20.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:20 smithi106 ceph-mon[36505]: pgmap v6291: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:39:22.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:22 smithi078 ceph-mon[32429]: pgmap v6292: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:39:22.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:22 smithi107 ceph-mon[36485]: pgmap v6292: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:39:22.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:22 smithi106 ceph-mon[36505]: pgmap v6292: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:39:24.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:24 smithi078 ceph-mon[32429]: pgmap v6293: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:39:24.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:24 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 11038.260351 secs 2024-03-24T03:39:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:24 smithi107 ceph-mon[36485]: pgmap v6293: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:39:24.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:24 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 11038.260351 secs 2024-03-24T03:39:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:24 smithi106 ceph-mon[36505]: pgmap v6293: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:39:24.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:24 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 11038.260351 secs 2024-03-24T03:39:26.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:26 smithi078 ceph-mon[32429]: pgmap v6294: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:39:26.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:26 smithi107 ceph-mon[36485]: pgmap v6294: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:39:26.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:26 smithi106 ceph-mon[36505]: pgmap v6294: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:39:28.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:28 smithi078 ceph-mon[32429]: pgmap v6295: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:39:28.593 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:28 smithi107 ceph-mon[36485]: pgmap v6295: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:39:28.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:28 smithi106 ceph-mon[36505]: pgmap v6295: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:39:29.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:29 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 11043.260438 secs 2024-03-24T03:39:29.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:29 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:39:29.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:29 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 11043.260438 secs 2024-03-24T03:39:29.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:29 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:39:29.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:29 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 11043.260438 secs 2024-03-24T03:39:29.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:29 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:39:30.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:30 smithi106 ceph-mon[36505]: pgmap v6296: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:39:30.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:30 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:39:30.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:30 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:39:30.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:30 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:39:30.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:30 smithi078 ceph-mon[32429]: pgmap v6296: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:39:30.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:30 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:39:30.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:30 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:39:30.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:30 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:39:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:30 smithi107 ceph-mon[36485]: pgmap v6296: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:39:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:30 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:39:30.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:30 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:39:30.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:30 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' 2024-03-24T03:39:31.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:31 smithi106 ceph-mon[36505]: pgmap v6297: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:39:31.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:31 smithi106 ceph-mon[36505]: pgmap v6298: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:39:31.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:31 smithi078 ceph-mon[32429]: pgmap v6297: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:39:31.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:31 smithi078 ceph-mon[32429]: pgmap v6298: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:39:31.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:31 smithi107 ceph-mon[36485]: pgmap v6297: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-24T03:39:31.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:31 smithi107 ceph-mon[36485]: pgmap v6298: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-24T03:39:33.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:33 smithi106 ceph-mon[36505]: pgmap v6299: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:39:33.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:33 smithi078 ceph-mon[32429]: pgmap v6299: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:39:33.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:33 smithi107 ceph-mon[36485]: pgmap v6299: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:39:34.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:34 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 11048.260496 secs 2024-03-24T03:39:34.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:34 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 11048.260496 secs 2024-03-24T03:39:34.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:34 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 11048.260496 secs 2024-03-24T03:39:35.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:35 smithi106 ceph-mon[36505]: pgmap v6300: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:39:35.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:35 smithi078 ceph-mon[32429]: pgmap v6300: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:39:35.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:35 smithi107 ceph-mon[36485]: pgmap v6300: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:39:36.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:36 smithi106 ceph-mon[36505]: pgmap v6301: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:39:36.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:36 smithi078 ceph-mon[32429]: pgmap v6301: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:39:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:36 smithi107 ceph-mon[36485]: pgmap v6301: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:39:39.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:38 smithi078 ceph-mon[32429]: pgmap v6302: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:39:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:38 smithi107 ceph-mon[36485]: pgmap v6302: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:39:39.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:38 smithi106 ceph-mon[36505]: pgmap v6302: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-24T03:39:40.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:39 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 11053.260580 secs 2024-03-24T03:39:40.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:39 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 11053.260580 secs 2024-03-24T03:39:40.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:39 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 11053.260580 secs 2024-03-24T03:39:41.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:40 smithi078 ceph-mon[32429]: pgmap v6303: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:39:41.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:40 smithi107 ceph-mon[36485]: pgmap v6303: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:39:41.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:40 smithi106 ceph-mon[36505]: pgmap v6303: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:39:43.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:42 smithi078 ceph-mon[32429]: pgmap v6304: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:39:43.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:42 smithi107 ceph-mon[36485]: pgmap v6304: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:39:43.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:42 smithi106 ceph-mon[36505]: pgmap v6304: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:39:45.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:44 smithi078 ceph-mon[32429]: 2 slow requests, 0 included below; oldest blocked for > 11058.260662 secs 2024-03-24T03:39:45.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:44 smithi078 ceph-mon[32429]: pgmap v6305: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:39:45.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:44 smithi107 ceph-mon[36485]: 2 slow requests, 0 included below; oldest blocked for > 11058.260662 secs 2024-03-24T03:39:45.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:44 smithi107 ceph-mon[36485]: pgmap v6305: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:39:45.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:44 smithi106 ceph-mon[36505]: 2 slow requests, 0 included below; oldest blocked for > 11058.260662 secs 2024-03-24T03:39:45.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:44 smithi106 ceph-mon[36505]: pgmap v6305: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:39:46.686 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi078.front.sepia.ceph.com 2024-03-24T03:39:46.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T03:39:46.686 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T03:39:46.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T03:39:46.704 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2024-03-24T03:39:46.760 INFO:teuthology.orchestra.run.smithi078.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T03:39:46.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T03:39:46.761 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T03:39:46.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T03:39:46.834 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link delete ceph-brx 2024-03-24T03:39:46.928 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-24T03:39:46.929 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-24T03:39:46.929 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T03:39:46.945 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056355,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=397 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/e7b1b482a113a0ff7e53cc7c3e69a311ae73cfa73ed294530d851ffffd03a9fb/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/GBRB25UQHVZKFGLHRMQ47OY5RJ:/var/lib/containers/storage/overlay/l/OUUWFT2EIJAISQ5EKDZZLCX2I2,upperdir=/var/lib/containers/storage/overlay/e7b1b482a113a0ff7e53cc7c3e69a311ae73cfa73ed294530d851ffffd03a9fb/diff,workdir=/var/lib/containers/storage/overlay/e7b1b482a113a0ff7e53cc7c3e69a311ae73cfa73ed294530d851ffffd03a9fb/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/836075a9eb57d2f287a1c08cae3365e912ef285bd576dfbd4529bb8a00fc0094/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/GBRB25UQHVZKFGLHRMQ47OY5RJ:/var/lib/containers/storage/overlay/l/OUUWFT2EIJAISQ5EKDZZLCX2I2,upperdir=/var/lib/containers/storage/overlay/836075a9eb57d2f287a1c08cae3365e912ef285bd576dfbd4529bb8a00fc0094/diff,workdir=/var/lib/containers/storage/overlay/836075a9eb57d2f287a1c08cae3365e912ef285bd576dfbd4529bb8a00fc0094/work,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,inode64 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/660b0fa5df42f8f1ac2a83b5e91aacb4a0bf276fd34107ccd3e5c06e96ea805a/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/GBRB25UQHVZKFGLHRMQ47OY5RJ:/var/lib/containers/storage/overlay/l/OUUWFT2EIJAISQ5EKDZZLCX2I2,upperdir=/var/lib/containers/storage/overlay/660b0fa5df42f8f1ac2a83b5e91aacb4a0bf276fd34107ccd3e5c06e96ea805a/diff,workdir=/var/lib/containers/storage/overlay/660b0fa5df42f8f1ac2a83b5e91aacb4a0bf276fd34107ccd3e5c06e96ea805a/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/d13b30b2053acb039309feee811247995f10a8aa81af3835ac2444d9f005f3af/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/GBRB25UQHVZKFGLHRMQ47OY5RJ:/var/lib/containers/storage/overlay/l/OUUWFT2EIJAISQ5EKDZZLCX2I2,upperdir=/var/lib/containers/storage/overlay/d13b30b2053acb039309feee811247995f10a8aa81af3835ac2444d9f005f3af/diff,workdir=/var/lib/containers/storage/overlay/d13b30b2053acb039309feee811247995f10a8aa81af3835ac2444d9f005f3af/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/31caf2291fe46f3f576a490ad8794999585c7ea587c14858a05884722f024f7f/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/GBRB25UQHVZKFGLHRMQ47OY5RJ:/var/lib/containers/storage/overlay/l/OUUWFT2EIJAISQ5EKDZZLCX2I2,upperdir=/var/lib/containers/storage/overlay/31caf2291fe46f3f576a490ad8794999585c7ea587c14858a05884722f024f7f/diff,workdir=/var/lib/containers/storage/overlay/31caf2291fe46f3f576a490ad8794999585c7ea587c14858a05884722f024f7f/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/a3bbbb6b63f54fc44b0914f7a5bc8fdaa4b2744163a16d501d6a5b8eb8e9ab09/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/GBRB25UQHVZKFGLHRMQ47OY5RJ:/var/lib/containers/storage/overlay/l/OUUWFT2EIJAISQ5EKDZZLCX2I2,upperdir=/var/lib/containers/storage/overlay/a3bbbb6b63f54fc44b0914f7a5bc8fdaa4b2744163a16d501d6a5b8eb8e9ab09/diff,workdir=/var/lib/containers/storage/overlay/a3bbbb6b63f54fc44b0914f7a5bc8fdaa4b2744163a16d501d6a5b8eb8e9ab09/work,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 2024-03-24T03:39:46.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T03:39:46.945 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2024-03-24T03:39:46.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:46 smithi078 ceph-mon[32429]: pgmap v6306: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:39:47.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T03:39:47.002 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link delete ceph-brx 2024-03-24T03:39:47.072 INFO:teuthology.orchestra.run.smithi078.stderr:Cannot find device "ceph-brx" 2024-03-24T03:39:47.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T03:39:47.074 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-24T03:39:47.085 INFO:tasks.cephadm:Teardown begin 2024-03-24T03:39:47.085 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-24T03:39:47.085 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T03:39:47.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:46 smithi107 ceph-mon[36485]: pgmap v6306: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:39:47.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:46 smithi106 ceph-mon[36505]: pgmap v6306: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:39:47.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.217+0000 7fae6aadc640 1 -- 172.21.15.78:0/436268244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae64107640 msgr2=0x7fae64107a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:47.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.217+0000 7fae6aadc640 1 --2- 172.21.15.78:0/436268244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae64107640 0x7fae64107a20 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fae58009930 tx=0x7fae5802bc30 comp rx=0 tx=0).stop 2024-03-24T03:39:47.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.217+0000 7fae6aadc640 1 -- 172.21.15.78:0/436268244 shutdown_connections 2024-03-24T03:39:47.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.217+0000 7fae6aadc640 1 --2- 172.21.15.78:0/436268244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae64101630 0x7fae64101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:47.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.217+0000 7fae6aadc640 1 --2- 172.21.15.78:0/436268244 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae64101ff0 0x7fae6410e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:47.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.217+0000 7fae6aadc640 1 --2- 172.21.15.78:0/436268244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae64107640 0x7fae64107a20 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:47.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.217+0000 7fae6aadc640 1 -- 172.21.15.78:0/436268244 >> 172.21.15.78:0/436268244 conn(0x7fae640fd140 msgr2=0x7fae640ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:39:47.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.217+0000 7fae6aadc640 1 -- 172.21.15.78:0/436268244 shutdown_connections 2024-03-24T03:39:47.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.217+0000 7fae6aadc640 1 -- 172.21.15.78:0/436268244 wait complete. 2024-03-24T03:39:47.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.218+0000 7fae6aadc640 1 Processor -- start 2024-03-24T03:39:47.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.218+0000 7fae6aadc640 1 -- start start 2024-03-24T03:39:47.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.218+0000 7fae6aadc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae64101630 0x7fae64197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:39:47.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.218+0000 7fae6aadc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae64101ff0 0x7fae641984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:39:47.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.218+0000 7fae6aadc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae64107640 0x7fae6419c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:39:47.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.218+0000 7fae6aadc640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae64111a50 con 0x7fae64101ff0 2024-03-24T03:39:47.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.218+0000 7fae6aadc640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae641118d0 con 0x7fae64101630 2024-03-24T03:39:47.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.218+0000 7fae6aadc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae64111bd0 con 0x7fae64107640 2024-03-24T03:39:47.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.218+0000 7fae63fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae64101ff0 0x7fae641984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:39:47.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.218+0000 7fae63fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae64101ff0 0x7fae641984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58730/0 (socket says 172.21.15.78:58730) 2024-03-24T03:39:47.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.218+0000 7fae68878640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae64101630 0x7fae64197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:39:47.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.218+0000 7fae63fff640 1 -- 172.21.15.78:0/1057865184 learned_addr learned my addr 172.21.15.78:0/1057865184 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T03:39:47.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.219+0000 7fae63fff640 1 -- 172.21.15.78:0/1057865184 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae64107640 msgr2=0x7fae6419c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:47.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.219+0000 7fae69079640 1 --2- 172.21.15.78:0/1057865184 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae64107640 0x7fae6419c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:39:47.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.219+0000 7fae63fff640 1 --2- 172.21.15.78:0/1057865184 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae64107640 0x7fae6419c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:47.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.219+0000 7fae63fff640 1 -- 172.21.15.78:0/1057865184 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae64101630 msgr2=0x7fae64197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:47.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.219+0000 7fae63fff640 1 --2- 172.21.15.78:0/1057865184 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae64101630 0x7fae64197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:47.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.219+0000 7fae63fff640 1 -- 172.21.15.78:0/1057865184 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae6419cfa0 con 0x7fae64101ff0 2024-03-24T03:39:47.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.219+0000 7fae68878640 1 --2- 172.21.15.78:0/1057865184 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae64101630 0x7fae64197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T03:39:47.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.219+0000 7fae63fff640 1 --2- 172.21.15.78:0/1057865184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae64101ff0 0x7fae641984f0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fae5000c970 tx=0x7fae5000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:39:47.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.219+0000 7fae61ffb640 1 -- 172.21.15.78:0/1057865184 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae50007bb0 con 0x7fae64101ff0 2024-03-24T03:39:47.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.219+0000 7fae61ffb640 1 -- 172.21.15.78:0/1057865184 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fae50007d50 con 0x7fae64101ff0 2024-03-24T03:39:47.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.219+0000 7fae61ffb640 1 -- 172.21.15.78:0/1057865184 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5000f450 con 0x7fae64101ff0 2024-03-24T03:39:47.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.219+0000 7fae6aadc640 1 -- 172.21.15.78:0/1057865184 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae6419d290 con 0x7fae64101ff0 2024-03-24T03:39:47.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.219+0000 7fae6aadc640 1 -- 172.21.15.78:0/1057865184 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae641a4b20 con 0x7fae64101ff0 2024-03-24T03:39:47.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.220+0000 7fae6aadc640 1 -- 172.21.15.78:0/1057865184 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae640688e0 con 0x7fae64101ff0 2024-03-24T03:39:47.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.221+0000 7fae61ffb640 1 -- 172.21.15.78:0/1057865184 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fae50005020 con 0x7fae64101ff0 2024-03-24T03:39:47.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.222+0000 7fae61ffb640 1 --2- 172.21.15.78:0/1057865184 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fae540a2970 0x7fae540a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:39:47.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.222+0000 7fae68878640 1 --2- 172.21.15.78:0/1057865184 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fae540a2970 0x7fae540a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:39:47.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.222+0000 7fae61ffb640 1 -- 172.21.15.78:0/1057865184 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 13278+0+0 (secure 0 0 0) 0x7fae500cfec0 con 0x7fae64101ff0 2024-03-24T03:39:47.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.222+0000 7fae68878640 1 --2- 172.21.15.78:0/1057865184 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fae540a2970 0x7fae540a4e30 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fae58005ab0 tx=0x7fae58005fd0 comp rx=0 tx=0).ready entity=mgr.25243 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:39:47.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.227+0000 7fae61ffb640 1 -- 172.21.15.78:0/1057865184 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fae50014030 con 0x7fae64101ff0 2024-03-24T03:39:47.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.437+0000 7fae6aadc640 1 -- 172.21.15.78:0/1057865184 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae6402d070 con 0x7fae64101ff0 2024-03-24T03:39:47.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.437+0000 7fae61ffb640 1 -- 172.21.15.78:0/1057865184 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 307 v307) v1 ==== 95+0+5908 (secure 0 0 0) 0x7fae5009b990 con 0x7fae64101ff0 2024-03-24T03:39:47.439 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T03:39:47.439 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":307,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":307,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:18:43.592215+0000","modified":"2024-03-24T00:39:04.275409+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":154,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24500,"mds_1":15747,"mds_2":24464},"failed":[],"damaged":[],"stopped":[],"info":{"gid_15747":{"gid":15747,"name":"j","rank":1,"incarnation":300,"state":"up:active","state_seq":5,"addr":"172.21.15.78:6835/3027423976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3027423976},{"type":"v1","addr":"172.21.15.78:6835","nonce":3027423976}]},"join_fscid":-1,"laggy_since":"2024-03-24T00:35:39.142109+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/2231611477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":2231611477},{"type":"v1","addr":"172.21.15.106:6835","nonce":2231611477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6837/2939571540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2939571540},{"type":"v1","addr":"172.21.15.107:6837","nonce":2939571540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24464":{"gid":24464,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.106:6837/2090968633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":2090968633},{"type":"v1","addr":"172.21.15.106:6837","nonce":2090968633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":308,"addr":"172.21.15.106:6839/2482071304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2482071304},{"type":"v1","addr":"172.21.15.106:6839","nonce":2482071304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24464,24500,15747]},"id":1}]} 2024-03-24T03:39:47.439 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 307 2024-03-24T03:39:47.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.439+0000 7fae6aadc640 1 -- 172.21.15.78:0/1057865184 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fae540a2970 msgr2=0x7fae540a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:47.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.439+0000 7fae6aadc640 1 --2- 172.21.15.78:0/1057865184 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fae540a2970 0x7fae540a4e30 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fae58005ab0 tx=0x7fae58005fd0 comp rx=0 tx=0).stop 2024-03-24T03:39:47.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.439+0000 7fae6aadc640 1 -- 172.21.15.78:0/1057865184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae64101ff0 msgr2=0x7fae641984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:47.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.439+0000 7fae6aadc640 1 --2- 172.21.15.78:0/1057865184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae64101ff0 0x7fae641984f0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fae5000c970 tx=0x7fae5000ce30 comp rx=0 tx=0).stop 2024-03-24T03:39:47.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.440+0000 7fae6aadc640 1 -- 172.21.15.78:0/1057865184 shutdown_connections 2024-03-24T03:39:47.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.440+0000 7fae6aadc640 1 --2- 172.21.15.78:0/1057865184 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fae540a2970 0x7fae540a4e30 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:47.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.440+0000 7fae6aadc640 1 --2- 172.21.15.78:0/1057865184 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae64107640 0x7fae6419c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:47.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.440+0000 7fae6aadc640 1 --2- 172.21.15.78:0/1057865184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae64101ff0 0x7fae641984f0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:47.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.440+0000 7fae6aadc640 1 --2- 172.21.15.78:0/1057865184 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae64101630 0x7fae64197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:47.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.440+0000 7fae6aadc640 1 -- 172.21.15.78:0/1057865184 >> 172.21.15.78:0/1057865184 conn(0x7fae640fd140 msgr2=0x7fae640fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:39:47.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.440+0000 7fae6aadc640 1 -- 172.21.15.78:0/1057865184 shutdown_connections 2024-03-24T03:39:47.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.440+0000 7fae6aadc640 1 -- 172.21.15.78:0/1057865184 wait complete. 2024-03-24T03:39:47.451 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T03:39:47.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.551+0000 7fbf56df4640 1 -- 172.21.15.78:0/4046383690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf50100f40 msgr2=0x7fbf5010d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:47.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.551+0000 7fbf56df4640 1 --2- 172.21.15.78:0/4046383690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf50100f40 0x7fbf5010d450 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fbf40009930 tx=0x7fbf4002bc30 comp rx=0 tx=0).stop 2024-03-24T03:39:47.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.551+0000 7fbf56df4640 1 -- 172.21.15.78:0/4046383690 shutdown_connections 2024-03-24T03:39:47.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.551+0000 7fbf56df4640 1 --2- 172.21.15.78:0/4046383690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf50100580 0x7fbf50100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:47.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.551+0000 7fbf56df4640 1 --2- 172.21.15.78:0/4046383690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf50100f40 0x7fbf5010d450 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:47.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.551+0000 7fbf56df4640 1 --2- 172.21.15.78:0/4046383690 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf501055a0 0x7fbf50105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:47.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.551+0000 7fbf56df4640 1 -- 172.21.15.78:0/4046383690 >> 172.21.15.78:0/4046383690 conn(0x7fbf500776d0 msgr2=0x7fbf50077ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:39:47.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.552+0000 7fbf56df4640 1 -- 172.21.15.78:0/4046383690 shutdown_connections 2024-03-24T03:39:47.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.552+0000 7fbf56df4640 1 -- 172.21.15.78:0/4046383690 wait complete. 2024-03-24T03:39:47.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.552+0000 7fbf56df4640 1 Processor -- start 2024-03-24T03:39:47.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.552+0000 7fbf56df4640 1 -- start start 2024-03-24T03:39:47.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.552+0000 7fbf56df4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf50100580 0x7fbf50196e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:39:47.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.552+0000 7fbf56df4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf50100f40 0x7fbf50197340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:39:47.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.552+0000 7fbf56df4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf501055a0 0x7fbf5019b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:39:47.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.552+0000 7fbf56df4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf501109a0 con 0x7fbf50100580 2024-03-24T03:39:47.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.552+0000 7fbf56df4640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf50110820 con 0x7fbf501055a0 2024-03-24T03:39:47.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.552+0000 7fbf56df4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf50110b20 con 0x7fbf50100f40 2024-03-24T03:39:47.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.552+0000 7fbf54b90640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf50100580 0x7fbf50196e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:39:47.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.552+0000 7fbf54b90640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf50100580 0x7fbf50196e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58746/0 (socket says 172.21.15.78:58746) 2024-03-24T03:39:47.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.552+0000 7fbf54b90640 1 -- 172.21.15.78:0/1637101674 learned_addr learned my addr 172.21.15.78:0/1637101674 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T03:39:47.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.552+0000 7fbf55391640 1 --2- 172.21.15.78:0/1637101674 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf501055a0 0x7fbf5019b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:39:47.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.553+0000 7fbf4ffff640 1 --2- 172.21.15.78:0/1637101674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf50100f40 0x7fbf50197340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:39:47.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.553+0000 7fbf54b90640 1 -- 172.21.15.78:0/1637101674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf50100f40 msgr2=0x7fbf50197340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:47.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.553+0000 7fbf54b90640 1 --2- 172.21.15.78:0/1637101674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf50100f40 0x7fbf50197340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:47.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.553+0000 7fbf54b90640 1 -- 172.21.15.78:0/1637101674 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf501055a0 msgr2=0x7fbf5019b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:47.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.553+0000 7fbf54b90640 1 --2- 172.21.15.78:0/1637101674 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf501055a0 0x7fbf5019b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:47.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.553+0000 7fbf54b90640 1 -- 172.21.15.78:0/1637101674 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5019bdf0 con 0x7fbf50100580 2024-03-24T03:39:47.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.553+0000 7fbf4ffff640 1 --2- 172.21.15.78:0/1637101674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf50100f40 0x7fbf50197340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T03:39:47.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.553+0000 7fbf54b90640 1 --2- 172.21.15.78:0/1637101674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf50100580 0x7fbf50196e00 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fbf4800ebc0 tx=0x7fbf4800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:39:47.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.553+0000 7fbf4dffb640 1 -- 172.21.15.78:0/1637101674 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf48003b70 con 0x7fbf50100580 2024-03-24T03:39:47.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.553+0000 7fbf4dffb640 1 -- 172.21.15.78:0/1637101674 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf48003d10 con 0x7fbf50100580 2024-03-24T03:39:47.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.553+0000 7fbf56df4640 1 -- 172.21.15.78:0/1637101674 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf5019c0e0 con 0x7fbf50100580 2024-03-24T03:39:47.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.553+0000 7fbf4dffb640 1 -- 172.21.15.78:0/1637101674 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf48019980 con 0x7fbf50100580 2024-03-24T03:39:47.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.553+0000 7fbf56df4640 1 -- 172.21.15.78:0/1637101674 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf501a3970 con 0x7fbf50100580 2024-03-24T03:39:47.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.554+0000 7fbf56df4640 1 -- 172.21.15.78:0/1637101674 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf50067c00 con 0x7fbf50100580 2024-03-24T03:39:47.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.555+0000 7fbf4dffb640 1 -- 172.21.15.78:0/1637101674 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fbf48019b20 con 0x7fbf50100580 2024-03-24T03:39:47.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.556+0000 7fbf4dffb640 1 --2- 172.21.15.78:0/1637101674 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fbf380a2970 0x7fbf380a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:39:47.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.556+0000 7fbf4dffb640 1 -- 172.21.15.78:0/1637101674 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 13278+0+0 (secure 0 0 0) 0x7fbf480d0900 con 0x7fbf50100580 2024-03-24T03:39:47.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.556+0000 7fbf4ffff640 1 --2- 172.21.15.78:0/1637101674 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fbf380a2970 0x7fbf380a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:39:47.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.557+0000 7fbf4ffff640 1 --2- 172.21.15.78:0/1637101674 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fbf380a2970 0x7fbf380a4e30 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fbf440058c0 tx=0x7fbf44005850 comp rx=0 tx=0).ready entity=mgr.25243 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:39:47.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.560+0000 7fbf4dffb640 1 -- 172.21.15.78:0/1637101674 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbf4809c3d0 con 0x7fbf50100580 2024-03-24T03:39:47.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.777+0000 7fbf56df4640 1 -- 172.21.15.78:0/1637101674 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf50198150 con 0x7fbf50100580 2024-03-24T03:39:47.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.777+0000 7fbf4dffb640 1 -- 172.21.15.78:0/1637101674 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 307 v307) v1 ==== 95+0+5908 (secure 0 0 0) 0x7fbf4809bcd0 con 0x7fbf50100580 2024-03-24T03:39:47.778 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T03:39:47.779 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":307,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":307,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:18:43.592215+0000","modified":"2024-03-24T00:39:04.275409+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":154,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24500,"mds_1":15747,"mds_2":24464},"failed":[],"damaged":[],"stopped":[],"info":{"gid_15747":{"gid":15747,"name":"j","rank":1,"incarnation":300,"state":"up:active","state_seq":5,"addr":"172.21.15.78:6835/3027423976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3027423976},{"type":"v1","addr":"172.21.15.78:6835","nonce":3027423976}]},"join_fscid":-1,"laggy_since":"2024-03-24T00:35:39.142109+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/2231611477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":2231611477},{"type":"v1","addr":"172.21.15.106:6835","nonce":2231611477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6837/2939571540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2939571540},{"type":"v1","addr":"172.21.15.107:6837","nonce":2939571540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24464":{"gid":24464,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.106:6837/2090968633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":2090968633},{"type":"v1","addr":"172.21.15.106:6837","nonce":2090968633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":308,"addr":"172.21.15.106:6839/2482071304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2482071304},{"type":"v1","addr":"172.21.15.106:6839","nonce":2482071304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24464,24500,15747]},"id":1}]} 2024-03-24T03:39:47.779 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 307 2024-03-24T03:39:47.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.780+0000 7fbf56df4640 1 -- 172.21.15.78:0/1637101674 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fbf380a2970 msgr2=0x7fbf380a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:47.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.780+0000 7fbf56df4640 1 --2- 172.21.15.78:0/1637101674 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fbf380a2970 0x7fbf380a4e30 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fbf440058c0 tx=0x7fbf44005850 comp rx=0 tx=0).stop 2024-03-24T03:39:47.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.780+0000 7fbf56df4640 1 -- 172.21.15.78:0/1637101674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf50100580 msgr2=0x7fbf50196e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:47.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.780+0000 7fbf56df4640 1 --2- 172.21.15.78:0/1637101674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf50100580 0x7fbf50196e00 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fbf4800ebc0 tx=0x7fbf4800c6a0 comp rx=0 tx=0).stop 2024-03-24T03:39:47.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.780+0000 7fbf56df4640 1 -- 172.21.15.78:0/1637101674 shutdown_connections 2024-03-24T03:39:47.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.780+0000 7fbf56df4640 1 --2- 172.21.15.78:0/1637101674 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fbf380a2970 0x7fbf380a4e30 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:47.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.780+0000 7fbf56df4640 1 --2- 172.21.15.78:0/1637101674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf50100f40 0x7fbf50197340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:47.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.780+0000 7fbf56df4640 1 --2- 172.21.15.78:0/1637101674 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf501055a0 0x7fbf5019b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:47.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.780+0000 7fbf56df4640 1 --2- 172.21.15.78:0/1637101674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf50100580 0x7fbf50196e00 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:47.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.780+0000 7fbf56df4640 1 -- 172.21.15.78:0/1637101674 >> 172.21.15.78:0/1637101674 conn(0x7fbf500776d0 msgr2=0x7fbf50103140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:39:47.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.780+0000 7fbf56df4640 1 -- 172.21.15.78:0/1637101674 shutdown_connections 2024-03-24T03:39:47.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.780+0000 7fbf56df4640 1 -- 172.21.15.78:0/1637101674 wait complete. 2024-03-24T03:39:47.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:47 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1057865184' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T03:39:47.791 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T03:39:47.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.892+0000 7f5f55309640 1 -- 172.21.15.78:0/3630318077 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f50106670 msgr2=0x7f5f50106a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:47.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.892+0000 7f5f55309640 1 --2- 172.21.15.78:0/3630318077 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f50106670 0x7f5f50106a50 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f5f44009930 tx=0x7f5f4402bc30 comp rx=0 tx=0).stop 2024-03-24T03:39:47.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.892+0000 7f5f55309640 1 -- 172.21.15.78:0/3630318077 shutdown_connections 2024-03-24T03:39:47.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.892+0000 7f5f55309640 1 --2- 172.21.15.78:0/3630318077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f50100fe0 0x7f5f5010d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:47.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.892+0000 7f5f55309640 1 --2- 172.21.15.78:0/3630318077 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f50100620 0x7f5f50100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:47.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.892+0000 7f5f55309640 1 --2- 172.21.15.78:0/3630318077 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f50106670 0x7f5f50106a50 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:47.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.892+0000 7f5f55309640 1 -- 172.21.15.78:0/3630318077 >> 172.21.15.78:0/3630318077 conn(0x7f5f50077700 msgr2=0x7f5f50077b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:39:47.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.892+0000 7f5f55309640 1 -- 172.21.15.78:0/3630318077 shutdown_connections 2024-03-24T03:39:47.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.892+0000 7f5f55309640 1 -- 172.21.15.78:0/3630318077 wait complete. 2024-03-24T03:39:47.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.892+0000 7f5f55309640 1 Processor -- start 2024-03-24T03:39:47.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.892+0000 7f5f55309640 1 -- start start 2024-03-24T03:39:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.893+0000 7f5f55309640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f50100620 0x7f5f5010df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:39:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.893+0000 7f5f55309640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f50100fe0 0x7f5f5010e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:39:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.893+0000 7f5f55309640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f50106670 0x7f5f5010e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:39:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.893+0000 7f5f55309640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f50111af0 con 0x7f5f50106670 2024-03-24T03:39:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.893+0000 7f5f55309640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f50111970 con 0x7f5f50100fe0 2024-03-24T03:39:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.893+0000 7f5f55309640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f50111c70 con 0x7f5f50100620 2024-03-24T03:39:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.893+0000 7f5f4f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f50106670 0x7f5f5010e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:39:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.893+0000 7f5f4f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f50106670 0x7f5f5010e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58778/0 (socket says 172.21.15.78:58778) 2024-03-24T03:39:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.893+0000 7f5f4f7fe640 1 -- 172.21.15.78:0/2046124982 learned_addr learned my addr 172.21.15.78:0/2046124982 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T03:39:47.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.893+0000 7f5f4e7fc640 1 --2- 172.21.15.78:0/2046124982 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f50100fe0 0x7f5f5010e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:39:47.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.893+0000 7f5f4effd640 1 --2- 172.21.15.78:0/2046124982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f50100620 0x7f5f5010df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:39:47.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.893+0000 7f5f4f7fe640 1 -- 172.21.15.78:0/2046124982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f50100620 msgr2=0x7f5f5010df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:47.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.893+0000 7f5f4f7fe640 1 --2- 172.21.15.78:0/2046124982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f50100620 0x7f5f5010df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:47.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.893+0000 7f5f4f7fe640 1 -- 172.21.15.78:0/2046124982 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f50100fe0 msgr2=0x7f5f5010e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:47.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.893+0000 7f5f4f7fe640 1 --2- 172.21.15.78:0/2046124982 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f50100fe0 0x7f5f5010e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:47.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.893+0000 7f5f4f7fe640 1 -- 172.21.15.78:0/2046124982 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f501a6450 con 0x7f5f50106670 2024-03-24T03:39:47.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.893+0000 7f5f4effd640 1 --2- 172.21.15.78:0/2046124982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f50100620 0x7f5f5010df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T03:39:47.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.893+0000 7f5f4f7fe640 1 --2- 172.21.15.78:0/2046124982 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f50106670 0x7f5f5010e9b0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f5f4800eb10 tx=0x7f5f4800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:39:47.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.893+0000 7f5f37fff640 1 -- 172.21.15.78:0/2046124982 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f48004b20 con 0x7f5f50106670 2024-03-24T03:39:47.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.893+0000 7f5f55309640 1 -- 172.21.15.78:0/2046124982 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f501a6740 con 0x7f5f50106670 2024-03-24T03:39:47.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.893+0000 7f5f37fff640 1 -- 172.21.15.78:0/2046124982 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f48004cc0 con 0x7f5f50106670 2024-03-24T03:39:47.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.893+0000 7f5f37fff640 1 -- 172.21.15.78:0/2046124982 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f48010760 con 0x7f5f50106670 2024-03-24T03:39:47.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.893+0000 7f5f55309640 1 -- 172.21.15.78:0/2046124982 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f501a6ce0 con 0x7f5f50106670 2024-03-24T03:39:47.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.894+0000 7f5f55309640 1 -- 172.21.15.78:0/2046124982 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f500688e0 con 0x7f5f50106670 2024-03-24T03:39:47.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.896+0000 7f5f37fff640 1 -- 172.21.15.78:0/2046124982 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f5f48005000 con 0x7f5f50106670 2024-03-24T03:39:47.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.896+0000 7f5f37fff640 1 --2- 172.21.15.78:0/2046124982 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f5f380a2970 0x7f5f380a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:39:47.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.896+0000 7f5f4effd640 1 --2- 172.21.15.78:0/2046124982 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f5f380a2970 0x7f5f380a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:39:47.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.896+0000 7f5f37fff640 1 -- 172.21.15.78:0/2046124982 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 13278+0+0 (secure 0 0 0) 0x7f5f480d0b90 con 0x7f5f50106670 2024-03-24T03:39:47.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.897+0000 7f5f4effd640 1 --2- 172.21.15.78:0/2046124982 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f5f380a2970 0x7f5f380a4e30 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f5f44009900 tx=0x7f5f44005fd0 comp rx=0 tx=0).ready entity=mgr.25243 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:39:47.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:47.900+0000 7f5f37fff640 1 -- 172.21.15.78:0/2046124982 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5f4809c660 con 0x7f5f50106670 2024-03-24T03:39:48.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.075+0000 7f5f55309640 1 -- 172.21.15.78:0/2046124982 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f50108b40 con 0x7f5f50106670 2024-03-24T03:39:48.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.077+0000 7f5f37fff640 1 -- 172.21.15.78:0/2046124982 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v157) v1 ==== 74+0+30992 (secure 0 0 0) 0x7f5f4809bf60 con 0x7f5f50106670 2024-03-24T03:39:48.078 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-24T03:39:48.079 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":157,"fsid":"c7e2806a-e972-11ee-95cd-87774f69a715","created":"2024-03-24T00:10:44.134563+0000","modified":"2024-03-24T01:24:14.909719+0000","last_up_change":"2024-03-24T00:17:50.237686+0000","last_in_change":"2024-03-24T00:17:29.788699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:14:04.534391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:18:42.178418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:18:42.800424+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":23,"snap_epoch":155,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:18:46.112800+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":23,"snap_epoch":155,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b950388a-c437-426b-b703-3f8ae821dd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6803","nonce":1816879397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6805","nonce":1816879397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6809","nonce":1816879397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1816879397},{"type":"v1","addr":"172.21.15.78:6807","nonce":1816879397}]},"public_addr":"172.21.15.78:6803/1816879397","cluster_addr":"172.21.15.78:6805/1816879397","heartbeat_back_addr":"172.21.15.78:6809/1816879397","heartbeat_front_addr":"172.21.15.78:6807/1816879397","state":["exists","up"]},{"osd":1,"uuid":"b8d64714-dd65-400f-b3f4-673ab14e1365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6801","nonce":1647915612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6803","nonce":1647915612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6807","nonce":1647915612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":1647915612},{"type":"v1","addr":"172.21.15.106:6805","nonce":1647915612}]},"public_addr":"172.21.15.106:6801/1647915612","cluster_addr":"172.21.15.106:6803/1647915612","heartbeat_back_addr":"172.21.15.106:6807/1647915612","heartbeat_front_addr":"172.21.15.106:6805/1647915612","state":["exists","up"]},{"osd":2,"uuid":"fcfc4dd1-aaf8-4131-a023-5e832212158d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6801","nonce":2010770435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6803","nonce":2010770435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6807","nonce":2010770435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2010770435},{"type":"v1","addr":"172.21.15.107:6805","nonce":2010770435}]},"public_addr":"172.21.15.107:6801/2010770435","cluster_addr":"172.21.15.107:6803/2010770435","heartbeat_back_addr":"172.21.15.107:6807/2010770435","heartbeat_front_addr":"172.21.15.107:6805/2010770435","state":["exists","up"]},{"osd":3,"uuid":"47c718ab-a0f4-4e52-83c5-b89386b262d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6811","nonce":1912251346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6813","nonce":1912251346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6817","nonce":1912251346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":1912251346},{"type":"v1","addr":"172.21.15.78:6815","nonce":1912251346}]},"public_addr":"172.21.15.78:6811/1912251346","cluster_addr":"172.21.15.78:6813/1912251346","heartbeat_back_addr":"172.21.15.78:6817/1912251346","heartbeat_front_addr":"172.21.15.78:6815/1912251346","state":["exists","up"]},{"osd":4,"uuid":"7dad64dd-677e-4c1c-9638-e4f4f2195983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6809","nonce":374641205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6811","nonce":374641205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6815","nonce":374641205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":374641205},{"type":"v1","addr":"172.21.15.106:6813","nonce":374641205}]},"public_addr":"172.21.15.106:6809/374641205","cluster_addr":"172.21.15.106:6811/374641205","heartbeat_back_addr":"172.21.15.106:6815/374641205","heartbeat_front_addr":"172.21.15.106:6813/374641205","state":["exists","up"]},{"osd":5,"uuid":"bbd524e5-328d-4309-8877-1149cbe110dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6809","nonce":3272806073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6811","nonce":3272806073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6815","nonce":3272806073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3272806073},{"type":"v1","addr":"172.21.15.107:6813","nonce":3272806073}]},"public_addr":"172.21.15.107:6809/3272806073","cluster_addr":"172.21.15.107:6811/3272806073","heartbeat_back_addr":"172.21.15.107:6815/3272806073","heartbeat_front_addr":"172.21.15.107:6813/3272806073","state":["exists","up"]},{"osd":6,"uuid":"5696389a-918f-4ea8-b20d-4e2a0d4cb3bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6819","nonce":3091148508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6821","nonce":3091148508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6825","nonce":3091148508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3091148508},{"type":"v1","addr":"172.21.15.78:6823","nonce":3091148508}]},"public_addr":"172.21.15.78:6819/3091148508","cluster_addr":"172.21.15.78:6821/3091148508","heartbeat_back_addr":"172.21.15.78:6825/3091148508","heartbeat_front_addr":"172.21.15.78:6823/3091148508","state":["exists","up"]},{"osd":7,"uuid":"f6b764e8-ed4e-4c9a-a85b-94af6dc6b924","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6817","nonce":4262103080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6819","nonce":4262103080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6823","nonce":4262103080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4262103080},{"type":"v1","addr":"172.21.15.106:6821","nonce":4262103080}]},"public_addr":"172.21.15.106:6817/4262103080","cluster_addr":"172.21.15.106:6819/4262103080","heartbeat_back_addr":"172.21.15.106:6823/4262103080","heartbeat_front_addr":"172.21.15.106:6821/4262103080","state":["exists","up"]},{"osd":8,"uuid":"ba16039f-8093-4fa0-87ad-e28b7d142b23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6817","nonce":4032799356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6819","nonce":4032799356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6823","nonce":4032799356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":4032799356},{"type":"v1","addr":"172.21.15.107:6821","nonce":4032799356}]},"public_addr":"172.21.15.107:6817/4032799356","cluster_addr":"172.21.15.107:6819/4032799356","heartbeat_back_addr":"172.21.15.107:6823/4032799356","heartbeat_front_addr":"172.21.15.107:6821/4032799356","state":["exists","up"]},{"osd":9,"uuid":"651357f2-0669-42e6-aa91-1d5c02d00c5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6827","nonce":214023431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6829","nonce":214023431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6833","nonce":214023431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":214023431},{"type":"v1","addr":"172.21.15.78:6831","nonce":214023431}]},"public_addr":"172.21.15.78:6827/214023431","cluster_addr":"172.21.15.78:6829/214023431","heartbeat_back_addr":"172.21.15.78:6833/214023431","heartbeat_front_addr":"172.21.15.78:6831/214023431","state":["exists","up"]},{"osd":10,"uuid":"2375119d-346e-426c-8a65-b64af71ad63a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6825","nonce":400029804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6827","nonce":400029804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6831","nonce":400029804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":400029804},{"type":"v1","addr":"172.21.15.106:6829","nonce":400029804}]},"public_addr":"172.21.15.106:6825/400029804","cluster_addr":"172.21.15.106:6827/400029804","heartbeat_back_addr":"172.21.15.106:6831/400029804","heartbeat_front_addr":"172.21.15.106:6829/400029804","state":["exists","up"]},{"osd":11,"uuid":"b6b609d9-dd78-4c01-ae66-ba2a662ab22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6825","nonce":4191916230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6827","nonce":4191916230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6831","nonce":4191916230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":4191916230},{"type":"v1","addr":"172.21.15.107:6829","nonce":4191916230}]},"public_addr":"172.21.15.107:6825/4191916230","cluster_addr":"172.21.15.107:6827/4191916230","heartbeat_back_addr":"172.21.15.107:6831/4191916230","heartbeat_front_addr":"172.21.15.107:6829/4191916230","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:13.728602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:13:37.320809+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:01.269452+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:25.758281+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:14:50.298863+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:14.345219+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:15:38.980489+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:04.749214+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:29.992673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:16:56.269202+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:22.270316+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:17:48.074907+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6834/1985874955":"2024-03-25T00:35:20.737219+0000","172.21.15.78:6835/3414536872":"2024-03-25T00:35:04.921316+0000","172.21.15.78:6834/3414536872":"2024-03-25T00:35:04.921316+0000","172.21.15.78:6837/2612396245":"2024-03-25T00:34:49.129252+0000","172.21.15.78:6834/93340378":"2024-03-25T00:34:30.894376+0000","172.21.15.78:6837/878237454":"2024-03-25T00:34:15.858120+0000","172.21.15.78:6836/878237454":"2024-03-25T00:34:15.858120+0000","172.21.15.78:6835/2163967090":"2024-03-25T00:34:00.891452+0000","172.21.15.78:6834/2163967090":"2024-03-25T00:34:00.891452+0000","172.21.15.78:6836/1186751954":"2024-03-25T00:33:45.900090+0000","172.21.15.78:6835/1416685123":"2024-03-25T00:33:30.602155+0000","172.21.15.78:6834/1416685123":"2024-03-25T00:33:30.602155+0000","172.21.15.78:6836/1077781429":"2024-03-25T00:33:14.880383+0000","172.21.15.78:6839/2480119933":"2024-03-25T00:32:59.099199+0000","172.21.15.78:6838/2480119933":"2024-03-25T00:32:59.099199+0000","172.21.15.78:6837/2239323994":"2024-03-25T00:32:40.627803+0000","172.21.15.78:6839/3377749691":"2024-03-25T00:32:25.708147+0000","172.21.15.78:6836/1071274623":"2024-03-25T00:32:10.632135+0000","172.21.15.78:6839/4176145760":"2024-03-25T00:31:55.146631+0000","172.21.15.78:6838/4176145760":"2024-03-25T00:31:55.146631+0000","172.21.15.78:6840/1658072731":"2024-03-25T00:31:39.075579+0000","172.21.15.78:0/442826396":"2024-03-25T00:21:09.220548+0000","172.21.15.107:6832/4067467714":"2024-03-25T00:29:09.638984+0000","172.21.15.78:6837/1077781429":"2024-03-25T00:33:14.880383+0000","172.21.15.107:6839/3131891440":"2024-03-25T00:24:40.372092+0000","172.21.15.107:6838/3949153161":"2024-03-25T00:25:43.971852+0000","172.21.15.107:6833/794068267":"2024-03-25T00:28:05.611074+0000","172.21.15.107:6838/3131891440":"2024-03-25T00:24:40.372092+0000","172.21.15.107:6833/1751539133":"2024-03-25T00:27:35.655923+0000","172.21.15.107:6834/4046882186":"2024-03-25T00:24:25.648083+0000","172.21.15.107:6839/3949153161":"2024-03-25T00:25:43.971852+0000","172.21.15.78:6835/1985874955":"2024-03-25T00:35:20.737219+0000","172.21.15.78:6801/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6841/3424505910":"2024-03-25T00:29:25.370378+0000","172.21.15.78:0/1982202541":"2024-03-25T00:11:40.199569+0000","172.21.15.107:6834/108936439":"2024-03-25T00:23:06.644918+0000","172.21.15.107:6832/794068267":"2024-03-25T00:28:05.611074+0000","172.21.15.78:6838/3377749691":"2024-03-25T00:32:25.708147+0000","172.21.15.107:6839/3531100722":"2024-03-25T00:23:22.383674+0000","172.21.15.107:6838/117119462":"2024-03-25T00:22:50.169074+0000","172.21.15.78:0/2236879401":"2024-03-25T00:11:40.199569+0000","172.21.15.107:6833/3086318443":"2024-03-25T00:27:05.852992+0000","172.21.15.78:6837/1186751954":"2024-03-25T00:33:45.900090+0000","172.21.15.107:6834/23104891":"2024-03-25T00:24:55.644935+0000","172.21.15.78:0/1502752625":"2024-03-25T00:11:14.637881+0000","172.21.15.107:6834/2447512361":"2024-03-25T00:26:33.985150+0000","172.21.15.78:0/359613096":"2024-03-25T00:21:09.220548+0000","172.21.15.78:6800/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6801/1983101026":"2024-03-25T00:21:09.220548+0000","172.21.15.78:0/465691596":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/3962523609":"2024-03-25T00:11:14.637881+0000","172.21.15.78:0/1510740971":"2024-03-25T00:21:09.220548+0000","172.21.15.78:6840/1960340276":"2024-03-25T00:31:02.121119+0000","172.21.15.78:6836/2612396245":"2024-03-25T00:34:49.129252+0000","172.21.15.78:6801/2913847518":"2024-03-25T00:11:14.637881+0000","172.21.15.78:6800/2748920178":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6836/2239323994":"2024-03-25T00:32:40.627803+0000","172.21.15.107:6835/2950120609":"2024-03-25T00:25:25.629098+0000","172.21.15.78:0/1933054339":"2024-03-25T00:11:40.199569+0000","172.21.15.78:6838/4117918862":"2024-03-25T00:31:18.139126+0000","172.21.15.107:6838/607202225":"2024-03-25T00:25:10.629186+0000","172.21.15.78:0/1685151467":"2024-03-25T00:21:09.220548+0000","172.21.15.107:6838/3531100722":"2024-03-25T00:23:22.383674+0000","172.21.15.78:6838/2758357551":"2024-03-25T00:30:15.603375+0000","172.21.15.78:6800/1983101026":"2024-03-25T00:21:09.220548+0000","172.21.15.107:6835/758053763":"2024-03-25T00:25:59.620946+0000","172.21.15.107:6835/4046882186":"2024-03-25T00:24:25.648083+0000","172.21.15.107:6835/108936439":"2024-03-25T00:23:06.644918+0000","172.21.15.106:6833/2367018868":"2024-03-25T00:26:15.443626+0000","172.21.15.78:6835/93340378":"2024-03-25T00:34:30.894376+0000","172.21.15.107:6835/2447512361":"2024-03-25T00:26:33.985150+0000","172.21.15.107:6835/23104891":"2024-03-25T00:24:55.644935+0000","172.21.15.106:6832/2611554095":"2024-03-25T00:26:49.797488+0000","172.21.15.106:6833/2611554095":"2024-03-25T00:26:49.797488+0000","172.21.15.107:6832/3086318443":"2024-03-25T00:27:05.852992+0000","172.21.15.106:6832/2299054161":"2024-03-25T00:27:20.754602+0000","172.21.15.107:6832/1006503957":"2024-03-25T00:29:44.040588+0000","172.21.15.106:6832/2367018868":"2024-03-25T00:26:15.443626+0000","172.21.15.106:6833/2299054161":"2024-03-25T00:27:20.754602+0000","172.21.15.107:6832/1751539133":"2024-03-25T00:27:35.655923+0000","172.21.15.106:6832/459568830":"2024-03-25T00:27:50.719456+0000","172.21.15.106:6833/459568830":"2024-03-25T00:27:50.719456+0000","172.21.15.106:6832/93515314":"2024-03-25T00:28:20.713453+0000","172.21.15.107:6833/3928685127":"2024-03-25T00:28:35.891033+0000","172.21.15.107:6839/117119462":"2024-03-25T00:22:50.169074+0000","172.21.15.106:6832/2104934388":"2024-03-25T00:28:54.025064+0000","172.21.15.78:6837/1071274623":"2024-03-25T00:32:10.632135+0000","172.21.15.106:6833/2104934388":"2024-03-25T00:28:54.025064+0000","172.21.15.78:6841/1658072731":"2024-03-25T00:31:39.075579+0000","172.21.15.107:6833/4067467714":"2024-03-25T00:29:09.638984+0000","172.21.15.78:6840/3424505910":"2024-03-25T00:29:25.370378+0000","172.21.15.107:6834/758053763":"2024-03-25T00:25:59.620946+0000","172.21.15.78:6838/2645579087":"2024-03-25T00:30:46.105435+0000","172.21.15.107:6834/2950120609":"2024-03-25T00:25:25.629098+0000","172.21.15.78:6840/1196331852":"2024-03-25T00:29:59.753104+0000","172.21.15.107:6833/1006503957":"2024-03-25T00:29:44.040588+0000","172.21.15.78:6841/1196331852":"2024-03-25T00:29:59.753104+0000","172.21.15.107:6839/607202225":"2024-03-25T00:25:10.629186+0000","172.21.15.78:6841/1759489401":"2024-03-25T00:30:30.711566+0000","172.21.15.78:6839/2758357551":"2024-03-25T00:30:15.603375+0000","172.21.15.78:6840/1759489401":"2024-03-25T00:30:30.711566+0000","172.21.15.78:6839/2645579087":"2024-03-25T00:30:46.105435+0000","172.21.15.107:6832/3928685127":"2024-03-25T00:28:35.891033+0000","172.21.15.78:6841/1960340276":"2024-03-25T00:31:02.121119+0000","172.21.15.106:6833/93515314":"2024-03-25T00:28:20.713453+0000","172.21.15.78:6839/4117918862":"2024-03-25T00:31:18.139126+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T03:39:48.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.080+0000 7f5f55309640 1 -- 172.21.15.78:0/2046124982 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f5f380a2970 msgr2=0x7f5f380a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:48.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.080+0000 7f5f55309640 1 --2- 172.21.15.78:0/2046124982 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f5f380a2970 0x7f5f380a4e30 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f5f44009900 tx=0x7f5f44005fd0 comp rx=0 tx=0).stop 2024-03-24T03:39:48.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.080+0000 7f5f55309640 1 -- 172.21.15.78:0/2046124982 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f50106670 msgr2=0x7f5f5010e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:48.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.080+0000 7f5f55309640 1 --2- 172.21.15.78:0/2046124982 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f50106670 0x7f5f5010e9b0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f5f4800eb10 tx=0x7f5f4800c2d0 comp rx=0 tx=0).stop 2024-03-24T03:39:48.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.080+0000 7f5f55309640 1 -- 172.21.15.78:0/2046124982 shutdown_connections 2024-03-24T03:39:48.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.080+0000 7f5f55309640 1 --2- 172.21.15.78:0/2046124982 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f5f380a2970 0x7f5f380a4e30 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:48.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.080+0000 7f5f55309640 1 --2- 172.21.15.78:0/2046124982 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f50106670 0x7f5f5010e9b0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:48.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.080+0000 7f5f55309640 1 --2- 172.21.15.78:0/2046124982 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f50100fe0 0x7f5f5010e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:48.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.080+0000 7f5f55309640 1 --2- 172.21.15.78:0/2046124982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f50100620 0x7f5f5010df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:48.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.080+0000 7f5f55309640 1 -- 172.21.15.78:0/2046124982 >> 172.21.15.78:0/2046124982 conn(0x7f5f50077700 msgr2=0x7f5f500ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:39:48.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.080+0000 7f5f55309640 1 -- 172.21.15.78:0/2046124982 shutdown_connections 2024-03-24T03:39:48.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.080+0000 7f5f55309640 1 -- 172.21.15.78:0/2046124982 wait complete. 2024-03-24T03:39:48.092 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T03:39:48.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:47 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1057865184' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T03:39:48.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:47 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1057865184' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T03:39:48.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.194+0000 7f43da5e8640 1 -- 172.21.15.78:0/2050016679 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43d41066c0 msgr2=0x7f43d4106aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:48.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.194+0000 7f43da5e8640 1 --2- 172.21.15.78:0/2050016679 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43d41066c0 0x7f43d4106aa0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f43c4009930 tx=0x7f43c402bc30 comp rx=0 tx=0).stop 2024-03-24T03:39:48.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.194+0000 7f43da5e8640 1 -- 172.21.15.78:0/2050016679 shutdown_connections 2024-03-24T03:39:48.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.194+0000 7f43da5e8640 1 --2- 172.21.15.78:0/2050016679 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43d4101670 0x7f43d410d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:48.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.194+0000 7f43da5e8640 1 --2- 172.21.15.78:0/2050016679 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43d4100cb0 0x7f43d4101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:48.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.194+0000 7f43da5e8640 1 --2- 172.21.15.78:0/2050016679 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43d41066c0 0x7f43d4106aa0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:48.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.194+0000 7f43da5e8640 1 -- 172.21.15.78:0/2050016679 >> 172.21.15.78:0/2050016679 conn(0x7f43d4076b60 msgr2=0x7f43d40fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:39:48.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.194+0000 7f43da5e8640 1 -- 172.21.15.78:0/2050016679 shutdown_connections 2024-03-24T03:39:48.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.194+0000 7f43da5e8640 1 -- 172.21.15.78:0/2050016679 wait complete. 2024-03-24T03:39:48.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.195+0000 7f43da5e8640 1 Processor -- start 2024-03-24T03:39:48.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.195+0000 7f43da5e8640 1 -- start start 2024-03-24T03:39:48.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.195+0000 7f43da5e8640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43d4100cb0 0x7f43d4196e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:39:48.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.195+0000 7f43da5e8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43d4101670 0x7f43d4197370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:39:48.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.195+0000 7f43da5e8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43d41066c0 0x7f43d419b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:39:48.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.195+0000 7f43da5e8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d4110950 con 0x7f43d41066c0 2024-03-24T03:39:48.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.195+0000 7f43da5e8640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d41107d0 con 0x7f43d4100cb0 2024-03-24T03:39:48.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.195+0000 7f43da5e8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d4110ad0 con 0x7f43d4101670 2024-03-24T03:39:48.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.195+0000 7f43d8b85640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43d41066c0 0x7f43d419b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:39:48.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.195+0000 7f43d8b85640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43d41066c0 0x7f43d419b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58792/0 (socket says 172.21.15.78:58792) 2024-03-24T03:39:48.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.195+0000 7f43d8b85640 1 -- 172.21.15.78:0/1422734752 learned_addr learned my addr 172.21.15.78:0/1422734752 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T03:39:48.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.195+0000 7f43d3fff640 1 --2- 172.21.15.78:0/1422734752 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43d4100cb0 0x7f43d4196e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:39:48.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.195+0000 7f43d37fe640 1 --2- 172.21.15.78:0/1422734752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43d4101670 0x7f43d4197370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:39:48.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.196+0000 7f43d8b85640 1 -- 172.21.15.78:0/1422734752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43d4101670 msgr2=0x7f43d4197370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:48.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.196+0000 7f43d8b85640 1 --2- 172.21.15.78:0/1422734752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43d4101670 0x7f43d4197370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:48.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.196+0000 7f43d8b85640 1 -- 172.21.15.78:0/1422734752 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43d4100cb0 msgr2=0x7f43d4196e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:48.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.196+0000 7f43d8b85640 1 --2- 172.21.15.78:0/1422734752 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43d4100cb0 0x7f43d4196e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:48.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.196+0000 7f43d8b85640 1 -- 172.21.15.78:0/1422734752 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d419be20 con 0x7f43d41066c0 2024-03-24T03:39:48.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.196+0000 7f43d8b85640 1 --2- 172.21.15.78:0/1422734752 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43d41066c0 0x7f43d419b6f0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f43cc00cad0 tx=0x7f43cc00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:39:48.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.196+0000 7f43d17fa640 1 -- 172.21.15.78:0/1422734752 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43cc00ae20 con 0x7f43d41066c0 2024-03-24T03:39:48.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.196+0000 7f43d17fa640 1 -- 172.21.15.78:0/1422734752 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f43cc005020 con 0x7f43d41066c0 2024-03-24T03:39:48.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.196+0000 7f43da5e8640 1 -- 172.21.15.78:0/1422734752 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43d419c110 con 0x7f43d41066c0 2024-03-24T03:39:48.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.196+0000 7f43d17fa640 1 -- 172.21.15.78:0/1422734752 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43cc00f6f0 con 0x7f43d41066c0 2024-03-24T03:39:48.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.196+0000 7f43da5e8640 1 -- 172.21.15.78:0/1422734752 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43d41a39a0 con 0x7f43d41066c0 2024-03-24T03:39:48.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.197+0000 7f43da5e8640 1 -- 172.21.15.78:0/1422734752 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43d40688e0 con 0x7f43d41066c0 2024-03-24T03:39:48.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.198+0000 7f43d17fa640 1 -- 172.21.15.78:0/1422734752 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f43cc0051d0 con 0x7f43d41066c0 2024-03-24T03:39:48.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.198+0000 7f43d17fa640 1 --2- 172.21.15.78:0/1422734752 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f43bc0a2970 0x7f43bc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:39:48.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.199+0000 7f43d3fff640 1 --2- 172.21.15.78:0/1422734752 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f43bc0a2970 0x7f43bc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:39:48.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.199+0000 7f43d17fa640 1 -- 172.21.15.78:0/1422734752 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 13278+0+0 (secure 0 0 0) 0x7f43cc0d0170 con 0x7f43d41066c0 2024-03-24T03:39:48.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.199+0000 7f43d3fff640 1 --2- 172.21.15.78:0/1422734752 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f43bc0a2970 0x7f43bc0a4e30 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f43c402efd0 tx=0x7f43c4000bc0 comp rx=0 tx=0).ready entity=mgr.25243 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:39:48.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.203+0000 7f43d17fa640 1 -- 172.21.15.78:0/1422734752 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f43cc014030 con 0x7f43d41066c0 2024-03-24T03:39:48.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.418+0000 7f43da5e8640 1 -- 172.21.15.78:0/1422734752 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f43d4108550 con 0x7f43d41066c0 2024-03-24T03:39:48.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.422+0000 7f43d17fa640 1 -- 172.21.15.78:0/1422734752 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v308) v1 ==== 161+0+0 (secure 0 0 0) 0x7f43cc09bc40 con 0x7f43d41066c0 2024-03-24T03:39:48.422 INFO:teuthology.orchestra.run.smithi078.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T03:39:48.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.425+0000 7f43da5e8640 1 -- 172.21.15.78:0/1422734752 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f43bc0a2970 msgr2=0x7f43bc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:48.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.425+0000 7f43da5e8640 1 --2- 172.21.15.78:0/1422734752 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f43bc0a2970 0x7f43bc0a4e30 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f43c402efd0 tx=0x7f43c4000bc0 comp rx=0 tx=0).stop 2024-03-24T03:39:48.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.425+0000 7f43da5e8640 1 -- 172.21.15.78:0/1422734752 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43d41066c0 msgr2=0x7f43d419b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:48.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.425+0000 7f43da5e8640 1 --2- 172.21.15.78:0/1422734752 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43d41066c0 0x7f43d419b6f0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f43cc00cad0 tx=0x7f43cc00cf90 comp rx=0 tx=0).stop 2024-03-24T03:39:48.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.425+0000 7f43da5e8640 1 -- 172.21.15.78:0/1422734752 shutdown_connections 2024-03-24T03:39:48.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.425+0000 7f43da5e8640 1 --2- 172.21.15.78:0/1422734752 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f43bc0a2970 0x7f43bc0a4e30 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:48.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.425+0000 7f43da5e8640 1 --2- 172.21.15.78:0/1422734752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43d4101670 0x7f43d4197370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:48.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.425+0000 7f43da5e8640 1 --2- 172.21.15.78:0/1422734752 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43d41066c0 0x7f43d419b6f0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:48.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.425+0000 7f43da5e8640 1 --2- 172.21.15.78:0/1422734752 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43d4100cb0 0x7f43d4196e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:48.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.425+0000 7f43da5e8640 1 -- 172.21.15.78:0/1422734752 >> 172.21.15.78:0/1422734752 conn(0x7f43d4076b60 msgr2=0x7f43d40fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:39:48.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.425+0000 7f43da5e8640 1 -- 172.21.15.78:0/1422734752 shutdown_connections 2024-03-24T03:39:48.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.425+0000 7f43da5e8640 1 -- 172.21.15.78:0/1422734752 wait complete. 2024-03-24T03:39:48.436 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-24T03:39:48.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.540+0000 7f1d7359d640 1 -- 172.21.15.78:0/3389598191 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d6c107640 msgr2=0x7f1d6c107a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:48.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.540+0000 7f1d7359d640 1 --2- 172.21.15.78:0/3389598191 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d6c107640 0x7f1d6c107a20 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c009930 tx=0x7f1d5c02bc30 comp rx=0 tx=0).stop 2024-03-24T03:39:48.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.540+0000 7f1d7359d640 1 -- 172.21.15.78:0/3389598191 shutdown_connections 2024-03-24T03:39:48.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.540+0000 7f1d7359d640 1 --2- 172.21.15.78:0/3389598191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1d6c101ff0 0x7f1d6c10e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:48.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.540+0000 7f1d7359d640 1 --2- 172.21.15.78:0/3389598191 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d6c101630 0x7f1d6c101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:48.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.540+0000 7f1d7359d640 1 --2- 172.21.15.78:0/3389598191 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d6c107640 0x7f1d6c107a20 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:48.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.540+0000 7f1d7359d640 1 -- 172.21.15.78:0/3389598191 >> 172.21.15.78:0/3389598191 conn(0x7f1d6c078aa0 msgr2=0x7f1d6c078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:39:48.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.540+0000 7f1d7359d640 1 -- 172.21.15.78:0/3389598191 shutdown_connections 2024-03-24T03:39:48.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.540+0000 7f1d7359d640 1 -- 172.21.15.78:0/3389598191 wait complete. 2024-03-24T03:39:48.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.541+0000 7f1d7359d640 1 Processor -- start 2024-03-24T03:39:48.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.541+0000 7f1d7359d640 1 -- start start 2024-03-24T03:39:48.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.541+0000 7f1d7359d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1d6c101630 0x7f1d6c197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:39:48.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.541+0000 7f1d7359d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d6c101ff0 0x7f1d6c198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:39:48.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.541+0000 7f1d7359d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d6c107640 0x7f1d6c19c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:39:48.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.541+0000 7f1d7359d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6c111a60 con 0x7f1d6c101ff0 2024-03-24T03:39:48.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.541+0000 7f1d7359d640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6c1118e0 con 0x7f1d6c107640 2024-03-24T03:39:48.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.541+0000 7f1d7359d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6c111be0 con 0x7f1d6c101630 2024-03-24T03:39:48.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.541+0000 7f1d70b38640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d6c101ff0 0x7f1d6c198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:39:48.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.541+0000 7f1d70b38640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d6c101ff0 0x7f1d6c198480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58808/0 (socket says 172.21.15.78:58808) 2024-03-24T03:39:48.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.541+0000 7f1d70b38640 1 -- 172.21.15.78:0/4102978462 learned_addr learned my addr 172.21.15.78:0/4102978462 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T03:39:48.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.541+0000 7f1d71b3a640 1 --2- 172.21.15.78:0/4102978462 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d6c107640 0x7f1d6c19c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:39:48.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.541+0000 7f1d70b38640 1 -- 172.21.15.78:0/4102978462 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1d6c101630 msgr2=0x7f1d6c197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:48.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.541+0000 7f1d71339640 1 --2- 172.21.15.78:0/4102978462 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1d6c101630 0x7f1d6c197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:39:48.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.541+0000 7f1d70b38640 1 --2- 172.21.15.78:0/4102978462 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1d6c101630 0x7f1d6c197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:48.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.541+0000 7f1d70b38640 1 -- 172.21.15.78:0/4102978462 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d6c107640 msgr2=0x7f1d6c19c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:48.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.541+0000 7f1d70b38640 1 --2- 172.21.15.78:0/4102978462 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d6c107640 0x7f1d6c19c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:48.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.541+0000 7f1d70b38640 1 -- 172.21.15.78:0/4102978462 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d6c19cf30 con 0x7f1d6c101ff0 2024-03-24T03:39:48.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.541+0000 7f1d71b3a640 1 --2- 172.21.15.78:0/4102978462 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d6c107640 0x7f1d6c19c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T03:39:48.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.541+0000 7f1d70b38640 1 --2- 172.21.15.78:0/4102978462 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d6c101ff0 0x7f1d6c198480 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f1d6800d950 tx=0x7f1d6800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:39:48.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.541+0000 7f1d627fc640 1 -- 172.21.15.78:0/4102978462 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d68017030 con 0x7f1d6c101ff0 2024-03-24T03:39:48.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.541+0000 7f1d627fc640 1 -- 172.21.15.78:0/4102978462 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1d68004020 con 0x7f1d6c101ff0 2024-03-24T03:39:48.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.542+0000 7f1d7359d640 1 -- 172.21.15.78:0/4102978462 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d6c19d220 con 0x7f1d6c101ff0 2024-03-24T03:39:48.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.542+0000 7f1d627fc640 1 -- 172.21.15.78:0/4102978462 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d680047a0 con 0x7f1d6c101ff0 2024-03-24T03:39:48.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.542+0000 7f1d7359d640 1 -- 172.21.15.78:0/4102978462 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d6c1a4ab0 con 0x7f1d6c101ff0 2024-03-24T03:39:48.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.542+0000 7f1d7359d640 1 -- 172.21.15.78:0/4102978462 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d6c0688e0 con 0x7f1d6c101ff0 2024-03-24T03:39:48.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.544+0000 7f1d627fc640 1 -- 172.21.15.78:0/4102978462 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f1d680041d0 con 0x7f1d6c101ff0 2024-03-24T03:39:48.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.544+0000 7f1d627fc640 1 --2- 172.21.15.78:0/4102978462 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f1d580a2970 0x7f1d580a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:39:48.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.544+0000 7f1d71339640 1 --2- 172.21.15.78:0/4102978462 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f1d580a2970 0x7f1d580a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:39:48.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.544+0000 7f1d627fc640 1 -- 172.21.15.78:0/4102978462 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 13536+0+0 (secure 0 0 0) 0x7f1d680cfc70 con 0x7f1d6c101ff0 2024-03-24T03:39:48.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.545+0000 7f1d71339640 1 --2- 172.21.15.78:0/4102978462 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f1d580a2970 0x7f1d580a4e30 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c036040 tx=0x7f1d5c02df90 comp rx=0 tx=0).ready entity=mgr.25243 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:39:48.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.549+0000 7f1d627fc640 1 -- 172.21.15.78:0/4102978462 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1d6809b640 con 0x7f1d6c101ff0 2024-03-24T03:39:48.688 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mon-a[32425]: 2024-03-24T03:39:48.420+0000 7f94c8b27700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T03:39:48.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:48.766+0000 7f1d7359d640 1 -- 172.21.15.78:0/4102978462 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f1d6c02d070 con 0x7f1d6c101ff0 2024-03-24T03:39:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:48 smithi107 ceph-mon[36485]: pgmap v6307: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:39:48.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:48 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1637101674' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T03:39:48.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:48 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2046124982' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T03:39:48.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:48 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1422734752' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-24T03:39:48.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:48 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T03:39:48.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:48 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem has a failed mds daemon (FS_WITH_FAILED_MDS) 2024-03-24T03:39:48.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:48 smithi107 ceph-mon[36485]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T03:39:48.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:48 smithi107 ceph-mon[36485]: Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2024-03-24T03:39:48.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:48 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1422734752' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-24T03:39:48.844 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:48 smithi107 ceph-mon[36485]: osdmap e158: 12 total, 12 up, 12 in 2024-03-24T03:39:48.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:48 smithi107 ceph-mon[36485]: fsmap cephfs:0/3, 3 failed 2024-03-24T03:39:48.845 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 24 03:39:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-c[67198]: ignoring --setuser ceph since I am not root 2024-03-24T03:39:48.845 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 24 03:39:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-c[67198]: ignoring --setgroup ceph since I am not root 2024-03-24T03:39:48.845 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 24 03:39:48 smithi107 ceph-mds[67204]: -- 172.21.15.107:0/27391609 <== mon.1 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5631edb4ba40 con 0x5631edb1f000 2024-03-24T03:39:48.845 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 24 03:39:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-c[67198]: starting mds.c at 2024-03-24T03:39:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:48 smithi106 ceph-mon[36505]: pgmap v6307: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:39:48.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:48 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1637101674' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T03:39:48.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:48 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2046124982' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T03:39:48.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:48 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1422734752' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-24T03:39:48.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:48 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T03:39:48.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:48 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem has a failed mds daemon (FS_WITH_FAILED_MDS) 2024-03-24T03:39:48.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:48 smithi106 ceph-mon[36505]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T03:39:48.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:48 smithi106 ceph-mon[36505]: Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2024-03-24T03:39:48.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:48 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1422734752' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-24T03:39:48.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:48 smithi106 ceph-mon[36505]: osdmap e158: 12 total, 12 up, 12 in 2024-03-24T03:39:48.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:48 smithi106 ceph-mon[36505]: fsmap cephfs:0/3, 3 failed 2024-03-24T03:39:48.904 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 24 03:39:48 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-e[66368]: ignoring --setuser ceph since I am not root 2024-03-24T03:39:48.904 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 24 03:39:48 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-e[66368]: ignoring --setgroup ceph since I am not root 2024-03-24T03:39:48.904 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 24 03:39:48 smithi106 ceph-mds[66372]: -- 172.21.15.106:0/3855076583 <== mon.2 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5652e66a9a40 con 0x5652e667cc00 2024-03-24T03:39:48.904 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 24 03:39:48 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-e[66368]: starting mds.e at 2024-03-24T03:39:48.905 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 24 03:39:48 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-b[66985]: ignoring --setuser ceph since I am not root 2024-03-24T03:39:48.905 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 24 03:39:48 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-b[66985]: ignoring --setgroup ceph since I am not root 2024-03-24T03:39:48.905 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 24 03:39:48 smithi106 ceph-mds[66989]: -- 172.21.15.106:0/1917832884 <== mon.2 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f737ef1a40 con 0x55f737ec5000 2024-03-24T03:39:48.905 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 24 03:39:48 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-b[66985]: starting mds.b at 2024-03-24T03:39:48.906 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 24 03:39:48 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-h[66675]: ignoring --setuser ceph since I am not root 2024-03-24T03:39:48.906 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 24 03:39:48 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-h[66675]: ignoring --setgroup ceph since I am not root 2024-03-24T03:39:48.906 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 24 03:39:48 smithi106 ceph-mds[66679]: -- 172.21.15.106:0/2244277816 <== mon.2 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5636ce483a40 con 0x5636ce456800 2024-03-24T03:39:48.906 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 24 03:39:48 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-h[66675]: starting mds.h at 2024-03-24T03:39:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:48 smithi078 ceph-mon[32429]: pgmap v6307: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-24T03:39:49.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:48 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1637101674' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T03:39:49.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:48 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2046124982' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T03:39:49.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:48 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1422734752' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-24T03:39:49.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:48 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T03:39:49.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:48 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem has a failed mds daemon (FS_WITH_FAILED_MDS) 2024-03-24T03:39:49.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:48 smithi078 ceph-mon[32429]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T03:39:49.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:48 smithi078 ceph-mon[32429]: Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2024-03-24T03:39:49.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:48 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1422734752' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-24T03:39:49.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:48 smithi078 ceph-mon[32429]: osdmap e158: 12 total, 12 up, 12 in 2024-03-24T03:39:49.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:48 smithi078 ceph-mon[32429]: fsmap cephfs:0/3, 3 failed 2024-03-24T03:39:49.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.423+0000 7f1d627fc640 1 -- 172.21.15.78:0/4102978462 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v309) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1d6809af40 con 0x7f1d6c101ff0 2024-03-24T03:39:49.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.426+0000 7f1d7359d640 1 -- 172.21.15.78:0/4102978462 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f1d580a2970 msgr2=0x7f1d580a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:49.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.426+0000 7f1d7359d640 1 --2- 172.21.15.78:0/4102978462 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f1d580a2970 0x7f1d580a4e30 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c036040 tx=0x7f1d5c02df90 comp rx=0 tx=0).stop 2024-03-24T03:39:49.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.426+0000 7f1d7359d640 1 -- 172.21.15.78:0/4102978462 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d6c101ff0 msgr2=0x7f1d6c198480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:49.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.426+0000 7f1d7359d640 1 --2- 172.21.15.78:0/4102978462 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d6c101ff0 0x7f1d6c198480 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f1d6800d950 tx=0x7f1d6800de10 comp rx=0 tx=0).stop 2024-03-24T03:39:49.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.426+0000 7f1d7359d640 1 -- 172.21.15.78:0/4102978462 shutdown_connections 2024-03-24T03:39:49.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.426+0000 7f1d7359d640 1 --2- 172.21.15.78:0/4102978462 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f1d580a2970 0x7f1d580a4e30 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:49.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.426+0000 7f1d7359d640 1 --2- 172.21.15.78:0/4102978462 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d6c107640 0x7f1d6c19c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:49.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.426+0000 7f1d7359d640 1 --2- 172.21.15.78:0/4102978462 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d6c101ff0 0x7f1d6c198480 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:49.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.426+0000 7f1d7359d640 1 --2- 172.21.15.78:0/4102978462 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1d6c101630 0x7f1d6c197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:49.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.426+0000 7f1d7359d640 1 -- 172.21.15.78:0/4102978462 >> 172.21.15.78:0/4102978462 conn(0x7f1d6c078aa0 msgr2=0x7f1d6c103fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:39:49.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.427+0000 7f1d7359d640 1 -- 172.21.15.78:0/4102978462 shutdown_connections 2024-03-24T03:39:49.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.427+0000 7f1d7359d640 1 -- 172.21.15.78:0/4102978462 wait complete. 2024-03-24T03:39:49.441 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-24T03:39:49.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.552+0000 7f4d25d10640 1 -- 172.21.15.78:0/1772664962 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d2006a750 msgr2=0x7f4d2010e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:49.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.552+0000 7f4d25d10640 1 --2- 172.21.15.78:0/1772664962 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d2006a750 0x7f4d2010e510 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f4d18009cd0 tx=0x7f4d1802c0f0 comp rx=0 tx=0).stop 2024-03-24T03:39:49.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.552+0000 7f4d25d10640 1 -- 172.21.15.78:0/1772664962 shutdown_connections 2024-03-24T03:39:49.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.552+0000 7f4d25d10640 1 --2- 172.21.15.78:0/1772664962 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d2006a750 0x7f4d2010e510 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:49.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.552+0000 7f4d25d10640 1 --2- 172.21.15.78:0/1772664962 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d20069d90 0x7f4d2006a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:49.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.552+0000 7f4d25d10640 1 --2- 172.21.15.78:0/1772664962 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d20107700 0x7f4d20107ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:49.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.552+0000 7f4d25d10640 1 -- 172.21.15.78:0/1772664962 >> 172.21.15.78:0/1772664962 conn(0x7f4d20076920 msgr2=0x7f4d20076d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:39:49.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.552+0000 7f4d25d10640 1 -- 172.21.15.78:0/1772664962 shutdown_connections 2024-03-24T03:39:49.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.552+0000 7f4d25d10640 1 -- 172.21.15.78:0/1772664962 wait complete. 2024-03-24T03:39:49.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.553+0000 7f4d25d10640 1 Processor -- start 2024-03-24T03:39:49.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.553+0000 7f4d25d10640 1 -- start start 2024-03-24T03:39:49.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.553+0000 7f4d25d10640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d20069d90 0x7f4d20197df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:39:49.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.553+0000 7f4d25d10640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d2006a750 0x7f4d20198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:39:49.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.553+0000 7f4d25d10640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d20107700 0x7f4d2019c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:39:49.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.553+0000 7f4d25d10640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d20111a10 con 0x7f4d20107700 2024-03-24T03:39:49.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.553+0000 7f4d25d10640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d20111890 con 0x7f4d2006a750 2024-03-24T03:39:49.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.553+0000 7f4d25d10640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d20111b90 con 0x7f4d20069d90 2024-03-24T03:39:49.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.553+0000 7f4d1ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d20107700 0x7f4d2019c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:39:49.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.553+0000 7f4d1ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d20107700 0x7f4d2019c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58840/0 (socket says 172.21.15.78:58840) 2024-03-24T03:39:49.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.553+0000 7f4d1ffff640 1 -- 172.21.15.78:0/525372620 learned_addr learned my addr 172.21.15.78:0/525372620 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T03:39:49.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.553+0000 7f4d1effd640 1 --2- 172.21.15.78:0/525372620 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d2006a750 0x7f4d20198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:39:49.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.553+0000 7f4d1f7fe640 1 --2- 172.21.15.78:0/525372620 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d20069d90 0x7f4d20197df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:39:49.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.553+0000 7f4d1ffff640 1 -- 172.21.15.78:0/525372620 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d20069d90 msgr2=0x7f4d20197df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:49.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.553+0000 7f4d1ffff640 1 --2- 172.21.15.78:0/525372620 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d20069d90 0x7f4d20197df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:49.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.553+0000 7f4d1ffff640 1 -- 172.21.15.78:0/525372620 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d2006a750 msgr2=0x7f4d20198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:49.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.553+0000 7f4d1ffff640 1 --2- 172.21.15.78:0/525372620 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d2006a750 0x7f4d20198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:49.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.553+0000 7f4d1ffff640 1 -- 172.21.15.78:0/525372620 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d2019cd90 con 0x7f4d20107700 2024-03-24T03:39:49.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.554+0000 7f4d1ffff640 1 --2- 172.21.15.78:0/525372620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d20107700 0x7f4d2019c660 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f4d18004af0 tx=0x7f4d18032ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:39:49.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.554+0000 7f4d1cff9640 1 -- 172.21.15.78:0/525372620 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d18042020 con 0x7f4d20107700 2024-03-24T03:39:49.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.554+0000 7f4d1cff9640 1 -- 172.21.15.78:0/525372620 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4d18034640 con 0x7f4d20107700 2024-03-24T03:39:49.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.554+0000 7f4d25d10640 1 -- 172.21.15.78:0/525372620 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d2019d020 con 0x7f4d20107700 2024-03-24T03:39:49.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.554+0000 7f4d1cff9640 1 -- 172.21.15.78:0/525372620 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d18039040 con 0x7f4d20107700 2024-03-24T03:39:49.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.554+0000 7f4d25d10640 1 -- 172.21.15.78:0/525372620 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d201a4910 con 0x7f4d20107700 2024-03-24T03:39:49.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.555+0000 7f4d25d10640 1 -- 172.21.15.78:0/525372620 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d200688e0 con 0x7f4d20107700 2024-03-24T03:39:49.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.556+0000 7f4d1cff9640 1 -- 172.21.15.78:0/525372620 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f4d18045020 con 0x7f4d20107700 2024-03-24T03:39:49.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.556+0000 7f4d1cff9640 1 --2- 172.21.15.78:0/525372620 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f4d080a2970 0x7f4d080a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:39:49.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.557+0000 7f4d1f7fe640 1 --2- 172.21.15.78:0/525372620 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f4d080a2970 0x7f4d080a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:39:49.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.557+0000 7f4d1cff9640 1 -- 172.21.15.78:0/525372620 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 13536+0+0 (secure 0 0 0) 0x7f4d180f21e0 con 0x7f4d20107700 2024-03-24T03:39:49.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.557+0000 7f4d1f7fe640 1 --2- 172.21.15.78:0/525372620 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f4d080a2970 0x7f4d080a4e30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4d1000ab60 tx=0x7f4d1000a2d0 comp rx=0 tx=0).ready entity=mgr.25243 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:39:49.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.562+0000 7f4d1cff9640 1 -- 172.21.15.78:0/525372620 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4d1804c020 con 0x7f4d20107700 2024-03-24T03:39:49.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:49.740+0000 7f4d25d10640 1 -- 172.21.15.78:0/525372620 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4d201a4ae0 con 0x7f4d20107700 2024-03-24T03:39:50.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:49 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/4102978462' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T03:39:50.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:49 smithi078 ceph-mon[32429]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T03:39:50.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:49 smithi078 ceph-mon[32429]: Health check cleared: FS_WITH_FAILED_MDS (was: 1 filesystem has a failed mds daemon) 2024-03-24T03:39:50.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:49 smithi078 ceph-mon[32429]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T03:39:50.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:49 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/4102978462' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-24T03:39:50.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:49 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.107:6832/4268837038,v1:172.21.15.107:6833/4268837038] up:boot 2024-03-24T03:39:50.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:49 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6832/1975181730,v1:172.21.15.106:6833/1975181730] up:boot 2024-03-24T03:39:50.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:49 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6834/4218160011,v1:172.21.15.106:6835/4218160011] up:boot 2024-03-24T03:39:50.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:49 smithi078 ceph-mon[32429]: mds.? [v2:172.21.15.106:6836/1316362992,v1:172.21.15.106:6837/1316362992] up:boot 2024-03-24T03:39:50.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:49 smithi078 ceph-mon[32429]: fsmap 4 up:standby 2024-03-24T03:39:50.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:49 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T03:39:50.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:49 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T03:39:50.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:49 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T03:39:50.093 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:49 smithi078 ceph-mon[32429]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T03:39:50.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:49 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/4102978462' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T03:39:50.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:49 smithi107 ceph-mon[36485]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T03:39:50.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:49 smithi107 ceph-mon[36485]: Health check cleared: FS_WITH_FAILED_MDS (was: 1 filesystem has a failed mds daemon) 2024-03-24T03:39:50.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:49 smithi107 ceph-mon[36485]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T03:39:50.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:49 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/4102978462' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-24T03:39:50.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:49 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.107:6832/4268837038,v1:172.21.15.107:6833/4268837038] up:boot 2024-03-24T03:39:50.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:49 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6832/1975181730,v1:172.21.15.106:6833/1975181730] up:boot 2024-03-24T03:39:50.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:49 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6834/4218160011,v1:172.21.15.106:6835/4218160011] up:boot 2024-03-24T03:39:50.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:49 smithi107 ceph-mon[36485]: mds.? [v2:172.21.15.106:6836/1316362992,v1:172.21.15.106:6837/1316362992] up:boot 2024-03-24T03:39:50.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:49 smithi107 ceph-mon[36485]: fsmap 4 up:standby 2024-03-24T03:39:50.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:49 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T03:39:50.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:49 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T03:39:50.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:49 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T03:39:50.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:49 smithi107 ceph-mon[36485]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T03:39:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:49 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/4102978462' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T03:39:50.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:49 smithi106 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T03:39:50.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:49 smithi106 ceph-mon[36505]: Health check cleared: FS_WITH_FAILED_MDS (was: 1 filesystem has a failed mds daemon) 2024-03-24T03:39:50.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:49 smithi106 ceph-mon[36505]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T03:39:50.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:49 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/4102978462' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-24T03:39:50.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:49 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.107:6832/4268837038,v1:172.21.15.107:6833/4268837038] up:boot 2024-03-24T03:39:50.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:49 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6832/1975181730,v1:172.21.15.106:6833/1975181730] up:boot 2024-03-24T03:39:50.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:49 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6834/4218160011,v1:172.21.15.106:6835/4218160011] up:boot 2024-03-24T03:39:50.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:49 smithi106 ceph-mon[36505]: mds.? [v2:172.21.15.106:6836/1316362992,v1:172.21.15.106:6837/1316362992] up:boot 2024-03-24T03:39:50.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:49 smithi106 ceph-mon[36505]: fsmap 4 up:standby 2024-03-24T03:39:50.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:49 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T03:39:50.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:49 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T03:39:50.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:49 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T03:39:50.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:49 smithi106 ceph-mon[36505]: from='mgr.25243 172.21.15.78:0/3009555442' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T03:39:50.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.700+0000 7f4d1cff9640 1 -- 172.21.15.78:0/525372620 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v159) v1 ==== 181+0+0 (secure 0 0 0) 0x7f4d180bdcc0 con 0x7f4d20107700 2024-03-24T03:39:50.700 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' removed 2024-03-24T03:39:50.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.703+0000 7f4d25d10640 1 -- 172.21.15.78:0/525372620 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f4d080a2970 msgr2=0x7f4d080a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:50.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.703+0000 7f4d25d10640 1 --2- 172.21.15.78:0/525372620 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f4d080a2970 0x7f4d080a4e30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4d1000ab60 tx=0x7f4d1000a2d0 comp rx=0 tx=0).stop 2024-03-24T03:39:50.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.703+0000 7f4d25d10640 1 -- 172.21.15.78:0/525372620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d20107700 msgr2=0x7f4d2019c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:50.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.703+0000 7f4d25d10640 1 --2- 172.21.15.78:0/525372620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d20107700 0x7f4d2019c660 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f4d18004af0 tx=0x7f4d18032ec0 comp rx=0 tx=0).stop 2024-03-24T03:39:50.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.703+0000 7f4d25d10640 1 -- 172.21.15.78:0/525372620 shutdown_connections 2024-03-24T03:39:50.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.703+0000 7f4d25d10640 1 --2- 172.21.15.78:0/525372620 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f4d080a2970 0x7f4d080a4e30 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:50.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.703+0000 7f4d25d10640 1 --2- 172.21.15.78:0/525372620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d20107700 0x7f4d2019c660 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:50.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.703+0000 7f4d25d10640 1 --2- 172.21.15.78:0/525372620 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d2006a750 0x7f4d20198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:50.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.703+0000 7f4d25d10640 1 --2- 172.21.15.78:0/525372620 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d20069d90 0x7f4d20197df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:50.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.703+0000 7f4d25d10640 1 -- 172.21.15.78:0/525372620 >> 172.21.15.78:0/525372620 conn(0x7f4d20076920 msgr2=0x7f4d200fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:39:50.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.703+0000 7f4d25d10640 1 -- 172.21.15.78:0/525372620 shutdown_connections 2024-03-24T03:39:50.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.703+0000 7f4d25d10640 1 -- 172.21.15.78:0/525372620 wait complete. 2024-03-24T03:39:50.720 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-24T03:39:50.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.831+0000 7f523b305640 1 -- 172.21.15.78:0/1675806055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5234102910 msgr2=0x7f5234102cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:50.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.831+0000 7f523b305640 1 --2- 172.21.15.78:0/1675806055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5234102910 0x7f5234102cf0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f5230009900 tx=0x7f523002bcd0 comp rx=0 tx=0).stop 2024-03-24T03:39:50.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.831+0000 7f523b305640 1 -- 172.21.15.78:0/1675806055 shutdown_connections 2024-03-24T03:39:50.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.832+0000 7f523b305640 1 --2- 172.21.15.78:0/1675806055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5234103230 0x7f523410b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:50.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.832+0000 7f523b305640 1 --2- 172.21.15.78:0/1675806055 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f523410bce0 0x7f523410e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:50.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.832+0000 7f523b305640 1 --2- 172.21.15.78:0/1675806055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5234102910 0x7f5234102cf0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:50.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.832+0000 7f523b305640 1 -- 172.21.15.78:0/1675806055 >> 172.21.15.78:0/1675806055 conn(0x7f5234077700 msgr2=0x7f5234077b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:39:50.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.832+0000 7f523b305640 1 -- 172.21.15.78:0/1675806055 shutdown_connections 2024-03-24T03:39:50.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.832+0000 7f523b305640 1 -- 172.21.15.78:0/1675806055 wait complete. 2024-03-24T03:39:50.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.832+0000 7f523b305640 1 Processor -- start 2024-03-24T03:39:50.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.832+0000 7f523b305640 1 -- start start 2024-03-24T03:39:50.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.832+0000 7f523b305640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5234102910 0x7f5234197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:39:50.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.832+0000 7f523b305640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5234103230 0x7f5234198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:39:50.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.832+0000 7f523b305640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f523410bce0 0x7f523419c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:39:50.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.832+0000 7f523b305640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5234111af0 con 0x7f5234103230 2024-03-24T03:39:50.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.832+0000 7f523b305640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5234111970 con 0x7f5234102910 2024-03-24T03:39:50.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.832+0000 7f523b305640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5234111c70 con 0x7f523410bce0 2024-03-24T03:39:50.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.832+0000 7f52388a0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5234103230 0x7f5234198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:39:50.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.833+0000 7f52388a0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5234103230 0x7f5234198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58854/0 (socket says 172.21.15.78:58854) 2024-03-24T03:39:50.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.833+0000 7f52388a0640 1 -- 172.21.15.78:0/2891878710 learned_addr learned my addr 172.21.15.78:0/2891878710 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T03:39:50.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.833+0000 7f52390a1640 1 --2- 172.21.15.78:0/2891878710 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5234102910 0x7f5234197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:39:50.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.833+0000 7f52398a2640 1 --2- 172.21.15.78:0/2891878710 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f523410bce0 0x7f523419c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:39:50.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.833+0000 7f52388a0640 1 -- 172.21.15.78:0/2891878710 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f523410bce0 msgr2=0x7f523419c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:50.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.833+0000 7f52388a0640 1 --2- 172.21.15.78:0/2891878710 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f523410bce0 0x7f523419c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:50.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.833+0000 7f52388a0640 1 -- 172.21.15.78:0/2891878710 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5234102910 msgr2=0x7f5234197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:50.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.833+0000 7f52388a0640 1 --2- 172.21.15.78:0/2891878710 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5234102910 0x7f5234197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:50.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.833+0000 7f52388a0640 1 -- 172.21.15.78:0/2891878710 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f523419cf90 con 0x7f5234103230 2024-03-24T03:39:50.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.833+0000 7f52390a1640 1 --2- 172.21.15.78:0/2891878710 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5234102910 0x7f5234197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T03:39:50.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.833+0000 7f52388a0640 1 --2- 172.21.15.78:0/2891878710 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5234103230 0x7f5234198530 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f522400c920 tx=0x7f522400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:39:50.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.833+0000 7f522a7fc640 1 -- 172.21.15.78:0/2891878710 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5224007c70 con 0x7f5234103230 2024-03-24T03:39:50.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.833+0000 7f522a7fc640 1 -- 172.21.15.78:0/2891878710 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5224007e10 con 0x7f5234103230 2024-03-24T03:39:50.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.833+0000 7f523b305640 1 -- 172.21.15.78:0/2891878710 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f523419d280 con 0x7f5234103230 2024-03-24T03:39:50.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.833+0000 7f522a7fc640 1 -- 172.21.15.78:0/2891878710 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5224018890 con 0x7f5234103230 2024-03-24T03:39:50.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.833+0000 7f523b305640 1 -- 172.21.15.78:0/2891878710 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52341a4b10 con 0x7f5234103230 2024-03-24T03:39:50.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.834+0000 7f523b305640 1 -- 172.21.15.78:0/2891878710 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52340688e0 con 0x7f5234103230 2024-03-24T03:39:50.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.836+0000 7f522a7fc640 1 -- 172.21.15.78:0/2891878710 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f5224018a50 con 0x7f5234103230 2024-03-24T03:39:50.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.836+0000 7f522a7fc640 1 --2- 172.21.15.78:0/2891878710 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f52200a29f0 0x7f52200a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:39:50.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.836+0000 7f52390a1640 1 --2- 172.21.15.78:0/2891878710 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f52200a29f0 0x7f52200a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:39:50.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.836+0000 7f522a7fc640 1 -- 172.21.15.78:0/2891878710 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 12982+0+0 (secure 0 0 0) 0x7f52240d03e0 con 0x7f5234103230 2024-03-24T03:39:50.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.837+0000 7f52390a1640 1 --2- 172.21.15.78:0/2891878710 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f52200a29f0 0x7f52200a4eb0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5230006010 tx=0x7f523002cf90 comp rx=0 tx=0).ready entity=mgr.25243 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:39:50.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:50.840+0000 7f522a7fc640 1 -- 172.21.15.78:0/2891878710 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5224014030 con 0x7f5234103230 2024-03-24T03:39:51.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.017+0000 7f523b305640 1 -- 172.21.15.78:0/2891878710 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f52341992f0 con 0x7f5234103230 2024-03-24T03:39:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:50 smithi078 ceph-mon[32429]: pgmap v6309: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:39:51.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:50 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/525372620' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T03:39:51.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:50 smithi107 ceph-mon[36485]: pgmap v6309: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:39:51.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:50 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/525372620' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T03:39:51.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:50 smithi106 ceph-mon[36505]: pgmap v6309: 129 pgs: 129 active+clean; 27 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-24T03:39:51.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:50 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/525372620' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T03:39:51.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.707+0000 7f522a7fc640 1 -- 172.21.15.78:0/2891878710 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v160) v1 ==== 169+0+0 (secure 0 0 0) 0x7f52341992f0 con 0x7f5234103230 2024-03-24T03:39:51.707 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' removed 2024-03-24T03:39:51.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.709+0000 7f523b305640 1 -- 172.21.15.78:0/2891878710 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f52200a29f0 msgr2=0x7f52200a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:51.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.709+0000 7f523b305640 1 --2- 172.21.15.78:0/2891878710 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f52200a29f0 0x7f52200a4eb0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5230006010 tx=0x7f523002cf90 comp rx=0 tx=0).stop 2024-03-24T03:39:51.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.709+0000 7f523b305640 1 -- 172.21.15.78:0/2891878710 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5234103230 msgr2=0x7f5234198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:51.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.709+0000 7f523b305640 1 --2- 172.21.15.78:0/2891878710 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5234103230 0x7f5234198530 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f522400c920 tx=0x7f522400cde0 comp rx=0 tx=0).stop 2024-03-24T03:39:51.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.709+0000 7f523b305640 1 -- 172.21.15.78:0/2891878710 shutdown_connections 2024-03-24T03:39:51.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.709+0000 7f523b305640 1 --2- 172.21.15.78:0/2891878710 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7f52200a29f0 0x7f52200a4eb0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:51.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.709+0000 7f523b305640 1 --2- 172.21.15.78:0/2891878710 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f523410bce0 0x7f523419c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:51.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.709+0000 7f523b305640 1 --2- 172.21.15.78:0/2891878710 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5234103230 0x7f5234198530 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:51.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.709+0000 7f523b305640 1 --2- 172.21.15.78:0/2891878710 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5234102910 0x7f5234197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:51.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.709+0000 7f523b305640 1 -- 172.21.15.78:0/2891878710 >> 172.21.15.78:0/2891878710 conn(0x7f5234077700 msgr2=0x7f52340fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:39:51.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.709+0000 7f523b305640 1 -- 172.21.15.78:0/2891878710 shutdown_connections 2024-03-24T03:39:51.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.709+0000 7f523b305640 1 -- 172.21.15.78:0/2891878710 wait complete. 2024-03-24T03:39:51.724 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-24T03:39:51.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.831+0000 7fd98c2f8640 1 -- 172.21.15.78:0/2557531280 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd984100580 msgr2=0x7fd984100a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:51.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.831+0000 7fd98c2f8640 1 --2- 172.21.15.78:0/2557531280 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd984100580 0x7fd984100a00 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fd974009900 tx=0x7fd97402bc30 comp rx=0 tx=0).stop 2024-03-24T03:39:51.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.831+0000 7fd98c2f8640 1 -- 172.21.15.78:0/2557531280 shutdown_connections 2024-03-24T03:39:51.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.831+0000 7fd98c2f8640 1 --2- 172.21.15.78:0/2557531280 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd984100f40 0x7fd98410d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:51.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.831+0000 7fd98c2f8640 1 --2- 172.21.15.78:0/2557531280 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd984100580 0x7fd984100a00 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:51.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.831+0000 7fd98c2f8640 1 --2- 172.21.15.78:0/2557531280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd984106640 0x7fd984106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:51.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.831+0000 7fd98c2f8640 1 -- 172.21.15.78:0/2557531280 >> 172.21.15.78:0/2557531280 conn(0x7fd984076920 msgr2=0x7fd984076d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:39:51.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.831+0000 7fd98c2f8640 1 -- 172.21.15.78:0/2557531280 shutdown_connections 2024-03-24T03:39:51.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.831+0000 7fd98c2f8640 1 -- 172.21.15.78:0/2557531280 wait complete. 2024-03-24T03:39:51.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.832+0000 7fd98c2f8640 1 Processor -- start 2024-03-24T03:39:51.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.832+0000 7fd98c2f8640 1 -- start start 2024-03-24T03:39:51.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.832+0000 7fd98c2f8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd984100580 0x7fd984197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:39:51.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.832+0000 7fd98c2f8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd984100f40 0x7fd984198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:39:51.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.832+0000 7fd98c2f8640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd984106640 0x7fd98419c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:39:51.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.832+0000 7fd98c2f8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd984110920 con 0x7fd984100580 2024-03-24T03:39:51.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.832+0000 7fd98c2f8640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9841107a0 con 0x7fd984106640 2024-03-24T03:39:51.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.832+0000 7fd98c2f8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd984110aa0 con 0x7fd984100f40 2024-03-24T03:39:51.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.832+0000 7fd98a094640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd984100580 0x7fd984197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:39:51.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.832+0000 7fd98a094640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd984100580 0x7fd984197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58870/0 (socket says 172.21.15.78:58870) 2024-03-24T03:39:51.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.832+0000 7fd98a094640 1 -- 172.21.15.78:0/1104227311 learned_addr learned my addr 172.21.15.78:0/1104227311 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-24T03:39:51.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.832+0000 7fd989893640 1 --2- 172.21.15.78:0/1104227311 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd984100f40 0x7fd984198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:39:51.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.832+0000 7fd98a094640 1 -- 172.21.15.78:0/1104227311 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd984100f40 msgr2=0x7fd984198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:51.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.832+0000 7fd98a094640 1 --2- 172.21.15.78:0/1104227311 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd984100f40 0x7fd984198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:51.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.832+0000 7fd98a094640 1 -- 172.21.15.78:0/1104227311 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd984106640 msgr2=0x7fd98419c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:51.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.832+0000 7fd98a094640 1 --2- 172.21.15.78:0/1104227311 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd984106640 0x7fd98419c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:51.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.832+0000 7fd98a094640 1 -- 172.21.15.78:0/1104227311 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd98419cea0 con 0x7fd984100580 2024-03-24T03:39:51.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.832+0000 7fd98a094640 1 --2- 172.21.15.78:0/1104227311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd984100580 0x7fd984197f00 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fd98000d770 tx=0x7fd98000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:39:51.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.832+0000 7fd97b7fe640 1 -- 172.21.15.78:0/1104227311 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd980017070 con 0x7fd984100580 2024-03-24T03:39:51.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.832+0000 7fd97b7fe640 1 -- 172.21.15.78:0/1104227311 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd980005500 con 0x7fd984100580 2024-03-24T03:39:51.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.833+0000 7fd98c2f8640 1 -- 172.21.15.78:0/1104227311 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd98419d190 con 0x7fd984100580 2024-03-24T03:39:51.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.833+0000 7fd97b7fe640 1 -- 172.21.15.78:0/1104227311 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd980004e00 con 0x7fd984100580 2024-03-24T03:39:51.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.833+0000 7fd98c2f8640 1 -- 172.21.15.78:0/1104227311 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9841a4a20 con 0x7fd984100580 2024-03-24T03:39:51.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.834+0000 7fd98c2f8640 1 -- 172.21.15.78:0/1104227311 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd984101ca0 con 0x7fd984100580 2024-03-24T03:39:51.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.835+0000 7fd97b7fe640 1 -- 172.21.15.78:0/1104227311 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fd980004fa0 con 0x7fd984100580 2024-03-24T03:39:51.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.835+0000 7fd97b7fe640 1 --2- 172.21.15.78:0/1104227311 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fd9700a2970 0x7fd9700a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:39:51.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.836+0000 7fd989893640 1 --2- 172.21.15.78:0/1104227311 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fd9700a2970 0x7fd9700a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:39:51.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.836+0000 7fd97b7fe640 1 -- 172.21.15.78:0/1104227311 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 12452+0+0 (secure 0 0 0) 0x7fd9800cfcc0 con 0x7fd984100580 2024-03-24T03:39:51.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.836+0000 7fd989893640 1 --2- 172.21.15.78:0/1104227311 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fd9700a2970 0x7fd9700a4e30 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd984105c60 tx=0x7fd97402c750 comp rx=0 tx=0).ready entity=mgr.25243 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:39:51.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:51.839+0000 7fd97b7fe640 1 -- 172.21.15.78:0/1104227311 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd98009bac0 con 0x7fd984100580 2024-03-24T03:39:52.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:52.027+0000 7fd98c2f8640 1 -- 172.21.15.78:0/1104227311 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd9841a4c50 con 0x7fd984100580 2024-03-24T03:39:52.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:51 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/525372620' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T03:39:52.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:51 smithi078 ceph-mon[32429]: osdmap e159: 12 total, 12 up, 12 in 2024-03-24T03:39:52.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:51 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2891878710' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T03:39:52.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:51 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/525372620' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T03:39:52.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:51 smithi107 ceph-mon[36485]: osdmap e159: 12 total, 12 up, 12 in 2024-03-24T03:39:52.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:51 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2891878710' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T03:39:52.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:51 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/525372620' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T03:39:52.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:51 smithi106 ceph-mon[36505]: osdmap e159: 12 total, 12 up, 12 in 2024-03-24T03:39:52.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:51 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2891878710' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T03:39:52.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:52.709+0000 7fd97b7fe640 1 -- 172.21.15.78:0/1104227311 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v161) v1 ==== 178+0+0 (secure 0 0 0) 0x7fd98009b3c0 con 0x7fd984100580 2024-03-24T03:39:52.709 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data_ec' removed 2024-03-24T03:39:52.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:52.711+0000 7fd98c2f8640 1 -- 172.21.15.78:0/1104227311 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fd9700a2970 msgr2=0x7fd9700a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:52.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:52.711+0000 7fd98c2f8640 1 --2- 172.21.15.78:0/1104227311 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fd9700a2970 0x7fd9700a4e30 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd984105c60 tx=0x7fd97402c750 comp rx=0 tx=0).stop 2024-03-24T03:39:52.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:52.711+0000 7fd98c2f8640 1 -- 172.21.15.78:0/1104227311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd984100580 msgr2=0x7fd984197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:39:52.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:52.711+0000 7fd98c2f8640 1 --2- 172.21.15.78:0/1104227311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd984100580 0x7fd984197f00 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fd98000d770 tx=0x7fd98000dc30 comp rx=0 tx=0).stop 2024-03-24T03:39:52.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:52.711+0000 7fd98c2f8640 1 -- 172.21.15.78:0/1104227311 shutdown_connections 2024-03-24T03:39:52.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:52.711+0000 7fd98c2f8640 1 --2- 172.21.15.78:0/1104227311 >> [v2:172.21.15.78:6800/3337965055,v1:172.21.15.78:6801/3337965055] conn(0x7fd9700a2970 0x7fd9700a4e30 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:52.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:52.711+0000 7fd98c2f8640 1 --2- 172.21.15.78:0/1104227311 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd984100f40 0x7fd984198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:52.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:52.711+0000 7fd98c2f8640 1 --2- 172.21.15.78:0/1104227311 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd984106640 0x7fd98419c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:52.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:52.711+0000 7fd98c2f8640 1 --2- 172.21.15.78:0/1104227311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd984100580 0x7fd984197f00 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:39:52.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:52.711+0000 7fd98c2f8640 1 -- 172.21.15.78:0/1104227311 >> 172.21.15.78:0/1104227311 conn(0x7fd984076920 msgr2=0x7fd9840ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:39:52.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:52.711+0000 7fd98c2f8640 1 -- 172.21.15.78:0/1104227311 shutdown_connections 2024-03-24T03:39:52.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-24T03:39:52.711+0000 7fd98c2f8640 1 -- 172.21.15.78:0/1104227311 wait complete. 2024-03-24T03:39:52.727 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T03:39:52.756 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T03:39:52.790 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T03:39:52.826 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-24T03:39:52.826 DEBUG:teuthology.orchestra.run.smithi078:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T03:39:52.842 DEBUG:teuthology.orchestra.run.smithi106:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T03:39:52.859 DEBUG:teuthology.orchestra.run.smithi107:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T03:39:52.883 INFO:tasks.cephadm:Stopping all daemons... 2024-03-24T03:39:52.883 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-24T03:39:52.883 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mon.a 2024-03-24T03:39:53.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:52 smithi078 ceph-mon[32429]: pgmap v6311: 97 pgs: 97 active+clean; 21 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T03:39:53.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:52 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/2891878710' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T03:39:53.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:52 smithi078 ceph-mon[32429]: osdmap e160: 12 total, 12 up, 12 in 2024-03-24T03:39:53.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:52 smithi078 ceph-mon[32429]: from='client.? 172.21.15.78:0/1104227311' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T03:39:53.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:52 smithi078 systemd[1]: Stopping Ceph mon.a for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T03:39:53.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:52 smithi107 ceph-mon[36485]: pgmap v6311: 97 pgs: 97 active+clean; 21 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T03:39:53.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:52 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/2891878710' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T03:39:53.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:52 smithi107 ceph-mon[36485]: osdmap e160: 12 total, 12 up, 12 in 2024-03-24T03:39:53.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:52 smithi107 ceph-mon[36485]: from='client.? 172.21.15.78:0/1104227311' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T03:39:53.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:52 smithi106 ceph-mon[36505]: pgmap v6311: 97 pgs: 97 active+clean; 21 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T03:39:53.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:52 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/2891878710' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T03:39:53.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:52 smithi106 ceph-mon[36505]: osdmap e160: 12 total, 12 up, 12 in 2024-03-24T03:39:53.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:52 smithi106 ceph-mon[36505]: from='client.? 172.21.15.78:0/1104227311' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T03:39:53.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mon-a[32425]: 2024-03-24T03:39:53.206+0000 7f94ce332700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:39:53.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mon-a[32425]: 2024-03-24T03:39:53.206+0000 7f94ce332700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-24T03:39:53.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:53 smithi078 podman[141568]: 2024-03-24 03:39:53.258004463 +0000 UTC m=+0.180427292 container died c766f205ef0efd9d287f7e718d1e87cb3ca7b62fe6851449dbfa54159e05dc51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mon-a, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T03:39:53.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:53 smithi078 podman[141568]: 2024-03-24 03:39:53.640491706 +0000 UTC m=+0.562914535 container cleanup c766f205ef0efd9d287f7e718d1e87cb3ca7b62fe6851449dbfa54159e05dc51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mon-a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T03:39:53.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:53 smithi078 bash[141568]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mon-a 2024-03-24T03:39:54.100 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mon.a.service' 2024-03-24T03:39:54.125 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:53 smithi078 podman[141583]: 2024-03-24 03:39:53.849969926 +0000 UTC m=+0.586321155 container remove c766f205ef0efd9d287f7e718d1e87cb3ca7b62fe6851449dbfa54159e05dc51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mon-a, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-24T03:39:54.125 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:54 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-24T03:39:54.126 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:54 smithi078 systemd[1]: Stopped Ceph mon.a for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T03:39:54.126 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 24 03:39:54 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mon.a.service: Consumed 2min 19.162s CPU time. 2024-03-24T03:39:54.161 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:39:54.161 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-24T03:39:54.161 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-24T03:39:54.162 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mon.b 2024-03-24T03:39:54.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:54 smithi106 systemd[1]: Stopping Ceph mon.b for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T03:39:55.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:54 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mon-b[36501]: 2024-03-24T03:39:54.767+0000 7f3879c35700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:39:55.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:54 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mon-b[36501]: 2024-03-24T03:39:54.767+0000 7f3879c35700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-24T03:39:55.153 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:54 smithi106 podman[121355]: 2024-03-24 03:39:54.910785193 +0000 UTC m=+0.549045582 container died e957d719501dc48f6c0219f9b38cc9815beb59f84c48403797c6eb31cfd0afa6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mon-b, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T03:39:55.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:55 smithi106 podman[121355]: 2024-03-24 03:39:55.377508094 +0000 UTC m=+1.015768525 container cleanup e957d719501dc48f6c0219f9b38cc9815beb59f84c48403797c6eb31cfd0afa6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mon-b, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T03:39:55.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 24 03:39:55 smithi106 bash[121355]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mon-b 2024-03-24T03:39:55.800 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mon.b.service' 2024-03-24T03:39:55.848 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:39:55.848 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-24T03:39:55.849 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-24T03:39:55.849 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mon.c 2024-03-24T03:39:56.174 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:55 smithi107 systemd[1]: Stopping Ceph mon.c for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T03:39:56.533 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:56 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mon-c[36481]: 2024-03-24T03:39:56.173+0000 7f6061ad0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:39:56.533 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:56 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mon-c[36481]: 2024-03-24T03:39:56.173+0000 7f6061ad0700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-24T03:39:56.533 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:56 smithi107 podman[124129]: 2024-03-24 03:39:56.235353053 +0000 UTC m=+0.191545146 container died 37d0db3ece2c5741886d1c9dc03a6f86a03ade018620b958b1ecfc944b6cccfd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mon-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-24T03:39:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:56 smithi107 podman[124129]: 2024-03-24 03:39:56.724826593 +0000 UTC m=+0.681018676 container cleanup 37d0db3ece2c5741886d1c9dc03a6f86a03ade018620b958b1ecfc944b6cccfd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-24T03:39:56.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 24 03:39:56 smithi107 bash[124129]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mon-c 2024-03-24T03:39:57.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mon.c.service' 2024-03-24T03:39:57.109 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:39:57.109 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-24T03:39:57.110 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-24T03:39:57.110 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mgr.x 2024-03-24T03:39:57.340 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 03:39:57 smithi078 systemd[1]: Stopping Ceph mgr.x for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T03:39:57.736 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 03:39:57 smithi078 podman[141701]: 2024-03-24 03:39:57.458474705 +0000 UTC m=+0.163618184 container died f41546bd5b7fa8bf8e03bbf855c6658b6aadab51760348f365de1cdf2dfeb9fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-24T03:39:58.045 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 03:39:57 smithi078 podman[141701]: 2024-03-24 03:39:57.819213888 +0000 UTC m=+0.524357367 container cleanup f41546bd5b7fa8bf8e03bbf855c6658b6aadab51760348f365de1cdf2dfeb9fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T03:39:58.045 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 03:39:57 smithi078 bash[141701]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x 2024-03-24T03:39:58.289 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mgr.x.service' 2024-03-24T03:39:58.314 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 03:39:58 smithi078 podman[141715]: 2024-03-24 03:39:58.045823378 +0000 UTC m=+0.584515052 container remove f41546bd5b7fa8bf8e03bbf855c6658b6aadab51760348f365de1cdf2dfeb9fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-x, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS) 2024-03-24T03:39:58.314 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 03:39:58 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mgr.x.service: Deactivated successfully. 2024-03-24T03:39:58.314 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 03:39:58 smithi078 systemd[1]: Stopped Ceph mgr.x for c7e2806a-e972-11ee-95cd-87774f69a715. 2024-03-24T03:39:58.314 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 24 03:39:58 smithi078 systemd[1]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mgr.x.service: Consumed 7min 19.816s CPU time. 2024-03-24T03:39:58.328 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:39:58.328 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-24T03:39:58.328 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-24T03:39:58.328 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mgr.y 2024-03-24T03:39:58.652 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 03:39:58 smithi106 systemd[1]: Stopping Ceph mgr.y for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T03:39:58.975 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 03:39:58 smithi106 podman[121483]: 2024-03-24 03:39:58.66932032 +0000 UTC m=+0.138164727 container died 1d774065926dab3f34293996a29433ad313959694acd46f4c90964e8c2e6d9be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-24T03:39:59.359 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 03:39:59 smithi106 podman[121483]: 2024-03-24 03:39:59.082755519 +0000 UTC m=+0.551599918 container cleanup 1d774065926dab3f34293996a29433ad313959694acd46f4c90964e8c2e6d9be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T03:39:59.359 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 24 03:39:59 smithi106 bash[121483]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-y 2024-03-24T03:39:59.492 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mgr.y.service' 2024-03-24T03:39:59.528 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:39:59.529 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-24T03:39:59.529 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-24T03:39:59.529 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mgr.z 2024-03-24T03:39:59.843 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 03:39:59 smithi107 systemd[1]: Stopping Ceph mgr.z for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T03:40:00.129 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 03:39:59 smithi107 podman[124251]: 2024-03-24 03:39:59.857713359 +0000 UTC m=+0.141077406 container died 2ac0a0181356e05ede1d3448e3882e017ca1374fc1481655daa6589f10714ced (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-24T03:40:00.547 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 03:40:00 smithi107 podman[124251]: 2024-03-24 03:40:00.229566006 +0000 UTC m=+0.512930044 container cleanup 2ac0a0181356e05ede1d3448e3882e017ca1374fc1481655daa6589f10714ced (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-24T03:40:00.547 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 24 03:40:00 smithi107 bash[124251]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mgr-z 2024-03-24T03:40:00.671 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mgr.z.service' 2024-03-24T03:40:00.708 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:40:00.708 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-24T03:40:00.709 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-24T03:40:00.709 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.0 2024-03-24T03:40:01.015 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 24 03:40:00 smithi078 systemd[1]: Stopping Ceph osd.0 for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T03:40:01.340 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 24 03:40:01 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-0[44470]: 2024-03-24T03:40:01.014+0000 7f7028441700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:40:01.340 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 24 03:40:01 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-0[44470]: 2024-03-24T03:40:01.014+0000 7f7028441700 -1 osd.0 161 *** Got signal Terminated *** 2024-03-24T03:40:01.341 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 24 03:40:01 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-0[44470]: 2024-03-24T03:40:01.014+0000 7f7028441700 -1 osd.0 161 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T03:40:06.732 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 24 03:40:06 smithi078 podman[141826]: 2024-03-24 03:40:06.445917076 +0000 UTC m=+5.543975750 container died b307c21203417f1e0e87b38bc2006248822e082cbb85d9b1dea0c172aaa3f819 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-24T03:40:07.090 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 24 03:40:06 smithi078 podman[141826]: 2024-03-24 03:40:06.82388465 +0000 UTC m=+5.921943325 container cleanup b307c21203417f1e0e87b38bc2006248822e082cbb85d9b1dea0c172aaa3f819 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T03:40:07.090 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 24 03:40:06 smithi078 bash[141826]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-0 2024-03-24T03:40:07.447 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 24 03:40:07 smithi078 podman[141843]: 2024-03-24 03:40:07.099940782 +0000 UTC m=+0.648345831 container remove b307c21203417f1e0e87b38bc2006248822e082cbb85d9b1dea0c172aaa3f819 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T03:40:07.698 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 24 03:40:07 smithi078 podman[141936]: 2024-03-24 03:40:07.350184054 +0000 UTC m=+0.018839685 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T03:40:07.698 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 24 03:40:07 smithi078 podman[141936]: 2024-03-24 03:40:07.450883777 +0000 UTC m=+0.119539441 container create c13d1b4f13f5a7a4e36cb56bf1b8dca045af7bda3d16fd909c1d2b2b89065943 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-0-deactivate, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-24T03:40:07.699 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 24 03:40:07 smithi078 podman[141936]: 2024-03-24 03:40:07.695447707 +0000 UTC m=+0.364103333 container init c13d1b4f13f5a7a4e36cb56bf1b8dca045af7bda3d16fd909c1d2b2b89065943 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-24T03:40:07.986 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 24 03:40:07 smithi078 podman[141936]: 2024-03-24 03:40:07.698992199 +0000 UTC m=+0.367647824 container start c13d1b4f13f5a7a4e36cb56bf1b8dca045af7bda3d16fd909c1d2b2b89065943 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-24T03:40:07.986 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 24 03:40:07 smithi078 podman[141936]: 2024-03-24 03:40:07.811872981 +0000 UTC m=+0.480528613 container attach c13d1b4f13f5a7a4e36cb56bf1b8dca045af7bda3d16fd909c1d2b2b89065943 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-0-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-24T03:40:08.339 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 24 03:40:07 smithi078 podman[141936]: 2024-03-24 03:40:07.986816644 +0000 UTC m=+0.655472271 container died c13d1b4f13f5a7a4e36cb56bf1b8dca045af7bda3d16fd909c1d2b2b89065943 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-24T03:40:08.676 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.0.service' 2024-03-24T03:40:08.718 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:40:08.719 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-24T03:40:08.719 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-24T03:40:08.719 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.1 2024-03-24T03:40:09.038 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 24 03:40:08 smithi106 systemd[1]: Stopping Ceph osd.1 for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T03:40:09.402 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 24 03:40:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-1[41094]: 2024-03-24T03:40:09.037+0000 7f461cb9d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:40:09.402 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 24 03:40:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-1[41094]: 2024-03-24T03:40:09.037+0000 7f461cb9d700 -1 osd.1 161 *** Got signal Terminated *** 2024-03-24T03:40:09.403 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 24 03:40:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-1[41094]: 2024-03-24T03:40:09.037+0000 7f461cb9d700 -1 osd.1 161 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T03:40:14.697 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 24 03:40:14 smithi106 podman[121610]: 2024-03-24 03:40:14.407325079 +0000 UTC m=+5.491814132 container died 08dd91ac5525c9f66e922e7f49961af1fcdba1d2f108107b71369cdc444d9ccf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-1, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319) 2024-03-24T03:40:14.989 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 24 03:40:14 smithi106 podman[121610]: 2024-03-24 03:40:14.796852363 +0000 UTC m=+5.881341405 container cleanup 08dd91ac5525c9f66e922e7f49961af1fcdba1d2f108107b71369cdc444d9ccf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-1, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319) 2024-03-24T03:40:14.989 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 24 03:40:14 smithi106 bash[121610]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-1 2024-03-24T03:40:15.402 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 24 03:40:14 smithi106 podman[121627]: 2024-03-24 03:40:14.990055784 +0000 UTC m=+0.579782868 container remove 08dd91ac5525c9f66e922e7f49961af1fcdba1d2f108107b71369cdc444d9ccf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-1, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319) 2024-03-24T03:40:15.676 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 24 03:40:15 smithi106 podman[121721]: 2024-03-24 03:40:15.30612335 +0000 UTC m=+0.017713627 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T03:40:15.676 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 24 03:40:15 smithi106 podman[121721]: 2024-03-24 03:40:15.416163779 +0000 UTC m=+0.127754055 container create 8f3e57204a1fb567cf0ddc904136ef6de4bb086d874ead10825313ad8dd8157c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-24T03:40:15.677 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 24 03:40:15 smithi106 podman[121721]: 2024-03-24 03:40:15.67713463 +0000 UTC m=+0.388724913 container init 8f3e57204a1fb567cf0ddc904136ef6de4bb086d874ead10825313ad8dd8157c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS) 2024-03-24T03:40:15.970 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 24 03:40:15 smithi106 podman[121721]: 2024-03-24 03:40:15.680348326 +0000 UTC m=+0.391938594 container start 8f3e57204a1fb567cf0ddc904136ef6de4bb086d874ead10825313ad8dd8157c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-24T03:40:15.971 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 24 03:40:15 smithi106 podman[121721]: 2024-03-24 03:40:15.777252579 +0000 UTC m=+0.488842867 container attach 8f3e57204a1fb567cf0ddc904136ef6de4bb086d874ead10825313ad8dd8157c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-24T03:40:16.321 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 24 03:40:15 smithi106 podman[121721]: 2024-03-24 03:40:15.972082383 +0000 UTC m=+0.683672670 container died 8f3e57204a1fb567cf0ddc904136ef6de4bb086d874ead10825313ad8dd8157c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-24T03:40:16.648 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.1.service' 2024-03-24T03:40:16.689 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:40:16.690 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-24T03:40:16.690 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-24T03:40:16.690 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.2 2024-03-24T03:40:17.057 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 24 03:40:16 smithi107 systemd[1]: Stopping Ceph osd.2 for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T03:40:17.326 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 24 03:40:17 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-2[41647]: 2024-03-24T03:40:17.056+0000 7f64411d6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:40:17.326 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 24 03:40:17 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-2[41647]: 2024-03-24T03:40:17.056+0000 7f64411d6700 -1 osd.2 161 *** Got signal Terminated *** 2024-03-24T03:40:17.326 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 24 03:40:17 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-2[41647]: 2024-03-24T03:40:17.056+0000 7f64411d6700 -1 osd.2 161 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T03:40:22.789 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 24 03:40:22 smithi107 podman[124376]: 2024-03-24 03:40:22.475173833 +0000 UTC m=+5.588937626 container died c7ecb3310794e65625564d5d0941ded51ccd8b78be1c6bbe5713e603cccf78bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T03:40:23.093 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 24 03:40:22 smithi107 podman[124376]: 2024-03-24 03:40:22.790016074 +0000 UTC m=+5.903779836 container cleanup c7ecb3310794e65625564d5d0941ded51ccd8b78be1c6bbe5713e603cccf78bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-2, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T03:40:23.093 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 24 03:40:22 smithi107 bash[124376]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-2 2024-03-24T03:40:23.093 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 24 03:40:23 smithi107 podman[124406]: 2024-03-24 03:40:23.025275174 +0000 UTC m=+0.548027895 container remove c7ecb3310794e65625564d5d0941ded51ccd8b78be1c6bbe5713e603cccf78bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-24T03:40:23.703 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 24 03:40:23 smithi107 podman[124501]: 2024-03-24 03:40:23.349168668 +0000 UTC m=+0.017994703 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T03:40:23.703 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 24 03:40:23 smithi107 podman[124501]: 2024-03-24 03:40:23.459897223 +0000 UTC m=+0.128723242 container create 7f7f0fa815968a8270faf8d2585b9d95a79f890336b160d40e2134c27aca0eca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-24T03:40:23.703 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 24 03:40:23 smithi107 podman[124501]: 2024-03-24 03:40:23.703849103 +0000 UTC m=+0.372675123 container init 7f7f0fa815968a8270faf8d2585b9d95a79f890336b160d40e2134c27aca0eca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-24T03:40:23.994 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 24 03:40:23 smithi107 podman[124501]: 2024-03-24 03:40:23.707193916 +0000 UTC m=+0.376019936 container start 7f7f0fa815968a8270faf8d2585b9d95a79f890336b160d40e2134c27aca0eca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-2-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T03:40:23.994 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 24 03:40:23 smithi107 podman[124501]: 2024-03-24 03:40:23.804109896 +0000 UTC m=+0.472935921 container attach 7f7f0fa815968a8270faf8d2585b9d95a79f890336b160d40e2134c27aca0eca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T03:40:24.257 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 24 03:40:23 smithi107 podman[124501]: 2024-03-24 03:40:23.995123113 +0000 UTC m=+0.663949143 container died 7f7f0fa815968a8270faf8d2585b9d95a79f890336b160d40e2134c27aca0eca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-24T03:40:24.610 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.2.service' 2024-03-24T03:40:24.651 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:40:24.651 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-24T03:40:24.651 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-24T03:40:24.652 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.3 2024-03-24T03:40:24.958 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 24 03:40:24 smithi078 systemd[1]: Stopping Ceph osd.3 for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T03:40:25.217 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 24 03:40:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-3[51438]: 2024-03-24T03:40:24.958+0000 7f31d67a4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:40:25.217 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 24 03:40:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-3[51438]: 2024-03-24T03:40:24.958+0000 7f31d67a4700 -1 osd.3 161 *** Got signal Terminated *** 2024-03-24T03:40:25.217 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 24 03:40:24 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-3[51438]: 2024-03-24T03:40:24.958+0000 7f31d67a4700 -1 osd.3 161 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T03:40:27.652 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:27 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:27.544+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:40:27.753 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:27 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:27.466+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:40:27.840 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 24 03:40:27 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-3[51438]: 2024-03-24T03:40:27.504+0000 7f31ced90700 -1 osd.3 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.713862+0000 front 2024-03-24T03:40:04.713814+0000 (oldest deadline 2024-03-24T03:40:27.013959+0000) 2024-03-24T03:40:28.593 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:28.484+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:40:28.840 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 24 03:40:28 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-3[51438]: 2024-03-24T03:40:28.480+0000 7f31ced90700 -1 osd.3 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.713862+0000 front 2024-03-24T03:40:04.713814+0000 (oldest deadline 2024-03-24T03:40:27.013959+0000) 2024-03-24T03:40:28.902 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:28 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:28.585+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:40:29.840 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 24 03:40:29 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-3[51438]: 2024-03-24T03:40:29.529+0000 7f31ced90700 -1 osd.3 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.713862+0000 front 2024-03-24T03:40:04.713814+0000 (oldest deadline 2024-03-24T03:40:27.013959+0000) 2024-03-24T03:40:29.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:29.470+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:40:29.843 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5[48372]: 2024-03-24T03:40:29.594+0000 7fd9de157700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.614044+0000 front 2024-03-24T03:40:04.614010+0000 (oldest deadline 2024-03-24T03:40:28.714168+0000) 2024-03-24T03:40:29.902 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:29 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:29.623+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:40:30.659 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 24 03:40:30 smithi078 podman[142042]: 2024-03-24 03:40:30.374556143 +0000 UTC m=+5.520114776 container died a88b01b13dd0225bf2b0164f9645d499c29ad47c402062ddfd32dad4cf9bbe0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-3, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T03:40:30.843 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:30 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5[48372]: 2024-03-24T03:40:30.570+0000 7fd9de157700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.614044+0000 front 2024-03-24T03:40:04.614010+0000 (oldest deadline 2024-03-24T03:40:28.714168+0000) 2024-03-24T03:40:30.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:30 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:30.502+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:40:30.902 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:30 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:30.645+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:40:31.009 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 24 03:40:30 smithi078 podman[142042]: 2024-03-24 03:40:30.766904525 +0000 UTC m=+5.912463155 container cleanup a88b01b13dd0225bf2b0164f9645d499c29ad47c402062ddfd32dad4cf9bbe0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-3, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True) 2024-03-24T03:40:31.010 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 24 03:40:30 smithi078 bash[142042]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-3 2024-03-24T03:40:31.310 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 24 03:40:31 smithi078 podman[142070]: 2024-03-24 03:40:31.009944876 +0000 UTC m=+0.638228348 container remove a88b01b13dd0225bf2b0164f9645d499c29ad47c402062ddfd32dad4cf9bbe0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-3, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-24T03:40:31.580 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 24 03:40:31 smithi078 podman[142161]: 2024-03-24 03:40:31.213344967 +0000 UTC m=+0.017818471 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T03:40:31.580 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 24 03:40:31 smithi078 podman[142161]: 2024-03-24 03:40:31.327773332 +0000 UTC m=+0.132246818 container create ff3e4b4f9b96c3a4d09aab51d752dabda16eb49b78d1179cd2f7efed2b008199 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-24T03:40:31.841 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 24 03:40:31 smithi078 podman[142161]: 2024-03-24 03:40:31.580631123 +0000 UTC m=+0.385104615 container init ff3e4b4f9b96c3a4d09aab51d752dabda16eb49b78d1179cd2f7efed2b008199 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-24T03:40:31.841 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 24 03:40:31 smithi078 podman[142161]: 2024-03-24 03:40:31.584014714 +0000 UTC m=+0.388488203 container start ff3e4b4f9b96c3a4d09aab51d752dabda16eb49b78d1179cd2f7efed2b008199 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0) 2024-03-24T03:40:31.841 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 24 03:40:31 smithi078 podman[142161]: 2024-03-24 03:40:31.671841133 +0000 UTC m=+0.476314641 container attach ff3e4b4f9b96c3a4d09aab51d752dabda16eb49b78d1179cd2f7efed2b008199 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T03:40:31.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:31 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:31.479+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:40:31.843 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:31 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5[48372]: 2024-03-24T03:40:31.555+0000 7fd9de157700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.614044+0000 front 2024-03-24T03:40:04.614010+0000 (oldest deadline 2024-03-24T03:40:28.714168+0000) 2024-03-24T03:40:32.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:31 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:31.659+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:40:32.165 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 24 03:40:31 smithi078 podman[142161]: 2024-03-24 03:40:31.883054559 +0000 UTC m=+0.687528049 container died ff3e4b4f9b96c3a4d09aab51d752dabda16eb49b78d1179cd2f7efed2b008199 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T03:40:32.533 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:32 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:32.164+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:40:32.560 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.3.service' 2024-03-24T03:40:32.604 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:40:32.604 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-24T03:40:32.605 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-24T03:40:32.605 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.4 2024-03-24T03:40:32.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:32 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:32.430+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:40:32.843 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:32 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5[48372]: 2024-03-24T03:40:32.560+0000 7fd9de157700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.614044+0000 front 2024-03-24T03:40:04.614010+0000 (oldest deadline 2024-03-24T03:40:28.714168+0000) 2024-03-24T03:40:32.902 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:32 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:32.698+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:40:32.902 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 24 03:40:32 smithi106 systemd[1]: Stopping Ceph osd.4 for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T03:40:33.402 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 24 03:40:32 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-4[48051]: 2024-03-24T03:40:32.928+0000 7fc69860a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:40:33.403 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 24 03:40:32 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-4[48051]: 2024-03-24T03:40:32.928+0000 7fc69860a700 -1 osd.4 161 *** Got signal Terminated *** 2024-03-24T03:40:33.403 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 24 03:40:32 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-4[48051]: 2024-03-24T03:40:32.928+0000 7fc69860a700 -1 osd.4 161 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T03:40:33.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:33 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:33.173+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:40:33.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:33 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:33.471+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:40:33.843 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:33 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5[48372]: 2024-03-24T03:40:33.577+0000 7fd9de157700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.614044+0000 front 2024-03-24T03:40:04.614010+0000 (oldest deadline 2024-03-24T03:40:28.714168+0000) 2024-03-24T03:40:34.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:33 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:33.681+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:40:34.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:34 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:34.135+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:40:34.833 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:34 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5[48372]: 2024-03-24T03:40:34.563+0000 7fd9de157700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.614044+0000 front 2024-03-24T03:40:04.614010+0000 (oldest deadline 2024-03-24T03:40:28.714168+0000) 2024-03-24T03:40:34.834 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:34 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:34.446+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:40:35.093 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:34 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:34.834+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:40:35.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:34 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:34.668+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:40:35.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:35 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:35.180+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:40:35.843 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:35 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5[48372]: 2024-03-24T03:40:35.575+0000 7fd9de157700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.614044+0000 front 2024-03-24T03:40:04.614010+0000 (oldest deadline 2024-03-24T03:40:28.714168+0000) 2024-03-24T03:40:35.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:35 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:35.440+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:40:36.152 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 24 03:40:35 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-4[48051]: 2024-03-24T03:40:35.845+0000 7fc690bf6700 -1 osd.4 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.917635+0000 front 2024-03-24T03:40:12.917662+0000 (oldest deadline 2024-03-24T03:40:35.817741+0000) 2024-03-24T03:40:36.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:35 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:35.674+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:40:36.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:35 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:35.863+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:40:36.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:36 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:36.223+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:40:36.843 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:36 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5[48372]: 2024-03-24T03:40:36.621+0000 7fd9de157700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.614044+0000 front 2024-03-24T03:40:04.614010+0000 (oldest deadline 2024-03-24T03:40:28.714168+0000) 2024-03-24T03:40:36.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:36 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:36.484+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:40:37.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:36 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:36.692+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:40:37.152 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 24 03:40:36 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-4[48051]: 2024-03-24T03:40:36.856+0000 7fc690bf6700 -1 osd.4 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.917635+0000 front 2024-03-24T03:40:12.917662+0000 (oldest deadline 2024-03-24T03:40:35.817741+0000) 2024-03-24T03:40:37.232 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:36 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6[58136]: 2024-03-24T03:40:36.907+0000 7f8a3fa66700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:10.563069+0000 front 2024-03-24T03:40:10.563156+0000 (oldest deadline 2024-03-24T03:40:36.463161+0000) 2024-03-24T03:40:37.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:36 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:36.848+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:40:37.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:37 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:37.232+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:40:37.843 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:37 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5[48372]: 2024-03-24T03:40:37.593+0000 7fd9de157700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.614044+0000 front 2024-03-24T03:40:04.614010+0000 (oldest deadline 2024-03-24T03:40:28.714168+0000) 2024-03-24T03:40:37.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:37 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:37.490+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:40:38.152 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 24 03:40:37 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-4[48051]: 2024-03-24T03:40:37.824+0000 7fc690bf6700 -1 osd.4 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.917635+0000 front 2024-03-24T03:40:12.917662+0000 (oldest deadline 2024-03-24T03:40:35.817741+0000) 2024-03-24T03:40:38.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:37 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:37.705+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:40:38.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:37 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:37.705+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:40:38.242 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:37 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6[58136]: 2024-03-24T03:40:37.908+0000 7f8a3fa66700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:10.563069+0000 front 2024-03-24T03:40:10.563156+0000 (oldest deadline 2024-03-24T03:40:36.463161+0000) 2024-03-24T03:40:38.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:37 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:37.850+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:40:38.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:38 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:38.242+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:40:38.636 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 24 03:40:38 smithi106 podman[121840]: 2024-03-24 03:40:38.346995252 +0000 UTC m=+5.554387497 container died 561904edeeb39444b525cc75673188362137df7318dd2654a380f42aaa6fc655 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-4, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-24T03:40:38.637 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:38.371+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.182631+0000 front 2024-03-24T03:40:13.182383+0000 (oldest deadline 2024-03-24T03:40:37.882570+0000) 2024-03-24T03:40:38.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:38.523+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:40:38.843 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5[48372]: 2024-03-24T03:40:38.590+0000 7fd9de157700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.614044+0000 front 2024-03-24T03:40:04.614010+0000 (oldest deadline 2024-03-24T03:40:28.714168+0000) 2024-03-24T03:40:38.843 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5[48372]: 2024-03-24T03:40:38.590+0000 7fd9de157700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.415384+0000 front 2024-03-24T03:40:13.415477+0000 (oldest deadline 2024-03-24T03:40:38.115366+0000) 2024-03-24T03:40:38.902 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 24 03:40:38 smithi106 podman[121840]: 2024-03-24 03:40:38.727335424 +0000 UTC m=+5.934727679 container cleanup 561904edeeb39444b525cc75673188362137df7318dd2654a380f42aaa6fc655 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-4, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_CLEAN=True) 2024-03-24T03:40:38.902 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 24 03:40:38 smithi106 bash[121840]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-4 2024-03-24T03:40:38.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:38.675+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:40:38.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:38 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:38.675+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:40:39.271 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:38 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6[58136]: 2024-03-24T03:40:38.878+0000 7f8a3fa66700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:10.563069+0000 front 2024-03-24T03:40:10.563156+0000 (oldest deadline 2024-03-24T03:40:36.463161+0000) 2024-03-24T03:40:39.338 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 24 03:40:39 smithi106 podman[121857]: 2024-03-24 03:40:39.012213629 +0000 UTC m=+0.665075170 container remove 561904edeeb39444b525cc75673188362137df7318dd2654a380f42aaa6fc655 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-4, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-24T03:40:39.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:38 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:38.881+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:40:39.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:39 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:39.271+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:40:39.636 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:39.420+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.182631+0000 front 2024-03-24T03:40:13.182383+0000 (oldest deadline 2024-03-24T03:40:37.882570+0000) 2024-03-24T03:40:39.636 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 24 03:40:39 smithi106 podman[121948]: 2024-03-24 03:40:39.241334285 +0000 UTC m=+0.017771935 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T03:40:39.636 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 24 03:40:39 smithi106 podman[121948]: 2024-03-24 03:40:39.355031237 +0000 UTC m=+0.131468912 container create 3c265c53904fc09559c215aa0b081ea05673623eae3a135853c5e26deda751ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-24T03:40:39.636 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 24 03:40:39 smithi106 podman[121948]: 2024-03-24 03:40:39.540691484 +0000 UTC m=+0.317129135 container init 3c265c53904fc09559c215aa0b081ea05673623eae3a135853c5e26deda751ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-24T03:40:39.636 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 24 03:40:39 smithi106 podman[121948]: 2024-03-24 03:40:39.544116177 +0000 UTC m=+0.320553804 container start 3c265c53904fc09559c215aa0b081ea05673623eae3a135853c5e26deda751ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T03:40:39.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:39 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:39.532+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:40:39.843 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:39 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5[48372]: 2024-03-24T03:40:39.564+0000 7fd9de157700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.614044+0000 front 2024-03-24T03:40:04.614010+0000 (oldest deadline 2024-03-24T03:40:28.714168+0000) 2024-03-24T03:40:39.844 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:39 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5[48372]: 2024-03-24T03:40:39.564+0000 7fd9de157700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.415384+0000 front 2024-03-24T03:40:13.415477+0000 (oldest deadline 2024-03-24T03:40:38.115366+0000) 2024-03-24T03:40:39.902 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 24 03:40:39 smithi106 podman[121948]: 2024-03-24 03:40:39.640723125 +0000 UTC m=+0.417160771 container attach 3c265c53904fc09559c215aa0b081ea05673623eae3a135853c5e26deda751ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T03:40:39.903 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 24 03:40:39 smithi106 podman[121948]: 2024-03-24 03:40:39.832468987 +0000 UTC m=+0.608906623 container died 3c265c53904fc09559c215aa0b081ea05673623eae3a135853c5e26deda751ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-24T03:40:39.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:39.636+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:40:39.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:39 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:39.636+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:40:40.090 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:39 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6[58136]: 2024-03-24T03:40:39.832+0000 7f8a3fa66700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:10.563069+0000 front 2024-03-24T03:40:10.563156+0000 (oldest deadline 2024-03-24T03:40:36.463161+0000) 2024-03-24T03:40:40.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:39 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:39.908+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:40:40.472 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.4.service' 2024-03-24T03:40:40.513 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:40:40.514 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-24T03:40:40.514 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-24T03:40:40.514 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.5 2024-03-24T03:40:40.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:40 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:40.319+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:40:40.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:40 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:40.319+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:40:40.810 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:40 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:40.506+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:40:40.810 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:40 smithi107 systemd[1]: Stopping Ceph osd.5 for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T03:40:40.810 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:40 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5[48372]: 2024-03-24T03:40:40.592+0000 7fd9de157700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.614044+0000 front 2024-03-24T03:40:04.614010+0000 (oldest deadline 2024-03-24T03:40:28.714168+0000) 2024-03-24T03:40:40.810 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:40 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5[48372]: 2024-03-24T03:40:40.592+0000 7fd9de157700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.415384+0000 front 2024-03-24T03:40:13.415477+0000 (oldest deadline 2024-03-24T03:40:38.115366+0000) 2024-03-24T03:40:40.810 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:40 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5[48372]: 2024-03-24T03:40:40.809+0000 7fd9e5b6b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:40:40.811 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:40 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5[48372]: 2024-03-24T03:40:40.809+0000 7fd9e5b6b700 -1 osd.5 161 *** Got signal Terminated *** 2024-03-24T03:40:40.811 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:40 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5[48372]: 2024-03-24T03:40:40.809+0000 7fd9e5b6b700 -1 osd.5 161 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T03:40:40.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:40 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:40.407+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.182631+0000 front 2024-03-24T03:40:13.182383+0000 (oldest deadline 2024-03-24T03:40:37.882570+0000) 2024-03-24T03:40:40.902 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:40 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:40.590+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:40:40.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:40 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:40.590+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:40:41.090 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:40 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6[58136]: 2024-03-24T03:40:40.830+0000 7f8a3fa66700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:10.563069+0000 front 2024-03-24T03:40:10.563156+0000 (oldest deadline 2024-03-24T03:40:36.463161+0000) 2024-03-24T03:40:41.093 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:40 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:40.870+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:40:41.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:41 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:41.314+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:40:41.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:41 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:41.314+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:40:41.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:41 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:41.498+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:40:41.843 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:41 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5[48372]: 2024-03-24T03:40:41.632+0000 7fd9de157700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.614044+0000 front 2024-03-24T03:40:04.614010+0000 (oldest deadline 2024-03-24T03:40:28.714168+0000) 2024-03-24T03:40:41.844 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:41 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5[48372]: 2024-03-24T03:40:41.632+0000 7fd9de157700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.415384+0000 front 2024-03-24T03:40:13.415477+0000 (oldest deadline 2024-03-24T03:40:38.115366+0000) 2024-03-24T03:40:41.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:41 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:41.454+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.182631+0000 front 2024-03-24T03:40:13.182383+0000 (oldest deadline 2024-03-24T03:40:37.882570+0000) 2024-03-24T03:40:41.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:41 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:41.578+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:40:41.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:41 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:41.578+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:40:42.091 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:41 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6[58136]: 2024-03-24T03:40:41.785+0000 7f8a3fa66700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:10.563069+0000 front 2024-03-24T03:40:10.563156+0000 (oldest deadline 2024-03-24T03:40:36.463161+0000) 2024-03-24T03:40:42.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:41 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:41.905+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:40:42.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:42 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:42.331+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:40:42.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:42 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:42.331+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:40:42.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:42 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:42.531+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:40:42.843 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:42 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5[48372]: 2024-03-24T03:40:42.682+0000 7fd9de157700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.614044+0000 front 2024-03-24T03:40:04.614010+0000 (oldest deadline 2024-03-24T03:40:28.714168+0000) 2024-03-24T03:40:42.844 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:42 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5[48372]: 2024-03-24T03:40:42.682+0000 7fd9de157700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.415384+0000 front 2024-03-24T03:40:13.415477+0000 (oldest deadline 2024-03-24T03:40:38.115366+0000) 2024-03-24T03:40:42.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:42 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:42.495+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.182631+0000 front 2024-03-24T03:40:13.182383+0000 (oldest deadline 2024-03-24T03:40:37.882570+0000) 2024-03-24T03:40:42.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:42 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:42.620+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:40:42.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:42 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:42.620+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:40:43.090 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:42 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6[58136]: 2024-03-24T03:40:42.784+0000 7f8a3fa66700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:10.563069+0000 front 2024-03-24T03:40:10.563156+0000 (oldest deadline 2024-03-24T03:40:36.463161+0000) 2024-03-24T03:40:43.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:42 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:42.934+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:40:43.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:43.282+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:40:43.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:43.282+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:40:43.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:43.532+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:40:43.843 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5[48372]: 2024-03-24T03:40:43.644+0000 7fd9de157700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.614044+0000 front 2024-03-24T03:40:04.614010+0000 (oldest deadline 2024-03-24T03:40:28.714168+0000) 2024-03-24T03:40:43.844 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5[48372]: 2024-03-24T03:40:43.644+0000 7fd9de157700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.415384+0000 front 2024-03-24T03:40:13.415477+0000 (oldest deadline 2024-03-24T03:40:38.115366+0000) 2024-03-24T03:40:43.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:43 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:43.501+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.182631+0000 front 2024-03-24T03:40:13.182383+0000 (oldest deadline 2024-03-24T03:40:37.882570+0000) 2024-03-24T03:40:43.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:43 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:43.501+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:18.383583+0000 front 2024-03-24T03:40:18.383626+0000 (oldest deadline 2024-03-24T03:40:43.083704+0000) 2024-03-24T03:40:43.902 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:43 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:43.652+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:40:43.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:43 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:43.652+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:40:44.090 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:43 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6[58136]: 2024-03-24T03:40:43.747+0000 7f8a3fa66700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:10.563069+0000 front 2024-03-24T03:40:10.563156+0000 (oldest deadline 2024-03-24T03:40:36.463161+0000) 2024-03-24T03:40:44.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:43.970+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:40:44.344 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:43 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:43.970+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:17.321155+0000 front 2024-03-24T03:40:17.321175+0000 (oldest deadline 2024-03-24T03:40:43.221253+0000) 2024-03-24T03:40:44.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:44.296+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:40:44.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:44.296+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:40:44.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:44 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:44.571+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:40:44.843 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:44 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5[48372]: 2024-03-24T03:40:44.605+0000 7fd9de157700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.614044+0000 front 2024-03-24T03:40:04.614010+0000 (oldest deadline 2024-03-24T03:40:28.714168+0000) 2024-03-24T03:40:44.844 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:44 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5[48372]: 2024-03-24T03:40:44.605+0000 7fd9de157700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.415384+0000 front 2024-03-24T03:40:13.415477+0000 (oldest deadline 2024-03-24T03:40:38.115366+0000) 2024-03-24T03:40:44.844 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:44 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5[48372]: 2024-03-24T03:40:44.605+0000 7fd9de157700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:18.115706+0000 front 2024-03-24T03:40:18.115685+0000 (oldest deadline 2024-03-24T03:40:44.015832+0000) 2024-03-24T03:40:44.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:44 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:44.506+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.182631+0000 front 2024-03-24T03:40:13.182383+0000 (oldest deadline 2024-03-24T03:40:37.882570+0000) 2024-03-24T03:40:44.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:44 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:44.506+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:18.383583+0000 front 2024-03-24T03:40:18.383626+0000 (oldest deadline 2024-03-24T03:40:43.083704+0000) 2024-03-24T03:40:44.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:44 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:44.630+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:40:44.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:44 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:44.630+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:40:45.090 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:44 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6[58136]: 2024-03-24T03:40:44.765+0000 7f8a3fa66700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:10.563069+0000 front 2024-03-24T03:40:10.563156+0000 (oldest deadline 2024-03-24T03:40:36.463161+0000) 2024-03-24T03:40:45.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:44 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:44.945+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:40:45.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:44 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:44.945+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:17.321155+0000 front 2024-03-24T03:40:17.321175+0000 (oldest deadline 2024-03-24T03:40:43.221253+0000) 2024-03-24T03:40:45.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:45 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:45.291+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:40:45.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:45 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:45.291+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:40:45.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:45 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:45.556+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:40:45.843 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:45 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5[48372]: 2024-03-24T03:40:45.635+0000 7fd9de157700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.614044+0000 front 2024-03-24T03:40:04.614010+0000 (oldest deadline 2024-03-24T03:40:28.714168+0000) 2024-03-24T03:40:45.844 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:45 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5[48372]: 2024-03-24T03:40:45.635+0000 7fd9de157700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.415384+0000 front 2024-03-24T03:40:13.415477+0000 (oldest deadline 2024-03-24T03:40:38.115366+0000) 2024-03-24T03:40:45.844 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:45 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5[48372]: 2024-03-24T03:40:45.635+0000 7fd9de157700 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:18.115706+0000 front 2024-03-24T03:40:18.115685+0000 (oldest deadline 2024-03-24T03:40:44.015832+0000) 2024-03-24T03:40:45.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:45 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:45.533+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.182631+0000 front 2024-03-24T03:40:13.182383+0000 (oldest deadline 2024-03-24T03:40:37.882570+0000) 2024-03-24T03:40:45.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:45 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:45.533+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:18.383583+0000 front 2024-03-24T03:40:18.383626+0000 (oldest deadline 2024-03-24T03:40:43.083704+0000) 2024-03-24T03:40:45.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:45 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:45.624+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:40:45.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:45 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:45.624+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:40:46.090 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:45 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6[58136]: 2024-03-24T03:40:45.753+0000 7f8a3fa66700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:10.563069+0000 front 2024-03-24T03:40:10.563156+0000 (oldest deadline 2024-03-24T03:40:36.463161+0000) 2024-03-24T03:40:46.343 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:46 smithi107 podman[124606]: 2024-03-24 03:40:46.179191871 +0000 UTC m=+5.474294604 container died d7016c8c7fe072cdb55b813624a5068bcaa97dabdce7f82ef4f3761f25e2660f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T03:40:46.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:45 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:45.961+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:40:46.344 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:45 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:45.961+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:17.321155+0000 front 2024-03-24T03:40:17.321175+0000 (oldest deadline 2024-03-24T03:40:43.221253+0000) 2024-03-24T03:40:46.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:46 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:46.280+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:40:46.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:46 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:46.280+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:40:46.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:46 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:46.280+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.963619+0000 front 2024-03-24T03:40:21.963771+0000 (oldest deadline 2024-03-24T03:40:45.463653+0000) 2024-03-24T03:40:46.810 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:46 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:46.537+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:40:46.810 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:46 smithi107 podman[124606]: 2024-03-24 03:40:46.542679064 +0000 UTC m=+5.837781766 container cleanup d7016c8c7fe072cdb55b813624a5068bcaa97dabdce7f82ef4f3761f25e2660f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T03:40:46.810 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:46 smithi107 bash[124606]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5 2024-03-24T03:40:46.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:46 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:46.576+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.182631+0000 front 2024-03-24T03:40:13.182383+0000 (oldest deadline 2024-03-24T03:40:37.882570+0000) 2024-03-24T03:40:46.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:46 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:46.576+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:18.383583+0000 front 2024-03-24T03:40:18.383626+0000 (oldest deadline 2024-03-24T03:40:43.083704+0000) 2024-03-24T03:40:46.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:46 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:46.594+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:40:46.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:46 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:46.594+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:40:46.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:46 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:46.594+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:40:47.090 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:46 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6[58136]: 2024-03-24T03:40:46.797+0000 7f8a3fa66700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:10.563069+0000 front 2024-03-24T03:40:10.563156+0000 (oldest deadline 2024-03-24T03:40:36.463161+0000) 2024-03-24T03:40:47.093 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:46 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:46.931+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:40:47.093 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:46 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:46.931+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:17.321155+0000 front 2024-03-24T03:40:17.321175+0000 (oldest deadline 2024-03-24T03:40:43.221253+0000) 2024-03-24T03:40:47.093 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:46 smithi107 podman[124623]: 2024-03-24 03:40:46.810665069 +0000 UTC m=+0.625576040 container remove d7016c8c7fe072cdb55b813624a5068bcaa97dabdce7f82ef4f3761f25e2660f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T03:40:47.414 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:47 smithi107 podman[124719]: 2024-03-24 03:40:47.007059626 +0000 UTC m=+0.017806858 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T03:40:47.414 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:47 smithi107 podman[124719]: 2024-03-24 03:40:47.183211122 +0000 UTC m=+0.193958399 container create 1efa9061705da45415fa4871664e20bb5316a145adc7f32c04f06e32e8090289 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5-deactivate, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T03:40:47.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:47 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:47.251+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:40:47.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:47 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:47.251+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:40:47.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:47 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:47.251+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.963619+0000 front 2024-03-24T03:40:21.963771+0000 (oldest deadline 2024-03-24T03:40:45.463653+0000) 2024-03-24T03:40:47.704 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:47 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:47.571+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:40:47.704 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:47 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:47.571+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:40:47.704 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:47 smithi107 podman[124719]: 2024-03-24 03:40:47.41427688 +0000 UTC m=+0.425024102 container init 1efa9061705da45415fa4871664e20bb5316a145adc7f32c04f06e32e8090289 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T03:40:47.705 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:47 smithi107 podman[124719]: 2024-03-24 03:40:47.418185613 +0000 UTC m=+0.428932838 container start 1efa9061705da45415fa4871664e20bb5316a145adc7f32c04f06e32e8090289 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-24T03:40:47.705 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:47 smithi107 podman[124719]: 2024-03-24 03:40:47.514310955 +0000 UTC m=+0.525058179 container attach 1efa9061705da45415fa4871664e20bb5316a145adc7f32c04f06e32e8090289 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-24T03:40:47.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:47 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:47.598+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.182631+0000 front 2024-03-24T03:40:13.182383+0000 (oldest deadline 2024-03-24T03:40:37.882570+0000) 2024-03-24T03:40:47.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:47 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:47.598+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:18.383583+0000 front 2024-03-24T03:40:18.383626+0000 (oldest deadline 2024-03-24T03:40:43.083704+0000) 2024-03-24T03:40:47.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:47 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:47.615+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:40:47.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:47 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:47.615+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:40:47.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:47 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:47.615+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:40:47.957 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 24 03:40:47 smithi107 podman[124719]: 2024-03-24 03:40:47.705465659 +0000 UTC m=+0.716212893 container died 1efa9061705da45415fa4871664e20bb5316a145adc7f32c04f06e32e8090289 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-24T03:40:48.090 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:47 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6[58136]: 2024-03-24T03:40:47.759+0000 7f8a3fa66700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:10.563069+0000 front 2024-03-24T03:40:10.563156+0000 (oldest deadline 2024-03-24T03:40:36.463161+0000) 2024-03-24T03:40:48.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:47 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:47.957+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:40:48.344 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:47 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:47.957+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:17.321155+0000 front 2024-03-24T03:40:17.321175+0000 (oldest deadline 2024-03-24T03:40:43.221253+0000) 2024-03-24T03:40:48.395 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.5.service' 2024-03-24T03:40:48.435 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:40:48.436 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-24T03:40:48.436 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-24T03:40:48.436 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.6 2024-03-24T03:40:48.590 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:48 smithi078 systemd[1]: Stopping Ceph osd.6 for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T03:40:48.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:48.281+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:40:48.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:48.281+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:40:48.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:48.281+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.963619+0000 front 2024-03-24T03:40:21.963771+0000 (oldest deadline 2024-03-24T03:40:45.463653+0000) 2024-03-24T03:40:48.643 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:48.589+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:40:48.643 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:48.589+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:40:48.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:48 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:48.597+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.182631+0000 front 2024-03-24T03:40:13.182383+0000 (oldest deadline 2024-03-24T03:40:37.882570+0000) 2024-03-24T03:40:48.903 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:48 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:48.597+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:18.383583+0000 front 2024-03-24T03:40:18.383626+0000 (oldest deadline 2024-03-24T03:40:43.083704+0000) 2024-03-24T03:40:48.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:48 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:48.636+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:40:48.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:48 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:48.636+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:40:48.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:48 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:48.636+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:40:49.090 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6[58136]: 2024-03-24T03:40:48.752+0000 7f8a43472700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:40:49.091 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6[58136]: 2024-03-24T03:40:48.753+0000 7f8a43472700 -1 osd.6 161 *** Got signal Terminated *** 2024-03-24T03:40:49.091 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6[58136]: 2024-03-24T03:40:48.753+0000 7f8a43472700 -1 osd.6 161 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T03:40:49.091 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6[58136]: 2024-03-24T03:40:48.799+0000 7f8a3fa66700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:10.563069+0000 front 2024-03-24T03:40:10.563156+0000 (oldest deadline 2024-03-24T03:40:36.463161+0000) 2024-03-24T03:40:49.091 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:48 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6[58136]: 2024-03-24T03:40:48.799+0000 7f8a3fa66700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:17.564255+0000 front 2024-03-24T03:40:17.564287+0000 (oldest deadline 2024-03-24T03:40:48.165203+0000) 2024-03-24T03:40:49.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:48.912+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:40:49.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:48 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:48.912+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:17.321155+0000 front 2024-03-24T03:40:17.321175+0000 (oldest deadline 2024-03-24T03:40:43.221253+0000) 2024-03-24T03:40:49.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:49.292+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:40:49.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:49.292+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:40:49.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:49.292+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.963619+0000 front 2024-03-24T03:40:21.963771+0000 (oldest deadline 2024-03-24T03:40:45.463653+0000) 2024-03-24T03:40:49.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:49 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:49.564+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:40:49.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:49 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:49.564+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:40:49.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:49 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:49.571+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.182631+0000 front 2024-03-24T03:40:13.182383+0000 (oldest deadline 2024-03-24T03:40:37.882570+0000) 2024-03-24T03:40:49.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:49 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:49.571+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:18.383583+0000 front 2024-03-24T03:40:18.383626+0000 (oldest deadline 2024-03-24T03:40:43.083704+0000) 2024-03-24T03:40:49.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:49 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:49.605+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:40:49.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:49 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:49.605+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:40:49.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:49 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:49.605+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:40:50.320 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6[58136]: 2024-03-24T03:40:49.845+0000 7f8a3fa66700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:10.563069+0000 front 2024-03-24T03:40:10.563156+0000 (oldest deadline 2024-03-24T03:40:36.463161+0000) 2024-03-24T03:40:50.320 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:49 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6[58136]: 2024-03-24T03:40:49.845+0000 7f8a3fa66700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:17.564255+0000 front 2024-03-24T03:40:17.564287+0000 (oldest deadline 2024-03-24T03:40:48.165203+0000) 2024-03-24T03:40:50.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:49 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:49.915+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:40:50.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:49 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:49.915+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:17.321155+0000 front 2024-03-24T03:40:17.321175+0000 (oldest deadline 2024-03-24T03:40:43.221253+0000) 2024-03-24T03:40:50.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:50.319+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:40:50.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:50.319+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:40:50.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:50.319+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.963619+0000 front 2024-03-24T03:40:21.963771+0000 (oldest deadline 2024-03-24T03:40:45.463653+0000) 2024-03-24T03:40:50.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:50 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:50.593+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.182631+0000 front 2024-03-24T03:40:13.182383+0000 (oldest deadline 2024-03-24T03:40:37.882570+0000) 2024-03-24T03:40:50.903 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:50 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:50.593+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:18.383583+0000 front 2024-03-24T03:40:18.383626+0000 (oldest deadline 2024-03-24T03:40:43.083704+0000) 2024-03-24T03:40:50.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:50 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:50.626+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:40:50.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:50 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:50.626+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:40:50.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:50 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:50.626+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:40:50.932 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:50 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:50.593+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:40:50.932 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:50 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:50.593+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:40:51.090 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6[58136]: 2024-03-24T03:40:50.833+0000 7f8a3fa66700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:10.563069+0000 front 2024-03-24T03:40:10.563156+0000 (oldest deadline 2024-03-24T03:40:36.463161+0000) 2024-03-24T03:40:51.091 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:50 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6[58136]: 2024-03-24T03:40:50.833+0000 7f8a3fa66700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:17.564255+0000 front 2024-03-24T03:40:17.564287+0000 (oldest deadline 2024-03-24T03:40:48.165203+0000) 2024-03-24T03:40:51.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:50 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:50.932+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:40:51.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:50 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:50.932+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:17.321155+0000 front 2024-03-24T03:40:17.321175+0000 (oldest deadline 2024-03-24T03:40:43.221253+0000) 2024-03-24T03:40:51.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:51 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:51.289+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:40:51.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:51 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:51.289+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:40:51.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:51 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:51.289+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.963619+0000 front 2024-03-24T03:40:21.963771+0000 (oldest deadline 2024-03-24T03:40:45.463653+0000) 2024-03-24T03:40:51.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:51 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:51.568+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:40:51.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:51 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:51.568+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:40:51.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:51 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:51.603+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.182631+0000 front 2024-03-24T03:40:13.182383+0000 (oldest deadline 2024-03-24T03:40:37.882570+0000) 2024-03-24T03:40:51.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:51 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:51.603+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:18.383583+0000 front 2024-03-24T03:40:18.383626+0000 (oldest deadline 2024-03-24T03:40:43.083704+0000) 2024-03-24T03:40:51.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:51 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:51.624+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:40:51.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:51 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:51.624+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:40:51.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:51 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:51.624+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:40:51.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:51 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:51.624+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:40:52.090 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:51 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6[58136]: 2024-03-24T03:40:51.795+0000 7f8a3fa66700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:10.563069+0000 front 2024-03-24T03:40:10.563156+0000 (oldest deadline 2024-03-24T03:40:36.463161+0000) 2024-03-24T03:40:52.091 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:51 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6[58136]: 2024-03-24T03:40:51.795+0000 7f8a3fa66700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:17.564255+0000 front 2024-03-24T03:40:17.564287+0000 (oldest deadline 2024-03-24T03:40:48.165203+0000) 2024-03-24T03:40:52.091 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:51 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6[58136]: 2024-03-24T03:40:51.795+0000 7f8a3fa66700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:28.165663+0000 front 2024-03-24T03:40:28.165623+0000 (oldest deadline 2024-03-24T03:40:51.665724+0000) 2024-03-24T03:40:52.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:51 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:51.887+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:40:52.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:51 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:51.887+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:17.321155+0000 front 2024-03-24T03:40:17.321175+0000 (oldest deadline 2024-03-24T03:40:43.221253+0000) 2024-03-24T03:40:52.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:52 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:52.307+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:40:52.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:52 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:52.307+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:40:52.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:52 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:52.307+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.963619+0000 front 2024-03-24T03:40:21.963771+0000 (oldest deadline 2024-03-24T03:40:45.463653+0000) 2024-03-24T03:40:52.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:52 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:52.583+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:40:52.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:52 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:52.583+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:40:52.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:52 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:52.569+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.182631+0000 front 2024-03-24T03:40:13.182383+0000 (oldest deadline 2024-03-24T03:40:37.882570+0000) 2024-03-24T03:40:52.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:52 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:52.569+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:18.383583+0000 front 2024-03-24T03:40:18.383626+0000 (oldest deadline 2024-03-24T03:40:43.083704+0000) 2024-03-24T03:40:52.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:52 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:52.601+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:40:52.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:52 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:52.601+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:40:52.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:52 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:52.601+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:40:52.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:52 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:52.601+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:40:53.090 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:52 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6[58136]: 2024-03-24T03:40:52.825+0000 7f8a3fa66700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:10.563069+0000 front 2024-03-24T03:40:10.563156+0000 (oldest deadline 2024-03-24T03:40:36.463161+0000) 2024-03-24T03:40:53.091 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:52 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6[58136]: 2024-03-24T03:40:52.825+0000 7f8a3fa66700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:17.564255+0000 front 2024-03-24T03:40:17.564287+0000 (oldest deadline 2024-03-24T03:40:48.165203+0000) 2024-03-24T03:40:53.091 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:52 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6[58136]: 2024-03-24T03:40:52.825+0000 7f8a3fa66700 -1 osd.6 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:28.165663+0000 front 2024-03-24T03:40:28.165623+0000 (oldest deadline 2024-03-24T03:40:51.665724+0000) 2024-03-24T03:40:53.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:52 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:52.925+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:40:53.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:52 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:52.925+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:17.321155+0000 front 2024-03-24T03:40:17.321175+0000 (oldest deadline 2024-03-24T03:40:43.221253+0000) 2024-03-24T03:40:53.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:52 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:52.925+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:26.722380+0000 front 2024-03-24T03:40:26.722425+0000 (oldest deadline 2024-03-24T03:40:52.622419+0000) 2024-03-24T03:40:53.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:53.279+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:40:53.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:53.279+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:40:53.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:53.279+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.963619+0000 front 2024-03-24T03:40:21.963771+0000 (oldest deadline 2024-03-24T03:40:45.463653+0000) 2024-03-24T03:40:53.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:53 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:53.279+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:27.164667+0000 front 2024-03-24T03:40:27.164627+0000 (oldest deadline 2024-03-24T03:40:53.064884+0000) 2024-03-24T03:40:53.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:53 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:53.570+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:40:53.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:53 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:53.570+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:40:53.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:53 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:53.570+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:40:53.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:53 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:53.612+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.182631+0000 front 2024-03-24T03:40:13.182383+0000 (oldest deadline 2024-03-24T03:40:37.882570+0000) 2024-03-24T03:40:53.903 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:53 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:53.612+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:18.383583+0000 front 2024-03-24T03:40:18.383626+0000 (oldest deadline 2024-03-24T03:40:43.083704+0000) 2024-03-24T03:40:53.903 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:53 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:53.612+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.485667+0000 front 2024-03-24T03:40:29.485706+0000 (oldest deadline 2024-03-24T03:40:53.585733+0000) 2024-03-24T03:40:53.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:53 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:53.621+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:40:53.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:53 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:53.621+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:40:53.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:53 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:53.622+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:40:53.904 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:53 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:53.622+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:40:54.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:53 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:53.881+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:40:54.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:53 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:53.881+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:17.321155+0000 front 2024-03-24T03:40:17.321175+0000 (oldest deadline 2024-03-24T03:40:43.221253+0000) 2024-03-24T03:40:54.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:53 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:53.881+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:26.722380+0000 front 2024-03-24T03:40:26.722425+0000 (oldest deadline 2024-03-24T03:40:52.622419+0000) 2024-03-24T03:40:54.486 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:54 smithi078 podman[142268]: 2024-03-24 03:40:54.23017169 +0000 UTC m=+5.593855278 container died fbf7e964a41f4d21ea93560a989280611de74214bbadb7fed30f5432310a5d8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-24T03:40:54.486 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:54 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:54.278+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:40:54.486 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:54 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:54.278+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:40:54.486 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:54 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:54.278+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.963619+0000 front 2024-03-24T03:40:21.963771+0000 (oldest deadline 2024-03-24T03:40:45.463653+0000) 2024-03-24T03:40:54.486 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:54 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:54.278+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:27.164667+0000 front 2024-03-24T03:40:27.164627+0000 (oldest deadline 2024-03-24T03:40:53.064884+0000) 2024-03-24T03:40:54.744 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:54 smithi078 podman[142268]: 2024-03-24 03:40:54.568996294 +0000 UTC m=+5.932679892 container cleanup fbf7e964a41f4d21ea93560a989280611de74214bbadb7fed30f5432310a5d8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T03:40:54.745 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:54 smithi078 bash[142268]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6 2024-03-24T03:40:54.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:54.557+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:40:54.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:54.557+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:40:54.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:54.557+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:40:54.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:54 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:54.622+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.182631+0000 front 2024-03-24T03:40:13.182383+0000 (oldest deadline 2024-03-24T03:40:37.882570+0000) 2024-03-24T03:40:54.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:54 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:54.622+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:18.383583+0000 front 2024-03-24T03:40:18.383626+0000 (oldest deadline 2024-03-24T03:40:43.083704+0000) 2024-03-24T03:40:54.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:54 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:54.622+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.485667+0000 front 2024-03-24T03:40:29.485706+0000 (oldest deadline 2024-03-24T03:40:53.585733+0000) 2024-03-24T03:40:54.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:54 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:54.647+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:40:54.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:54 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:54.647+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:40:54.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:54 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:54.647+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:40:54.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:54 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:54.647+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:40:55.090 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:54 smithi078 podman[142287]: 2024-03-24 03:40:54.745209005 +0000 UTC m=+0.510334222 container remove fbf7e964a41f4d21ea93560a989280611de74214bbadb7fed30f5432310a5d8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux ) 2024-03-24T03:40:55.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:54.921+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:40:55.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:54.921+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:17.321155+0000 front 2024-03-24T03:40:17.321175+0000 (oldest deadline 2024-03-24T03:40:43.221253+0000) 2024-03-24T03:40:55.344 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:54 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:54.921+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:26.722380+0000 front 2024-03-24T03:40:26.722425+0000 (oldest deadline 2024-03-24T03:40:52.622419+0000) 2024-03-24T03:40:55.514 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:55 smithi078 podman[142382]: 2024-03-24 03:40:55.179375012 +0000 UTC m=+0.114491342 container create ca69baadd0832e789dc9f446b4d2b5042dc70d2cc4ef890491223403a42f538b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux ) 2024-03-24T03:40:55.515 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:55 smithi078 podman[142382]: 2024-03-24 03:40:55.083602841 +0000 UTC m=+0.018719173 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T03:40:55.515 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:55 smithi078 podman[142382]: 2024-03-24 03:40:55.415483447 +0000 UTC m=+0.350599780 container init ca69baadd0832e789dc9f446b4d2b5042dc70d2cc4ef890491223403a42f538b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-24T03:40:55.515 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:55 smithi078 podman[142382]: 2024-03-24 03:40:55.418670013 +0000 UTC m=+0.353786341 container start ca69baadd0832e789dc9f446b4d2b5042dc70d2cc4ef890491223403a42f538b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-24T03:40:55.515 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:55 smithi078 podman[142382]: 2024-03-24 03:40:55.514935147 +0000 UTC m=+0.450051485 container attach ca69baadd0832e789dc9f446b4d2b5042dc70d2cc4ef890491223403a42f538b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T03:40:55.515 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:55 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:55.237+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:40:55.515 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:55 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:55.237+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:40:55.515 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:55 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:55.237+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.963619+0000 front 2024-03-24T03:40:21.963771+0000 (oldest deadline 2024-03-24T03:40:45.463653+0000) 2024-03-24T03:40:55.516 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:55 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:55.237+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:27.164667+0000 front 2024-03-24T03:40:27.164627+0000 (oldest deadline 2024-03-24T03:40:53.064884+0000) 2024-03-24T03:40:55.840 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 24 03:40:55 smithi078 podman[142382]: 2024-03-24 03:40:55.699076378 +0000 UTC m=+0.634192707 container died ca69baadd0832e789dc9f446b4d2b5042dc70d2cc4ef890491223403a42f538b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-24T03:40:55.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:55.561+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:40:55.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:55.561+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:40:55.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:55.561+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:40:56.152 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:55 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:55.664+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.182631+0000 front 2024-03-24T03:40:13.182383+0000 (oldest deadline 2024-03-24T03:40:37.882570+0000) 2024-03-24T03:40:56.152 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:55 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:55.664+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:18.383583+0000 front 2024-03-24T03:40:18.383626+0000 (oldest deadline 2024-03-24T03:40:43.083704+0000) 2024-03-24T03:40:56.153 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:55 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:55.664+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.485667+0000 front 2024-03-24T03:40:29.485706+0000 (oldest deadline 2024-03-24T03:40:53.585733+0000) 2024-03-24T03:40:56.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:55 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:55.659+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:40:56.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:55 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:55.659+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:40:56.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:55 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:55.659+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:40:56.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:55 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:55.659+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:40:56.340 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:56 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:56.260+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:40:56.341 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:56 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:56.260+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:40:56.341 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:56 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:56.260+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.963619+0000 front 2024-03-24T03:40:21.963771+0000 (oldest deadline 2024-03-24T03:40:45.463653+0000) 2024-03-24T03:40:56.341 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:56 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:56.260+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:27.164667+0000 front 2024-03-24T03:40:27.164627+0000 (oldest deadline 2024-03-24T03:40:53.064884+0000) 2024-03-24T03:40:56.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:55.914+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:40:56.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:55.914+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:17.321155+0000 front 2024-03-24T03:40:17.321175+0000 (oldest deadline 2024-03-24T03:40:43.221253+0000) 2024-03-24T03:40:56.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:55 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:55.914+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:26.722380+0000 front 2024-03-24T03:40:26.722425+0000 (oldest deadline 2024-03-24T03:40:52.622419+0000) 2024-03-24T03:40:56.413 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.6.service' 2024-03-24T03:40:56.455 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:40:56.455 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-24T03:40:56.455 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-24T03:40:56.455 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.7 2024-03-24T03:40:56.822 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:56 smithi106 systemd[1]: Stopping Ceph osd.7 for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T03:40:56.822 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:56 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:56.622+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.182631+0000 front 2024-03-24T03:40:13.182383+0000 (oldest deadline 2024-03-24T03:40:37.882570+0000) 2024-03-24T03:40:56.823 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:56 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:56.622+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:18.383583+0000 front 2024-03-24T03:40:18.383626+0000 (oldest deadline 2024-03-24T03:40:43.083704+0000) 2024-03-24T03:40:56.823 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:56 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:56.622+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.485667+0000 front 2024-03-24T03:40:29.485706+0000 (oldest deadline 2024-03-24T03:40:53.585733+0000) 2024-03-24T03:40:56.823 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:56 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:56.663+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:40:56.823 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:56 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:56.663+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:40:56.823 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:56 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:56.663+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:40:56.823 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:56 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:56.663+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:40:56.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:56 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:56.561+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:40:56.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:56 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:56.561+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:40:56.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:56 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:56.561+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:40:57.152 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:56 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:56.821+0000 7f655e500700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:40:57.152 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:56 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:56.821+0000 7f655e500700 -1 osd.7 161 *** Got signal Terminated *** 2024-03-24T03:40:57.152 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:56 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:56.821+0000 7f655e500700 -1 osd.7 161 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T03:40:57.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:56 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:56.948+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:40:57.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:56 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:56.948+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:17.321155+0000 front 2024-03-24T03:40:17.321175+0000 (oldest deadline 2024-03-24T03:40:43.221253+0000) 2024-03-24T03:40:57.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:56 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:56.948+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:26.722380+0000 front 2024-03-24T03:40:26.722425+0000 (oldest deadline 2024-03-24T03:40:52.622419+0000) 2024-03-24T03:40:57.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:57 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:57.286+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:40:57.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:57 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:57.287+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:40:57.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:57 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:57.287+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.963619+0000 front 2024-03-24T03:40:21.963771+0000 (oldest deadline 2024-03-24T03:40:45.463653+0000) 2024-03-24T03:40:57.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:57 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:57.287+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:27.164667+0000 front 2024-03-24T03:40:27.164627+0000 (oldest deadline 2024-03-24T03:40:53.064884+0000) 2024-03-24T03:40:57.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:57 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:57.521+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:40:57.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:57 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:57.521+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:40:57.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:57 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:57.521+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:40:57.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:57 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:57.641+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.182631+0000 front 2024-03-24T03:40:13.182383+0000 (oldest deadline 2024-03-24T03:40:37.882570+0000) 2024-03-24T03:40:57.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:57 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:57.641+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:18.383583+0000 front 2024-03-24T03:40:18.383626+0000 (oldest deadline 2024-03-24T03:40:43.083704+0000) 2024-03-24T03:40:57.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:57 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:57.641+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.485667+0000 front 2024-03-24T03:40:29.485706+0000 (oldest deadline 2024-03-24T03:40:53.585733+0000) 2024-03-24T03:40:57.902 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:57 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:57.691+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:40:57.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:57 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:57.691+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:40:57.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:57 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:57.691+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:40:57.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:57 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:57.691+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:40:58.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:57 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:57.968+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:40:58.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:57 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:57.968+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:17.321155+0000 front 2024-03-24T03:40:17.321175+0000 (oldest deadline 2024-03-24T03:40:43.221253+0000) 2024-03-24T03:40:58.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:57 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:57.968+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:26.722380+0000 front 2024-03-24T03:40:26.722425+0000 (oldest deadline 2024-03-24T03:40:52.622419+0000) 2024-03-24T03:40:58.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:58 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:58.320+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:40:58.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:58 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:58.320+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:40:58.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:58 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:58.320+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.963619+0000 front 2024-03-24T03:40:21.963771+0000 (oldest deadline 2024-03-24T03:40:45.463653+0000) 2024-03-24T03:40:58.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:58 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:58.320+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:27.164667+0000 front 2024-03-24T03:40:27.164627+0000 (oldest deadline 2024-03-24T03:40:53.064884+0000) 2024-03-24T03:40:58.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:58.549+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:40:58.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:58.549+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:40:58.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:58.549+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:40:59.152 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:58 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:58.661+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.182631+0000 front 2024-03-24T03:40:13.182383+0000 (oldest deadline 2024-03-24T03:40:37.882570+0000) 2024-03-24T03:40:59.152 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:58 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:58.662+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:18.383583+0000 front 2024-03-24T03:40:18.383626+0000 (oldest deadline 2024-03-24T03:40:43.083704+0000) 2024-03-24T03:40:59.152 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:58 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:58.662+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.485667+0000 front 2024-03-24T03:40:29.485706+0000 (oldest deadline 2024-03-24T03:40:53.585733+0000) 2024-03-24T03:40:59.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:58 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:58.681+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:40:59.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:58 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:58.681+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:40:59.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:58 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:58.681+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:40:59.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:58 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:58.681+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:40:59.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:58.989+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:40:59.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:58.989+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:17.321155+0000 front 2024-03-24T03:40:17.321175+0000 (oldest deadline 2024-03-24T03:40:43.221253+0000) 2024-03-24T03:40:59.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:58 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:58.989+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:26.722380+0000 front 2024-03-24T03:40:26.722425+0000 (oldest deadline 2024-03-24T03:40:52.622419+0000) 2024-03-24T03:40:59.840 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:59.369+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:40:59.840 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:59.369+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:40:59.841 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:59.369+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.963619+0000 front 2024-03-24T03:40:21.963771+0000 (oldest deadline 2024-03-24T03:40:45.463653+0000) 2024-03-24T03:40:59.841 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:40:59 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:40:59.369+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:27.164667+0000 front 2024-03-24T03:40:27.164627+0000 (oldest deadline 2024-03-24T03:40:53.064884+0000) 2024-03-24T03:40:59.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:59 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:59.543+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:40:59.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:59 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:59.543+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:40:59.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:40:59 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:40:59.543+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:40:59.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:59 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:59.659+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.182631+0000 front 2024-03-24T03:40:13.182383+0000 (oldest deadline 2024-03-24T03:40:37.882570+0000) 2024-03-24T03:40:59.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:59 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:59.659+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:18.383583+0000 front 2024-03-24T03:40:18.383626+0000 (oldest deadline 2024-03-24T03:40:43.083704+0000) 2024-03-24T03:40:59.903 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:59 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:59.659+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.485667+0000 front 2024-03-24T03:40:29.485706+0000 (oldest deadline 2024-03-24T03:40:53.585733+0000) 2024-03-24T03:40:59.903 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:40:59 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:40:59.659+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:33.586166+0000 front 2024-03-24T03:40:33.586190+0000 (oldest deadline 2024-03-24T03:40:58.886403+0000) 2024-03-24T03:40:59.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:59 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:59.647+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:40:59.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:59 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:59.647+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:40:59.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:59 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:59.647+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:40:59.903 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:40:59 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:40:59.647+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:41:00.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:59 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:59.984+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:41:00.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:59 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:59.984+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:17.321155+0000 front 2024-03-24T03:40:17.321175+0000 (oldest deadline 2024-03-24T03:40:43.221253+0000) 2024-03-24T03:41:00.344 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:40:59 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:40:59.984+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:26.722380+0000 front 2024-03-24T03:40:26.722425+0000 (oldest deadline 2024-03-24T03:40:52.622419+0000) 2024-03-24T03:41:00.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:00 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:00.334+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:41:00.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:00 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:00.334+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:41:00.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:00 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:00.334+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.963619+0000 front 2024-03-24T03:40:21.963771+0000 (oldest deadline 2024-03-24T03:40:45.463653+0000) 2024-03-24T03:41:00.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:00 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:00.334+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:27.164667+0000 front 2024-03-24T03:40:27.164627+0000 (oldest deadline 2024-03-24T03:40:53.064884+0000) 2024-03-24T03:41:00.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:00 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:00.584+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:41:00.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:00 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:00.584+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:41:00.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:00 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:00.584+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:41:01.109 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:41:00 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:41:00.700+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.182631+0000 front 2024-03-24T03:40:13.182383+0000 (oldest deadline 2024-03-24T03:40:37.882570+0000) 2024-03-24T03:41:01.109 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:41:00 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:41:00.700+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:18.383583+0000 front 2024-03-24T03:40:18.383626+0000 (oldest deadline 2024-03-24T03:40:43.083704+0000) 2024-03-24T03:41:01.110 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:41:00 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:41:00.700+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.485667+0000 front 2024-03-24T03:40:29.485706+0000 (oldest deadline 2024-03-24T03:40:53.585733+0000) 2024-03-24T03:41:01.110 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:41:00 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:41:00.700+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:33.586166+0000 front 2024-03-24T03:40:33.586190+0000 (oldest deadline 2024-03-24T03:40:58.886403+0000) 2024-03-24T03:41:01.110 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:00 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:00.656+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:41:01.110 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:00 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:00.656+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:41:01.110 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:00 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:00.656+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:41:01.110 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:00 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:00.656+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:41:01.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:01 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:01.019+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:41:01.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:01 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:01.019+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:17.321155+0000 front 2024-03-24T03:40:17.321175+0000 (oldest deadline 2024-03-24T03:40:43.221253+0000) 2024-03-24T03:41:01.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:01 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:01.019+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:26.722380+0000 front 2024-03-24T03:40:26.722425+0000 (oldest deadline 2024-03-24T03:40:52.622419+0000) 2024-03-24T03:41:01.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:01 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:01.291+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:41:01.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:01 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:01.291+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:41:01.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:01 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:01.291+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.963619+0000 front 2024-03-24T03:40:21.963771+0000 (oldest deadline 2024-03-24T03:40:45.463653+0000) 2024-03-24T03:41:01.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:01 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:01.291+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:27.164667+0000 front 2024-03-24T03:40:27.164627+0000 (oldest deadline 2024-03-24T03:40:53.064884+0000) 2024-03-24T03:41:01.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:01 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:01.544+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:41:01.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:01 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:01.544+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:41:01.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:01 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:01.544+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:41:02.153 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:41:01 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:41:01.679+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.182631+0000 front 2024-03-24T03:40:13.182383+0000 (oldest deadline 2024-03-24T03:40:37.882570+0000) 2024-03-24T03:41:02.153 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:41:01 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:41:01.679+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:18.383583+0000 front 2024-03-24T03:40:18.383626+0000 (oldest deadline 2024-03-24T03:40:43.083704+0000) 2024-03-24T03:41:02.153 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:41:01 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:41:01.679+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.485667+0000 front 2024-03-24T03:40:29.485706+0000 (oldest deadline 2024-03-24T03:40:53.585733+0000) 2024-03-24T03:41:02.153 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:41:01 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7[55083]: 2024-03-24T03:41:01.679+0000 7f655aaf4700 -1 osd.7 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:33.586166+0000 front 2024-03-24T03:40:33.586190+0000 (oldest deadline 2024-03-24T03:40:58.886403+0000) 2024-03-24T03:41:02.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:01 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:01.677+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:41:02.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:01 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:01.677+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:41:02.154 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:01 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:01.677+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:41:02.154 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:01 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:01.677+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:41:02.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:02 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:02.029+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:41:02.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:02 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:02.029+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:17.321155+0000 front 2024-03-24T03:40:17.321175+0000 (oldest deadline 2024-03-24T03:40:43.221253+0000) 2024-03-24T03:41:02.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:02 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:02.029+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:26.722380+0000 front 2024-03-24T03:40:26.722425+0000 (oldest deadline 2024-03-24T03:40:52.622419+0000) 2024-03-24T03:41:02.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:02 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:02.029+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:36.024265+0000 front 2024-03-24T03:40:36.024248+0000 (oldest deadline 2024-03-24T03:41:01.924103+0000) 2024-03-24T03:41:02.489 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:41:02 smithi106 podman[122054]: 2024-03-24 03:41:02.191932544 +0000 UTC m=+5.546047787 container died 77a85d20b30e1fd4e65dc5c4e2baca702fd06ad7272926157d285406df6b4c46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0) 2024-03-24T03:41:02.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:02 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:02.336+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:41:02.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:02 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:02.336+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:41:02.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:02 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:02.336+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.963619+0000 front 2024-03-24T03:40:21.963771+0000 (oldest deadline 2024-03-24T03:40:45.463653+0000) 2024-03-24T03:41:02.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:02 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:02.336+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:27.164667+0000 front 2024-03-24T03:40:27.164627+0000 (oldest deadline 2024-03-24T03:40:53.064884+0000) 2024-03-24T03:41:02.839 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:41:02 smithi106 podman[122054]: 2024-03-24 03:41:02.589032713 +0000 UTC m=+5.943147951 container cleanup 77a85d20b30e1fd4e65dc5c4e2baca702fd06ad7272926157d285406df6b4c46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T03:41:02.840 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:41:02 smithi106 bash[122054]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7 2024-03-24T03:41:02.840 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:02 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:02.703+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:41:02.840 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:02 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:02.703+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:41:02.840 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:02 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:02.703+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:41:02.840 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:02 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:02.703+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:41:02.841 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:02 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:02.703+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:36.470431+0000 front 2024-03-24T03:40:36.470487+0000 (oldest deadline 2024-03-24T03:41:02.271075+0000) 2024-03-24T03:41:02.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:02 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:02.539+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:41:02.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:02 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:02.539+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:41:02.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:02 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:02.539+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:41:03.148 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:41:02 smithi106 podman[122084]: 2024-03-24 03:41:02.840408075 +0000 UTC m=+0.648005194 container remove 77a85d20b30e1fd4e65dc5c4e2baca702fd06ad7272926157d285406df6b4c46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T03:41:03.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:03 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:03.049+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:41:03.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:03 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:03.049+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:17.321155+0000 front 2024-03-24T03:40:17.321175+0000 (oldest deadline 2024-03-24T03:40:43.221253+0000) 2024-03-24T03:41:03.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:03 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:03.049+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:26.722380+0000 front 2024-03-24T03:40:26.722425+0000 (oldest deadline 2024-03-24T03:40:52.622419+0000) 2024-03-24T03:41:03.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:03 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:03.049+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:36.024265+0000 front 2024-03-24T03:40:36.024248+0000 (oldest deadline 2024-03-24T03:41:01.924103+0000) 2024-03-24T03:41:03.402 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:41:03 smithi106 podman[122176]: 2024-03-24 03:41:03.052358889 +0000 UTC m=+0.017760228 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T03:41:03.402 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:41:03 smithi106 podman[122176]: 2024-03-24 03:41:03.149764932 +0000 UTC m=+0.115166257 container create 88ace296fa6d0b1091f79e686899e96bc1cae408094300cc03e4f02bf9de8e77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-24T03:41:03.403 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:41:03 smithi106 podman[122176]: 2024-03-24 03:41:03.377202103 +0000 UTC m=+0.342603425 container init 88ace296fa6d0b1091f79e686899e96bc1cae408094300cc03e4f02bf9de8e77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-24T03:41:03.403 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:41:03 smithi106 podman[122176]: 2024-03-24 03:41:03.38049246 +0000 UTC m=+0.345893783 container start 88ace296fa6d0b1091f79e686899e96bc1cae408094300cc03e4f02bf9de8e77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-24T03:41:03.670 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:41:03 smithi106 podman[122176]: 2024-03-24 03:41:03.494395132 +0000 UTC m=+0.459796474 container attach 88ace296fa6d0b1091f79e686899e96bc1cae408094300cc03e4f02bf9de8e77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0) 2024-03-24T03:41:03.840 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:03.360+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:41:03.841 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:03.360+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:41:03.841 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:03.360+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.963619+0000 front 2024-03-24T03:40:21.963771+0000 (oldest deadline 2024-03-24T03:40:45.463653+0000) 2024-03-24T03:41:03.841 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:03.360+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:27.164667+0000 front 2024-03-24T03:40:27.164627+0000 (oldest deadline 2024-03-24T03:40:53.064884+0000) 2024-03-24T03:41:03.841 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:03 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:03.360+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.166393+0000 front 2024-03-24T03:40:37.166530+0000 (oldest deadline 2024-03-24T03:41:03.066159+0000) 2024-03-24T03:41:03.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:03 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:03.499+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:41:03.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:03 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:03.499+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:41:03.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:03 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:03.499+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:41:03.988 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 24 03:41:03 smithi106 podman[122176]: 2024-03-24 03:41:03.67168088 +0000 UTC m=+0.637082217 container died 88ace296fa6d0b1091f79e686899e96bc1cae408094300cc03e4f02bf9de8e77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-24T03:41:03.988 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:03.688+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:41:03.988 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:03.688+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:41:03.988 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:03.688+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:41:03.989 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:03.688+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:41:03.989 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:03 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:03.688+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:36.470431+0000 front 2024-03-24T03:40:36.470487+0000 (oldest deadline 2024-03-24T03:41:02.271075+0000) 2024-03-24T03:41:04.332 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.7.service' 2024-03-24T03:41:04.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:04 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:04.068+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:41:04.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:04 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:04.068+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:17.321155+0000 front 2024-03-24T03:40:17.321175+0000 (oldest deadline 2024-03-24T03:40:43.221253+0000) 2024-03-24T03:41:04.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:04 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:04.068+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:26.722380+0000 front 2024-03-24T03:40:26.722425+0000 (oldest deadline 2024-03-24T03:40:52.622419+0000) 2024-03-24T03:41:04.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:04 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:04.068+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:36.024265+0000 front 2024-03-24T03:40:36.024248+0000 (oldest deadline 2024-03-24T03:41:01.924103+0000) 2024-03-24T03:41:04.376 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:41:04.376 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-24T03:41:04.376 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-24T03:41:04.377 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.8 2024-03-24T03:41:04.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:04.326+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:41:04.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:04.326+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:41:04.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:04.326+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.963619+0000 front 2024-03-24T03:40:21.963771+0000 (oldest deadline 2024-03-24T03:40:45.463653+0000) 2024-03-24T03:41:04.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:04.326+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:27.164667+0000 front 2024-03-24T03:40:27.164627+0000 (oldest deadline 2024-03-24T03:40:53.064884+0000) 2024-03-24T03:41:04.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:04 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:04.326+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.166393+0000 front 2024-03-24T03:40:37.166530+0000 (oldest deadline 2024-03-24T03:41:03.066159+0000) 2024-03-24T03:41:04.702 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:04 smithi107 systemd[1]: Stopping Ceph osd.8 for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T03:41:04.702 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:04 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:04.535+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:41:04.702 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:04 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:04.535+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:41:04.703 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:04 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:04.535+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:41:04.703 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:04 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:04.535+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.746407+0000 front 2024-03-24T03:40:37.746383+0000 (oldest deadline 2024-03-24T03:41:03.646218+0000) 2024-03-24T03:41:05.024 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:04 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:04.702+0000 7f9d5f9ee700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:41:05.025 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:04 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:04.702+0000 7f9d5f9ee700 -1 osd.8 161 *** Got signal Terminated *** 2024-03-24T03:41:05.025 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:04 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:04.702+0000 7f9d5f9ee700 -1 osd.8 161 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T03:41:05.025 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:05 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:05.024+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:41:05.025 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:05 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:05.024+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:17.321155+0000 front 2024-03-24T03:40:17.321175+0000 (oldest deadline 2024-03-24T03:40:43.221253+0000) 2024-03-24T03:41:05.025 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:05 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:05.024+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:26.722380+0000 front 2024-03-24T03:40:26.722425+0000 (oldest deadline 2024-03-24T03:40:52.622419+0000) 2024-03-24T03:41:05.025 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:05 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:05.024+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:36.024265+0000 front 2024-03-24T03:40:36.024248+0000 (oldest deadline 2024-03-24T03:41:01.924103+0000) 2024-03-24T03:41:05.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:04 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:04.717+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:41:05.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:04 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:04.717+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:41:05.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:04 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:04.717+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:41:05.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:04 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:04.717+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:41:05.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:04 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:04.717+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:36.470431+0000 front 2024-03-24T03:40:36.470487+0000 (oldest deadline 2024-03-24T03:41:02.271075+0000) 2024-03-24T03:41:05.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:05 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:05.329+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:41:05.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:05 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:05.329+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:41:05.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:05 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:05.329+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.963619+0000 front 2024-03-24T03:40:21.963771+0000 (oldest deadline 2024-03-24T03:40:45.463653+0000) 2024-03-24T03:41:05.592 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:05 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:05.329+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:27.164667+0000 front 2024-03-24T03:40:27.164627+0000 (oldest deadline 2024-03-24T03:40:53.064884+0000) 2024-03-24T03:41:05.592 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:05 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:05.329+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.166393+0000 front 2024-03-24T03:40:37.166530+0000 (oldest deadline 2024-03-24T03:41:03.066159+0000) 2024-03-24T03:41:05.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:05 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:05.580+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:41:05.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:05 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:05.580+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:41:05.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:05 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:05.580+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:41:05.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:05 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:05.580+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.746407+0000 front 2024-03-24T03:40:37.746383+0000 (oldest deadline 2024-03-24T03:41:03.646218+0000) 2024-03-24T03:41:06.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:05 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:05.757+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:41:06.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:05 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:05.757+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:41:06.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:05 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:05.757+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:41:06.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:05 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:05.757+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:41:06.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:05 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:05.757+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:36.470431+0000 front 2024-03-24T03:40:36.470487+0000 (oldest deadline 2024-03-24T03:41:02.271075+0000) 2024-03-24T03:41:06.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:05 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:05.987+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:41:06.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:05 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:05.987+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:17.321155+0000 front 2024-03-24T03:40:17.321175+0000 (oldest deadline 2024-03-24T03:40:43.221253+0000) 2024-03-24T03:41:06.344 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:05 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:05.987+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:26.722380+0000 front 2024-03-24T03:40:26.722425+0000 (oldest deadline 2024-03-24T03:40:52.622419+0000) 2024-03-24T03:41:06.344 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:05 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:05.987+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:36.024265+0000 front 2024-03-24T03:40:36.024248+0000 (oldest deadline 2024-03-24T03:41:01.924103+0000) 2024-03-24T03:41:06.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:06.294+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:41:06.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:06.294+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:41:06.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:06.294+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.963619+0000 front 2024-03-24T03:40:21.963771+0000 (oldest deadline 2024-03-24T03:40:45.463653+0000) 2024-03-24T03:41:06.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:06.294+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:27.164667+0000 front 2024-03-24T03:40:27.164627+0000 (oldest deadline 2024-03-24T03:40:53.064884+0000) 2024-03-24T03:41:06.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:06 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:06.294+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.166393+0000 front 2024-03-24T03:40:37.166530+0000 (oldest deadline 2024-03-24T03:41:03.066159+0000) 2024-03-24T03:41:07.030 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:06 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:06.629+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:41:07.030 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:06 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:06.629+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:41:07.030 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:06 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:06.629+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:41:07.030 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:06 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:06.629+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.746407+0000 front 2024-03-24T03:40:37.746383+0000 (oldest deadline 2024-03-24T03:41:03.646218+0000) 2024-03-24T03:41:07.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:06 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:06.795+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:41:07.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:06 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:06.795+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:41:07.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:06 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:06.795+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:41:07.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:06 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:06.795+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:41:07.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:06 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:06.795+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:36.470431+0000 front 2024-03-24T03:40:36.470487+0000 (oldest deadline 2024-03-24T03:41:02.271075+0000) 2024-03-24T03:41:07.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:07 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:07.029+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:41:07.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:07 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:07.029+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:17.321155+0000 front 2024-03-24T03:40:17.321175+0000 (oldest deadline 2024-03-24T03:40:43.221253+0000) 2024-03-24T03:41:07.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:07 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:07.029+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:26.722380+0000 front 2024-03-24T03:40:26.722425+0000 (oldest deadline 2024-03-24T03:40:52.622419+0000) 2024-03-24T03:41:07.344 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:07 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:07.029+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:36.024265+0000 front 2024-03-24T03:40:36.024248+0000 (oldest deadline 2024-03-24T03:41:01.924103+0000) 2024-03-24T03:41:07.344 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:07 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:07.029+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:41.924532+0000 front 2024-03-24T03:40:41.924620+0000 (oldest deadline 2024-03-24T03:41:06.624627+0000) 2024-03-24T03:41:07.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:07 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:07.293+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:41:07.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:07 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:07.293+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:41:07.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:07 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:07.293+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.963619+0000 front 2024-03-24T03:40:21.963771+0000 (oldest deadline 2024-03-24T03:40:45.463653+0000) 2024-03-24T03:41:07.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:07 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:07.293+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:27.164667+0000 front 2024-03-24T03:40:27.164627+0000 (oldest deadline 2024-03-24T03:40:53.064884+0000) 2024-03-24T03:41:07.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:07 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:07.293+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.166393+0000 front 2024-03-24T03:40:37.166530+0000 (oldest deadline 2024-03-24T03:41:03.066159+0000) 2024-03-24T03:41:07.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:07 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:07.293+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.767266+0000 front 2024-03-24T03:40:44.767332+0000 (oldest deadline 2024-03-24T03:41:07.067121+0000) 2024-03-24T03:41:07.997 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:07 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:07.650+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:41:07.998 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:07 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:07.650+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:41:07.998 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:07 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:07.650+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:41:07.998 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:07 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:07.650+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.746407+0000 front 2024-03-24T03:40:37.746383+0000 (oldest deadline 2024-03-24T03:41:03.646218+0000) 2024-03-24T03:41:08.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:07 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:07.802+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:41:08.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:07 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:07.802+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:41:08.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:07 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:07.802+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:41:08.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:07 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:07.802+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:41:08.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:07 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:07.802+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:36.470431+0000 front 2024-03-24T03:40:36.470487+0000 (oldest deadline 2024-03-24T03:41:02.271075+0000) 2024-03-24T03:41:08.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:07 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:07.997+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:41:08.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:07 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:07.997+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:17.321155+0000 front 2024-03-24T03:40:17.321175+0000 (oldest deadline 2024-03-24T03:40:43.221253+0000) 2024-03-24T03:41:08.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:07 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:07.997+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:26.722380+0000 front 2024-03-24T03:40:26.722425+0000 (oldest deadline 2024-03-24T03:40:52.622419+0000) 2024-03-24T03:41:08.344 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:07 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:07.997+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:36.024265+0000 front 2024-03-24T03:40:36.024248+0000 (oldest deadline 2024-03-24T03:41:01.924103+0000) 2024-03-24T03:41:08.344 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:07 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:07.997+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:41.924532+0000 front 2024-03-24T03:40:41.924620+0000 (oldest deadline 2024-03-24T03:41:06.624627+0000) 2024-03-24T03:41:08.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:08.251+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:41:08.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:08.251+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:41:08.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:08.251+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.963619+0000 front 2024-03-24T03:40:21.963771+0000 (oldest deadline 2024-03-24T03:40:45.463653+0000) 2024-03-24T03:41:08.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:08.251+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:27.164667+0000 front 2024-03-24T03:40:27.164627+0000 (oldest deadline 2024-03-24T03:40:53.064884+0000) 2024-03-24T03:41:08.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:08.251+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.166393+0000 front 2024-03-24T03:40:37.166530+0000 (oldest deadline 2024-03-24T03:41:03.066159+0000) 2024-03-24T03:41:08.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:08 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:08.251+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.767266+0000 front 2024-03-24T03:40:44.767332+0000 (oldest deadline 2024-03-24T03:41:07.067121+0000) 2024-03-24T03:41:08.990 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:08 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:08.630+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:41:08.990 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:08 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:08.630+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:41:08.990 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:08 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:08.630+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:41:08.990 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:08 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:08.630+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.746407+0000 front 2024-03-24T03:40:37.746383+0000 (oldest deadline 2024-03-24T03:41:03.646218+0000) 2024-03-24T03:41:09.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:08 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:08.849+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:41:09.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:08 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:08.849+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:41:09.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:08 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:08.849+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:41:09.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:08 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:08.849+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:41:09.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:08 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:08.850+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:36.470431+0000 front 2024-03-24T03:40:36.470487+0000 (oldest deadline 2024-03-24T03:41:02.271075+0000) 2024-03-24T03:41:09.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:08 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:08.850+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:42.271882+0000 front 2024-03-24T03:40:42.271944+0000 (oldest deadline 2024-03-24T03:41:08.171852+0000) 2024-03-24T03:41:09.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:08 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:08.990+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.920168+0000 front 2024-03-24T03:40:13.920046+0000 (oldest deadline 2024-03-24T03:40:34.420174+0000) 2024-03-24T03:41:09.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:08 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:08.990+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:17.321155+0000 front 2024-03-24T03:40:17.321175+0000 (oldest deadline 2024-03-24T03:40:43.221253+0000) 2024-03-24T03:41:09.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:08 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:08.990+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:26.722380+0000 front 2024-03-24T03:40:26.722425+0000 (oldest deadline 2024-03-24T03:40:52.622419+0000) 2024-03-24T03:41:09.344 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:08 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:08.990+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:36.024265+0000 front 2024-03-24T03:40:36.024248+0000 (oldest deadline 2024-03-24T03:41:01.924103+0000) 2024-03-24T03:41:09.344 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:08 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8[54934]: 2024-03-24T03:41:08.990+0000 7f9d57fda700 -1 osd.8 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:41.924532+0000 front 2024-03-24T03:40:41.924620+0000 (oldest deadline 2024-03-24T03:41:06.624627+0000) 2024-03-24T03:41:09.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:09 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:09.284+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:41:09.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:09 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:09.284+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:41:09.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:09 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:09.284+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.963619+0000 front 2024-03-24T03:40:21.963771+0000 (oldest deadline 2024-03-24T03:40:45.463653+0000) 2024-03-24T03:41:09.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:09 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:09.284+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:27.164667+0000 front 2024-03-24T03:40:27.164627+0000 (oldest deadline 2024-03-24T03:40:53.064884+0000) 2024-03-24T03:41:09.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:09 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:09.284+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.166393+0000 front 2024-03-24T03:40:37.166530+0000 (oldest deadline 2024-03-24T03:41:03.066159+0000) 2024-03-24T03:41:09.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:09 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:09.284+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.767266+0000 front 2024-03-24T03:40:44.767332+0000 (oldest deadline 2024-03-24T03:41:07.067121+0000) 2024-03-24T03:41:10.086 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:09 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:09.641+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:41:10.087 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:09 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:09.641+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:41:10.087 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:09 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:09.641+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:41:10.087 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:09 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:09.641+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.746407+0000 front 2024-03-24T03:40:37.746383+0000 (oldest deadline 2024-03-24T03:41:03.646218+0000) 2024-03-24T03:41:10.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:09.889+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:41:10.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:09.889+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:41:10.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:09.889+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:41:10.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:09.890+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:41:10.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:09.890+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:36.470431+0000 front 2024-03-24T03:40:36.470487+0000 (oldest deadline 2024-03-24T03:41:02.271075+0000) 2024-03-24T03:41:10.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:09 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:09.890+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:42.271882+0000 front 2024-03-24T03:40:42.271944+0000 (oldest deadline 2024-03-24T03:41:08.171852+0000) 2024-03-24T03:41:10.343 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:10 smithi107 podman[124837]: 2024-03-24 03:41:10.087300068 +0000 UTC m=+5.522396925 container died 5518906f734532c97dbf0c20eba06d4453e5c209b14a1bd57e161f2dceb583a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-24T03:41:10.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:10 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:10.315+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:41:10.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:10 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:10.315+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:41:10.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:10 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:10.315+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.963619+0000 front 2024-03-24T03:40:21.963771+0000 (oldest deadline 2024-03-24T03:40:45.463653+0000) 2024-03-24T03:41:10.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:10 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:10.315+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:27.164667+0000 front 2024-03-24T03:40:27.164627+0000 (oldest deadline 2024-03-24T03:40:53.064884+0000) 2024-03-24T03:41:10.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:10 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:10.315+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.166393+0000 front 2024-03-24T03:40:37.166530+0000 (oldest deadline 2024-03-24T03:41:03.066159+0000) 2024-03-24T03:41:10.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:10 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:10.315+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.767266+0000 front 2024-03-24T03:40:44.767332+0000 (oldest deadline 2024-03-24T03:41:07.067121+0000) 2024-03-24T03:41:10.602 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:10 smithi107 podman[124837]: 2024-03-24 03:41:10.434981955 +0000 UTC m=+5.870078814 container cleanup 5518906f734532c97dbf0c20eba06d4453e5c209b14a1bd57e161f2dceb583a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-24T03:41:10.602 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:10 smithi107 bash[124837]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8 2024-03-24T03:41:10.991 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:10 smithi107 podman[124855]: 2024-03-24 03:41:10.602743762 +0000 UTC m=+0.509368280 container remove 5518906f734532c97dbf0c20eba06d4453e5c209b14a1bd57e161f2dceb583a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T03:41:10.991 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:10.656+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:41:10.991 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:10.656+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:41:10.991 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:10.656+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:41:10.991 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:10.656+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.746407+0000 front 2024-03-24T03:40:37.746383+0000 (oldest deadline 2024-03-24T03:41:03.646218+0000) 2024-03-24T03:41:10.991 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:10 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:10.656+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.147186+0000 front 2024-03-24T03:40:44.147077+0000 (oldest deadline 2024-03-24T03:41:10.547680+0000) 2024-03-24T03:41:11.178 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:10 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:10.904+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:41:11.178 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:10 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:10.904+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:41:11.178 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:10 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:10.904+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:41:11.178 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:10 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:10.904+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:41:11.179 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:10 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:10.904+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:36.470431+0000 front 2024-03-24T03:40:36.470487+0000 (oldest deadline 2024-03-24T03:41:02.271075+0000) 2024-03-24T03:41:11.179 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:10 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:10.904+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:42.271882+0000 front 2024-03-24T03:40:42.271944+0000 (oldest deadline 2024-03-24T03:41:08.171852+0000) 2024-03-24T03:41:11.272 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:10 smithi107 podman[124942]: 2024-03-24 03:41:10.894460932 +0000 UTC m=+0.017683637 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T03:41:11.272 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:11 smithi107 podman[124942]: 2024-03-24 03:41:11.037141423 +0000 UTC m=+0.160364121 container create 891aeaf7a84ed49fa8890826daa1bab43065505c395652ff832ca03ac9d860b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8-deactivate, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-24T03:41:11.272 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:11 smithi107 podman[124942]: 2024-03-24 03:41:11.272753097 +0000 UTC m=+0.395975792 container init 891aeaf7a84ed49fa8890826daa1bab43065505c395652ff832ca03ac9d860b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-24T03:41:11.564 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:11 smithi107 podman[124942]: 2024-03-24 03:41:11.275922075 +0000 UTC m=+0.399144768 container start 891aeaf7a84ed49fa8890826daa1bab43065505c395652ff832ca03ac9d860b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8-deactivate, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T03:41:11.564 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:11 smithi107 podman[124942]: 2024-03-24 03:41:11.355973987 +0000 UTC m=+0.479196677 container attach 891aeaf7a84ed49fa8890826daa1bab43065505c395652ff832ca03ac9d860b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T03:41:11.840 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:11 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:11.343+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:41:11.840 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:11 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:11.343+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:41:11.841 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:11 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:11.343+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.963619+0000 front 2024-03-24T03:40:21.963771+0000 (oldest deadline 2024-03-24T03:40:45.463653+0000) 2024-03-24T03:41:11.841 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:11 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:11.343+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:27.164667+0000 front 2024-03-24T03:40:27.164627+0000 (oldest deadline 2024-03-24T03:40:53.064884+0000) 2024-03-24T03:41:11.841 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:11 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:11.343+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.166393+0000 front 2024-03-24T03:40:37.166530+0000 (oldest deadline 2024-03-24T03:41:03.066159+0000) 2024-03-24T03:41:11.841 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:11 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:11.343+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.767266+0000 front 2024-03-24T03:40:44.767332+0000 (oldest deadline 2024-03-24T03:41:07.067121+0000) 2024-03-24T03:41:11.843 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 24 03:41:11 smithi107 podman[124942]: 2024-03-24 03:41:11.565388446 +0000 UTC m=+0.688611151 container died 891aeaf7a84ed49fa8890826daa1bab43065505c395652ff832ca03ac9d860b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-8-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T03:41:11.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:11 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:11.687+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:41:11.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:11 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:11.687+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:41:11.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:11 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:11.687+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:41:11.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:11 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:11.687+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.746407+0000 front 2024-03-24T03:40:37.746383+0000 (oldest deadline 2024-03-24T03:41:03.646218+0000) 2024-03-24T03:41:11.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:11 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:11.687+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.147186+0000 front 2024-03-24T03:40:44.147077+0000 (oldest deadline 2024-03-24T03:41:10.547680+0000) 2024-03-24T03:41:12.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:11 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:11.859+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:41:12.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:11 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:11.859+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:41:12.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:11 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:11.859+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:41:12.154 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:11 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:11.859+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:41:12.154 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:11 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:11.859+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:36.470431+0000 front 2024-03-24T03:40:36.470487+0000 (oldest deadline 2024-03-24T03:41:02.271075+0000) 2024-03-24T03:41:12.154 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:11 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:11.859+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:42.271882+0000 front 2024-03-24T03:40:42.271944+0000 (oldest deadline 2024-03-24T03:41:08.171852+0000) 2024-03-24T03:41:12.191 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.8.service' 2024-03-24T03:41:12.233 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:41:12.233 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-24T03:41:12.233 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-24T03:41:12.234 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.9 2024-03-24T03:41:12.554 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:12 smithi078 systemd[1]: Stopping Ceph osd.9 for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T03:41:12.554 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:12 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:12.366+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:41:12.554 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:12 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:12.366+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:41:12.554 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:12 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:12.366+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.963619+0000 front 2024-03-24T03:40:21.963771+0000 (oldest deadline 2024-03-24T03:40:45.463653+0000) 2024-03-24T03:41:12.555 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:12 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:12.366+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:27.164667+0000 front 2024-03-24T03:40:27.164627+0000 (oldest deadline 2024-03-24T03:40:53.064884+0000) 2024-03-24T03:41:12.555 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:12 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:12.366+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.166393+0000 front 2024-03-24T03:40:37.166530+0000 (oldest deadline 2024-03-24T03:41:03.066159+0000) 2024-03-24T03:41:12.555 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:12 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:12.366+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.767266+0000 front 2024-03-24T03:40:44.767332+0000 (oldest deadline 2024-03-24T03:41:07.067121+0000) 2024-03-24T03:41:12.555 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:12 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:12.553+0000 7f08a2cd1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:41:12.555 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:12 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:12.553+0000 7f08a2cd1700 -1 osd.9 161 *** Got signal Terminated *** 2024-03-24T03:41:12.555 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:12 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:12.553+0000 7f08a2cd1700 -1 osd.9 161 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T03:41:13.093 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:12 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:12.734+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:41:13.093 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:12 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:12.734+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:41:13.093 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:12 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:12.734+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:41:13.094 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:12 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:12.734+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.746407+0000 front 2024-03-24T03:40:37.746383+0000 (oldest deadline 2024-03-24T03:41:03.646218+0000) 2024-03-24T03:41:13.094 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:12 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:12.734+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.147186+0000 front 2024-03-24T03:40:44.147077+0000 (oldest deadline 2024-03-24T03:41:10.547680+0000) 2024-03-24T03:41:13.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:12 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:12.882+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:41:13.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:12 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:12.882+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:41:13.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:12 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:12.882+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:41:13.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:12 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:12.882+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:41:13.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:12 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:12.882+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:36.470431+0000 front 2024-03-24T03:40:36.470487+0000 (oldest deadline 2024-03-24T03:41:02.271075+0000) 2024-03-24T03:41:13.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:12 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:12.882+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:42.271882+0000 front 2024-03-24T03:40:42.271944+0000 (oldest deadline 2024-03-24T03:41:08.171852+0000) 2024-03-24T03:41:13.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:13.329+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:41:13.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:13.329+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:41:13.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:13.329+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.963619+0000 front 2024-03-24T03:40:21.963771+0000 (oldest deadline 2024-03-24T03:40:45.463653+0000) 2024-03-24T03:41:13.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:13.329+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:27.164667+0000 front 2024-03-24T03:40:27.164627+0000 (oldest deadline 2024-03-24T03:40:53.064884+0000) 2024-03-24T03:41:13.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:13.329+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.166393+0000 front 2024-03-24T03:40:37.166530+0000 (oldest deadline 2024-03-24T03:41:03.066159+0000) 2024-03-24T03:41:13.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:13 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:13.329+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.767266+0000 front 2024-03-24T03:40:44.767332+0000 (oldest deadline 2024-03-24T03:41:07.067121+0000) 2024-03-24T03:41:14.093 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:13.735+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:41:14.093 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:13.735+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:41:14.093 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:13.735+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:41:14.094 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:13.735+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.746407+0000 front 2024-03-24T03:40:37.746383+0000 (oldest deadline 2024-03-24T03:41:03.646218+0000) 2024-03-24T03:41:14.094 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:13 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:13.735+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.147186+0000 front 2024-03-24T03:40:44.147077+0000 (oldest deadline 2024-03-24T03:41:10.547680+0000) 2024-03-24T03:41:14.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:13 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:13.870+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:41:14.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:13 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:13.870+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:41:14.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:13 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:13.870+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:41:14.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:13 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:13.870+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:41:14.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:13 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:13.870+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:36.470431+0000 front 2024-03-24T03:40:36.470487+0000 (oldest deadline 2024-03-24T03:41:02.271075+0000) 2024-03-24T03:41:14.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:13 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:13.870+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:42.271882+0000 front 2024-03-24T03:40:42.271944+0000 (oldest deadline 2024-03-24T03:41:08.171852+0000) 2024-03-24T03:41:14.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:14.295+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:41:14.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:14.295+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:41:14.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:14.295+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.963619+0000 front 2024-03-24T03:40:21.963771+0000 (oldest deadline 2024-03-24T03:40:45.463653+0000) 2024-03-24T03:41:14.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:14.295+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:27.164667+0000 front 2024-03-24T03:40:27.164627+0000 (oldest deadline 2024-03-24T03:40:53.064884+0000) 2024-03-24T03:41:14.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:14.295+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.166393+0000 front 2024-03-24T03:40:37.166530+0000 (oldest deadline 2024-03-24T03:41:03.066159+0000) 2024-03-24T03:41:14.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:14 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:14.295+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.767266+0000 front 2024-03-24T03:40:44.767332+0000 (oldest deadline 2024-03-24T03:41:07.067121+0000) 2024-03-24T03:41:15.093 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:14.692+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:41:15.093 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:14.692+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:41:15.093 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:14.692+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:41:15.094 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:14.692+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.746407+0000 front 2024-03-24T03:40:37.746383+0000 (oldest deadline 2024-03-24T03:41:03.646218+0000) 2024-03-24T03:41:15.094 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:14 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:14.692+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.147186+0000 front 2024-03-24T03:40:44.147077+0000 (oldest deadline 2024-03-24T03:41:10.547680+0000) 2024-03-24T03:41:15.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:14 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:14.821+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:41:15.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:14 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:14.821+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:41:15.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:14 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:14.821+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:41:15.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:14 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:14.821+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:41:15.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:14 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:14.821+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:36.470431+0000 front 2024-03-24T03:40:36.470487+0000 (oldest deadline 2024-03-24T03:41:02.271075+0000) 2024-03-24T03:41:15.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:14 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:14.821+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:42.271882+0000 front 2024-03-24T03:40:42.271944+0000 (oldest deadline 2024-03-24T03:41:08.171852+0000) 2024-03-24T03:41:15.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:14 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:14.821+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-24T03:40:50.472808+0000 front 2024-03-24T03:40:50.472825+0000 (oldest deadline 2024-03-24T03:41:14.572697+0000) 2024-03-24T03:41:15.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:15 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:15.336+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:41:15.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:15 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:15.336+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:41:15.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:15 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:15.336+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.963619+0000 front 2024-03-24T03:40:21.963771+0000 (oldest deadline 2024-03-24T03:40:45.463653+0000) 2024-03-24T03:41:15.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:15 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:15.336+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:27.164667+0000 front 2024-03-24T03:40:27.164627+0000 (oldest deadline 2024-03-24T03:40:53.064884+0000) 2024-03-24T03:41:15.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:15 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:15.336+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.166393+0000 front 2024-03-24T03:40:37.166530+0000 (oldest deadline 2024-03-24T03:41:03.066159+0000) 2024-03-24T03:41:15.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:15 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:15.336+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.767266+0000 front 2024-03-24T03:40:44.767332+0000 (oldest deadline 2024-03-24T03:41:07.067121+0000) 2024-03-24T03:41:16.093 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:15 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:15.649+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:41:16.094 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:15 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:15.649+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:41:16.094 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:15 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:15.649+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:41:16.094 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:15 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:15.649+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.746407+0000 front 2024-03-24T03:40:37.746383+0000 (oldest deadline 2024-03-24T03:41:03.646218+0000) 2024-03-24T03:41:16.094 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:15 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:15.649+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.147186+0000 front 2024-03-24T03:40:44.147077+0000 (oldest deadline 2024-03-24T03:41:10.547680+0000) 2024-03-24T03:41:16.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:15 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:15.789+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:41:16.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:15 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:15.789+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:41:16.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:15 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:15.789+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:41:16.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:15 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:15.789+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:41:16.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:15 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:15.789+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:36.470431+0000 front 2024-03-24T03:40:36.470487+0000 (oldest deadline 2024-03-24T03:41:02.271075+0000) 2024-03-24T03:41:16.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:15 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:15.789+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:42.271882+0000 front 2024-03-24T03:40:42.271944+0000 (oldest deadline 2024-03-24T03:41:08.171852+0000) 2024-03-24T03:41:16.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:15 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:15.789+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-24T03:40:50.472808+0000 front 2024-03-24T03:40:50.472825+0000 (oldest deadline 2024-03-24T03:41:14.572697+0000) 2024-03-24T03:41:16.840 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:16 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:16.350+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:41:16.840 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:16 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:16.350+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:41:16.841 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:16 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:16.350+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.963619+0000 front 2024-03-24T03:40:21.963771+0000 (oldest deadline 2024-03-24T03:40:45.463653+0000) 2024-03-24T03:41:16.841 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:16 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:16.350+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:27.164667+0000 front 2024-03-24T03:40:27.164627+0000 (oldest deadline 2024-03-24T03:40:53.064884+0000) 2024-03-24T03:41:16.841 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:16 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:16.350+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.166393+0000 front 2024-03-24T03:40:37.166530+0000 (oldest deadline 2024-03-24T03:41:03.066159+0000) 2024-03-24T03:41:16.841 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:16 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:16.350+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.767266+0000 front 2024-03-24T03:40:44.767332+0000 (oldest deadline 2024-03-24T03:41:07.067121+0000) 2024-03-24T03:41:17.093 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:16 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:16.651+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:41:17.093 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:16 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:16.651+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:41:17.094 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:16 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:16.651+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:41:17.094 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:16 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:16.651+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.746407+0000 front 2024-03-24T03:40:37.746383+0000 (oldest deadline 2024-03-24T03:41:03.646218+0000) 2024-03-24T03:41:17.094 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:16 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:16.651+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.147186+0000 front 2024-03-24T03:40:44.147077+0000 (oldest deadline 2024-03-24T03:41:10.547680+0000) 2024-03-24T03:41:17.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:16 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:16.813+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:41:17.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:16 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:16.813+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:41:17.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:16 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:16.813+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:41:17.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:16 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:16.813+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:41:17.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:16 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:16.813+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:36.470431+0000 front 2024-03-24T03:40:36.470487+0000 (oldest deadline 2024-03-24T03:41:02.271075+0000) 2024-03-24T03:41:17.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:16 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:16.813+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:42.271882+0000 front 2024-03-24T03:40:42.271944+0000 (oldest deadline 2024-03-24T03:41:08.171852+0000) 2024-03-24T03:41:17.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:16 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:16.813+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-24T03:40:50.472808+0000 front 2024-03-24T03:40:50.472825+0000 (oldest deadline 2024-03-24T03:41:14.572697+0000) 2024-03-24T03:41:17.590 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:17 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:17.320+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:05.660240+0000 front 2024-03-24T03:40:05.660356+0000 (oldest deadline 2024-03-24T03:40:31.560428+0000) 2024-03-24T03:41:17.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:17 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:17.320+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:12.661916+0000 front 2024-03-24T03:40:12.662010+0000 (oldest deadline 2024-03-24T03:40:40.262507+0000) 2024-03-24T03:41:17.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:17 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:17.320+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.963619+0000 front 2024-03-24T03:40:21.963771+0000 (oldest deadline 2024-03-24T03:40:45.463653+0000) 2024-03-24T03:41:17.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:17 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:17.320+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:27.164667+0000 front 2024-03-24T03:40:27.164627+0000 (oldest deadline 2024-03-24T03:40:53.064884+0000) 2024-03-24T03:41:17.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:17 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:17.320+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.166393+0000 front 2024-03-24T03:40:37.166530+0000 (oldest deadline 2024-03-24T03:41:03.066159+0000) 2024-03-24T03:41:17.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:17 smithi078 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9[65171]: 2024-03-24T03:41:17.320+0000 7f089b2bd700 -1 osd.9 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.767266+0000 front 2024-03-24T03:40:44.767332+0000 (oldest deadline 2024-03-24T03:41:07.067121+0000) 2024-03-24T03:41:17.994 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:17 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:17.627+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:41:17.995 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:17 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:17.627+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:41:17.995 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:17 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:17.627+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:41:17.995 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:17 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:17.627+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.746407+0000 front 2024-03-24T03:40:37.746383+0000 (oldest deadline 2024-03-24T03:41:03.646218+0000) 2024-03-24T03:41:17.995 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:17 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:17.627+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.147186+0000 front 2024-03-24T03:40:44.147077+0000 (oldest deadline 2024-03-24T03:41:10.547680+0000) 2024-03-24T03:41:18.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:17 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:17.809+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:41:18.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:17 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:17.809+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:41:18.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:17 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:17.809+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:41:18.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:17 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:17.809+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:41:18.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:17 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:17.809+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:36.470431+0000 front 2024-03-24T03:40:36.470487+0000 (oldest deadline 2024-03-24T03:41:02.271075+0000) 2024-03-24T03:41:18.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:17 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:17.809+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:42.271882+0000 front 2024-03-24T03:40:42.271944+0000 (oldest deadline 2024-03-24T03:41:08.171852+0000) 2024-03-24T03:41:18.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:17 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:17.809+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-24T03:40:50.472808+0000 front 2024-03-24T03:40:50.472825+0000 (oldest deadline 2024-03-24T03:41:14.572697+0000) 2024-03-24T03:41:18.230 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:17 smithi078 podman[142498]: 2024-03-24 03:41:17.961284663 +0000 UTC m=+5.532228199 container died e4a6cd44d703e99a4eca6f496ef19cddbc5012f4c4c3b1a7dc1dd318ec790bb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T03:41:18.588 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:18 smithi078 podman[142498]: 2024-03-24 03:41:18.329318103 +0000 UTC m=+5.900261638 container cleanup e4a6cd44d703e99a4eca6f496ef19cddbc5012f4c4c3b1a7dc1dd318ec790bb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True) 2024-03-24T03:41:18.588 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:18 smithi078 bash[142498]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9 2024-03-24T03:41:18.840 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:18 smithi078 podman[142515]: 2024-03-24 03:41:18.588710705 +0000 UTC m=+0.623518306 container remove e4a6cd44d703e99a4eca6f496ef19cddbc5012f4c4c3b1a7dc1dd318ec790bb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-24T03:41:19.093 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:18.598+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:41:19.093 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:18.598+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:41:19.093 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:18.598+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:41:19.094 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:18.598+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.746407+0000 front 2024-03-24T03:40:37.746383+0000 (oldest deadline 2024-03-24T03:41:03.646218+0000) 2024-03-24T03:41:19.094 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:18 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:18.598+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.147186+0000 front 2024-03-24T03:40:44.147077+0000 (oldest deadline 2024-03-24T03:41:10.547680+0000) 2024-03-24T03:41:19.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:18 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:18.799+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:41:19.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:18 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:18.799+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:41:19.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:18 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:18.799+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:41:19.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:18 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:18.799+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:41:19.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:18 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:18.799+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:36.470431+0000 front 2024-03-24T03:40:36.470487+0000 (oldest deadline 2024-03-24T03:41:02.271075+0000) 2024-03-24T03:41:19.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:18 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:18.799+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:42.271882+0000 front 2024-03-24T03:40:42.271944+0000 (oldest deadline 2024-03-24T03:41:08.171852+0000) 2024-03-24T03:41:19.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:18 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:18.799+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-24T03:40:50.472808+0000 front 2024-03-24T03:40:50.472825+0000 (oldest deadline 2024-03-24T03:41:14.572697+0000) 2024-03-24T03:41:19.258 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:18 smithi078 podman[142609]: 2024-03-24 03:41:18.847037431 +0000 UTC m=+0.018323355 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T03:41:19.258 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:18 smithi078 podman[142609]: 2024-03-24 03:41:18.947927813 +0000 UTC m=+0.119213769 container create ee9982244bdfb536399f6a2a080346b49b232305a55d16d2327de54af862c925 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True) 2024-03-24T03:41:19.258 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:19 smithi078 podman[142609]: 2024-03-24 03:41:19.183949605 +0000 UTC m=+0.355235524 container init ee9982244bdfb536399f6a2a080346b49b232305a55d16d2327de54af862c925 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-24T03:41:19.258 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:19 smithi078 podman[142609]: 2024-03-24 03:41:19.187361056 +0000 UTC m=+0.358646976 container start ee9982244bdfb536399f6a2a080346b49b232305a55d16d2327de54af862c925 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T03:41:19.258 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:19 smithi078 podman[142609]: 2024-03-24 03:41:19.258449424 +0000 UTC m=+0.429735352 container attach ee9982244bdfb536399f6a2a080346b49b232305a55d16d2327de54af862c925 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9-deactivate, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-24T03:41:19.591 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 24 03:41:19 smithi078 podman[142609]: 2024-03-24 03:41:19.48460615 +0000 UTC m=+0.655892079 container died ee9982244bdfb536399f6a2a080346b49b232305a55d16d2327de54af862c925 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T03:41:20.080 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.9.service' 2024-03-24T03:41:20.093 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:19.602+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:41:20.093 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:19.602+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:41:20.093 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:19.602+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:41:20.094 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:19.602+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.746407+0000 front 2024-03-24T03:40:37.746383+0000 (oldest deadline 2024-03-24T03:41:03.646218+0000) 2024-03-24T03:41:20.094 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:19.602+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.147186+0000 front 2024-03-24T03:40:44.147077+0000 (oldest deadline 2024-03-24T03:41:10.547680+0000) 2024-03-24T03:41:20.094 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:19 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:19.602+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-24T03:40:52.848586+0000 front 2024-03-24T03:40:52.848744+0000 (oldest deadline 2024-03-24T03:41:19.249181+0000) 2024-03-24T03:41:20.121 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:41:20.122 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-24T03:41:20.122 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-24T03:41:20.122 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.10 2024-03-24T03:41:20.150 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:19 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:19.804+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:41:20.150 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:19 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:19.804+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:41:20.151 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:19 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:19.804+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:41:20.151 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:19 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:19.804+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:41:20.151 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:19 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:19.804+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:36.470431+0000 front 2024-03-24T03:40:36.470487+0000 (oldest deadline 2024-03-24T03:41:02.271075+0000) 2024-03-24T03:41:20.151 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:19 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:19.804+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:42.271882+0000 front 2024-03-24T03:40:42.271944+0000 (oldest deadline 2024-03-24T03:41:08.171852+0000) 2024-03-24T03:41:20.151 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:19 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:19.804+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-24T03:40:50.472808+0000 front 2024-03-24T03:40:50.472825+0000 (oldest deadline 2024-03-24T03:41:14.572697+0000) 2024-03-24T03:41:20.402 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:20 smithi106 systemd[1]: Stopping Ceph osd.10 for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T03:41:20.789 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:20 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:20.459+0000 7f8503d93700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:41:20.790 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:20 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:20.459+0000 7f8503d93700 -1 osd.10 161 *** Got signal Terminated *** 2024-03-24T03:41:20.790 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:20 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:20.459+0000 7f8503d93700 -1 osd.10 161 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T03:41:20.790 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:20 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:20.789+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:41:20.790 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:20 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:20.789+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:41:20.790 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:20 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:20.789+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:41:20.790 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:20 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:20.789+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:41:20.790 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:20 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:20.789+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:36.470431+0000 front 2024-03-24T03:40:36.470487+0000 (oldest deadline 2024-03-24T03:41:02.271075+0000) 2024-03-24T03:41:20.791 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:20 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:20.789+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:42.271882+0000 front 2024-03-24T03:40:42.271944+0000 (oldest deadline 2024-03-24T03:41:08.171852+0000) 2024-03-24T03:41:20.791 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:20 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:20.789+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-24T03:40:50.472808+0000 front 2024-03-24T03:40:50.472825+0000 (oldest deadline 2024-03-24T03:41:14.572697+0000) 2024-03-24T03:41:21.093 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:20 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:20.596+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:41:21.093 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:20 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:20.596+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:41:21.094 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:20 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:20.596+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:41:21.094 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:20 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:20.596+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.746407+0000 front 2024-03-24T03:40:37.746383+0000 (oldest deadline 2024-03-24T03:41:03.646218+0000) 2024-03-24T03:41:21.094 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:20 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:20.596+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.147186+0000 front 2024-03-24T03:40:44.147077+0000 (oldest deadline 2024-03-24T03:41:10.547680+0000) 2024-03-24T03:41:21.094 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:20 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:20.596+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-24T03:40:52.848586+0000 front 2024-03-24T03:40:52.848744+0000 (oldest deadline 2024-03-24T03:41:19.249181+0000) 2024-03-24T03:41:21.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:21 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:21.589+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:41:21.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:21 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:21.589+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:41:21.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:21 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:21.589+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:41:21.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:21 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:21.589+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.746407+0000 front 2024-03-24T03:40:37.746383+0000 (oldest deadline 2024-03-24T03:41:03.646218+0000) 2024-03-24T03:41:21.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:21 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:21.589+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.147186+0000 front 2024-03-24T03:40:44.147077+0000 (oldest deadline 2024-03-24T03:41:10.547680+0000) 2024-03-24T03:41:21.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:21 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:21.589+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-24T03:40:52.848586+0000 front 2024-03-24T03:40:52.848744+0000 (oldest deadline 2024-03-24T03:41:19.249181+0000) 2024-03-24T03:41:22.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:21 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:21.829+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:41:22.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:21 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:21.829+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:41:22.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:21 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:21.829+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:41:22.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:21 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:21.829+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:41:22.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:21 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:21.829+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:36.470431+0000 front 2024-03-24T03:40:36.470487+0000 (oldest deadline 2024-03-24T03:41:02.271075+0000) 2024-03-24T03:41:22.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:21 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:21.829+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:42.271882+0000 front 2024-03-24T03:40:42.271944+0000 (oldest deadline 2024-03-24T03:41:08.171852+0000) 2024-03-24T03:41:22.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:21 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:21.829+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-24T03:40:50.472808+0000 front 2024-03-24T03:40:50.472825+0000 (oldest deadline 2024-03-24T03:41:14.572697+0000) 2024-03-24T03:41:22.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:22 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:22.545+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:41:22.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:22 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:22.545+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:41:22.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:22 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:22.545+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:41:22.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:22 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:22.545+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.746407+0000 front 2024-03-24T03:40:37.746383+0000 (oldest deadline 2024-03-24T03:41:03.646218+0000) 2024-03-24T03:41:22.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:22 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:22.545+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.147186+0000 front 2024-03-24T03:40:44.147077+0000 (oldest deadline 2024-03-24T03:41:10.547680+0000) 2024-03-24T03:41:22.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:22 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:22.545+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-24T03:40:52.848586+0000 front 2024-03-24T03:40:52.848744+0000 (oldest deadline 2024-03-24T03:41:19.249181+0000) 2024-03-24T03:41:23.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:22 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:22.790+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:41:23.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:22 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:22.790+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:41:23.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:22 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:22.790+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:41:23.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:22 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:22.790+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:41:23.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:22 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:22.790+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:36.470431+0000 front 2024-03-24T03:40:36.470487+0000 (oldest deadline 2024-03-24T03:41:02.271075+0000) 2024-03-24T03:41:23.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:22 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:22.790+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:42.271882+0000 front 2024-03-24T03:40:42.271944+0000 (oldest deadline 2024-03-24T03:41:08.171852+0000) 2024-03-24T03:41:23.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:22 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:22.790+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-24T03:40:50.472808+0000 front 2024-03-24T03:40:50.472825+0000 (oldest deadline 2024-03-24T03:41:14.572697+0000) 2024-03-24T03:41:23.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:23 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:23.591+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:41:23.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:23 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:23.591+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:41:23.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:23 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:23.591+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:41:23.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:23 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:23.591+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.746407+0000 front 2024-03-24T03:40:37.746383+0000 (oldest deadline 2024-03-24T03:41:03.646218+0000) 2024-03-24T03:41:23.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:23 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:23.591+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.147186+0000 front 2024-03-24T03:40:44.147077+0000 (oldest deadline 2024-03-24T03:41:10.547680+0000) 2024-03-24T03:41:23.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:23 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:23.591+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-24T03:40:52.848586+0000 front 2024-03-24T03:40:52.848744+0000 (oldest deadline 2024-03-24T03:41:19.249181+0000) 2024-03-24T03:41:24.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:23 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:23.749+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:41:24.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:23 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:23.749+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:41:24.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:23 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:23.749+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:41:24.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:23 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:23.749+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:41:24.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:23 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:23.749+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:36.470431+0000 front 2024-03-24T03:40:36.470487+0000 (oldest deadline 2024-03-24T03:41:02.271075+0000) 2024-03-24T03:41:24.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:23 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:23.749+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:42.271882+0000 front 2024-03-24T03:40:42.271944+0000 (oldest deadline 2024-03-24T03:41:08.171852+0000) 2024-03-24T03:41:24.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:23 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:23.749+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-24T03:40:50.472808+0000 front 2024-03-24T03:40:50.472825+0000 (oldest deadline 2024-03-24T03:41:14.572697+0000) 2024-03-24T03:41:25.093 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:24.623+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:41:25.093 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:24.623+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:41:25.094 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:24.624+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:41:25.094 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:24.624+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.746407+0000 front 2024-03-24T03:40:37.746383+0000 (oldest deadline 2024-03-24T03:41:03.646218+0000) 2024-03-24T03:41:25.094 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:24.624+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.147186+0000 front 2024-03-24T03:40:44.147077+0000 (oldest deadline 2024-03-24T03:41:10.547680+0000) 2024-03-24T03:41:25.094 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:24 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:24.624+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-24T03:40:52.848586+0000 front 2024-03-24T03:40:52.848744+0000 (oldest deadline 2024-03-24T03:41:19.249181+0000) 2024-03-24T03:41:25.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:24 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:24.713+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.364977+0000 front 2024-03-24T03:40:04.364802+0000 (oldest deadline 2024-03-24T03:40:27.264878+0000) 2024-03-24T03:41:25.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:24 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:24.713+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-24T03:40:13.666682+0000 front 2024-03-24T03:40:13.666709+0000 (oldest deadline 2024-03-24T03:40:37.166795+0000) 2024-03-24T03:41:25.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:24 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:24.713+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:20.668207+0000 front 2024-03-24T03:40:20.668051+0000 (oldest deadline 2024-03-24T03:40:45.967987+0000) 2024-03-24T03:41:25.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:24 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:24.713+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:25.968397+0000 front 2024-03-24T03:40:25.968482+0000 (oldest deadline 2024-03-24T03:40:51.268582+0000) 2024-03-24T03:41:25.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:24 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:24.713+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:36.470431+0000 front 2024-03-24T03:40:36.470487+0000 (oldest deadline 2024-03-24T03:41:02.271075+0000) 2024-03-24T03:41:25.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:24 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:24.713+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:42.271882+0000 front 2024-03-24T03:40:42.271944+0000 (oldest deadline 2024-03-24T03:41:08.171852+0000) 2024-03-24T03:41:25.153 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:24 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10[61723]: 2024-03-24T03:41:24.713+0000 7f84fc37f700 -1 osd.10 161 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-24T03:40:50.472808+0000 front 2024-03-24T03:40:50.472825+0000 (oldest deadline 2024-03-24T03:41:14.572697+0000) 2024-03-24T03:41:26.093 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:25 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:25.596+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:41:26.093 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:25 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:25.596+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:41:26.093 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:25 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:25.596+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:41:26.094 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:25 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:25.596+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.746407+0000 front 2024-03-24T03:40:37.746383+0000 (oldest deadline 2024-03-24T03:41:03.646218+0000) 2024-03-24T03:41:26.094 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:25 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:25.596+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.147186+0000 front 2024-03-24T03:40:44.147077+0000 (oldest deadline 2024-03-24T03:41:10.547680+0000) 2024-03-24T03:41:26.094 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:25 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:25.596+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-24T03:40:52.848586+0000 front 2024-03-24T03:40:52.848744+0000 (oldest deadline 2024-03-24T03:41:19.249181+0000) 2024-03-24T03:41:26.094 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:25 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:25.596+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6820 osd.7 since back 2024-03-24T03:40:59.249980+0000 front 2024-03-24T03:40:59.249913+0000 (oldest deadline 2024-03-24T03:41:25.149704+0000) 2024-03-24T03:41:26.152 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:25 smithi106 podman[122281]: 2024-03-24 03:41:25.863230199 +0000 UTC m=+5.555381795 container died 7e8f139366db5115d2da4e8724a85eceecc58f906071049d629238078892b4c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T03:41:26.509 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:26 smithi106 podman[122281]: 2024-03-24 03:41:26.250728265 +0000 UTC m=+5.942879852 container cleanup 7e8f139366db5115d2da4e8724a85eceecc58f906071049d629238078892b4c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T03:41:26.510 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:26 smithi106 bash[122281]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10 2024-03-24T03:41:26.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:26 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:26.588+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:41:26.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:26 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:26.588+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:41:26.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:26 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:26.588+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:41:26.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:26 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:26.588+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.746407+0000 front 2024-03-24T03:40:37.746383+0000 (oldest deadline 2024-03-24T03:41:03.646218+0000) 2024-03-24T03:41:26.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:26 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:26.588+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.147186+0000 front 2024-03-24T03:40:44.147077+0000 (oldest deadline 2024-03-24T03:41:10.547680+0000) 2024-03-24T03:41:26.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:26 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:26.588+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-24T03:40:52.848586+0000 front 2024-03-24T03:40:52.848744+0000 (oldest deadline 2024-03-24T03:41:19.249181+0000) 2024-03-24T03:41:26.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:26 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:26.588+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6820 osd.7 since back 2024-03-24T03:40:59.249980+0000 front 2024-03-24T03:40:59.249913+0000 (oldest deadline 2024-03-24T03:41:25.149704+0000) 2024-03-24T03:41:26.851 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:26 smithi106 podman[122297]: 2024-03-24 03:41:26.510106947 +0000 UTC m=+0.650253853 container remove 7e8f139366db5115d2da4e8724a85eceecc58f906071049d629238078892b4c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-24T03:41:27.112 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:26 smithi106 podman[122386]: 2024-03-24 03:41:26.852623445 +0000 UTC m=+0.112004234 container create 475405a9bea5aaf6bde0128afac4c68c5a4e3ba55538baa31ba697ab46dec380 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10-deactivate, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-24T03:41:27.113 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:26 smithi106 podman[122386]: 2024-03-24 03:41:26.758107089 +0000 UTC m=+0.017487885 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T03:41:27.113 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:27 smithi106 podman[122386]: 2024-03-24 03:41:27.030549622 +0000 UTC m=+0.289930430 container init 475405a9bea5aaf6bde0128afac4c68c5a4e3ba55538baa31ba697ab46dec380 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T03:41:27.113 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:27 smithi106 podman[122386]: 2024-03-24 03:41:27.034188883 +0000 UTC m=+0.293569675 container start 475405a9bea5aaf6bde0128afac4c68c5a4e3ba55538baa31ba697ab46dec380 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-24T03:41:27.113 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:27 smithi106 podman[122386]: 2024-03-24 03:41:27.113089932 +0000 UTC m=+0.372470748 container attach 475405a9bea5aaf6bde0128afac4c68c5a4e3ba55538baa31ba697ab46dec380 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-24T03:41:27.402 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 24 03:41:27 smithi106 podman[122386]: 2024-03-24 03:41:27.319703317 +0000 UTC m=+0.579084115 container died 475405a9bea5aaf6bde0128afac4c68c5a4e3ba55538baa31ba697ab46dec380 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-10-deactivate, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T03:41:27.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:27 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:27.557+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:41:27.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:27 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:27.557+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:41:27.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:27 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:27.557+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:41:27.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:27 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:27.557+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.746407+0000 front 2024-03-24T03:40:37.746383+0000 (oldest deadline 2024-03-24T03:41:03.646218+0000) 2024-03-24T03:41:27.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:27 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:27.557+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.147186+0000 front 2024-03-24T03:40:44.147077+0000 (oldest deadline 2024-03-24T03:41:10.547680+0000) 2024-03-24T03:41:27.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:27 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:27.557+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-24T03:40:52.848586+0000 front 2024-03-24T03:40:52.848744+0000 (oldest deadline 2024-03-24T03:41:19.249181+0000) 2024-03-24T03:41:27.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:27 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:27.557+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6820 osd.7 since back 2024-03-24T03:40:59.249980+0000 front 2024-03-24T03:40:59.249913+0000 (oldest deadline 2024-03-24T03:41:25.149704+0000) 2024-03-24T03:41:27.916 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.10.service' 2024-03-24T03:41:27.998 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:41:27.998 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-24T03:41:27.998 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-24T03:41:27.999 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.11 2024-03-24T03:41:28.293 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:28 smithi107 systemd[1]: Stopping Ceph osd.11 for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T03:41:28.565 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:28.293+0000 7fb0538e1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:41:28.566 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:28.293+0000 7fb0538e1700 -1 osd.11 161 *** Got signal Terminated *** 2024-03-24T03:41:28.566 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:28.293+0000 7fb0538e1700 -1 osd.11 161 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T03:41:28.566 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:28.565+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:41:28.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:28.565+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:41:28.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:28.565+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:41:28.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:28.565+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.746407+0000 front 2024-03-24T03:40:37.746383+0000 (oldest deadline 2024-03-24T03:41:03.646218+0000) 2024-03-24T03:41:28.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:28.565+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.147186+0000 front 2024-03-24T03:40:44.147077+0000 (oldest deadline 2024-03-24T03:41:10.547680+0000) 2024-03-24T03:41:28.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:28.565+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-24T03:40:52.848586+0000 front 2024-03-24T03:40:52.848744+0000 (oldest deadline 2024-03-24T03:41:19.249181+0000) 2024-03-24T03:41:28.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:28 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:28.565+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6820 osd.7 since back 2024-03-24T03:40:59.249980+0000 front 2024-03-24T03:40:59.249913+0000 (oldest deadline 2024-03-24T03:41:25.149704+0000) 2024-03-24T03:41:29.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:29.568+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:41:29.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:29.568+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:41:29.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:29.568+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:41:29.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:29.568+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.746407+0000 front 2024-03-24T03:40:37.746383+0000 (oldest deadline 2024-03-24T03:41:03.646218+0000) 2024-03-24T03:41:29.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:29.568+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.147186+0000 front 2024-03-24T03:40:44.147077+0000 (oldest deadline 2024-03-24T03:41:10.547680+0000) 2024-03-24T03:41:29.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:29.569+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-24T03:40:52.848586+0000 front 2024-03-24T03:40:52.848744+0000 (oldest deadline 2024-03-24T03:41:19.249181+0000) 2024-03-24T03:41:29.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:29 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:29.569+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6820 osd.7 since back 2024-03-24T03:40:59.249980+0000 front 2024-03-24T03:40:59.249913+0000 (oldest deadline 2024-03-24T03:41:25.149704+0000) 2024-03-24T03:41:30.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:30 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:30.528+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:41:30.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:30 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:30.528+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:41:30.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:30 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:30.528+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:41:30.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:30 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:30.528+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.746407+0000 front 2024-03-24T03:40:37.746383+0000 (oldest deadline 2024-03-24T03:41:03.646218+0000) 2024-03-24T03:41:30.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:30 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:30.528+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.147186+0000 front 2024-03-24T03:40:44.147077+0000 (oldest deadline 2024-03-24T03:41:10.547680+0000) 2024-03-24T03:41:30.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:30 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:30.528+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-24T03:40:52.848586+0000 front 2024-03-24T03:40:52.848744+0000 (oldest deadline 2024-03-24T03:41:19.249181+0000) 2024-03-24T03:41:30.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:30 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:30.528+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6820 osd.7 since back 2024-03-24T03:40:59.249980+0000 front 2024-03-24T03:40:59.249913+0000 (oldest deadline 2024-03-24T03:41:25.149704+0000) 2024-03-24T03:41:31.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:31 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:31.551+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:41:31.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:31 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:31.551+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:41:31.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:31 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:31.551+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:41:31.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:31 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:31.551+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.746407+0000 front 2024-03-24T03:40:37.746383+0000 (oldest deadline 2024-03-24T03:41:03.646218+0000) 2024-03-24T03:41:31.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:31 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:31.551+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.147186+0000 front 2024-03-24T03:40:44.147077+0000 (oldest deadline 2024-03-24T03:41:10.547680+0000) 2024-03-24T03:41:31.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:31 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:31.551+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-24T03:40:52.848586+0000 front 2024-03-24T03:40:52.848744+0000 (oldest deadline 2024-03-24T03:41:19.249181+0000) 2024-03-24T03:41:31.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:31 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:31.551+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6820 osd.7 since back 2024-03-24T03:40:59.249980+0000 front 2024-03-24T03:40:59.249913+0000 (oldest deadline 2024-03-24T03:41:25.149704+0000) 2024-03-24T03:41:32.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:32 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:32.551+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-24T03:40:04.238385+0000 front 2024-03-24T03:40:04.238337+0000 (oldest deadline 2024-03-24T03:40:26.538447+0000) 2024-03-24T03:41:32.843 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:32 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:32.551+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-24T03:40:21.443326+0000 front 2024-03-24T03:40:21.443117+0000 (oldest deadline 2024-03-24T03:40:47.343283+0000) 2024-03-24T03:41:32.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:32 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:32.551+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-24T03:40:29.545170+0000 front 2024-03-24T03:40:29.545188+0000 (oldest deadline 2024-03-24T03:40:53.045050+0000) 2024-03-24T03:41:32.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:32 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:32.551+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-24T03:40:37.746407+0000 front 2024-03-24T03:40:37.746383+0000 (oldest deadline 2024-03-24T03:41:03.646218+0000) 2024-03-24T03:41:32.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:32 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:32.551+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-24T03:40:44.147186+0000 front 2024-03-24T03:40:44.147077+0000 (oldest deadline 2024-03-24T03:41:10.547680+0000) 2024-03-24T03:41:32.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:32 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:32.551+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-24T03:40:52.848586+0000 front 2024-03-24T03:40:52.848744+0000 (oldest deadline 2024-03-24T03:41:19.249181+0000) 2024-03-24T03:41:32.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:32 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:32.551+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.106:6820 osd.7 since back 2024-03-24T03:40:59.249980+0000 front 2024-03-24T03:40:59.249913+0000 (oldest deadline 2024-03-24T03:41:25.149704+0000) 2024-03-24T03:41:32.844 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:32 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11[61990]: 2024-03-24T03:41:32.551+0000 7fb04becd700 -1 osd.11 161 heartbeat_check: no reply from 172.21.15.107:6820 osd.8 since back 2024-03-24T03:41:09.651469+0000 front 2024-03-24T03:41:09.651450+0000 (oldest deadline 2024-03-24T03:41:32.551449+0000) 2024-03-24T03:41:34.010 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:33 smithi107 podman[125062]: 2024-03-24 03:41:33.740535591 +0000 UTC m=+5.553503360 container died f3d648a121e2e6bb3176dcc6630542285ce6f2ec876a80a7cdbfcf8ea3e184a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319) 2024-03-24T03:41:34.276 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:34 smithi107 podman[125062]: 2024-03-24 03:41:34.092587969 +0000 UTC m=+5.905555735 container cleanup f3d648a121e2e6bb3176dcc6630542285ce6f2ec876a80a7cdbfcf8ea3e184a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T03:41:34.277 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:34 smithi107 bash[125062]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11 2024-03-24T03:41:34.593 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:34 smithi107 podman[125079]: 2024-03-24 03:41:34.277114123 +0000 UTC m=+0.526270916 container remove f3d648a121e2e6bb3176dcc6630542285ce6f2ec876a80a7cdbfcf8ea3e184a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-24T03:41:34.988 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:34 smithi107 podman[125170]: 2024-03-24 03:41:34.601656196 +0000 UTC m=+0.017906071 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T03:41:34.988 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:34 smithi107 podman[125170]: 2024-03-24 03:41:34.711500065 +0000 UTC m=+0.127749945 container create 5e23d197060a83ad937810457cf36816f3e679f367d12ba6d3d2569114067f02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-24T03:41:34.988 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:34 smithi107 podman[125170]: 2024-03-24 03:41:34.913506822 +0000 UTC m=+0.329756691 container init 5e23d197060a83ad937810457cf36816f3e679f367d12ba6d3d2569114067f02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-24T03:41:34.988 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:34 smithi107 podman[125170]: 2024-03-24 03:41:34.916664323 +0000 UTC m=+0.332914193 container start 5e23d197060a83ad937810457cf36816f3e679f367d12ba6d3d2569114067f02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-24T03:41:34.988 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:34 smithi107 podman[125170]: 2024-03-24 03:41:34.988419165 +0000 UTC m=+0.404669030 container attach 5e23d197060a83ad937810457cf36816f3e679f367d12ba6d3d2569114067f02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-24T03:41:35.343 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:35 smithi107 podman[125170]: 2024-03-24 03:41:35.202992979 +0000 UTC m=+0.619242832 container died 5e23d197060a83ad937810457cf36816f3e679f367d12ba6d3d2569114067f02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-24T03:41:35.736 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 24 03:41:35 smithi107 podman[125197]: 2024-03-24 03:41:35.716314621 +0000 UTC m=+0.508036669 container remove 5e23d197060a83ad937810457cf36816f3e679f367d12ba6d3d2569114067f02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-osd-11-deactivate, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-24T03:41:35.741 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@osd.11.service' 2024-03-24T03:41:35.782 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:41:35.782 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-24T03:41:35.782 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-24T03:41:35.782 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.a 2024-03-24T03:41:35.824 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.a.service' 2024-03-24T03:41:35.907 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:41:35.907 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-24T03:41:35.907 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-24T03:41:35.907 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.d 2024-03-24T03:41:35.985 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.d.service' 2024-03-24T03:41:36.066 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:41:36.066 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-24T03:41:36.066 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-24T03:41:36.067 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.g 2024-03-24T03:41:36.140 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.g.service' 2024-03-24T03:41:36.217 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:41:36.217 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-24T03:41:36.217 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-24T03:41:36.217 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.j 2024-03-24T03:41:36.289 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.j.service' 2024-03-24T03:41:36.363 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:41:36.364 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-24T03:41:36.364 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-24T03:41:36.364 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.b 2024-03-24T03:41:36.652 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 24 03:41:36 smithi106 systemd[1]: Stopping Ceph mds.b for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T03:41:36.914 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 24 03:41:36 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-b[66985]: 2024-03-24T03:41:36.659+0000 7f08d6699700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:41:36.915 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 24 03:41:36 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-b[66985]: 2024-03-24T03:41:36.659+0000 7f08d6699700 -1 mds.b *** got signal Terminated *** 2024-03-24T03:41:40.743 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 24 03:41:40 smithi106 podman[122491]: 2024-03-24 03:41:40.494087961 +0000 UTC m=+3.937572165 container died 1f15b94cc52ef0fb6cca0811dbb55fd69b05917ee2cd2b94359da2fc10d0be2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-24T03:41:41.152 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 24 03:41:40 smithi106 podman[122491]: 2024-03-24 03:41:40.843428491 +0000 UTC m=+4.286912696 container cleanup 1f15b94cc52ef0fb6cca0811dbb55fd69b05917ee2cd2b94359da2fc10d0be2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-b, ceph=True, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-24T03:41:41.152 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 24 03:41:40 smithi106 bash[122491]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-b 2024-03-24T03:41:41.313 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.b.service' 2024-03-24T03:41:41.350 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:41:41.350 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-24T03:41:41.350 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-24T03:41:41.351 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.e 2024-03-24T03:41:41.902 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 24 03:41:41 smithi106 systemd[1]: Stopping Ceph mds.e for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T03:41:41.903 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 24 03:41:41 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-e[66368]: 2024-03-24T03:41:41.663+0000 7f61f8d84700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:41:41.903 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 24 03:41:41 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-e[66368]: 2024-03-24T03:41:41.663+0000 7f61f8d84700 -1 mds.e *** got signal Terminated *** 2024-03-24T03:41:44.764 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 24 03:41:44 smithi106 podman[122624]: 2024-03-24 03:41:44.492257788 +0000 UTC m=+2.923534584 container died d421b8ece3df473359b1be72524413f52fa2d430cea617546e5c6418ab637952 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T03:41:45.039 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 24 03:41:44 smithi106 podman[122624]: 2024-03-24 03:41:44.83897383 +0000 UTC m=+3.270250612 container cleanup d421b8ece3df473359b1be72524413f52fa2d430cea617546e5c6418ab637952 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-e, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True) 2024-03-24T03:41:45.039 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 24 03:41:44 smithi106 bash[122624]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-e 2024-03-24T03:41:45.263 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.e.service' 2024-03-24T03:41:45.299 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:41:45.299 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-24T03:41:45.300 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-24T03:41:45.300 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.h 2024-03-24T03:41:45.592 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 24 03:41:45 smithi106 systemd[1]: Stopping Ceph mds.h for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T03:41:45.902 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 24 03:41:45 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-h[66675]: 2024-03-24T03:41:45.591+0000 7f7f00991700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:41:45.903 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 24 03:41:45 smithi106 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-h[66675]: 2024-03-24T03:41:45.591+0000 7f7f00991700 -1 mds.h *** got signal Terminated *** 2024-03-24T03:41:48.750 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 24 03:41:48 smithi106 podman[122745]: 2024-03-24 03:41:48.484103824 +0000 UTC m=+2.971025563 container died cb8a0c8e7dbcf4a3542c653bab6687bcb58fe96b3fb11bdffd0d5f2671dc80ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-h, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-24T03:41:49.018 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 24 03:41:48 smithi106 podman[122745]: 2024-03-24 03:41:48.834166848 +0000 UTC m=+3.321088585 container cleanup cb8a0c8e7dbcf4a3542c653bab6687bcb58fe96b3fb11bdffd0d5f2671dc80ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-h, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-24T03:41:49.018 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 24 03:41:48 smithi106 bash[122745]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-h 2024-03-24T03:41:49.225 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.h.service' 2024-03-24T03:41:49.259 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:41:49.259 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-24T03:41:49.259 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-24T03:41:49.259 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.k 2024-03-24T03:41:49.329 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.k.service' 2024-03-24T03:41:49.402 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:41:49.403 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-24T03:41:49.403 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-24T03:41:49.403 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.c 2024-03-24T03:41:49.716 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 24 03:41:49 smithi107 systemd[1]: Stopping Ceph mds.c for c7e2806a-e972-11ee-95cd-87774f69a715... 2024-03-24T03:41:50.031 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 24 03:41:49 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-c[67198]: 2024-03-24T03:41:49.716+0000 7ff8d938e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:41:50.032 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 24 03:41:49 smithi107 ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-c[67198]: 2024-03-24T03:41:49.716+0000 7ff8d938e700 -1 mds.c *** got signal Terminated *** 2024-03-24T03:41:52.774 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 24 03:41:52 smithi107 podman[125277]: 2024-03-24 03:41:52.483752303 +0000 UTC m=+2.893797747 container died cedd2902e47d42e03f52b748bd15350a8e09d557a6ae1b5dffae0e11b6f502a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-c, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T03:41:53.093 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 24 03:41:52 smithi107 podman[125277]: 2024-03-24 03:41:52.891183768 +0000 UTC m=+3.301229212 container cleanup cedd2902e47d42e03f52b748bd15350a8e09d557a6ae1b5dffae0e11b6f502a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-c, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-24T03:41:53.093 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 24 03:41:52 smithi107 bash[125277]: ceph-c7e2806a-e972-11ee-95cd-87774f69a715-mds-c 2024-03-24T03:41:53.306 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.c.service' 2024-03-24T03:41:53.343 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:41:53.343 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-24T03:41:53.343 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-24T03:41:53.344 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.f 2024-03-24T03:41:53.414 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.f.service' 2024-03-24T03:41:53.487 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:41:53.488 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-24T03:41:53.488 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-24T03:41:53.488 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.i 2024-03-24T03:41:53.558 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.i.service' 2024-03-24T03:41:53.631 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:41:53.631 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-24T03:41:53.632 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-24T03:41:53.632 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.l 2024-03-24T03:41:53.702 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c7e2806a-e972-11ee-95cd-87774f69a715@mds.l.service' 2024-03-24T03:41:53.775 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:41:53.775 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-24T03:41:53.775 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c7e2806a-e972-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T03:41:53.966 INFO:teuthology.orchestra.run.smithi078.stdout:Deleting cluster with fsid: c7e2806a-e972-11ee-95cd-87774f69a715 2024-03-24T03:41:56.272 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c7e2806a-e972-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T03:41:56.461 INFO:teuthology.orchestra.run.smithi106.stdout:Deleting cluster with fsid: c7e2806a-e972-11ee-95cd-87774f69a715 2024-03-24T03:41:58.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c7e2806a-e972-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T03:41:59.043 INFO:teuthology.orchestra.run.smithi107.stdout:Deleting cluster with fsid: c7e2806a-e972-11ee-95cd-87774f69a715 2024-03-24T03:42:01.459 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T03:42:01.492 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T03:42:01.526 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T03:42:01.554 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-24T03:42:01.556 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616430/remote/smithi078/crash 2024-03-24T03:42:01.557 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/crash -- . 2024-03-24T03:42:01.598 INFO:teuthology.orchestra.run.smithi078.stderr:tar: /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T03:42:01.598 INFO:teuthology.orchestra.run.smithi078.stderr:tar: Error is not recoverable: exiting now 2024-03-24T03:42:01.599 DEBUG:teuthology.misc:Transferring archived files from smithi106:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616430/remote/smithi106/crash 2024-03-24T03:42:01.600 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar c -f - -C /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/crash -- . 2024-03-24T03:42:01.642 INFO:teuthology.orchestra.run.smithi106.stderr:tar: /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T03:42:01.643 INFO:teuthology.orchestra.run.smithi106.stderr:tar: Error is not recoverable: exiting now 2024-03-24T03:42:01.645 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616430/remote/smithi107/crash 2024-03-24T03:42:01.646 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/crash -- . 2024-03-24T03:42:01.689 INFO:teuthology.orchestra.run.smithi107.stderr:tar: /var/lib/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T03:42:01.689 INFO:teuthology.orchestra.run.smithi107.stderr:tar: Error is not recoverable: exiting now 2024-03-24T03:42:01.690 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-24T03:42:01.690 DEBUG:teuthology.orchestra.run.smithi078:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-03-24T03:42:01.725 INFO:teuthology.orchestra.run.smithi078.stdout:2024-03-24T00:12:37.578413+0000 mon.a (mon.0) 188 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-24T03:42:01.729 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-24T03:42:01.729 INFO:tasks.cephadm:Compressing logs... 2024-03-24T03:42:01.729 DEBUG:teuthology.orchestra.run.smithi078:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T03:42:01.771 DEBUG:teuthology.orchestra.run.smithi106:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T03:42:01.774 DEBUG:teuthology.orchestra.run.smithi107:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T03:42:01.798 INFO:teuthology.orchestra.run.smithi078.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T03:42:01.798 INFO:teuthology.orchestra.run.smithi078.stderr:: No such file or directory 2024-03-24T03:42:01.798 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-03-24T03:42:01.825 INFO:teuthology.orchestra.run.smithi106.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T03:42:01.826 INFO:teuthology.orchestra.run.smithi106.stderr:: No such file or directory 2024-03-24T03:42:01.826 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.10.log 2024-03-24T03:42:01.826 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T03:42:01.826 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.b.log 2024-03-24T03:42:01.827 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.4.log 2024-03-24T03:42:01.827 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.1.log 2024-03-24T03:42:01.827 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.b.log: /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.k.log 2024-03-24T03:42:01.827 INFO:teuthology.orchestra.run.smithi106.stderr: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T03:42:01.828 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.h.log 2024-03-24T03:42:01.828 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.7.log 2024-03-24T03:42:01.828 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.e.log 2024-03-24T03:42:01.828 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.7.log: 91.6% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T03:42:01.829 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mgr.y.log 2024-03-24T03:42:01.829 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T03:42:01.831 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.6.log 2024-03-24T03:42:01.831 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/ceph-client.admin.log: gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mon.a.log 2024-03-24T03:42:01.832 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T03:42:01.832 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.6.log: /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.g.log 2024-03-24T03:42:01.832 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.j.log 2024-03-24T03:42:01.832 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.g.log: gzip -5 92.6% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-03-24T03:42:01.833 INFO:teuthology.orchestra.run.smithi078.stderr: --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.3.log 2024-03-24T03:42:01.833 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mgr.x.log 2024-03-24T03:42:01.833 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.3.log: /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.j.log: 91.5% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T03:42:01.833 INFO:teuthology.orchestra.run.smithi078.stderr: 93.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T03:42:01.834 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.0.log 2024-03-24T03:42:01.834 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T03:42:01.837 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph.log 2024-03-24T03:42:01.837 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph.cephadm.log: 88.0% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T03:42:01.838 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph.log 2024-03-24T03:42:01.839 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph.cephadm.log: 88.4% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T03:42:01.839 INFO:teuthology.orchestra.run.smithi107.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T03:42:01.839 INFO:teuthology.orchestra.run.smithi107.stderr:: No such file or directory 2024-03-24T03:42:01.839 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.l.log 2024-03-24T03:42:01.839 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T03:42:01.839 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.i.log 2024-03-24T03:42:01.840 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.l.log: /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.11.log 2024-03-24T03:42:01.840 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.i.log: --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mgr.z.log 2024-03-24T03:42:01.840 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.8.log 2024-03-24T03:42:01.840 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.5.log 2024-03-24T03:42:01.840 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.f.log 2024-03-24T03:42:01.840 INFO:teuthology.orchestra.run.smithi107.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T03:42:01.840 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T03:42:01.840 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.f.log: 91.6% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T03:42:01.841 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph.log 2024-03-24T03:42:01.841 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph.cephadm.log: 88.4% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T03:42:01.841 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.2.log 2024-03-24T03:42:01.841 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.c.log 2024-03-24T03:42:01.841 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T03:42:01.842 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T03:42:01.843 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mon.c.log 2024-03-24T03:42:01.845 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-volume.log: 94.2% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mgr.z.log.gz 2024-03-24T03:42:01.847 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T03:42:01.851 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mon.b.log 2024-03-24T03:42:01.857 INFO:teuthology.orchestra.run.smithi107.stderr: 90.2%/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mon.c.log: -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T03:42:01.861 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.9.log 2024-03-24T03:42:01.870 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.d.log 2024-03-24T03:42:01.879 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.a.log 2024-03-24T03:42:01.887 INFO:teuthology.orchestra.run.smithi107.stderr: 91.1% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T03:42:01.891 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-volume.log: /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mon.b.log: 90.2% 91.1% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T03:42:01.894 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.d.log: 90.2% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T03:42:01.902 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.a.log: 91.1% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T03:42:01.902 INFO:teuthology.orchestra.run.smithi106.stderr: -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T03:42:01.906 INFO:teuthology.orchestra.run.smithi106.stderr: 94.2% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mgr.y.log.gz 2024-03-24T03:42:01.966 INFO:teuthology.orchestra.run.smithi107.stderr: 92.3% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.i.log.gz 2024-03-24T03:42:01.970 INFO:teuthology.orchestra.run.smithi107.stderr: 92.2% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.l.log.gz 2024-03-24T03:42:01.978 INFO:teuthology.orchestra.run.smithi106.stderr: 92.2% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.k.log.gz 2024-03-24T03:42:01.995 INFO:teuthology.orchestra.run.smithi078.stderr: 92.2% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.j.log.gz 2024-03-24T03:42:02.016 INFO:teuthology.orchestra.run.smithi078.stderr: 92.1% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.g.log.gz 2024-03-24T03:42:02.030 INFO:teuthology.orchestra.run.smithi107.stderr: 92.7% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.f.log.gz 2024-03-24T03:42:02.094 INFO:teuthology.orchestra.run.smithi078.stderr: 92.2% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.a.log.gz 2024-03-24T03:42:02.137 INFO:teuthology.orchestra.run.smithi078.stderr: 92.2% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.d.log.gz 2024-03-24T03:42:02.403 INFO:teuthology.orchestra.run.smithi106.stderr: 94.3% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.h.log.gz 2024-03-24T03:42:02.472 INFO:teuthology.orchestra.run.smithi107.stderr: 95.1% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.c.log.gz 2024-03-24T03:42:02.576 INFO:teuthology.orchestra.run.smithi106.stderr: 95.0% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.e.log.gz 2024-03-24T03:42:03.948 INFO:teuthology.orchestra.run.smithi078.stderr: 90.3% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mgr.x.log.gz 2024-03-24T03:42:04.390 INFO:teuthology.orchestra.run.smithi107.stderr: 92.0% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mon.c.log.gz 2024-03-24T03:42:05.812 INFO:teuthology.orchestra.run.smithi106.stderr: 91.9% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mon.b.log.gz 2024-03-24T03:42:07.567 INFO:teuthology.orchestra.run.smithi106.stderr: 93.6% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mds.b.log.gz 2024-03-24T03:42:08.463 INFO:teuthology.orchestra.run.smithi078.stderr: 90.0% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-mon.a.log.gz 2024-03-24T03:42:08.900 INFO:teuthology.orchestra.run.smithi107.stderr: 93.0% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.11.log.gz 2024-03-24T03:42:08.921 INFO:teuthology.orchestra.run.smithi106.stderr: 92.9% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.1.log.gz 2024-03-24T03:42:09.011 INFO:teuthology.orchestra.run.smithi106.stderr: 92.8% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.7.log.gz 2024-03-24T03:42:09.283 INFO:teuthology.orchestra.run.smithi078.stderr: 93.0% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.0.log.gz 2024-03-24T03:42:09.467 INFO:teuthology.orchestra.run.smithi107.stderr: 92.5% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.8.log.gz 2024-03-24T03:42:09.805 INFO:teuthology.orchestra.run.smithi078.stderr: 92.8% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.9.log.gz 2024-03-24T03:42:09.865 INFO:teuthology.orchestra.run.smithi107.stderr: 93.0% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.2.log.gz 2024-03-24T03:42:11.956 INFO:teuthology.orchestra.run.smithi106.stderr: 92.7% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.4.log.gz 2024-03-24T03:42:12.005 INFO:teuthology.orchestra.run.smithi078.stderr: 92.2% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.6.log.gz 2024-03-24T03:42:12.278 INFO:teuthology.orchestra.run.smithi107.stderr: 92.6% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.5.log.gz 2024-03-24T03:42:12.279 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-24T03:42:12.280 INFO:teuthology.orchestra.run.smithi107.stderr:real 0m10.492s 2024-03-24T03:42:12.280 INFO:teuthology.orchestra.run.smithi107.stderr:user 0m35.653s 2024-03-24T03:42:12.280 INFO:teuthology.orchestra.run.smithi107.stderr:sys 0m1.265s 2024-03-24T03:42:12.353 INFO:teuthology.orchestra.run.smithi078.stderr: 92.7% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.3.log.gz 2024-03-24T03:42:12.355 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-03-24T03:42:12.355 INFO:teuthology.orchestra.run.smithi078.stderr:real 0m10.570s 2024-03-24T03:42:12.355 INFO:teuthology.orchestra.run.smithi078.stderr:user 0m43.886s 2024-03-24T03:42:12.355 INFO:teuthology.orchestra.run.smithi078.stderr:sys 0m1.411s 2024-03-24T03:42:12.454 INFO:teuthology.orchestra.run.smithi106.stderr: 92.1% -- replaced with /var/log/ceph/c7e2806a-e972-11ee-95cd-87774f69a715/ceph-osd.10.log.gz 2024-03-24T03:42:12.456 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-03-24T03:42:12.456 INFO:teuthology.orchestra.run.smithi106.stderr:real 0m10.668s 2024-03-24T03:42:12.456 INFO:teuthology.orchestra.run.smithi106.stderr:user 0m44.650s 2024-03-24T03:42:12.456 INFO:teuthology.orchestra.run.smithi106.stderr:sys 0m1.558s 2024-03-24T03:42:12.456 INFO:tasks.cephadm:Archiving logs... 2024-03-24T03:42:12.457 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616430/remote/smithi078/log 2024-03-24T03:42:12.458 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T03:42:18.384 DEBUG:teuthology.misc:Transferring archived files from smithi106:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616430/remote/smithi106/log 2024-03-24T03:42:18.385 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T03:42:24.311 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616430/remote/smithi107/log 2024-03-24T03:42:24.312 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T03:42:29.557 INFO:tasks.cephadm:Removing cluster... 2024-03-24T03:42:29.558 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c7e2806a-e972-11ee-95cd-87774f69a715 --force 2024-03-24T03:42:29.748 INFO:teuthology.orchestra.run.smithi078.stdout:Deleting cluster with fsid: c7e2806a-e972-11ee-95cd-87774f69a715 2024-03-24T03:42:30.062 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c7e2806a-e972-11ee-95cd-87774f69a715 --force 2024-03-24T03:42:30.252 INFO:teuthology.orchestra.run.smithi106.stdout:Deleting cluster with fsid: c7e2806a-e972-11ee-95cd-87774f69a715 2024-03-24T03:42:30.560 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c7e2806a-e972-11ee-95cd-87774f69a715 --force 2024-03-24T03:42:30.753 INFO:teuthology.orchestra.run.smithi107.stdout:Deleting cluster with fsid: c7e2806a-e972-11ee-95cd-87774f69a715 2024-03-24T03:42:31.047 INFO:tasks.cephadm:Removing cephadm ... 2024-03-24T03:42:31.048 DEBUG:teuthology.orchestra.run.smithi078:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T03:42:31.066 DEBUG:teuthology.orchestra.run.smithi106:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T03:42:31.081 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T03:42:31.096 INFO:tasks.cephadm:Teardown complete 2024-03-24T03:42:31.097 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-24T03:42:31.107 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-24T03:42:31.108 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T03:42:31.110 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T03:42:31.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T03:42:31.237 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-24T03:42:31.248 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-24T03:42:31.249 DEBUG:teuthology.orchestra.run.smithi078:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T03:42:31.251 DEBUG:teuthology.orchestra.run.smithi106:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T03:42:31.254 DEBUG:teuthology.orchestra.run.smithi107:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T03:42:31.265 INFO:teuthology.orchestra.run.smithi106.stderr:bash: line 1: ntpq: command not found 2024-03-24T03:42:31.266 INFO:teuthology.orchestra.run.smithi078.stderr:bash: line 1: ntpq: command not found 2024-03-24T03:42:31.267 INFO:teuthology.orchestra.run.smithi107.stderr:bash: line 1: ntpq: command not found 2024-03-24T03:42:31.270 INFO:teuthology.orchestra.run.smithi106.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T03:42:31.270 INFO:teuthology.orchestra.run.smithi106.stdout:=============================================================================== 2024-03-24T03:42:31.270 INFO:teuthology.orchestra.run.smithi106.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 42 +732us[ +732us] +/- 107ms 2024-03-24T03:42:31.270 INFO:teuthology.orchestra.run.smithi106.stdout:^* hv02.front.sepia.ceph.com 3 7 377 44 -669us[ -647us] +/- 58ms 2024-03-24T03:42:31.270 INFO:teuthology.orchestra.run.smithi106.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 42 +312us[ +312us] +/- 63ms 2024-03-24T03:42:31.271 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T03:42:31.272 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T03:42:31.272 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2024-03-24T03:42:31.272 INFO:teuthology.orchestra.run.smithi107.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 108 +764us[ +764us] +/- 106ms 2024-03-24T03:42:31.273 INFO:teuthology.orchestra.run.smithi107.stdout:^* hv02.front.sepia.ceph.com 3 8 377 254 -691us[ -683us] +/- 55ms 2024-03-24T03:42:31.273 INFO:teuthology.orchestra.run.smithi107.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 98 +314us[ +314us] +/- 62ms 2024-03-24T03:42:31.273 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T03:42:31.273 INFO:teuthology.orchestra.run.smithi078.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T03:42:31.273 INFO:teuthology.orchestra.run.smithi078.stdout:=============================================================================== 2024-03-24T03:42:31.273 INFO:teuthology.orchestra.run.smithi078.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 39 +741us[ +741us] +/- 107ms 2024-03-24T03:42:31.273 INFO:teuthology.orchestra.run.smithi078.stdout:^* hv02.front.sepia.ceph.com 3 7 377 42 -677us[ -662us] +/- 58ms 2024-03-24T03:42:31.273 INFO:teuthology.orchestra.run.smithi078.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 105 +308us[ +323us] +/- 62ms 2024-03-24T03:42:31.274 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T03:42:31.274 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-24T03:42:31.284 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-24T03:42:31.285 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-24T03:42:31.298 DEBUG:teuthology.orchestra.run.smithi078:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T03:42:31.414 DEBUG:teuthology.orchestra.run.smithi106:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T03:42:31.498 DEBUG:teuthology.orchestra.run.smithi107:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T03:42:31.581 DEBUG:teuthology.orchestra.run.smithi078:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T03:42:31.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T03:42:31.619 DEBUG:teuthology.orchestra.run.smithi106:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T03:42:31.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T03:42:31.649 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T03:42:31.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T03:42:31.680 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-24T03:42:31.690 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-24T03:42:31.701 INFO:teuthology.task.internal:Duration was 13188.805772 seconds 2024-03-24T03:42:31.701 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-24T03:42:31.713 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-24T03:42:31.713 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T03:42:31.716 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T03:42:31.718 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T03:42:31.758 INFO:teuthology.orchestra.run.smithi106.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T03:42:31.762 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T03:42:31.763 INFO:teuthology.orchestra.run.smithi078.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T03:42:32.183 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-24T03:42:32.183 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi078.front.sepia.ceph.com 2024-03-24T03:42:32.183 DEBUG:teuthology.orchestra.run.smithi078:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T03:42:32.203 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi106.front.sepia.ceph.com 2024-03-24T03:42:32.203 DEBUG:teuthology.orchestra.run.smithi106:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T03:42:32.242 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi107.front.sepia.ceph.com 2024-03-24T03:42:32.242 DEBUG:teuthology.orchestra.run.smithi107:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T03:42:32.262 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-24T03:42:32.262 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T03:42:32.264 DEBUG:teuthology.orchestra.run.smithi106:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T03:42:32.284 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T03:42:32.330 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-24T03:42:32.331 DEBUG:teuthology.orchestra.run.smithi078:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T03:42:32.333 DEBUG:teuthology.orchestra.run.smithi106:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T03:42:32.353 DEBUG:teuthology.orchestra.run.smithi107:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T03:42:33.096 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-24T03:42:33.108 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-24T03:42:33.108 DEBUG:teuthology.orchestra.run.smithi078:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T03:42:33.135 DEBUG:teuthology.orchestra.run.smithi106:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T03:42:33.162 DEBUG:teuthology.orchestra.run.smithi107:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T03:42:33.189 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-24T03:42:33.200 DEBUG:teuthology.orchestra.run.smithi078:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T03:42:33.202 DEBUG:teuthology.orchestra.run.smithi106:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T03:42:33.204 DEBUG:teuthology.orchestra.run.smithi107:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T03:42:33.225 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = core 2024-03-24T03:42:33.227 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern = core 2024-03-24T03:42:33.254 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = core 2024-03-24T03:42:33.284 DEBUG:teuthology.orchestra.run.smithi078:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T03:42:33.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T03:42:33.297 DEBUG:teuthology.orchestra.run.smithi106:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T03:42:33.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T03:42:33.311 DEBUG:teuthology.orchestra.run.smithi107:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T03:42:33.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T03:42:33.338 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-24T03:42:33.349 INFO:teuthology.task.internal:Transferring archived files... 2024-03-24T03:42:33.350 DEBUG:teuthology.misc:Transferring archived files from smithi078:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616430/remote/smithi078 2024-03-24T03:42:33.350 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T03:42:33.398 DEBUG:teuthology.misc:Transferring archived files from smithi106:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616430/remote/smithi106 2024-03-24T03:42:33.398 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T03:42:33.438 DEBUG:teuthology.misc:Transferring archived files from smithi107:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616430/remote/smithi107 2024-03-24T03:42:33.438 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T03:42:33.479 INFO:teuthology.task.internal:Removing archive directory... 2024-03-24T03:42:33.479 DEBUG:teuthology.orchestra.run.smithi078:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T03:42:33.481 DEBUG:teuthology.orchestra.run.smithi106:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T03:42:33.484 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T03:42:33.527 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-24T03:42:33.540 INFO:teuthology.task.internal:Not uploading archives. 2024-03-24T03:42:33.540 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-24T03:42:33.551 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-24T03:42:33.551 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T03:42:33.554 DEBUG:teuthology.orchestra.run.smithi106:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T03:42:33.556 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T03:42:33.566 INFO:teuthology.orchestra.run.smithi078.stdout: 266042 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 03:42 /home/ubuntu/cephtest 2024-03-24T03:42:33.567 INFO:teuthology.orchestra.run.smithi106.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 03:42 /home/ubuntu/cephtest 2024-03-24T03:42:33.581 INFO:teuthology.orchestra.run.smithi107.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 03:42 /home/ubuntu/cephtest 2024-03-24T03:42:33.582 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-24T03:42:33.594 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-24T03:42:33.618 INFO:teuthology.nuke:Checking targets against current locks 2024-03-24T03:42:33.644 DEBUG:teuthology.nuke:shortname: smithi078 2024-03-24T03:42:33.644 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T03:42:33.668 DEBUG:teuthology.nuke:shortname: smithi106 2024-03-24T03:42:33.669 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T03:42:33.680 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616430', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 23:54:08.246705', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNlgIrdQHTUm2YYUyP8OOqEpcJBHk0sJPKor9xkGjd7G77koK5yCB7Xl8n2XEaD96yV9TYEuyXcol4jclHKgm1w='} 2024-03-24T03:42:33.690 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi106.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616430', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 23:54:08.250442', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOU+yW0i5FrJ74ve1qM70CraxjpC6KFblwTr27caooJCq5unNp8xntIF0qN5C6lhlcJ38CkQZEV1HdL7ie/qIUk='} 2024-03-24T03:42:33.694 DEBUG:teuthology.nuke:shortname: smithi107 2024-03-24T03:42:33.694 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T03:42:33.713 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616430', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 23:54:08.248785', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEhgHjr9J4ZU0C3A4afjwXidRGS3mBNzk3UgRevilq1xOPxqzr44vijyI6iJRJBsQr+a5Ie4pRxQLYFT3wjMfNY='} 2024-03-24T03:42:33.724 INFO:teuthology.orchestra.console.smithi078:Power off 2024-03-24T03:42:33.725 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T03:42:33.740 INFO:teuthology.orchestra.console.smithi106:Power off 2024-03-24T03:42:33.740 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T03:42:33.752 DEBUG:teuthology.orchestra.console.smithi078:power off output: Chassis Power Control: Down/Off 2024-03-24T03:42:33.753 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T03:42:33.764 DEBUG:teuthology.orchestra.console.smithi106:power off output: Chassis Power Control: Down/Off 2024-03-24T03:42:33.764 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T03:42:33.779 INFO:teuthology.orchestra.console.smithi107:Power off 2024-03-24T03:42:33.779 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T03:42:33.788 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on 2024-03-24T03:42:33.789 DEBUG:teuthology.orchestra.console.smithi106:check power output: Chassis Power is on 2024-03-24T03:42:33.800 DEBUG:teuthology.orchestra.console.smithi107:power off output: Chassis Power Control: Down/Off 2024-03-24T03:42:33.800 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T03:42:33.821 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-03-24T03:42:37.789 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T03:42:37.798 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T03:42:37.822 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T03:42:37.900 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on 2024-03-24T03:42:37.908 DEBUG:teuthology.orchestra.console.smithi106:check power output: Chassis Power is on 2024-03-24T03:42:37.932 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-03-24T03:42:41.905 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T03:42:41.913 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T03:42:41.932 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T03:42:42.015 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is off 2024-03-24T03:42:42.023 DEBUG:teuthology.orchestra.console.smithi106:check power output: Chassis Power is off 2024-03-24T03:42:42.041 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is off 2024-03-24T03:42:42.116 INFO:teuthology.orchestra.console.smithi078:Power off completed 2024-03-24T03:42:42.123 INFO:teuthology.orchestra.console.smithi106:Power off completed 2024-03-24T03:42:42.142 INFO:teuthology.orchestra.console.smithi107:Power off completed 2024-03-24T03:42:42.248 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} duration: 13188.80577158928 failure_reason: 'Command failed (workunit test suites/dbench.sh) on smithi078 with status 124: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh''' flavor: default owner: mchangir sentry_event: https://sentry.ceph.com/organizations/ceph/?query=11c22a897d5742f3b23baba2cdec83a6 status: fail success: false 2024-03-24T03:42:42.249 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T03:42:42.375 INFO:teuthology.run:FAIL